Towards Better Understanding and Better Generalization of Low-shot Classification in Histology Images with Contrastive Learning
Authors: Jiawei Yang
Hanbo Chen
Jiangpeng Yan
Xiaoyu Chen
Jianhua Yao
Venue: ICLR 2022 Poster (2022)
Keywords: Few shot learning
Histology Image
Knowledge Transferring

Abstract: Few-shot learning is an established topic in natural images for years, but few work is attended to histology images, which is of high clinical value since well-labeled datasets and rare abnormal samples are expensive to collect. Here, we facilitate the study of few-shot learning in histology images by setting up three cross-domain tasks that simulate real clinics problems. To enable label-efficient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build a few-shot system. CL learns useful representations without manual labels, while LA transfers semantic variations of the base dataset in an unsupervised way. These two components fully exploit unlabeled training data and can scale gracefully to other label-hungry problems. In experiments, we find i) models learned by CL generalize better than supervised learning for histology images in unseen classes, and ii) LA brings consistent gains over baselines. Prior studies of self-supervised learning mainly focus on ImageNet-like images, which only present a dominant object in their centers. Recent attention has been paid to images with multi-objects and multi-textures. Histology images are a natural choice for such a study. We show the superiority of CL over supervised learning in terms of generalization for such data and provide our empirical understanding for this observation. The findings in this work could contribute to understanding how the model generalizes in the context of both representation learning and histological image analysis. Code is available.

Published as a conference paper at ICLR 2022
TOWARDS BETTER UNDERSTANDING AND BETTER GENERALIZATION OF FEW-SHOT CLASSIFICATION IN HISTOLOGY IMAGES WITH CONTRASTIVE LEARNING
Jiawei Yang Hanbo Chen Jiangpeng Y., Xiaoyu Chen Jianhua Yao 1,*
Tencent Al Lab, UCC Los Angeles, Tsinghua University, 4 *Xiamen University
ABSTRACT
Few-shot learning is an established topic in natural images for years, but few work is attended to histology images, which is of high clinical value since well-labeled datasets and rare abnormal samples are expensive to collect. Here, we facilitate the study of few-shot learning in histology images by setting up three cross-domain tasks that simulate real clinics problems. To enable label-efficient learning and better generalizability, we propose to incorporate contrastive learning (CL) with latent augmentation (LA) to build few-shot system. CL learns useful represen- tations without manual labels, while LA transfers semantic variations of the base dataset in an unsupervised way. These two components fully exploit unlabeled experi- training data and can scale gracefully to other sabel-hungry problems. ments, we find i) models Iearned by CL generalize better than supervised learning for histology images in unseen classes, and ii) LA brings consistent gains over baselines. Prior studies of self-supervised learning mainly focus on ImageNet-like images, which only present a dominant object in their centers. Recent attention has been paid images with multi-objects and multi-textures (Chen & Li. 2020). Histology images are a natural choice for such study. We show the superiority of CL over supervised learning in terms of generalization for such data and pro- vide our empirical understanding for this observation. The findings in this work could contribute to understanding how the model generalizes in the context of both representation learning and histological image analysis. Code is available at WSI
INTRODUCTION
Histological images provide crucial phenotypical and diagnostic information for disease assessment and prognosis (Srinidhi et al., (020)). Thus, computer-aides histological image classification systems are highly demanded but expensive to build due to the scarcity of well-annotated data. Besides, histological images diversify many aspects, including acquisition protocols, body sites and tissue types. Such heavy domain shifts and variations pose more challenging data-hungry issues. How to train robust models with limited annotated samples becomes the key of general diagnosis systems.
We sort for shw- learning (FSL) to tackle the aforementioned issues. Recent works have demon- strated FSL's success natural images, but it is much unexplored in histological image analysis. Here, we facilitate the study of FSL and generalized FSL (GFSL) for histology images by setting up cross-domain tasks where there exist near-, middle- and out-domain shifts from base class to novel class. In addition, we investigate the impact of homogeneous and heterogeneous shot selection problem, i.e., few-shot samples come from the same whole slide image (WSI) or different ones.
To enable label-efficient learning and better generalizability, learning (CL) with latent augmentation (LA) to build a few-shot system. Concretely, CL learns meaningful encoder during pre-training, while LA inherits knowledge from "unlabeled" base datasets by transferring semantic variations in latent space. These two components fully exploit base dataset by using its legacy: the learned model weights, and the captured latent variations.
propose to incorporate contrastive
*: Equal contribution. 1; Work done during intern Tencent AT Lab. ": Corresponding author.

---

Published as a conference paper at ICLR 2022
Background
Aclpose (ND)
Debrs (DERR)
Lymphooyyes (LY1))
Mucus (MUC))
Muscle (MUSS)
Normal colon mucosa (NDRM)
Cancor assodiated Strame
Coloreciall ader coarcir ooma
Figure 1: Example images from NCT (Kather et al. 2018). Each column contains two samples from the same class (column name). More example images can be found in Appendix A.I
Also to our surprise, we show the generalization gap between state-of-thee of-the--ttt-tttt art CL models and the supervised models larger histological images than that in natural images. Previously studies of CL. mainly focus on iconic natural images where only dominant object occupies image centers, while images like histological ones, where multiple small objects (e.g- cells, nucleus) and various textures (e.g. muscle, mucus) are densely presented, are much unexplored. We also aim to fill the gap of studying CL for non-iconic, multi-object and multi-texture histological images, and empirically explain why the large generalization gap between CL models and supervised ones exists for them.
To summarize, our key findings and contributions are:
We, as one of early works, study FSL in histological data with domain-specinf problems.
We propose a simple label-efficient method for few-shot learning, which incorporates con- trastive learning and latent augmentation to fully exploit training data in an unsupervised way. Extensive experiments confirm their consistent gains and improved generalizability.
We show that, at slight odds with findings in iconic natural images, CL learned models generalize better than the supervised counterparts for histology images by large margin. We analyze and provide our empirical explanations for this observation, which we believe could contribute understanding how model generalizes to novel samples in the context of representation learning and histology image analysis.
2 PRELIMINARIES AND PROBLEM FORMULATION
Whole-Slide Image (WSI). Whole-slide images are digital scans of histology tissue slides col- lected by biopsy or surgery. Given micron-size pixels and centimeter-size slides, WSI is usually of gigapixel size and thus have to be divided into hundreds or thousands of small "patches" for computational analysis. A patch represents the basic unit in patch-level classification problem. As tissues context and staining quality could vary across WSIs, the extracted patches' styles are con- fined to their source WSIs. leading to inter-WSS WSI domain shift. Besides, unlike iconic natural images which only present a dominant object in their centers, histological patches contain multiple small objects and multiple texture-like tissues (see Figure T). making the process different from the major recognition systems which only need to focus on dominant objects.
Few-shot Learning (FSL). Few-shot classification is to learn from large "base" dataset, and afterwards generalize to unseen classes with only limited labeled data. Formally, a base dataset defined as Diase = {(x, y Naase C Xhase x Vouse, where Xpase is sample set and Vbase is their label space. Novel dataset Dnove! = [(x;, y:) ) } Naovel has a disjoint label space, i.e. Vbase Vnoved = 0. where Vnowet is the novel label space. A few-shot learner is trained on Dhase and evaluated a series of meta-tasks sampled from Dnovel- Such task is defined as T = {(S,, 2 1== where
S {(xi, vr) Dnooeet is small training set, called support set, and Q = [x,),N
Xnown!
another small test set, called query set, and is the number of tasks. This formulation termed as N N-way K-shot (Q (Q-query) task, since N classes
sampled from Noovet, each with K labeled samples for training and Q unlabeled samples for testing. Usually, K is less than c.g. K = 1,5, and Q 15. The evaluation stage is often referred as meta-testing stage.
Generalized few-shot learning (GFSL). Unlike in FSL, GFSL samples meta-tasks from joint dataset Dj joint = Drase U D novels with a joint label space Djoint = Vhase U Unnonet- Now, support sets and query sets contain both seen base classes and unseen novel classes.
2

---

Published as a conference paper at ICLR 2022
Prestraining
I.A.
Enference
Augmentation (I.A)
Unlabeled WSI patches
patches
WSI patches predict
f&
feature extractor
Dictionary Construction
Hase Diclionaty s
prosy boad(s)
K-Means
Laarsablee medill
classifier
Fixed model
Contrastive Learning other pricy tasks
Few-shot Classification
Prediction
Figure 2: Overview. With pre-trained feature extractor (a), N-way K-shot classifiers are learnt (b) based on LA (d) to classify WSI patches (c). Given a novel representation z, LA generates its new features from the most likely variation in the base dictionary, so few-shot novel samples can be proliferated in a reasonable way, and the decision boundary could therefore be improved.
METHODS
Consider few-shot classifier f fo o where fo is an embedding function, i.e., feature extractor that maps high-dimensional input image E RR into low-dimensional latent space Rd, and fo a classifier trained on support set S and predicts results for query set Q. Φ and their corresponding parameters. Our method consists of two phases a) pre-training fo base datasets and b) training fe on support sets with latent augmentation during meta-testing stage. Figure shows the overview of our methods. We elaborate them in the followings.
3.1 PRE-TRAINING
Current paradigms in FSL for training So lie in two folds: meta-training, also known episodic training, where base datasets are divided into various episodic N-way K-shot meta-tasks that sim- ulate meta-learning: and ii) standard training, which does standard fully supervised classification pre-training without splitting data. The former one emphasizes the idea of meta-learning for fast adaption (Schmidhuber 1987: Finn et al. 2017). while the latter one attributes the success of FSL to feature reuse (Raghu et al. 2019) or good representations (Chen et al. 2019 Tian et al. 2020 We follow the latter one and believe better learned encoders lead to stronger generalizability.
Fully-supervised pre-training (FSP). Given a base dataset, we jointly train a feature extractor fo and a proxy classifier fo using the standard cross-entropy loss. Once pre-trained, only fo is kept and fixed for downstream tasks. We term the embedding functions learned by FSP as f fFSS
Contrastive-leannig pre-training (CLP). Self-supervised learning methods alleviate the need for data annotation. Here we focus on a contrastive learning method MoCo-v3 (Chen et al. 2021b), which currently holds the state-of-the-art performance. It consists of three components: Teature extractor (backbone) fa. projection head fg and a prediction head fg. Given an unlabeled base training dataset Dinse {x, Nooce the model learns minimize the contrastive loss function w.r.t. unlabeled batch data:
x FCLP (Fg fo(x), fg fg (x)):
66.9" = arg min E
(1)
6.9.4
where LeLP is contrastive loss function; x' two views of the same images obtained by applying random data augmentation and denote the momentum updated copies of é and g. In short, contrastive learning aims to maximize the similarity between positive pairs (two augmented views of same image), while minimizing the similarity between negative pairs (two different im- ages). We leave more detailed descriptions of MoCo-v3 and LCLP to Appendix B Once CLP is done, two auxiliary heads. fg and fy. are removed, while fo kept and fixed, termed as f FCLP
3.2 LATENT AUGMENTATION
The pre-trained feature extractor fo only transfers parts of available knowledge in base datasets by reusing the learned weights. The more transferable knowledge is inherent data representations.
3

---

Published as a conference paper at ICLR 2022
is reasonable to assume that base classes and novel classes share similar modes of variations (Wang jet al. 2018) since they are all cattologyrreattee Such inductive biases allow us to transfer variations from seen tissues styles to unseen ones. Here we propose to transfer the representation variations in a simple unsupervised way. Below, we first introduce latent augmentation and then discuss our motivations and intuitions about it.
Base dictionary and Latent augmentation (LA), Our goal is fully exploit training data. This is done not only by reusing pre- pre-trained model weights fa, but also by transferring possible semantic shifts of clustered representations. Given an unlabeled base dataset, we perform K-Means on the representations z = (x) to obtain C' clusters (Figure (a). red arrows). The base dictionary is constructed as B= = {(c, E where c; is the i-th cluster prototype (i.e., mean representation) and Σ, denotes its intra-cluster covariance matrix. Roughly, B captures how the pre-traimed fe thinks samples from base dataset would vary in latent space conditioned on cluster i, e.g. using a multivariate Gaussian (c., Σ.). Given the base dictionary B, during meta-testing stage, LA uses original representations z to query the most likely variations from B, followed by additive augmentation ă = z + (Figure (b,d). This is done by sampling 6 ~ ~O Σ;-) where i* is selected by finding the maximum cosine similarity between and c,. The classifier fo is then trained on both the original representations 2 and the augmented representations 2 (Figure (c)).
3.2.1 INTUITIONS AND MOTIVATIONS ON LATENT AUGMENTATION
Why transferring variations works. LA aims at transferring the knowledge of variations. Such knowledge brings semantic diversity from base classes to novel classes. For example, tumorous cells mutated from normal cells; when given limited tumorous samples, LA may replicate how normal cells alter under this mutation using the captured variations in base dictionaries. This closely resem- bles how pathologist builds his/her knowledge on unseen phenotypes from seen phenotypes. From the view of undeer-rerresentatie learning (Yin et al., (219)). replicating latent variations encourages under-represented distributions to be closer regular ones. From the view of low-data learning, distribution of few samples is not well calibrated (Yang et al. 2021). so using established distribu- tions in base class calibrate untrustworthy novel class may help. Besides, LA can be seen as a consistency regularization technique. Enforcing the classifiers' predictions to be consistent across different perturbation is known to help in low-data regime (Bachman et al., 2014; Berthelot et al.) 2019 Sohn et al. 2020). In fact, I.A is stronger alternative against data augmentation (DA), as we later show in $4.3 that I.A outperforms DA by a large margin and can cover the role played by DA.
Why linear additive augmentation is meaningful. When well trained, deep networks are hy- pothesized to be good at linearizing deep features (Bengio et al., 2013 Upchurch et al. (017). This gives the rationality behind linearly inter/cxtrapolating features, i.e., using "add" operation to gen- erate new features. Recently, Cheung & Yeung (2020) study the universal label-preserving additive augmentations in latent space that can be used in different data modalities, showing the effectiveness of simple linearly transforming latents.
Why base dictionary construction is warranted in both FSP and CLP. FSP uses classification task as proxy task to learn useful encoders fe. During optimization, features are incentivized to maximize their dot-product similarity with class weights in fe. thus forming meaningful metric space. In the regard of CLP, the contrastive loss, a form of metric- loss, brings similar features closer while spreading dissimilar representations farther, which also results in an informative met- ric space. Thus, feature distance in the representation space of both FSP and CLP is meaningful, justifying the use of unsupervised clustering method to construct base dictionary.
4 EXPERIMENTS
4.1 SETUP
Datasets. Since tissues vary across body sites, we use three public histology datasets from differ- ent body sites to construct three tasks with different degrees of domain shift. They are: NCT-CRC- HE-100K collected from colon site (Kather et al. 2018). LC25000 collected from lung and colon sites (Borkowski et al. 2019), and PAIP19 collected from liver site (Kim et al. [021]; we term them as NCT, LC-25K and PAIP respectively. NCT consists of classes with 100k norevverlap patches in total, each of size 224 224. LC-25K has 5 classes with 5,000 patches in each class; each patch is of size 768 x 768, PAIP is compused of 50 WSIs, each of size about 45k 45k with 3 mask annotated classes. For I. LC-25K, all patches are resized to 224 224, For PAIP, the foreground tissues
4

---

Published as a conference paper at ICLR 2022
cropped into 75k patches of size 224 x 224 with the same pixel resolution NCT and labels are assigned by majority voting. More details are in Appendix [A] When novel and base classes are from different organs, we view the novel classes as out-domain classes. When from the same organ, we consider them as near-domain classes only when the data is collected from the same source. Otherwise, they are considered middle-domain classes due to difference in imaging protocols.
Task i) Near-domain task (to study GFSL). We randomly split NCT by 80%/20% to construct a training set (80k images) and a test set (20k images). Then, we do leave-one-llas--ou-------o class to the training set to construct 9 base datasets and use the test set as joint for evaluation. This results in 9 sub-tasks, each of which has one class regarded as novel class and samples belonging to it are excluded from pre- pre-training datasets.
Task ii Mixture-doman
task (to study FSL). We use the entire training set (80k images) of NCT as D Drase and use I.C-55K as Duvvet for evaluation. Two of five classes in I.C-55K are colon-related momdile-domain novel classes) and the remaining three are lung-related (out-domain novel classes).
Task iii) Out-domain task (to study FSL). Similar to mixture-domain task, we use the NCT train- ing set Doase and PAIP as the novel dataset D novel- Liver tissues from PAIP are different from colon tissues in NCT. We thus regard them as are out-domain novel classes. To study heterogeneous and homogeneous shot selection problem, we use WSI ID information to split PAIP into a support WSI set (15 WSIs with 22.5k images) and a query WSI set (35 WSIs with 52.5k images). During evaluation, support samples and query samples are drawn from the support WSI set and the query WSI set respectively. Heterogeneous strategy selects few-shot samples from different support WSIs, while homogeneous strategy selects them from a single randomly chosen support WSI.
Evaluation. If not otherwise specified, for near-domain task, we evaluate methods in 1000 x (9 sub-tasks) random meta-tasks; for mixture- and out-domain tasks, we evaluate methods in 1000 randomly sampled meta-tasks. All meta-tasks use 15 samples per class as the query set. We report the average Fs-score and 95% confidence interval. To handle the unequal numbers of base classes and novel classes, we follow convention in GZSL (Xian et al. 2018) and GFSL (Shi et al.) 2020) to report their average harmonic mean. More details about evaluation metries are in Appendix A
Implementations. I. Pre-training. We use R8 as the embedding function fa. and follow previous arts in FSL (Tian et al. 2020; Chen et al. 2019) to use lz-normalized features for clustering and downstream meta-tasks II. Latent Augmentation. We use faiss (Johnson al.) 2019). library for clustering, to perform K-means with a fixed seed for reproducibility. The number of prototypes in the base dictionary is 16 (C = 16, discussed in ablation $4.3). each meta-task, each sample is augmented 100 times (including the original one) by LA. More details are in Appendix C
Compared methods. Recent works (Chen et al., 2019 Tian et al. 2020). including a concurrent work for histology image (Shakeri et al.) 2021), show that sophisticated episodic training (meta- training) is no better than standard pre-training. Hence, we summarize methods using standard pre-training 1) NearestCentroid. It computes class centroids from support sets and classifies query samples to their nearest centroids. Related works using such strategy includes |Wang et al.) (2019a), [nell et al. (201)). and[Chen et al. (2020c), to list few; 2) LinearClassifier It trains new fully-connected layer with different loss functions (Chen et al. 2019 Lee et al. 2019) w.r.t. support samples, or directly uses linear models in scikit-learn (Pedregosa et al.) (2011), e.g., Logistic sion (Yang et al. 2021 Tian et al. 2020). For the ease of implementation and consistency, we use NearestCentroid, and two lg-regularized linear classifiers Legisticeegressoon and RidgeClassi- fier, all from scikit-learn APIs (Pedregosa et al. 2011).
4.2 MAIN RESULTS
Fully-supervised fo FFSP v.s. Self-supervised fC Results in Table I show that CLP generalizes better to novel classes than FSP by a large margin. Comparing the best vanilla entries (w/o. LA) using two types of pre-training methods, we observe an average advantage in HarmMean of 4%, 5% and 8% in 1-/5-/10-shot settings by CLP in near-domain task and 10%, 19%, 16% in mixture-domain task. Besides, CLP representations benefit more from the increase of number of shots than FSP's in both tasks, e.g. +17% vs. +11% and + +12% vs. +10% when 1-shot -> 5-shot for linear classifiers in near-domain, and mexture-domain tasks respectively. Despite the inevitable advantage of FSP in base classes under full supervision, CLP demonstrates stronger generalizability to novel classes. Furthermore, Table also confirms the superiority of CLP over FSP in out-domain task where
5

---

Published as a conference paper at ICLR 2022
Table 1: Main results in near//mixture-domaih tasks. In near-domain task, the "'ove" columns report average Fl-scores of the base/novel classes; the "HarmMean" columns report their average harmonic mean, In mixture-domain task, the same metries are reported w.r.t. middle-domal classes and out-domain classes. "i" numbers denote 95% confidence interval across multiple runs. "LA" denotes latent augmentation. More 10-shot results are in Table D.1
I-shot
5 S-shot
10-shot
9-way K shot
Near domain task
Methods
Base
Novel
HarmMean
Novel
HarmMean
HarmMean
Fully-supervseed pre-training (FSP)
NearestCeotroid
77.38=0.96
43.80:+1.12
54,84=1003
88 38.6440.41
57.67=0.80
68.36-20.33
71.00=0 46
Legistickegression
7,.14=1.03
37,8011117
48.84=1.09
88.45=0.40
48.7610.93
59.9910.55
66.3910.45
RidgeClassifir
7..89-1.02
37.55.11.18
48.7511.09
88.4410.40
45.73.10.97
56.96.10.57
60.33.10.45
I Logistic Regression + L.A (ours)
78.88-0.94
43.424+.14
54.83+1.022
90.85+0.366
63,54+0074
73.63-00.48
78.14+00.39
RidgeClassifeer + LA (ours)
76.19=1.03
40.71+1.16
51.95=1.07
88.86:+0.41
53,90=20.90
64.8 04.87:00.55
66.96=0046
Contrastive leaariiivieeaiiinngggg pre-training (CLP)
NearestC cutroiid
71.45=0.95
51.95+11003
58.81=0.98
03,11=0.52
65,36 80
72.51=0.62
75.18:20.05
Reggesscnn
70.83=:1.01
48.76:11.12
56.13:1.066
84.0410.50
62.69:10887
70.89:10.22
76.83:1:0511
RidgeClussifier
71.24:1.99
49.18.11122
$6.56 1.05
85.89 10.46
66. 66.12.10.83
13.53
79.45 1:0.4
LogisticRegrent Regression + LAA (ours)
72.11+0.95
53.154-1008
59.82-11011
86.43-40.66
76.68+00.61
80.67-00551
85,48--0.44
RidgeClassifier+ LA (ours)
72.60=0.99
54.50::.111
60.89=1004
08.18
78.00=0060
0.52
86.17:00440
5-way-K-shot
Mixture-domain task
Methods
Middle
Out
HarmMean
Middle
Out
HarmMean
HarmMean
Fully-supervised pre-training (FSP)
Nearest
45.65.1.27
54.94
49.8711.24
49.01
61.28.0078
$4.56.10.90
55.75
RogisticRegresseee
40.07+1.33
48.004-1
13.68+1.39
49.42+1.02
54.18+1.044
51 31.69-41.00
56.12-0999
RidgeClassifier
11.46=1.36
48,74 18.74+1.43
44.81=1.39
58.28=0098
56. 12:11.005
55,70=1.001
60.77=00 88
Logistic Regression + LA (ours)
16.98=1.33
53.34+1.30
49.95=1.31
65.51==0.1
62.64:00877
6..04:10.44
67.60:0073
RidgeClasifier LA (ours)
47.70
52.1311.35
1.38221.36
67.45.1..8
60.97:10.95
64.04.1..66
67.23
Contrastive Learariivee-lamiingg pre-truining (CLP)
NeurestCentrui
71.42.1.14
52.01.11.05
60.19.11009
84.50.10.49
65.31
73.68.1.58
76.30.10.49
I Logistic Regressin
72.16-11066
51.14+0.97
59.86-1100 1.01
$3.91-40.49
61.98+0.71
71.29++0.58
74,89+00.4
RidgeClassifier
72.57=1.04
51.1340.96
59.99=1.000
85.22++0.43
62.47-0.72
72.09=00.54
75.84+0 46
Logistic Regression + LA (ours)
71.77=1.09
52.734+1.03
60.79=1.06
07.51
72.92-0.65
09.55=10448
84.95:10.41
RidgeClassfiier + LA (ours)
71.8611.08
52.92.11.04
60.95-100
88.55 1..38
74.04
80.64 0..48
86.32.00.9 0..39
Table 2: Ablations on covariance type. See text for more details.
Table 3: Results in out-domain tasks. Average Fl-scores from 1000 meta-tasks are reported.
Cov Type
Base
Novel
HMean
RldgeCCnsstfier
Homogeneoms
Heterogencoos
None
85.8520.78
53.2721.63
65.7411.06
3-way K-shot
ISP
CLP
CLP+LA
FSP
CLP
CLP-LA
Tied
79.35:1.08
65.32::1.21
71.6521.14
K
36.90
42.56
43.14
Ding
85.9110.88
6..66.11.42
7..46.1.008
39.00
48.91
49.83
Spberical
85.7810.87
62,0021.39
71.9721.07
K = 10
40.26
50.57
$1.62
45.91
55.96
5M.35
K
41.53
51.76
53.71
50.54
61.88
65.38
Full (default)
87,5120.80
65,7911.36
75.11:1.01
52.74
larger domain shift exists. Such generalization gap between FSP and CLP in histology images is slight odds with observations in natural images, where they show similar generalizability. We study and discuss it at $4.4 We also provide the linear evaluation results of all the 20 pre-trained models in Appendix C.3 for a reference to see how each model perform in NCT dataset.
Latent augmentation brings consistent improvement. Regardless of pre-training methods, LA brings consistent gains over baseline linear classifiers, confirming its effectiveness. With base dic- tionaries, limited few-shot samples are able to proliferate in a reasonable way by transferring latent variations. Such boost maintains its significance from near-domain task to mixture-domean
task (Ta- b but turns smaller in out-domain task (Table This is in expectation since the threc classes (non-tumor, viable-tumor and other) defined in PAIP are extremely coarse goarse-grained: it may include couples of real fine-grained classes (c.f. Figure A.4 in Appendix). Few samples are unable to well represent their entangled semantics. Hence, this observation does not repudiate the effectiveness of latent augmentation but re-ensures its tenability.
4.3 ABLATIONS
To ablate design choices, we exclude two cancer-related classes, i.e., cancer- stroma (STR) and colorectal adenocarcinoma epithelium (TUM), from NCT to be novel classes, and use the rest as base classes. If not otherwise specified, all ablations are conducted on CLP models with Ridge- Classifier for 300 meta-tasks in 5-ohot setting.

---

Published as a conference paper at ICLR 2022
IIN-LA
AA Base
Nowe-AA
A LA
LAVOA
DM
Nove
TAw lipiets
LAW? pow
Average 1-ccores
of F1 F1l-score (99)
86
Ave Tvergge Harmoni
12
Aparage sconns
LA superaied dict.
70
78
55
128 256
6
8"
10
12
16
Novel
HMlton
Number of prototypas
Number of augmentation times
of namber of prototypes
(b) Efffet mumber augmertation times
(c) Fffect using labels calibration
Figure 3: Ablations latent augmentation. (a) The effect of different number of prototypes. Dash lines are the baselines for the solid lines of same colors. (b) The effect of number of augmentation times. The harmonic mean are plotted. "LAx ILAxDA DA" denotes T latent augmentations are applied after T' traditional data augmentations (leads to 2 times in total). (c) The effect of using labels and calibration. "DC": Distribution Calibration. "calib.": calibration; we introduce it in Appendix DD3
Number of prototypes in base dictionary. Figure 3 (a) shows how performance varies with the number of prototypes C. We observe the similar tendency between base class and novel class, where their harmonic means peak at C = 16; we subsequently choose C' = 16 for all experiments. Besides, the performance of base classes and novel classes shows opposite trends from C 4 C = 16. The trade-off off exists here that as the granularity of clusters increases (C 1), the intra-cluster variance decreases, which results in better grouping accuracy but brings less semantic variation. The novel classes benefit from larger variation while the base classes benefit. from more accurately estimated variation since they have been exposed in training. Nevertheless, LA demonstrates its robustness by consistent improvement over baselines (solid vs. dashed lines of same color in Fig. (a)).
DA vs. LA, and number of augmentation times. Here we compare LA with data augmenta- tion (DA), and their combination. DA's details are in Appendix D.2 Figure (b) shows that LA outperforms DA by large margin. The boost brought by DA saturates easily and keeps dropping thereafter, while LA keeps improving with all tested cases. Besides, DA can marginally improve LA (LAxDA v.s. LA). We conclude that LA has already covered the role played by DA in an implicit way since the most of gains are brought by LA. It is worthy to emphasize the computation budget involved in LA (addition in I& space) significantly lower than DA (image augmentation in R3HW space and encoder forwards). Therefore, we run all experiments only with LA.
Using label information. LA constructs the base dictionary without any label information, e.g. the number of classes. When label is available, similar methods such as Distribution Calibration (DC) (Yang et al. (021) can be used. Figure (c) shows the comparisons of using labels and calibration (introduced in Appendix [D3).. Under supervision, "DC" and "[A+ssuervised diet." achieve competitive performance. Surprisingly, once given the number of base classes, "LA w/ 7-proto" can attain better results than using 16-prototype and be comparable as supervised DC. Calibration could further improve LA. This implies that, with LA, knowing the number of base classes can be sufficient for gaining as descent results as knowing all examples' labels.
Covariance type. Here we explore more types of covariances that LA can use. Specifically, we also include: "Tied", where all clusters share a covariance matrix estimated from the entire base dataset, 2) "Diag", where each cluster has its own diagonal covariance matrix, i.e., diagonal elements are a variance vector and non-diagonal elements are zeros, 3) "Spherical", where each cluster has its own single ecalar-variane shared by all feature dimensions. Table showw the results. LA with all types of covariances improves performance. This emphasizes the importance of diversifying few samples with variation. Using full covariance estimation achieves the best performance. We further show how different covariance types perform with smaller or larger cluster sizes in Appendix D.4
Heterogeneous v.s. Homogeneous patch selection. We investigate the hetero-/homo-genne patch selection strategies defined in out-domain task ($4.I). Table[3]shows the results. We observe: i) heterogeneous selection shows higher baselines than homogeneous one; and ii) LA brings more gains for heterogeneous selection. Heterogeneous patches provide reliable and diverse "anchor" samples than homogeneous ones, thus can benefit more from bootstrapping the basc dictionary.
Additional ablation studies. We conduct two more ablation studies: (a) clustering with differ- ent random seeds in K-Means, (b) studying the effect of l2 n-mommalzzttion to covariance. We show that LA's improvement is stable under different random seeds used in K-Means (Appendix D.5).
7

---

Published as a conference paper at ICLR 2022
CLP
FSP
Novel Class
Abs. Sim. Rel. Sim.
k=2
k=4
k=6
Abs. Sim.
Ral Sim.
k=2
k=4
k=6
Low
Middle
High
Camiarity
Chudaan
Base Class
High
Figure 4: Visualization of samples learned by CLP and FSP. "Abs./Rel. Sim." columns show the absolute/relative cosine similarity between the global feature and the local ones. Relative similarity the min-max normalized absolute similarity. "Low", "middle" and "high" denote using features from stage-3, 4, and 5 from a ResNet. See text in $4.4 for discussion. Visualization procedures and more examples are provided Appendix D.8
indicate the cluster numbers used by K-Means.
and covariances estimated before and after Iz-normalization are highly correlated, and further 12- normalizing augmented samples marginally degenerates LA's performance (Appendix D.6). A.p). In ad- dition, we reproduce ô-encoder (Schwartz al. 2018) for ablation task. However, we observe performance drops from baseline (e.g., -2.5% HMean), and none of the tested cases can outperform our method, More results and discussions are in Appendix
4.4 MORE DISCUSSION
Disparity between FCLP and f FFSP influences the choice of base learner. In Table we find i) stronger baselines for CLP and FSP vary, and ii) simple NearestCentroid can sometimes outper- form the vanilla (2 regularized linear classifiers for FSP. Here we briefly discuss our understandings. Representations produced by CLP can have different distributions compared to FSP, as also no- ticed by He et al. (2020). With limited training samples, different classifiers have their biases in building decision boundary, leading to different generalizability. Besides, no regulariza- tion techniques DropBlock (Lee et al., 2019 Tian et al.] 2020) or "distill" regularization (Tian et al.] 2020). The lin- classifiers, though with la penalty, may still be overfitted in such representation space when only limited samples are provided. Subsequently, the simplest NearestCentrood model can yield better results than these overfitted linear models, as it has the least complexity.
used during FSP (Chen et al. (020a), e.g., weight decay (Krogh & Hertz 1992),
Why do CLP models generalize better than FSP ones in histology images? To study why the large generalization gap exists, in Figure 4 we follow Chen & Li (2020) to see how features aggregate in space. Specifically, we visualize the cosine similarity between feature map (a set of local representations) and its global averaging (global representation), and run K-Means on the feature maps from different layers (i.e., stage 3, 4, 5 of ResNet) with different cluster numbers. We observe: the FSP model maintains high global-local similarity low-/middlle-evelss, while the CLP model holds it in high-level (solid boxes). Besides, the CLP model extracts low-level features that are edge-related, and afterwards successively agglomerates adjacent similar structures (dashed boxes). In contrast, the FSP model can differentiate nuclei in www-&middle-levee but fails to encode structure-related features a deeper layer.
We further visualize some samples from base classes (bottom of Figure and D.6p. and find that such disparity between FSP and CLP does not only exist in a previously unseen class but also in seen classes. In the bottom row of Figure 4 FSP only pays attention to the most discriminative parts, leaving the rest "redundant parts" disorder (dashed box). However, the discriminative parts are likely to alter when new class is presented. ESP's inability to fully encode meaningful information may lead to its failure in generalizing to new classes. Meanwhile, CLP encodes most of tissue- structure-related features that may be useful for novel class recognition, possibly resulting in better generalizability. However, FSP models and CLP models are shown to perform similarly. instead of differently, under the same visualization procedure in ImageNet dataset (see the (website| for a comparison). What might cause this disparity? ImageNet has more diverse classes (1000 classes) and samples (~ 1.28M images) than those in histology datasets. ESP models in ImageNet need to recognize the discriminative parts of all 1000 classes. In such case, the redundant information
8

---

Published as a conference paper at ICLR 2022
in class might contribute to the recognition of another class. Therefore FSP may eventually encode most of available information for new classes that are related to ImageNet classes. However, histology datasets usually lack enough diverse annotated classes that help to build know-everything FSP model. A topic we leave for future work is to explore whether CLP always generalizes better than FSP when pre-training on base dataset with limited number of annotated classes and if the generalization gap would increase as the label diversity decreases. We point out that the visualization results and the large generalization gap shown in our work still remains as empirical observations. Our discussion is about what the reasons behind them could be. We hope our work would be helpful for representation learning, histology image analysis, and beyond.
RELATED WORK
Few-shot learning (FSL). FSL. has been tackled from different perspectives, egg. metric-based and optimization-based (Finn et al. 2017 Rusu et al.) 2018). This paper follows a "pre-training & fine-tuning" scheme in metric-based branch, where previous works typically learn a shared metric space by standard fully-supervised pre- pre-training (TTaa et al. 2020 Chen et al.) 2019; Wang et al.) 2018). In contrast, we propose to incorporate self self-supervised pre- wre-training to enable label-efficient learning, and show that it can yield stronger generalization than supervised pre-training.
FSL in medical images. FSL in medical images is at its early stage, especially for histology im- ages. Mahajan et al.) (2020) investigate FSL methods in skin-disease classification, while Chen et al. (221a) tackle COVID-19 CT image classification using contrastive pre-training and prototypical network fine-tuning. In the regard of histology image, MMedela al. (2019) use triplet loss (Schroff) et al. 2015) to pre-train an encoder with followed fine-tuned SVM classifier for few-shot domain adaptation. Sikaroudi et al. ( and Tch & Taylor (2020) study learning with less data in histol- ogy images. Concurrent our work, (Shaeeii et al. (2021) simultaneously propose a benchmark for few-shot classification of histological images. Our work has similar but different settings, with more investigations condueted, e.g., GFSL task, and cetero-/homo-geneouu few shots selection.
Self-supervised learning. Self-supervised learning aims to learn good representations without true labels. Recent state-of-the-arr variants can be categorized as contrastive based learning (Chen et al. 20221b 2020a] He et al.) 2020). cluster- based learning (Caron et al., 2018; (020) and expec- tation prediction based learning (Grill et al., 2020; Chen & Hel (021). However, this line of works focus on pre-training on ImageNet-like images, and recent attention has been attracted to images with multi-objects and multi-texture presented (Chen & Li] (020). We see histology image as a natural choice for such study, and show that contrastive learning can agglomerate structural "part- whole" information and maintain "global-local consistency", which make it generalize better for such data than supervised counterparts (see $4.4 and Table C.1 in Appendix C
Representation variation augmentation. The idea of exploiting feature variations dates back to a decade ago (Heller et al. 2009) Salakhutdinnv et al. 2012). Recent variants further develop this idea. For example. Hariharan & Girshick (2017) and |Schwartz et al.] (2018) use a generator to generate "hallucinated" novel features from variation of given base samples. This method is later extended by not relying on given base samples (Wang al.] 20188) |Wang et al. (2019b) use class variance to perform semantic augmentation for classification and segmentation (Wang et al. 2021), 1, while Yin et al. 2019) and Liu et al. (2020) utilize intra-class variance of head classes augment tail classes for 'longtail" face recogmition problem. Yang et al. (202I) use the distribution information, i.c. mean and variance, of base classes to calibrate novels' distribution. Cheung & Yeung 20200) propose a novel and systematical framework to apply automated augmentation with more considerations in label-preserving transformations. This work follows the line of Yang al. (202I): Wang et al. 2001); Liu et al. (2020), but in contrast to them, we obtain and transfer variations without relying on any label information, allowing our method scale gracefully to other label-hungry problems.
CONCLUSION
This work has studied, as an early attempt, the few-shot learning problem for histology images. We incorporate contrastive learning and latent augmentation to fully exploit training data in an unsuper- vised way, which means our method can gracefully scale to other large label-hungry problems. More importantly, we show that the generalization gap between the state-of-the-art contrastive learning pre- pre-training method and supervised pre-training in histological images is larger than that in Ima- geNet experiments. We analyze the underlying reasons and provide our empirical understandings. We hope our work could contribute to the study of representation learning and generalization for both self-supervised learning community well as histology image analysis community.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural information processing systems, 77:3365-3373 2014.
Yoshua Bengio. Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep represen-
tations. In International conference on machine learning, pp. 552-560. PMLR, 2013.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmenta- tion anchoring. arXiv preprint arXiv: 101.9...097.... 2019.
Andrew A. Borkowski, Marilyn M. Bui, L. Brannon Thomas, Catherine P. Wilson, Lauren A. De- Land, and Stephen M. Mastorides. Lc25000 lung and colon histopathological image dataset. 2019. http://ss/ggiibb.oo//aappppppppp
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.
Ting Chen and Lala Li. Intriguing properties of contrastive losses. arXiv 2020.
arXiv:201 22000
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pp. 1597-1607. PMLR, 2020a.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira. Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2019.
Xiaocong Chen, Lina Yao, Tao Zhou, Jinming Dong, and Yu Zhang. Momentum contrastive learning for few-shot covid-19 diagnosis from chest ct images. Pattern recognition, 113:107826, 2021a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04.977 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057 2021b.
Yinbo Chen, Xiaolong Wang. Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning, arXiv preprint arXiv:2003.04390 2020c.
Tsz-Him Cheung and Dit-Yan Yeung. Modals: Modality-annostic automated data augmentation in the latent space. In International Conference on Learning Representations, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learming for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135 PMLR, 2017.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.0r7 07733, 2020.
10

---

Published as a conference paper at ICLR 2022
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3018- 3027, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 9729-9738, 2020.
Katherine A Heller, Adam Sanborn, and Nick Chater. Hierarchical learning of dimensional biases in human categorization. In NIPS, pp. 727-735. Citeseer, 2009.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.
Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue, April 2018. htLL:/tpss///o..orr//0055//////2 1214ee4
Yoo Jung Kim, Hyungjoon Jang, Kyoungbun Lee, Seongkeun Park, Sung-Gyu Min, Choyeon Hong. Jeong Hwan Park, Kanggeun Lee, Jisoo Kim, Wonjae Hong, et al. Paip 2019: Liver cancer segmentation challenge. Medical Image Analysis, 67:101854, 2021.
Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
K Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2970-2979, 2020.
Kushagra Mahajan, Monika Sharma, and Lovekesh Vig. Meta-dermndaggooss: few-shot skin disease identification using meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 730-731, 2020.
Alfonso Medela, Artzai Picon, Cristina L. Saratxaga, Oihana Belar, Virginia Cabezón, Riccardo Cicchi, Roberto Bilbao, and Ben Glover. Few shot learning in histopathological images: reducing the need of labeled data on biological datasets. 2019 JEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1860-1864. IEEE, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 1877v:1807.03748, 2018.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-28300 2011.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint 0rXiv:1909.09153 2019.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 2018
Ruslan Salakhutdinov, Joshua Tenenbaum, and Antonio Torralba. One-shot learning with hierar- chical nonparametric bayesian model. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pp. 195-206. JMLR Workshop and Conference Proceedings, 2012.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-mela-.. hook. PhD thesis, Technische Universität München, 1987.

---

Published as a conference paper at ICLR 2022
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 815-
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, and Alex Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. Advances in Neural Information Processing Systems, 31, 2018.
Fereshteh Shakeri, Malik Boudiaf, Sina Mohammadi, Ivaxi Sheth, Mohammad Havaei, Ismail Ben Ayed, and Samira Ebrahimi Kahou, Fhist: A benchmark for few-shot classification of histological images. 2021.
Xiahan Shi, Leonard Salewski, Martin Schiegg, and Max Welling. Relational generalized few- shot learning. In 31st British Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020.
Milad Sikaroudi, Amir Safarpoor, Benyamin Ghojogh, Sobhan Shafiei, Mark Crowley, and Hamid R Tizhoosh. Supervision and source domain impact on representation learning: A histopathology case study. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). pp- 1400-1403, IEEE, 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint 100.:v:1703.05100 2017.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying supervupe learning with consistency and confidence. arXiv preprint arXiv:2001.07653 2020.
Chetan L Srinidhi, Ozan Ciga, and Anne L Martel. Deep neural network models for computational histopathology: A survey. Medical Image Analysis, pp. 101813, 2020.
Eu Wern Tch and Graham W Taylor. Learning with less data via weakly labeled patch classification in digital pathology. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 471-47555 IEEE, 2020.
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 266-282 Springer, 2020.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064-7073, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04622 2019a.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recogni- fion, pp.
2018.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. Advances in Neural Information Processing Systems, 32: 12635-12644 2019b.
Yulin Wang, Gao Huang, Shiji Song, Xuran Pan, Yitong Xia, and Cheng Wu. Regularizing deep networks with semantic data augmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
12

---

Published as a conference paper at ICLR 2022
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning- comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on paltern analysis 2018.
and machine intelligence. 41(9):2251-228
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learn- ing for face recognition with under-represented data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5704-5713, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, arXiv preprint arXiv:2/03.032:00, 2021.
13

---

Published as a conference paper at ICLR 2022
A DATASET
A.1 NCT
Data. NCT-CRC-HE-IOOK dataset (Kather al. 2018) contains 100, 000 non-overlapping patches extracted from hematoxylin and cosin (H&E) stained human colorectal cancer and nor- mal tissues. Each image is of size 224 224 0.5 MPP (20x magnification). Tissue classes and their label index are: 0) background (BACK), 1) adipose (ADI), 2) debris (DEB), 3) lymphocytes (LYM), 4) mucus (MUC), 5) smooth muscle (MUS), 6) normal colon mucosa (NORM), 7) cancer- associated stroma (STR), 8) colorectal adenocarcinoma epithelium (TUM). Figure A.. shows the class distribution and Figure(A.) J.s]sos
shows examples per class from NCT. These images are from Na- tional Center for Tumor Diseases (Heidelberg. Germany) and University Medical Center Mannheim (Mannheim, Germany). Their acquisition protocols differ across organizations, which lead to inter- souree domain shift.
Processing. We use the "NCT-CRCC-I--IOO--NNNOON dataset, which does not apply color normalization images. We randomly split NCT dataset (100k images) into a training set (80k images) and a test set (20k images). The class distribution is the same for the training set and test set, i.e. data are sampled w.r.t. each class. Since each image in NCT is of size 224 224, we do not further resize it.
Ncar-domain task. NCT dataset contains tissue patches from multiple sources, which bears slight domain shift since the staining intensity varies. To better study the generalizability in FSL problem, we do caae--oe--lass-oute tatasss-uut-s--oooe novell--ovel--lass to the training set of NCT. This procedure closely resem- bles the leave-one-out cross-validation. Since histology datasets usually have far less number of classes compared to natural image dataset (e.g. classes in NCT v.s. 100 classes in manelmageNet and 608 classes in fieredlmageNet), simulate the generalization error bound when number of classes small. After the leave-one-class novell-novel-llas split, 9 sub-base datasets are constructed, each of which has one class regarded as the novel class and samples belonging to it are excluded from pre-training. For each sub-base dataset, a CLP model and a FSP model will be trained on Therefore, total of 18 models will be obtained (9 CLP models and 9 FSP models). All models are evaluated on the NCT test set. It contains both seen classes and unseen novel classes, so as to simulate generalizes few-shot classifi- cation.
believe such
better
Evaluation metric. Typical FSL methods use accuracy as metrics. However, to study GFSL prob- lem, the performance of novel classes and base classes should be appropriately considered. Accuracy joint label space. Hence, we choose F1 Pl-score as our
w.r.t. specific classes cannot be computed metric. Besides, the numbers of novel classes and base classes unequal in each sub-task, follow convention in GFSL to compute their harmonic mean. The final metrics are
Mbase (i)
A()). $E(0..9911)
Σ
(A.1)
Mnovel (1) = Fi)),
(A.2)
2
H MMammi) 1/Minc (i) + /Mooet
(A.3)
Base 1
1
Mbase (i), Novel
M Mnonet (i), HarmMean
MMean(i), (A.4)
where Fj) denotes the Fl-score of j-th class. Muuse (1) (Equation A.1 computes the average F1-score of base classes in i-th sub-task Equation A.2 and Equation A.3 compute the Fl-score of novel class and the harmonic mean of Moase and Mnowet in i-th sub-task sub-taskrespeetivee. Equation A.4 shows the final metrics report in this work.
14

---

Published as a conference paper at ICLR 2022
BACK
ADI
DEB
LYM
MUC
MUS
NORM
STR
TUM
Figure A.I: Example images from NCT dataset. Each column represents a different class.
12002
10002
0003
4002
Figure A.2: Class distribution in NCT dataset. Labels denote: 0-ADI, 1-BACK, 2-DEB, 3-LYM, 4-MUC, 5-MUS, 6-NORM, 7-STR and 8-TUM. See Appendix A full names of labels.
A.2 LC25000
Data. The LC25000 (LC-25K) dataset contains 25,000 color images from 5 classes. They are: 0) colon ademocarcinoma (Colon ACA), 1) benign colonic tissue (Colon (Colon.benign), 2) lung adenocar- cinoma Lung_ACA), 3) benign lung tissue (Lung_ benign), and 4) lung squamous cell carcinoma LLung_SCC) The class distribution is balanced, i.e. each class has 5000 images. All images are of size 768 768. LC25K constructed by augmenting 1250 images (250 images for each class). The augmentations are: left and right rotations (within 25 degrees, p=1.0) and vertical and horizontal flips (p=0.5), where "p" represents the probability.
Processing. We resize all images from 768 768 to 224 224. No other process is taken.
Mixture-domain task. LC-25K dataset includes tissues from colon and lung sites. Two of five classes in LC-25K are colonic tissues, which suffers moderate domain shift due to difference in imaging protocol and pixel resolutions. Thus, we see them as middle-domain novel classes. The rest three dunatrelated classes organ. To study few-shot classification, we use the entire training set (80k images) NCT as Doases and use LC-25K as Dnovel for evaluation.
regarded out-domain novel classes since they from different
Evaluation metric. Similar near-domain task evaluation in Appendix A. Middle ] (0) + (1)) Out = 3 (2) + Fi (3) + (4)), HarmMean 1/Midde+1//O Here (i) denotes the Fl-score of i-th class. Class 0 and class 1 are colon-related, termed as middle-domain novel classes, and the rest three classes are lung-related, termed as out-domain novel classes.
compute
15

---

Published as a conference paper at ICLR 2022
Colon AGA
Colon_benign
Lung_ACA
benign
Lung_SC
Figure A.3: Example images from LCC5000 dataset. Each column represents a different class. The classes are: colon adenocarcinoma (Colon_ACA), benign colonic tissue (Colon_benign) lung adenocarcinoma (Lung_ACA), benign lung tissue (Lung. Lung_benign), and lung squamous cell carcinoma (Lung_SCC).
A.3 PAIP19
Data. The PAIP 2019 (Kim et al. 2021 is composed of 50 H&E stained WSIs at 20 magnifica- tion. Each WSI approximately of size 45, 000 x 45, 000 with XML annotation. contains three classes, which are: 0) non-tumor liver tissue, 1) viable tumor, and 2) other tissues in whole tumor area but not viable tumor. The class 2 may include intratumoral hemorrhage, necrosis or non-tumor tissue in whole tumor area. Figure A.4 shows example WSI of PAIP19. This dataset provides "case number" (WSI identifier) information.
Processing. We use mask information to extract 500 randomly selected non-overlapping patches 20x magnification for each class from each WSI, leading dataset with 75, 000 samples. We define patches from 15 randomly selected WSIs as the support WSI set and patches from the rest 35 WSIs query WSI set.
WSI
Non- Tum-r
Viable-Tumor
Other
Figure A.4: Examples from PAIP 2019 dataset. The Leftmost image shows the overview of a whole- slide image, where tissues outside yellow boundary are defined as "non-tumor" class, and tissues inside green boundary are "viable-tumor" class, and tissues inside yellow but outside green are "other" class. Right columns show examples of extracted patches from PAIP19 dataset. The "non- tumor" class and "other" class are coarse-defined, since they include many real fine-grained classes such as hepatocyte, vein, artery, stroma, fibrosis, fatty infiltration, etc.
B CONTRASTIVE LEARNING PRE-TRAINING: MoCo-v3
MoCo-v3 (Chen et al. 2021b) aims for an empirical study for vision transformer pre-training, but has ResNet variant. We adopt the ResNet variant.
16

---

Published as a conference paper at ICLR 2022
Overview. MoCo-v3 follows its ancestors MoCo v12 (He al., 2020 Chen al. 2020b) with straightforward modification. As shown in Figure B. each batch is augmented twice under stochas- tic augmentation t obtain two views, denoted 21, t(x), where represents the im- ages. They are then encoded by query encoder and key encoder respectively. The query encoder composed of three components: backbone fá- projector (projection head) fg and predictor (prediction head) fq. formulated as o The key encoder, also known as momentum encoder, consists of momentum copies of backbone and projector, formulated as fa fz.
Backbone
Projector
Predictor
View
Image
Momentum update
View
Backbone
Projector
L
ctr
fg
Query encoder. J ffa
Key encoder. ffg
Figure B.1: Abstraction MoCo-v3 structure. Each image is augmented twice under stochastic augmentation / to obtain two views (denoted by blue and red color). The key encoder (bottom, Ig o J momentum updated by parts query encoder (top, Ja o Sg fa). The "symbol denotes momentum updated parameters.
Momentum update.
In iteration k the momentum update rule is: gs mğj (1 m)gk.
d -mx + (1 m)
(A.5)
where m is the momentum.
Loss function. The contrastive loss function is form of InfoNCE (Ordd et al. 2018)
exp (u u+/r)
Cetr(u, v) - log exp (u ut /r) + -.- exp (u-o=/T)
(A.6)
Here, a denotes the positive sample, i.e. the other augmented view of u. and w denotes the neg- ative samples, i.e. other representations in the batch. T is hyper-parameter, known as temperature
(Wu et al. 2018). The final loss is a symmetric sum: LCLP = (21, ē2) + Ler (22, Z /2, where the subscript indicates view source, 12 lif o fg o fo(x)), la fo o fz(z) and lz|-] means l2 normalization.
C IMPLEMENTATION DETAILS
C.1 FULLY-SUPERVISED PRE- TRAINING
Optimization. We use SGD optimizer with Ir 0.5, momentum 0.9, and weight decay used, i.e. weight decay=0. The batch size is 512. We train for 100 epochs with "step decay" learning schedule. The Ir is multiplied by 0.1 at 30, 60 and 90 epochs respectively.
Data augmentation. We use RandomRes zet.ommeeszedrropop. lowed by normalization (subtract mean and divide st using ImageNet (Deng et al. (0099) statis- tics, i.e. mean (0.485, 0.456, 0.406) and std = (0.229, 0.224, 0.225). During testing, we only resize the image to 224 224, followed by ImageNet normalization. Following previous work (Chen et al. 2019 Tian et al. 2020), we further lg-normalize features.
CandomHorizontalli
fol-
C.2 CONTRASTIVE-EEARNII PRE-TRAINING
The detailed description of MoCo-v3 is in Appendix B
17

---

Published as a conference paper at ICLR 2022
The linear evaluation accuracy of FSP and CLP models. FSP means fully-supervised pre- training. while CLP denotes contrastivellerning pre-training. We train LogisticRegression models on whole training features, and report the accuracy for whole test features. Base-Base means the novel class is excluded when we train and test linear classifiers. Joint-Joint means the linear clas- sifiers are trained on all training features (80k samples) and then evaluated on test features (20k samples). The "Average" row reports the average accuracy of 9 sub-tasks (the top 9 rows).
Table C.I:
Base Base
Joint-Joint
Novel Class
FSP
CLP
FSP
CLP
98.46
95.81
97.73
95.98
97.80
95.79
97.41
95.72
98.59
96.28
90.60
95.10
96.67
95.85
93.92
95.88
95.50
96.49
92.66
95.67
97.60
97.39
92.43
94.33
98.72
96.09
94.75
95.22
7
98.49
97.68
92.66
95.52
8
97.44
95.75
92.63
94.80
Average
97.80
96.35
93.87
95.36
No Novel Class
98.43
96.00
We the codebase of OpenSelfSur[] For projector, we use 3-layer Nont-inearNeckkimmiii (Chen al. 2020a). with the dimension transitions of 512 - 1024 + 1024 + 256. For predictor, we use 2-layer MonLinearNech
Architecture. The architecture of MoCo-v3 is ResNet Besaaett---rofeeccoo-rrraaaaaaa
imCLR, with the dimension transitions of 256 - 1024 + 256. The base momentum m to update key encoder is 0.996 and is linearly increased to I as training iteration goes. The temperature r for contrastive loss is set to 1.
Optimization. We follow Chen al. (022b) to use LARS optimizer with initial learning rate of 0.3, weight decay of 1.5e 6, momentum 0.9, and use CosineAnnearling learning schedule, We train all models with batch size of 256 for 200 epochs.
Data augmentation. Following previous work (rrill et al. 2020 Chen al. 2021b), use strong data augmentation: RandomResizedCropp Randomlooriznta.ll ColorJitter with (brightness=0.4, contrast=0.4, saturation=0.4, hue=0.4) and probability of 0.8, RandomGraysca e with probability of 0.2, Gaussi antassinBBlr with probability 0.5, and Solarizate with probability of 0.2.
and
C.3 LINEAR EVALUATION FOR PRE-TRAINING
In $4.1 mamually construct 9 sub-tasks in near-domain task, thus resulting pre-trained models for each pre-training method (FSP and CLP). Besides, for mixture-domain and out-domain tasks, models pre-trained the entire NCT training dataset with no class excluded.
Here we report their performance w.r.t. linear classifier. Specifically, we use the pre-trained models extract features from NCT training set and set. Then, we train LegisticRegression models on whole training features, and report the accuracy for whole test features. We report the results of Base-Base and Joint-Joint. Base-Base means the novel class excluded when we train and test the linear classifiers, while Joint-Joint means the linear classifiers are trained on all training features (80k samples) and then evaluated on all test features (20k samples).
Table C.I shows the results. With full supervision, FSP models can achieve better results in base label space (Base-Base). However, when evaluated joint label space (Joint-Joint), they are worse than CLP models. Besides, in few-shot setting, CLP models underperform FSP models in base label space (c.f. the "Base" column in Table near-domain task), but when the number training samples
18

---

Published as a conference paper at ICLR 2022
Table D.I: Results of 10-shot setting in near-domain task and mixture-domain task. In near-domain task, the "Base" and "Novel" columns report the average F1-score of the base/novel class respec- tively; the "HarmMean" columns report their average harmonic mean. In mixture-domain task, the same metrics are reported w.r.t. middle-domain and out-domain novel classes. "LA" denotes latent augmentation. The bold numbers denote the best while the underscored numbers denote the second best. Results in this table are in complementary to Table [
10-shot Near-domain Task
10-shot Mixture-domain Task
Methods
Basc
Novel
HarmMcan
Middle
HarmMean
Fupll-supervised pre-training (FSP)
NearesiCentrridd
90.08:10.35
60.96.10.72
71.00:10.46
50.63-0096
62.0410.75
S5.75.10.84
LogisticRegression
90.65:10.32
55.92:10.78
66 0.449
55.4220.87
56.8411.00
16.12.10.93
RidgoClassifiee
90.14:1..34
49.62:10.86
60.33=0.48
62.72=0.78
58.93+1.01
60.7711088
LogisticRegressonn + LA (ours)
92.75=+0.29
69.10=0063
78.14=0.39
70.10=00.69
65.26=#0.78
67.60+10.73
RidgeCllsssifier LA (ours)
90.69++..3
56.65-+00.80
66.96=0.46
71.09
63.77=0.84
67.23=0.74
Contrastive-learinng pre-training (CLP),
NcaresiCeentroid
85.15+0.445
68.18+00.7
75.18=0.54
85.97=0.40
68.58+0.65
76.30++0:49
LngisticRegression
87.20++0.41
69.83++0.71
76.83+0.55
85.82-0.38
66.42+0.66
74.89+0.48
RidgeClussifier
89.02-10.36
73.03-10.66
79.45 10.45
0..04
67.1911.66
75.84.10:6
LogisticRegressionn LA (ours)
89.62-10.36
82.40:10.48
48.00.0.40
44...4
80.08=0.51
84.95=041
RidgeClassifier LA (ours)
89.4510.36
83.8110.47
8..17=0.40
91.77=0.32
81.4710.00
86.3210.39
Table D.2: Ablation on the effect of the number of prototypes in the base dictionary. shows the numerical results of Figure (a).
This table
5-shot ablation task
Number of prototypes
Base
Novel
HarmMean
No Latent Augmentation
85.85
$3.27
65.74
76.84
66.99
71.58
79.83
67.92
73.39
84.87
67.59
75.25
16
87.37
66.51
75.53
32
87.08
65.94
75.05
64
86.65
65.41
74.55
128
86.52
65.32
74.44
256
86.
64.25
73.60
increases (no longer few-shot setting), CLP models can achieve similar results as FSP models (Table "BaeeBase" column).
D DETAILED RESULTS AND MORE ABLATIONS
D.1 10-SHOT RESULTS
D.I the results of 10-shot settings in near-domain and mixture-domail task, in comple-
mentary to results Table
D.2 ABLATION ON DATA AUGMENTATION
For ablation study, we exclude cancer-related classes, i.e. cancer-associated stroma (STR) and col- orectal adenocarcinoma epithelium (TUM), from NCT training set, and use them as novel classes. An additional CLP model is trained on this base dataset for ablation. D.2 shows the numerical results of Figure (a), which studies the effect of the number of prototypes in the base dictionary.
Figure D.I shows the full results of comparison between data augmentation (DA), latent augmen- tation (LA), and their combination (LAx DA). It can be seen that, DA brings slight gain when the number of augmentation times is small, and brings negative gain when the number of augmentation times increases. In contrast, LA and LA x DA can consistently improve the baselines for base class and novel class, resulting in best performance in harmonic mean.
19

---

Published as a conference paper at ICLR 2022
The data augmentations used in this experiment are RandomResizedCrog (scale= (0.8, 1.0)) with probability 1.0, RandomllorizontaPPlip with probability 0.5, and ColorJitter (brightness=O, contrast=0 4, saturation=0, 4, hue=0.2 with probability 0.8. These augmentations are applied sequentially and jointly on original images.
LA LAXDA
DA
LA
LAXDA
DA
LAHDA
DA
80
68
F1-score (N)
61
57
86
86
90
2
47
62
8
10a
12
14
16
22
47
6
8-
10a
122
142
16
23
42
6
8
10a
122
142
16"
Number of augmentation times
Number of augmentation times
Number of sugmentation times
(a) Effect of times in Buse Class
(b) Effect of times in Novel Class
(c) Effect of aug. times of their HarmMean
Figure D.I: Comparison of DA, LA and LAxDA. "LAxDA" denotes T latent augmentation applied after T traditional data augmentation (leads to T times in total).
D.3 ABLATION ON USING LABEL INFORMATION AND CALIBRATION
Supervised base dictionary. LA constructs the base dictionary using the unsupervised K-Means clustering. Alternatively, it use label information, if available, construct the base dictionary. Specifically, the prototypes and their covariance matrices are computed with respect to each class instead of each cluster.
Calibration. Distribution Calibration (Yang et al. 2021), recent state-of-the-art in few-shot clas- sification, proposes to use the statistics of base classes to calibrate the statistics of novel classes. Specifically, they compute calibrated novel class distribution: μ'' (Eies Ms + (k + 1), E' (Eies /k) /s + o,, where and a are two hyper-parameters, and |S] = k. Here, S is the top-k similar classes set the novel sample k denotes the number of base classes used to calibrate the novel class, and a controls the the degree of dispersion from the calibrated covariance. The augmented samples are then generated from N E¹). In their experiments, they find calibrating from two base classes, i.e., k = 2 achieve the best results. In our experiments, we set = 1, a = 0, since the number of base classes our problem is 7, which significantly smaller than 64, 160, and 100 base classes in natural images (Yang et al. 202I).
D.4 ABLATION ON COVARIANCE TYPES
By default, LA estimates the "full" covariance matrix for each cluster. Following notations in Gaus- sianMixture model API in scikit-learn library (Pedregosa al.) (01IT) we further compute different types of covariance: 1) "tied", where all clusters share the same general covariance matrix estimated from the entire base dataset; 2) "diag". where each cluster has its own diagonal covariance matrix; 3) "spherical", where each cluster has its own single scalar variance. Specifically, "diag" covariance matrix only computes the variance of each channel (each dimension) without considering the cor- relation between channels, and "spherical" covariance further averages diagonal elements in each single scalar variance. Results are shown in Table 2
"diag" covariance matrix, resulting
Table 2 reports results of our default setting, where 16 clusters are used. Here we further explore more options cluster numbers (leading to larger or smaller clusters). Figure DD2] the results. LA with full covariance matrix does degenerate performance the cluster size decreases (larger number of prototypes). In contrast, LAs with isotropic covariance matrices ("spherical'""diag" perform more stably when cluster size alters. However, they underperform "full" covariance in all cluster sizes. Besides, similar performance degeneration also observed when using isotropic covariance, but becomes slight. Note, the "Tied" covariance is estimated from the whole dataset.
20

---

Published as a conference paper at ICLR 2022
Therefore the results should only be affected by randomness in latent augmentation (yellow line in Figure D.2).
Diag
Spherical
76
Average Harmonk ic mesi of ol acare
74
160
73
72
71
16
32
128
256
Number protolyppes
Figure D.2: Ablation study on cluster sizes v.s. covariance types.
D.5 ABLATION ON K-MEANS RANDOM SEEDS
LA relies on K-Means clustering, which might be effected by random initialization. Here we explore the K-Means API from faiss Johnson
how LA performs under different random seeds. Still, et al. 2019), a library for clustering. Table D.3 reports the results. LA is robust to the choice of random seeds. Among different seeds, the numerical differences of performance are within 95% confidence interval. We choose the random seed 66 for reproducibility in main paper.
Table D.3: Ablation on different K-Means seeds. Results of ablation tasks are reported, i.e. 300 meta-tasks under 5-shot setting.
Random Seed
Base
Novel
HMean
Baseline
85.8520.78
53.27+1.63
65.74
86.85+0.88
65.70+: 1.0
74.81
20
87.3420.80
16.74+1.33
75.66+1.00
30
87.71=0.81
66.48.11.5
75.63+1.01
40
87.31
65. 1..04+1..3
74.55 .02
50
87.83+0.78
66.40
75 03..6300.99
66 (default)
87.37=0.81
66.51+1.39
75.53+1 .03
Average
87.40=0.81
66.1441.37
75.30+1.01
D.6 ABLATION ON COVARIANCE & L2 NORMALIZATION.
In our experiments, we compute the covariance matrices after Iz-normalization To inspect whether the covariance is still meaningful after normalization. We compute the covariance matrices of NCT test set features before and after ng-normalization. Then, we plot the covariance matrix as heatmaps. Since the length of feature vector is 512, and inspecting a 512 512 heatmap might be difficult, we first compute the whole covariance matrix, then randomly select 128 channels to visualize; the upper triangle elements of this 128 x 128 matrix are masked out. Figure D.3 D.4 D.5 are results of three runs. We observe that the covariance before and after Ig-normalization are highly corre- lated, where the Pearson correlation coefficients are 0.9908, 0.9900, 0.9910 between before/after l --normalization covariance in Figure D.3 D.4 and D.5 respecttiv... Due to this high correlation, the covariance after 12-normalization may still be informative to some extend. Besides, a recent work self- sef-ssuerrviee learning, BarlowTwins (Zbontar et al. 2021), also compute covariance af- ter l2 my-nommalizttion in their ablation study (Section 4, "Loss Function Ablations").
After ly nonoalizzaiion, all original samples sional manifold, i.e. Sd-1), but the augmented samples generated by LA are not necessarily on the surface. We therefore further normalize the augmented samples to whether this impacts
the surface of unit- mni-hhyperpphere (a d-1 dimen-
21

---

Published as a conference paper at ICLR 2022
performance. In the same setting ablation study (300 meta-tasks, 5-sot), observe the HMean marginally drops from 75.30= 1.01 to 74.69=1.01. Although the lg-normalized features are on the surface of unit-hypersphere, the linear classifiers, typically, still fitted in the original R space and no special consideration is taken. Generating augmented samples that are not on the surface of unit-hypersphere may still help. For more analysis about the niceness of unit-hypersphere, we refer the reader to the discussion section in Wang & Isola (2020).
Before L2-norm
After L2-norm
0.0
0.5
1.0
00001 0.000 0.001 0.002
0.003
Figure D.3: Heatmap visualization of the covariance matrix of randomly selected 128 channels before and after Iz-normalization. Pearson correlation coefficients between them are 0.9908.
Before L2-norm
After L2-norm
-05
0.0
0.5
1.0
00.011
0.000
0.001
0.002
0.003
Figure D.4: Heatmap visualization of the covariance matrix of randomly selected 128 channels before and after (z-normalization. Pearson correlation coefficients between them are 0.9901.
D.7 COMPARISON TO DELTA-ENCODER
D.7.1 REPRODUCTION
We follow an open-source repository the batch size from 128 to 512 for faster training speed. Other hyper-parameten remain unaltered.
2 to implement 8-encoder (Schwartz al. 2018 and change
Note that, 8-encoder trains different generators for different shot settings. e.g., one generator for 1-shot setting, and another for 5-shot. This differs from our flexible and universal augmentation pipeline. Besides, in both the official and the wytorch-version repositories, the best generator is chosen based on the best performance test set. We do not follow this setting, and directly use the last-epoch model for augmentation.
http:///ghhhhaaccmmmmmmmmotittin pytorch
22

---

Published as a conference paper at ICLR 2022
Before L2-norm
After L2-norm
-0.25 0.00 0.25
0.50
-0.01
0.000
0.001
0.002
Figure D.5: Heatmap visualization of the covariance matrix of randomly selected 128 channels before and after Iz-normalization. Pearson correlation coefficients between them are 0.9910.
For 5-shot ablation task, we exclude the tumor-related classes be novel class and use the remaining as base dataset to train d-encoder. All 6-encoders are trained for 20 epochs (the original paper trains 12-epoch for 5-shot). We find the 20-epoch d-encoder performs slightly better than 12-epoch and 50-epoch d-encoders.
D.7.2 RESULTS AND DISCUSSION
Results. Table D.4 shows the results. When using cluster labels supervision, find train- ing --encoder on non-lz-normalized features and transferring to l2-normalized features gives the best results. Normalizing the original features and generated features further improves the results (63.40% + 64.92%). Using ground truth labels, unfortunately and surprisingly, leads to worse per- formance (64. (64.92% - especially no further nzooommalization is applied on generated features (63.30% +48.1 14%)....... In our re-implementation, none of tested cases can outperform baseline and our method.
During augmentation, 8-encoder randomly samples pair of base features from same class, referred as reference pair, and use as guidance to augment novel samples. It, by construc- tion, does not utilize any similarity information between base features and novel features. This could potentially cause problems as the base samples and novel samples can differ drastically, making the semantic transferring meaningless. In contrast, LA queries the most likely variation from base dictionary via cosine similarity, which more strict constraint than randomly sampling as 5- encoder. Besides, in d-encoder, the reference pair used for augmentation could come from different base classes every time for a same novel class. For example, it could sample LYM-LYM pair ADI-ADI pair augment another novel
Discussion.
augment one novel sample from TUM class, and sample from TUM class This step might lead to feature inconsistency, which could degenerate performance.
D.8 VISUALIZATION
D.8.1 VISUALIZATION PROCEDURE
Models. All images are visualized with models that have never seen their classes during pre- training. For example, the class "NORM" in NCT regarded as novel class, samples belonging to it are excluded from this sub- pre-training dataset. Models trained on this sub-dataset are used to visualize "NORM" class.
Similarity map. To better understand why the generalization gap exists between CLP and FSP models, we visualize how CLP models and FSP models are attended to different local features for non-iconic, multi-object and multi-texture histology images. To this end, we resize images in
LYM,, ADI and TUM class in NCT dataset
23

---

Published as a conference paper at ICLR 2022
Table D.4: Comparison d-encoder in ablation task. ô-encoder is supervised method. "Label source" column marks using unsupervised clustering labels or ground truth labels as supervision. We study training d-encoder on Iz-normalized (L2) and non-normalized (Non-L2) base features, marked in "Trained on" columns. Given the trained d-encoder, we study how the target features applied on affect the results, marked in the "Transfer to" column. last, we study how 12- normalization classifier affects final performance, marked in "CIf train&infer" column. Underlined numbers denote the best entries among each group rows. Bold numbers denote the best entries among all rows. Results are shown in percentage (%).
Method
Label source
Train on
Transfer to
Clf train infer
Base
Novel
Hmean
KMeans(K=16)
1.2
1.2
1.2
82.40
48.76
61.26
KMeans(K=16)
1.2
1.2
Non-L.2
82.04
48.15
60.28
KMeans(K=16)
L.2
Non-L2
L2
81.16
45.31
58.15
KMeans(K=16)
L2
Non-L2
Non-L2
77.89
45.58
57.51
8-encoeer
KMeans(K=1))
Non-L2
L2
L2
79.68
54.77
64.92
KMeans(K=16
Non-L2
L2
Non-L2
81.65
51.82
63.40
KMeans(K=16)
Non-L2
Non-L2
L2
81.52
48.21
60.59
KMcans(K=16)
Non-L2
Non-L2
Non-L2
77.82
46.89
58.52
Ground Truth
Non-L2
80.18
52.29
63.30
Ground Truth
Non-L2
Non-1.2
82.67
33.93
48.11
Baseline
85.85
53.27
65.74
ours
Baseline+LA
87.68
66.71
75.77
the NCT test set to 448 448, and the pre-trained CLP models and FSP models to extract io-normllized features from stage-2, stage-3, and stage-4 of ResNet-18. To obtain similarity heat maps, we compute the cosine similarity between global representations (after pooling) and local representations (before pooling). For absolute similarity ("Abs. Sim."), we directly rescale the similarity values by multiplying 255. For normalized relative similarity ("Rel. Sim."), we first rescale the similarity values to [0, 1] by subtracting the minimum and dividing the maximum, and then multiply them by 255.
Local feature agglomeration. We follow Chen & Li 2020) to see how local features are agglom- erated across layers. To this end, we run K-means in leikit-learn Pedregosa et al. (2011) on the normalized local features (before pooling) from stage-2, stage-3, and stage-4 of ResNet-18 with different numbers clusters, i.e. 2, 4, 6.
D.8.2 MORE EXAMPLES
Figure D.6 show more visualizations of a base class that has been exposed during both supervised pre-training and self-supervised pre-training.
Figure D.7 and Figure D.8 show more example images in the NCT dataset. Figure D.9 and Figure D.10 the results of the LC-25K dataset and the PAIP dataset respectively. We visualize the ab- solute and relative cosine similarity between the global average pooled feature and the local features before pooling ("Abs. Sim." and "Rel. Sim." columns accordingly). To inspect correlation between local features, we conduct k-means clustering with different values (columns with "k=*"). "Low", "middle" and "high" represent using features from stage-2, 3, and 4 from ResNet-18.
24

---

Published as a conference paper at ICLR 2022
CLP
FSPP
Abs. Sim. Rel.
2=2
k=4
k=6
Abs. Sim.
Rel. Sim.
k=2
k=4
k=6
Low
Middle
High
Low
Middle
High
Low
Middle
High
Figure D.6: Visualization of base classes in NCT dataset. FSP show high correspondence the most discriminative parts (Abs. Sim. and Rel. Sim. columns). However, the cluster results of non-discriminative parts appear to be meaningless, which may implies FSP's failure in encoding all available semantic information in the image.
25

---

Published as a conference paper at ICLR 2022
CLP
FFSP
Abs. Sim.
Sim.
k=2
k=4
k=6
Abs. Sim.
Rel. Sim.
k=2
k=4
k=6
Middle
High
Middle
High
Middle
High
Low
Middle
High
Low
Middle
High
Figure D.7: Visualization in the NCT dataset White indexes in black circles represent class IDs. They 0) background (BACK), adipose (ADI), 2) debris (DEB), 3) lymphocytes (LYM), and 4) mucus (MUC).
26

---

Published as a conference paper at ICLR 2022
CLP
SSP
Abs. Sim. Sim.
k=2
k=4
k=6
Abs. Sim.
Rel. Sim.
k=2
k=4
k=6
Low
Middle
High
Low
Middle
High
Low
Middle
High
Low
Middle
High
Figure D.8: Visualization in the NCT dataset White indexes in black circles represent class IDs. They are: 5) smooth muscle (MUS), 6) normal colon mucosa (NORM), 7) cancer-associated stroma (STR), 8) colorectal adenocarcinoma epithelium (TUM).
27

---

Published as a conference paper at ICLR 2022
JCLP
JFSP
Abs. Sim.
Rel. Sim.
k=2
k=4
k=6
Abs. Sim.
Rel. Sim.
k=2
k=4
k=6
Low
Middle
High
Low
Middle
High
Low
Middle
High
Low
Middle
High
Low
Middle
High
Figure D.9: Visualization in the LC-25K dataset. White indexes in black circles represent class IDs. They are: 0) colon adenocarcinoma, 1) benign colonic tissue, 2) lung adenocarcinoma, 3) benign lung tissue, and 4) lung squamous cell carcinoma.
28

---

Published as a conference paper at ICLR 2022
CLPP
SP
$
Sim.
k-2
k=4
k=6
Abs. Sim.
Rel. Sim.
k-2
k=4
k=6
Low
Middle
High
Low
Middle
High
Low
Middle
High
Low
Middle
High
Low
Middle
High
Figure D.10: Visualization in the PAIP dataset. White indexes in black circles represent class IDs.
They are: 0) non-tumor, viable-tumor and 2) other tissues.
29


Reviews:
[{"review_id": "vjx-BXfXmTW", "reviewer": "Reviewer_PqMe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed LA is interesting and shows superior data augmentation including RandomResizedCrop, RandomHorizontalFlip, and ColorJitter.\n2. The paper is rich in contents and provided a comprehensive appendix.\n3. The study of few shot learning on histological images are at its early stage. This paper's result show that CL can be better than supervised pretraining, which is a encouraging finding. It will be inspiring for researchers in this field.\n\nWeakness:\n1. The technical novelty of this paper is limited. The CL strategy is the existing MoCo v3. LA is simple and effective, but it has supervised counterparts (as stated in the related work part). \n2. Lack of comparisons. The authors compared \"CL vs. supervised learning\" and \"LA vs. stardard data augmentation\", but did not compare with other few-shot learning, semi-supervised learning and data augmentation algorithms. These methods can also be used in problems with few labeled samples.\n3. In Table 1, LA is compared with no data augmentation (if I understand correctly), which is unfair because data augmentation have become a standard step, so LA should be compared with other data augmentation methods.\n4. Other self-supervised learning (SSL) papers, such as \"Unsupervised Learning of Dense Visual Representations, NeurIPS 2020\", \"Dense contrastive learning for self-supervised visual pre-training, CVPR 2021\", \"Momentum Contrast for Unsupervised Visual Representation Learning, CVPR 2020\", have found that SSL sometimes outperform supervised pretraining. Therefore, the finding of this paper is not so surprising. In this paper, the novel classes are improved more by SSL pretraining, while supervised pretraining still prevails in the base classes, which is consistent with existing papers which found that SSL are better when the task or dataset is different in pretraining and finetuning.\n5. The paper is rich in contents and provided a comprehensive appendix. However, it may make the main body not very self-contained. It may be better to submit the full paper to a journal in this field such as Medical Image Analysis. For example, the core algorithm, MoCo-v3, should be explained in the main body.\n\nQuestion: Is self-supervised learning performed on manual crops of WSIs? Since each patch may contain one manually assigned label, does that mean labels have been somehow encorporated in the self-supervised learning process? Ideally, self-supervised learning should be performed on WSIs without manual cropping.", "summary_of_the_paper": "The paper proposes to use contrastive learning (CL) with a novel data augmentation (latent augmentation, LA) strategy to build a few-shot system for histology image classification. Two empirical findings are: i) models learned by CL generalize fairly better than supervised learning for histology images, and ii) LA brings consistent gains over baselines without data augmentation. Analyses are given to discuss these findings.", "summary_of_the_review": "The paper has slight novelty and interesting findings but neither novelty nor finding is significant enough. It may be better to be submit it to medical image journals.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hZe5i_lZGe", "reviewer": "Reviewer_hJxe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- Great results, showing a consistent and significant performance boost (~ +20% F1 score) few-shot classification of histology images.\n- The proposed latent augmentation is simple to implement, and it consistently outperforms the standard input augmentation. It has the potential to be adopted by the community.\n- Very well structured and written paper. The paper feels complete.\n- Extensive experiments and ablation studies, with sufficient implementation details provided (e.g., MoCo method is explained)\n- Extensive discussion of the findings.\n\n\n**Weaknesses**\n- The proposed method, \"latent augmentation\" isn't compared to other variation data augmentation methods. This makes the reader wonder whether \"latent augmentation\" is superior to other variation augmentation methods. Why didn't you compare with other \" variation augmentation\" methods? For example, how well does adding white noise to the latent variables perform?\n- Add a small investigation of the shape of the clusters (i.e., the covariance matrix). K-Means corresponds to GMM with an isotropic covariance matrix. If it turns out that isotropic covariance matrices are sufficient, this information would be helpful in scenarios with small clusters (when the covariance approximations are poor).\n- How stable is the method to the randomness in K-means? Did you notice any significant performance variation when running K-means with different random seeds?\n\n**Minor comments**\n- The experiment (and the caption) of figure 4 is unclear, even though the appendix provides many details. Please refactor the explanation so that it doesn't require reading the appendix (for example, explain what the features are inside the caption).\n- It's unclear how l2 feature normalization is used in clustering and downstream meta-tasks. As of now, I understand that l2 feature normalization is applied before doing clustering. However, all feature vectors would be on the unit hypersphere, giving a meaningless cluster covariance matrix. Please clarify.\n- Some paragraphs are too general and can be reduced in size. For example, the first paragraph in section 3.1 could be smaller. The fully supervised pre-training (equation 1) is just the standard cross-entropy loss. The few-shot learning section from related works could be reduced in size by a third.\n- Replace the formula for choosing the cluster in latent augmentation (penultimate row in section 3.2) with simply saying that i* is selected by using cosine similarity between the representation z and c_i.\n- The following terms are ambiguous and should be changed: \nsaying that the results \"fairly better.\"\n-- \"space is shown to be highly linearized\" is ambiguous. Do you mean the space allows linear interpolation?\n-- \"legacy dictionary\"\n- Does the +- numbers in Table 1 specify the confidence interval or the standard deviation across multiple runs?\n- Replace \"golden labels\" with \"true labels\"\n- Typo: \"significantly lower than LA (image augmentations\" should be DA instead of LA\n- Will the code be released?\n", "summary_of_the_paper": "The paper investigates few-shot learning for histology images. So far, contrastive learning has been studied for natural images, which differ from medical images. The paper shows that contrastive pre-training provides a significant performance boost (~ +10% F1 score) compared to standard supervised pre-training in few-shot learning. Further, they introduce a new self-supervised method (called \"latent augmentation\") for performing augmentation in the feature space created by the model. The prediction model (e.g., logistic regression) trained on the augmented data provides ~ +10% higher F1 score for small and moderate shifts in the pre-training and base dataset (for pre-training the feature extractor) and the support dataset (in the meta-tasks). Overall, the results are very promising, and the performance boost is significant. The paper has the potential to make contrastive learning the standard pre-training methodology for non-natural images and make the \"latent augmentation\" a common data augmentation technique in few-shot learning.\n", "summary_of_the_review": "The paper shows very promising results for few-shot learning in histology images and introduces a simple and effective method for doing data augmentation. The paper would benefit from some additional experiments with competing augmentation methods and a few straightforward ablations. Overall, the paper is very well written, feels complete, and has an extensive discussion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dCfyObzWoIJ", "reviewer": "Reviewer_ijae", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. This paper proposes latent augmentation (LA), a new augmentation method to diversify training samples in the latent space. By sampling variations learned from the base classes and adding to novel class samples, LA can essentially generate more diversified samples and improve the model generalization ability.\n2. Extensive experiments show the proposed LA improves the baseline counterparts on the three low-shot learning histology image classification tasks, although the improvements on the out-domain task are much smaller.\n3. Ablation experiments provide more details about the influence of prototype number and augmentation times in latent augmentation.\n\nWeakness\n1. The proposed latent augmentation relies on the random initialization of k-means clustering. How does the randomness affect the result? In other words, how should the seed for K-means clustering be chosen? If a different seed is used, will the performance be much worse?\n2. The variation from base classes can also be calculated from the whole dataset without K-means clustering. It seems to be a simple alternative to be compared.\n3. The explanation of why CLP models generalize better than FSP models is not convincing. While the lack of dominant object in histology images is a major difference compared to ImageNet natural images, how does it result in the higher global-local feature similarity in higher feature level? Also, what is absolute and relative similarity in Figure 4?\n4. Why are no regularization techniques used during fully supervised pre-training?\n5. In page 7, section \"DA vs. LA, and number of augmentation times\", is the computation budget of LA lower than \"DA\", instead of \"LA\" again?", "summary_of_the_paper": "This paper studies the low-shot learning problem with application to histology image classification. Specifically, contrastive learning pre-training is used for representation learning in an unsupervised manner. For low-shot classification, a new latent augmentation is proposed to augment samples by transferring variations from base classes to novel classes in the latent feature space. Three low-shot classification tasks are set up with different levels of domain shift, i.e., near-domain, mixture domain, and out-domain, using public histology image datasets. Experiments are conducted to verify the proposed method's effectiveness for these three low-shot learning tasks.", "summary_of_the_review": "This paper proposed a new augmentation method for low-shot classification in histology images. It is well-motivated and reasonable. The experiment results also confirmed its effectiveness. There are also some issues regarding the proposed method, i.e., the influence of K-means clustering randomness, and the explanation of CLP's superiority over FSP. Overall, this paper's quality is good.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qs-JNJGjtud", "reviewer": "Reviewer_qUzb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main strengths:\n1) The paper focuses on a real limitation which hinders widespread deployment of ML-based pipelines in histological image analysis: the issues relative to out-of-domain adaptation, especially with respect to the introduction of a new label set (i.e., different types of tissues/anomalies). \n2) The proposed approach, which makes large use of some of the latest development in both FSL and data augmentation strategies, is interesting and convincing.\n3) Most importantly, the experimental section is comprehensive and well-devised. Three different adaptation scenarios are implemented, with varying degrees of domain adaptation required. \n\nMain weaknesses:\n1) Lack of novelty: while probably novel in their combination and use case, none of the implemented techniques (i.e., contrastive and fully-supervised classification, domain adaptation through FSL) is very novel. The only exception might be “latent augmentation”, which follows in the footsteps of previous latent space augmentation techniques but - differently from most of them - doesn’t require labels. This aspect is interesting, but since the experiments are limited to histological images, it is difficult to assess its impact on image classification in general.\n2) Potentially questionable take-home message: amongst their most important contributions, the authors state that “Also to our surprise, we show that state-of-the-art CL models generalize fairly better than the supervised counterparts for histological images”. In their Discussion, they add “The lack of dominant objectiveness makes FSP (Fully-supervised pre-training) tend to overfit severely in high level, but CLP (contrastive-learning pre-training) can capture the overall patterns from the crudely scattered tissues. We believe this is the main reason why CLP models generalize better than FSP models, in pathology.” To me, this is more a statement about the observed outcome rather than an explanation. Additionally, I’m not sure that the results shown by Chen & Li (2020) are directly comparable to these, which are relative to the same comparison but performed in an out-of-domain (i.e., previously unseen class) setting. It seems rather intuitive to me that FSP-learned representations would be unreliable when a patch showing a previously unseen class is presented. In CLP, on the other hand, the goal is learning to identifying similarities and differences in images, which should provide more class-agnostic mid-level features. I would appreciate a more detailed discussion of this comparison.\n\nMinor comments:\n1. Typos (page 5): “samples belong”, “we summarizes”\n2. Typo (page 15): “oragan”. Also, a “plus” seems missing from the “Out” equation.\n3. Typo (page 7): “The trade-off EXISTS”\n4. Typos (page 9): “Later methods is attracted to transfer learning”, “This method is later extended by not relying given base samples”\n5. Please consider citing and briefly discussing the following paper on latent space augmentation from ICLR 2021: Tsz-Him Cheung, Dit-Yan Yeung. MODALS: Modality-agnostic Automated Data Augmentation in the Latent Space. ICLR 2021 \n\nChen & Li (2020): https://contrastive-learning.github.io/intriguing", "summary_of_the_paper": "The paper focuses on learning useful representations of histological patches for classification tasks. Acknowledging the large domain shifts typical of histological imaging (e.g., differences in examined organs/tissues, imaging protocols, staining intensities), the authors frame the classification problem as a few-shot learning (FSL) one. In their experiments, they compare FSL pre-training on a base dataset both using full supervision and contrastive learning. During low-shot training, they make use of an unsupervised approach for data augmentation in the latent space (referred to as “latent augmentation”). In the results, they note that contrastive learning with latent augmentation provides the most reliable representations, especially for out-of-domain tasks. ", "summary_of_the_review": "Overall, I vote for acceptance. While the novelty might be relatively limited, the comparison between fully-supervised and contrastive pre-training in an FSL setting (with and without latent augmentations) for histopathological images is interesting. The experimental design and ablations provide a valuable overview of the inner workings of these methods, which could be valuable to a large portion of ICLR’s audience (potentially beyond applications to histopathology).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IvYtuMHXPa-", "reviewer": "Reviewer_BmyB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of few-shot learning in biomedical applications is of great interest, in particular in histology where whole slide scanners can capture a lot of data, but annotations are scarce and expensive. The datasets used in the study are openly available and cover a large range of variations in tissue, staining and pathologies.\n\nThe paper is well structured, but hard to follow. The language/text of the paper should be revised. At various places expressions used are ambiguous which affects the scientific value to the point of incorrectness and makes the steps taken very hard to understand. Articles are often omitted, there are a number of grammatical mistakes and some words are used erroneously throughout the paper (reflect, confront, deprecate...).\n\nSome vocabulary used is not conventional or precise. Some examples: \nThese two components [...] can scale gracefully [...]. (abstract)\n[...] CLP generalizes fairly better [...]. (sec. 4.2)\nSuch boost [...] becomes humble in out-domain task. (sec. 4.2)\n[...] DA can slightly polish LA [...]. (sec 4.3)\n\nIt should be addressed more clearly what is meant by “near-domain”, “mixture-domain”, “middle-domain” and “out-domain”. \nIn 4.1 it says “We use faiss to perform K-means with a fix seed.” - it should be pointed out that faiss is a library for clustering.\nFor readers outside this immediate field of deep learning (e.g. more application oriented bioimage analysts in digital pathology), I suggest to use the same terminology throughout the paper, currently the authors switch for example between “low-shot” and “few-shot” and similar.\n\nAt multiple places it was hard to understand what the authors were trying to say, a few examples are: \nin 3.2.1 “From the view of low-data learning, distribution of few samples is not well calibrated, so using established distributions in base class to calibrate untrustworthy novel class can help.”; \nin 4.1 “The rest three lung-related classes are regarded as out-domain novel classes.”; \nin 4.3 “It is worthy to emphasize the computation budget involved in LA (addition in $\\mathbb{R}^d$ space) is significantly lower than LA (image augmentation in $\\mathbb{R}^{3HW}$ space and encoder forwards)“\nIn 4.3 “Patches from a single WSI are confronted to its stain degree (impacts color) and body site (impacts pattern and context)” - what does it mean they are confronted to these properties?\nIn 4.4 “The lack of dominant objectiveness makes FSP tend to overfit severely in high level, but CLP can capture the overall patterns from the crudely scattered tissues.”\n\n\nScientific content:\nFSL description in Sec. 2: Is $\\mathcal{S}$ a subset of $\\mathcal{D}$_{novel} and $\\mathcal{Q}$ a subset of $\\mathcal{X}$_{novel}? If so, the notation needs to be revised. Currently it states they are distributed by $\\mathcal{D}$_{novel} and $\\mathcal{X}$_{novel} respectively. Furthermore, the sets $\\mathcal{S}$ and $\\mathcal{Q}$ are indexed by i=1 to NK and i=1 to NQ, but immediately after K and Q are referred to what probably should be NK and NQ.\nThe authors investigate how augmentation in the latent space compares to data augmentation w.r.t. To F1-score. However, I cannot find the specifications of what kind of data augmentation was used in the main paper, which is a crucial aspect. I eventually found some details in the appendix, but I would like to see some intuition behind the choice of data augmentation (crops, horizontal flips and color jittering) and how it is expected to be comparable to the latent augmentation in terms of adding variety to the dataset. It is common to use more extensive data augmentation - rotations, mirroring, blurr,... I also lack the information if at every iteration one augmentation was drawn at random, or a combination of the augmentations. \nAdditionally, the data augmentation used for the fully supervised pretraining (crops, horizontal flips and normalization) was chosen differently than in the ablation study, why?\nIt is not clear to me what is meant with augmentation times in the experiments of LA vs DA. Is it that the augmentations are being performed on already augmented versions of the original input? If so, how does the additive color jittering affect the images?\nHow are the patches resized in the LC-25K dataset? What interpolation is used?\nFig. 4, visualizations of CLP features from stage 4 using 4 and 6 clusters for k-means: it looks as if there are three clusters inherent to the data present in the shown image, but for k=4 and k=6, the data is split into 4 or 6 clusters due to how k-means work, but I wonder why those “additional” clusters are distributed along the edges of the image as they are? Are the edge effects in these feature maps that cause this behavior or is it due to initialization of the seeds for k-means? Similar behavior can be observed in the visualizations provided in the appendix.\n \nNovelty:\nThe authors say that few-shot learning “is much unexplored in medical images, especially in histological ones.” and they “pioneer the study of low-shot learning for histology images [...]”. However, there are a number of studies on this topic, some included in the paper’s related work section, but there are other recent works on this topic also, “Learning with Less Data Via Weakly Labeled Patch Classification in Digital Pathology“ by Teh et al. (ISBI 2020), “Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study” by Sikaroudi et al. (EMBC 2020) to name a few.\n", "summary_of_the_paper": "The paper addresses few-shot classification for histological images. They compare the generalization capabilities of self-supervised contrastive learning and fully supervised classification as pretraining. They perform experiments to evaluate the impact of latent augmentation vs. data augmentation. They use three publicly available histological datasets (spanning multiple tissues, pathologies and number of available classes). They construct a number of tasks given the tissue labels provided in those datasets and compare the classification performance in N-way-K-shot classification experiments for a number of different N and K.\n", "summary_of_the_review": "The paper addresses few-shot learning for the classification of histological images. This topic is of significant interest to the biomedical community, as annotations are very expensive to obtain. Exploring the generalizability of different pretraining methods and augmentations to exploit the available ground truth to its fullest extent is of great importance. However, I found it very hard to understand the text and followingly some of the experimental setup. On multiple occasions, the phrasing was so ambiguous, that it became unclear to me what the authors tried to convey. The text needs revisions to correct for semantic and grammatical errors. Due to this unclarity, I suggest to reject this submission, but encourage the authors to revise the text and submit to another venue.\n\n\n\n------------- After author's response and revision of the submission ------------------\nThe authors addressed my concerns and questions in detail and the revised submission has improved in clarity. I believe the topic is of significant interest to the biomedical imaging community, which is why I am suggesting to accept the revised submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper presents an approach and evaluation setting for few-shot learning in histology images. The approach leverages contrastive learning pretraining, and latent augmentation (LA) for data augmentation. The evaluation examines in-domain few-shot learning, mixed domain few-shot learning, and out of domain few-shot learning.  

Latent augmentation is an approach to learn how categories vary between samples within unsupervised clusters in a base dataset, and transfer that variation to the few-shot sampled classes. 

Pros:
- A couple reviewers have claimed as a strength the novelty of the proposed latent augmentation method, but as other reviewers point out, there is much work in this field, some of which wasn't cited (i.e. Delta-Encoder, NeurIPS 2018).
- The latent augmentation method is simple to implement, and outperforms standard input augmentation approaches.
- The paper is rich in content and details of experiments.
- Examining learning over a variety of domain shift settings is interesting.
- Shows contrastive learning can outperform supervised pretraining for this application domain.

Cons:
- Multiple reviewers raise concerns about technical novelty. This work applies mostly previously proposed methods, or variations thereof, to the domain of medical imaging. May be more suited to a medical imaging venue. 
- Some of the results are consistent with prior reports, such as finding that self-supervised learning can outperform supervised pretraining. In that regard the results are not surprising.
- One reviewer raised issues about lack of comparison to other relevant few-shot works. Authors argue that fine-tuning is a competitive baseline. Authors did add comparison to one other variation augmentation approach, distribution calibration. But as mentioned, delta-encoder is a very related work, which has not been cited nor compared against. Biggest difference is that delta-encoder uses labels, but the unsupervised clusters can trivially be supplied as labels in this setting. AC feels authors should have done a more comprehensive comparison to related learned augmentation works. 
- Authors initially did not address how latent augmentation is affected by random seeds, but authors have replied to reviewers with additional data.

Reviewer consensus, excluding 1 reviewer, favors accept, though significant concerns regarding technical novelty and comparisons to other relevant works persist (especially in regards to works that learn how to augment as the proposed LA method does).
Decision: Accept (Poster)