Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the low-shot learning problem with application to histology image classification. Specifically, contrastive learning pre-training is used for representation learning in an unsupervised manner. For low-shot classification, a new latent augmentation is proposed to augment samples by transferring variations from base classes to novel classes in the latent feature space. Three low-shot classification tasks are set up with different levels of domain shift, i.e., near-domain, mixture domain, and out-domain, using public histology image datasets. Experiments are conducted to verify the proposed method's effectiveness for these three low-shot learning tasks.
main_review: Strength
1. This paper proposes latent augmentation (LA), a new augmentation method to diversify training samples in the latent space. By sampling variations learned from the base classes and adding to novel class samples, LA can essentially generate more diversified samples and improve the model generalization ability.
2. Extensive experiments show the proposed LA improves the baseline counterparts on the three low-shot learning histology image classification tasks, although the improvements on the out-domain task are much smaller.
3. Ablation experiments provide more details about the influence of prototype number and augmentation times in latent augmentation.

Weakness
1. The proposed latent augmentation relies on the random initialization of k-means clustering. How does the randomness affect the result? In other words, how should the seed for K-means clustering be chosen? If a different seed is used, will the performance be much worse?
2. The variation from base classes can also be calculated from the whole dataset without K-means clustering. It seems to be a simple alternative to be compared.
3. The explanation of why CLP models generalize better than FSP models is not convincing. While the lack of dominant object in histology images is a major difference compared to ImageNet natural images, how does it result in the higher global-local feature similarity in higher feature level? Also, what is absolute and relative similarity in Figure 4?
4. Why are no regularization techniques used during fully supervised pre-training?
5. In page 7, section "DA vs. LA, and number of augmentation times", is the computation budget of LA lower than "DA", instead of "LA" again?
summary_of_the_review: This paper proposed a new augmentation method for low-shot classification in histology images. It is well-motivated and reasonable. The experiment results also confirmed its effectiveness. There are also some issues regarding the proposed method, i.e., the influence of K-means clustering randomness, and the explanation of CLP's superiority over FSP. Overall, this paper's quality is good.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use contrastive learning (CL) with a novel data augmentation (latent augmentation, LA) strategy to build a few-shot system for histology image classification. Two empirical findings are: i) models learned by CL generalize fairly better than supervised learning for histology images, and ii) LA brings consistent gains over baselines without data augmentation. Analyses are given to discuss these findings.
main_review: Strength:
1. The proposed LA is interesting and shows superior data augmentation including RandomResizedCrop, RandomHorizontalFlip, and ColorJitter.
2. The paper is rich in contents and provided a comprehensive appendix.
3. The study of few shot learning on histological images are at its early stage. This paper's result show that CL can be better than supervised pretraining, which is a encouraging finding. It will be inspiring for researchers in this field.

Weakness:
1. The technical novelty of this paper is limited. The CL strategy is the existing MoCo v3. LA is simple and effective, but it has supervised counterparts (as stated in the related work part). 
2. Lack of comparisons. The authors compared "CL vs. supervised learning" and "LA vs. stardard data augmentation", but did not compare with other few-shot learning, semi-supervised learning and data augmentation algorithms. These methods can also be used in problems with few labeled samples.
3. In Table 1, LA is compared with no data augmentation (if I understand correctly), which is unfair because data augmentation have become a standard step, so LA should be compared with other data augmentation methods.
4. Other self-supervised learning (SSL) papers, such as "Unsupervised Learning of Dense Visual Representations, NeurIPS 2020", "Dense contrastive learning for self-supervised visual pre-training, CVPR 2021", "Momentum Contrast for Unsupervised Visual Representation Learning, CVPR 2020", have found that SSL sometimes outperform supervised pretraining. Therefore, the finding of this paper is not so surprising. In this paper, the novel classes are improved more by SSL pretraining, while supervised pretraining still prevails in the base classes, which is consistent with existing papers which found that SSL are better when the task or dataset is different in pretraining and finetuning.
5. The paper is rich in contents and provided a comprehensive appendix. However, it may make the main body not very self-contained. It may be better to submit the full paper to a journal in this field such as Medical Image Analysis. For example, the core algorithm, MoCo-v3, should be explained in the main body.

Question: Is self-supervised learning performed on manual crops of WSIs? Since each patch may contain one manually assigned label, does that mean labels have been somehow encorporated in the self-supervised learning process? Ideally, self-supervised learning should be performed on WSIs without manual cropping.
summary_of_the_review: The paper has slight novelty and interesting findings but neither novelty nor finding is significant enough. It may be better to be submit it to medical image journals.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates few-shot learning for histology images. So far, contrastive learning has been studied for natural images, which differ from medical images. The paper shows that contrastive pre-training provides a significant performance boost (~ +10% F1 score) compared to standard supervised pre-training in few-shot learning. Further, they introduce a new self-supervised method (called "latent augmentation") for performing augmentation in the feature space created by the model. The prediction model (e.g., logistic regression) trained on the augmented data provides ~ +10% higher F1 score for small and moderate shifts in the pre-training and base dataset (for pre-training the feature extractor) and the support dataset (in the meta-tasks). Overall, the results are very promising, and the performance boost is significant. The paper has the potential to make contrastive learning the standard pre-training methodology for non-natural images and make the "latent augmentation" a common data augmentation technique in few-shot learning.

main_review: **Strengths**
- Great results, showing a consistent and significant performance boost (~ +20% F1 score) few-shot classification of histology images.
- The proposed latent augmentation is simple to implement, and it consistently outperforms the standard input augmentation. It has the potential to be adopted by the community.
- Very well structured and written paper. The paper feels complete.
- Extensive experiments and ablation studies, with sufficient implementation details provided (e.g., MoCo method is explained)
- Extensive discussion of the findings.


**Weaknesses**
- The proposed method, "latent augmentation" isn't compared to other variation data augmentation methods. This makes the reader wonder whether "latent augmentation" is superior to other variation augmentation methods. Why didn't you compare with other " variation augmentation" methods? For example, how well does adding white noise to the latent variables perform?
- Add a small investigation of the shape of the clusters (i.e., the covariance matrix). K-Means corresponds to GMM with an isotropic covariance matrix. If it turns out that isotropic covariance matrices are sufficient, this information would be helpful in scenarios with small clusters (when the covariance approximations are poor).
- How stable is the method to the randomness in K-means? Did you notice any significant performance variation when running K-means with different random seeds?

**Minor comments**
- The experiment (and the caption) of figure 4 is unclear, even though the appendix provides many details. Please refactor the explanation so that it doesn't require reading the appendix (for example, explain what the features are inside the caption).
- It's unclear how l2 feature normalization is used in clustering and downstream meta-tasks. As of now, I understand that l2 feature normalization is applied before doing clustering. However, all feature vectors would be on the unit hypersphere, giving a meaningless cluster covariance matrix. Please clarify.
- Some paragraphs are too general and can be reduced in size. For example, the first paragraph in section 3.1 could be smaller. The fully supervised pre-training (equation 1) is just the standard cross-entropy loss. The few-shot learning section from related works could be reduced in size by a third.
- Replace the formula for choosing the cluster in latent augmentation (penultimate row in section 3.2) with simply saying that i* is selected by using cosine similarity between the representation z and c_i.
- The following terms are ambiguous and should be changed: 
saying that the results "fairly better."
-- "space is shown to be highly linearized" is ambiguous. Do you mean the space allows linear interpolation?
-- "legacy dictionary"
- Does the +- numbers in Table 1 specify the confidence interval or the standard deviation across multiple runs?
- Replace "golden labels" with "true labels"
- Typo: "significantly lower than LA (image augmentations" should be DA instead of LA
- Will the code be released?

summary_of_the_review: The paper shows very promising results for few-shot learning in histology images and introduces a simple and effective method for doing data augmentation. The paper would benefit from some additional experiments with competing augmentation methods and a few straightforward ablations. Overall, the paper is very well written, feels complete, and has an extensive discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper focuses on learning useful representations of histological patches for classification tasks. Acknowledging the large domain shifts typical of histological imaging (e.g., differences in examined organs/tissues, imaging protocols, staining intensities), the authors frame the classification problem as a few-shot learning (FSL) one. In their experiments, they compare FSL pre-training on a base dataset both using full supervision and contrastive learning. During low-shot training, they make use of an unsupervised approach for data augmentation in the latent space (referred to as “latent augmentation”). In the results, they note that contrastive learning with latent augmentation provides the most reliable representations, especially for out-of-domain tasks. 
main_review: Main strengths:
1) The paper focuses on a real limitation which hinders widespread deployment of ML-based pipelines in histological image analysis: the issues relative to out-of-domain adaptation, especially with respect to the introduction of a new label set (i.e., different types of tissues/anomalies). 
2) The proposed approach, which makes large use of some of the latest development in both FSL and data augmentation strategies, is interesting and convincing.
3) Most importantly, the experimental section is comprehensive and well-devised. Three different adaptation scenarios are implemented, with varying degrees of domain adaptation required. 

Main weaknesses:
1) Lack of novelty: while probably novel in their combination and use case, none of the implemented techniques (i.e., contrastive and fully-supervised classification, domain adaptation through FSL) is very novel. The only exception might be “latent augmentation”, which follows in the footsteps of previous latent space augmentation techniques but - differently from most of them - doesn’t require labels. This aspect is interesting, but since the experiments are limited to histological images, it is difficult to assess its impact on image classification in general.
2) Potentially questionable take-home message: amongst their most important contributions, the authors state that “Also to our surprise, we show that state-of-the-art CL models generalize fairly better than the supervised counterparts for histological images”. In their Discussion, they add “The lack of dominant objectiveness makes FSP (Fully-supervised pre-training) tend to overfit severely in high level, but CLP (contrastive-learning pre-training) can capture the overall patterns from the crudely scattered tissues. We believe this is the main reason why CLP models generalize better than FSP models, in pathology.” To me, this is more a statement about the observed outcome rather than an explanation. Additionally, I’m not sure that the results shown by Chen & Li (2020) are directly comparable to these, which are relative to the same comparison but performed in an out-of-domain (i.e., previously unseen class) setting. It seems rather intuitive to me that FSP-learned representations would be unreliable when a patch showing a previously unseen class is presented. In CLP, on the other hand, the goal is learning to identifying similarities and differences in images, which should provide more class-agnostic mid-level features. I would appreciate a more detailed discussion of this comparison.

Minor comments:
1. Typos (page 5): “samples belong”, “we summarizes”
2. Typo (page 15): “oragan”. Also, a “plus” seems missing from the “Out” equation.
3. Typo (page 7): “The trade-off EXISTS”
4. Typos (page 9): “Later methods is attracted to transfer learning”, “This method is later extended by not relying given base samples”
5. Please consider citing and briefly discussing the following paper on latent space augmentation from ICLR 2021: Tsz-Him Cheung, Dit-Yan Yeung. MODALS: Modality-agnostic Automated Data Augmentation in the Latent Space. ICLR 2021 

Chen & Li (2020): https://contrastive-learning.github.io/intriguing
summary_of_the_review: Overall, I vote for acceptance. While the novelty might be relatively limited, the comparison between fully-supervised and contrastive pre-training in an FSL setting (with and without latent augmentations) for histopathological images is interesting. The experimental design and ablations provide a valuable overview of the inner workings of these methods, which could be valuable to a large portion of ICLR’s audience (potentially beyond applications to histopathology).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper addresses few-shot classification for histological images. They compare the generalization capabilities of self-supervised contrastive learning and fully supervised classification as pretraining. They perform experiments to evaluate the impact of latent augmentation vs. data augmentation. They use three publicly available histological datasets (spanning multiple tissues, pathologies and number of available classes). They construct a number of tasks given the tissue labels provided in those datasets and compare the classification performance in N-way-K-shot classification experiments for a number of different N and K.

main_review: The topic of few-shot learning in biomedical applications is of great interest, in particular in histology where whole slide scanners can capture a lot of data, but annotations are scarce and expensive. The datasets used in the study are openly available and cover a large range of variations in tissue, staining and pathologies.

The paper is well structured, but hard to follow. The language/text of the paper should be revised. At various places expressions used are ambiguous which affects the scientific value to the point of incorrectness and makes the steps taken very hard to understand. Articles are often omitted, there are a number of grammatical mistakes and some words are used erroneously throughout the paper (reflect, confront, deprecate...).

Some vocabulary used is not conventional or precise. Some examples: 
These two components [...] can scale gracefully [...]. (abstract)
[...] CLP generalizes fairly better [...]. (sec. 4.2)
Such boost [...] becomes humble in out-domain task. (sec. 4.2)
[...] DA can slightly polish LA [...]. (sec 4.3)

It should be addressed more clearly what is meant by “near-domain”, “mixture-domain”, “middle-domain” and “out-domain”. 
In 4.1 it says “We use faiss to perform K-means with a fix seed.” - it should be pointed out that faiss is a library for clustering.
For readers outside this immediate field of deep learning (e.g. more application oriented bioimage analysts in digital pathology), I suggest to use the same terminology throughout the paper, currently the authors switch for example between “low-shot” and “few-shot” and similar.

At multiple places it was hard to understand what the authors were trying to say, a few examples are: 
in 3.2.1 “From the view of low-data learning, distribution of few samples is not well calibrated, so using established distributions in base class to calibrate untrustworthy novel class can help.”; 
in 4.1 “The rest three lung-related classes are regarded as out-domain novel classes.”; 
in 4.3 “It is worthy to emphasize the computation budget involved in LA (addition in $\mathbb{R}^d$ space) is significantly lower than LA (image augmentation in $\mathbb{R}^{3HW}$ space and encoder forwards)“
In 4.3 “Patches from a single WSI are confronted to its stain degree (impacts color) and body site (impacts pattern and context)” - what does it mean they are confronted to these properties?
In 4.4 “The lack of dominant objectiveness makes FSP tend to overfit severely in high level, but CLP can capture the overall patterns from the crudely scattered tissues.”


Scientific content:
FSL description in Sec. 2: Is $\mathcal{S}$ a subset of $\mathcal{D}$_{novel} and $\mathcal{Q}$ a subset of $\mathcal{X}$_{novel}? If so, the notation needs to be revised. Currently it states they are distributed by $\mathcal{D}$_{novel} and $\mathcal{X}$_{novel} respectively. Furthermore, the sets $\mathcal{S}$ and $\mathcal{Q}$ are indexed by i=1 to NK and i=1 to NQ, but immediately after K and Q are referred to what probably should be NK and NQ.
The authors investigate how augmentation in the latent space compares to data augmentation w.r.t. To F1-score. However, I cannot find the specifications of what kind of data augmentation was used in the main paper, which is a crucial aspect. I eventually found some details in the appendix, but I would like to see some intuition behind the choice of data augmentation (crops, horizontal flips and color jittering) and how it is expected to be comparable to the latent augmentation in terms of adding variety to the dataset. It is common to use more extensive data augmentation - rotations, mirroring, blurr,... I also lack the information if at every iteration one augmentation was drawn at random, or a combination of the augmentations. 
Additionally, the data augmentation used for the fully supervised pretraining (crops, horizontal flips and normalization) was chosen differently than in the ablation study, why?
It is not clear to me what is meant with augmentation times in the experiments of LA vs DA. Is it that the augmentations are being performed on already augmented versions of the original input? If so, how does the additive color jittering affect the images?
How are the patches resized in the LC-25K dataset? What interpolation is used?
Fig. 4, visualizations of CLP features from stage 4 using 4 and 6 clusters for k-means: it looks as if there are three clusters inherent to the data present in the shown image, but for k=4 and k=6, the data is split into 4 or 6 clusters due to how k-means work, but I wonder why those “additional” clusters are distributed along the edges of the image as they are? Are the edge effects in these feature maps that cause this behavior or is it due to initialization of the seeds for k-means? Similar behavior can be observed in the visualizations provided in the appendix.
 
Novelty:
The authors say that few-shot learning “is much unexplored in medical images, especially in histological ones.” and they “pioneer the study of low-shot learning for histology images [...]”. However, there are a number of studies on this topic, some included in the paper’s related work section, but there are other recent works on this topic also, “Learning with Less Data Via Weakly Labeled Patch Classification in Digital Pathology“ by Teh et al. (ISBI 2020), “Supervision and Source Domain Impact on Representation Learning: A Histopathology Case Study” by Sikaroudi et al. (EMBC 2020) to name a few.

summary_of_the_review: The paper addresses few-shot learning for the classification of histological images. This topic is of significant interest to the biomedical community, as annotations are very expensive to obtain. Exploring the generalizability of different pretraining methods and augmentations to exploit the available ground truth to its fullest extent is of great importance. However, I found it very hard to understand the text and followingly some of the experimental setup. On multiple occasions, the phrasing was so ambiguous, that it became unclear to me what the authors tried to convey. The text needs revisions to correct for semantic and grammatical errors. Due to this unclarity, I suggest to reject this submission, but encourage the authors to revise the text and submit to another venue.



------------- After author's response and revision of the submission ------------------
The authors addressed my concerns and questions in detail and the revised submission has improved in clarity. I believe the topic is of significant interest to the biomedical imaging community, which is why I am suggesting to accept the revised submission.
review_title: null

Meta-review: Paper presents an approach and evaluation setting for few-shot learning in histology images. The approach leverages contrastive learning pretraining, and latent augmentation (LA) for data augmentation. The evaluation examines in-domain few-shot learning, mixed domain few-shot learning, and out of domain few-shot learning.  

Latent augmentation is an approach to learn how categories vary between samples within unsupervised clusters in a base dataset, and transfer that variation to the few-shot sampled classes. 

Pros:
- A couple reviewers have claimed as a strength the novelty of the proposed latent augmentation method, but as other reviewers point out, there is much work in this field, some of which wasn't cited (i.e. Delta-Encoder, NeurIPS 2018).
- The latent augmentation method is simple to implement, and outperforms standard input augmentation approaches.
- The paper is rich in content and details of experiments.
- Examining learning over a variety of domain shift settings is interesting.
- Shows contrastive learning can outperform supervised pretraining for this application domain.

Cons:
- Multiple reviewers raise concerns about technical novelty. This work applies mostly previously proposed methods, or variations thereof, to the domain of medical imaging. May be more suited to a medical imaging venue. 
- Some of the results are consistent with prior reports, such as finding that self-supervised learning can outperform supervised pretraining. In that regard the results are not surprising.
- One reviewer raised issues about lack of comparison to other relevant few-shot works. Authors argue that fine-tuning is a competitive baseline. Authors did add comparison to one other variation augmentation approach, distribution calibration. But as mentioned, delta-encoder is a very related work, which has not been cited nor compared against. Biggest difference is that delta-encoder uses labels, but the unsupervised clusters can trivially be supplied as labels in this setting. AC feels authors should have done a more comprehensive comparison to related learned augmentation works. 
- Authors initially did not address how latent augmentation is affected by random seeds, but authors have replied to reviewers with additional data.

Reviewer consensus, excluding 1 reviewer, favors accept, though significant concerns regarding technical novelty and comparisons to other relevant works persist (especially in regards to works that learn how to augment as the proposed LA method does).
Decision: Accept (Poster)