Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper improves the self-supervised contrastive learning framework, specifically MoCo, by making the features in the intermediate layers closer. Although not the first attempt at this idea, this paper dig to investigate the choice of loss terms and the property of learned features. MSE loss and a recently represented Barlow Twins loss are studied. For analyzing learned features, layer-wise probing measures the performance of using intermediate features for downstream tasks (higher means better). Feature reuse is measured by quantifying the similarity of features between models after pertaining, finetuning on labeled data. The idea is that the features change less over finetuning means self-supervised learning learns better features. The output distributions of modes finetuned with different amounts of labels are also compared with the best-performed model as a metric for finetuning with limited labeled data. This paper focuses on medical images in the study.
main_review: Strength:
- This paper explores a nature idea to extend the current contrasting frameworks by adding intermediate features. Two loss terms used in the paper don't need negative pairs, which saves the computation cost. Both losses improve the original MoCo in linear eval and e2e fine-tuning except the NIH chest X-ray dataset and MSE loss works better than Balow twins in more cases. 
- They analyze learned features from a few interesting perspectives and may inspire other research.

Weakness/Concerns/Suggestions:
- It is unclear why 3.4.1 "feature reuse" and 3.4.3 "Feature utility for small datasets choose" different comparison targets: 3.4.1 choose the 100% labels finetuned model for each method while 3.4.3 compare to the best-performed model. I am a bit concerned about the choice of 3.4.3. The distance may not only be related to the performance but also the approach itself. In other words, is it really true that the probability similarity with the best model is strongly correlated with the actual performance? There are counter-cases e.g. in 6% labeled  Breast cancer data, the supervised model has a similar distance with MoCo-MSE and is better than MoCO while it has the worst accuracy. Maybe checking the correlation between the distances and the actual performance is helpful to answer this question.

- It will be helpful if the paper can discuss more (clearly) on that what properties the three analysis can provide about regarding the feature quality that the downstream task performance cannot. In other words, why do people working on self-supervised representation learning want to do these analyses on their approaches?   

- A ablation study on applying the intermediate feature similarity to not all positions is missing. 

- Some details are missing:
  - Why use KS distance but not other metrics e.g. KL divergence?
  - How is the similarity/distance computed. My best guess is the average of the per-sample results over the test set.
  - section 3.3, how do you use 1000 bootstrap replicates to measure the performance? 

Overall, I think this paper presents a reasonable approach to extend MoCo that shows good performance. The other contribution is the three ways of analyzing feature quality. However, authors should highlight/discuss more their value as general metrics for evaluating learned representation if they are not only used for the approach in this paper.  Consider the limited scope of studied model/backbone/datasets (discussed in the conclusion though), I am slightly lean to reject this paper.
  

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose to learn augmentation invariant features early on in a network by, in addition to a global contrastive loss, also optimizing for an intermediate loss function to ensure similarity between features in early layers of the network. They test two loss functions for this purpose - MSE and a minimization of the difference between the cross-correlation matrix of the intermediate representations and the identity matrix. They hypothesize that this way the network learns high quality features early on.
The authors also test the quality of features learnt with their method in a self-supervised setting, by comparing the similarity of the features learnt under self-supervision to features learnt after fine-tuning on a labeled dataset following the self-supervision. They do so for a variety of different labelled set sizes, as well as end-to-end fine-tuning, and only tuning the last layer.

They test their method on three publicly available medical datasets (chest X-Ray, diabetic retinopathy and histopathological whole slide images), which are of different imaging modalities.

main_review: The paper is well structured and well written. The idea is clearly presented, as well as the means how to evaluate the feature quality. The claims are supported by experiments on three available datasets of different modalities which shows that the versatility of the proposed idea to regularize intermediate feature representations by an additional similarity loss term. The concept can be used for any backbone architecture and enforce various invariances of the representations depending on the data augmentation. Seeing how the method shows major improvements in classification performance over the baseline in case of only small labelled datasets, this work could be very valuable, for biomedical tasks in particular.



There are however a number of questions to address in the rebuttal:



It should be stated more clearly in the methodology section how the InfoNCE loss is combined with the loss on the intermediate layers. 

I think it is very relevant to point out precisely what augmentations were used during the self-supervision pretraining (on top of citing Chen et al) as they will lead to the representations’ invariances.

Assuming rotations were part of the augmentation, I wonder if the drop in performance of the proposed method vs only MoCo is due to the orientation of the data in case of the NIH Chest X-Ray dataset? The histopathological images are rotationally and translationally invariant by nature, the diabetic retinopathy images to some extend also, if the meta data available to the dataset (left/right eye, inversion of the photo or not) was not used to correct for it (which is how I understand the dataset was used here for both the MoCo experiments as well as for the proposed method). However, the chest X-Ray images have a set orientation, so learning rotationally invariant features could possibly be counterproductive. Similarly, invariances to color distortion could prohibit the network to pick up on darker or lighter regions in the X-Ray which may be crucial for classification.  Hence my suggestion (possibly for future work) would be to adjust the augmentation such that only invariances are learnt that are in line with the clinical background knowledge available and see if the performance can be further improved that way.

Why were the sizes, in percent, of the training/test/validation sets chosen differently for the DR dataset than the other two datasets?

What interpolation method was used for the resizing? Seeing how the patches from the whole slide images increased its pixel count by more than 26x, resized from 50x50 to 256x256, the interpolation method will play a significant role in the resulting input images.

Why was AUC reported for the DR binary classification task, but the F1-measure for the histopathology dataset? I strongly suggest to report both values for both datasets to make comparison of the improvement of the proposed method feasible. 



Minor remarks: 
It would be good to point out in the caption of Table 3 that CKA was used as a similarity measure.

It would be beneficial to adjust Fig. 1 to the same notation used in Eqn. (1) (views of the same image denoted as $\tilde{x_i}$ in the equation, but as $v$ and $v’$ in the figure for example).

It is unclear to me from the paper what part the “momentum” plays portrayed in Fig. 1.

Some minor spelling/grammar mistakes, mostly with singular/plural or capitalization (lines 17, 23, 34, 38, 68, 158, 166, 180, 201, 204, 222, 228, 240, 247, 248, 278, other instances of X-ray)
Some issues in the citations, e.g Neyshabur et al 2021 missing journal/proceedings

-------------------------------------------------------------------------------------------------------------------
After rebuttal: 
The authors promise to include details which I was missing in the submission, in the camera-ready version and I higher my score to 7.
limitations_and_societal_impact: The authors did not discuss the negative societal impact in sufficient detail. Their method is in no way limited to medical images, i.e. they should reflect upon other data in which self-supervision is key.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper examines the utility of minimizing the distance between the values of intermediate layers in a neural network model, when performing supervised learning with the momentum contrastive (MoCo) method on two augmented versions of the same image. Two metrics for minimizing said distance are tested: mean squared error, and a cross-correlation matrix by the Barlow twins loss function (Zbontar et al., 2021). Experiments on three medical image datasets show that enforcing intermediate layer closeness can produce improve classification performance, especially when the proportion of labelled data for training is small. Additional experiments suggest that intermediate layer features trained in this way are more suited for reuse for various downstream tasks, and are of higher quality.
main_review: 1. It might be clarified whether the intermediate feature similarity enforcements apply only to positive pairs, and the standard MoCo for negative pairs (in their respective encoder) remains unchanged; if so, it might be commented as to whether intermediate features may be exploited for negative pairs too.
2. It is stated that Resnet-50 is the backbone architecture for all baseline self-supervised methods (apparently MoCo) and our approach(es) (+MSE/+Barlow Twins) (Line 222). The architecture and training methodology for the supervised baseline might be stated as well, and justified to be a fair comparison. 
3. For linear probing (LL fine-tuning) as reported in Section 4.3.1/Table 1, what was the label fraction used, and is there any reason why various label fraction values were not investigated, as was done for end-to-end (E2E) fine-tuning? It might also be considered to standardize the format of Table 1 with Table 2 (e.g. “Supervised” column to come first)
4. The class distributions for the 1% and 6% label fraction experiments might be stated. In particular, are the original class distributions maintained? This is of interest as medical image datasets often have highly imbalanced class distributions, and rare classes may thus possibly not be represented when sampling for low label fractions. Ideally, the class label distributions/quantities for each dataset might be provided, for training/validation/test splits, possibly in the supplementary Appendix.
5. It might be interesting to investigate saliency heatmaps for the various methods (supervised, MoCo, MoCo+MSE, MoCo+Barlow Twins) if possible, to further verify whether intermediate layer similarity affects classification attribution.
6. Minor issues:
(Line 62) “have high performance” might be “has high performance”
(Line 86) “We don’t” might be “We do not”
(Line 236) “The” is repeated.
(Line 259) “proliferate diabetic retinopathy” might be “proliferative diabetic retinopathy”
---
We thank the authors for addressing our major concerns in their reply.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an augmentation to MoCo whereby they augment the standard InfoNCE loss with an additional loss that encourages the intermediate layers of a ResNet50 to be close for multiple views of the same image. They demonstrate the efficacy of this loss on several medical imaging classification tasks. 
main_review: 
Originality: 
The idea is a natural extension to a popular technique in a very trendy field. The loss functions used in the intermediate layer are either the first-choice when encouraging similarity (MSE), or a recent development that has been shown to work well (BT). However the use of an MLP projection for the BT loss or the average pooling for MSE are nontrivial and indicate that the authors have had to work a little and not simply do the most-naive thing for these losses. 

Quality: 
The emphasis on purely medical imaging tasks is fairly nonstandard and makes the comparison of the proposed approach to other techniques challenging. When comparing against vanilla MoCo, the intermediate layer losses provide a boost in performance in almost all cases, and surprisingly outperform the fully supervised setting in certain cases. However, I am slightly skeptical about the comparison to the baseline. Intuitively, the training times of models trained with the intermediate-loss will be larger than the vanilla MoCo training times (on a per-epoch basis), and there is no assertion that the vanilla MoCo was trained to maximum convergence. An alternative hypothesis that could explain these results (and thus needs to be ruled out) could be that vanilla MoCo is able to attain the same performance as the proposed methods, but requires more epochs of training; the different training times per epoch, however, could mean that vanilla MoCo can attain this equivalent performance using the same wall-clock time. I would like to see training curves, some notion of wall-clock time, and some argument that the proposed methods actually do outperform MoCo regardless of training duration/hyperparameter tuning. Another way to ameliorate my concern would be to demonstrate the performance of the MoCo+X on datasets more commonly used in the SSL literature. Further, comparisons against other SSL techniques should be performed: is MoCo+MSE better than {SWaV, Barlow Twins, etc...} on these tasks (ignoring the benifits that Moco+X is doesn't require the overhead of negative examples)?

Significance: 
Learning models that are able to extract useful features without labels is very important, and is doubly so in instances where labeled data is hard to obtain (such as in medical applications). While this work seems like a natural training trick without strong theoretical backing, this trick is applicable to many different self-supervised learning schemes and seems to improve the performance with very limited overhead or hyperparameter tuning, and may even provide more useful features. 

Clarity: 
The paper is well-written and easy to follow. 

Overall: I think this paper is marginally above the acceptance threshold, as a cute and easy improvement to self-supervised learning techniques that seems to improve performance in most cases. I'm hesitant to push for a full-accept because of the experimental concerns I pointed out in the 'quality section', and the emphasis on nonstandard datasets that makes a thorough comparison to SOTA hard to tease out. 
limitations_and_societal_impact: The authors have adequately addressed the societal impact and limitations of their work.
review_title: null

Meta-review: This paper proposes a straight-forward but clever and useful extension of MoCo by adding intermediate features. 
A similar approach has been proposed earlier (Gong et al., withdrawn from ICLR, on openreview, cited in the proposed paper) this paper makes a great case for it.  The paper tests on three very different, biomedical and openly available datasets and achieve very good results. They have addressed and promised to include details which the reviewers requested from the original submission so overall this paper was viewed positively by all reviewers and meta-reviewer. 


Decision: Accept (Poster)