Intermediate Layers Matter in Momentum Contrastive Self Supervised Learning
Authors: Aakash Kaku
Sahana Upadhya
Narges Razavian
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep learning
Self-supervised learning
MoCo
Momentum contrastive self supervised learning
Histopathology
Chest Xray
Diabetic Retinopathy
Medical imaging

Abstract: We show that bringing intermediate layers' representations of two augmented versions of an image closer together in self-supervised learning helps to improve the momentum contrastive (MoCo) method. To this end, in addition to the contrastive loss, we minimize the mean squared error between the intermediate layer representations or make their cross-correlation matrix closer to an identity matrix. Both loss objectives either outperform standard MoCo, or achieve similar performances on three diverse medical imaging datasets: NIH-Chest Xrays, Breast Cancer Histopathology, and Diabetic Retinopathy. The gains of the improved MoCo are especially large in a low-labeled data regime (e.g. 1% labeled data) with an average gain of 5% across three datasets. We analyze the models trained using our novel approach via feature similarity analysis and layer-wise probing. Our analysis reveals that models trained via our approach have higher feature reuse compared to a standard MoCo and learn informative features earlier in the network. Finally, by comparing the output probability distribution of models fine-tuned on small versus large labeled data, we conclude that our proposed method of pre-training leads to lower Kolmogorov–Smirnov distance, as compared to a standard MoCo. This provides additional evidence that our proposed method learns more informative features in the pre-training phase which could be leveraged in a low-labeled data regime.

Intermediate Layers Matter in Momentum Contrastive Self Supervised Learning
Aakash Kaku Center for Data Science New York University New York, NY 10011 ark576@nyu edu
Sahana Upadhya
Department of Computer Science
Courant Institute of Mathematical Sciences
New York, NY 10011
su575dnyu. edu
Narges Razavian
Departments of Population Health and Radiology
NYU Grossman School of Medicine and NYU Center for Data Science
New York, NY 10016
Narges Razavian@nyulangaann org
Abstract
We show that bringing intermediate layers' representations of two augmented versions of an image closer together in self supervised learning helps to improve the momentum contrastive (MoCo) method. To this end, in addition to the con- trastive loss, we minimize the mean squared error between the intermediate layer representations make their cross-correlation matrix closer trix. Both loss objectives either outperform standard MoCo, or achieve similar performances three diverse medical imaging datasets: NIH- OhH-Chest Xrays, Breast Cancer Histopathology, and Diabetic Retinopathy. The gains of the improved MoCo especially large in low-labeled data regime (e.g. 1% labeled data) with an average gain of 5% across three datasets. We analyze the models trained using our novel approach via feature similarity analysis and layer-wise probing. Our analysis reveals that models trained via our approach have higher feature reuse compared to standard MoCo and learn informative features earlier the network. Finally, by comparing the output probability distribution of models fine tuned on small versus large labeled data, we conclude that our proposed method of pre-training leads to lower Kolmogorov-Sminno distance, as compared to a standard MoCo. This provides additional evidence that our proposed method learns more informative features in the pre-traue pre-training phase which could be leveraged in a low-labeled data regime.
identity
Introduction
Self-supervised learning (SSL) Noroozi and Favaro 2016 Doersch et al.) 2015 Zhang et al. 2016
He al., 2020 Chen et al. 2020 Caron et al. [2200] Zbootar et al.] [202]] Caron et al.] 2021
helps in learning useful representations from large unlabeled dataset that can be used as pre- traitrdined
initialization points for different downstream tasks. These pretrained models are particularly useful in settings where we have small labeled dataset to learn from. Recent advances have shown that such self-supervised methods are capable of learning representations that are competitive with those learned in fully supervised manner for natural images [Wu et al.) 2018 Hjelm et al.) 2018 Zbontar ctt al.) 202 Caron et al. 2021]. In application domains such as medical imaging, where high quality labeled data scarce and data modalities shift rapidly, SSL has potential to make significant impact.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Contrastive self superviee learning methods [He et al. 2020 Chen et al. 2020 Caron et al. 2020 Zbontar et al., 202T Caron et al. 202T] are currently the state of the art in SSL. In these methods, final layer's feature representations for positive pairs of images are brought closer and those for negative pairs are pushed away using a contrastive loss functions such as InfoNCE [Oord et al.. 2018 or NT--Xent [Chen et al. 2020]. A positive pair of image could be two different views of the same image while a negative pair could be views of different images. These contrastive methods have also been applied to various medical imaging tasks such as chest x-ray interpretation (Sowrirajan et al.
202T Vu et al., 202T]. image segmentation [Chaitanya et al. 2020], covid diagnosis [Chen et al.
2021 and dermatology classification [Azizi et al.) [0221]
In this work, we extend the idea of contrastive learning further and hypothesize that ensuring closeness of representations for the intermediate layers, and not just the last one, significantly improves the
learning of network parameters. To this end, we hypothesize that the features learned by the method should be more reusable for the downstream task than a standard contrastive learning method. We use Momentum Contrastive (MoCo) method (He et al. (020) as the base SSL method, and study two additional loss terms based on closeness of feature representation between intermediate layers. The first loss function is based on mean squared error, and the second loss function is based on distance between cross-correlation matrix and identity matrix [zbontar et al. 2021).
We test our method on three diverse medical datasets:NIH chest x-ray, diabetic retinopathy and breast cancer histopathology. We show that both proposed loss functions either outperform standard MoCo or have comparable performance for classification tasks on all three datasets. The performance gap between the standard MoCo and our approach higher in the case where we have small labeled dataset to fine tune our model. This suggests that the features learned via our self-supervision approach are of higher quality than the standard MoCo. To confirm this finding, we first investigate how much the features are reused by the model, by comparing the similarity of the features before and after fine-tuning on the labeled dataset. Based on prior work [Neyshabur et al., 202T], higher feature reuse indicates better feature quality. Additionally, we also perform layer- wise probing of the model to conclude that indeed the features learned earlier in the model by our approach are more informative than standard MoCo. Finally, to understand the importance of the feature quality for learning from a small labeled dataset, we compare the output probability distribution of models fine tuned on 1 1% and 6% of labeled data versus on 100% of labeled data. The proposed method of pre-training leads to lower as compared to a standard MoCo. To the best of our knowledge, this is the first work to show that the proposed approach has high performance on such diverse non-natural imaging datasets. Additionally, this is the first work to investigate the feature quality of the self-sun supervieed method by going beyond the linear probing and focusing on feature reuse and on learning informative features earlier in the model.
Aaoomongrov-Smrinne
(KS) distance against the best performing model,
2 Relevant work
2.1 Pretext Task based SSL
self-supervised learning helps to learn feature representation of data in an unsupervised manner. This is accomplished by performing a formulated task where the labels for the task extracted from the data itself. Classic examples for handcrafted formulated tasks include solving a jigsaws puzzle (Norooii and Favaro 2016), relative patch prediction (Doersch et al.) 2015) and coloriza- tion IZhang et al. 2016). These tasks are based on heuristics and therefore the representations learned using these tasks may not be useful for performing downstream tasks. Consequently, contrastive meth- ods to learn representations in an unsupervised manner emerged. They outperformed the heuristics task based methods and gradually have also reached a fully supervised level of performance [Caron et al. 202T]. Some of the recent contrastive methods include SimCLR [Chen et al.) 2020). MoCo [e al. 2020]. BYO (Grill et al. [2020]. SwAV [Caron et al. 2020). Barlow Twins [boontar et al.
2021 and Dino [Caron et al. [222].
All the contrastive methods treat the backbone model as a black box and work with the final layer's
feature representations only. A recent work [Chengyue Gong [02T] attempted to incorporate contrastive loss for the intermediate layers of a model trained via MoCo, with a focus on accelerating the model training. They employed partial back-propagation - by randomly choosing layer to back-propage proe-pro from for reducing model computations and training time. In contrast, our work T focuses on investigating the impact of the intermediate loss terms on feature quality, using state of the art analysis methods such as feature re-use, layer- wise probing and model output similarity. We

---

investigate this intermediate loss functions with two distinct formulations (mean squared error, and loss based on cross correlation matrix) and on three diverse medical imaging datasets. We do not employ partial back propagation, and our formulation of intermediate layer loss does not require storing intermediate representations for the negative pairs.
2.2 Contrastive self' supervised learning in medical imaging
Out of above mentioned contrastive methods, MoCo has been widely adopted by the research community across many domains including medical imaging. The momentum contrastive method (MoCo) differentiates from other contrastive methods by having an additional memory bank of negative examples. The memory bank helps the model to learn even when the batch size is small, and is especially important for medical imaging tasks, because medical images are typically large (e.g. 1024 1024) and it becomes computationally infeasible to train with large batch size. Increasingly, many recent works have used momentum contrastive method with minor modifications as the choice of self-supervised method perform diverse medical imaging tasks such as histopathology classification [Ciga et al. 2020 Dehaene et al. 22200, chest x-ray interpretation [Sowrirajan et al.) 2021) and dermatology classification [Azizi et al. [02T]. Given widespread adoption of the momentum contrastive method, we focus on it in our work.
3 Methodology
In section 3.1 first, we briefly discuss the standard MoCo method. Then, describe proposed approach in secction3.2 In section we describe the fine tuning and evaluation methodology for the self-supervised methods. Finally, in section 3.4 we describe the methods used for feature analysis which helps us quantify the quality of the representations learned by different methods.
3.1 Standard MoCo
With the help of MoCo, we want to learn feature representation for images in an unsupervised manner. We achieve this by minimizing InfoNCE loss [Oord et al., 2018
exp(g(E1).
LInfONCE(1) -log
exp(g(zi).M + Σ exp (g(Z)),((E)/T))
where j and 122 are different views/augmentations of the same image x. कi and 2i are called positive pair. There are K negative pairs of 4i and E,S where & is random augmentation of a different image. E, either comes from queue in the case of MoCo or a minibatch in the case of SimCLR [Chen] et al. 2020). We minmmize equation to optimize the parameters of the encoder g. h is an identical copy of g whose parameters are an exponential moving average of g. T is a temperature-scaling hyper-parameter.
3.2 MoCo with intermediate feature similarity
In our work, we extend the idea of contrastive learning and propose that the encoder should be encouraged to learn also for intermediate layers. We make of an additional loss function ensure closeness in the representations for the intermediate layers. Such intermediate loss have shown to improve performance of deep learning models such as InceptionNet (Szegedy al. 2015). PSPnet (Zhao et al.) 2017] and DART's [Liu et al.) 2018). Intermediate loss helps to improve performance for various reasons. In Mostajabi al. [2018) it helped regularizer. In InceptionNet, PSPnet and DARTs, it improved training of the model by facilitating gradient flow. In our case, we hypothesize that it would facilitate learning of augmentation-invant
representations not only at the end of the encoder but
features early in the model thereby contributing to learning of high quality features for the downstream tasks.
We experiment with two intermediate loss functions that ensure closeness of representations: mean square error, and a loss function inspired by Zbontar et al. [2021) that minimizes the difference between cross-correlatio matrix of representations and an identity matrix. Figure 3.2 includes an overview of our approach.
3.2.1 Mean squared error of intermediate features
We compute the mean squared error of the intermediate features between the two views of the same image. We extract the intermediate features and pass them through adaptive average pooling layer
3

---

Memory Bank 2)
view
lpprr
1/1
embedding
Contrastive Loss
2
lastre
3/2
MSEiBarlow Twins
Figure I: Schematic diagram for our approach of supe-supe learning. In addition to contrastive loss, we use intermediate loss like mean squared error (MSE) or Barlow twins loss bring interme- diate representations of the two views of same image closer together. The "momentum" in the figure denotes that the weights of the second encoder are an exponential moving average of the weights of the first encoder.
to reduce the spatial variances in the two views that may be caused due to augmentations such as rotations. For ResNet-50 the backbone architecture, the spatial resolutions of the features after the average pooling layers are 16 x 16, 16 x 16, 4 x and 4 x 4 for "Block 1", "Block 2", "Block 3" and "Block 4" of ResNet-50, respectively. The mean squared error is calculated between these spatially down-sampled features of the two views for each block.
3.2.2 Cross-correlation based loss function for intermediate features
We took inspiration from Zbontar et [02T] for this loss function. The loss function is termed Barlow Twins (BT), and is designed to bring the cross-correlation matrix of two feature representations close to an identity matrix. The loss function is defined follows:
LBT = C1- ΣΣCH J7H
C.p
2
B EB
26,
A
Cis = VLA (=A)) V (=
redundancy roduction term where, is weight that trades off between the invariance term and redundancy term. Each CE [-1, is the cross-correlation matrix of two feature representations of d dimension. Here, batch of intermediate feature representations for two augmented versions of the
invariance
= and = same image with batch size b. and j are feature components of d dimensional feature.
The invariance term tries to equate the diagonal entries the correlation matrix to one, thereby making the representations invariant to augmentations. The redundancy term tries to equate the non- diagonal terms to zero and thereby decorrelating the different components of the feature representation. This decorrelation helps to reduce redundancy encoded in different components of the representation.
We follow Zbontar et 202 's procedure for implementing the Barlow twins loss function. We pass the intermediate feature representation of dimension CC xl through global average pooling and obtain feature vector IR dimension. Here, C' = number of channels, H = height of the each channel, and W = width of each channel. The resultant feature vector is passed through three linear layers with ouput dimension being 2048 for each linear layer. The first two linear layers are followed by ReLU non-liny and batch normalization. The resultant output representations are fed to the Barlow twins loss function.
Intermediate feature similarity applies only to positive pairs. For both loss functions, we not need negative pairs to learn the representations. We, therefore, do not have to store the intermediate representations for the negative pairs which could be computationally expensive.
3.3 Model fine tuning methodology
In order to evaluate the utility of the representations learned by models using our approach and standard MoCo, we use the pre-trained models to perform downstream classification tasks. While fine tuning the models for performing the downstream tasks, we perform two types of fine tuning: fine tuning only the final linear classifier (LL-fine-tuning) fine tuning the entire model (E2E

---

fine-tuning). We fine tume the model using different fraction of labeled training data i.e. 1%, 6% and 100%. For example, if model is fine tuned with 1% label fraction, the model will have access to only 1% of the labels of the training data which was used for self model. We ensured that the class distribution of the 1% and 6% label fraction matches the 100% label fraction training data (see Appendix [). Fine tuned model with best validation performance is used for obtaining predictions for test set. We report all the performance measures with 95% confidence interval which we calculate using 1000 bootstrap replicates of the test set. We calculate the upper and lower confidence limit using fr + 1.960 / /N where μ and a are mean and standard deviation of the performance measure for 1000 bootstrap replicates, respectively. N is number of bootstrap replicates.
pre-training of the
3.4 Analysis of the features
We analyze the features of the models learned via self-supervised methods to understand their quality for the downstream tasks. To objectively measure the quality, we investigate the re-usability of the features. Additionally, we also probe the pre-trained models in a layer-wise manner to understand how informative the intermediate features for the downstream tasks are. Finally, we compute the KS distance between the distributions of output probability of the model fine tuned on small vs large data. This shows us the utility of different self-supervised methods when there is a small labeled dataset available for learning the downstream task.
3.4.1 Feature reuse
Higher feature reuse indicates better feature quality (Neyshabur al. 2021] To assess feature reuse in SSL, we measure the similarity of the features of the models trained by our proposed method, before and after fine tuning them on a labeled dataset. We compute the similarity of features extracted from different layers of the pre-trained and E2E-fine-tuned models. Intuitively, if two models pre- trained via self-supervised learning reach a similar performance after fine-tuning on a downstream task, the SSL method that leads to the model with features closer to the fine-tuned model has learned better/mooreusseful features during the self supervised learning.
Following Neyshabur et al.] 202T], we use Centered Kernal Alignment (CKA) (Kombliihh al., 2019] as the measure of feature similarity. CKA has desirable properties such as invariance to invertible linear transformations, orthogonal transformations and isotopic scaling. We use CKA with RBF kernel and = 0.8 following prior work. To compute the feature similarity, we extract the intermediate features from the model which has a dimensions of RC where, C == number of channels, H and W are height and width of each channel. We pass each channel through global average pooling layer and use the resultant vector of IRC dimension for computing feature similarity.
3.4.2 Layer-wise probing
We probe the intermediate layers of the pre-trained models to understand how informative the intermediate features are for performing the downstream tasks. We expect the intermediate features learned via our approach to be more informative than the standard MoCo, because by bringing the intermediate features of two augmented version of an image closer together, our approach encourages the model to learn augmentation- invariant features not only late in the network but at all levels.
Procedure for layer-wise probing: We extract the intermediate features from the model which has a dimensions of Hx We pass it through global average pooling layer and use the resultant feature vector of size RC for performing the downstream task. The feature vector is finally passed through a linear classifier. The entire network is frozen except the final linear classifier. Since all our experiments are performed on ResNet-50 as the backbone architecture, we extract features at the end of each ResNet block namely, "Block 1", "Block 2", "Block 3" and "Block 4".
3.4.3 Feature utility for small datasets
In medical domain, we typically encounter a small labeled dataset from which we want to learn a generalizable model. This is often challenging and in such scenarios, self-supervised pre-trained models can be useful. In this analysis, we investigate how close a self-supervised model, fine tuned on a small labeled dataset, gets to one that utilizes a larger labeled dataset. To this end, we compare the distributions of the output probability of models, fine tuned on small versus large labeled data (KS) distance (Massey Jr. 1951]. KS distance between two distributions is the greatest separation between their cumulative distribution functions (CDFs). We compute KS distance between models trained via our SSL methods, fine tuned on 1% and 6% of the labeled training data, and the best model fine-tuned on 100% of the labeled training data (Detailed procedure
using Bolmogorov-Smirme
5

---

to compute KS distance is described in Appendix G KS distance not only depends on the difference in the performance of the two models but also on how similatly/diffrently calibrated the two models are [Gupta et al. 2020]. Therefore, this metric goes beyond the standard linear probing evaluation as also takes into account the model calibration. Lower KS distance indicates better method for self supervised learning, when only small labeled dataset is available.
Experiment and Results
4.1 Experimental settings
We use Resnet-50 as the backbone architecture for all the baseline self-supervised methods, supervised models and our approach. We use pytorch-lightning' 's [Falcon et al. [019] implementation of MoCo as the basis for our analysis. A preliminary ablation analysis showed that applying intermediate loss to all blocks yielded superior results (See Appendix FF. Therefore, for our approach, we apply the intermediate feature loss after each of the four ResNet blocks. Specifically, the four intermediate feature losses from each block of the Resnet-50 encoder are added to the InfoNCE loss using a scaling factor. The scaling factor multiplies the intermediate feature loss by 0.25 (in case of MSE loss) or by 5e-5 (in case of BT loss). We chose these scaling factors so that the contrastive loss and intermediate losses are in the same order of magnitude for the initial epochs.
The training methodology for the supervised models is similar to E2E finetuming of the self- supervied models. The only difference being in E2E finetuning of models, we initialize the model weights with the SSL learned weights whereas for supervised models, we randomly intialized the weights.
To ensure that the improvement for our approach over the vanilla MoCo (baseline method) isn't due to different training times, we trained the baseline method to convergence (See Appendix [ The baseline method and our approach are pretrained for same number of epochs, and same batch size. These values are different for different datasets and are mentioned in section 4.2 The details of the data augmentations used for training the models are mentioned in Appendix parameters such learning rate, embedding dimension, number of negative pairs, temperature scaling, encoder momentum, weight decay and sgd momentum are kept same for all the models. The value for the above hyper-parameters are as follows: learning rate 0.3, embedding dimension 128, number of negative pairs = 65536, temperature scaling = 0.07, encoder momentum = 0.99, weight decay= le-4 and sgd momentum 0.9. We ran all our experiments on Nvidia Tesla V100 GPU with a memory of 32 GB. The code is available at https://// github. com/ aakashrkaku intermdiate laaee_llaeee_____ra_s_ mmatteretssssss
Other hyper-
4.2 Datasets
4.2.1
NIH chest x-ray dataset
NIH Chest X-ray dataset [Wang et al. 2017] has 1 112,120 x-ray images with disease labels from 30,805
unique patients. Authors use natural language processing and the associated radiological reports to text-mine disease labels. There are 14 disease labels: Atelectasis (10416 images), Cardiomegaly (2532
images), Effusion (12015 images), Infiltration (17852 images), Mass (5121 images). Nodule (5710 images), Pneumonia (1220 images). Pneumothorax (4794 images), Consolidation (4220 images), Edema (2103 images). Emphysema (2308 images), Fibrosis (1520 images), Pleural Thickening (3013 images) and Hernia (186 images), and one label for "No Findings" " (54333 images) associated with each x-ray. We use full resolution images of size 1024 x 1024 to train models. The downstream task is to predict binary label for each of the 14 disease labels for each x-ray. The data is split, by patient, into training (70%), validation (10%) and test (20%) sets. We evaluate the model performance using the macro average area under the receiver operating characteristics curve (AUROC) for 14 classes because the prior works [Wagg et al. 2017 Seyyed-Kalantee et al.. 2020) report the same. We also report the AUROC for each class for different self-supervised methods in Appendix and Appendix[B [.. We trained the models by all self supervised methods until the learning saturates which happens around 50 epochs with batch size of 16.
4.2.2 Diabetic retinopathy (DR) dataset
The EyePACS diabetic retinopathy (Voets et al. 2019b Kaggle 2015) dataset consists of 88,702 retinal fundus images. The training and validation set consists 57,146 images with a 20% split for validation set, and the test set consists of 8790 images. We resize the images to 299 x 299 for training models. Each image has been rated by a clinician on a scale of 0-4, ranging from no diabetic retinopathy to proliferative diabetic retinopathy. Labels up to I are considered non-referable DR

---

788,784 images) and 2-4 are considered referable DR (rDR) (17,152 images). The downstream task is a binary classification task to predict referable vs non-referable diabetic retinopathy. We evaluate the model performance using macro average area under the receiver operating characteristics (AUROC) curve for the 2 classes because the prior works [Voets al.) 2019a Gulshan et al., (016) working on this dataset report the same. We trained all the self supervised models until the learning saturates which happens around 100 epochs with a batch size of 32.
4.2.3 Breast cancer histopathology dataset
Invasive ductal carcinoma (IDC) is the most common form of breast cancer in women. The dataset (Janowezyk and Madabhushi, 2016 Cruz-Roa et al. 2014] consists of 277,524 patches (198,738 IDC negative and 78,786 IDC positive) of size 50 x 50 from 162 whole slide images of the tumor biopsies. We resize the images to 256 x 256 for training models. We use Bilinear interpolation to resize the 50 x 50 patch of breast cancer slides 256 256. The downstream task is to predict a binary label for each patch. The data is split based on the whole slide images, into training (70%), validation (10%) and test (20%) sets. We evaluate the model performance using the F1-measure. F1-measure is the harmonic mean of precision and recall. The prior works (Janowczyk and Madabhushi [016 Cruz-Roa et al.) 2014) working this dataset also reported the F1-measure which makes the comparison easier. For readers' reference, we also report the AUROC score in the Appendix H We trained all the self supervised models until the learning saturates which happens around 75 epochs with a batch size of 32.
4.3 Results
4.3.1
Performance for linear probing (LL fine-tuning) of models
Table shows the results for linear probing of different self-supervised methods. We see that MoCo MSE significantly outperforms other methods for diabetic retinopathy and breast cancer histopathology datasets. For NIH chest x-ray dataset, it is marginally better than a standard MoCo but not statistically significant. Table I also shows the performance of fully supervised models for reference.
Table 1: Performance of sele-super methods for the downstream task after fine tuning only the linear classifier: For diabetic retinopathy and breast cancer histopathology datasets, MoCo + MSE significantly outperforms the standard MoCo. Whereas for NIH chest x-ray dataset, MoCo MSE and MoCo lead to comparable performance.
MoCo Barlow Twins
Dataset / Method
MoCo
MoCo + MSE
Supervised
NIH Chest X- X-ray (AUC (95% CT)
74.4
74.8
73.5
79.8
(73.9-75.0)
(74.2-75.4)
(72.9-74.0)
(79.2-80.3)
Diabetic Retinopathy (AUC (95% CT)
74.6
84.8
79.7
94.1
(74.5-74.7)
(84.6-85.0)
(79.6-79.7)
(94.1-94.2)
Breast Cancer Histopathology (Fl-score (95% CT)
80.7
82.5
82.3
82.7
(80.4-81.1)
(82.2 82.9)
(82.0-82.7)
(82.4-83.1)
4.3.2 Performance for E2E fine tuning of models
Table [4 shows the results for E2E fine tuning of models trained via different self-supervised methods, fine tuned over different fractions of the labeled data (label fractions). All the models pre- trained via self-supervised learning methods outperform the models trained via standard supervised learning, indicating that self supervised learning provides better initialization for the downstream classification tasks. For smaller labeled datasets (1% and 6% label fractions), models trained via MoCo + MSE and MoCo + Barlow twins significantly outperform the models trained via a standard MoCo. Compared to standard MoCo, our approach leads to an average performance gain of 5% and 2% for 1% and 6% label fractions, respectively, across three datasets. For NIH chest x-ray dataset trained with 100% label fraction, standard MoCo leads to marginally better performance than MoCo + MSE but this difference is not statistically significant.
7

---

Table 2: Performance of models trained via supervie learning methods in downstream classification tasks,
after E2E fine tuning different label fractions. For smaller labeled dataset (1% and 6% label fractions), MoCo + MSE and MoCo Barlow twins lead significantly better performance compared standard MoCo. For 100% label fraction, except NTH chest x-ray dataset, MoCo + MSE and MoCo + Barlow twins lead to better
performance compared to standard MoCo.
Label fraction
MoCo
MoCo + MSE
MoCo + Barlow Twins
Supervised
IIHI Chest X-ray (AUC (95% CI))
100%
82.4 (81.7-83.0)
81.5 (80.9-82.1)
80.0 (79.5-807)
79.8 (79.2-80.3)
6%
69.8 (69.3-70.4)
70.5 (69.9-71.0)
70.0 (69.2-70.6)
65.2 (64.6-65.8)
1%
59.2 (58.6-59.9)
61.4 (60.7-62.0)
62.9 (62.3-63.5)
57.8 (57.2-5884))
Diabetic Retinopathy (AUC (95% CT)
100%
94.6 (94.3-94.6
96.6 (96.6-9..7)
95.7 (95.7-95.8) 8)
94.1 (94,1 1-9.2)
6%
92.4 (92.2-92.6)
95.1 (94.8-95.2)
94.0 (94.0-94.3)
69.1 (69.0-69.2)
1%
88.1 (88.1-88.4)
93.6 (93.2-93.6)
92.5 (92.2-92.7)
65.5 (65.4-65.6)
Breast Cancer Histopathology (F) -score (95% CT))
100%
82.9 (82.6-8333)
85.7 (85.4-86.0)
86.4 (86.1-86.7)
82.7 (82.4-83.1)
6%
82.8 (82.4-83.2)
84,6 (84.2-84.9)
84.5 (84.2-84.)
82.7 (82.4-83.1)
1%
82.8 (82.5-83.2)
85.1 (84.7-85.4)
84.4 (84.1-84.7)
80.6 (80.3-81.0)
4.4 Features analysis
4.4.1 Feature reuse
Each row of 3 the feature similarity between pre-trained model and E2E fine tuned model for different intermediate layers. We perform this analysis for small labeled datasets because those are the datasets where we would like to leverage the features learned via supervipeiiiion In that for almost all the blocks, models trained via MoCo+MSE and MoCo+Barlow twins have a higher feature reuse compared to the model trained via a standard MoCo. Additionally, the table shows that features from the earlier blocks are more similar before and after finetuning, compared to the features in the later blocks for all the self-supervised learning methods. This observation is consistent with Asano et al. [2019] which stated that self-supervised learning methods help to learn the low-level statistics of the data. These low-level image statistics are captured in the initial layers of the model.
Table 3
Table 3: Feature Reuse: Feature similarity (measure using CKA) for different intermediate layers before and after E2E fine tuning of the models. Models trained via MoCo MSE and MoCo Barlow twins show higher feature similarity and higher performance than those trained via the standard MoCo. This suggests the features learned via MoCo + MSF and MoCo + Barlow twins are more reusable in various downstream tasks.
1% labeled data
6% labeled data
Method
Block
Block
Block 3
Block
Performance Block
Black 2
Block 3
Block 4
Performance
NIH Chest X-ray (Pertormance in AUC)
McCo
0.81
0.57
59,2
0.72
0.75
0.39
69.8
MoCo MSE
0.97
0.83
0.65
0.42
61.4
0.97
0.84
0.55
0.43
70.5
McCo+ + Barlow Twins
0.99
0,98
0.76
0.38
62.9
0.97
0.92
0.79
0.41
70.0
Diabetic Retinopatlly (Performance in AUC)
MoCo
0.87
0,80
0.51
0.19
88.1
0.81
0.69
0.50
0.14
92.4
MoCo MSE
0.96
0.78
0.33
0.26
93.6
0.95
0.73
0.25
0.12
95.1
McCo Barlow Twins
0.98
0.83
0.58
0.24
92.5
0.93
0.75
0.50
0.15
94.0
Breast Cancer Histopathology (Performance in F1-score)
MoCo
0.50
0.55
0.98
0.16
82.8
0.57
0.48
0.97
0.19
82.8
MoCo MSE
0.77
0.82
0.58
0.42
85.1
0.75
0.79
0.55
0.31
84.6
McCo + Darlow Twins
0.77
0.74
0.54
0.36
84.4
0.76
0.76
0.58
0.38
84.5
4.4.2 Layer-wise probing
Table shows the performance of models trained via different self-supervised learning methods when their intermediate features are used to perform the downstream task. In this analysis, we only train a linear classifier on the top of extracted intermediate features to perform the downstream tasks. Except for Block I of NIH chest x-ray, for rest of the blocks and datasets, the intermediate features of models trained via either MoCo+MSE or MoCo+Barlow twins outperform the model trained via a standard MoCo. This indicates that these models have better quality intermediate features.
8

---

Table 4: Layer-wise probing: Performance of models trained via self-supervised learning methods in the
downstream classification tasks, using the intermediate features of the models. Training model via MoCo + MSE and MoCo + Barlow twins leads to higher performance compared to standard MoCo, indicating that more informative features are learned via these methods.
Block 1
Block 2
Block 3
Block 4
NIH Chest X-ray (AUC (95% CI))
MoCo
58.8 (58.4-59.3)
59.5 (59. 0-600)))0)
65.3 (64.8-65.8)
74.4 (73.9-75.0)
MoCo MSE
57.6 (56.9-58.3)
59.9 (59.4-60.4)
69.2 (68.70-69.7)
74.8 (74.2-75.4)
MoCo + Barlow Twins
56.6 (56.2-57.0)
56.5 (56.1-56.9)
64.2 (63.7-64.6)
73.5 (72.9-74.0)
Diabetic Retinopathy (AUC (95% CI))
MoCo
68.1 (68.0-68 I)
68.2 (68.2-68 (68.2-68.3)
69.2 (69.2-69.5)
74.6 (74.5774.7)
MoCo MSE
68.3 (68.2-68.3)
70.1 (70.0-70.1)
71.2 (71.1-71.3)
84.8 (84.6-85.0)
MoCo + Barlow Twins
67.2 (67.2-67.3)
68.6 (68.5-68.7)
69.9 (69.4-69.9)
79.7 (79.6-79.7)
Breast Cancer Histopathology (F1 -ccoe
(95% CD)
MoCo
80.9 (80.5-81.3)
81.1 (80.8-81.5)
81.1 (80.7-81.5)
80.7 (80.4-81.1)
MoCo MSE
80.6 (80.2-81.0)
81.3 (81.0-81.7)
82.7 (82.4-83.)
82.5 (82.2-82.9)
MoCo Barlow Twins
81.0 (90.7-81.4)
81.1 (80.7-81.5)
82.2 (819-826)
82.3 (82.0-82.7)
4.4.3 Probability comparison
5 ssowss the KS distance between the output probabilities of E2E fine tuned models on small labeled datasets and the best performing models. The best performing models for different datasets are as follows: NIH chest x-ray dataset a model trained via standard MoCo and fine tuned on 100% label fraction; Diabetic retinopathy dataset = model trained via MoCo + MSE fine and tuned on 100% label fraction and breast cancer histopathology a model trained via MoCo + Barlow twins fine tuned on 100% label fraction. For small labeled data, Table 5 shows that models trained via MoCo + MSE and MoCo + Barlow twins and E2E fine tuned on small data are closer to the best performing models compared to standard MoCo.
Table 5: Soaamonnrov-Smrinnn distance hetween the output probability distribution of the best performing
models, and the models trained via self-supervised learning methods and E2F fine tuned on 1% or 6% label fractions. Except for NIH chest x-ray dataset with 6% label fraction, MoCo MSE and MoCo Barlow twins lead to models that are closer to the best performing model, compared to standard MoCo.
Label fraction
MoCo
MoCo + MSE
MoCo + Barlow Twins
Supervised
NIH Chest X-ray (Comperdd to MoDo- Fine tanel on 100% laboled dat)
6%
0.028
0.039
0.034
0.040
1%
0.244
0.094
0.104
0.260
Diabetic Retinopathy (Compared MoCo - MSE -inee toned on 1006 labeled dam)
6%
0.21
0.12
0.30
0.37
1%
0.31
0.18
0.22
0.60
Breast Cancer Histopathology (Compered to MoDo+ Barlow rwins Fine nuned 100% lsheled dets)
6%
0.082
0.036
0.026
0.040
1%
0.043
0.082
0.033
0.155
5
Discussion
The results from Table 1 and Table 14 show that the proposed approach of minimizing intermediate feature dissimilarity improves performance of the models for diverse downstream tasks and on diverse medical image datasets, as compared to standard MoCo. The gains are even larger for a setting where we have small labeled data (1% and 6% label fraction) to fine tune the models. The small labeled data settings are common medical imaging domains, as expert manual effort is needed to annotate the images, leading to expensive and time-consuming labeling process. With this context, the gains achieved by our approach for small labeled datasets becomes more important.
Going beyond only assessing SSL. in terms of predictive performance of downstream tasks, and using novel methods such as feature reuse analysis, layer-wise probing and probability distribution

---

comparisons, further helped us analyse the self self-supervised learning methods, and quantify the quality of features learned via each self-supervision approach. Our finding suggests that better self supervised learning methods lead to objectively better features, in terms of reuse and informativeness.
Limitations: (1) Although we showed that the proposed approach improves performance for diverse datasets, due to computation resource constraints, we only showed it for one backbone architecture (ResNet-50) and one self-supervised baseline method (MoCo). But our approach of using intermediate layers to learn better features is simple and can be adapted to other backbone architectures and self- supervised learning methods. It would be interesting to investigate if our findings hold for other backbone architectures and self-supervised learning methods. (2) Our main interest was medical imaging, but our finding may also hold true in natural imaging domain. Due to computational constraints we did not test this method on ImageNet.
Potential negative societal impact: Our work primarily focuses on medical imaging. Artificial intelligence (AI) in medicine is still in early stage. Hence, any commercial deployment of the tools/models presented our work should not be done without sufficient evaluations. Proper FDA and other regulatory approval should be received to ensure appropriate use of the tools/models presented in our work.
Conclusion
We found that our proposed self-supervised learning method with the additional MSE or Barlow twins loss in the intermediate layers generates higher quality representations than the standard MoCo in three different medical imaging tasks. Our proposed methods also improved the performance of the models downstream classification tasks, especially on low label fractions of the data. We evaluated the quality of the learned features by measuring the feature similarity before and after finetuning the model and probing the intermediate layers the pre-trained model. The results from our proposed method indicates that the representations learned by the models are more reusable and informative compared to the standard MoCo. Finally, we showed that the models fine-tuned on low label fractions with the pretrained weights from method get closer to the best performing model compared to standard MoCo, based on KS distance between output probability distributions.
To the best of our knowledge, this work the first to show the importance of intermediate layers in momentum contrastive self superviedd learning and in diverse set of medical imaging tasks. Despite the differences in the modalities of data, sulf-supervision via MoCo+MSE/Barlow twins leads to models that consistently outperformed or had similar performance on the downstream tasks.
Acknowledgements
We would like to thank NYU Langone HPC team for assisting us with our computational needs. This work was supported by NYU Langone Health, Predictive Analytics Unit, and NYU Langone Radiology Department (NR) and NSF NRT-HDR Award 1922658 (AK).
References
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, or what we can learn from single image. arXiv preprint arXiv:1904.13 13132, 2019.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big supervised models 2021.
advance medical image classification. arXiv preprint arXiv:2101.0522
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. arXiv preprint arXiv:2006. 10511, 10571, 2020.
10

---

Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pages 1597-1607. PMLR, 2020.
Xiaocong Chen, Lina Yao, Tao Zhou, Jinming Dong, and Yu Zhang. Momentum contrastive learning for few-shot covid-19 diagnosis from chest et images. Pattern Recognition, 113:107826, 2021.
Giang Liu Chengyue Gong, Xingehao Liu. Fast training of contrastive learning with intermediate contrastive loss. http:://ppeerrrveeeenneodddddddp
2021.
Ozan Ciga, Anne I. Martel, and Tony Xu. Self supervised contrastive learning for digital histopathol- ogy. arXiv preprint arXiv:201 13971, 2020.
Angel Cruz-Roa, Ajay Basavanhally, Fabio González, Hannah Gilmore, Michael Feldman, Shridar Ganesan, Natalie Shih, John Tomaszewski, and Anant Madabhushi. Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks. Medical Imaging 2014: Digital Pathology, volume 9041, page 904103. International Society for Optics and Photomics, 2014.
Olivier Dehaene, Axel Camara, Olivier Moindrot, Axel de Lavergne, and Pierre Courtiol. Self- supervision closes the gap between weak and strong supervision in histology. arXiv preprint arXiv:2012.03583, 2020.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
William Falcon et al. Pytorch lightning. GitHub. Note: httpss//jihuuaccoo//yyyyytttiiit lightingg, 3, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284, Curran Associates, Inc., 2020. URL https: http::::////cceed/eeeeeee neurips cc/ .pff
Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama, 316(22):2402-22) 2410, 2016.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. arXiv preprint arXiv:2006.12800 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729- 9738,, 2020.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808 06670, 2018.
Andrew Janowezyk and Anant Madabhushi. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. Journal of pathology informatics, 2016.
Kaggle. Diabetic retinopathy detection (data), 2015. Available from: https://www. www.w.a.gl....mm com/ crdiabetic-eetiinpatyyydedeeeete
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural
network representations revisited. International Conference on Machine Learning, pages 3519-3529, PMLR. 2019.

---

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 10X0:1880600905,, 2018.
Frank Massey Jr. The kolmogorov-smmrnox test for goodness of fit. Journal of the American statistical Association, 46(253):68-78, 1951.
Mohammadreza Mostajabi, Michael Maire, and Gregory Shakhnarovich. Regularizing deep networks by modeling and predicting label structure. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5629-5638, 2018.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?, 2021.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European conference on computer vision, pages 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Laleh Seyyed-Kalantari, Kalantari, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: Proceedings of the Pacific Symposium, pages 232-243. World Scientific, 2020.
Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar. Moco-exr: Moco pretraining improves representation and transfcrability of chest x-ray models. 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
Mike Vocts, Kajsa Møllersen, and Lars Ailo Bongo Reproduction study using public data of: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. PloS one, ((6)):0211541, 2019a.
Mike Voets, Kajsa Møllersen, and Lars Ailo Bongo Reproduction study using public data of: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. PLOS ONE, 14(6), 2019b.
doi: 10. 1..337//ournal.ponee 0217541.
Yen Nhi Truong Vu, Richard Wang, Niranjan Balachandar, Can Liu, Andrew Y Ng, and Pranav Rajpurkar. Medaug: Contrastive learning leveraging patient metadata improves representations for chest x-ray interpretation. arXiv preprint arXiv:2102.10663, 2021.
Xiaosong Wang, Yifan Peng. Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032.00 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2881-2890, 2017.
12


Reviews:
[{"review_id": "teIifEFWlhb", "reviewer": "Reviewer_aDCN", "summary": "The authors propose an augmentation to MoCo whereby they augment the standard InfoNCE loss with an additional loss that encourages the intermediate layers of a ResNet50 to be close for multiple views of the same image. They demonstrate the efficacy of this loss on several medical imaging classification tasks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality: \nThe idea is a natural extension to a popular technique in a very trendy field. The loss functions used in the intermediate layer are either the first-choice when encouraging similarity (MSE), or a recent development that has been shown to work well (BT). However the use of an MLP projection for the BT loss or the average pooling for MSE are nontrivial and indicate that the authors have had to work a little and not simply do the most-naive thing for these losses. \n\nQuality: \nThe emphasis on purely medical imaging tasks is fairly nonstandard and makes the comparison of the proposed approach to other techniques challenging. When comparing against vanilla MoCo, the intermediate layer losses provide a boost in performance in almost all cases, and surprisingly outperform the fully supervised setting in certain cases. However, I am slightly skeptical about the comparison to the baseline. Intuitively, the training times of models trained with the intermediate-loss will be larger than the vanilla MoCo training times (on a per-epoch basis), and there is no assertion that the vanilla MoCo was trained to maximum convergence. An alternative hypothesis that could explain these results (and thus needs to be ruled out) could be that vanilla MoCo is able to attain the same performance as the proposed methods, but requires more epochs of training; the different training times per epoch, however, could mean that vanilla MoCo can attain this equivalent performance using the same wall-clock time. I would like to see training curves, some notion of wall-clock time, and some argument that the proposed methods actually do outperform MoCo regardless of training duration/hyperparameter tuning. Another way to ameliorate my concern would be to demonstrate the performance of the MoCo+X on datasets more commonly used in the SSL literature. Further, comparisons against other SSL techniques should be performed: is MoCo+MSE better than {SWaV, Barlow Twins, etc...} on these tasks (ignoring the benifits that Moco+X is doesn't require the overhead of negative examples)?\n\nSignificance: \nLearning models that are able to extract useful features without labels is very important, and is doubly so in instances where labeled data is hard to obtain (such as in medical applications). While this work seems like a natural training trick without strong theoretical backing, this trick is applicable to many different self-supervised learning schemes and seems to improve the performance with very limited overhead or hyperparameter tuning, and may even provide more useful features. \n\nClarity: \nThe paper is well-written and easy to follow. \n\nOverall: I think this paper is marginally above the acceptance threshold, as a cute and easy improvement to self-supervised learning techniques that seems to improve performance in most cases. I'm hesitant to push for a full-accept because of the experimental concerns I pointed out in the 'quality section', and the emphasis on nonstandard datasets that makes a thorough comparison to SOTA hard to tease out. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the societal impact and limitations of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "l019mPzpW-9", "reviewer": "Reviewer_FLzm", "summary": "This paper examines the utility of minimizing the distance between the values of intermediate layers in a neural network model, when performing supervised learning with the momentum contrastive (MoCo) method on two augmented versions of the same image. Two metrics for minimizing said distance are tested: mean squared error, and a cross-correlation matrix by the Barlow twins loss function (Zbontar et al., 2021). Experiments on three medical image datasets show that enforcing intermediate layer closeness can produce improve classification performance, especially when the proportion of labelled data for training is small. Additional experiments suggest that intermediate layer features trained in this way are more suited for reuse for various downstream tasks, and are of higher quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. It might be clarified whether the intermediate feature similarity enforcements apply only to positive pairs, and the standard MoCo for negative pairs (in their respective encoder) remains unchanged; if so, it might be commented as to whether intermediate features may be exploited for negative pairs too.\n2. It is stated that Resnet-50 is the backbone architecture for all baseline self-supervised methods (apparently MoCo) and our approach(es) (+MSE/+Barlow Twins) (Line 222). The architecture and training methodology for the supervised baseline might be stated as well, and justified to be a fair comparison. \n3. For linear probing (LL fine-tuning) as reported in Section 4.3.1/Table 1, what was the label fraction used, and is there any reason why various label fraction values were not investigated, as was done for end-to-end (E2E) fine-tuning? It might also be considered to standardize the format of Table 1 with Table 2 (e.g. “Supervised” column to come first)\n4. The class distributions for the 1% and 6% label fraction experiments might be stated. In particular, are the original class distributions maintained? This is of interest as medical image datasets often have highly imbalanced class distributions, and rare classes may thus possibly not be represented when sampling for low label fractions. Ideally, the class label distributions/quantities for each dataset might be provided, for training/validation/test splits, possibly in the supplementary Appendix.\n5. It might be interesting to investigate saliency heatmaps for the various methods (supervised, MoCo, MoCo+MSE, MoCo+Barlow Twins) if possible, to further verify whether intermediate layer similarity affects classification attribution.\n6. Minor issues:\n(Line 62) “have high performance” might be “has high performance”\n(Line 86) “We don’t” might be “We do not”\n(Line 236) “The” is repeated.\n(Line 259) “proliferate diabetic retinopathy” might be “proliferative diabetic retinopathy”\n---\nWe thank the authors for addressing our major concerns in their reply.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "i7LlIbj11Xs", "reviewer": "Reviewer_ZiVQ", "summary": "The authors propose to learn augmentation invariant features early on in a network by, in addition to a global contrastive loss, also optimizing for an intermediate loss function to ensure similarity between features in early layers of the network. They test two loss functions for this purpose - MSE and a minimization of the difference between the cross-correlation matrix of the intermediate representations and the identity matrix. They hypothesize that this way the network learns high quality features early on.\nThe authors also test the quality of features learnt with their method in a self-supervised setting, by comparing the similarity of the features learnt under self-supervision to features learnt after fine-tuning on a labeled dataset following the self-supervision. They do so for a variety of different labelled set sizes, as well as end-to-end fine-tuning, and only tuning the last layer.\n\nThey test their method on three publicly available medical datasets (chest X-Ray, diabetic retinopathy and histopathological whole slide images), which are of different imaging modalities.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well structured and well written. The idea is clearly presented, as well as the means how to evaluate the feature quality. The claims are supported by experiments on three available datasets of different modalities which shows that the versatility of the proposed idea to regularize intermediate feature representations by an additional similarity loss term. The concept can be used for any backbone architecture and enforce various invariances of the representations depending on the data augmentation. Seeing how the method shows major improvements in classification performance over the baseline in case of only small labelled datasets, this work could be very valuable, for biomedical tasks in particular.\n\n\n\nThere are however a number of questions to address in the rebuttal:\n\n\n\nIt should be stated more clearly in the methodology section how the InfoNCE loss is combined with the loss on the intermediate layers. \n\nI think it is very relevant to point out precisely what augmentations were used during the self-supervision pretraining (on top of citing Chen et al) as they will lead to the representations’ invariances.\n\nAssuming rotations were part of the augmentation, I wonder if the drop in performance of the proposed method vs only MoCo is due to the orientation of the data in case of the NIH Chest X-Ray dataset? The histopathological images are rotationally and translationally invariant by nature, the diabetic retinopathy images to some extend also, if the meta data available to the dataset (left/right eye, inversion of the photo or not) was not used to correct for it (which is how I understand the dataset was used here for both the MoCo experiments as well as for the proposed method). However, the chest X-Ray images have a set orientation, so learning rotationally invariant features could possibly be counterproductive. Similarly, invariances to color distortion could prohibit the network to pick up on darker or lighter regions in the X-Ray which may be crucial for classification.  Hence my suggestion (possibly for future work) would be to adjust the augmentation such that only invariances are learnt that are in line with the clinical background knowledge available and see if the performance can be further improved that way.\n\nWhy were the sizes, in percent, of the training/test/validation sets chosen differently for the DR dataset than the other two datasets?\n\nWhat interpolation method was used for the resizing? Seeing how the patches from the whole slide images increased its pixel count by more than 26x, resized from 50x50 to 256x256, the interpolation method will play a significant role in the resulting input images.\n\nWhy was AUC reported for the DR binary classification task, but the F1-measure for the histopathology dataset? I strongly suggest to report both values for both datasets to make comparison of the improvement of the proposed method feasible. \n\n\n\nMinor remarks: \nIt would be good to point out in the caption of Table 3 that CKA was used as a similarity measure.\n\nIt would be beneficial to adjust Fig. 1 to the same notation used in Eqn. (1) (views of the same image denoted as $\\tilde{x_i}$ in the equation, but as $v$ and $v’$ in the figure for example).\n\nIt is unclear to me from the paper what part the “momentum” plays portrayed in Fig. 1.\n\nSome minor spelling/grammar mistakes, mostly with singular/plural or capitalization (lines 17, 23, 34, 38, 68, 158, 166, 180, 201, 204, 222, 228, 240, 247, 248, 278, other instances of X-ray)\nSome issues in the citations, e.g Neyshabur et al 2021 missing journal/proceedings\n\n-------------------------------------------------------------------------------------------------------------------\nAfter rebuttal: \nThe authors promise to include details which I was missing in the submission, in the camera-ready version and I higher my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not discuss the negative societal impact in sufficient detail. Their method is in no way limited to medical images, i.e. they should reflect upon other data in which self-supervision is key.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "9agPhxn9th", "reviewer": "Reviewer_vuAC", "summary": "This paper improves the self-supervised contrastive learning framework, specifically MoCo, by making the features in the intermediate layers closer. Although not the first attempt at this idea, this paper dig to investigate the choice of loss terms and the property of learned features. MSE loss and a recently represented Barlow Twins loss are studied. For analyzing learned features, layer-wise probing measures the performance of using intermediate features for downstream tasks (higher means better). Feature reuse is measured by quantifying the similarity of features between models after pertaining, finetuning on labeled data. The idea is that the features change less over finetuning means self-supervised learning learns better features. The output distributions of modes finetuned with different amounts of labels are also compared with the best-performed model as a metric for finetuning with limited labeled data. This paper focuses on medical images in the study.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- This paper explores a nature idea to extend the current contrasting frameworks by adding intermediate features. Two loss terms used in the paper don't need negative pairs, which saves the computation cost. Both losses improve the original MoCo in linear eval and e2e fine-tuning except the NIH chest X-ray dataset and MSE loss works better than Balow twins in more cases. \n- They analyze learned features from a few interesting perspectives and may inspire other research.\n\nWeakness/Concerns/Suggestions:\n- It is unclear why 3.4.1 \"feature reuse\" and 3.4.3 \"Feature utility for small datasets choose\" different comparison targets: 3.4.1 choose the 100% labels finetuned model for each method while 3.4.3 compare to the best-performed model. I am a bit concerned about the choice of 3.4.3. The distance may not only be related to the performance but also the approach itself. In other words, is it really true that the probability similarity with the best model is strongly correlated with the actual performance? There are counter-cases e.g. in 6% labeled  Breast cancer data, the supervised model has a similar distance with MoCo-MSE and is better than MoCO while it has the worst accuracy. Maybe checking the correlation between the distances and the actual performance is helpful to answer this question.\n\n- It will be helpful if the paper can discuss more (clearly) on that what properties the three analysis can provide about regarding the feature quality that the downstream task performance cannot. In other words, why do people working on self-supervised representation learning want to do these analyses on their approaches?   \n\n- A ablation study on applying the intermediate feature similarity to not all positions is missing. \n\n- Some details are missing:\n  - Why use KS distance but not other metrics e.g. KL divergence?\n  - How is the similarity/distance computed. My best guess is the average of the per-sample results over the test set.\n  - section 3.3, how do you use 1000 bootstrap replicates to measure the performance? \n\nOverall, I think this paper presents a reasonable approach to extend MoCo that shows good performance. The other contribution is the three ways of analyzing feature quality. However, authors should highlight/discuss more their value as general metrics for evaluating learned representation if they are not only used for the approach in this paper.  Consider the limited scope of studied model/backbone/datasets (discussed in the conclusion though), I am slightly lean to reject this paper.\n  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a straight-forward but clever and useful extension of MoCo by adding intermediate features. 
A similar approach has been proposed earlier (Gong et al., withdrawn from ICLR, on openreview, cited in the proposed paper) this paper makes a great case for it.  The paper tests on three very different, biomedical and openly available datasets and achieve very good results. They have addressed and promised to include details which the reviewers requested from the original submission so overall this paper was viewed positively by all reviewers and meta-reviewer. 


Decision: Accept (Poster)