Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles
Authors: Bram Wallace
Devansh Arpit
Huan Wang
Caiming Xiong
Venue: ICLR 2022 Submitted (2022)
Keywords: k-NN
ensemble
self-supervised learning

Abstract: Pretraining convolutional neural networks via self-supervision, and applying them in transfer learning, is an incredibly fast-growing field that is rapidly and iteratively improving performance across practically all image domains. 
Meanwhile, model ensembling is one of the most universally applicable techniques in supervised learning literature and practice, offering a simple solution to reliably improve performance. But how to optimally combine self-supervised models to maximize representation quality has largely remained unaddressed.
In this work, we provide a framework to perform self-supervised model ensembling via a novel method of learning representations directly through gradient descent at inference time.
This technique improves representation quality, as measured by k-nearest neighbors, both on the in-domain dataset and in the transfer setting, with models transferable from the former setting to the latter.
Additionally, this direct learning of feature through backpropagation improves representations from even a single model, echoing the improvements found in self-distillation.

Under review as a conference paper at ICLR 2022
LEARNING RICH NEAREST NEIGHBOR REPRESENTA- TIONS FROM SELF-SUPERVISED ENSEMBLES
Anonymous authors Paper under double-blind review
ABSTRACT
Pretraining convolutional neural networks via self-supervision, and applying them in transfer learning, is an incredibly fast- fast-growing field that is rapidly and itera- tively improving performance across practically all image domains. Meanwhile, model ensembling is one of the most universally applicable techniques super- vised learning literature and practice, offering simple solution to reliably im- prove performance. But how to optimally combine self-supervised models to maximize representation quality has largely remained unaddressed. In this work, we provide a framework to perform self-supervised model ensembling via a novel method of learning representations directly through gradient descent at inference time. This technique improves representation quality, as measured by k-nearest neighbors, both the in-domain dataset and in the transfer setting, with models transferable from the former setting to the latter. Additionally, this direct learning of feature through backpropagation improves representations from even single model, echoing the improvements found in self-distillation.
INTRODUCTION
The widespread application of pretrained convolutional neural networks in computer vision is one of the most important tools in the field. State-of-the-art on many benchmarks ranging from classi- fication, to object detection, to pose estimation has been set using a pretrained model, such as an ImageNet classifier, as network initialization (Kornblith et al. 2019 He et al., 2019; Chen al.] 2020a Grill et al. 2020 Kolesnikov et al.) 2019). Transter learning is an entire field focused on studying and utilizing this phenomenon. While supervised ImageNet classifiers were the dominant feature extractors of choice for many years, recently self- euf-ssupere models have begun to take their place. Methods such as MoCo(v2), SimCLR(v2), SimSiam, PIRL, Swav, BYOL, and Barlow Twins all claim transferability competitive with or superior to that of ImageNet classifiers (He et al. 2019
Chen et al. 2020c a b Chen & He 2020 Misra & Maaten 2020 Caron et al., [021] Grill et al, 2020 Zbontar al. 2021 As such, the question of what initialization to use has arisen; benchmark
studies have sought to compare methods under dozens of different settings (Goyal et al. 2019 Zhaii al. 2019). Even when decision has been made use particular feature extractor, the utility and knowledge of other options is then left unutilized.
To address this concern, we consider ensembling, a common practice in the supervised setting (Diet- terich 2000 Hinton et al. 2015). Ensembling models involves combining the predictions obtained by multiple different models in some way, typically by averaging or similar operation. In the supervised setting, such outputs are aligned and such an operation easily captures the combined knowledge of the models. In the supf-supe setting, however, such alignment is not guaranteed, particularly when dealing with independently trained contrastive learners which many pretrained models of choice are. Averaging the features still useful, and obtains reasonably strong image representations (Section 4.. but we show that it is possible to build yields richer, stronger representations than the mean feature. We do so without training any new CNN components, allowing for the same frozen backbone be used across applications.
ensembling strategy that
How to approach ensembling in the self-supervised setting? We contend that the goal of a model ensemble is to capture the useful information provided by the different models in single representa- tion. We consider the "capture" of information from a recoverability perspective: if every network's features can be recovered by some fixed operation representation vector, for all data samples,

---

Under review as a conference paper at ICLR 2022
Training
Testing
Unlabeled Train Images
Unlabeled Test Images
Feature Extractor (1)
Feature Extractor (
Feature
Feature Extractor (1)
Feature
Peature
Extractor (m)
Extractor (p
Extractor (m)
Features (1)
Features (j)
Features (m)
Features (1)
Features (j)
Features (m)
Ground Truth
Ground Truth
Ground Truth
Cosine Loss
Features (1)
Fearures ())
Features (m)
Features
Features (j)
Features (m)
Reconstructed
Reconstructed
Reconstructed
Reconstructed
Reconstructed
Reconstructed
MLP (3)
MLP i
MLP (m)
MLP (1)
MLP (i)
MLP (m)
SGD-Leamed Train Representations
Being learned
SGD-Learned Text Representations
Figure 1: A schematic of our method. We wish to ensemble M feature extractors over training dataset images, M MLPs are initialized as well representations for each of the n images. The training objective is for all of the features of the M models to be recoverable by feeding the learned representations through the respective MLP. The MLPs and learned representations are simultane- ously optimized via gradient descent, using cosine loss. At inference time, the MLPs frozen, and solely the image representation is optimized.
Algorithm PyTorch-like pseudocode of our method
Training Phose
J Initia_ize representations to average reature trein fea.fffatteeiiee
Inet (images) for ensemble]
avg_feat average
festura_list; shape points feature dimansion)
learned_traie wtrp_ttrrrnreeeeeee
detacc (1) initialize params with avg_teat enseable]
mips [XLP for
feature extracter Lesrned_train__ee
opt SGD par.mmm....aa.erraa....
optinize both mapping KLPs input
representations
for imagos_idx, images in trainloader: in ensemble]
ensemble_reate [net (imagea)
ensemble teaturea cutputs [mip tteeerne__traa___ee www.seeggsieeeeeee (images__dde for mip in mips] # Map learned representations through eifferent KLPS coaiine_los (ensembblll teata, coss.backward()
Inference Phase
test hat__faturr__ll
[not (Images)
in ensamble]
tteetttetttttteee..teeee opt ieleaared_test__eee for images_idx, images in leatleader:
Freeze KLPs st inference time
Same as training loop
MI.P): lalti-layer perceptron model
Parameter(t): Pytorch function that takes the argument arryy and initializes trainable parameters with that value
then such representations are useful. While concatenation of features can trivially achieve this ob- ject, we show that such an operation in fact suboptimal in terms of the behavior the derived feature space. We instead propose directly lear via gradient descent set of representations that contain all of the information necessary to derive the ensemble features. Our architecture is shown in Figure with example pseudocode in Algorithm Specifically, we show that extracting features from an ensemble of self-supervised models using this technique improves the nearest neighbor (NN) performance when evaluated downstream supervised tasks.
RELATED WORK
Supervised ensembling: It is a ubiquitous technique in machine learning (Dietterich) 2000). In ad- dition to the large number of online contests through such approaches (Andres), ensembling has
2

---

Under review as a conference paper at ICLR 2022
also been demonstrated to achieve tf-te--f-thearrr performance on standard computer vision bench- marks (Huang et al. 2017). Most approaches employ Bayesian ensembling, where the predictions of separate networks averaged (Wu et al., 202I; Hinton et al. 2013 Dietterich] 2000). While this relied on the alignment of objectives between the networks, we show that such an averaging on the intermediate features does indeed generate representations superior that of individual models. Our method differs from this literature, however, in the learning done on top of the ensemble as well as the no-label setting of our representation learning.
Knowledge distillation (Hinton et al., 2015); This is a related vein of work is that of where the knowledge of an ensemble is distilled into single neural network, or a single model is distilled into another network. This second version, known as self-distillatii relevant to our single-model setting, as we are able to obtain improvements while operating purely in the feature space of a single model. Our goal is not to discard the ensembled models as it is in knowledge distillation, but our method bears similarities to that of Hydra (Mandt et al.), where network is trained to output representations capable of recovering the ensembled outputs. We note that our resulting accuracies consistently surpass average ensembling, a baseline that Hydra considers as an upper-bound to their method (but the differing settings do not lend themselves to apples-to-apples comparisons).
(Zhang et al.) 2219)) is highly
Xu al. (2020) propose self-supervisce knowledge distillation algorithm that uses both supervised and sulf-supervised loss to train a teacher network, and then distill this combined knowledge to a student network. They show that this combination improves the student performance compared to traditional KD. In contrast to their work, our goal is not to learn a separate student network and we do not assume access labeled data, but instead, our main goal is to extract rich nearest neighbor representations from an ensemble of self-supervised models.
k-Nearest neighbor (k-NN) classifiers: j--N a -onrammmettiic classifier which has been shown to be a consistent approximator (Devroye et al. (1966)). i.c., asymptotically its empirical risk goes to zero as k oo and k // + 0, where N is the number of training samples. While these theoretical conditions may not be true in practice, main advantage of using k-NN is that it is parameter- free outside of the choice of l (which typically does not change qualitative rankings) and thus is a consistent and easily replicable measurement of representation quality. Additionally, k-NN makes the decision process interpretable (Papernot & McDaniel 2018 Dalitz 2009), which is important in various applications (Mehrabi al.) [021 Vellido 2019 Gilpin et al. 22018.. For these reasons, our paper focuses on extracting rich features from supervieee ensembles that are condusive to k-NN classification.
On a related note, SLADE (Duan et al. learning, which is essentially the setting of our evaluation framework. While the goal is similar, SLADE uses supervised training to initialize learning and generate pseudolabels for the unlabeled points, our method assumes zero label access. Additionally, SLADE is concerned with learning a new network from scratch, as opposed to an ensembling framework.
2021) leverages unlabeled data to improve distance metric
Our method relies the efficacy of pretrained self-supervised models. Spe- cific methods we employ are SimCLR, SwAV, Barlow Twins, RotNet, PIRL, as well as traditional label supervision. In addition to the above works: Goyal et al. (2019) Zhai et al. (2019); Kolesnikov et al. 2019)) benchmark and demonstrate the generalization efficacy of pretrained models in various settings.
Pretrained Models:
Gradient descent at inference time: One atypical facet to our method is the use of gradient descent at inference time to directly learn new representations. While this approach quite uncommon, we are not the first to leverage backpropagation at inference-time. Zadeh et al. (2019) uses backpropaga- tion at inference time to learn representations from images using a generative "auto-decoder" frame- work and Park et al. (2019); Sitzmann et al.] (2020) employ similar approaches to learn implicit representations of shapes. Sun et al. (2020) considers new samples as one-image self-supervised learning problems, and perform brief optimization of self-supervised learning objective on new image (modifying the feature extractor's ' parameters) before performing inference, and Shocher al. 2018) train CNN test-time for the purpose of super-resolution.
3

---

Under review as a conference paper at ICLR 2022
3 METHODS
We present method for directly learning rich ensembled unsupervised representations of images through gradient descent. Consider a training collection of images X = {xi|f=1 and an ensemble of convolutional neural networks feature extractors @ {0, -رح In this work the 03 have previously been trained in a self-supervised manner on ImageNet classification and are ResNet-50s (Deng et al. 2009 He et al., 2016). Denote the L2-normalized features obtained by removing the linear/MLP heads of these networks and extracting intermediate features post-pooling (and ReLU) as Z ([=())
ya 1m j= here 2,6) denotes the intermediate feature corresponding to (1))).
We initialize a memory bank of representations of X, with one entry for each Iy these entries have the same feature dimensionality as the z/. This memory bank is analagous to the type use in early contrastive learning works such as Wu al. (2018). We denote this memory bank v// { in
km
Σ #2
Each tha is initialized to the 12-normalized average representation of the ensemble t/ = || || &f
note that the sum operation is equivalent to averaging due to the normalization being performed.
To map the memory bank to the ensembled features, we employ a set of multi-layer perceptrons (MLPs), & {or ) each corresponding to a feature extractor 0z. Unless noted otherwise in our experiments, these te are layers, both of output dimension the same as their input (2048 for ResNet50 features). ReLU activations are used after both layers, the first as a traditional activation function, and the second to align the network mapping to the post-relu set Z.
During training, a batch of images {xi are sampled with indices 1 C {1...n). The corre- sponding ensemble features, Z₁ = (f2(") he 1. / sentations , = { ker- Note that no image augmentations are included our framework, meaning that the 2,") are typically cached to lessen computational complexity. Each banked repre- sentation is then fed through each of the m MLPs, 4. resulting in a set mapped representations (VV,) = (<be(vis) (EE. The goal of the training is to maximize the alignment of these mapped features &b 7 with the original ensemble features Z₁. This is done by training both the networks and the representations using a cosine loss between ( and Z,, gradients are computed for both the MLPs and representations for each batch.
are retrieved as are the memory bank repre-
Once training is completed, the de are frozen. During inference, when a new image x' is given, the above process is repeated with the frozen MLPs. Concretely, the ensemble features de (x') are com- puted and averaged to initialize a new representation v/. v/' is then optimized via gradient descent to maximize the cosine similarity of each (e((")) with (z(),, vi then serves our representation of x'.
The described method results in representations superior to either the average concatenated feature in terms of nearest-nett -nirhshr accuracy. We highlight several exciting aspects of our method:
The learning of a representation at test time via gradient descent is an uncommon approach. Existing methods do exist, such as auto-decoders in Zadeh et al. (2019) implicit 3D rep- resentation literature (Park et al. 2019 Sitzmann et al., (020). There are also techniques, such as Sun al. (020 for generalization, which use gradients shape the parameters prior to inferenc.
The vast majority of ensembling literature focuses on the supervised setting, where the training objectives of the ensembled networks are identical and thus aligned. Very little work has been performed on improving a group of self-supervised features with outside auxiliary signal. Hydra (Mandt et al.) considers similar setting, but with a focus on knowledge distillation of the ensemble into a single network.
Our method is extremely adaptable to different settings. Networks trained on multiple objectives, with different head architectures, can be usefully ensembled as demonstrated
in Sec. 4 This could trivially be extended to using networks of different architectures as well (e.g. VGG + ResNet + AlexNet) (Simonyan & Zisserman 2014; He et al., 2016 Krizhevsky et al., 2012). The flexibility of our approach additionally extends to the input data. While we use networks pretrained on ImageNet, the ensembling provides benefit on both ImageNet as well in the self-supervised transfer learning setting. This transfer can
4

---

Under review as a conference paper at ICLR 2022
either be performed by training new & the target dataset by training & on ImageNet and then using the frozen MLPs to learn representations on the target dataset.
Our method as presented requires only a single forward pass of each image through each ensembled model as we do use data augmentation. This allows eaching of CNN features and fast training.
3.1 TECHNICAL DETAILS
Models The MLPs are as previously described. The pretrained ensemble é consists convolutional neural networks (all ResNet50s with features extracted between the stem and head of the network) that have had pretraining on ImageNet. The method used in said pretraining varies and is subject of study in this work. Methods considered include SimCLR(v2), SwAV, Barlow Twins, PIRL, Learning by Rotation (RotNet, Gidaris et al. (2018)), and supervised classification. Pretrained models are obtained from the VISSL Model Zoo (Goyal et al. 202I), specific model choices provided in the appendix.
Data For our ImageNet experiments, we use the standard ILSVRC 2012 (Deng et al. 2009) train- ing/validation split with per-channel normalization. We additionally include several datasets to eval- uate the efficacy of our method in self-supervised transfer learning setting. These datasets are CIFAR10/100, Street View House Numbers (SVHN), Food101, and EuroSat with splits as provided by VISSL, the same per-channel normalizations used when inputting these images into the pre- trained 0, (Krizhevsky et al. 2009 Netzer al. 2011 Bossard et al. 2014 Helber al. 2019). No data augmentations used this work.
Training Adam optimizers with learning rate 3- used throughout all experiments with batch sizes of 4096 for all ImageNet training and 256 on all other settings (Kingma & Ba (014)). For ImageNet experiments. 20 epochs are performed for both training and inference, otherwise. 50 epochs are used. In practice, we find it useful to warmup the MLPs (Φ) for half of the training epochs before allowing the learned representations to shift from their average initialization.
Evaluation We employ k-nearest neighbor (k-NN) evaluation throughout to measure the qual- ity of features, qualitative comparisons remain constant under variance in the choice of k. The choice of k-nearest neighbor is made as it a parameter-free evaluation method that requires min- imal tuning. Under regularization-fie free linear evaluation transfer learning, ensembled features outperform both our ensembled feature baselines as well as their component state-of-the-art trans- fer learning models (e.g. SimCLR). Under heavy regularization cross-validation standardized in self sup--supeeeeee learning works such as Grill et al. (2020); Kornblith et al. (2019). however, the ensembled baselines can exceed performance of all the above (see Appendix).
Baselines compare against several ensembling baselines throughout this work.
Concatenation: for image x; the concatenation of all 2; into single vector 2f E R 2048m
the average feature Em |X
Averaging:
(the initialization our learned v/)
Individual: a single model of the ensemble being used (in each case we detail specifically which model this is)
RESULTS
4.1 ENSEMBLING
In Figures and we use our method on an ensemble consisting of 4 SimCLR models. Figure demonstrates the efficacy of our method the source dataset, ImageNet. All of the ensembled models were trained in a self-supervised fashion on this source, but our method extracts an additional 2% of performance, increasing the nearest-neighhor accuracy over 58%.
5

---

Under review as a conference paper at ICLR 2022
SimCLR Ensemble (4 Models)
SimCLR Ensemble Madels)
60
90-
Concalemation
Neares Neighber Accurssy
Averaging
58
Individus!
Ours
56
60
50
52
50
Concatonan Concatonatonn
Avernging
Individual
Our
CIFAR10
CIAAR1OD
EuroSat
SWHN
Food10]
Moan
Figure Our method applied to non-ImageNet datasets, leveraging the generalization of our pretrained feature extractors. Performance improved across all datasets.
Figure 2: Nearest neighbor accuracies on the validation split of ImageNet. Our method im- proves over all baselines by over 2%.
In Figure 3] learning is performed novel datasets, this can be thought of self- transfer learning. Labels are not made available until evaluation time, and then only to measure the k-NN accuracy. Using the frozen SimCLR features extractors, model learns representations which achieve over 2.5% higher k-NN accuracy on average, with a smallest win of 0.6% (over Averaging on EuroSat).
We employ different types of ensemble in Figures and Figure emmpoys an ensemble consisting of five differently trained self-supervised models: Barlow Twins, PIRL, RotNet, SwAV, and Sim- CLR. These represent various approaches to self- self-suup learning: SwAV and SimCLR are more standard contrastive methods, while Barlow Twins achieves state-of-the-ard performance using an information redundancy reduction principal. SwAV is clustering method in the vein of DeepClus- ter (Caron et al., 2018) and RotNet is heuristic pretext from the family of Jigsaw or Colorization (Noroozi & Favaro 2016 Zhang et al. 2016). Here use Barlow Twins as the "Individual" com- parison as it notably achieves the highest individual k-NN accuracy on every dataset. This setting is intriguing from two different perspectives: firstly, the varying strengths of the underlying ensembled models is challenging as noisy signal from the weaker models can drown out that of the strongest; second, the varied pretraining methods results in different strengths. For example, RotNet is by far the weakest model the ensemble with an average transfer k-NN accuracy almost 10% lower than any other model. However, non-Barlow methods by over 4 % (the efficacy of such geometric heuristic tasks symbolic datasets as previously been noted in Wallace & Hariharan (2020)). Our model seems benefit from this, achieving its largest win over Barlow Twins (8.2%) on this dataset. This demonstrates the ability of our model effectively include multiple varying sources of information.
SVHN (a digit recognition task), it performs very well, beating all
we consider the effect of using our method on supervised ensemble. While the pre- training goals of these models are aligned and thus traditional techniques (e.g. prediction averaging) could be used, we demonstrate that our model successfully improves upon the ensembled interme- diate features further demonstrating its agnosticity towards pretraining tasks.
In Figure 5
4.2 EFFICACY ON INDIVIDUAL MODELS
While our method is designed as an ensembling technique, we discover that it is surprisingly effec- tive when employed on a single model. These result are quite remarkable, as the improvement of features without access to their corresponding images or additional supervision is quite challenging. This setting is especially remarkable as here the input initialization and targets are identical; we find that the MLP, ø. does not converge to perfect identity function during the warmup period and the movement of the representations stage, the MLP output of the average feature is close to identity (0.97 cosine similarity). but only by learning the representation through gradient descent does the similarity improve to near near-perfect (0.99+) similarity. We discuss possible reasons for this behavior in Section 5
fact help enable near- sear-perfect target recovery. In the inference
perfect

---

Under review as a conference paper at ICLR 2022
Supervised Ensemble (3 Models)
90
Concatenation
Concatenation
Averaging
Averaging
Barlow Twins
Badlow
Outs
70
60
60
50
CIPARLO
CIFARI D0
EuroSat
SWNN
Frod1 01
Moan
CTPARTO
CIFARD00
Eurosat
svitN
Feodiut
Moall
Figure 4:
Ensembling varied models. We ob- serve that our method seems to capture spe- cialties/strengths of the component feature ex- tractors, particularly the symbolic-dataset effi- cacy of RotNet.
Figure Despite not utilizing the consistency of the supervised classification objective, our method effectively combines supervised models to improve upon the performance of all other en- sembles considered.
Single-Model ImageNet Distillation
Barlow Twins Single Model
Original
Barlow Twins
65
70
60
60
50
50
Barlow Twins
SimCLR
Saperdised
CIFAR10
CIPARIUG
EaroSat
SVINN
Foodi01
Meun
Figure 6: Paralleling the efficacy of self- distillation (Zhang et al.) 2019), our "ensem- bling" method proves to provide performance gains even when just one model is employed.
Figure We experiment with a single Barlow Twins model on our generalization benchmark. Performance is gained on datasets, with a mean improvement of over full percent.
In Figure we see that our "ensembling" technique benefits individual models substantially (1.8, 1.3 and 0.4% respectively) when our representations are trained on ImageNet. Given the degree to which the original self-supervised models' objectives are optimized, the margin the improvement quite impressive. This benefit carries over the self-supervised transfer learning well (Figure 7 Here we use our method in conjunction with Barlow Twins model. Our method offers a mean k-NN accuracy gain of over 1,, once again despite additional information, augmentations, or images being made available besides the CNN's 's features themselves. While this gain is relatively minor, show in the appendix that superiority to the baseline features is maintained across wide range of hyperparameter choices.
4.3 TRANSFERRING MIPS FROM IMAGENET
So far we have only considered the scenario where the MLPs, t. are trained on the same dataset as the representations V, where inference is ultimately performed. This is not a necessary assump- tion of our framework, however, once MLPs are trained a dataset they can be re-used to learn representations V on arbitrary imagery. We conduct experiments where (specifically those generated for in Figures 2 & ( are re-used in the transfer setting. Because the MLPs are frozen. no parameters of any networks are being changed during training, solely the representations V are being learned. The results of these experiments shown in Figures &
trained on ImageNet

---

Under review as a conference paper at ICLR 2022
Barlow Twins (Single Model)
SimCLR Ensemble (4 Models)
90
Concatenation
90
Barlow Twins
Ours (Standard)
Averaging
Barlow Twins
80
Ours (Transder)
(Standard)
70
Ours
(Irnssfer)
60
50
40
CIFAR1O
CIFAR100
EuroSat
SVHN
Frod 01
Maan
CIFAR1O
CIFARICO
EuroSal
SVHN
Food 11
Mean
Figure 9: In the single-model case, transfer- ring & still provides benefit over the baseline, but is less effective than learning the MLPs per- dataset.
Figure 8: The MLPs do necessarily need to be trained on the target dataset, but can be transferred from ImageNet to other downstream datasets. Here no parameter tuning is done on each dataset, gradients are computed solely to directly learn the representations. We find in fact that this transfer approach maintains the performance of directly learning new MLPs.
In the ensemble setting, the performance is largely maintained when re-using MLPs from ImageNet. Both method (Ours (Standard) and Our (Transfer MLPs)) are still superior the baselines across all datasets. This result is intriguing, as the observed ensemble gains are relatively large, and this method requires no pretraining on targeted transfer set, simply inference via gradient descent of Vr; such generalization properties are part of what makes the base pretrained models so valuable in application.
In the single-mode model setting, the Barlow Twins model + MLP trained on ImageNet is re-used across transfer datasets. Here we see notable decline in performance, but the transferred model still maintains improvement over the baseline on 4 out of 5 datasets (all but EuroSat).
ANALYSIS
5.1
DEEPER & REGULARIZE V
One hypothesis for the efficacy of our method in the single-model setting is that Φ acts as a regular- izer. The findings Arora et al. (2019) demonstrate that stochastic gradient descent in deep neural networks tends to recover low-rank solutions when performing matrix factorization, this helps to explain the generalization properties of deeper networks: deeper networks lead to simpler solutions which tend to generalize better. This relates to our method, as our improvements hinge upon the network & not learning a perfect identity function during warmup; it did, then the gradients with respect to I would vanish and no change of representation would occur. Arora et al.] (2019) suggests then that a deeper network might improve the ultimate quality of V, as the points would need to be recoverable by a lower-rank MIP.
We confirm this phenomenon Figure varying the depth from I to layers while learning representations directly our varied dataset benchmark using Barlow Twins model. Increasing depth improves accuracy incrementally over a total of '% on average until the network is 6 layers deep, more than triple that of our default setting. Some but not all of this performance boost is recoverable by adding in small amounts of traditional weight decay (e.g. le 6) to the parameters of the MIP.
5.2 BEHAVIOR MODEL
Now that we have established partial explanation of why model works, by learning represen- tations which preserve information under the regularization of an SGD-leamed deep network, we now investigate what specific changes method makes to the feature space.
8

---

Under review as a conference paper at ICLR 2022
Sorted Singuler Values of Feature Matrix CIPARIO CIFARI00 EuroSat SVHN
fnod101
10a
Moan Per- Dar-Datasstt Accuracy
Network Depth Ablation
74
101
73
72
10"
71
6
Orig.
Orig.
Owr
Ours
Figure 10: Ablation of MLP depth, possibly sug- gesting that the low-rank tendency of deeper net- works serves as regularizer on the learned rep- resentations. This results in improved representa- tion quality with network depth up to 6 layers.
Train
Test
Test
Figure 11: Sorted singular value curves for our method vs. the baseline features an apples- to-apples setting (learning á restricted to non- negative). Our method learns features with a more balanced set of singular values, indicating a more uniformly spread bounding space.
First, Figure I1 examine the distribution of v. We do so by training representations from a single Barlow Twins model similar to previously, with the important distinction that we restrict our points be non-negative (i.e. in the first n-tant of feature space), to make an apples-to-apples comparison to the baseline features. We examine the singular values of this (constrained) feature matrix compared that of the original features. In general, the singular value distribution of are less heavy-tailed: meaning the volume occupied by the features is larger and more uniform in each dimension than the bascline features. This an indication of learning a regularized form of the original Z. The above finding indicates that the feature representations the learning process. We also investigate what happens to clusters in Figure [5]in in the Appendix. In summary, the findings suggest that our methods success is partially attributable accentuation of existing clusters in the dataset.
spread out result of
DISCUSSION
In this work, we presented novel self-supervised ensembling framework which learns represen- tations directly through gradient descent. The intuition behind our method is capture all of the knowledge contained in the ensembled features by learning a set of representations from which the former are fully recoverable. We demonstrated the efficacy of our method in Section 4 and analyzed causes and effects of the representation improvement in Section 5 We hope that this work lays the groundwork for further forays into the problem of utilizing combinations of the powerful pretrained models that are becoming plentiful in the computer vision literature.
As previously noted, while we demonstrate improvement under k-NN this paper the average fea- ture baseline surpasses our method under linear evaluation when regularization is heavily optimized under grid sweep as standardized in Grill et al. 22000); Kornblith et al. (2019). We tried applying various regularizations during MLP training, including traditional L2 weight decay, LI regulariza- tion of V, the dimensionality of V, and the depth/width the MLPs Φ. While some of these modifications further increased the k-NN performance improvements, when representations evaluated under linear regression with cross-validated regularization none consistently surpassed the average ensemble baseline.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Eliot
Andres.
Kaggle
past
solutions.
URL Sttps://ndren: me/
kacgle-past--olutt.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization, Advances in Neural Information Processing Systems, 32:7413-7424, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. European Conference on Computer Vision, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.00 05009, 2020a.
Ting Chen, Simon Korbllith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hin- ton. Big self-supervised models are strong semi-supervised learners. In Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neu- Curran Associates, Inc., 2020b. URL https://proceeeinngs neurips. cc//paperr/22///iiee/ Fcbc95ccdd551em dal 81 1070001140005555 Papeed5555pppe.ppp
ral Information Processing Systems, volume 33. pp.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint
1.Xiv:0011110500 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020c.
Christoph Dalitz. Reject options and confidence measures for knn classifiers. Schriftenreihe des Fachbereichs Elektrotechnik und Informatik Hochschule Niederrhein, 8:16-38, 2009.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Lue Devroye, Laszlo Gyorfi, and Gabor Lugosi. Consistency of the k-nearest neighbor rule. In A Probabilistic Theory of Pattern Recognition, pp. 169-185. Springer, 1996.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multi- ple classifier systems, pp. 1-15. Springer, 2000.
Jiali Duan, Yen-Liang Lin, Son Tran, Larry S Davis, and C-C Jay Kuo. Slade: self-training framework for distance metric learning. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 9644-9653, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL
net fote:/ffmmmmm--122221111
Leilani Н Gilpin, David Bau, Ben Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning. arXiv preprint arXiv: 1rXiv:1055.011 2019.
10

---

Under review as a conference paper at ICLR 2022
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski. Armand Joulin, and Ishan Misra. Vissl. https: /ttpgtgttttutt/t./
2021.
Mann-Batieen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. Advances in Neural Information Processing Systems, 33, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 1911:09911000222 2019.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7)2217-2226, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Weinberger. Snapshot ensembles: Train 1, get m for free. arXiv preprint arXiv: 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 2014.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXiv: 1XXiv:9922.1377 2019.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Stephan Mandt, Jasper Snoek, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35, 2021.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariamt representa- tions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representions by solving jigsaw puzzles. In ECCV, 2016.
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv: 1803.0/80.0444 04765, 2018.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The JEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.

---

Under review as a conference paper at ICLR 2022
Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super- super-resolution using deep internal learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3118-3126, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. arXiv preprint arXiv:2006. 09662, 2020.
Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexci A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In JCML, 2020.
Alfredo Vellido. The importance of interpretability and visualization in machine learning for appli- cations in medicine and health care. Neural Computing and Applications, pp. 1-15, 2019.
Bram Wallace and Bharath Hariharan. Extending and analyzing self-supervised learning across domains, 2020.
Yanzhao Wu, Ling Liu, Zhongwei Xie, Ka-Ho Chow, and Wenqi Wei. Boosting ensemble aceu- racy by revisiting ensemble diversity metries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16469-16477, June 2021.
Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference on Computer Vision, pp- 588-604. Springer, 2020.
Amir Zadeh, Yao-Chong Lim, Paul Pu Liang, and Louis-Philippe Morency. Variational auto- decoder: A method for neural generative modeling from incomplete data. arXiv preprint 1:XY::1903000400 2019.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. CoRR, abs/2103.03230, 2021. URL https://arxiv.
org/abs/ 21 03. 03230
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski. Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. CoRR, abs/1910.04867, 2019. httht
wwwp:ii.rrg//bs...o. 1910. 04867
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
A APPENDIX
A.1 ROBUSTNESS TO HYPERPARAMETER CHOICES
In Figures 12 and 13 examine the robustness of our method (Single Barlow Twins Model on var- ied dataset benchmark) to choices of learning rate and batch size. We find that superior performance to the baseline is maintained across wide choice of settings (despite this being one of the settings where our margin improvement is the smallest) and that there fact room for further ter-task optimization via cross- cross-vlldattion of hyperparameters.
12

---

Under review as a conference paper at ICLR 2022
Mean Per-Dataset Aocuracy
Mean Per Per-Dataaet Acourscy
Batch Size Ablation
Learning Rate Ablation
74
74
Baseline Features
Baseline Features
Default Setting
73
Default: Setting
72
72
71
71
70
70
10-5
10
10-3
10
32
256
512
Batch Size
Learning Rate
Figure 13: Mean per-dataset accuracy, single Barlow Twins model varied dataset bench- mark. Batch size ablation. Linear learning rate scaling rule followed.
Figure 12: Mean per-dataset accuracy, single Barlow Twins model on varied dataset bench- mark. Learning rate ablation.
Cross-Validann Linear Regression
Single SimCLR
SimCLR Ensemble (Average)
90
Ours
(Single)
Ours
(Ens emble)
80
70
60
CIFAR1O
CIFAR100
EuroSat
SVHN
Food101
Menn
Figure 14: Linear regression accuracies with cross-validated 1.2-regularization. Relative perfor-
mance of our method is worse compared to the k-NN evaluation setting.
A.2 CROSS- VALIDAAADDDD LINEAR REGRESSION
We follow a protocol similar to Grill et al. (2020): Kornblith et al. (2019). A validation subset (10% of the training set) sampled and held out during training. Training on the remaining 90% of the data is performed for hyperparameter sweep with performance on the validation subset being measured. We employ an SGD optimizer for 1000 epochs with a batch size 4096 and learning rate 1.6. A weight decay sweep of {le 6, le 5. le 4, le 3, 1e 2} is performed. Results the single-model setting suffers substantially
are shown in Figure [4] We see that our method compared to the baseline. In the ensembled setting, there is still improvement on-average, but the gains are much more inconsistent than under k-NN evaluation. While this is a current limitation of model the benefits under N-NN indicate fundamental utility in our ensembling method to learn new reprsentations.
A.3 MODELS USED
Barlow Twins
bar ow www.wwwwwwonoeooooootoooeoweewsw
iaapiiin________ee 4node_ httpa//opppo.oooooooooomoooooooe
13

---

Under review as a conference paper at ICLR 2022
CIFARIO
CIFAR100 00
EuroSat
SVHN
Food
Ours Original
2
2
2
Normalized Similarity of Nearest Neighbor
The normalized maximum similarity (measure of how close each test point's nearest neighbor in the trainset is relative to average) for each method-dataset pair. Dashed lines indicate the median for each distribution We see that the proposed approach generally has higher normalized maximum similarities, indicating relatively tighter clustering behavior.
Figure 15:
SimCLR
200 epoch http
200/simell rn50 model final chacspolit_ppaaae19..ooocl
Btttoodeeeeee___ee__e______ 8nw.e_nnod___e_eee resnet 16 26. 00
400 epoch training:
2oo/simelr rn50 http:://a...aaalllllaee..o////.2
model_ffnal.. _
800 epoch http://ii..nnngnggg.........a...
model_ |
1000 epoch
Supervised
Varied Models
PIRL: https://dl. ostps///ddllfffaappilolccceeee.s
jigsaw 4node
22_07__222 243777554599
RoiNet (note, trained on ImageNeet22kk):
torch
Swav:
inik_rn50
2ttpa:000000eaaaeoo.eeeeerrrrra. f final Shaa__chheeppcinn_phhee///.....d
B BEHAVIOUR OF MODEL (ADDITIONAL EXPERIMENTS)
This is continuation of section 5.2 in the main text.
14

---

Under review as a conference paper at ICLR 2022
Here wish investigate what happens clusters. We examine this again through the lens of nearest-neighhos: for each 1// in the test set, we calculate the maximum cosine similarity of points in the train set. This maximum similarity is then normalized by the mean similarity for each method-dataset pair. The resulting normalized similarity provides a measurement of how relatively close points to their nearest neighbor relative closeness). Histograms of this metric are shown in Figure [5] For 4 out of the 5 datasets, our method results tighter neighbor matchings than the baseline. Intriguingly, the dataset for which this does not hold is EuroSat, is also the dataset where our models consistently yielded the lowest benefit. These findings suggest that our methods success is partially attributable accentu- ation of existing clusters in the dataset. It is interesting to note as aside that the mean pairwise similarity across the entire dataset significantly lower for our method, cementing the findings from Figure Ti that the features are more distributed across space.
average pair of points (with higher value indicating
15


Reviews:
[{"review_id": "w6f6O5MST3j", "reviewer": "Reviewer_HHFN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. Figure 1 and Algorithm 1 do a good job in explaining the proposed method.\n2. I appreciate the extensive experiments the authors have done both on ImageNet and on other datasets.\n3. I appreciate the authors specify the checkpoints used in the experiment and clarify all the hyper-parameters.\n\n**Weaknesses**\n1. Paper writing.\n- The description of the method in general needs to be improved. The figure 1 and algorithm 1 helps a lot with understanding the proposed method, however, the description itself in Section 3 is not clear. I find it difficult to understand which parameters are optimized during the training, it takes me quite a while to get that the inputs to the MLPs are optimized, but not the backbone parameters, this may leads to some confusion in understanding the proposed method.\n- The description of the some key components of the method is not clear. For example, the details of the kNN evaluation. At the beginning of section 3, the authors only mentioned which feature is used for the test images. However, they do not specify which image feature is used for the training images, this makes the kNN method in the evaluation in section 3.1 not clear. There are a variety of choices for training images features that I can think of to be used in the kNN. For example, this could be the $z_i^j$s, $\\psi_k$s, or $\\phi_{l}(\\psi_k)$s. I think this issue needs to be addressed since which feature is used in the kNN may make a huge difference in terms of the experiment result.\n\n2. I do not get the idea behind the proposed inference procedure. \n- For example, why frozen MLP? What happens if the MLPs are not frozen? \n- Why use $\\psi'$, what happens if $\\bar{\\theta}_{\\ell}(x')$ is optimized and used for the inference. \n\n3. More experiments need to be done in order to show the superiority of the proposed method. Below are several experiment that the reviewer is interested.\n- What if different training and/or test feature are used in the kNN, would this make a difference in terms of the experiment result?\n- The image feature is optimized by maximizing the cosine similarity in the proposed approach, comparing to the baseline methods used for comparison in the experiments. What if other self-supervised learning approaches used, would the proposed method still be useful? For example, what if minimize the L2 distance between the reconstructed feature and the extracted feature.\n- Based on the experiment result in Section 4.3, it seems like the parameter values of the MLPs does not matter, but optimizing the representation matters. A further experiment should be carried out to see what happens if random initializations for the MLPs are used and are fixed.\n- What happens if MLPs are not frozen during the test? \n- In Figure 15, the authors find the clusters produced by the proposed method has larger within class similarity and may lead to the better performance. If this is the case, another baseline for comparison should be the $z_j^i$s are all included in the kNN. In another word, there are m*n test image features.\n\n**Minor Issue**\n- The paper needs more careful proofread. For example, the legends in the figures are out of the box (see figure 14), the sentence in caption of Figure 15 is not complete. The citation of Mandt et al. is not right.", "summary_of_the_paper": "The paper proposes a new self-supervised ensembling method to get a better feature representation. Instead of using the conventional averaging or concatenating to ensemble multiple features, this paper proposes a different inference scheme where the backbone is also updated during the test time in a self supervised fashion. Experiments are conducted to empirically show the superiority of their methods compared to some existing methods.", "summary_of_the_review": "The overall approach of using self-supervised ensembling may potentially be interesting for the community. However, some details of the proposed method needs to be addressed and more experiments need to be done to explore potential weakness/strengths of the proposed method. I am happy to review the rating if these concerns are addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Xj47sq9lEkH", "reviewer": "Reviewer_94Ck", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The improvement is marginal considering the training and testing cost (20 epochs on ImageNet and 50 epochs on other datasets). Noting that the proposed method needs to fine-tune during validation. Is there any possibility of information leakage from validation data?\n2. This paper only compares the proposed method with the provided baseline. It is necessary to compare with other advanced ensembling methods.\n3. What are the differences between \"4 SimCLR models\" in Fig. 2?\n4. Some experimental issues:\n  - The baseline performance seems to be too low in Fig. 2.\n  - Only KNN results are reported. What about the classification accuracy under the linear evaluation protocol, which is widely adopted in self/un-supervised methods?\n  - It is claimed that no data augmentation is used. What about the resizing augmentation", "summary_of_the_paper": "This paper proposes a framework to perform self-supervised model ensembling via a novel method of learning representations directly through gradient descent at inference time. The effectiveness of the proposed method is evaluated by k-nearest neighbors accuracy.", "summary_of_the_review": "The main concern comes from the fine-tune manner in the testing phase, which may cause cheating for the model from the validation data. The key contribution needs to be further clarified as the applicable scenario for this method seems to be limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ty67-J_yW50", "reviewer": "Reviewer_mwTd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1) The problem of model ensemble for self-supervised learning studied in this paper is interesting and important. It could potentially improve the performance of current self-supervised learning methods and achieve robust results in real-world applications.\n2) The proposed method of learning both feature vectors and MLP is reasonable.\n3) Experimental results show that the proposed method can improve the performance of an existing representation from a pre-trained self-supervised learning model.\n\nCons\n1) The main experimental comparisons are based on KNN classification without data augmentation, which is not a common practice in self-supervised learning. Self-supervised representation learning methods usually compare the quality of the learned representation on different tasks, for example linear classification, object detection or semi-supervised learning. The reported KNN result of a 58% accuracy is far behind the state-of-the-art performance of self-supervised learning on ImageNet.\n2) I have doubts about the comparison with the learned representation vectors with baselines. As reported in Figure 7, the learning process with MLP has already improved upon original individual features with one model. Then it is unclear to me whether the method would still improve upon similar learnings on top of concatenation or averaging, since the baselines from Fig. 2 to 5 directly use features without learning if I am understanding correctly. Besides, I think an important baseline is missing, which is simply doing ensemble based on the outputs of linear classifiers trained on top of every models.\n3) Figures in this paper are indistinguishable when not viewing in colour.\n\nI am curious about the results in Figure 9. It seems that transferring the MLP does not lead to improved performance. But how many data examples are needed for the target task so that the MLP per-dataset can be well-trained to have the ability of proper generalization?", "summary_of_the_paper": "This paper presents a representation learning method to optimize individual data representation vectors as well as an MLP encoder so that the learned representation vectors can recover the features from multiple pre-trained self-supervised learning models. Experimental studies show that such learning method can outperform baselines including individual feature, concatenation and averaging in terms of KNN accuracies.", "summary_of_the_review": "This paper works on combining the pre-trained models from self-supervised learning, and the proposed method is well-motivated and reasonable, though not technically novel enough. However, experimental evaluations are not convincing enough and lacks comparisons in many situations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QmhlPuRBb35", "reviewer": "Reviewer_TyZ7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strength:\nThe paper is well-written clearly structured in general. And the performance looks well on the K-NN classification.\n\nweakness:\nThe paper proposes a simple ensembling method that optimizes the self-supervised pretrained model. But some results and explanations are not very clear in the paper and the experimental section has to be improved. I think the contribution of this paper is not significant enough for ICLR.", "summary_of_the_paper": "This paper propose a new way to learn self-supervised model ensembling. Their novel approach learns representations via gradient descent directly at inference time after having pretrained feature extractors. And the authors conduct a series of experiments to show the efficacy of their method. ", "summary_of_the_review": "1) As mentioned in the appendix, the ensembling method seemingly does not work on the linear regression evaluation but only boosts the performance on the K-NN classification. How to explain that phenomenon? Why not use the linear probing or finetune protocol to evaluate the representation? Also, I suspect the superior performance is highly dependent on the self-supervised methods, e.g., Barlow Twins，MoCo, SimcCLR, etc. Since the feature extractor is the ultimate goal in self-supervised learning, it would be more convincing if authors can propose an approach to train end-to-end in a self-supervised manner. \n\n2) Does the proposed method only boost K-NN classification performance? If so, I think the contribution of this work is not enough because K-NN classification is only a very limited downstream task for self-supervised learning. In addition, the proposed method need to be trained based on a pretrained model, and be fine-tuned during test stage. Compared with the cost of time, the improvement is trivial. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to perform self-supervised model ensembling by learning representations directly through gradient descent at inference. The effectiveness is evaluated by k-nearest neighbors accuracy.

The reviewers agreed that the paper studies an important and interesting problem of leveraging model ensembling for self-supervised learning, which could improve both the performance and robustness of the learned representations. However, the reviewers also agreed that there were issues with the soundness of the empirical evaluation, which was a key reason for rejection.
Decision: Reject