Weighted Ensemble Self-Supervised Learning
Authors: Yangjun Ruan
Saurabh Singh
Warren Richard Morningstar
Alexander A Alemi
Sergey Ioffe
Ian Fischer
Joshua V. Dillon
Venue: ICLR 2023 poster (2023)
Keywords: self-supervised learning
ensemble
representation learning

Abstract: Ensembling has proven to be a powerful technique for boosting model performance, uncertainty estimation, and robustness in supervised learning. Advances in self-supervised learning (SSL) enable leveraging large unlabeled corpora for state-of-the-art few-shot and supervised learning performance. In this paper, we explore how ensemble methods can improve recent SSL techniques by developing a framework that permits data-dependent weighted cross-entropy losses.  We refrain from ensembling the representation backbone; this choice yields an efficient ensemble method that incurs a small training cost and requires no architectural changes or computational overhead to downstream evaluation. The effectiveness of our method is demonstrated with two state-of-the-art SSL methods, DINO (Caron et al., 2021) and MSN (Assran et al., 2022). Our method outperforms both in multiple evaluation metrics on ImageNet-1K, particularly in the few-shot setting. We explore several weighting schemes and find that those which increase the diversity of ensemble heads lead to better downstream evaluation results. Thorough experiments yield improved prior art baselines which our method still surpasses; e.g., our overall improvement with MSN ViT-B/16 is 3.9 p.p. for 1-shot learning.

Published as a conference paper at ICLR 2023
WEIGHTED ENSEMBLE SELF-SUPERVISED LEARNING
Yangjun Ruan"t
Saurabh Singh
Warren Morningstar
Alexander A. Alemi
Sergey loffe
Ian Fischer!
Joshua V. Dillon'
Google Research
ABSTRACT
Ensembling has proven to be a powerful technique for boosting model perfor- mance, uncertainty estimation, and robustmess in supervised learning. Advances in self-supervised learning (SSL) enable leveraging large unlabeled corpora for state-of-the-art few-shot and supervised learning performance. In this paper, we explore how ensemble methods can improve recent SSL techniques by developing framework that permits data-dependent weighted cross-entropy losses. We frain from ensembling the representation backbone; this choice yields an efficient ensemble method that incurs small training cost and requires architectural changes or computational overhead to downstream evaluation. The effectiveness of our method is demonstrated with state-of-the-art SSL methods, DINO (Caron et al. 202T) and MSN (Assran et al. 2022). Our method outperforms both in multiple evaluation metries on ImageNet- IK. particularly in the few-shot setting. We explore several weighting schemes and find that those which increase the di- versity of ensemble heads lead better downstream evaluation results. Thorough experiments yield improved prior art baselines which our method still surpasses; e.g., our overall improvement with MSN ViT-B/16 is 3.9 p.p. for 1-shot learning.
INTRODUCTION
INDIOO VIT-R/8
1.2
t 0.9
DDINO ENT (Our))
The promise of self-supervised learning (SSL)
3.1
1 1.1
to extract information from unlabeled data and leverage this information in downstream
Accuracy
70
6.1
tasks (He et al. 2020 Caron et al., 2202 ); e.g., semi-supervised learning (Chen et al. 2020a b).
60
7.5
robust learning (Radford et al. 1021] Ruan et al.
30
2022 Lee al. 2021), 1), few-shot learning (Ass-
I-shot
2-shot
5-shot
1%
KNN Linear
ran et al. 2022), and supervised learning (Toma-
Figure 1: Our improvements to DINO, including
sev al., 2022). These successes have en-
baseline improvements and ensembling.
couraged increasingly advanced SSL techniques (e.g..
al. 2020; Zbontar et al., 2021 He et al. 2022). Perhaps surprisingly however, a simple and otherwise common idea has received limited consideration: ensembling.
(e.g..Grrll Grill
Ensembling combines predictions from multiple trained models and has proven effective improving model accuracy (Hansen & Salamon 1990; Peerrone & Cooper 1992) and capturing predictive
uncertainty in supervised learning (Lakshminarayanan et al.) [01 7 Ovadia et al.) [209). Ensembling in the SSL regime is nuanced, however; since the goal is to learn useful representations from unlabeled data, it is less obvious where and how to ensemble. We explore these questions in this work.
We develop efficient ensemble method tailored for SSL that replicates the non-representation parts (e.g., projection heads) the SSL model. In contrast with traditional "post-training" ensembling, our ensembles only used during training to facilitate the learning of single representation encoder, which yields no extra cost in downstream evaluation. We further present a family of weighted cross- entropy losses effectively train the ensembles. The key component of our losses is the introduction of data-dependant importance weights for ensemble members. We empirically compare different choices from our framework and find that the choice of weighting schemes critically impacts ensemble diversity, and that greater ensemble diversity correlates with improved downstream performance, Our method is potentially applicable to many SSL methods; focus on DINO (Caron et al., (02T)) and MSN (Assran et al.] (002) to demonstrate its effectiveness. Fig. I shows DINO improvements from using our ensembling and weighted cross-entropy loss.
"University of Toronto & Vector Institute. Work done as a student researcher at Google. 1 "Correponndence yjruandcs. toronto. edu, aiansf, jvdi 1 8goog le. com.

---

Published as a conference paper at ICLR 2023
In summary, core contributions to:
Develop a downstream-effin ensemble method suitable for many SSL techniques Secc. 3.1)
Characterize an ensemble loss family weighted cross-entropy objectives (Sec. 3.2
Conduct extensive ablation studies that improve the prior art baselines by up to 6.3 p.p.
Sec.. 5.1
Further improve those baselines with ensembling (e.g., up to 5.5 p.p. gain for 1-shot) (Table 2b.
2
BACKGROUND
In this section, we frame SSL methods from the perspective of maximum likelihood estimation (MLE) and use this as the notational basis to describe the state-of-the-an clustering- SSL methods as well as derive their ensembled variants in Sec.
From Maximum Likelihood to SSL Denote unnormalized KI divergence (Dikmen al. 2014) between non-negative integrable functions q by K(p(X), ( = Hx qX)),
X]]
where H* (x(X),(XX) = fx p(x) log g(x) flrr+J. Jx ((x)dx is the unnormalized cross-entropy (with log 0 = 0) and H[p(X)] = H* [p(X), (.),,((. p(X)] These quantities simplify their usual definitions when q are normalized, but critically they enable flexible weighting of distributions for the derivation of our weighted ensemble losses in Sec. 3.2]
Let v ((,Y) = (X be nature's distribution input/target pairs over the space x y and XX) be a predictive model of target given the input parameterized by 9 T. Supervised maximum likelihood seeks the minimum expected conditional population risk with respect to 8,
Ex(x) K(r(Y|X), s (X) Ex(x) HM [x(Y s(1 x)] Ex(x) H[)(Y]x)). (I)
Henceforth omit (x) H[)(Y[X)) since it is constant in 6. Since vxX, Y) is unknown, a finite sample approximation is often employed. Denote a size-n i.ddd training set by Dn = {2 heemi ~ e and empirical distribution by ((X Y)
(y 8(X - =))(( y) where 8: R - {0,1) is when x and otherwise. The sample risk is thus - ErED, H () s sY
In SSL,, we interpret v(Y (x) as being the oracle teacher under a presumption of how the rep- downstream task. This assumption is similar that made
resentations will be evaluated in Arora et al. (2019); Nozawa et al. (2020). We also assume D(Y (X) is inaccessible and/or un- reliable. Under this view, some SSL techniques substitute (({(z) (z) for weakly learned target or "teacher", tx)]x). We don't generally expect t(Y[x: to recover v(Y [z); we only assume that an optimal teacher exists and it (Y(r.. With the teacher providing the targets, the loss becomes
E.ED, H [t(( (r), sY x).
Teacher and student in clustering SSL methods Clustering SSL methods such as SWaV (Caron al. 2020), DINO (Caron al., 2021), and MSN (Assran et al., 2022) employ a student model characterized by proximity between leameed codebook entries and data dependentn code,
s(Y(0, softmax (f
(hy (z) My y e [c] (dl)
(1hpp
(w, v, (Hyyeele) TT,
where the encoder Tw X + Z produces the representations used for downstream tasks, and the projection head h. + 2d and codebook entries {ny]eey Rd characterize the SSL loss. Eq. (2) can be viewed as "soft clustering". where the input is assigned to those centroids that are closer to the projection head's output. The projection head and codebook are used during training away for evaluation, which is empirically found vital for downstream tasks (Chen al. 2020a b). Hyperparameters T E Rxo, c E Zzo represent temperature and codebook size. The teacher is defined <(Y[=) = s(Y stopgrad (g))), x²) where g
thrown
T. Commonly s(0) is exponential moving average gradient descent iterates and the teacher uses a lower temperature than the student.
To capture desirable invariances and prevent degeneracy, data augmentation and regularization (e.g.. Sinkhorn- Sinkhorn-Knopp normalization (Caron et al. 22020), mean entropy maximization (Assran et al. 2022 essential. As these not directly relevant method, omit them for brevity.
2

---

Published as a conference paper at ICLR 2023
3 METHOD
Ensembling is technique that combines models to boost performance, and has been especially successful in supervised learning. We are interested in ensembling methods that carry over this success SSL approaches. However, SSL has key differences, such as throw-away "projection heads", from supervised learning that result in multitude of possibilities for how to ensemble. With this in mind, we propose first where to ensemble, and then how ensemble. Those proposals result in an efficient "peri-training" ensembling technique specifically tailored for SSL and family of weighted ensemble objectives; we subsequently suggest different ways to select the weights.
3.1 WHERE TO ENSEMBLE?
Encoders
Heads
Preds.
Losses
Denote the teacher/student ensembles by and define
fn
(t and each as in Sec. 2 parameters 0 = {0,} } | T independently initialized, all students temperature and all teachers another. We asymmet- rically denote 1, (Y(z)) and s(Y [0]]0) 2r) emphasize that teachers' gradients are zero and that the students are distinct solely by way of 0, 0j. Studying heterogeneous architectures and/or different teacher parameterizations is left for future work.
WP11
Teacher Fz
Pym
H
moving average
H
Student r
Mmnm
Recall that O parameterizes the encoder, pro- jection head, and codebook parameters: 0 (wg, thy, (Mig hvey). We further restrict 7 such that
Figure 2: Overview of (hy. ensemmbbe Two augmented inputs are encoded by the teacher/student into representations, and then processed by an ensemble of heads. The loss for each head weighted and summed into the final loss. Strike-through edges indicate stop-gres cop-graddents. See Appp. A for pseudocode.
wi = Wy, i.e., we limit our consideration to ensem- bles of projection heads hes, and/or codebooks μi but not encoders Tw- This choice makes our en- semble method inherently different from traditional supervised ensembling or encoder Tw ensembling: the ensembled parts not used for evaluation but for improving the learning of non-ensembled representation encoder during training, thus it requires change of downstream evaluation computational cost.
Ensembling of rw is left for future work.
3.2 HOw TO ENSEMBLE?
We would like extend the loss support an ensemble of teacher/student pairs while respecting the MLE intuition of the loss as in Sec. 2 Additionally, we want to facilitate data-dependent importance weights, thus enabling preferential treatment of some teacher/student pairs. We therefore propose weighted average (unnormalized) cross-entropy loss,
(.) = Σ
Σ
H HM 2 (Y((1=),)(1103))
(4)
n
zED i.jE[m]
softmax (( ; e [ml})
where Wijy
(5)
The notation why O (Y(()))) denotes Hadamard product; i.e., the product of event-specific weights and probabilities for each y E ). The hyperparameter y is the temperature. The function Jijy is defined for brevity and discussed in the following section.
This objective admits generality and flexibility for introducing various weighting schemes, as it supports potential interactions between all teacher/student pairs and allows the weights to be both model- and data-dependent. Up to constant independent of 0, it is importance weighted average of (unnormalized) KL divergences between each teacher and each student; i.e., a mixture of MLE-like objectives. We stop the gradient of Wijy to 0 in order to keep the overall gradient weighted average of students' log-like likeliiodoood gradients, similar to Eq. (I). We also normalize the weights such that each data point equally contributes the loss.
3

---

Published as a conference paper at ICLR 2023
3.3 HOw TO WEIGHT?
In this section, we present several instantiations of our losses with different weighting schemes. We empirically show in Sec. 5 that the particular choice of weighting scheme is critical for the representation performance and the induced diversity of (he, -ensemblss.. For simplicity we assume 7 in this section. We indicate with O
that loss has the same arg min [q.
For additional analysis and discussion, see see Appx.
Uniform weighting (UnIF) The simplest strategy is treat different teacher/student pairs inde- pendently and average each with uniform weighting: i.e.,
Jaju log 5(i -
UNIF L (0)
Σ
HJ HY kE/m)
Σ
(6)
This strategy introduces uniform weights w; m ensemble elements. The role of log 8(i j) (here and elsewhere) is to sub-select corresponding teacher/student pairs rather than all m² pairs.
Probability weighting (PROB) An alternative to using the average cross- cross-entropy loss (UNIF) is to compute the cross-entropy loss of the average predictions whose gradient weighted by Wajy (see Appx. D.I. At 1, those gradient weights simplify into an average over the student probabilities:
fug = log xx)
PROB (0) =
H RED
Σ
Σ (Y(2)) <E(m
1 Σ SY
(7)
m
m JELMJ
Averaging the predictive distributions introduces correspondence between codes from different heads; thus different heads are no longer independent but instead cooperale to match the student to the teachers. The loss favors student heads with more confident predictions (i.e., larger Kyll0,=0) ):) Further motivation for averaging predictions comes from multi-sample losses studied in Morningstar al. (2022). Note that the joint convexity of (unnormalized) KL divergence implies that this loss is upper bounded by the UNIF loss up to some constant in (see Appx. D)
Although the PROB strategy favors confident student predictions, the weights change as a function of E y. This may be in conffict with our intuition that SSL is like maximum likelihood (Sec. 2). since under that view, the teacher responsible for weighting outcomes.
Entropy weighting (ENT) Another way to favor heads with more confident predictions is to directly weight by their predictive entropies; i.e.
fejy
+ log 5(i -)
(8)
CNNT (0) Σ Σ
softmax; H(4 (Yk)) i' e [m]]) H* (c(Y)),,&(Y10,
(9)
DED mm(m
where the weight w: softmax, H(4 : :'' E [m]}} is inversely correlated with the entropy of teacher predictions. In other words, the head whose teacher has lower entropy (i.c., higher confidence about its prediction) is given a larger importance weight for learning the representation. Like PROB, this strategy encourages "data specialists" by emphasizing strongly opinionated teacher heads for different inputs. Like UNIF, different heads are treated more independent (than PROB), since interaction between different heads is introduced only through the weight computation. By preferring low-entropy teachers we also favor low variance teachers; this aligns with the intuition that using a lower-variance teacher benefits representation quality (Wang al. 2022).
Countless other weighting schemes is impossible to fully explore the space weightings; the following might also be interesting to study in detail but were omitted due to resource constraints.
Jzju
(Favors all pairs of teachers/students equally)
(10)
fajy log (y(l:)
(Favors opinionated teachers)
(11)
friv = - H[s()|0g..)
(Favors low-entropy students)
(12)
fAjU ((/(())a),<((1000
(Favors disagreeing teacher/student pairs)
(13)
4

---

Published as a conference paper at ICLR 2023
fujv - log (M [ + +c)
(Favors low variance teachers; e.g., < 122
(14)
Note that "aligned" versions of all schemes are possible by using fivv + log ő(i j). We did early experiments exploring Eqq. (IT) and (12) but the results were inferior and are largely omitted below.
4 RELATED WORK
Self-supervised learning Recent work on self-supervised learning (SSL. focuses on discriminative generative approaches. Most discriminative approaches seek to learn iggnenttaionniivariant representations by enforcing the similarity between augmented pairs of the same image while
utilizing different techniques avoid collapse. Contrastive methods (Chen al. 2020a He et al.)
2020 Wu et al. 2018 Hjelm et al. 2018: Bachman et al. 2019 Tian et al. 2020) use large number
of negative samples with a
objective (Gutmann & Hyvärinen, 2010; Oord et al.)
2018). A large body of followup work eliminates the necessity of explicit negative samples with
various techniques, including clustering assignment constraints (Caron et al. 2018 2020 2
Asano et al.) 2019), bootstrapping (Grill et al. 2020) or self-distillation (Caron et al. [021] inspired
by mean teacher (Tarvainen & Valpolal 2017). asymmetric architecture design (Grill et al. 2020
Chen & He 2021), or redundancy reduction (Zbontar al., [021] Bardes et al. 2021). Recent
generative approaches that use masked image modeling as the pretraining task (Dosovitskiy et al.
2020 Bao al.) [021] He et al.) 2022] Zhou et al. [2022] Xie et al. (022) have achieved competitive
finetuning performance. Our method may be applicable to all of the above methods that have some sort of "projection head", such as most of the discriminative approaches.
Ensemble methods Ensembling has been extensively studied for improving model performance Dietterich] (2000) and uncertainty estimation 2019 in supervised learning and semi-supervised
(Hansen & Salamon 1990, Perrone & Cooper
1992:
[Lakshminarayanss et al. 2017] et al.
learning (Laine & Aila| (0116). A major research direction to train efficient ensembles with partial
parameter sharing (Lee et al. 2013 Wen et al.
2020 Dusenberry et al. 2020; Havasi et al.) 2020) or
intermediate checkpointing (Huang et al. 2017 Garipov al. [2018). Our method also shares the encoder parameters across ensembles, which is closely related to heali--headed networks (Lee et al.
2015 Tran et al. 2020). Ensemble methods for SSL are less explored. Some recent work studies ensembles of supervised models adapted from pretrained SSL models. [Gontjo-LLppss et al. (2022) conduct an empirical study of ensembles adapted from different SSL models and find that higher divergence in SSL methods leads to less correlated errors and better performance. Wortsman et al. (2022) ensemble multiple finetuned models adapted from the same SSL model by averaging their weights, which boosts the performance without any inference cost. Our method differs from them in that (1) applies to the SSL training stage to directly improve representation quality, rather than aggregates multiple models in the post-training/finttnni
stage: (2) introduces little training cost and no evaluation cost; and (3) is complementary to these post-training/fieeuning ensembling methods.
EXPERIMENTS
We carefully study the impact of (he. -.eeemmbeesssss and our selected weighted ensemble losses (UNIF, PROB, and ENT) on smaller DINO models in [Sec. 5.1] Using what we learned in those experiments, in Secc. 5.2 we present new state-of-the for multiple model sizes by ensembling both DINO- and MSN-based models. Finally, we explore ensemble evaluations in the transfer learning setting in [Sec. 5.3 Additional experimental details and results are in Appx. and Appx. C. respectively.
results on ImageNet-IK on various metrics
Experimental setup We assessed the effectiveness of our method with two SSL methods: DINO (Caron et af., (021) and MSN (Assran et al.) 2022). In order to ensure that we are comparing against strong baselines, we consider three different classes of baselines: (1) evaluation numbers reported in the original works (Caron et al. (2021) Assran et al. (2022), and Zhou et al. (2022) for an additional baseline iBOT); (2) evaluation of our implementation using the hyperparameters reported in the original works (DINO only, for space reasons) to validate our implementation; and (3) evaluation of implementation using the best hyperparameters that found by tuning the baselines (DINO and MSN) for fair comparisons. In almost all models and evaluations, our retuned baselines give non- trivial performance improvements on top of previously reported numbers. These type (3) baselines
5

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of different ensemble strategies. ENT and PROB significantly improve over the non-ensembled baseline, while UNIF leads to no gains. Ensembling both the projection head and the codebook works the best. All models are DINO ViT-S/16 trained for 300 epochs. Averages and standard deviations are over 3 initialization seeds. The linear evaluation results on ImageNet-Ie IK with different amounts of labeled data are reported here (see Table 11
Appx. C.3
for all metrics).
Where
# of Labels Per Class
How
Proj. Aig
Code.
5
10 (1%)
Full
Base
40.6 +
57.9 + 0.3
63.4 = 0.2
74.4 + 0.1
UNIF
40.4 = 0.4
57.6 + 0.3
63.3 + 0.3
74.5 + 0.2
PROB PROB
/
39.8 + 0.5 1 0.9
57.4 +: 0.4 | 0.5
63.0 0.4 | 0.4
74.8 + 0.1 f 0.4
41.9 = 0.3 1 1.3
59.6 = 0.4 1.7
65.1 1.7
75.4 + 0.1 1.0
ENT-ST
40.0 + 4 0.6
57.3 + 0.6 0.6
62.7 0.7
0.4 + 7..
ENT ENT
40.8 = 0,4
58.0 - 0.4
63.5 + 0.4
74.5 + 0.3
43.0 = 0.6 1 2.4
59.7 = 0.7
1 1.8
64.8 0.5 1.4
75.1 + 0.4 0.7
ENT
44.0 = 0.2 f 3.4
60.5 + 0.3 2.6
65.5 0.1
2.2
75.3 + 0.1 1 0.9
we label DINO" and MSN', and we use them as the base models for our experiments with (hy (+)- ensembles and weighted ensemble losses. Appx. B.2.1 describes the details for getting such strong performance for DINO and MSN In particular, we find that the projection head has a crucial impact on label efficiency of representations and using a smaller head (3-layer MLP with hidden size 1024) significantly improves few-shot evaluation performance (see Appx. C.2).
We compared models trained with and without our (hw (4) -eeeembbess by mea- suring various evaluation metrics on ImageNet-IK (Deng et al., 2009). The evaluation metrics reflect the decodability and the label efficiency of learned representations. We measured the decodability with respect both the linear classifier following the common linear evaluation protocol and the k-NN classifier following Caron et al. (202I). We measured the label efficiency by evaluating the linear evaluation performance in few-shot settings, including 1 1% (~~1-shots) labeled data evaluation (Chen et al. 2020a) and 1-/2-/5-shot evaluations (Assran et al., 2022). All evaluations used frozen representations of the teacher encoder we did not fine tune the models. See AceeApp. B.3 for details.
Evaluation metrics
5.1 EMPIRICAL STUDY OF ( ENNSEMLESS
Table I compares different strategies for where and how to ensemble. Fig. 4 compares the impact of the weighted ensemble loss on (ha. -eeseemble diversity. Fig. shows the effect of increasing the number of ensembles, adjusting the temperature y, and increasing baseline projection head parameters. In these experiments, we used DINO" with ViT-S/16 6 trained for 300 epochs as the base model. We compared different ensemble methods applied the base model with which we found to work the best. For the ENT strategy in Table I the entropy weighting temperature y is set to 0.05 log(c) by default which is selected from f0.0125, 0.025, 0.05, 0.1, 0.2} x log(c), where the scale log(c) gives the maximum entropy of the codebook size c. For PROB, we keep 7 1.
16 heads
Where to ensemble We study the where question by ensembling either the projection head h the codebook μ. or both with the ENT and the PROB ensemble strategies, as shown in Table I We find that ensembling both he and μ provides the largest gains for both losses, probably due to the increased flexibility for learning diverse ensemble. Interestingly, only ensembling hy also works well for the ENT strategy.
How to ensemble We study the how question by considering four different loss variants: UNIF, PROB, ENT, and the variant of ENT with student entropy weighting. We find that when we ensemble both the projection head he and the codebook μ. the ENT ensemble strategy leads to the most significant gains (e.g.. 3.4 p.p. gains for 1-shot and 0.9 p.p. gains for full-data). The PROB strategy also consistently improves the performance with a slightly larger gain (1 p.p.) in full-data evaluation. In contrast, that uses the student entropy (i.c.. Eq. (12) with the log 5(i - j) term) for the importance weights (denoted as ENT-ST). ENT-ST performs much worse than ENT and is even worse than the baseline.
no gains for the UNIF strategy over the baseline. We also study a variant of ENT

---

Published as a conference paper at ICLR 2023
749
753
75.0
74.
66.3
65.4
Accuracy
$4.4
Accuracy
04-
43.5
46.0
44.0
Defaut Our Ourbaslin
cata
0.005
0.005
0.02
0.1
02
10
10"
10"
Num of Ensembbles
Entropy Weight Temperature
Number of Heed Peremters
(a) Scaling of (hep enssemblss..
(b) Effect of ENT temperature y.
(c) Comparing different heads.
Figure Empirical study of (he, w,, ensembllss.. (a) The gains of (hw, enseembbllss start to diminish above 16 heads. (b) The temperature for entropy weighting has larger impact on few-shot performance. 16 heads are used and y is scaled by (o)( (c) Our (he: jl)-emmeblles outperform all non-ensembled baselines when controlling for number of parameters. A too powerful non-ensembled projection head significantly harms accuracy. 1%-data evaluation is shown. Also see Fig.
We conjecture that this is because the student predictions typically have larger variance than teacher predictions (Wang et al. (2022) especially when multi-crop augmentation (Caron et al.] 2020) (021) is applied to the student. Similar experiments on Eq. (11) and/or y 0 variants of PROB also resulted in inferior performance (see Table 12).
Analysis of (hg, p-)-emmeble diversity The previous exper- iments showed that the choice of ensemble weighting strategy has a large impact on performance. We hypothesize that this choice substantially impacts the diversity of the codebook en- sembles. Since the codes in different heads may not be aligned, align them by the similarity of their code assignment prob- abilities across different input images, which measures how the codes are effectively used to 'cluster' the data. See Sce|Appx. C.4 for detailed explanations and results. In Fig. 4 we visualize the decay patterns of the similarity score between aligned codes (1.0 means the most similar) in random pair of heads for each weighting strategy. ENT decays the fastest and UNIF decays the slowest, indicating that ENT learns the most diverse codebooks while UNIF least diverse. This shows a positive correlation between the diversity of (h.. -e)-embeblss and the empirical
1.0
Unif
Prob
0.5
Ent
Figure 4: Visualization of code similarity. ENT learns the most di- verse (h -e)-ennebless reflected by the fastest decay of similarity scores between aligned codes in dif- ferent heads. UNIF has low diver- sity between heads.
performance of the ensemble strategies from Table 1 Finally, for UNIF, we find that different heads tend to learn the same semantic mappings when randomly initialized; i.e., the code assignments in different heads become homogeneous up to permutation. See Fig. 8 for a visualization.
Number of (he, (2) ensembles We study the effect of increasing the number (hg. -enemmbbss for ENT in Fig. 3al Having more (t),, ensembless boosts the performance until m 16. Interestingly, using as few as m = heads already significantly improves over the baseline.
Effect of ENT temperature 7 Fig. 3b studies the effect entropy weighting temperature y for different evaluation metrics. We observe that has a relatively larger impact on few-shot evaluation performance. ? should be neither too high nor too low: high temperature leads to under-specialization (i.e. less diversity) of heads similar to UNIF (y oo) and low temperature may otherwise lead to over-specialization (i.e., only a single head used for each input).
Comparison of different projection heads Our method linearly increases projection head param- eters, thus natural question is: Is the gain of (hy, (i)-.--eeeessees due to the increased power (or number of parameters) in projection heads? We answer this question with an empirical study of non-ensembled projection heads. Specifically, we studied non-ensembled h2 with (depth, width) searched over (2,3.4) x (512, 1024, 2048, 4096) and measured the linear evaluation performance with different amounts of labeled data. In Fig. 3c) plot the 1%-data evaluation result with respect to the number of parameters of the projection head both for ensembled and ensembedele baselines. See Appx. C.2 for detailed analysis and extra results for other metrics. Our key findings are:
7

---

Published as a conference paper at ICLR 2023
Table 2: Effectiveness of ensemble heads for DINO"/MSN with different VIT models. Our ensemble heads consistently improve all downstream evaluation metrics on ImageNet- IK and achieve new state-of-the- art for few-shot evaluations. For ViT-S/16, report linear evaluation results probed from the last layer (left) and from the last 4 layers (right, following DINO). TW evaluated the few-shot settings using DINO's publicly-availate
pretrained weights in the cases those results were not reported in Caron et al. 2002)) MSS VřT-B/16 and ViT-B/8 are both trained for 600 epochs in Assran et al. (202)) whereas models are trained for only 400, 300 epochs, respectively. For each architecture, we highlight the best DINO baseline and weighted ensemble in blue. For MSN, the corresponding highlights are yellow. The best results for each architecture and metric are bolded.
Few-shot
Full-data
Method
5
~0 (1%)
k-NN
Linear
VIT-S/16, epochs
iOOT
40.4 0.5
50.8 - 0.8
59.9 0.2
65,9
75.2
/ 77.9
DINO
38.9 0.4
48.9
0.1
64.5
74.5
76.1 77.0
DINO (Repro)
0.5
58.6 0.2
64.7
74.3
7
DINO (Retuned)
44.6 0.2
0..
61.1 0.2
66.2
74.1
7
MSN
47.1 0.1
62.8 0.6
67.2
766.9
MSN (Retuned)
47.4
5..3
62.8 + 0.2
67.1
73.3
76.6/
DINO PROB (16)
45.2 0.4
54.9
62.5 0.2
67.3
75.1
76.6
DINO -NNT (4)
46.3 0.1
55.5
6.3
67.5
74.8
76.2 77.2
DINO -ENT (16)
47.6 + 0.1 3.0
56.8 0.5
6..0
68.3 f 2.1
75.3
76.8 77.7 1 0.8
MSN ENT (2)
48.8 +
57.5
64.0 0.2
67.9
74.6
76.0 76.9
MSN ENT (8)
50.1 + 0.1 1 2.7
58,9 + 0.6
65.1 0.3
68.7 1.6
75.2
76.4/77.44 0.8
ViT-B/16, 400 epochs
BOOT
46.1 +: 0.3
56.2 2: 0.7
64.7 + 0.3
69.7
77.1
79.5
DINO
43.0 + 0.2
52.7 + 0.5
61.8 + 0.2
67.4
76.1
78.2
DINO (Retuned)
49.3 - 0.1
58.1 = 0.5
65.0 0.3
69.1
76.0
78.5
MSN
49.8 + 0.2
58.9 + 0.4
65.5
MSN (Retuned)
50.7 +
59.2
65.9 0.2
74.7
78.1
DINO ENNT (16)
52.8 + 0.1
1 3.5
61.5 1 0.4
67.6 + 0.3
71.1 2.0
77.1
79.1
0.6
MSN ENT (8)
53.7 1 0.2
3.0
62.4 + 0.6
68.3 + 0.2
71.5 1.8
77.2
78.9 1 0.8
VT-B/8, 300 epochs
DINO
47.5 0.2
0.5
65.4 0.3
70.3
77.4
80.1
DINO (Retuned)
49.5 0.5
0.6
65.9 $..9000 0.3
70.7
77.1
80.2
MSN!
55.1
71.6 0.7
MSN (Retuned)
51.9 =
61.1
67.7 0.3
71.7
75.7
80.3
DINO (16)
55.0 0.4 5.5
63.4 1.44000. 0.6
69.5 0.3
73.4
2.7
78.6
81.0 1
0.8
MSN*-ENT (8)
55.6 + 0.2 3.7
64.5 + 0.5
70.3 0.2
73.4
1.7
78.9
80.8
0.5
too powerful non-ensembled hy significantly hurts the label efficiency of learned representa- tions. This result is similar to Chen et al. (2020b), which found that probing from intermediate layers of projection heads (which can be viewed as using a shallower head) could improve semi-supervised learning (1%-/10% labeled data) results.
The default head (3/2048, denoted as 'Default") used recent SSL methods (SimCLRv2, DINO, MSN, etc.) does not perform as well in few-shot evaluations, probably because it is selected by looking at full-data evaluation metrics. contrast, our baseline (3/1024, denoted as 'Our baseline') significantly improves few-shot evaluation performance.
Our (hy, p)) -eneembees outperform all non-ensembled baselines and lead consistent improve- ments in all evaluation metrics, despite the increase of parameters.
5.2 IMPROVING SOTA RESULTS WITH ENSEMBLEING
Next we apply (hg p)eenmebbles to DINO and MSN* and compare with the state-of-the-arr results. We experimented with model architectures ViT-S/16, ViT-B/16, VIT-B/8 trained for 800, 400, 300 epochs respectively following Caron et al. 2021). We include both the published results and our retuned versions to ensure strong baselines. For clarity, we denote (ensemble strategy] (# of heads)", e.g., DINO (4). We tuned both baselines and our methods for all architectures. We report the best hyperparameters for all models in Appx. B.2.2
method as "I "[baseline] }-
8

---

Published as a conference paper at ICLR 2023
Table 2 compares the results of (hg. ensembles and baselines. We find that (hg, ensembllss with ENT consistently improve all evaluation metrics (full-data, few-shot) across both SSL methods (DINO", MSN') and all architectures (ViT-S/16, ViT-B/16, ViT-B/8) over their non-ensembled counterparts. The gains in few-shot evaluation are particularly substantial, providing a new state-of- the-art for ImageNet- K evaluation from ImageNet pretraining.
5.3 MORE EVALUATIONS FOR (his BNNSSEBBBES
Table 3:
Comparison of transfer performance. ViT-S/16 used. Our ensemble heads lead to consistent improvements for MSN and comparable results for DINO
Food101
CIFARIO
CIFAR100
SUN399
Cars
DTD
Pets
Caltech- 101
Flowers
Avg.
DINO
78.4
93.8
81.0
66.1
66.7
74.6
92.0
94,9
94.4
82.43
DINO EENO (16)
79.1
93.8
81.4
66.5
66.8
74.9
92.8
94.6
93.9
82.64
MSN MSN"-ENT (8)
77.7
93.1
79.8
64.6
63.3
72.2
92.4
94.7
92.7
81.17
78.4
93.9
81.1
65.2
68.0
73.2
93.1
95,4
92,8
82.34
Transfer learning In Table we compare the transfer learning performance of (he. jeneembbell
and non ensembldelled baselines. We used ViT-S- 16 models trained on ImageNet- IK for 800 epochs and evaluated on 9 natural downstream datasets from Chen et al. (2020a) with linear evaluation (details in Appx. B.3). (he, je--emmebbllss lead to consistent improvements in transfer performance for MSN and comparable results for DINO
Training overhead In Table 4] we benchmark the compu- tational overhead of (heg, -eneembees at training time. We used medium sized model, DINO with ViT-B/16, trained with the same setting used in all of our experiments. We bench- marked the wall-clock time and peak memory on 128 TPUv3 cores. (hy. p1) ensembling is relatively cheap in terms of train- ing cost because the ensembled parts typically account for small portion of total computation, especially when the back- bone encoder is more computationally expensive (e.g.. ViT-B/8).
Table 4: Training overhead. Wall- clock time and peak memory per core for training with different num- bers of ensembles.
m
Wall Time
Peak Memory
5.81h
5.25G
4
5,91h
5,40G
16
6.34h
5,89G
Again, we emphasize that there is no evaluation overhead when (hy, meeessembls are removed.
6 CONCLUSION & DISCUSSION
We introduced an efficient ensemble method for SSL where multiple projection heads are ensembled to effectively improve representation learning. We showed that with carefully designed ensemble losses that induce diversity over ensemble heads, our method significantly improves recent state-of- the-art SSL methods in various evaluation metrics, particularly for few-shot evaluation. Although ensembling is a well-known technique for improving evaluation performance of a single model, we demonstrated that, for models with throw-away parts such the projection heads in SSL, ensembling these parts can improve the learning of the non-ensembled representation encoder and also achieve significant gains in downstream evaluation without introducing extra evaluation cost.
Our ensemble method is applicable to many SSL methods beyond the two we explored. For example, one may consider generalization to BYOL (Grill et al. (2020) or SimSiam (Chen & He (021) that ensembles projection and/or prediction heads, or MAE (He et al. (202)) that ensembles the decoders (which introduces more training cost though). Our weighted ensemble losses can also be applied as long as the original loss can be reformulated as MLE for some t, and Y, e.g., the MSE loss in these methods MLE under multivariate normal distributions. We hope our results and insights will motivate more future work for extending our method or exploring more ensemble techniques for SSL.,
In future work, we also hope to remove three limitations of our setting. First, considering ensembling strategies that include the representation encoder, Tw. may lead to further improvements in the performance of weighted ensemble SSL, at the cost of increased computation requirements during both training and evaluation. Second, considering heterogenous architectures in the ensemble may further improve the learned representations (e.g., mixing Transformers with ResNets), whether the heterogeneity is in rw. hg, both. Third, considering other possibilities for fejy may also reveal performance gains and improve our understanding of the critical aspects that lead to good SSL representations, similar what we learned about the importance of ensemble diversity.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like thank Mathilde Caron and Mahmoud Assran for their extensive help in reproducing DINO and MSN baselines. We would also like to thank Ting Chen and Yann Dubois for their helpful discussions and encouragements.
REPRODUCIBITLITY STATEMENT
We include detailed derivations for all our proposed losses in Appx. D details in Appx. B] including the implementation details for reproducing the baselines (Appx. B. I). training and evaluating our methods (Appx. B.2.I), and all hyper-parameters (Appx. B.2.2) used in our experiments for reproducing our results Table
We report experimental
REFERENCES
TensorFlow Datasets, collection of ready-to-use datasets. ftt.ss//ww..eensof...oom
datasets
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprini arXivv1902.09929, 2019.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. arXiv preprint arXiv:1911.0537) 2019.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.07141, 2022.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2 08254, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: for self-supervised learning. arXiv preprint 1.3iv:2005.04906 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-100 101--mimmiigggg discriminative compo- nents with random forests. In European conference computer vision, pp. 446 Springer,
2014.
Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR, 2016.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 99-49499999, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 13:22243-22255 2020b.
10

---

Published as a conference paper at ICLR 2023
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, PP. 15750-15758, 2021.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Thomas M Cover and Joy Thomas. Elements of Information Theory. John Wiley & Sons, 1999.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1-15. Springer, 2000.
Onur Dikmen, Zhirong Yang. and Erkki Oja. Learning the information divergence. IEEE transactions on pattern analysis and machine intelligence, 37(7):1442-144 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Sever S Dragomir. A generalization of f-divergence measure to convex functions defined on linear spaces. Communications in Mathematical Analysis, 15(2):1-14, 2013.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma. Jasper Snoek, Katherine Heller, Balaji Lakshninarayanann and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pp. 2782-2792. PMLR, 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson.
Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations, 2022.
4gbGq IBG
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 13:21271-21288 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Abner Miizann-Rivera Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems, 25, 2012.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10):933-1001, 1990.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv preprint or arxiv:2010.666 2020.

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808. 06670, 2018.
Karan Grewal, Phil Bachman, Adam
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 846-6614 Springer, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. arXiv preprint D0Xiv:/704.00110 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ine-grained categorization. Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-99) 1955.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1rXiv:1610.022222, 2016.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems, 34:19538-19552 2021.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 151.0663 14, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Warren Morningstar, Alex Alemi, and Joshua V Dillon. Pacm-bayes: Narrowing the empirical risk gap in the misspecified bayesian regime. In International Conference Artificial Intelligence and Statistics, pp. 8270-8298. PMLR, 2022.
Wai Ho Mow. A tight upper bound on discrete entropy. IEEE Transactions on Information Theory, 44(2):775-778, 1998.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate o(1/k?). Proceedings of the USSR Academy of Sciences, 269:543-547, 1983.
Maria-Elema Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729. IEFE, 2008
Kento Nozawa, Pascal Germain, and Benjamin Guedj. Pac-bayesian contrastive unsupervised repre- sentation learning. In Jonas Peters and David Sontag (eds.), Proceedings of the 36th Conference Uncertainry in Artificial Intelligence (UAI), volume 124 of Proceedings Machine Learning Research, pp. 21-30. PMLR, 03-06 Aug 2020. URL https: ottps:://proceedlngs.s mlr press/ v1 21/4///zaawa html
12

---

Published as a conference paper at ICLR 2023
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Michael P Perrone and Leon N Cooper. When networks disagree: Ensemble methods for hybrid neural networks. Technical report, Brown Univ Providence Ri Inst for Brain and Neural Systems, 1992.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. PMLR, 2021.
Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal representations for covariate shift. In International Conference on Learning Representations, 2022. URL /ttpps://opeereeiee
net Btte://ormmiimee 3
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 151):1999-11988 2014.
llya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International conference on machine learning, pp. 1139-1147. PMLR, 2013.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.051/2 19, 2022.
Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint arXiv:2001.000 044694, 2020.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16570-16579, 2022.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lpess, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23998 PMLR, 2022.
13

---

Published as a conference paper at ICLR 2023
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-37422 2018.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URLbttps: /owpr//eeee.eeeeeeeeeooeo net forum? wttp://oruun..ddddooooeeeeee
14

---

Published as a conference paper at ICLR 2023
A PSEUDOCODE
Algorithm 1: Pseudocode for computing ensemble loss
b, by 0 batch 818e, number of ensemble heads, codebooks 6ize log_pe, log_pt: student, teacher log probabilities with ensembles strategy: ensemble loss avezage strategy tau_ent: temperature
entropy weighting
ensemble_ ps 1og_ptr strategy, teu_ent): b, tog_pt.ahopa it axis correspenda to ensemble 1og_pt stop_grad (log_pt) stop gradlent for teacher
if strategy -- "Unif":
loss (exp (log_pt) log_ps) aox(iigg--)))
loss = [asss i average over ensembles
strategy "prob":
log_mean_p: logaumexp pt, axis=1, b=1/n) rean teacher
_og_mean_pa Logaumexp (lo9_p8, axis=1, b-1/n) nean student
loss
(log_mean_pt) poo__mean_ps)esum.le
elif strategy "Ent";
(exp lug_pt)..umlaxiiee
tecchor entropy
weight softmax (tetmaxx-entttaaeentt axis=l) i entropy weights
loss (exp (log_pt) log_ps) aus
1089 (loas we'ght) sum entropy weighted average
mess. average samples
Algorithm 2: Pseudocode for ensemble heads with simplified DINO
3 Dr oy etas number of ensemble heads, codebook size, momentum updaLe cace f6, ft: student, teacher encoders
3 he_ene, at_ens: student, Leacher peotection heads with ensemblea, 11st with length a mus_ens, aut_ens: student, teacher codebooks with a ensembles, 1ist with length a student, teacher temperatures
# strategy: ensemble loas average strategy tau_ent: Lemperature for entropy weighting
for in dataloader: # load batch with a samples
xt (xgs(x), augt (x) 3 random augmentations 25, 2= =s (x(ss, ft (xt) zepresentations, (b, 1)
# all rollowing computation can be parallelized with batch computation
log_ps, log_pt [I, []
for in range (a):
he_1, ht_ hilens(il, ht_ens(d) j-th projecsion head mus_], mut_i mus_ensjll, mut_ens[j] j-th codebook, (d,
85_1, et_ (75), ht d-th embedding, (b,
rs_j - [es_] mus_j) fer_jiaarml keepdine-True) mus_j.nozm(axie-)
keepdinz=izue) / taus i studens logits, (a, cl rt_] let_1 mot jet_j.normmaxiset keepdins=True) not___.oom/
keepdins=izue)) taut i teacher logits,
c)
[og_pa_1 logsottmax (ra_i, axis--1) (b,
log_p-_] logsoftmax [xt_], axis- 1) F (b, e)
log_p=_] zenorm (log_ptt)))t_) adjust seacher predictions with centering or sinkhorn, onitred here for simplicity
log_ps. (log______)_ pog_pr.appond
log_ps stack (log_ps_j, axis=1) stacked student log probablities, (b, nr c)
log_p: stack (lag_pt__, axis=l) stacked teacher log probablities, (b, nr c)
loas ensemble_ (l, 1tg_pt, strategy=strateqy) compute ensemble loss
loas. (oak.back baok-paoaa
agd_update as_ens, mus_ens) apply gradient uodate student ema_update (ft, at_,as, mut_ens, retereta) # apply momentum update for teacher
15

---

Appx. C.2), and found that the design of Published as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
In this section, we provide details for our experiments. In Appx. B.1 we describe how we reproduced and improved the baseline DINO/MSN models. We give the implementation details for SSL training and evaluation in Appx. B.2 and Appx. B.3 respectively. All the hyper-parameters used in our
experiments are in Appx. B.2.2
B. REPRODUCING & IMPROVING BASELINES
We carefully reproduced and further improved baseline methods (denoted as DINO and MSN respectively) with an extensive study and hyperparameter search (see|Appx. B.I). particular, we systematically study the projection head design (which we found is crucial for few-shot evaluation performance Appx. C.2 and different techniques for avoiding collapse used in both methods (Appx. C.I. DINO performs significantly better than DINO few-shot evaluation (e.g.. 2~6 percentage point (p.p.) gains for shot) and maintains the full-data evaluation performance. The main adjustments of DINO are: (i) A 3-layer projection head with hidden dimension of 1024 (instead of 2048); (ii) Sinkhorn-Knopp (SK) normalization (instead of centering) is applied to teacher predictions, combined with a smaller teacher temperature T 0.025 and codebook size =004 or 4096, MSN' uses the same projection head as DINO and applies ME-MAX regularization without SK normalization (which is applied in MSN by default). Further details for DINO and MSN can be found below.
B.1.1 DINO
Table 5: Reproducing & Improving DINO. Our reproduce results match the public numbers. We further improve the DINO baseline (DINO") by studying projection heads and collapse-avoiding techniques. The evaluation results of DINO/DIÑO" ViT-S/16 trained with 800 epochs are reported.
Few-shot
Full-data
2
S
13 (1%)
k-NN
Linear
DINO (Caron et al. 2 DINO (Ours reproduced)
38.9 + 0.4
48.9 + 0.3
58.5 + 0.1
64.5
74.5
76.1 77.00
39.1 1 0.3
49.1
0.5
58.6 1 0.2
64.7
74.3
75.8/76,9
DINO (Retuned)
44.6 0.2
53.6 0.3
61.1 1 0.2
66.2
74.1
75.8/
Reproducing DINO We carefully reproduced DINO with JAX following the official DINO im- report the evaluation results of DINO using ViT-S trained with 800
plementation In Table epochs following the exact training configuration for ViT-S/16 in the official DINO code. The official results of full-data evaluation and 1 1% evaluation
from Caron et al. (021), the other few-shot evaluation results are evaluated by Assran et al.] (002) and also validated by us. Note that for consistency of full-data linear evaluation, we report the results with both the [CLS] token representations of the last layer and the concatenation of the [CLS] ] token representations from the last layers following Cewiin et al. (202T). For 1-/2-/5-shots evaluation results, we report the mean accuracy and standard deviation across 3 random splits of the data following AAssan et al. (022).. As shown in Taale 5] our reproduced results are all comparable with the published numbers which validates the implementation of our training and evaluation pipelines.
We improved the DINO baseline with a systematic empirical study of some important components. We first empirically compared different techniques for avoiding collapse (see Appx. C.I) and find that Bonkhorn-Knopp (SK) normalization is a more effective and also simpler technique for encouraging codebook usage than the centering operation used in DINO. We thus applied SK normalization, which enabled us to use a smaller teacher temperature (instead of 0.07) and much smaller codebook size =1024 or 4096 (instead of 65536). These modifications lead to similar performance as DINO with a much smaller codebook (up to 1M parameters, compared to 16M parameters for DINO). Next we empirically studied the effect of projection heads for different evaluation metrics (see
Improving DINO
0.025
16

---

Published as a conference paper at ICLR 2023
projection heads is crucial for few-shot evaluation metrics and an too power powerful projection head (e.g., the 3-layer MLP with a hidden dimension of 2048 used in DINO/MSN/etc.) could significantly hurt the few-shot performance. With empirically study of projection head architectures, we found that a simply reducing the hidden dimension to 1024 could significantly improves the few-shot evaluation performance while maintaining full-data evaluation performance. The improved results of DINO are shown in Table 5
B.1.2 MSN
Table 6: Reproducing& & improving MSN. We implement MSN" by adding ME-MAX regularization and masking to DINO", , which surpasses public MSN results. The evaluation results of MSN/MSN* ViT-S/16 trained with 800 epochs are reported.
Few-shot
Full-data
2
5
10 (1%)
k-NN
Linear
MSN (Assran et al. 2022
47.1 + 0.1
55.8 + 0.6
62.8 0.3
67.2
/ 76.9
MSN (Repro)
39.1
+ 0.3
49.2
+
0.3
58.4 0.1
64.3
72.8
74.7 75.5
MSN (Retuned)
0.1
56.3
+ 0.4
62.8 + 0.2
67.1
73.3
75.6/
We carefully implemented MSN by adding its main components, i.e., ME-MAX regularization and masking, to the DINO implementation (denoted as MSN*), which surpassed public results as shown in Table 6) Note that the implementation of MSN does not exactly match the public implementation in the public MSN code? where the main differences are:
MSN applies ME-MAX with Sinkhorn Knopp normalization by default (as in the released training configuration), which we empirically find does not work very well (see Table 9). MSN does not apply SK normalization and tunes the regularization strength for ME-MAX.
Some differences in implementation details, e.g., schedules for learning rate/weight decay, batch normalization in projection heads, specific data augmentations, etc. MSN' uses the exact same setup as DINO" which follows original DINO implementation.
We initially tried to exactly reproduce the original MSN following the public MSN code, but the results are much below the public ones, as shown in Table 6 Incorporating the two differences above bridges the gap and makes MSN surpass the public results.
B.2 PRETRAINING DETAILS
In this subsection, we provide the general implementation details in Appx. B.2. and specific hyper- parameters in Appx. B.2.2 Appx. B.2.2 for reproducibility.
B.2.1 IMPLEMENTATION DETAILS
Common setup We experimented with DINO (Caron et al.
2021) and MSN (Assran et al. 2022
models on ImageNet ILSVRC-2012 dataset (Deng et al. 2009). We mainly followed the training setup in Caron et al. (202I). I). In particular, all models were trained with AdamW optimizer (Loshchloov Hutter 2018) and a batch size of 1024. The learning rate was linearly warmuped to 0.002
0.001 x batch size/512) 12) and followed cosine decay schedule. The weight decay followed cosine schedule from 0.04 to 0.4. The momentum rate for the teacher was increased from 0.996 to with a cosine schedule following BYOL (Grill al. 2020). A stochastic depth (Huang et al. 2016) of 0.1 was applied without dropout (Srivastava et al., 2014). The student temperature t is set to 0.1. As with DINO, we used the data augmentations of BOL and multi-crop augmentation of SWAV (Caron et al. 2020). In particular, 2 global views with a 224 x 224 resolution and crop area range [0.25, 1.0] were generated for the teacher and student, and another local views with 96x96 resolution and crop area range [0.08, 0.25] were used as extra augmented inputs for the student. For MSN, we used the exact same setup and incorporated its major component: 1) mean entropy maximization (ME-MAX) regularization: 2) masking as extra augmentation applied to the student global view.
17

---

Published as a conference paper at ICLR 2023
Main modifications We retuned the baselines (DINO* and MSN") as detailed in Appx. B.1 and the main adjustments are as followed. We used 3-layer projection head with hidden dimension of 1024. The output embedding (i.e., (h r. (z)) and the codes (i.e., μu) both have dimension of 256 and are 1.2 normalized. For DINO", Sinkhorn-Knopp (SK) normalization was applied to teacher predictions. For MSN", ME-MAX was used without SK normalization and the regularization strength was tuned over [3, 4. 5). For all models, we used teacher temperature T = 0.025 which was linearly decayed from 0.05 for the first 30 epochs. The codebook size с was selected over {1024, 4096) for all models, and typically =4096 was selected for baseline methods and c = 1024 was selected for ours, For our (hy. ensembbls with ENT, entropy weighting temperature y is linearly decayed from 0.5 to the specified value.
B.2.2
We report the hyperparameters for training our models for reproducibility:
Table 7: Hyper-paramette for training the DINO" model.
VIT-S/16
VIT-B/16
VIT-B/8
Hyper-parameter
DINO
DINO" PRos (16)
DINO ENT (4/16)
DINO
DINO EENT (16)
DINO
DINO"-ENTTIN (16)
mriining epach
800
400
300
butch size
1024
1024
1024
learning
2e-3
2e-3
2e-3
warmup epoch
10
30
10
min lr
le-5
le.5
4e-5
weight decay
0.04
0.04 0.4
0.04
stochastic depth
0.1
0.1
0.1
gradient clip
3.0
1.0
3.0
momentum
0.996 > 1.0
0.996 >>> 1.D
0.996 -> 1.0
of multi-crops masking retio
10
10
10
proj. layer
3
3
pooj. hidden dim
J024
1024
1024
emhi dim d'
256
256
256
rop. dimt
codebook size
4096
1024
1024
4096
1024
4096
1024
student temp.
0.1
teacher temp.
0.025
0.025
0.025
temp, decay epoch center
30
30
30
SK norm
ME.MAX weight
cnt. weight temp.
0.05
0.05
0.05
init.
y dacay epach
30
30
30
B.3 EVALUATION PROTOCALS
Few-shot linear evaluation We followed the few-shot evaluation protocal in Assran et al. (0022). Specifically, we used the 1-/2-/5-shot ImageNet dataset split& in Assran al. (2022) and 1% (~-3-shot) ImageNet dataset splits"] For given labelled images, we took single central crop of size 224 x 224 without additional data augmentations, and extracted the output [CLS] token representations from the frozen pretrained model. Then we trained linear classifier with multi-class logistic regression on top of the extracted representations. We used the sci saiking -iiitlea rn package (Pedregosa et al. I for the logistric regression classifier. For all sew- evaluations, we searched the L2 regularization strength over (Ic-4, 3e-4, 1c-3, 3e-3, le-2, 3c-2, 1c-1, 3e-1, 1, 3, 10).
Full-data linear evaluation We followed the linear evaluation protocal in (Caron et al. 2021 Specifically, we trained a linear classifier on top of the representations extracted from the frozen pretrained model. The linear classifier is optimized by SGD with Nesterov momentum (Nesterov) 1983; Sutskever et al. (2013) of 0.9 and batch size of 4096 for 100 epochs on the whole ImageNet dataset, following cosine learning rate decay schedule. We did not apply any weight decay.
available at wttpaiatttttt...t.t..ooooooottts 'Publicly available htttee///ggtoob..oo//oooo/eeeerr http://rrrrr.ammmmmmert///eaaaaa
imagenet_subsets
18

---

Published as a conference paper at ICLR 2023
Table 8: Hyper-paraaetien for training the MSN model.
VIT-S/16
VIT-B/16
VIT-B/S
Hyper parameter
DINO
MSNN-ENT (2/8)
MSN"
MSN"-ENT (8)
MSN"
MSN -ENT (8)
training opoch
800
400
300
botch size
1024
1024
learning rale
2e-3
2e-3
2e:3
warmup epoch
20
min Ir
le-5
4e-5
4c-5
weight decay
0.04 - 0.4
0.04 -> 0.4
0.04 -> 0.4
stochastic depth
0.1
gradicnt clip
1.0
1.0
1.0
momentum
0,996 1.0
0.996 - 1.0
0,996 - 1.0
multi-crops
10
10
masking ratio
0.2
0.2
0.15
proj. layer
3 1024
proj. hidden dim
1024
1024
emb. dim
256
rep. dim
384
768
768
codebaok size
4096
1024
4096
1024
4096
1024
student temp.
0.1
0.1
0.1
teacher temp.
0.025
0.025
te. temp. decay epoch
30
30
30
center
X
SK norm
x
x
ME-MAX weight
4.0
4.0
cot. weight tomp. y
0.01
0.005
0.01
0.5
docay epoch
30
30
30
During training, we only applied basic data augmentations including random resized crops of size 224 x 224 and horizontal flips. During testing, we took single central crop the same size. For ViT-S/16, Caron et al. (202T) found that concatenating the [CLS] token representations from the last (specifically. / 4) layers (c.f. Appendix F.2 in Caron et al. (202I) improved the results by about p.p. We followed the same procedure, but reported linear evaluation results with both / = 1 and I = 4 in Table 2 for consistency. In our empirical study with ViT-S/16, we used the result with 1= 1. For larger models (e.g., ViT-B/16), followed et al. (2021); Zhou et al. (022) to the concatenation of the [CLS] token representation and the peerage-poooed patch tokens from the last = 1 layer for linear evaluation. For all linear evaluations, we searched the base learning rate over {4.8e-3, 1.6e-2, 4.8e-2, 1.6e-1, 4.8e-1, 1.6).
Full-data k-NN evaluation We followed the k-NN evaluation protocal in Caron al. 2022 Wu et al. (018). Specifically, for each image in the given dataset, we took a single central crop of size 224 x 224 without additional data augmentations, and extracted the output [CLS] token representations from the frozen pretrained model. The extracted representations are used for weighted k- -eiehhott-Neihhbor classifier. In particular, denote the stored training representations and labels as D = {(2,, y:) 1- For a test image with extracted representation 2, denote the set of its top k--N training samples as Da[=] C D and [Da[=]] k. The k-NN set Dz[2] is used make the prediction for the test image with a weighted vote, i.e., arg max, (Z(g))))) 0g dy1y=ms) where 1 1y=p is the one-hot vector corresponding to label yj and aj is the weight induced by the cosine similarity between and Zj. i.e., ag = exp (1 z'zg We set = 0.07 without tuning as in Caron al. (202T) :: Wu et al. (0088). For all k-NN evaluations, we searched over (5, 10, 20, 50, 100} and found that k 10 or k 20 was consistently the best.
|
Transfer evaluation via linear probing We mainly followed the transfer evaluation protocal in (Grill et al. 2020; Chen et al.. 2020a). In particular, we used 9 of their 13 datasets that are available in Censorfeiirreeeee fwwwwllowwaaoo asets (fdd). namely Food- -011 (Bossard et al. 2014]. CIFARIO (rizhevsky
al. 2009), CIFAR 100 (Krizhevsky et al. 2009), SUN397 scene dataset (Xiao et al. 2010), Stanford
Cars Krause et al. 2013), Describable Textures Dataset (Cimpoi et al. 2014 DTD), Oxford-IIIT Pets
(Parkhi al. 2012). Caltech-100 (Fei-Fei et al.
2004). Oxford 102 Flowers (Nilsback & Zisserman
19

---

Published as a conference paper at ICLR 2023
Table 9: Empirical study of different techniques for avoiding collapse. Using Sinkhorn-Know normalization instead of centering for DINO leads to improved performance, and matches the original DINO even with much smaller codebook. The ME-MAX regularization of MSN is very effective and leads to significant improvement for few-shot evaluations,
Technique
Few-shot
Full-data
Center
Sinkhorn
ME-MAX
5
1/ (1%)
k-NN
Lincar
37.8 0.4
47.4 = 0.3
56.9 + 0.4
63.0
72.4
74.9
DINO
39.1 0.3
49.4 0.3
58.7 0.2
64.8
74.1
76.0
36.0
46.6 0.6
56.5 0.2
63.2
73.2
75.2
MSN
43.9 0.2
53.0
61.1 0.2
66.0
74.0
75.8
Table 10: ME-MAX regularization is sensitive to hyper-parameters.
Few-shot
Full-date
Weight
2
1= (1%)
KNN
Linear
1.0
37.6
- 0.2
48.0 + 0.4
57.7 = 0.2
64.0
73.5
75.6
3.0
43.9
+
0.2
$3.0
+ 0.3
61.1
0.2
66.0
74.0
75.8
43.6
+ 0.2
52.6 + 0.4
60.4 + 0.1
65.5
73.9
75.6
2008). Following their evaluation metrics, reported mean per-class accuracy for Oxford-IIIT Pets. Caltech- and Oxford 102 Flowers datasets and reported top-1 accuracy for other datasets. We transferred the models pretrained on ImageNet (Deng al., 2009) to these datasets by training linear classifer on top of frozen representations. In particular, we resized given images to 256 x 256 and took single central crop of size 224 224 without additional data augmentations. We extracted the output [CLS] ] token representations from the frozen pretrained model. Then we trained a linear classifier with multi-class logistic regression seikit-leat package (Pedregosa et al. 2011) for the logistric regression classifier. For all transfer evaluations, we searched the L2 regularization strength over (1e-6, 1e-5, le-4, 3e-4, le-3, 3e-3, le-2, 3c-2, le-1, 3e-1, 1, 3, lel, 3e1, 1e2, 1e3, le4, le5).
top of the extracted representations. We used the
C ADDITIONAL RESULTS
C.1 EMPIRICAL STUDY OF TECHNIQUES FOR AVOIDING COLLAPSE
Most self-supervised learning methods utilize some techniques to avoid collapse of representations with, e.g., contrastive loss (Chen et al. 2020a He et al., 2020). batch normalization (Gril al. 2020), asymmetric architecture design with predictor (Grill et al. 2020 Chen & He 2021 ). etc. In DINO and MSN, learnable codebook is used for the learning objective and different techniques are applied to encourage the effective codebook usage. There are two potential cases of collapse (as discussed in Caron et al. (202)))
Dominating codes. This is the of "winner-take-all:: only small portion of codes being predicted while others are inactive, Typical solutions for avoiding this include applying Sinkhorn-Knan normalization (Cuturi 2013) in SWaV (Caron al. (020), centering teacher logits as in DINO (Caron et al., (021). and applying mean-entropy maximization regularization (ME-MAX) as in MSN (Assran al. 2022). Note that in MSN, ME-MAX is combined with Sinkhorn Knopp normalization by default.
Uniform codes. This is the case where all codes are treated equally and the predictions reduce be uniform over codes. A simple and effective solution is to applying sharpening, i.c., using a lower temperature for computing the teacher prediction.
We systematically study different techniques in a unified setup. In particular, we used DINO with the ViT-S backbone, a 3-layer MLP projection head with hidden dimension 2048, and codebook of size 4096 and dimension 256. We applied different techniques to DINO and searched the teacher temperature in (0.0125, 0.025, 0.05) for each. For ME-MAX, we searched regularization weight in
20

---

Published as a conference paper at ICLR 2023
1,0
Normalized Metric
40
0.9
Accuracy
35
0.8
Ful-data
0.7
196 data
30
base
I-shot
Ensembic
0.6
Defsult
25
Non-ensemble
0.5
10
10
Num. of Head Paramters
Number of Head Paramters
(a) Merged
(b) 1-shot
65
75
Accuracy
Accuracy
74.
60
Default
Default
Our baseine
73-
Our baseline
Ensemble
Ensemble
-on-eeneemble
55
Non-ensamble
10
10
10
10
10'
10
Number Head Paramters
Number of Head Paramters
(c) 1%-data
(d) Full-data
Figure 5: Effect of projection heads for different evaluation metries. We compare non-ensemble projection heads with different depths and widths as well as our -l)esembless and evaluate linear evaluation performance with different amount of labeled data. (a) shows the comparison of normalized metrics for non-ensembles. (b)-(d) compares non-ensemble and (hw, -e)-enbebless by unnormalized metries. 'Default' denotes the default projection heads used in many SSL methods. See analysis in Appx. C.2 for details.
(1.0, 3.0, 5.0). For ME-MAX combined with Sinkhorn, we followed Assran al. (2022) and used default regularization weight of 1.0. The results are in Table 10 We observed that:
DINO's centering operation is not as strong as other techniques, and it favours a larger teacher temperature (e.g., 0.05). It does not work well when the codebook size (4096) is not as large as the one used in the original DINO model (65536). Switching to use Sinkhorn Sinkhorn-Knopp normalization leads to much more improved performance, and matches the performance of original DINO TTbll 5) with much smaller codebook.
MSN's ME-MAX regularization is very effective, and leads to significant improvements others. We also found it is sensitive the regularization weight and teacher temperature (c.f. Table 10p. However, we observed that combining ME-MAX with Sinkhorn does not work well without tuning the regularization weight (which is recommended by Assran et al. 2022
C.2 EMPIRICAL STUDY OF PROJECTION HEADS
In this subsection, we systematically study the effect of projection heads for different evaluation metrics. In particular, used DINO VIT-S/16 as the base model and used different projection heads with (depth, width) searched over {2, 3, 4} x (512, 1024, 2048, 4096). All models are trained with 300 epochs using exact the same of hyper-parametert We measured the linear evaluation performance with different amount of labeled data (i.e., full-data, 1%-data, 1-shot).
In Fig. 5a plot different evaluation metrics (normalized respectively by the best of each) versus the number of projection head parameters. In In[Figs. 5b 5d] we plot each unnormalized evaluation metric respectively for different heads well our (To,: -eseembbss... Our key findings
21

---

Published as a conference paper at ICLR 2023
75
73.9
74.4
73.6
73
722
-
Accuracy
64
63.1
63.4
61.3
59.5
*
Full data
* I-shot
60
-X-
1% data
Ensemble
44
41.7
42
40.6
40
38
37.1
37.6
0.005
0.01
0.025
0.05
Teacher Temperature
Figure Effect of teacher temperature for non-ensemble DINO". DINO" with a lower temperature achieve better few-shot performance, but still under-performs ensemble method (DINO" MOINO"EEN -ENT with 16 heads, orange lines). DINO V trained for 300 epochs is used and T = 0.025 is used for DINO"-ENT.
The projection head has relatively larger impact on few-shot evaluation metrics, as reflected by the relative magnitudes of different metries in Fig. 5a An too powerful non-ensemble projection head significantly hurts the label efficiency of learned representations, reflected by much larger drop in few-shot evaluation performance (up 18 p.p. for 1-shot, p.p. for 1a-d)ta). This result is also partially observed in [Chn et al. (2020b), where they found that probing from intermediate layers of projection heads (which can be viewed as using shallower head) could improve the semi-supervised learning (1 (%-11%) results.
The optimal projection head for different metrics differ lot. A weaker head improves label efficiency (few-shot performance), while a stronger (but not too strong) head improves linear decodability. As result, the default projection head (3/2048) that is widely used in SimCLR v2 (Chen et al. (2020b), DINO (Caron et al. 2021), iBOT (Zhou et al. (022). MSN (Assran al. (022). etc.. does not perform well in few-shot evaluations (as shown by the green cross denoted as 'Default',, probably because it is selected by full-data evaluation metrics.
There exist some projection heads that performs decently well on all evaluation metries, e.g., the baseline model (3/1024) used in our experiments (pink star denoted as 'Our base').
Compared to naively tuning projection head architectures, our (hw,, -eneembblss (orange curves Figs. tb 5d) consistently improve all metrics with different amount of labeled data, despite it also increases the number of parameters in projection heads. Our (hg, ())- ensembles outperform all non-ensembles, which also include the counterparts of probing from intermediate layers from the a deeper head (i.e., shallower heads).
C.3 EMPIRICAL STUDY OF (he. ENNSMBBLE
Are the gains of ENT purely from sharper teacher predictions? Our ENT strategy assigns higher weights to the heads that predict with lower entropies, thus effectively uses sharper teacher predictions as the targets. One may be curious about how this effect accounts for the gains of the ENT strategy. We empirically answer this question by studying the non-ensemble baseline that uses a sharper teacher predictions in a indeennndeenneen manner (in contrast to ENT, which uses data-dependent entropy weights). Specifically, temperature T E {0.005, 0.01, 0.025, 0.05) and also our DINO (16) with 7 = 0.025, as shown We find that the teacher temperature has big impact evaluation results especially for few-shot evaluation. Compared to our default baseline that uses 7 = 0.025, a lower temperature (e.g.,
compare the non-ensemble DINO that use different teacher
in Fig. 6
0.01) indeed improve the 1-shot performance (at the cost of worse full-data performance). However, an too low temperature (r = 0.005) will hurt the performance. Our DINO*-ENT (16) consistently outperform all the baselines, which implies the importance of selecting sharper teacher predictions in
manner.
22

---

Published as a conference paper at ICLR 2023
Table 11: Full table of Table 1 including all metries for comparing different ensemble strategies.
ENT and PROB significantly improves over the non-ensemble baseline, while UNIF leads to no gains. Ensembling the whole projection head works the best. All models are DINO" ViT-S/16 trained for 300 epochs. The means and standard deviations over 3 initialization seeds for all evaluation results are reported.
Where
Few-shot
Full-data
How
Proj. Head
Codcbook
2
5
~033 (1%)
I--N
Linear
Base
40.6 +
49.8 0.2
57.9 = 0.3
63.4 0.2
72.3 0.1
74.4 + 0.1
UNIF
40.4 + 0.4
49.5 +: 0.4
57.6 + 0.3
63.3 4: 0.3
72.2 + 0.2
74.5 + 0.2
PROB
v
39.7 + 0.5
49.0 : 0.5
57.4 = 0.4
63.0 + 0.4
72.8 + 0.2
74.8 + 0.1
PROB
41.9 + 0.3
51.5 2
59.6 = 0.4
65.1 1.1::00 0.3
73.7 + 0.3
75.4 0.1
Ext
40,6 + 0.4
19.5
58.0 0.4
63.5
72.1 0.3
74.5 + 0.3
EXT
43.0 + 0.6
52.2 0.8
59.7 = 0.7
64.8 0.5
72.9 + 0.6
75.1 + 0.4
ENt
44.0 + 0.2
53.0 0.5
60.5 0.3
65.5 +.
73.2 + 0.1
75.3 + 0.1
EXT-ST
40.0 1 0.5 39.2 1 0.6
57.3 1 0.5
62.7 0.5
71.9
1 0.4
74.0 1 0.4
Table 12: Comparison of different varaints of PROB. The PROB strategy used in our experiments performs the best. '_' in the table denotes training divergence for PROB-MAX The experimental setup is the same as Table 11
Where
Few-shot
Full-data
How
Weight by
Temp.
2
10 (19)
k-NN
Lineur
Base
40.6 0.2
49.8 + 0.2
$7.9 + 0.3
63.4 + 0.2
72,3 +: 0.1
74.4 + 0,1
PROB
student
41.9 : 0.3
51.5 + 0,5
59.6 0.4
65.1 + 0,3
73.7 + 0.3
75.4 = 0,1
PROB-TE
teacher
41.5 + 0.2
50.4 + 0.3
58.3 +: 0.3
63.7 + 0.1
72.3 +: 0.2
74.6 0.1
PROB-MAX
student
PROB-MAX-TB
teacher
41.4 0.2
50.3 + 0.3
58.1 + 0.3
63.6 + 0.2 72.3 0.2
74.5 0.2
Comparison of different ensemble strategies and variants We present the full table of Table that includes all the metrics in Table 11] The same observation holds for all metries.
For all previous studies, we considered a specific instantiation of PROB strategy, i.e., weight by student predicted probabilities fkjy log s()) and 7 1, which has nice interpretation of model average (see See. 3.3). We also studied different variants of the PROB strategy (see Apex. D.I.
PROB-TE: weight by teacher füjy log 1: (vlar) and y 1;
PROB-MAX: weight by student fuja = log 8, (y/xr) and 7 + O:
PROB-MAX-TE: weight by teacher fijy = log 4, (ylzr and 7 + 0
Table 12 compares the downstream performance for all the variants. We find that the our PROB (used in our empirical studies) performs better than other variants. Interestingly, weighting by the teacher (PrOB-Te) performs worse than PROB. We conjecture that this is because the important weights turn out to give a weighted average of teacher predictions as the surrogate target that shared across all students (like PROB) but does not give effective preferential treatment across students which are directly optimized (unlike PROB-TE). Furthermore, PROB-MAX which sharpens the importance weights leads to training divergence. This is probably because the student predictions have higher variance based on which sharp weights lead to unstable training. In contrast, TROB-MAX: TE which uses the (lower-variance teacher gives reasonable results and comparable PROB-TE.
Number of ensembles for MSN In Fig. 7a we study the effect of increasing the number of (he. -leeensebless for MSN* ENT with VIT-S/I6 trained for 800 epochs. The scaling trend is similar to DINO -ENT Fig. 3ab and the gains start to diminish when the number of heads increases above
Effect of ENT temperature y for MSN Fig. 7b studies the effect of entropy weighting temper- ature y for -ENT-ENT. We observed that MSN is more robust to small temperatures, and the
23

---

Published as a conference paper at ICLR 2023
76
75.6
76.0
76.3
76.4
X
76.6
76
76.2
x
76.3
76.2
76.2
76.0
76.0
67.9
68.3
68.6
65
64.9
64.8
66.0
64.4
64.1
Accuracy
Accuracy
67.1
50.1
49.9
63
60
49
49.3
-
Full data
49
48.8
49.3
49.2
49.0
48.5
48.3
-
data
-X-
Full data
1-shot
47.4
I-shot
-X- S-shot
Baseline
4 Num. of Ensembles
0.005
0.01
0.02
0.04
0.07
0.1
Entropy Weight Temperature
(a) Scaling ensembles
(b) Effect of ENT temperature 7
Figure 7: Empirical study for MSNT-ENT. (a) The gains by increasing the number of (he p)- ensembles start to diminish when it over heads. (b) MSN* prefers smaller temperature for entropy weighting than DINO
best = 0.01 is smaller than that of DINO (y = 0.05). When the temperature is too high, the performance drops as result of under-speciallzattin (i.e., less diversity) as with DINO
C.4 ANALYZING (he, JENSEMBE DIVERSITY
Visualizing (hw; -eneembbee similarity We analyze the diversity between different heads by visualizing the similarity matrix between their codes. Directly measuring the similarity between codes in two heads could not work, because 1) they may live in different subspaces because of the ensembled projection heads; 2) they may not align the natural order but permuted order.
Therefore, we seek to align codes between different heads by how they are effectively used to 'cluster' the data. In particular, we use set of randomly sampled inputs {a²) } of size b 51200 to
obtain an empirical code assignment matrix A E R for each (hyp -nseembeee E [m]] where the i-th of A corresponds the teacher predictions (Y|(x)) For the k-th code in the head extract the k-th column from A (i.e., its empirical assignment) as its embedding. For two codes, we measure their similarity by the cosine similarity between their embeddings. For a pair of heads; and j', we align their codes using the Hungarian algorithm (Kuhn) 1955) maximize the sum of cosine similarity. After that, we plot the similarity matrix which is aligned and reordered by the similarity value on the diagonal (in an descending order). Note that it is not necessary to do the alignment procedure for the PROB strategy since it is naturally aligned because of the direct distribution averaging over (ha. ensembless. but did for fair comparison with other strategies.
We applied the same procedure for different ensemble weighting strategies using DINO with 4 (hy- mi)-ensembles.
randomly picked a pair of heads and visualize the similarity matrix before (top row) and after (bottom row) the blignment-reorderin alignment procedure, the similarity matrix of the PROB strategy already mostly aligns because explicitly introduces code correspondence between different heads. Furthermore, by analyzing the similarity decay pattern on the diagonal, is clear that ENT learns the most diverse (ho. -neemmbbll while UNIF learns the least ones, which may explain the difference of their empirical performance. For completeness, we also include the visualization of aligned similarity matrices for all pairs of heads in Figs. to 11 the observations the same.
setup in Fig. 8 We found that before the
24

---

Published as a conference paper at ICLR 2023
1.0
0.8
0.5
0.4
0.2
0.0
(a) UNIF
(b) PROB
(c) ENT
Visualization of (he, eensembee diversity. ENT learns the most diverse (he p)) ensembles while UNIF learns the least ones. We visualize the code similarity matrix between pair of randomly selected projection heads. Top row shows the original similarity matrix (i.e., in natural order) and the bottom row shows the aligned similarity matrix which aligns codes by empirical assignment probabilities. DINO ViT-S/16 with 4 heads is used. Best viewed in color.
Figure 8:
Head 1- Head 2
Head 1 Head 3
Head 1 Head
1.0
088
0.4
022
0.0
Head 2 Head
Head 2 Head
Head 3 Head
Figure Visualization of (hep ju) ensemble diversity between all pairs heads for DINO UNIF. The UNIF strategy does not learn diverse (hy,, -eseeeeel.... DINO with ViT-S/16 and heads used. Best viewed in color.
25

---

Published as a conference paper at ICLR 2023
Head 1- Head 2
Head 1 Head 3
Head 1 Head 4
0.8
0.6
022
0.0
Head 2 Head
Head 2
Head
Head Head
Visualization of (hes enneembe diversity between all pairs of heads for DINO PROB. The PROB strategy learns more diverse (lip, Hi)-ensebbl than UNIF, DINO" with ViT-S/16 and 4 heads used. Best viewed in color.
Figure 10:
Head Head
Head
Head
Head Head
1.0
0.8
0.6
0.4
0.0
Head Head
Head Head
Head Head
Figure 11: Visualization of (he -nseembll diversity between all pairs of heads for DINO ENT. The ENT strategy learns the most diverse (hes enseemmbles... DINO" with VIT-S/16 and heads is used. Best viewed in color.
26

---

Published as a conference paper at ICLR 2023
D ANALYSIS
D.1 DERIVATIONS
In this subsection, we provide derivations for some non-trivial losses that we explore within our framework.
Recall that our weighted cross-entropy loss is of the form,
(.( = Σ
Σ
H (uig (1(1)))
(15)
71 zEDa f.jmmm]
Σ
Σ
(16)
ZED, 1.JE/m] vEy
Wijy = softmax ((t/ssn(stopgrra())..) i E [ml}]
where
(17)
Fuethermore, observe that,
V Σ H 04((1)), 1.EEmm]
E
(ily log s0y00, (18)
(JE(m]]
This indicates that the proposed weighted ensemble SSL loss is simply reweighted log-likelihood loss. We use this fact in our derivation of probability weighting (PROB) loss.
Uniform weighting (UNIF) Our UNIF strategy in Eq. (6) uses fijy = log 8(i - j) which gives oi j) (for any choice of 7), thus the loss,
Wijy
UNIF (0) = Σ
Σ
Σ
1
5(i 5)4(() log ,(vl0,,
(19)
m
xEDa (JE/m] DEy
VE)
Σ
Σ
H (Y((Y)))
(20)
:E|m]
This loss assigns equal weights to m pairs of pairwised student/teacher.
An straightforward generalization is to assign equal weights all possible pairs (m²) of stu- dent/teacher with fijy = O and Wijy m².
1 which gives the UNIF-ALL loss,
1 Σ Σ H* (
C UNLI (0)
(21)
ZED
Probability weighting (PROB)
Recall our PROB loss in
Eq. (7) has the form,
PROD (0)
Σ
H
Σ (Y()))
1 Σ s sY100,,
(22)
72
m se(m)
m
ZED
jE(m)
We derive equivalence with our general loss with Jijy = log ( x) and y = 1 in terms of the gradients,
PROB (0)
Σ
4(ylz) log
syl10,,)dd
(23)
2E(m)
je(m]
1 Σ
1
log m
(24)
m iE(m]
JE
1 Σ
1 jE[m]
V Vos(yl0,.2)
dy
(25)
=mmm
yy
T70
ΣiE(
s
27

---

Published as a conference paper at ICLR 2023
1 Ejmmm x log ())
Σ
tz(ylx)
m.
dy
(26)
# - s(y) 03),
m iE/m]
1
8(v105.z)
te(yla:)
Vo loygs(yy0))
(27)
m
Eye(m) s1yl0y,
V
Σ [J
HJ luiy 2
(28)
where wijy E 3. (or equivalenty, fajo = log sy) x)) and 1). The last equality
<(y (,,x)
is because Wijy is stopped gradient with respect to 0. This is the same analysis as done in Burda et al. (2016). The above formation establishes the equivalence of gradients between two losses, which implies the same behavior (e.g., optimum) using gradient-base based optimization, as the common practice of deep learning.
We also generalize this loss to some variants which we explore in Table 12 teacher predictions fijy = log t; (y/zz) instead of student ones; this implies Wijy Ere(m Golalz]
"dual" variant is to use
f(gla)
and the PROB-TE loss,
C
PROB TE (0))
1 - DE (,7[(m] DEy
Σ Σ
tx x
to (y]xz) log swyl0,
(29)
Evejm tor (y|xr
Note that this simply reduces to use weighted teacher predictions Evelo for (vlz) 44 (ylar) as the surrogate target that is shared across all students.
"hard" weighting, i.e., - O, which gives the PROB-MAX loss that only assigns weight to the most confident student,
Another generalization
C PROB-MAA (0)
1 Σ Σ
Σ
(ijyl((ylx) log #()
(30)
ZED i.j vEy
where Wijy
8(i j )²), (i (())")) arg max fijy Vy E V.
(31)
This loss reduces to generalization of multiple choice learning (Guzman Rivera al. 2012) used in multi-headed networks (Leee et al.) 2015) in our ensemble SSL. setup. Similarly we can also derive the dual variant of that uses the teacher predictions, which is omitted here for brevity.
Entropy weighting (ENT) The derivation of ENT loss in Eg. (9) is similar to the UNIF loss but applies an entropy weights. Recall that use fejy H[t, [x] + log 8(i - -)), which gives Wijy softmax; (1- H[t, (Y [z)] i' [m]]] and,
1 Σ
CENT (+)
softmax, (1- (Y[x)] : i' [m]}) HM (
(32)
DED, (E[m)
One also generalizes it to its dual variant which uses the student entopies, i.e.ffij - -Hss(Yl0,, = + log ò(i j), which gives the ENT-ST loss,
(ENT-ST
1
Σ
softmax =)) (1s [m]]] H* (Y11),)
n
#E D., sE[m]
(33)
D.2 RELATING SOME LOSSES
Here, relate some losses derived above. Specifically, relate the uniform weighting (UNIF, UNIF-ALL) and probability weighting (PROB) in Appx. D.2.I and relate entropy weighting (ENT) and variance weighting in Appx. D.2.2
28

---

Published as a conference paper at ICLR 2023
D.2.1 UNIFORM & PROBABILITY WEIGHTING
We first establish the relation between UNIF and PROB using the joint convexity of unnormalized KL divergence and the fact that our weighted cross-entropy loss is weighted unnormalized KL divergence up to constant in 0. In particular, the joint convexity of unnormalized KL divergence can be shown by combining the facts that Csiszär f-divergences are jointly convex (Proposition in Dragomir (2013) and unnormalized KL divergence corresponds the convex generator, f(u) log + 1, as required by the proposition.
First, our weighted cross-entropy loss is unnormalized KI. divergence up to some constant in 0:
1
L (0)
((Y)), + constant
(34)
zEDn
m
PROB (0)
1 71
Σ Z
K
(Y
Σ jelm
s 6Y
+
constant
(35)
m
vE(m)
Therefore, the joint convexity of (unnormalized) KI. divergence directly implies an ordering of the loss up some constant in 0, i.c.
C.ROB UNIF
(36)
Furthermore, we can also relate PROB and UNIF-ALL using the fact that the (unnormalized) cross- entropy Hx q())) linear the first argument p but convex the second argument g, which implies,
C
PROB < Ln
CUNIC-ALL
(37)
D.2.2 ENTROPY & VARIANCE WEIGHTING
Suppose p(X) is discrete distribution (normalized) X = [c]. It can be shown that,
H[p(X)] < log (Var, [X] + } + 1 log (2TTe)
(38)
where Var [X] Exele ((=)(<= 20)2 and μ Ex = Exelel p(x)xr (Theorem 9.7.1, Cover Thomas (199)). Note, tighter bound (Mow 1998) also exists but places stronger restrictions on This relationship suggests that choosing weights proportional to exp( ([([((((())))) (as in ENT) is
potentially related to choosing weights proportional to weighting by variance (Var,,(Ylw) Y +))+))+14
where (c 1.
29


Reviews:
[{"review_id": "wnvOVNHaudW", "reviewer": "Reviewer_8Pcm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the benefits of ensembling for self-supervised learning methods. The paper proposes to ensemble the “non-representation” part of the SSL models, and proposes different weighting schemes. Extensive experiments are conducted to analyze and evaluate the proposed ensemble method and weighting schemes, which lead to improvements over the non-ensemble models.", "summary_of_the_review": "The paper studies a timely topic on ensembling for SSL models. The proposed method is simple and could fit with a variety of SSL models. While the paper can be made better by comparing to more related ensembling baselines to guide practical use, the current presented empirical results is quite comprehensive on its own, and the usefulness of the proposed method is well-justified by the empirical results.", "strength_and_weaknesses": "Strengths:\n- This could be a pretty timely work that explores ensembling for self-supervised learning models. I like the idea of only ensembling the non-representation part as that incurs little training overhead, plus no extra cost for downstream tasks.\n- I appreciate that the authors included the analysis on the ensemble diversity. This gives a deeper understanding of the effectiveness of different weighting schemes, and could potentially guide the designs of future weighting schemes.\n- The paper includes thorough analysis and ablations on the proposed methods, and shows strong empirical results compared to the baselines and SOTA.\n\nWeaknesses:\n- The connection between the Maximum Likelihood view of SSL and the proposed ensemble method design is not very clear to me. The presentation could be improved with stronger connections between Sec.2 and Sec.3.\n- While the paper does a good job in evaluating a variety of ablations on the proposed ensemble method, it seems to lack other possible baselines for ensembling, e.g., instead of only ensembling the projection head, one baseline is to consider ensembling the base models. It’d be good to compare this baseline to understand whether there is a trade-off between the benefits of ensembling and the cost for extra parameters. In general, the paper should include comparisons to the other related ensemble baseline mentioned in Sec. 4.\n\nOther questions/comments:\n- Can one use the proposed ensembling method on pre-trained SSL models (e.g. DINO)? What’s the performance difference of that to applying ensembling in the pre-training from scratch? I think this is a relevant question to look into since there are already a lot of publicly available pre-trained SSL models, and it’d be best if the ensembling technique can be applied on top of these pre-trained models and still offer improvements.\n- How does one set $\\gamma$ for ENT in practice? From Figure 3.b, it seems to affect the performance pretty much.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "pbwmyclW1co", "reviewer": "Reviewer_wYk7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper systematically studies the ensemble of self-supervised learning. Ensemble has been one of the oldest time-proven strategies in machine learning and has been well-studied in supervised learning. Here, the authors introduce ensemble into the field of SSL. They mainly consider ensembling the projection head and codebook and leaving the encode part as future work. This proposed method is straightforward with rich analysis and proper results. ", "summary_of_the_review": "I believe this paper, albeit short on technical novelty, does provide a sufficiently novel and important empirical evaluation and details analysis of self-supervised learning ensembling methods and could be accepted - assuming the authors can address the issues I have detailed in my review.", "strength_and_weaknesses": "## Strength\n(1) The proposed method is straightforward and easy to implement with compelling results.\n\n(2) I believe this paper includes some important and thorough experimental work on ensemble of self-supervised models.\n\n(3) Detailed hyperparameters and training configurations are provided, improving the reproducibility.\n\n## Weakness\n(1) The ensembling of the projection head and codebook is a natural extension of the classifier ensemble in supervised models. It is not too surprising to see a performance improvement. \n\n(2) This paper mainly evaluated ensemble on transfer learning, while the performance on upstream tasks (Imagenet) has been overlooked. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eZWNUMja_-", "reviewer": "Reviewer_pJds", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how ensemble methods can improve self-supervised learning algorithms. The considerations are limited to ensembles of projection heads and codebooks but not encoders, to ensure no extra computational cost involved. The authors propose a kind of data-dependent weighted cross-entropy losses. Two sota SSL methods, DINO and MSN are considered as baselines. In various experimental settings, the ensemble methods are observed to largely improve the baselines. ", "summary_of_the_review": "Overall,  the problem this paper studies is interesting. To simplify the design and retain the computational cost, this paper is limited to ensembles of projection heads and codebooks but not encoders, which limits the technical novelty. It is interesting to see the significant improvement in performance under various settings and evaluation metrics. Therefore, I think this is a borderline paper.", "strength_and_weaknesses": "Strength:\n- This paper is well-written and easy to follow.\n- The ensemble methods only incur a small training cost.\n- It is interesting to see that sota SSL algorithms can be further improved by ensembling in various settings (linear eval, KNN, few-shot, etc.).\n\nWeaknesses:\n- The proposed framework only suits contrastive SSL methods (i.e., align positive pairs). Other SSL methods such as MAE can not be applied.\n- When encoders are not considered for the ensemble, the possibilities for where to ensemble are not too many. The proposed ensemble framework somewhat lacks novelty. The main contribution is to test different weighting schemes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EQxLYIKPo09", "reviewer": "Reviewer_37Hd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Model ensembling is a common technology to further boost performance. While most paper study how to ensemble on supervised models, this paper focus on self-supervised models (SSL). Then, this paper discusses where to ensemble and how to ensemble the models training from SSL, proposing a downstream-efficient ensemble method based on weighted cross-entropy objectives. The results on few-shot learning such as imagenet-1k (1%) improve baselines of SSL method (DINO/MSN).", "summary_of_the_review": "Model ensembling for SSL model is an interesting idea to explore, however, it is still in the preliminary stage, and the above mentioned methods and experiments need to be further supplemented. ", "strength_and_weaknesses": "With the development of SSL, it is necessary to study SSL model ensemble. The paper is rich in theoretical basis, Explaining clustering-based self-supervised learning methods from the perspective of maximum likelihood distinctly. This paper proposed a method that can be effective solutions to ensemble multi models, because of ensembling only applying to the head and codebook, not the encoder. This paper is easy to follow, and experiments are enough to support the claims made in this paper. \n\nCouple questions come to mind:\n\n1.This method is effective but I think it's not enough to just ensemble the head and codebook, most researchers are probably more concerned how to ensemble the encoder part.\n\n2.How extensible is this method? The method is verified in the double-branch structure (DINO/MSN) and can it be applied to the most popular MIM methods such as MAE. \n\n3.How to acquire multiple models to ensemble and how to ensure the diversity of the models, it seems to be less described in the paper.\n\n4.If I am not wrong, the authors only fuse different heads, and how these different sub-models are obtained, from the intermediate checkpoints, and how can ensure the diversity of these heads. In my opinion, the diversity of the head counts for then ensemble, and how about choosing other heads from even smaller backbones but with large diversity. How is the method compared with another simple method model soups [1] that ensembles the encoders of different checkpoints and do not increase the inference time.  \nOverall, although the results are impressive, I still wonder the rationale of the methods.\n[1] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time\n\n5. It is better to add average number in Table 3 for clarify.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster