Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Authors: Samuel Lavoie
Christos Tsirigotis
Max Schwarzer
Ankit Vani
Michael Noukhovitch
Kenji Kawaguchi
Aaron Courville
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Self-Supervised learning
Representation learning
Pre-training

Abstract: Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a \texttt{softmax} operation. This procedure conditions the representation onto a constrained space during pretraining and imparts an inductive bias for group sparsity. For downstream classification, we formally prove that the SEM representation leads to better generalization than an unnormalized representation.
Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and ImageNet. Finally, when used in a downstream classification task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.

SIMPLICIAL EMBEDDINGS IN SELF-SUPERVISED LEARNING AND DOWNSTREAM CLASSIFICATION
Samuel L Christos Tsirigotis", Max Schwarzer", Ankit Vani", Michael Noukhovitch", Kenji Kawaguchi', Aaron Courville
o Mila, Université de Montréal, National University of Singapore, + CIFAR Fellow
ABSTRACT
Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into L simplices of V dimen- sions each using a softmax operation. This procedure conditions the representation onto constrained space during pre-training and imparts inductive bias for discrete representations. For downstream classification, we provide an upper bound and argue that using SEM leads better expected error than the unnormalized representation. Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR- 100 and ImageNet. Finally, when used in a downstream classífication task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically- relevvan classes.
INTRODUCTION
Self-supervised learning (SSL) is an emerging family of meth- ods that aim to learn representations of data without manual supervision, such as class labels. Recent works (Hjelm et al., 2019; Grill et al., 2020; Saced et al., 2020; You et al., 2020)
BYOL+SEM (x4)
Top-1 accuracy
(x2)
76-
BYOL [x4]
learn dense representations that can solve complex tasks by simply fitting linear model top of the learned represen- tation. While SSL is already highly effective, we show that
74
BYOL (x2)
changing the lype of representation learned can improve both the performance and interpretability of these methods.
72
BYOL
0.4
1.1
3.7
FLOPs (10*)
For this we draw inspiration from overcomplete representa- tions: representations of an input that are non-unique combi- nations of number of basis vectors greater than the input's dimensionality (Lewicki & Sejnowski, 2000). Mostly stud-
Figure 1: Linear probe accuracy of BYOL and BYOL SEM on Ima- geNet trained for 200 epochs with ied in the context of the sparse coding literature (Gregor ResNet-50 architecture.
LeCun, 2010; Goodfellow et al., 2012; Olshausen, 2013), sparse overcomplete representations have been shown to increase stability the presence of noise (Donoho et al., 2006), have applications in neuroscience (Olshausen & Field, 1996; Lee et al., 2007), and lead to more interpretable representa- tions (Murphy et al., 2012; Fyshe et al., 2015: Faruqui et al., 2015). But, the basis vector is learned using linear models (Lewicki & Sejnowski. 2000; Tch et al., 2003).
In this work, we show that SSL may be used to learn discrete, sparse and overcomplete representations. Prior work has considered sparse representation but not sparse and overcomplete representation learning with SSL; for example, Dessi et al. (2021) propose to discretize the output of the encoder in SSL model using Gumbel--Softax (Jang al., 2017). However, we show that discretization during pre-training is not necessary to achieve a sparse representation. Instead, we propose to project the encoder's 's output into L vectors of V dimensions onto which we apply a softmax function impart an inductive bias toward sparse one-hot vectors (Correia et al., 2019; Goyal et al., 2022), also alleviating the need to use virianer gradient estimators to train the encoder. We refer to this embedding as Simplicial Embeddings (SEM), as the softmax functions map the unnormalized representations onto simplices. The procedure to induce SEM is simple, efficient, and generally applicable.
"Correspondence samuel. wawoie...loii..mm..... www.oille@@mai.coommmmm

---

The SSL pre-training phase, used with SEM, learss set of L approximately one-hot vectors. Key to controlling the inductive bias of SEM during pre- training is the softmax temperature parameter: the lower the temperature, the stronger the bias toward sparsity. Consistent with earlier attempts sparse representation learning (Coates & Ng. 201 1), we find that the optimal sparsity for pre-training need not match the optimal level for downstream learning.
For downstream classification, we may discretize the learned representation by, for example, taking the argmax for each simplex. But, we can also use SEM to control the representation's expressivity via the softmax's 's temperature. We provide a theoretical bound showing that the expected error follows a trade-off between the training error and the representations' expressivity that can be controlled by the softmax's 's temperature used to normalize the representation for downstream classification. Our bound also shows improved expected error as we increase I, and for SEM.
SEM is generally applicable to recent SSL methods. Applying it to seven different SSL methods (Chen et al., 2020b; He al., 2020; Grill et al., 2020; Caron et al., 2020; 2021: Zbontar et al., 2021; Bardes et al., 2022), we find accuracy increases of 2% to 4% on CIFAR-IO0 1000 We observe monotonic improve- ment as we increase the number of vectors L, showing the benefit of the overcomplete representations learned by SEM, while this improvement is absent when we do not use softmax normalization. When training a SSL method with SEM on ImageNet we also observe improvements on in-distribution compared to the baseline (Figure 1). We also observe improvement out-of of-diiitriuutiinn test sets, semi-supervised learning benchmark and transfer learning datasets, demonstrating the potential of SEM for large scale applications. Finally, find that SEM learns features that closely aligned to the semantic categories in the data. This demonstrates that SEM learns disentangled and interpretable representations, as previously observed in overcomplete representations (Faruqui al., 2015).
2 RELATED WORK
The softmax operation has been used in other contexts, notably models to attend to context-dependent queries via, for example, an attention mechanism (Bahdanau al., 2016; Vaswani et al., 2017; Correia et al., 2019; Goyal et al., 2022), a mixture of experts (Jordan & Jacobs, 1993) or memory augmented networks (Graves et al., 2014). This operation is also used for the computation of several SSL objectives such as InfoNCE (van den Oord et al., 2018: Hjelm et al., 2019). and as a normalization of the output to compute the objective in DINO and SWaV (Caron al., 2020; 2021). Different from these, our method places the softmax at the output of an encoder to constrain the representation into set of L sparse vectors.
architectural component for
Similar to our approach, other architectural constraints such as Dropout (Srivastava et al., 2014), BatchNorm (loffe & Szegedy, 2015) and LayerNorm (Ba et al., 2016) also improve the training of large neural networks. However, contrary to SEMs, they are not used to induce sparsity on the representation or control its expressivity for downstream tasks. Closer to our work, Liu et al. (2021) propose to constrain the expressivity of the representation of neural network with a set of discrete-valued symbols obtained using a set of Vector Quantized (Oord et al., 2018 8) bottlenecks. Similarly, Dessi et al. (2021) propose a communication game with a discrete bottleneck. The idea of discretizing the encoder's is output is similar to using SEM vectors that are one-hot (c.g. temperature = 0) and only one symbol (e.g. L = 1, V = 2048). In our work, we find success in removing the hard-discretization and having L 1, which can be interepreted combining several symbols.
3 SIMPLICIAL EMBEDDINGS
Simplicial Embeddings (SEM) are representations that can be integrated easily into contrastive learning model (Hjell et al., 2019; Chen et al., 2020b), the BYL method (Grill et al., 2020), and other SSL methods (Caron et al., 2020; 2021: Zbontar al., 2021). For example, in BYOL, we insert the SEM after the encoder and before the projector and the rest is unchanged as shown in Figure 2c. In this figure, and E' are augmentations defined by the practitioner, & are parameters of the target network that are updated as moving average of the parameters 8 of the online networks trained with SGD. So, & are updated as follow: & at (1 a)1,, with [0, 1].
To produce SEM representation, the encoder's 's output e is embedded into L vectors 2; E R.V. A temperature parameter 7 scales 2,, and then softmax re-normalizes each vector 2; to produce ž,.
2

---

4
SEM
10.0
t: 0.1
normalization
3
1.0
0.01
t: 0.5
2
92
P/
Concat
1
0.5
1.5 Entropy H(2,)
1.0
2.0
2.5
(a)
encoderg,
embedder
SEM
projector
predictior
Co
hormallzation,
pø
g
x
loss
v'
encoder:
embedder;
SEM
projector;
F
stop-ored
>sg(p
>g
normalisation.
Figure 2: (a) Procedure to obtain Simplicial Embeddings (SEM). A matrix : E L contains vectors 24 E RRV. The vectors 2; are normalized with dr, the softmax operation with temperature The normalized vectors are concatenated into the vector Σ. (b) Normalized histogram of the entropies (2,) of each simplex & for the sample CIFAR's training dataset at the end of pre-training with various т. The peak at In(2) for = 0.01 and = 0.1 are large number of simplices with two elements close 0.5. (c) Integration of SEM with BYOL (Grill et al., 2020). The encoder outputs latent vector which is embedded into the matrix E R R and then transformed into SEM.
Finally, the normalized vectors 2, are concatenated to produce the vector of length L V. We illustrate SEM in Figure 2a. Formally, the re-normalization as follows:
EU/T
3 (z)), (=())
Concat (3.... Er.); W WE (1)
Ev esalT''
3.1 INDUCTIVE BIAS TOWARDS SPARSITY DURING PRE-TRAINING
In SEM, I. controls the numbers of simplices and V controls the dimensionality of each simplex. As such, the higher V is, the sparser the representation can be. During pre-training, the constraint induced by embedding the representation into a simplex biases each vector towards sparse vectors by creating a zero-sum competition between the components the vector. In order for a component to increase a, then the other elements must decrease by a; and all elements networks to learn useful features and minimize their objective, they must prioritize some components the expense of others. The strength of this bias is controlled via the pretraining temperature Tp of the softmax, and the size of the vectors V as was noted in the context of attention (Vaswani et al., 2017; Wang al., 2021b). For SSL methods with a target network, the temperature for the target network can be different to the online network's 's as no gradient is back prok-prop through it.
bounded by 0. For
To visualize the effect of the temperature on SEM after pre- pre-training, we interpret each simplex as a probability mass function p(ā;)) where, for E [Z], ΣV p(z,)) and p(āg) 2 Vj. The entropy of a simplex ž, defined as (2,) Σソ P(Zzy) log p(2)). informs whether the simplex a sparse or dense vector. That is, if (= = then the vector one-hot. On the other hand, ((= In(V) then the vector is dense and uniform. While the temperature Tp is merely scaling of the logits, it has an important control over the learned representation' 's entropy and resulting SEM CIFAR- 100 using BYOL, and analyze the entropies of the resulting simplices. In Figure 2b, we plot the histogram of the entropies (2,). for given Tp. of each simplex for each sample in the training set of CIFAR-100. We observe that
sparsity. We demonstrate this by learning representation
3

---

even after pre-training, small temperatures (Tp 0.01) yields representations that are close to one-hot vectors while high temperatures yields vectors that are close to uniform vectors.
By pre-training using softmax, SEMs create representations that are conditioned to fit onto simplices. In pre- ure-training we select T for optimal inductive bias: Tp too small yiclds vanishing gradients (Wang al., 2021b) and Tp too large yields a bias that is too weak. We may select different optimal Ta for downstream performance as discussed formally in the next subsection.
3.2 SEM IMPROVEMENT ON THE GENERALIZATION OF THE DOWNSTREAM CLASSIFIER
In this subsection, we theoretically demonstrate the benefit of training downstream classifier with SEM normalized input compared to a baseline classifier with unnormalized input. We show that: (I) there is trade-off between the training loss and the generalization gap, which is controlled by the value Ta (denoted T == Td in this subsection), (2) SEM can improve the base model performance when we attain good balance in this trade-off, and (3) the improvement due to SEM is expected to increase or stay constant as I. and V increase. In the remainder of this subsection, we introduce the notation and assumptions needed understand and derive the result, then present our theoretical claim and discuss its implications.
Notation. We use a training dataset S S (2())
of n samples for supervised training of a classifier, using the representation 2 extracted from the pre-trained model and the corresponding label E y where y is the space of possible labels. Assume that 2 E 2 = [-1 + +1]6XV, which means that is matrix with I. rows and V columns. We denote the element of row and column j as Zij. Let g represent the downstream classifier. We refer to the baseline downstream model with unnormalized input as fhase. and fbase (2) g(z). The corresponding downstream model trained with the SEM normalization is (sFM(()(() = (go (z),, where o, is applied element- along eRig/r each row of 2 such that σ, (2ij) Eist e'w// for j = l...., V. Moreover, we define f and SEM()) the base and the SEM normalized models obtained by fitting the dataset S. Finally, let H be the union of the hypothesis spaces of (SEM(r) and fhase-
To compare the quality of the base model and the model with SEM normalization, we analyze the generalization gap Ez,y('(ss())) v)] a Ei= ((ss(2()), (1) for each Is E (SEM)r)) ) where 1: R x y + R2o is the per-sample loss.
The key insight that we exploit for the theorem is that the softmax operation o, controls the ex- pressivity of the input's representation to g via the temperature т. We denote f as an upper bound on the expressivity of z; for the baseline model Shase: and feEMs:- the upper bound on the expressivity of a, (2,) for the model with SEM normalization fSEM(r)- The formal definition of PPosse and (PSsmmer) requires proof devices that will hinder the readability of this section, so we refer the reader to Appendix A for detailed definition. Let y E { from (SSSEM))) }. Intuitively, iss measures the largest possible distance that two embeddings can have such that the largest component remains the same for both embeddings. We note that this measure depends only on V for Jbases and on both V and r for (SEM(r). We use f (V,7) to denote the measure given by either model and note that has effect for fbase-
Assumptions. We assume that the sar-sample loss is bounded such that [[f(2), v) S B for f E H and for all (z, E Z x Yr For example. B 1 for the 0-1 loss. Next, let ly be the per-sample loss given y. We assume that o,og are uniformly Lipschitz functions for all y E ) and g E Gs. where Gs is the set of classifiers g returned by the training algorithm using the dataset S. Let R be such uniform Lipschitz constant. This means that ((y o g) (2)) - (Ly ( S R((a)(()) where 1g o (2)) = 1(g ()(2), y), and o a+ when f = fSEMG) and o is identity when f= fbase- Finally, we assume that there exists A > O such that for all representations 2 of the underlying distribution we have that for any i [4] k- arg max Zij, then 2ik 2 Zi + A for any j # k. Since A can be arbitrarily small (e.g. as small as machine precision), this assumption typically holds in practice. We are now ready to state our theoretical claim.
(z')ll
Theorem illuminates the advantage of SEM and the effect of the syper-parrametr on the perfor- mance of the downstream classifier. We present the proof in Appendix A.
'In this subsection, we refer the extracted representation as z, the embedder's output
4

---

Theorem 1. Ler 2 2. For any 1280 with probability least 1 - 8, the following holds for any fs E ( F8 basel:
In(2/0)
Engl((s(())ww) S (.s()()) + By Lipss( +
1=1
where c >> O is a constant in (n, J. H, 5, 8). Moreover,
(EEM()))) 0 as T + 0 and
P P
S ( V) <0 Vr
The first statement of Theorem 1 shows that the expected loss is bounded by the three terms: the training loss Ei= (ff((=(ss)) y()),, the second term R Lipss and the third term cV In(2/4) Since e is a constant in (n, f. H. ò, S), the third term goes to zero with and without SEM. Thus, for the purpose assessing the impact of SEM, we can focus on the second term, where a difference arises. Theorem I shows that /LLLps goes to zero with SEM; i... (((SEM)))) + as + Also, for any > 0, the second term with SEM strictly smaller than that without SEM ( SI S (1-1 V)
oo and is the same
and demonstrates that the improvement due to SEM is expected to asymptotically increase as V increases. Moreover, I. is a multiplicative constant of op which shows that, L increases, the improvement due SEM is also expected be higher. Overall, Theorem shows the benefit of SEM as well as the trade-off with т. When + 0, the second term goes zero, but the training loss (the first term) can increase due to underfitting resulting from the reduction in representation expressivity. Thus, 7 should be chosen to optimally balance this trade-off.
EMPIRICAL ANALYSIS
We empirically study the effect of SEM on the representation SSL methods and demonstrate that SEM improves the test set accuracy on CIFAR- 100 (Krizhevsky, 2009). We compare SEM with other methods for inducing sparse representations during pretraining and demonstrate that SEM lead to better downstream accuracy. On IMAGENET (Deng et al., 2009), study the effect of SEM on robustness, supervupee learning and transfer learning datasets, demonstrating consistent improvement attributed to SEM. Finally, present evidences that features produced by SEMs are more naturally aligned with the semantic categories of the data. The code for reproducing the results is available at: https://github.coo http://wiiiiisssspppialleeeeeeee
Training setup. For all experiments, we build off the implementation of the baseline models from the Solo-Learn library (da Costa al., 2021). We probe the encoder's output for the baseline methods, as typically done the literature. For models with SEM, we probe the SEM normalized representation (i.e. 2). In our experiments, the embedder is a linear layer followed by BatchNorm (Ioffe & Szegedy, 2015). Unless mentioned otherwise, we use L = 5000 and V = 13 for the SEM representation. We do not perform any search for the non-SEM selected by using a validation set of 10% of the training set of CIFAR- and 10 samples per class on the distribution dataset for IMAGENET. The test accuracy is obtained by retraining the model with all of the training data using the parameters found with the validation set. We pre-train the SSL models for 200 epochs on IMAGENET and 1000 epochs on CIFAR-100.
papper--raamters... The SEM hyper-parameter are
Table 1: Linear probe top-1 accuracy CIFAR- 100 trained for 1000 epochs with a ResNet- 18/50 encoder. We compare the test accuracy of several SSL models with and without SEM. Boldface indicates highest accuracy. Green indicate SSL method SEM.
SIMCLR MoCo BYOL BARLOW-TWING SWAV DINO VICREG
ResNet-18:
Baseline
66.8
69.3
70.7
70.7
64.6
66.8
68.5
With SEM
69.5
71.0
73.9
73.0
67.7
69.2
71.4
ResNer- 50:
Bascline
70.5
73.24
74.2
72.0
70.8
With SEM
73.2
75.8
77.5
73.3
73.3
5

---

4.1 SEM IMPROVES ON DOWNSTREAM CLASSIFICATION
Baseline comparison. We evaluate the effect of adding SEMs in seven modern SSL approaches. We take standard SimCLR (Chen et al., 2020b), MoCo-v2 (He et al., 2020), BYOL (Grill al., 2020) Barlow-Twins (Zbontar et al., 2021). SwAV (Caron et al., 2020), DINO (Caron et al., 2021) and VicReg (Bardes et al., 2022) models and implement SEM after the encoder. We compare our approach on CIFAR-100 with ResNet- 18 and ResNet-50 in Table 1. We found SWaV and DINO to be unstable with ResNet-50 thus have decided not to compare them with SEM. For every SSL methods, using SEMs improves the baseline methods by 2% to 4% demonstrating that SEM is general approach that improves in-distribution generalization for SSL methods.
Increasing the representation's size of SEM increases the performance. We find that increasing L (the number of sim- plices of SEM) beyond the over-complete regime increases the downstream accuracy. This increased performance is not ob- served when we abstain from using the softmax normalization of SEM. In Figure 3, using ResNet-50 encoder, we compare BYOL + SEM, with an identical model without the Softmax normalization which we call BYOL- Embed and BYOL to which we increase the representation' 's size before the mean- pooling using the method proposed in (Dubois et al., 2022) and described in their Appendix F. To be clear, the extracted representation of BYOL- Embed is the embedder's output zg and the extracted representation for BYOL is the encoder's output eg. We fix V 13 and scale L [10, 10000] to get range of representation sizes.
80-
BYOL
SEM
78
BYOL
BYDI Embed
76:
72
70
69
130
650
13K
130k
Embedding size
Figure 3: Effect of the Softmax when scaling up L on the linear probe accuracy. Using a RN-50.
Table 2: Comparing SEM with hard discretization using Gumbel Soft- max (G.S.) and Vector Quantization (V.Q.). RN-18 base on CIFAR-100.
Comparison of SEM with hard discretization approaches. Several other methods can be used to induce a sparse and over-
Accuracy
e9 29
complete representation during pre-training and downstream classification. For example, we may sample I. discrete one-hot codes of V dimensions using Gumbel Softmax (Jang et al., 2017) as done in Dessi et al. (2021). We can also use Vector Quantization (VQ) (Oord et al., 2018) and consider L latent embedding spaces with V embedding vectors each, wherein the vectors are in Rd. In contrast to SEM, it is not possible to propagate the gradient through the bottleneck trivially and VQ uses sraiigh-thhouuhh estimation in the embedding space to back- propagate the gradient to the encoder. Here, we observe that these alternative approaches exhibit a considerable decrease in performance in comparison to the baseline demonstrated in Table 2. In this table, we reproduce the same setup as SEM but we replace the Softmax with hard discretization baselines methods. For discretization with Gumbel Straight-Througe estimation, we use the same setup as SEM with L = 5000 and V = 13, that is 5000 one-hot vectors of 13 dimensions and 7 = 2f. For VQ, we found that = 512 and V 128 led to the best performance. That is, we have 512 latent embedding spaces, each with 128 possible embedding vectors that are in 32
BYOL
70.7
BYOL+G.S.
63.3
54.5
BOOL+V.Q.
65.6
60.3
BYOL+SEM (Ta = 0) 73.2
BYOL+SEM (Ta = 0.1) 73.9
We note that while we have not found pirr--dscretization be successful during pre-training, we may hard-discretize a SEM representation for downstream task. In Table 2, we also present SEM with TDs = O. which correspond to using the discretized representation for downstream classification. We obtain the discrete representation by taking the argmax for each simplex. This result demonstrating that SEM with pre-training can be used to learn meaningful discrete codes for downstream applications and yields better performance than the baselines, implying that pre-training with SEM could be be used in applications that require discretization.
Memory and computational efficiency of SEM. SEM's performance improvements come at cost of increased memory allocation (VRAM) due to additional parameters needed to perform the matrix multiplication, and slightly more computation (FL.PP//ammlle) For very large over-complete representation the increased memory requirement can impede practical application. We propose a more efficient version of SEM by sparsifying the matrix multiplication of the embedder and of the projector and detail this procedure in Appendix D. As shown in Table 17, SEM with sparse matrix
A hapae-parameter search was performed to select the hest performing hapar-parameter.

---

multiplication use only slightly more memory and compute but outperforms the BYOL baseline on CIFAR- 100 though underperforming the regular SEM. We also note that SEM's memory cost scale up the encoder. As well, the computational cost of SEM is
becomes relatively minor small compared to the total cost of pre-training and achieves higher accuracy using fewer FL.OPs compared to scaling the encoder shown in Figure 1.
4.2 ANALYZING THE PARAMETERS OF SEM
We present two figures in this section to better understand the effect of the parameters of SEM on the downstream accuracy. In Figure 4, we evaluate the effect of changing T and Ta the downstream accuracy. In Figure 5, we evaluate the effect of L and V on the downstream accuracy and also contrast Staee and (TEM = 1) by using the same encoder pre-trained with SEM. This allows to relate some observations to the theorey presented in Section 3.2. Now, we discuss the effect of each of SEM's parameter on the resulting downstream classification.
Increasing V yields steep performance increase for small V but quickly plateau. In Figure 5b, we observe a steep increase of the accuracy for V < followed by a plateau for V > 13. In Figure 4a, we observe that the optimal accuracy obtained for V = 1024 and L 64 is similar to the one obtained for
(Embedding size=650) in Figure 3.
Increasing I yields monotonical improvement for downstream classification. In the regime that we can test it, increasing I. lead to consistent improvement on the downstream accuracy as observed in Figure 3 and Figure 5a. Using SEM in pre- pre-training only is not enough and using in the downstream classifier is necessary for the improved performance as demonstrated in Figure 5a.
The optimal Tp depends on V. As previously noted in the context of Attention (Vaswani et al., 2017; Wang et al., 202 2021a), the optimal attention's 's temperature is proportional to attention's 's vector size. We also observe this in SEM. As presented in Figure 4a, the optimal Tр for larger V is higher.
Models with larger are more robust to smaller Tл. In Figure 4, we observe that SSL models are more robust smaller Ta L increase. We speculate that the information can be scattered across the simplices for large L, allowing to reduce the expressivity each vector with minimal impact on the downstream accuracy.
Accuracy
Accuracy
Accuracy
Lo5000, Vi
70
fus
bue
10224
60
L-202
LH022Z
140020
0.01
0.2
05
2.01
63
0.5
10
5301000
5000
Pretrain tau
Downstream tau
(a)
(b)
(a)
(b)
Figure Effect of Tp and Ta on RN-50.
Figure S: Comparing fSEM and foase on RN-18.
4.3 SEM IMPROVEMENT ON -ARGE-SCALE DATASETS WITH IMAGENET
Figure in the introduction demonstrates that using SEM leads to better in distribution generalization for IMAGENET and is a more efficient method of scaling up the model as compared scaling up the width of the ResNet-50 encoder. Here, we demonstrate that SEM generally improves the accuracy on several robustness test sets, sumi-supervised learning benchmark and transfer learning datasets. We use BOL++MEM with an embedding size of 105 000 features (T. 5000 and V 21) for these experiments. The embedding is pre-trained for 200 epochs using the BYO. SSL procedure.
Robustness to -ittof--istriutionn test sets. We perform comparative study using several test sets: (IN) the in-distribution test set provided IMAGENET: (IN-C) IMAGENET-C, which exhibits a set of common image corruptions (Hendrycks & Dietterich, 2019); (IN-R) IMAGENET-R (Hendrycks et al., 2021) which consists of different renderings for several IMAGENET classes; and (IN-V2) IMAGENET- V2 (Recht et al., 2019). distinct test set for IMAGENET collected using the same process; (IN-A) IMAGENET-A A (Chen et al., 2020a) contains a set of samples that are miclassifier by a IMAGENET ResNet-50 classifier. We the methodology and software proposed in Djolonga al. (2020; 2021)
7

---

Table 3: Robustness via linear probe top- Table 4: Top-1 transfer learning accuracy from IMA- 1 test accuracies on IMAGENET variant GENET pre-trained representation. datasets, using representations pre-trained for 200 epochs. * Taken from (Chen & He, 2020)
FOOD101 C10 C100 SUN DTD FLOWER
Linear probe:
BYOL
74.2
91.8
74.9
60.9 72.2
88.9
IN IN-V IN-
B BYOL+SEM
74.7
93.5
78.6 62.1
71.9
91.5
BYOL*
70.6
Fine-tuned:
BYOL
71.9 59.2 18.8 39.5 1.65
BYOL
83.1
97.2 83.6 59.1
69.2
85.4
BYOL+SEM 74.1 61.2 22.1 43.4 2.53
B SOM
84.7
97.2
85.6 63.3 71.3
91.7
to perform our experiments. We observe that BYOL + SEM outperforms BYOL on every robustness datasets probed, demonstrating that SEM also improves generalization to out-of-distribution test sets.
Transfer learning, We probe the effectiveness of SEM in BYOL and MoCo when transferring representations trained on IMAGENET to other classification tasks. We follow the linear evaluation and fine-tuning methodologies described in previous works (Grill et al., 2020; Lee et al., 2021), which entails training a linear classifier with logistic regression using sklearn (Pedregosa et al., 201 on the embeddings of the samples and fine-tuning the encoder respectively. To avoid out-of memmor issues that may occur in the linear probe experiment with the sklearn solver when the number of features, we discretize our features and use sparse matrix to fit the logistic regression. This is equivalent to forcing тι = for all the experiments. For the fine-tuning experiments, we fix Ta = 1 since the evaluation method allows for mini-batch gradient descent. We perform transfer learning experiments on the following datasets: FOOD (Bossard et al., 2014), CIFAR-10 (C-10) (Krizhevsky, 2009), CIFAR-100 (C-100) (Krizhevsky, 2009), SUN (Xiao et al., 2010), DTD (Cimpoi et al., 2014) and FLOWER (Nilsback & Zisserman, 2008).
This task evaluates the generality of the encoder it has to encode samples from various out-of- distribution domains with categories that it may not have seen during training. We present our results in Table 4 and observe that SEM improves the transfer accuracy over the baseline for every datasets but DTD for the linear probe experiment. For DTD, we hypothesize that the drop in performance is due to the fact that we use temperature that is too small. Since this is a texture dataset with higher frequency, it might be the case that we need more expressivity to correctly fit the data. We support the conjecture with the fine-tuning experiment where B BOO SEM out-performs the baseline.
Semi-supervised learning. We evaluate the effect of us- ing SEM when fine-tuning on a classification task with a small subset of IMAGENET's training set. We follow the cemi-supervised learning procedure of Chen et al. (2020b); Grill et al. (2020) and use the same fixed splits of 1 1% and 10% of ImageNet labelled training set. In Table 5, we demonstrate that using SEM lead to an important increased performance, especially in the low supervised data regime.
Table 5: Semi-supervised learning accu-
racy by fine-tuning on IMAGENET
Top-1
Top-5
1%
10%
1%
10%
BYOL BYOL+SEM 56.7
51.6
67.5
78.0
88.9
69.9
81.0
90.0
4.4 SEMANTIC COHERENCE OF SEM FEATURES
Here we demonstrate that SEM features are coherently aligned with the semanties present in the training data. Qualitatively, we visualize the most predictive features of a downstream linear classifier trained on CIFAR- 100 and see that the classes with similar predictive features are semantically related. Quantitatively we propose a metric that returns the ratio of features mostly predictive for a classes that are in the same super class to total number of class predictive for this feature.
For both our analysis, we use linear classifier trained on the features extracted from BYOL with and without SEM. Consider the trained linear classifier with weight matrix W E N with N features, and C' classes. By preserving the top K parameters of the weight matrix W for each class and pruning the features predictive for only one class, we create a bipartite graph between two set of nodes: the CIFAR- 100 classes and the features of the representation. We denote this graph WK.
The qualitative analysis is given by plotting the subset Ws. obtained by taking the top 5 features for each class. We present subset of the graph for BYOL+SEM in Figure 6a and for BYOL in Figure 6b. The full graphs are presented in the Appendix. In the SEM plot, a set of connected components emerge, and the connected components of the graph are semantically related. For example, the
8

---

Bortle Assicinn
man
bay
BYOL embed
Semantic coherence
(a) BYOL+ SEM
ICA
table
Pipuroos for
worrn
Reconon
cammel
Murk
Chros wolff
Vrangg
clair peraao
Top features
(c)
(b) BYOL
Figure 6: Semantic coherence of the features. (a) and (b) Subset of Wi, the bipartite graph of the
most 5 highest magnitude features BYOL + SEM features (a) and BYOL on the encoded features (b). (c) Coherence of the top K features to the semantics of the super-class of the categories of CIFAR 100. It is taken the number of pairwise categories in the same super-class for which feature is among its top K most predictive features over the total number of pairwise categories.
first set of connected components are flowers, and the last set of connected components are aquatic m F. The same class coherence not observed with either the BYOL baseline or with BYOL augmented with a large representation. In particular, we do not see small number of semantically related connected components. Instead, we see large fully connected graphs
Next, we describe how we quantitatively measure the semantic coherence of the features, Notice that two classes share common predictive feature on WK if they are 2-neighbour. Let (c(s) returns all pairs (c,, sÿ) for all 2-neighbour of q. Moreover, define the operation is super(c;, Cj) which returns e; and ej are from the same CIFAR- 100 superclass and otherwise. We reproduce the superclass of CIFAR-100 in Table 22 in the Appendix. We measure semantic coherence as follows:
C E(eney)eN(an) is super(c,, Cj)
Coherence(Wx)
[((c)]
1=1
where C' = 100 for CIFAR-100 100 and is the cardinality of a set.
We compare the semantic coherence of B YOL+ with the control experiments BYO:: regular BYOL, YOL with an embedding of the same size as B GOL++EM but without the normalization and BYOL which we applied linear ICA (Hyvärinen Oja, 2000) in an attempt to disentangle the features. In Figure 10, we plot the full graph W3 for BYOL+SEM and the baselines. We observe that using the SEM yields semantically coherent features for the classes CIFAR-100. This observation is consistent with the qualitative and quantitative experiments presented earlier and demonstrates that SEM's inductive bias during pre- training leads to features that are semantically coherent with the semantic categories extant in the data. This arguably have important implications for improving the interpretability of SSI. representations.
CONCLUSION
SEM simple, drop-in module that induces discrete sparse overcomplete representations for standard SSL. methods using softmax operation. This simple modification leads to improved generalization on downstream classification across several state-of-the-art SSL methods. Furthermore, SEM improves performance on cittof-dditributionn semi-supervised, and transfer learning tasks across the board and also scales with encoder size. By analyzing semantic coherence, we find that SEMs naturally disentangle data into semantic categories without any explicit training objectives.
Although "flatfish" may seem out of place in the third set, manually checking CIFAR images showed that many images labelled "flatfish" often humans holding flatfish.

---

ACKNOWLEDGEMENTS
The authors are grateful for the insightful discussions with Xavier Bouthillier, Hattie Zhou, Sébastien Lachapelle, Tristan Deleu, Yuchen Lu, Eeshan Dhekane, Maude Lizaire, Julien Roy and David Dobre. We acknowledge funding support from Samsung and Hitachi, as well as support from Aaron Courville's CIFAR CCAI chair. We also wish to acknowledge Mila and Compute Canada for providing the computing infrastructure that enabled this project. Finally, this project would not have been possible without the contribution of the following open source projects: Pytorch (Paszke et al., 2019), Orion (Bouthillier et al., 2022), Solo-Learn (da Costa et al., 2021), Scikit-Leery (Pedregosa et al., 2011), and Numpy (Harris et al., 2020).
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
https: /etssv//aaiv...//e org/ wwrg/abs//ee abs /tos/1607 607. 06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. arX9v:1409.044 [cs, stat], May 2016. URL http://arxiv
org/abs/ 1409. 0473. arXiv: 1409.0473.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-ovaaaa regularization for self-supervised learning. International Conference on Learning Representations, 2022. URL
.nt forum? wtt.ruummidimmmmemmeee Xm6YD62DUUb.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014, pp. ISBN 978-3-319-1059944
Cham, 2014. Springer International Publishing.
Xavier Bouthillier, Christos Tsirigotis, François Corneau-Tremblay, Thomas Schweizer, Lin Dong, Pierre Delaunay, Fabrice Normandin, Mirko Bronzi, Dendi Suhubdy, Reyhane Askari, Michael Noukhovitch, Chao Xue, Satya Ortiz-Gagné, Olivier Breuleux, Arnaud Bergeron, Olexa Bilaniuk, Steven Bocco, Hadrien Bertrand, Guillaume Alain, Dmitriy Serdyuk, Peter Henderson, Pascal Lamblin, and Christopher Beckham. Epistimio/orion: Asynchronous Distributed Hyperparameter Optimization, March 2022. URL httph::////// /ww. org/ 10. 5281 5281/zenodo
zenodo. 34 78592.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Asso- ciates, Inc., 2020. URL tttps://ppoceeding: neurips cc wttppppr//0200/////////// file/ f0 6e0238f74 (ateee441aa2222ff8f 60220
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. arXiv:2104. 104.144 [cs], May 2021. URL http: /arxivvv..v. org/ abs /214 14294. arXiv: 2104.14294.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training fine-tuning. In CVPR 2020, June 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566, 2020.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEÉ Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
10

---

Adam Coates and Andrew Y. Ng. The importance of encoding versus training with sparse coding and vector quantization. In ICML, pp. 921-928, 2011. URL https: 485 wwt.iimmmpppp...mmmm
Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. Adaptively sparse transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2174-2184, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1tt///3//1/111111/0 1223. URL https: ac www.anaonanoooooaaacaoao org/Di9 9-1233.
Victor G. Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. Solo-learn: A library of self-supervised methods for visual representation learning, 2021. URL httph://///// /github..
com/ vttrrisi/////i solo-learn.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Roberto Dessi, Eugene Kharitonov, and Marco Baroni. Interpretable agent communication from scratch(with a generic visual processor emerging on the side). CoRR, abs/2106.04258, 2021. URL pettss:/arree org/ abss 2206.. 04258.
Roberto Dessi, Eugene Kharitonov, and Marco Baroni. Interpretable agent communication from scratch (with generic visual processor emerging on the side). In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL https::////. http///pppeeeeeeeeeeeee/ee net 1ort
1M4H
Josip Djolonga, Frances Hubis, Matthias Minderer, Zachary Nado, Jeremy Nixon, Rob Romijnders, Dustin Tran, and Mario Lucic. Robustness Metries, 2020. URL https: : / tuu...uu....... com/
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, Syl- vain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On Robustness and Transferabil- ity of Convolutional Neural Networks. arXiv:2007. 08558 [cs], March 2021. URL http:
//arxivv.o org/ abs / 2007 08558. arXiv: 2007.08552
D.L. Donoho, M. Elad, and V.N. Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Transactions on Information Theory, 52(1))6-18, 2006. doi: 10.1 10.1000/TT.2005.8660000 860030.
Yann Dubois, Stefano Ermon, Tatsunori Hashimoto, and Percy Liang. Improving supervid learning by characterizing idealized representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. net forum? i ii=iidattt aa=aagDGD222ppp
URL https::///// httpps://peerevvee openrevieweeeeeeeeeeeeeeeereeeme
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. Sparse over- complete word vector representations. In P Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers). pp. 1491-1500, Beijing, China, July URL https:
2015. Association for Computational Linguistics. doi: s/aclanthe ogy www./o/g//11111
Alona Fyshe, Leila Wehbe, Partha P. Talukdar, Brian Murphy, and Tom M. Mitchell. A compositional and interpretable semantic space. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 32 41, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/v/N15- 1004. URL https: /aa/aaaatttoicccco
org/Ni 5-1044
Ian J. Goodfellow, Aaron Courville, and Yoshua Bengio. Large-scale feature learning with spike-and- slab sparse coding. In Proceedings of the 29th International Coference International Conference on Machine Learning, ICML' 12, pp. 9781450312851.
Madison, WI, USA, 2012. Omnipress. ISBN
11

---

Anirudh Goyal, Aniket Rajiv Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Ra- haman, Jonathan Binas, Charles Blundell, Michael Curtis Mozer, and Yoshua Bengio. Coordination among neural modules through shared global workspace. In International Conference Learning Representations, 2022. URL ht ttsp::: / wpwpoeeeevveeeeeeeeeeeeeeeeeeeee ew .et.et.eeeeeeeeeeeeeeeeeee.ee fo forum? (
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing Machines. arXivv1410.5401 [cs], December 2014. URL http: http://aaavvvvv/vi wwtp://wwwrgb/bb//////ea/e
5401. arXiv: 1410.5401.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML 10, pp. 399-406, Madison, WI, USA, 2010. Omnipress. ISBN 978 7781605589077.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284. Curran Associates, Inc., 2020. URL https: https::/ppocceddii http://pccceeeiig.cceo..o neur cc/ paper/20200 filellllll e Bee.aaaa@055c444
01 42b1 7b81 9222152229988e Paae22995--Ppprr.... Pappr.ppd.s
Charles Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825))357-336) September 2020. doi: 10.1038/441586-000-2222 URL https::tp//..........
/ttp:://dolio http://ooooog///////// 1038/00.300//000 586 2649-2
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv:1911.0577 [cs], March 2020. URL http:
//arxiv.org aba/ / 911. 05722. arXiv: 1911.05722.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
URL hnttps://openreeette /wwpnreveiwww..ccooooppoppoo..wm net wttp:/iwwwweeeeotoeieeeeeeeeee id= (twoormmii-f02ttiiii1ie
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of Out-of-Distriuttion Generaliza- tion. arXiv:2006. 16241 [cs, stat], July 2021. URL http: //tp:v///aaii.aa//a wwtpoogg///bbe/e///oooeee 16241. arXiv: 2006.16241
R Devon Hjelm, Alex Fedorov, Samuel Maric--ar,,,,,onn Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019, URL https: / //openreview..o net www.:nerttotttee/eeee wwwoommmiidBaar -rr.dd--kkkrrkk CXX..
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks, 13:411 13441-4430,1 2000.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference
Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448-456 456, Lille, France, 07-09 Jul 2015. PMLR. URL https: / /ptpceeeeioc...... w.lr.press.e v 37/ 1o ffef5. html.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https::/ n forum? d=rkE3y85e..
Li Jing, Pascal Vincent. Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. URL https: / wpwpeeeeeeeeeeeeeeoeepp net fo dow.um?s..eeesss05np..
12

---

M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the algorithm. In Proceedings of 1993 International Conference on Neural Networks N3-Noyy,, Japan), volume 2, pp. 1339-1344 vol.2, 1993. doi: 10.1 10.1109/ICNNN.193..66677
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representa- tion learning. CoRR, 1es/1991..9005, 2019. URL
1901 wtwpa////9...0......... 09005.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. URL https: //www. cs toronto. parronto.ddu/kkrer.. wattd///iiii/leemiinnnn learning- featugge-eeuesseeeeee 2009- TR. TR.ppf.
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area v2. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL httptttts///prrceeeeeee wnw.ooeeeeeiiein.....ooo hewpeeiiiinnnnnnnn.....e....... neurips. http:///0....eeeeaaaaaaaaaaeaeaa
Paper. pdf.
Kuang-Huei Lee. Anurag Arab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive Visual Representations. arXiv:2109. [cs, math], September 2021. URL http:://rxii org/abs/ 2109. 12909.
arXiv: 2109.12909.
Michael S. Lewicki and Terrence J. Sejnowski. Learning Overcomplete Representations. Neural
Computation, 12(2)3337-365, 02 2000. ISSN 0899-7667. doi: 10.1 16220889996600000005888222
URL https: doi org/ 12w./11222// 1111////966600000055533
Dianbo Liu, Alex M Lamb, Kenji Kawaguchi, Anirudh Goyal ALIAS PARTH GOYAL. Chen Sun, Michael C Mozer, and Yoshua Bengio. Discrete-valued neural communication. In Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad-
M. Ranzato, A. Beygelzimer, vances in Neural Information Processing Systems, volume 34, pp. 2109-21911 Curran Asso- ciates, Inc., 2021. URL httpt::tt/........t. httppceee/////ceeeeeeeee ngs. neurips cctppapar//222//1111. 1090781 31777811999229911 635876 62 Papeeee.1-pppppppppp. pdf.
Brian Murphy, Partha Pratim Talukdar, and Tom Michael Mitchell. Learning effective and inter- pretable semantic models using non-nee -en-eeattvee sparse embedding. In COLING, 2012.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pp. 722-729, 2008. doi: 10.1 109/CVVIPP.2008..7
B.A. Olshausen and D.J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-640 -609, June 1996.
Bruno A. Olshausen. Highly overcomplete sparse coding. In Bernice Rogowitz, Thrasyvoulos N. Pappas, and Huib de Ridder (eds.), Human Vision and Electronic Imaging XVIII, volume 8651 of Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, pp. 86510S, March 2013. doi: 11/1111//2221111
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learning. 17Xvv:711.00093) [cs], May 2018.
abs 11111.
arXiv: 1711.00937.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: wwtppppppeeerriiiipprppppppppppe at---yytoch-at--inmme pdf.
neuri cc cctppper//cccecccccccccccceeeeee
ivee sww.sttllylggee wwwthiigghhhhreeee wormarooreeeeeeeeee -ewpppeoreeeeeeeeeeeeeeee wwwedeeepplllraeeee lewnneng-nnnnnnraaaaa
Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2836 2011.
13

---

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning, pp. 5389-5400 PMLR, May 2019. URL wttptttsppppppppeppeeeee mlr. presss/ 197/techtt 19. html. ISSN: 2640-3498.
Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive Learning of General-Purpose Audio Representations. arXiv:2010. 10915 [cs, eess], October 2020. URL htttp://axxiv.org arXiv: 10100109115
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-199) 2014. URL http: http:mmll..rrrg wttp:/pppppppp.......eeeee
srivastaval aa.alaa.hie html.
Yee Whye Teh, Max Welling, Simon Osindero, and Geoffrey E. Hinton. Energy-based models for sparse overcomplete representations. J. Mach. Learn. Res., ((null):1235-122) dec 2003. ISSN 1532-4435.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/ 00744,3748 2018. URL http:://axxiv..rrgg// abs /1807. 03748.
Aad W. van der Vaart and Jon A. Wellner. Weak Convergence and Empirical Processes. Springer New York, 1996. doi: 10.1007/978- 1-1-788---47722555--5 URL htttps://dbi.orgg/111// 2F 978-1-4457-2544
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:17066 03762 [cs], December 2017. URL httpthtp::/aartee
org/ abs/ 1706. 03762. arXiv: 1706.03762 version: 5.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to Unseen Domains: Survey on Domain Generalization. arXiv:2103.0300 [cs], December 20 URL ptstp://arxte wwtp://pgp/......e///////// 03097. arXiv: 2103.03097.
Shulun Wang, Bin Liu, and Feng Liu. Escaping the gradient vanishing: Periodic alternatives of softmax in attention mechanism, 2021b. URL httts::: http::::////xxv..../ wwtpr/xiivoogabb/////aa
07153.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: scage scene recognition from abbey to 700. In 2010 IEFE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. CoRR, abs/2010.139022, 2020. URL https://arxiv org/abs/ 2010 0 13902.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.33230, 2021.
14

---

A PROOF OF THEOREM
Let us introduce additional notations used in the proofs. Define (z. R. (
Cym.i...kke = ((2,0) E E = y: K2 arg max Z/et Vj [
BE[V]
and
2 42 {a E 2: K2 = arrg max Zjet EEE E [L]].
mrg rE[V]
We then define Cs to be the flatten version of Cy.iiinneei ie.. {C*]k=
with C C1, C2 C2.1,...!,, C/yl Cyia..... Cpi+1 = C1,,,1,.,,,, Capl = 2...,,........ and so on. Similarly, define 2n to be the flatten version of 2 " We also use Q, [9 |-1,+1]V : arg max 9,), Ix Ih fi E [n] and (h) == E,(e(h, r)(r E Cx). Moreover, we define (( supte(v) suPareER 119 Ll2
E Ck),
-//7 and (((SEM))) = supie(v] su (()((g) o, (q') where o (g)s = Σ Elaree exe for j ...
We first decompose the generalization gap into two terms using the following lemma: Lemma 1. For any d > O. with probability at least 1 ò.the f following holds for all h E H.
E, (((h,,)) - - Σ e(h rz) S
1
K
((h.rr) + ov (2/6))
Ixl
a (h)
IA
EE
Proof. We first write the expected error the sum of the conditional expected error:
K
E, = Σ E,(((h,r)) E Pr(r E C+)
Σ
Er, [e(h, Pr(r Ck),
k=1
1=1
where ты is the random variable for the conditional with r E Cs. Using this, we decompose the generalization error into two terms:
1 n2
E, [e(h, r)
r(h
(3)
Σ E., (((h (Pe(r
Cu)
I&
Σ Er,
(e rk)
Iel 1 n
> ((h,rr))
n
n
The second term in the right-hand side of (3) is further simplified by using
l(h, ri) =
E(h, rs),
n
TE
as
K
K Σ rh) = Σ
I&l 1
E (h, rl)
I&l
Σ E(h,
72 k=1
In
EEIN
Substituting these into equation (3) yields
E, [e(h, r)]
rs)
(4)
n
Σ Erg [e(h rw)l
[Pr(r E Cx)
[&l
[zl
Σ
E(h, ri)
n
n
[Tl
sEZ.
K
B
Pr(r E Ct)
Izl
Ixl
By [eh
1
Σ ((h,rs)
In
k=1
FEI
15

---

By using the Bretagnolle-Hubeed Haae--uuberr-arre inequality (van der Vaart & Wellner, 1996, A6.6 Proposition), we have that for any 5 > O, with probability at least 8,
K
[T
2K In(2/6)
Pr(r E Cx)
(5)
n
Here, notice that the term of Σ Pr(r E Cb)
does not depend h E H. Moreover,
note that for any (f. h. M) such that M > O and B 2 O for all X, we have that P(J(X) 2 M) 2 P(f(X) > M) 2 (( h(X)= > BM + +(()), where the probability is with respect to the randomness of X. Thus, by combining (4) and (5), we have that for any E H. for any > 0, with
probability at least 1 - 8, the following holds for all / E H.
E, [e),
Σ (hh,
IAl
O+k (+(hh
Σ TET
e(h.ra))
ee
In (n/2)))
[I
In particular, the first term from the previous lemma will bounded with the following lemma: Lemma 2. For any f E {) Fianch
K
1
Inl
(f(f)
Σ ((f.ri))
n 1=1
In
SE
Proof. By using the triangle inequality,
IRl
1
Σ (Uni))
[IA
k=
iEIx
K
I
B.(((f.))lr Cxl
Σ e(fire)
I&l
EET
Furthermore, by using the triangle inequality,
B,(((f.r)(rr E Cn]
1
Σ ((frr))
1 E.(((Fr)) Cal
1
Σ ((frm))
In
I&
[Ta]
SEI
iEIN
<
|B(((f,))) cal r(frr:)
[Tal
TET
s sup
|((f.r e(f,r')
rreec
f = (SE (SEM(+) 00,, since {EEM(T) E Gs. by using the Lipschitz continuity, boundedness, and non-negativity,
sup (((,r) e(S,r')) r,r'Ec
sup sup
1(l, (SEM)))) (hy ((=
yEY E,
< R sup
100, (z) or (2('))lE
22
v
R sup
Σ Σ (a-(2))) ()( t=1 j=1
2,
R
sup sup
| or('')13
iE(V] E.E'E.
R Lip (EPEE))))))
16

---

Similarly, if f fhaso -bassn since P
Gs, by using the Lipschitz continuity, boundedness,
and non-negativity,
sup (e(f.r) -((f,r) r.r'Ck
11 o Rase (hy Rasse (2')1
= sup sup vEV 2,z'EZ
R sup
|| 2'|1r
< RV/ Lipl Lipls
Therefore, for any ( Fbsse
K
1
I&l
ax(f)
e(f.ri)
In R R RV())f).
I&
TEIN
Combining Lemma and Lemma 2, we obtain the following upper bound on the gap:
Lemma 3. For any > O, with probability at least 1 - 6, the following holds for any f E (JSEM(r) R
1 E (f.r) RVLL(S) + vo In(2/8)
[((((r)
Proof. This follows directly from combining Lemma and Lemma 2.
We now provide upper bound on ((f)))))) in the following lemma: Lemma 4. For any O,
1
1
(((SEM())) 2
1+ (V - 1)e-2/T
1+ (V 1)e- A/T
1
+ -V)
I+ee/T (V 2) 2/T) 1+ 2/T + (V 2)e A/T)
Proof. Recall the definition:
* <(SSEM))) = sup sup (|r((g) oz (r'))1
iE(V] a.''Q.
where
etsl
or
Σ e9//7
for j = 1,...., V. By the symmetry and independence over E [V] inside of the first supremum, we have
+(SSEM)))
sup ||0 (9))) q.
For any 9. o e Qı and e (,...., v} (with g = (q....., qv) and g' .oh...., Iv)),, there exists
8,, 8 > 0 such that
= - 8
and
df = dه %%
Here, since Zik
A Zi from the assumption, we have that for all E (2...., V}.
6; 0 > A 0.
17

---

Thus, can rewrite
cilT = - + Σ ((
V
= =691/T + 091/77
Σ
61/7
1=2
-91/T
1+ |
bult
i=2
Similarly,
v 1+ I+Ees
cit = c/ilT
6-///
6//7
i=2
Using these,
0r(4)) e///
e01/T
Et=l ea//
e91/t
1+ Σこ
2i=2
E-b/T
b,/T
and for all j E (2.....V V}.
cail
0+(9);
-/01-6/)/7
e91/T
(1+ Di=2 Ek=ge-bi1 e-bilT
b
1 Ei=ze-011 6//7
1
1 + +EEE,, clô,-b)//r
where 1, := (2...., V} \ (i) Similarly,
1
I a,(4) = 1 Ek
and for all j E (2..... V}.
1
or(d'); = 1 8/T + +Yter, -8)/=
Using these, for any q.d6 E Q1,
1
(a+(a)
1+ 2 -b/T
1 DI V
88/7
1
1 + ΣV 2//
1+ 1i=2 AA/T
1
1 (V 1)e-2 2/T
+ (V 1//
18

---

and for all j E (2..... V}.
o (r); or((())
1 - 01/7 + LiEE, el6y -d)/- 1 1 5/T + Lie,, (6 -8///
22/5 + -EiE,, el2-2)/1 1 + e/T + EiEr, e1A-2)/T
1 1 eA/T +VV - 2)c(A-2)/7 2)/T e 62/T + (V 2)ee2-Ael)
1 1 eA/T(1 + (V -V- 2))-2(1)
1 e (V 2/T-
By combining these,
- oo=')11
sup and' EQi
r Σ
(r)( - (r)())
= sup
9.0' q EQ1
j=1
$
1 (V 1)e 2/-
1+ (V 1)7-
+ -V-1) -1 1)
e/T (V 2/T) 1 e2/ (V - 2)e 4/T)
Using the previous lemma, we will conclude the asymptotic behavior of (P(fEMM())) in the following lemma:
Lemma 5. It holds that
v(SEM))) + 0 as - O.
Proof.
Using Lemma 4,
1
1
lim o ((SEM)))) T-0
lim 7-+0
1 (V - -1ee- 2/7
1+ (V - 1)e A/T
+ nV - 1)
lim 7
1
(V 2/1)
14 2/7 (V 2----/// 2/7)
Moreover,
2
lim
0.
1 (V -2/T 2/T 0
1 (V 1)e- A/r
and
lim
1
= 000- 0
1 It L+e//T(1+ (V - 2)77
1 21/22/111 (V - 2)e A/T))
Therefore,
lim p(SE <
Since yp ((SEEM))) 2 0, this implies the statement of this lemma.
19

---

As we have analyzed ((fSEM()) in the previous two lemmas, now ready compare p((SEM())) and (ofbase) which done in the following lemma: Lemma 6. For any
3 o0 (((EM)))) @(hhood S
$11 (1-V)- < 0.
4
Proof. From Lemma 4, for any O,
2
1 1 (V- -1)e-2/1 2/T
1
(((SEM))))
1+ (V 1)e-A/T
+ +n(V -1)
1 + (V - -:0e-2ee22 2/1)
1+ 2/T (V 2)e 2)e-A/7)
1
+ (V 1)e-2/1 2/7
1 - -V-) 1)
1
1
+ +VV-1
1 (1 (V- 17--21 2/rr
1+e2/ + (V 2))
2
2
1
+ (V 1)
1+ (V
1)e-2/7
V
2 + (V - 22e 2/7
1 1 2
-
2
1
2
+V 1)
0
V
2
1 1 1)
1
(V -
4
Recall the definition of
= sup sup |14-00 - q 1'112 xE[V] E..'
By choosing an element the set over which the supremum is taken, for any 8 2 > 0,
p 2 sup
11g o' 14 1111 Σ - (2 - 8)3V,
q.q'EQi
J=1
where ? = 1, 1 1- for j E V}. Ø = 5-1, 1, and 07 - -1 for j E ( V}.
=1
By combining those, for for any and 5 2 A > 0,
1 1
((FSEM))) - ((S <
+ (V- 10-11) 11 (2 - 8)2V
1 V
1
S1+
----- - -2 8VVV
V
4
3 1
v- 8
4
3
V
(2- 8)2
4
3
(1- 1
<
V
3
2 (1 - V)
4
20

---

We combine the lemmas above prove Theorem I, which is restated below with its proof:
Theorem 1. Ler V 2 For any Σ 8 > O, with probability at least 1 - 6, the following holds for any Ss US L ::
In(2/0)
[=,g(((s())) S
Σ (.s(=(()) + RV (.sss +oy
n
where e >> O is a constant in (n, f, H. 5, T, S). Moreover.
9 (S
0 0 and PREET
Py base
S 7 (1 < 0 V7 0.
Proof. The first statement directly follows from Lemma 3. The second statement is proven by Lemma 5 and Lemma 6.
B EXPERIMENT DETAILS FOR IMAGENET
B. IMAGE AUGMENTATION
The augmentation applied in order during training are:
Random Resize crop 224 224 image. A random patch of the image is selected and resized to a 224 x 224 image.
Random color jitter. Modifying the brightness, the contrast, the saturation and the hue.
Random gray scale. Randomly applying gray scale filter to the image
Random Gaussian blur. Randomly applying Gaussian bluer filter.
Random solarization. Randomly applying solarization filter.
The parameters of the augmentations are presented in Table 16. At validation and test time, resize the images to 256 x 256 and then center crop patch of 224 x 224.
For both training and evaluation, we re-normalize the image using the statistic of the training set.g
B.2 LINEAR EVALUATION
We follow the evaluation protocol from (Chen et al., 2020b). The linear evaluation is done by training a linear classifier on the frozen representation the ImageNet training samples. We train linear classifier with cross-entropy objective for 100 epochs using SGD with nesterov, momentum of 0.9 and batch size 256. We perform learning rate scheduling at epoch 60 and epoch 80 where we divide the learning rate by factor of 10. During training,
apply random resized crop 224 x 224 pixels and random horizontal flip. We sweep over a set of 4 learning rates: (0.5, 0.1, 0.05, 0.01 } 11 weight decays: (0, le 6, 1e 5) and 3 тa for SEM: {0.01, 0.1, 1). using validation set of 10 images per class and re-traing using the full training set. We report the results on the test set.
B.3 ROBUSTNESS EXPERIMENTS
We follow the evaluation procedure from (Lee al., 2021). We treated the robustness datasets as additional "test sets" in that we simply evaluated them using the evaluation procedure described above. The images were resized a 256 256 before being center cropped 224 224 image. The evaluation procedure was performed using the public robustness benchmark evaluation code of (Djolonga et al., 2020)
B.4 TRANSFER LEARNING LINEAR PROBE
We follow the linear evaluation protocol of (Kolesnikov al., 2019; Chen al., 2020b) We train linear classifier using a regularized multinomial logistic regression from the scikit-learn package (Pe- dregosa al., 2011). The representation is frozen, so that we do not train the encoder backbone nor
http://iiiibbboooeeeeeeeeeee/ees metriiss
21

---

the nonch-oomalizaiion statistics. We do not perform any augmentations and the images resized to 224 pixels using bicubic resampling and the normalized using the statistics on ImageNet's 's training set. We tune the regularizer term from a range of 45 logaribmiicall-ssacee values between 10 and 10 using a small validation set and re-train using the full training set. For SEM, we set Tu = 0 for all experiments.
B.5 TRANSFER LEARNING FINE-TUNING
We follow the same fine-tuning protocol of (Chen al., 2020b; Grill et al., 2020). We initialize the encoder with the pre-traines model and a classifier head with random initialization. We train for 20,000 steps with batch size of 256 using SGD with Nesterov momentum of 0.9. We set the momentum parameter for the batch normalization to be max(1 - 10/s, 0.9) where 8 is the number of steps per epoch. During pre-training, we use random resize 224 224 pixels and random horizontal flipping. At test time, we resize the images along the shortest size to 256 pixels using cubic resampling following by a center resize to 224 x 224 pixels. Due to computational constraint, we only tune the learning rate using a search of 7 values spaces on logarithmic scales between 0.0001 and 0.1. 1. For SEM, we set Fa 1. for all experiments After choosing the best learning rate of a validation set, re-run the models using the full training set and evaluate it on the test set, which we use to report the numbers.
B.6 SEMI-SUPERVISED LEARNING
We follow the semi- cumi-supereieed learning protocol of (Chen al., 2020b; Grill al., 2020). We initialize the network using the pre-trained representation and initialize a classification head using random initialization. We fine-tune the encoder while training the classification head using a small subset of ImageNet. We choose the same subset used in prior works which is defined in the MaasooFFow-Datasse software. During training, we random resize the images to 224 x 224 pixels along the shorter size using bicubic resampling followed by center crop and random horizontal flipping. At test time, we resize the image to 224 x 224. We optimize the cross entropy loss with nestorov and momentum of 0.9 using batch sizes of 224. We train models for (30, 50) and take the best performing on the validation set. The learning rate used is chosen among a set of 5 learning rates: {0.01, 0.02, 0.05, 0.1, 0.005). For SEM, we also search Ta {0.01, 0.1, 1}. We perform the search on the best performing one on the validation set and the number are returned are obtained using the test set after re-training using the full training set.
C HYPERPARAMETERS
The implementation of the SSL methods used in this work are taken from Solo-Learn (da Costa et al., 2021) to which we added the SEM module. The pre-training hyper-parameter of every SSL methods trained on CIFAR-100 100 repository of Solo-Learn. The hyper-parameters are also provided in the launch scripts accompanying this work. Due to the large number of SSL. methods probed in this work and the amount of space it would require to exhaustively detail all of the pypar-parameters, we refer the reader to the code.
with ResNet- 18 used in this work are the default provided in the companion
For the CIFAR-100 results obtained with BYOL and a ResNet-50, we have slightly modified the default parameters. Otherwise, the baseline BYOL model would not obtain competitive results. The hyper-parameters were tuned using the BYOL baseline and the SEM module was not considered in the selection of the SSL tyrem--arameters. The B YOL script accompanying this work and presented below for completeness.
are presented in the launch
For the ImageNet experiments, we took the hyper-parameters proposed in the launch scripts of Solo-Learn to which only modified the amount of epochs (100 epochs to 200 epochs.)
Here, we present all of the SEM hyper-parametee used in every experiments. These pareetpprs can also be found in the launch scripts accompanying this work.
We present the paremparreeters used to train for BYOL+SEM and MoCo+SEM on CIFAR100. Unless mentioned otherwise, these are the parameters used.
22

---

Table 7: CIFAR-100
SEM SimCLR RN-18 for
Tp
p
5000
13
0.17
0.78
Table 8: SEM MoCo RN-18 for CIFAR- 100
Table 6: BYOL with ResNet-50 for CIFAR-100.
L
V
Tp
5000
13
0.04
0.01
precision
16
Learning rate
0.5
Table 9: SEM BYOL RN-18 for CIFAR-
Weight-decay
le-4
100
Optimizer
sgd lars
L
V
Tp
LR scheduler eta lars exclude bias norm (lars)
warmup cosine
5000
13
1.0
1.0
0.001
True
Table 10:
SEM SWAV RN-18 for
batch size base momentum
256
CIFAR-100
0.99
V
T
final ema momentum proj output dim proj hidden dim pred hidden dim augmentations:
1.0
5000
13
0.85
1.5
256
4096
Table 11: SEM DINO RN-18 for CIFAR- 100
4096
L
V
Tp
solarization_prat
view 1: 0 view 2: 0.2
5000
13
1.0
1.0
crop size
32
hue saturation
0.1
Table 12: SEM Barlow RN-18 for
0.2
CIFAR- 100
contrast brightness
0.4
V
0.4
P
5000
13
1.0
0.99
Table 13: SEM VicREG RN-18 for CIFAR-100
L
V
Tp
T
5000
13
1.0
1.0
Table 14: SEM BYOL RN-50 for CIFAR-100
Table 15: SEM BYOL all ResNets for ImageNet
L
Tp
L
V
Tp
T
5000
13
5000
21
0.16
0.04
C.1 COMPUTATIONAL RESOURCES
For all CIFAR- 100 training, used 1 RTX-8000 per experiment. For ImageNet experiments, we used parallel training with 2 40GB A100 for the training with ResNet50 and ResNet50-x2 and 40GB A100 for the training with ResNet50-x4. With this setup, the training takes about week for the ResNet50 experiments and about 10 days for the ResNet50-x2 and ResNet50-x4 experiments.
D ADDITIONAL STUDIES OF SEM
In Section 4.2, we discussed the effect of scaling and V as well as changing the Softmax temperature during pre-training of the online network and changing the Softmax temperature for the downstream task. Here, we propose additional studies SEM to provide a better mastery of the method. We provide a method for reducing the memory overhead of SEM and experiments demonstrating that despite this version still largely outperform the baseline. We additionally present the effect of modifying the embedder contributing to the insight how get the most out of SEM. Next, have discussion with a study of the spectrum of the covariance matrix of the SEM representation and the B BYOL representation, showing insight how SEM particularly improve the training signal
23

---

Table 16: BYOL with all ResNet-50 architectures for ImageNet.
precision
16
Learning rale Weight-decay
0.4
le-6
Optimizer
sgd + lars
LR scheduler
warmup cosine
eta lars
0.001
exclude bias norm (lars)
True
batch size
256
base ema momentum final ema momentum
0.99
1.0
proj output dim proj hidden dim pred hidden dim augmentations:
256
4096
4096
solarization propatiiiooppobbbboopooo gaussian_ prob crop size hue saturation contrast
view 1: view 0.2
view I: 1.0 view 2: 0.1
224
0.1
0.2
0.4
brightness
0.4
Table 17: # of parameters, # of activations, allocated memory, computation efficiency (FLOPs/sample) and CIFAR-100 accuracy of BYOL, BYOL with SEM and its memory-efficient variant with blocks (denoted B BYOL+ + SEM/8).
# params
activations
vRAM (GiB)
FLOPs
Accuracy
Resnet-18:
BYOL
16.5M
0.731M
4.0
7.20e8
70.7
BYOL+SEM
313.7M
0.797M
13.1
1.01e9
73.9
BYOL+SEM/8
51.9M
0.796M
5.3
7.46e8
73.3
Resner-50:
BYOL
35M
4.05M
11.1
1.65e9
74.3
BYOL+SEM 425.6.M
4.12M
21.9
2.04c9
77.4
BYOL+SEM/8
76.7M
4.12M
11.8
1.69e9
76.6
during pre-training. We provide a scaling analysis of BYOL and BYOL + SEM on CIFAR-I00. We end with experiment showing that pre-training with SEM is necessary to get the best performance.
D.1 AN EFFICIENT VARIANT OF SEM
A large over-complete representation may induce a significant memory footprint due to the additional parameters of the fully connected linear layer used map and from the representation. For SEM require two such mappings as depicted in Figure 2c for BYOL. To reduce the amount of parameters, we propose to sparsify the weight matrix of the fully connected linear layer. We propose do so by taking the block diagonal of the parameters of the matrix multiplication and setting the parameters outside the block diagonal to O. Formally, let b w E R. and y = w be the fully connected matrix multiplication. Instead, partition into blocks with zf R& and define smaller w' E R
where i [L] is the tth block. Then, we perform batch matrix multiplication of o² and w² that concatenate follows: y² v² u² and y² Concat ([y² ... "I).. Thus, the amount of parameters of this matrix multiplication seales in O( mm=-0) allowing us to reduce the memory consumption by increasing n, the number of blocks.
We perform an experiment where we partition the embedder and the first linear layer of the projector into blocks. We present the results in Table 17 in which compare the of parameters, the
24

---

# of activations, the allocated vRAM by pytorch, the FLOPs/sample and the aceuracy of BYOL, BYOL+SEM and BYOL+SEM/8 representing the model with blocks obtained following the method described above. We observe that partitioning the matrix multiplications of SEM allows vastly reduce the computation parameters while still yielding an important improvement over the baseline. This result demosntrate that SEM can be beneficial while inducing minimal computational overhead.
Attentive readers may notice that this performance is better compared to the ablation presented in Figure 3. The difference in performance due to probing the embedder's output (i.e. =q) in Figure and probing the encoder's output (i.e. eq) in Table 17. Using the each ablation's 's representation for probing the other recovers the performance observed by each.
D.2 ADDITIONAL ABLATION THE SEM PARAMETERS
Ablating the embedder In the main text, we mentioned that we use batch normalization)) at the output of the embedder. The reason we use batch normalization is mostly due the fact that we wanted to avoid tuning any syper-parametee that were not related to SEM to emphasize its contribution. Using BatchNorm gave best performance without tuning the hyper-parameters of the baseline models.
Here, we want to emphasize that SEM can used without batch norm, more prperpparameters might need to be tuned for it to perform as well as the model with batch norm in the encoder. For example, we found that using no weight decay was important to get better performance when we not have batch normalization as illustrated in Table table 18. We leave the full study of the interaction of SEM with the SSL related parameters for future work.
Table 18: Understanding the relationship between the use of BatchNorm in the embedder and the weight decay hyper-paree parameerr..
BatchNorm
weight decay
Accuracy
0 67.2
le-5
57.9
0
68.3
le-5
73.9
Another decision is to linear layer as the embedder. Other alternative may include using the Identidy function (i.c. the output the encoder is used for SEM). However, we want to systematically use the same encoder the SSL model, then are constrained representation size that is the one of the ResNet encoder (i.e. 512 for ResNet-18).
Finally, we showcase that using recommend practitioner to limit the expressivity of their embedder.
expressive embedder leads exacerbated performance and
Table 19: Comparing alternative embedders.
Accuracy
Identity
63.0
Linear
73.9
hidden layer MLP
65.0
A very very large embedding Using a ResNet-18 encoder and the method proposed in Section D. we further scale the embedding size of SEM to see where the performance saturates for classification. In Figure 7 observe that the performance saturates for L 10000 for the classification task. We conjecture that the optimal might be different for other tasks, but we leave that study for future work.
D.3 ANALYZE OF THE SPECTRUM OF THE COVARIANCE MATRIX OF THE REPRESENTATION
To obtain a better insight on why the SEM representation leads to better downstream performance, we analyze the spectrum of the covariance matrix of the representation using the methodology presented
25

---

73.5
Top-1 Accuracy
72.5
72.0
71.5
1000
5000
10000
20000
50000
Figure 7: Study of very very large L using ResNet-18 backbone and 8 SEM/8 blocks using the method described in Section D.I.
in Jing et al. (2022). That is, we collect the embedding vectors of the test set of CIFAR-106 100 using a pre-trained model using ResNet-50. For B YOL,, we have additional embedder without softmax normalization (as done in Figure 3). For BYOL and BYOL+SEM we use the embedder's 's output (zg) perform the evaluation. To compute the covariance matrix C E RL-VLLV of the embedding layer z, we define ΣN z//N the average representation over the N samples and compute the covariance follows:
C Σ (z- ()=
To plot the spectrum of the covariance matrix, we take the singularalue decomposition of the matrix (C USV with the diagonal of the singular values, which we plot in sorted order and logarithm scale in Figure 8.
This experiment demonstrates that the softmax normalization counters the dimensionality collapse that was discussed in Jing et al. (2022). Interestingly, the drop observed with SEM with T. 2 500 occurs at the index 2048 which the dimensionality output of the ResNet-50 encoder.
SEM L= 1000
SEM L-500
-10
log
-20
2000
10000
12000
eigenvalue index
Figure 8: Spectrum of the covariance matrix of the represention for BYOL and BYOL + SEM obtained with ResNet-50 encoder.
D.4 SCALING THE RESNET ENCODER FOR CIFAR- 100
We perform a scaling experiment on CIFAR-100 where we compare the scaling behaviour of BYOL and BYOL + SEM. We evaluate the computational cost of the methods and the resulting downstream accuracy for a range of four resnets: ResNet-13 18, ResNet-50, ResNet-50 x2 and ResNet-50 x4. In Figure 9, we observe that SEM has a better scaling behaviour than the baseline, especially as increase the width of the ResNet-50. For B BYOL, observe that the performance decays for
26

---

RN50-X4
RN50
RN50-X2
Top-1 Accuracy
76
@NN18 RN50
74
RN50-X2
72
RN18
70
BYOL
66
BYOL
RM50-X4
64
7.2e+08
1.65e+09
4.65e+09
1.52e+10
FLOPS
Figure 9: Scaling the ResNet encoder for CIFAR-100.
Table 20: Downstream accuracy of training a classifier with SEM normalization of the representation while using unormalized representation during pretraining. Experiments performed with ResNet-50 encoder.
Pre-train model BYOL Embed
Probe location Embedder Embedder Embedder
SEM(T 0.1)
Accuracy
No
69.8
BYOL Embed
Yes
72.3
BYOL+ SEM
Yes
77.3
ResNet-50 with width x2 and x4. This is not unprecendented, as prior works as demonstrated other methods where scaling up the capacity of model led to decrease in performance. When comparing the discrepancy with Figure 1, we attribute that to the fact that CIFAR-100 is small dataset. In fact, observe that the training accuracy stays constant about 79% for all the ResNet-50 scales demonstrating overfitting for the baseline BYOL. Nevertheless, SEM prevents the decrease in performance and even lead further improved performance we increase the scale of the ResNet-50.
D.5 THE ROLE OF PRE-TRAINING WITH SEM
We probe the downstream accuracy obtained of a model pre-trained without SEM and add SEM normalization only for the downstream classification. For this experiment, we take pre-trained model with embedder (i.e. BYOL + embed) with L = 5000 and V = 13 and add the softmax normalization only for the downstream classification. We do not use SEM during pre-training. We observe that using SEM for downstream classification leads to an improvement even when the model is not pre-trained with SEM, demonstrating the utility of SEM downstream classification. However, note that the performance of the model pre-trained without SEM is much weaker and thus demonstrates the imprtance of also pre-training using SEM.
E CIFAR- 10 RESULTS
We confirm that our method also yield improvement on simpler datasets such as CIFAR-10. Here, we compare BYOL and BYOL SEM ResNet-50 and observe and improvement of 1.6%.
Table 21: Downstream accuracy of training a classifier with SEM normalization of the representation while using unormalized representation during pretraining. Experiments performed with ResNet-50 encoder.
Pre-train model
TOP-1 Accuracy
BYOL
94.2
BYOL + SEM
95.8
27

---

F CIFARI00 SUPERCLASS
The 100 classes of 100 (Krizhevsky, 2009) are grouped into 20 superclasses. The list of superclass for each class in Table 22
Table 22: Set of classes for each superclass on CIFAR-100.
Superclass
Classes
aquatic mammals
beaver, dolphin, otter, seal, whale aquarium fish, flatfish, ray, shark, trout orchids, poppies, roses, sunflowers, tulips
fish
flowers
food containers
bottles, bowls, cans, cups, plates apples, mushrooms, oranges, pears, sweet peppers clock, computer keyboard, lamp, telephone, television bed, chair, couch, table, wardrobe bee, beetle, butterfly, caterpillar, cockroach bear, leopard, lion, tiger, wolf bridge, castle, house, road, skyscraper cloud, forest, mountain, plain, camel, cattle, chimpanzee, elephant, kangaroo fox, porcupine, possum, raccoon, skunk crab, lobster, smail, spider, worm baby, boy, girl, man, woman crocodile, dinosaur, lizard, snake, turtle hamster, mouse, rabbit, shrew, squirrel maple, oak, palm, pine, willow bicycle, bus, motoreycle, pickup truck, train lawn-mower, rocket, streetcar, tank, tractor
fruit and vegetables
household electrical devices
household furniture
insects
large carnivores
large man-made outdoor things large natural outdoor scenes large omnivores and herbivores medium-sized mammals
non-insect invertebrates people
reptiles
small mammals
trees
vehicles
vehicles
28

---

G ADDITIONAL CIFAR-100 COHERENCE GRAPHS
29

---

5128
rabbit
5148
5239
whale
5300
hale
5365
sa5
sorew
glain
5336
5274
accoon
mnby
5146
hhane
wark
s3
luwg mowe
crecadile
amp
Tatfish
baby
actor
5233
Kilip
555
Bed
Mighin
panz
6422
Adman
Beycle
shale
atter
turtle
578
phir
touch
lale
5404
Frest
tree
key
536
clock
uum
532
orkmach
571
palp free
Gee
$25
ppar
5327
maple
tree.
orde
Ruuk
m palle e troe
S61
tan
5321
bottle
587
Seal
5291
plate
bawl
fox
Gasnge
foorr
5195
snake
5210
Kuish
wardrobe
baby
occidd
pappy
Vyle Merlly esee
$190
welf
5188
man
orange
prokep
drab
S319
squire
5138
5106
strepica
train
Whest
5415
jobster
aple
hhir
sunfoow
pep
530
566
moshroo
5189
pear
sneil
make
sirob
huster
worm
sweet pep
Tows
5434
tel phin
578
60a
urte
5227
skyscraper
Ruuse
Spider
ccon
figer
5272
ma do tr
5309
camel
5216
Wolf
willo lollow tr
seal
$182
pickup truck
5360
tauch
chimpanzee
S284
56
apple
5256
skunk
561
Séal
eeskffoach
rocket
tractor
5279
529
hoous Boy
5184
palm, tree
cloud
5350
5408
lawa mower
motorcycle
tank
plain
computer keybo
5107
5249
train
castle
archid
549
house
51.44
road
5276
cattle
loopard
tank
5398
tiger
caterpillar
kangaroo
lamp
5194
teout
5240
bear baetle
5382
bettle
51.78
cockroach
kangaroo
spider
5363
otter
516
Betfish
$248
bee
(a) BYOL baseline
(b) BYOL baseline with large rep- resentation
(c) BYOL + SEM
Figure 10: Comparison of the full semantic coherence graph W3 between BYOL and BYOL SEM.
30


Reviews:
[{"review_id": "rnAuSnk8uf", "reviewer": "Reviewer_E9n7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests using simplicial embedding (SEM) -- instead of the standard R^d embedding -- as the feature for self-supervised learning and downstream classification. Specifically, given $L*V$-dim feature, SEM applies softmax operation for each $V$-dim subvector and concatenates the sparsified vectors. The paper claims the superiority of SEM theoretically and empirically. First, the paper provides a new generalization bound where the complexity measure $\\varphi$ is determined by the type of final representation. Here, the paper claims that the complexity measure of SEM is strictly smaller than the one of standard R^d embedding. Second, the paper demonstrates that SEM improves the classification performance of various self-supervised learning methods.", "summary_of_the_review": "Overall, I think the paper has a clear contribution. However, I'm on the borderline since empirical evidence is not convincing enough. I'm willing to raise my score if my concerns are addressed:\n- ImageNet results comparing with BYOL in 800 epoch\n- CIFAR-10 results and explanation of why SEM is more effective for many-class classification\n- Results on wide models such as ResNet-50 x16 instead of adding overcomplete layer\n- Discussion on why SEM should be used for SSL - why not supervised learning?\n- Comparison with MoE methods - would SEM be a better way to scale large models?\n", "strength_and_weaknesses": "**Strength**\n\nThis paper makes a good attempt to validate their claims, providing a theory and empirical evidence. For example, the paper provides an ablation study on the size of $L$ and $V$, a comparison with hard discretization, and an analysis of semantic coherence. The overall presentation was well-written and supported by proper evidence.\n\n---\n\n**Weakness**\n\nWhile the paper suggests an interesting alternative to designing an embedding, my biggest concern is whether future research would really use this technique. Here are some reasons why its practicality is not convincing yet.\n\n* **Weak baselines.** The paper shows an improvement over underfitted BYOL trained for 200 epoch (IN acc.: 70.6) instead of standard 800 epoch (IN acc.: 74.3). Since many methods converge faster but saturates eventually (or even collapse later), the paper should provide the results at 800 epoch. Also, this repo (https://github.com/HobbitLong/SupContrast) shows that SimCLR trained on CIFAR-100 shows acc. of 70.7, although they use ResNet-50, which is 5% higher than this paper. Thus, the overall baseline results are undervalued, and the gain of this method can be exaggerated. Providing the results in the SOTA setup would make the benefit of SEM more convincing.\n\n* **CIFAR-10 results.**\nThe paper only demonstrates the CIFAR-100 results instead of more standard CIFAR-10. I suspect that SEM is less effective for CIFAR-10. Still, it would be informative if the paper provided CIFAR-10 results in Appendix. How can we explain this if SEM is less effective for few-class classification? Discussion for coarse vs. fine-grained classification would be informative.\n\n* **Heavy computation.**\nAs the paper mentions, SEM needs an additional layer to expand the embedding overcomplete, which requires many parameters, memory, and computation. Although Table 17 in Appendix suggests an efficient version, it still uses 2~3x of parameters. Due to this, the actual baseline should be BYOL with some additional nonlinear layers instead of the vanilla BYOL.\nI appreciate that the paper shows that naively adding the overcomplete layer to BYOL does not work, as shown in Figure 3. However, I think a better way to sell this method is to emphasize the scaling part -- instead of adding an overcomplete layer (which may be arguable for a fair comparison) -- achieving SOTA using very-wide models would be more impactful. For example, current self-supervised learning (SSL) methods achieve SOTA on models like ResNet-50 x4. If the paper claims that prior work does not scale for larger models such as ResNet-50 x16 but SEM can, it could be a game changer.\n\n* **Why self-supervised learning (SSL)?**\nWhile the paper targets SSL, the idea of SEM could also be applied to supervised learning. Why the paper's scope should be SSL? Would SEM be more effective for SSL than supervised learning?\n\n* **Comparison with a mixture of experts (MoE).**\nConceptually, SEM may be thought of as a simple instantiation of MoE [1] or MCL [2] where each $V$-dim subvector is routing the specialized feature for each class. The bipartite in Figure 6 shows the learned features are indeed specialized to some classes. Here, the MoE technique is popularly used to train large-scale models. Since one major benefit of SEM is robust training on larger overcomplete features, it may discuss the relation to MoE and may show superiority over them.\n\n[1] Fedus et al. A Review of Sparse Expert Models in Deep Learning. arXiv 2022.\\\n[2] Guzmán-rivera et al. Multiple Choice Learning: Learning to Produce Multiple Structured Outputs. NeurIPS 2012.\n\nTypo:\n- Gubel -> Gumbel in the caption of Table 2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TS058GPVyQ", "reviewer": "Reviewer_u5uQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In self-supervised learning (SSL), a representation is learned and it is usually one multi-dimensional vector. This paper proposes an effective representation using multiple sparsified embeddings (SEM) inspired by over-complete representation. It is obtained by projecting the SSL embedding into multiple smaller dimensional vectors each through a softmax operation. The sparsity of the smaller vectors can be controlled via the temperature parameter in the softmax function. The author gives theoretical analysis on the benefit of using SEM and also conducts extensive empirical experiments. The results show the superiority of the proposed method. ", "summary_of_the_review": "The author proposes a simple yet effective component to build more interpretable and more effective representation. The experiments are solid and extensively demonstrate the effectiveness of the method. There, I think it is a good paper - \"accept\".", "strength_and_weaknesses": "Strength: \n\nClarity and quality of writing. Extensive experiments.\nThe paper is well written with a pretty clear introduction of background and related work. I also like how the author draws inspirations from the over-complete representation literature to motivate the proposed method. The experimental results show both quality improvement (with reasonable computation increase) and interpretability improvement.\n\nOriginality. The work seems pretty novel to me. \nPractical application: Being aware of the increased memory usage from SEM, the author also studies how to efficiently reduce memory usage at inference time. \n\nThough the findings in the proposed work is largely based on prior work in SSL, the SEM idea is pretty interesting and effective (observation from the empirical results).\n\nWeakness:\nQuestions to the author(s):\n\n1. Regarding Figure 5 (a), or effects of increasing $L$. Have you investigated when the improvement of accuracy starts to diminish if we further increases $L$ (e.g, increase $L$ to 20K, 50K)?  A larger number of basic vectors could be very helpful in defining finer granularities in some applications.\n\n2. Could we increase the embedding dimension in BYOL to achieve the similar quality improvement as achieved by SEM?\n\n3. Is the BYOL metric in Figure 1 a strong baseline? This is not something I am familiar with. Please clarify.\n\n4. After SSL embedding is obtained, one usage of the SSL embedding is to perform clustering to obtain needed level of granularity and interpretability. Let's say we cluster the embeddings into 5K groups. And also on the other hand, we have the SEM embedding (when L= 5K), what would be the pros and cons of the two approaches ?\n\nI will be willing to increase my score once the questions above are well clarified/explained.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NoXzf-92JC", "reviewer": "Reviewer_Jv3d", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Examines the utility of including a large number of parallel softmax operations, also called simplicial embedding (SEM), within a SSL method.  Includes theory showing improvements in generalization error bounds from using SEM, as well as extensive experimental evaluations showing benefits to classification, transfer learning, robustness to input corruption, as well as demonstrating interpretability of the SEM features (clustering by class).  The experiments also give insight on how to choose the number of simplices and the dimensionality of the simplex, as well as the temperature parameter.", "summary_of_the_review": "This paper puts a lot of thought into presenting and analyzing a potentially high-impact addition to semi-supervised learning techniques.  The author's rebuttal addressed many concerns about the fairness of the comparisons and overclaiming. The impact of the technique (around 1-2 percent accuracy) would be interesting to a general audience but perhaps not at spotlight level.", "strength_and_weaknesses": "Strengths\n\n+ Simple idea with interesting connections to sparse coding (in neuroscience)\n+ Goes beyond looking at performance and also shows improved interpretability.\n+ Includes a theoretical analysis.\n+ Provides useful guidance on setting hyperparameters.\n+ Evaluations show benefits in several problems: classification (without and without corruption) and transfer learning.\n\n\nWeaknesses\n\n- Why are some of the competing methods doing so poorly?  Is the poor performance of BYOB+Gumbel on CIFAR-100 really due to SEM being a better method, or perhaps the Gumbel approach was not implemented as well as possible? [after rebuttal: addressed]\n- It would be nice to how changing the architecture affects results. [after rebuttal: addressed]\n- A minor weakness, but the use of softmax is so commonplace that it doesn't even seem to warrant a mention in related work.\n- The practical utility of the theoretical result is unclear.\n- As reviewer E9n7 and 4CQ8 noted, the baselines could be improved.  BYOL should be trained until convergence, and increasing the number of nonlinear layers or increasing the size of the network to match BYOL+SEM would be more convincing. [after rebuttal: addressed]\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Easvwq9qiJM", "reviewer": "Reviewer_4CQ8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to apply a  subcomponent-wise softmax to SSL representations (z -> partition -> softmax -> concatenate -> new z), dubbed SEM. They empirically show that this simple transformation combined with a large increase in the dimensionality of the representation yields significant improvements in standard SSL benchmarks (in and out of distribution). They also try to explain the gains by providing generalization bounds of the downstream predictors.", "summary_of_the_review": "The paper is enjoyable to read and could be impactful if the author's main claim (usefulness of SEM) holds. My main issue with the current version is that I have doubts about: (1) whether the theory is meaningful; (2) whether the gains are due to SEM rather than increasing the dimension of the representation. \n", "strength_and_weaknesses": "**Updated during rebuttal** the authors removed some strong claims and results showing that the gains are due to the proposed method rather than only increasing dimensionality of the representation. I thus update my score 5->8 .\n\n-------\n\n**Strengths**\n- **Paper is well written** the paper is generally easy to understand and enjoyable to read.\n- **Simplicity and generality** the main strength of the transformation is that it is simple and widely applicable. \n- **Significant empirical gains** the empirical results show uniformly significant gains across settings. Assuming that the baseline is indeed meaningful (see below), the results clearly indicate that the proposed method would be useful. \n\n**Weaknesses**\n- **Is the theory meaningful?** my main issue with the theory is that it might be meaningless and unrelated to SEM[^1]. Specifically, it seems that using the arguments of your theoretical section one would conclude that any transformation that shrinks the support of Z would be better. For example, let $f_{\\mathrm{new}}(z)=g(z/2)$ be a function that is defined like for SEM but the transformation is halving rather than SEM (other transformations like relu also works). Then if I followed your definitions you would essentially[^2] have $\\psi_{f_{new}} - \\psi_{f_{base}} \\leq 2V - 4V = -2V < 0 $. Using the rest of your arguments you would thus conclude that \"halving representation provably leads to better performance\".  You can further use a parameter $t$ instead of $2$ to say $\\psi_{f_{new}} \\to 0$ as $\\frac{1}{t} \\to 0$. For the theorem to be meaningful for SEM it should give rise to the same conclusion for any transformation just because it shrinks the support. Am I missing something?\n- (addressed in rebuttal) **Are gains only due to increasing dimensionality of the representations?** my main issue with the experimental results is that it is unclear to me whether the gains are only due to the increasing dimensionality of the representation rather than to the use of SEM. In particular, I think that your experiments and previous work show that: SEM+high dim helps, SEM only does not help, and high dim only helps. It is thus currently unclear what is the gain of SEM. More specifically: \n    - **Previous papers show similar gains by only increasing dimensionality** [1] argues theoretically + shows empirically that increasing dimensionality of representations helps downstream performance. There figure 7c shows a monotonic improvement in embedding size similar to BYOL+SEM in your figure 3. The line of work on dimensionality collapse, eg, [2], also suggests that higher effective dimensionality of the representation should improve performance regardless of SEM.\n    - **Inadequate baseline** As just mentioned [1] finds a significant monotonic improvement when increasing dimensionality without SEM while your baseline doesn't. I think that the problem is that the baseline you are using does not actually increase the effective dimensionality of the representation. Indeed, your embedding layer is linear (linear layer + BN) and as a result, it cannot increase the dimensionality of the span of the representations (ie you are only increasing the ambient space but not the real/effective dimensionality). To bypass this issue [1] increases the dimensionality before the avg pooling layer (see Appx F1). Another standard way of increasing the dimensionality of representations of resnets which can lead to significant gains is to change the pooling layer (eg see [this code](https://github.com/facebookresearch/vissl/blob/main/configs/config/benchmark/linear_image_classification/imagenet1k/eval_resnet_8gpu_transfer_in1k_linear.yaml#L65) ). For this reason, the current figure 3 does not adequately support the hypothesis that SEM is what drives the performance up.\n    - **SEM without increasing dimensionality seems to show no gains** BYOL + SEM without increasing dimension in Figure 3 seems to perform similarly to standard BYOL (table 17), thus suggesting that SEM without increasing dimensionality does not help.\n - (addressed in rebuttal) **Some over overclaiming**\n    -  in the abstract, you say \"we formally prove that the SEM representation leads to better generalization than normalized\". This is a strong claim, which I do not think is supported given that you only show one upper bound is smaller than the other (without a discussion about their tightness). To support such a statement you could for example provide a lower bound on the generalization gap of $f_{\\mathrm{base}}$ and show that it is larger than the upper bound of $f_{\\mathrm{ERM}}$\n    - section 4.2 you say \"Figure 5 [...] allowing us to confirm two predictions made in Section 3.2: the expected generalization [...]\". Your experiments do not actually test the generalization gap of the probe (as suggested by your theory) but the general test performance. One potential explanation (which would be my guess) is that the training performance of the linear probe is actually what drives the performance up (due to the increase of dimensionality) rather than the shrinking of the generalization gap. I think that showing the generalization gap is needed if you want to make any claims about the relation between your theory and experiments. Another way of testing your claims would be consider the standard \"semi-supervised\" setting where the downstream probe uses only 1% of ImageNet. My guess is that your representation will perform worst there (due to the high dimensionality) while your theory would suggest that it actually performs better due to the sparsity.\n\n[^1] Note that I haven't read all appendices in detail. If the argument breaks please refer me to the exact line and I will look at it.\n\n[^2] I removed $\\delta$ for conciseness but the argument still holds with those.\n\n[1] [Improving Self-Supervised Learning by Characterizing Idealized Representations](https://arxiv.org/abs/2209.06235)\n\n[2] [On Feature Decorrelation in Self-Supervised Learning](https://arxiv.org/abs/2105.00470)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%