Compressive Visual Representations
Authors: Kuang-Huei Lee
Anurag Arnab
Sergio Guadarrama
John Canny
Ian Fischer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-Supervised Learning
Information Theory
Compression
Visual Representations
Vision

Abstract: Learning effective visual representations that generalize well without human supervision is a fundamental problem in order to apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representation, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing a tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related to robustness, this provides a new explanation for why compressed models are more robust. Our experiments confirm that adding compression to SimCLR and BYOL significantly improves linear evaluation accuracies and model robustness across a wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x.

Compressive Visual Representations
Kuang-Huel Lee Google Research leekh@google.com com
Anurag Arnabt Google Research aarnablgoogle. com
Sergio Guadarrama
John Canny Google Research
Ian Fischer Google Research i iansf lansf@google com
Google Research sguadadgoogle... com
canny@google. com
Abstract
Learning effective visual representations that generalize well without human su- pervision is fundamental problem order apply Machine Learning to a wide variety of tasks. Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respec- tively, have made significant progress. In this work, we hypothesize that adding explicit information compression to these algorithms yields better and more robust representations. We verify this by developing SimCLR and BYOL formulations compatible with the Conditional Entropy Bottleneck (CEB) objective, allowing us to both measure and control the amount of compression in the learned representa- tion, and observe their impact on downstream tasks. Furthermore, we explore the relationship between Lipschitz continuity and compression, showing tractable lower bound on the Lipschitz constant of the encoders we learn. As Lipschitz continuity is closely related robustness, this provides new explanation for why compressed models are more robust. Our experiments confirm that adding compression SimCLR and BYOL significantly improves linear evaluation accu- racies and model robustness across wide range of domain shifts. In particular, the compressed version of BYOL achieves 76.0% Top-1 linear evaluation accuracy on ImageNet with ResNet-50, and 78.8% with ResNet-50 2x
Introduction
Individuals develop mental representations of the surrounding world that generalize over different views of shared context. For instance, shared context could be the identity of object, as does not change when viewed from different perspectives or lighting conditions. This ability to represent views by distilling information about the shared context has motivated a rich body of self-supervised learning work [54] 4 12
30 33 47]. For concrete example, we could consider an image from the ImageNet training set 60 as a shared context, and generate different views by repeatedly applying different data augmentations. Finding stable representations of shared context corresponds learning minimal high-level description since not all information is relevant or persistent. This explicit requirement of learning concise representation leads us to prefer objectives that are compressive and only retain the relevant information.
Recent contrastive approaches to self-supervised visual representation learning aim to learn rep- resentations that maximally capture the mutual information between transformed views of an The primary idea of these approaches is that this mutual information
image 54
12 33 40).
Main contributors
Code available
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

corresponds to general shared context that is invariant to various transformations of the input, and it assumed that such invariant features will be effective for various downstream higher-level tasks. However, although existing contrastive approaches maximize mutual information between augmented views of the same input, they do not necessarily compress away the irrelevant information from these views [22 33 As shown in 26 277, retaining irrelevant information often leads to less stable representations and to failures in robustness and generalization, hampering the efficacy of the learned representations. An alternative state-of-the-art self-supervised learning approach is BYOL (30), which uses a slow-moving average network to learn consistent, view-invariant representations of the inputs. However, it also does not explicitly capture relevant compression in its objective.
In this work, we modify SimCLR [12], a state-of-the-an art contrastive representation method, by adding information compression using the Conditional Entropy Bottleneck (CEB) [27]. Similarly, we show how BYOL (30 representations can also be compressed using CEB. By using CEB we are able to measure and control the amount of information compression in the learned representation [26], and observe its impact on downstream tasks. We empirically demonstrate that our compressive variants of SimCLR and BOL, which we name C-SimCLR and C-BYOL, significantly improve accuracy and robustness to domain shifts across a number of scenarios. Our primary contributions are:
Reformulations of SimCLR and BOOL such that they are compatible with information-theoreti compression using the Conditional Entropy Bottleneck [26].
An exploration of the relationship between Lipschitz continuity, SimCLR, and CEB compression, as well as a simple, tractable lower bound on the Lipschitz constant. This provides an alternative explanation, in addition the cngrmatiion-hhoretiii view 226 27 22 for why CEB compression improves SimCLR model robustness.
Extensive experiments supporting our hypothesis that adding compression to the state-of-thh self-supervised representation methods like SimCLR and BYOL can significantly improve their performance and robustness to domain shifts across multiple datasets. In particular, linear evaluation accuracies of C-BYOL are even competitive with the supervised baselines considered by SimCLR [12] and BYOL [30]. C-BYOL reaches 76.0% and 78.8% with ResNet-50 and ResNet-50 2x respectively, whereas the corresponding supervised baselines are 76.5% and 77. 77.8% respectively.
2 Methods
In this section, we describe the components that allow us to make distributional, compressible versions of SimCLR and BYOL. This involves switching to the Conditional Entropy Bottleneck (CEB) objective, noting that the von Mises-Fisher distribution is the exponential family distribution that corresponds to the cosine similarity loss function used by SimCLR and BYOL, and carefully identifying the random variables and the variational distributions needed for CEB in SimCLR and BYO.. We also note that SimCLR and CEB together encourage learning models with a smaller Lipschitz constant, although they do not explicitly enforce that the Lipschitz constant be small.
2.1 The Conditional Entropy Bottleneck
In order to test our hypothesis that compression can improve visual representation quality, we need to be able to measure and control the amount of compression in our visual representations. To achieve this, we use the Conditional Entropy Bottleneck (CEB) 26), an objective function the Information Bottleneck (IB) [66] family.
Given an observation X, a target Y, and learned representation Z of X, CEB can be written as:
CF. = min 3T(X; 2 ?(Y;
(1)
2nn min ((H(()((())))))))))))))))))))))) -()))) H(Z) + H(Z))) H(YI)
(2)
2
min ((H()((-(ZZ H(ZYY) + H(YIZ)
(3)
where H(-) and (-I--) denote entropy and conditional entropy respectively. We can drop the H(Y term because it is constant with respect to Z. T(Y: 2) is the useful information relevant to the task, or the prediction target Y. T(X; ZY) the residual information Z captures about X when we already know Y, which we aim to minimize. Compression strength increases as 3 increases.

---

We define e(2|z) as the true encoder distribution, where is sampled from; b(zly), variational approximation conditioned on y; d(y]z), the decoder distribution (also variational approximation) which predicts y conditioned on z. As shown in 126] CEB can be variationally upper-bounded:
min =CEB <(=|x) <(c)))))(()l),d((()3))
E
(1log e(2)x) - log b(=l)) - log d(ylz)
There is requirement that all three distributions have learned parameters. At one limit, model's parameters can be restricted to any one the three distributions; at the other limit, all three distribu- tions could have learned parameters. If e(-) has learned parameters, its distributional form may be restricted, as we must be able to take gradients through the 2 samples 2 The only requirement on the b(-) and d()) distributions is that be able take gradients through their log probability functions.
As shown in [26]. besides parameterizing d(ylz), it is possible to reuse b(zly) to make a variational bound on the H(Y (Z) term. As I(Y; Z) = (Y) H(Y Z) and H (Y) is constant with respect to Z:
InfoNCE.
HY S E..y-p((ss) log b(2lg)
(5)
b(2ly))
where K is the number of examples in a minibatch. Eq. (5) is also known as the contrastive InfoNCE bound [54 [6]. The inner term,
b( y)
d d(ul=)
1 b(2ly))
valid variational approximation of the true but unknown p(y]2). Fischer (26) calls Eq. O the CatGen decoder because it is a categorical distribution over the minibatch that approximates the generative decoder distribution.
2.2 C-SimCLR: Compressed SimCLR
The InfoNCE bound [54] enables many contrastive visual representation methods to use it to capture
shared context between different views of an image a self-supervised objective [12 13 33 1S 40)
In this work, we show how to compress the SimCLR [2] model, but the method we discuss generally applicable to other InfoNCE-based models.
SimCLR applies randomized augmentations to image create two different views, æ and y (which we also refer to as æ'), and encodes both of them with a shared encoder, producing representations Tz and ry. Both Ta and ry are Ig-normalized. The SimCLR version of the InfoNCE objective is:
LNCE(Ta:Ty) = = log
ΣK
where r a temperature term and is the number of views in minibatch, SimCLR further makes its InfoNCE objective bidirectional, such that the final objective becomes
drTry
e
LNCE(Ta, + L = log
log
(8)
Σ
k=
We can observe the following: exp( (Tp( r2) in Eq. 7 corresponds to the unnormalized b(2ly) in Eq. D). e(-1z) generates Tz. whilst Ty and ryu are distribution parameters of b(-ly) and b(-lye) respectively. e(-(2r) and b(-ly) share model parameters.
von Mises-Fisher Distributional Representations. The cosine-similarrty-biee -hbnil loss (Eq. (7) is commonly used in contrastive learning and can be connected to choosing the von Mises-Fisher (vMF) distribution for e(-Iz) and b(-1g) [32 [3)] vMF is a distribution on the (n - -)-dimensionnll
hyper-sphere. The probability density function is given by fn, (2,,,, μ, x) (x) where μ and i. are denoted as the mean direction and concentration parameter respectively. We assume k is
For example, e(2|2) could not generally be mixture distribution, sampling the mixture distribution has discrete component, and cannot easily take gradients through discrete samples.

---

sample
x
hx Fx
e(-1a)
Zx
Conv
Projection
CEB
y hy
Fyy
DC bC-12)
Zzy
Figure 1: C-SimCLR explicitly defines encoder distributions e(-[z) and b(-l)) where x and are two augmented views of an image. is also referred as z'. The upper and lower encoder outputs are used to specify mean directions of e and 6, and the two encoders share parameters. Yay fy are lz-normalized. Our modifications to SimCLR are highlighted in blue. No new parameters are added.
constant. The normalization term C. (k) is function of and equal to (2m) In)) where I denotes the modified Bessel function of the first kind at order v.
xn/2-1
By setting the mean direction μ to Ty. concentration K2 of b(-ly) to 1/T, and rz 2, connect the SimCLR objective (Eq. (7)) to the distributional form of InfoNCE (Eq. (5))
Cn(n)ee
b(rzly)
Σ FF
Σ (R
k= fn Rb)
Zi=11 b(ralyn)
Fx is a deterministic unit-length vector, which is equivalent to a vMF with rz as the mean direction and Ke - co. We can further extend the forward encoder to have non-infinite Ke, which results in stochastic These allow to have SimCLR a distributional form with explicit distributions e(-[z) and b(-ly) and satisfy the requirements of CEB discussed in Sec. 2.1
view e(-lx) as spherical delta distribution.
Compressing SimCLR with Bidirectional CEB. Figure illlustrates the Compressed SimCLR (C-SimCLR) model. The model learns compressed representation of an view X that only preserves information relevant to predicting different view Y by switching to CEB. As can be seen in Eq. (3)
the CEB objective treats X and Y asymmetrically. However, shown in (26), it is possible to learn a single representation of both and Y by having the forward and backward encoders act as variational approximations of each other:
Bridir = min 8x T(X; Z - T(Y; 2) By; - T(X;
(10)
Z
min Bx (x((- (2 + H(Y
(11)
2
H(
min (-)))----)..(-)))(
Ex Exy--ta.
(12)
E B (m) [xx (log c(22lz) - log b(zn b)) log d(vl2z)]
E. -e By (log(e(z,)) (y) log(b(zylz)) log c(x)zz))) [
where d(-l-) and c(-l-) are the InfoNCE variational distributions of b( and e(-l-) respectively. and b use the same encoder to parameterize mean direction in SimCLR setting. Since SimCLR is trained with bidirectional InfoNCE objective, Eq. (12) gives an easy way compress its learned representation. As in SimCLR, the deterministic hrr (in Fig. T is still the representation used on downstream classification tasks.
2.3 C-BYOL: Compressed BYOL
In this section we will describe how to modify BYOL to make compatible with CEB, as summarized in Fig. [ BYOL [30] learns an online encoder that takes an augmented view of given image, as input and predicts outputs of a target encoder which encodes x', different augmented view of the same image. The target encoder's parameters are updated not by gradients but an exponential moving average of the online encoder's 's parameters. The loss function is simply the mean square error, which is equivalent the cosine similarity between the online encoder output Me and the target encoder output y as both Me and y are lz-oormalized:
Loyol = life 2'112 = p. pe + o -2p = 2 2- 2-2p
(13)

---

h
r
g(-)
e(+1x)
Z
I(.)
dc
Conv
Projection
sample
CEB
Target ene.
h
Ft
y y=sy(r))
peb
D(-11)
x
hi'
y'=sg(ri)
Figure 2: C-BYOL, The upper online encoder path takes an augmented view I as input and produces e(-(ar) and d(.1z). The lower two paths use the same target encoder (shaded), which is moving average of the online encoder (Conv + Projection). The target encoder maps x and another view x' to and ri- sg(rt) (sg: stop gradients) is our target y- y leads to b(-ly). sg(r;) is our perturbed target y'. rts I's Mex (bb D are lz-normalized. These yield the components required by CEB. We highlight changes to BYOL in blue.
This iterative "latent bootstrapping" allows BYOL learn a iiea-invariant representation. In contrast to SimCLR, BYOL does not rely on other samples in a batch and does not optimize the InfoNCE bound. It is simple regression task: given input x. predict y'. To make BYOL. CEB-compatible, we need to identify the random variables X, Y. Z, define encoder distributions e(z[æ) and b(zly), and define the decoder distribution d(y)
(see Equation
We define e(z(z) to be a vMF distribution parameterized by Me. and sample from e(z|z):
e(z|z) Cn (Ne))e""
(14)
We use the target encoder encode and output rt, an l2- ng-normalized vector. We choose r. be y. We then add 2-layer MLP on top of and lg-normalize the output, which gives M4g We denote this transformation as μa = m(y) and define b(=ly) to be the following vMF parameterized by ps:
b(lly) Cn (Na))e us
(15)
For d(ylz), add linear transformation on with L2 normalaiizniinn =(2). and define vMF parameterized by ÿ:
d(yl=) = Cx (Ma)e
(16)
In the deterministic case where = is sampled, this corresponds adding linear layer with lz-normalization on Pe which does not change the model capacity and empirical performance.
any stochastic function of Z generate Y. In our implementation, replace the generative decoder log d(y]z) with log d(y'|z), where we use the target encoder to encode x' and output y'. Given that X + X is stochastic transformation and both X and X' go through the same the target encoder function, Y - Y is also a stochastic transformation. d(y'lz) can be considered having stochastic perturbation to d(y]z). Our eCEB objective becomes
In principle,
Lo, x') = Blog <(z|z) log b(=ly)) - log d(y'l=).
(17)
We empirically observed the best results with this design choice. d(y/lz) (z) can be directly connected the standard B BOL regression objective: When Ka 2, log (1 (1/(=)) = -ka "T (o.((,.(Ra))
is equivalent to Eq. [3) when constants are ignored.
Although it seems that additionally apply the target encoder x compared BYOL, this does not increase the computational cost practice. As in BYOL, the learning objective applied symmetrically in our implementation: Lebyot (zz, x²) Labyot (x', æ). Therefore, the target encoder has to be applied to both æ and x' no matter in BYOL or C-BYOL. Finally, note that like in B BYO,, (Fig.) is the deterministic representation used for downstream tasks.
2.4 Lipschitz Continuity and Compression
Lipschitz continuity provides a way of measuring how smooth function is. For some function f and distance measure f(2))) Lipschitz continuity defines upper bound how quickly can change as x changes:
A|| 2 D(f(z), Ar)),
(18)
5

---

where L is the Lipschitz constant, Ar is the vector change in æ, and ||a||| > 0. If we define f(z) to be our encoder distribution e(2 ()) (which is a vMF and always positive), and the distance measure, D, to be the absolute difference of the logs of the functions, we get function of of Lipschitz value, such that:
I(=)
loge(2|a) log c(2/2 + Az)
(19)
A
As detailed in Sec. by taking expectations with respect to can obtain lower bound on the encoder distribution's s squared Lipschitz constant
L
| max (KL(e())s)lle((t< Ar)], KL[e(z/r + [o)
(20)
To guarantee smoothness of the encoder distribution, we would like to have upper bound on L, rather than a lower bound. Minimizing a lower bound does not directly yield any optimality guarantees relative to the bounded quantity. However, in this case, minimizing the symmetric KL below is consistent with learning a smoother encoder function:
inf (L)e + Ax)] + KL|e(=|æ + Az)]le(2(=)) e(a)-)
(21)
By consistent, we mean that, if we could minimize this symmetric KI. at every pair (I, x + Ar) in the input domain, we would have smoothed the model. In practice, for high-dimensional input domains, that is not possible, but minimizing Eq. (2I) at subset of the input domain still improves the model's smoothness, at least at that subset.
The minimization in Eq. 21) corresponds almost exactly to the CEB compression term in the bidirectional SimCLR models. We define y Az. At samples of the augmented observed variables, X, Y. the C-SimLLR models minimize upper bounds on the two residual informations:
(x; Z|Y) + (Y; S Ex,y<p(a..) KL(e(2) (t)e(=//)]]((((]])) + KL(e(\/))le(((/o)
(22)
The only caveat to this is that we use b(zly) instead of e(2ly) in C-SimCLR. b and e share weights but have different K values in their vMF distributions. However, these are hyperparameters, so they are not part of the trained model parameters. They simply change the minimum attainable KLs in thereby adjusting the minimum achievable Lipschitz constant for the models (see Sec. G
Eq. (22
Directly minimizing Equation (20) would require normalizing the symmetric KL per-example by || The symmetric CEB loss does not do this. However, the residual information terms in Equation (22) are multiplied by a hyperparameter 3 < 1. Under simplifying assumption that the ||Aa||| values generated by the sampling procedure are typically of similar magnitude, we can extract the average || into the hyperparameter 33. We note that in practice using per-example values of
||Ax|| would encourage the optimization process to smooth the model more strongly at observed (z. Ar) pairs where it is least smooth, but we leave such experiments to future work.
should expect that the C-SimCL.R models are locally more smooth around the observed data points. We reiterate, though, that this is not a proof of increased global Lipschitz smoothness,
Due to Eq. [22)
are minimizing lower bound the Lipschitz constant, rather than minimizing an upper bound. It is still theoretically possible to learn highly non-smooth functions using CEB in this manner. It would be surprising, however, if the C-SimCLR were somehow less smooth than the corresponding SimCLR models.
The Lipschitz continuity property is closely related to model robustness to perturbations 99. including robustness to adversarial examples [71] 23 73). Therefore, we would expect to see that the C-SimLLR models are more robust than SimCLR models common robustness benchmarks. It is more difficult to make the same theoretical argument for the C-BYOL models, as they do not use exactly the same encoder for both and y. Thus, the equivalent conditional information terms from Eq. (22) not directly minimizing a lower bound on the Lipschitz. constant of the encoder. Nevertheless, we empirically explore the impact of CEB on both SimCLR and BYOL models next Sec.
"Note that by taking an expectation we get a KL divergence, which violates the triangle inequality, even though we started from a valid distance metric. Squaring the Lipschitz constant addresses this in the common case where the KL divergence grows quadratically in [Ar]], as detailed in Section G

---

3
Experimental Evaluation
We first describe our experimental set-up in Sec. 3.1 before evaluating the image representations learned by our self-supervised models in linear evaluation settings in Sec. 3.2 We then analyse the robustness and generalization of our self-supervised representations by evaluating model accuracy across a wide range of domain and distributional shifts in Sec. 3.3 Finally, we analyse the effect of compression strength in Sec. 3.4 Additional experiments and ablations can be found in the Appendix.
3.1 Experimental Set-up
Implementation details. Our implementation of SimCLR, BYOL, and their compressed versions is based off of the public implementation of SimCLR (12). Our implementation consistently reproduces BYOL results from [30] and outperforms the original SimCLR, as detailed in Sec.
We use the same set of image augmentations as in BYL 30 for both BYOL and SimCLR, and also use BYOL's (4096, 256) two-layer projection head for both methods. We follow SimCLR and BYOL to use the LARS optimizer [74] with a cosine decay learning rate schedule 49] over 1000 epochs with warm-up period, detailed in Sec. A.4 For ablation experiments we train for 300 epochs instead. As in SimCLR and BYOL, we use batch size of 4096 split over 64 Cloud TPU v3 cores. Except for ablation studies of compression strength, /3 is set to 1.0 for both C-SimCLR and C-BYOL. We follow SimCLR and BYO their hyperparameter choices unless otherwise stated, and provide exhaustive details in Sec. A Pseudocode can be found in Sec. H
Evaluation protocol. We assess the performance of representations pretrained on the ImageNet training set (60) without using any labels. Then we train a linear classifier on different labeled datasets on top of the frozen representation. The final performance metric is the accuracy of these classifiers. As our approach builds on SimCLR [13 and BOO 330, we follow the same evaluation protocols. Further details are in Sec. B
3.2 Linear Evaluation of Self-supervised Representations
Linear evaluation on ImageNet. We first evaluate the representations learned by our models by training a linear classifier top of frozen features the ImageNet training set, following standard practice 12
As shown in Table our compressed objectives provide strong improvements state-of-ie of-te-of-the-ar the SimCLR [2 and BYOL [30 models across different ResNet architectures 34 of varying widths (and thus number of parameters) [75]. Our reproduction of the SimCLR baseline (70.7% top-1 accuracy) outperforms that of the original paper (69.3%). Our implementation of BOI,, which obtains mean Top-1 accuracy of 74.2% (averaged over three trials)
30 43 44].
matches that of 30 within a standard deviation.
Table shows that improvements remain consistent for both 300 epochs, and the longer 1000 epoch schedule which achieves the best results. In addition to the Top-1 and Top-5 accuracies, we also compute the Brier score [8] which measures model calibration. Similar to the predictive accuracy, observe that our compressed models obtain consistent improvements.
Current self-supervised methods benefit from longer training schedules 12 1
15
33
Learning with a few labels on ImageNet. After self-supervised pretraining on ImageNet, we learn linear classifier on small subset (1% or 10%) of the ImageNet training set, using the class labels this time, following the standard protocol of 122 30 We expect that with strong feature representations, we should be able to learn an effective classifier with limited training examples.
Table 2 shows that the compressed models once again outperform the SimCLR and BYOL coun- terparts. The largest improvements are observed in the low-data regime, where we improve upon the state-of-the-art BYO by 5.1% % and SimCLR by 1.8%, when using only 1% of the ImageNet labels. Moreover, note that self-supervised representations sigmificantly outperform fully-supervised ResNet-50 baseline which overfits significantly in this low-data scenario.
Comparison to other methods. Table 3 compares C-SimCLR and C-BYOL to other recent self- supervised methods from the literature (in the standard setting using two augmented views) on ImageNet linear evaluation accuracy. We present accuracy for models trained for 800 and 1000
7

---

Table 1:
ImageNet accuracy of linear classifiers trained on representations learned with SimCLR [2] and BYOL [30]. with and without CEB compression. A lower Brier score corresponds to better model calibration. We report mean accuracy and standard deviations over three trials.
SimCLR Top-5
BYOL Top-5
Method
Top-1
Brier
Top-1
Brier
ResNet-50, 300 epochs
Uncompressed
69.110.089
89.110.034
42.1111.66
12.810.155
91.010.072
37.3110.899
Compressed 70.110.177
89.610.099
01.0:20.077
73.6.10.039
10.500
30.540
ResNet-50, 1000 epocks Uncompressed Compressed
70.7:10094
10.100.081
40.0= 123
03.
91.74:0.441
35.7
71.640.084
90.5+0..67
39.7=0.877
0..610.151
92.7+00076
34.0+0.127
ResNet-50 2x, 1000 epocks
Uncompressed Compressed 75.0=0.082
74.540.014
92. 10.031
35.2=0.038
77.2+0.057
93.5+0.036
31.820.073
92.410.086
34.7=0.129
78.8+0.088
94.5+0.016
29.8:0.028
Table 2: ImageNet accuracy when training linear classifiers with 1% and 10% of the labeled Ima- geNet data, on top of frozen, self-supervised repre- sentations learned on the entire ImageNet dataset without labels. For the supervised baseline, the whole ResNet-50 network is trained from random initialisation. We report mean results of 3 trials.
Table 3: Comparison to other methods on ImageNet linear evaluation and supervised baselines. trained for 800 epochs. Other methods 1000 epochs.
ResNet-50
Method
1x
2x
SimCLR 12
69.3
74.2
Top-1
Top-5
SwAV (2 crop) 11
15
71.8
Method
1%
10%
1%
10%
InfoMin Aug 65
73.0
Supervised
77
25.4
56.4
48.4
80.4
Barlow Twins
73.2
BYOL 30
74.3
77.4
SimCLR
49.3
63.3
75.8
85.9
C-SimLLR
51.1
64,5
77.2
86.5
C-SimCLR (ours)
71.6
75.0
C-BYOL (ours)
75.6
78.8
BYOL
55.5
68.2
79.7
88.4
C-BYOL
60.6
70.5
83.4
90.0
Supervised
12
30
76.5
77.8
epochs, depending on what the original authors reported. C-BYOL achieves the best results compared to other state-of-the-art methods. Moreover, we can improve C-BYOL with ResNet-50 even further 76.0 Top-1 accuracy when train it for 1500 epochs.
Comparison to supervised baselines. As shown in Table 3, SimCLR and BYO use supervised baselines of 76.5% for ResNet-50 and 77.8% for ResNet-50 2x [12 30 respectively. In comparison, the corresponding compressed BYOL models achieve 76.0% for ResNet-50 and 78.8% for ResNet-50 2x, effectively matching or surpassing reasonable supervised baselines
The results Tables to support our hypothesis that compression of SSL techniques can improve their ability generalize in variety of settings. These results are consistent with theoretical understandings of the relationship between compression and generalization [61 the results in Table 5 that show that performance improves with increasingly strong compression (corresponding to higher values of (3), up to some maximum amount of compression, after which performance degrades again.
68 22]. as are
3.3 Evaluation of Model Robustness and Generalization
In this section, we analyse the robustness of our models to various domain shifts. Concretely, we use the models, with their linear classifier, from the previous experiment, and evaluate them suite
"We note that comparing supervised and sulf--upervvied methods is difficult, it only be system-wise. Various complementary techniques can used to further improve evaluation results both settings. For example, the appendix of (30) reports that various techniques improve supervised model accuracies, whilst [30 [3] report various techniques to improve evaluation accuracy of self supervied representations. We omit these in order to follow the common supervised baselines and standard evaluation protocols used in prior work.

---

Table Evaluation of robustness and generalization of self supervveee models, using a ResNet-50 backbone trained on ImageNet for 1000 epochs. We report the mean Top-1 accuracy over trials on a range of benchmarks (detailed in Sec. 3.3 and the appendix) which measure an ImageNet-trained model's generalization to different domains and distributions.
Method
ImageNet-A
ImageNet-C
ImageNet-R
ImageNet-v2
ImageNot-Vid
YouTube-BB
ObjectNet
SimCLR
57.7
18.7
C-SimCLR
1.4
36.8
19.6
58.7
64.7
59.5
20.8
RYOL
42.7
24.4
62.1
67.9
60.7
23.4
C-BYOL
2.3
45.1
25.8
63.9
70.8
63.6
25.5
Table 5: Ablation study CEB compression using C-SimCLR models trained for 300 epochs with a ResNet-50 backbone. / controls the level of CEB compression. We evaluate linear-evaluation ImageNet, and model robustness on the remaining datasets as described in Sec, 3.3
Method
ImageNet
ImageNet-A
ImageNer-C
ImageNet-R
ImageNet.v2
ImageNer-Vas Vid
YouTube DB
ObjectNet
SimCLR
69.0
1.2
32.9
17.8
$6.0
61.1
58.3
17.6
69.7
17.6
62.5
58.4
18.5
0.01
69.7
1.2
36.2
17.5
57.2
61.2
58.5
18.7
0.1
70.1
1.1
36.1
17.6
56.9
62.4
58.6
18.4
70.2
1.1
36.7
18.2
57.5
62.6
60.4
19.2
= 1.5
69.7
1.1
36.4
18.3
57.3
62.0
57.9
18.5
of robustness benchmarks that have the same label set the ImageNet dataset. We the public robustness benchmark evaluation code of [19 18]. As a result, we can evaluate our network and report Top-1 accuracy, shown in Table without any modifications to the network.
consider "natural adversarial examples" with ImageNet-A 37 which consists of difficult images which ResNet-50 classifier failed on. ImageNet-C [37] adds synthetic corruptions to the ImageNet validation set, ImageNet-R [36] considers other naturally occuring distribution changes in image style while ObjectNett [5] presents more difficult test set for ImageNet where the authors control for different parameters such as viewpoint and background. ImageNet- Vid and YouTube-BB [62] evaluate the robustness of image classifiers natural perturbations arising in video. Finally, ImageNet- v2 (58) a new validation set for ImageNet where the authors attempted to replicate the original data collection process. Further details of these robustness benchmarks in Section F
Table shows that SimCLR and BYOL models trained with CEB compression consistently out- perform their uncompressed counterparts across all seven robustness benchmarks. This is what we hypothesized in the SimCLR settings based on the Lipschitz contimuity argument in Sec. 2.4 and the appendix. All models performed poorly ImageNet-A, but this is not surprising given that ImageNet-A was collected by (37) according to images that a ResNet-50 classifier trained with full supervision ImageNet misclassified, and we evaluate with ResNet-50 models too.
3.4 The Effect of Compression Strength
BableBsstudies the effect of the CEB compression term, on linear evaluation accuracy on ImageNet, as well as on the same suite of robustness datasets. We observe that = 0, which corresponds to no explicit compression, but a stochastic representation, already achieves improvements across all datasets. Further improvements are observed by increasing compression (8), with 3 11 obtaining the best results. But overly strong compression can harmful. Large values of 3 correspond to high levels of compression, and can cause training to collapse, which observed for 3 2.
4 Related Work
Most methods for learning visual representations without additional annotation can roughly grouped into three families: generative, discriminative, and bootstrapping. Generative approaches build a latent embedding that models the data distribution, but suffer from the expensive image generation step 66]
59 28 39 42). While many early discriminative approaches used heuristic 53 multi- view contrastive methods are among the recent of-te-of-theartt [2 33
pretext tasks 20
14 13
54
35
can be connected to the information bottleneck principle [66] 67 3 but in form of unconditional compression as
Some previous contributions the multi-view contrastive family 65
76

---

they are agnostic of the prediction target, i.e. the target view in multiview contrastive learning.
As discussed in [26] 277, CEB performs conditional compression that directly optimizes for the information relevant to the task, and is shown theoretically and empirically better 26 27 A multi- view self-supervised formulation of CHB, which C-SimCLR can be linked to, was described in [26]_ Federici et al. [24] later proposed practical implementation of that, leveraging either label information or data augmentations. In comparison to [24], we apply our methods with large ResNet models to well-studied large-scale classification datasets like ImageNet and study improvements in robustness and generalization, rather than using two layer MLPs on smaller scale tasks. This shows that compression can still work using state-of-the-an art models challenging tasks. Furthermore, we use the vMF distribution rather than Gaussians in eiggh-imenssonal spaces, and extend beyond contrastive learning with C-BYOL.
Among the bootstrapping approaches 31 modified B BOL 30 to leverage Projected Gradient Descent (0
10
30
which BYOL.
30 belongs to, BYORL 29
to learn more adversarially robust encoder. The focus is, however, different from ours as we concentrate on improving the generalization gap and robustness to domain shifts.
variety of theoretical work has established that compressed representations yield improved gen- eralization, including [6]]
68 22). Our work demonstrates that these results are valid in practice, for important problems like ImageNet, even in the setting of self-supervised learning. Our theo- retical analysis linking Lipschitz continuity to compression also gives a different way of viewing the relationship between compression and generalization, since smoother models have been found to generalize better (e.g., ()). Smoothness is particularly important in the adversarial robustness setting 71 23 73. although we do not study that setting in this work.
5
Conclusion
We introduced compressed versions of two state-of-the-art self-supervised algorithms, SimCLR 12 and BYOL [30], using the Conditional Entropy Bottleneck (CEB) (27). Our extensive experiments verified our hypothesis that compressing the information content of self-supervised representations yields consistent improvements in both accuracy and robustness to domain shifts. These findings were consistent for both SimCLR and BYO across different network backbones, datasets and training schedules. Furthermore, we presented an alternative theoretical explanation of why C-SimCLR models are more robust, in addition to the information-theoreis view 226 27 21, by connecting Lipschitz continuity to compression.
Limitations. We note that using CEB often requires explicit and restricted distributions. This adds certain constraints on modeling choices. It also requires additional effort to identify create required random variables, and find appropriate distributions for them. Although we did not need additional trainable parameters for C-SimCLR, we did for C-BYOL, where we added a linear layer to the online encoder, and a 2-layer MLP to create b(-ly). It was, however, not difficult to observe the von Mises-Fisher distribution corresponds loss function of BYL and SimCLR, well as other recent InfoNCE-based contrastive methods
33
Potential Negative Societal Impact. Our work presents self-supervised methods for learning effective and robust visual representations. These representations enable learning visual classifiers with limited data (as shown by our experiments on ImageNet with 1% or 10% training data), and thus facilitates applications in many domains where annotations expensive or difficult to collect.
Image classification systems are generic technology with a wide range of potential applications. We are unaware of all potential applications, but are cognizant that each application has its own merits and societal impacts depending on the intentions of the individuals building and using the system. We also note that training datasets contain biases that may render models trained on them unsuitable for certain applications. is possible that people use classification models (intentionally or not) to make decisions that impact different groups in society differently.
Acknowledgments and Disclosure of F Funding
We thank Toby Boyd for his help in making our implementation open source.
10

---

John Canny is also affiliated with the University of California, Berkeley.
References
[1] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):1947-1980, 2018.
[2] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, (M(12):2897-2900) 2018.
[3] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
[4] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in Neural Information Processing Systems, 15555-15555, 2019.
[5] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in Neural Information Processing Systems, 32:9453-9468 2019.
[6] Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2018,, 2014.
[7] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In European conference computer vision, pages 461. Springer, 2014.
[8] Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
[9] Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-15 1886, 2013.
[10] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference Computer Vision (ECCV), pages 132-149, 2018
[11] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[13] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33:22243-2223 2020.
[14] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003. 04497, 2020.
[15) Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[16] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.

---

[17] Joshua V Dillon, Ian Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffman, and Rif A Saurous. Tensorflow distributions, arXiv preprini arXiv: 10604, 2017.
[18] Josip Djolonga, Frances Hubis, Matthias Minderer, Zachary Nado, Jeremy Nixon, Rob Romi- jnders, Dustin Tran, and Mario Lucie. Robustness Metrics, 2020. https://githhb. com/
[19] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexan- der Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. arXiv preprint arXiv:2007.08558, 2020.
[20] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
[21] Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, and Chris J Maddison. Lossy compression for lossless prediction. arXiv preprini urXiv:2106. 10800, 2021.
[22] Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18674-18690. Curran Associates, Inc., 2020.
[23] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George J Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. NeurIPS, 2019.
[24] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi view information bottleneck. arXiv preprint arXiv:2002.07011, 2020.
[25] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pullern recognition workshop, pages 178-178. IEEE, 2004.
[26] Ian Fischer. The conditional entropy bottleneck. Entropy, 22(9), 2020.
[27] Ian Fischer and Alexander A. Alemi. Ceb improves model robustness. Entropy, 22(10), 2020.
[28] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[29] Sven Gowal, Po-Sen Huang, Aaron van den Oord, Timothy Mann, and Pushmeet Kohli. Self- supervised adversarial robustness for the low-label, high-data regime. In International Confer- ence on Learning Representations, 2021.
[30] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33. pages 21271-2111 2020.
[31] Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean- Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap pareint-re representations for multitask reinforcement learning. In International Conference on Machine Learning, pages 3875-3886. PMLR, 2020.
[32] Md Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, and Liming Chen. von mises- fisher mixture model-based deep learning: Application to face verification. arXiv preprint 17Xiv:1706.04260 2017.
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9732 2020.
12

---

[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[35] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In Interna- tional Conference on Machine Learning, pages 4182 4192. PMLR, 2020.
[36] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006. /6241, 2020.
[37] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
[38] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
[39] Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 2006.
[40] R Devon Hjelm, Alex Fedorov, Samuel Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: a 06670, 2018.
Karan Grewal, Phil Bachman,
[41] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 188-456. PMLR, 2015.
[42] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 13266113 2013.
[43] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1920-1929, 2019.
[44] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2661-2671, 2019.
[45] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. In The 2nd Fine- Grained Visual Categorization Workshop, 2013.
[46] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[47] Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in RL. Advances in Neural Information Processing Systems, 33:11890-11901, 2020.
[48] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021.
[49] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 1608. 03983, 2016
[50] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 11f.:1170000088 2017.
[51] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 1rXiv:130..5151, 2013.
[52] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
13

---

[53] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
[54] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint Ian::0374700 2018.
[55] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[56] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[57] Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large high-precision CVPR, 2017.
data set for object detection in video.
[58] Benjamin Recht, Rehecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? ICMI,, 2019.
[59] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and variational inference in deep latent gaussian models. In International Conference on Machine Learning, 2014.
[60] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
[61] Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the informa- tion bottleneck. Yoav Freund, László Györfi, György Turán, and Thomas Zeugmann, editors, Algorithmic Learning Theory, pages 92-107. Springer Berlin Heidelberg, 2008
[62] Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? arXiv preprint arXiv:1906.02/66 2019.
[63] Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In Proceedings of the 21 Annual Conference Learning Theory, pages 403-414, 414, 2008.
[64] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
[65] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
[66] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprini physics/0004057, 2000.
[67] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
[68] Matías Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of information complexity and randomization in representation learning. arXiv preprint 100.00180..05355, 2018.
[69] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-11000 1103, 2008
Manzagol. Extracting and
[70] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning, pages 9929-9993999 PMLR, 2020.
14

---

[71] Tsui- Wei Weng, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustmess of neural networks: An extreme value theory approach. In International Conference on Learning Representations, 2018.
[72] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scale scene recognition from abbey to zoo. In 2010 JEEE computer society conference on computer vision and pattern recognition, pages 3485- 3492. IEEE, 2010.
[73] Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. Advances in Neural Information Processing Systems, 33, 2020.
[74] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 108.::7088,3888, 2017.
[75] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1605v01605.077 07/ (46, 2016.
[76] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.
[77] Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4L: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1476-1485, 2019.
15


Reviews:
[{"review_id": "uHAgInvstyh", "reviewer": "Reviewer_1tWc", "summary": "This work adds a conditional entropy bottleneck (CEB) objective to two existing frameworks for self-supervised learning, namely SimCLR and BYOL. The motivation for adding CEB is to encourage information compression in the representations learned from these models which can make the representations more robust. The authors measure the performance of SimCLR and BYOL with added CEB objective empirically on standard benchmarks such as linear evaluation protocol on ImageNet with different amounts of training data. They provide evidence that adding CEB outperforms other existing self-supervised learning models on these benchmarks, including vanilla SimCLR and BYOL. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- While the information theoretic CEB objective is not novel, to the best of my knowledge, incorporating it into SimCLR and BYOL has not been done before and is a well-motivated idea. This approach can be applied in other settings. \n- The experimental setup is exhaustive. Мodels with the proposed modified objective are evaluated on popular benchmarks (including the standard ImageNet linear classification protocol and linear classification on versions of ImageNet designed to evaluate robustness and generalization) and show strong empirical performance when compared to other existing methods. \n- The paper provides a link between Lipschitz continuity and CEB compression in the case of SimCLR.\n- This is a well written paper and it is organized clearly. \n\n###Questions & Comments\n1. How many seeds were used for results in Tables 2-5? What was the standard deviation?\n2. What is the full range of values that were used for $\\beta$? \n3. I think it would be beneficial for the main text to clearly indicate the best values for $\\beta$ for both SimCLR and BYOL in all the different experiments.\n\n###Nitpicks\n- Equation (6): $z_x$ and $z_y$ should be replaced by $r_x$ and $r_y$, the sum in the denominator should be over lower case $k$ (not capital $K$)\n- Line 155: in “distribution of ||z||” should use LaTeX, $\\||z\\||$\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "As68sE9-b1D", "reviewer": "Reviewer_ZEAE", "summary": "The paper shows how to use the Conditional Entropy Bottleneck (CEB)[1] in a self-supervised fashion.  Thus resulting in a new method for training approximately minimal self-supervised  (SSL) representations, i.e., a new member of the SSL information bottleneck family (although extremely similar to [2]). Specifically, they show how to use CEB in conjunction with SimCLR and BYOL. Furthermore, they provide good arguments to suggest that their proposed objective will learn smoother encoders than standard self-supervised models, which can help robustness. Through extensive experiments,  they show significant performance gains of the proposed methods compared to SimCLR and BYOL.\n\n- [1] Fischer, Ian. \"The conditional entropy bottleneck.\" Entropy 22.9 (2020): 999.\n- [2] Federici, Marco, et al. \"Learning robust representations via multi-view information bottleneck.\" arXiv preprint arXiv:2002.07017 (2020).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Post-rebuttal**: I've updated my score given that the authors (seem to) have addressed all my major concerns (related work / mathematical mistakes / practical details) in the updated manuscript.\n\n----------\n\n**Overall**: This paper has interesting ideas with strong experimental results. It will likely be of interest to the SSL and IB sub-communities. I nevertheless do not think that the current version is ready for publication given that (1) it fails to discuss very related work; (2) has mathematical mistakes; (3) it's very hard to understand what authors do in practice. Fortunately, all these problems can be easily solved and I am happy to increase my score if the authors make the necessary changes. \n\n**Strengths**:\n- **Significant empirical gains and thorough experimental section**. The experimental results show significant gains against very strong baselines. These results are even more impressive given the thorough evaluation protocol. \n- **Interest to the community**: The link between the proposed CEB and smoothness of the learned objective is interesting and to my knowledge novel. Such a smoothness argument might be useful beyond robustness. Furthermore, although SSL+IB has already been considered (see \"related work\" below), the final objective used with vMF variational family is novel and has many potential advantages compared to previous work (which I hope the authors will discuss in the final version).\n- **Thorough appendices**: The supplemental material is thorough and very useful.\n\n**Weaknesses**:\n- **Missing important related work**: The discussion of related work is insufficient. Most importantly, multi-view IB [1] provides a loss that is extremely similar to your C-simlcr objective (the only real difference seems to be the variational family) and provides experiments in robustness and standard SSL just as you. More generally, the paper reads as if it is the first (although the authors do not explicitly say so)  to investigate the use of \"compressed\"/\"low information\" representations in SSL, which is definitely not the case. The paper also fails to cite previous work that uses/notice the link between von Mises-Fisher distribution and contrastive learning with normalized representations. This definitely diminishes the novelty of the paper, but I do not find that to be an issue as long as it is correctly discussed. Authors should take advantage of their \"related work section\" to discuss the advantages/limits of the proposed extension of CEB compared to previous work. More details can be found in my \"related work\" section below.\n- **Some mathematical mistakes/imprecisions** currently there are many mathematical issues that obscure some parts/equations of the paper. The biggest issues are the following (see \"suggestions\" for other issues):\n     -  Equation (5) has many mistakes, to the point that I don't even know what you were trying to show and how many mistakes there are. The mistakes that are obvious are: (1) what do you mean by lower bounding a conditional entropy by a distribution? surely you meant $E[-\\log d(y|z)]$; (2) you say that you drop $H[Y]$ because that's a constant, but that's only true if you have some minimization, do you mean taking min on both sides? (3) you say that this is the InfoNCE bound but you are removing $\\frac{1}{k}$ from the denominator, is that on purpose? If so say that it's InfoNCE up to $log K$. I actually think there are deeper issues, but I'm not even sure what you aiming for due to the previous issues. Here's what I think you meant by using InfoNCE bound: $I[Z,Y] \\geq I_{NCE}$ so $H[Y] - I[Z,Y] = H[Y|Z]  \\leq H[Y] - I_{NCE}$ so $\\inf_{e(z|x)} H[Y|Z]  \\leq \\inf_{e(z|x)} - I_{NCE} =  \\inf_{e(z|x)} - \\log K - E[ \\log d(y|z)]$ so $\\inf_{e(z|x)} H[Y|Z]  \\leq \\inf_{e(z|x)} E[- \\log d(y|z)]$. But if that is the case you are: (1) missing a negative sign; (2) used lower bound instead of an upper bound; (3) should really just use the standard cross-entropy bound instead of InfoNCE to get the same result.\n    - I think that the “distance\" measure that you are using is not an actual metric (it’s only a pseudo metric) because it does not satisfy $d(x_1,x_2) = 0 \\iff x_1=x_2$, indeed, you can have $e(z|x_2) = e(z|x_1)$ for different $x_1,x_2$.  As a result, you cannot formally talk about Lipchitz continuity and the results from the paper you cite in line 191 might not hold. I would suggest adding a footnote explaining that you are actually using a pseudometric so you are not exactly considering Lipchitz continuity but hypothesize that this provides similar robustness guarantees. I actually don’t think that it makes much difference but this has to be flagged.\n- **Hard to understand what is done in practice and missing important implementation details**: it's actually quite hard to get what is done in practice because there are many moving components and these are scattered in different places of the paper (e.g. (5) and (11) are the equations that are needed for C-SimCLR but separated by much text and equations that is not used for the model). It would help a lot to have clear algorithms for C-SimCLR and C-BYOL. Other non-trivial and important details are missing, e.g., how you sample (with gradient flows) from the vMF which is known to be very hard in high dimensions.\n\n**Related work**\n- **Multi-View Information Bottleneck**: As previously stated [1] is extremely similar to your paper on many fronts. As a result,  I think that this paper should be cited in your introduction where you should explain what you are contributing which they didn't. Furthermore, the exact difference needs to be discussed in the text. \n- **Self supervised learning + information bottleneck**:  besides [1] there are other (albeit less related) papers that consider \"compressed\"/\"low information\" representations in SSL. For example, Barlow Twins[2] can be seen from an SSL IB point of view  (see their Appx. A).  [3] provides some theory for this type of SSL+IB. [4] provides different losses for learning compressed SSL representations (although this seems to be online since March so contemporary work).  The only similar work that you cite is InfoMin Aug, which arguably is the least related.\n- **Von Mises-Fisher + cosine similarity**: the relation between vMF and contrastive loss with cosine similarity has already been pointed out/used by others, e.g., [5-7]. Please add some discussion about those or at least acknowledge some of them around line 113.\n- **Minimality and generalization**. Your experimental results clearly show that compressed / low-information representations help downstream performance. There’s a large body of work that proves/suggests that this will be the case and I think that these should be cited.   E.g., [8] proves that a minimal representation ensures a generalization gap of 0 (i.e. tight but only for optimal representations), while [9,10] provides upper bounds (although loose) on the generalization gap based on $I[Z, X]$. \n\n**Suggestions / specific issues:**\n- I would say very early on that you are conditioning on a second view instead of the label (as usual in CEB). This was not clear to me when reading until I arrived at section 2.2.\n- The use and meaning of $r_x,r_y$ vs $z_x,z_y$ is unclear. Can you please clarify around lines 102-103 how $r$ relates to $z$? From section 2.2. it seems that $r$ refers to parameters of the density of $z$ but this should be clearer much sooner. Also, in equation (6) should it be $L_{nce}(r_x,r_y)$?\n- I find section 2.3 hard to follow. I would suggest simplifying this section, although to be honest, I’m not sure how, because there are many moving components in BYOL.  \n- Section 3.2. is missing some motivation/explanation as to why compression should help in standard iid setting  (i.e. it’s missing a paragraph like at lines 271-273 for robustness). As I said in “related work” I would cite the body of work that proves [8] or suggests [9,10] that low information representations provide strong guarantees on generalization to give an intuition as to why this makes sense and why you consider the experimental setting 3.2. \n\n**Questions**\n- Right now in equation (11) it seems that you are estimating the KL of the vMF distributions using samples, have you considered using an upper bound on the KL between vMF such as in [11]? This would essentially maximize $r_x^Tr_y$ which is much simpler and already symmetric. \n- How do you sample in practice from the vMF with a reparametrization trick in high dimension? Implementations of vMF usually suffer from numerical issues as the Bessel function is very hard to evaluate in high dimensions (e.g. TensorFlow's implementation does not work for more than 5 dimensions).  E.g., [12] had to come up with a clever parametrization trick for vMF, is that what you are using?\n\n**Minor Suggestions**\n- Equation (3): this should be an equal sign instead of define\n- Line 81: say that d(y|z) is also a variational approximation\n- Line 82: bounded -> upper-bounded\n- Line 88: cannot take gradients -> cannot easily take gradients\n- Line 107: $r_{y_k}$ -> $r_y$\n- Line 107: corresponds to $b(z|y)$ ->corresponds to the unnormalized  $b(z|y)$       (or use $\\frac{1}{C}exp$instead of exp, where $C$ is normalising constant).\n- Line 114: n-dimensional -> (n-1)-dimensional     (or change pdf to n+1 in 115)\n- Line 123: “relax” makes it sound as if the resulting $I_{NCE}$ is not a proper lower bound anymore, but that’s not the case, right? If it’s still a lower bound I’d use “extend”.\n- Equation 15: I don’t think the first equality is correct: both densities have different domains and even different graphs. I think you mean that $e(z|x)=e(w \\cdot \\phi |x)$ where  $e(w ,\\phi |x) = e(w | x) \\cdot e( \\phi |x)$.\n- Line 158: “and since C is a constant.” I don’t understand what you mean here?\n- Line 161: Continuity -> continuity\n- Equation 18: say that L is maxed over L(z)\n- Line 175: isn’t -> is not \n- I would replace all $\\min$ with $\\inf$ (e.g. equation 4,9,10,11)\n\n**References:**\n- [1] Federici, Marco, et al. \"Learning robust representations via multi-view information bottleneck.\" arXiv preprint arXiv:2002.07017 (2020).\n- [2] Zbontar, Jure, et al. \"Barlow twins: self-supervised learning via redundancy reduction.\" arXiv preprint arXiv:2103.03230 (2021).\n- [3] Sridharan, Karthik, and Sham M. Kakade. \"An information-theoretic framework for multi-view learning.\" (2008): 403.\n- [4] Dubois, Yann, et al. \"Lossy Compression for Lossless Prediction.\" arXiv preprint arXiv:2106.10800 (2021).\n- [5] Wang, Tongzhou, and Phillip Isola. \"Understanding contrastive representation learning through alignment and uniformity on the hypersphere.\" International Conference on Machine Learning. PMLR, 2020.\n- [6] Hasnat, Md, et al. \"von mises-fisher mixture model-based deep learning: Application to face verification.\" arXiv preprint arXiv:1706.04264 (2017).\n- [7] Kumar, Sachin, and Yulia Tsvetkov. \"Von mises-fisher loss for training sequence to sequence models with continuous outputs.\" arXiv preprint arXiv:1812.04616 (2018).\n- [8] Dubois, Yann, et al. \"Learning optimal representations with the decodable information bottleneck.\" arXiv preprint arXiv:2009.12789 (2020).\n- [9] Shamir, Ohad, Sivan Sabato, and Naftali Tishby. \"Learning and generalization with the information bottleneck.\" International Conference on Algorithmic Learning Theory. Springer, Berlin, Heidelberg, 2008.\n- [10] Vera, Matías, Pablo Piantanida, and Leonardo Rey Vega. \"The role of information complexity and randomization in representation learning.\" arXiv preprint arXiv:1802.05355 (2018).\n- [11] Diethe, Tom. \"A Note on the Kullback-Leibler Divergence for the von Mises-Fisher distribution.\" arXiv preprint arXiv:1502.07104 (2015).\n- [12] Davidson, Tim R., et al. \"Hyperspherical variational auto-encoders.\" arXiv preprint arXiv:1804.00891 (2018).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation and societal impact are generally well discussed. Although, as previously stated, I would like to see a comparison (and thus potential limitations of the current method) compared to previous SSL + IB papers.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15 hours"}, "extra_text": {}}, {"review_id": "6pgrmA5_ivR", "reviewer": "Reviewer_ofpQ", "summary": "Contrastive learning approaches learn salient representations that will be effective for various downstream tasks, while not all learned information is relevant to the task target. Given this insight, the authors aim to learn compressed representations by adding information compression objectives to existing algorithms. \n\nThey applied conditional entropy bottleneck (CEB) objective to SimCLR and BYOL to compress away the irrelevant information from the learned representations. In addition, they discovered that enforcing the information compression entails the encoder with bounded Lipschitz constant which is closely related to robustness. Extensive experiments confirm their hypotheses that adding compression regularizer can improve the performance in terms of accuracy and robustness to domain shift.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "They present their work with reasonable motivations. Overall, I think this paper would be interesting to the NeruIPS audience.\n\nI have some technical concerns:\n- It is unclear to me what are the architectures and outputs of decoders $d(.), c(.)$. Are they supposed to predict the inputs $x, y$?\n- Is it possible to insert decoders $d(.), c(.)$ into Figure 1&2 just like you did for $e(.), b(.)$. I believe that would help people to get a better understanding of your work.\n- I like the diagrams in Figure 1&2. But I would suggest to spend a few sentences to explicitly explain the work flows. For example, I think you can make them more friendly for readers if you can explain what are the red dash lines implying. \n- Although the absolute difference is small, I would suggest to report both mean and standard deviation. For instance, in table 1, the last these accuracies are just marginally better than baselines, if std dev is available that would help review to clearly see the significance of experimental results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "the authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "4lMR1enHKg", "reviewer": "Reviewer_GtXm", "summary": "The paper posits that compressing the representation during learning improves the robustness of said representation. To demonstrate their argument, they derive an objective for SimCLR and BYOL that explicitly compresses the representation using the Conditional Entropy Bottleneck. The addition of this compression improves the performance for both BYOL and SimCLR on ImageNET. They also demonstrate that their method improves the robustness of the encoder and its representation to perturbation of the input data.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a principled and novel approach to learning a compressed representation in the context of contrastive methods. While the method is well detailed for both SimCLR and BYOL, it could be helpful to have a pseudo-code of the objective function detailing how the computation is done in practice. Other than that, the paper is very well written. The argument is concise and easy to follow.\n\nThe experiments convincingly show the benefits of the approach proposed. While the authors demonstrate that their method performs better than the state-of-the-art of ImageNet, they also provide an ablation demonstrating that the improvement is due to their modifications. They also demonstrate that their method improves robustness on various benchmarks.\n\nOverall, the paper provides a good demonstration that the compression of latent representations improves robustness. \n\n### Question / comments\n* L107: I am confused about the correspondance between $exp(\\dfrac{1}{\\tau} r_{yk}^\\top r_x)$ and $b(z|y)$. Isn't $y_k$ the *negative* views and $y$ the *positive* view. I.e., shouldn't the former be a correspondance with $b(z|y_k)$?\n* While the authors claim that they will release their code upon acceptance, they could have released an anonymized version.\n* The practical realization of $e$, $b$, $c$ and $d$ were not clear to me. The confusion could be resolved by presenting a pseudo-code of the practical implementation of the method.\n* It is not clear to me why a model Lipschitz constant generally relates to model robustness. The authors present some works that relate \"types of model robustness\" to the model's Lipschitz constant. What are the types of model robustness? How do these works relate to your model?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some limitations of their work.\n\nThe authors present a generic discussion of the societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies conditional entropy bottleneck (CEB) objective to two existing frameworks for self-supervised learning: SimCLR and BYOL.  The paper clearly demonstrates the advantages of this approach and is well-grounded in the existing theory on CEB. Adapting CEB to SimCLR and BYOL seems to be novel. Interestingly, presented approach can be applied in several similar settings, thus the results can be potentially impactful also beyond the main scope of the paper.
Decision: Accept (Poster)