Improving Transformation Invariance in Contrastive Representation Learning
Authors: Adam Foster
Rattana Pukdee
Tom Rainforth
Venue: ICLR 2021 Poster (2021)
Keywords: contrastive learning
representation learning
transformation invariance

Abstract: We propose methods to strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance as representations are learned, we look to more directly enforce invariance in the encoding process. To this end, we first introduce a training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation. We show that representations trained with this objective perform better on downstream tasks and are more robust to the introduction of nuisance transformations at test time. Second, we propose a change to how test time representations are generated by introducing a feature averaging approach that combines encodings from multiple transformations of the original input, finding that this leads to across the board performance gains. Finally, we introduce the novel Spirograph dataset to explore our ideas in the context of a differentiable generative process with multiple downstream tasks, showing that our techniques for learning invariance are highly beneficial.

Published as a conference paper at ICLR 2021
IMPROVING TRANSFORMATION INVARIANCE IN CONTRASTIVE REPRESENTATION LEARNING
Adam Foster", Rattana Pukdee' & Tom Rainforth
Department of Statistics
University of Oxford (adam. foster, rainforthhessttssssomo
ABSTRACT
We propose methods strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance representations force invariance in the encoding process, To this end, we first introduce a train- ing objective for contrastive learning that uses novel regularizer to control how the representation changes under transformation. We show that representations trained with this objective perform better on downstream tasks and are more ro- bust to the introduction of nuisance transformations at test time. Second, we pro- pose a change to how test time representations are generated by introducing a feature averaging approach that combines encodings from multiple transforma- tions the original input, finding that this leads to across the board performance gains. Finally, introduce the novel Spirograph dataset to explore ideas in the context of a differentiable generative process with multiple downstream tasks, showing that techniques for learning invariance are highly beneficial.
learned, look more directly en-
INTRODUCTION
Learning meaningful representations of data is a central endeavour in artificial intelligence. Such representations should retain important information about the original input whilst using fewer bits to store it (van der Maaten et al. 2009; Gregor et al. (016)). Semantically meaningful representations may discard great deal of information about the input, whilst capturing what relevant. Knowing what to discard, as well as what to keep, is key to obtaining powerful representations.
By defining transformations that are believed a priori to distort the original without altering semantic features of interest, we can learn representations that are (approximately) invariant to these transfor- mations (Hadsell et al. 2006). Such representations may be more efficient and more generalizable than lossless encodings. Whilst less effective for reconstruction, these representations are useful in many downstream tasks that relate only to the semantic features of the input. Representation invari- ance is also critically important task in of itself: it can lead to improved robustness and remove noise (Du et al., 2020), afford fairness in downstream predictions (Jaiswal et al., (020), and enhance interpretability (Xu al. 2018).
Contrastive learning is a recent and highly successful self-supervized approach to representation learning that has achieved state-of-the -hee--rrtte-rttttt performance in tasks that rely semantic features, rather than exact reconstruction (van den Oord et al. 2018 Hjelm et al. 2018 Bachman et al. 2019 He al. 2019) These methods learn match two different transtormations of the same object in representation space, distinguishing them from contrasts that are representations of other objects.
The objective functions used for contrastive learning encourage representations to remain similar under transformation, whilst simultancously requiring different inputs to be well spread out in rep- resentation space (Wang & Isola 2020). such, the choice of transformations key to their success (Chen et 2020a). Typical choices include random cropping and colour distortion.
However, representations are compared using a similarity function that can be maximized even for representations that are far apart, meaning that the invariance learned is relatively weak. Unfor-
Equull contribution

---

Published as a conference paper at ICLR 2021
tunately, directly changing the similarity measure hampers the algorithm (Wu et al. 2018 Chen jet al. 2020a). We therefore investigate methods to improve contrastive representations by explic- itly encouraging stronger invariance to the set of transformations, without changing the core self- supervized objective; we look to extract more information about how representations are changing with respect to transformation, and use this to direct the encoder towards greater invariance.
To this end, we first develop a gradient regularization term that, when included in the training loss, forces the encoder to learn representation function that varies slowly with continuous transforma- tions. This can be seen as constraining the encoder to be approximately transformation invariant. We demonstrate empirically that while the parameters of the transformation can be recovered from standard contrastive learning representations using just linear regression, this is no longer the case when our regularization is used. Moreover, our representations perform better on downstream tasks and are robust to the introduction of nuisance transformations at test time.
Test representations are conventionally produced using untransformed inputs (Hjelm et al. 2018) Kolesnikov et al. 2019), but this fails to combine information from different transformations and views of the object, or to emulate settings which transformation noise cannot simply be removed at test time. Our second key proposal is to instead create test time representations by feature averaging over multiple, differently transformed, inputs to address these concerns and to more directly impose invariance. We show theoretically that this leads to improved performance under linear evaluation protocols, further confirming this result empirically.
We evaluate approaches first CIFAR-10 and CIFAR-100 (Krizhevsky al. 2009), using transformations appropriate to natural images and evaluating on a downstream classification task. To validate that our ideas transfer other settings, and to use our gradient regularizer within fully differentiable generative process, we further introduce a new synthetic dataset called Spirograph. This provides greater variety of downstream regression tasks, and allows us to explore the interplay between nuisance transformations and generative factors of interest. We confirm that using our regularizer during training and our feature averaging test time both improve performance in terms of transformation invariance, downstream tasks, and robustness to westt distributional shift.
In summary, the contributions of this paper are as follows:
We derive novel contrastive learning objective that leads to more invariant representations.
We propose test time feature averaging enforce further invariance.
We introduce the Spirograph dataset.
We show empirically that our approaches lead to more invariant representations and achieve state-of-the-ard performance for existing downstream task benchmarks.
2 PROBABILISTIC FORMULATION OF CONTRASTIVE LEARNING
The goal of unsupervized representation learning is to encode high-dimensional data, such as im- ages, retaining information that may be pertinent to downstream tasks and discarding information that is not. To formalize this, we consider a data distribution p(x) on X' and an encoder fa X - Z which is a parametrized function mapping from data space to representation space.
Contrastive learning is a self-supervized approach to representation learning that learns to make rep- resentations of differently transformed versions of the same input more similar than representations of other inputs. Of central importance is the set of transformations, also called augmentations (Chen al. 2020a) or views (Tian et al., 2019), used to distort the data input x. In the common appli- cation of computer vision, it is typical to include resized cropping: brightness, contrast, saturation and hue distortion; greyscale conversion; and horizontal flipping. We will later introduce the Spiro- graph dataset which uses quite different transformations. In general, transformations to change the input only cosmetically, so all semantic features such as the class label are preserved; the set of transformations indicates changes which can be safely ignored by the encoder.
assumed
Formally, we consider a transformation set T C {t : X + X} and probability distribution p(t) on this set. A representation of x is obtained by applying random transformation t to and then encoding the result using fy. Therefore, we do not have one representation of x, but an implicit distribution p(z) x). A sample of p(z|x) is obtained by sampling p(t) and setting (t(x)))
2

---

Published as a conference paper at ICLR 2021
If the encoder is to discard irrelevant information, would expect different encodings of x formed with different transformations to be close in representation space. Altering the transformation should not lead to big changes in the representations of the same imput. In other words, the distri- bution p(z/x) should place most probability mass in a small region. However, this does not provide sufficient training signal for the encoder fo it fails penalize trivial solutions in which all are mapped to the same z. To preserve meaningful information about the input x whilst discarding purely cosmetic features, we should require p(z|x) to be focused around a single whilst simul- taneously requiring the representations of different inputs not to be close. That is, the marginal p(z) Ep(x) [p(x))) should distribute probability mass over representation space.
This intuition is directly reflected in contrastive learning. Most afrte--rrr contrastive learning methods utilize the InfoNCE objective (van den Oord et al., 2018), or close variants of (Chen et al. 2020a). InfoNCE uses a batch x1:..., xx of inputs, from which we form pairs of representa- tions 1,1 2 .... (2x. < by applying two random transformations each input followed by the encoder fg. In probabilistic language
x; ~ pxx) for l...., K
2, z ~ p(z)x = x;) conditionally independently given x,, for i 1,..., K, such that 2;, x = (o(t(x)),fem la (t((x) for i.i.d. transformations t, ~ p(t). Given a learnable similarity score SΦ 2 2 R, contrastive learning methods minimize the following loss
(2)
K
K log Σ
1 K
1 K
6)
(2
2 +
exp [s0(2;, 21)]
(3)
Written in this way, we see that the loss will be minimized when so (z, 2/) is large, but 8 (2, 2/) small for * j. In other words, InfoNCE makes the two samples 2,, 2 of p(z|x = xx) similar, whilst making samples z,, z, of p(z) dissimilar. This can also be understood through the lens of
mutual information, for more details see Appendix [A]
In practice, the similarity measure used generally takes the form (Chen et al. 2020al
so z') = 9
2 (z)
(z')
(4)
= (z) (z))))
where go is a small neural network and perfectly invariant to the transformations, then z, = z and s z[) will be maximal. However, there are many ways to maximize the InfoNCE objective without encouraging strong invariance in the encoder[]in In this paper, we show how we can learn stronger invariances, above and beyond what is learned through the above approach, and that this benefits downstream task performance.
temperature hyperparameter. If the encoder fo is
INVARIANCE BY GRADIENT REGULARIZATION
Contrastive learning with InfoNCE can gently encourage invariance by maximizing s& (z z'). but does not provide strong signal to ensure this invariance. Our first core contribution is to show how we can use gradient methods to directly regulate how the representation changes with the trans- formation and thus ensure the desired invariance. The key underlying idea is to differentiate the representation with respect to the transformation, and then encourage this gradient to be small so that the representation changes slowly as the transformation is varied.
To formalize this, we begin by looking more closely at the transformations T which are used to de- fine the distribution p(z(x). Many transformations, such brightness adjustment, are controlled by a transformation parameter. We can include these parameters in our set-up by writing the transformation
a map from both input space X' and transformation parameter space Ut i.e. t : X x l > X. In this formulation, we sample random transformation parameter from u ~ p(u) which is distribution on U. A sample from p(z|x) is then obtained by taking fo(t(x, u)), with t now regarded as a fixed function.
This is because the function ge is not injection, so we may have gz - (z') but 2 * z'. Johnson & [.indenstrauss (1984) gives conditions under which a projection of this form will preserve approximate distances, particular, the required projection dimension is much larger than the typical value 128.
3

---

Published as a conference paper at ICLR 2021
The advantage of this change of perspective is that it opens up additional ways to learn stronger invariance of the encoder. In particular, it may make sense to consider the gradient Vuz. which describes the rate of change of z with respect to the transformation. This only makes sense for some transformation parameters- can differentiate with respect to the brightness scaling but not with respect horizontal flip.
To separate out differentiable and non-differentiable parameters we write u = =a, 3 where a are the parameters for which it makes sense consider the derivative Vaz. Intuitively, this gradient should be small to ensure that representations change only slowly as the transformation parameter or varied. For clarity of exposition, and for implementation practicalities, it is important consider gradients of scalar function, so we introduce an arbitrary direction vector e E Z and define fo fo(t(x, a. 3)) F(a, B, x, e) [t, ax, 3))12
(5)
R calculates the scalar projection of the normalized representation z/ e/||z||2 in the direction. To encourage encoder that is invariant changes in a, we would like to minimize the expected conditional variance of F with respect to a:
so that F x x
V (p(x))(3)p(e) [Varp(a]] 3,, e) x. 3.
where we have exploited independence to write p(x, e) = p(x)p(()p(ee) Defining V requires distribution for be specified. For this, make components of independent Rademacher random variables, justification for which included Appendix B
A naive estimator of V can be formed using direct nested Monte Carlo estimator (Rainforth al. 2018) of sample variances, which, including Bessel's correction, is given by
L(L L(L- Σ F(au, Fla B.) 1"
V~ K
Σ Bl
where X,, Br e, ~ p(x)p(3)p(() and ai ~ p(a). However, this estimator requires LK forward passes through the encoder fo to evaluate. As alternative to this computationally prohibitive approach, we consider a first-order approximation
F(a', 3, e) F(a. B, e) V. F(a (a' - a)) + +ol -
(8)
and the following alternative form for the conditional variance (see Appendix B for a derivation)
Varp(a) [F x, e) x. 3, e] = { (, - F(a', B, x, e))² x, e
(9)
Combining these two ideas, we have
V ((()))(()p(e))
Fp(a))(a)) (F(a, B,x B.x, e))² x 1, e e]] Ep(x)r(3)n(e) pp())(aa) (VaF(a, B,x,, (a' a))² 1x
(10)
(11)
Here we have an approximation of the conditional variance V that uses gradient information. In- cluding this regularizer within contrastive learning will encourage the encoder reduce the magnitude of the conditional variance V, forcing the representation to change slowly as the trans- formation is varied and thus inducing approximate invariance the transformations.
An unbiased estimator of equation
I
using a batch XI,..., XX is
Vegullarizrr
1 Σ
Σ
VaF Bi, c;) (al, a;)]
(12)
K
2L
where X,, 0i,, Bi, , p(x)p((app(3)p(c) a' ~ p(a). We can cheaply use large number of
samples for a' without having to take any additional forward passes through the encoder: we only require K evaluations of F. Our final loss function is
K
K Σ
K Σ
C(0, () = K
S (Z
2 + K
log
exp [so 25,]
1=1
(13)
K
L
+
V (a., x;, (al, - aas)
LK
== j=1
We use the notation a(z) o(b(z)) ((z)/b(z))
0
4

---

Published as a conference paper at ICLR 2021
where À is hyperparameter controlling the regularization strength. This loss does not require us to encode a larger number of differently transformed inputs. Instead. it uses the gradient at (x, a, [3,e) control properties of the encoder in neighbourhood of a. This can effectively reduce the representation gradient along the directions corresponding to many different transformations. This, in turm, creates encoder that is approximately invariant to the transformations.
BETTER TEST TIME REPRESENTATIONS WITH FEATURE AVERAGING
At test time, standard practice (Hjelm al. 2018 Kolesnikov al. 2019) dictates that test repre- sentations be produced by applying the encoder to untransformed inputs (possibly using a central crop). It may be beneficial, however, to aggregate information from differently transformed versions of inputs to enforce invariance more directly, particularly when our previously introduced gradient regularization can only be applied to a subset of the transformation parameters. Furthermore, in real-world applications, may not be possible to remove nuisance transformations at test time or, as in our Spirograph dataset, there may not be only one unique "untransformed" version of x.
To this end, we propose combining representations from different transformations using feature averaging. This approach, akin ensembling, does not directly use one encoding from the network fo as representation for an input Instead, we sample transformation parameters C......, Q&M ~ 0(ax),1,, ..., BM p(3) independently. and average the encodings of these differently transformed versions of to give single feature averaged representation
M
z (M) (M)
Σ m=1
ft(t(x, 0mm; Bm.).
(14)
M
Using z (M) aggregates information about x by averaging over range of possible transformations, thereby directly encouraging invariance. Indeed, the resulting representation has lower conditional variance than the single-sample alternative, since
Var ar())ar)
e z (dr)) (x) xx)
M Varp(an)<(91))
(e zz (x)
(15)
Further, unlike gradient regularization, this approach takes account of all transformations, including those which cannot differentiate with respect to (e.g. left-right flip). It therefore forms natural test time counterpart to our training methodology to promote invariance.
We do not recommend using feature averaged representations during training. During training, we need a training signal to recognize similar and dissimilar representations, and feature averaging will weaken this training signal. Furthermore, the computational cost of additional encoder passes is modest when used once at test time, but more significant when used at every training iteration.
As a test time tool though, feature averaging powerful. In Theorem ! below, we show that for certain downstream tasks the feature averaged representation will always perform better than the single-sample transformed alternative. The proof presented in Appendix
Theorem 1. Consider evaluation downstream task by fitting linear classification model with softmax loss or linear regression model with square error loss on with representations as features. For fixed classifier or regressor and M' 2 M we have
(2(("),)) [
(2( .v)
S appxxiiploraa))(B...
(16)
Empirically we find that, using the same encoder and the same linear classification model, feature averaging can outperform evaluation using untransformed inputs. That is, even when it is possible to remove the transformations at test time, it is beneficial to retain them and use feature averaging.
5 RELATED WORK
Contrastive learning (van den Oord et al. 2018 Hénaff et al. 2019) has progressively refined the role of transformations in learning representations, with Bachman et al. (2019) applying repeated data augmentation and Tian et al. (2019) using Lab colour decomposition to define powerful self- supervized tasks. The range of transformations has progressively increased (Chen et al. 2020a
5

---

Published as a conference paper at ICLR 2021
Figure 1: Samples from the spirograph dataset. Two sets of four images (left and right); each set shows different transformations applied to the same generative factors of interest.
whilst changing transformations can markedly improve performance (Chen et al. 2020c). Recent work has attempted to further understand and refine the role of transformations (Tian al. 2020).
The idea of differentiating with respect to transformation parameters dates back to the tangent prop- agation algorithm (Simard al. 1998 Rifai et al. 2011). Using the notation of this paper, tangent propagation penalizes the norm the gradient of a neural network evaluated at a = 0, encourag- ing local transformation invariance near the original input. In work, target the conditional variance (Equation 6). leading to gradient evaluations across the a parameter space with random p(ax) and regularizer that not gradient norm (Equation [2
Our gradient regularization approach also connects to work on gradient regularization for Lipschitz constraints. small Lipschitz constant has been shown to lead better generalization (Sokolić et al. 2017) and improved adversarial robustness (Cisse et al. 2017 Tsuzuku et al. 20188 Barrett al. 2021 Previous work focuses on constraining the mapping x z to have a small Lipschitz constant which is beneficial for adversarial robustness. In our work we focus on the influence of a: on z, which gives rise to transformation robustness. Appendix Daaiixx]pp provides a more comprehensive discussion of related work.
EXPERIMENTS
6.1 DATASETS AND SET-UP
The methods proposed in this paper learn representations that discard some information, whilst taining what is relevant. To more deeply explore this idea, we construct dataset from generative process controlled by both generative factors of interest and nuisance transformations. Represen- tations should be able to recover the factors of interest, whilst being approximately invariant to transformation. To aid direct evaluation of this, introduce new dataset, which refer to the Spirograph dataset. Its samples are created using four generative factors and six nuisance transfor- mation parameters. Figure [
sets of four samples with the generative factors fixed in each set. Every Spirograph sample is based on hypotrochoid one of a parametric family of curves that describe the path traced out by a point on one circle rolling around inside another. This generative process is fully differentiable in the parameters, meaning that our gradient regularization can be ap- plied to every transformation. We define four downstream tasks for this dataset, each corresponding to the recovery of one of the four generative factors of interest using linear regression. The final dataset consists 100k training and 20k test images of size 32 x 32, For full details of this dataset, see Appendix B
As well as the Spirograph dataset, we apply our ideas to CIFAR-I0 and CIFAR- 100 (Krizhevsky et al., 2009). We base our contrastive learning set-up SimCLR (Chen al. 2020a). To use our gradient regularization, we adapt colour distortion (brightness, contrast, saturation and hue ad- justment) fully differentiable transformation giving four dimensional a; we also included random cropping and flipping but we did not apply a gradient regularization to these. We used ResNet50 (He et al. 2016) encoders for CIFAR and ResNet18 for Spirograph, and regularization parameters À= = 0.I for CIFAR and À = 0.01 for Spirograph. For comprehensive details of our set-up and additional plots, see Appendix F For an open source implementation of our methods, see htap::///qtthsse com/ ae- foster invelr
6.2 GRADIENT REGULARIZATION LEADS TO STRONGLY INVARIANT REPRESENTATIONS
We first show that our gradient penalty successfuly leams representations that are more invariant to transformation than standard contrastive learning. First, we estimate the conditional variance

---

Published as a conference paper at ICLR 2021
77
1.0
71
Too acuuacc
70
0.8
69
0.6
88
57
No regularizetion
66
reqularization
Gradient regularization
Gradient
No regulorization
regularization
Gradient regularization
0.2 0.2 6.3 04 055 DE 0.7 0.5
n0
Propertion of train data
train
(a) CIFAR-10
(h) CIFAR-100
(c) Spirograph
Figure 3: Downstream task performance of gradient regularized representations. (a)(b) Top-1 test accuracy for various levels of semi- superviseon (higher better). (c) Test loss on four downstream regression tasks on Spirograph that recover the generative factors of interest (lower better). The loss rescaled for legibility, see Table for raw values. Error bars are 1 standard error from runs.
of the representation that was used as the starting point for motivating approach, i.e. Equation 6 using the slower, but more exact, nested Monte Carlo
0.8
Conditionnal variacee
0.5
estimator of Equation 7 to evaluate this. In Figure 2 we see that the gradient penalty strikingly reduces the conditional variance on CIFAR-10 compared to
0.5
No regularization
Gradient regularization
0.2
0.1
standard contrastive learning.
200
400
600 Epoch
800
1000
As an additional measure of representation invari- ance, we fit linear regression model that predicts a from z, for which higher loss indicates a greater degree of invariance. We also compute reference loss: the loss that would be obtained when predict- ing using only constant. In Table I that unlike standard contrastive learning. after train- ing with gradient regularization the linear regression model cannot predict a from z any better than us- ing constant prediction. The loss is actually higher than the reference value because the former is ob- tained by training a regressor for a finite number of steps, whilst the latter is a theoretical optimum value. Similar results for other datasets are in Appendix(F
Figure 2:
Conditional variance for CIFAR- 10 per Equation 6 Error bars represent +1 standard error from runs.
Table 1:
Test loss when linear regression is used to predict a from z on CIFAR-10. The reference value is Mean, Var(a;). We present the mean and + I s.c. from 3 runs.
Test loss 0.0353 + 0.0002 0.0415 + 0.00006
No regularization Regularization Reference value
0.0408
6.3 GRADIENT REGULARIZATIR FOR DOWNSTREAM TASKS AND TEST TIME DATASET SHIFT
We now show that these more invariant representations perform better on downstream tasks. For CIFAR, we produce representations for each element of the training and test set (by applying the encoder fg to untransformed inputs). We then fit linear classifier on the training set, using different fractions of the class labels. This allows us assess our representations at different levels of super- vision. We use the entire test set to evaluate each of these classifiers. In Figures 3 aa) and (b), we see that the test accuracy improves across the board with gradient regularization.
For Spirograph, we take a similar approach to evaluation: we create representations for the training and test sets and fit linear regression models with representations as features for each of the four downstream tasks. In Figure c). we see the test loss on each task with the baseline scaled to 1. Here we see huge improvements across all tasks, presumably due to the ability to apply gradient regularization to all transformations (unlike for CIFAR).
We further study the effect transformation at test time, showing that gradient penalized repre- sentations can be more robust to shifts in the transformation distribution. For CIFAR-10, we apply colour distortion transformations at test time with different levels variance. By focusing on colour distortion at test time, we isolate the transformations that the gradient regularization targetted. In Figure (a) we see that when the test time distribution is shifted to have higher variance than the training regime, our gradient penalized representations perform better than using contrastive learn-

---

Published as a conference paper at ICLR 2021
0.000
No regulertzation Gradient
regulenication
0.007
regubrization
TeG: Soportt
0.505
01.5
91.0
00.5
No regullarzziinn
0.00
2,035
90.0
Gradlett
requomaatoo
5,022
88.
23
0.00
-0.6
-0.1
-2.3
0.0
0.1
0.6
0.2 chdribztion
0.3
8-012 butermmnaa 20111
(a) CIFAR-I0 var shift
(b) Spirograph mean shift
(c) Spirograph var shift
Figure 4: Assessing representation robustness to test time distribution shift. (a) Changing the vari- ance of colour distortions; is no transformation and 0.5 is the training regime. (b) Mean shifting of the distribution of the transformation parameter h. (c) Variance shifting of the background colour
distribution. (b)(c), shift indicates the training regime. Error bars are s.e. from runs.
Gradient
0.0035
0.0000
04.5
Tist accuroc
0.0025
72
0.0020
0.0015
0.0010
95.0
0.009
20
102
20
(a) CIFAR-10
(b) CIFAR-100
(c) Spirograph
Figure 5: The impact of feature averaging CIFAR-IO. CIFAR-100 and Spirograph. (a)b) Test accuracy for CIFAR-IO and CIFAR-100 respectively for various values M. Dashed lines represent evaluation with untransformed inputs. (c) Test mean square enror averaged over all four tasks for Spirograph (untransformed inputs not valid here). Error bars are +1 s.c. from 3 runs.
ing alone. For Spirograph, we investigate changing both the mean the transformation distribution, moving the entire test distribution away from the training regime, and increasing the variance of transformations to add noise. Results are shown Figure and (c). In []) in particular, we see that gradient regularized representations are robust to greater level of distortion at test time.
6.4 FEATURE AVERAGING FURTHER IMPROVES PERFORMANCE
We now assess the impact of feature averaging on test time performance. For CIFAR, we apply feature averaging using all transformations, including random crops etc., and compare with the standard protocol of using untransformed inputs to form the test representations. Figures a) and (b) show that feature averaging leads to significant improvements. This adds the result Theorem which implies that test loss decreases as M is increased. In Figure ((c), we see that feature averaging has an equally beneficial impact on Spirograph. is interesting note that in both still significant residual benefit from gradient regularization, even with large value of M.
there
6.5 OUR METHODS COMPARE FAVOURABLY WITH OTHER PUBLISHED BASELINES
Our primary aim was to show that both gradient regularization and feature averaging lead to im- provements compared to baselines that are in other respects identical. Our methods are applicable to almost any base contrastive learning approach, and we would expect them to deliver improve- ments across this range of different base methods. In Table 2
present published baselines
Comparative best test accuracy of various self self-supervized representation learning tech- niques, evaluated using linear classification.
Table 2:
Method
CIFAR- 10
CIFAR- 100 acc.
AMDIM small (Bachman et al.
2019
89.5%
68.1%
AMDIM large Bachman et 2019
91.2%
70.2%
SimCLR (Chen et al. 2020a)
94.0%
Ours (SiMCLR base)
94.9%
75.1%
8

---

Published as a conference paper at ICLR 2021
Tutal Loz. sqa.eellllllll
0.02300
conditoral
0,0225
0.03799
(a) Conditional variance
(b) Downstream task performance
Figure 6: The impact of the regularization hyperparameter À on representation learning with the Spirograph dataset. (a) Conditional variance of Equation [6] (b) The total mean square error all four downstream tasks. Error bars are +1 s.e. from 3 runs. Smaller is better in both cases giving an optimum around 10-3, but with stable performance as is increased above this.
Table 3: Results for representation learning on CIFAR- 100 with MoCo v2 as the base contrastive learning method, with gradient regularization in isolation and in combination with feature averaging. We trained two different encoder architectures. We present test accuracy from linear classification evaluation. Feature averaging uses M Errors are 1 s.e. from multiple runs.
Method
ResNer18 8 acc.
ResNet50 acc.
MoCo v2
52.3% + 0.3
57.9% = 0.1
MoCo v2 with gradient penalty
54.1% + 0.1
58.9% 0.2
MoCo v2 with gradient penalty and feature averaging
60.6% + 0.1
64.4% + 0.2
CIFAR datasets, along with the results that we obtain using our gradient regularization and feature averaging with SimCLR base method. This is the default base method that recommend, and that was used in our previous experiments. Interestingly, the best ResNet50 encoder from our experiments achieves an accuracy of 94.9% on CIFAR-10, which outperforms the next best pub- lished result from the contrastive learning literature by almost 1,, and 75.1% 15.1% on CIFAR-100, an almost 5% improvement over significantly larger encoder architecture. As such, we see our results actually provide performance that is tate-offthhe-att for contrastive learning on these benchmarks. In fact, our performance increases almost entirely close gap to the state-of-the-ar performance for fully supervized training with the same architecture on CIFAR-10 (95.15 1%, Chen et al. (2020a)).
To demonstrate that our ideas generalize to other contrastive learning base methods, we apply our ideas to MoCo v2 (Chen et al. 2020c). Table 3 shows that, whilst MoCo v2 itself does not perform as well as SimCLR on CIFAR- 100, the addition gradient regularization and feature averaging still leads significant improvements in its performance. Table 3 further illustrates that both gradient regularization and feature averaging contribute to the performance improvements offered by our approach and that techniques generalize across diffrent encoder architectures.
6.6 HYPERPARAMETER SENSITIVITY
As further ablation study, we investigated the sensitivity of our method to changes in the gradient regularization hyperparameter À (as defined in Equation 13}. In Fig we see that, as expected, the conditional variance of representations decreases À is increased. The downstream task per- formance Figure 6|b) similarly improves as we increase X, reaching an optimum around X = 10-3 before beginning to increase due over-regularization. We that wide range of values of deliver good performance and the method is not overly sensitive to careful tuning of X.
7 CONCLUSION
Viewing contrastive representation learning through the lens of representation invariance to trans- formation, we derived a gradient regularizer that controls how quickly representations can change with transformation, and proposed feature averaging test time to pull in information from multi- ple transformations. These approaches led to representations that performed better on downstream tasks. Therefore, work provides evidence that invariance is highly relevant to the success of con- trastive learning methods, and that there is scope to further improve upon these methods by using invariance guiding principle.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
AF gratefully acknowledges funding from EPSRC grant no. EP/N509711/1, AF would also like to thank Benjamin Bloem- Reeed for helpful discussions about theoretical aspects of this work.
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp- 15535-15545, 2019.
Ben Barrett, Alexander Camuto, Matthew Willetts, and Tom Rainforth. Certifiably robust variational autoencoders. arXiv preprint arXiv:2102.075599 2021.
Olivier Catoni. Pac-bayesian supervised classification: the thermodynamics of statistical learning. arXiv preprint arXiv:0712.02488, 2007.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil the details: Delving deep into convolutional nets. arXiv preprint trXivv:105.335 2014.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. Invariance reduces variance: Understanding data augmentation in deep learning and beyond. arXiv preprint /907. 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXivv2002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020c.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. arXiv preprint arXhv:1004.08847, 2017.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999, 2016.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. arXiv preprint arXiv.1801.10130, 2018.
Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropa- gation. IEEE Transactions on Neural Networks, 3(6):991-997, 1992.
Wenchao Du, Hu Chen, and Hongyu Yang. Learning invariant representation for unsupervised image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14483-14499 14492, 2020.
Benjamin Graham. Fractional max-pooling, arXiv preprint arXiv:1412.6071, 2014.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In Advances In Neural Information Processing Systems, pp. 3549-3557,
2016.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein GANs. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- fion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 2019.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint (XXix:190500927. 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: /woXivv/808..om 1108.880800000 2018.
Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premkumar Natarajan. Invariant representations through adversarial forgetting. In AAAI, pp. 0772-479,, 2020.
William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26( 36(189-206):, 1984.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In Proceedings of the IEFE conference on Computer Vision and Pattern Recog- nition, pp. 1920-1929, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
lyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint I60iv 03983, 2016.
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin (loem-Reddy On the benefits of invariance in neural networks. arXiv preprini arXiv:2005.00118, 2020.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Ser- manet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pp. 15578-15588, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On varia- tional bounds of mutual information. arXiv preprint arXiv:1905.0699) 2019.
Tom Rainforth, Rob Cornish, Hongseok Yang. Andrew Warrington, and Frank Wood. On nesting Monte Carlo estimators. In International Conference on Machine Learning, pp. 4267-4276. PMLR, 2018.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. Advances in neural information processing systems, 24:2294-2300 2302, 2011.
Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorii. Transformation invariance in pattern recognitiontangent distance and tangent propagation. In Neural networks: tricks of the trade, pp. 239-274. Springer, 1998.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, (211)))426-220000 2017.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-ande canslation-equivariaee neural networks for 3d point 2018.
clouds. arXiv preprint

---

Published as a conference paper at ICLR 2021
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
Yonglong Tian, Chen Sun, Ben Poole. Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certifi- cation of perturbation invariance for deep neural networks. In Advances in neural information processing systems, pp. 6541-6550, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:
2018.
Laurens van der Maaten, Eric Postma, and Jaap van den Herik. Dimensionality reduction: com- parative. Journal of Machine Learning Research, 10(66-71)::1) 2009.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. arXiv preprini urXiv:2005./0242, 2020.
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph Ledsam, Patricia WacWiliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved
detection. arXiv preprint arXiv:2007.05566, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3733-3742, 2018.
Tian Xu, Jiayu Zhan, Oliver GB Garrod, Philip HS Torr, Song-Chun Zhu, Robin AA Ince, and Philippe Schyns. Deeper interpretability deep networks. arXiv preprint arXiv:1811.07880 2018.
Donggeun Yoo, Sunggyun Park, Yoon-yung Lee, and In So Kweon. Multi-scale pyramid pooling for deep convolutional representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 71-80, 2015.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 700.00888..88 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances neural information processing systems, pp. 3391- 3401, 2017.
12

---

Published as a conference paper at ICLR 2021
A MUTUAL INFORMATION
In Section we saw that the InfoNCE objective equation fulfills the need to make p(z|x) tightly focused on single point whilst simultaneously requiring p(z) to be well spread out represen- tation space. In this appendix, we show that the same general principle of making p(z) x) tightly focused on a single point whilst simultaneously requiring p(z) be well spread out over represen- tation space connects to mutual information maximization.
To establish the connection to mutual information, we take the differential entropy as our measure of 'spread'. Recall the differential entropy of a random variable w is
H [p((w]] ppt log p(w)].
(17)
We then translate our intuition make p(z|x) tightly focused on single point whilst simultaneously requiring p(z) to be well spread out over representation space into requiring Ep(x) [H [p(z/x)]] to be minimized whilst H [p(z]] should be simultancously maximized. This suggests the following loss function
Centroy (Ep(x)([p((zx)) - H[p(z)] = (xx;z)
(18)
which is the (negative) mutual information between x and z. Note that in this formulation, it the distribution p( x)) as much as the InfoMax principle which determines how this loss will behave. Finally, there a clear connection between the InfoNCE loss and mutual information, specifically the InfoNCE loss is, in expectation and up an additive constant, lower bound on / [xx; z) van den
Oord et al. 2018); Poole et al. (2019).
B METHOD
B. AN ALTERNATIVE VARIANCE FORMULA
We present derivation of our alternative formula for the variance (dropping the conditioning from the notation for conciseness)
E placpior) [(F(a,3,x,)) F,x',
2 Eplaipaa))
[(F(a, 4a
-Epajjaaa
+ Epla)pia'
(F((a,8,x,
(F(c,,3xxcc (4 Fa
Varp(a) (F,,,,,x,e
B.2 MOTIVATING THE RADEMACHER DISTRIBUTION
We are interested in the conditional variance with respect to a. but as is vector valued random variable we properly need to consider the conditional covariance matrix Σ Cova (z)x, B). We henceforth consider x, 3 to be fixed. To reduce conditional variance all directions, it makes sense to reduce the trace Tr Σ. Due to computational limitations, at each iteration, instead we must estimate Var(e z) = e Ee. However, by carefully selecting the distribution for expectation over e. Specifically, suppose that the components of are independent Rademacher random variables (=1 with equal probability). Then
cannot directly estimate this trace
can effectively target the trace of the covariance matrix by taking the
Ep(c) le Σe] = Ep(c)
Σ Eeeee
- ij
EENdd [eeez] = Eijdes = T.
(19)
i5
C THEORY
We present the proof of Theorem which is restated for convenience.
13

---

Published as a conference paper at ICLR 2021
Theorem 1. Consider evaluation downstream task by fitting linear classification model with softmax loss or a linear regression model with square ervor loss on with representations as features. For fixed classifier or regressor and M' 2 M we have
[ (2())) .0] - bwwx.hhllansarrbbassss) (2
(16)
Proof. We have the softmax loss
f(z. v) = -w; + log Σ exp (w0 2)
(20)
or the square error loss
f(z. 1) y/ - w z 2
(21)
We first show that both loss functions considered are convex in the argument To show this, we fix
S p 1pp1-1 S 1. For softmax loss, we have
e (p + (22,8)
(22)
= -w,, (nz) + (22) + log Σ exp (w] (pz) + +gz))
(23)
= =-ww, 21 - gwy 22 + log
Σ exp (w] 21)" exp (w] (w]]22) 222)
(24)
4
-pww, 21 - qw, 22 + log
> exp (w, 21)
Σ exp (w) 22)
(25)
by Hölder's Inequality
-pw zı qw,, 22 plog
[esp
Σ exp (w]zz)) )
q +glog Σ exp (w] 22)
(26)
pl (21,2/1 + oe (zz, v) and for square error loss we have
(27)
8. (yz) + (22, y) = |y w (11) + q²2)1
(28)
1p(y w 21) q0y w TT 22)
(29)
ply w T "z g/y w zzl² 2 + (p² - p) [o/ww> 21 w 22
(30)
S ply w T 2 gly w 22 2
(31)
pd (Zi,v) v) + ge (z2.v) v)
(32)
For the inequality in the Theorem, consider drawing M' M samples, and randomly choosing an M subset. Let S represent this subset and let 2s (M) that uses the subset S. We have
represent the feature averaged representation
E Pp(x.s)planaarpp0.. (p(Ba))))))))) (2(40, E Pp(x.))p(ttaa pla (p( )(())
[e
(M)
2is .1)
(33)
(P [ (Ep(()
25
(M))
.11
(34)
bo i 2 (M') y/
(35)
where the inequality at equation 34 by Jensen's Inequality. This completes the proof.
We provide an informal discussion of other theoretical results that relate to work. Lyle et al. (2020) explored -ayyesan approaches to analyzing the role of group invariance in generaliza- tion of supervized neural network models. The central bound based on Catoni (2077) is given in
14

---

Published as a conference paper at ICLR 2021
Theorem 1 of Lyle et al. (2020) and depends the empirical risk Re(Q, Dn,) and the KL term L P) which represents the PAC-Bayesian KL divergence between distributions on hypothe- sis space. Theorem 7 of Lyle et al. (2020) shows KL(Q")IPP) I [L( where Q and p are formed by symmetrization such as feature averaging over the group of transformations. In our context, although the transformations do not form a group, we could still consider symmetriza- tion operation with feature averaging. If the symmetrization does not affect the empirical risk, then Theorem 9 Lyle al. (2020) would apply to our setting and we would be able to obtain tighter generalization bound for our suggested approach of feature averaging.
D RELATED WORK
D.1 THE ROLE OF TRANSFORMATIONS CONTRASTIVE LEARNING
Recent work on contrastive learning. initiated by the development of Contrastive Predictive Coding (van den Oord al. 2018 Hénaff et al., 2019). has progressively moved the transformations to a more central position in understanding and improving these approaches. In Bachman et al. (2019), multiple views of a context are extracted, on images this utilizes repeated data augmentation such as random resized crop, random colour jitter, and random conversion to grayscale, and the model is trained to maximize information between these views using an InfoNCE style objective. Other approaches are possible, for instance Tian et al. (2019) obtained multiple views of images using Lab colour decomposition. In SimCLR (Chen et al., 2020a b), the approach of applying multiple data augmentations (including flip and blur, well as crops, colour jitter and random grayscale) and using an InfoNCE objective was simplified and streamlined, and the central role of the aug- mentations was emphasized. By changing the set of transformation operations used, [Cen et al.. 2020c) were able to improve their contrastive learning approach and achieve excellent performance on downstream detection and segmentation tasks. TTan et al. (2020) studied what the best range of transformations for contrastive learning is. The authors found that there is a 'sweet spot' in the strength of transformations applied in contrastive learning. with transfornations that are too strong or weak being less favourable. Winkens et al.] (2020) showed that contrastive methods can be suc- cessfully applied to -iitr--distribttion detection. We note that for tasks such as -uitt--dist detection, transformation covariance may be more relevant property than invariance.
D.2 GRADIENT REGULARIZATION TO ENFORCE LIPSCHITZ CONSTRAINTS
Constraining neural network to be Lipschitz continuous bounds how quickly its output can change as the input changes. In supervized learning, small Lipschitz constant has been shown to lead to better generalization (Sokolić et al. (017)) and improved adversarial robustness (Cisse et al. 2017; Tsuzuku et al.] 20188. One practical method for constraining the Lipschitz constant is gradient regularization (rrucker & Le Cun 1992 Guirajani et al. 2017). Lipschitz constraints have also been applied a self-supervized context: in Ozair et al. (2019). the authors used a Wasserstein dependency measure in contrastive learning setting by using gradient penalization to ensure that the function x' + s (fx)),,, fo (x')) ) is I-Lipschitz. Our work uses a gradient regularizer to control how quickly representations can change, but unlike existing work we focus on how representations change with a as x is fixed, instead of how they change with x.
D.3 GROUP INVARIANT NEURAL NETWORKS
A large body of recent work has focused on designing neural network architectures that are perfectly invariant, or equivariant, to a set of transformations T in the case when T forms group. Cohen & Welling (2016) showed how convolutional neural networks can be generalized to have equivariance to arbitrary group transformations applied to their inputs. This can apply, for instance, to rotation groups on the sphere (Cohen et al. 2018), rotation and translation groups on point clouds (Thomas) ct al. 2018), and permutation groups on sets (Zaheer et al. (2017). Transformations that form group cannot remove information from the input (because they must be invertible) and can be composed in any order. This means that the more general transformations considered in our work cannot form group- tthey cannot be composed (repeated decreasing of brightness to zero is not allowed) nor inverted (crops are not invertible). We have therefore considered methods that improve invariance under much more general transformations.
15

---

Published as a conference paper at ICLR 2021
(a) Line drawing
(h) Pixel intensity
(c) Colour sample
Figure 7: A sample from the Spirograph dataset with m
4, = 0.4, = 2, =
(fr fg f6) (0.9 0.8 0.7), (br bg bb) = (0.3 0.4 0.5).
D.4 FEATURE AVERAGING AND POOLING
The concepts of sum- max- and mean-pooling have a rich history in deep learning (Krizhevsky al.) 2012: Graham 2014). For example, pooling can be used down-scale representations in convolutional neural networks (CNNs) as part of single forward pass through the network with single input. our work, however, we apply feature averaging, or mean-pooling, using multiple, differently transformed versions of the same input. This is more similar to Chatfield et al. (2014), who considered pooling or stacking augmented inputs as part of a CNN, and Yoo et al. (2015) who proposed multi-scale pyramid pooling approach. Unlike these works, we apply pooling in unsupervized contrastive representation learning context. Our feature averaging occurs on the final representations, rather than in pyramid, and not intermediate layers of the network. We also use the transformation distribution that is used to define the self-supervized task itself. Other work has explored theoretical aspects of feature averaging (Chen et al., 2019 Lyle et al.. 2020) in the supervized learning setting, showing conditions on the invariance properties of the underlying data distribution that can be exploited obtain improved generalization using feature averaging. For
detailed discussion of Lyle et al. (020) and its connections with our own work, see Section
E SPIROGRAPH DATASET
We propose new dataset that allows the separation of generative of interest from nuisance transformation factors and that is formed from fully differentiable generative process. A stan- dalone implementation of this dataset can be found at
com/ ratttoup/ spi rggrrograhh Our dataset is inspired by the beautiful spirograph patterns some of us drew as children, which are mathematically hypotrochoids given by the following equations
(m h)t
x (m h) cos(t) + cos
(36)
5
(m h)
g (m- h) sin(t) sin
(37)
shows example. To create image dataset from such curves, choose 40 equally spaced points f, with t = and 440 = 2x, giving a sequence points (æ1, (11, (æ40: y40) on the chosen hypotrochoid. For smoothing parameter a, the pixel intensity at point (u, v) is given by
Figure 7
1 40
(u x, - y/)
i(u,v) v)
exp
(38)
For grid of pixel, the intensity values are normalized so that the maximum intensity is equal to 1. Figure (]bb) shows the pixel intensity with 0.5. Finally, for a foreground colour with RGB values (fr: fg. fo) and background colour (b,, by. by). the final RGB values at a point (u, v) is
br
c(u,v) i(u, v)
f
+ i(u, v))
b
(39)
bb
16

---

Published as a conference paper at ICLR 2021
The final coloured sample image is shown in Figure 7c).
The Spirograph sample is fully specified by the parameters m, b, h, a, fr, fg. fo. by, by. bb. our experiments, treat m, b, fr parameters of interest. We treat h and the remaining colour pa- rameters as nuisance parameters. That is, we take x = (m, o, fr) and a = (h, fg: fo. b bg. bp) and the transformation t(x, a:) is the full generative process described above. There tional parameters 3 for this dataset. Figure [ shows two sets of four samples from the Spirograph dataset, in each set the generative factors of interest are fixed and the nuisance parameters ied. In general for the Spirograph dataset, the distinction between generative factors of interest and nuisance parameters can be changed to attempt to learn different aspects of the data. The transfor- mation t is fully differentiable, meaning that we can apply gradient penalization to all the nuisance parameters of the generative process. In our experiments, we took the following distributions to sample random values of the parameters: m ~ (2, 5), ~ (0,11, 1.1), U(0.5, 2.5), ~ (0.25 1), fr: Jg. fo (0.4, 1), b,., by. by (O, 0.6). We synthesized 100,000 training images and 20,000 test images with dimension 32 x 32.
addi-
var-
F EXPERIMENT DETAILS
Our experiments were implemented in PyTorch (Paszke et al. 2019) and ran on 8 Nvidia GeForce GTX 1080Ti GPUs. See htttps://iithiti cowee/ubbboommmeeeeeeeeeeeeeeeee com/ ae foster/invclle for an implementa- tion of our approaches.
F.1 DIFFERENTIABLE COLOUR DISTORTION
We want to improve the representations learned from contrastive methods by explicitly encouraging stronger invariance the set of transformations. Our method is to restrict gradients of the repre- sentations with respect to certain transformations. Ensuring that the transformations are practically differentiable within PyTorch (Paszke et al., (019)) required thorough study of the transformations. The subset of transformations we apply gradient regularization to includes colour distortions which are conventionally treated as a part of data preprocessing. Rewriting this as a differentiable module within the computational graph allows us to practically compute the gradient regularizer of equa- tion II We will consider adjusting brightness, contrast, saturation, hue an image. In fact, most of these transformations are simply linear transformations of the original image. First, the brightness adjustment simply defined as
Xbett X00tt
(40)
when is scale factor. If we write x r, g. b. for the three colour channels of x, then greyscale conversion of given by
gs 0.299r + 0.587g 0.114b.
(41)
Adjusting the saturation of x is a linear combination of x and Xys, the greyscale version of x
X X0 + Xg.
Ossa))
(42)
when Osa is scale factor. Adjusting the contrast of is linear combination of and mean mean(xgs). which the mean over all spatial dimensions of Xyy. With a scaling parameter Oreon we have
Xçon X0 + mean Xgs) -Ccoa)-
(43)
We utilize linear approximation for hue adjustment. We perform hue adjustment by converting to the YIQ colour space, and then applying rotation on the IQ components. The transformation between RGB and YIQ colour space is given by the following linear transformation
0.299
0.587
0.114
R
R G B
0.5959
-0.2746
0.3213
G TYIQ
(44)
0.2115
-0.5227
0.3112
B
Note that the component is exactly the greyscale version Xgs defined above. We transform YIQ back to RGB by
R
0.956
0.619
G
1
-0.272
0.647
7
T TRGB
T
(45)
B
1
-1.106
1.703
Q
Q
17

---

Published as a conference paper at ICLR 2021
Parameter
CIFAR
Spirograph
Encoder model
ResNet50
ResNet18
Training batch size
512
512
Training epochs
1000
50
Optimizer
LARS
LARS
Scheduler
Cosine annealing
Cosine annealing
Learning rate
3
Momentum
0.9
0.9
Temperature T
0.5
0.5
Table 4: Hyperparameters used for CIFAR-I0, CIFAR-100 and Spirograph
In YIQ format, can adjust hue of image by 0 2mee by multiplying with rotation matrix
0
0
RS =
cos
sin cos 0
(46)
sin 0
Therefore, our hue adjustment is given by
Xhue T RRG Tyx TYiQx
(47)
where the matrices operate on the three colour channels of x and in parallel over all spatial dimen- sions. Each operation is followed by pointwise clipping of pixel values to the range [0, 1).
F.2
SET-UP
Our set-up is quite similar to the setup in Chen et al. (2020a) with two main differences: we treat colour distortions as a differentiable module while Chen et al. (2020a) the transformation was performed in the preprocessing step, and we add the gradient penalty term in addition to the original loss Chen et (2020a).
F.2.1 TRANSFORMATIONS
First, for a batch X1, XK of inputs, we form a pair of (x1, xf,xx).... (XK x'x) by applying two random transformations: random resized crop and random horizontal flip for each input. We then apply our differentiable colour distortion function which is composed of random colour jitter with probability 0.8 and random greyscale with probability = 0.2. (Colour jitter is the composi- tion of adjusting brightness, adjusting contrast, adjusting saturation, adjusting hue in this order.) We sample a, the parameter that controls how strong the adjustment is for each image from the follow- ing distributions: brightness, contrast and saturation adjustment parameters from (0.6, 1.4) and hue adjustment parameter from U U(-0.1,0.1) We call the resultant pairs (x1, x()..., .... (XK xxx).
F.2.2 CONTRASTIVE LEARNING
Similar to Chen et al. (020aa, we use the transformed xx), (xK, xxx) as an input to an encoder to learn a pair representations (Z), 2 ..., (žx: z'x) ). The final loss function that we use for training is equation 13 Table shows all hyperparameters that were used for training. The small neural network 9ø is a MLP with the two layers consisting of fully connected linear map, ReLU activation and batch normalization. We use LARS optimizer (You et al. 2017) and apply cosine annealing (Loshchilov & Hutter 2016) to the learning rate.
F.2.3 GRADIENT REGULARIZATION
this part, we explain our setup for calculating the gradient penalty as in equation 12 random vector with independent Rademacher components and independently for each sample in the batch. We generate samples of a for each element of the batch to compute the regularizer. Finally, clip the penalty from above prevent instability the onset of training. In practice, this meant the gradient regularization was not enforced for about the first epoch of training. Table 5 shows hyperparameters that used within gradient penalty calculation.
We sample
18

---

Published as a conference paper at ICLR 2021
Parameter
CIFAR-10
Spirograph
L
100
100
À
0.1
0.01
Clip value
1
1000
Table 5: Hyperparameters for gradient penalty calculation
Parameter
CIFAR-10
Spirograph
Evaluation model
Linear classification
Linear regression
Evaluation loss
Cross entropy loss
Mean squared error 10-8
Weight decay
10-5
Optimization
L-BFGS 500 steps
L-BFGS 500 steps
Table 6: Hyperparameters for model evaluation
F.2.4 EVALUATION
We representations as features in linear classification and regression tasks. We train these linear models with L-BFGS with hyperparameters as shown in Table on the training set and eval- uate performance the test set.
F.2.5 MoCo v2
To empirically demonstrate that our ideas transfer to alternative base contrastive learning methods, applied both gradient regularization and feature averaging base set-up. We also explored two different ResNet (He et al., 2016) architectures. We closely fol- lowed the MoCo v2 implementation ht/ttss: //gs::///iii / liggtthhhn hubb. com/ http///ccboookeeercchhccoccc /moco As for SimCLR, we adapted the transformations to be a differentiable module, We also made adap- tations for CIFAR- removed batch normalization in the projection head gs: we used SGD optimization with learning rate 0.06 for a batch size of 512, and used the MoCo parameters K 2048 and m = 0.99 for ResNet18 and K = 4096, sweeps, but did investigate larger values of K which did not lead improved performance on CIFAR-100. (In particular, the original settings K 65536, well on this dataset.) Other hyperparameters and settings were identical to Chen et al. (020cc. We did 3 independent runs with ResNet18 and 2 runs with a ResNet50. We conducted linear classi- fication evaluation with fixed representations in exactly the same way as for our other experiments. Feature averaging results used M 40.
the MoCo v2 (Chen al. 2020c
an identical way as in our previous experiments. As in MoCo v2, we
0.99 for ResNet50. We did not conduct extensive hyperparameter
0.999 appeared to perform less
F.2.6 COMPUTATIONAL COST
We found that gradient regularization increased the total time to train encoders by factor at most 2. For feature averaging at test time with fixed dataset, the computation of features (M) is an O(M) operation, whilst the training and testing of the linear classifier is O(1). Training time remained by far the larger in experiments by orders of magnitude.
F.3 ADDITIONAL EXPERIMENTAL RESULTS
F COMPARISON WITH ENSEMBLING
Feature averaging is approach that bears much similarity with ensembling. To experimentally compare these two approaches, we applied both approaches to encoders trained on CIFAR-10. To provide a suitable comparison with feature averaging using (MM) we first a trained a linear classi- fier p(y z) using an M-fold augmented dataset of representations with a standard cross-entropy loss using L-BFGS optimization using the same weight decay as for feature averaging. For CIFAR-10, which has training set of length 50000, the feature averaging classifier was trained using 50000 averaged representations, whereas the ensembling classifier was trained with 50000. examples using data augmentation. At test time, we averaged prediction probabilities using M different rep-
19

---

Published as a conference paper at ICLR 2021
0.250
0.255
* Feature averaging
Augiennatiinn ensembligg
0.175
20
00
20
40
60
80
102
M
(a) Accuracy
(b) Cross-entropy loss
Figure 8: A comparison between feature averaging and augmentation ensembling using representä- tions obtained with gradient regularization on CIFAR-10. Error bars are s.c. from runs.
6.22
$4.0
averaging Augieentation ensemblinn
93.5
0.175
(a) Accuracy
(b) Cross-entropy loss
Figure 9: A comparison between feature averaging and augmentation ensembling using representa- tions obtained with SimCLR on CIFAR-10, Error bars are s.e. from runs.
resentations of each test image. Specifically, if p(yzz) is the classifier trained by the aforementioned procedure and (.,,..., OKM p(())( fI,.... BM ~ p(33) are independent transformation parameters, the probability of assigning label y to input x given by
M
Ponsemble (v|x)
Σ
plylfo(t(x, am, Bm))).
(48)
M
The results outlined in Figure g show that ensembling gives very similar performance to feature averaging in terms of accuracy, but is significantly worse in terms of loss. We can understand this result intuitively because ensembling includes probabilities from every transformed version of the input (including where the classifier is uncertain or incorrect) whereas feature averaging combines transformations in representation space and uses only one forward pass of the classifier. More for- mally, the difference in test loss makes sense in light of Theorem T Figure 9 shows additional results obtained using representations trained with standard SimCLR CIFAR-10. We see the same pattern-a a similar test accuracy but worse test loss when using augmentation ensembling.
F.3.2 GRADIENT REGULARIZATIO LEADS STRONGLY INVARIANT REPRESENTATIONS
We first show that our gradient penalty successfully learns representations that have greater invariance to transfor- mation than their counterparts generated by contrastive learning. We consider two metries: the conditional vari-
Conditional voriance
0.6
0.4
No regularization Gradient regulartzation
ance targetted directly by the gradient regularizer, and the loss when is used to predict with linear regression. Table 7 and Figure Tr]are are the equivalents of Table [ and
0.3
0.1
Figure 2 for CIFAR-TO0, showing the conditional vari- ance and the regression loss for predicting a respectively. In T we present the same results for Spirograph. We see very similar results to CIFAR-10 in both cases-the gradient penalty dramatically reduces conditional vari- ances, and prediction of a by linear regression gives loss that is better than a constant prediction only for standard contrastive representations.
230
400
600
800
1000
Figure 10: The conditional variance of Equation on CIFAR-100. Error bars represent s.e. from 3 runs.
F.3.3 GRADIENT REGULARIZED REPRESENTATIONS PERFORM BETTER ON DOWNSTREAM TASKS AND ARE ROBUST TO TEST TIME TRANSFORMATION
For downstream performance on Spirograph, we evaluate the performance encoders trained with gradient regularization and without gradient regularization on the task of predicting the generative
20

---

Published as a conference paper at ICLR 2021
Table 7: The loss when linear regression model is used to predict a from on CIFAR-100. The reference value is Mean; Var(a;). We present the mean and + s.e. from 3 runs.
Test loss
No regularization Regularization
0.0357 0.0003 0.0417 + 0.00007
Reference value
0.0408
Table 8: Invariance metrics for Spirograph. We present the conditional variance, and the test loss when a linear regression model is used predict from z. The reference value is Mean, Var(o;;. We present the mean and + s.e. from runs.
Conditional variance 0.789 0.0069
Test loss
No regularization Regularization 0.0016 + 0.00004
0.0751 + 0.0003 0.0808 + 0.00009 0.0806
Reference value
parameters of interest. In our set-up, we use 100,000 train images and 20,000 test images and train the encoders the training set for 50 epochs. For evaluation, train a linear regressor on the representations from encoders to predict the actual generative parameters. Setting for the linear regressor is shown in the Table 6 To accompany the main results in Figure c), we include the exact values used in this figure in Table
We now turn to our experiments used to investigate robustness-we investigate scenarios when we change the distribution of transformation parameters a at test time, but use encoders that were trained with the original distribution. We investigate on both CIFAR and Spirograph datasets.
For CIFAR, we chose to vary the distribution of pa- rameters for colour distortions at test time. We could write the distribution of parameter of brightness, sat-
72.5
72.0
71.5
uration, contrast as U(1 0.8.51 + 0.85) and the distribution of hue ((-0.22, 0.2S) where S is a parameter controlling the strength of the distor- tion. In the original setup, we have S = 0.5. By varying the value of S used at test time, we can in- crease the variance of the nuisance transformations, including stronger transformations than those that present when encoders
71.0
70.5
70.0
69.5
No regulerization Gradient
60.0
01 02 0.3 0.4 0. 0.6 07 0.8 0.
Vartance sealing
trained. This is visualized in Figure II] Figure 14 is companion plot for Figure a) applied on CIFAR-100. We see broadly similar trends- our representations outper- form those from standard contrastive learning across a range of test time distributions.
Figure 14: Robustness of performance on CIFAR-100 under variance scaling of trans- formation parameters.
For robustness on Spirograph recall a Jgr Jo, by. by bb) when ~ U 2.5), fg. So U00.4, 1), br, by, by UO, 0.6). We chose to vary the distribution of the background colour (bp, byz bn) and the distribution of which is structure-related transformation parameter. We con- sider two approaches, mean shifting where shift the uniform distribution by S, for example U U(a.b) (a + S 5) and we consider changing variance where we increase the range of the
Table 9: The raw values used produce Figure Cc.. Each of the downstream tasks is generative parameters. Values are the test mean square error + s.e. from runs.
Parameters
No regularization
Gradient regularization 0.0005073 4- 0.0000078
m
0.0006773 + 0.0000509
b 0.0112480 + 0.0002555
0.0073607 + 0.0001079
0.0000914 + 0.0000108
0.0000527 + 0.0000026
fr
0.0000232 + 0.0000004
0.0000028 + 0.0000001
21

---

Published as a conference paper at ICLR 2021
(a) Scale 0.1
(b) Scale 0.2
(c) Scale 0.3
(d) Scale 0.4
Figure 11: Visualization of test time distortions applied to CIFAR-10 for various variance scalings.
(a) 1t = 0.5
(b) l: = 1.0
(c) = 1.5
(d) 1: = 2.0
Figure 12: The effect of varying h on the structure of Spirograph image.
(a) Shift 0
(b) Shift 0.15
(c) Shift 0.30
(d) Shift 0.45
Figure 13: The effect of shifting the background colour distribution used for Spirograph images.
uniform distribution by 2S, for example, U(a,b) + U(a - S S). We compare performance of the trained encoders epoch 50 predicting the generative parameters (m, b, o, f+) and the same setting for linear regressors as in Table
Figure 12 visualization of the effect of varying h from 0.5 2.0 while other parameters are kept constant. The figure 13 shows the effect of varying the background colour of an image by adding 0.15, 0.30, 0.45 to each of the background RGB channels.
For varying the distribution of h, we consider shifting the mean of h ~ U0055, 2.5) by S 10.3, #0.5 #0.5 and increasing the variance of by S 0.1, 0.3, 0.5. For the distribution of the background colour (b,, bg. bb), we consider shifting the distribution of (b,, by. bb) by S 0.1, 0.2, 0.3, 0.4 and increasing variance by the same amount. We note that (b, by. bb) controls the background colour of an image, we are varying the 3 distributions at the same time. Since, the foreground colour has the distribution fr, fg: fo U(0.4, 1), we are shifting the distribution of (br, by. 6p) toward (fr, fo. fo) and this will make the background and foreground colours more similar. For example, with S 0.4, when we apply a mean shift we change the distribution of (b,, bg. bb) by bg. U0.4, 1). and when increase the variance the distribution becomes by. by. by UO, 1).
22


Reviews:
[{"review_id": "kABLHqQ5Pxc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary\nGiven one image, the paper first generates different views which are controlled by differentiable parameter \\alpha, and then minimizes the additional \"conditional variance\" term~(expectation of these views' squared differences). Therefore, the paper encourages representations of the same image remain similar under the augmentation. A testing strategy is further proposed by voting features with different augmentations. Results demonstrate the effectiveness.\n\n2. Strength\n    * The proposed \"conditional variance\" and its first-order approximation which is driven by a learnable augmentation control factor \\alpha is a clear contribution，and makes it efficient to minimize the representations of different views of the same image. \n    * Results in Cifar and the new proposed dataset validate the effectiveness of the methods.\n\n3. Weakness\n    * Correct me if I were wrong. It seems more than half of the improvements come from the testing strategies. The claimed contribution of \"feature averaging\" during testing looks more like a testing time augmentation for me, although the paper has some differences in the augmentation details. I also think that the claim of SOTA is not rigorous if it has benefited from testing strategies.\n    * It seems the proposed methods can be plugged into many recent studies. Thus more comprehensive experiments are needed to validate its generalization, e.g., combining with other baselines and conducting on ImageNet.\n\n---------------------------\nAfter reading the authors’ feedback and comments from other reviewers, I raised my score from 5 to 6.  I agree the feature averaging is another testing strategy.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_PoHiGTLFAR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n- Proposes new method to improve transformation invariance in contrastive representation learning and demonstrates utility on downstream tasks\n- Proposes using feature averaging from multiple transformations at test time leading to further improvements \n- Introduces Spirograph dataset to explore the importance of learning feature invariances in the context of contrastive learning\n\nClarity\n- Paper very well written and easy to follow. Figures supplement the text well\n- Experiments include error bars to show statistical significance of results\n- Supplementary material clarifies experimental setup and very comprehensive\n- Perhaps consider changing “Self-supervized” -> “self-supervised”? \n\nNovelty/Significance\n- New contrastive objective with gradient regularizer term to encourage transformation invariance and the Spirograph dataset are well motivated \n- Results over contrastive baselines suggest the contributions are important improvements to the contrastive training recipe \n\nQuestions/Comments/Clarifications\n- “Unfortunately, directly changing the similarity measure hampers the algorithm.” - Please add a citation to validate this claim. Is it coming from the experiments in SimCLRv1 - Chen et al, 202\n\n- “However, there are many ways to maximize the InfoNCE objective without encouraging strong invariance in the encoder.” - Please add a citation to validate this claim\n\n- Eq 9 and 10:  (Fe(α, β, x) − Fe(α’ , t, x))^2. Should t not be β?\n\n- “It may be beneficial, however, to aggregate information from differently transformed versions of inputs to enforce invariance more directly” -> it is unclear why invariances need to enforced at test time if the learned representation is already invariant\n\n- The authors point this out but the gradient regularization term is unfortunately encouraging invariance only to differentiable \n transforms and this is a key limitation\n\n- Worth pointing out for certain tasks likely near OOD detection, you may want to be transformation covariant rather than invariant. Would be interesting to see results of the proposed method on OOD detection benchmarks following https://arxiv.org/abs/2007.05566\n- One major limitation is lack of baselines beyond vanilla contrastive training. For example, it would have been good to compare test time feature averaging with test time augmentation ensembling. Similarly instead of gradient regularization, the model could directly predict augmentation parameters and have the gradients of that loss penalized/ reversed. Adding such additional baselines, would solidify the improvements as best in class.\n- It is also unclear how much train/test time compute model adds.\n\nOverall, this is a very nicely written paper and very solid contribution. If the authors address my concerns, would be happy to increase my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IAfGi7oYcfk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on a subtle different yet seemingly novel approach to learn invariance, i.e. instead of forcing the representations of two transformed inputs to be similar, it minimizes the gradient of the representation w.r.t to the transformation. This is indeed closer if not exactly the definition of tolerance or invariance to transformations is.\n\nThe problem is certainly an important one. The paper uses trick to to use a different representation of the conditional variance and first order approximation to arrive at a more computationally viable formulation. The formulation overall is interesting and described in sufficient detail. The paper does provide some relations to adversarial robustness as well, indeed it is quite intuitive to find connections between robustness and invariance.\n\nThe paper also provides validation experiments to show that indeed the conditional variance is reduced using this regularization.  However, there seems to be missing plots depicting this difference as a function of lambda. Is the method highly sensitive to lambda? or does a large range provide similar effects? If its the latter case, why so? what is preventing a large decrease in conditional variation given this loss augmentation?\n\nAveraging across test samples does seem to help significantly. This is an important property in practice when there might be multiple transformations available that one needs to optimally utilize. This is one solution for such a case.\n\nHow would these ideas apply to more complicated nuisance transformations such as object pose?\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper recieved three consistently positive reviews. While I agree with most of them, I have two major concerns regarding the novelty of the paper, which the authors are strongly recommended to address in the final version.

1. Taking derivative with respect to the parameters of transformation isn't novel. The standard tangent prop algorithm has been around for over a decade:

P. Simard, Y. LeCun, J. S. Denker, and B. Victorri. Transformation invariance in pattern recognition-tangent distance and tangent propagation. In Neural Networks: Tricks of the Trade. 1996.
(see Eq 26 in https://halshs.archives-ouvertes.fr/halshs-00009505/document)

Salah Rifai, Yann N. Dauphin, Pascal Vincent, Yoshua Bengio, Xavier Muller. The Manifold Tangent Classifier. NIPS 2011.
(see Eq 6 therein)

I understand there is some normalization in Eq 5, sampling of \alpha, \alpha', and the direction, and using the expectation to approximate the norm of the gradient. But such novelty is really incremental, or at least some empirical comparison will be necessary. It will also be necessary to cite the tangent distance/prop literature.

2. The new gradient based regularizer in Eq 11 and 12 appears completely decoupled with contrastive learning. It can be applied to any representation learning where f_\theta is an encoder. It does not use any substantial element from contrastive learning, although it might be "inspired" by contrastive learning. One may argue that such generality is an advantage, but 1) there is really no need to take such a big detour into contrastive learning just in order to derive the invariance regularizer in Eq 12, and 2) writing in this way can be quite confusing and/or misleading.
Decision: Accept (Poster)