Explicit homography estimation improves contrastive self-supervised learning
Authors: David Torpey
Richard Klein
Venue:  (2021)

Abstract: The typical contrastive self-supervised algorithm uses a similarity measure in latent space as the supervision signal by contrasting positive and negative images directly or indirectly. Although the utility of self-supervised algorithms has improved recently, there are still bottlenecks hindering their widespread use, such as the compute needed. In this paper, we propose a module that serves as an additional objective in the self-supervised contrastive learning paradigm. We show how the inclusion of this module to regress the parameters of an affine transformation or homography, in addition to the original contrastive objective, improves both performance and rate of learning. Importantly, we ensure that this module does not enforce invariance to the various components of the affine transform, as this is not always ideal. We demonstrate the effectiveness of the additional objective on two recent, popular self-supervised algorithms. We perform an extensive experimental analysis of the proposed method and show an improvement in performance for all considered datasets. Further, we find that although both the general homography and affine transformation are sufficient to improve performance and convergence, the affine transformation performs better in all cases.

Under review as a conference paper at ICLR 2021
EXPLICIT HOMOGRAPHY ESTIMATION IMPROVES CON- TRASTIVE SELF- SUPERISDD LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The typical contrastive self-supervised algorithm uses similarity measure in la- tent space as the supervision signal by contrasting positive and negative images directly or indirectly. Although the utility of self-supervised algorithms has im- proved recently, there are still bottlenecks hindering their widespread use, such as the compute needed. In this paper, we propose module that serves an ad- ditional objective in the sulf-supervised contrastive learning paradigm. We show how the inclusion of this module to regress the parameters of an affine transfor- mation or homography, in addition to the original contrastive objective, improves both performance and learning speed. Importantly, we ensure that this module does not enforce invariance to the various components of the affine transform, as this is not always ideal. We demonstrate the effectiveness of the additional objec- tive on two recent, popular self-supervised algorithms. We perform an extensive experimental analysis of the proposed method and show an improvement per- formance for all considered datasets. Further, we find that although both the gen- eral homography and affine transformation are sufficient to improve performance and convergence, the affine transformation performs better in all cases.
INTRODUCTION
There is an ever-increasing pool of data, particularly unstructured data such as images, text, video, and audio. The vast majority of this data unlabelled. The process of labelling is time-consuming, labour-intensive, and expensive. Such an environment makes algorithms that can leverage fully unlabelled data particularly useful and important. Such algorithms fall within the realm of unsuper- vised learning. A particular subset of unsupervised learning is known as Self-Superviser Learning (SSL). SSL is a paradigm in which the data itself provides a supervision signal to the algorithm.
Somewhat related is another core area of research known transfer learning (Wang et al. 2020 In the context computer vision, this means being able to pre-train an encoder network offine on large, varietal dataset, followed by domain-specine fine-tuning the bespoke task at hand. The state-of-the-art for many transfer learning applications remains dominated by supervised learning techniques (Tan et al., 2020, Martinez et al., 2019 Donahue et al. 2014 Girshick al. 2014), in which models are pre-trained on a large labelled dataset.
However, self-supervised learning techniques have more recently come to the fore potential al- ternatives that perform similarly on downstream tasks, while requiring no labelled data. Most self- supervised techniques create a supervision signal from the data itself in one of two ways. The one approach are techniques that define a pre-text task beforehand that a neural network is trained to solve, such as inpainting (Pathak et al. 2016) or jigsaw puzzle (Noroozi & Favaro 2016). this way, the pre-text task is a kind of proxy that, if solved, should produce reasonable representations for downstream visual tasks such as image or video recognition, object detection, or semantic seg- mentation. The other approach is class of techniques known contrastive methods (Chen et al.) 2020a He et al.) 2019) Chen et al. 2020b). These methods minimise the distance (or maximise the similarity) between the latent representations of two augmented views of the same input image, while simultaneously maximising the distance between negative pairs. In this way, these methods enforce consistency regularisation (Sohn et al., 2020), well-known approach to semi-supervised learning. These contrastive methods often outperform the pre-text task methods and are the current state-of-the-art in self-supervised learning. However, most of these contrastive methods have several

---

Under review as a conference paper at ICLR 2021
drawbacks, such as requiring prohibitively large batch sizes memory banks, in order retrieve the negative pairs of samples (Chen et al. 2020a He et al. 2019).
The intuition behind our proposed module is that any system tasked with understanding images can benefit from understanding the geometry of the image and the objects within it. An affine transformation is geometric transformation that preserves parallelism of lines. It can be composed of any sequence of rotation, translation, shearing, and scaling. homography a generalisation of this notion include perspective warping. A homography need not preserve parallelism of lines, however, it ensures lines remain straight. Mathematically, homography is shown in Equation vector homogenous coordinates. An affine
I It has 8 degrees of freedom and is applied transformation has the same form, but with the added constraint that Φ3,1 03,2 =
11,1
11.2
11.3
H =
21
22
$2.3
Ø3,1
Ø3,2
The ability to know how a source image was transformed to get to a target image implicitly means that you have learned something about the geometry of that image. An affine transformation or, more generally, a homography is a natural way to encode this idea. Forcing the network to estimate the parameters of random homography applied to the source images thereby forces it learn semantics about the geometry. This geometric information can supplement the signal provided by contrastive loss, or loss the latent space.
In this paper, we propose additional module that can be used in tandem with contrastive self- supervised learning techniques to augment the contrastive objective (the additional module high- lighted in Figure D. The module is simple, model-agnostic, and can be used to supplement a con- trastive algorithm to improve performance and supplement information learned by the network to converge faster. The module is essentially an additional stream of the network with the objec- tive of regressing the parameters of an affine transformation or homography. In this way, there is a multi-task objective that the network must solve: 1. minimising the original contrastive objective, and 2. learning the parameters of a homography applied to one of the input images from a vector difference of their latent representations. We force the latent space to encode the geometric trans- formation information by learning to regress the parameters of the transformation in an MLP that takes the vector difference of two latent representations of input, æ. and its transformed analogue, x.. By including the information in this way, the network is not invariant to the components of the transformation but still able to use them as supervise signal for learning. Moreover, this approach serves as novel hybrid of the pre-text tasks and contrastive learning by enforcing consistency regularisation (Sohn et al. 2020).
4
A1
C$
MSE
NT-Xent
Figure 1: Proposed architecture. The highlighted box highlights the additional proposed module tasked with regressing the parameters of an affine transformation or homography.
Through extensive empirical studies, we show that the additional objective of regressing the trans- formation parameters serves useful supplementary task for self- self- superviedd contrastive learning, and improves performance for all considered datasets in terms of linear evaluation accuracy and convergence speed.
The remainder of the paper is structured as follows. In Section 2 the area of seff-supervised learning, going into detail where necessary. In Section we detail
we cover the related work in
2

---

Under review as a conference paper at ICLR 2021
proposed method. We first introduce framework and set of notation to make the formalisation of the approach clear. We then delve into the details behind the architecture and choices for the various part of the system. This is followed by comprehensive set of experiments in Section [ including results of various datasets, as well as an ablative study. Finally, the paper is concluded with some closing remarks in Section 5
2 RELATED WORK
SSL is popular research area within computer vision. Previous approaches can be broadly classed into two main categories. The first is where pre-text tasks are manually defined, and the goal of the algorithms is to solve these hand-crafted tasks (Lee al. 2020 Doersch et al.] 2015 Gidaris al.) 2018 ZZang et al. 2016 MMisa & Maaten (020). Examples of such methods include inpainting (Pathak al.] 2016). 6). colourising (Zhang al.j 2016), jigsaw puzzles (Noroozi & Favaro, 2 patch prediction (Doersch et al., 2015], and geometric image transformations (Dosovitskiy et al., 2014 such as using rotation as the pre-text task (Gidaris al., 2018 Feng et al. 2019). Some of these pre-text approaches that deal with geometric image transtormations are similar in spirit to our method. Gidaris et al.) (2018; Feng et al. (2019) are two variants of predicting image rotations as an auxiliary task for unsupervised learning. Perhaps closer to our method is Dosovitskiy et al. (2014), in which a set of transformations is applied to image patches, and the network is trained in uully-unsueerviied manner to predict surrogate classes defined by a set of transformed image patches by minimising the log loss. Our method, however, investigates a different, particular set of transformations (those that define affine transformation of general homography), and show this can be used to aid self-supervised performance, using the transformation parameters themselves as targets that need to be regressed (using mean-squared error) by the contrastive algorithm in a multi- task manner. The discrepancy in the network's is ability to predict the actual values of the parameters of the affine transformationtoomgrrayy serves as our additional supervision signal.
A somewhat related approach to our proposed method within the pre-text task domain is proposed by Lee et al. (2020). They propose augment the learning process of supervised learning algorithm with additional labels constructed using self-supervised labels. These labels are rotation classes and colour permutations. Importantly, they create a loss function which is based on a joint distribution of the original (supervised) labels and the self- supesuueee (augmented) labels. In this way, the network not forced be invariant the transformations under consideration, since this has been shown to hurt performance (Lee et al. 2220)) Our method is different to this in that we propose module to be integrated specifically with self-supervised algorithms. Additionally, we regress the transformation parameters in real vector space and do not create classes for the parameters.
The other broad category of SSL is based on contrastive learning (Chen et al. 2020a He et al, 2019 Caron et al. 2020), and this class of techniques represent the current ofate-ofthheee in self- supervised learning, outperforming the hand-crafted pre-text task methods. These approaches learn representations by contrasting positive pairs of samples from negative pairs of samples in latent space. Such methods typically require that careful attention be paid to the negative samples. Ad- ditionally, they have the disadvantage of requiring prohibitively large batch sizes (4096-16000), memory banks, or other mechanisms to retrieve the relevant negative samples.
One popular such method is known as SimCLR (Chen al. 2020a).. SimCLR is a general frame- work for contrastive learning, and in its vanilla formulation consists of an encoder network param- eterised by a CNN (usually variant of ResNet (He et al. 2016)) and MLP projection head. An input image is sampled, and two distinct views of that same input image are computed using ran- dom augmentation. The augmentation consists of colour jiterring, Gaussian blurring, and random cropping. The two views are sent through the encoder network to produce two latent representa- tions. These latent vectors are then sent through the projection head to produce final latent vectors. It is from these vectors that the loss is computed. In the case of SimCLR, the loss is normalised Cempperte
temperatured cross-entropy (NT-Xent).
A recent approach proposed in Grill al. (2020) (BYOL) somewhat overcomes the aforementioned disadvantages of requiring negative pairs of samples (which implicitly requires a large batch size). Two separate networks with their weights used in tandem to learn the representation. An online network (consisting of an encoder, MLP projection head, and MLP prediction network) is trained to predict the representation outputted by a target network. During training, the online
3

---

Under review as a conference paper at ICLR 2021
network parameters updated using backpropagation of error derivatives computed using mean- squared error loss. However, the target network parameters are updated using an exponential moving average. In this way, BYOL overcomes collapsed solutions in which every image produces the same representation. We test our module with both SimCLR and BYOL, since these two methods serve two popular, recent approaches to contrastive SSL..
Some helpful findings for guiding self-supervised research were demonstrated in Kolesnikov et al. (2019). Core among these are that 1) standard architecture designs that work well in the fully- supervised setting do not necessarily work well in the self-supervised setting, 2) in the self- supervised setting larger CNNs often means higher quality learned representations. and 3) the linear evaluation paradigm for assessing performance may take long time to converge. Moreover, NNwell Deng (2020) find that the effectiveness of self-supervised pretraining decreases as the amount of labelled data increases, and that performance on one particular downstream task is not necessarily indicative of performance on other downstream tasks.
PROPOSED METHOD
We first introduce a mathematical framework for discussing our method. Let B1 be set of base transformations. A base transformation is transformation that cannot be decomposed into more basic transformations and is interpreted as per Grill et al. (2020); Chen et al. (2020a). Examples of base transformations include colour jittering. cropping, and horizontal flipping. We define the possible base transformations a-priori, and |B1| < cc. Next, we define a new set of base spatial transformations B3 that correspond the general affine transformations (i.e. rotation, translation, scaling and shearing) or the full homography (i.e. affine transformations and perspective projection). Further, impose the following condition:
B1 n B2 0
The reason for this restriction will be apparent later.
A transformation to.e is parameterised by its associated base transformation b E B1 U B2 and trans- formation parameters E O Then, the set of all possible transformations for a particular base transformation set B may be defined as:
T (to,olb B, 0 e θ}
(3)
Clearly, we may have that [T]] 30, since some parameters may take on any value within compact subsets of IR. This is important because we want to be able to sample from an infinite sample space during training to ensure the network sees a variety samples.
We can now define augmentation, which is an ordered sequence of n transformations. As such, each unique ordering will necessarily produce a unique augmentation (e.g. flipping and then crop- ping is diffeerntt from cropping and then flipping). Formally, augmentation a is defined as:
a x 10 ... lin li B, (x)
(4)
Denote the set of all possible augmentations for transformation set T, as AT,. Under this defini- tion, Ar, is the set of all possible affine or homographic transformations. Examples of the affine transformations and homographies can be seen in Appendix A
Now, consider an input image r sampled at random from dataset of images X C X, where X' is the sample space of images. We sample augmentations a, b E AT,, and apply them to x to produce augmented views #1 and 22, respectively. We then sample an affine/homographic transformation Co E AAz and apply to 27 to produce a/{. Note that æ1 and a/ are related by a homography. This assumption relied upon by further inductive biases introduce into model.
We now describe the proposed architecture as depicted in Figure [ be parameterised by CNN, and the mappings y -2 R and I2P - R. be parameterised by MLPs, where p. k. and m are the dimensionality of the encoder latent vector, projection head latent vector, and homography parameter vector, respectively. of and are the encoder and projection head from the original SimCLR (Chen et al., 2020a) and BYO (Grill et al.) (020) formulations, respectively, whereas is new MLP tasked with estimating the homography parameters. Note that
Let the mapping f X RP
4

---

Under review as a conference paper at ICLR 2021
if we regressing all parameters of a general affine transformation, then 6, whereas for full homography we have m = 8. For brevity, we have denoted both streams in the architecture to be a network with the same shared weights, although it may be the case that the two streams consist of networks with different weights (as is the case with BYOL).
The loss function for our method contains two terms. First is the original loss function defined by the original method: NT-Xent for SimCLR and squared Lg for BYOL. We define this first term as Li (z1; 22), where ž1 = g( (x) )) and 22 = (x(())) The second term be seen as forcing the network to explicitly learn the affine transformation or homography between 2) and z4. Let the latent representations of 21 and z be 1 f(x1) and f(z)). We send the vector difference I 'f through h to produce an estimate of the homography's parameters. We regress to these parameters using mean-squared error: (2(h((1 - li); Φ), where are the ground truth affine transformation parameters. Thus, the complete loss function is given by:
Li(21, 22) + C2(h(11 ())
(5)
The vector difference naturally describes the transformation needed move from 1 to I1- With architecture and learning objective, we force this vector difference transformation vector to encode the homography between 1 and z/. This interpretation may be seen as natural and intuitive. Hence, the L term enforces invariance to the transformations in B, and L2 enforces non-invariance to the transformations in B2-
Note that this is still completely self-supervised. Moreover, the restriction imposed in Equation iis necessary because we cannot have any transformations in co 's sequence that would destroy the fact that Iı and z' are related through a homography. For example, adding cropping transformation would break the homography assumption. One may add transformations that do not break this restriction (e.g. colour jitter), however, we do not explore this here.
We may interpret this extended architecture as solving multi-task learning objective in which 1) the contrastive loss between differing augmented views of the image must be minimised, and 2) another network must be able to estimate the homography between images, which explicitly forees the latent space to encode this spatial information during training.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
This section presents an empirical study comparing the original SimCLR and BYOL techniques on the CIFARIO, CIFAR 100 and SVHN benchmark datasets, with and without our proposed mod- ule. Our goal is not to achieve near state-of-the-art performance on the datasets, but is rather to demonstrate the effectiveness of the proposed additional homography estimation objective under consistent experimental settings. In all cases, the proposed module improves the performance of a linear classifier the learned representation and improves the learning speed.
The experimental setup for the self-supervised training of SimCLR and BYOL can be found in Table The batch size is somewhat lower than the original methods since the original methods focused on performance on ImageNet, which requires a considerably larger batch size to perform well. In some additional experiments, sizes larger than 256 for all methods (original SimCLR and BYOL, as well as our method). Further, we found alternative optimised hyperparameter values (learning rate, optimiser, and weight decay) that worked better than those proposed in the original formulations of SimCLR and BYOL, which can be attributed to similar reasons as the batch size arguments. We use the same type of learning rate decay as the previous methods, and train for the same number of epochs (and warmup epochs) as SimCLR. We use a temperature of 0.5 for the NT-Xent loss and keep all images at their default resolution of 32 32. Lastly, all reported confidence intervals are the average across 10 trials of the full pipeline trained from scratch (SSL pretraining + linear evaluation).
find performance decreased for our datasets with batch
Performance is measured per the literature, using linear evaluation the relevant dataset. The experimental setup for linear evaluation can be seen in Table ! optimise the weights of a final linear layer using cross-entropy.
We freeze the encoder and only
5

---

Under review as a conference paper at ICLR 2021
Table 1: Experimental setup.
SSL
Linear Evaluation
SimCLR
BYOL
SimCLR
BYOL
Batch size
256
256
Batch size
64
64
Optimiser
Adam
SGD
Optimiser
Adam
Adam
LR
3e-04
0.03
LR
3e-04
3e-04
Momentum
0.9
Epochs
200
200
Weight decay
10e-06
4c-04
Hardware
Epochs (warmup)
100 (10)
100 (10)
V100 16GB GPU
LR schedule
Cosine decay
Cosine decay
We parameterise as ResNer50, while g and are parameterised as two-layer ReLU MLPs (Figure D. Further, to ensure consistency with SimCLR, we have that B frandom crop, random horizontal flip, colour jitter, Gaussian blur, random grayscale). The output of h is a six-dimensional real vector, where the six components
defined according to the parame- ters of a general affine transform: 1) rotation angle, 2) vertical translation, 3) horizontal translation, 4) scaling factor, 5) vertical shear angle, and 6) horizontal shear angle. For homography, the output of h is instead an eight- ligh--oimennional vector. For details about the transformations, see Appendix A
4.2 AFFINE AND HOMOGRAPHY OBJECTIVE
From Tables 2 and (++ H' and '+ A' for homography and affine, respectively) we can see that the estimation of the affine transformation and the homography both assist performance and allow for faster learning. In particular, we note statistically significant improvements across all datasets for both SimCLR and BYOL with the affine objective We posit that the ability explicitly estimate the affine transformation or homography between input images in this way allows the encoder to learn complementary information early on in training that is not available from the contrastive su- pervision signal. The ability to estimate the affine transform or homography means that the network encoding the geometry of the input images. This explicit geometric information is not directly available from the contrastive signal. Interestingly, the affine objective outperforms the full homog- raphy in all cases, even though an affine transformation is a subset of a homography. We perform sweep of the distortion amount for the homography and find it consistently performs similar to or a little worse than the affine transform (see Appendix B When the distortion factor becomes too large, accuracy drops noticeably as the images too distorted to learn effectively. We note that incorporating our module into a network results in an average 30% additional training time versus the respective original methods.
Table Performance with SimCLR on various datasets (mean + 99% confidence interval).
CIFARTO
CIFAR100
SVHN
SimCLR
63.34 + 0.0016
28.53 = 0.0017
83.19 = 0.0017
SimCLR + H
64.04 0.0029
29.10 = 0.0025
82.37 = 0.0024
SimCLR A
64.71 + 0.0023
31.33
0.0024
83.85
0.0017
Table 3: Performance with BYOL on various datasets (mean + 999 confidence interval).
CIFARIO
CIFAR 100
SVHN
BYOL
56.56 0.0026
23.25 0.0026
72.34 + 0.0047
BYOL + H
58.78 - 0.0033
25.88 + 0.0029
76.56 + 0.0054
BYOL + A
60.19 + 0.0016
28.10 + 0.0018
78.71 + 0.0050
Figure 2 shows the linear evaluation accuracy trained on the embeddings extracted from the model each epoch during the SSL training. We can see that performance and convergence improves with the inclusion of the proposed module. The module and its accompanying additional objective of
Using a t-test and significance level of 1%

---

Under review as a conference paper at ICLR 2021
CIRAR1 SIIG
CIRARO BYGL
CIARRO0 SMOI
CIAAIDO
0.7
CEARDO2 Durs
CIRRRDO Ours
Livor Evalustor Accuna
SAN Ours
0.4
SVIRV
0.3
0.-
011
300
100
tpoch
tpoch
(a) SimCLR results (measured every 10 epochs).
(b) BYOL results (measured every 10 epochs).
Figure 2: Graph of the linear evaluation accuracy at various points during training for both Sim- CLR and BYOL with and without our proposed module. The shaded region indicates one standard deviation from the mean.
regressing the affine transform/homoorrphy may be seen as regulariser for the original contrastive objective. This is further evidenced by the shaded regions in the figures, in which the proposed method results in more stable performance.
We note that the relative benefit of our proposed module diminishes with longer training time for SimCLR and BYOL. This makes sense as the relative benefit of the module decreases with time the model learns to estimate the affine transformation epochs progress. We performed additional experiments on SimCLR and BYOL training the model for longer, and note that the proposed module still outperforms performs similarly the original methods on all datasets. This is shown in Table These results also verify the findings of previous works that find that larger models trained for longer benefits self-supervised architectures (Chen)
homography more accurately as the
et al. 2020a: Grill et al. 2020; Chen et al. 2020b Kolesnikov et al. 2019).
Table Performance comparison of SimCLR and BYOL on various datasets trained for 500 epochs.
CIFARIO
CIFARI00
SVHN
SimCLR SimCLR + A BYOL
76.96 + 0.0031
42.17 + 0.0001
88.37 + 0.0030
76.80 + 0.0020
42.53 + 0.0014
88.83 = 0.0015
78.00 0.0060
38.05 = 0.0007
90.78
0.0067
B
77.45 + 0.0020
40.91 = 0.0005
90.74 = 0.0025
We note that the performance gap between SimCLR and BYOL evident in Tables and i general can be attributed the fact that in the original works, BYOL was trained for 10x long as SimCLR, whereas we trained both for the same number of epochs as the original SimCLR work. We posit that BYOL has simply not converged sufficiently, since BYOL eventually outperforms SimCLR evidenced by 4). This is consistent with the findings from the original works.
4.3 INVARIANCE IS NOT ALWAYS DESIRABLE
In order for a function f to be invariant to a transformation T, we must have that, for all x, fx) = f(Tx). Thus, one way to encourage invariance to 7 in neural network is to add term to the loss function which minimises:
L(f(x), fTTx))
(6)
for some measure of similarity L. If we rewrite our loss function from Equation 5 in terms of our input image and augmentations ca. we get:
+ Lz(h(((az) - ((((f(aa))f(cazz)))) ()
The first term of the above loss. corresponding the SimCLR/BYOL loss, is clearly of the form of Expression This means that we are encouraging our representation to be invariant to the transfor- mations within B, However, the second term in the loss (i.e. the term corresponding to the affine

---

Under review as a conference paper at ICLR 2021
transfrmation/homograiyy parameter estimation) is not of the form of Expression since we have recast the objective into a parameter prediction task. Thus, we are not encouraging invariance to the transformations within B2. We provide some empirical evidence for this in Table When we cast the module to minimise T.2 (x1),, f(x(() (I.2 being the mean squared error loss), performance decreases notably on all datasets, with average relative decrease of over 8%. This is because, with this loss, we have enforced invariance to the transformations in B2. In particular, we have encouraged invariance all the elements of affine transformation, which proves problematic.
Table 5: Performance for SimCLR for transformation invariant and non-transformation invariant representations using the affine objective.
CIFARIO
CIFARI00 26.13 + 0.0074
SVHN
'6" vs '9'
Invariant
61. .1.43 0.0053
80.32 + 0.0047
68.64 + 0.0085
Not Invariant
64.71 0.0023
31.33 0.0024
83.85 + 0.0017
72.35 + 0.0058
To delve deeper into the effect of transformation invariance on performance, we extract only the and "9' classes of the SVHN dataset as new dataset and repeat the SSL pre-training and linear evaluation tasks. The goal of this experiment is to observe how performance degrades when the neural network is encouraged be invariant to certain transformations including rotation in setting where certain invariance (i.e. rotation) is not desirable. Results can also be seen in Table 5 This further suggests that invariance certain transformations is not always desirable. Evidence from Table suggests that transformation invariance (for this particular class of transformations) in SSL may not always be desirable, and may, in fact, hurt performance, even when this may not be expected (as in the case with CIFARI0 and CIFARI00 100, since no classes of these seem as if they should be affected by transformation invariance like the '6' vs '9' case). For more details on the invariance analyses, see Appendix C
4.4 TRANSFORM COMPONENT ANALYSIS
Table 6 shows the performance of the various components of an affine transformation in terms of linear evaluation accuracy on the dataset F mapping h needs to be changed accordingly. Namely, rotation, translation, scale, and shear have corresponding output dimensionalities m of 1, 2. 1, and 2, respectively. Interestingly, shear alone outperforms the three other transforms all datasets for both SimCLR and BYOL. We hypothesise that this is because shear corrupts the image the most out of the four transforms, but still a recognisable way. This forces the networks learn more complex geometry and information about the object that the other transforms. We leave further investigation of this to future work.
To compute these results, the output dimensionality of
Table 6: Performance comparison of the components of an affine transformation for SimCLR and BYOL. Best- peestteerfoming transformation highlighted in bold for each dataset.
SimCLR
BYOL
CIFARI0
CIFARI00
SVHN
CIFARIO
CIFAR100
SVHN
Rotation
63.57 + 0.0031
29.40 1 0.0035
83.46 1 0.0036
58.66 : 0.0020
26.46 = 0.0042
76.34 = 0.0073
Translation
64,36 + 0.0036
29.83 1 0.0008
82,60 1 0.0053
58.45 : 0.0038
25.74 = 0.0022
77.50
0.0059
Scale
64.05 0.0035
30.34 1 0.0027
82.80 0.0037
59.12 0.0034
25.47 = 0.0023
76.53 0.0102
Shear
64,56 1 0.0051
31.05 1 0.0033
84.25 0.0020
60.21 0,0048
26.94
0.0038
77.69 0.0059
4.5
ADDITIONAL ABLATIONS
We perform various additional experiments to motivate the choice of architecture. We experiment with other means of encoding the latent transformation, specifically, concatenation instead of vector difference. However, this results in marginal performance gains of average of 0.28 percentage points across the 3 datasets. These results do not seem to justify the noticeable additional compu- tational cost from the transformation representation being twice the size. For primarily this reason,
2 Due to apparent instability in training BYOL with our module for these Jow-dimensional outputs (e.g. single real value output for rotation and scale), we temporarily replace MSE with logcosh, which stabilises training this setting
8

---

Under review as a conference paper at ICLR 2021
opt stick with vector difference. Further, experiment with having the module operate on the output of g instead of f. Performance degrades for all datasets (SimCLR): 64.38 for CIFARIO, 29.99 for CIFARI00, and 82.49 for SVHN. Lastly, we perform some preliminary experiments into having two modules: one operating on 21 and the other operating on i2 (instead of just one module as per our original experimental setup). The resulting performance difference is negligible with this setup: CIFARIO 10 65.28 + 0.61, CIFAR 100 31.68 + 1.04, and SVHN 84.10 = 0.23. We posit that this is because if the one module can solve the homography estimation for 3 then module oper- ating on will have to be able to solve the homography estimation for it, since the same types of random homographies/affine transformations are being applied both streams.
CONCLUSION
Network size and time of training is bottleneck in modern SS architectures that can compete on a performance level like supervised alternatives. We have shown that the proposed module that regresses the parameters of an affine transformation or homography sists this training bottleneck with faster convergence and better performance. The architecture of the module does not encourage invariance to the affine or homographic transformation, invariance has been previously shown to be potentially harmful (Lee et al. 2020). Rather, the proposed module en- courages these transformations to be encoded within the latent space itself by directly estimating the parameters of the transformation. Lastly, we note that the affine transformation performs better in all cases than the full homography, even though the homography is superset of affine transformations. The experiments suggest that the additional ability of perspective transform in homography does not yield any tangible benefit over a regular affine transformation in such low-resolution settings.
additional objective as-
REFERENCES
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey F. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Carl Doersch, Abhinav Gupta, and Alexci A. Efros. Unsupervised visual representation learning by context prediction. 2015.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang. Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. volume 32 of Proceedings of Machine Learning Research, pp. 647-655, 2014.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 1, NIPS' 14, pp- 766-774, 2014.
Z. Feng. C. Xu, and Tao. Self-supervised representation learning by rotation feature decoupling. In 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10356 10366, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ArXiv, abs/1803.00728, 2018.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580-587, 2014.
Jean- Man-BBstien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A approach to self-supervised learning, 2020.

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2019.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting supe-supe visual repre- sentation learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929, 2019.
Hankook Lee, Sung Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transformations, 2020.
Brais Martinez, Davide Modolo, Yuanjun Xiong, and Joseph Tighe. Action recognition with spatial- temporal discriminative filter banks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
I. Misra and L. D. Maaten, Self-supervised learning of pretext-invariant representations. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6706-6716, 2020.
Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? pp. 7343-7352, 06 2020.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. coders: Feature learning by inpainting. 2016.
Context en-
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint 20X1v:2001.07685, 2020.
Mingxing Tan, Ruoming Pang, and Quoe V. Le. Efficientder: Scalable and efficient object detection. In Proceeding of the 2020 IEEE Conference Computer Vision and Pattern Recognition, 2020.
Kafeng Wang, Xitong Gao, Y. Zhao, Xingjian Li, D. Dou, and C. Xu. Pay attention to features, transfer learn faster cnns. In ICLR, 2020.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
A DATA AUGMENTATION DETAILS
Tables and g detail the parameter values and value ranges used in the experiments for the base transformation sets B, and B2, respectively. The transformations from B1 are applied with a speci- fied probability. We also normalise the parameters of the affine transformations in the following way. Consider a rotation angle a, translation values tz ty. and shear angles $2, Sy. We perform the fol- lowing normalisation on these parameters: == a e/360; t, := fz/ /::Wt := tty H;; 8z = s smxxx 8y == Sy/ Smux, where H, W are the image height and width, respectively, and 8max the maximum allowed shear.
B AFFINE VS HOMOGRAPHY
In addition to the perspective distortion factor of 0.5, perform sweep across this parameter for the values {0.1, 0.2, 0.8). The results can be seen in Table 9 Interestingly, most distortion factors these datasets, with distortion factor of 0.5 performing best average. However, when the factor gets too large, as is the case for 0.8. then the images become too corrupted for the neural network to seemingly learn anything useful.
perform similarly
10

---

Under review as a conference paper at ICLR 2021
Table Parameter values for base transformation B1.
Transformation
Parameter Value
Probability
Colour Jitter (brightness)
0.8
0.8
Colour Jitter (contrast)
0.8
0.8
Colour Jitter (saturation)
0.8
0.8
Colour Jitter (hue)
0.2
0.8
Random Resized Crop
(0.08, 1.0)
Random Horizontal Flip
0.5
Random Grayscale
0.2
Gaussian Blurring (kernel size)
3
Gaussian Blurring (variance)
[0.1,2]
Table 8: Parameter values for base transformation set B2.
Base Transformation
Parameter Value
Rotation
-00, 90
Translation (æ: and y)
0%, 25%
Scaling
0.7, 1.3
Shear and y)
25, 25)
Perspective
0.5
Table Sweep across the distortion factor for the homography using SimCLR.
CIFAR10
CIFAR100
SVHN
Factor 0.8
57.17 = 0.0319
23.13 = 0.0089
78.67
Factor = 0.5
64.04 + 0.0029
29.1 + 0.0025
82.38 + 0.0024
Factor 62.82 + 0.0094
29.39 + 0.0028
82.73
0.0030
Factor 0.1
62.45 = 0.152
29.07 = 0.0035
82.89 + 0.0014
C INVARIANCE ANALYSIS
Figures aaddd 5 shows the confusion matrices for a particular run of the SVHN dataset for SimCLR when enforcing (and not enforcing) transformation invariance. Interestingly, transformation invari- ance negatively affects most classes of the dataset. Unsurprisingly, the classes '6' and '9' are most negatively affected when transformation invariance enforced. Rotation invariance in this context is prohibitive, and performance subsequently drops. By recasting the module as proposed encode the transformation and predict its parameters. do enforce invariance, and instead allow the network to learn from a richer supervision signal by learning to estimate a homography.

---

Under review as a conference paper at ICLR 2021
(b) CIFARIO affine transforma- tion.
(a) CIFARIO original.
(c) CIFARIO homography.
(e) CIFAR100 affine transforma- tion.
(d) CIFARI00 original.
(f) CIFARI00 homography.
19
19
(g) SVHN original.
(h) SVHN affine transformation.
(i) SVHN homography.
Figure 3: Example affine transformations and homographies for all considered datasets.
6.2059
0.1644
2.344
0.555
0.1034
5.0682
0:0044
0:0046
9.024
0.001
0355
01.
0.2044
0.054
6.8522
0.00011
0.000
3,00744
0::2114
0111
0:12
5.001
0:71
3.072
0:0001
01.
2.014
0.8222
0.5027
3,015
0.0553
0.2244
0.015
0.0495
3.022
0.21
0.72
5.1
Figure Confusion matrix on SVHN when enforeing transformation invariance.
6.5522
0.2052
0.20234
0.5036
0.0232
2.112
61
0255
2.004
5.025
0.0044
0:244
9/027
0
0
0.6177
.015
1.055
1.000
0.5
8:22-14
007
01 8:2244
5.115
0.47
8-2212
0.775
0:2278
0.5244
0:58
3,003
5.023
3.012
3.013
2.002
0:0211
0.0222
0.92
0.0422
1.033
Figure 5: Confusion matrix on SVHN when not enforcing transformation invariance.
12


Reviews:
[{"review_id": "pjGUiMqNmgw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**1. Summary:**\n\nThe authors propose a module that regresses the parameters of an affine transformation or homography as an additional objective in the contrastive self-supervised learning framework. The authors argue that the geometric information encoded in the proposed module can supplement the signal provided by a contrastive loss, improving both performance and convergence speed. The authors validate their claims with two recent contrastive self-supervised learning approaches (i.e., SimCLR, BYOL) on several benchmark datasets showing effective results.\n\n**2. Strengths:**\n\n+ The authors extend the typical contrastive objective with an additional homography estimation objective to enforce non-invariance to the affine transformation or homography. This extension is well motivated, and the authors provide a clear intuition for the proposed module.\n\n+ The authors evaluate the proposed module on several benchmarks showing nice results.\n\n**3. Weaknesses:**\n\nWhile the authors demonstrate the effectiveness of the proposed module on several considered datasets, some points make me concerned about the actual usefulness of this module:\n\n- The authors conduct the empirical study on three small-scale datasets (i.e., CIFAR10, CIFAR100, SVHN). This is fine in itself, however, without experiments on large-scale datasets like ImageNet, it is unknown whether the conclusion still holds since most of the previous effective contrastive learning methods (e.g., SimCLR, BYOL) are originally conducted on ImageNet. It would be better for the authors to provide such experiments to further validate the claims. In this case, there is no need to perform experiments with large batch size to achieve state-of-the-art performance, just a normal batch size (e.g., 256) can validate the effectiveness of the proposed module.\n\n- The authors show significant improvements across all datasets with the proposed module in Table 2 and Table 3. However, as shown in Table 4, when pretrained for longer epoches, the relative benefit diminishes and even degrades the performance a little. Although the authors give a brief explanation on this phenomenon, I am concerned whether the proposed module is actually useful when SimCLR and BYOL have sufficiently converged.\n\n- The authors use the linear evaluation to measure the quality of learned representations by pre-training and fine-tuning on the same dataset. Although it is one of the common evaluation protocols, pre-training and fine-tuning on the same dataset may induce some inevitable biases. Therefore, it would be better to provide more empirical evidence by transferring to other datasets to evaluate the effectiveness of the proposed module.\n\nTo sum up, although the paper is interesting, I think it requires more work (see points W1-W3 above) in order to become more complete and convincing. Therefore, I am leaning towards rejection.\n\n====================================================================================\n\n**Post-Rebuttal**\n\nAfter reading the rebuttal and the other reviewers' comments, my concerns persist:\n\n- The technical contribution is limited. Adding a pretext task of homography prediction itself brings little insights regarding how it improves upon contrastive representation learning from a different perspective. \n\n- The experimental results are not convincing compared to the recent advances. The authors are encouraged to include ImageNet results as well as transfer learning evaluation.\n\nTherefore, I would like to keep my initial rating as rejection.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lWjdMe-p2PY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "====================================================\n\n**Update after rebuttal**\n\nI want to thank the authors for a long and highly detailed rebuttal. They clarified a lot of my questions and hopefully in the process they were able to improve the paper. However, my listed weaknesses still stand:\n* there is no transfer learning experiments\n* there not seem to be any consistent gains with the proposed approach over other methods, as seen in Table 4, after 500 epochs (when the models indeed have probably converged). Gains can be seen for 100 epochs, but from the absolute numbers  it is obvious that models havent really convered at that time. Furthermore, as the authors clarified,  they do require  30% higher training time, something that make the claim that they learn faster even weaker\n* There is no analysis It is unclear why/how forcing the feature vectors difference to be predictive (ie encode) the transformation and learning representations that are non-invariant to homography transforms actually helps for learning better representations. In fact, It turns out that simpler versions of the proposed module than homography estimation are the ones giving the best results. \n\nI will therefore retain my rejection rating.\n\n\n====================================================\n\nSummary:\n\nThe authors propose a module and objective that can be added to recent contrastive self-supervised learning (SSL) frameworks like SimCLR or BYOL. The module p`erforms an affine or homography transformation $\\phi$ on the input $x_1$. Then the difference of the features of $x_1$ and the geometrically transformed version are the input to an MLP that tries to regress to the parameters of  transformation $\\phi$, using an MSE loss. The authors show gains on CIFAR and SVHN by adding this module over SimCLR and BYOL. \n\nStrong points: The authors show that adding this module speeds up SSL pre-training during early epochs.\n\n\nWeak points: \n\nA) The authors do not measure transfer learning performance and evaluate *only* on a very superficial setup, where they train linear classifiers on top of encoders that are pre-trained with contrastive SSL learning on the exact same datasets.  Although this is indeed an experiment popular among SSL papers,  in all recent SSL works like SimCLR and BYOL the linear evaluation on the same dataset (usually imagenet) is only a small part of the evaluation suite that also involves testing performance on many different tasks and datasets. Transfer learning performance here is not measured, nor is performance on  semi-supervised learning tasks.\n\nB) The authors show some gains and faster learning for the first epochs of SSL pre-training. However, results should be taken with a pinch of salt as their gains are only visible in a realm where performance is still clearly suboptimal and learning is far from converged (eg for CIFAR100,  31.33 at epoch 100 vs 42.53 at epoch 500) To their credit, the authors themselves clearly say this in the text and show in Table 4. However they only present results at 100 and then 500 epochs, so it is hard to understand when (at which epoch) the two lines \"cross\" - at which epoch does the proposed approach stop giving gains? \n\nC) There is no analysis It is unclear why/how forcing the feature vectors difference to be predictive (ie encode) the transformation and learning representations that are non-invariant to homography transforms actually helps for learning better representations. In fact, It turns out that simpler versions of the proposed module than homography estimation are the ones giving the best results. In Table 5, the authors show that most of the gain can be achieved by only predicting the 2-dim sheer tranformation; this is something potentially interesting that the authors however do not investigate any further.\n\nQuestions and notes:\n* Although not used for SSL, the authors should cite and discuss the Spatial Transformer module of [Jaderberg et al 2015]\n* The notation could be clearer (eg although clearly explained in the text, it is a bit confusing that augmentation/transformation $a_2$ is sampled from set $T_1$ and $a_\\phi$ from $T_2$)\n* Design ablations: The authors use vector difference as the input to MLP h to regress to the transformation parameters, but other reasoning modules could also be used. How would performance be affected for other input functions beyond difference, eg concatenation or elementwise multiplication?\n* There area number of hyperparameter choices in Table 1 that do not match the corresponding papers (eg BYOL uses LARS not SGD). Why wasn't the original protocol followed, and were all the differences ablated?\n* How do you balance the two losses in Eq(5)? Is there no hyperparameter? what would happen if you favor one or the other loss?\n* How would performance change if this module was added after g() instead of f()?\n* Why only on $x_1$? Could there be extra gains if the MSE loss was also applied over $x_2$?\n* What is the added cost of the module in terms of training time? \n* It is unclear to me what the variance measures: Is it for results after multiple runs, and if so, how many runs? Were the multiple runs from scratch or only for the linear evaluation?\n* In Figure 2, how often was performance tested? markers would help understand where datapoints are.\n* The datasets used are relatively small and more importantly in very low resolution. Would the same hold for higher resolution datasets?\n* How dataset biased are the results presented? E.g. invariance to homographies might not be beneficial to digits (6 and 9 is a good example) but it would be useful in eg landmark datasets. Maybe a more suiting dataset/task would be something related to localization\n* [Tian et al 2020] is another related paper that could be discussed - there the authors explore different augmentations to get invariance to, and show highly increased performance even after longer training.\n\nReferences:\n[Jaderberg et al 2015] Spatial transformer networks.\" Advances in neural information processing systems. 2015.\n[Tian et al 2020] What makes for good views for contrastive learning, Arxiv May 2020, accepted at NeurIPS 2020 (not officially published yet)\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "i-22EnCSm3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper test a hypothesis that adding  affine transformation and homography estimation as an auxilliary loss in an image recognition task will improve performance. In experiments on CIFAR10, CIFAR100 and SVHN, improvement of a up to a few % points is observed.\n\nThe technique is close to augmentation and experimenting with A and H is not novel. \n\nThere are issues with the experiments:\n1. a certain representation of A and H was chosen. Other decompositions are possible. Is the ad hoc chosen one suitable? Have other options been considered?\n2. the representation of A and H influences the sampling. \n3. the three problems used for demonstrations are quite similar. It is unclear what type of vision problems would benefit from this technique.\n\nParts of the paper state well-known facts (description of H and A, there properties), some are irrelevant - for instance par 2. in the intro about transfer learning. More details should be provided about SimCLR and BYOL to make the submission more self-contained.\n\nI see very limited benefit to the reader.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U6Oj9TQQNkb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: the paper introduces a novel technique for self-supervised learning where additionally to the self-supervised loss, the model is forced to predict a parameter of the homography between two images given the difference of its embedding vectors. Authors evaluate the proposed method in BYOL and SimCLR.\n\nStrengths:\n\n- Self-supervised learning has shown a lot of promise these last years. I believe that this work explores an interesting addition to traditional self-supervised methods. \n\n- The paper evaluates this addition in two of the state-of-the-art self-supervised models (BYOL and SimCLR).\n\n- Authors provide very detailed explanation of the experimental set-up, which is very good for reproducibility. \n\n\nWeaknesses:\n\n- In my opinion, the paper needs to evaluate the methods in more large-scale benchmarks. Authors only use CIFAR and street view house numbers dataset, which are very specific benchmarks. However, the community is moving towards using larger and larger benchmarks and I think that the paper should follow this trend to be publicable. I think authors should at least apply their method to ImageNet training. \n\n- The performance improvement shown does not seem enough to merit publication, given the benchmark used. In particular, larger differences are shown only when one method does worse in absolute terms (Table 3). I think the combination of these two factors (the numerical improvement and the benchmarks used) make the paper weak.\n\n- I think it would be useful for the reader to see some examples of the homography to estimate in the main paper. I believe it helps the reader understand the task the model is attempting and how hard or difficult this is?\n\n- Have the authors considered using using a concatenation of the two vectors x1, x1' instead of its difference to predict the homography? I am missing some experiments motivating the choice of architecture. \n\n\nConclusion: I think the research direction is interesting but the paper needs to evaluate in larger dataset to show results that merit publication. Furthermore, I think authors should motivate better their architectural choices. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers raised concerns on the limited technical novelty and insufficient experiments. They unanimously recommended a rejection. I carefully read the authors' rebuttal but did not find strong reasons to go against the reviewers' recommendations. The reviewers made excellent points to further improve the paper. The authors are encouraged to incorporate those for a future submission.
Decision: Reject