Explicit homography estimation improves contrastive self-supervised learning

1 INTRODUCTION . There is an ever-increasing pool of data , particularly unstructured data such as images , text , video , and audio . The vast majority of this data is unlabelled . The process of labelling is time-consuming , labour-intensive , and expensive . Such an environment makes algorithms that can leverage fully unlabelled data particularly useful and important . Such algorithms fall within the realm of unsupervised learning . A particular subset of unsupervised learning is known as Self-Supervised Learning ( SSL ) . SSL is a paradigm in which the data itself provides a supervision signal to the algorithm . Somewhat related is another core area of research known as transfer learning ( Wang et al. , 2020 ) . In the context of computer vision , this means being able to pre-train an encoder network offline on a large , varietal dataset , followed by domain-specific fine-tuning on the bespoke task at hand . The state-of-the-art for many transfer learning applications remains dominated by supervised learning techniques ( Tan et al. , 2020 ; Martinez et al. , 2019 ; Donahue et al. , 2014 ; Girshick et al. , 2014 ) , in which models are pre-trained on a large labelled dataset . However , self-supervised learning techniques have more recently come to the fore as potential alternatives that perform similarly on downstream tasks , while requiring no labelled data . Most selfsupervised techniques create a supervision signal from the data itself in one of two ways . The one approach are techniques that define a pre-text task beforehand that a neural network is trained to solve , such as inpainting ( Pathak et al. , 2016 ) or a jigsaw puzzle ( Noroozi & Favaro , 2016 ) . In this way , the pre-text task is a kind of proxy that , if solved , should produce reasonable representations for downstream visual tasks such as image or video recognition , object detection , or semantic segmentation . The other approach is a class of techniques known as contrastive methods ( Chen et al. , 2020a ; He et al. , 2019 ; Chen et al. , 2020b ) . These methods minimise the distance ( or maximise the similarity ) between the latent representations of two augmented views of the same input image , while simultaneously maximising the distance between negative pairs . In this way , these methods enforce consistency regularisation ( Sohn et al. , 2020 ) , a well-known approach to semi-supervised learning . These contrastive methods often outperform the pre-text task methods and are the current state-of-the-art in self-supervised learning . However , most of these contrastive methods have several drawbacks , such as requiring prohibitively large batch sizes or memory banks , in order to retrieve the negative pairs of samples ( Chen et al. , 2020a ; He et al. , 2019 ) . The intuition behind our proposed module is that any system tasked with understanding images can benefit from understanding the geometry of the image and the objects within it . An affine transformation is a geometric transformation that preserves parallelism of lines . It can be composed of any sequence of rotation , translation , shearing , and scaling . A homography is a generalisation of this notion to include perspective warping . A homography need not preserve parallelism of lines , however , it ensures lines remain straight . Mathematically , a homography is shown in Equation 1 . It has 8 degrees of freedom and is applied to a vector in homogenous coordinates . An affine transformation has the same form , but with the added constraint that φ3,1 = φ3,2 = 0 . Hφ = [ φ1,1 φ1,2 φ1,3 φ2,1 φ2,2 φ2,3 φ3,1 φ3,2 1 ] ( 1 ) The ability to know how a source image was transformed to get to a target image implicitly means that you have learned something about the geometry of that image . An affine transformation or , more generally , a homography is a natural way to encode this idea . Forcing the network to estimate the parameters of a random homography applied to the source images thereby forces it to learn semantics about the geometry . This geometric information can supplement the signal provided by a contrastive loss , or loss in the latent space . In this paper , we propose an additional module that can be used in tandem with contrastive selfsupervised learning techniques to augment the contrastive objective ( the additional module is highlighted in Figure 1 ) . The module is simple , model-agnostic , and can be used to supplement a contrastive algorithm to improve performance and supplement the information learned by the network to converge faster . The module is essentially an additional stream of the network with the objective of regressing the parameters of an affine transformation or homography . In this way , there is a multi-task objective that the network must solve : 1. minimising the original contrastive objective , and 2. learning the parameters of a homography applied to one of the input images from a vector difference of their latent representations . We force the latent space to encode the geometric transformation information by learning to regress the parameters of the transformation in an MLP that takes the vector difference of two latent representations of an input , x , and its transformed analogue , x′ . By including the information in this way , the network is not invariant to the components of the transformation but is still able to use them as a self-supervised signal for learning . Moreover , this approach serves as a novel hybrid of the pre-text tasks and contrastive learning by enforcing consistency regularisation ( Sohn et al. , 2020 ) . Through extensive empirical studies , we show that the additional objective of regressing the transformation parameters serves as a useful supplementary task for self-supervised contrastive learning , and improves performance for all considered datasets in terms of linear evaluation accuracy and convergence speed . The remainder of the paper is structured as follows . In Section 2 , we cover the related work in the area of self-supervised learning , going into detail where necessary . In Section 3 we detail our proposed method . We first introduce a framework and set of notation to make the formalisation of the approach clear . We then delve into the details behind the architecture and choices for the various part of the system . This is followed by a comprehensive set of experiments in Section 4 , including results of various datasets , as well as an ablative study . Finally , the paper is concluded with some closing remarks in Section 5 . 2 RELATED WORK . SSL is a popular research area within computer vision . Previous approaches can be broadly classed into two main categories . The first is where pre-text tasks are manually defined , and the goal of the algorithms is to solve these hand-crafted tasks ( Lee et al. , 2020 ; Doersch et al. , 2015 ; Gidaris et al. , 2018 ; Zhang et al. , 2016 ; Misra & Maaten , 2020 ) . Examples of such methods include inpainting ( Pathak et al. , 2016 ) , colourising ( Zhang et al. , 2016 ) , jigsaw puzzles ( Noroozi & Favaro , 2016 ) , patch prediction ( Doersch et al. , 2015 ) , and geometric image transformations ( Dosovitskiy et al. , 2014 ) such as using rotation as the pre-text task ( Gidaris et al. , 2018 ; Feng et al. , 2019 ) . Some of these pre-text approaches that deal with geometric image transformations are similar in spirit to our method . Gidaris et al . ( 2018 ) ; Feng et al . ( 2019 ) are two variants of predicting image rotations as an auxiliary task for unsupervised learning . Perhaps closer to our method is Dosovitskiy et al . ( 2014 ) , in which a set of transformations is applied to image patches , and the network is trained in a fully-unsupervised manner to predict surrogate classes defined by a set of transformed image patches by minimising the log loss . Our method , however , investigates a different , particular set of transformations ( those that define an affine transformation of general homography ) , and show this can be used to aid self-supervised performance , using the transformation parameters themselves as targets that need to be regressed ( using mean-squared error ) by the contrastive algorithm in a multitask manner . The discrepancy in the network ’ s ability to predict the actual values of the parameters of the affine transformation/homography serves as our additional supervision signal . A somewhat related approach to our proposed method within the pre-text task domain is proposed by Lee et al . ( 2020 ) . They propose to augment the learning process of a supervised learning algorithm with additional labels constructed using self-supervised labels . These labels are rotation classes and colour permutations . Importantly , they create a loss function which is based on a joint distribution of the original ( supervised ) labels and the self-supervised ( augmented ) labels . In this way , the network is not forced to be invariant to the transformations under consideration , since this has been shown to hurt performance ( Lee et al. , 2020 ) . Our method is different to this in that we propose a module to be integrated specifically with self-supervised algorithms . Additionally , we regress the transformation parameters in real vector space and do not create classes for the parameters . The other broad category of SSL is based on contrastive learning ( Chen et al. , 2020a ; He et al. , 2019 ; Caron et al. , 2020 ) , and this class of techniques represent the current state-of-the-art in selfsupervised learning , outperforming the hand-crafted pre-text task methods . These approaches learn representations by contrasting positive pairs of samples from negative pairs of samples in latent space . Such methods typically require that careful attention be paid to the negative samples . Additionally , they have the disadvantage of requiring prohibitively large batch sizes ( 4096-16000 ) , memory banks , or other mechanisms to retrieve the relevant negative samples . One popular such method is known as SimCLR ( Chen et al. , 2020a ) . SimCLR is a general framework for contrastive learning , and in its vanilla formulation consists of an encoder network parameterised by a CNN ( usually a variant of ResNet ( He et al. , 2016 ) ) and an MLP projection head . An input image is sampled , and two distinct views of that same input image are computed using a random augmentation . The augmentation consists of colour jiterring , Gaussian blurring , and random cropping . The two views are sent through the encoder network to produce two latent representations . These latent vectors are then sent through the projection head to produce final latent vectors . It is from these vectors that the loss is computed . In the case of SimCLR , the loss is normalised temperatured cross-entropy ( NT-Xent ) . A recent approach proposed in Grill et al . ( 2020 ) ( BYOL ) somewhat overcomes the aforementioned disadvantages of requiring negative pairs of samples ( which implicitly requires a large batch size ) . Two separate networks with their own weights are used in tandem to learn the representation . An online network ( consisting of an encoder , MLP projection head , and MLP prediction network ) is trained to predict the representation outputted by a target network . During training , the online network parameters are updated using backpropagation of error derivatives computed using a meansquared error loss . However , the target network parameters are updated using an exponential moving average . In this way , BYOL overcomes collapsed solutions in which every image produces the same representation . We test our module with both SimCLR and BYOL , since these two methods serve as two popular , recent approaches to contrastive SSL . Some helpful findings for guiding self-supervised research were demonstrated in Kolesnikov et al . ( 2019 ) . Core among these are that 1 ) standard architecture designs that work well in the fullysupervised setting do not necessarily work well in the self-supervised setting , 2 ) in the selfsupervised setting larger CNNs often means higher quality learned representations , and 3 ) the linear evaluation paradigm for assessing performance may take a long time to converge . Moreover , Newell & Deng ( 2020 ) find that the effectiveness of self-supervised pretraining decreases as the amount of labelled data increases , and that performance on one particular downstream task is not necessarily indicative of performance on other downstream tasks .

Summary: The authors propose a module that regresses the parameters of an affine transformation or homography as an additional objective in the contrastive self-supervised learning framework. The authors argue that the geometric information encoded in the proposed module can supplement the signal provided by a contrastive loss, improving both performance and convergence speed. The authors validate their claims with two recent contrastive self-supervised learning approaches (i.e., SimCLR, BYOL) on several benchmark datasets showing effective results.