Viewmaker Networks: Learning Views for Unsupervised Representation Learning

1 INTRODUCTION . Unsupervised representation learning has made significant recent strides , including in computer vision , where view-based methods have enabled strong performance on benchmark tasks ( Wu et al. , 2018 ; Oord et al. , 2018 ; Bachman et al. , 2019 ; Zhuang et al. , 2019 ; Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020a ) . Views here refer to human-defined data transformations , which target capabilities or invariances thought to be useful for transfer tasks . In particular , in contrastive learning of visual representations , models are trained to maximize the mutual information between different views of an image , including crops , blurs , noise , and changes to color and contrast ( Bachman et al. , 2019 ; Chen et al. , 2020a ) . Much work has investigated the space of possible image views ( and their compositions ) and understanding their effects on transfer learning ( Chen et al. , 2020a ; Wu et al. , 2020 ; Tian et al. , 2019 ; Purushwalkam & Gupta , 2020 ) . The fact that views must be hand designed is a significant limitation . While views for image classification have been refined over many years , new views must be developed from scratch for new modalities . Making matters worse , even within a modality , different domains may have different optimal views ( Purushwalkam & Gupta , 2020 ) . Previous studies have investigated the properties of good views through the lens of mutual information ( Tian et al. , 2020 ; Wu et al. , 2020 ) , but a broadly-applicable approach for learning views remains unstudied . In this work , we present a general method for learning diverse and useful views for contrastive learning . Rather than searching through possible compositions of existing view functions ( Cubuk et al. , 2018 ; Lim et al. , 2019 ) , which may not be available for many modalities , our approach produces views with a generative model , called the viewmaker network , trained jointly with the encoder network . This flexibility enables learning a broad set of possible view functions , including input-dependent views , without resorting to hand-crafting or expert domain knowledge . The viewmaker network is trained adversarially to create views which increase the contrastive loss of the encoder network . Rather than directly outputting views for an image , the viewmaker instead outputs a stochastic perturbation that is added to the input . This perturbation is projected onto an ` p sphere , controlling the effective strength of the view , similar to methods in adversarial robustness . This constrained adversarial training method enables the model to reduce the mutual information between different views while preserving useful input features for the encoder to learn from . In summary , we contribute : 1 . Viewmaker networks : to our knowledge the first modality-agnostic method to learn views for unsupervised representation learning 2 . On image data , where expert-designed views have been extensively optimized , our viewmaker-models achieve comparable transfer performance to state of the art contrastive methods while being more robust to common corruptions . 3 . On speech data , our method significantly outperforms existing human-defined views on a range of speech recognition transfer tasks . 4 . On time-series data from wearable sensors , our model significantly outperforms baseline views on the task of human activity recognition ( e.g. , cycling , running , jumping rope ) . 2 RELATED WORK . Unsupervised representation learning Learning useful representations from unlabeled data is a fundamental problem in machine learning ( Pan & Yang , 2009 ; Bengio et al. , 2013 ) . A recently successful framework for unsupervised representation learning for images involves training a model to be invariant to various data transformations ( Bachman et al. , 2019 ; Misra & Maaten , 2020 ) , although the idea has much earlier roots ( Becker & Hinton , 1992 ; Hadsell et al. , 2006 ; Dosovitskiy et al. , 2014 ) . This idea has been expanded by a number of contrastive learning approaches which push embeddings of different views , or transformed inputs , closer together , while pushing other pairs apart ( Tian et al. , 2019 ; He et al. , 2020 ; Chen et al. , 2020a ; b ; c ) , as well as non-contrastive approaches which do not explicitly push apart unmatched views ( Grill et al. , 2020 ; Caron et al. , 2020 ) . Related but more limited setups have been explored for speech , where data augmentation strategies are less explored ( Oord et al. , 2018 ; Kharitonov et al. , 2020 ) . Understanding and designing views Several works have studied the role of views in contrastive learning , including from a mutual-information perspective ( Wu et al. , 2020 ) , in relation to specific transfer tasks ( Tian et al. , 2019 ) , with respect to different kinds of invariances ( Purushwalkam & Gupta , 2020 ) , or via careful empirical studies ( Chen et al. , 2020a ) . Outside of a contrastive learning framework , Gontijo-Lopes et al . ( 2020 ) study how data augmentation aids generalization in vision models . Much work has explored different handcrafted data augmentation methods for supervised learning of images ( Hendrycks et al. , 2020 ; Lopes et al. , 2019 ; Perez & Wang , 2017 ; Yun et al. , 2019 ; Zhang et al. , 2017 ) , speech ( Park et al. , 2019 ; Kovács et al. , 2017 ; Tóth et al. , 2018 ; Kharitonov et al. , 2020 ) , or in feature space ( DeVries & Taylor , 2017 ) . Adversarial methods Our work is related to and inspired by work on adversarial methods , including the ` p balls studied in adversarial robustness ( Szegedy et al. , 2013 ; Madry et al. , 2017 ; Raghunathan et al. , 2018 ) and training networks with adversarial objectives ( Goodfellow et al. , 2014 ; Xiao et al. , 2018 ) . Our work is also connected to the vicinal risk minimization principle ( Chapelle et al. , 2001 ) and can be interpreted as producing amortized virtual adversarial examples ( Miyato et al. , 2018 ) . Previous adversarial view-based pretraining methods add adversarial noise on top of existing handcrafted views ( Kim et al. , 2020 ) or require access to specific transfer tasks during pretraining ( Tian et al. , 2020 ) . In contrast , our method is more general : it is neither specialized to a particular downstream task , nor requires neither human-defined view families . Outside of multi-view learning paradigms , adversarial methods have also seen use for representation learning in GANs ( Donahue et al. , 2016 ; Donahue & Simonyan , 2019 ) or in choosing harder negative samples ( Bose et al. , 2018 ) , as well as for data augmentation ( Antoniou et al. , 2017 ; Volpi et al. , 2018 ; Bowles et al. , 2018 ) . Adversarial networks that perturb inputs have also been investigated to improve GAN training ( Sajjadi et al. , 2018 ) and to remove “ shortcut ” features ( e.g. , watermarks ) for self-supervised pretext tasks ( Minderer et al. , 2020 ) . Learning views Outside of adversarial approaches , our work is related to other studies that seek to learn data augmentation strategies by composing existing human-designed augmentations ( Ratner et al. , 2017 ; Cubuk et al. , 2018 ; Zhang et al. , 2019 ; Ho et al. , 2019 ; Lim et al. , 2019 ; Cubuk et al. , 2020 ) or by modeling variations specific to the data distribution ( Tran et al. , 2017 ; Wong & Kolter , 2020 ) . By contrast , our method requires no human-defined view functions , does not require first pretraining a generative model , and can generate perturbations beyond naturally-occurring variation observed in the training data ( e.g . brightness or contrast ) , potentially conferring robustness benefits , as we explore in Section 4.3 . 3 METHOD . In contrastive learning , the objective is to push embeddings of positive views ( derived from the same input ) close together , while pushing away embeddings of negative views ( derived from different inputs ) . We focus mainly on the simple , yet performant , SimCLR contrastive learning algorithm ( Chen et al. , 2020a ) , but we also consider a memory bank-based algorithm ( Wu et al. , 2018 ) in Section 4 . As our method is agnostic to the specific pretraining loss used , it is naturally compatible with other view-based algorithms such as MoCo ( He et al. , 2020 ) , BYOL ( Grill et al. , 2020 ) , and SwAV ( Caron et al. , 2020 ) by similarly substituting the data transformation pipeline with a viewmaker network . Formally , given a batch of N pairs of positive views ( i , j ) the SimCLR loss is L = 1 2N NX k=1 [ ` ( 2k 1 , 2k ) + ` ( 2k , 2k 1 ) ] where ` ( i , j ) = log exp ( si , j/⌧ ) P2N k=1 [ k 6=i ] exp ( si , k/⌧ ) and sa , b is the cosine similarity of the embeddings of views a and b . We generate views by perturbing examples with a viewmaker network V , trained jointly with the main encoder network M . There are three attributes desirable for useful perturbations , each of which motivates an aspect of our method : 1 . Challenging : The perturbations should be complex and strong enough that an encoder must develop useful representations to perform the self-supervised task . We accomplish this by generating perturbations with a neural network that is trained adversarially to increase the loss of the encoder network . Specifically , we use a neural network that ingests the input X and outputs a view X + V ( X ) . 2 . Faithful : The perturbations must not make the encoder task impossible , being so strong that they destroy all features of the input . For example , perturbations should not be able to zero out the input , making learning impossible . We accomplish this by constraining the perturbations to an ` p sphere around the original input . ` p constraints are common in the adversarial robustness literature where perturbations are expected to be indistinguishable . In our experiments , we find the best results are achieved with an ` 1 sphere , which grants the viewmaker a distortion budget that it can spend on a small perturbation for a large part of the input or a more extreme perturbation for a smaller portion . 3 . Stochastic : The method should be able to generate a variety of perturbations for a single input , as the encoder objective requires contrasting two different views of an input against each other . To do this , we inject random noise into the viewmaker , such that the model can learn a stochastic function that produces a different perturbed input each forward pass . Figure 2 summarizes our method . The encoder and viewmaker are optimized in alternating steps to minimize and maximize L , respectively . We use an image-to-image neural network as our viewmaker network , with an architecture adapted from work on style transfer ( Johnson et al. , 2016 ) . See the Appendix for more details . This network ingests the input image and outputs a perturbation that is constrained to an ` 1 sphere . The sphere ’ s radius is determined by the volume of the input tensor times a hyperparameter ✏ , the distortion budget , which determines the strength of the applied perturbation . This perturbation is added to the input image and optionally clamped in the case of images to ensure all pixels are in [ 0 , 1 ] . Algorithm 1 describes this process precisely . Algorithm 1 : Generating viewmaker views Input : Viewmaker network V , C ⇥W ⇥H image X , ` 1 distortion budget ✏ , noise Output : Perturbed C ⇥W ⇥H image X P V ( X , ) // generate perturbation

Summary: The paper presents a generative model to automatically generate data that is needed for contrastive learning, with a focus on the SimCLR framework, while the method itself is general. Experiments were conducted across multiple modalities, including image, speech and wearable sensor data. The results demonstrate the effectiveness of the proposed model as compared to data augmentation methods relying on human domain knowledge.