Scarf: Self-Supervised Contrastive Learning using Random Feature Corruption

1 INTRODUCTION . In many machine learning tasks , unlabeled data is abundant but labeled data is costly to collect , requiring manual human labelers . The goal of self-supervised learning is to leverage large amounts of unlabeled data to learn useful representations for downstream tasks such as classification . Selfsupervised learning has proved critical in computer vision ( Grill et al. , 2020 ; Misra & Maaten , 2020 ; He et al. , 2020 ; Tian et al. , 2019 ) and natural language processing ( Song et al. , 2020 ; Wang et al. , 2019 ; Raffel et al. , 2019 ) . Some recent examples include the following : Chen et al . ( 2020 ) showed that training a linear classifier on the representations learned by their proposed method , SimCLR , significantly outperforms previous state-of-art image classifiers and requires 100x fewer labels to achieve state of art results ; Brown et al . ( 2020 ) showed through their GPT-3 language model that by pre-training on a large corpus of text , only few labeled examples were required for task-specific fine-tuning for a wide range of tasks . A common theme of these advances is learning representations that are robust to different views or distortions of the same input ; this is often achieved by maximizing the similarity between views of the same input and minimizing those of different inputs via a contrastive loss . However , techniques to generate views or corruptions have thus far been , by and large , domain-specific ( e.g . color distortion ( Zhang et al. , 2016 ) and cropping ( Chen et al. , 2020 ) in vision , and token masking ( Song et al. , 2020 ) in NLP ) . Despite the importance of self-supervised learning , there is surprisingly little work done in finding methods that are applicable across domains and in particular , ones that can be applied to tabular data . In this paper , we propose SCARF , a simple and versatile contrastive pre-training procedure . We generate a view for a given input by selecting a random subset of its features and replacing them by random draws from the features ’ respective empirical marginal distributions . Experimentally , we test SCARF on the OpenML-CC18 benchmark ( Vanschoren et al. , 2013 ; Bischl et al. , 2017 ; Feurer et al. , 2019 ) , a collection of 72 real-world classification datasets . We show that not only does SCARF pre-training improve classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled . Moreover , we show that combining SCARF pre-training with other solutions to these problems further improves them , demonstrating the versatility of SCARF and its ability to learn effective task-agnostic representations . We then conduct extensive ablation studies , showing the Classification head . effects of various design choices and stability to hyperparameters . Our ablations show that SCARF ’ s way of constructing views is more effective than alternatives . We show that SCARF is less sensitive to feature scaling and is stable to various hyperparameters such as batch size , corruption rate , and softmax temperature . 2 RELATED WORKS . A number of self-supervised learning techniques have been proposed in computer vision ( Zhai et al. , 2019 ; Tung et al. , 2017 ; Jing & Tian , 2020 ) . One framework involves learning features based on generated images through various methods , including using a GAN ( Goodfellow et al. , 2014 ; Donahue et al. , 2016 ; Radford et al. , 2015 ; Chen et al. , 2018 ) , predicting pixels ( Larsson et al. , 2016 ) , predicting colorizations ( Zhang et al. , 2016 ; Larsson et al. , 2017 ) , ensuring local and global consistency ( Iizuka et al. , 2017 ) , and learning synthetic artifacts ( Jenni & Favaro , 2018 ) . Another line of work involves training on generated labels with semantic meanings that can be obtained without any human labelers ( e.g . using segmentation masks ) ( Ren & Lee , 2018 ; Li et al. , 2016 ; Pathak et al. , 2017 ) . Leveraging the spatial relations of image patches and temporal ones of videos has also been successful . Such approaches include the image jigsaw puzzle ( Noroozi & Favaro , 2016 ; Ahsan et al. , 2019 ) , context prediction ( Doersch et al. , 2015 ) , clustering ( Xie et al. , 2016 ; Caron et al. , 2018 ; Yang et al. , 2016 ) , and geometric transformation recognition ( Lee et al. , 2017 ; Xu et al. , 2019 ) . Finally , most related to our approach , are the similarity-based approaches that contrast different views of the image ( Tian et al. , 2019 ; Hassani & Khasahmadi , 2020 ; Oord et al. , 2018 ; Henaff , 2020 ; Li et al. , 2016 ; He et al. , 2020 ; Bojanowski & Joulin , 2017 ; Wang & Gupta , 2015 ; Gidaris et al. , 2018 ) . In particular , our framework is similar to SimCLR ( Chen et al. , 2020 ) , which involves generating views of a single image via image-based corruptions like random cropping , color distortion and blurring ; however , we generate views that are applicable to tabular data . Self-supervised learning has had an especially large impact in language modeling ( Qiu et al. , 2020 ) . One popular approach is masked language modeling , wherein the model is trained to predict input Algorithm 1 SCARF pre-training algorithm . 1 : input : unlabeled training data X ⊆ RM , batch size N , temperature τ , corruption rate c , encoder network f , pre-train head network g. 2 : let X̂j be the uniform distribution over Xj = { xj : x ∈ X } , where xj denotes the j-th coordinate of x . 3 : let q = bc ·Mc be the number of features to corrupt . 4 : for sampled mini-batch { x ( i ) } N i=1 ⊆ X do 5 : for i ∈ [ N ] , uniformly sample subset Ii from { 1 , ... , M } of size q and define x̃ ( i ) ∈ RM as follows : x̃ ( i ) j = xj if j 6∈ Ii , otherwise x̃ ( i ) j = v , where v ∼ X̂j . # generate corrupted view . 6 : let z ( i ) = g ( f ( x ( i ) ) ) , z̃ ( i ) = g ( f ( x̃ ( i ) ) ) , for i ∈ [ N ] . # embeddings for views . 7 : let si , j = z ( i ) > z̃ ( j ) / ( ‖z ( i ) ‖2 · ‖z̃ ( j ) ‖2 ) , for i , j ∈ [ N ] . # pairwise similarity . 8 : define Lcont : = 1N ∑N i=1− log ( exp ( si , i/τ ) 1 N ∑N k=1 exp ( si , k/τ ) ) . 9 : update networks f and g to minimize Lcont using SGD . 10 : end for 11 : return encoder network f . tokens that have been intentionally masked out ( Devlin et al. , 2018 ; Raffel et al. , 2019 ; Song et al. , 2019 ) as well as enhancements to this approach ( Liu et al. , 2019 ; Dong et al. , 2019 ; Bao et al. , 2020 ; Lample & Conneau , 2019 ; Joshi et al. , 2020 ) and variations involving permuting the tokens ( Yang et al. , 2019 ; Song et al. , 2020 ) . Denoising autoencoders have been used by training them to reconstruct the input from a corrupted version ( produced by , for example , token masking , deletion , and infilling ) ( Lewis et al. , 2019 ; Wang et al. , 2019 ; Freitag & Roy , 2018 ) . Finally , most related to our approach , is contrastive learning , which involves learning based on similarity of pairs of inputs . Such approaches include randomly replacing words and distinguishing between real and fake phrases ( Collobert et al. , 2011 ; Mnih & Kavukcuoglu , 2013 ) , random token replacement ( Mikolov et al. , 2013 ; Clark et al. , 2020 ) , and adjacent sentences ( Joshi et al. , 2020 ; Lan et al. , 2019 ; de Vries et al. , 2019 ) . Recently , Yao et al . ( 2020 ) adapted the similarity-based contrastive framework to large-scale recommendation systems in a similar way to our approach . The key difference is in the way the methods generate multiple views . Yao et al . ( 2020 ) proposes masking random features in a correlated manner and applying a dropout for categorical features , while our approach involves randomizing random features based on the features ’ respective marginal training distribution ( in an uncorrelated way ) . Generating such views for a task is a difficult problem : there has been much work done in understanding and designing them ( Wu et al. , 2018 ; Purushwalkam & Gupta , 2020 ; Gontijo-Lopes et al. , 2020 ; Lopes et al. , 2019 ; Perez & Wang , 2017 ; Park et al. , 2019 ) and learning them ( Ratner et al. , 2017 ; Cubuk et al. , 2020 ; Ho et al. , 2019 ; Lim et al. , 2019 ; Zhang et al. , 2019b ; Tran et al. , 2017 ; Tamkin et al. , 2020 ) . In this work , we present a simple method to generate multiple views that ’ s effective for unsupervised representation learning of tabular data , which may be of interest to other applications that benefit from such data augmentation . Within the contrastive learning framework , the choice of loss function is significant . InfoNCE ( Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ) , which can be interpreted as a non-parametric estimation of the entropy of the representation ( Wang & Isola , 2020 ) , is a popular choice . Since then there have been a number of proposals ( Zbontar et al. , 2021 ; Grill et al. , 2020 ; Hjelm et al. , 2018 ) ; however , we show that InfoNCE is effective for our framework . 3 SCARF . We now describe our proposed method ( Algorithm 1 ) , which is also described in Figure 1 . For each mini-batch of examples from the unlabeled training data , we generate a corrupted version x̃ ( i ) for each example x ( i ) as follows . We sample some fraction of the features uniformly at random and replace each of those features by a random draw from that feature ’ s empirical marginal distribution , which is defined as the uniform distribution over the values that feature takes on across the training dataset . Then , we pass both x ( i ) and x̃ ( i ) through the encoder network f , whose output we pass through the pre-train head network g , to get z ( i ) and z̃ ( i ) respectively . Note that the pre-train head network ` 2-normalizes the outputs so that they lie on the unit hypersphere – this has been found to be crucial in practice ( Chen et al. , 2020 ; Wang & Isola , 2020 ) . We train on the InfoNCE contrastive loss , encouraging z ( i ) and z̃ ( i ) to be close for all i and z ( i ) and z̃ ( j ) to be far apart for i 6= j , and we optimize over the parameters of f and g via SGD . Then , to train a classifier for the task via fine-tuning , we take the encoder network f and attach a classification head h which takes the output of f as its input and predicts the label of the example . We optimize the cross-entropy classification loss and tune the parameters of both f and h. While pre-training can be run for a pre-determined number of epochs , much like normal supervised training , how many is needed largely depends on the model and dataset . To this end , we propose using early stopping on the validation InfoNCE loss . Given unlabeled validation data , we cycle through it for some number of epochs , running our proposed method to generate ( x ( i ) , x̃ ( i ) ) pairs . Once built , the loss on this static set is tracked during pre-training . Prototypical loss curves are shown in the Appendix .

Summary: The paper presents a method for self-supervised pre-training on tabular data to improve the performance in supervised transfer. The method adapts the successful contrastive learning framework to tabular data by defining “augmentations” of the data wherein randomly chosen feature columns are replaced by sampling from their corresponding marginal distribution. The method is evaluated on tabular classification tasks of the OpenML-CC18 benchmark, both in the fully supervised setting and in the semi-supervised setting and under 30% label noise.  The method outperforms several baselines.