Scarf: Self-Supervised Contrastive Learning using Random Feature Corruption

1 INTRODUCTION . In many machine learning tasks , unlabeled data is abundant but labeled data is costly to collect , requiring manual human labelers . The goal of self-supervised learning is to leverage large amounts of unlabeled data to learn useful representations for downstream tasks such as classification . Selfsupervised learning has proved critical in computer vision ( Grill et al. , 2020 ; Misra & Maaten , 2020 ; He et al. , 2020 ; Tian et al. , 2019 ) and natural language processing ( Song et al. , 2020 ; Wang et al. , 2019 ; Raffel et al. , 2019 ) . Some recent examples include the following : Chen et al . ( 2020 ) showed that training a linear classifier on the representations learned by their proposed method , SimCLR , significantly outperforms previous state-of-art image classifiers and requires 100x fewer labels to achieve state of art results ; Brown et al . ( 2020 ) showed through their GPT-3 language model that by pre-training on a large corpus of text , only few labeled examples were required for task-specific fine-tuning for a wide range of tasks . A common theme of these advances is learning representations that are robust to different views or distortions of the same input ; this is often achieved by maximizing the similarity between views of the same input and minimizing those of different inputs via a contrastive loss . However , techniques to generate views or corruptions have thus far been , by and large , domain-specific ( e.g . color distortion ( Zhang et al. , 2016 ) and cropping ( Chen et al. , 2020 ) in vision , and token masking ( Song et al. , 2020 ) in NLP ) . Despite the importance of self-supervised learning , there is surprisingly little work done in finding methods that are applicable across domains and in particular , ones that can be applied to tabular data . In this paper , we propose SCARF , a simple and versatile contrastive pre-training procedure . We generate a view for a given input by selecting a random subset of its features and replacing them by random draws from the features ’ respective empirical marginal distributions . Experimentally , we test SCARF on the OpenML-CC18 benchmark ( Vanschoren et al. , 2013 ; Bischl et al. , 2017 ; Feurer et al. , 2019 ) , a collection of 72 real-world classification datasets . We show that not only does SCARF pre-training improve classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled . Moreover , we show that combining SCARF pre-training with other solutions to these problems further improves them , demonstrating the versatility of SCARF and its ability to learn effective task-agnostic representations . We then conduct extensive ablation studies , showing the Classification head . effects of various design choices and stability to hyperparameters . Our ablations show that SCARF ’ s way of constructing views is more effective than alternatives . We show that SCARF is less sensitive to feature scaling and is stable to various hyperparameters such as batch size , corruption rate , and softmax temperature . 2 RELATED WORKS . A number of self-supervised learning techniques have been proposed in computer vision ( Zhai et al. , 2019 ; Tung et al. , 2017 ; Jing & Tian , 2020 ) . One framework involves learning features based on generated images through various methods , including using a GAN ( Goodfellow et al. , 2014 ; Donahue et al. , 2016 ; Radford et al. , 2015 ; Chen et al. , 2018 ) , predicting pixels ( Larsson et al. , 2016 ) , predicting colorizations ( Zhang et al. , 2016 ; Larsson et al. , 2017 ) , ensuring local and global consistency ( Iizuka et al. , 2017 ) , and learning synthetic artifacts ( Jenni & Favaro , 2018 ) . Another line of work involves training on generated labels with semantic meanings that can be obtained without any human labelers ( e.g . using segmentation masks ) ( Ren & Lee , 2018 ; Li et al. , 2016 ; Pathak et al. , 2017 ) . Leveraging the spatial relations of image patches and temporal ones of videos has also been successful . Such approaches include the image jigsaw puzzle ( Noroozi & Favaro , 2016 ; Ahsan et al. , 2019 ) , context prediction ( Doersch et al. , 2015 ) , clustering ( Xie et al. , 2016 ; Caron et al. , 2018 ; Yang et al. , 2016 ) , and geometric transformation recognition ( Lee et al. , 2017 ; Xu et al. , 2019 ) . Finally , most related to our approach , are the similarity-based approaches that contrast different views of the image ( Tian et al. , 2019 ; Hassani & Khasahmadi , 2020 ; Oord et al. , 2018 ; Henaff , 2020 ; Li et al. , 2016 ; He et al. , 2020 ; Bojanowski & Joulin , 2017 ; Wang & Gupta , 2015 ; Gidaris et al. , 2018 ) . In particular , our framework is similar to SimCLR ( Chen et al. , 2020 ) , which involves generating views of a single image via image-based corruptions like random cropping , color distortion and blurring ; however , we generate views that are applicable to tabular data . Self-supervised learning has had an especially large impact in language modeling ( Qiu et al. , 2020 ) . One popular approach is masked language modeling , wherein the model is trained to predict input Algorithm 1 SCARF pre-training algorithm . 1 : input : unlabeled training data X ⊆ RM , batch size N , temperature τ , corruption rate c , encoder network f , pre-train head network g. 2 : let X̂j be the uniform distribution over Xj = { xj : x ∈ X } , where xj denotes the j-th coordinate of x . 3 : let q = bc ·Mc be the number of features to corrupt . 4 : for sampled mini-batch { x ( i ) } N i=1 ⊆ X do 5 : for i ∈ [ N ] , uniformly sample subset Ii from { 1 , ... , M } of size q and define x̃ ( i ) ∈ RM as follows : x̃ ( i ) j = xj if j 6∈ Ii , otherwise x̃ ( i ) j = v , where v ∼ X̂j . # generate corrupted view . 6 : let z ( i ) = g ( f ( x ( i ) ) ) , z̃ ( i ) = g ( f ( x̃ ( i ) ) ) , for i ∈ [ N ] . # embeddings for views . 7 : let si , j = z ( i ) > z̃ ( j ) / ( ‖z ( i ) ‖2 · ‖z̃ ( j ) ‖2 ) , for i , j ∈ [ N ] . # pairwise similarity . 8 : define Lcont : = 1N ∑N i=1− log ( exp ( si , i/τ ) 1 N ∑N k=1 exp ( si , k/τ ) ) . 9 : update networks f and g to minimize Lcont using SGD . 10 : end for 11 : return encoder network f . tokens that have been intentionally masked out ( Devlin et al. , 2018 ; Raffel et al. , 2019 ; Song et al. , 2019 ) as well as enhancements to this approach ( Liu et al. , 2019 ; Dong et al. , 2019 ; Bao et al. , 2020 ; Lample & Conneau , 2019 ; Joshi et al. , 2020 ) and variations involving permuting the tokens ( Yang et al. , 2019 ; Song et al. , 2020 ) . Denoising autoencoders have been used by training them to reconstruct the input from a corrupted version ( produced by , for example , token masking , deletion , and infilling ) ( Lewis et al. , 2019 ; Wang et al. , 2019 ; Freitag & Roy , 2018 ) . Finally , most related to our approach , is contrastive learning , which involves learning based on similarity of pairs of inputs . Such approaches include randomly replacing words and distinguishing between real and fake phrases ( Collobert et al. , 2011 ; Mnih & Kavukcuoglu , 2013 ) , random token replacement ( Mikolov et al. , 2013 ; Clark et al. , 2020 ) , and adjacent sentences ( Joshi et al. , 2020 ; Lan et al. , 2019 ; de Vries et al. , 2019 ) . Recently , Yao et al . ( 2020 ) adapted the similarity-based contrastive framework to large-scale recommendation systems in a similar way to our approach . The key difference is in the way the methods generate multiple views . Yao et al . ( 2020 ) proposes masking random features in a correlated manner and applying a dropout for categorical features , while our approach involves randomizing random features based on the features ’ respective marginal training distribution ( in an uncorrelated way ) . Generating such views for a task is a difficult problem : there has been much work done in understanding and designing them ( Wu et al. , 2018 ; Purushwalkam & Gupta , 2020 ; Gontijo-Lopes et al. , 2020 ; Lopes et al. , 2019 ; Perez & Wang , 2017 ; Park et al. , 2019 ) and learning them ( Ratner et al. , 2017 ; Cubuk et al. , 2020 ; Ho et al. , 2019 ; Lim et al. , 2019 ; Zhang et al. , 2019b ; Tran et al. , 2017 ; Tamkin et al. , 2020 ) . In this work , we present a simple method to generate multiple views that ’ s effective for unsupervised representation learning of tabular data , which may be of interest to other applications that benefit from such data augmentation . Within the contrastive learning framework , the choice of loss function is significant . InfoNCE ( Gutmann & Hyvärinen , 2010 ; Oord et al. , 2018 ) , which can be interpreted as a non-parametric estimation of the entropy of the representation ( Wang & Isola , 2020 ) , is a popular choice . Since then there have been a number of proposals ( Zbontar et al. , 2021 ; Grill et al. , 2020 ; Hjelm et al. , 2018 ) ; however , we show that InfoNCE is effective for our framework . 3 SCARF . We now describe our proposed method ( Algorithm 1 ) , which is also described in Figure 1 . For each mini-batch of examples from the unlabeled training data , we generate a corrupted version x̃ ( i ) for each example x ( i ) as follows . We sample some fraction of the features uniformly at random and replace each of those features by a random draw from that feature ’ s empirical marginal distribution , which is defined as the uniform distribution over the values that feature takes on across the training dataset . Then , we pass both x ( i ) and x̃ ( i ) through the encoder network f , whose output we pass through the pre-train head network g , to get z ( i ) and z̃ ( i ) respectively . Note that the pre-train head network ` 2-normalizes the outputs so that they lie on the unit hypersphere – this has been found to be crucial in practice ( Chen et al. , 2020 ; Wang & Isola , 2020 ) . We train on the InfoNCE contrastive loss , encouraging z ( i ) and z̃ ( i ) to be close for all i and z ( i ) and z̃ ( j ) to be far apart for i 6= j , and we optimize over the parameters of f and g via SGD . Then , to train a classifier for the task via fine-tuning , we take the encoder network f and attach a classification head h which takes the output of f as its input and predicts the label of the example . We optimize the cross-entropy classification loss and tune the parameters of both f and h. While pre-training can be run for a pre-determined number of epochs , much like normal supervised training , how many is needed largely depends on the model and dataset . To this end , we propose using early stopping on the validation InfoNCE loss . Given unlabeled validation data , we cycle through it for some number of epochs , running our proposed method to generate ( x ( i ) , x̃ ( i ) ) pairs . Once built , the loss on this static set is tracked during pre-training . Prototypical loss curves are shown in the Appendix .

Summary: This paper proposes a data augmentation method (SCARF) used for self-supervised learning for tabular data. SCARF generates different views by corrupting a random subset of features (via sampling from the marginal distribution). The experimetanl results demonstrate that SCARF not only improves accuracy in the fully-supervised setting but also in the presence of label noise and in the semi-supervised setting. The authors also conduct ablation studies and compare with various methods.  I think adapting SSL to tabular data is a very important direction and I thank the authors for their efforts in this direction.