Scarf: Self-Supervised Contrastive Learning using Random Feature Corruption
Authors: Dara Bahri
Heinrich Jiang
Yi Tay
Donald Metzler
Venue: ICLR 2022 Spotlight (2022)
Keywords: self-supervised learning
tabular data
pre-training
contrastive learning
openML

Abstract: Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data. However, such methods are domain-specific and little has been done to leverage this technique on real-world \emph{tabular} datasets. We propose \textsc{Scarf}, a simple, widely-applicable technique for contrastive learning, where views are formed by corrupting a random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark, \textsc{Scarf} not only improves classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled. We show that \textsc{Scarf} complements existing strategies and outperforms alternatives like autoencoders. We conduct comprehensive ablations, detailing the importance of a range of factors.

Published as a conference paper at ICLR 2022
SCARF: SELF-SUPERVISED CONTRASTIVE LEARNING USING RANDOM FEATURE CORRUPTION
Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler
Google Research (dbahri, hei nri chii yitay, metzler)
ABSTRACT
sulf-supe contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-ae performance with orders of magnitude less labeled data. However, such methods are domain- specific and little has been done to leverage this technique on real-world tabular datasets. We propose SCARF, a simple, widely- -aplicabpee technique for contrastive learning, where views are formed by corrupting random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark, SCARF not only improves classifi- cation accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only fraction of the available training data is labeled. We show that SCARF complements existing strategies and outperforms alternatives like autoencoders. We conduct comprehensive ablations. detailing the importance of range of factors.
INTRODUCTION
In many machine learning tasks, unlabeled data is abundant but labeled data is costly to collect, requiring manual human labelers. The goal of self-supervised learning is leverage large amounts of unlabeled data to learn useful representations for downstream tasks such as classification. Self- supervised learning has proved critical computer vision (Grill et al., 2020; Misra & Maaten, 2020; He et al., 2020; Tian et al., 2019) and natural language processing (Song et al., 2020; Wang et al., 2019; Raffel al., 2019). Some recent examples include the following: Chen et al. (2020) showed that training a linear classifier on the representations learned by their proposed method, SimCLR, significantly outperforms previous state-of-art art image classifiers and requires 100x fewer labels to do Brown et al. (2020) showed through their GPT-3 language model that by pre-training large corpus of text, only few labeled examples were required for task-specific fine-tuning for a wide range of tasks.
common theme of these advances is learning representations that are robust to different views or distortions of the same input; this is often achieved by maximizing the similarity between views of the same input and minimizing those of different inputs via a contrastive loss. However, techniques to generate views or corruptions have thus far been, by and large, domain-specifie (e.g. color distortion (Zhang et al., 2016) and cropping (Chen et al., 2020) in vision, and token masking (Song al., 2020) in NLP). Despite the importance of self-supervised learning, there is surprisingly little work done in finding methods that are applicable across domains and in particular, ones that can be applied to tabular data.
In this paper, we propose SCARF, simple and versatile contrastive pre-training procedure. We generate a view for given input by selecting random subset of its features and replacing them by random draws from the features' respective empirical marginal distributions. Experimentally, we test SCARF on the OpenML-CC11 benchmark (Vanschoren et al., 2013; Bischl et al., 2017: Feurer et al., 2019), a collection of 72 real-world classification datasets. We show that not only does SCARF pre- pre-training improve classification accuracy in the sully-super setting but does so also in the presence of label noise and in the cemi-supervised setting where only a fraction of the available training data is labeled. Moreover, we show that combining SCARF pre-training with other solutions to these problems further improves them, demonstrating the versatility of SCARF and its ability to

---

Published as a conference paper at ICLR 2022
Unsupervised SCARF Pre-training
Similarity
Input
Unchanged
Pre-training head g (shared weights)
Encoder (shared weights)
Sample hel the intiosa comupt.
Comuut drawwing from the resp. fealure's marginll.
Supervised Fine-Tuning
Input
Classification head
Classification Loss
h
Figure 1: Diagram showing unsupervised SCARF pre-training (Top) and subsequent supervised fine- tuning (Bottom). During pre-training, networks f and g are learned to produce good representations of the input data. After pre-training, is discarded and classification head h is applied on top of the learned f and both f and are subsequently fine-tuned for classification.
learn effective task-agnostic representations. We then conduct extensive ablation studies, showing the effects of various design choices and stability hyperparameters. Our ablations show that SCARF's way of constructing views is more effective than alternatives. We show that SCARF is less sensitive feature scaling and stable to various hyperparameters such as batch size, corruption rate, and softmax temperature.
RELATED WORKS
number self-supervised learning techniques have been proposed in computer vision (Zhai et al., 2019; Tung et al., 2017; Jing & Tian, 2020). One framework involves learning features based on generated images through various methods, including using GAN (Goodfellow et al., 2014; Donahue al., 2016; Radford al., 2015; Chen al., 2018), predicting pixels (Larsson al., 2016), predicting colorizations (Zhang et al., 2016; Larsson et al., 2017), ensuring local and global consistency (lizuka al., 2017), and learning synthetic artifacts (Jenni & Favaro, 2018). Most related to our approach are contrastive learning ones (Tian et al., 2019; Hassani & Khasahmadi, 2020; Oord al., 2018: Henaff, 2020: Li al., 2016; He al., 2020; Bojanowski & Joulin, 2017: Wang Gupta, 2015; Gidaris et al., 2018). In particular, our framework is similar to SimCLR (Chen et al., 2020), which involves generating views of single image image-based corruptions like random cropping. color distortion and blurring; however, we generate views that are applicable to tabular data.
Self-supervised learning has had an especially large impact in language modeling (Qiu et al., 2020). One popular approach is masked language modeling, wherein the model is trained to predict input tokens that have been intentionally masked out (Devlin et al., 2018; Raffel et al., 2019; Song et al., 2019) as well enhancements to this approach (Liu et al., 2019; Dong et al., 2019; Bao et al., 2020; Lample & Conneau, 2019; Joshi et al., 2020) and variations involving permuting the tokens (Yang et al., 2019; Song al., 2020). Denoising autoencoders have been used by training them to reconstruct the input from a corrupted version (produced by, for example, token masking, deletion, and infilling) (Lewis et al., 2019; Wang al., 2019; Freitag & Roy, 2018). Contrastive approaches
2

---

Published as a conference paper at ICLR 2022
Algorithm 1 SCARF pre-training algorithm.
1: input: unlabeled training data S RRM,, batch size N, temperature т, corruption rate e, encoder network J. pre-train head network y.
2: let X3 be the uniform distribution over X3 {ay : x E X3, where zy denotes the j-th coordinate of x.
3: let q - [c- M| be the number of features corrupt.
4: for sampled mini-batch (xx(0 Ji=1
) S X' do
5: for E [NV] uniformly sample subset Z, from {I, .M of size and define z(0) R- as # generate corrupted view. # embeddings for views.
follows: jj 2i if j & I, otherwise x.fi) where v ~ Xz.
6: let 20 9 (f (z())), z(0) = g (z(1)) for [N].
7: let 81 - (+) 500) (1|=(0 |12 ||500| l12). for i, E [N].
pairwise similarity,
8: define Loont := N Σ log Σレ- explst, a/T)
explst.1/7)
9: update networks f and g to minimize feomt using SGD.
10: end for
11: return encoder network ʃ.
include randomly replacing words and distinguishing between real and fake phrases (Collobert et al., 2011; Mnih & Kavukcuoglu, 2013), random token replacement (Mikolov et al., 2013; Clark et al., 2020), and adjacent sentences (Joshi et al., 2020; Lan et al., 2019; Vries et al., 2019).
Within the contrastive learning framework, the choice of loss function is significant. InfoNCE (Gut- mann Hyvärinen, 2010; Oord al., 2018), which can be interpreted as a non-nonnnmee cana-mamettric estimation of the entropy of the representation (Wang & Isola, 2020), is a popular choice. Since then there have been a number of proposals (Zbontar al., 2021; Grill et al., 2020; Hjelm et al., 2018); however, we show that InfoNCE is effective for our framework.
Recently, Yao et al. (2020) adapted the contrastive framework to large-scale recommendation systems in way similar to our approach. The key difference is in the way the methods generate multiple views. Yao et al. (2020) proposes masking random features in a correlated manner and applying a dropout for categorical features, while our approach involves randomizing random features based the features' respective empirical marginal distribution (in an uncorrelated way). Generating such views for task is difficult problem: there has been much work done in understanding and designing them (Wu et al., 2018; Purushwalkam & Gupta, 2020; Gontijo-Lopes et al., 2020; Lopes et al., 2019; Perez & Wang. 2017; Park al., 2019) and learning them (Ratner et al., 2017: Cubuk al., 2020; Ho et al., 2019; Lim et al., 2019; Zhang et al., 2019b; Tran et al., 2017; Tamkin et al., 2020).
Lastly, also similar our work VIME (Yoon al., 2020), which proposes the same corruption technique for tabular data that we do. They pre-train an encoder network on unlabeled data by attaching "mask estimator" and "feature estimator" heads on top of the encoder state and teaching the model to recover both the binary mask that was used for corruption well as the original uncorrupted input, given the corrupted input. The pre-trained encoder network is subsequently used for semi- supervised learning via attachment of a task-specific head and minimization of the supervised loss as well as an auto-encoder reconstruction loss. VIME was shown to achieve state-of-art results on genomics and clinical datasets. The key differences with our work is that we pre-train using a contrastive loss, which we show to be more effective than the denoising auto-encoder loss that partly constitutes VIME, Furthermore, after pre-training fine-tune all model weights, including the encoder (unlike VIME, which only fine-tunes the task head), and we do so using task supervision only.
SCARF
We now describe our proposed method (Algorithm 1), which is also described in Figure 1. For each mini-batch of examples from the unlabeled training data, we generate corrupted version 5(() for
3

---

Published as a conference paper at ICLR 2022
each example 20() as follows. We sample some fraction of the features uniformly at random and replace each of those features by a random draw from that feature's empirical marginal distribution, which is defined as the uniform distribution over the values that feature takes across the training dataset. Then, we pass both x²(i) and z(0) through the encoder network f. whose output we pass through the tre-train head network 9, to get 200) and 2(0) respectively. Note that the pre-train head network & -z-oorallsess the outputs so that they lie on the unit hypersphere this has been found to be crucial in practice (Chen et al., 2020; Wang & Isola, 2020). We train on the InfoNCE contrastive loss, encouraging 2 and x()) to be close for all i and (t() and g06 to be far apart for i # j, and we optimize over the parameters of f and 9 via SGD.
Then, to train a classifier for the task via fine-tuning, we take the encoder network f and attach a classification head which takes the output of f as its input and predicts the label of the example. We optimize the cross-entropy classification loss and tune the parameters of both f and h.
While pre-training can be run for pre-determined number of epochs, much like normal supervised training, how many is needed largely depends on the model and dataset. To this end, we propose using early stopping on the validation InfoNCE loss. Given unlabeled validation data, we cycle through it for some number of epochs, running our proposed method to generate (a!*), ,() pairs. Once built, the loss on this static set is tracked during pre-training. Prototypical loss curves are shown in the Appendix.
4 EXPERIMENTS
We evaluate the impact of SCARF pre-training on test accuracy after supervised fine-tuning in three distinct settings: on the full dataset, on the full dataset but where only 25% of samples have labels and the remaining 75% do not, and on the full dataset where 30% of samples undergo label corruption.
Datasets. We use 69 datasets from the public OpenML-CCC18 benchmark' under the CC-BY licence. consists of 72 real-world classification datasets that have been manually curated for effective benchmarking. Since we're concerned with tabular datasets in this work, we remove MNIST, Fashion- MNIST, and CIFAR10. For each OpenML dataset, we form 70%/10% 100% /20 train/validationntest splits, where different split is generated for every trial and all methods use the same splits. The percentage used for validation and test are never changed and only training labels are corrupted for the label noise experiments.
Dataset pre-processing. We represent categorical features by a one-hot encoding, and most of the comuptiin methods explored in the ablations are on these one-hot encoded representations of the data (with the exception of SCARF, where the marginal sampling is done before one-hot encoding). We pre-process missing data as follows: if feature column is always missing, we drop it. Otherwise, if the feature is categorical, we fill in missing entries with the mode, or most frequent, category computed over the full dataset. For numerical features, we impute with the mean. We explore scaling numerical features by z-score, min-max, and mean normalization. We find that for a vanilla network (i.e. control), z-score normalization performed the best for all but three datasets (OpenML. dataset ids 4134, 28, and 1 1468), for which no scaling was optimal. We thus do not scale these three datasets and z-score normalize all others.
Model architecture and training. Unless specified otherwise, we use the following settings across all experiments. As described earlier, we decompose the neural network into an encoder f. a pre- training head g, and classification head h, where the inputs to g and h are the outputs of f. We choose all three component models to be ReLU networks with hidden dimension 256. ff consists of layers, whereas both g and h have layers. Both SCARF and the autoencoder baselines use g (for both pre-training and co-training, described later), but for autoencoders, the output dimensionality is the input feature dimensionality, and the mean-squared error reconstruction loss is applied. We train all models and their components with the Adam optimizer using the default learning rate of 0.001. For both pre-training and fine-tuning we use 128 batch size. Unsupervised pre-training methods all use early stopping with patience 3 on the validation loss, unless otherwise noted. Supervised fine-tuning uses this same criterion (and validation split), but classification error is used as the validation metric for early stopping, as it performs slightly better. We set max number of fine-tune epochs of 200 and
hhttps://dcss openmll www.m//.ogg///////eee /ttg//nncchakk////////
4

---

Published as a conference paper at ICLR 2022
100% training
label smooth
100% training
SCARF AE
add. hoise AE
no noise AE
SCARF disc.
dropout
mixup
distill
control
control
SCARF
Worst
1.0
SCARF 23/25
22/75
24/90
2325
28/30
control
2150
315
0.8
SCARF
26-30
17:06
14000
24/4
0.24
0.8
SCARF
0.6
(00) UNM
SCARF AE
3002
2030
21
515
9/30
0.11
add. noise AE
604
318
6/13
9/13
11/08
add. noise AE
0.4
0.4
noise
noise
D:24
6.00
0.2
0.2
SCARF disc.
2b/3
0124
5000
30
6:18
SCARF disc.
R/7
10/14
511
15/1
0.0
0.0
SCARF
SCARF disc.
relative gain %
relative gain %
control
dispous
mieup
latel smootf
control
dropouot
mbnu
latel smootf
noise AE
add. noise AE
SCARF AE
relative gain %
relative gain %
relative gain
contral
drrponut
mixup
Lebel smooth
distel
control
dropaut
mieu
label smastt
deund
contral
drobout
mieo
lbbel smozl!!
Win matrices comparing pre-training methods against each other, and their im- provement to existing solutions. Bottom: Box plots showing pre-training methods over baselines (y-axis zoomed in). We see that SCARF pre-training adds value even when used in conjunction with known techniques.
Figure Top:
the relative improvement of different
pre-train epochs of 1000, We use 10 epochs to build the static validation set. Unless otherwise noted, we use a corruption rate of 0.6 and temperature of 1, for ScARF-based methods. All runs are repeated 30 times using different train/validation// splits. Experiments were run cloud cluster of CPUs, and we used about one million CPU core hours in total for the experiments.
Evaluation methods. We use the following to effectively convey the results across all datasets.
Win matrix. Given M methods, we compute "win" matrix W of size M x M, where the (i, entry is defined as:
69 1 [mehhod beats j on dataset d]
W
Dde -1 [method beats j on dataset d] + 1l [mehhod i loses to j on dataset d]
"Beats" and "loses" are only defined when the means are not a statistical tie (using Welch's 's f-test with unequal variance and p-value of 0.05). A win ratio of 0/1 means that out of the 69 (pairwise) comparisons, only one was significant and it was a loss. Since 0/1 and 0/69 have the same value but the latter is more confident indication that is worse than
present the values in fractional form
5

---

Published as a conference paper at ICLR 2022
and use a heat map. We add an additional column to the matrix that represents the minimum win ratio across each row.
Box plots. The win matrix effectively conveys how often method beats another but does not capture the degree by which. To that end, for each method, we compute the relative percent improvement some reference method each dataset. We then build box-plots depicting the distribution of the relative improvement across datasets, plotting the observations as small points. We only consider datasets where the means of the method and the reference are different with p-value 0.20. We use a larger y-value here than when computing the win ratio because otherwise some methods would not have enough points to make the box box-plots meaningful.
Baselines. We use the following baselines:
Label smoothing (Szegedy al., 2016), which has proved successful for accuracy (Müller et al., 2019) and label noise (Lukasik et al., 2020). We use a weight of 0.1 on the smoothing term.
Dropout. We use standard dropout (Srivastava et al., 2014) using rate 0.04 on all layers. Dropout has been shown to improve performance and robustness to label noise (Rusiecki, 2020).
Mixup (Zhang et al., 2017), using a = 0.2.
Autoencoders (Rumelhart al., 1985). We use this as our key ablative pre-training baseline. We use the classical autoencoder ("no noise AE"), the denoising autoencoder (Vincent et al., 2008; 2010) using Gaussian additive noise ("add. noise AF") as well as SCARF'S corruption method ("SCARF AE"). We use MSE for the reconstruction loss. We try both pre-training and co-training with the supervised task, and when co-training, we add 0.1 times the autoencoder loss to the supervised objective. We discuss co-training in the Appendix as it is less effective than pre-training.
SCARF data- aatta-aummetttionn. technique, we skip pre-training and instead train on the corrupted inputs during supervised fine-tuning. We discuss results for this baseline the Appendix as is less effective than the others.
order isolate the effect of our proposed feature corruption
Discriminative SCARF. Here, our pre-training objective is to discriminate between original input features and their counterparts that have been corrupted using proposed technique. To this end, we update our pre-training head network to include a final linear projection and swap the InfoNCE with binary logistic loss. We classification error, not logistic loss, as the validation metric for early stopping, as we found it to perform slightly better.
Self-distillation (Hinton et al., 2015; Zhang et al., 2019a). We first train model on the labeled data and then train the final model on both the labeled and unlabeled data using the first models' predictions as soft labels for both sets.
Deep k-NV (Bahri al., 2020), a recently proposed method for label noise. We set k 50.
Bi-tempered loss (Amid et al., 2019), recently proposed method for label noise. We use 5 iterations, 1 0.8, and t2 1.2.
Self-training (Yarowsky, 1995; McClosky et al., 2006). A classical semi-supervised method each iteration, we train on pseudo-labeled data (initialized to be the original labeled dataset) and add highly confident predictions the training set using the prediction the label. We then train our final model on the final dataset. We use a softmax prediction threshold of 0.75 and run for 10 iterations.
Tri-training (Zhou & Li, 2005). Like self- toff-training,, but using three models with different initial labeled data via bootstrap sampling. Each iteration, every model's training set updated by adding only unlabeled points whose predictions made by the other two models agree. It was shown to be competitive modern semi-supervised NLP tasks (Ruder & Plank, 2018). We use same hyperparameters self-training.
4.1 SCARF PRE-TRAINING IMPROVES PREDICTIVE PERFORMANCE
Figure 2 shows our results. From the first win matrix plot, we see that all five pre-training techniques considered improve over no pre-training (control), and that SCARE outperforms the others and has

---

Published as a conference paper at ICLR 2022
30% label noise
SCARF AE SCARF
add. Hoise AE
no noise AE
SCARF disc.
30% label noise
control
worst
1.0
dropout
mirup
label smooth
distil
deep knn
Diterpeeed
control
contral
SCARF
SCARF 21/36
24/44
20038
30443
28135
0.70
0.8
SCARF
1023
2413
1824
1217
1370
11:33
1422
SCARF AE
1306
0.6
0.6
add. noise AE
1308
3503
3417 1817
1220
1121
15/22
6/14
4/1
4078
6:34
0.4
nO noise AE
noise AE
3523
134-3
371
4.33
13723
0.19
0.2
0.2
SCARF
1002
SCARF disc.
SCARF
SCARF disc.
7.5
7.5
relative gain %
5.0
relative gain
2.5
2.5
0.0
0.0
-2.5
-2.5
-5.0
+9.0
-7.5
contrl
dinpoud
label distal
deep ertorpeene
control
dispooo
mikop
label prosotth
diee kmo1
Iitempennd
noise AE
add, noise
SCARF AE
relative gain %
7.5
relative gain %
7.5
@elative gain
2.5
2,5
0.0
3.0
550
55.0
-5.0
control
drqgosal
manp
distil
exterpreed
control
miag
usbel kmaeoth
dret krdt
anterppreed
miog
cabel smonth dianp
deep kint
biterpennd
deus
label
Figure 3: SCARF boosts baseline performance even when 30% the training labels are corrupted. Notably, it improves state-of-the-arr label noise solutions like Deep k-NN.
more statistically significant wins. The second win matrix shows that SCARF pre-training boosts the performance of mixup, label smoothing, distillation, and dropout, and it does so better than alternatives. In other words, pre-training complements existing solutions, suggesting that distinet mechanism is at play here. The box plots expand on the second win matrix, showing the relative improvement that each of the pre-training strategies confers over the basclines. Table summarizes the box-plots by the average relative gain. We observe that SCARF generally outperforms the alternatives and adds 1-2% relative gain across the board.
4.2 SCARF PRE-TRAINING IMPROVES PERFORMANCE IN THE PRESENCE TO LABEL NOISE
To show how pre-training improves model robustness when the data's 's labels are unreliable, we do as follows. Firstly, label noise robustness is often studied in two distinct settings (1) when some subset of the training data is guaranteed to be uncorrupted and that set is known in advance, (2) when the entire dataset is untrustworthy, For simplicity, we consider setting 2 in our experiments. We corrupt labels as follows: leaving the validation and test splits uncorrupted, we select a random 30% percent of the training data corrupt and for each datapoint, the classes, uniformly over all classes (including the datapoint's 's true class). Results are shown in Figure and Table 1. Again, see SCARF outperforms the rest and boosts all baselines by 2-3%.
replace its label by one of
7

---

Published as a conference paper at ICLR 2022
100% labeled training
SCARF
SCARF AE
no noise AE
add. noise AE
SCARF disc.
control
2.352
2.244
1.107
1.559
0.574
dropout
1.609
1.196
0.623
1.228
-1.312
mixup
1.72
1.183
-0.377
0.971
-0.307
label smooth
1.522
0.711
-0,002
1.04
-0.894
distill
2.392
2.186
0.823
1.431
-0.394
25% labeled training
control
3.692
1.702
0.777
1.662
0.233
dropout
2.212
1.848
2.013
1.155
-0.322
mixup
2.809
0.73
0.106
0.439
0.466
label smooth
2.303
0.705
-0.564
0.196
-0.206
distill
3.609
2.441
1.969
2.263
1.795
self-train
3,839
2.753
1.672
2.839
2.559
tri-train
3.549
2.706
1.455
2.526
1.92
30% label noise
control
2.261
1.988
0.914
1.612
-1,408
dropout
2.004
2.058
0.9
1.471
-2.54
mixup
2.739
1.723
0.116
1,409
0.189
label smooth
2.558
1.474
0.703
1.395
-1.337
distill
2.881
2.296
-0,239
1.659
-0,226
deep knn bitempered
2.001
1.281
0.814
1.348
0,088
2.68
2.915
0.435
1.387
-1.147
Table Results using the fully labeled training data, only 25% labeled training data, and full training data subject to 30% label noise. Shown is the average relative gain in accuracy when adding the pre-training methods (columns) to the reference methods (rows). Like the box-plots, we filter out datasets using p-value 0.20. We see that SCARF consistently outperforms alternatives, not only in improving control but also improving methods designed specifically for the setting.
4.3 SCARF PRE-TRAINING IMPROVES PERFORMANCE WHEN LABELED DATA LIMITED
To show how pre-training helps when there is more unlabeled data than labeled ones, we remove labels the training split so that only 25% of the original split remains labeled. Autoencoders, SCARF, self-training and tri-training all leverage the unlabeled remainder. Results are shown in Figure (Appendix) and Table I. SCARF outperforms the rest, adding a very impressive 2-4% to all baselines.
4.4 ABLATIONS
We now detail the importance of every factor in SCARE. We show only some of the results here; the rest are in the Appendix.
Other corruption strategies are less effective and we ablate the marginal sampling corruption technique we proposed, replacing it with the following other promising strategies, while keeping all else fixed.
sensitive to feature scaling, Here,
1. No corruption. We not apply any corruption i.e. (((() 20) in Algorithm
this case, the cosine similarity between positive pairs is always one and the model is learning to make negative pairs orthogonal possible. Under the recent perspective (Wang & Isola, 2020) that the contrastive loss comprises two terms one that encourages alignment between views of the same example- and one that encourages the hypersphere embeddings to be uniformly spread out see that with corruption, pre-training may just be learning to embed input examples uniformly on the hypersphere.
2. Mean corruption. After determining which features to corrupt, we replace their entries with the empirical marginal distribution's 's mean.
8

---

Published as a conference paper at ICLR 2022
3. Additive Gaussian noise. We add i.dd (V, noise features.
4. Joint sampling. Rather than replacing features by random draws from their marginals to form f(6), we instead randomly draw a(0) from training data X' i.e. draw from the empirical (joint) data distribution and then set 2/) = $ vj E I,
5. Missing feature corruption. We mark the selected features as "missing" and add one learnable value per feature dimension to our model. When a feature missing, it's filled in with the corresponding learnable missing value.
6. Feature dropout. We zero-out the selected features.
We also examine the corruption strategies under the following ways of scaling the input features.
1. Z-score scaling. Here, Ij = [zr mean (X2)) /sid (Xz).
2. Min-max scaling. I = [xy - min (2)) [max (X2) min (x2))-
3. Mean scaling. Iÿ =I mean (Zt)) [max (X₃) min (X2))-
Figure (Appendix) shows the results for z-score and min-max scaling. SCARF marginal sampling generally outperforms the other corruption strategies for different types of feature scaling. Marginal sampling is neat in that in addition to not having hyperparameters, it is invariant to scaling and preserves the "units" of each feature. In contrast, even a simple multiplicative scaling requires the additive noise to be scaled in the same way.
SCARF is not sensitive to batch size. Contrastive methods like SimCLR (Chen et al., 2020) have shown consistent improvements upon increasing the batch size, N. There is a tight coupling between the batch size and how hard the contrastive learning task is, since, in our case, the loss term for each example i involves I positive and N 1 negatives. The need for large (e.g. 5000) batch sizes has motivated engineering solutions to support them (Gao & Zhang. 2021) and have been seen as grounds for adopting other loss functions (Zbontar al., 2021). Figure 5 (Appendix) compares a range of batch sizes. We see that increasing the batch size past 128 did not result in significant improvements. reasonable hypothesis here is that higher capacity models and harder tasks benefit more from negatives.
SCARF is fairly insensitive to the corruption rate and temperature. We study the effect of the corruption rate e in Figure (Appendix). We see that performance stable when the rate in the range 50% 80%. We thus recommend default setting of 60%. We see similar stability with respect to the temperature hyperparameter (see Appendix). We recommend using a default temperature of 1.
Tweaks to the corruption do not work any better. The Appendix shows the effect of four distinct tweaks to the corruption method. We do not see any reason to use any of them.
Alternatives to InfoNCE do not work any better. We investigate the importance of our choice of InfoNCE loss function and see the effects of swapping it with recently proposed alternatives Alignment and Uniformity (Wang & Isola, 2020) and Barlow Twins (Zbontar et al., 2021). We found that these alternative losses almost match or perform worse than the original and popular InfoNCE in setting. See the Appendix for details.
5 CONCLUSION
Self-supervised learning has seen profound success in important domains including computer vision and natural language processing, but little progress has been made for the general tabular setting. We propose a self-supervised learning method that's simple and versatile and learns representations that are effective in downstream classification tasks, even in the presence of limited labeled data or label noise. Potential negative side effects of this method may include learning representations that reinforce biases that appear in the input data. Finding ways to mitigate this during the training process is potential direction for future research.

---

Published as a conference paper at ICLR 2022
REFERENCES
Ehsan Amid, Manfred K Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. arXiv preprint arXiv:1906.0336: 2019.
Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. International Conference on Machine Learning, pp. 540-550. PMLR, 2020.
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. Unilmv2: Pseudo-masked language models for unified language International Conference on Machine Learning, pp. 642-652. PMLR, 2020.
model pre-training
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael Man- tovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprini 1.0::0170...331. 2017.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International Conference on Machine Learning, pp. 517-526. PMLR, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2400:20005..155,, 2020.
Ting Chen, Xiaohua Zhai, and Neil Houlsby. Self-supervised gan to counter forgetting. arXiv preprint 1810. 11590, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Kevin Clark, Minh- Than-Thang Luong, Quoc Le, and Christopher D Manning. Electra: Pre-training text encoders discriminators rather than generators. arXiv preprint arXiv:2003. wwXiv:2003//0555, 2020.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12 (ARTICLEE):2993-- 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. Bertje: dutch bert model. arXiv preprint 191xv:1112.0955.. 2019.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1rX.::8800004805. 2018.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.09782. 2016.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03190 2019.
Matthias Feurer, Jan N. Rijn, Arlind Kadra, Pieter Gijshers, Neeratyoy Mallik, Sahithya Ravi, Andreas Mueller, Joaquin Vanschoren, and Frank Hutter. Openml-python: an extensible python api for openml. arXiv, 1911.02490, 2019. URL https::///// httpppp:///aaaio.oospp////////// 02490 pdf.
Markus Freitag and Scott Roy. Unsupervised natural language generation with denoising autoencoders. arXiv preprint o 1804.88894 2018.
Luyu Gao and Yunyi Zhang. Scaling deep contrastive learning batch size with almost constant peak
memory usage. arXiv preprint 10Xi::2001006983 2021.
10

---

Published as a conference paper at ICLR 2022
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 2018.
Raphael Gontijo-Lopes, Sylvia Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprini arXiv:2002.0887 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint 1406. 2661, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint 00t1::0006.07333, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. International Conference on Machine Learning, pp. 4116-4126. PMLR, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, pp. 4182-4192. PMLR, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018.
Daniel Ho, Eric Liang, Xi Chen, lon Stoica, and Picter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference on Machine Learning, pp. 2731-2741. PMLR, 2019.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):1-14, 2017.
Simon Jenni and Paolo Favaro. Self-supervised feature learning by learning to spot artifacts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2733-2742, 2018.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining, arXiv preprint arXiv:1901.07291, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 2019.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In European conference computer vision, pp. 577-593. Springer, 2016.

---

Published as a conference paper at ICLR 2022
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization proxy task for visual understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6874-6883, 2017. 2017.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-tu seneence-to-equeeee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 19:::99100 10. 13461, 2019.
Yin Li, Manohar Paluri, James M Rehg, and Piotr Dollár. Unsupervised learning of edges. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1619-1627, 2016.
Sunghin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. arXiv preprint urXiv:1905.00397 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 2019.
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint (.XXvv:906.02611 2019.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 2020.
PMLR,
David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing, In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pp. 152-159, 2006.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed repre- sentations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546. 2013.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6707-67 17, 2020.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. Advances in neural information processing systems, 26:2265-22777 2013.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? arXiv preprint ar SSt.::190600622999 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arX.v:1807.03748 2018.
Daniel S Park, William Chan, Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1xXiv:190..08777 2019.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint 1rXiv:1112.04221, 2017.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive supervsed learning: Invariances, augmentations and dataset biases. arXiv preprint arXiv:2007.13916. 2020.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pp. 1-26, 2020.
12

---

Published as a conference paper at ICLR 2022
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 15:::::11.666 1511.0643,, 2015.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019.
Alexander Ratner, Henry R Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learning compose domain-specific transformations for data augmentation. Advances in neural information processing systems, 30:3239, 2017.
Sebastian Ruder and Barbara Plank. Strong baselines for neural semiisuuervised learning under domain shift. arXiv preprint /804
2018
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Andrzej Rusiecki. Standard dropout remedy for training deep neural networks with label noise. In International Conference on Dependability and Complex Systems, pp. 534-542. Springer, 2020.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv: /wt.///05502202 02450, 2019.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1): ((1)):199--1953) 1958, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. arXiv preprint 2rX14:2010.07432, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 1506::9906005844 2019.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid, A bayesian data augmentation approach for learning deep models. arXiv preprint arXiv: 10564, 2017.
Hsiao- Yu Fish Tung, Hsiao-Wei Tung, Ersin Yumer, and Katerina Fragkiadaki. Self-supervised learning of motion capture. arXiv preprint arXiv:1712.013:: 2017.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explorations, 15(2):49-66 60, 2013. doi: 10.1145/2641190 19w.:/41199922911999999 URL http: doi. acm. wttg//.....00 http://0..11///2560 21155/2541119 264119.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoing Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Liang Wang, Wei Zhao, Ruoyu Jia, Sujian Li, and Jingming Liu. Denoising based scquence-to- sequence pre-training for text generation. arXiv preprint (00iv::990..0000 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. PMLR, 2020.
13

---

Published as a conference paper at ICLR 2022
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pp. 2794-2802, 2015.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. arXiv preprint arXiv:1805.01975 2018.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint WwXiv:/906.082.7 2019.
Transheng Yao. Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H Chi, Steve Tjoa, Jieqi Kang. al. Self-supervised learning for large-scale item recommendations. arXiv preprint 2xXiv:2007.2266, 2020.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33: 1033-11043, 2020.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032:0, 2021.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi-
supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476-1485, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9713-3722, 2019a.
Richard Zhang, Phillip Isola, and Alexei A Efros, Colorful image colorization. In European conference on computer vision, pp. 666. Springer, 2016.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. arXiv preprint arXhv:1912.11188, 2019b.
Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering, 17211)1159--154 2005.
14

---

Published as a conference paper at ICLR 2022
A
APPENDIX
We now present findings held out from the main text. Unless noted otherwise, we use the same hyperparameters (i.e. c 0.6, temperature 7 = etc.).
A.1 RESULTS FOR RATA-LIMITED EXPERIMENTS
Figure presents the results for the case when only 25% of the training labels are available. We see that SCARF outperforms the rest here well.
A.2 PRE-TRAINING LOSS CURVES
Figure shows prototypical pre-training training and validation loss curves for SCARF. We see that both losses tend to decrease rapidly at the start of training and then diminish slowly until the early stopping point, We also notice the noisy (high variance) nature the training loss this is due the stochasticity of our corruption method.
A.3 ABLATIONS CONTINUED
We present more ablation results, where the metric is accuracy using 100% of the labeled data.
IMPACT OF BATCH SIZE AND CORRUPTION RATES
Figure examines the impact of the batch size and corruption rate for the fully labeled, noiseless setting.
IMPACT OF CORRUPTION STRATEGIES
Figure 6 shows the performance of variety of alternative corruption strategies for SCARF under z-score and min-max feature normalization. Results for mean scaling are shown Figure 12.
IMPACT OF TEMPERATURE
Figure 8 shows the impact of the temperature term. While prior work considers temperature an important hyperparameter that needs to be tuned, we that default of (i.e. just softmax) works the best in our setting.
MORE CORRUPTION ABLATIONS
Figure 1l shows the following points.
Corrupting one view is better than corrupting both the views. The likely explanation for this that at a corruption rate of 60%, corrupting both views i.i.d means that fraction of feature indices that contain the same value for both views is small in other words, there is less information between the two views and the contrastive task is harder.
Corrupting the same set of feature indices within the mini-batch performs slightly worse than random sampling for every example in the mini-batch.
An alternative way to select feature indices to corrupt is use Bernoulli's: s: for each feature index, corrupt it with probability (corruption rate) c, ensuring that at least one index is corrupted. In the selection method we describe in Algorithm a constant number of indices are corrupted, whereas here it is variable. This altermative way of selecting indices performs roughly the same.
Once the feature indices to corrupt are determined, rather than corrupting them by drawing from the empirical marginal distribution, training set and use its feature entries for corrupting all views for the mini- batch. This performs worse
instead draw random example from the
15

---

Published as a conference paper at ICLR 2022
25% training
SCARF AE
add. Hoise AE
no noise AE
SCARF disc.
25% training
control
SCARF
worst
labe! smooth
1.0
dropout
mirup
distil
self-train
tr-train
control
SCARF 26-22
26/33
3007
22442
1443
0.79
0.8
SCARF
15:3
21/23
2227
1256
1270
1270
1423
SCARF AE
1503
0.6
0.6
add. noise AE
576
33/33
822 1217
1320
11:1
13/23
1/33
1524
6:14
0.4
no noise AE
noise AE
3522
5:42
30:25
9:24
23133
0.21
0.2
0.2
SCARF
1542
SCARF disc.
SCARF
SCARF disc.
15
15
relative gain %
relative gain %
10
10
0
control
drapeuu
uabll smooth
diang
setfrrain
bi-eran
contrrol
drapenul
mapp
label smoooh
disig
detforein
ti-tran
no noise AE
add. noise AE
SCARF AE
relative gain %
relative gain %
relative gain
10
10
control
draponot
saiftrain
trimein
conttol
dropoutt
mispp
latel smartt
diall!
Setf Selkirain
cantrol
minap
latel smouth
diaii!
Seildraah
traii
babel
Figure 4: SCARF shows even more significant gain in the semi-supervised setting where 25% of the data is labeled and the remaining 75% is not. Strikingly, pre-training with SCARF boosts the performance of self-trane self-training and tri-training by several percent.
feature corruption rate (%)
batch size
worst
20
50
60
80
90
26
6g
228
256
512
worst
1.0
1/7
0/20
0/22
0/26
0/26
031
1/29
322
0.00
20 6/7
0/12
0/16
0/23
0/25
023
2/26
3/24
0.00
0.8
20/21
1/37
022
0/28
025
0.00
30
40
22/22 16/6 7/7
144
1/8
1/12
1/14
7/18
0.07
0.6
oone WI
64
16/17
0.0
0.00
50
282623231311
3M
1/4
1/5
3/7
9/11
0.20
Tato
23/21
0/2
0.4
26/26
0.39
0.4
70
31/33 23/3366/7 11/12
4/5
112
2/2
7/7
0.50
256
40/40
26/28
8/8
2/2
0/1
0.00
0.2
80
28/29
512
26/29 17/22
33/18
0.0
0.0
Figure 5: Win matrix for various batch sizes (Left) and corruption rates (Right) for the fully labeled, noiseless setting.
16

---

Published as a conference paper at ICLR 2022
other corruptions, z-score norm.
other corruptions, min-max norm.
contral
SCARF
joint
zero
additive
missing
no comupt mean
worgt
control
SCARE
Joint
zero
additive
missing
no comupp mean
worst
1.0
1.0
control
206
8/27
3244
2:22
305
623
7/10
0.36
control
2/36
9/4
3:1
4:1
2/3
2/32
2115
9.00
SCARF
0.8
SCARF
30/31
0.8
joint 2427
6:11
611
13:35
30053
0.45
jaint
14:24
9/38
14122
10:20
29/90
mieu UIM
zero
552
1/
0.14
zero 25/3
310
23020
adeitive 2372
5/3
703
0.25
add additive
27/33
4/1
802
8:26
622
27/73
0.4
0.4
missing 93155
8/12
4/7
611
24/26
0.44
missing
4/12
4/16
10/31
0.00
0.2
30/33
0.2
no corrupt
810
224
322
2/25
2/26
422
0:38
corrupt
191
180
5:88
5V96
1/81
319
0:09
0.0
Figure 6: Left: Win matrix comparing different corruption strategies when z-score feature normaliza- tion is used in the fully labeled, noiseless setting. Right: The same matrix but when min-ma, feature scaling used. We see that SCARF better than alternative corruption strategies for different types of feature scaling.
pre-train training loss
pre-train validation loss
4.7
4.5
4.6
4.3
4.2
4.-
10000
8000
number of steps
number of steps
Figure 7: Training and validation loss curves for SCARF pre-training on Phonemes (dataset id 1489). We observe that both losses drop rapidly initially and then taper off. The training curve is jittery because of the random corruptions, but the validation curve isn't because the validation dataset is built once before training and is static throughout training.
ALTERNATIVE LOSSES
Figure 8 shows the impact of using two recently-rroosed alternatives the InfoNCE loss: Uniform and Align (Wang & Isola, 2020) and Barlow Twins (Zbontar et al., 2021). We use 5e-3 for the hyperparameter in Barlow, and equal weighting between the align and uniform loss terms. We find no benefit in using these other losses.
SCARF PRE- TRAINING OUTPERFORMS SCARF CO-TRAINING
Here we address the question: better to pre-train using SCARF or to apply it as term in the supervised training loss? In particular, Ceu Sappervised + Acumt Coont where Leomt is as described in Algorithm 1. Figure shows that pre- outperforms co-training for a range of different Agaat- We see this is also the case for additive noise autoencoders.
SCARF PRE-TRAINING OUTPERFORMS SCARF DATA AUGMENTATION
Figure shows that using SCARF only for data augmentation during supervised training performs far worse than using it for pre-training.
17

---

Published as a conference paper at ICLR 2022
infoNCE temperature
other losses
Oz
0,2
40
worst
uniform slign
1.0
InfoNCE
barlow
Wrorsz
1.0
0.01
0/13
4/20
1527
0.00
0.8
infoNCE
0.1
19013
4012
1525
0.33
0.6
oneo UM
0.6
barlow
30G2
aqp
1623
8/12
36117
0.67
0.4
0.2
mnlform-align
6G2
232
6.17
0.2
1227
1025
1217
0.06
0.0
Figure Left: Barlow Twins loss performs similar to InfoNCE while Uniform-Align performs worse. Right: InfoNCE softmax temperature 1 performs well.
pre-train co-train SCARF
pre-train co-train add. noise AE
pre-train
Eetrran 0.01
Co-train 0,1
co-train 1.
pro-train
co-train 0.01
co-train 0.1
co-briin
warst
Worst
pre train
3422
1175
2028
180
train
10
1922
2.1
0.8
0.8
co-train 0.01
317
311
0.6
co-train 0.01
1/1
1551
0.55
co-train 0.1
2025
0.4
co-train 0.1
0.2
0.2
co-train
co-train 1.
100
207
Figure 9: Left: Pre-training with SCARF beats co-training for range weights on the co-training term. Right: The same is true for additive noise autoencoders.
SCARF wre-train vs. SCARF data augment
val. loss metric (for early stopping)
SCARE data augmmnnt
InfoNCE loss
InfoNIC error
Worst
1.0
SCARF
Wors
0.8
SCARF
2204
8:92
0.6
0.6
0.4
InfoNCE error
SCARF data augment
204
0.2
0.2
Figure 10: Left: Using SCARF only for data augmentation during supervised training performs worse than using it for pre-training. Right: Using InfoNCE error instead of InfoNCE loss as the validation metric for early stopping degrades performance.
USING INFONCE ERROR AS THE PRE-TRAINING VALIDATION METRIC IS WORSE
SCARF uses the same loss function (InfoNCE) for training and validation. If we instead use the InfoNCE error for validation where enror occurs when the model predicts off-diagonal entry of the batch-size by batch-size similarity matrix downstream performance degrades, as shown in Figure 10.
18

---

Published as a conference paper at ICLR 2022
the same feature indices across batch?
corrupt one or views?
worst
1.0
one view
two views
worst
1.0
0.8
0.8
yes
view
0:75
0.6
oNE/ UNM
0.6
0.4
two views
1/4
0:25
nD
0.0
0.0
apply Bernoulli feature sampling?
corrupt using single draw per batch?
worst
1.0
worst
1.0
nl
0.8
0.8
yes
0.6
rato UM
0.6
Opel UM
0.4
0.4
yes
nD
0.0
0.0
Figure 11: Top left: Corrupting one view is better than corrupting both. Top right: Using different is better than using same set across the
random feature indices for each example in the batch. Bottom left: Selecting a variable number of feature indices via coin flips performs similar to the method described in Algorithm 1. Bottom right: Corrupting by replacing the features by the features of single drawn example for the whole mini-batch performs worse.
other corruptions, mean norm.
different uniform-align settings
control
SCARF
joint
zero
additive
missing
mean
corrupt
worst
a
a=1.0,
1.0.0,
6=1.0.
#=1.0
1.0
Us
u=0
Us
uw?
Un3
worst
127
2029
223
SCARF
49
712
23/00
0.8
joint 2127
5:7
25:30 0113
1012
51153
0.45
2-1.0, u-0
4:34
1:22
227
225
0.3
zero
32766
00e2 Uh
additvve
1829
427 5:20
470
473
0.23
13/00
0.35
a-1.0,
0.4
missing
3.1.0. u-2
2002
2107
10
0.50
mean 11/22
3/2
5:35
1/3
2002
204
23/2
0.35
0.2
no corrupt
8-1.0, u-3
0.0
0.0
Figure 12: Left: SCARF'S corruption strategy remains competitive for "mean" feature scaling, was the case for both z-score and min-max scaling. Right: Comparison of different hyperparameters for the uniform-align loss. and are the weights on the align and uniform terms respectively.
MORE ON UNIFORM-ALIER LOSS
Figure 12 compares SCARF using the uniform-align loss, for different weights between the align and uniform loss terms. The best performance is achieved with equal weight between the two, and Figure 8 shows that this underperforms vanilla InfoNCE. We thus recommend using the vanilla InfoNCE loss.
19

---

Published as a conference paper at ICLR 2022
ABSOLUTE PERFORMANCE
We have thus far presented results terms of relative improvement over control since relative gain is easier to summarize across the many datasets we consider. In this section, we present the absolute test accuracy for all methods and datasets, for the keen reader. We also add traddeen-bbostee decision tree baseline using the XGBoost package, since decision trees are common among practitioners working with tabular data. We use the Python API 2, version 0.6, and choose the default settings for XGBClassifier (max depth of 3, 100 estimators, learning rate of 0.1).
Tables 2, 3. show the absolute test accuracies (averaged over the 30 trials) for each dataset and baseline for the 100% training data, the 30% label noise, and the 25% (semi-supervised) training data settings respectively.
TRAINING SPEED
We use an early stopping criteria when fine-tuning SCARE and baselines. Table shows the number of actual training epochs used (averaged over the 30 trials) for each dataset and baseline for the 100% training data setting.
readthedocs. http::/wh...aaaaooooooeo/eeeeeea htm1
20

---

Published as a conference paper at ICLR 2022
6
64.69
95.75
94.6
94.56
94.36
95.1
$8.01
6332
74.24
75.4
71.76
69.35
66.6
69.
77.59
54
76.75
75.14
88
76.16
50
90.54
97.53
96. 98.68
97.33
95.91
96.66
90.61
469
18.54
19.08
19.89
19.14
19.79
19.17
21.21
458
98.96
99.51
99.34
99.2
99.11
99.19
98.46
4538
56.28
60.32
56.78
$6.89
56.84
$7.06
56.31
4534
96.08
95.8 85
95.91
95.8
95,9
96.46
94.66
4134
75.53
75.51
76.25
75.52
75.77
75.47
78.31
81.75
40094
90.97
90.91
91.00 01.02
91.5
90.99
90.86
94.29
40984
90. 56
91.97 97
90.32
90.46
90.53
90.69
92.
40983
59
98.7
98.66
98.77
98.6
40982
73-48
73.27
73.47
73.7
73.56
72.99
78.07
40978
94 84
95.13 13
95.36
95.81
94.75
94.74
97.33
40975
97.26
96
95.87
40966
98.44
98.77
98 98.21
98.46
98.65 18.68
97.87
97.05
40923
2.17
2.19
2.19
2.19
2.19
2.19
77.49
40701
92.36
92.36
95.31
4067U
92.37
92. 88
92.88
93.27
92.25
92.53
96.27
40499
05
98.9
98 665
98.64
98.95
98.63
97.58
98.53
97.72
97.65
97.6
38
97.14
97.75
97.09
97.19
97.18
97.45
98.42
37
75.21
75.36
75.15
75.58
75.57
74.58
76.47
32
99.25
98.39
31
73.55
73.86
73.3
72.68
72.92
71.92
76.38
300
94 46
94.37
94.3
94.44
94.18
94.47
94.73
29
86.35
85.65 35.62
85.57
85.14
86.16
28
97.56
97.88
97.63
97.76
97.44
97.55
97.29
23
54.25
53.22
$3.27
53.03
53.41
55.77
23517
52
23381
60.59
$9.07
56.78
$6.68
53.22
$4.62
$9.7
18
72.59
74.06
73.51
72.94
73.69
73.06
72.12
188
64.24
67.34
66.09
64.95
64.7
64.89
182
89.21
90.43
89.15
89.17
88.98
89,42
89.63
16
95.60 6R
96.55
95.51
95.55
95.47
95.
94.85
1590
85.55
85.6
85.51
85.51
85.54
85.5
86.45
151
83.55
83.27
1510
96.94
96.49
96.27
96.54
96.21
96.58
96.26
1501
91.41
93.76
91.55
91.6
92.49
92.35
91.33
14
79.27
80.99
81.05
80.67
80.89
78,82
$2.34
1497
88.73
93.65
89.
88.51
88.99 98.94
89.21
99.69
1494
76
1489
85.25
87.55
85.84
85,86
85.03
87.15
85.63
1487
94.25
94.27
94.31
94.27
93.96
94.34
1486
95.74
.6
95.74
95.79
95.76
95.74
95.63
1485
54.2
70.16
68.76
68.24
64.12
56.73
71.6
1480
70.
69.74
69.49
69.19
69.23
69.64
68.01
1478
96. 85
97.16
97.08
97. 12
97.07
94,98
98. 38
1468
93.45
94.04
92.95
91.68
92.72
85,79
91.27
1464
79.20
78.53
79.31
79.13
78.04
77.58
1462
99.5
99.94
99.75
99.72
99.77
99.92
99.49
1461
90.42
90.46
90.43
90.43
90.41
90.45
90.36
12
96.82
96.46
95.96
95.99
95.97
96.42
96.17
11
95.98
97.27
96.87
97.15
97.05
97.16
87.95
1068
1067
85.6
85 69
85.66
85,57
85:45
85,46
85,36
1063
84.17
84.06
84.06
83.71
82.16
83.7
83.27
1053
81.47
81.19
81.28
81.21
81.27
81.19
81.44
1050
89.80 84
89.49
89.24
89.27
89.4
89.7
88.9
1049
90.65
89.97
90.2
90.38
90.74
90.17
91.2
Table 2: Absolute test accuracy (percent; averaged over 30 trials) for the 100% training data setting, for each dataset and every baseline, including gradient-boosted decision tree (XGBoost) baseline (max depth of 3, 100 estimators).
21

---

Published as a conference paper at ICLR 2022
no
6
90.45
90.57
90.02
90.01
89.65
90.7
85.53
6332
69.
70.99
63.41
62.78
60.25
62.96
72.75
72.21
70.84
71.11
72.76
72.92
50
79.64
87.56
90.52
86.7
85.14
88. 12.12
87.4
93.05
469
19.47
19.02
20.08
19.89
19.41
18.
19.52
458
96.65
99.27
98. 98.06
97.97
97.1
95.47
96.07
4538
52.24
54.25
52.07
52.15
$1.94
52.35
54.88
4534
94.13
94.38
94.2
94.09
94
94,31
93.66
4134
72.09
71.17
71.99
72.13
72.2
72.31
76.95
41027
40994
91.33
91.37
90.74
91.16
91.53
91.91
90.12
40984
88.63
89. .99
88,
88.51
88.08
88.68
90.71
40983
97.8 87
98.5
98.5
14
98.57
97.86
40982
70.54
70.92
71.26
70.97
71.11
70.19
73.69
92.43
40978
91.87
92.77
93.81
94,12
93.64
92.52
96.48
40975
91 1..42
90.91
91.36
40966
90.78
88.33 52
88.
90.12
91.86
87.37
89.58
40923
2.16
2.17
2.17
2.17
2.17
2.17
75.47
40701
88,
88
89.41
40670
84.41
90.34
89.6
94.39
87.63
87.66
93.94
73.67
40499
98.15 18
97.85
97.82
98. 11
98.16
98.29
95.69
94.07
94.77
94.93
95.
95.21
93.59
96.07
96.2
96. 88
96.1
96.19
95.9
96.3
97.81
75.68
74.5
73.99
73.8
74.27
73.17
72.71
98.47
98.56
97.12
31
71.15
72.12
70.66
71.04
70.63
66.9%
73
78.7
74.95
300
89.13
87.21
89. 1
89.21
88.9
89.2
90.89
84.12
83.21
84.28
83.86
84.18
81.53
82.44
95,77
96.54
96.1
95.94
95.77
95.45
95.22
$1.89
50.29
50.03
50.21
51.37
53.5
23381
58.07
57.88
$5.73
55.73
$4.7
$3.25
55.47
70.8
73.25
72.63
71.93
73.1
72.72
71.23
188
58.3
53.19
$8.
57.64
57.09
55.3
60.29
182
88.31
88.88 88
87.82
87.83
87.36
87.59
88.25
88,10
94.62
92.02
91.56
89.75
87.96
89.57
1500
84,81
85.19
85.07
85.06
85.06
85.05
86.23
151
80.08
$1.15
80
80.52
82.97
1510
94.58
94.37
92.51
92.24
92.18
93.83
92.13
1501
78.73
89.
81.16
81.73
82.16
84.43
85.6
72.34
78.57
77.87
77.52
74,92
71.53
77.64
1497
82,85
90.2
82. 64
82.24
82.82
82.72
98.75
1489
83.29
85.39
$4. 16
84.
83.48
84. 62
84.26
92.61
92.97
93.75
93.41
92.9
93.55
1486
94.89
94.71
94,91
94.95
94,91
94.96
95.12
1485
53,97
67.03
66.74
66.62
62.55
54.21
65.67
1480
71.63
69.72
68.86
68.95
68.9
69.53
68.06
1478
94, 13
94.42
94,75
94.71
94.71
94,74
96.37
1468
83.07
90.76
83, 39
79.58
78.46
64.88
87.3
1464
76.65
76.91
77.69
78.25
76.63
75.71
1462
98.97
99.55
99.14
99.21
99.52
99.69
98.07
90.06
90.21
90.
90.11
90.11
90.05
90.37
93.99
95.18
93.93
91.77
93.8
94.2
92.98
90.25
88.77
89.77
89.11
89.88
89. 68
82.43
92.45
92.24
92.1
1067
84,74
84,87
84.96
84.53
84.62
84.33
85.33
1063
83.81
83.76
83.59
83.63
82.95
83.95
80.95
1053
81.05
81.05
81.06
81.04
81.09
81.04
81.07
1050
89.08
94
88. 38.96
89.11
89.46
89.32
88.81
1049
88.95
87.45
88.53
88.69
88.93
88.58
89.2
Table 3: Absolute test accuracy (percent; averaged over 30 trials) for the 30% label noise setting, for each dataset and every baseline, including gradient-boosted decision tree (XGBoost) baseline (max depth of 3, 100 estimators).
22

---

Published as a conference paper at ICLR 2022
6
87.22
89.71
87.12
87.38
87.01
88.87
85.02
6332
63.67
66.71
58.53
57.82
57.25
61.5
69.41
54
69.27
66.59
68
69.04
50
72.63
79.2
86.85
82.9
75.78
88.62
81.34
469
18.02
18.53
19:31
19.16
19.11
17.85
18.0
458
98.03
98.48
98.6
98.21
97.54
98.16
95.58
4538
50.65
53.75
50.79
50.32
50.45
50.03
54.05
4534
94.1
94.31
93.
94,06
93.93
94.3
94.22
4134
68.31
68.44
69.65
69
68.78
68.45
74.58
40094
90.63
90.96
90.96
90.93
90.97
90,96
92.60
40984
86.48
88.05
86.
85.83
86.34
86.49
89.
40983
94.72
97.65
94.85
94.65
97.72
98.58
97.92
40982
68. 25
67.99
67.62
68.08
68.62
67.11
70.14
40978
86.05
89.95
91.12
91.49
88.9
85.88
96.32
72.47
90.55
91.14
91.
40966
86. 53
84.11
83.98
86.01
88.3
83,48
80.65
40923
2.16
2.16
2.16
2.16
2.16
2.16
74.35
40701
87,4 44
89.59
88.
88.34
93.46
4067U
81.67
90.97
89.69
90.27
87.58
88.22
94.37
40499
97. 88
97 65
97.13
97.59
98.13
97.78
94.59
94.02
94.84
94.34
38
93.13
96.92
95.39
95.17
94.53
95.39
97.7
37
70.37
72.24
71.15
71.2
70.21
71.58
72.
32
98.4
31
70.53
71 .666
68 48
68.54
65.82
69.33
73.12
300
91 66
87.0 07.01
89.98
90.42
90.58
91.06
90.3
29
83.6
83.57
80.7
81.46
81.93
82,28
84.59
28
96.43
96.56
96.38
96.08
96.23
96.12
95.07
23
51 .55
45.64
46.48
44.84
49.95
51.48
23517
23381
58.83
58.7
55.62
55.37
53.37
$6.43
55.5
73.72
18
70.71
71.37
71.05
70.53
72.39
71.11
70.11
188
55.18
46.41
50.56
53.42
$3.06
$1.72
$7.82
182
86.98
88.36
87.37
87.34
86.53
86.62
87.89
16
88.25
93.91
90.7
90.44
89.6
89.97
87.54
1590
84.9
85.01
84.89
84.89
84.92
84,86
86.34
80.7
79.61
79.4
68.68
1510
94 63
94,36
93.8
93.87
92.73
94.23
94.12
1501
81.01
S8.24
79.66
80.16
81.53
83.97
81.09
14
73.13
76.71
76.51
76.3
75.2
72.6
76.36
1497
80.15 19
90.15
80.88
80. 14
80.18
82,09
98.
1494
1489
95
84.81
83.27
82.72
82.04
84.18
83.97
1487
9
93.72
93.5
94.02
1486
95.75
94.64
94.75
94.84
94.83
94.81
95,38
1485
53.60 69.69
66.82
67.1
66.82
62.54
53,86
61.14
1480
70.5
69.57
68.72
69.42
69.62
70,63
68.52
1478
93.88 82
94.06
93.92
94.41
93.68
93.45
96.9
1468
85.0 01
88.74
80. 14
74.45
77.73
71.39
$1.67
1464
75.37
75.7
75.36
75.4
75.17
75.82
1462
98.48
99.13
98.9
98.78
99.39
99,58
97.36
1461
$9.87 87
89.94
90.04
89.94
89.92
89.81
90.22
12
91.88 83
94.38
92.78
92.68
92.95
92.86
91.49
11
88.38
87.99
85.45
85.84
89.13
84.27
82.43
1068
1067
84,88 88
84.87
84.77
84.7
84,27
84,06
84.24
1063
82.78
81.81
81.3
80.33
82.38
81.21
1053
80.82
80.91
80.87
80.89
80.95
80,94
80.98
1050
90.2
89.3
89.46
89.
89.06
89.73
88.43
1049
87.59
87.14
87.33
87.47
88.1
87.95
89.18
Table 4: Absolute test accuracy (percent; averaged over 30 trials) for the 25% training data (semi- supervised) setting, for each dataset and every baseline, including gradient-boostee decision tree (XGBoost) baseline (max depth of 3, 100 estimators).
23

---

Published as a conference paper at ICLR 2022
dataset ld
control
SCARF 17.23
SCARE
add. AE
SCARF disc.
20.1
19.3
18.42
18.77
18.1
12.53
11.1
9.85
54
12.03
12.68
11.83
12
11.5
13.95
12.92
13.58
10.63
11.65
14.63
9,02
9.32
7.8
7.55
7.15
7.53
7.93
469
7.36
7.38
7.43
7.02
6.97
7.9
4538
13.17
15.18
16.3
15.38
14.97
14,13
14.03
12.1
13.93
13.2
12.68
44
8.86
8.55
9.85
9.3
9.78
7.52
4134
9.13
12.45
11.07
11.05
10.85
10.48
41027
31.3
33.37
35.62
37.9
37.62
36.2
40994
5
5.3
5.37
6.03
5.28
5
40983
9.91
7.78
8.8
9.3
8.45
6,75
40982
10.94
10.45
12.5
12.38
11.45
12.57
40979
10.65
7.77
10
10.65
10.32
40978
1.3
13.38
13.15
13.92
13.3
40975
15.09
12.63
14.12
13.78
14.13
15.33
40966
12.62
11.37
12.28
11.7
11.1
13.87
40701
10.53
12.72
11.25
11.48
11.72
10.38
40670
9.91
8.93
10.4
9.92
9.72
10.27
40668
10.93
12.3
11,45
12.07
12.15
11.92
40499
13.19
10.95
12.55
12
11.35
13.17
12.07
11.9
11.5
12.5
11.63
38
9.53
9.03
9.83
10.53
9.9
9.88
7.77
7.05
32
10.73
9.73
10.68
10.35
11.3
10.62
31
8.45
8.35
9.58
9.13
9.87
7.5
307
17.08
13.27
16.78
16.7
17.43
16.88
300
12.2
16.5
13.38
14.4
13.28
14.4
8.05
9.15
7,38
28
10.68
9.85
11.83
12.15
11.6
10.98
10.48
10.65
7.8
23517
8.37
7.83
8.45
8.4
7.8
7.62
23381
7
6.67
6.95
7.17
6.83
10.8
11.55
10.48
10.05
12.2
18
13.2
10.52
12.73
11.93
11.07
12.52
182
12.23
12.35
13.4
12.6
12.03
14.77
12.68
10.25
10.63
11.52
10.82
5.59
5.3
6.05
7,02
6.72
1,590
7.5
7.12
7.67
7.53
7.33
151
14.55
18.27
19.07
18.37
16.77
1510
7.24
6.67
7.42
7.37
7.85
7.25
14
10.65
10.02
11.22
11.38
11.38
12.12
1497
13.422
11.82
15.87
14.83
14:45
13.85
1494
8.68
8.28
8.65
8.55
8:25
9.02
1489
13.84
12.52
14.12
13.97
14.62
12.6
7.81
7,98
8.17
7.33
7.75
1486
11.3
11.35
11.05
11.6
10.33
11.5
1480
5.44
6.63
6.75
6.57
6.6
6.25
1478
11.4
12.38
12.17
11.78
12.57
1475
13,16
11.57
14.57
13.65
13.42
14
1468
12.25
11.12
14.1
16.35
14.93
13.62
7.83
8.6
6.95
8.12
1462
8.03
6.4
7.58
7.43
7.28
5.68
12
11.16
8.13
9.73
10.07
9.7
11.68
10.31
10.97
11,48
10.78
9.58
11.38
1068
5.69
6.78
6.18
6.3
6.47
1067
7.97
9.87
8.23
8.55
8.1
9.03
7.03
6.8
1053
8.44
8.78
8.12
9.07
8.43
9.1
1049
8.68
10.92
9.38
9.42
8.8
9.5
Table Number of actual training epochs used (averaged over 30 trials) for the 100% training data setting, for each dataset and every baseline.
24


Reviews:
[{"review_id": "y2Xk6xo4z_l", "reviewer": "Reviewer_swwj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Exploring self-supervision on novel domains (e.g., tabular data) is valuable\n- The method for generating ‘augmentations’ for contrastive learning via random replacement from the marginal feature distribution makes sense and is novel to the best of my knowledge\n- The idea is simple but seems quite effective\n- The paper is well written, and the method is represented well\n- Experiments show relative improvements over several baselines\n\nWeaknesses:\n- All the experiments only show relative comparisons. It is also important to get a sense of the absolute performance on these benchmarks, e.g., to assess how strong the baseline is compared to the existing state-of-the-art. An obvious baseline on tabular data would be a tree ensemble model (like XGBoost).\n- Some of these relative comparisons are also difficult to interpret: It is, for example, unclear what the second Win-matrix is showing. Is it SCARF pre-training vs. Dropout or SCARF with vs. without Dropout? How does it follow that SCARF is complementary to Dropout from the Figure when we do not see any absolute performance numbers?\n- The method from Yao et al. (2020) sounds quite similar, but it is not clear what the advantage of SCARF would be, and a comparison is also missing.\n- The joint sampling ablation is flawed if the replacement rate is set to the default c=0.6. This would only make sense when c<<0.5. The augmented sample otherwise has more resemblance with the randomly drawn instance. ", "summary_of_the_paper": "The paper presents a method for self-supervised pre-training on tabular data to improve the performance in supervised transfer.\nThe method adapts the successful contrastive learning framework to tabular data by defining “augmentations” of the data wherein randomly chosen feature columns are replaced by sampling from their corresponding marginal distribution.\nThe method is evaluated on tabular classification tasks of the OpenML-CC18 benchmark, both in the fully supervised setting and in the semi-supervised setting and under 30% label noise. \nThe method outperforms several baselines. ", "summary_of_the_review": "The paper introduces a simple adaptation of contrastive learning to tabular data by introducing a novel ‘data augmentation’ on tabular data. While the method outperforms several baselines, the experimental setup is not very convincing. Strong baselines like XGBoost are missing, a comparison to the related prior work by Yao et al. is missing, and only relative performance improvements over baselines are reported while absolute performance numbers are missing altogether. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sGtRhIPdEq5", "reviewer": "Reviewer_7Frm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n1) Easy to read paper with very comprehensive results across a variety of datasets. They also try >10 baselines and show very comprehensive improvements on top of these baseline methods.\n2) SCARF also shows improved performance even in presence of label noise, which suggests that SCARF is also more robust to corruption in the datasets as well.\n3) SCARF is also less sensitive to feature scaling as compared to previous methods.\n\nWeakness:\n1) Role of negative samples: one thing I’m not sure about is what is the role of negative samples in SCARF. From the results in Figure 5, it can be seen that increasing the batch size doesn’t really have an impact on the performance. Can it be that a negative sample doesn't have much impact on the results? It would be interesting to see a ablation showing the performance dip as we decrease the #negative samples.\n2) Results using other non contrastive methods:  Results using BYOL and other non-contrastive methods would be interesting to see. I can see that Barlow twins perform very similarly to Info-NCE methods, but I’m not sure how many hyperparameter searches have been made on Barlow-Twins. Generally I’m asking if it is possible to outperform infonce methods with non contrastive methods. Even if it doesn't work well; it would be nice to see some analysis on why non-contrastive methods don’t work as well on tabular data while they are working better on Image datasets.\n3) Temperature = 1: usually in contrastive learning temperature plays a huge role in representation ability of the network. But in this case the temp is 1, which effectively means there is no effect of temperature. Can the authors comment more on that ?\n", "summary_of_the_paper": "This work proposes a good method to use SSL methods on tabular datasets. They achieve this by randomly corrupting a subset of the features. They show comprehensive results and improved performances on 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark.\n", "summary_of_the_review": "Overall I like the paper, there are few concerns regarding use of negative samples and below par performance on non contrastive methods. It would be nice to have some more analysis on this front.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_tB3Mw_MGg_", "reviewer": "Reviewer_y3t1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) The motivation of this paper is clear and the problem they studied is valuable. InfoNCE loss (or the contrastive learning paradigm) has been well studied in computer vision but it is not obvious to design augmentations (i.e., generate different views) for tabular data. Images are invariant under various transformations (e.g., translation, rotation, color jittering or cropping) but it is hard to design such transformations for tabular data.\n\n(2) The experimental results are very sufficient and the authors provide a wide range of comparison methods. They experimented with 69 datasets and repeated 30 times for each run. Hence, I think the experimental results are convincing. \n\n(3) The authors investigate the label-noise and semi-supervised setting. There has been works in computer vision that have studied applying SSL under label-noise and semi-supervised setting, but this is the first time I have seen for the tabular datasets.\n\n(4) Some conclusions given by the authors are instructive for further study. For instance, corrupting one view is better than corrupting both the views for tabular.\n\n-------------------------------------------------------------------------------------------------------------------------------\nWeakness:\n\n(1) When comparing accuracies with and without self-supervised pre-training, is the number of fine-tuning epochs the same?  As noted in the paper, the authors set a max number of fine-tune epochs of 200 and pre-train epochs of 1000. Should we fine-tune for longer epochs (i.e., >200) if we don't conduct self-supervied pre-training for fair comparisons? \n\n(2) Is label information useful when sampling? For instance, when using joint sampling or marginal samling, what if we consider the label $y_i$ for $x_i$ and only consider the instances in the same class when sampling? Although pre-training is not unsupervised in this case, I don’t think this is a problem because labels are also used in the fine-tuning phase.\n\n(3) I was curious about whether SCARF pre-training outperforms using SCARF augmentation during fine-tuning and I appericiate the authors for providing such results. However, as noted in weakness (1), the training details (especially epochs) are not clear and I am not sure whether it is a fair comparison.\n\n(4) Some conclusions need further explainations. For instance, it is common practice in computer vision to corrupt both views during SSL but why corrupting one view is better than corrupting both views in this paper. Is it due to the characteristics of the tabular data or the proposed method?\n", "summary_of_the_paper": "This paper proposes a data augmentation method (SCARF) used for self-supervised learning for tabular data. SCARF generates different views by corrupting a random subset of features (via sampling from the marginal distribution). The experimetanl results demonstrate that SCARF not only improves accuracy in the fully-supervised setting but also in the presence of label noise and in the semi-supervised setting. The authors also conduct ablation studies and compare with various methods.\n\nI think adapting SSL to tabular data is a very important direction and I thank the authors for their efforts in this direction. ", "summary_of_the_review": "This paper is well-writtern and the motivation of paper is clear. I agree with the contributions of this paper but I still have some concerns (see the weakness). I am happy to raise my score if my concerns are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QRU4C0pxcD", "reviewer": "Reviewer_6ZpH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are several merits of the paper: \n - While the augmentation scheme looks very simple and intuitive, it is superior to all conceivable baselines.\n - Thorough evaluation scheme using 'win matrix', measured how many datasets SCARF beats other baselines. To compute the average performance of the algorithm, they run 30 times on each dataset and method pair\n- Wide range of findings - showed its superiority in label-corrupted and partially-labeled settings, compared between several corruption strategies, done sensitivity analysis with regards to key hyperparameters.\n - As for the experiment, the authors also explored recently-proposed loss alternatives including Uniform and Align and Barlow Twins, and showed that InfoNCE is a good choice.\n\nThe paper could be more complete if there was a discussion between two self-supervised learning approaches for tabular data: contrastive learning and pre-text learning models such as VIME (NeurIPS 2020) or TabNET (AAAI 2021).", "summary_of_the_paper": "This paper proposes SCARF, which is a contrastive pretraining procedure for tabular data. SCARF generates an augmentation of a data point by selecting a random subset of features and replace them with their marginal distributions. Unsupervised pre-training is done by update networks consists of an encoder and pretraining head. Contrastive InfoNCE loss is used to pull the original input and its perturbed version, while pushing it away from other data points in mini-batch. On 69 datasets from OpenML-CC18 benchmarks, SCARF is compared with dozens of baselines and showed its efficacy. SCARF pretraining improves model robustness in label-corrupted settings and also helps classification in partially-labeled settings.", "summary_of_the_review": "Simple and intuitive contrastive learning framework for tabular data, backed up by extensive experiments. Considering the sufficient experiments, the proposed SCARF algorithm is expected to perform well on any tabular dataset, and the simplicity of the algorithm is expected to draw attention and will bring subsequent methodologies.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores self-supervised learning on tabular data and proposes a novel augmentation method via corrupting a random subset of features. The idea is simple but effective. Experiments include 69 datasets and compare with a number of methods. The result shows its superiority. It would be inspiring more work for SSL on the tabular domain.
Decision: Accept (Spotlight)