Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for self-supervised pre-training on tabular data to improve the performance in supervised transfer.
The method adapts the successful contrastive learning framework to tabular data by defining “augmentations” of the data wherein randomly chosen feature columns are replaced by sampling from their corresponding marginal distribution.
The method is evaluated on tabular classification tasks of the OpenML-CC18 benchmark, both in the fully supervised setting and in the semi-supervised setting and under 30% label noise. 
The method outperforms several baselines. 
main_review: Strengths:
- Exploring self-supervision on novel domains (e.g., tabular data) is valuable
- The method for generating ‘augmentations’ for contrastive learning via random replacement from the marginal feature distribution makes sense and is novel to the best of my knowledge
- The idea is simple but seems quite effective
- The paper is well written, and the method is represented well
- Experiments show relative improvements over several baselines

Weaknesses:
- All the experiments only show relative comparisons. It is also important to get a sense of the absolute performance on these benchmarks, e.g., to assess how strong the baseline is compared to the existing state-of-the-art. An obvious baseline on tabular data would be a tree ensemble model (like XGBoost).
- Some of these relative comparisons are also difficult to interpret: It is, for example, unclear what the second Win-matrix is showing. Is it SCARF pre-training vs. Dropout or SCARF with vs. without Dropout? How does it follow that SCARF is complementary to Dropout from the Figure when we do not see any absolute performance numbers?
- The method from Yao et al. (2020) sounds quite similar, but it is not clear what the advantage of SCARF would be, and a comparison is also missing.
- The joint sampling ablation is flawed if the replacement rate is set to the default c=0.6. This would only make sense when c<<0.5. The augmented sample otherwise has more resemblance with the randomly drawn instance. 
summary_of_the_review: The paper introduces a simple adaptation of contrastive learning to tabular data by introducing a novel ‘data augmentation’ on tabular data. While the method outperforms several baselines, the experimental setup is not very convincing. Strong baselines like XGBoost are missing, a comparison to the related prior work by Yao et al. is missing, and only relative performance improvements over baselines are reported while absolute performance numbers are missing altogether.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a good method to use SSL methods on tabular datasets. They achieve this by randomly corrupting a subset of the features. They show comprehensive results and improved performances on 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark.

main_review: 
Strengths:
1) Easy to read paper with very comprehensive results across a variety of datasets. They also try >10 baselines and show very comprehensive improvements on top of these baseline methods.
2) SCARF also shows improved performance even in presence of label noise, which suggests that SCARF is also more robust to corruption in the datasets as well.
3) SCARF is also less sensitive to feature scaling as compared to previous methods.

Weakness:
1) Role of negative samples: one thing I’m not sure about is what is the role of negative samples in SCARF. From the results in Figure 5, it can be seen that increasing the batch size doesn’t really have an impact on the performance. Can it be that a negative sample doesn't have much impact on the results? It would be interesting to see a ablation showing the performance dip as we decrease the #negative samples.
2) Results using other non contrastive methods:  Results using BYOL and other non-contrastive methods would be interesting to see. I can see that Barlow twins perform very similarly to Info-NCE methods, but I’m not sure how many hyperparameter searches have been made on Barlow-Twins. Generally I’m asking if it is possible to outperform infonce methods with non contrastive methods. Even if it doesn't work well; it would be nice to see some analysis on why non-contrastive methods don’t work as well on tabular data while they are working better on Image datasets.
3) Temperature = 1: usually in contrastive learning temperature plays a huge role in representation ability of the network. But in this case the temp is 1, which effectively means there is no effect of temperature. Can the authors comment more on that ?

summary_of_the_review: Overall I like the paper, there are few concerns regarding use of negative samples and below par performance on non contrastive methods. It would be nice to have some more analysis on this front.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a data augmentation method (SCARF) used for self-supervised learning for tabular data. SCARF generates different views by corrupting a random subset of features (via sampling from the marginal distribution). The experimetanl results demonstrate that SCARF not only improves accuracy in the fully-supervised setting but also in the presence of label noise and in the semi-supervised setting. The authors also conduct ablation studies and compare with various methods.

I think adapting SSL to tabular data is a very important direction and I thank the authors for their efforts in this direction. 
main_review: Strengths:

(1) The motivation of this paper is clear and the problem they studied is valuable. InfoNCE loss (or the contrastive learning paradigm) has been well studied in computer vision but it is not obvious to design augmentations (i.e., generate different views) for tabular data. Images are invariant under various transformations (e.g., translation, rotation, color jittering or cropping) but it is hard to design such transformations for tabular data.

(2) The experimental results are very sufficient and the authors provide a wide range of comparison methods. They experimented with 69 datasets and repeated 30 times for each run. Hence, I think the experimental results are convincing. 

(3) The authors investigate the label-noise and semi-supervised setting. There has been works in computer vision that have studied applying SSL under label-noise and semi-supervised setting, but this is the first time I have seen for the tabular datasets.

(4) Some conclusions given by the authors are instructive for further study. For instance, corrupting one view is better than corrupting both the views for tabular.

-------------------------------------------------------------------------------------------------------------------------------
Weakness:

(1) When comparing accuracies with and without self-supervised pre-training, is the number of fine-tuning epochs the same?  As noted in the paper, the authors set a max number of fine-tune epochs of 200 and pre-train epochs of 1000. Should we fine-tune for longer epochs (i.e., >200) if we don't conduct self-supervied pre-training for fair comparisons? 

(2) Is label information useful when sampling? For instance, when using joint sampling or marginal samling, what if we consider the label $y_i$ for $x_i$ and only consider the instances in the same class when sampling? Although pre-training is not unsupervised in this case, I don’t think this is a problem because labels are also used in the fine-tuning phase.

(3) I was curious about whether SCARF pre-training outperforms using SCARF augmentation during fine-tuning and I appericiate the authors for providing such results. However, as noted in weakness (1), the training details (especially epochs) are not clear and I am not sure whether it is a fair comparison.

(4) Some conclusions need further explainations. For instance, it is common practice in computer vision to corrupt both views during SSL but why corrupting one view is better than corrupting both views in this paper. Is it due to the characteristics of the tabular data or the proposed method?

summary_of_the_review: This paper is well-writtern and the motivation of paper is clear. I agree with the contributions of this paper but I still have some concerns (see the weakness). I am happy to raise my score if my concerns are addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes SCARF, which is a contrastive pretraining procedure for tabular data. SCARF generates an augmentation of a data point by selecting a random subset of features and replace them with their marginal distributions. Unsupervised pre-training is done by update networks consists of an encoder and pretraining head. Contrastive InfoNCE loss is used to pull the original input and its perturbed version, while pushing it away from other data points in mini-batch. On 69 datasets from OpenML-CC18 benchmarks, SCARF is compared with dozens of baselines and showed its efficacy. SCARF pretraining improves model robustness in label-corrupted settings and also helps classification in partially-labeled settings.
main_review: There are several merits of the paper: 
 - While the augmentation scheme looks very simple and intuitive, it is superior to all conceivable baselines.
 - Thorough evaluation scheme using 'win matrix', measured how many datasets SCARF beats other baselines. To compute the average performance of the algorithm, they run 30 times on each dataset and method pair
- Wide range of findings - showed its superiority in label-corrupted and partially-labeled settings, compared between several corruption strategies, done sensitivity analysis with regards to key hyperparameters.
 - As for the experiment, the authors also explored recently-proposed loss alternatives including Uniform and Align and Barlow Twins, and showed that InfoNCE is a good choice.

The paper could be more complete if there was a discussion between two self-supervised learning approaches for tabular data: contrastive learning and pre-text learning models such as VIME (NeurIPS 2020) or TabNET (AAAI 2021).
summary_of_the_review: Simple and intuitive contrastive learning framework for tabular data, backed up by extensive experiments. Considering the sufficient experiments, the proposed SCARF algorithm is expected to perform well on any tabular dataset, and the simplicity of the algorithm is expected to draw attention and will bring subsequent methodologies.
review_title: null

Meta-review: The paper explores self-supervised learning on tabular data and proposes a novel augmentation method via corrupting a random subset of features. The idea is simple but effective. Experiments include 69 datasets and compare with a number of methods. The result shows its superiority. It would be inspiring more work for SSL on the tabular domain.
Decision: Accept (Spotlight)