Q-Match: Self-Supervised Learning For Tabular Data by Matching Distributions Induced by a Queue
Authors: Thomas Mulc
Debidatta Dwibedi
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
deep learning for tabular data

Abstract: In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We show that Q-Match outperforms previous self-supervised learning techniques on tabular datasets when measuring downstream classification performance. Furthermore, we show that our method is sample efficient, both in terms of labels required for both downstream task training and amount of unlabeled data required for pre-training.

Under review as a conference paper at ICLR 2023
Q- :-MATCH: SELF-SUPERVISED LEARNING FOR TABU- LAR DATA BY MATCHING DISTRIBUTIONS INDUCED BY A QUEUE
Anonymous authors Paper under double-blind review
ABSTRACT
In semi-supervised learning, student-teacher distribution matching has been suc- cessful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We show that Q-Match outperforms pre- vious supersupere learning techniques tabular datasets when measuring downstream classification performance. Furthermore, we show that our method is sample efficient, both in terms of labels required for both downstream task training and amount of unlabeled data required for pre-training.
INTRODUCTION
Tabular data the most common form data for problems industry. While many robust techniques exist to solve real-world machine learning problems on tabular data, most of these techniques require access labels. Leveraging unlabeled data to learn good representations remains key open problem in the tabular domain. In this work, we propose a flexible and powerful framework using deep learning that helps use unlabeled data in the tabular domain.
Deep learning has been successful in processing data in many different domains like images, audio, and text. Learning non-linear features using deep architectures has been shown to be key feature in improving performance across wide variety of problems like image recognition (Krizhevsky et al. 2017). speech recognition (Deng et al. 2013). and machine translation (Singh et 2017). Until recently, achieving state-of-the-ar performance would not have been possible without large, manually annotated datasets. However, a class learning algorithms called self-supervised learning (Wu et al. 2018 Devlin et al.) 2018) has shown that highly performant features can be learned without large-labeled datasets well. In this work, we propose new self-supervised learning algorithm and study its effectiveness in the tabular data domain.
In self-supervised learning, a task known as the pretext task is first solved on a dataset that is typically large and unlabeled. The aim of self-supervised learning learn an encoding function parameterized by 9 that captures variances and invariances in the dataset without the need for human-annotated labels. This initial training is usually referred to the pre-training stage. The parameters learned from the pretext task during pre-training are then used to solve a new objective called the downstream task. Concretely. f is function with parameters 0 that maps sample from the input dimensionality to an embedding size that is f (X, 0) - Rd where x a single data-point in the input space X. After the pretext algorithm updates the model parameters 0 during the pre-training stage, the downstream data (typically a manually annotated dataset) is either used to fine-tune or learn the parameters of linear classifier on the output of f.
Recently proposed self-supervised approaches for tabular data (Darabi al. 2021 Yoon al. 2020; Ank & Pfister 2021]; Lee et al. (2020) have shown encouraging results. In this work, we propose novel method for self-supervised learning called Q-Match which is closely related to the semi-supervised learning framework called FixMatch (Sohn al., 2020). For labeled data, FixMatch uses the standard supervised learning loss. For unlabeled data. FixMatch proposes to match the student and teacher distributions over the set of classes used in the downstream task. In

---

Under review as a conference paper at ICLR 2023
self- supervised setup, however, access the relevant classes any labeled data during pre-training limited. For this reason, Q-Match uses a queue of embeddings instead of known classes to induce the teacher and the student distributions. The queue is implemented similar to MoCo He et al. (2020) and NNCLR (Dwibedi al. 202T) by updating a list of embeddings with newer embeddings and discarding older ones as training proceeds. We use individual samples (via their embeddings) to generate the student and teacher distributions required for training. We find Q-Match leads to improvements not only in the final performance of the downstream model, but also reduces the number of labeled samples required for the downstream task.
2 PROPOSED APPROACH
Motivation. Our self superviee approach is based on the success of the semi-supervised learning algorithm FixMatch (Sohn et al. 2020). In their method, the authors show it is possible to leverage large unlabeled dataset and few labeled samples to improve the performance of the model downstream task. They do so by matching the class distributions of the student and teacher views produced by augmenting the input in two different ways. We hypothesize that it might be possible to adapt their framework the self-supervised setup by removing the dependency on the known classes during training. To do so, we keep a queue of past embeddings that can serve as a proxy for classes. We use this queue produce the target and student distributions used to train model. The training then proceeds by performing continuous knowledge distillation from the teacher to the student model such that the student ultimately learns to predict the distribution induced by the teacher.
Method. In Figure I we outline our method for the pretext training approach used in Q-Match. We corrupt the input 2; two times independently using the method proposed in VIME (Yoon et al.) 2020) to produce the student view zi and the teacher view Zi,t. We pass Ti,s through the student model to produce the student embedding ai.a. Similarly, we pass Ti, through the teacher model to produce the teacher embedding Zi, We want this pair of embeddings to induce similar probability distributions over a representative set of samples of the dataset. In other words, we want our encoder to maintain similar relationships between different data samples in spite of the corruption introduced while generating the views. As the dataset can be quite large, we maintain fixed size queue of past embeddings like MoCo (He et al./ 2020) and ČNCLR (Dwibedi et al. 2021). A stop-gradient is applied to the teacher embeddings, so only the student weights are updated at each iteration. We multiply the student embeddings and the teacher embeddings with the embeddings in the queue to produce the student and teacher logits. After taking softmax of these logits, we produce the student distribution Pt.. and teacher distribution Pa.: respectively. The teacher distribution Pi, is the target distribution which the student distribution Pt.s should match. We define the distribution matching loss as follows:
Zi.t -.. . softmax Tt
Zi.a Q
2)) Ts
QM (pisssPi.)) H ssftmax
where is scalar temperature value that controls the sharpness of the distribution produced by the teacher, T, is scalar temperature value that controls the sharpness of the student distribution,
Q E RR is the queue of many previous teacher embeddings, and H(p, 9) is the cross-entropy loss between two distributions p and q. We normalize the views 24, and zi,z using L2 normalization. The queue size is constant and is refreshed at every training iteration with the previous batch of teacher embeddings while the oldest embeddings are removed from the queue. The parameters of the teacher model are updated using the Exponential Moving Average (EMA) of the student model parameters. Empirically, we observe that corrupting the teacher with smaller probability leads to better performance (discussed later Section 3.3
Model Architecture. In all experiments, we use an MLP as our encoding function f. We follow the same architecture from i-Mix (Lee et al. 2020) which includes five fully connected layers (2048- 2048-4096-409-881999 The final layer uses a 4-set max-out activation (Goodfellow et al., 2013). All layers except the output layer have batch normalization followed by a ReLu. We use a linear projector of 128 dimensions on top of the encoder in all our experiments. Similar i-Mix, we added a 2-layer (512-128) MLP projection head, but we noticed that it performed similar to the model without the projection head for Q-Match training.
Data Preprocessing. For data preprocessing, we compute the normalization statisties by using a batch normalization layer without the learnable parameters of scale and bias just after the input layer. During evaluation, the accumulated exponential moving average statistics are applied to the
2

---

Under review as a conference paper at ICLR 2023
2g
Ps
Strongly Augmented
student network
Prodictions
X
EMA
queue
Distributtion
Matching Loss H P.
Weakly
teacher network
Bf Pt
Augmented
Pseudo labels
Figure/ 1: Q-Match Training Method. The network is trained by performing continuous self- distillation by minimizing the cross-entropy between student-teach distributions. The two slanted lines denote stop-gradient operation. The queue is updated with 2, at each training step.
original inputs normalizing them. We found this method works just as well as normalizing the data by calculating the statistics the full dataset. For categorical features, one-hot encoded all categorical features. Additionally, we experimented with quantile transformation (Gorishniy et al.) 2022) of the inputs. However, this only showed benefits in the Adult dataset. Hence, we do not use quantile transformation for other datasets.
3 EXPERIMENTS
Datasets. In this work we use the following datasets: Higgs (Baldi et al. 2014). Cover Type Asuncion & Newman (007), Adult (Kohavi et al. 1996) and ( el al. 2010). We take various subsets of the original datasets for different experiments. For the exact splits used in different experiments, please refer to Table 8 in the Appendix.
Implementation. We implement our method in JAX (Bradbury et al. 2018). The experiments were conducted on single V100 and P100 GPUs.
3.1 COMPARISON WITH BASELINES
In this section, we want to measure how our method compares with other self- methods that have been evaluated on tabular data. We find that different papers evaluate their approach on different datasets with different split settings. The pretext set size and labeled set size vary in the original experiments conducted in these papers. To be fair in our comparison to prior work, we report the performance Q-Match with the same downstream dataset sizes as used by the original authors.
Higgs Dataset. First, we compare Q-Match against three other sulf-supe methods for tabular data on the Higgs (Baldi al.) 2014) dataset: TabNet (Arik & Phister 2021). i-Mix (Lee et al. 2020). and CORE (Han Ranganath 2021). We report the results this experiments Table | Q- Q-Match outperforms all the baselines under all the different splits. In particular, we observe that Q-Match outperforms TabNet with pretext dataset size that is 100 times smaller.
Cover Type Dataset. Next, we compare the performance of our method with baselines on the Cover Type dataset (Asuncion & Newman 2007). We report the results of our experiment in Table There are two commonly used splits: Cover Type 10% and Cover Type 15k. In the fine-tuning setup, we observe that Q-Match outperforms Contrastive MixUp by a margin of about 10%. We hypothesize this gain in performance is due the fact that in the contrastive loss the encoder mistakenly considers samples belonging to the same class as negatives. The Q-Match loss does not suffer from this problem. The encoder is free to learn the similarities between different samples and does not consider all items the batch as negatives. In the linear evaluation setup, we observe Q-Match is about 0.7% to 1.6% better than two versions of i-Mix that use the contrastive loss. Q-Match is slightly worse (0.3%) than i-Mix BYOL in the linear evaluation setup.
MNIST 10% Dataset. Next, we perform a similar comparison on the MNIST (LeCun et al. 2010) dataset. While it is an image dataset, prior work (Yoon et al. (2020) Darabi et al. (2021) has used pixels an image into flattened vector. In the past, only 10% of the training set is used as labeled data while the rest 90% of the training set is used as pretext data. We report the results of this experiment in Table 3 We observe
MNIST 10% for research tabular domain by converting
3

---

Under review as a conference paper at ICLR 2023
Method
Pretext Dataset Size
Labeled Dataset Size
Accuracy
CORE (Han & Ranganath| 2021
50k
5k
66.92 + 0.55
Q-Match (Ours)
50k
5k
68.13 + 1.02
TabNet (Arik & Pfister 2021
10M
10k
68.96 + 0.39
Q-Match (Ours)
10M
10k
71.13 + 0.21
TabNet (Arik & Pfister i-Mix (Lee et al. 2020
2021
10M
100k
73.19 + 0.15
100k
100k
72.9
Q-Match (Ours)
100k
100k
73.27 + 0.19
Table 1: Higgs experiments. CORE, TabNet, and Q-Match report fine tuning accuracy, while i-Mix reports the linear classification accuracy.
that while Q-Match outperforms VIME by a margin of about 2%, our method is comparable in performance with the Contrastive Mixup method without using the MixUp augmentation.
Method
Cover Type 10% Accuracy
Cover Type 15k Accuracy
Constrastive MixUp (Darabi et 2021)
80.41 .255
i-Mix N-Pair (Lee et al. 2020)
72.1 + 0.2
i-Mix MoCo v2 (Lee al. 2020
73.1 + 0.1
i-Mix BYOL, (Lee et al. 2020
74.1 0.2
Q-Match Finetune (Ours) Q-Match Linear (Ours)
90.26 .11
82.76 = .07
73.79 = .22
Table Cover Type experiments. For Cover Type 15k, i-Mix uses a linear classifier on top of the pretext features. For Constrastive MixUp, the evaluation uses fine tuning. We present results of Q-Match for both tasks.
Method
Accuracy
VIME (Yoon et al. 2020)
95.77 + .22
Constrastive MiXUp (Darabi al. 2022
97.58 .08
Q-Match (Ours)
97.67 + .21
Table 3: MNIST 10% % experiments. Fine tuning accuracy for VIME, Constrastive MixUp, and Q-Match.
3.2 DATA SCALING EXPERIMENTS
In the previous subsection, we show that Q-Match either outperforms or is par with the other self-supervised baselines when the pretext size is large and all the samples in the labeled downstream dataset are used for evaluation. the amount of labeled data and the pretext dataset size changes. To compare fairly against all the baselines, for this set of experiments, we re-implement the following methods: TabNet, VIME, and i-Mix (N-Pair version).
this experiment, we want to measure how these methods perform
Training Details. In all the experiments in this section, we first train the encoder on the pretext task. We then proceed the downstream tasks which can either be fine-tuning linear evaluation. We always compare against the supervised learning baseline. This baseline refers to the method where we do not train model with any pretext task but begin downstream task training from random initialization. We perform a grid search over relevant hyper parameters for each method. For all methods, we search over pretext task and downstream task learning rates. For the TabNet, VIME, SimCLR. SimSiam, DINO. VICReg, and Q-Match algorithms, we added the probability of corrupting a column (as defined in Yoon et al. (2020)) to the grid. For the N-Pair i-Mix algorithm, we also add the loss temperature to the grid. For the Q-Match algorithm, we also add the queue size and the student temperature to the grid. Please refer Table O the Appendix for more details on the parameters. We picked the best parameters according to the validation dataset for each downstream task. For all experiments, report the average and standard deviation over trials. We train all
4

---

Under review as a conference paper at ICLR 2023
pretext tasks using maximum of 200 epochs with early stopping and patience of 32 (using the pretext validation dataset). For the supervised tasks, we train for a maximum of 500 epochs with a patience of 32 on the validation accuracy. All tasks use a batch size of 512 and the parameters are updated using Adam citepadamw optimizer with weight decay. All algorithms used zero weight decay during pretraining and 10 11 weight decay during the downstream tasks.
Corruption Function. One important factor that affects the performance is the choice of the corruption function used to create the two augmented views. Both VIME and TabNet augment the orignal data by randomly corrupting columns. The VIME corruption function replaces corrupted values with samples from the pretext dataset while the TabNet corruption function replaces values with zeros. We always use the VIME corruption function in all our experiments. Even for our implementation of the TabNet algorithm, we use the VIME corruption function as we found the VIME corruption performed better. Unless otherwise stated, we do not corrupt the teacher view and only corrupt the student view.
Few Shot Learning. In this experiment, compare the representations learned by different learning algorithms by only using I 1% of the total available labels for evaluation. This scenario is common in industry when number of available labels for downstream task might be less but there might a lot of unlabeled data available to learn an encoder. In addition the tabular self-supervised algorithms we used as comparisons for other experiments, we also include results for our implementations of the following methods that have previously been used for image sulf-superviiion tasks: SimCLR (Chen et al. 2020b), SimSiam (Chen & He| p, DINO (Caron et al. 2021], and VICReg (Bardes) et al. 2021). b. For SimCLR, we also include a large batchsize (4096) version, since this has been Tound to be helpful (Chen al. 2020b). For all datasets, we only use one-percent of the original labels for downstream training and keep the rest of the data for pretext training and validation. We report the performance of learning linear classifier in Table and finettuning the entire encoder in Table For the Linear Classification Task, Q-Match outperforms all other methods on all datasets except VICReg on MNIST, which it performed similarly. For the Finetuning task, Q-Match was competitive on all datasets, but did especially well on the Cove rType and Higgs datasets, where had the top accuracies.
Dataset
Algorithm
Cover Type 1%
Higgs100k 1%
Adult 1%
MNIST 1%
Supervised Baseline
70.33 + 1.45
56.29 + 2.58
78.14 + 0.70
88.81 + 0.34
VIME
71.47 + 0.25
62.90 = 2.29
79.70 + 1.81
92.85 + 0.48
TabNet
70.34 + 0.88
62.12 = 1.3
78.18 + 1.17
86.94 + 1.88
i-Mix
71.45 = 0.23
59.63 2.79
77.66 + 1.50
92.12 0.42
SimCLR
71.80 = 0.41
64.26
80.32 1.47
94.67
SimCLR (large batch)
71.86 0.19
80.13 # 1.01
93.97
SimSiam
71.95 0.32
1.28
78.45 3.38
93.03
VICReg
69.92 + 1.47
61.15 3,80
80.62 + 0.42
+ 97.40 0.09
DINO
72.58 + 0.54
56.69 - 4.11
78.35 + 0.81
90.10 + 1.20
Q-Match (Ours)
72.69 + 0.34
65.19 = 1.25
79.05
+ 2.80
96.32 + 0.54
Table 4: Accuracy of our method versus other pretext algorithms on the finetuning classification task.
Varying Downstream Dataset Size. In this experiment we want to measure how downstream task performance changes as more labels are available. We increase the fraction of labeled data available in the Higgs dataset and train the downstream task. Note that in this experiment the pretext size fixed for all methods at 100k samples. We report the results of this experiment in Figure 2 We observe that Q-Match outperforms other methods across different fractions of labeled data in both the fine-tuning and linear evaluation tasks. The difference in performance between and other methods increases as the number of labeled samples becomes less. In other words, Q-Match is more sample efficient in terms of labels required. Also note that in the low labeled data regime, self-supervised pre-training using Q-Match provides good initialization for the downstream task. This initialization leads to about 10% improvement in performance than simply performing supervised learning from random initialization.
Varying Pretext Dataset Size. In this experiment we want to measure how downstream task performance changes as more unlabeled data is available for the pretext task training. We increase the fraction of unlabeled data available in the Higgs dataset and run both the pretext training and
5

---

Under review as a conference paper at ICLR 2023
Dataset
Algorithm
Cover Type 1%
Higgs100k 1%
Adult 1%
MNIST 1%
Supervised Bascline
70.18 + 0.20
60.35 = 0.34
78.54 + 0.27
85.77 + 0.22
VIME
68.55 + 0.32
64.36 = 0.62
76.87 + 3.04
87.48 + 0.34
TabNet
48.07 + 7.04
60.57 + 0.69
76.83 + 1.08
23.42 + 8.48
i-Mix
67.52 0.39
60.05 0.5
75.62 = 0.57
90.58 =
SimCLR
69.69 + 0.19
65.22 0.83
79.83 + 0.38
92.26
SimCLR (large batch)
70.13 + 0.20
6
77.45 + 2.87
SimSiam
64.66 + 1.43
3.660
79.27 1.95
92.21
VICReg
65.72 + 1.53
6 0.22
76.88 1.49
97.52 + 0.06
DINO
61.47 + 3.08
57.17 2.49
77.11 0.99
67.94 = 5.21
Q-Match (Ours)
70.39 + 0.59
67.22 = 0.29
80.53 + 0.38
97.11 + 0.47
Table 5: Linear classification accuracy of our method versus other pretext algorithms.
downstream task training. Note that in this experiment the downstream labeled set size is fixed for all methods 100k samples. We report the results of this experiment in Figure 3 We observe that Q-Match outperforms other methods. The difference performance is especially stark when fewer samples are available. In other words, Q-Match is more sample efficient even in terms of unlabeled data requirements. Depending on the amount of unlabeled data available, Q-Match can increase the performance on downstream task by 5 5%-8% in the finetuning setup.
Fine Tuning Accuracy vs Labeled Dataset Size
Linear Classification Accuracy vs Labeled Dataset Size
0.200
Accurscy
0850
0.50
Q-Match
Tabnet
0600
0.600
-iix Supervised
0330
0350
20%
35
20%
100%
data labeled
Percentage data labeled
(a) Tuning Classification Task
(b) Linear Classification Task
Figure 2: Classification performance as the size of the labeled data increases for the Higgs 10kk dataset.
Fine Tuning Accuracy Pretext Dataset Size
Linear Classification Accuracy vs Pretext Dataset Size
070
0.68
Accuracy
0.66
Q-Match
VIME
Tabnet
0.62
0.60
Supervised
10
10a
105
101
10'
Pretext
Pretext
(a) Fine Tuning Classification Accuracy
(b) Linear Classification Accuracy
Figure 3: Accuracy for different sizes of the pretext set for the Higgs dataset.

---

Under review as a conference paper at ICLR 2023
3.3 SENSITIVITY ANALYSIS
In this subsection, study how some hyperparameter choices affect downstream task performance. In particular, we find our method's final performance is affected significantly by the corruption probability and the queue size. Below we report how sensitive the learning algorithm is for these datasets: Higgs and Cover Type.
Corruption Probability. We experiment varying the corruption probability in both views in Q- Match. The linear classification results shown in Figure 4 The more yellow/bright the color of the grid, the better the performance while more blue/dark means the performance worse. Note that in both the Cover Type and the Higgs experiments, there appears be both maximum teacher and student corruption probability that beneficial for pre-training with Q-Match. After this maximum value, there is too much corruption of the original input to learn useful representation. Additionally, there is also critical maximum value of the of both probabilities going beyond which, the model other words both the student teacher are corrupted with high corruption probability, it leads to sub- cub-optimal performance. We recommend using small or zero value for the teacher corruption, and moderate value for the student corruption achieve optimal downstream performance.
fails to learn useful representations using Q-Match,
On the other hand, we find the fine-tuning results to be fairly robust to these parameters. The initialization found by performing Q-Match with different values of corruption probability is good enough for downstream fine-tuning to achieve optimal performance.
Linear Classification Accuracy
Linear Classification Accuracy
0.5
Teacher Corruption probabiltty
Teacher Corruption probability
03
03
04
04
06
05
25
86-
86
07
07
88
88
034
60
01
67
63
04
05
56
07
68
50
01
07
03
04
05
06
57
08
09
Student Corruption Probability
Student Corruption Probability
(a) Cover Type 1%
(b) Higgs 1%
Figure 4: Linear classification accuracy as corruption of the features changes for both the student and the teacher views.
Queue size. We examine the effect of changing the size of the queue used in Q-Match for the Higgs, Cover Type, and MNIST datasets. The linear classification performance (in the few shot setting) as a function of the queue size is shown in Figure 5 First, we observe that when the queue size becomes very small, the final performance declines. Second. we note the queue size increases, there the less variance in the downstream results. We recommend using a larger queue (2 10²) to achieve optimal performance.
7

---

Under review as a conference paper at ICLR 2023
Cover Type Linear Classification Accuracy
Higss Linear Classification Accuracy
MNIST cation Accuracy
0.71
0.9/5
0.650
Accurecy
Accuracy
Accuracy
0.950
0.70
0.625
0.925
0.600
0.69
0.900
0.575
0.875
10
10
10a
102
10a
10.
10
10a
10
Queue Size
Queue Size
Queue Size
(a) CovType 1%
(b) Higgs 1%
(c) MNIST 1%
Figure 5: Effect of the queue size on the linear classification accuracy.
4 DISCUSSION AND RELATED WORK
Contrastive Selr-supervised Learning. In the context of self-supervised learning, a class of ap- proaches that have been effective are built on top of the InfoNCE loss (Oord et (0188) where for any data point Hu and its corresponding embedding 2; = f(z,, 0) there exists a set of positives P. and negatives N₁, loss [nfoNCEE is defined as follows:
EXTER, exp (z. 2+/7)
ClaloNCE - -logg
Ex+Er exp (z+ - 2t /T) Ex-eN, EN. exp (21-2 /T)
where (zi, 255) are positive pairs, (2,, ) are negative pairs and is the softmax temperature. This loss aims to attract the positives closer to each other while repelling the negatives farther from each other. In SimCLR (Chen et al., (020a)). the positives are two views of the same data and negatives are all the other elements the current mini-batch. In MoCo (He al. 2020), the positives the same as SimCLR but the negatives are derived from a queue of past embeddings produced by the model.
Both Contrastive Mixup and i-Mix N - Pair use the InfoNCE loss. We find Q-Match outperforms both these methods consistently. This could be due to the fact the contrastive loss mistakenly considers samples from the same class as negatives. For a dataset whose classes are uniformly distributed, the probability that at least one sample in the batch of negatives belongs to the same downstream class as the positives is equal to 1 \ ((B-1) where N is the number of classes in the downstream task and the batch size used during training. This means the occurrence of this event is less in a large-scale and diverse image datasets like ImageNet. Concretely, with a batch size of 512 and the number of downstream classes equal 1000, there is probability of approximately 0.4 that an element of the same class will be considered a negative. However in the tabular setting where the number of classes in the downstream task is usually much less (say 10), the probability of considering items of the same class as negatives is much higher (very close 1). We hypothesize that this might be the reason that using the vanilla N-pair contrastive loss usually results in sub-par performance.
Non-Contrastive Self-supervised Learning. In order to remove the dependency explicit nega- tives during training. researchers have proposed two types of losses. The first class of approaches like B BYOL et al.] 2020), SimSiam (Chen & He| (22TT) aim to directly minimize the distance between the positive embeddings using Mean Squared Error (MSE) loss.
MSE
(2)
is derived from different view that is passed through momentum encoder while in SimSiam + comes from different view passed through the same encoder except has a stop-grad operation to prevent embedding collapse. We compare Q-Match with i-Mix BYOL on the Cover Type dataset in Table 2 and find is competitive with our method. But our implementation of i-Mix B BOL failed to achieve good performance. We leave exploring i-Mix B BYL on low labeled data regime future work.
In BYOL Grill et al. 2020
The second class of losses that do not require explicit negatives use prototypes to minimize the cross-entropy between two distributions induced by the positive pair. Examples of this approach are SWAV (Caron al. 2020) and DINO (?).
8

---

Under review as a conference paper at ICLR 2023
DINO H1 x . P C
where P is a list of learnable prototypes maintained by the model. To avoid embedding collapse DINO uses momentum encoding with a combination of centering and sharpening while SwAV uses Sinkhorn clustering.
Like DINO and SWAV, Q-Match also uses the cross-entropy between two distributions to learn an encoder. But Q-Match differs from them in two aspects. First, while DINO and SWAV use learnable prototypes to induce the target distribution, we use queue to produce the target distribution. Second, instead of relying on centering and sharpening or Sinkhorn clustering, we use a queue of past embeddings to prevent embedding collapse. Since the queue of embeddings keeps getting refreshed, is non-trivial for the model to collapse.
Reconstruction-base based Self-supervised Learning. A commonly used approach for self-supervised learning is using reconstruction loss to solve de-noising pretext task. Methods, like VIME (Yonn et al. 2020) and (Arik & Plister (02I)) take the original sample in the data, and corrupt some of its values using& corruption mask sampled from Bernoulli distribution. They then learn an encoding function as well as a reconstruction function which aims to reconstruct the original sample using both the learned parameters of the encoder and the parameters specific to the reconstruction task. TabNet is pre-trained to only predict the reconstructed values. On the other hand, VIME also proposes predicting the corruption mask in addition to predicting the reconstructed values. We find the corruption function introduced in these papers useful for producing student and teacher views in Q-Match. However, we find the distribution matching loss to be more effective in the low data regime than the reconstruction based losses.
Semi-supervised Learning Our approach is also closely related to the semi-supervised approaches like FixMatch (Sohn et al., 2020) and PAWS (Assran et al. (02T)) that learn encoders by matching student-teacher distributions. The difference is that we focus on the self-supervised setup where the downstream task fine-tuning happens after the sulf-super pre-training stage. Hence, we cannot assume knowledge of any known classes during the pre-training stage. Instead, we use queue to induce the student and teacher distributions. Additionally, study the problem in the context of tabular datasets while the above mentioned papers are evaluated on image datasets.
CONCLUSION
We introduced new self-supervised algorithm. Q-Match, that learns new, useful representations of tabular data entirely from unlabeled data. Q-Match utilizes a queue to perform continuous self- distillation by matching the student distribution to the teacher distribution as training proceeds. We show that Q-Match outperforms existing self-supervised algorithms and supervised learning on tabular datasets in terms of classification accuracy. Additionally, we show that Q-Match is more efficient than existing methods in terms of sizes of the unlabeled pretext dataset and of the labeled downstream dataset. Q-Match continues to outperform other methods when the size of both the pretext and the downstream datasets is increased.
6 BROADER IMPACT
Q-Match has minimal assumption about the distribution or the domain of data. Hence, it can be applied to data that is not tabular mix of tabular and non-tabular domains well. We show that Q-Match reduces the amount of unlabeled pretext data and labeled downstream data. This can be advantageous in domains when collecting data itself is expensive, when samples can only be labeled after waiting a long time, or even when the total number of possible labels are naturally limited (e.g.. rare medical diagnoses).
On the other hand, even if the encoder is learned in self-supervised manner without labels it might be biased against certain groups, especially they are rarely represented in the data. It recommended to visualize embeddings to identify such cases. Performing detailed group-wise analysis on the downstream task metric will also surface any bias learned by the encoder.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 6679-6687, 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 8443-8452. 2021.
Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communicatioons, 5(1), jul 2014. doi: 10. 10.1138//comms5308 URL
https://doi.osg http///...1022///ooooooom 5308
Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg: Variance- sariane-invariane--cvvaaannn regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://githun cww./oogle///aaaaaooooooc
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-96660 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.
1597-1607. PMLR, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 15750-15758, 2021.
Sajad Darabi, Shayan Fazeli, Ali Pazoki, Sriram Sankararaman, and Majid Sarrafzadeh. Contrastive mixup: Self- and semi-supervised learning for tabular domain, 2021. URL http::t/pa/aiiii
org ab///bb//tt 2108 12296
Li Deng, Geoffrey Hinton, and Brian Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In 2013 IEEE international conference on acousties, speech and signal processing, pp. 8599-86600 IEEE, 2013.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04400 2018.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588-9597, 2021.
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. 2013. doi: 00448550/ARXIV.1302.44 URL htllitttss//acz...ooo htl.lccppps/aaav..o////////333 4389
Yury Gorishniy. Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning, 2022. URL http:s///aaaiiv.oor/////////////
10

---

Under review as a conference paper at ICLR 2023
Dean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Xintian Han and Rajesh Ranganath. Core: Self-and semi-supervised tabular learning with conditional regularizations. 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVE Conference on Computer Vixion and Pattern Recognition (CVPR), pp. 9726-9735, 2020. doi: 10.100CCYPR2200022200000777
Ron Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. Kdd, volume 96, pp. 202-207, 1996.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-99) 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.a www.:/awwwwecc.c.oooo wttpw//wwwwwww//////s//.s/// 2, 2010.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix; A domain-agmostic strategy for contrastive representation learning. 2020. URL https://arxiv
brg/abs/2010..o 08887
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.03748 2018.
Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. Machine translation using deep learning: An overview. In 2017 international conference on computer, communications and electronics (comptelix), pp. 162-167. IEEE, 2017.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-60 6088 2020.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via con-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 133111033-1044 2020.
A APPENDIX
A. ALGORITIM
In Algorithm I
we show the pseudo code of our method.
A.2 HYPER PARAMETERS
In Table we show the values of hyper-parameters search over. For each method we perform a grid search over its relevant hyperparameter values and choose the best one. We use a value of TEMA 0.9 for the momentum encoder and teacher temperature, Tt, of 0.04 in Q-Match.
11

---

Under review as a conference paper at ICLR 2023
Algorithm Pseudo Code for Single Q-Match Training Step
# Create two views of the same data. teacher_viee view aug (x, teacher_corruptine crudent_view aug (x, student_corruptiin
Pass views through the model.
teacher_embed encoder (teacher_view, ema_params) student_embed encoder seeeuudet_vieww. params)
Normalize embeddings.
teacher_enbed_nomm stop_gradient (12_oomaaiieee (teacher_embed)
student_embed_non = 12_normalize ( (studdnt___eeedd
Compute student and teacher distributions.
teacher ciccherloggiss teacher teacher_embed_noe 2
Q.T T
teacher_
teacher_ softmax (tacceertttch
student_logits student_embed__ne [ Q.T student_dist
student
# Compute loss.
loss cross _ttrsssentppppppeeeee (teacher_diss) dist, student_dist
gradients = compute_gradiente (loss, params)
Update params, ema params, and queue. params = update_pad paatte__paaassse (gradients, params)
ema params tau ema params
tau) params
update_queus (Q, @eacher_embed_noxee
Parameter Name
Search Space
Relevant Algorithm
Learning rate Pre-text learning rate Temperature, т Corruption probability Student corruption
[10-5, 10-4, 10-3, 10 2]
All
[10-5, 10-1,1011000 10-31
All
[0.04, 0.10, 0.15, 0.20, 33]
i-Mix
[.3, 4,
VIME, TabNet, SimCLR, SimSiam, VICReg, DINO
[.3, 4, .5]
Q-Match
Student temperature, T.
[0.05, 0.1. 0.2]
Q-Match
Queue size
[2 2
Q-Match
Table Hyper parameter spaces for algorithms used during training.
A.3 IMAGENET EXPERIMENTS
As our proposed approach is Nomain-agnosse it can also be used to learn image features in self- supervised manner. We pre-train a ResNet-50 model with the Q-Match loss. We use the multi-crop training setup of NNCLR Dwibedi et al. (2021) and SWAV Caron et al. (2020) where 2 crops of 224 x 224 and 6 crops of 96 x 96 are used in single batch. The two larger sized images serve as the teacher views. We pre-train the model for 800 epochs. We use student temperature of and teacher temperature of 0.04. We do not use any color jitter augmentation on the teachers view to produce the weakly augmented view. Crop augmentation is used on both views. We use queue size of 98304, embedding size of 256, momentum of 0.99, and the same projection MLP used NNCLR.
We report the results of linear evaluation of the 2048-d output from the ResNet-50 model in Table 7 We find that training with the Q-Match loss results in better performance than training with the baseline methods. In particular, the performance improvement over DINO is interesting because both DINO and Q-Match learnable prototypes to induce these distributions. Q-Match uses queue of past embeddings. This further validates the utility of the queue in the suuent-ttecche distribution matching framework. Furthermore, even though Q-Match was developed primarily with tabular datasets with low number of downstream classes, it is capable of strong performance on image datasets with larger number of classes.
the student-teacher distribution matching loss. While DINO
12

---

Under review as a conference paper at ICLR 2023
Method
Accuracy
SWAV (Caron al. 2020)
75.3
DINO
Caron et al.
202T0
75.3
NNCLR (Dwibedi et al. 2021
75.6
Q-Match (Ours)
76.0
Table 7: ImageNet Linear Evaluation. Linear evaluation accuracy for NNCLR, SWAV, and Q-
Match.
Dataset Name
Pretext Training
Downstream Training
Downstream Test Set Size
Experiments
Set Size
Set Size
Higgs 1%
98k
980
500k
Few Shot, Sensitivity
Higgs 5k
50k
5k
25k
Baseline
Higgs 10k
10M
10k
500k
Baseline
Higgs 100k
100k
100k
500k
Baseline
Higgs Variable Labeled
98k
(980, 9.80k, 19.6k, 98k}
500k
Data Scaling
Higgs Variable Pretext
(10k, 40k, 160k, 640k, 2.56M)
500k
Data Scaling
Cover Type
113400
1134
429812
Few Shot, Sensitivity
Cover Type 10%
464809
46480
116203
Baseline
Cover Type 15k
11340
11340
565892
Baseline
Adult 1%
8170
86
16281
Few Shot
MNIST 1%
57k
600
10k
Few Shot, Sensitivity
MNIST 10%
60k
10k
10k
Baseline
Table Dataset splits.
A.4 DATASETS AND SPLITS
In Table we show in detail the datasets and their splits. Different experiments required us to create different splits. For comparison with baseline methods, we attempt to stick to the splits used in the baseline papers. For data scaling and few-shot experiments, we create our own splits and train all methods on the same splits.
13


Reviews:
[{"review_id": "ka4kKp-Pib", "reviewer": "Reviewer_VckF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a queue of embeddings of samples from the unlabeled dataset to improve the performance of self-supervised learning. It presents that the proposed method have achieved success on the tabular datasets.", "summary_of_the_review": "The technical novelty of this paper is very limited. ", "strength_and_weaknesses": "Strength:\n\nThis paper has conducted some interesting experiments.\n\nWeaknesses:\n\n1. The technical novelty of this paper is very limited. The proposed method is just a combination of some existing methods. \n\n2. The motivation of the proposed method is unclear. This paper has not explained why the proposed method can achieve proper performance. \n\n3. The writing of this paper requires significantly improvement. There are many typos and grammar mistakes. For example, in the **method** subsection of Chapter 2, the word \"student\" in the sentence \"We pass $x_{i,s}$ Through the student model to produce the student embedding $z_{i,s}$\" should be \"teacher\", In the **Semi-supervised Learning** subsection of Charpter 4, the word \"teach\" in the second last sentence may be \"teacher\". ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a_yJEItXh9", "reviewer": "Reviewer_CX2D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adapts the Fixmatch method [Sohn et al., 2020] from semi-supervised to self-supervised learning, and significantly improves the results compared with the benchmarks in experiments.", "summary_of_the_review": "The contribution of this article is insufficient to be accepted to the conference. However,  I believe the paper is an excellent workshop paper.\n\n\n###########\n\nI thank the authors for their response. However, I still think the contribution of the article is relatively small so keep my score unchanged.", "strength_and_weaknesses": "Strength:\n\n1. The paper is relatively well-written and easy to follow.\n2.  The paper conducts extensive experiments to validate the proposed method empirically.\n\nWeaknesses.\n\n1. The work has limited novelty on top of Fixmatch [Sohn et al., 2020].", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VGaPHQiYZR", "reviewer": "Reviewer_gtA5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Q-match, a self-supervised method for tabular data. Similar to Fixmatch and DINO, the proposed method matches the feature distribution induced by a teach network and a student network, with a feature queue similar to moco. Results on several tabular data benchmarks indicate that the proposed method outperform previous self-supervised methods.", "summary_of_the_review": "Self-supervised learning has been a very successful technique across vision and NLP. This paper extends the common self-supervised techniques to tabular data and shows its effectiveness. Although the authors could have provided more justifications of their method, I tend to accept this paper.", "strength_and_weaknesses": " Strengths:   \n1.\tThe paper is well-written and easy to follow. The algorithm is described clearly. For each part of the algorithm, the authors provide the motivation and mention the counterpart in vision methods.    \n2.\tThe proposed method is simple and has good performance. The method does not include many tricks, but a rather original implementation of the principled self-supervised learning method. It is encouraging to see it works well on tabular data, across various datasets and settings.    \nWeaknesses:    \n1.\tDespite the good performance, I’m also curious to see the contribution of each part of the model. How does loss function affect the performance? What if you use the VCIReg loss? How does the queue affect the performance? If you remove the queue and use large batchsize following SimCLR, or simply use Simsiam, what will happen? It would be better if the authors include them as an ablation study.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mu9YfN7iiVB", "reviewer": "Reviewer_8ARq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a self-supervised learning method for tabular data. Specifically, it adopts a pipeline similar to DINO – it matches the categorical distribution produced from an EMA teacher network and a student network but the classifier is built with a queue of past embeddings instead of a learned linear layer. The paper shows empirical results and ablation studies on multiple datasets and demonstrates the effectiveness of the proposed method on tabular data.", "summary_of_the_review": "The idea of the proposed SSL method is straightforward and seems to be effective. However, it shares too many similarities with SSL methods for images without any pieces which are special to tabular data. It is therefore unclear if the performance gain is unique for the specific design in this paper.", "strength_and_weaknesses": "Strength\n1. The motivation of the proposed method is clear: to build a SSL method similar to the existing semi-supervised methods for tabular data.\n2. All the building pieces (with slight differences) exist in SSL for image data and therefore the proposed method is simple and straightforward.\n\nWeakness\n1. The novelty is limited to empirical application to tabular data.\n2. The paper does not seem to have compared against many similar methods it mentioned (SwAV for instance). As the proposed method is  not tailored for tabular data from a pure methodology perspective, it is unclear if the proposed method makes more sense than just applying DINO, SwAV, etc. to the tabular datasets used in the experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject