Confident Sinkhorn Allocation for Pseudo-Labeling
Authors: Vu Nguyen
Sachin Farfade
Anton van den Hengel
Venue: Submitted to ICLR 2023 (2023)
Keywords: pseudo-labeling
semi-supervised learning
tabular data

Abstract: Semi-supervised learning is a critical tool in reducing machine learning’s dependence on labeled data. It has been successfully applied to structure data, such as image and language data, by exploiting the inherent spatial and semantic structure therein with pretrained models or data augmentation. Some of these methods are no longer applicable for the data where domain structures are not available because the pretrained models or data augmentation can not be used.

Due to simplicity, existing pseudo-labeling (PL) methods can be widely used without any domain assumption, but are vulnerable to noise samples and to greedy assignments given a predefined threshold which is typically unknown. This paper addresses this problem by proposing a Confident Sinkhorn Allocation (CSA), which assigns labels to only samples with high confidence scores and learns the best label allocation via optimal transport. CSA outperforms the current state-of-the-art in this practically important area of semi-supervised learning.

Under review as a conference paper at ICLR 2023
CONFIDENT SINKHORN ALLOCATION FOR PSEUDO- LABELING
Anonymous authors Paper under double-blind review
ABSTRACT
sunii-suervised learning is a critical tool reducing machine learning's 's depen- dence on labeled data. It has been successfully applied to structure data, such image and language data, by exploiting the inherent spatial and semantic structure therein with pretrained models or data augmentation. Some of these methods are no longer applicable for the data where domain structures are not available because the pretrained models or data augmentation can not be used.
Due to simplicity, existing pseudo-labeling (PL.) methods can be widely used without any domain assumption, but are vulnerable to noise samples and to greedy assignments given a predefined threshold which is typically unknown. This paper addresses this problem by proposing Confident Sinkhorn Allocation (CSA), which assigns labels to only samples with high confidence scores and learns the best label allocation via optimal transport. CSA outperforms the current state-of-the-an this practically important area of semi-supervised learning.
INTRODUCTION
The impact machine learning continues to grow in fields disparate as biology (Libbrecht & Noble, 2015; Tunyasuvunakool et al., 2021), quantum technology (Biamonte et al., 2017; van Esbroeck al., 2020; Nguyen al., 2021), brain stimulation (Boutet et al., 2021; van Bueren al., 2021), and computer vision (Esteva et al., 2021; Yoon et al., 2022). Much of this impact depends on the availability of large numbers of annotated examples for the machine learning models to be trained on. The data annotation task by which such labeled data is created is often expensive, and sometimes impossible, however. Rare genetic discases, stock market events, and eyber-security threats, for example, are hard to annotate due the volumes of data involved, the rate which the significant characteristics change, or both.
Related work. Fortunately, for some classification tasks, we can overcome a scarcity of labeled data using semi-supervised learning (SSL) (Zhu, 2005; Huang al., 2021; Killamsetty al., 2021; Olsson et al., 2021). SSL exploits an additional set of unlabeled data with the goal of improving on the performance that might be achieved using labeled data alone (Lee al., 2019; Carmon al., 2019; Ren et al., 2020; Islam et al., 2021).
Domain specific: Supi-super learning for image and language data has made rapid progress (Oymak & Gulcu, 2021; Zhou, 2021; Sohn et al., 2020) largely by exploiting the inherent spatial and semantic structure images (Komodakis & Gidaris, 2018) and language (Kenton Toutanova, 2019). This is achieved typically either using pretext tasks (eg. (Komodakis & Gidaris, 2018; Alexey et al., 2016)) or contrastive learning (eg. (Van den Oord et al., 2018; Chen et al., 2020)). Both approaches assume that specific transformations applied to each data element will not affect the associated label.
Greedy lsbudo-labeling: Without domain assumption, simple but effective way for SSL is pseudo- labeling (PL) (Lee et al., 2013) which generates "pseudo-labels' for unlabeled samples using a model trained on labeled data. A label k is assigned to unlabeled sample x; where a predicted class probability larger than a predefined threshold y as
o 1 k x,) 2 r
(1)

---

Under review as a conference paper at ICLR 2023
Table Comparison with the related approaches terms of properties and their relative trade-off's
Algorithms
Not domain specific
Uncertainty consideration
Non-greedy
Pseudo-Labeling (Lee et al., 2013)
FlexMatch (Zhang et al. 2021)
X
Vime (Yoon al., 2020)
x
NA
MixMatch (Berthelot et al., 2019)
X
X
NA
FixMatch (Sohn et al., 2020)
X
NA
UPS (Rizve et al., 2021)
SLA (Tai et al., 2021)
X
CSA
where y [0, I] is a threshold used to produce hard labels and p(y = k x:) is the predictive probability of the i-th data point belonging to the class k-th. A classifier can then be trained using both the original labeled data and the newly pseudo-labeles data. Pseudo labeling is naturally an iterative process, with the next round of pseudo-labels being generated using the most-recently trained classifier. The key advantage of pseudo-labeling is that it does not inherently require any domain assumption and can be generally applied to most domains, including tabular data.
Greedy PL with uncertainty: Rizve et al. (2021) propose an uncertainty aware pseudo-label selection (UPS) that aims to reduce the noise in the training process by using the uncertainty score together with the probability score for making assignments:
= p - k x.) 2 v
where Yu is additional threshold on the uncertainty level and (p)) is the uncertainty of a prediction p. As shown in Rizve et al. (2021), selecting predictions with low uncertainties greatly reduces the effect of poor calibration, thus improving robustness and generalization.
However, the aforementioned works in PL are greedy in assigning the labels by simply comparing the prediction value against predefined threshold y irrespective of the relative prediction values across samples and classes. Such greedy strategies will be sensitive to the choice of threshold.
Non- pseudo-labeling: FlexMatch (Zhang et al., 2021) considers adaptively selecting a thresh- old n for each class based on the level of difficulty. This threshold is adapted using the predictions across classes. However, the selection process is still heuristic in comparing the prediction score with an adjusted threshold. Recently, Tai et al. (2021) provide a novel view in connecting the pseudo- labeling assignment task to optimal transport problem, called SLA, which inspires our work. SLA and Flex are better than existing PL in that their non-greedy label assignments not only use the single prediction value but also consider the relative importance of this value across rows and columns in holistic way. However, both SLA and FlexMatch overconfidently assign labels to noise samples and have not considered utilizing uncertainty values in making assignments.
Contributions. We propose here semi-supervised learning method that does not require any domain-specific assumption for the data. We hypothesize that this is by far the most common case for the vast volumes of data that exist. The method we propose based on pseudo-labeling of set of unlabeled data using Confident Sinkhorn Allocation (CSA). Our method is theoretically driven by the role of uncertainty in robust label assignment in SSL. CSA utilizes Sinkhorn's 's algorithm (Cuturi, 2013) to assign labels to only the data samples with high confidence scores. By learning the label assignment with optimal transport, CSA eliminates the need to predefine the heuristic thresholds used in existing beeudo-labeling methods, which can be greedy. The proposed CSA is applicable to any data domain, and could be used in concert with consistency-ban is particularly useful for data domain where pretext tasks and data augmentation are not applicable, such a tabular data.
approaches (Sohn et al., 2020), but
2 CONFIDENT SINKHORN ALLOCATION (CSA)
We consider the supi-super learning setting whereby we have access to dataset consisting of labeled examples D, = ) and one of unlabeled examples D, = {x/) Mi, where x; E Re and
y; E (I,..., K). We define also X [xx), == (1,...,: + N.). Our goal is to utilize D, UD, to
2

---

Under review as a conference paper at ICLR 2023
Probabilities from M classifiers
T-test
Estimate Q using Sinkhorn
Prob
Prob Pи
Confidence
Cost C= log(P)
Assignment
0
3.0
3.0
2
2.5
4
Uniabeler Data
2.0
2.0
0.6
1.5
0.4
10
12
0.5
14
1 Class
Class
Class
Remove low confident samples
Class
Figure I: A depiction of CSA in application. We estimate the ensemble of predictions P on unlabeled data using classifiers which can result in different probabilities. We then identify high-confidence samples by performing T-test. Next, estimate the label assignment Q using Sinkhomn's 's algorithm. The cost C the optimal transport problem the negative of the probability averaging across classifiers, C C= log We repeat the process the remaining unlabeled data required.
learn predictor X + y that is more accurate than predictor trained using labeled data Dj alone. notation summary provided in Appendix Table
Generating high-quality pseudo labels is critical to the final classification performance, as erroneous label assignment can quickly lead the iterative pseudo-labeling process astray. We provide in Sec. 2.1 a theoretical analysis of the role and impact of uncertainty in as far as we are aware. Based on the theoretical result, we propose two approaches to identify the high-confidence samples for assigning labels and use the Sinkhorn's algorithm to find the best label assignment. We name the proposed algorithm Confident Sinkhorn Allocation (CSA). provide diagram demonstrating CSA in Fig 1, and comparison against related algorithms in Table 1.
the first such analysis
THEORETICAL ANALYSIS ON THE EFFECT OF UNCERTAINTY IN PSEUDO-LABELING
Our theoretical results highlight two properties of PL settings: (i) less noise and uncertainty are beneficial, and (ii) more unlabeled data is beneficial. For the analysis, make a popular assumption that input features X; E Dj and 2, are sampled i.i.d. from feature distribution Px. and the labeled data pairs (x) in Dj drawn from joint distribution Px,r
We consider the binary classification problem in one-dimensional space as follows: generate the feature x y= +1 sid N(H,, o²) and similarly x y = - jid ((,,,,²), where μ4 and μ_ are the means of the positive and negative classes respectively, is the standard deviation indicating the level of data noisiness. Without loss of generality, let μ4+ > μ.... The optimal Bayes's classifier is f(x) = s sign (x ". eg. classify y; = +1 if > (u4 + /2) Therefore, the following, we measure the probability bound to learn (M+ /-)/2 criterion for achieving good performance.
Let 1 and (8, } be the sets of unlabeled data whose pseudo-labels are +1 and 11, re- spectively. Let {4, 1 X 7 o²) and otherwise X; ~ o2), Similarly, we define {4, //=1 indicating correct assignment for negative class. Instead of generating from Bernoulli distribution as used in Yang & Xu (2020), we generate the binary indicators 1,+ and 1,- from classifier and denote the expectations E(7, ). E(E and variances Var(4,)) Var(( respectively. In the uncertainty estimation literature (Der Kiureghian & Ditlevsen, 2009), o² called the aleatoric uncertainty (noise from the obser- vations) and Var(I;) Var(()) are the epistemic incertainties (from the model knowledge). As mentioned above, aim learn M++ via the extra unlabeled data. It is natural from the binary
Hit be the binary indicators of correct assignments, such as if 1," = 1, then

---

Under review as a conference paper at ICLR 2023
Confidence = 1 Total Variance
Confidence T-test
16
0.84
14
0.80
12
0.76
0.72
8
0.68
0.64
44
0.56
Figure 2: Examples using total variance and T-test for estimating the confidence level on = classes. The yellow area indicates the highest confidence. We exclude the samples falling into the dark area where the value T-test less than 2. We have simplified the tabular data (with possibly mixed categorical--oomiious--- features) into the d = dimensional representation.
classification setting above to construct our estimate as 0 = 1 R x /
+
+
X / Please
Appendix A.1 for details.
Theorem 1. For 8 > O, with probability at least - 2exp {
+..
9 Varrll Varl,
900
estimate Ô satisfies θ Almi- (u++
< 8 where A IE(4,) E(T,
Proof. Our proof (see Appendix A.1) extends the result from Yang & Xu (2020) taking into account the data noisiness o² and uncertainty Var(4")) Var(4,) for pseudo-labeling
The thcoretical result reveals several interesting aspects. First, training data imbalance affects the accuracy of our estimation, as discussed in Yang & Xu (2020). The more imbalanced the data is, the larger the gap A would be, which influences the closeness between the estimated 0 and desired value _ Note that data imbalance is not the focus of this paper. In
Interpretation.
addition, the second term of the bound R- shows that more unlabeled data A+ 1 or Ā 1 is helpful for good estimation. We empirically validate this property in Fig. 5. Finally, our analysis takes step further to show that both aleatoric uncertainty o² and epistemic uncertainty V Var(() reduce the probability of obtaining good estimation. In other words, less uncertainty is more helpful. Note that the interpretation through the uncertainty has not been explored in Yang & Xu (2020). To the best of our knowledge, we are the first theoretically characterize the role of data noise 2 and uncertainties Var(4") ) Var(4)) affecting the performance in SSL.
A.
2.2 IDENTIFYING CONGIECNN SAMPLES FOR ASSIGNING LABELS
In Theorem 1, show that the data sample noise and uncertainty can worsen the estimated classifier. Therefore, we propose to identify and ignore the high-uncertain samples from being assigned labels.
is notorious that machine learning predictions can vary with different choices of hyperparameters. Under these variations, it is unclear to identify the most probable class to assign labels for some data points. Not only the predictive values but also the ranking order of these predictive samples vary with different hyperparameters. The variations in the prediction can be explained due to (i) the uncertainties of the prediction coming from the noise observations (aleatoric uncertainty) and model knowledge (epistemic uncertainty) (Hüllermeier & Waegeman, 2019) and (ii) the gap between the highest and recond-highest scores is small. These bring a challenging fact that the highest score class can be changed with different set of hyperparameters, This leads to the confusion for assigning pscudo labels because (i) the best set of hyperparameter is unknown given limited labeled data and (ii) we consider ensemble learning setting wherein multiple models used together.
To address the two challenges above, we are motivated by the theoretical result in Theorem 1 to only assign labels to samples in which the most probable class is statistically significant than the second-mo most probable class. For this purpose, we propose to use two criteria for measuring the confidence level: (i) Welch's T-test and (ii) total variance across classes.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Confident Sinkhorn Label Allocation
Input:
lab data {x, unlab data X,,, Sinkhorn reg fraction of assigned label p [0, Output: Allocation matrix Q RNxX"
for I,... do
Train M models oj...., GM given the limited labelled data X,,y1
Obtain confidence set Xy Xu using T-value from Eq. (3) where :: [Xy]
Define cost matrix C from Eq. (16) using Xy NVC -bb 1x) and [b4N:N(1 pb! 1x)
Set marginal distributions
/+ Sinkhorn's s algorithhr
while j I,...JJ until converged do Initialize a k; Update b(i+1)
exp( Cul-Ca/ and a(++1)
exp( (a(-CCale 60+H)
Obtain Q = diag exp (CCe/E) diag(b()) Augment the assigned labels to (X,,)) and remove these points from the unlabeled data X,,
// the ass gnment matrix
Welch's T-test. For each data point i, we define two empirical distributions (see Appendix Fig. 6) of predicting the highest N (Htto, o7,o) and second-highest class N (Mi o. estimated from the predictive probability across M classifiers, such as Mio .7 >M Pm = x;) and Pio = I EM (y C x4) are the empirical means; 0f and of are the variances. We consider Welch's T-test (Welch, 1947) to compare the statistical significance of two empirical distributions:
Hi,o Hi,2
T-value(x;)
Vil. 7
The degree of freedom for the above statistical significance is (M (M-l)(a As a standard practice of
in statistical testing (Neyman & Pearson, 1933; Fisher, 1955), calculate the degree of freedom and look at the T-value distribution. In this particular setting, we get the following rules if the T-value less than 2, the two considered classes are from the same distribution - eg.. the sample might fall into the dark area in right Fig. 2. Thus, we exclude sample from assigning labels when its T-value less than
The estimation of the T-test above encourages separation between the highest and second- highest classes relates to entropy minimization (Grandvalet Bengio, 2004) which encourages classifier to output low entropy predictions on unlabeled data.
Total variance. multilabel classification settings (Kapoor et al., 2012), multiple high score classes can be considered together. Thus, the Welch's T-test between the highest and second-highes is no longer applicable. We consider the following total variance across classes as the second criteria. This total variance for uncertainty estimation has also been used in UPS (Rizve et al., 2021) using Monte Carlo dropout (Gal& & Ghahramani, 2016) in the context of deep neural network:
Vpp x))
M
Pm (mmly= = x
Pme
k x)
(4)
M
M
variance of assigning class
We exclude the data point with high uncertainty measured by the total variance. It makes sense because consensus of multiple classifiers is generally good indicator of the labeling quality. In our setting, a high consensus is represented by low variance or high confidence. While the T-test naturally has threshold of 2 reject data point, the total variance does not have such a threshold and thus we need to impose our own value. In the experiment, we reject 50% of the points with high total variance score. We visualize the confidence estimated by T-test and total variance Fig. 2.
We further consider the entropy criteria proposed in Malinin et al. (2020) as another way of estimating the confidence score in Appendix B.I.
Discussion. In Appendix Table 4, we empirically compare the performance using different uncertainty choices. We show that the Welch T-test is the best for this lebeling problem which can
Denote the highest score class by o: o() and second-highest score class by == @((i) for data point i.
5

---

Under review as a conference paper at ICLR 2023
explicitly capture the uncertainty gap in predicting the highest and second-highest classes while total variance and entropy can not.
2.3 OPTIMAL TRANSPORT ASSIGNMENT
We use the confidence scores to filter out the uncertain data points and assign labels to the remaining using Sinkhorn algorithm (Cuturi, 2013). Partieularly, follow the novel view in Tai et al. (2021) to interpret the label assignment process as an optimal transportation problem between examples and classes, wherein the cost of assigning an example class is dictated by the predictions of the classifier.
Let denote N S No be the number of accepted points, eg. by T-test, from the unlabeled set. Let define an assignment matrix Q RA*K of N unlabeled data points to K classes such that we assign X class k if Qi O. We seek an assignment Q that minimizes the total assignment cost Li CaCa where Ca is the cost of assigning an example i to class k given by the corresponding negative
probability as used in Tai et al. (2021), i.e. Ca == p(y
x;) where S p(v x,) S
minimizee
(2.c)
(5) minimizeq.uv.
(2.c) Qa 2 0,u 0, Qlx+u= ly
(10)
Qn 20
(6)
s.t.
(11)
Qlx S In
(7)
(12)
1'1N S Nw+
(8)
e' + v= W+N
(13)
11 Npw 1x
(9)
u Iw -= T-NT= (N11-PW) N=N(1-ww11k 1k
(14)
+ -N(w
1w
(15)
where 1a and ly are the vectors one in K and N dimensions, respectively; p E [0,1] is the fraction of assigned label, i.e., p 1 is full allocation; W+, w- E R are the vectors of upper and lower bound assignment per class which can be estimated empirically from the class label frequency in the training data or from prior knowledge.
Our formulation has been motivated and modified from the original SLA (Tai et al., 2021) introduce the lower bound w specifying the minimum number of data points to be assigned in each class. We refer Appendix B.6 for the ablation study on the effect of this lower bound and Appendix A.2 for the derivations on the equivalence between the inequality in Eqs. (6,7,8,9) for linear programming and equality in Eqs. (11,12)13,14415)) for optimal transport.
We define the marginal distributions for c= iw,NNN(11 -1-w'" 1x) E RK+1 We define the prediction matrix over the unlabeled set, which satisfied the confidence test, P E MMXNx such that each element PoLiX Pm x)) and the averaging prediction over M models P matrices are defined below
[1N:NWW] - pw]1x)]]
RN+1 and column
M WM
Pm.,,- RANK, The cost and the assignment
log P
Ow 0
(16)
Ø =
Q
E ((++))((K+),
(17)
0%
We derive in Appendix A.3 the optimization process learn Q by initializing bD 1 and iteratively updating for j == 1...J iterations or until convergence:
((+1)
r
(18)
b(+1)=
(19)
C.. exp
Cia T
al+t)
exp
After estimating Ô, we get Q from Eq. (17) and assign labels to unlabeled data, i.e., where 2i > and repeat the whole pseudo-labeling process for few iterations shown in Algorithm 1. The optimization process above can also be estimated using mini-batches (Fatras et al., 2020) if needed.
Discussion. Instead of performing greedy selection using threshold like other PL algorithms, CSA specifies the frequency of assigned labels including the lower bound w and upper bound w per class well as the fraction of data points p (0, 1) be assigned. Then, the optimal transport will automatically perform row and column scalings to find the 'optimal' assignments such that the selected element, Qu+ > 0, is among the highest values in the row (within data point i-th) and the

---

Under review as a conference paper at ICLR 2023
Madelon
Emotions
57.5
68
Test Accuracy
57.0
Test Precision
56.5
$6.0
05-
55.5
$5.0
Pseudo-label Iteration
Pseudo-label Iteration
Supervised Learning
Pseudo-Labeling
Figure 3: Comparison with the pseudo-labeling methods on tabular data for single-label classification
(leff) and multi-label classification (righ).. x-axis indicates the iteration = I....T.
same time receive the highest values in the column (within class k-th). In contrast, existing PL will either look at the highest values in row column, separately - but not jointly like ours. Here, the high value refers to high probability or low-cost C in Eq. (5).
Multi-label. The resulting matrix Q can naturally assign each data point to multiple classes. By that way, we can perform multi-label classification (Read et al., 2011; Nguyen et al., 2016) by assigning a vector of labels & to each data point s.t. Qi > 0. This multi-label option is also readily available for other pseudo-labeling methods (Lee et al., 2013). In this multi-label setting, we will use the total variance the main criteria for confidence estimation the T-test is not suitable to consider the co-existence of multiple labels.
3 EXPERIMENTS
Baselines. We select to compare our CSA with the following baselines which are suitable for the data setting without data augmentation: Pseudo-labels labellng (PL) (Lee al., 2013), FlexMatch (Zhang et al., 2021), UPS (Rizve et al., 2021) and SLA (Tai et al., 2021). We adapt the label assignment mechanisms in Zhang et al. (2021), 1): Rizve et al. (2021) 1) for tabular domains although their original designs are for computer vision tasks. Without explicitly stated, 0.8 for PL., FlexMatch, UPS. We further vary this threshold with different values in Section 3.2. We also compare with the supervised learning method trained using the labeled data. The multi-layer perceptron (MLP) implemented in Yoon et al. (2020) includes two layers using 100 hidden units. Additional to these pseudo-labeling based approaches, compare the proposed method with Vime.
pre-defined threshold of
Data. We use public datasets for single-label and multi-label classifications from UCI repository (Asuncion Newman, 2007). Since we do not use any domain assumption, these data are presented as vectorized or tabular data formats. In addition, we obtain three products, renamed as A, B, and C from Amazon catalog. The Amazon data include product numerical and categorical attributes as well as text features of product review. We summarize all dataset statistics in Appendix Table 7. Our data covers various domains including: image, language, medical, biology, audio, finance and retail.
Classifier and byperparameters. Given limited labeled data presented in tabular format, we choose XGBoost (Chen & Guestrin, 2016) as the main classifier which typically outperforms state-of-the-art deep learning approaches in this setting (Shwartz-Ziv & Armon, 2022) although our method is more general and not restricted to XGBoost. We use = 20 XGBoost models for ensembling, We refer to Appendix B.3 on the empirical analysis with different choices of M. The ranges for these hyperparameters are summarized in Appendix Table 6.
Setting. Given the setting with limited labels data, do not validation set for tuning hyperparameters of the XGBoost model. We repeat the experiments 30 times with different random seeds, then report the and standard error. All Python implementations will be released in the final version.
3.1 COMPARISON OF PSEUDO-LABELING FOR SINGLE AND MULTI-LABEL CLASSIFICATION
We use accuracy as the main metric for evaluating single-label and precision for multi-labe classifi- cation (Wu & Zhou, 2017). We first compare CSA with the methods in pseudo-labeling family.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison with pseudo-labeling methods.
Supervised
Learning
Datasets
Vime
PL
Flex
UPS
SLA
CSA
XGBoost
MLP
Segment
95.42
94.63=1
92.71 =1
95.68 =
95.68 =
95.67 -1
95.80
95,90=
Wdbe
89.20
91.33
91.8315
91.23
91.23
91.6 3
90.61 -2
91.83 1
Analcatdata
91.63+2
96.17
96.13=2
90.95+22
90.62+33
91.33+33
90.98+22
96.60 +
German-credit
2
71.10
70.50 =4
70.72
70.72
71.15
70.72
71.47 1
Madelon
3
50.80
52.97
56,4514
56.7414
57.1313
56.5314
57.51
Dna
88.53
76.50 +
79.07
88. 18+1
88.17+1
8
88.09
89.24
Agar Lepiota
$7.63
63.80
63.83 =
58.98
59.53=1 1
58.88
58. 38.96=1
59.53
Breast cancer
93,20
86,40 6
85,87 1
92.8912
92,8912
93,38
92.76.12
93.55 2
Digits
82.23
86.80
84.10=1
81.67#3
81,44+3
83.78+3
81,51=3
88.10
Category A
72.71
57.15
50.55 -
72.48.13
72.35 -5-3
72.71 =3
72.24.13
73.97 14
Category B
3
42.85+2
36.45
79.41
80.49
79.45
79.82=2
80.62
Category C
61.35
$1.90
52,80
61.23=1
61.98
61.65
62.04=1
63.01
We show the performance achieved on the held-out test set as we iteratively assign pseudo-label to the unlabeled samples, i.c., varying I.....T. As presented in Fig. and Appendix Fig. 11, CSA significantly outperforms the baselines by a wide margin in the datasets of Analcatdata, Synthetic Control and Digits. CSA improves approximately 6% compared with fully supervised learning. CSA gains 2% Madelon dataset and in the range of [0.4%, 0.8%) on other datasets.
Unlike other labeligg methods (PL, FlexMatch, UPS), CSA can get rid of the requirement to predefine the suitable threshold y E [0. 1). which unknown in advance and should vary with different datasets. In addition, the selection in CSA is non-greedy that considers the relative importance within and across rows and columns by the Sinkhorn's ' algorithm while the existing PL methods can be greedy when only compared against predefined threshold.
Furthermore, our CSA outperforms SLA (Tai et al., 2021), which does not take into consideration the confidence score. Thus, SLA may assign labels to 'uncertain' samples which can degrade the performance. From the comparison with SLA, we have shown that uncertainty estimation provides us tool identify susceptible points ignore from the data and thus can be beneficial for SSL.
We then compare our CSA with Vime (Yoon et al., 2020), the current state-of-the-art methods in semi-supervised learning for tabular data in Table 2. CSA performs better than Vime in most cases, except the Agaricus Lepiota dataset which has an adequately large size (6500 samples, see Table 7) thus the deep network component in Vime can successfully learn and perform well.
Multi-label classification. The advantage of CSA is further demonstrated to assign labels in multi- label classification settings. We present the comparison in the bottom row of Fig. 3 where we show that CSA performs competitively and better than the baselines on the three datasets considered.
3.2 COMPARISON OF CSA AGAINST PSEUDO-LABELING WITH DIFFERENT THRESHOLDS
One of the key advantages of using optimal transport is that it can learn the non-greedy assignment by performing row-scaling and column-scaling to get the best assignment such that the cost C is minimized, or the likelihood is maximized with respect to the given constraints from the row r and column e marginal distributions. By doing this, our CSA can get rid of the necessity to define the threshold y and perform greedy selection in most of the existing pseudo-labeling methods.
We may be wondering if the optimal transport assignment Q can be achieved by simply varying the thresholds y in pseudo-labeling methods? The answer is no. To back up our claim, we perform two analyses. First, visualize and compare the assignment matrix achieved by CSA and by varying threshold y E (0.5, 0.7, } in PL. As shown in left Fig. 4, the outputs are different and there are some assignments which could not be selected by PL with varying Y, eg., the following frow index, column index): 3). {9,3} and { (14,3) annotated in red square in left Fig. 4.
Second, we empirically compare the performance of CSA against varying 7 (0.9,0.8.0.990099) in PL. We present the results in right Fig. 4, which again validates our claim that the optimal assignment in CSA will lead to better performance consistently than PI. with changing values of y.
8

---

Under review as a conference paper at ICLR 2023
Prob P
CSA
PL 0.9
0.7
0.5
German Credit
72,50
0-7.
0=
CSA
y-0.8
y=0.95
72.00
Untablled Data
4-
Accuracy
1
71.50
8-
71.25
10-
70.75
70.50
Pseudo-label Iteraiion
Figure 4: Comparison between CSA versus pseudo-labeling with different thresholds y. Left: the assignments by CSA can not be simply achieved by varying the threshold PL, eg. see the locations highlighted in red square. Right: Comparison when varying thresholds in PL against CSA.
3.3 ABLATION STUDIES
Varying the number of labels and unlabels. Different approaches exhibit substantially different levels of sensitivity to the amount of labeled and unlabeled data, mentioned in Oliver et al. (2018). We, therefore, validate our claim in Theorem by empirically demonstrating the model performance with increasing the number of unlabeled examples. In left Fig. 5, we show that the model will be beneficial with increasing the number of unlabeled data points.
Limitation. In some situations, such as when labeled data is small in size contains outliers, our CSA will likely assign incorrect labels to the unlabeled points at the earlier iteration. This erroneous will be accumulated further and lead to poor performance, shown left Fig. 5 when the number of labels is 100 and the number of unlabels is less than 500. We note, however, that other pseudo-labed labeling methods will share the same limitation.
Varying LLbbled #Unlabeled
Further analysis. We refer to Appendix B for other empirical analysis which can be sensitive to the perfor- mance, including (i) different choices of pseudo-labeling iterations T and (ii) different number of choices for XGB models M, (iii) computational time for each component and (iv) statistics on the number of points selected by each component per iteration.
#1ab-200
80
100
300
500
760
500
1100
CONCLUSION AND FUTURE WORK
#Unlabeled Samples
Figure 5: Performance on Digit w.r.t the number of unlabeled samples given the number labeled samples as 100,200,500, respectively.
Although there has been significant progress in SSL for images and text domains, limited attention has been in- vested for unstructured domains in which data augmenta- tion or pretraining is not available, such as data presenting in tabular format. We propose CSA, a new method for pseudo-labeling particularly suitable for tabular domains but not restricted to.
Our CSA has two key ingredients. First, it estimates and takes into account the confidence levels in assigning labels. Second, it learns the optimal assignment using Sinkhorn algorithm which appropriately scales rows and columns probability to get assignment without the greedy selection using a threshold used in most of PL methods.
The proposed CSA maintains the benefits of PI. its simplicity, generality, and ease of implementa- tion while CSA can significantly improve PL performance by addressing the overconfidence issuc and better assignment with optimal transport.
Future work can extend the ensembling process by not only using XGBoost, but also other classifiers, such as AdaBoost (Wyner et al., 2017) or CatBoost (Prokhorenkova et al., 2018). Another extension is CSA the main label assignment method for SSL and integrate it into training deep learning model to build CSAMatch, analogous to FixMatch (Sohn et al., 2020), MixMatch (Berthelot et al., 2019) and Flex Match (Zhang et al., 2021).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dosovitskiy Alexey, Philipp Fischer, Jost Tobias, Martin Riedmiller Springenberg, and Thomas Brox. Diseriminative, unsupervised feature learning with exemplar convolutional, neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9):133-11747, 2016.
Arthur Asuncion and David Newman. Uci machine learning repository. 2007.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019. 2,
Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd. Quantum machine learning. Nature, 549(767 (::6661):19555 195-202, 2017.
Alexandre Boutet, Radhika Madhavan, Gavin JB Elias, Suresh E Joel, Robert Gramer, Manish Ranjan, Vijayashankar Paramanandam, David Xu, Jurgen Germann, Aaron Loh, et al. Predicting optimal deep brain stimulation parameters for parkinsons disease using functional mri and machine learning. Nature communications, 12(1):1-13, 2021.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang, Unlabeled data improves adversarial robustness. Advances in Neural Information Processing Systems, 32, 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SigKDD International Conference on Knowledge Discovery and Data Mining, pp- 785-794. ACM, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020. ]
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural
information processing systems, 26:2292-2300, 2013. 2, 6, 15, 16
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does it matter? Structural safety,
31(2):105-112, 2009. 3, 13
Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. Computational optimal transport: Complexity by accelerated gradient descent better than by sinkhorn's algorithm. International conference on machine learning, pp. 1367-1376. PMLR, 2018. 15
Andre Esteva, Katherine Chou, Serena Yeung. Nikhil Naik, Ali Madani, Ali Mottaghi, Yun Liu, Eric Topol, Jeff Dean, and Richard Socher. Deep learning-enabled medical computer vision. NPJ digital medicine, 4(1):1 1, 2021.
Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein: asymptotic and gradient properties. In AISTATS 2020-23nd International Conference Artificial Intelligence and Statistics, volume 108, pp. 1-20, 2020. 6
Ronald Fisher. Statistical methods and scientific induction. Journal of the Royal Statistical Society:
Series (Methodological), 17(1):69-78, 1955. 5
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model
uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016. 5
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances
in neural information processing systems, 17, 2004. 5
Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. Advances in Neural Information Processing Systems, 34, 2021.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning:
A tutorial introduction. 2019. 4,
10

---

Under review as a conference paper at ICLR 2023
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke. Dynamic distillation network for cross-domain few-shot recognition with unlabeled data. Advances in Neural Information Processing Systems, 34:3584-3595, 2021.
Ashish Kapoor, Raajay Viswanathan, and Prateek Jain. Multilabel classification using bayesian compressed sensing. In Advances in Neural Information Processing Systems, pp. 2645-2653, 2012.
Jacob Devlin Ming-Woi Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-IILT, pp. 4171- 4186, 2019.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in Neural Information Processing Systems, 34, 2021.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations (ICLR), 2018.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896. 2013. 2,
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 312-321, 2019.
Maxwell W Libbrecht and William Stafford Noble. Machine learning applications in genetics and genomics. Nature Reviews Genetics, 16(6):321-33,, 2015.
Andrey Malinin. Uncertainty estimation in deep learning with application to spoken language assessment. PhD thesis, University of Cambridge, 2019. 18
Andrey Malinin, Liudmila Prokhorenkova, and Aleksei Ustimenko. Uncertainty in gradient boosting via ensembles. In International Conference on Learning Representations, 2020. 5, 18
Jerzy Neyman and Egon Sharpe Pearson. Ix. on the problem of the most efficient tests of statistical
hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Muthematical or Physical Character, 231 2316699776) 1933. 5
V Nguyen, SB Orbell, Dominic T Lennon, Hyungil Moon, Florian Vigneau, Leon C Camenzind, Liuqi Yu, Dominik M Zumbühl, G Andrew D Briggs, Michael A Osborne, et al. Deep reinforcement learning for efficient measurement of quantum devices. npj Quantum Information, 7(1):1-9, 2021.
Vu Nguyen, Sunil Gupta, Santu Rana, Cheng Li, and Svetha Venkatesh. A Bayesian nonparametric approach for multi-label classification. In Proceedings of The 8th Asian Conference on Machine Learning (ACML), pp. 254-269, 2016. 7
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Advances neural information processing systems, 31, 2018. 9
Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation- based data augmentation for semi-supereised learning. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1369-1378, 2021.
Samet Oymak and Talha Cihad Gulcu. A theoretical characterization of semi- eemi-supervised learning with self-training for gaussian mixture models. In International Conference on Artificial Intelligence
and Statistics, pp. 3601-3609. PMLR, 2021. 1
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018. 9
11

---

Under review as a conference paper at ICLR 2023
Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-labed classification. Machine learning, 85(3):333-355) 2011. 7
Zhongzheng Ren, Raymond Yeh, and Alexander Schwing. Not all unlabeled data equal: Learning to weight data in semi-supervised learning. Advances in Neural Information Processing Systems, 33:21786-21797, 2020.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-avee awerre
aware pseudo-label selection framework for sumi-super learning. In Internationaa Conference on Learning Representations, 2021. 2, 5, 7, 18
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information
Fusion, 81 81:84-90, 2022. 7
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chum-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
1,2, 9
Kai Sheng Tai, Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning, pp. 10065-10075. PMLR, 2021. 2, 6, 7, 8, 15, 21, 22
Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Zidek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, et al. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590-599) 596, 2021.
Nienke ER van Bueren, Thomas L Reed, Vu Nguyen, James G Sheffield, Sanne HG van der Ven, Michael A Osborne, Evelyn H Kroesbergen, and Roi Cohen Kadosh. Personalized brain stimulation for effective neurointervention across participants. PLoS compulational biology, 17(9): 2021.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Nina M van Esbroeck, Dominic T Lennon, Hyungil Moon, Vu Nguyen, Florian Vigneau, Leon C Camenzind, Liuqi Yu, Dominik M Zumbühl, G Andrew D Briggs, Dino Sejdinovic, et al. Quantum device fine-tuning using unsupervised embedding learning. New Journal of Physics, 22(9):095003, 2020.
Journal
Bernard Welch. The generalization of student problem when several different population varlances are involved. Biometrika, 34(1-2):28 35, 1947. 5
Xi-Zhu Wu and Zhi-Hua Zhou A unified view of multi-label performance measures. In International Conference on Machine Leurning, pp. 3780-3788. PMLR, 2017. 7
Abraham J Wyner, Matthew Olson, Justin Bleich, and David Mease. Explaining the success of adaboost and random forests interpolating classifiers. The Journal of Machine Learning Research, 10011558-11990 2017. 9
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. In NeurlPS, 2020. 3, 4, 13, 14
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. Advances in Neural Information Processing Systems, 33:1 1033-11043, 2020. 2, 7,
Sora Yoon, Aditi Chandra, and Golnaz Vahedi. Stripenn detects architectural stripes from chromatin conformation data using computer vision. Nature Communications, 13(1):1-14, 2022.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pscudo labeling.
Advances in Neural Information Processing Systems, 34, 2021. 7,9
Zhi-Hua Zhou. Semi-supervised learning. In Machine Learning, pp. 315-341, Springer, 2021. 1
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
12


Reviews:
[{"review_id": "xqPSsamaj2", "reviewer": "Reviewer_hVwg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel problem for pseudo-labeling based semi-supervised learning. Especially, the authors proposed Confident Sinkhorn Allocation (CSA), which assigns labels to only examples with high confidence scores and learns the best label allocation via optimal transport.", "summary_of_the_review": "I like the solid theoretical analyses of this paper, although the experiments are comparatively weak. I would temporarily give a positive mark for this paper. ", "strength_and_weaknesses": "(+) The paper is generally well-written.\n\n(+) The theoretical analyses of this paper are solid. \n\n(+) The experiments are sufficient.\n\n\n(-) I feel that the proposed method is computational expensive. In each iteration, M models should be trained on labeled data, and an inner loop exists for Sinkhorn’s algorithm. Therefore, I want to see the total running time comparison of the proposed CSA and baseline methods. Besides, how about the scalability of the proposed method?\n\n(-) Also regarding the model assembling, I doubt that the performance improvement of CSA over other comparators in Table 2 is due to this operation. Moreover, I found that the performance improvement on various dataset is quite marginal. Therefore, I want to know whether the comparison is fair, and whether the comparison demonstrates the effectiveness of the proposed way for calculating the uncertainty. Maybe, the authors should conduct statistical significance test to justify the superiority. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "o1o_rVOCrW", "reviewer": "Reviewer_MB4o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an improved pseudo-labeling method for semi-supervised learning methods. The paper first proposes a method for filtering out some unlabeled data that has high uncertainty based on  Welch’s T-test and  Total variance.  The paper then extends the methods of SLA [1] to assign labels for unlabeled data.\n\n\n", "summary_of_the_review": "This paper proposes an improved pseudo-labeling method for semi-supervised learning methods, but the novelty of the proposed methods is limited.  ", "strength_and_weaknesses": "**Strengths**:\n\n* The paper proposes a method that does not need a predefined threshold to filter the unlabeled data\n\n\n**Weakness**:\n\n* The proposed method has limited novelty. First, considering uncertainty in pseudo-labeling is not new, e.g. [2] considered this scheme. Second, using optimal transport is not new as well. It is proposed in SLA [1].\n\n\n* The proposed method does not need a predefined threshold to filter the unlabeled data, but the method needs additional hyper-parameters, such as fraction of assigned label $\\rho$, which counteract the benefit of the proposed methods.\n\n\n* The proposed methods need additional computation cost due to optimal transport calculation and multiple model training on the labeled data.  \n\n\nReferences:\n\n[1] Sinkhorn label allocation: Semi-supervised classification via annealed self-training. ICML.2021\n\n[2] In defense of pseudo labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. ICLR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f7aF9BI6zQ", "reviewer": "Reviewer_AnF4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to improve the well-known pseudo-labeling by Confident Sinkhorn Allocation (CSA) which confidently assigns the labels via optimal transport. The authors demonstrated CSA’s performance by comparing it with other SSL strategies.", "summary_of_the_review": "I have listed my concerns about this work above and I am expecting the authors' response.", "strength_and_weaknesses": "Strength:\n1. CSA first conducts sample selection by removing low-confidence samples and then focuses on assigning labels. The overall framework is convincing and easy to follow. \n2. The variance of correct assignment, i.e., epistemic uncertainty, is considered for evaluating the uncertainty, which is an interesting idea.\n\nWeaknesses:\n1. The so-called domain-specific and pseudo-labeling has been attributed to two separate components which were thought based on the different designing principles. Please refer to [1].\n2. I understand that authors would like to show correct assignments affect the optimal decision boundary by theorem 1. As it extends Yang&Xu’s theory, I am curious if it gives a better theoretical conclusion, e.g., closer to the decision boundary.\n3. The discussion about the most provable class and second-most probable class is related to the theorem presented in the work [1], because both considered the relative relation between outputs.\n4. T-test for sample selection is based on training M models, which incurs much computation cost. Note that the dropout used in Rizves et al is only involving inference, but CSA is with an ensemble style. An explanation about it should be added.\n5. Another concern is that T-test is used for sample selection, but it has not used epistemic uncertainties, i.e., the variance of I. Notice that total variance is designed for multilabel classification, is it feasible for multiclass semi-supervised classification?\n6. The contribution compared with Tai et al from the label assignment section is not very convincing to me. Does CSA outperform theirs by T-test and some additional constraints over class population size?\n\n[1] Taming Overconfident Prediction on Unlabeled Data from Hindsight, https://arxiv.org/abs/2112.08200\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject