Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
Authors: Harsh Rangwani
Shrinivas Ramasubramanian
Sho Takemori
Kato Takashi
Yuhei Umeda
Venkatesh Babu Radhakrishnan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Semi-Supervised Learning
Long-Tailed Learning
Non-Decomposable Objectives

Abstract: Self-training based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only a fraction of labeled data. However, the majority of work on self-training has focused on the objective of improving accuracy whereas practical machine learning systems can have complex goals (e.g. maximizing the minimum of recall across classes, etc.) that are non-decomposable in nature. In this work, we introduce the Cost-Sensitive Self-Training (CSST) framework which generalizes the self-training-based methods for optimizing non-decomposable metrics. We prove that our framework can better optimize the desired non-decomposable metric utilizing unlabeled data, under similar data distribution assumptions made for the analysis of self-training.  Using the proposed CSST framework, we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks.  Our results demonstrate that CSST achieves an improvement over the state-of-the-art in majority of the cases across datasets and objectives.

Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
Harsh Rangwani", Shrinivas Ramasubramanian" Sho Takemori Takashi Kato", Yuhei Umeda", and R. Venkatesh Babu'
Video Analytics Lab, Indian Institute of Science, Bengaluru, India FFujitsu Limited, Kanagawa, Japan harshr@iisc ac. in, shrinivas mannasubrrmmaninn@@maaa com, takemori sww.ooiisssofuuitsuuuu.uuu com, kato. bakashi_O1@fu] com, umeda. wwwhhi@fujitssu.omm com, venkydiisc. ac. in
Abstract
Self-training based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only fraction of labeled data. However, the majority of work on self-training has focused on the objective of improving accuracy whereas practical machine learning systems plex goals (c.g. maximizing the minimum of recall across classes etc.) that are non-decomposable in nature. In this work, introduce the Cost-Sensitive Self- Training (CSST) framework which generalizes the self-training based methods for optimizing non-decomposable metrics. We prove that our framework can better optimize the desired non non-decomposable metric utilizing unlabeled data, under similar data distribution assumptions made for the analysis of self self-training. Using the proposed CSST framework we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks. Our results demonstrate that CSST achieves an improvement over the state-of-the in majority of the cases across datasets and objectives.
have com-
Introduction
In recent years, semi-supervisee learning algorithms are increasingly being used for training deep neural networks [3, 9, 31, 37]. These algorithms lead to accurate models by leveraging the unlabeled data in addition to the limited labeled data present. For example, it's possible to obtain model with minimal accuracy degradation (< 1%) using 5% of labeled data with semi-supervised algorithms compared to supervised models trained using 100% labeled data [31]. Hence, the development of vast reduction in the requirement for expensive labeled data.
these algorithms has resulted
Self-training is one of the major paradigms for semi-supervised learning. It involves obtaining targets (e.g. pscudo-labels) from network from the unlabeled data, and using them to train the network further. The modern self-training methods also utilize additional regularizers that enforce prediction consistency across input transformations (e.g., adversarial perturbations [18], augmentations [36, 3 etc.) enabling them to achieve high performance using only tiny fraction of labeled data. Currently, the enhanced variants of self eeff-training with consistency regularization [40, 25] are among the state-of- the-art (SOTA) methods for semi- cuni-supereiee learning.
Equal Contribution. Code: http:://ttha..coooaaaicccoooaaas
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Optimizing Min Recall for CIFAR10-LT
Figure 1: We show a comparison of the SOTA CSL [20] method with the Self-training-bart based Semi-Supervised methods, for optimizing the minimum recall objective on the CIFARIO-LT dataset. Our proposed CSST framework pro- duces significant gains the desired metric leveraging additional unlabeled data through our proposed weighted novel consistency reg. ularizer and thresholding mechanism.
Superstsed
CSL (SOTA)
Semi Supervised
Method
Self- Seif-Training
Sensitive Belf-Training (CSST)
Minimum Recall Across Classes
Despite the popularity of self-training methods, most of the works [36, 1, 31] have focused on the objective of improving prediction accuracy. However, there nuanced objectives in real-world based on the application requirements. Examples include minimizing the worst-case recall [19] used for federated learning, classifier coverage for minority classes for ensuring fairness [6], etc. These objectives are complex and cannot be expressed just by using a loss function on the prediction of single input (i.e., non-decompooabbee.. There has been considerable effort in optimizing non- decomposable objectives for different supervised machine learning models [20, 30]. However, as supervision can be expensive, in this work aim answer the question: Can we optimize non- decomposable objectives using self-training-baat baasedd methods developed for semi-supervised learning?
We first demonstrate that vanilla self olfftrraiingg methods (e.g., FixMatch [31], UDA [36], etc.) can produce unsatisfactory results for non-decomposable metrics (Fig. 1). We then generalize the Cost- Sensitive Loss for Self- Teainiig by introducing novel weighted consistency regularizer, for partic- ular non-decomposable metric. Further, for training neural networks we introduce appropriate loss functions and pseudo label selection (thresholding) mechanisms considering the non-decomposable metric we aim to optimize. We also prove that we can achieve better performance on desired non- decomposable metric through our framework utilizing self-training, under similar assumptions on data distributions as made for theoretical analysis of self-training [35]. We demonstrate the practical application by optimizing various non-decomposable metrics by plugging existing methods (e.g. FixMatch [31] etc.) into our framework. Our framework leads to a significant average improvement in desired metric of minimizing worst-case recall while maintaining similar accuracy (Fig. 1).
In summary: a) we introduce a Cost-Sensitive SelfnTrannng (CSST) framework for optimizing non- decomposable metries that utilizes unlabeled data in addition to labeled data. (Sec. 4) b) we provably demonstrate that our CSST framework can leverage unlabeled data to achieve better performance baseline desired non-decomposable metric (Sec. 3)
show that by combining CSST with self-training frameworks (e.g. FixMatch [31], UDA [36] etc.) leads to effective optimization of non-decomposable metrics, resulting in significant improvement over vanilla baselines. (Sec. 5)
Preliminaries
2.1 Non-Decomposable Objectives and Reduction to Cost-Sensitive Learning
Table 1: Metrics defined using entries of a confusion matrix.
We consider the K-class classification problem with an instance space X' and the set of labels
[K].
The data distribution on X' x [K] is denoted by D. For i E [K], we denote by n; the class prior P(y = i). Notations commonly used across paper are in Table
Metric
Definition
Recall (rec,[F))
C.d
Cis(
1 present in Appendix. For classifier F X K]. we define confusion matrix C(F) E R by C. E (z,))>> (1 = i, F(x) = j)]. Many met- rics relevant to classification can be defined as functions of entries confusion matrices such as class-coverage, recall and accuracy to name few. We introduce more complex metries, which are of practical importance in the case of imbalanced distributions [4] (Tab. 1).
Coverage (cov, (Fov;[F))
E, Fy.lF)
Cral
Precision (prec,[F))
Σx Ce,(FF
C..s(P]
Worst Case Recall
min; >, (F.
Accuracy
Σ, C.,

---

A classifier often tends to suffer low recalls on tail (minority) classes in such cases. Therefore, one may want to maximize the worst case recall, min rec:[F]. F [E[
Similarly, on long-tailed datasets, the tail classes suffer from low coverage, lower than their respective priors. An interesting objective in such circumstances is to maximise the mean recall, subject to the coverage being within given margin.
0.95 S.L. [P] 2 [Fc
max F
Σ KE(K]
vj E [K].
(1)
K
K
Many of these metrics are non-decomposable, i.e., one cannot compute these metrics by simply calculating the average of scores on individual examples. Optimizing for these metrics can be regarded as instances of cost-sensitive learning (CSL). More specifically, optimization problems of the form which can be written as a linear combination of Gi.j and Cigy (F) will be our focus in this work where G is K x K matrix.
Σ C.C.(PP,, [EE[K]
The entry Giy represents the reward associated with predicting class when the true class is i. The matrix G is called a gain matrix [20]. Some
complex non-decomposable objectives for classification can be reduced to CSL [22, 32, 20]. For instance, the aforementioned two complex objectives can be reduced to CSL using continuous relaxation Lagrange multiplier as bellow. Let AK-1 C Rx be the K - 1-dimensional probability simplex. Then, maximizing the minimum recall equivalent the saddle-point optimization problem:
max min AEAR
Σ AE[R]
A
(3)
Thus, for a fixed X, the corresponding gain matrix is given as a diagonal matrix with G2 \// m, for 1 < i < K. Similarly, using Lagrange multipliers À E R Eq. (1) is rewritten as a max-min optimization problem [20, Sec. 2]:
Gk)
max min XERR
EN, Σ CalFI/T++
Σ --114-00 -095//1 (E[K]
*E[K]
SE[K]
5yg In this case, the corresponding gain matrix G is given as Giz= k= + +)g, where dig is the Kronecker's delta. One can solve these max-min problems by alternatingly updating À (using exponented gradient or projected gradient descent) and optimizing the cost-sensitive objectives [20].
2.2 Loss Functions for Non-Decomposable Objectives
The cross entropy loss function is appropriate for optimizing accuracy for deep neural networks, however, learning with CE can suffer low performance for cost-sensitive objectives [20]. Following [20], we introduce calibrated loss functions for given gain matrix G. We let pm - AK-1 R be prediction function of model, where AK-1 the K 1-dimensional probability simplex. For gain matrix G. the corresponding loss function is given as a combination of logit adjustment [17] and loss re-weighting [24]. We decompose the gain matrix G as G MD, where D (Giag(Gn1,... ( be a diagonal matrix, with D > 0, Vi E [K] and M E k For y E [K] and model prediction pm (I), the hybrid loss is defined as follows:
(pm(x));/Da
obyb (y. P.. (x)) = Σ Mgi log
(5)
ΣSE[K] (pm(z)), /D
SE[K]
To make the dependence of G explicit, we also denote ghyb Pm (x)) as ghyb (y.Pm Pm (x): G). The average loss on training sample 5 c X' is defined as clyb(xx) = [ST Exes phyb (1, Pm (x)).
(u.Pmn Narasimhan and Menon [20] proved that the hybrid loss calibrated, that is learning with ghyb gives the Bayes optimal classifier for G (c.f., [20, Proposition 4], of which we provide formal statement in Sec. 1. If G is a diagonal matrix (i.e., M 1x), the hybrid loss is called the logit adjusted (LA) loss and ghyb (y. Pm (z)) is denoted by GLA (y, Pm (x)).
3

---

2.3 Consistency Regularizer for Semi-Supervised Learning
Modern self-training methods not only leverage pseudo labels, but also forces consistent predictions of a classifier on augmented examples or neighbor examples [35, 18, 36, 31]. More formally, classifier F is trained so that the consistent regularizer R(F) small while supervised loss or a loss between pseudo labeler are minimized [35, 31]. Here the consistency regularizer R( ()) is defined as
Eg (F(()) + F( 3z' s.t. x' is neighbor of an augmentation of x)].
In existing works, consistency regularizers are considered for optimization of accuracy. In the subsequent sections, consider consistency regularizers for cost-sensitive objectives.
3 Cost-Sensitive Self-Training for Non-Decomposable Metrics
3.1 CSL and Weighted Error
In the case of accuracy O- --error, a self-training based SSL. algorithm using a consistency regularizer achieves the state-of-the-arr performance across variety of datasets [31] and its effectiveness has been proved theoretically [35]. This section provides theoretical analysis of self-training based SSL algorithm for non-decomposable objectives by generalizing [35]. More precisely, the main result of this section (Theorem 5) states that an SSL method using consistency regularizer improves a given pseudo labeler for non-decomposable objectives. We provide all the omitted proofss in Appendix for theoretical results in the paper.
In Sec. 2, we considered non- non-decomposable metrics and their reduction to cost-sensitive learning objectives defined by Eq. (2) using a gain matrix. In this section. we consider an equivalent objective using the notion of weighted error. For weight matrix w (way)s and classifier F:X- [K]. weighted error is defined as follows:
E = Σ WigEaR [1F(() 5)),
[JE(K]
where, Pz(x)) denotes the class conditional distribution P(x y 1). If
diag(1/K: K 1/K), then this coincides with the usual balanced error [17]. Since Ci,[F] =
(( [1 (() = 5)] (vy = i) Pyy Ee (x(()) + j)], we can write:
GiyC,, (Py = Pyy = i)E (F(()) # 5)) = Giy( -, E. (1(F()) fi)
for 1 S S K as before. Hence CSL objec- tive (2) ie. max En, Gij CiFFF is equivalent to minimizing the negative term above i.e. Eep,, [I(F(x) j)l which is same Err,, (F) with wi; Giym, for S i,j < K. Hence, the notion of weighted error is equivalent to CSL, which we will also use later for deriving loss functions. We further note that if we add matrix with the same columns (c1 > 0) to the gain matrix G. still the maximizers of CSL (2) are the same as the original problem. Hence, without loss of generality, assume Way 2 O. We assume + 0, i... |w/1
Here T, is the class prior Py
for avoiding degenerate solutions.
In the previous work [35], it is assumed that there exists ground truth classifier F* X + [K] and the supports of distributions {PihigiH are disjoint. However, if supports of distributions (Pi) are disjoint, a solution of the minimization problem miny Err,,(F) independent of w in some cases. More precisely, if w diag(w1,... wx) i.e. a diagonal matrix and >> O, Vi, then the optimal classifier is given as x + [=[K] wy Px(x) (this follows from [20, Proposition 1J). If supports are disjoint, then the optimal classifier the same as x + argmaжжеΙк] Ps (æ), which coincides with the ground truth classifier. Therefore, we not assume the supports of P, are disjoint nor ground truth classifier exists unlike [35]. See Fig. 2 for an intuitive explanation.
3.2 Weighted Consistency Regularizer
For improving accuracy, the consistency in prediction equally important across the distributions P: for 1 < < K [31, 35]. However, for our case of weighted error, if the entries of the ith row of the weight matrix w are larger than the other entries for some ig, then the consistency of model predictions on examples drawn from the distribution P., are more important than those the other

---

wy) (1)
[b] w:)
(wi, (1, 10)
(w.. wy]= (10.
w:P(00
W2P100)
opt. decision boundary
Using simple example, we explain a difference in theoretical assumptions compared to [35] that assumes {PitisieK have disjoint supports. Here, we consider the case when K = 2. = diag(w1, (w))). and P1 and P2 are distributions on X' RR. (a), (b): perfect setting where two distributions Pi and P3 have disjoint supports, the Bayes optimal classifier for the CSL is identical to the ground truth classifier (r + argmax, (x))) for any choices of weights (w1, W2). (c), (d): In more generalized setting, the Bayes optimal classifier + argmax, wx):(z) depends on the choice of weights (i.e., gain matrix). The optimal decision boundary (the green line) for the CSL moves to the right as w1/ W2 increases.
Figure 2:
W2/
examples. In this case, we require more restrictive consistency regularizer for distribution for Pio- Thus, we require a weighted (or cost-sensitive) consistency regularizer, which we define below.
We assume that the instance space X' is normed vector space with norm |. and T is set of augmentations, i.e., each T E T is a map from X to itself. For fixed r > O, we define (xx) by {a' E X' 3T E s.t. |x' T(x) < r). For each i [K]. define conditional consistency regularizer by (F)) = Eg-P, (3x' (3x E B(x) s.t. F(x) & F(x')). Then, we define the weighted ΣiJEKK] Way (F). If the prediction of classifier F is robust augmentation T E and small noise, then RB, FF small. For 3 > 0, we consider the following optimization objective for finding classifier F:
consistency regularizer by R8,
min Err, (F) subject to R8 (F) S B.
A solution of the problem (6) is denoted by F*. We let Fpt: X [K] pseudo labeler (a classifier) with reasonable performance (we elaborate on this in Section 3.4). The following mathematically informal assumption below is required to interpret our main theorem.
ASSUMPTION 1. We assume both 3 and Errw (Fr (F*) are sufficiently small so that they are negligible compared to (Fp)).
Assumption assumes existence of an optimal classifier F* that minimizes the error Err, (i.e. Bayes Optimal) among the class of classifiers which are robust data augmentation (i.e. low weighted consistency R. (F)) (F)). As we operate in case of overparameterized neural networks such a classifier fs is bound to exist, but is unknown in our problem setup. In the case of the balanced error, the validity of this assumption is justified by the fact that the existing work [31] using consistency regularizer on data augmentation obtains classifier F' that achieves high accuracy (i.e., low balanced errors) for balanced datasets. Also in Appendix C.I, we provide an example that supports the validity of the assumption in the case of Gaussian mixtures and diagonal weight matrices.
3.3 Expansion Property
For x E X. we define the neighborhood (²x) of z by {a' e X: ((z') & 0}. For a subset S C X, neighborhood of S is defined (S) UzesN(x). Similarly to [35], consider the following property on distributions.
DEFINITION 2. Let (0, + [1, 2o) be a non-increasing function. For a distribution Q on X' we say Q has c-expansion property if (S)(S) 2 c(Q(S))Q(S)) for any measurable $ C X.
The c-expansion property implies that if Q(S) decreases, then the "expansion factor" Q(N (S) (S)) increases. This is a natural condition, because it roughly requires that if Q(S) is small, then QN large compared to Q(S). For intuition let us consider a ball of radius depicting S R with volume Q(S) and its neighborhood (SS) expands to ball with radius 1 + 1. The expansion factor here would be ((1 1)/1)á, hence as 1 (i.e. Q(S)) increases (1 +1+1 1/1)d (i.e. Q(V(S))Q((S) decreases. Hence, it's natural to expect e to be a non-decreasing function. The c-expansion property
(on each P.) considered here is equivalent to the (a, (), -xexanssoo property, which is shown to be
5

---

realistic for vision and used for theoretical analysis of self self-training in [35], where E (0. 1) and č > (see Sec. N.2 in Appendix). In addition, we also show that is also satisfied for mixtures of Gaussians and mixtures of manifolds (see Example in Appendix for more details). Thus, the c-expansion property a general property satisfied for a wide class of distributions.
ASSUMPTION 3. For weighted probability measure Pw on X by (u)(U) Eiserko "4P for 21.jeik] Miy
U C X. We assume Pw, satisfies c-expansion for a non-increasing function e: (0, (0)
3.4 Cost-Sensitive Self-Training with Weighted Consistency Regularizer
In this section we first introduce the assumptions on the pseudo labeler FpI and then introduce the theoretical Cost-Sensitive Self-Training (CSST) objective. Pp can be any classifier satisfying the following assumption, however, typically it is a classifier trained on a labeled dataset.
ASSUMPTION 4. We assume that Err (Pp)) + Err, (F*) S |w/1- Let y c(pw), where Pic = Brr. ( 1u/1
We also assume 7 > 3.
Since is non-increasing, 7 (as function of Err (Fp:)) is a non-increasing function of Frr (Pp) (and (f",(F")). Therefore, the assumption y > 3 roughly requires that ErT (Fpt) is "small". We provide concrete conditions for ErT., (Fpi) that satisfy y > 3 in the case of mixture of isotropic d-dimenional Gaussians for a region B(x) defined by in Appendix (Example 2). In the example, we show that the condition y > 3 is satisfied Err., (Fp)) < 0.17 the case when r = 1/(2v and satisfied if Err (Pp)) < 0.33 in the case when r= 3/(2v/) d), where X' S RR. Since we assume Err (F*)) negligible compared Err,, (F)t) (Assumption I), the former condition in Assumption 4 is approximately equivalent to Errw (rrw(FF)) S |w]1 which is satisfied by the definition of Errw-
We define [ (F, F*) = EEWP, [I(F(x) # F' (x))] Then, consider the following objective:
7 ++
2
min (w) where Lw
7-1 Lo (F.
7-1 (F).
Here (F, Fpt) is defined as EijeiK] Wey ma.--g.1 sensitive self- self-training (with Fpi) objective with weighted consistency regularization. We provide following theorem which relates the weighted error of classifier F learnt using the above objective to the weighted error of the pseudo labeler (Fpi).
(1) (F, Fpi). The above objective corresponds cost-
THEOREM 5. Any learnt classifier F using the loss function Lw (i.e. argming Lu (F)) satisfies:
Err. 7- Errw Y + 1
2
4y 7 7-1 (B,.((("))
Err, (F*)
REMARK. Since both Brr.:(FF) and Rg (F*) S B are negligible compared to Errw (Fii) and 7 > 3, Theorem 5 asserts that f learnt by minimizing semi-supervised loss L (F, Fp1) with the consistency regularizer Rg (F) can achieve superior performance than the pseudo labeler Fo in terms of the weighted error Err,.. The above theorem is generalization of [35, Theorem 4.3), which provided similar result for balanced 0-1-error in the case of distributions with disjoint supports. In Appendix Sec. D. following [35, 34]. we also provide generalization bound for Err. (F) using all-layer margin [34] in the case when classifiers neural networks.
4 CSST in Practice
In the previous section, we proved that by using self-training (CSST), we can achieve a superior classifier F in comparison to pseudo labeler Fp through weighted consistency regularization. As we have established the equivalence of the weighted error Err.. to the CSL objective expressed in terms of G (Sec. 3.1), we can theoretically optimize a given -on-decomposable metric expressed by G better using CSST, utilizing the additional unlabeled data via self-training and weighted consistency regularization. We now show how CSST can be used in practice for optimizing non- deccomposabll metrics in the case of neural networks.
The practical self self-training methods utilizing consistency regularization (e.g., FixMatch [31], etc.) for semi-supervised learning have supervised loss Cg for labeled and consistency regularization loss for

---

unlabeled samples (i.e., Lw) with a thresholding mechanism to select unlabeled samples. The final loss for training the network is L, + >n,Cu, where A, is the hyperparameter. The supervised loss fas can be modified conveniently based on the desired non-decomposable metric by using suitable G (Sec. 2.1). We will now introduce the novel weighted consistency loss and its corresponding thresholding mechanism for unlabeled data in CSST, used for optimizing desired non-decomposable metric.
Weighted Consistency Regularization. As the idea of consistency regularization is to enforce consistency between model prediction different augmentations of input, this is usually achieved by minimizing some kind of divergence D. A lot of recent works [18, 31, 36] in semi-supervised learning use DKL to enforce consistency between the model's prediction on unlabeled data and its augmentations, Pm (z) and pm (A(z)). ). Here A usually denotes form of strong augmentation. Across these works, the distribution of confidence of the model's prediction is either sharpened or used to get a hard pseudo label to obtain (1). As we aim to optimized the cost-sensitive learning objective, we aim match the distribution of normalized distribution (i.e. norm G fm(()) (emm(G"mmmm)) = GT Pm (x)) Ei (GT (mTT)(()))) (=)):) (Proposition 2 [20] also in Prop. 7) ) with the pm (A(xr) by minimizing the Kiv-Divereence between these. We now propose to use the following weighted consistency regularizer loss function for optimizing the same:
K
(GTTm par (m G) = Σ (GT pmn (z)( log( (og()m(A(())))) Pm
(8)
1m
PROPOSITION 6. The minimizer of Du: mization of KL. Divergence i.e. DKL P (G"T pm (2)()))))p (x)))l Vx E
B..T Emen, B. e (fm Pm (A(x)) G) leads to mini-
As the above loss is similar in nature to the cost sensitive losses introduced by Narasimhan and Menon [20] (Sec. 2.1) we can use the logit-adjusted variants (i.e. gLA and ghyl based type of G) of these in our final loss formulations (C for training overparameterized deep networks. We further show in Appendix Sec. B that the above loss fxt approximately minimizes the theoretical weighted consistency regularization term RB, (F)) defined in Sec. 3.4.
Threshold Mechanism for CSST. In the usual semi-supervised learning formulation [31] we use the confidence threshold (Pax,(m (Ir)a) > 7) as the function to select samples for which consistency regularization term is non-zero. We find that this leads to sub-optimal results in particularly the case of non-diagonal G as only few samples cross the threshold (Fig. 4). As in the case of cost-sensitive loss formulation the samples may not achieve the high confidence to cross the threshold of consistency regularization. This is also theoretically justified by the following proposition:
PROPOSITION 7 ([20) Proposition 2). Let (x))( be the optimal softmax model function obtained by optimizing the cost-sensitive objective in Eq. (2) by averaging weighted loss function gat (y. Pen (x))
K Gyys log Σ, Pm (x))
(pmi()))) Then optimal po (x)) is: yopt (x) Σ, Gv.3
G G.d
= norm GT (x,
Here y is the one-hot representation vector for a label y. This proposition demonstrates that for particular sample the high confidence Pm. may not be optimal based on G. We now propose our novel way of thresholding samples for which consistency regularization is applied in CSST. Our thresholding method takes into account the objective of optimizing the non-decomposable metric by taking G into account. We propose to use the threshold on KL-Divergence of the softmax of the sample pmn (2x) with the optimal softmax (i.e. norm GT pm (z))) for given G corresponding the pseudo label (or sharpened) Pm (I), using which we modify the consistency regularization loss term:
1
(yr(
Σ Z.B..
L(D (norm (m. |I Pm (z)) (pm (x), Pm (A(x)), G)
(9)
Bul
We name this proposed combination of KL- KL-Thresholding and weighted consistency regularization as CSST in our experimental results. We find that for non-diagonal gain matrix G the proposed thresh- olding plays a major role in improving performance over supervised learning. This is demonstrated by comparison of CSST and CSST w/o <L-Thheshooding (without proposed thresholding mechanism) in Fig. 4 and Tab. 3. We will now empirically incorporate CSST by introducing consistency based losses and thresholding mechanism for unlabeled data, into the popular semi-supervised methods of FixMatch [31] and Unsupervised Data Augmentation for Consistency Training (UDA) [36]. The exact expression for the weighted consistency losses utilized for UDA and FixMatch have been provided in the Appendix Sec. H.2 and H.3.
7

---

Table 2: Results of maximizing the worst-case recall over all classes
CIFAR-10 Long Tall (g-108, gm4)
(col 2-3) and over just the head and tail classes (col 4-7).
5000
Method
CIFARIO-LT
[o - 100)
CIEARIO--L (p = 10)
Imagenet(00-1
L00--TT Go
10)
3090
Avg. Rec
Min. Rec
Avg. Rec
Min. HT Rec
Avg. Rec
Min HT Rec
ERM
0.26
0.14
0.30
7003
CSL
LA
0.51
0.64
0.38
0.57
0.43
0.36
0.41
0.35
0.48
0.52
0.52
0.47
1000
(HxMarch)
Vanilla
0.78
0.48
0.63
0.36
0.49
0
(FixMarch)
CEST
0.76
0.72
0.63
0.61
0.64
0.63
Clocs irdec
Figure 3: CIFAR-I0 Long tail distribution P 100, μr = 4.
5 Experiments
We demonstrate that the proposed CSST framework shows significant gains in performance on both vision and NLP tasks on imbalanced datasets, with an imbalance ratio defined on the training set as max, Ply=i) min; P(y=i) We assume the labeled and unlabeled samples come from a similar data distribution and the unlabeled samples are much more abundant (μ times) the labeled. The frequency of samples follows an exponentially decaying long-tailed distributed as seen in Fig. 3, which closely imitates the distribution of real-world data [33, 10]. For CIFAR- 10 [11], IMDb [16] and DBpedia- 14 [14], we use P = 100 and p = 10 for CIFAR- 100 [11] and ImageNet- 100 [27] 2 datasets. We compare our method against supervised methods of ERM, Logit Adjustment (LA) [17] and Cost Sensitive Learning (CSL) [21] trained on the same number of labeled samples as used by semi-supervised learning methods, along with vanilla semi-supervised methods of FixMatch (for vision) and UDA (for NLP tasks). We use WideResNetst WRN) [39], specifically WRN-28-2 and WRN-28-8 for CIFAR-10 10 and CIFAR-100 respectively. For ImageNet, we use ResNet-50 [7] network for our experiments and finetuned (Es uncased) [29] for IMDb and DBpedia-14 14 datasets. We divide the balanced held-out set for each dataset equally into validation and test sets. A detailed list of typee-parameters and additional experiments can be found in the Appendix Tab, 4 and Sec. O respectively.
Maximizing Worst-Case Recall. For CIFAR-10, IMDb, and DBpedia-14 14 datasets, maximize the minimum recall among all classes (Eq. (3)). Given the low number of samples per class for datasets with larger number of classes like CIFAR-100 and ImageNet-1000 we pick objective (10). We define the head classes (H) and tail classes (T) as the first 90 classes and last 10 classes respectively. The Min. HT recall objective can be mathematically formulated as:
AM
CF[F]
AT
Ca[F]
max F
min (AAA
(10)
H
R
T
iET
The corresponding gain matrix Gs is diag(
Ay
Aw
AJ FK| Since G is diagonal
72 PE
#K-1
here, use (Sst((ixMatch) loss function Eq. (9) with the corresponding gwt being substituted by CLL as define in Sec. 2.1. Also for labeled samples we use CtA as G is diagonal, we then combine the loss and train network using SGD. Each few steps of SGD, were followed by an update on the X and G based on the uniform validation set (See Alg. 1 in Appendix). We find that CSst(FixMatch) significantly outperforms the other baselines in terms of the Min. recall and Min. Head-Tail recall for all datasets, the metrics which we aimed to optimize (Tab. 2), which shows effectiveness of CSST framework. Despite optimizing worst-case recall we find that our framework is still able to maintain reasonable average (Avg.) recall in comparison to baseline vani vamilla(FixMatch, which demonstrates it's practical applicability. We find that optimizing Min. recall across NI.P tasks of classification on long-tailed data by plugging UDA into CSST (UDA) framework shows similar improvement in performance (Tab. 4). This establishes the generality of our framework even self-training methods across domain of NLP as well. As the G is a diagonal matrix for this objective, the proposed KL-Based Thresholding here is equivalent to the confidence based threshold of FixMatch in this case. Despite the equivalence of thresholding mechanism, we see significant gains in min-recall (Tab. 2) just using the regularization term. We discuss their equivalance in Appendix Sec. I.
Maximizing Mean Recall Under Coverage Constraints. Maximizing mean recall under coverage constraints objective seeks result in a model with good average recall, yet at the same time
8

---

Table Results of maximizing the mean recall subject to coverage constraint all classes (col 2-3) and over the head and tail classes (col 4-7). Proposed CSST(FixMatch) approach compares favorably to ERM,LA, CSL vanilla(FixMatc) and CSST(FixMatch) w/o KL.-Thresh.. It the best at both maximizing mean recall and coming close to satisfying the coverage constraint.
Method
CIFARIO-LT
CIFAR100-LT
ImageNet100-4
Per-class Coverage
Head-Taie Coverage
Head- Tail Coverage
(p 100. igt: 0.1)
(p
10, lgt 0.01)
(p
10, tgt 0.01)
Avg. Rec
Min. Cov
Avg. Rec
Min. HT Cov
Avg. Rec
Min. HT Cov
ERM
0.32
0.034
0.36
0.004
0.40
0.006
LA
0,51
0.039
0,36
0.009
0.48
0.009
CSL
0.60
0.090
0.45
0.010
0.48
0.010
Vanilla (FixMatch)
0.78
0.055
0.63
0.004
0.58
0.007
(Sst(FixMatch) w/o
0.67
0.093
0.47
0.010
0.26
0.010
KL-Thresh.
(SST(FixMatch)
0.80
0.092
0.63
0.010
0.58
0.010
Table 4: Results of maximizing the min recall over all classes for classification on NLP datasets. Proposed CSST(UDA) approach outperforms ERM and vanilla baselines.
Method
IMDb (p
10)
IMDb (p-
100)
DBpedia-14 (p
100)
Avg Rec
Min Rec
Avg Rec
Min Rec
Avg Rec
Min Rec
ERM
0.79
0.61
0.50
0.00
0.95
0.58
vanilla (UDA)
0.82
0.66
0.50
0.00
0.96
0.65
CUST(UDA)
0.89
0.88
0.77
0.75
0.99
0.97
constraints the proportion of predictions for each class to be uniform across classes. The ideal target coverage under a balanced evaluation set(or such circumstances) given as cov:[ _े. V [K]. Along similar lines to objective (10) we modify the objective (4) to maximize the average recall subject to the both the average head and tail class coverage (HT Coverage) being above given 85 threshold 0.95 The G non-diagonal here and Gij = (I Au + 15ET ) Km. [
K
CH(FI Km;
C
0.95 K
Σ
C
0.95
max
min (Ax.AT)ERS
H
+XT
ITI
K
(E(K)
JE[K],JH
jE[K]JTT
(11)
As these objectives corresponds to non-diagonal G as shown in Eq. (4) in Sec. 2.1. Hence, for introducing CSST into FixMatch we replace first supervised loss L. with Clyb, For the unlabeled data introduce ghyb in L (Eq. (9)). Hence, the final objective L. is defined as, L Lhyb Awt We update the parameters of the cost-sensitive loss (G and X) periodically after few of SGD on the model parameters (Alg. in Appendix). In this case our proposed thresholding mechanism in cssT(FixMatth) introduced in Sec. 4, leads to effective utilization of unlabled data resulting in improved performance over the naive CSST(FixMatch) without (w/o) KL-Thresholding (Tab. 3). In these experiments, the mean recall of our proposed approach either improves or stays same to the vanil sanilla(FixMathh implementation but only ours is the one that comes close to satisfying the coverage constraint. We further note that among all the methods, the only methods that come close to satisfying the coverage constraints are the ones with ghyb included in them.
Ablation on amount unlabeled data. Here, we ablated the total number of labeled sampled while keeping the number of unlabeled samples constant. We observe (in Fig. 5a) that number of labeled samples the mean recall improved. This is because more labeled samples helps in better pseudo-labeling sample, the models errors on the pseudo-labels increase causing a reduction in mean recall. Hence, any addtional labeled data can be easily used improve CSST performance.
increase the
the unlabeled samples and similarly as decrease the number of labeled
blation on r threshold. Fig. 5b shows that when the KL divergence threshold too high, large number of samples with very low degree of distribution match are used for generating the sharpened target (or pseudo-labels), this leads worsening mean recall as many targets are incorrect. We find that keeping conservative of 0.3 works well across multiple experiments.

---

(201))
6.
Mean Recalll
0.00
Mean Recall
273
Trecken of
costiaeteet costralteet
2.11
1.0
176
00
25
3.0
35
4.0
4.5
5.0
02
04
08
0.8
122
KL Threshold
(b)
Figure 4: Fraction of unlabelled Figure 5: Maximizing average recall under coverage constraints data used for maximizing average for CIFAR-10 10 Long tail (p = 100) (Sec. 5). Fig. shows compar- recall under coverage constraints ison of (a) increasing the ratio of unlabeled samples to labeled for CIFAR-100 (p = 10, p 4) samples given fixed number of unlabeled samples (b) Ablation on KL diveregence based threshold for CSST(Fix.Math))
(Sec. 5).
6 Related Work
Self-Training Self-training algorithms have been popularly used for the tasks of semi-supervised
learning [1, 37, 31, 13] and unsupervised domain adaptation [28, 41]. In recent years several regularizers which enforce consistency in the neighborhood (either an adversarial perturbation [18] or augmentation [36]) of given sample have further enhanced the applicability and performance of self-training methods, when used in conjunction. However, these works have focused mostly on improving the generic metric of accuracy, unlike the general non-dccomposable metrics consider.
Cost-Sensitive Learning. refers to problem settings where the cost of error differs for sample based on what class it belongs to. These settings are very important for critical real world applications like disease diagnosis, wherein mistakenly classifying diseased person as healthy can be disastrous. There have been pletheora of techniques proposed for these which can be classified into: importance weighting [15, 38, 5] and adaptive margin [2, 38] based techniques. For overparameterized models Narasimhan et al. [22] show that loss weighting based techniques are ineffective and propose logit-adjustment based cost-sensitive loss which we also use in our framework.
Complex Metries for Deep Learning. There has been a prolonged effort on optimizing more complex metrics that take into account practical constraints [21, 26, 23]. However most work has focused on linear models leaving scope for works in context of deep neural networks. Sanyal et al. [30] train DNN using reweighting strategies for optimizing metries, Huang al. [8] use reinforcement learning strategy to optimize complex metrics, and Kumar et al. [12] optimize complex AUC (Area Under Curve) metric for deep neural network. However, all these works have primarily worked in supervised learning setup and are not designed to effectively make use of available unlabeled data.
7 Conclusion
In this work. we aim to optimize practical con-decomposable metrics readily used in machine learning through self-training with consistency regularization, class of semi-supervised learning methods. We introduce cost-sensitive self-training framework (CSST) that involves minimizing a cost-sensitive error pseudo labels and consistency regularization. We show theoretically that we can obtain classifiers that can better optimize the desired non-decomposable metric than the original model used for obtaining pseudo labels, under similar data distribution assumptions used for theoretical analysis Self-training. We then apply CSST to practical and effective self teff-training method of FixMatch and UDA, incorporating a novel regularizer and thresholding mechanism based a given non-decomposable objective. We find that CSST leads to a significant gain in performance of desired non-decomposable metric, in comparison to vanilla self- training- based baseline. Analyzing the CSST framework when the distribution of unlabeled data significantly differs from labeled data is a good direction to pursue for future work.
Acknowledgements: This work was supported in part by Fujitsu Research grant. Harsh Rangwani supported by Prime Minister's Research Fellowship (PMRF).
10

---

References
[1] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learn- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors. Advances Neural Information Processing Systems. Curran As- sociates, Inc., 2019. URL https: pttps://prooeeddng..nnoe hwt.gs..eurss... wwtrriis.ccppppppe pttpppp/p///////////// (1l11800999264441000000bbbbbaba0
pdf. 2. 10
[2] Kaidi Cao. Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with lubel-distrrbution-aaarr margin loss. Advances in neural information processing systems, 32, 2019. 10
[3] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks, 20(3)552-554. 2009.
[4] Andrew Cotter, Heinrich Jiang, Maya R Gupta, Serena Wang, Taman Narayan, Seungil You, and Karthik Sridharan. Optimization with einediffereet constraints with applications to fairness, recall, churn, and other goals. J. Mach. Learn. Res., 20(172):1-59, 2019. 2
[5] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268-9277, 2019. 10
[6] Gabriel Goh, Andrew Cotter, Maya Gupta, and Michael P Friedlander. Satisfying real-world goals with dataset constraints. Advances in Neural Information Processing Systems, 29, 2016. 2
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 8
[8] Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Bautista Martin, Shih- Yu Sun, Carlos Guestrin, and Josh Susskind. Addressing the loss-metric mismatch with adaptive loss alignment. In International conference on machine learning, pages 2891-2900 PMLR, 2019. 10
[9] Durk Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
[10] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017. 8
[11] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009. Technical report, University of Toronto. 8
[12] Abhishek Kumar, Harikrishna Narasimhan, and Andrew Cotter. Implicit rate-constrained opti- mization of nonooecomeo objectives. In International Conference on Machine Learning, pages 5861-5871. PMLR, 2021. 10
[13] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint
arXiv:1610.02242 2016. 10
[14] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large- scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195, 2015. 8
[15] Yi Lin, Yoonkyung Lee, and Grace Wahba. Support vector machines for classification in nonstandard situations. Machine learning, 46(1) I): 1):1-22222 2002. 10
[16] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Pouts. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Compulational Linguistics: Humun Language Technologies, pages 142- 150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URI http:// /ttt:///ww... aclweb. org/ httpe//thoooooo/11-l1aaa 1015. 8

---

[17] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail taig-taalll learning via logit adjustment. In International Conference on Learning Representations, 2020. 3, 4,
[18] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993, 2018. I, 7, 10
[19] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In International Conference on Machine Learning, pages 4615-4625. PMIR, 2019. 2
[20] Harikrishna Narasimhan and Aditya K. Menon. Training over-parameterized models with non-decomposable objectives. Advances in Neural Information Processing Systems, 34, 2021.
2, 3, 4,
[21] Harikrishna Narasimhan, Rohit Vaish, and Shivani Agarwal. On the statistical consistency of plug-in classifiers for coomppoooeoeabee performance measures, Advances in neural information processing systems, 27, 2014. 8, 10
[22) Harikrishna Narasimhan, Harish Ramaswamy, Aadirupa Saha, and Shivani Agarwal. Consistent multiclass algorithms for complex performance measures. In International Conference on Machine Learning, pages 2398-2407. PMLR, 2015, 3, 10
[23] Nagarajan Natarajan, Oluwasanmi Koyejo, Pradeep Ravikumar, and Inderjit Dhillon. Optimal classification with multivariate losses. In International Conference on Machine Learning, pages 1538. PMLR, 2016. 10
[24] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944-1952, 2017. 3
[25] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 11557-11568, 2021.
[26] Shameem Puthiya Parambath, Nicolas Usunier, and Yves Grandvalet. Optimizing f-measures by cost-sensitive classification. Advances in neural information processing systems, 27, 2014.
10
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jouaatiiaanoounaaa of computer vision, 1 115(3)2211-25, 2015.
[28] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetrie tri-training for unsupervised domain adaptation. In International Conference on Machine Learning, pages 9998-2997.
PMLR, 2017. 10
[29] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019. 8
[30] Amartya Sanyal, Pawan Kumar, Purushottam Kar, Sanjay Chawla, and Fabrizio Sebastiani. Optimizing non-decomposable measures with deep networks. Machine Learning, 107(8): 1597-1620, 2018. 2, 10
[31] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems, 33:596-608, 2020. 1, 4, 5, 6, 7, 10
[32] Shiv Kumar Tavker, Harish Guruprasad Ramaswamy, and Harikrishna Narasimhan. Consistent plug-in classifiers for complex objectives and constraints. Advances in Neural Information Processing Systems, 33:20366-20377 2020.
[33] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
[34] Colin Wei and Tengyu Ma. Improved sample complexities for deep neural networks and robust classification via an all-layer margin. International Conference on Learning Representations, 2019. 6
12

---

[35] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self- ceff-training with deep networks on unlabeled data. In International Conference on Learning Representations. 2020. 2, 5, 6
[36] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Le. Unsupervised data aug- mentation for consisteney training. Advances in Neural Information Processing Systems, 33: 2020. 1, 2, 4, 10
[37] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020. 1, 10
[38] Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate example weighting. In Third IEEE international conference on data mining, pages 435-442.
IEEE, 2003. 10
[39] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1 1-8.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.87 URL https::/t//.. /dx, doi. org/10. 5244/2.4222 30 87.
[40] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34, 2021.
[41] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5982-5991, 2019. 10
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section A in Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section A in Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Assumption Assumption 3, and Assumption 4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section C, D and Section E in Appendix.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] We will release code here: https://github.e com//al
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Refer Section 5, and more detail is available in Section L in supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] As experiments are done neural networks which are expensive to train we haven't included the error bars in result currently. However, we do plan to provide error bars for a certain subset of experiments in Appendix Tab.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section G in Appendix
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes] See Section G in
Appendix
(b) Did you mention the license of the assets? [Yes] See Section Appendix
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xV5DLIaey2w", "reviewer": "Reviewer_GPmX", "summary": "Non-decomposable metrics take an important role in machine learning. This paper proposes to improve self-training by optimizing non-decomposable objectives for semi-supervised learning. This strategy can bring a significant average improvement in desired metric of minimizing worst-case recall while maintaining similar accuracy compared with SOTA methods. Empirical results show that proposed Cost-Sensitive Self-Training (CSST) framework help improve the performace of baselines, e.g., FixMatch, UDA.", "questions": "See above.", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- Propose to improve self-training method by optimizing non-decomposable metrics that utilizes unlabeled data in addition to labeled data.\n- Propose a weighted consistency regularizer for Cost-Sensitive Self-Training.\n- The superiority of desired non-decomposable metric is theoretically justified.\n- Empirical results have validated the effectness of the proposed method, and bring improvement over baselines.\n\n\nI am overall satisfied with this paper and would like to give some comments that may make this paper better.\n- It is expected to release the source code of all experiments to benifit the community.\n- It is promising to compare with methods that improves the recall of Fixmatch, e.g., Flexmatch.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Y8VcHju7DXP", "reviewer": "Reviewer_FQ9c", "summary": "The paper propose a new loss function for self learning that can take into account also non-decomposable cost functions. The work both analyzes the performance when using this new cost function and show empirically its advantage over previous solutions for self-training. In the theoretical part, the author claim that their loss function leads to a lower error compared to standard optimization over the loss. In the practical part, the authors show that their approach, which combines also a smart thresholding for keeping only part of the examples for the self training, leads to improved results over previous solutions. Both vision and NLP benchmarks are being considered. ", "questions": "see above ", "limitations": "Didn't see limitations being mentioned ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper is that it combines a theoretical analysis with some strong empirical results compared to existing baselines. \nA new loss function is being proposed and a thresholding approach. Both seems to improve the performance for the task at hand\n\nWeaknesses: \n\n1. The paper is very (very very) hard to read. The notation are not well presented. Some symbols are defined only after they are being first used. The writing is quite bad and it is very hard to follow. Few examples:\na.  the recall and precision on line 65-66 are defined the same. \nb. F_{pl =} is used in Assumption 1 but defined only in Section 3.4\nc. The notation is also not very consistent. Sometimes the authors use P_w where w is the weight and in the same equation use P_i where i is an index. It is very unclear and confusing. \nd. Less critical compared to the above but part of the things that are defined should be better defined. For example c that is used in assumption 4 is mentioned without saying what it is. Indeed, it is mentioned in the c-expansion but the statement should start with mentioning that there is a function c and then say what is done with it. The comment here, is indeed a matter of style (unlike point a-c, which are not a matter of style but just make the paper very hard to follow)\n\n2. The assumption made in theoretical part are not justified. Specifically, in assumption 1 if we assume beta is negligible then it means that R_B,w is basically equal to zero. Is it really the constraint we want to use? \nAlso, if we assume that the error of the found function F^* is smaller than the one of F_{pl}, why Theorem 5 is surprising? Is not it just saying that if the error is smaller than the error is smaller? The authors assume something and then claim something more complicated that more or less states the same. At the end of the day, they want to show that F^* is better. So they assume that and then claim they proved that. So what is the point? \n\n3. The experiments are interesting but I wonder whether the main benefit is just from the new thresholding being used. There is an ablation that show the impact of this new threshoding approach on the proposed CSST approach. Yet, the real missing experiment is checking what is the impact of this approach on existing baselines (e.g., FixMatch). Clearly, the problem with the threshold has already been mentioned in previous works as the authors admit. So proposing a solution for it should be checked also with previous works. \n\n4. Finally, why only one baseline is being compared to in each task?  Is FixMatch/UDA the only work that was proposed sine 2020 for this problem (Clearly there are and the authors should compare to them). Note that this is the least important problem in the paper that I find compared to the other points raised above. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TWq2WAOc8g4", "reviewer": "Reviewer_Vi6Y", "summary": "The paper proposes a technique for optimizing non-decomposable metrics in self-supervised learning settings. The method is based on the hybrid loss for optimizing non-decomposable metrics in supervised learning settings (Narashiman & Menon, 2021). The authors apply the reduction of the non-decomposable metric to cost-sensitive learning and form a weighted consistency regularizer. The framework is then used to improve the FixMatch algorithm for self-supervised training. Finally, the authors demonstrate the benefit of the models in real-world datasets.", "questions": "Please answer my concerns and questions in the previous section.\n", "limitations": "The authors have not discussed the limitation of the proposed model. A discussion on the model's limitations is suggested.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Optimizing non-decomposable metrics are relevant to many real-world problems\n- The authors provide theoretical justifications for the proposed models.\n- The experiments show the benefit of the proposed algorithm compared to baselines.\n\nWeaknesses:\n- My biggest complaint about the paper is the clarity of the paper. The paper is hard to understand for someone that is not already familiar with the related works. In many places, the authors refer to the related works without providing enough explanation for the reader to understand. In short, I do not think the authors have made the paper self-contained.\n- Some terminologies in the paper are used before even defining them, for example, Assumption 1 in Sec 3.2 uses F_pl, but it is defined only after Sec 3.4.\n\n----------- post rebuttal -----------\n\nThanks to the authors for providing the detailed explanation. \nThe authors has added more explanation in the appendix. \nHowever, I still think the paper need more explanation on the main paper to add clarity and help reader that is not already familiar with the related works. As also mentioned by another reviewer, the paper is very hard to read.\nFor the technical contribution of the paper, I could not asses it thoroughly due to the difficulty of understanding the contribution of the paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept