Don’t fear the unlabelled: safe semi-supervised learning via debiasing
Authors: Hugo Schmutz
Olivier HUMBERT
Pierre-Alexandre Mattei
Venue: ICLR 2023 poster (2023)
Keywords: Semi-supervised learning
deep learning
empirical risk minimisation
control variate
variance reduction
asymptotic statistics

Abstract: Semi-supervised learning (SSL) provides an effective means of leveraging unlabelled data to improve a model’s performance. Even though the domain has received a considerable amount of attention in the past years, most methods present the common drawback of lacking theoretical guarantees. Our starting point is to notice that the estimate of the risk that most discriminative SSL methods minimise is biased, even asymptotically. This bias impedes the use of standard statistical learning theory and can hurt empirical performance. We propose a simple way of removing the bias. Our debiasing approach is straightforward to implement and applicable to most deep SSL methods.  We provide simple theoretical guarantees on the trustworthiness of these modified methods, without having to rely on the strong assumptions on the data distribution that SSL theory usually requires. In particular, we provide generalisation error bounds for the proposed methods. We evaluate debiased versions of different existing SSL methods, such as the Pseudo-label method and Fixmatch, and show that debiasing can compete with classic deep SSL techniques in various settings by providing better calibrated models. Additionally, we provide a theoretical explanation of the intuition of the popular SSL methods.  An implementation of a debiased version of Fixmatch is available at
https://github.com/HugoSchmutz/DeFixmatch

Published as a conference paper at ICLR 2023
DON'T FEAR THE UNLABELLED: SAFE SEMI-
LEARNING VIA DEBIASING
Hugo Schmutz +f
Olivier Humbert Pierre-Alexandre Mattei
hugo. s schmut z0 @inrii..iiaa.. fr
ABSTRACT
Semi-supervised learning (SSL) provides an effective means of leveraging un- labelled data to improve model's performance. Even though the domain has received a considerable amount of attention in the past years, most methods present the common drawback of lacking theoretical guarantees. Our starting point is notice that the estimate of the risk that most discriminative SSL methods minimise is biased, even asymptotically. This bias impedes the use of standard statistical learning theory and can hurt empirical performance. We propose a simple way of removing the bias. Our debiasing approach is straightforward to implement and applicable most deep SSL methods. We provide simple theoretical guarantees on the trustworthiness of these modified methods, without having to rely on the strong assumptions on the data distribution that SSL theory usually requires. In particular, we provide generalisation error bounds for the proposed methods. We evaluate debiased versions of different existing SSL methods, such as the Pseudo- label method and Fixmatch, and show that debiasing can compete with classic deep SSL techniques in various settings by providing better calibrated models. Additionally, we provide a theoretical explanation of the intuition of the popular SSL methods. An implementation of a debiased version of Fixmatch is available at https: /ttp::://qtthus/ www.:/uggsccmutt//////oooooo// DeF xmatch
INTRODUCTION
The promise of semi-supervised learning (SSL) is to be able to learn powerful predictive models using partially labelled data. In turn, this would allow machine learning to be less dependent on the often costly and sometimes dangerously biased task of labelling data. Farly SSL approaches- e.g. Scudder's ' (1965) untaught pattern recognition predictions made by some estimate of the predictive model and used the obtained psendo-labels to refine their initial estimate. Other more complex branches of SSL have been explored since, notably using generative models (from McLachlan 1977 to Kingma et al. 2014) or graphs (notably following Zhu al., 2003).. Deep neural networks, which are of supervised predictors, have been trained successfully using SSL. Somewhat surprisingly, the main ingredient of their success is still the notion of pseudo-labed labels (or one of its variants), combined with systematic use of data augmentation (e.g. Xie et al. 2019 Sohn et al. 2020 Rizve et al. 202T)
machine-simply replaced unknown labels with
An obvious SSL baseline is simply throwing away the unlabelled data. We will call such a baseline the complete case, following the missing data literature (e.g. [Tsiaiis 2006). As reported in van Engelen & Hoos (2020), the main risk of SSL is the potential degradation caused by the introduction of unlabelled data. Indeed, semi semi-supervised learning outperforms the complete case baseline only in specific cases (Singh et al. 2008 Schölkopf et al. 2012: Li & Zhou 2014). This degradation risk for generative models has been analysed in Chapelle et al. (2006) Chapter 4). To overcome this issue, previous works introduced the notion of safe semi-supervised learning for techniques which never reduce predictive performance by introducing unlahelled data (Li & Zhou 2014 Guo et al. 2022)). Our loose definition of safeness is as follows: an SSL algorithm is safe if has theoretical guarantees that are similar or stronger the complete case baseline. The "theoretical" part of the definition
" "Université Côte d' Azur, Inria, Maasai, LIAD, CNRS, Nice, France 1 Université Côte Azur, TIRO-MATOs, UMR E 4320 CEA, Nice, France Centre Antoine Lacassagne, Nice, France

---

Published as a conference paper at ICLR 2023
14
True distribution
06
Pseudlo-labll
Complete case
DePseudo label
Figure 1: (Left) Data histogram. (Right) Posterior probabilities p[1[z) of the same model trained following either complete case (only labelled data), Pseudo-label or our DePseudo-label.
is motivated by the fact that any empirical assessment of generalisation performances of an SSL algorithm jeopardised by the scarcity labels. "Similar or stronger guarantees" can be understood in broad sense since there are many kinds of theoretical guarantees (e.g. the two methods may be both consistent, have similar generalisation bounds, be both asymptotically normal with related asymptotic variances). Unfortunately, popular deep SSL techniques generally do not benefit from theoretical guarantees without strong and essentially untestable assumptions on the data distribution (Mey & Loog 2022) such as the smoothness assumption (small perturbations on the features do cause large modification in the labels, p(y ((((pet((x)) R p(y(z)) or the cluster assumption (data points distributed discrete clusters and points in the same cluster are likely to share the same label).
Most semi-supervised methods rely on these distributional assumptions to ensure performance in entropy minimisation. pseudo-labelling and consistency-basee methods. However, no proof given that guarantees the effectiveness of state-of-the-a
methods (Tarvainen & Valpola 2017, MMyato al. 2018 Sohn al.) 2020 Pham et al. 2021). To illustrate that SSL requires specific assumptions, we show in toy example that pseudo- labelling can fail. To do so, we draw samples from two uniform distributions with small overlap. Both supervised and semi-supervised neural networks are trained using the same labelled dataset. While the supervised algorithm learns perfectly the true distribution of p(1[z), the semi-supervised learning methods (both entropy minimisation and pseudo-label) underestimate [r) for [1, (see Figure []. We also test our proposed method (DeSSL) on this dataset and show that the unbiased version of each SSL technique Icarns the true distribution accurately. See Appendix A for the results with Entropy Minimisation. Beyond this toy example, a recent benchmark (Wang et al. 2022) of the last SSL methods demonstrates that no one method is empirically better than the others. Therefore, the scarcity of labels brings out the need for competitive methods that benefit from theoretical guarantees. The main motivation of this work is to show that these competitive methods can be easily modified to benefit from theoretical guarantees without performance degradation.
1.1 CONTRIBUTIONS
Rather than relying on the strong geometric assumptions usually used in SSL theory, we use the missing completely at random (MCAR) assumption, standard assumption from the missing data literature (see e.g. [Little Rubin 2019) and often implicitly made most SSL works. With this only assumption on the data distribution, we propose a new safe SSL method derived from simply debiasing common SSL risk estimates. Our main contributions are:
We introduce debiased SSL (DeSSL), safe method that can be applied to most deep SSL algorithms without assumptions on the data distribution;
We propose theoretical explanation of the intuition of popular SSL methods. We provide theoretical guarantees on the safeness of using DESSL. both on consistency, calibration and asymptotic normality. We also provide generalisation error bound;
We show how simple it is to apply DeSSL to the most popular methods such as Pseudo-label and Fixmatch, and show empirically that DeSSL leads to models that are never worse than their classical counterparts, generally better calibrated and sometimes much more accurate.
2

---

Published as a conference paper at ICLR 2023
2 SEMI-SUPERVISED LEARNING
The ultimate objective of most of the learning frameworks is to minimise risk R, defined as the expectation of particular loss function L over data distribution p(a;, y), on of models fo(x), parametrised by 8 E O. The distribution p(z, y) being unknown, we generally minimise a Monte Carlo approximation of the risk, the empirical risk R(0) computed on sample of n i.dd points drawn from p(x, 2). R is unbiased and consistent estimate of (()) under mild assumptions. Its unbiased nature is one of the basic properties that used for the development of traditional learning theory and asymptotic statisties (van der Vaart 1998 Shalev-Shwarte & Ben-David 2014).
Semi-supervised learning leverages both labelled and unlabelled data to improve the model's per- formance and generalisation. Further information on the distribution p(z) provides better under- standing of the distributions p(a., y) and also ply| Indeed, p(æ:) may contain information on p(ylar)
(Schölkopf al. 2012 Goodfellow al. 2016 Chapter 7.6. van Engelen & Hoos 2020).
In the following, we have access to n samples drawn from the distribution p(z, y) where some of the labels are missing. We introduce new binary random variable (xx)) that governs whether
or not a data point is labelled (r = 0 missing, r = 1 observed, and т E (0, 1) is the probability of being labelled). The labelled (respectively the unlabelled) datapoints are indexed by the L (respectively 2), Cul- = (I.....n. We note ny the number of labelled and ny the number of unlabelled datapoints.
The MCAR assumption states that the missingness of a label is independent of its features and the value of the label: p(a, y. r) = p(z, v)p(r). This is the case when neither features nor labels carry information about the potential missingness of the labels. This description of semi sumi-supervised learning as a missing data problem has already been done in multiple works (see e-g. [Seeger] 2000:
Ahfock & Mclaachann] 2019). Moreover, the MCAR assumption is implicitly made in most of the SSL works to design the experiments, indeed, missing labels are drawn completely as random in datasets such MNIST, CIFAR SVHN (Tarvainen & Valpola| 2017 7 Miyato et al., [018 Xie et al. [019 Sohn et al. 2020). Moreover, the definition of safety in the introduction is not straightforward without the MCAR assumption the complete case is not an unbiased estimator of the risk in non-MCAR
settings (see e.g. Liu & Goldberg, 2020 Section 3.1).
2.1 COMPLETE CASE: THROWING THE UNLABELLED DATA AWAY
In missing data theory, the complete case is the learning scheme that only uses fully observed instances, namely labelled data. The natural estimator of the risk is then simply the empirical risk computed on the labelled data. Fortunately, in the MCAR setting, the complete case risk estimate keeps the same good properties of the traditional supervised one: it is unbiased and converges point wisely to RØ)). Therefore, traditional learning theory holds for the complete case under MCAR While these observations are hardly new (see e.g. [iu & Goldberg, (020), they can be seen as particular cases of the theory that we develop below. The risk to minimise
(c)
Σ SEL
[(0; 34 yt).
2.2 INCORPORATING UNLABELLED DATA
A major drawback of the complete framework is that lot of data ends up not being exploited. A class of SSL approaches, mainly inductive methods with respect to the taxonomy of van Engelen & Hoos (022)) generally aim minimise modified estimator of the risk by including unlabelled data. Therefore, the optimisation problem generally becomes finding i that minimises the SSL risk,
1 Σ L(0; M.)
ReSL(0) (0) = T4
H(00
ThE
VEC
BEU
where H is a term that does not depend on the labels and À is a sealar weight which balances the labelled and unlabelled terms. In the literature, H can generally be seen surrogate of L. Indeed, it looks like the intuitive choices of H are equal or equivalent to a form of expectation of L on a distribution given by the model.
3

---

Published as a conference paper at ICLR 2023
2.3 SOME EXAMPLES OF SURROGATES
A recent overview of the recent SSL techniques has been proposed by van Engelen & Hoos (020). In this work, we focus on methods suited for discriminative probabilistic model po byya x that approximates the conditional p(y(x). We categorised methods into two distinct sections, entropy and consistency-based.
Entropy-basee methods Entropy-based methods aim to minimise term of entropy of the predic- tions computed on unlabelled data. Thus, they encourage the model to be confident on unlabelled data, implicitly using the cluster assumption. Entropy-based methods can all be described as an expectation of L. under a distribution Ty computed at the datapoint 2:
H x) ET,,, (2. (0; 5, 7)]
For instance, Grandvalet & Bengio 2004) simply use the Shannon entropy as H H0; z) which can be rewritten as equation equation 3 with We (z, 37 8g pz) (p] where 6z is the dirac distribution in z. Also, pseudo-label methods, which consist in picking the class with the maximum predicted probability as pseudo-label for the unlabelled data (Scudder) 1965). can also be described as Equation 3 See Appendix for complete description of the entropy- -based literature (Berthelot et al. 2019 2020
Xie et al. 2019, Sohn et al., 2020;
Rizve et al.) 2021) and further details.
Consistency-based methods Another range of SSL methods minimises a consistency objective that encourages invariant prediction for perturbations either on the data to enforce stability on model predictions. These methods rely on the smoothness assumption. In this category, we cite IT-model (Sajjadi al.] (0166), temporal ensembling (Laine & Aila (017). Mean-teacher (Tarvainen & Valpola| 2017). virtual adversarial training (VAT, Miyato et al.) [201 8) and interpolation consistent training (ICT. Verma et al. 2019)). These objectives H are equivalent to an expectation of L (see Appendix B). The general form of the unsupervised objective can be written as
the model in order
C E. IL(; T, 7)) S ((; = Div(J (ert( )) S C2 [[0; I. ū)], (4) where fa is the predictions of the model, the Div is a non non-negative function that measures the divergence between two distributions, θ is fixed copy of the current parameter 0 (the gradient is not propagated through é). pert is perturbation applied to the model or the data and 0 S C1 S C2.
Previous works also remarked that H is an expectation of L for pseudo-label (Zhu et al.] [2022] Amminian et al. (022). We describe more general framework covering further methods and provide with our theory an intuition on the choice of H.
minimisation and
2.4
SAFE
LEARNING
The main risk of SSL the potential degradation caused by the introduction unlabelled data when distributional assumptions are not satisfied (Singh et al.) 2008 Schölkopf et al. 2012 Li & Zhou
2014). specifically in settings where the MCAR assumption does not hold anymore OOlieer et al.
2018. Guo al. 2020). Additionally, in Zhu et al. (2022). the authors show disparate impacts of
pseudo-labelling on the different sub-classes of the population. As remarked by /Oliver et al. 2018).
SSL performances are enabled by leveraging large validation sets which is not suited for real-world applications. To mitigate these problems, previous works introduced the notion safe semi-supervised
learning for techniques which never reduce learning performance by introducing unlabelled data (Li
& Zhou 2014 Kawakita & Takeuchi 2014 Li et al. 2016: Gan et al. 2017: Trapp et al.) 2017 Guo al. 2020). While the spirit of the definition of safe given by these works the same, there are different ways of formalising it. To be more precise, we listed in Appendix C. theoretical guarantees
given by these different works. In our work, we propose to call safe an SSL algorithm that has thcoretical guarantees that are similar to stronger than those of the complete case baseline.
Even though the methods presented in Section 2.3 produce good performances in a variety of SSL benchmarks, they generally do not benefit from theoretical guarantees, even elementary. Moreover, Schölkopf et al. (2012) identify settings on the causal relationship between the features x and the target y where SSL may systematically fail, even if classic SSL assumptions hold. Our example of Figure [ also shows that classic SSL may fail to generalise in a very benign setting with a large number of labelled data. Presented methods minimise biased version of the risk under the MCAR
4

---

Published as a conference paper at ICLR 2023
assumption and therefore classical learning theory cannot be applied anymore, precisely in Appendix C.2 Learning over a biased estimate of the risk is not necessarily unsafe but it is difficult to provide theoretical guarantees on such methods even if some works try to do with strong assumptions on the data distribution (Mey & Loog 2022, Sections 4 and 5). Additionally, the choice of H can be confusing as seen in the literature. For instance, (Grnnvvalet & Bengio (2004) and Corduneanu & Jaakkola (2003) perform respectively entropy and mutual information minimisation whereas Pereyra al. (2017) and Krause et al. (2010) perform maxinisation of the same quantities.
argue more
Some other SSL methods have theoretical guarantees, unfortunaeely, so far these methods come with either strong assumptions or important computational burdens. [i & Zhou (2014) introduced a safe semi-superviset SVM and showed that the accuracy of their method is never worse than SVMs trained with only labelled data with the assumption that the true model is accessible. However, if the distributional assumptions are not satisfidd, no improvement or degeneration is expected. [Sakai et al. (2017) proposed an unbiased estimate of the risk for binary classification by including unlahelled data. The key idea is to use unlabelled data to better evaluate the one hand the risk of positive class samples and on the other the risk of negative samples. They provided theoretical guarantees on its variance and generalisation error bound. The method is designed only for binary classification and has not been tested in a deep-learning setting. It has been extended to ordinal regression in follow-up work (Tsuchiya al., (02T). In the context of kernel machines, [Liu & Goldberg (2020) used an unbiased estimate of risk, like ours, for specific choice of H. Guo et al. (2020) proposed DS L. safe method that needs to approximately solve a bi-level optimisation problem. In particular, the method is designed for a different setting, not under the MCAR assumption, where there is a class mismatch between labelled and unlabelled data. The resolution of the optimisation problem provides a solution not worse than the complete case but comes with approximations. They provide a generalisation error bound. [Sokolovska et al. (2008) proposed method with asymptotic guarantees using strong assumptions such that the feature space is finite and the marginal probability distribution of a fully known. Fox-Robert, & Rosten (2014) proposed an unbiased estimator the generative setting applicable to a large range of models and they prove that this estimator has a lower variance than the one of the complete case.
3 DESSL: UNBIASED SEMI-SUPERVISED LEARNING
To overcome the issues introduced by the second term in the approximation of the risk for the semi- supervised learning approach, we propose DeSSL, unbiased version of the SSL estimator using labelled data to annul the bias. The idea here is to retrieve the properties of classical learning theory. Fortumately, we will see that the proposed method eventually have better properties than the com- plete case, in particular with regard to the variance of the estimate. The proposed DeSSL objective is
RDSSSI
1 EL(0;22 20; 91)+ Th REC
Σ ((0
Σ (0, IEC
(5)
Th
T4
iEN
Under the MCAR assumption, this estimator is unbiased for any value of the parameter A. For proof of this result see Appendix D We prove the optimality of debiasing with the labelled dataset in Appendix F Intuitively, for entropy-based methods, H should be applied only on unlabelled data to enforce the confidence of the model only unlabelled datapoints. Whereas, for sesiistency-basee methods, H can be applied to any subset of data points. Our theory and proposed method remain the same whether H is applied to all the available data not (see Appendix K.
3.1 DOES THE DESSL RISK ESTIMATOR MAKE SENSE?
The most intuitive interpretation is that by debiasing the risk estimator, we get back to the basics of learning theory. This way of debiasing closely related the method of control variates (Owen) 2013 Chapter 8) which is common variance reduction technique. The idea is to add an additional term to a Monte-Carlo estimator with a null expectation in order to reduce the variance of the estimator without modifying the expectation. Here, DeSSL can also be interpreted as a control variate on the risk's 's gradient itself and should improve the optimisation scheme. This idea is close to the optimisation schemes introduced by Johnson & Zhang (2013) and Defazio al.) (2014) which reduce the variance of the gradients' estimate to improve optimisation performance. As a matter of fact, we study the gradient's variance reduction of DeSSI. in Appendix E.I
5

---

Published as a conference paper at ICLR 2023
Another interesting way to interpret DeSSL is constrained optimisation problem. Indeed, min- imising RDESSL is equivalent minimising the Lagrangian of the following optimisation problem:
min 8
(C)
1 H((::::)
Σ H(0.z.). TI iEC
(6)
s.t.
nu IEM
The idea of this optimisation problem minimise the complete case risk estimator by assessing that some properties represented by H are on average equal for the labelled data and the unlabelled data. For example, if we consider datropy-minimisatiin the same confidence on the unlabelled examples on the labelled ones.
this program encourages the model have
The debiasing term of our objective will penalise the confidence of the model on the labelled data. Pereyra et al. (2017) show that penalising the entropy in a supervised context for supervised models improves on the state-of-the-ard on common benchmarks. This comforts us the idea of debiasing using labelled data in the case of entropy-minimisation Moreover, the debiasing term in pseudo- Coandorff-Nielseee 1976). Intuitively we understand the
label is similar to plausibility inference benefits of debiasing the estimator with labelled data to penalise the confidence of the model on these datapoints. But the debiasing can be performed on any subset of the training data (labelled or unlabelled). However, in regards to the variance of the estimator, only the labelled data or the whole dataset are both optimal and equivalent (see Appendix F.
prove that debiasing with
Our objective also resembles doubly-robust risk estimates used for SSL in the context of kernel chines by [iu & Goldberg (2020) and for deep learning by Hu et al. (2022). In both cases, their focus quite different, they consider weaker conditions than MCAR, but very specific choices of H.
3.2 IS RDSSSL AN ACCURATE RISK ESTIMATE?
Because of the connections between our debiased estimate and variance reduction techniques, we have a natural interest in the variance of the estimate. Having a lower-variance estimate of the risk would mean estimating it more accurately, leading to better models. Similarly to traditional control variates (Owen 2013), the variance can be computed, and optimised in X:
Theorem 3.1. The function À (RR (0)(r) reaches its minimum for:
The Cov(L y), ((;;x))
Aopt=
(7)
v(H(0:x))
n
and at Aopt!
V ((RDess((0)))) = =(1-
Tu Pi., v(Rcc(0))r) (((cc((0)))
(8)
n
where PI.H Co H(;x)))
Additionally, we have a variance reduction regime V(RDessz(()) (0)lr) < V(Roc(())r) for all between 0 and 2Nopt- A proof of this theorem is available as Appendix E This theorem provides a formal justification to the heuristic idea that H should be a surrogate of L. Indeed, DeSSL is more accurate risk estimate when H is strongly positively correlated with L, which likely to be the case when is equal or equivalent to an expectation of L. Then, choosing À positive is a coherent choice. We demonstrate in Appendix that L and H are positively correlated when L is the negative likelihood and H is the entropy. Finally, we experimentally validate this theorem and the unbiasedness of our estimator in Appendix E. Other SSL methods have variance reduction guarantees, see Fox- FFox-Roberts & Rosten (2014) and Sakai et al. (2017). In purely supervised context, Chen et al. (2020) show that the effectiveness of data augmentation techniques lies partially in the variance reduction of the risk estimate. A natural application of this theorem would be to tune X automatically by estimating Aopt- In our case however, the estimation of Cov(L(#: x, y). (6; x)) with few labels led to unsatisfactory results. However, estimate it more accurately using the test set (which of course impossible in practice) on different datasets and methods to provide intuition on the order of Appt and the range of the variance reduction regime in Appendix M.2

---

Published as a conference paper at ICLR 2023
3.3 CALIBRATION, CONSISTENCY AND ASYMPTOTIC NORMALITY
The calibration of model is its capacity of predicting probability estimates that are representative of the true distribution. This property is determinant real-world applications when we need reliable predictions. In the sense of scoring rules (Gneiting & Raftery (007) theory, we prove DeSSL is as well-calibrated as the complete case. See Theorem G.1 and proof in Appendix G
We say that Ô is consistent d(0, 0*) & when n - oc, where d a distance on ④ The properties of f depend on the behaviours of the functions L and H. We will thus use the following standard assumptions.
Assumption 3.2. The minimum 0 R is well well-separated: info:d(0 ,022 R(0) > R(0*). Assumption 3.3. The uniform weak law of large numbers holds for both L and H.
Theorem 3.4. Under the MCAR assumption, Assumption 3.2 and Assumption
3.3
0
arg min R.SS.. is consistent.
For proof of this theorem see Appendix G (1998) Theorem 5.7 proving the consistency of an M-estimator. The result holds for the complete case, with À = 0 which proves that the complete case is solid baseline under the MCAR assumption.
This theorem is simple application of van der Vaart's s
Coupling of ny and n, under the MCAR assumption Under the MCAR assumption, ny and ny random variables. We have ~ B(rr) (i.e. any z has the probability of being labelled). Then, nig
with n growing to infinity, we have ni = mr-nn > π. Therefore, both ny and na grow to infinity and niL Au
_-1 This implics nu (n4) when goes infinity.
Going further, we prove the asymptotic normality of Ô DeSSI. by simplifying a bit the learning objective. The idea is replace n, with an to remove the dependence between samples. However, this modification is equivalent to our objective by modifying X. We also show that the asymptotic variance can be optimised with respect to A. We define the cross-covariance matrix between random vectors ( x. v) and V V (0;x) as Keli.j) (Cv((V (0; y)i, V H x)j).
Theorem 3.5. Suppose L and H are smooth functions in (O, Assume R(f) admits a second- order Taylor expansion at 8 with non-singular second order derivative Vo-. Under the MCAR assumption, 0DeSsI. asymptotically normal and the trace of the covariance can be minimised. Indeed, Tr Tr(EDesSL) reaches its minimum at
Tr Kg- Vg- Tr EIV 2); Vy
Dopt (1 πt
(9)
and at Aopts have Tr(EDesst) Tr(Loo)) 0.
See Appendix I for proof of this result. The Theorem shows there exists a range of X for which DeSSL is better model's parameters than the complete case. In this context, we have a variance reduction regime on the risk estimate but even more importantly on the parameters estimate.
3.4 RADEMACHER COMPLEXITY AND GENERALISATION BOUNDS
In this section, we prove an upper bound for the generalisation error of DeSSL.. To simplify, we use the same modification as for the asymptotic variance. The unbiasedness of RDSS can directly be used derive generalisation bounds based on the Rademacher complexity (Bartlett Mendelson 2002). defined in our case as
Σ (0: )] (10)
R, = E(es)sn
sup EEE
Σ ELL(0; v.)
A
e ((;
A
nH iEC
TEST
n01
FEC
VeM
where e are i.i.d. Rademacher variables independent of the data. In the particular of X we recover the standard Rademacher complexity of the complete case.
Theorem 3.6. We assume that labels are MCAR and that both I. and H are bounded. Then, there exists a constant k. > 0, that depends on X, T,, H, and the ratio of observed labels, such that, with probability at least - for all 0 θ,
log(4/8)
R()) S (essi(()) (0)+ +2,, + K
(11)
7
7

---

Published as a conference paper at ICLR 2023
The proof follows Shwaart- & Gen-David (0044 Chapter 26), and is available in Appendix
4 EXPERIMENTS
We evaluate the performance of DeSSL against different classic methods. In particular, we perform experiments with varying À on different datasets MNIST, DermaMNIST, CIFAR-10, CIFAR-I00 (Krizhevsky 2009) and five small datasets of MedMNIST (Yang al. 20221] 2023 with fixed X. The results of these experiments are reported below. In our figures, the error bars represent the size of the 95% confidence interval (CT). Finally, modified the implementation of Fixmatch (Sohn et al. 2020) and compare with its debiased version on CIFAR-10, CIFAR- 100 and on STL10 (Coates al. 2011). Finally, we show how simple it is to debias an existing implementation, by demonstrating it on the consistency- based models benchmarked by (Oliver et al.) 2018), namely VAT, model and MeanTTeachrr on CIFAR-IB and SVHN (Netzer et al., 2011). We observe similar performances between the debiased and biased versions for the different methods, both in terms of cross- entsopy and accuracy. See Appendix O
4.1 PSEUDO LABEL
We compare Pseudol Label and DePseudol on CIFAR-I0 and MNIST. We test the influence of the hyperparameters and report the accuracy. the cross-entropy and the expected calibration crror (ECE, Guo et al. 2017) at the epoch of best validation accuracy using 10% of ny the validation set.
MNIST is an advantageous dataset for SSL since classes are well-separated. We train a LeNet-like architecture using n, 1000 labelled data 10 different splits of the training dataset into a labelled and unlabelled set. Models are then evaluated using the standard 10, 000 test samples. Results are reported in Figure and Appendix L In this example SSL and DeSSL have almost the same accuracy for all X, however, DeSSL seems to be always better calibrated.
CIFAR-10 We train a CNN-13 from Tarvainen & Valpola (2017) on 5 different splits. We use n4 4000 and use the rest of the dataset unlabelled. Models then evaluated using the standard test samples. Results are reported Figure 3 We report the ECE Appendix [M] The performance of both methods on CIFAR-100 with nıl 10000 are reported in Appendix M We observe DeSSL provides both a better cross-entropy and ECE with the same accuracy for small A. For larger X, DeSSL performs better in all the reported metrics. We performed paired Student's t-test to ensure that our results are significant and reported the p-values in Appendix M The p-values indicate that for À close to 10, DeSSL is often significantly better in all the metrics. Moreover, DeSSL for large provides a better cross-entropy and ECE than the complete case whereas SSL never does.
is large-scale MNIST-like collection of biomedical images. We selected the five smallest 2D datasets of the collection, for these datasets it is likely that the cluster assumption no longer holds. Results are reported Appendix L We show DePseudoLabel competes with PseudoLabel in terms of accuracy and even success when PseudoLabel 's accuracy is less than the complete case. Moreover, DePseudoLabel is always better in terms of cross-entropy, so calibration, whereas PseudoLabel is always worse than the complete case. We focus the unbalanced dataset DermaMNIST with a majority class and we compare the accuracy per class of the complete case, PseudoLabee and DePseudoL Label, see Figure 4 In this context, PseudoLabel fails to learn correctly the minority classes as the majority class is over-represented in the pseudo labels. DePseudoL outperforms both the Complete Case and PseudoLabel in term of balanced accuracy.
MedMNIST
Figure 2: The influence of À on Pseudo-label and DePseudo-label for Lenet trained MNIST with Yu = 1000: (Left) Mean test accuracy; (Right) Mean test cross-entropy, with 95% CI.
Figure 3: Influence of X on Pseudo-label and DePseudo-Label for CNN trained CIFAR with yu 4000: (Left) Mean test accuracy; (Right) Mean test cross-entropy, with 95% CI.
8

---

Published as a conference paper at ICLR 2023
Figure 4: Class accuracies (without the majority class) on DermaMNIST trained with my = 1000 labelled data on five folds. (Left) CompleteCase (B Ae- 26.88 + 2.26%) (Middle) PscudoLabel (B-Acc: 22.03 = 1.45%); (Right) DePscudoLabel (B-Ace: 28.84 + 1.02%) % with 95% CI.
4.2 FIXMATCH (SOHN AL.. 2020
The efficiency of the debiasing method lies in the correlation the labelled objective and the unla- belled objective shown in Theorem 3.1] So, we debiased version of Fixmatch that includes strong augmentations in the labelled objective. See Appendix for further details. We also compare our results to the original Fixmatch (Fixmatch" in Table T. While the modified version is slightly worse than the original Fixmatch, debiasing it works. For this experiment, we use nı = 4000 on 5 different folds for CIFAR-10 and лı 10000 one fold for CIFAR-I00. First, report that a strong com- plete case baseline using data augmentation reaches 87.27% accuracy on CIFAR-I0 10 (resp. 62.62% on CIFAR-100 100). A paired Student's t-test ensures that our results are significantly better on CIFAR- 10 (p-value in Appendix N). Then, we observe that the debiasing method improves both the accuracy and cross-entropy of this modified version of Fixmatch. Inspired by Zhu al. 2022). we analyse individual classes on CIFAR-10 and show that our method improved performance on "poor" classes more equally than the biased version. Pseudo-label-basee methods with a fixed selection threshold draw more pseudo-labels in the "easy" subpopulations. For instance, in the toy dataset (Figure 6 PseudoLabel will always draw samples of the class blue in the overlapping area. DeSSL prevents the method to overfit and be overconfident Fixmatch improves Fixmatch by 1.57% (resp. 1.94%) overall but by 4.91% on the worst class (resp. 8.00%). We report in Appendix N.1 the accuracy per class and in Appendix N.2 the results on STL10.
the "easy" classes with the debiasing term. Indeed, De-
Table 1: Test accuracy, worst class accuracy and cross-entropy of Complete Case, Fixmatch and DeFixmatch on 5 folds of CIFAR- and one fold of CIFAR- 100. Fixmatch" the original version of Fixmatch, that uses strong augmentations only on the unlabelled.
CIFAR-10 10AR-1O-e (n:
4000)
CIFAR- 100 (i)
10000)
Complete Case
Fixmatch
DeFismatihh
Fismatchh
Complete Case
Fismotcch
DeFisintch
Fixmmich"
Accuracy
87.27 :: 0.25
93.87 = 0.13
95.44 0.10
95.48 0.17
62.62
69.28
71.22
71.93
Worst cluss accunacy
70.88 + 0.93
82,25 2.27
87.16 + 0.46
86,88 0.78
28.00
23.00
31,00
29.00
Cross entropy
0.01
0.27 0.01
0.20
0.20 0.01
1.87
1.52
1.42
1.51
Brier score
0214 0.005
0.101
20003
0.076 0.001
0.000
0.56
0.44
0.44
CONCLUSION
Motivated by the remarks of Van Engelen & Hoos (2020) and Oliver et al. (018)) on the missingness of theoretical guarantees in SSL, we proposed simple modification of SSL frameworks. We consider frameworks based on the inclusion of unlabelled data in the computation of the risk estimator and debias them using labelled data. We show theoretically that this debiasing comes with several theoretical guarantees. We demonstrate these theoretical results experimentally on several common SSL datasets. DeSSL shows competitive performance in terms of accuracy compared its biased version but improves significantly the calibration. There are several future directions open to us. Our experiments suggest that DeSSL will perform as well as SSL when classic SSL assumptions are true (such as the cluster assumption), for instance with MNIST in Figure 2 Still, for MNIST, DeSSL does not outperform in terms of accuracy but does in terms of cross- cross-entropy and expected calibration error. However, when these assumptions do not hold anymore DeSSL will outperform its SSL counterparts. We showed that Aopt exists (Theorem 3.1 and therefore our formula provides guidelines for the optimisation of X. Finally, an interesting improvement would be to go beyond the MCAR assumption by considering settings with a distribution mismatch between labelled and unlabelled data (Guo al. 2020 Cao al. 2021
Hu et al. 2022

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work has been supported by the French government, through the 31A Côte d' Azur, Investment in the Future, project managed by the National Research Agency (ANR) with the reference number ANR-19-P31A-000)
We thank Jes Frellsen for suggesting the interpretation of DeSSL constrained optimisation problem. We are also grateful to the OPAL infrastructure from Université Côte d' Azur for providing resources and support.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jía, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Lcrge-scale machine learning on heterogeneous systems, 2015. URL htttp:: /wwp://wwwww//.// tensor flww org/ from tensorflow.org.
Olah,
Software available
Daniel Ahfock and Geoffrey J McLachlan. On missing label patterns in semi-supervised learning. arXiv preprini 1xXiv:194.002885 2019.
Gholamali Aminian, Mahed Abroshan, Mohammad Mahdi Khalili, Laura Toni, and Miguel Ro- approach to supervipe learning under covariate-shift.
drigues. An International Conference on Artificial Intelligence and Statistics, 2022.
Athanassios N Avramidis and James Wilson. splitting scheme for control variates. Operations Research Letters, 1993.
Ole Barndorff-Niclsen. Plausibility inference. Journal of the Royal Statistical Society: Series B (Methodological), 1976.
Peter L Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 2002.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Informaiinn Processing Systems, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-supervised learning with distribution matching and augmentation anchoring. International conference on Learning Representations, 2020.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning, 2021.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. MIT Press, 2006.
Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 2020.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. International Conference on Artificial Intelligence and Statistics, 2011.
Adrian Corduneanu and Tommi Jaakkola. On information regularization. Uncertainty Artificial Intelligence, 2003.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. Conference on Computer Vision and Pattern Recognition Workshops, 2020.
10

---

Published as a conference paper at ICLR 2023
Aaron Defazio, Francis Bach, and Simon Lacoste-Julies SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems, 2014.
Patrick Fox-Roberts and Edward Rosten. Unbiased generative semi-supervised learning. The Journal of Machine Learning Research, 2014.
Haitao Gan, Zhenhua Li, Yingle Fan, and Zhizeng Luo. Dual learning- based safe semi-supervised learning. IEEE Access, 6:2615-2621, 2017.
Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 2007.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems, 2004.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International Conference on Machine Learning, 2017.
Lan-Zhe Guo, Zhen-Ye Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. International Conference on Machine Learn- ing, 2020.
Charles Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg. Nathamiel J. Smith, Robert Kern, Matti Pieus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre (éeradd-Marchen Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis Oliphant. Array programming with NumPy. Nature, 2020.
Xinting Hu, Yulei Niu, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. On non-random missing labels in semi-supervised learning. International Conference on Learning Representations, 2022.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in Neural Information Processing Systems, 2013.
Masanori Kawakita and Jun't ichi Takeuchi. Safe semi-supervised learning based weighted likelihood. Neural Networks, 2014.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. Advances Neural Information Processing Systems, 2014.
Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized informa- tion maximization. Advances in neural information processing systems, 23, 2010.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, MIT, NYU, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. International Conference on Learning Representations, 2017.
Dong-Hyun Lee. Pseudo-Label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, International conference on machine learning, 2013.
Yu-Feng Li and Zhi- Hua Zhou. Towards making unlabeled data never hurt. IEEE transactions on pattern analysis and machine intelligence, 2014.
11

---

Published as a conference paper at ICLR 2023
Yu-Feng Li, James T Kwok, and Zhi-Hua Zhou. Towards safe semi-supervisee learning for multivari- ate performance measures. AAAI Conference on Artificial Intelligence, 2016.
Roderick JA Little and Donald B Rubin. Statistical Analysis with Missing Data. John Wiley & Sons, 2019.
Tiantian Liu and Yair Goldberg. Kernel machines with missing responses. Electronic Journal of Statistics, 2020.
Marco Loog. Contrastive pessimistic likelihood estimation for semi-supervised classification. IEEE transactions on pattern analysis and machine intelligence, 2015.
Fredrik Lundh, M Ellis, et al. Python imaging library (PIL), 2012.
Wes McKinney et al. Data structures for statistical computing in python. Python in Science Conference, 2010.
Geoffrey John McLachlan. Estimating the linear discriminant function from initial samples containing small number of unclassified observations. Journal of the American statistical association, 1977.
Alexander Mey and Marco Loog. Improved generalization in semi-supervised learning: survey of theoretical results. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning, IEEE transactions pattern analysis and machine intelligence, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Handbook of econometrics, 1994.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evalua- tion of deep semi-supervised learning algorithms. Advances in Neural Information Processing Systems, 2018.
Art B. Owen. Monte Carlo theory, methods and examples. 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py PyTorch: An imperative style, nighpeerfrmance deep learning library. Advances in Neural Information Processing Systems, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. Journal of machine learning research, Octobre 2011.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. Workshop track, International Conference on Learning Representations, 2017.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. Conference on Computer Vision and Pattern Recognition, 2021.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. International Conference on Learning Representations, 2021.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in Neural Information Processing Systems, 2016.
12

---

Published as a conference paper at ICLR 2023
Tomoya Sakai, Marthinus Christoffel Plessis, Gang Niu, and Masashi Sugiyama. Semi-supervisee classification based on classification from positive and unlabeled data. International conference on machine learning, 2017.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij.
On causal and anticausal learning. Internation conference on machine learning, 2012.
Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transac- tions on Information Theory, 1965.
Matthias Seeger. Learning with labeled and unlabeled data. Technical report, 2000.
Denis Serre. Matrices. Theory and Applications (Second edition). Springer, 2010.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge university press, 2014.
Aarti Singh, Robert Nowak, and Jerry Zhu. Unlabeled data: Now it helps, now it doesn't 't. Advances in Neural Information Processing Systems, 2008.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. FixMatch: Simplifying semi cemi-supervised learning with consistency and confidence. Avances Neural Information Processing Systems, 2020.
Nataliya Sokolovska, Olivier Cappé, and François Yvon. The asymptoties of superviee learning in discriminative probabilistic models. International Conference on Machine Learning, 2008.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve sunii-upeeeised deep learning results. Advances in Neural Information Processing Systems, 2017.
Martin Trapp.
Tamas Madl, Robert Peharz, Franz Pernkoff, and Robert Trappl. Safe sumi-supervised learning of sum- pro-pprduct networks. Conference on Uncertainty in Artificial Intelligence, 2017.
Anastasios A Tsiatis. Semiparametric theory and missing data. Springer, 2006.
Taira Tsuchiya, Nontawat Charoenphakdec, Issei Sato, and Masashi Sugiyama. Semi-supervised ordinal regression based empirical risk minimization. Neural Computation, 2021.
Aad W van der Vaart. Asymptotic statistics. Cambridge university press, 1998.
Jesper E van Engelen and Holger H Hoos. A survey semi-supervised learning. Machine Learning, 2020.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. International Joint Conference on Artificial Intelligence, 2019.
Yidong Wang, Hao Chen, Yue Fan, SUN Wang, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang. Zhi Zhou, Lan-Zhe Guo, et al. USB: A unified semi-supervised learning benchmark for classification. Advances Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Michael Waskom, Olga Botvinnik, Drew O'Kane, Paul Hobson, Saulius Lukauskas, David C Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Jordi Warmenhoven, Julian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram, Tal Yarkoni. Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, Chris Fonnesbeck, Antony Lee, and Adel Qalich. Seaborn: v0.8.1, September 2017. URL httph: //o::::///oooooo http:::///dooooooo http://o.....ooo..ooo 52811
zenodoo 883859
ker.odo883859)
Colin Wei. Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks unlabeled data. International Conference Learning Representations, 2021.
13

---

Published as a conference paper at ICLR 2023
Qizhe Xie, Zihang Dai, Eduard Hovy. Minh- Minn-Thang Luong, and Quoe V Le. Unsupervised data augmnentation for consistency training. Advances in Neural Information Processing Systems, 2019.
Jiancheng Yang, Rui Shi, and Bingbing Ni. MedMNIST classification decathlon: A lightweight Au- toML benchmark for medical image analysis. IEEE 18th International Symposium Biomedical Imaging, 2021.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. MedMNIST v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Nature Scientific Data, 2023.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. FlexMatch. Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 2021a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. Internation Conference on Learning Representations, 2017.
Shuai Zhang, Meng Wang. Sijia Liu, Pun-Yu Chen, and Jinjun Xiong. How unlabeled data improve
generalization in self-training? A one-hidden-layer theoretical analysis. International Conference on Learning Representations, 2021b.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi- supervisd learning using Gaussian fields and harmonic functions. International Conference on Machine Learning, 2003.
Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semi-supervised learning. International Conference on Learning Representations, 2022.
14

---

Published as a conference paper at ICLR 2023
A TOY EXAMPLE
We trained a 4-layer neural network (1/20/100/2/11 with ReLU activation function using 25, 000 labelled and 25, 000 unlabelled points drawn from two ID uniform laws with an overlap. We used = 1 and a confidence threshold for Pseudo-label T 0.70. We optimised the model's weights using a stochastic gradient descent (SGD) optimiser with learning rate of 0.1.
Figure 5: Data histogram
1/4
True distribution Complete case
1/-
Thue distribution
Pseudo-label
EntMin
Sabelddolabbll
DeEnthin
00100
04100
-2
-1
332
334
330-
326
326
1928
1928
40200
50000
Figure 6: 4-layer neural net trained using SSL methods on ID dataset drawn from two uniform laws. (Top-left) Posterior probabilities p(1[xr of the same model trained following either complete case (only labelled data), Pseudo-label our DePseudo-label. (Top-right) Same for EntMin and DeEntMin (Bottom-left) Training cross- entropy Pseudo-lahel DePseudo-label (Bottom-right) Training cross-entropy EntMin DeEntMin
15

---

Published as a conference paper at ICLR 2023
B DETAILS ON SURROGATES AND MORE EXAMPLES
We provide this appendix further details on our classification of SSL methods between entropy- based and consistency-baed (see Section 2.3). We detail general framework for both of these methods' classes. We also show how popular SSL methods are related to our framework.
B.1 ENTROPY-BASED
We class as entropy-beae based, methods that aim to minimise a term of entropy such as Grandvalet & Bengio (004) which minimises Shannon's 's entropy or pseudo-label which is form of entropy, Remark E.5 These methods encourage the model to be confident on unlabelled data, implicitly using the cluster assumption. We recall those entropy- -based methods can all be described as expectation of L under a distribution Ta computed at the datapoint x:
H0; x) = (2. (0,;
(12)
Pseudo-label: As presented in the core article, the unsupervised objective of pseudo-label can be written an expectation of L on the distribution T. 5) = 6₂ (e) (5)Ø). The Pseudo-label unsu- pervised term can be written as an entropy term of Rényi entropy. Hox (z) For this reason, we class Pseudo-labe methods entropy prropy-baseed methods. Recently, Lee (0033) encouraged the pseudo-labels method for deep semi-supervised learning. Then, Rizve et al. (202)) recently improved the pscudo-label selection by introducing uncertainty-awant mechanism
max, log p
the confidence of the model concerning the predicted probabilities. Pham et al. (2021) reaches
state-of- of- the-art the Imagenet challenge using pseudo-labels large dataset of additional images.
B.2 PSEUDO-LABEL AND DATA AUGMENTATION
Recently, several methods based on data augmentation have been proposed and proven to perform well on a large spectrum of SSL tasks. The idea is have model resilient to strong data augmentation of the input (Berthelot et al., 2019; 2020 Sohn et al. 2020 Xie et al. 2019 Zhang et al. 2021a). These methods rely both on the cluster assumption and the smoothness assumption and are at the border between entropy-he bbased and consistency-based methods. The idea is to have the same prediction for an input and an augmented version it. For instance, in Sohn et al. (2020), we first compute pseudo-labels predicted using a and then minimise the likelihood with the predictions of the model on a strongly augmented version of In Xie al. (2019), the method is little bit different the prediction of the model on æ and the predictions of an augmented version. both cases, the unsupervised part of the risk estimator can be reformulated Equation 12
version of (flip-and-s)) hopp-and-shift data augmentation)
minimise the cross entropy between
Fixmatch: In Fixmatch, Sohn et al. 2020), the unsupervised objective can be written as:
H(#; x) = 1|max (B(l)))) 7 =] x2, arg max pa (g(u)x))
(13)
where is fixed copy of the current parameters 0 indicating that the gradient is not prop- agated through it, Ii is a weakly-augmented version of x and Se strongly-augmented one. Therefore, we write H as an expectation of L on the distribution ny (z, Sag (5)6arg max, polula.) (t) (m)x ()())11) x1) r].
UDA: In UDA, Xie et al. 2019), the unsupervised objective can be written as:
H(0:x) = Σ Epg(vla)L(0;01..)
(14)
where i is fixed copy of the current parameters 0 indicating that the gradient is not propagated through it and x, is an augmented version of x. Therefore, we write H as an expectation on the distribution ,,, (f,0) ū) = dz, (7)))
16

---

Published as a conference paper at ICLR 2023
Others: Recently, have been proposed in the literature Zhang et al. 2021aa and Rizve et al. 22021) The former is an improved version of Fixmatch with a variable threshold T with respect to the class and the training stage. The latter introduces measurement of uncertainty in the laeudo-labelling step to improve the selection. They also introduce negative pseudo-labels to improve the single-label classification.
B.3 CONSISTENCY BASED
Consistency-based methods aim to smooth the decision function of the models have more stable predictions. These objectives are not directly a form of expectation of but are equivalent to an expectation of L. For all the following methods
can write the unsupervised objective H such that:
C (EE (1;; x. 5) S H(0;)) S (2 (1;;
(15)
with & C1 & C2.
Indeed, consistency-basee methods minimise unsupervised objective that is divergence between the model predictions and modified version of the input (data augmentation) or perturbation the model. Using the fact that all norms are equivalent in ditenssiinn space such as the space of the labels, we have the equivalence between a consistency-bast -bssd H and an expectation of L.
VAT The virtual adversarial training method proposed by (Miyato al. 2018 generates the most impactful perturbation Tady to add to æ. The objective is to train a model robust to input perturbations. This method is closely related to adversarial training introduced by Goodfellow al. (2014).
H(0 x (z. Tadvi :)
where the Div is a non-negative function that measures the divergence between two distributions, the cross-entropy or the KI. divergence for instance. If the divergence function is the cross-entropy, it is straightforward to write the unlabelled objective as Equation 3 divergence, we can write the objective as
If the objective function the KI.
(16) with (.. (z)g))())) Therefore, variation of H with respect O are the same as (m+(a)))))) (0; T. 7).. VAT is also method between consistency-basee and entropy-based methods long we the KL-divergence the cross-entropy the measure of divergence.
HO; = Ex, (#+r, [(0 9)] - Ex,,
Mean-Teacher A different form of pseudo-labelling is the Mean-Teacher approach proposed by (Tarvainen Valpola, 2017) where pseudo-labels are generated by teacher model for student model. The parameters of the student model are updated, while the teacher's s are moving average of the student's parameters from the previous training steps. The idea is to have a more stable pseudo-labelling using the teacher than in the classic Pseudo-label. Final predictions are made by the student model. A generic form of the unsupervised part of the risk estimator is then
H(0: ((po()l=) p& p
where 9 are the fixed parameters the teacher.
II-Model
The II-Models are intrinsically stochastic models (for example a model with dropout) model. The SSL loss is using the stochastic behaviour of the model where the model Se and penalises different predictions for the same x Sajjadi et al. 2016). Let's note fo ))1 and fe(z. .)2 two passes of through the model fa. A generic form of the unsupervised part of the risk estimator is then
encouraged to make consistent predictions through several passes of the same
H(;; x) = (z,
(17)
where Div is a measure of divergence between two distributions (often the Kullback-Leibler diver- gence).
17

---

Published as a conference paper at ICLR 2023
Temporal ensembling Temporal ensembling (Laine & Aila 2017) is form of II-Model where we compare the current prediction the model on the input a with an accumulation the previous passes through the model. Then, the training is faster as the network is evaluated only once per imput on each epoch and the perturbation expected to be less noisy than for IT-models.
ICT Interpolation consistency training Veerma al. 2019) is an SSL method based on the mixup operation (Zhang et al. (2017). The model trained is then consistent to predictions at interpolations. The unsupervised term of the objective then computed on two terms:
( 21, x2) Div (fo(am) +
(f -) (1 (x
(18)
with a drawn with from a distribution ((a, a). With the exact same transformation, we will be able to show that this objective is equivalent to a form of expectation of L.
18

---

Published as a conference paper at ICLR 2023
C SAFE SEMI-SUPERVISED LEARNING
C.I ON THE DEFINITION OF SAFE SEMI-SUPERVISED LEARNING
Previous works introduced the notion safe semi-supervised learning for techniques which never reduce learning performance by introducing unlabelled data (Li & Zhou 2014 Kawakita & Takeuchi 2014 Li al. 2016 Trapp et al., 2017 Guo et al. 2020). While the spirit the definition of safe given by these works is the same, there are different ways of formalising it. In the following, we list the theoretical guarantees given on the performance of SSL methods by each of these works:
& Zhou (2014) prove that the accuracy of their method, namely S4VM, is never worse than the complete case in a transductive setting under the low-density assumption.
Kawakita & Takeuchi (2014) prove that the model learned is asymptotically normal and has a lower asymptotic variance without assumption on the data distribution but only when
S Ru.
Li et al. (2016) prove that their method builds a model which better for a chosen measure of performance in the range of top-k precision, F3 score AUC than the complete under either the cluster assumption, the low-density assumption or the manifold assumption.
Trapp al. (2017) prove that the Sum-Product network learned has a lower train loss than the one with the complete case.
Guo et al. 2020) prove that the empirical risk of the SSL model on the labelled train set is lower or equal than the one of the complete case with no assumption on the data distribution nor the missingness mechanism. Additionally, they prove generalisation error bounds.
Some other works proposed theoretical guarantees in the use of their SSL methods without calling the method safe. regard to the definition and the safe SSL methods listed above, we can categorise the following method as safe. Among them, we cite the following:
Sokolovska et al. 2008) prove that the model learned is asymptotically normal and has lower asymptotic variance with the assumptions that the feature space is finite and nhe -> co.
Fox-Roberts & Rosten (2014) propose unbiased estimator of the likelihood in a generative setting and prove that the estimator is unbiased and has a lower variance than its complete case baseline with no assumption on the data distribution.
Loog (2015) proposes method for which the log- log-likelihood of the SSL model the labelled train set is lower or equal to the one of the complete case with the gaussian distribution assumption of Linear Discriminant analysis.
Sakai et al. (2017) prove that their risk estimator is unbiased and has lower variance than the complete case baseline. Additionally, they prove generalisation error bounds and the bounds decrease with respect to the number of unlabelled data without assumption on the data distribution but with strong assumption on the chosen loss function.
C.2 ON THE SEMI-SUPERVISED BIAS
We provide in this appendix further explanation of the risk induced by the SSL bias introduced in Section 2.4
Presented methods minimise a biased version of the risk under the MCAR assumption and therefore classical learning theory does not apply anymore,
E[RSsL(0) = E[L(O: æ, + + R(0).
(19)
Learning over a biased estimate of the risk is not necessarily unsafe but it is difficult to provide theoretical guarantees on such methods even if some works try to do so with strong assumptions on the data distribution (Mey & Loog 2022] Section 4 and 5, Z.anng et al. 2021b). Previous works proposed generalisation error bounds SSL methods under strong assumptions on the data distribution or the true model. We refer the survey by Mey & Loog (2022). More recently, Wei al. (202T) proves
19

---

Published as a conference paper at ICLR 2023
an upper bound for training deep models with the pseudo-label method under strong assumptions. Under soft assumptions, Aminian et al. (2022) provides an error bound showing that the choice of H crucial to provide good performances.
Indeed, the unbiased nature of the risk estimate is crucial in the development of learning theory. This bias the risk estimate may look like one of regularisation, such as ridge regularisation. However, SSL and regularisation are intrinsically different for several reasons:
Regularisers have a vanishing impact in the limit of infinite data whereas SSL usually do not in the proposed methods, see Equation 19 A solution would be to choose À with respect the number of data points and make vanish when n goes to infinity. However, in most works, the choice of is independent of the number of ny (Oliver al. 2018 Sohn ct al. 2020).
One of the main advantages of regularisation to turn the learning problem into "more convex" problem, see (Shalev-Shwards & Ben-David (2014) Chapter 13). Indeed, ridge regularisation will often turn a convex problem into a strongly-convex problem. However, SSL faces the danger turn the learning problem into non-convex previously noted by
Sokolovska et al. (2008).
The objective of regulariser is bias the risk towards optimum with smooth decision functions whereas entropy-based SSL will lead to sharp decision functions.
Regularisation usually does not depend on the data whereas H does in the SSL framework.
An entropy bias has been actually used by Pereyra et al. (2017) as a regulariser but as entropy maximisation which should have the exact opposite effect of the SSL method introduced by Grandvalet Bengio (2004), the entropy minimisation.
20

---

Published as a conference paper at ICLR 2023
D PROOF THAT ReSSL (0) IS UNBIASED UNDER MCAR
Theorem D.1. Under the MCAR assumption, RDDSSL(() is an unbiased estimator of R((0).
As consequence of the theorem, under the MCAR assumption, Rcc(0) also unbiased special case of DeSsL for À
Proof: We first recall that the DeSSL risk estimator R.DESSL is defined for any by
RDSSSL = Σ
1
L(0:
24,
Ui) +
Σ
H(f:
H(::
T4 iEC
Thu
iEll
iEc
(20)
Tt L(0:2,)0)) +
1-ri
Ti
H
H(::
By the law of total expectation:
E[RDessL(0)] =E, (EnsE.y)
As far as we are under the MCAR assumption, the data (x. y) and the missingness variable are independent thus, E, (R.yyRrsssss (0)]r] Er [Rnss(
We focus on Er,y RR.yS (0)). First, we replace RDESSL (0) by its definition and then use the linearity of the expectation. Then,
Eg,y (DDSSL
E
1
L(0; we) + Thu
Σ H(0;
A H(;;
by definition
Th iEC
IEC
iEN
X
[[((0: 24, 9:)1
+
E(H(0;:))
E
[6;
by linearity
T1 SEC
na
Th
kel
iEC
The couples (2i, vi) are i...d. samples following the same distribution. Then, we have
En.y RRDSSI
1 Σ E|L(O;x,V)
Σ
[[H(()x)]]
A
Σ E[H(O;x)) i.id samples
iEC
SEU
IEC
E [L(0; x, v)]
R()).
Finally, we have the results that, RDSSI (0) is unbiased R()) is constant,
E(RDessI(0) = EE RED.SR (0))r] E, [R(0]] R(0).
(21)
21

---

Published as a conference paper at ICLR 2023
E PROOF AND COMMENTS ABOUT THEOREM 3.1
Theorem 3.1
The function + ((DDeSS((0)r) reaches its minimum for:
Thu Cov z, y), (0;
Aopt
(22)
(0;)
and
((RDessL(0)1)) 2opp
hape -(1 (1-
Bu pi.H) v(Rcc(0))
n S v(Rcc(0)),
(23)
where PL,H = Corr(L(0; v, y), ())
Proof:
For any À E IR,we want compute the variance: V(RDessL(0)))
Under the MCAR assumption, and y are both jointly independent Also, the couples (x,, ye. rz) are independent. Therefore, we have
((RDessL(O)r)) =
Σ V (ons)))pteaal))
T''s L(0, Fz, 9:)
+
1- ri
H zzi)
i..d samples
nuu
Σ ((sa)))p(
Ft L(0, z4, yi)
(1- riri
ri
H
H(8,2.) 221)
(x,3) and r independent
Tuu
n1
Using the fact that the couples (x,, yi) are i..dd samples following the same distribution, we have
(RDesssL(0))) =
((.))~(())))
Th L(0, x, v)
1- ri
H(0,2)
T2
Yhu
1 11-r
((T(0,,,,)) (1, + x2
v(H(0,x))
using covariance
m
ni
+ 2x Ti
1
Cov Cov H()
n1
T4
Now, we remark that the variable is binary and therefore 72 = (1- r) = 1 and r(1 (1-r) = Using that and simplifying, have
((DD,ss((0)))=
rs Σ
(LL(,, 2,,00 x2 (1 + r.m V (HH(f,)))
(1-1))rr r:)nf
n
n?n
21 Cov(L x. y). H ((,)))
Finally, by summing and simplifying the expression (note that ny ny n)), we compute the expression variance,
((DDess(())))== = 1 ((L((,,,,)) + X2 " V(H((,,))) 2A H(0 m
So (()DEssL(()))) is quadratic function in and reaches its minimum for Aopt such that:
nu Cov x. v). H(0,
Appt
(OH(0,)))
22

---

Published as a conference paper at ICLR 2023
And, at Aopt the variance of RDDSSL (0)[r) becomes
1 V(L(O,, 2, (1 Th Cov x. v). H)))))))
((DDess((0)r)
n (LL),
TI I V(L(O, v, U))
Tu Corr(L(e, v). H(8, x))²
TM
(1 The pi.
1 V(L(O,x,y)) n
Remark E. If H is perfectly correlated with L (PL,H = 1), then the variance of the estimator is equal to the variance of the estimator with no missing labels.
The data distribution p(r, y) being unknown, the computation of Aopt is not possible directly. Therefore, need to use an estimator of the covariance Cov(L x, y). H (0;z)) and the variance ((H(0;)) (See Equation 24p. Also, we have be careful using the training set, Appt becomes dependent on x and and therefore R.DSSSL (0) rr) becomes biased. A solution would be to validation dataset for its computation. Another approach is to compute it using the splitting method (Avramidis & Wilson] 1993). Moreover, the computation of Aopt is tiresome and time-consuming practice it has to be updated for every different value of so at each gradient step.
Remark E.2. Is it possible to estimate Aopt in practice ?
introduce new bias with the computation of Appt indeed, if compute
1 Ty (H(0;))) (())²
(0))
Appt
(24)
where H((0) = = E. (0;;;;)) and T.(0) = //2 Eiec T(0; y:)
Remark E.3. About the sign of A The theorem still has quantitative merit when it comes choosing A. by telling that the sign of À is positive when H and L are positively correlated which will generally be the case with the examples mentioned the article. For instance. concerning the entropy minimisation technique, the following proposition proves that the log- ligliikel is negatively correlated with its entropy and therefore it justifies the choice of X > the entropy minimisation.
Proposition E.4. The log-likel of the true distribution log pl y) (z) is negatively correlated with its entropy Hy(p(ūlx)) = -Eg-p(((>) [log p(ū]x))
Cov(logp(ula), Ha(p(ū(x))
(25)
Proof.
Cov(logp(ulz)) (,(]) = Ez.
- [/ap://)))eEEoooo/////////////00
E (og.gloo)()))))) (log [(}==)))
En, [e
= By the law total expectation, we have that E. [.,, then
Hy = - F xr Po [ogr(() E (og)
- x²)² - log. pog
On the other hand, also with the law of total expectation, E.g. [[Eygl(logg((u) [log (og
En p(glo) Jog (ū)l))
log p()
E E Dogr( 2 E Ezygooggyll)))
Jensen's 's inequality total expectation law
23

---

Published as a conference paper at ICLR 2023
Finally, we have the results,
Cov(log p(ul)), (p(((())))
[og p(y[x)]²
0
Remark E.5. We can also the Pscudo-label form of entropy. Indeed, modulo the confidence selection on the predicted probability, the Pscudo-label objective is the inverse of the Rényi min- entropy:
(x) = max log p(ylæ)
E.1 ON RISK ESTIMATION QUALITY
We train CNN-13 on CIFAR-10 using only 4,000 labelled data and then split the test dataset into labelled and unlabelled data to estimate the PscudoLabel (PL) and DePseudoLabel (DePL) risks that we compared to the oracle risk estimate using all the test set. For this experiment, we split 100 times the test set into 40 labelled data and 9960 unlabelled data order to estimate the variance of the risk estimator. We compute Aopt using the entire test set. This experiment illustrates that DePL is unbiased for any value (Figure Left) and its variance can be optimised and at Aopts we have (())) S (0)c((0) (Figure Right)
De
DePL
ViRe 6911
1.01
10P
10
10>
162
10P
Figure 7: (Left) Risk estimate value for PseudoLabes (PL) and DePseudol (DePL. compared to the true value of the risk. (Right) The influence of on the ratio (
The gradient of DeSSL also an unbiased estimator of the risk's gradient, the Complete Case. We compare the quality of these two estimators under the same setting. We compare the variance of the gradient DePL risk to the complete case and show that DePL. reduces the variance considerably (see Figure 8). Variance reduction techniques applied on the gradients have shown promising results in the amelioration of optimisation algorithms (Defazio al. 2014 Johnson & Zhang 2013).
2.0
0.5
6.0
10
10
10a
Figure 8: The influence of À the ratio V(V WWWR
24

---

Published as a conference paper at ICLR 2023
F WHY DEBIASING WITH THE LABELLED DATASET?
We remark that the debiasing can be performed with any subset of the training data, labelled and unlabelled. The choice of debiasing only with the labelled data can be explained both intuitively and computationally in regard to the Theorem 3.1 Intuitively, the debiasing term penalises the confidence on the labelled datapoints and then prevents the overfitting on the train dataset. As remarked in section
3.1. Pereyra et al. (2017) showed that penalising low entropy models acts as strong regulariser supervised settings. This comforts the idea of penalising low entropy on the labelled dataset, i.e. debiaising the entropy minimisation with the labelled dataset. Considering Seeud-LLabel-bee methods, the objective for the labelled data to predict the correct labels with moderate confidence. This is also similar to the concept of plausibility inference described by Srrnddorff-Neelsee (1976).
In regard to Theorem 3.1 we show that the optimum choice of subset for debiaising is either only the labelled data or the whole dataset and both are equivalent.
We consider a subset of the training set. We defined a as follow:
{
1/|AI
if oz E A otherwise
ag
The unbiased estimator is then:
RDEESSL.A0)) =
1 Σ
L(0; v,, vi)
Σ a ((O
(26)
nu
IEC
Nell
We compute the variance of this quantity as in the proof of Theorem 3.1 and show that:
n
) ((H(()
((DDsS5L.(0))))- Σ V(L 2, v)) +22
ai
Th
(27)
2X ria Cov (Lv(L((, (0,
m
Suppose that no labelled datapoints are in A. Then, the last term of the variance is null. Hence, having no labelled datapoints in A leads to variance increase. We also remark that debiasing with the entire dataset is equivalent that debiasing with the labelled datapoints. Indeed
DeSSI (0)
1 Σ L(0 +
A
Σ
H(00
H(0;
The sell
SEC
i=1
A H(0;
H(0:
H(0: keu
m iec
Tue sel
iec
Xni
An
L(0: Fi, y/)
+
Σ H(0:z:))
H(0:
nu
ny REC
SEC
sell
which is equivalent debiasing with only the labelled dataset by replacing À by A
At this point, we can still sample a random subset composed of labelled datapoints and u unlabelled datapoints. Therefore a, 1/(1 + u)1(x, A}, we show in the following that the optimum choice of the couples (1, u) are (nz, 0) and (nu, n,), so only the labelled or the whole dataset.
We sample/ labelled and unlabelled datapoints debiased the estimator, by simplifying the term in the sum of Equation 27 as follow:
if 3; E A and r;
1
if x; E A and r, 1 if s; & A and r;
ai
(] u+=
na
if I; & AA and r;
25

---

Published as a conference paper at ICLR 2023
Then, by summing the temn and simplifying, get:
1 (())) - V(L(O, x, + X²
u -
1
+
1
au
v(H(0,z))
Th
(
(l+ u)+
n
2x 11(1 + u)
Cov(L(O, 1), H(0,,))
1 -[ x,,) x2 nu nu u +1 ((H(0xx))
nu 1+ u
22
Cov(L x, v) H())))
1e(1 + u)
We want to minimise (()))))((0)r) with respect (A,1, u). V(RDessL (0)]r) reaches is mini- mum in X at
n4
Cov(L(0, x, y). x))
Aopt
Bu u uu
((H(0,z))
Then,
((RDsss) 1 VV(L(O (0,00,,,1))
Thu
12
Cov x, y), H(0, ((H(0,z))
T1
71 (nu u + 1)(
nu and S < nu. We can easily show that the (ma- - u + ()(1 + u) reaches its minimum for u = 0 or u Tly and for both value:
We now want to minimise with respect
(0))) 1 V(L(O, 2,
Cov(L(O, x, v). H(0,))))
The +1
V(H
Then 1/(ne + 1) is increasing function, then reaches its maximum a = ny. So finally, the optimal choices for the couple (r4, 0) and (ng n.).. We showed that these couples are equivalent.
26

---

Published as a conference paper at ICLR 2023
G PROOF OF THEOREM G.1
The calibration of a model is its capacity of predicting probability estimates that are representative of the true distribution. This property is determinant real-world applications when we need reliable predictions. A scoring rule S is a function assigning a score to the predictive distribution po(y)æ)) relative to the event y/a p(y]2), S(po. (x, y)), where p(x, y) the true distribution (see e.g. Gneiiing & Raftery 2007). A scoring rule measures both the accuracy and the quality of predictive uncertainty, meaning that better calibration is rewarded. The expected scoring rule defined S(po.p) p) (1.
A proper scoring rule is defined as scoring rule such that S(pe. p) S S(p,p) (Gneiting & Raftery 2007). The motivation behind having proper scoring rules comes from the following: suppose that the true data distribution is accessible by set models. Then, the scoring rule encourages to predict Pa = p. The opposite of a proper scoring rule can then be used to train model to encourage the calibration of predictive uncertainty: L(#; x, y) = --S (z, y)). The most common losses used to train models are proper scorings rules such log-likelihodd.
Theorem G.1. If S(pe, (z, -L(0 a. v) is proper scoring rule, then S' (z, (2.u,r) ( L(0; x. y) An( 1-t ", (+))))) is also a proper scoring rule.
The proof follows directly from unbiasedness and the MCAR assumption. The main interpretation of this theorem is that we can expect DeSSL to be well-calibrated as the complete case.
Proof. The scoring rule considered in our SSL framework is:
n L x. v) + Am( - -)(H(())) nu
S'op (z,
n2
The proper scoring rule of the fully supervised problem is
S(pe, (2, v,r))) = L(0; x, v)-
Let p be the true distribution of the data (x, y. Under MCAR. is independent of a and y, then p(z, y, r) = p(r)p(x,
/
(')
(x, u, r)) dx dy dr = / p(z, ((r,w)((r) (po, (r, y: r)) dæ dr
/ ((x,w)p(r) L(0;,,,) An 1-r ___ The
_)H((;; - du dy dr
m
m
[ J.n p(
rn dr p(r)
T(0; xr, y) dx dy
m
An / p(z,8) (fplo)
(1-r--) dr
(1-r_ -
H #0; ) dr dy
21
nu
T41
p(z, L(O y) dr dy
B.M
S(pe.P) p)
Therefore, if S(pe, (x, L(:0x,,)))) is proper scoring rule, then S' (z, v,r)) -("7 L(f; x) An( r, (0; is also proper scoring rule.
27

---

Published as a conference paper at ICLR 2023
H
PROOF OF THEOREM 3.4
Assumption
3.2
the minimum 0 of R is well-separated.
inf R(0) R((0)
(28)
0:d(0~,0)) ,02c
Assumption 3.3
uniform weak law of large numbers holds for function L if:
sup
Σ
L(0 Mi) EIL(0,,,,))) 2 0
(29)
AE
YI
Theorem 3.4
Under assumption and assumption B for both L and H, 0 = arg min R.sSs asympiotically consistent with respect to n.
This result is direct application of Theorem 5.7 from van der Vaart 1998 Chapter 5) that states that under assumption A and B for L, 0 arg min R. is asymptotically consistent with respect Assumption A remains unchanged as we have M-estimators of the same R. We now aim to prove that under assumption for both L and H, we have the assumption on rm L(B: I. y) X(1
22t (0; x)).
Lemma H.1. If the uniform law of large number holds for both L and H, then it holds for 0 22 L(0 x, v) + X(1 my (0;x).
Proof. Suppose assumption for L, then the same result holds we replace n with ny as n and my coupled by the law of Indeed, when grows to infinity, ny too and inversely. Therefore,
sup eEe
Σ REC
1(0 xi, vi E[L(O; x. v)
40
T1
Now, suppose we have assumption for H, then we can make the same remark than for L. Now, we have show that:
supp ACB
r L(0,z,)) An
(1-r r nu
H(0:z)
E[L(0;
1=1 T4
T4.
We first split the absolute value and the sup operator
1-r
sup 8Ee
L(0; 2. + An
H(0: x))
E[L(B: v)]
T2
n4
11
n
1-
sup
L(0; xz
EE[L(O; Ir, v)]
+
An
H(fx)))
#
n
n4
11
1-r
sup
L(0; w)
([ w)
sup 828
Xn
H (0;00)
8c8
111
SEC
nla
mi
E
So we now have to prove that the second term is also converging to 0 in probability. Again by splitting the absolute value and the sup, we have
sup AEG
An
1 -
r
H(#: x)
sup 000
(1 T)
rn H(0; x)
nes
ni
t=1
i=1
1=1
28

---

Published as a conference paper at ICLR 2023
Then have that,
sup 8E
H();a)
r
x0
1=1
1=1
n
sup
r); - [1H(0,I,,)),
r1((0x) - [[H w)]
==1
H(0; a:) E(H x; 2.00) m/
sup ac0
H(0:2) E[H(O,I,U))
+=
SEC
< sup
Σ
H(0:z) E.H (0: x)
+ sup
((0;))
ElH (0:2,00)))
Tuu
711
(=
iEC
Thus,
sup dEθ
n
r L(0; x, 3) + +An
(1-r1- _
-(0: 8)]
E
T1
n2u
And we now just have to apply the results van der Vaart [1988 consistent arg min DeSLL
Theorem 5.7) have the asymptotic
Remark H.2. A sufficient condition on the function H to verify assumption B, the uniform weak law of large numbers. to be bounded (Newey McFadden 1994 Lemma 2.4). For instance, the entropy H = - E, Pa (v/ar) log (p((v=)) is bounded and therefore, the entropy minimisation is asymptotically consistent.
29

---

Published as a conference paper at ICLR 2023
I ASYMPTOTIC NORMALITY OF DESSL
In the following, we study a modified version of the objective to simplify the proof. Let us consider the following DeSSL objective L' 2,,0,,,,r) (0; y) + ( +)
((;;)) which has the
same properties than the original one (unbiasedness, variance reduction property. consistency and benefit from generalisation error bounds). The idea replace ny with mn to simplify the expression. The value лu converges to then the following Theorem should hold with the true DeSSL objective.
We define the cross-covariance matrice between random vectors VL(0; x. v) and (O;;; x) as (o) Cov(VV((0x,, y);, V
(0;
Theorem I.1. Suppose L and H are smooth functions in (e, RR. Assume R(0) admit a second- order Taylor expansion 0" with non-singular second order derivative Vo-. Under the MCAR assumption, have that DDssL is asymptotically normal with covariance:
1 VØ- [V
DeSSL.
V
X² x01
go F V// Vg-
Va- Kg- Va-
As consequence, we can minimise the trace of the covariance. Indeed, Tr(E-Dest)) reaches its minimum at
Tr Kg. Vg-1) Tr(V, L [
Aopt = (1 T)
(30)
and at Xopt
Tr(EpessL) Tr(Ecc) 1 -
Tr(V: Kg- V==
S 0.
(31)
Tr VH
The complete case the special case of DeSSL with X Then, the Theorem holds for the complete case.
H(O;a) +)==a))((;a) The assumptions of the theorem are sufficient assumptions to apply Theorem 5.23 of Van der Vaart 1998 to the couple (ÔDeSSL: I/). Hence, we obtain the following representation for representation @DeSSI:
Proof. We define L'(0;z,y,r) = EL(0;x,1) + + =)H((;a))
Vn(ODessL - 00) 1 nn
Pi π
VV((0"2,,,0) + (11
rs π
r VH(0 xx) + (1).
(32)
V - 0*) E, N(0, EpSSLL)
The asymptotic normality follows with variance:
EDSSSL V EVL'' 2.,00 ;2, Va-¹.
Using the MCAR assumption, simplify the expression of EpsSL:
30

---

Published as a conference paper at ICLR 2023
: " V
DDSSL = Vg
22 Vo. ' [[V V ; Vg.
+ +
(1 - T)2 (
1-- + x
b) Va-
X
Vo-'E (
V
n7²
00 Cov (0(v ; V-)
X²
V E (
VØ=
Vo- Ko- Vo-
We remark that the complete case the particular case of DeSSL with X 0. Then,
A2 7(1
DeSSL Doc
π V-
v0 (,, Va=
Vo- Kg. Vg-
The asymptotic relative efficiency of consequence, the asymptotic relative efficiency pESSL com- Tr(En.ssL)
This quotient can be minimised with respect to pared to Ico is defined as the quotient Tr(Ece) X:
Tr Kg. Vo-²) Aopt (1 Tr(V, E [VM
(33)
and at Aopt :
Tr(Enessi) Tr(Ecc)
=1 1-
Tr(V- Kg- Vo-1
$1. 1.
(34)
Tr(V g- V
Vg- ]r((Ec))
Remark I.2. On the sign of X. It is easy to show that sufficient condition to have Aap > 0 is to have Ka- positive semi-definite. Indeed, using that Vg- is positive definite and Proposition 6.1 of Serre 2010), we show that Tr(V-- Kg- V0- > 0 and then Aopt >> 0.
Remark 1.3. Why minimising the trace EDeSSL? Minimising the trace of EDSSSL leads to an estimator with smaller asymptotic MSE, see Chen et al.] (2020).
Remark L.4. Fully supervised setting. We also remark that our theorem matches the theorem for the supervised setting. Indeed, observing all the labelled corresponds to the case π = and we obtain:
DdsSt Ecc = EFully supervised
31

---

Published as a conference paper at ICLR 2023
J PROOF OF THEOREM 3.6
Our proof will be based on the following result from Shalev-Shwartz & Ben-David 2014 Theorem 26.5).
Theorem J.1. Let H be set of parameters, D random variable living in space 2, 0,
and L: H 2 + [-c, cl. We denote
m2
LD(h) [E](((h,))] =)1, and Ls(h)
(h,,,))
(35)
m
where 2 Fm ii.dd samples from D. For any 8 0, with probability least 6, we have
1 E,e( za) E:e(b,2z))
]]
1oog(4/0)
LD(h) S (h) (esei)m
sup bER
+ 4og
(36)
m
where E.. Em. are i.i.d. Rademacher variables independent from "1,...: Fm
We can now restate and prove our generalisation bound.
Theorem 3.6
We assume that both and H are bounded and that the labels are MCAR. Then, there exists constant > O, that depends on X, L, H, and the ratio of observed labels, such that, with probability at least for all 0 E O
log(4/6)
R())= S Resst 2R, k1
(37)
where Ry, is the Rademacher complexity
R. = =Ees)ss sup 6EB
e:L(O; A Σ (.H((O + n(1 T)
Σ (0:
)] (38)
YLT REC
with EL, .. Em i.i.d. Rademacher variables independent from the data.
Proof. We use Theorem J.1 with (x, r), H θ m n,, and
((h, 1 L(0 vi)
1 Ti
(39)
The unbiasedness of our estimate under the MCAR assumption, proven in Appendix D ensures that the condition of Equation equation 35 is satisfied with LD(h) = R(0) and (h) = Rossst(f)
Now, since L and H are bounded, there exists M 0 such that L M and H < M. We can then bound e:
18(h, =) < M
{ -
1 M
A Amxx
(40)
Now that we have chosen that bounds f, we can use Theorem and finally get Equation equation 37 with x 4cy2.
32

---

Published as a conference paper at ICLR 2023
K
DESSL WITH H APPLIED ON ALL AVAILABLE DATA
For consistency-based SSL methods it is common to use all the available data for the consistency term:
RSSL (0)
Σ
1(0: U4)
Σ H(0)
(41)
TH iEC
With the same idea, debias the risk estimate with the labelled data:
1 El(8:
RDSSSL(0)
L(0: 21 9/
11
72
IEC
(42)
H(0:
EEC
Under MCAR, this risk estimate is unbiased and the main theorem of the article hold with minor modifications. In Theorem 3.1 Aopt is slightly different and the expression of the variance at Xppt remains the same. The scoring rule in Theorem i. is different but the theorem remains the same. Both Theorem 3.4 3.5 and 3.6 remain the same with very similar proofs.
Theorem K.1. The function a V(RDessLL()) reaches its minimum for:
Cov x. y). H(0; x))
Appt
(43)
v(H(0:z))
and
v(R home = (1 n
nu 2 ((Rc((())(()))
(44)
< <V((cc(01)
where PL.H Corr(L(O;x,, v). H (0;x)))
When H is applied on all labelled and unlabelled data, the scoring rule used in the learning process
then S' S'(po, (I, y, r)) - L(0:2.)))) A(1 rm H x)) and have S' is proper scoring rule.
mi
33

---

Published as a conference paper at ICLR 2023
L MNIST AND MEDMNIST
L.1
MNIST
DePaeutstasel
1340
Pssucelabel
8045
$.92
5003
Figure 9: The influence of Pseudo-label and DePseudo-label for Lenet trained MNIST with 1000: (Left) Test accuracy; (Middle) Mcan cross-entropy; (Right) Mcan test ECE, with 95% CI
L.2
MNIST LABEL NOISE
6.1
DePonutttasel
Pssudolabel
k250
5224
AU2
3135
800
Figure 10: The influenec of À on Pscudo-label and DePseudo-label for Lenet trained on MNIST with label noise with nu 1000: (Left) Mcan test accuracy; (Middle) Mean test cross-entropy; (Right) Test ECE, with 95% CI.
L.3 MEDMNIST
We trained 5-layer CNN with a fixed = and n, 10% the training data. We report in Table 2 the mean accuracy and cross-entropy on 5 different splits of the labelled and unlabelled data and the number of labelled data used. We report the AUC in Appendix L DePseudoLabes competes with PseudoLabel in terms of accuracy and even success when PseudoLabel's accuracy is less than the complete case. Moreover, DePseudoLabel is always better in terms cross-entropy, so calibration, whereas Pseudol abel is always worse than the complete case.
34

---

Published as a conference paper at ICLR 2023
Table 2: Test accuracy and cross-entropy of Complete Case (CC), Lseud (PI.) and DePseu- doLabel (DePL) on five datasets of MedMNIST.
DATASET
R
CC
PL
DEPL
CROSS-ENTROPY
ACCURACY
CROSS-ENTROPY
ACCURACY
CROSS-ENTROPY
ACCURACY
DERMA
1000
1.95 + 0.09
68,99= 1.20
2.51 = 0.20
68.88 1.03
1,88
0.12
69.3011 0.85
PNEUMONIA
585
1.47 0.04
83,94
2.40
2.04 1: 0.04
85,831
2.13
1.40
1: 0.06
84.36 1: 3.79
RETINA
160
1.68
0.03
48.30= 3.06
1.80 - 0.18
47.754 2.50
1.67
49.40 2.62
BREAST
78
0.80
= 0.04
76.15= 0.75
1.00 = 0.26
74,74 1.04
0.70 = 0.03
76.67 = 1.32
84.094:
Table 3: Test AUC of Complete Case, PseudoLabel and DePseudoLabel on five datasets of MedM- NIST.
DATASET
COMPLETE CASE
PSEUDOLABEL
DEPSHUDOLABEL
DERMA
84.26 + 0.50
82.64 + 1.19
83.82 = 0.95
PNECMONIA
94.28 - 0.46
94.34 0.91
94.15 0.33
RETINA
70.70 + 0.74
70.12
69.97 = 1.44
BREAST
74.67 + 3.68
74.86
75.33 + 3.05
BLOOD
97.83 + 0.23
97.83 + 0.23
97.72 0.15
35

---

Published as a conference paper at ICLR 2023
M PSEUDOLABEL AND DEPSEUDOLABEL ON CIFAR: P-VALUES
M.1 CIFAR-10
Pscudotabel
5.1
5.5
620
Figure 11: The influence of À on Pseudo-label and DePseudo-label on CIFAR-10 with nl= 4000:
(Left) Mean test accuracy; (Middle) Mean test cross-entropy; (Right) Test ECE, with 95% CI.
08-
0.6
0.4
02
02
0.0
Figure 12: p-values of paired student test between PscudoLabel and DePscudoLabel (Right) DePseudoL is better than LseudoLabel; (Left) DePseudoLabel is worst than Lseu.
M.2 COMPUTATION Aept THE TEST SET.
As explained in the main text, the estimation of Cov y). (0:x)) (() with few labels led to extremely unstable unsatisfactory results. However, we test the formula on CIFAR-10 and different methods to provide intuition on the order Aopt and the range of the variance reduction regime (between 0 and 2Xopt). To do so, estimate Aoyt the test set for CIFAR-10 by training a CNN13 using only 4, 000 labelled data on 200 epochs. The value of Aopt is 1.67, 31.16 and 0.66 for entropy minimisation, pseudo label and Fixmatch. Therefore, the reduced variance regime covers the intuitive choices of in the SSL literature. Unfortunately, computing Aopt on the test set is not applicable in practice.
36

---

Published as a conference paper at ICLR 2023
M.3 CIFAR-100
440
Dehaobiiad
140
Peouttab011
3.73
3.02
3225
Figure 13: The influence of À on Pseudo-label and DePseudo-label on CIFAR-100 with nl= 4000: (Left) Mean test accuracy: (Middle) Mean test entsopy: (Right) Test ECE, with 95% CI.
37

---

Published as a conference paper at ICLR 2023
N FIXMATCH (SOHN ET AL.. 2020
We performed a paired Student t-test to ensure that our performances significandy better CIFAR-I0 on both accuracy, cross-entropy and Brier score. The results show that DeFixmatch is significantly better with p-value of 6.5x10 in accuracy, 3.3x10 in cross-entropy and 7. 7.6x10 in Brier score.
N.1 PER CLASS ACCURACY
In recent work, Zhu et al. (2022) exposed the disparate effect of SSL different classes. Indeed, classes with high complete case accuracy benefit more from SSL than classes with a low baseline accuracy. They introduced metric called the benefit ratio (BR) that quantifies the impact of SSL on a class C::
BR(C) accssI, - acccc(()) accs(C) acccc((C)
(45)
where acessL( (Cecc(C) and accs(C) are respectively the accuracy of the class with an SSL trained model, a complete-case model and a fully supervised model (a model that has access to all labels). Inspired by this work, we report the per-class accuracy and the benefit ratio in Table We that the "poor" classes such as bird, cat and dog tend to benefit from DeFixmatch much more than from Fixmatch. We compute (CCs(C) using pre-trained model with the same architecture'. Zhu al. (2022) also promote the idea that fair SSL algorithm should benefit different sub-classe equally, then having (R(C)) = (R(C")) for all C, C". While perfect equality seems unachievable in practice, we propose to look the standard deviation of the BR through the different classes. While the standard deviation of Fixmatch 0.12, the one of DeFixmatch is 0.06. Therefore, DeFixmatch improves the sub-populations accuracies more equally.
Table 4: Mcan accuracy per class and mean benefit ratio (BR) 5 folds for Fixmatch, DeFixmatch and the Complete Case. Bold: "poor" complete case accuracy classes.
COMPLETE CASE
FIXMATCH
DEFIXMATCH
ACCURACY
ACCURACY
BR
ACCURACY
BR
AIRPLANE
86.94
95.94
0.88
96.62
0.94
AUTOMOBILE
95.26
97.54
0.68
98.22
0.89
BIRD
80.46
90.80
0.68
92.64
0.80
CAT
70.08
82.50
0.56
87.16
0.78
DEER
88.88
95.86
0.78
97.26
0.94
DOG
79.66
87.16
0.53
90.98
0.81
FROG
93.12
97.84
0.80
98.62
0.94
HORSE
90.96
96.94
0.83
97.64
0.92
SHIP
94.12
97.26
0.67
98.06
0.84
TRUCK
93.18
96.82
0.84
97.20
0.93
N.2 STL10
DeFixmatch also performs better than Fixmatch STL-10 using 4200 labelled data for training and 800 for validation.
N.3 FIXMATCH DETAILS
As first detailed in Appendix B Fixmatch is secudo-label-base method with data augmentation. Indeed, Fixmatch uses weak augmentations of x (flip-and-shiff) for the pseudo-labela selection and then minimises the likelihood with the prediction of the model strongly augmented version of
38

---

Published as a conference paper at ICLR 2023
Table 5: Test accuracy, worst class accuracy and cross-entropy of Complete Case, Fixmatch and DeFixmatch on 5 folds of CIFAR-10, one fold of CIFAR- 100 and one fold of STL10.
CIFAR-10 8000)
CIFAR-100 (oz 18000)
(y) 5000)
Complec Case
DeFicmatch
Camplese
Fixmatch
DeFismotch
Complete Cloe
DeFixoutch
Accuracy
57.27 = 0.25
4187 = 0.13
95.44 = 0.10
02.62
09.28
71.22
$3.69
92.64
93.23
Worst class sccuracy
70.08 0.93
82.25 = 2.27
87.16 - 1.46
28.00
23,00
31,00
70.75
85.00
88.25
Croes euttropy
0.60 0.01
0.27 0.00
0.20 0.01
1.57
1.52
1.42
0.42
0.30
0.25
Brier soree
0.214 1 0,005
0.001 10.00
0,06
0.001
0.56
0.47
0.44
0.171
0.117
0,109
Weak augmentations are also used for the supervised part of the loss. In this context,
L(0; xr y) Emy mw)a(())) - (1)(1))
and
((;xx)= (E (w)
(o r] sommm(()) log(po (arg max Po(v)as) |لa))]]
where T1 is weak augmentation of and 22 is strong augmentation. We tried debias implementation of Fixmatchhhoor Thowever training was very unstable and led to model that were much worst than the complete case. We believed that this behaviour is because the supervised part of the loss does not include strong augmentation. Indeed, our theoretical results encourage to have strong correlation between L and H, therefore including strong augmentations the supervised term. Moreover, solid baseline for CIFAR-15 10 using only labelled data integrated strong augmentations (Cubuk al. 2020p. We modify the implementation, see Code ? Therefore, the supervised loss term can be written as:
1 L(O; v) (eak))) log(ps (1)x1))] + Ezy=sttmgg(=)) (ogepl
(46)
where 21 is weak augmentation of and T2 is strong augmentation. This modification encourages us to choose A= == as the original Fixmatch implementation used À = We also remark that this modification degrades the performance of Fixmatch (less than 2%) reported in the work of (Shnn et al. (222)). However, including strong augmentations in the supervised part greatly improves the performance of the Complete Case.
39

---

Published as a conference paper at ICLR 2023
O CIFAR AND SVHN: OLIVER ET AL. (2018) IMPLEMENTATION OF CONSISTENCY-BASED MODEL.
In this section, we present the results on CIFAR and SVHN by debiasing the implementation of OOliver et al. 2018) of II-Model, Mean-Teacher and VAT We mimic the experiments of Oliver et al. (2018) figure-4) with the same configuration and the exact same hyperparameters (Oliver et al.) 2018 Appendix B and C). We perform an early stopping independently on both cross-entropy and accuracy. As reported below, we reach almost the same results as the biased methods.
0.1 CIFAR-10
Grrgkc
Puf stde
Cone pirca Cose
deleas Tesche
Figure 14: Test accuracy for each SSL approaches on CIFAR-10 with various amounts labelled data ny (Left) I1-model and Dell-model. (Right) VAT+EntMin and VAT+DeEntMin, (Bottom) Mcan-ter teacher and DeMean- teacher. Shadows represent 95% CI.
WT REND
15
Mou Tre
Cord Mi Lais
Figure 15: Test cross-entropy for each SSL approaches CIFAR-10 with various amounts of labelled data (Leff) II-model and Dell-model. (Right) VAT+En VAT+DeEmtMin. (Bottom) Mean-teacher and DeMean-teacher. Shadows represent 9,5% CI.
https:://githubb.omm//rrr httes://gtthhao.commmmareeeeeeee istiiaaiiiiiiiaaoa -e--sss--vva uuation
40

---

Published as a conference paper at ICLR 2023
O.2 SVHN
WT FUNK
425
Figure 16: Test accuracy for each SSL approaches on CIFAR-10 with various amounts labelled data ny (Left) II-model and Dell-model. (Right) VAT+EntMin and VAT+DeEntMin, (Bottom) Mcan-test teacher and DeMean- teacher. Shadows represent 95% CI.
Masi
Cum Core
Foaster
Figure 17: Test cross-entropy for each SSL approaches on CIFAR-10 with various amounts of labelled data (Ie(f) TI-model and Dell-model. (Right) VnT+EntMin and VAT+DeEntMin. (Bottom) Mean-teacher and CoMean-teacher Shadows represent 95% Cl.
41

---

Published as a conference paper at ICLR 2023
P COMPUTATION DETAILS
P.1 COMPUTATION RESOURCES
Deep Learning experiments of this work required approximately 11,250 hours of GPU computation. In particular, Fixmatch was trained using 4 GPUs. Here are the details:
MNIST: 300 hours
medMNIST: 3 hours
CIFAR-10: 525 hours
CIFAR-100: 1500 hours
Fixmatch 3000 hours
Realistic SSL evaluation on both CIFAR and SVHN: 5880 hours
P.2
COMPUTATION LIBRARIES AND TOOLS
Python Van Rossum & Drake Jr
(995)
PyTorch Paszke al. (0119)
TensorFlow Abadi et al. 2015
Scikit-leamn Pedregosa el al.
Seabomn Waskom et al.
2017)
Python imaging library Lundh et (2012)
Numpy Harris al. (2020)
Pandas McKinney al. 2010)
RandAugment(Cuhuu Cubuk et al. (2020)
Fixmatc--Pytorch|]
RealisticSSLL-evatuttio Oliver et al. (2018)
42


Reviews:
[{"review_id": "t2FAwx7HQc", "reviewer": "Reviewer_6iPj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to give theoretical analysis about the safeness of semi-supervised learning, and proposed a debiased method that can be applied to the classical semi-supervised learning objectives. Experimental results show the proposal can improve the performance of FixMatch algorithm.", "summary_of_the_review": "Based on the above discussion, the paper is not ready to be published, so I tend to reject this paper.\n--------------------------------\n\nThe author's responses address my concerns. Based on the author's efforts and other reviewers' comments,  I change the score to borderline accept.\n", "strength_and_weaknesses": "Strength:\n1) The proposal is clear and easy to understand.\n2) The proposed debias method is general and can be applied to various semi-supervised learning methods.\n\nWeakness:\n1) The safety of semi-supervised learning means its performance will be better than simple supervised learning. Why the deep SSL, such as FixMatch will be unsafe? From the experimental results of other SSL methods, it seems they do not suffer from this problem.\n2) The theoretical analysis seems did not give the guarantee that the performance of the proposed method will be better than the supervised learning method. So it may be inconsistent with the author's claim.\n3) The experimental results are not convincing. The author should apply the proposal to more SSL methods (such as PL, MixMatch, FixMatch, ReMixMatch, UDA, etc) and more commonly adopted datasets (such as STL-10, SVHN, Image-Net).\n4) The paper writing needs to be improved. For example, Figure 1 needs to be reorganized.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "iZEv7a9-Nu", "reviewer": "Reviewer_1cgk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Empirical Risk Minimization (ERM) is an established technique and is one of the basis of supervised (plus some other types) learning. The ERM objective is to minimize the risk when fitting a set of labeled data points. In the context of Semi Supervised Learning (SSL), as we need to deal with data points with and without labels, existing conventional approaches combine (usually through some weighting) two independent losses; however, they lack concrete theoretical guarantee (one of the claims of this paper). One of the the objectives of this paper is to address this issue and ensure the unbiased (in the context of data distribution) semi supervised learning.\n \nTraditional SSL techniques (such as low density separation and consistency regularization based) heavily depend on labeled data points and sometimes overweight them when compared to the corresponding unlabeled set. This may look obvious as unlabeled data come with label uncertainty (compared to labelled data). The hypothesis is even unlabeled data come with uncertain labels; they shouldn’t be weighted poorly (compared to labeled data) to learn a proper model.\n \nIn this work (DeSSL), the SSL problem has been formulated as s ERM problem which also includes a debiasing term to control/minimize the labeled data bias. The main hypothesis is unlabeled data should be taken as complementary not the competitor of the available labeled data. The debiasing term can be thought as a regularizer to penalize higher weights to labeled data and thus control the data bias.\n \nThe proposed approach has been tested on CIFAR-10, CIFAR-100 and compared against FixMatch. Reported results are found to be encouraging.", "summary_of_the_review": "I have gone through the paper more than once including the appendices. Overall, the debiasing idea is quite sound, well articulated, and the document is found easy to follow. Learning by balancing the contribution from both labeled and unlabelled data is well supported through experiments and  corresponding results. Some important SSL theoretical derivations have made the paper even stronger. ", "strength_and_weaknesses": "Strength: The proposed DeSSL model is based on the hypothesis that existing SSL techniques put less weight on unlabeled data and more on the labelled set and thus bias the learning. If this biasing problem can be solved, it is likely that SSL model performance will improve. The debiasing methodology, presented in this work, comes with a set of theorems and corresponding proofs that make the paper theoretically strong. The experiment results, especially the boosted worst-class  performance, signals that  the debiasing idea is working to an extent (for classification tasks). \n \nThe proposed approach has been tested on CIFAR-10, CIFAR-100 and compared against FixMatch, a known technique in SSL.  The most interesting result is the DeSSL worst-case accuracy, which is noticeably better than FixMatch for both CIFAR-10 and 100 benchmarks although the overall accuracy gain for CIFAR-100 was marginal.\n\nAn additional advantage is that the proposed  DeSSL method can be easily applied to some of the available SSL algorithms with minimal changes/efforts.\n\nWeakness: Although the results reported in section 4 look encouraging it is difficult to judge whether the reported gains are statistically significant. It is suggested that authors perform some statistical tests to measure if the performance gains are statistically significant.\n\nThe overall accuracy gain for CIFAR-100 was marginal (compared to CIFAR-10) which signals scalability limitations of the approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XxkpnTzH7f", "reviewer": "Reviewer_NKdq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work identifies that the existing approaches in semi-supervised learning minimize a biased risk and hence, are devoid of theoretical guarantees unless there are strong assumptions. This works gives a method to de-bias the loss with a simple estimator which also allows them to give theoretical guarantees on the risk. They also evaluate their debiased versions of existing semi supervised learning approaches on various datasets and show that they lead to improved calibration and improved accuracy in certain settings.", "summary_of_the_review": "The idea is interesting, simple and natural with theoretical guarantees. The paper is very well written. But, I think the experimental section is weak as discussed above.", "strength_and_weaknesses": "Semi-supervised learning is an interesting area and coming up with theoretically sounds methods is an important problem. The idea of debasing the risk is simple and interesting and also leads to an unbiased estimator of the risk with theoretical guarantees. The toy example provided is appealing. Their method also leads to improved calibration and better accuracy on certain subgroups.\n\nThe main weakness is that their method does not lead to improved accuracy on many of the datasets. It would be nice to have some discussion of the properties of the dataset which leads this method to outperform others in terms of accuracy.  They also show that the debiased version of Fixmatch leads to better accuracy but not the debased version of Pseudolabels. Is there some reason for why this is the case? It would also be good to discuss why their method leads to better accuracy on certain subgroups when using Fixmatch. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T23PTp7wOa-", "reviewer": "Reviewer_DVba", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes DESSL, a general methodology to debias the risk estimate of the SSL objective under the MCAR assumption (Equation 5). Authors conducted comprehensive theoretical investigation of the approach, including its connection with control variates and constrained optimization, its finite-sample guanrantee in estimation variance, its asymptotic guanrantee in calibration, consistency and normality, as well as its generalization bound in terms of Radamacher complexity (Section 3). Authors applied DESSL to popular SSL methods (EntMin, PseudoLabel and FixMatch) and tested them on simple benchmarks (MNIST, CIFAR, MedMNIST), showing clear improvement over baselines.", "summary_of_the_review": "This work proposes a simple approach to improve risk objective estimation in SSL. The method is simple to implement, backed by comprehensive theory, and is sufficiently evaluated on standard benchmarks. I believe the experiments should be extended to very some of the key theorems in practice (e.g., variance reduction and calibration), but otherwise the quality is sufficient to pass the ICLR bar.", "strength_and_weaknesses": "Strength: \n* A simple debiasing method with strong and comprehensive theoretical backing. \n\nWeakness:\n\nI am rather happy with the simplicity of the method and the theoretical backing of the approach, and just have some suggestions in terms of experiment evaluation, and in terms of adding theoretical comments around assumption violation.\n\n* Missing intrinsic evaluation: Authors derived variance-reduction and calibration property of the method. Consequently, I do expect to see some experiment results on risk estimation quality (bias and variance) and calibration property (e.g., measured by proper scoring rule) to validate the theoretical argument in practice.\n* Missing theoretical comments / empirical discussion on how the method may perform if MCAR is violated.\n* Investigation is restricted to simple academic benchmarks, evaluation on realistic large-scale benchmarks (e.g., ImageNet) with rich data distribution and large number of output categories is missing. (Although I believe the current empirical evaluation is sufficient given the theoretical contribution of the paper).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster