Doubly-Robust Self-Training
Authors: Banghua Zhu
Mingyu Ding
Philip Jacobson
Ming Wu
Wei Zhan
Michael Jordan
Jiantao Jiao
Venue: NeurIPS 2023 poster (2023)
Keywords: semi-supervised learning
self-training
auto-labeling
self-labeling
doubly robust

Abstract: Self-training is a well-established technique in semi-supervised learning, which leverages unlabeled data by generating pseudo-labels and incorporating them with a limited labeled dataset for training. The effectiveness of self-training heavily relies on the accuracy of these pseudo-labels. In this paper, we introduce doubly-robust self-training, an innovative semi-supervised algorithm that provably balances between two extremes. When pseudo-labels are entirely incorrect, our method reduces to a training process solely using labeled data. Conversely, when pseudo-labels are completely accurate, our method transforms into a training process utilizing all pseudo-labeled data and labeled data, thus increasing the effective sample size. Through empirical evaluations on both the ImageNet dataset for image classification and the nuScenes autonomous driving dataset for 3D object detection, we demonstrate the superiority of the doubly-robust loss over the self-training baseline.

Doubly Robust Self-Training
Banghua Zhu Department EECS
Mingyu Ding Department EECS
Philip Jacobson Department EECS
UC Berkeley banghua@berkeley. edu
UC Berkeley mydknggberkeley..dd
UC Berkeley philip_.accbsnbberrlleeeyy.ce
Ming Wu Department of EECS
Wei Zhan Department of EECS UC Berkeley wzhan@berkeley.ee edu
Michael Jordan Department of EECS
UC Berkeley mingmu@berkelley.ee edu
UC Berkeley jordan@berkeley.ee edu
Jiantao Jiao Department of EECS UC Berkeley iantao@berkeley. edu
Abstract
Self-training is an important technique for solving semi-supervised learning problems. It leverages unlabeled data by generating pseudo-labels and combining them with limited labeled dataset for training. The effectiveness of self-training heavily relies the accuracy of these pseudo-labels. In this paper, we introduce doubly robust self-training, novel semi-supervised algorithm that provably balances between two extremes. When the pseudo-labels are entirely incorrect, our method reduces to training process solely using labeled data. Conversely, when the pseudo-labels are completely accurate, our method transforms into a training process utilizing all speudo-labeleds data and labeled data, thus increasing the effective sample size. Through empirical evaluations on both the ImageNet dataset for image classification and the nuScenes autonomous driving dataset for 3D object detection, we demonstrate the superiority of doubly robust loss over the standard self-training baseline.
1 Introduction
Semi-supervised learning considers the problem of learning based small labeled dataset together with large unlabeled dataset. This general framework plays an important role in many problems in machine learning, including model fine-tuning, model distillation, self-training. transfer learning and continual learning (Zhu, 2005; Pan and Yang, 2010; Weiss et al., 2016; Gou et al., 2021; De Lange al., 2021). Many of these problems also involve some form of distribute shift, and accordingly, to best utilize the unlabeled data, an additional assumption is that one has access to teacher model obtained from prior training. important to study the relationships among the datasets and the teacher model. In this paper, we ask the following question:
Given teacher model, large unlabeled dataset and small labeled dataset, how can we design a principled learning process that ensures consistent and sample-efficient learning of the true model?
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Self-training is one widely adopted and popular approach in computer vision and autonomous driving for leveraging information from all three components (Lee, 2013; Berthelot al., 2019b.a; Sohn et al., 2020a; Xie et al., 2020; Jiang et al., 2022; Qi et al., 2021). This approach involves using teacher model to generate pseudo-labels for all unlabeled data, and then training a new model on a mixture of both pseudo-labeled and labeled data. However, this method can lead to overreliance on the teacher model and can miss important information provided by the labeled data. As a consequence, the self-training approach becomes highly sensitive to the accuracy of the teacher model. Our study demonstrates that even the simplest scenario mean estimation, this method can yield significant failures when the teacher model lacks accuracy.
To overcome this issue, we propose an alternative method that is doubly robust-whne the covariate distribution of the unlabeled dataset and the labeled dataset matches, the estimator is always consistent no matter whether the teacher model accurate or not. On the other hand, when the teacher model an accurate predictor, the estimator makes full use of the pseudo-labeled dataset and greatly increases the effective sample size. The idea is inspired by and directly related to missing-date inference and causal inference (Rubin, 1976; Kang and Schafer, 2007; Birhanu et al., 2011: I; Ding and Li, 2018), to semiparametric mean estimation (Zhang et al., 2019), and to recent work on prediction-powered inference (Angelopoulos et al., 2023).
1.1 Main results
The proposed algorithm is based for self-training. Assume that we are given set of unlabeled samples, D {2,, Xm.. drawn from fixed distribution Px. set of labeled samples D2 {(Xm+1, 1m+1), (Xm+2, Ym+2),.... "*,")))),.,,,t, Ym+ {n+n)), Px x Py|x, and a teacher model f. Let v) be pre-specified loss function that characterizes the prediction error of the estimator with parameter 0 on the given sample (X,Y) Y). Traditional self-training aims minimizing the combined loss for both labeled and unlabeled samples, where pseudo-labels for unlabeled samples are generated using f:
simple modification of the standard loss
drawn from some joint distribution
m Σ
m++n Σ i=m+1
SL (0)
f(x,))
lo Y.)
m + n
Note that this can also be viewed as first using f to predict all the data, and then replacing the originally labeled points with the known labels:
CSL D2
(0)
1
m++n
fo(x),f(X.))
1
Σ Co(X,, f(Xi)) + m + n
m++-
Σ
lo Yi).
m +
m + n
i=++1
i=++1
Our proposed doubly robust loss instead replaces the coefficient 1/(m + +n) with 1/n in the last two terms:
CD Di,D.
(0)
1
Σ + Eo f(x.)) - Σ
m++
lo (o(x,, + Σ
m+n
Y.),
1m++
imm++1
This seemingly minor change has major beneficial effect the estimator becomes consistent and doubly robust.
Theorem (Informal). Let H* be defined as the minimizer arg min, E(x.)) xFv (Xo(X,Y), Under certain regularity conditions, we have
6*
[V m+n'
when Y f(x),
V O
D..DD (00) S
otherwise.
m
On the other hand, there exists instances such that | 1 D. (0') 2 2 C. always holds true no matter how large m. are.
The result shows that the true parameter is also a local minimum of the doubly robust loss, but not a local minimum of the original self-training loss. We flesh out this comparison for the special example of mean estimation in Section 2, and present empirical results on image and driving datasets in Section 3.

---

1.2 Related work
Missing-dan data inference and causal inference. The general problem of causal inference can be formulated missing-data inference problem as follows. For each unit in experiment, at most one of the potential outcomes- the one corresponding to the treatment to which the unit is viewed as missing (Holland, 1986; Ding
exposed is observed, and the other potential outcomes and Li, 201 Two of the standard methods for solving this problem are data imputation Rubin (1979) and propensity score weighting Rosenbaum and Rubin (1983). A doubly robust causal inference estimator combines the virtues of these two methods. The estimator is referred to as "doubly robust" due to the following property: if the model for imputation is correctly specified then it is consistent no matter whether the propensity score model is correctly specified; on the other hand, if the model propensity score model is correctly specified, then it consistent no matter whether the model for imputation is correctly specified (Scharfstein et al., 1999; Bang and Robins, 2005; Birhanu et al., 2011; Ding and Li, 2018).
We note in passing that double machine learning is another methodology that is inspired by the doubly robust paradigm in causal inference (Semenova et al., 2017; Chernozhukov et al., ba,b; Foster and Syrgkanis, 2019). The problem in double machine learning is related to the classic semiparametric problem of inference for low-dimensional parameter in the presence of high-dimensional nuisance parameters, which is different goal than the predictive goal characterizing semi-supervised learning.
The recent work of prediction poolioedd inference (Angelopoulos et al., 2023) focuses on confidence estimation when there are both unlabeled data, labeled data, along with a teacher model. Their focus is the inferential problem of obtaining confidence set, while ours is the doubly robust property of point estimator. Since they focus on confidence estimation, an important, strong, yet biased baseline point-estimate algorithm that directly combines the ground-truth labels and pseudo-labels is not considered in their case. In our paper, we show with both theory and experiments that the proposed doubly-robust estimator achieves better performance than the naive combination of ground-truth labels and pseudo-labels.
Self-training. Self-training is a popular semi-supervised learning paradigm in which machine- generated lseudo-labels are used for training with unlabeled data (Lee, 2013; Berthelot et al., 2019b,a, Sohn et al., 2020a). To generate these pseudo-labels, a teacher model is pre-trained on a set of labeled data, and its predictions on the unlabeled data are extracted as pseudo-labels. Previous work seeks to address the noisy quality of pseudo-labels in various ways. MixMatch (Berthelot et al., 2019b) ensembles pseudo-labels across several augmented views of the input data. ReMixMatch (Berthelot et al., 2019a) extends this by weakly augmenting the teacher inputs and strongly augmenting the student inputs. FixMatch (Sohn et al., 2020a) uses confidence thresholding to select only high-quality pseudo-labels for student training.
Self-training has been applied in both 2D computer vision problems (Liu et al., 202 Jeong et al., 2019; Tang et al., 2021; Sohn et al., 2020b; Zhou et al., 2022) and 3D problems (Park al., 2022; Wang et al., 2021: Li et al., 2023; Liu et al., 2023) object detection. STAC (Sohn et al., 2020b) enforces consistency between strongly augmented versions of confidence- filtered pseudo-labels. Unbiased teacher (Liu et al., 202 a) updates the teacher during training with exponential moving average (EMA) of the student network weights. Dense Pseudo-Labe (Zhou et al., 2022) replaces box pseudo-labels with the raw output features of the detector to allow the student to learn richer context. In the 3D domain, 3DloUMatch (Wang et al., 2021) thresholds pscudo-labels using a predicetd Intersection-over Uvinnn (IoU). DetMatch (Park et al., 2022) performs detection in both the 2D and 3D domains and filters pseudo-labels based on 2D-3D correspondence. HSSDA (Liu et al., 2023) extends strong augmentation during training with patch-based point cloud shuffling augmentation. Offboard3D (Qi et al., 2021) utilizes multiple frames of temporal context to improve pseudo-label quality.
There has been limited amount of theoretical analysis of these methods, focusing on semi-supervised methods for mean estimation and linear regression (Zhang et al., 2019; Azriel et al., 2022). Our analysis bridges the gap between these analyses and the doubly robust estimators in the causal inference literature.
3

---

2 Doubly Robust Self-Training
We begin with the where the marginal distributions for the covariates of the labeled and unlabeled datasets are the same. Assume that we are given a set of unlabeled samples, D1 = {_1,, Xm drawn from fixed distribution 1Px supported on X, a set of labeled samples D2 = {(Xm+1, xm++) 1), (Xm+2, mm+2), ... (Xm+n, {n+n)), drawn from some joint distribution
Px x Prix supported on X' V. and pre-trained model, f X' y. Let (o(-,-) X y I be a pre-specified loss function that characterizes the prediction error of the estimator with parameter 0 the given sample (X, Y). Our target is to find some o E θ that satisfies
0 arg min ((x,Y))PxxPxv (X) Age
For given loss (x, y), consider naive estimator that ignores the predictor f and only trains on the labeled samples:
m. Σ (o(X,, Yi). π i=++1
D, (
Although naive, this is a safe choice since it is an empirical risk minimizer. As n + oo, the loss converges to the population loss. However, it ignores all the information provided in and the unlabeled dataset, which makes it inefficient when the predictor is informative.
On the other hand, traditional self-training aims at minimizing the combined loss for both labeled and unlabeled samples, where the pseudo-labels for unlabeled samples are generated using f:!
1
m++
CSL Dr.Dz (0)
Σ
XX,, f(X))+
Σ (X,
m-+ Σ
m lo lo(x,, ((x))
Σ
Σ
Co Eo(X.,
+ m + n
Co Yt).
m +
As is shown by the last equality, the self-training loss can be viewed as first using f to predict all the samples (including the labeled samples) and computing the average loss, then replacing that part of the loss corresponding to the labeled samples with the loss on the original labels. Although the loss uses the information arising from the unlabeled samples and f. the performance can be poor when the predictor is not accurate.
We propose an alternative loss, which simply replaces the weight 1/(m + n)) in the last two terms with 1/n:
m
1 mtto
m mt+ Σ 1
CDR (0) D.
m + n Σ (o, f(x,))
Σ TI 1m
(X f(x))
Lo Y2). (1)
As we will show later, this is a doubly robust estimator. We provide an intuitive interpretation here:
In the case when the given predictor is perfectly accurate, i.c., f(X) Y always holds (which also means that X|X- = I is a deterministic function of 2), the last two terms cancel, and the loss minimizes the average loss, min Ext"" (o(X,, f(X.)), on all of the provided data. The effective sample size is + compared with effective sample size n for training only on labeled dataset using CLL In this case, the loss CDR is much better than CL and comparable to LSL.
We may well relax the assumption of f(X) Y to (I, f(x)) E (X, Y).. As n grows larger, the loss is approximately minimizing the average loss 1- f(..).
traditional self seff-training loss. For example, Xie et al. (2020) introduce an extra weight (m + n2/n on the labeled samples, and add noise to the student model; Sohn et al. (2020a) use confidence thresholding to filter unreliable pseudo-labels. However. both of these alternatives still suffer from the inconsistency issue. In this paper focus the simplest form ELL
There are several variants

---

On the other hand, no matter how bad the given predictor is, the difference between the first two terms vanishes as either of m, n goes to infinity since the labeled samples X n arise from the same distribution as XI. Xm. Thus asymptotically the loss minimizes Di mm Ea (X,, Y:), which discards the bad predictor f and focuses only the labeled dataset. Thus, in this case the loss CDR is much better than C and comparable LTL
This loss appropriate only when the covariate distributions between labeled and unlabeled samples match. In the case where there is distribution mismatch, we propose an alternative loss; see Section 2.3.
2.1 Motivating example: Mean estimation
As concrete example, in the case of one-dimensional mean estimation we take () = (0-Y)? Our target is to find some 8* that satisfies
0r
arg min Fxxx))=xxxxxxx [0 ?
8
One can see that 0 = [[ In this case, the loss for training only on labeled data becomes
m+n
(0) (1)=
n Σ
(.)
i=++1
Moreover, the optimal parameter is ÔTL D m Y. which is a simple empirical average over all observed Y's.
For given pre-existing predictor f, the loss for self seff-training becomes
m
m2 m++
SL D. (0)
((x)()
Σ
Y2)
m + n
It is straightforward to see that the minimizer of the loss is the unweighted average between the unlabeled predictors f(X;)'s and the labeled Y/s::
m+n Σ
05
Y
m + n
1=1
1=m+1
In the case of >> n, the mean estimator is almost the same as the average all the predicted values on the unlabeled dataset, which can far from 0* when the predictor f is inaccurate.
On the other hand, for the proposed doubly robust estimator, we have
CDR DD,,D,(0)
m+n
Σ f(x)))²
m+n
Σ (
-(x)))
1
M+n
Σ (0 Y;)
m + n
n
n
i=++1
i=m+1
m-aa
Σ (0-
f(-)() 2(x.))
Σ 2(f(x)) -Y)0 f(x.)²
i=++1
Note that the loss is still convex, and we have
min
0R -
f(x)
(f(x))
i=++1
This recovers the estimator preictioon-oowete inference (Angelopoulos et al., 2023). Assume that is independent of the labeled data. We can calculate the mean mean-squared error the three estimators as follows.
Proposition 1. Let Var[f(X)= - (()[=E(((x----)))) - v B[(f(x) Y)}3 We have
E[( 0T 1
Var[Y)]
n
2m2
2m
2n
E(( 0s S (m + n)²
B((f(X) 2 +
Varlf(X) YY - (m+n)- Var[Y],
(m + n)²
E[(0" - - 2| < min
Var[Y]] + m 2n
m 2n Varlf(X) Y m + n Var Var[f(X)),
Varly]
(m + n)n
(m n)m
5

---

The proof is deferred to Appendix F. The proposition illustrates the Nouble-robustness. of 0DR no matter how poor the estimator f(X) is, the rate is always upper bounded by 2. (Var[y]] + Var[f(X)]]
On the other hand, when f(X) is an accurate estimator of (i.e., [xr - LJ small), the rate can be improved to m+n Var[Y]. In contrast, the self- self-training loss always has non-vanishing term, 1.2.Vaa
2m (m++))² [l( - Y) when vn > unless the predictor f is accurate.
On the other hand, when f(xx = - + 31 is a linear predictor trained on the labeled data with 3 = arg ( 1/2 Di=+t (3 (11 X; + 31 Y)), our estimator reduces to the semi- supervised mean estimator in Zhang et al. (2019). Let X = [1, x]. In this case, we also know that the self-training reduces to training only on labeled data, since ÔTL is also the minimizer of self-training loss. We have the following result that reveals the superiority the doubly robust estimator compared to the other two options.
Proposition ((Zhang et al., 2019)). We establish the asymptotic behavior of various estimators when f is linear predictor trained on the labeled data:
Training only on labeled data ÔTL is equivalent to self-training ÖSL, which gives unbiased estimator but with larger variance:
vn(ōnL- 8*) + N((O,E -B
Doubly Robust ŌDR is unbiased with smaller variance:
valōOR - 0²) - N0, E((Y - 3 x)
n m + n ((-1) E3
Here B= arg ming E[(Y- 33 2 and = E((x x T
2.2 Guarantee for general loss
In the general case, we show that the doubly robust loss function continues to exhibit desirable properties. In particular, n, goes to infinity, the global minimum of the original loss is also critical point of the doubly robust loss, matter how inaccurate the predictor f.
Let l" be the minimizer of Ep [X, Y)]. Let f be pre-existing model that does not depend on the datasets D1, D2. We also make the following regularity assumptions.
Assumption 1. The loss to (z. y) is differentiable 6* for any x. y-
Assumption 2. The random variables (X, and Volo(X,Y) have bounded first and second moments.
Given this assumption, we denote Σn EY- Cov/ oox,Voo((X,f(()) Volo(X, and let Σ/}
Cov|Volo(X. (X, Σ Cov\Volo(X,
Theorem 2. Under Assumptions and 2, we have that with probability least 5,
Ve. (0') S C'mi C'inn (12bb llay
+ +112/-11 1122
(m + nnjō
nd.
E2/- l2
d (m + n288
d +V
+
E
1l
d
no
no
where C/ a universal constant, and CDR
D,.D, is defined in Equation (1).
The proof is deferred to Appendix G. From the example of mean estimation we know that one can design instances such that V SL (f 2 C for some positive constant C.
When the loss VoLDR F p
is convex, the global minimum of V VDCDR D2 converges to 0* as both m, n
go infinity. When the loss VoCDR, is strongly convex, it also implies that ||0 0* 0*112 converges to zero both m, go infinity, where 0 is the minimizer of V.CD, VDCDR

---

When ff is perfect predictor with f(x) = Y (and Y = x is deterministic), one has D., (6*)
m+n - LØ Y.). The effective sample size is m + r instead of n in D1. < D2 (0).
When f is also trained from the labeled data, one may apply data splitting to achieve the same guarantee up to a constant factor. We provide further discussion Appendix
2.3 The case of distribution mismatch
We also consider the case in which the marginal distributions of the covariates for the labeled and unlabeled datasets are different. Assume in particular that we are given a set of unlabeled samples, D1= = [X1. X2, ... Xm), drawn from fixed distribution Px, set of labeled samples, D2 ((Xm+1: Y m+1). (Xm+2, mm+2). ... (Xm+n: Yn+n)). drawn from some joint distribution Qx x Pylx. and a pre-trained model f. In the case when the labeled samples do not follow the same distribution as the unlabeled samples, we need introduce an importance weight x (x:). This yields the following doubly robust estimator:
1 m Σ
D..D, D2 (0)
lo f(x.))
Σ 1=m+1
to(X,, f(x:)) +
Σ
Co (X, Yi).
xT (X,))
(X,))
s=m++
Note that we not only introduce the importance weight æ, but we also change the first term from the average of all the m + n samples to the average of n samples.
Proposition 3. We have BICD,,, (0) Exx,y (X, as long as one of the following two assumptions hold:
For any x, (()) (x(w)
(x(=))
For any x, Lo( f(x)) By- Y)
The proof is deferred to Appendix H. The proposition implies that as long as either π or f is accurate, the expectation of the loss is the same that of the target loss. When the distributions for the unlabeled and labeled samples match each other, this reduces to the case in the previous sections. In this case, taking π (x) 1 guarantees that the expectation of the doubly robust loss is always the same as that of the target loss.
Experiments
To employ the new doubly robust loss in practical applications, need to specify an appropriate optimization procedure, in particular one that is based on (mini-batched) stochastic gradient descent observed
so to exploit modern scalable machine learning methods. In preliminary experiments that directly minimizing the doubly robust loss in Equation (1) with stohastic gradient can lead to instability, and thus, we propose instead to minimize the curriculum-basee loss in each epoch:
D C. D. (0)
1
m++
Σ
(x(X,,fXx)) or
Σ i=
m+n
lo(X,, f(x,)
m++
Σ (o(X,,))) Y.)
m + n
7
7
As we show in the experiments below, this choice yields stable algorithm. We set a₁ t/T, where T is the total number of epochs. For the object detection experiments, we introduce the labeled samples only in the final epoch, setting epoch. Intuitively, we start from the training with samples only from the pseudo-labels, and gradually introduce the labeled samples in the doubly robust loss for fine-tuning.
0 for all epochs before setting = 1 in the final
We conduct experiments on both image classification task with ImageNet dataset (Russakovsky et al., 2015) and 3D object detection task with autonomous driving dataset nuScenes (Caesar et al., 2020). The code is available https :/ //github.. com/ dingmyy/Doubll yeeellyoooo -oooubly--obuut--o SeretsSSefffrrriiie
3.1 Image classification
Datasets and settings. We evaluate our doubly robust self-training method on the 1ma dataset, which contains random subset of 100 classes from ImageNet-Ik (Russakovsky et al., 2015).
7

---

4/4
24
50
461
734
75
Top-1 Apc (%)
45
Top-5 Ace (%)
70
16
37.0
02.
35-
30
54.0
25.88
28.7
Labeled Only
Labeled Only
23,3
Psuedo Only
50
Psuedo Only
20 19g
Psoudo
42.
Doubly-Rohnn (Duns
Doubly-Rbuut (Ours)
15 18,0
16.0
10
30
60
100
20
30
50
100
Labeled Data Fraction (%)
Labeled Data Fraction (96
(a) Top-1 on DaViT
(b) Top-5 on DaViT
4h
74.9
45-
26.5
Top-I Ace (%)
40
37.0
320
40g
Top-5 Ace Acp-SAce (%)
70-
35
30
362
366
385
37.6
60
58.1
21.
18
Labeled Only
50
Labeled Only
25
20.5
Paundo Only
Psuedo Only
20
4
Labeled Poende
40
484
Labeled Prando
Roubly-Rolust
Roubly (Ous)
184
10
30
40
60
80
90
100
20
30
50
60
70
80
90
100
Labeled Data Fraction (%6
Labeled Data Fraction (%)
(c) Top-1 on ResNe150
(d) Top-5 on ResNet50
Figure I: Comparisons ImageNet 100 using two different network architectures. Both Top-1 and Top-5 accuracies are reported. All models are trained for 20 epochs.
with 120K training images (approximately 1,200 samples per class) and 5,000 validation images (50 samples per class). To further test the effectiveness of our algorithm in low-data scenario, we create a dataset that we refer to as mini-ImageNett00 by randomly sampling 100 images per class from ImageNet100. Two models were evaluated: (1) DaViT-T (Ding et al., 2022), a popular vision transformer architecture with state-of- the-art performance ImageNet, and (2) ResNet50 (He et al., 2016), a classic convolutional network to verify the generality of our algorithm.
Baselines. To provide comparative evaluation of doubly robust self- eeff-rrainnng, we establish three baselines: (1) 'Labeled Only' for training on labeled data only (partial training set) with loss CTL (2) 'Pseudo Only' for training with pseudo labels generated for all training samples, and (3) 'Labeled + Pseudoo for a mixture of pseudo-labell and labeled data, with the loss CSL, See the Appendix for further implementation details and ablations.
Results ImageNet100. We first conduct experiments on ImageNet100 by training the model for 20 epochs using different fractions of labeled data from 1% to 100%. From the results shown in Fig. I. we observe that: (1) Our model outperforms all baseline methods on both two networks by large margins. For example, we achieve 5.5% and 5.3% gains (Top-1 Acc) on Da over the "Labeled + Pseudo' method for 20% and 80% labeled data, respectively. (2) The 'Labeled + Pseudo" method consistently beats the 'Labeled Only' baseline. (3) While 'Pseudo Only' works for smaller fractions of the labeled data (less than 30%) on DaVIT, inferior to 'Labeled Only' on ResNet50.
Results on maie-0m We also perform comparisons on mini-ImageNet1000 to demonstrate the performance when the total data volume is limited. From the results in Table 1, we see our model generally outperforms all baselines. As the dataset size decreases and the number of training epochs increases, the gain of our algorithm becomes smaller. This is expected, as (1) the models are not adequately trained and thus have noise issues, and (2) there are an insufficient number of ground truth labels to compute the last term of our loss function. extreme cases, there is only one labeled sample (1 (%) per class.
3.2
3D object detection
Doubly robust object detection. Given a visual representation of a scene, 3D object detection aims generate set of 3D bounding box predictions (b and set of corresponding class predictions {c)))e()n) Thus, each single ground-truth annotation Y, e Y is set Y, (b,, c.)
8

---

Table 1: Comparisons on mini Mini--mageNet1O0 all models trained for 100 epochs.
Labeled Only
Pscudo Only
Labeled
Pseudo
Doubly robust Loss
Labeled Duta Percent
Top1
Top5
topl
top5
topl
top5
topl
top5
2.72
2.81
9.57
2.73
9.55
2.75
9.73
3.92
13.34
4.27
13.66
4.27
4.89
16,38
6.76
20.84
7.27
21.64
7.65
22.48
21.90
20
12.3
31.3
13.46
30.79
13.94
32.63
13.50
32.17
50
20.69
46.86
20.92
45,2
24.9
50.77
25.31
51.61
80
27.37
53.57
25.57
50.85
30.63
58.85
30.75
59.41
100
31,07
60.62
28.95
$5.35
34.33
62.78
34.01
63.04
Table 2: Performance comparison on nuScenes val set.
Labeled Only
Labeled + Pseudo
Doubly robust
Labeled Data Fraction
mAP1
NDS
mAPt
NDSt
mAPt
NDS1
1/24
7.56
18.01
7.60
17.32
8.18
18.33
1/16
11.15
20.55
11.60
21.03
12,30
22.10
1/4
25.66
41.41
28.36
43.88
27.48
43.18
containing a box and class. During training, the object detector supervised with a sum of the box regression loss Cio and the classification loss Cats. i.c. Lobj Ltoc + Cets-
In the self-training protocol for object detection, pseudo-labels for given scene X; are selected from the labeler predictions f(;;) based some user-defined criteria (typically the model's detection confidence). Unlike in standard classification or regression, Y, will contain a differing number of labels depending the number of objects in the scene. Furthermore, the number of extracted pseudo-labels (X;) will generally not be equal to number of scene ground-truth labels Y, due to false positive/negative detections. Therefore it makes sense to express the doubly robust loss function terms of the individual box labels as opposed to the scene-level labels. We define the doubly robust object detection loss follows:
M+NNa
Σ ((xx.,((x.)) 1 M+N,, Σ 6o(x ((X) No 1=++1
M+N Σ 1=++1
1
CDR (0)
l Yi),
M Np
N
where M is the total number of pseudo-label boxes from the unlabeled split, N is the total number of labeled boxes, X/ the scene with pseudo-label boxes from the labeled split, and Nps is the total number of pseudo-label boxes from the labeled split. We note that the last two terms now contain summations over differing number of boxes, consequence of the discrepancy between the number of manually labeled boxes pseudo-labeled boxes. Both components of the object detection loss focalization/classifccaa
adopt this form of doubly robust loss.
Dataset and setting, To evaluate doubly robust self- self-training in the autonomous driving setting, we perform experiments on the large-scale 3D detection dataset nuScenes (Caesar et al., 2020). The nuScenes dataset comprised of 1000 scenes (700 training. 1550 validation and 150 test) with each frame containing sensor information from RGB camera, LiDAR, and radar scans. Box annotations are comprised of 10 classes, with the class instance distribution following a long-tailed distribution, allowing us to investigate detection metrics for nuScenes are mean Average Precision (mAP) and the nuScenes Detection Score (NDS), a dataset-specific metric consisting of weighted average of mAP and five other true-positive metrics. For the sake of simplicity, we train object detection models using only LiDAR sensor information.
self-training approach for both common and rare classes. The main 3D
Results. After semi-supervised training, we evaluate our student model performance the nuScenes val set. We compare three settings: training the student model with only the available labeled data
Table 3: Per-class mAP (%) comparison on nuScenes val set using 1/16 of total labels in training.
Car
Ped
Truck
Bus
Trailer
Barrier
Traffic Cone
Labeled Only
48.6
30.6
8.5
6.2
4,0
6.8
4.4
Labeled Pseudo
48.8
30.9
8.8
7.5
4.0
Improvement
+0.2
+0.3
+0.3
+1.3
+1.7
-0.1
-0.4
Doubly robust Loss
51.5
32.9
9.6
8.2
5.2
7.2
4.5
Improvement
+2.9
+2.3
+1.1
+2.0
+1.2
+0.4
+0.1

---

(i.e., equivalent to teacher training), training the student model on the combination of labeled/teacher- labeled data using the naive self-training loss, and training the student model on the combination of labeled/tecche--lbeleee data using our proposed doubly robust loss. We report results for training with 1/24, 1/16, and 1/4 of the total labels in Table 2. We find that the doubly robust loss improves both mAP and NDS over using only labeled data and the naive baseline in the lower label regime, whereas performance is slightly degraded when more labels are available. Furthermore, we also show per-class performance breakdown in Table 3. We find that the doubly robust Ioss consistently improves performance for both common (car, pedestrian) and rare classes. Notably, the doubly robust loss is even able to improve upon the teacher in classes for which pseudo-label training decreases performance when using the naive training (e.g., barriers and traffic cones).
Conclusions
We have proposed a novel doubly robust loss for self-training. Theoretically, analyzed the double-robustness property of the proposed loss, demonstrating its statistical efficiency when the pseudo-labels are accurate. Empirically, we showed that large improvements can be obtained in both image classification and 3D object detection.
As direction for future work, it would be interesting to understand how the doubly robust loss might be applied to other domains that have a missing- data aspect, including model distillation, transfer learning, and continual learning. It is also important to find practical and efficient algorithms when the labeled and unlabeled data do not match in distribution.
Acknowledgements
Banghua Zhu and Jiantao Jiao are partially supported by NSF IIS-1901252, CIF-1909499 and CIF- 221 1209. Michael I. Jordan is partially supported by NSF 19S-1902 Philip Jacobson is supported by the National Defense Science and Engineering Graduate (NDSEG) Fellowship. This work is also partially supported by Berkeley DeepDrive.
10

---

References
A. N. Angelopoulos, Bates, Fannjiang, M. I. Jordan, and T. Zmic. crediction-powertss inference. arXiv preprint arXhv:2301.09333, 2023.
E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN). IEFE, 2020.
D. Azriel, L. D. Brown, M. Sklar, R. Berk, A. Buja, and L. Zhao. Semi Bemi-supervised linear regression. Journal of the American Statistical Association, 117(540):2238 222122211112,,,,, 2022.
H. Bang and M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4)9662-9) 973, 2005.
D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 2019a.
D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel. Mixmatch: A holistic approach semi-supervised learning. arXiv preprint arXiv:1905.02244 2019b.
J. M. Bibby, K. V. Mardia, and J. T. Kent. Multivariate Analysis. Academic Press, 1979.
T. Birhanu, G. Molenberghs, C. Sotto, and M. G. Kenward. Doubly robust and based generalized estimating equations. Journal of Biopharmaceutical Statistics, 21(2):202-22 2011.
H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters, 2018a.
Chernozhukov, W. Newey, and R. Singh. De-biased machine learning of global and local parameters using regularized Riesz representers. arXiv preprint arXiv: 180200867, 2018h.
M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. continual learning survey: Defying forgetting in classification tasks. IEEE transuctions on pultern analysis and machine intelligence, 44(7):3366-3385, 2021.
M. Ding, B. Xiao, N. Codella, P. Luo, J. Wang, and L. Yuan. Davit: Dual attention vision transformers. In Computer Vision-ECCY 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 74 92. Springer, 2022.
P. Ding and F. Li. Causal inference. Statistical Science, 33(2):214-237, 2018
D. J. Foster and V. Syrgkanis. Orthogonal statistical learning, arXiv preprint arXiv:1901.0900 2019.
J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819, 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
P. W. Holland. Statistics and causal inference. Journal of the American Statistical Association. 81 (396):945-96 9960,,96,111 1986.
Jeong, Lec, J. Kim, and N. Kwak. Consistency- semi-supervised learning for object detection. In Advances in Neural Information Processing Systems, 2019.
C. M. Jiang, M. Najibi, C. R. Qi, Y. Zhou, and D. Anguelov. Improving the intra-class long-tail tail in 3d detection via rare example mining. In Computer VisionFECCV 2022: 17th European Conference, pages 158-175. Springer, 2022.

---

J. D. Kang and J. L. Schafer. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22:523-539, 2007.
A. H. Lang, S. Vora, H. Caesar, L. Zhou, Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12689-12697, 2019.
D.H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. JCML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Li, Z. Liu, Hou, and D. Liang. Dds3d: Dense pseudo-labele with dynamic threshold for semi- supervised 3d object detection. In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), 2023.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and Dollár. Focal loss for dense object detection. In ICCV, pages 2980-20000 2988, 2017.
C. Liu, Gao, F. Liu, P. Li, Meng, and X. Gao. Hierarchical supervision and shuffle data augmentation for 3d semi-supervised object detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Y.-C. Liu, C.-Y. Ma, Z. He, C.-W. Kuo, K. Chen, P. Zhang, B. Wu, Z. Kira, and P. Vajda. Unbiased teacher for semi superviper object detection. In Proceedings of the International Conference Learning Representations (ICLR), 2021a.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711005101, 2017.
A. W. Marshall and I. Olkin. Multivariate chebyshev inequalities. The Annals of Mathematical Statistics, pages 1001-1014, 1960.
S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 22110)1345-1135 2010.
J. Park, C. Xu, Y. Zhou, M. Tomizuka, and W. Zhan. Detmatch: Two teachers are better than one for joint 2D and 3D semi-supervised object detection. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, 2022.
C. R. Qi, Y. Zhou, M. Najibi, P. Sun, K. Vo, B. Deng, and D. Anguelov. Offboard 3D object detection from point cloud sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-555 1983.
D. B. Rubin. Inference and missing data. Biometrika, 63(3):581-59) 1976.
D. B. Rubin. Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association, 74(366a):318 1-888, 1979.
O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 11:2211225,, 2015.
D. O. Scharfstein, A. Rotnitzky, and J. M. Robins. Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94(448): 1096-1120, 1999.
V. Semenova, M. Goldman, V. Cheroozuuov, and M. Taddy. Estimation and inference on heterogeneous treatment effects in high-dimensional dynamic panels. arXiv preprint 17Xiv:1712.099888 2017.
12

---

L. N. Smith and N. Topin. learning rates, In Artificial Intelligence and Machine Learning for Multi-Doman Operations Applications, volume 1006, page 1100612. International Society for Optics and Photonics, 2019.
Very fast training of neural networks using large
K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel. Fixmatch: Simplifying semi- super-ieee learning with consistency and confidence. arXiv preprint arXiv:2001.07685, 2020a.
K. Sohn, Z. Zhang, C.-L. Li, H. Zhang, C.-Y. and T. Pfister. A simple sumi-super learning framework for object detection. In arXiv:2005.04757, 2020b.
Y. Tang, W. Chen, Y. Luo, and Y. Zhang. Humble teachers teach better students for semi-supervised object detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3131-3140, 2021.
H. Wang, Y. Cong, O. Litany, Y. Gao, and I.. Guibas. 3dioumatch: Leveraging iou prediction for semi-supervised 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14615-14624 2021.
K. Weiss, M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big Data, (1):1-40, 2016.
Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
T. Yin, X. Zhou, and P. Krähenbühl. Center-based 3D object detection and tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
A. Zhang, D. Brown, and T. Cai. Semi-supervised inference: General theory and estimation of means. Annals of Statistics, 47:2538-2566 2019.
H. Zhou, Z. Ge, S. Liu, W. Mao, Z. Li, H. Yu, and J. Sun. Dense teacher: Dense pseudo-labels for semi-supervised object detection. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision ECCV 2022, pages 35-50, 2022.
B. 7hu, 7. Jiang, X. Zhou, 7. Li, and G. Yu. Class grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.0944 2019.
X. J. Zhu. Semi-supervised learning literature survey. 2005.
13

---

A Implementation Details for Image Classification
We evaluate doubly robust self-training method the ImageNer110 and 0n
datasets, which are subsets of ImageNet-1k from ImageNet Large Scale Visual Recognition Challenge 2012 (Russakovsky al., 2015). Two models are evaluated: (1) DaViT-T (Ding et al., 2022), state-of-the-art 12-layer vision transformer architecture with a patch size of 4, a window size of and an embedding dim of 768, and (2) ResNer50 (He al., 2016), classic and powerful convolutional network with 50 layers and embedding dim 2048. We evaluate all the models on the same ImageNet100 validation set (50 samples per class). For the training, we use the same data augmentation and regularization strategies following the common practice in Liu et al. (2021b); Lin et al. (2017); Ding al. (2022). We train all the models with batch size of 1024 on 8 Tesla V100 GPUs (the batch size is reduced to 64 if the number of training data is less than 1000). We use A (Loshchilov and Hutter, 2017) optimizer and a simple triangular learning rate schedule (Smith and Topin, 2019). The weight decay is set to 0.05 and the maximal gradient norm is clipped to 1.0. The stochastic depth drop rates are set 0.1 for all models. During training, we crop images randomly to 224 x 224, while center crop is used during evaluation on the validation set. We use a curriculum setting where the or grows linearly or quadratically from to I throughout the training. To show the effectiveness of our method, we also compare model training with different curriculum learning settings and varying numbers of epochs.
B Additional Experiments in Image Classification
Table Ablation study on different curriculum settings on ImageNet- 100. All models are trained in
20 epochs.
30% GTss
50%
GTs
70% GTs
90% GTss
Methods
top1
top5
top1
top5
top1
top5
top1
top5
Naive Labeled + Pscudo doubly robust, os
28.01
54.63
37.6
66.72
43.76
73.42
47.74
77.15
28.43
56.65
38.06
67.18
43.22
73.18
48.52
77.21
doubly robust, du = 1/T (lincar)
30.87
60.98
40.18
71.06
46.60
75,80
50.44
78.88
doubly robust, (t/TT) (quadratic)
31.15
61.29
40.86
71.14
45,50
75.11
49.64
77.77
Ablation study on curriculum settings. There are three options for the curriculum setting: 1) a, 1 throughout the whole training, 2) grows linearly with training iterations a, t/T, 3) grows quadratically with training iterations O₂ = (t/T)². From results in Table 4, we see: the first option achieves comparable performance with the 'Naive Labeled Pseudo' baseline. Both the linear and quadratic strategies show significant performance improvements: the linear one works better when more labeled data is available, e.g., 70% and 90%, while the quadratic one prefers less labeled data, e.g. 30% and 50%.
Table 5: Ablation study on the number of epochs. All models are trained using 10% labeled data on ImageNet- 100.
Labeled Only
Pseudo Only
Labeled Pseudo top5
doubly robust Loss
Training epochs
top1 top5
top1
top5
topl
top1
top5
20
16.02
39.68
17.02
38.64
19.38
41.96
21.88
47,18
50
25.00
51.21
28.90
53.74
30.36
57.04
36.65
65.68
100
35.57
64.66
44.43
71.56
42.44
68.94
45.98
70.66
Ablation Study on the Number of Epochs. We conduct experiments on different training epochs. The results are shown in Table 5. Our model is consistently superior to the baselines. And we can observe the gain is larger when the number of training epochs is relatively small, e.g. 20 and 50.
Fully trained results (300 epochs) on ImageNet-100. In our original experiments, we mostly focus on a teacher model that is not super accurate, since our method reduces to the original pseudo-labeling when the teacher model is completely correct for all labels. In this experiment, we fully train the teacher model with 300 epochs on ImageNet- 100, leading to the accuracy of the teacher model as high as 88.0%. From Figure 2, we show that even in this case, our method outperforms the original pseudo-labeling bascline.
14

---

Figure 2: Results ImageNet-100 using fully trained (300 epochs) DaViT-T with different data fractions.
90.6
88.0
Top-1 Ace (%)
85
80
70-
67.7
Labeled Only
64.8
Labeled Pscudo
63.00
Doubly-Rousst (Ours)
20
50
80
100
Labeled Data Fraction (%)
Comparisons with previous SOTAs on CIFAR-10 and CiFAR-100. We compare with another 11 baselines in terms of error rate on CIFAR-I0-4K and CIFAR-100-100 under the same settings (i.e., Wide ResNet-28-22 for CIFAR-10 10 and WRN-28-8 for CIFAR-100), We show that our method is only 0.04 inferior to the best method Meta Pscudo Labels for CIFAR- 10-4K, and achieves the best performance for CIFAR- 10K.
Table 6: Comparisons with previous SOTAs on CiFAR- 10 and CIFAR-100.
Method
CIFAR-10-4K (error rate, %)
CIFAR-100-10K (error rate, %)
Pseudo-Labeling
16.09
36.21
LGA + VAT
12.06
Mean Teacher
9.19
35.83
ICT
7.66
SWA
5.00
28.80
MixMatch
4.95
25.88
ReMixMatch
4.72
23.03
5.35
UDA
4.32
24.50
FixMatch
4.31
23.18
Meta Pesudo Labels
3,89
Ours
3.93
22.30
C Implementation Details of 3D Object Detection
Our experiments follow the standard approach for semi-supervised detection: we first initialize two detectors, the teacher (i.e., labeler) and the student. First, random split of varying sizes is selected from the nuScenes training set. We pre-train the teacher network using the ground-truth annotations in this split. Following this, we freeze the weights in the teacher model and then use it to generate pseudo-labels on the entire training set. The student network is then trained on combination of the pscudo-labels and ground-truth labels originating from the original split. In all of our remi-supervised experiments, use CenterPoint with PointPillars backbone as our 3D detection model (Yin et al., 2021; Lang et al., 2019). The teacher pre-training and student training are both conducted for 10 epochs on 3 NVIDIA RTX A6000 GPUs. We follow the standard nuScenes training setting outlined in Zhu et al. (2019), with the exception of disabling ground-trute paste augmentation during training to prevent data leakage from the labeled split. To select the pseudo-labels be used training student, we simply filter the teacher predictions by detection confidence, using all detections above a chosen threshold. We use threshold of 0.3 for classes, as in Park et al. (2022). order to conduct training in batch-wise wise manner, compute the loss over only the samples contained within
15

---

the batch. We construct each batch to have consistent ratio of labeled/unlabeled samples to ensure the loss is well-defined for the batch.
D Additional Experiments in 3D Object Detection
Comparison with Semi-Supervised Baseline We compare our approach to another semi-supervised baseline on the 3D object detection task, Pseudo-labeling and confirmation bias Arazo al. (2020).
We shows that in multiple settings, our approach surpasses the baseline performance.
Ablation on Pseudo-label Confidence Threshold To demonstrate that uppropriate quality pseudo- labels are used to train the student detector, we performance ablation experiments varying the detection threshold used to extract pseudo-labels from the teacher model predictions. We show that training with threshold of r = 0.3 outperforms training with more stringent threshold, and is the appropriate experimental setting for our main experiments.
Table 7: Performance comparison with pseudo-labeling baseline nuScenes val set.
Labeled Only
Labeled + Pscudo
Doubly robust Loss
Pscudo-Labcling Confirmation
Labeled Fraction
mAPt
NDS1
MAPt
NDSt
MAPt
NDS1
MAPT
NDS1
1/24
7.56
18.01
17.32
8.18
18.33
7.80
16.86
1/16
11.15
20.55
11,60
21.03
12.30
22.10
12.15
22.89
Table 8: Doubly Robust Loss performance comparison with differing detection thresholds for pseudo-
labels.
0.3
0.5
Labeled Data Fraction
Labeled+Pseute
Doubly Robust Loss
Labuled+Pseudo
Doubly Robust Loss
MAPt
NDS
mAP1
NDSt
mAPt
NDS1
mAPt
NDS1
1/24
7.56
18.01
8.18
18.33
7.15
15.82
4.37
13.17
1/16
11.15
20.55
12.30
22.10
11.05
21.22
8.09
19.70
E
Considerations when f is trained from labeled data
In Theorem 2, we analyzed the double robustness of the proposed loss function when the predictor pre-existing and not trained from the labeled dataset. In practice, one may only have access to the labeled and unlabeled datasets without pre-existing teacher model. In this split the labeled samples D: into two parts. The last n samples are used to train f. and the first n/2 samples are used in the doubly robust loss:
may choose to
2 m++n2
m+t/22 Σ
DD2 D. (0)
(x f(x.))
Σ 1=m+1
Co Eo f(X,)) T(x))
Co (x, Yi).
((x,)
1=1
1=M+1
Since f independent of samples used the above loss, the result in Theorem continues to hold. Asymptotically, such doubly robust estimator is never worse than the estimator trained only on the labeled data.
F Proof of Proposition 1
For the labeled-only estimator êTL. we have
m+n E[y]
E -
0T E
Y
VYr
1=
16

---

For the self-training loss, we have
m
E((0- - -ss
E
E
Σ f(,))
Σ
Yi
m + n
I 2
E
m E[
m
1
f(X,)
+ E
E|
1 m+n
Σ i=++1
)1
m 1=1
+ n
n
2m
2m
2n
B[(f(x)- - - +
Varlf(X) 11+
Var[Y].
(m n)²
(m n)²
(m + n2)
For the doubly robust loss, on one hand, we have
27 (f(x.) -Yi)
m++
m+++
Ei - { = E
E
f(x.)
+
Σ i=m+1
Yzi)
m +
m++n
27
m+n
27
S 2E
Y
V
2
[[f(x)]
-
Σ f(X.)
n
n i=++1
i=++1
m+n
27
+ 2E
B[f(X)]
f(X;)
m + n
2 VarlY]+
Varlf(X]]-
7
m + n
n1
On the other hand, we have
+++
27
E 0DR) = E
[[]]
f(X,) + n
Σ (f(X.) - xi)
m +
i=m+1
27
27
m+n
(BLJ(X) -
m++n
S 2E
[8[y] m72 + n2
21E
Σ
(f(X))
n
i=1
1=++1
m++
27
+ 2E
[if((X) -Y
1
Σ (f(X,) Yi)
17 + n2
2
2) Varlf(x) Y]
2
Var[Y].
m + n
m + n
The proof is done by taking the minimum of the two upper bounds.
G Proof of Theorem
Proof. We know that
V a C...R
D. (0*) - E CDR (0")]l22
D.
m++ Σ
- +
Vo
(Volo - Vol ((x,))
- Vo (x ((X)1))
mtn
mtn Σ
m (Xl (Xx6,
+1
(Volo (X., (Vbb)(()))))Veecx.X(1(11
- E.Voll (olk)
17

---

From the we have that with probability at least 6/2, for some universal constant C.
Chebyshev inequality (Bibby et al., 1979; Marshall and Olkin, 1960),
1
- /11 (2
m + n2
(m + n2)
1=1
Similarly, we also have that with probability at least 1 - 8//2.
(Vl, YI - Vale-velllll i=++1
(xlx 50112/1111
11 d no
Furthermore, note that
E - (x.
Here we use Assumption and Assumption 2 to ensure that the expectation and differentiation are interchangeable. Thus we have that with probability at least 1 - 8,
VoLDi. sc ( (12dbllyy
d (m nj8
ley
12 1la nd
On the other hand, we can also write the difference as CD
0 D. (00) E (0>]]|22
m+n
m++ Σ
Vola-(X; ((X.) EiVolo (X,
K1)
m + T
m+++
Σ
-
m+
m
m++
(Volo- (X, f(x,) E -
+ +1|
Σ
(Volo Y.) EIVolo (x, YI,
m + n
m-+-
(Vol -X)
- - (X, f(x))))
1=++1
So 1 ll
(
d
(mmm+ +on+ny n)6
+
+
11
no
no
Here the last inequality uses the at least 1 - 8. This finishes the proof.
Chebyshev inequality and it holds with probability
H Proof of Proposition
Proof. We have
E E(CORR (0))
E
Ex (o, [(x)),
Σ
E Ex,exx
lo (o(xx.
x(X,))
m i=1
m+n
Σ
Ex,Nxx
Co (x Yi)
T(x,)
i=m+1
=Ex~py (X, - Ex-Ex
1
lo lolx, (xxx
x(X) x(X)
1
+ Ex-ex. X-Prx
lo (o(X
T(X)
18

---

In the first case when [xx)= = [x(x)
[x) we have
E|CD,, (0)] = Exxxx -ox - Ex-ex
(x(x) lo(xx
Px(X)
Xx
Bo(x.))
+ +EX~Qx,YNVII
Qxx (X)
Exxrx (6o f(x)) - Exaox [(x(x,
+ X (X)
= =Ex,y~xx,y (x,
In the second case when L0(2; f(x)) = By (x,, Y),, we have
E BICD,, (0) Exarx (xx - Ex ((x) Co ((xx,((x))
+ Ex Ey BY T(X) (o(X, X
Ex fx))) - Ex~ex
T(x)
1 (xx, + +xxexx
= =ExaRx (co (X,())
ExYxaxx Y)],
This finishes the proof.
19


Reviews:
[{"review_id": "jER9NTv9Ik", "reviewer": "Reviewer_yYuG", "summary": "This paper proposes a pseudo-labeling approach that balances out the supervised signal between the labeled and incorrect pseudo-labeled datapoints during the training process. The aim is to only account for the pseudo-labels when they are correctly labeled, which may happen when the covariate distribution of the unlabeled dataset and the labeled dataset matches. They show some analysis of the proposed loss and results in ImageNet100 (a subset with 100 random classes from ImageNet-1k) and mini-ImageNet100  and nuScenes dataset.", "questions": "Prior literature [2,3,4,5] has shown that predefined threshold values impact the overall performance in pseudo-labeling. Using 0.3 as the threshold seems to be too permissive, allowing too many noisy pseudo-labels. How sensitive is this threshold in your setup?\n\n[5] Oliver A, Odena A, Raffel CA, Cubuk ED, Goodfellow I. Realistic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems. 2018;31.", "limitations": "No limitation section is provided.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is clearly written, and the paper can easily be understood.\nThe paper with all necessary details and deep explanations for experimental results.\nComprehensive algorithmic analysis and clear motivation.\nThe proposed method effectively improves over vanilla pseudo-labeling [1]\n\n[1] D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL), 07 2013.", "weaknesses": "Novelty and missing prior work: Pseudo-labeling is the defacto method for entropy regularization techniques in semi-supervised learning problems. This is a paper that tackles the semi-supervised learning problem. Recent work has explored different ways to mitigate the error propagation from the teacher model and confirmation bias present in pseudo-labeling approaches. For example, [2,3,4] investigate the thresholding effect via fixed and curriculum based approaches, with flexible thresholds that are dynamically adjusted for each class according to the current learning status. With such prior exploration and no comparison with any of that work, it's difficult to assess the importance and impact of this work, which seems limited and incomplete.\n\nThe paper in its current state also fails to provide technical details to validate fair comparisons in the main text. Furthermore, no ablations for any of the technical selections are conducted.\nThe alternative loss proposed for distribution mismatch (Section 2.4) is only shown in the method but not in the empirical section.\n\n[2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In IJCNN, pages 1–8, 2020.\n\n[3] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6912–6920, 2021.\n\n[4] Zhang B, Wang Y, Hou W, Wu H, Wang J, Okumura M, Shinozaki T. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems. 2021 Dec 6;34:18408-19.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your valuable comments and suggestions. Please find our responses to each comment below.\n\n## Comment 1\n\n**Reviewer:** \n\n\n> Novelty and missing prior work: Pseudo-labeling is the defacto method for entropy regularization techniques in semi-supervised learning problems. This is a paper that tackles the semi-supervised learning problem. Recent work has explored different ways to mitigate the error propagation from the teacher model and confirmation bias present in pseudo-labeling approaches. For example, [2,3,4] investigate the thresholding effect via fixed and curriculum based approaches, with flexible thresholds that are dynamically adjusted for each class according to the current learning status. With such prior exploration and no comparison with any of that work, it's difficult to assess the importance and impact of this work, which seems limited and incomplete.\n\n**Response:** \n\nThank you for your comments! \n\n- We are happy to include comparisons with existing methods [2-4]. Please find the **Table 1 in the uploaded PDF / General Response** as our comparisons with other existing baselines. Due to the long-running time for all methods, for now we only include the comparisons with [2]. We also compare with other existing 11 benchmark algorithms for image classification in Table 1 of the uploaded PDF. One can see that our method still achieves better performance than existing methods. \n\n- Our proposed methodology is **fundamentally different from the existing pseudo-labeling idea in [2-4]**. In [2-4], the effect of thresholding is extensively studied. However, when the teacher model is very inaccurate, one can never expect the confidence of each label is given correctly, and thus the thresholding would not provide gain for this case. In fact, we can show that even in the simplest case of mean estimation, all the methods in [2-4] still fail to provide right solution even with infinite number of samples. \n\n- As an orthogonal approach, we propose a simple loss function that automatically uses the labeled samples to test the validity of the teacher model. It is guaranteed to perfectly interpolate both cases: when the teacher model is completely wrong, we will only use the labeled samples; when the teacher model is completely correct, it will use all pseudo-labels. This is not achievable by the existing algorithms in [2-4] since the confidence on each label might be inaccurate as well. \n\n- Our proposed method is **always unbiased and guaranteed to converge to the right solution, in contrast to any existing methods**. We show theoretically that the existing pseudo-labeling idea will give very poor performance when the teacher model is inaccurate. And only the proposed doubly-robust estimator will remain unbiased even in the simplest case such as mean estimation. Our experimental results validate our theoretical predictions.\n\n- As we also show in the experiments for 3D object detection, our method can be directly combined with any threshold method in [2-4] and improve over the existing threshold-based method. We believe the proposed method is a very important and necessary alternative / add-on for all the existing pseudo-labeling ideas. \n\n\n\n## Comment 2\n\n**Reviewer:** \n\n\n>  The paper in its current state also fails to provide technical details to validate fair comparisons in the main text. Furthermore, no ablations for any of the technical selections are conducted. The alternative loss proposed for distribution mismatch (Section 2.4) is only shown in the method but not in the empirical section.\n\n\n\n**Response:** \n\nThank you for the comments! \n\n- We will add the details about all the hyperparameters in the revised draft.\n\n- We **have ablation studies for both the curriculum settings and the number of epochs** in Appendix B. We have also added the new ablation study in Table 3 for the pseudo-labeling threshold in the uploaded new PDF file, we will add more data points and ablations on the original pseudo-labeling methods in the final revision. \n\n- The alternative loss for distribution mismatch requires the knowledge of the importance ratio between the target distribution and the original data distribution. We are happy to include simulated results. However, estimating such importance ratio in practical scenarios can be hard. \n\n\n\n## Comment 3\n\n**Reviewer:** \n\n\n> Prior literature [2,3,4,5] has shown that predefined threshold values impact the overall performance in pseudo-labeling. Using 0.3 as the threshold seems to be too permissive, allowing too many noisy pseudo-labels. How sensitive is this threshold in your setup?\n\n\n**Response:** \n\nThank you for your comments! We have added the new ablation study for the pseudo-labeling threshold in **Table 3 of the uploaded rebuttal PDF / General Response**. One can see that if we improve the threshold from 0.3 to 0.5 to allow less noisy pseudo-labels, the performance indeed downgrades. We will include more data points and the ablations for the original pseudo-labeling method in the final revision. \n\n\\\nWe wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments! Thank you!"}, {"review_id": "XUxSzWYf50", "reviewer": "Reviewer_AyWm", "summary": "The authors propose a very simple yet effective modification to the original loss for self-training by re-weighting terms of the loss function making. This change effectively balances between using the pseudo-labels when the predictor is strong and learning to not use it when it is unreliable, making it doubly robust. They provide a sound theoretical analysis and  empirical evaluation on classification and object detection tasks substantiating their claims.", "questions": "* In Section 2.1, the intuitive interpretation for when the predictor is bad is not super clear and paper would benefit from some elaboration. If just m goes to infinity, how would the two terms cancel out\n\n* In the case of distribution mismatch, what’s the motivation for changing the first term to average of (n) sample instead of (m+n)? Also how are $\\pi (x)$ ie importance weights chosen?\n\n* Did the authors explore tuning the predictor $\\hat{f}$ on the labelled examples before generating the pseudo-examples to train the student?\n\nTypos and possible errors\n- Inconsistent notation : line 155 should be $\\hat{\\theta}$ instead of $\\theta^{*}$\n- In line 161, I believe the upper bound should be $6/n(var[Y] ..) $ instead of $4/n$. Please verify!\n- in equation in line 171, it should be $\\theta_{SL}$ instead of $\\theta_{DR}$", "limitations": "The authors have adequately addressed some limitations. Other suggestions are listed above.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Strengths\n    - The paper is well written and easy to follow with sufficient background and motivated examples given to present the chain of reasoning well.\n    - For linear predictor, the proposed loss is unbiased with lower variance which is strictly better than self-training\n    - The results on both classification and 3D object detection highlight clear improvement over standard self distillation\n    - The technical novelty in the paper is limited with just a small modification to the overall loss. But the theoretical insights including guarantees for general loss and positive experimental results make it a meaningful contribution. The simplicity of the modification also make it much more likely to be adopted and have higher impact.", "weaknesses": "- Weaknesses\n    - The analysis provided is for very simplistic settings of linear predictor or mean-predictions and it’s unclear how much of it translates to realistic settings of over-parameterized deep nets trained on SGD.\n    - For image classification, the baselines used in the paper are meaningful but thorough comparison with other state of the art self-imitation methods like noisy-student etc is lacking.\n    - The evaluation is restricted to lower data regimes as the model shines when the training data is limited, making their evaluation a bit more contrived as compared to real world scenarios. It seems the proposed loss should shine more when unlabelled data scales, so some experimentation on the impact on performance as the unlabelled data scales would also be interesting to see\n    - One important use of self-distillation is using unlabelled data to show domain adaptation to that domain. I would like to see experiments for that as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We have corrected all the typos suggested. Please find our responses to each comment below.\n\n## Comment 1\n\n**Reviewer:** \n\n\n> The analysis provided is for very simplistic settings of linear predictor or mean-predictions and it’s unclear how much of it translates to realistic settings of over-parameterized deep nets trained on SGD.\n\n**Response:** \n\nThank you for your comments!  We would like to clarify that we have a **guarantee for arbitrary loss (including deep neural network) in Theorem 2 in Section 2.3**. This shows that even in the case of deep learning, our proposed algorithm still converges to a good point. In contrast, the existing method fails even in the simplest case of mean estimation. Our section 2.2 on mean estimation is only a motivating example for the general case.\n\n\n\n## Comment 2\n\n**Reviewer:** \n\n\n>  For image classification, the baselines used in the paper are meaningful but thorough comparison with other state-of-the-art self-imitation methods like noisy-student etc is lacking.\n\n\n\n**Response:** \n\nThank you for the comments! We have included comparisons with other baselines in the uploaded new PDF, which compares against another 12 baseline algorithms in CIFAR-10 and CIFAR-100 dataset. We will add more comprehensive comparisons in the revision as well.  \n\n\n\n## Comment 3\n\n**Reviewer:** \n\n\n> One important use of self-distillation is using unlabelled data to show domain adaptation to that domain. I would like to see experiments for that as well.\n\n\n**Response:** \n\nThank you for your comments! We have included theoretical results on the right algorithm for distribution mismatch. However, the proposed method requires knowledge about the importance ratio. Making the proposed algorithm practically implementable still remains an open problem. \n\n## Comment 4\n\n**Reviewer:** \n\n\n> In Section 2.1, the intuitive interpretation for when the predictor is bad is not super clear and paper would benefit from some elaboration. If just $m$ goes to infinity, how would the two terms cancel out?\n\n\n\n**Response:** \n\nThank you for your comments! The finite-sample guarantee when none of $m, n$ goes to infinity is given in Section 2.2 for mean estimation, and Section 2.3 for general loss functions. In short, if only $m$ goes to infinity, there will an additional noise introduced whose standard deviation is proportional to $1/\\sqrt{n}$. This can be seen from the third equation in Proposition 1, or Theorem 2.\n\n\n## Comment 5\n\n**Reviewer:** \n\n\n>  In the case of distribution mismatch, what’s the motivation for changing the first term to average of $(n)$ sample instead of $(m+n)$? Also how are the importance weights chosen? \n\n\n**Response:** \n\nThank you for your comments! Sorry here is a minor typo. We would like to change the first term from the average of $(m+n)$ samples to the average of $m$ samples. This is due to that the marginal distribution of the first $m$ (unlabeled) samples is different from the distribution of the last $n$ (labeled) samples. If we mix both distributions together, the resulting distribution won't be the same as the distribution of the unlabeled samples. And in this case, we cannot use let the first two terms cancel each other due to distribution mismatch. Thus in the first term, we only take average with the $m$ unlabeled samples. In practice, the importance weights are required to be known or estimated from the representations of the features $X$.\n\n## Comment 6\n\n**Reviewer:** \n\n\n>  Did the authors explore tuning the predictor on the labelled examples before generating the pseudo-examples to train the student?\n\n\n\n**Response:** \n\nThank you for your comments! Yes, we select the best checkpoint for the predictors on the labeled examples. And then generate the pseudo-examples to train the students. This applies to both the baseline algorithms and our proposed algorithm. \n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window."}, {"review_id": "aJUQvph4SJ", "reviewer": "Reviewer_bmAm", "summary": "The paper proposed a doubly robust loss for self-training. The proposed loss is analysed and shown to have preferable theoretical properties. ", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting: a simple change from 1/(m+n) to 1/n (line 51 - 53) lead to a doubly robust loss function for self-training. \n2. The writing is clear and easy to follow", "weaknesses": "1. While the proposed doubly robust loss for self-training enjoys theoretical advantages, directly minimizing the loss during network training leads to instability. The actual loss used in line 219 is very different especially in the early epochs. \n\n2. Lack of comparison to other stronger semi-supervised learning baselines. For example, the authors discuss MixMatch and FixMatch as related work but did not compare with them in experiments.\n\nminor:\n1. I don't think the description in line 79-87 is precise. MixMatch/FixMatch does not pre-trained a teacher model on 'labeled' data. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We have corrected all the typos suggested. Please find our responses to each comment below.\n\n## Comment 1\n\n**Reviewer:** \n\n\n> While the proposed doubly robust loss for self-training enjoys theoretical advantages, directly minimizing the loss during network training leads to instability. The actual loss used in line 219 is very different especially in the early epochs.\n\n**Response:** \n\nThank you for your comments! We would like to mention that the actual loss is an interpolation between two losses: in the early epochs, it is close to the original pseudo-labeling method, which first uses the pseudo-labels to learn a student model that is similar to the teacher model. In the later epochs, it utilizes the new proposed loss function to correct the learned student model. We find that it stabilizes the training greatly. \n\n\n## Comment 2\n\n**Reviewer:** \n\n\n> Lack of comparison to other stronger semi-supervised learning baselines. For example, the authors discuss MixMatch and FixMatch as related work but did not compare with them in experiments.\n\n\n\n**Response:** \n\nThank you for the comments! We have included comparisons with other baselines in **the uploaded new PDF / General Response**, which compares against MixMatch, FixMatch, and another 9 baselines on CIFAR-10, CIFAR-100 datasets. We will add it in the revision as well.  \n\n\n\n## Comment 3\n\n**Reviewer:** \n\n\n> I don't think the description in lines 79-87 is precise. MixMatch/FixMatch does not pre-trained a teacher model on 'labeled' data.\n\n\n\n**Response:** \n\nThank you for your suggestion! We have corrected our paper to make this part precise. MixMatch / FixMatch are methods that do not rely on pre-training a teacher model on the labeled dataset. \n\n\\\nThanks again for your time and effort! For any other questions, please feel free to let us know during the rebuttal window."}, {"review_id": "6NNbugpBMP", "reviewer": "Reviewer_8nT5", "summary": "This paper presents a very simple approach to semi-supervised learning that utilizes both labeled and unlabeled datasets. When there is a large amount of unlabeled data available, following the same distribution as the labeled dataset, the most effective method to leverage this unlabeled data for training is self-training, where pseudo labels are generated and used for training. The main limitation of self-training is that the performance can degrade when the pseudo-labels produced by the predictor for the unlabeled set are not accurate. \nIn this paper, the authors propose a very simple method to overcome this limitation by replacing the conventional loss for the labeled dataset and the unlabeled dataset in self-training. This modification allows the model to be trained effectively in both cases, whether the pseudo-labels are correct or not, leading to improved performance in all scenarios. The effectiveness of this approach is experimentally demonstrated on image classification benchmarks and 3D object detection benchmarks.", "questions": "see above.", "limitations": "Future work is discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, the method is very simple, intuitive, and quite novel. The paper is well-written and it provides a well-derived explanation using equations for both cases where the pseudo labels are accurate and when they are not. Additionally, the method is thoroughly analyzed from a theoretical perspective.", "weaknesses": "Although the theoretical derivation demonstrates the soundness of the method, there is doubt regarding its effectiveness in experiments.\n\n1. Especially, in the experiments (sec. 3.1), the authors use curriculum-based loss in each epoch. With $\\alpha_t  < 1 $, the proposed method will behave like self-training (exactly when $\\alpha_t = \\frac{n}{m+n}$) and the behavior/effectiveness of the proposed method could not be well represented in this case.  \n\n2. Image classification experiments were conducted using ImageNet-100. However, when the labeled set ratio is 100%, the top-1 accuracy of DaVIT and ResNet50 is remarkably lower at 47.8% and 46.7%, respectively, compared to the reported top-1 accuracy in other papers. Following the paper of DaVit and ResNet, the top-1 accuracy on the more complex task of ImageNet1k, compared to ImageNet-100, using DaVit-tiny and ResNet is 82.8% and 79.26%, respectively. Following [1], the top-1 accuracy of various resnet50-based methods for ImageNet-100 consistently surpasses 70%. It appears that the baseline has not been sufficiently well-trained.\n\n3. There is a lack of comparison with other semi-supervised methods. While this method compares with the basic self-training loss, it is necessary to compare it with various methods that utilize pseudo-labels for self-training [2], [3], [4]. Most semi-supervised image classification methods have performed experiments on benchmark datasets such as ImageNet1k and CIFAR100. These methods achieve a top-1 accuracy of 72% or higher when using a labeled dataset of 10% on ImageNet1k. Therefore, it is necessary to compare the effectiveness of this method on these benchmark datasets.\n\n4. Similarly, in semi-supervised 3D object detection, it is necessary to compare the results with existing baseline methods [5], [6].\n\n5. Although this paper proposes a method for the distribution mismatch case in Section 2.4, estimating the probability values of each sample, p(x) and q(x), for each distribution is not easy. Practical methods for this issue have not been provided, and experiments on this aspect are also lacking. Practical algorithms and experiments need to be provided in order to address this concern.\n\n[1] Zelin Zang, et al. DLME: Deep Local-flatness Manifold Embedding. ECCV 2022\n\n[2] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 2020\n\n[3] Kihyuk Sohn, David Berthelot, Zizhao Li, Chun-Liang Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In IEEE Conference on Computer Vision and Pattern Recognition, 2020\n\n[4]  Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le. Meta pseudo labels. In IEEE Conference on Computer Vision and Pattern Recognition, 2021\n\n[5] He Wang, Yezhen Cong, Or Litany, Yue Gao and Leonidas J. Guibas. 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D \nObject Detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2021\n\n[6] Na Zhao, Tat-Seng Chua and Gim Hee Lee. SESS: Self-Ensembling Semi-Supervised 3D Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions. We have corrected all the typos suggested. Please find our responses to each comment below.\n\n## Comment 1\n\n**Reviewer:** \n\n\n> In the experiments (sec. 3.1), the authors use curriculum-based loss in each epoch. With $\\alpha_t<1$, the proposed method will behave like self-training (exactly when $\\alpha_t = \\frac{n}{m+n}$) and the behavior/effectiveness of the proposed method could not be well represented in this case.\n\n**Response:** \n\nThank you for your comments! Our curriculum-based loss only matches the original self-training loss when exactly $\\alpha_t=0$ (which reduces to $\\frac{1}{m+n}\\sum_{i=1}^{m+n} \\ell_\\theta(X_i, \\hat f(X_i))$). For any $\\alpha_t>0$, it is an interpolation between the original self-training loss and our proposed loss function. Our experimental results also demonstrate a significant improvement over the original self-training loss with $\\alpha_t$ always equal to $0$.\n\nWe would like to also clarify that even when $\\alpha_t = \\frac{n}{m+n}$, this loss is **very different from self-training loss**. In this case, our loss becomes $\\frac{1}{m+n}\\sum_{i=1}^{m+n} \\ell_\\theta(X_i, \\hat f(X_i)) - \\frac{1}{m+n}\\sum_{i=m+1}^{m+n} \\ell_\\theta(X_i, \\hat f(X_i)) + \\frac{1}{m+n}\\sum_{i=m+1}^{m+n} \\ell_\\theta(X_i, Y_i)$. Note that none of the terms cancel since the second and third term is averaging over the labeled samples and reweighted, while the first term is average over both label and unlabeled samples. In contrast, the original self-training loss is $\\frac{1}{m+n}\\sum_{i=m+1}^{m+n} \\ell_\\theta(X_i, \\hat f(X_i))$. Thus our method is fundamentally different from the original self-training loss whenever $\\alpha_t>0$.\n\n\n## Comment 2\n\n**Reviewer:** \n\n\n>  Image classification experiments were conducted using ImageNet-100. However, when the labeled set ratio is 100%, the top-1 accuracy of DaVIT and ResNet50 is remarkably lower at 47.8% and 46.7%, respectively, compared to the reported top-1 accuracy in other papers. Following the paper of DaVit and ResNet, the top-1 accuracy on the more complex task of ImageNet1k, compared to ImageNet-100, using DaVit-tiny and ResNet is 82.8% and 79.26%, respectively. Following [1], the top-1 accuracy of various resnet50-based methods for ImageNet-100 consistently surpasses 70%. It appears that the baseline has not been sufficiently well-trained.\n\n\n\n**Response:** \n\n- Thank you for your suggestions. **We have included experiments for sufficiently well-trained ImageNet-100 results (training for full 300 epochs) in Figure 1 of the uploaded one-page PDF, and well-trained CIFAR-10-4K, CIFAR-100-10K in Table 1 of the PDF / General Response.** Even in these cases, our method still gives a universal improvement over the baselines. \n\n- Our method gives better improvement when the teacher model is not extremely accurate. If the teacher model is always correct, then our method reduces to the original self-training procedure. This is why previously we focus on the case when the baseline is not sufficiently well-trained. We will add more discussions and all the comparisons including well-trained case and less-trained case in the revision.\n\n\n\n## Comment 3\n\n**Reviewer:** \n\n\n> There is a lack of comparison with other semi-supervised methods. While this method compares with the basic self-training loss, it is necessary to compare it with various methods that utilize pseudo-labels for self-training [2], [3], [4]. Most semi-supervised image classification methods have performed experiments on benchmark datasets such as ImageNet1k and CIFAR100. These methods achieve a top-1 accuracy of 72% or higher when using a labeled dataset of 10% on ImageNet1k. Therefore, it is necessary to compare the effectiveness of this method on these benchmark datasets. Similarly, in semi-supervised 3D object detection, it is necessary to compare the results with existing baseline methods [5], [6].\n\n\n\n**Response:** \n\nThank you for your suggestion! **We have included comparisons with other 11 SOTA baselines in Table 1 the uploaded new PDF.** Overall, our method still gives the best improvement, and can be combined with existing baselines. We will add the results and more discussions in the revision as well.  \n\n## Comment 4\n\n**Reviewer:** \n\n\n> Although this paper proposes a method for the distribution mismatch case in Section 2.4, estimating the probability values of each sample, p(x) and q(x), for each distribution is not easy. Practical methods for this issue have not been provided, and experiments on this aspect are also lacking. Practical algorithms and experiments need to be provided in order to address this concern.\n\n**Response:** \n\nThank you for your comments! We agree that estimating the importance ratio can be hard for practical scenarios when we don't have access to the marginal distributions. We are happy to provide simulated results for this case. However, we would also like to point out that most of the self-training pipeline is based on the setting when there is no distribution mismatch between the labeled and unlabeled samples. Our proposed method makes a first attempt towards provable method to address the distribution mismatch phenomenon, and explains the name `doubly-robust' in statistics. However, we will mark that the proposed method for distribution shift may not be practical due to the difficulty in estimating the importance ratio, which is left as an open problem for future research.\n\n\\\nWe wish that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments! Thank you!"}]

Author rebuttal:
We would like to thank the reviewers for the valuable comments and suggestions, which help us greatly improve our paper. Besides individual responses, we summarize the revision and new experimental results in the one-page PDF uploaded. We also include the markdown table for your reference, which are the same results as the PDF.

### Additional Experiment 1 (Table 1): Comparisons with previous SOTAs on CIFAR-10 and CiFAR-100

In Table 1 of the uploaded PDF, we compare with **another 11 baselines** in terms of error rate on CIFAR-10-4K and CIFAR-100-10K under the same settings (i.e., Wide ResNet-28-2 for CIFAR-10 and WRN-28-8 for CIFAR-100). We show that our method is only 0.04 inferior to the best method Meta Pseudo Labels for CIFAR-10-4K, and achieves the best performance for CIFAR-100-10K. (The numbers are not reported in some of the methods in CIFAR-100-10K, we will try to re-implement and re-run these methods in the future.)

| Method     | CIFAR-10-4K (error rate, \%) | CIFAR-100-10K (error rate, \%) |
| ------------------ | ------------------- | --------------------- |
| Pseudo-Labeling    | 16.09               | 36.21                 |
| LGA + VAT          | 12.06               | --                    |
| Mean Teacher       | 9.19                | 35.83                 |
| ICT                | 7.66                | --                    |
| SWA                | 5.00                | 28.80                 |
| MixMatch           | 4.95                | 25.88                 |
| ReMixMatch         | 4.72                | 23.03                 |
| EnAET              | 5.35            | --                    |
| UDA                | 4.32                | 24.50                 |
| FixMatch           | 4.31                | 23.18                 |
| Meta Pesudo Labels | **3.89**            | --                    |
| **Ours**           | 3.93                | **22.30**             |
### Additional Experiment 2 (Figure 1): Sufficiently well-trained ImageNet-100

In our original experiments, we mostly focus on a teacher model that is not super accurate, since our method reduces to the original pseudo-labeling when the teacher model is completely correct for all labels. In this experiment, we fully train the teacher model with 300 epochs on ImageNet-100, leading to the accuracy of the teacher model as high as 88.4%. We show that even in this case, our method outperforms the original pseudo-labeling baseline.

|  Data Fraction    |   Labeled Only  (acc, \%) |   Pseudo + Labeled (acc, \%)  |   Ours (acc, \%)  |
| ---- | ---- | ---- | ---- |
|   20   | 63.59 | 64.87 | **67.16** |
|   50   | 81.23 | 82.92 | **85.47** |
|   80   | 85.50 | 86.98 | **87.57** |
|   100   | 88.01 | 89.43 | **90.61** |

### Additional Experiment 3 (Table 2): Comparisons with previous SOTAs on nuScenes object detection dataset

We compare with the idea of pseudo-labeling + confirmation bias in [1]. And show that on the object detection dataset, our method still gives better performance for various labeled data fractions. Due to the time limit, we are still not yet finished all the experiments with varying labeled fractions and other methods. We will include more comparisons in the final revision. 

| Labeled Fraction | Labeled Only (mAP↑) | Labeled Only (NDS↑) | Labeled + Pseudo (mAP↑) | Labeled + Pseudo (NDS↑) | Doubly robust Loss (mAP↑) | Doubly robust Loss (NDS↑) | Pseudo-Labeling + Confirmation Bias (mAP↑) | Pseudo-Labeling + Confirmation Bias (NDS↑) |
|:----------------:|:-------------------:|:-------------------:|:-----------------------:|:-----------------------:|:-------------------------:|:-------------------------:|:-------------------------------------------:|:-------------------------------------------:|
| 1/24       | 7.56        | 18.01       | 7.60        | 17.32         | **8.18**       | **18.33**          | 7.80      | 16.86      |
| 1/16       | 11.15      | 20.55     | 11.60      | 21.03         | **12.30**      | 22.10           | 12.15        | **22.89**     |


### Additional Experiment 4 (Table 3): Ablation on the detection thresholds

We include ablation studies for different detection thresholds, showing that our choice of 0.3 is not too pessimistic. We will include more data points and the results on the original pseudo-labeling methods in the final revision.
| Labeled Data Fraction | τ = 0.3 (mAP↑) | τ = 0.3 (NDS↑) | τ = 0.5 (mAP↑) | τ = 0.5 (NDS↑) |
|:----------------------:|:--------------:|:--------------:|:--------------:|:--------------:|
| 1/24                   | **8.18**       | **18.33**      | 4.37           | 13.17          |
| 1/16                   | **12.30**      | **22.10**      | 8.09           | 19.70          |

### Novelty and Contributions

We would like to remark here that our proposed method is fundamentally different from **all existing pseudo-labeling-based methods.** 
- Our proposed method is the first to have convergence guarantee for any loss functions, including neural networks. (see Theorem 2). While most of the SOTA methods are built up on the original pseudo-labeling idea, which is biased even for the simplest setting of mean estimation.
- Based on the experimental results, our proposed method improves over almost all existing baselines, showing the power of the doubly-robust idea.
- Our method is the first that connects and unifies the idea of doubly robust estimator in statistics for propensity score estimation, the prediction-powered inference idea for confidence estimation, and the self-training idea in knowledge distillation and neural network training. 
- Our method proposes an alternative loss that uses the labeled data to `correct' the unlabeled samples. It can also be combined with any of the existing pseudo-labeling methods to further improve their performance. 

We hope that our pointwise responses below could clarify all reviewers’ confusion. We thank all reviewers’ time again and we are always ready to solve your concerns.

Meta-review: The paper received borderline to positive reviews. The paper proposes a novel methodology for self-supervised learning where the method has some robustness to inaccurate pseudo-labels. The paper provides both empirical and theoretical justifications that most reviewers found compelling. Suggestions were made to discuss and compare to additional methods and the authors have partially addressed this in the rebuttal. In general the paper seems to make some reasonable contribution to address a current weakness in self-supervised learning and the method demonstrates positive results. The paper is recommended for acceptance.
Decision: Accept (poster)