SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Banghua Zhu
Department of EECS UC Berkeley
banghua@berkeley.edu
Mingyu Ding
myding@berkeley.edu
Philip Jacobson
philip_jacobson@berkeley.edu
Ming Wu
mingwu@berkeley.edu
Wei Zhan
Department of EECS, UC Berkeley  
zhan@berkeley.edu
Department of EECS UC Berkeley jordan@berkeley.edu
Jiantao Jiao
jiantao@berkeley.edu
Self-training is an important technique for solving semi-supervised learning problems. It leverages unlabeled data by generating pseudo-labels and combining them with a limited labeled dataset for training. The effectiveness of self-training heavily relies on the accuracy of these pseudo-labels. In this paper, we introduce doubly robust self-training, a novel semi-supervised algorithm that provably balances between two extremes. When the pseudo-labels are entirely incorrect, our method reduces to a training process solely using labeled data. Conversely, when the pseudo-labels are completely accurate, our method transforms into a training process utilizing all pseudo-labeled data and labeled data, thus increasing the effective sample size. Through empirical evaluations on both the ImageNet dataset for image classification and the nuScenes autonomous driving dataset for 3D object detection, we demonstrate the superiority of the doubly robust loss over the standard self-training baseline.
Semi-supervised learning considers the problem of learning based on a small labeled dataset together with a large unlabeled dataset. This general framework plays an important role in many problems in machine learning, including model fine-tuning, model distillation, self-training, transfer learning and continual learning (Zhu, 2005; Pan and Yang, 2010; Weiss et al., 2016; Gou et al., 2021; De Lange et al., 2021). Many of these problems also involve some form of distribute shift, and accordingly, to best utilize the unlabeled data, an additional assumption is that one has access to a teacher model obtained from prior training. It is important to study the relationships among the datasets and the teacher model. In this paper, we ask the following question:
Given a teacher model, a large unlabeled dataset and a small labeled dataset, how can we design a principled learning process that ensures consistent and sample-efficient learning of the true model?
Self-training is one widely adopted and popular approach in computer vision and autonomous driving for leveraging information from all three components (Lee, 2013; Berthelot et al., 2019b,a; Sohn et al., 2020a; Xie et al., 2020; Jiang et al., 2022; Qi et al., 2021). This approach involves using a teacher model to generate pseudo-labels for all unlabeled data, and then training a new model on a mixture of both pseudo-labeled and labeled data. However, this method can lead to overreliance on the teacher model and can miss important information provided by the labeled data. As a consequence, the self-training approach becomes highly sensitive to the accuracy of the teacher model. Our study demonstrates that even in the simplest scenario of mean estimation, this method can yield significant failures when the teacher model lacks accuracy.
To overcome this issue, we propose an alternative method that is doubly robust—when the covariate distribution of the unlabeled dataset and the labeled dataset matches, the estimator is always consistent no matter whether the teacher model is accurate or not. On the other hand, when the teacher model is an accurate predictor, the estimator makes full use of the pseudo-labeled dataset and greatly increases the effective sample size. The idea is inspired by and directly related to missing-data inference and causal inference (Rubin, 1976; Kang and Schafer, 2007; Birhanu et al., 2011; Ding and Li, 2018), to semiparametric mean estimation (Zhang et al., 2019), and to recent work on prediction-powered inference (Angelopoulos et al., 2023).
1.1 Main results
The proposed algorithm is based on a simple modification of the standard loss for self-training. Assume that we are given a set of unlabeled samples,  $\mathcal{D}_1 = \{X_1,X_2,\dots ,X_m\}$ , drawn from a fixed distribution  $\mathbb{P}_X$ , a set of labeled samples  $\mathcal{D}_2 = \{(X_{m + 1},Y_{m + 1}),(X_{m + 2},Y_{m + 2}),\dots ,(X_{m + n},Y_{m + n})\}$ , drawn from some joint distribution  $\mathbb{P}_X\times \mathbb{P}_{Y|X}$ , and a teacher model  $\hat{f}$ . Let  $\ell_{\theta}(x,y)$  be a pre-specified loss function that characterizes the prediction error of the estimator with parameter  $\theta$  on the given sample  $(X,Y)$ . Traditional self-training aims at minimizing the combined loss for both labeled and unlabeled samples, where the pseudo-labels for unlabeled samples are generated using  $\hat{f}$ :
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {S L}} (\theta) = \frac {1}{m + n} \left(\sum_ {i = 1} ^ {m} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) + \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, Y _ {i})\right).
$$
Note that this can also be viewed as first using  $\hat{f}$  to predict all the data, and then replacing the originally labeled points with the known labels:
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {S L}} (\theta) = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) - \frac {1}{m + n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) + \frac {1}{m + n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, Y _ {i}).
$$
Our proposed doubly robust loss instead replaces the coefficient  $1 / (m + n)$  with  $1 / n$  in the last two terms:
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathsf {D R}} (\theta) = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) + \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, Y _ {i}).
$$
This seemingly minor change has a major beneficial effect—the estimator becomes consistent and doubly robust.
Theorem 1 (Informal). Let  $\theta^{\star}$  be defined as the minimizer  $\theta^{\star} = \arg \min_{\theta}\mathbb{E}_{(X,Y)\sim \mathbb{P}_X\times \mathbb{P}_{Y|X}}[\ell_{\theta}(X,Y)]$ . Under certain regularity conditions, we have
$$
\| \nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\text {D R}} (\theta^ {\star}) \| _ {2} \lesssim \left\{ \begin{array}{l l} \sqrt {\frac {d}{m + n}}, & \text {w h e n Y \equiv \hat {f} (X) ,} \\ \sqrt {\frac {d}{n}}, & \text {o t h e r w i s e .} \end{array} \right.
$$
On the other hand, there exists instances such that  $\| \nabla_{\theta}\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{SL}}(\theta^\star)\| _2\geq C$  always holds true no matter how large  $m,n$  are.
The result shows that the true parameter  $\theta^{\star}$  is also a local minimum of the doubly robust loss, but not a local minimum of the original self-training loss. We flesh out this comparison for the special example of mean estimation in Section 2.1, and present empirical results on image and driving datasets in Section 3.
Missing-data inference and causal inference. The general problem of causal inference can be formulated as a missing-data inference problem as follows. For each unit in an experiment, at most one of the potential outcomes—the one corresponding to the treatment to which the unit is exposed—is observed, and the other potential outcomes are viewed as missing (Holland, 1986; Ding and Li, 2018). Two of the standard methods for solving this problem are data imputation Rubin (1979) and propensity score weighting Rosenbaum and Rubin (1983). A doubly robust causal inference estimator combines the virtues of these two methods. The estimator is referred to as "doubly robust" due to the following property: if the model for imputation is correctly specified then it is consistent no matter whether the propensity score model is correctly specified; on the other hand, if the model propensity score model is correctly specified, then it is consistent no matter whether the model for imputation is correctly specified (Scharfstein et al., 1999; Bang and Robins, 2005; Birhanu et al., 2011; Ding and Li, 2018).
We note in passing that double machine learning is another methodology that is inspired by the doubly robust paradigm in causal inference (Semenova et al., 2017; Chernozhukov et al., 2018a,b; Foster and Syrgkanis, 2019). The problem in double machine learning is related to the classic semiparametric problem of inference for a low-dimensional parameter in the presence of high-dimensional nuisance parameters, which is different goal than the predictive goal characterizing semi-supervised learning.
The recent work of prediction-powered inference (Angelopoulos et al., 2023) focuses on confidence estimation when there are both unlabeled data, labeled data, along with a teacher model. Their focus is the inferential problem of obtaining a confidence set, while ours is the doubly robust property of a point estimator. Since they focus on confidence estimation, an important, strong, yet biased baseline point-estimate algorithm that directly combines the ground-truth labels and pseudo-labels is not considered in their case. In our paper, we show with both theory and experiments that the proposed doubly-robust estimator achieves better performance than the naive combination of ground-truth labels and pseudo-labels.
Self-training. Self-training is a popular semi-supervised learning paradigm in which machine-generated pseudo-labels are used for training with unlabeled data (Lee, 2013; Berthelot et al., 2019b,a; Sohn et al., 2020a). To generate these pseudo-labels, a teacher model is pre-trained on a set of labeled data, and its predictions on the unlabeled data are extracted as pseudo-labels. Previous work seeks to address the noisy quality of pseudo-labels in various ways. MixMatch (Berthelot et al., 2019b) ensembles pseudo-labels across several augmented views of the input data. ReMixMatch (Berthelot et al., 2019a) extends this by weakly augmenting the teacher inputs and strongly augmenting the student inputs. FixMatch (Sohn et al., 2020a) uses confidence thresholding to select only high-quality pseudo-labels for student training.
Self-training has been applied in both 2D computer vision problems (Liu et al., 2021a; Jeong et al., 2019; Tang et al., 2021; Sohn et al., 2020b; Zhou et al., 2022) and 3D problems (Park et al., 2022; Wang et al., 2021; Li et al., 2023; Liu et al., 2023) object detection. STAC (Sohn et al., 2020b) enforces consistency between strongly augmented versions of confidence-filtered pseudo-labels. Unbiased teacher (Liu et al., 2021a) updates the teacher during training with an exponential moving average (EMA) of the student network weights. Dense Pseudo-Label (Zhou et al., 2022) replaces box pseudo-labels with the raw output features of the detector to allow the student to learn richer context. In the 3D domain, 3DIoUMatch (Wang et al., 2021) thresholds pseudo-labels using a model-predicted Intersection-over-Union (IoU). DetMatch (Park et al., 2022) performs detection in both the 2D and 3D domains and filters pseudo-labels based on 2D-3D correspondence. HSSDA (Liu et al., 2023) extends strong augmentation during training with a patch-based point cloud shuffling augmentation. Offboard3D (Qi et al., 2021) utilizes multiple frames of temporal context to improve pseudo-label quality.
There has been a limited amount of theoretical analysis of these methods, focusing on semi-supervised methods for mean estimation and linear regression (Zhang et al., 2019; Azriel et al., 2022). Our analysis bridges the gap between these analyses and the doubly robust estimators in the causal inference literature.
2 Doubly Robust Self-Training
We begin with the case where the marginal distributions for the covariates of the labeled and unlabeled datasets are the same. Assume that we are given a set of unlabeled samples,  $\mathcal{D}_1 = \{X_1,X_2,\dots ,X_m\}$ , drawn from a fixed distribution  $\mathbb{P}_X$  supported on  $\mathcal{X}$ , a set of labeled samples  $\mathcal{D}_2 = \{(X_{m + 1},Y_{m + 1}),(X_{m + 2},Y_{m + 2}),\dots ,(X_{m + n},Y_{m + n})\}$ , drawn from some joint distribution  $\mathbb{P}_X\times \mathbb{P}_{Y|X}$  supported on  $\mathcal{X}\times \mathcal{Y}$ , and a pre-trained model,  $\hat{f}:\mathcal{X}\mapsto \mathcal{Y}$ . Let  $\ell_{\theta}(\cdot ,\cdot):\mathcal{X}\times \mathcal{Y}\mapsto \mathbb{R}$  be a pre-specified loss function that characterizes the prediction error of the estimator with parameter  $\theta$  on the given sample  $(X,Y)$ . Our target is to find some  $\theta^{\star}\in \Theta$  that satisfies
$$
\theta^{\star}\in \operatorname *{arg  min}_{\theta \in \Theta}\mathbb{E}_{(X,Y)\sim \mathbb{P}_{X}\times \mathbb{P}_{Y|_{X}}}[\ell_{\theta}(X,Y)].
$$
For a given loss  $\ell_{\theta}(x,y)$ , consider a naive estimator that ignores the predictor  $\hat{f}$  and only trains on the labeled samples:
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {T L}} (\theta) = \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, Y _ {i}).
$$
Although naive, this is a safe choice since it is an empirical risk minimizer. As  $n \to \infty$ , the loss converges to the population loss. However, it ignores all the information provided in  $\hat{f}$  and the unlabeled dataset, which makes it inefficient when the predictor  $\hat{f}$  is informative.
On the other hand, traditional self-training aims at minimizing the combined loss for both labeled and unlabeled samples, where the pseudo-labels for unlabeled samples are generated using  $\hat{f}$ :
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {S L}} (\theta) = \frac {1}{m + n} \left(\sum_ {i = 1} ^ {m} \ell_ {\theta} \left(X _ {i}, \hat {f} (X _ {i})\right) + \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, Y _ {i}\right)\right) \\ = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, \hat {f} \left(X _ {i}\right)\right) - \frac {1}{m + n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, \hat {f} \left(X _ {i}\right)\right) + \frac {1}{m + n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, Y _ {i}\right). \\ \end{array}
$$
As is shown by the last equality, the self-training loss can be viewed as first using  $\hat{f}$  to predict all the samples (including the labeled samples) and computing the average loss, then replacing that part of the loss corresponding to the labeled samples with the loss on the original labels. Although the loss uses the information arising from the unlabeled samples and  $\hat{f}$ , the performance can be poor when the predictor is not accurate.
We propose an alternative loss, which simply replaces the weight  $1 / (m + n)$  in the last two terms with  $1 / n$ :
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {D R}} (\theta) = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, \hat {f} \left(X _ {i}\right)\right) - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, \hat {f} \left(X _ {i}\right)\right) + \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} \left(X _ {i}, Y _ {i}\right). \tag {1}
$$
As we will show later, this is a doubly robust estimator. We provide an intuitive interpretation here:
- In the case when the given predictor is perfectly accurate, i.e.,  $\hat{f}(X) \equiv Y$  always holds (which also means that  $Y|X = x$  is a deterministic function of  $x$ ), the last two terms cancel, and the loss minimizes the average loss,  $\frac{1}{m+n} \sum_{i=1}^{m+n} \ell_{\theta}(X_i, \hat{f}(X_i))$ , on all of the provided data. The effective sample size is  $m+n$ , compared with effective sample size  $n$  for training only on a labeled dataset using  $\mathcal{L}^{\mathrm{TL}}$ . In this case, the loss  $\mathcal{L}^{\mathrm{DR}}$  is much better than  $\mathcal{L}^{\mathrm{TL}}$ , and comparable to  $\mathcal{L}^{\mathrm{SL}}$ .
We may as well relax the assumption of  $\hat{f}(X) = Y$  to  $\mathbb{E}[\ell_{\theta}(X, \hat{f}(X))] = \mathbb{E}[\ell_{\theta}(X,Y)]$ . As  $n$  grows larger, the loss is approximately minimizing the average loss  $\frac{1}{m+n} \sum_{i=1}^{m+n} \ell_{\theta}(X_i, \hat{f}(X_i))$ .
There are several variants of the traditional self-training loss. For example, Xie et al. (2020) introduce an extra weight  $(m + n) / n$  on the labeled samples, and add noise to the student model; Sohn et al. (2020a) use confidence thresholding to filter unreliable pseudo-labels. However, both of these alternatives still suffer from the inconsistency issue. In this paper we focus on the simplest form  $\mathcal{L}^{\mathrm{SL}}$ .
- On the other hand, no matter how bad the given predictor is, the difference between the first two terms vanishes as either of  $m, n$  goes to infinity since the labeled samples  $X_{m+1}, \dots, X_{m+n}$  arise from the same distribution as  $X_1, \dots, X_m$ . Thus asymptotically the loss minimizes  $\frac{1}{n} \sum_{i=m+1}^{m+n} \ell_\theta(X_i, Y_i)$ , which discards the bad predictor  $\hat{f}$  and focuses only on the labeled dataset. Thus, in this case the loss  $\mathcal{L}^{\mathrm{DR}}$  is much better than  $\mathcal{L}^{\mathrm{SL}}$ , and comparable to  $\mathcal{L}^{\mathrm{TL}}$ .
This loss is appropriate only when the covariate distributions between labeled and unlabeled samples match. In the case where there is a distribution mismatch, we propose an alternative loss; see Section 2.3.
2.1 Motivating example: Mean estimation
As a concrete example, in the case of one-dimensional mean estimation we take  $\ell_{\theta}(X,Y) = (\theta -Y)^2$  Our target is to find some  $\theta^{\star}$  that satisfies
$$
\theta^ {\star} = \underset {\theta} {\arg \min } \mathbb {E} _ {(X, Y) \sim \mathbb {P} _ {X} \times \mathbb {P} _ {Y | X}} [ (\theta - Y) ^ {2} ].
$$
One can see that  $\theta^{\star} = \mathbb{E}[Y]$ . In this case, the loss for training only on labeled data becomes
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {T L}} (\theta) = \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \left(\theta - Y _ {i}\right) ^ {2}.
$$
Moreover, the optimal parameter is  $\hat{\theta}_{\mathrm{TL}} = \frac{1}{n}\sum_{i = m + 1}^{m + n}Y_{i}$ , which is a simple empirical average over all observed  $Y$ 's.
For a given pre-existing predictor  $\hat{f}$ , the loss for self-training becomes
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {S L}} (\theta) = \frac {1}{m + n} \left(\sum_ {i = 1} ^ {m} \left(\theta - \hat {f} \left(X _ {i}\right)\right) ^ {2} + \sum_ {i = m + 1} ^ {m + n} \left(\theta - Y _ {i}\right) ^ {2}\right).
$$
It is straightforward to see that the minimizer of the loss is the unweighted average between the unlabeled predictors  $\hat{f}(X_i)$ 's and the labeled  $Y_i$ 's:
$$
\theta_ {\mathsf {S L}} ^ {\star} = \frac {1}{m + n} \left(\sum_ {i = 1} ^ {m} \hat {f} (X _ {i}) + \sum_ {i = m + 1} ^ {m + n} Y _ {i}\right).
$$
In the case of  $m \gg n$ , the mean estimator is almost the same as the average of all the predicted values on the unlabeled dataset, which can be far from  $\theta^{\star}$  when the predictor  $\hat{f}$  is inaccurate.
On the other hand, for the proposed doubly robust estimator, we have
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {D R}} (\theta) = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} (\theta - \hat {f} (X _ {i})) ^ {2} - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} (\theta - \hat {f} (X _ {i})) ^ {2} + \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} (\theta - Y _ {i}) ^ {2} \\ = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} (\theta - \hat {f} (X _ {i})) ^ {2} + \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} 2 (\hat {f} (X _ {i}) - Y _ {i}) \theta + Y _ {i} ^ {2} - \hat {f} (X _ {i}) ^ {2}. \\ \end{array}
$$
Note that the loss is still convex, and we have
$$
\theta_ {\mathsf {D R}} ^ {\star} = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} \hat {f} (X _ {i}) - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} (\hat {f} (X _ {i}) - Y _ {i}).
$$
This recovers the estimator in prediction-powered inference (Angelopoulos et al., 2023). Assume that  $\hat{f}$  is independent of the labeled data. We can calculate the mean-squared error of the three estimators as follows.
Proposition 1. Let  $\mathsf{Var}[\hat{f}(X) - Y] = \mathbb{E}[(\hat{f}(X) - Y)^2 - \mathbb{E}[(\hat{f}(X) - Y)]^2]$ . We have
$$
\mathbb {E} \left[ \left(\theta^ {\star} - \hat {\theta} _ {\mathrm {T L}}\right) ^ {2} \right] = \frac {1}{n} \operatorname {V a r} [ Y ],
$$
$$
\mathbb {E} \left[ \left(\theta^ {\star} - \hat {\theta} _ {\mathrm {S L}}\right) ^ {2} \right] \leq \frac {2 m ^ {2}}{(m + n) ^ {2}} \mathbb {E} \left[ (\hat {f} (X) - Y) \right] ^ {2} + \frac {2 m}{(m + n) ^ {2}} \mathsf {V a r} [ \hat {f} (X) - Y ] + \frac {2 n}{(m + n) ^ {2}} \mathsf {V a r} [ Y ],
$$
$$
\mathbb {E} [ (\theta^ {\star} - \hat {\theta} _ {\mathsf {D R}}) ^ {2} ] \leq 2 \min  \left(\frac {1}{n} \mathsf {V a r} [ Y ] + \frac {m + 2 n}{(m + n) n} \mathsf {V a r} [ \hat {f} (X) ], \frac {m + 2 n}{(m + n) n} \mathsf {V a r} [ \hat {f} (X) - Y ] + \frac {1}{m + n} \mathsf {V a r} [ Y ]\right).
$$
The proof is deferred to Appendix F. The proposition illustrates the double-robustness of  $\hat{\theta}_{\mathsf{DR}}$  — no matter how poor the estimator  $\hat{f}(X)$  is, the rate is always upper bounded by  $\frac{4}{n} (\mathsf{Var}[Y] + \mathsf{Var}[\hat{f}(X)])$ . On the other hand, when  $\hat{f}(X)$  is an accurate estimator of  $Y$  (i.e.,  $\mathsf{Var}[\hat{f}(X) - Y]$  is small), the rate can be improved to  $\frac{2}{m+n}\mathsf{Var}[Y]$ . In contrast, the self-training loss always has a non-vanishing term,  $\frac{2m^2}{(m+n)^2}\mathbb{E}[(\hat{f}(X) - Y)]^2$ , when  $m \gg n$ , unless the predictor  $\hat{f}$  is accurate.
On the other hand, when  $\hat{f}(x) = \hat{\beta}_{(-1)}^{\top}x + \hat{\beta}_1$  is a linear predictor trained on the labeled data with  $\hat{\beta} = \arg \min_{\beta = [\beta_1,\beta_{(-1)}]}\frac{1}{n}\sum_{i=m+1}^{m+n}(\beta_{(-1)}^{\top}X_i + \beta_1 - Y_i)^2$ , our estimator reduces to the semi-supervised mean estimator in Zhang et al. (2019). Let  $\tilde{X} = [1,X]$ . In this case, we also know that the self-training reduces to training only on labeled data, since  $\hat{\theta}_{\mathsf{TL}}$  is also the minimizer of the self-training loss. We have the following result that reveals the superiority of the doubly robust estimator compared to the other two options.
Proposition 2 ((Zhang et al., 2019)). We establish the asymptotic behavior of various estimators when  $\hat{f}$  is a linear predictor trained on the labeled data:
- Training only on labeled data  $\hat{\theta}_{\mathrm{TL}}$  is equivalent to self-training  $\hat{\theta}_{\mathrm{SL}}$ , which gives unbiased estimator but with larger variance:
$$
\sqrt {n} \left(\hat {\theta} _ {\mathsf {T L}} - \theta^ {\star}\right)\rightarrow \mathcal {N} \left(0, \mathbb {E} \left[\left(Y - \beta^ {\top} \tilde {X}\right) ^ {2} \right] + \beta_ {(- 1)} ^ {\top} \Sigma \beta_ {(- 1)}\right).
$$
- Doubly Robust  $\hat{\theta}_{\mathrm{DR}}$  is unbiased with smaller variance:
$$
\sqrt {n} (\hat {\theta} _ {\mathsf {D R}} - \theta^ {\star}) \to \mathcal {N} (0, \mathbb {E} [ (Y - \beta^ {\top} \tilde {X}) ^ {2} ] + \frac {n}{m + n} \beta_ {(- 1)} ^ {\top} \Sigma \beta_ {(- 1)}).
$$
Here  $\beta = \arg \min_{\beta}\mathbb{E}[(Y - \beta^{\top}\tilde{X})^{2}]$  and  $\Sigma = \mathbb{E}[(X - \mathbb{E}[X])(X - \mathbb{E}[X])^{\top}]$ .
2.2 Guarantee for general loss
In the general case, we show that the doubly robust loss function continues to exhibit desirable properties. In particular, as  $n, m$  goes to infinity, the global minimum of the original loss is also a critical point of the new doubly robust loss, no matter how inaccurate the predictor  $\hat{f}$ .
Let  $\theta^{\star}$  be the minimizer of  $\mathbb{E}_{\mathbb{P}_{X,Y}}[\ell_{\theta}(X,Y)]$ . Let  $\hat{f}$  be a pre-existing model that does not depend on the datasets  $\mathcal{D}_1, \mathcal{D}_2$ . We also make the following regularity assumptions.
Assumption 1. The loss  $\ell_{\theta}(x,y)$  is differentiable at  $\theta^{\star}$  for any  $x,y$