SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 2. The random variables  $\nabla_{\theta}\ell_{\theta}(X,\hat{f} (X))$  and  $\nabla_{\theta}\ell_{\theta}(X,Y)$  have bounded first and second moments.
Given this assumption, we denote  $\Sigma_{\theta}^{Y - \hat{f}} = \mathrm{Cov}[\nabla_{\theta}\ell_{\theta}(X,\hat{f} (X)) - \nabla_{\theta}\ell_{\theta}(X,Y)]$  and let  $\Sigma_{\theta}^{\hat{f}} = \operatorname {Cov}[\nabla_{\theta}\ell_{\theta}(X,\hat{f} (X))],\Sigma_{\theta}^{Y} = \operatorname {Cov}[\nabla_{\theta}\ell_{\theta}(X,Y)]$
Theorem 2. Under Assumptions 1 and 2, we have that with probability at least  $1 - \delta$
$$
\begin{array}{l} \| \nabla_ {\theta} \mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathsf {D R}} (\theta^ {\star}) \| _ {2} \leq C \min  \left(\| \Sigma_ {\theta^ {\star}} ^ {\hat {f}} \| _ {2} \sqrt {\frac {d}{(m + n) \delta}} + \| \Sigma_ {\theta^ {\star}} ^ {Y - \hat {f}} \| _ {2} \sqrt {\frac {d}{n \delta}}, \right. \\ \left\| \Sigma_ {\theta^ {\star}} ^ {\hat {f}} \right\| _ {2} \left(\sqrt {\frac {d}{(m + n) \delta}} + \sqrt {\frac {d}{n \delta}}\right) + \left\| \Sigma_ {\theta^ {\star}} ^ {Y} \right\| _ {2} \sqrt {\frac {d}{n \delta}}), \\ \end{array}
$$
where  $C$  is a universal constant, and  $\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathsf{DR}}$  is defined in Equation (1).
The proof is deferred to Appendix G. From the example of mean estimation we know that one can design instances such that  $\| \nabla_{\theta}\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{SL}}(\theta^\star)\| _2\geq C$  for some positive constant  $C$ .
When the loss  $\nabla_{\theta}\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{DR}}$  is convex, the global minimum of  $\nabla_{\theta}\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{DR}}$  converges to  $\theta^{\star}$  as both  $m,n$  go to infinity. When the loss  $\nabla_{\theta}\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{DR}}$  is strongly convex, it also implies that  $\| \hat{\theta} -\theta^{\star}\| _2$  converges to zero as both  $m,n$  go to infinity, where  $\hat{\theta}$  is the minimizer of  $\nabla_{\theta}\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{DR}}$ .
When  $\hat{f}$  is a perfect predictor with  $\hat{f}(X) \equiv Y$  (and  $Y|X = x$  is deterministic), one has  $\mathcal{L}_{\mathcal{D}_1, \mathcal{D}_2}^{\mathrm{DR}}(\theta^\star) = \frac{1}{m + n} \sum_{i=1}^{m+n} \ell_\theta(X_i, Y_i)$ . The effective sample size is  $m + n$  instead of  $n$  in  $\mathcal{L}_{\mathcal{D}_1, \mathcal{D}_2}^{\mathrm{SL}}(\theta)$ .
When  $\hat{f}$  is also trained from the labeled data, one may apply data splitting to achieve the same guarantee up to a constant factor. We provide further discussion in Appendix E.
2.3 The case of distribution mismatch
We also consider the case in which the marginal distributions of the covariates for the labeled and unlabeled datasets are different. Assume in particular that we are given a set of unlabeled samples,  $\mathcal{D}_1 = \{X_1,X_2,\dots ,X_m\}$ , drawn from a fixed distribution  $\mathbb{P}_X$ , a set of labeled samples,  $\mathcal{D}_2 = \{(X_{m + 1},Y_{m + 1}),(X_{m + 2},Y_{m + 2}),\dots ,(X_{m + n},Y_{m + n})\}$ , drawn from some joint distribution  $\mathbb{Q}_X\times \mathbb{P}_{Y|X}$ , and a pre-trained model  $\hat{f}$ . In the case when the labeled samples do not follow the same distribution as the unlabeled samples, we need to introduce an importance weight  $\pi (x)$ . This yields the following doubly robust estimator:
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {D R 2}} (\theta) = \frac {1}{m} \sum_ {i = 1} ^ {m} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \frac {1}{\pi (X _ {i})} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) + \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \frac {1}{\pi (X _ {i})} \ell_ {\theta} (X _ {i}, Y _ {i}).
$$
Note that we not only introduce the importance weight  $\pi$ , but we also change the first term from the average of all the  $m + n$  samples to the average of  $n$  samples.
Proposition 3. We have  $\mathbb{E}[\mathcal{L}_{\mathcal{D}_1,\mathcal{D}_2}^{\mathrm{DR2}}(\theta)] = \mathbb{E}_{\mathbb{P}_{X,Y}}[\ell_\theta (X,Y)]$  as long as one of the following two assumptions hold:
For any  $x$ ,  $\pi(x) = \frac{\mathbb{P}_X(x)}{\mathbb{Q}_X(x)}$ .
- For any  $x$ ,  $\ell_{\theta}(x, \hat{f}(x)) = \mathbb{E}_{Y \sim \mathbb{P}_{Y|X = x}}[\ell_{\theta}(x, Y)]$ .
The proof is deferred to Appendix H. The proposition implies that as long as either  $\pi$  or  $\hat{f}$  is accurate, the expectation of the loss is the same as that of the target loss. When the distributions for the unlabeled and labeled samples match each other, this reduces to the case in the previous sections. In this case, taking  $\pi(x) = 1$  guarantees that the expectation of the doubly robust loss is always the same as that of the target loss.
To employ the new doubly robust loss in practical applications, we need to specify an appropriate optimization procedure, in particular one that is based on (mini-batched) stochastic gradient descent so as to exploit modern scalable machine learning methods. In preliminary experiments we observed that directly minimizing the doubly robust loss in Equation (1) with stochastic gradient can lead to instability, and thus, we propose instead to minimize the curriculum-based loss in each epoch:
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {D R}, t} (\theta) = \frac {1}{m + n} \sum_ {i = 1} ^ {m + n} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) - \alpha_ {t} \cdot \left(\frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} \ell_ {\theta} (X _ {i}, Y _ {i})\right).
$$
As we show in the experiments below, this choice yields a stable algorithm. We set  $\alpha_{t} = t / T$ , where  $T$  is the total number of epochs. For the object detection experiments, we introduce the labeled samples only in the final epoch, setting  $\alpha_{t} = 0$  for all epochs before setting  $\alpha_{t} = 1$  in the final epoch. Intuitively, we start from the training with samples only from the pseudo-labels, and gradually introduce the labeled samples in the doubly robust loss for fine-tuning.
We conduct experiments on both image classification task with ImageNet dataset (Russakovsky et al., 2015) and 3D object detection task with autonomous driving dataset nuScenes (Caesar et al., 2020). The code is available in https://github.com/dingmyu/Doubly-Robust-Self-Training.
3.1 Image classification
Datasets and settings. We evaluate our doubly robust self-training method on the ImageNet100 dataset, which contains a random subset of 100 classes from ImageNet-1k (Russakovsky et al., 2015),
with 120K training images (approximately 1,200 samples per class) and 5,000 validation images (50 samples per class). To further test the effectiveness of our algorithm in a low-data scenario, we create a dataset that we refer to as mini-ImageNet100 by randomly sampling 100 images per class from ImageNet100. Two models were evaluated: (1) DaViT-T (Ding et al., 2022), a popular vision transformer architecture with state-of-the-art performance on ImageNet, and (2) ResNet50 (He et al., 2016), a classic convolutional network to verify the generality of our algorithm.
Baselines. To provide a comparative evaluation of doubly robust self-training, we establish three baselines: (1) 'Labeled Only' for training on labeled data only (partial training set) with a loss  $\mathcal{L}^{\mathrm{TL}}$ , (2) 'Pseudo Only' for training with pseudo labels generated for all training samples, and (3) 'Labeled + Pseudo' for a mixture of pseudo-labels and labeled data, with the loss  $\mathcal{L}^{\mathrm{SL}}$ . See the Appendix for further implementation details and ablations.
Results on ImageNet100. We first conduct experiments on ImageNet100 by training the model for 20 epochs using different fractions of labeled data from  $1\%$  to  $100\%$ . From the results shown in Fig. 1, we observe that: (1) Our model outperforms all baseline methods on both two networks by large margins. For example, we achieve  $5.5\%$  and  $5.3\%$  gains (Top-1 Acc) on DaViT over the 'Labeled + Pseudo' method for  $20\%$  and  $80\%$  labeled data, respectively. (2) The 'Labeled + Pseudo' method consistently beats the 'Labeled Only' baseline. (3) While 'Pseudo Only' works for smaller fractions of the labeled data (less than  $30\%$ ) on DaViT, it is inferior to 'Labeled Only' on ResNet50.
Results on mini-ImageNet100. We also perform comparisons on mini-ImageNet100 to demonstrate the performance when the total data volume is limited. From the results in Table 1, we see our model generally outperforms all baselines. As the dataset size decreases and the number of training epochs increases, the gain of our algorithm becomes smaller. This is expected, as (1) the models are not adequately trained and thus have noise issues, and (2) there are an insufficient number of ground truth labels to compute the last term of our loss function. In extreme cases, there is only one labeled sample (1%) per class.
3.2 3D object detection
Doubly robust object detection. Given a visual representation of a scene, 3D object detection aims to generate a set of 3D bounding box predictions  $\{b_i\}_{i\in [m + n]}$  and a set of corresponding class predictions  $\{c_i\}_{i\in [m + n]}$ . Thus, each single ground-truth annotation  $Y_{i}\in Y$  is a set  $Y_{i} = (b_{i},c_{i})$
containing a box and a class. During training, the object detector is supervised with a sum of the box regression loss  $\mathcal{L}_{loc}$  and the classification loss  $\mathcal{L}_{cls}$ , i.e.  $\mathcal{L}_{obj} = \mathcal{L}_{loc} + \mathcal{L}_{cls}$ .
In the self-training protocol for object detection, pseudo-labels for a given scene  $X_{i}$  are selected from the labeler predictions  $f(X_{i})$  based on some user-defined criteria (typically the model's detection confidence). Unlike in standard classification or regression,  $Y_{i}$  will contain a differing number of labels depending on the number of objects in the scene. Furthermore, the number of extracted pseudo-labels  $f(X_{i})$  will generally not be equal to the number of scene ground-truth labels  $Y_{i}$  due to false positive/negative detections. Therefore it makes sense to express the doubly robust loss function in terms of the individual box labels as opposed to the scene-level labels. We define the doubly robust object detection loss as follows:
$$
\mathcal {L} _ {o b j} ^ {\mathrm {D R}} (\theta) = \frac {1}{M + N _ {p s}} \sum_ {i = 1} ^ {M + N _ {p s}} \ell_ {\theta} (X _ {i}, f (X _ {i})) - \frac {1}{N _ {p s}} \sum_ {i = M + 1} ^ {M + N _ {p s}} \ell_ {\theta} (X _ {i} ^ {\prime}, f (X _ {i} ^ {\prime})) + \frac {1}{N} \sum_ {i = M + 1} ^ {M + N} \ell_ {\theta} (X _ {i}, Y _ {i}),
$$
where  $M$  is the total number of pseudo-label boxes from the unlabeled split,  $N$  is the total number of labeled boxes,  $X_{i}^{\prime}$  is the scene with pseudo-label boxes from the labeled split, and  $N_{ps}$  is the total number of pseudo-label boxes from the labeled split. We note that the last two terms now contain summations over a differing number of boxes, a consequence of the discrepancy between the number of manually labeled boxes and pseudo-labeled boxes. Both components of the object detection loss (localization/classification) adopt this form of doubly robust loss.
Dataset and setting. To evaluate doubly robust self-training in the autonomous driving setting, we perform experiments on the large-scale 3D detection dataset nuScenes (Caesar et al., 2020). The nuScenes dataset is comprised of 1000 scenes (700 training, 150 validation and 150 test) with each frame containing sensor information from RGB camera, LiDAR, and radar scans. Box annotations are comprised of 10 classes, with the class instance distribution following a long-tailed distribution, allowing us to investigate our self-training approach for both common and rare classes. The main 3D detection metrics for nuScenes are mean Average Precision (mAP) and the nuScenes Detection Score (NDS), a dataset-specific metric consisting of a weighted average of mAP and five other true-positive metrics. For the sake of simplicity, we train object detection models using only LiDAR sensor information.
Results. After semi-supervised training, we evaluate our student model performance on the nuScenes val set. We compare three settings: training the student model with only the available labeled data
(i.e., equivalent to teacher training), training the student model on the combination of labeled/teacher-labeled data using the naive self-training loss, and training the student model on the combination of labeled/teacher-labeled data using our proposed doubly robust loss. We report results for training with 1/24, 1/16, and 1/4 of the total labels in Table 2. We find that the doubly robust loss improves both mAP and NDS over using only labeled data and the naive baseline in the lower label regime, whereas performance is slightly degraded when more labels are available. Furthermore, we also show a per-class performance breakdown in Table 3. We find that the doubly robust loss consistently improves performance for both common (car, pedestrian) and rare classes. Notably, the doubly robust loss is even able to improve upon the teacher in classes for which pseudo-label training decreases performance when using the naive training (e.g., barriers and traffic cones).
We have proposed a novel doubly robust loss for self-training. Theoretically, we analyzed the double-robustness property of the proposed loss, demonstrating its statistical efficiency when the pseudo-labels are accurate. Empirically, we showed that large improvements can be obtained in both image classification and 3D object detection.
As a direction for future work, it would be interesting to understand how the doubly robust loss might be applied to other domains that have a missing-data aspect, including model distillation, transfer learning, and continual learning. It is also important to find practical and efficient algorithms when the labeled and unlabeled data do not match in distribution.
Banghua Zhu and Jiantao Jiao are partially supported by NSF IIS-1901252, CIF-1909499 and CIF-2211209. Michael I. Jordan is partially supported by NSF IIS-1901252. Philip Jacobson is supported by the National Defense Science and Engineering Graduate (NDSEG) Fellowship. This work is also partially supported by Berkeley DeepDrive.
L. N. Smith and N. Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, page 1100612. International Society for Optics and Photonics, 2019.
K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685, 2020a.
K. Sohn, Z. Zhang, C.-L. Li, H. Zhang, C.-Y. Lee, and T. Pfister. A simple semi-supervised learning framework for object detection. In arXiv:2005.04757, 2020b.
Y. Tang, W. Chen, Y. Luo, and Y. Zhang. Humble teachers teach better students for semi-supervised object detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3131-3140, 2021.
H. Wang, Y. Cong, O. Litany, Y. Gao, and L. J. Guibas. 3dioumatch: Leveraging iou prediction for semi-supervised 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14615-14624, 2021.
K. Weiss, T. M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big Data, 3 (1):1-40, 2016.
Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
T. Yin, X. Zhou, and P. Krahenbuhl. Center-based 3D object detection and tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
A. Zhang, L. D. Brown, and T. T. Cai. Semi-supervised inference: General theory and estimation of means. Annals of Statistics, 47:2538-2566, 2019.
H. Zhou, Z. Ge, S. Liu, W. Mao, Z. Li, H. Yu, and J. Sun. Dense teacher: Dense pseudo-labels for semi-supervised object detection. In S. Avidan, G. Brostow, M. Cisse, G. M. Farinella, and T. Hassner, editors, Computer Vision - ECCV 2022, pages 35-50, 2022.
B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.09492, 2019.
X. J. Zhu. Semi-supervised learning literature survey. 2005.
A Implementation Details for Image Classification
We evaluate our doubly robust self-training method on the ImageNet100 and mini-ImageNet100 datasets, which are subsets of ImageNet-1k from ImageNet Large Scale Visual Recognition Challenge 2012 (Russakovsky et al., 2015). Two models are evaluated: (1) DaViT-T (Ding et al., 2022), a state-of-the-art 12-layer vision transformer architecture with a patch size of 4, a window size of 7, and an embedding dim of 768, and (2) ResNet50 (He et al., 2016), a classic and powerful convolutional network with 50 layers and embedding dim 2048. We evaluate all the models on the same ImageNet100 validation set (50 samples per class). For the training, we use the same data augmentation and regularization strategies following the common practice in Liu et al. (2021b); Lin et al. (2017); Ding et al. (2022). We train all the models with a batch size of 1024 on 8 Tesla V100 GPUs (the batch size is reduced to 64 if the number of training data is less than 1000). We use AdamW (Loshchilov and Hutter, 2017) optimizer and a simple triangular learning rate schedule (Smith and Topin, 2019). The weight decay is set to 0.05 and the maximal gradient norm is clipped to 1.0. The stochastic depth drop rates are set to 0.1 for all models. During training, we crop images randomly to  $224 \times 224$ , while a center crop is used during evaluation on the validation set. We use a curriculum setting where the  $\alpha_{t}$  grows linearly or quadratically from 0 to 1 throughout the training. To show the effectiveness of our method, we also compare model training with different curriculum learning settings and varying numbers of epochs.
B Additional Experiments in Image Classification
Ablation study on curriculum settings. There are three options for the curriculum setting: 1)  $\alpha_{t} = 1$  throughout the whole training, 2) grows linearly with training iterations  $\alpha_{t} = t / T$ , 3) grows quadratically with training iterations  $\alpha_{t} = (t / T)^{2}$ . From results in Table 4, we see: the first option achieves comparable performance with the 'Naive Labeled + Pseudo' baseline. Both the linear and quadratic strategies show significant performance improvements: the linear one works better when more labeled data is available, e.g.,  $70\%$  and  $90\%$ , while the quadratic one prefers less labeled data, e.g.  $30\%$  and  $50\%$ .
Ablation Study on the Number of Epochs. We conduct experiments on different training epochs. The results are shown in Table 5. Our model is consistently superior to the baselines. And we can observe the gain is larger when the number of training epochs is relatively small, e.g. 20 and 50.
Fully trained results (300 epochs) on ImageNet-100. In our original experiments, we mostly focus on a teacher model that is not super accurate, since our method reduces to the original pseudo-labeling when the teacher model is completely correct for all labels. In this experiment, we fully train the teacher model with 300 epochs on ImageNet-100, leading to the accuracy of the teacher model as high as  $88.0\%$ . From Figure 2, we show that even in this case, our method outperforms the original pseudo-labeling baseline.
Comparisons with previous SOTAs on CIFAR-10 and CiFAR-100. We compare with another 11 baselines in terms of error rate on CIFAR-10-4K and CIFAR-100-10K under the same settings (i.e., Wide ResNet-28-2 for CIFAR-10 and WRN-28-8 for CIFAR-100). We show that our method is only 0.04 inferior to the best method Meta Pseudo Labels for CIFAR-10-4K, and achieves the best performance for CIFAR-100-10K.
C Implementation Details of 3D Object Detection
Our experiments follow the standard approach for semi-supervised detection: we first initialize two detectors, the teacher (i.e., labeler) and the student. First, a random split of varying sizes is selected from the nuScenes training set. We pre-train the teacher network using the ground-truth annotations in this split. Following this, we freeze the weights in the teacher model and then use it to generate pseudo-labels on the entire training set. The student network is then trained on a combination of the pseudo-labels and ground-truth labels originating from the original split. In all of our semi-supervised experiments, we use CenterPoint with a PointPillars backbone as our 3D detection model (Yin et al., 2021; Lang et al., 2019). The teacher pre-training and student training are both conducted for 10 epochs on 3 NVIDIA RTX A6000 GPUs. We follow the standard nuScenes training setting outlined in Zhu et al. (2019), with the exception of disabling ground-truth paste augmentation during training to prevent data leakage from the labeled split. To select the pseudo-labels to be used in training the student, we simply filter the teacher predictions by detection confidence, using all detections above a chosen threshold. We use a threshold of 0.3 for all classes, as in Park et al. (2022). In order to conduct training in a batch-wise manner, we compute the loss over only the samples contained within
the batch. We construct each batch to have a consistent ratio of labeled/unlabeled samples to ensure the loss is well-defined for the batch.
D Additional Experiments in 3D Object Detection
Comparison with Semi-Supervised Baseline We compare our approach to another semi-supervised baseline on the 3D object detection task, Pseudo-labeling and confirmation bias Arazo et al. (2020). We show that in multiple settings, our approach surpasses the baseline performance.
Ablation on Pseudo-label Confidence Threshold To demonstrate that appropriate quality pseudolabels are used to train the student detector, we performance ablation experiments varying the detection threshold used to extract pseudo-labels from the teacher model predictions. We show that training with a threshold of  $\tau = 0.3$  outperforms training with a more stringent threshold, and is the appropriate experimental setting for our main experiments.
E Considerations when  $\hat{f}$  is trained from labeled data
In Theorem 2, we analyzed the double robustness of the proposed loss function when the predictor  $\hat{f}$  is pre-existing and not trained from the labeled dataset. In practice, one may only have access to the labeled and unlabeled datasets without a pre-existing teacher model. In this case, one may choose to split the labeled samples  $\mathcal{D}_2$  into two parts. The last  $n / 2$  samples are used to train  $\hat{f}$ , and the first  $n / 2$  samples are used in the doubly robust loss:
$$
\mathcal {L} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} ^ {\mathrm {D R 2}} (\theta) = \frac {1}{m} \sum_ {i = 1} ^ {m} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) - \frac {2}{n} \sum_ {i = m + 1} ^ {m + n / 2} \frac {1}{\pi (X _ {i})} \ell_ {\theta} (X _ {i}, \hat {f} (X _ {i})) + \frac {2}{n} \sum_ {i = m + 1} ^ {m + n / 2} \frac {1}{\pi (X _ {i})} \ell_ {\theta} (X _ {i}, Y _ {i}).
$$
Since  $\hat{f}$  is independent of all samples used in the above loss, the result in Theorem 2 continues to hold. Asymptotically, such a doubly robust estimator is never worse than the estimator trained only on the labeled data.
F Proof of Proposition 1
For the labeled-only estimator  $\hat{\theta}_{\mathrm{TL}}$  , we have
$$
\mathbb {E} [ (\theta^ {\star} - \hat {\theta} _ {\sf T L}) ^ {2} ] = \mathbb {E} \left[ \left(\mathbb {E} [ Y ] - \frac {1}{n} \sum_ {i = m + 1} ^ {m + n} Y\right) ^ {2} \right] = \frac {1}{n} \mathsf {V a r} [ Y ].
$$
For the self-training loss, we have