Title: Doubly-Robust Self-Training
Venue: NeurIPS 2023
Accepted: yes
Keywords: semi-supervised learning

self-training

auto-labeling

self-labeling

doubly robust
Abstract: Self-training is a well-established technique in semi-supervised learning, which leverages unlabeled data by generating pseudo-labels and incorporating them with a limited labeled dataset for training. The effectiveness of self-training heavily relies on the accuracy of these pseudo-labels. In this paper, we introduce doubly-robust self-training, an innovative semi-supervised algorithm that provably balances between two extremes. When pseudo-labels are entirely incorrect, our method reduces to a training process solely using labeled data. Conversely, when pseudo-labels are completely accurate, our method transforms into a training process utilizing all pseudo-labeled data and labeled data, thus increasing the effective sample size. Through empirical evaluations on both the ImageNet dataset for image classification and the nuScenes autonomous driving dataset for 3D object detection, we demonstrate the superiority of the doubly-robust loss over the self-training baseline.

Doubly Robust Self-Training

Abstract

Self-training is an important technique for solving semi-supervised learning problems. It leverages unlabeled data by generating pseudo-labels and combining them with a limited labeled dataset for training. The effectiveness of self-training heavily relies on the accuracy of these pseudo-labels. In this paper, we introduce doubly robust self-training, a novel semi-supervised algorithm that provably balances between two extremes. When the pseudo-labels are entirely incorrect, our method reduces to a training process solely using labeled data. Conversely, when the pseudo-labels are completely accurate, our method transforms into a training process utilizing all pseudo-labeled data and labeled data, thus increasing the effective sample size. Through empirical evaluations on both the ImageNet dataset for image classification and the nuScenes autonomous driving dataset for 3D object detection, we demonstrate the superiority of the doubly robust loss over the standard self-training baseline.

1 Introduction

Semi-supervised learning considers the problem of learning based on a small labeled dataset together with a large unlabeled dataset. This general framework plays an important role in many problems in machine learning, including model fine-tuning, model distillation, self-training, transfer learning and continual learning (Zhu, 2005; Pan and Yang, 2010; Weiss et al., 2016; Gou et al., 2021; De Lange et al., 2021). Many of these problems also involve some form of distribute shift, and accordingly, to best utilize the unlabeled data, an additional assumption is that one has access to a teacher model obtained from prior training. It is important to study the relationships among the datasets and the teacher model. In this paper, we ask the following question:

Given a teacher model, a large unlabeled dataset and a small labeled dataset, how can we design a principled learning process that ensures consistent and sample-efficient learning of the true model?

Self-training is one widely adopted and popular approach in computer vision and autonomous driving for leveraging information from all three components (Lee, 2013; Berthelot et al., 2019b,a; Sohn et al., 2020a; Xie et al., 2020; Jiang et al., 2022; Qi et al., 2021). This approach involves using a teacher model to generate pseudo-labels for all unlabeled data, and then training a new model on a mixture of both pseudo-labeled and labeled data. However, this method can lead to overreliance on the teacher model and can miss important information provided by the labeled data. As a consequence, the self-training approach becomes highly sensitive to the accuracy of the teacher model. Our study demonstrates that even in the simplest scenario of mean estimation, this method can yield significant failures when the teacher model lacks accuracy.

To overcome this issue, we propose an alternative method that is doubly robust -when the covariate distribution of the unlabeled dataset and the labeled dataset matches, the estimator is always consistent no matter whether the teacher model is accurate or not. On the other hand, when the teacher model is an accurate predictor, the estimator makes full use of the pseudo-labeled dataset and greatly increases the effective sample size. The idea is inspired by and directly related to missing-data inference and causal inference (Rubin, 1976; Kang and Schafer, 2007; Birhanu et al., 2011; Ding and Li, 2018), to semiparametric mean estimation (Zhang et al., 2019), and to recent work on prediction-powered inference (Angelopoulos et al., 2023).

1.1 Main results

The proposed algorithm is based on a simple modification of the standard loss for self-training. Assume that we are given a set of unlabeled samples, D 1 = { X 1 , X 2 , · · · , X m } , drawn from a fixed distribution P X , a set of labeled samples D 2 = { ( X m +1 , Y m +1 ) , ( X m +2 , Y m +2 ) , · · · , ( X m + n , Y m + n ) } , drawn from some joint distribution P X × P Y | X , and a teacher model ˆ f . Let ℓ θ ( x, y ) be a pre-specified loss function that characterizes the prediction error of the estimator with parameter θ on the given sample ( X,Y ) . Traditional self-training aims at minimizing the combined loss for both labeled and unlabeled samples, where the pseudo-labels for unlabeled samples are generated using ˆ f :

Note that this can also be viewed as first using ˆ f to predict all the data, and then replacing the originally labeled points with the known labels:

Our proposed doubly robust loss instead replaces the coefficient 1 / ( m + n ) with 1 /n in the last two terms:

This seemingly minor change has a major beneficial effect-the estimator becomes consistent and doubly robust.

Theorem 1 (Informal) . Let θ ⋆ be defined as the minimizer θ ⋆ = arg min θ E ( X,Y ) ∼ P X × P Y | X [ ℓ θ ( X,Y )] . Under certain regularity conditions, we have

On the other hand, there exists instances such that ∥∇ θ L SL D 1 , D 2 ( θ ⋆ ) ∥ 2 ≥ C always holds true no matter how large m,n are.

The result shows that the true parameter θ ⋆ is also a local minimum of the doubly robust loss, but not a local minimum of the original self-training loss. We flesh out this comparison for the special example of mean estimation in Section 2.1, and present empirical results on image and driving datasets in Section 3.

1.2 Related work

Missing-data inference and causal inference. The general problem of causal inference can be formulated as a missing-data inference problem as follows. For each unit in an experiment, at most one of the potential outcomes-the one corresponding to the treatment to which the unit is exposed-is observed, and the other potential outcomes are viewed as missing (Holland, 1986; Ding and Li, 2018). Two of the standard methods for solving this problem are data imputation Rubin (1979) and propensity score weighting Rosenbaum and Rubin (1983). A doubly robust causal inference estimator combines the virtues of these two methods. The estimator is referred to as 'doubly robust' due to the following property: if the model for imputation is correctly specified then it is consistent no matter whether the propensity score model is correctly specified; on the other hand, if the model propensity score model is correctly specified, then it is consistent no matter whether the model for imputation is correctly specified (Scharfstein et al., 1999; Bang and Robins, 2005; Birhanu et al., 2011; Ding and Li, 2018).

We note in passing that double machine learning is another methodology that is inspired by the doubly robust paradigm in causal inference (Semenova et al., 2017; Chernozhukov et al., 2018a,b; Foster and Syrgkanis, 2019). The problem in double machine learning is related to the classic semiparametric problem of inference for a low-dimensional parameter in the presence of high-dimensional nuisance parameters, which is different goal than the predictive goal characterizing semi-supervised learning.

The recent work of prediction-powered inference (Angelopoulos et al., 2023) focuses on confidence estimation when there are both unlabeled data, labeled data, along with a teacher model. Their focus is the inferential problem of obtaining a confidence set, while ours is the doubly robust property of a point estimator. Since they focus on confidence estimation, an important, strong, yet biased baseline point-estimate algorithm that directly combines the ground-truth labels and pseudo-labels is not considered in their case. In our paper, we show with both theory and experiments that the proposed doubly-robust estimator achieves better performance than the naive combination of ground-truth labels and pseudo-labels.

Self-training. Self-training is a popular semi-supervised learning paradigm in which machinegenerated pseudo-labels are used for training with unlabeled data (Lee, 2013; Berthelot et al., 2019b,a; Sohn et al., 2020a). To generate these pseudo-labels, a teacher model is pre-trained on a set of labeled data, and its predictions on the unlabeled data are extracted as pseudo-labels. Previous work seeks to address the noisy quality of pseudo-labels in various ways. MixMatch (Berthelot et al., 2019b) ensembles pseudo-labels across several augmented views of the input data. ReMixMatch (Berthelot et al., 2019a) extends this by weakly augmenting the teacher inputs and strongly augmenting the student inputs. FixMatch (Sohn et al., 2020a) uses confidence thresholding to select only high-quality pseudo-labels for student training.

Self-training has been applied in both 2D computer vision problems (Liu et al., 2021a; Jeong et al., 2019; Tang et al., 2021; Sohn et al., 2020b; Zhou et al., 2022) and 3D problems (Park et al., 2022; Wang et al., 2021; Li et al., 2023; Liu et al., 2023) object detection. STAC (Sohn et al., 2020b) enforces consistency between strongly augmented versions of confidence-filtered pseudo-labels. Unbiased teacher (Liu et al., 2021a) updates the teacher during training with an exponential moving average (EMA) of the student network weights. Dense Pseudo-Label (Zhou et al., 2022) replaces box pseudo-labels with the raw output features of the detector to allow the student to learn richer context. In the 3D domain, 3DIoUMatch (Wang et al., 2021) thresholds pseudo-labels using a model-predicted Intersection-over-Union (IoU). DetMatch (Park et al., 2022) performs detection in both the 2D and 3D domains and filters pseudo-labels based on 2D-3D correspondence. HSSDA (Liu et al., 2023) extends strong augmentation during training with a patch-based point cloud shuffling augmentation. Offboard3D (Qi et al., 2021) utilizes multiple frames of temporal context to improve pseudo-label quality.

There has been a limited amount of theoretical analysis of these methods, focusing on semi-supervised methods for mean estimation and linear regression (Zhang et al., 2019; Azriel et al., 2022). Our analysis bridges the gap between these analyses and the doubly robust estimators in the causal inference literature.

2 Doubly Robust Self-Training

We begin with the case where the marginal distributions for the covariates of the labeled and unlabeled datasets are the same. Assume that we are given a set of unlabeled samples, D 1 = { X 1 , X 2 , · · · , X m } , drawn from a fixed distribution P X supported on X , a set of labeled samples D 2 = { ( X m +1 , Y m +1 ) , ( X m +2 , Y m +2 ) , · · · , ( X m + n , Y m + n ) } , drawn from some joint distribution P X × P Y | X supported on X × Y , and a pre-trained model, ˆ f : X ↦→ Y . Let ℓ θ ( · , · ) : X × Y ↦→ R be a pre-specified loss function that characterizes the prediction error of the estimator with parameter θ on the given sample ( X,Y ) . Our target is to find some θ ⋆ ∈ Θ that satisfies

For a given loss ℓ θ ( x, y ) , consider a naive estimator that ignores the predictor ˆ f and only trains on the labeled samples:

Although naive, this is a safe choice since it is an empirical risk minimizer. As n → ∞ , the loss converges to the population loss. However, it ignores all the information provided in ˆ f and the unlabeled dataset, which makes it inefficient when the predictor ˆ f is informative.

On the other hand, traditional self-training aims at minimizing the combined loss for both labeled and unlabeled samples, where the pseudo-labels for unlabeled samples are generated using ˆ f : 1

As is shown by the last equality, the self-training loss can be viewed as first using ˆ f to predict all the samples (including the labeled samples) and computing the average loss, then replacing that part of the loss corresponding to the labeled samples with the loss on the original labels. Although the loss uses the information arising from the unlabeled samples and ˆ f , the performance can be poor when the predictor is not accurate.

We propose an alternative loss, which simply replaces the weight 1 / ( m + n ) in the last two terms with 1 /n :

As we will show later, this is a doubly robust estimator. We provide an intuitive interpretation here:

In the case when the given predictor is perfectly accurate, i.e., ˆ f ( X ) ≡ Y always holds (which also means that Y | X = x is a deterministic function of x ), the last two terms cancel, and the loss minimizes the average loss, 1 m + n ∑ m + n i =1 ℓ θ ( X i , ˆ f ( X i )) , on all of the provided data. The effective sample size is m + n , compared with effective sample size n for training only on a labeled dataset using L TL . In this case, the loss L DR is much better than L TL , and comparable to L SL .

We may as well relax the assumption of ˆ f ( X ) = Y to E [ ℓ θ ( X, ˆ f ( X ))] = E [ ℓ θ ( X,Y )] . As n grows larger, the loss is approximately minimizing the average loss 1 m + n ∑ m + n i =1 ℓ θ ( X i , ˆ f ( X i )) .

On the other hand, no matter how bad the given predictor is, the difference between the first two terms vanishes as either of m,n goes to infinity since the labeled samples X m +1 , · · · , X m + n arise from the same distribution as X 1 , · · · , X m . Thus asymptotically the loss minimizes 1 n ∑ m + n i = m +1 ℓ θ ( X i , Y i ) , which discards the bad predictor ˆ f and focuses only on the labeled dataset. Thus, in this case the loss L DR is much better than L SL , and comparable to L TL .

This loss is appropriate only when the covariate distributions between labeled and unlabeled samples match. In the case where there is a distribution mismatch, we propose an alternative loss; see Section 2.3.

2.1 Motivating example: Mean estimation

As a concrete example, in the case of one-dimensional mean estimation we take ℓ θ ( X,Y ) = ( θ -Y ) 2 . Our target is to find some θ ⋆ that satisfies

One can see that θ ⋆ = E [ Y ] . In this case, the loss for training only on labeled data becomes

Moreover, the optimal parameter is ˆ θ TL = 1 n ∑ m + n i = m +1 Y i , which is a simple empirical average over all observed Y 's.

For a given pre-existing predictor ˆ f , the loss for self-training becomes

It is straightforward to see that the minimizer of the loss is the unweighted average between the unlabeled predictors ˆ f ( X i ) 's and the labeled Y i 's:

In the case of m ≫ n , the mean estimator is almost the same as the average of all the predicted values on the unlabeled dataset, which can be far from θ ⋆ when the predictor ˆ f is inaccurate.

On the other hand, for the proposed doubly robust estimator, we have

Note that the loss is still convex, and we have

This recovers the estimator in prediction-powered inference (Angelopoulos et al., 2023). Assume that ˆ f is independent of the labeled data. We can calculate the mean-squared error of the three estimators as follows.

The proof is deferred to Appendix F. The proposition illustrates the double-robustness of ˆ θ DR -no matter how poor the estimator ˆ f ( X ) is, the rate is always upper bounded by 4 n ( Var [ Y ] + Var [ ˆ f ( X )]) . On the other hand, when ˆ f ( X ) is an accurate estimator of Y (i.e., Var [ ˆ f ( X ) -Y ] is small), the rate can be improved to 2 m + n Var [ Y ] . In contrast, the self-training loss always has a non-vanishing term, 2 m 2 ( m + n ) 2 E [( ˆ f ( X ) -Y )] 2 , when m ≫ n , unless the predictor ˆ f is accurate.

On the other hand, when ˆ f ( x ) = ˆ β ⊤ ( -1) x + ˆ β 1 is a linear predictor trained on the labeled data with ˆ β = arg min β =[ β 1 ,β ( -1) ] 1 n ∑ m + n i = m +1 ( β ⊤ ( -1) X i + β 1 -Y i ) 2 , our estimator reduces to the semisupervised mean estimator in Zhang et al. (2019). Let ˜ X = [1 , X ] . In this case, we also know that the self-training reduces to training only on labeled data, since ˆ θ TL is also the minimizer of the self-training loss. We have the following result that reveals the superiority of the doubly robust estimator compared to the other two options.

Proposition 2 ((Zhang et al., 2019)) . We establish the asymptotic behavior of various estimators when ˆ f is a linear predictor trained on the labeled data:

Training only on labeled data ˆ θ TL is equivalent to self-training ˆ θ SL , which gives unbiased estimator but with larger variance:

Doubly Robust ˆ θ DR is unbiased with smaller variance:

Here β = arg min β E [( Y -β ⊤ ˜ X ) 2 ] and Σ = E [( X -E [ X ])( X -E [ X ]) ⊤ ] .

2.2 Guarantee for general loss

In the general case, we show that the doubly robust loss function continues to exhibit desirable properties. In particular, as n, m goes to infinity, the global minimum of the original loss is also a critical point of the new doubly robust loss, no matter how inaccurate the predictor ˆ f .

Let θ ⋆ be the minimizer of E P X,Y [ ℓ θ ( X,Y )] . Let ˆ f be a pre-existing model that does not depend on the datasets D 1 , D 2 . We also make the following regularity assumptions.

Assumption 1. The loss ℓ θ ( x, y ) is differentiable at θ ⋆ for any x, y .

Assumption 2. The random variables ∇ θ ℓ θ ( X, ˆ f ( X )) and ∇ θ ℓ θ ( X,Y ) have bounded first and second moments.

Given this assumption, we denote Σ Y -ˆ f θ = Cov [ ∇ θ ℓ θ ( X, ˆ f ( X )) - ∇ θ ℓ θ ( X,Y )] and let Σ ˆ f θ = Cov [ ∇ θ ℓ θ ( X, ˆ f ( X ))] , Σ Y θ = Cov [ ∇ θ ℓ θ ( X,Y )] .

Theorem 2. Under Assumptions 1 and 2, we have that with probability at least 1 -δ ,

where C is a universal constant, and L DR D 1 , D 2 is defined in Equation (1).

The proof is deferred to Appendix G. From the example of mean estimation we know that one can design instances such that ∥∇ θ L SL D 1 , D 2 ( θ ⋆ ) ∥ 2 ≥ C for some positive constant C .

When the loss ∇ θ L DR D 1 , D 2 is convex, the global minimum of ∇ θ L DR D 1 , D 2 converges to θ ⋆ as both m,n go to infinity. When the loss ∇ θ L DR D 1 , D 2 is strongly convex, it also implies that ∥ ˆ θ -θ ⋆ ∥ 2 converges to zero as both m,n go to infinity, where ˆ θ is the minimizer of ∇ θ L DR D 1 , D 2 .

When ˆ f is a perfect predictor with ˆ f ( X ) ≡ Y (and Y | X = x is deterministic), one has L DR D 1 , D 2 ( θ ⋆ ) = 1 m + n ∑ m + n i =1 ℓ θ ( X i , Y i ) . The effective sample size is m + n instead of n in L SL D 1 , D 2 ( θ ) .

When ˆ f is also trained from the labeled data, one may apply data splitting to achieve the same guarantee up to a constant factor. We provide further discussion in Appendix E.

2.3 The case of distribution mismatch

We also consider the case in which the marginal distributions of the covariates for the labeled and unlabeled datasets are different. Assume in particular that we are given a set of unlabeled samples, D 1 = { X 1 , X 2 , · · · , X m } , drawn from a fixed distribution P X , a set of labeled samples, D 2 = { ( X m +1 , Y m +1 ) , ( X m +2 , Y m +2 ) , · · · , ( X m + n , Y m + n ) } , drawn from some joint distribution Q X × P Y | X , and a pre-trained model ˆ f . In the case when the labeled samples do not follow the same distribution as the unlabeled samples, we need to introduce an importance weight π ( x ) . This yields the following doubly robust estimator:

Note that we not only introduce the importance weight π , but we also change the first term from the average of all the m + n samples to the average of n samples.

Proposition 3. We have E [ L DR2 D 1 , D 2 ( θ )] = E P X,Y [ ℓ θ ( X,Y )] as long as one of the following two assumptions hold:

For any x , π ( x ) = P X ( x ) Q X ( x ) .

The proof is deferred to Appendix H. The proposition implies that as long as either π or ˆ f is accurate, the expectation of the loss is the same as that of the target loss. When the distributions for the unlabeled and labeled samples match each other, this reduces to the case in the previous sections. In this case, taking π ( x ) = 1 guarantees that the expectation of the doubly robust loss is always the same as that of the target loss.

3 Experiments

To employ the new doubly robust loss in practical applications, we need to specify an appropriate optimization procedure, in particular one that is based on (mini-batched) stochastic gradient descent so as to exploit modern scalable machine learning methods. In preliminary experiments we observed that directly minimizing the doubly robust loss in Equation (1) with stohastic gradient can lead to instability, and thus, we propose instead to minimize the curriculum-based loss in each epoch:

As we show in the experiments below, this choice yields a stable algorithm. We set α t = t/T , where T is the total number of epochs. For the object detection experiments, we introduce the labeled samples only in the final epoch, setting α t = 0 for all epochs before setting α t = 1 in the final epoch. Intuitively, we start from the training with samples only from the pseudo-labels, and gradually introduce the labeled samples in the doubly robust loss for fine-tuning.

We conduct experiments on both image classification task with ImageNet dataset (Russakovsky et al., 2015) and 3D object detection task with autonomous driving dataset nuScenes (Caesar et al., 2020). The code is available in https://github.com/dingmyu/Doubly-Robust-Self-Training .

3.1 Image classification

Datasets and settings. We evaluate our doubly robust self-training method on the ImageNet100 dataset, which contains a random subset of 100 classes from ImageNet-1k (Russakovsky et al., 2015),

10

20

30

40

50

60

70

80

90

100

Labeled Data Fraction (%)

15

20

25

30

35

40

45

50

55

Top-1 Acc (%)

16.0

16.9

23.9

29.8

32.8

36.5

39.6

43.1

45.0

47.8

17.0

18.2

24.4

29.7

31.9

35.2

37.3

39.7

41.2

42.8

19.4

20.3

28.0

34.0

37.6

41.4

43.8

46.1

47.7

47.8

23.3

25.8

32.6

37.0

39.7

45.5

48.4

51.4

51.5

51.8

Labeled Only

Labeled + Pseudo

Doubly-Robust (Ours)

10

20

30

40

50

60

70

80

90

100

Labeled Data Fraction (%)

40

50

60

70

80

Top-5 Acc (%)

39.7

40.0

50.0

58.0

62.6

66.3

69.9

72.6

75.2

77.1

38.6

39

48.9

55.0

58.0

62.4

65.6

67.8

69.5

70.3

42.0

44.5

54.6

62.7

66.7

71.8

73.4

75.3

77.2

77.1

50.5

54.0

62.1

67.9

70.3

74.6

77.9

79.4

79.9

80.0

Labeled Only

Labeled + Pseudo

Doubly-Robust (Ours)

(a) Top-1 on DaViT

(b) Top-5 on DaViT

10

20

30

40

50

60

70

80

90

100

Labeled Data Fraction (%)

15

20

25

30

35

40

45

50

Top-1 Acc (%)

15.7

21.6

25.8

29.1

31.7

34.8

37.0

40.0

43.6

46.7

15.6

20.8

24.8

27.3

28.9

32.6

33.9

35.7

37.6

39.5

16.5

23.1

28.9

33.0

36.8

40.3

43.5

45.8

46.6

46.7

20.5

25.0

32.0

37.0

41.9

43.9

46.8

48.4

49.9

51.4

Labeled Only

Labeled + Pseudo

Doubly-Robust (Ours)

10

20

30

40

50

60

70

80

90

100

Labeled Data Fraction (%)

40

50

60

70

80

Top-5 Acc (%)

38.3

48.2

54.1

58.4

62.1

66.3

68.4

71.8

74.9

77.0

35.8

44.9

50.3

53.7

56.0

59.8

62.1

64.4

65.7

67.9

38.6

50.5

57.9

63.0

67.6

71.1

73.7

75.0

76.6

77.0

43.3

55.1

63.4

66.9

70.5

72.9

74.9

76.8

77.6

79.0

Labeled Only

Labeled + Pseudo

Doubly-Robust (Ours)

(c) Top-1 on ResNet50

(d) Top-5 on ResNet50

with 120K training images (approximately 1,200 samples per class) and 5,000 validation images (50 samples per class). To further test the effectiveness of our algorithm in a low-data scenario, we create a dataset that we refer to as mini-ImageNet100 by randomly sampling 100 images per class from ImageNet100. Two models were evaluated: (1) DaViT-T (Ding et al., 2022), a popular vision transformer architecture with state-of-the-art performance on ImageNet, and (2) ResNet50 (He et al., 2016), a classic convolutional network to verify the generality of our algorithm.

Baselines. To provide a comparative evaluation of doubly robust self-training, we establish three baselines: (1) 'Labeled Only' for training on labeled data only (partial training set) with a loss L TL , (2) 'Pseudo Only' for training with pseudo labels generated for all training samples, and (3) 'Labeled + Pseudo' for a mixture of pseudo-labels and labeled data, with the loss L SL . See the Appendix for further implementation details and ablations.

Results on ImageNet100. We first conduct experiments on ImageNet100 by training the model for 20 epochs using different fractions of labeled data from 1% to 100%. From the results shown in Fig. 1, we observe that: (1) Our model outperforms all baseline methods on both two networks by large margins. For example, we achieve 5.5% and 5.3% gains (Top-1 Acc) on DaViT over the 'Labeled + Pseudo' method for 20% and 80% labeled data, respectively. (2) The 'Labeled + Pseudo' method consistently beats the 'Labeled Only' baseline. (3) While 'Pseudo Only' works for smaller fractions of the labeled data (less than 30%) on DaViT, it is inferior to 'Labeled Only' on ResNet50.

Results on mini-ImageNet100. We also perform comparisons on mini-ImageNet100 to demonstrate the performance when the total data volume is limited. From the results in Table 1, we see our model generally outperforms all baselines. As the dataset size decreases and the number of training epochs increases, the gain of our algorithm becomes smaller. This is expected, as (1) the models are not adequately trained and thus have noise issues, and (2) there are an insufficient number of ground truth labels to compute the last term of our loss function. In extreme cases, there is only one labeled sample (1%) per class.

3.2 3D object detection

Doubly robust object detection. Given a visual representation of a scene, 3D object detection aims to generate a set of 3D bounding box predictions { b i } i ∈ [ m + n ] and a set of corresponding class predictions { c i } i ∈ [ m + n ] . Thus, each single ground-truth annotation Y i ∈ Y is a set Y i = ( b i , c i )

containing a box and a class. During training, the object detector is supervised with a sum of the box regression loss L loc and the classification loss L cls , i.e. L obj = L loc + L cls .

In the self-training protocol for object detection, pseudo-labels for a given scene X i are selected from the labeler predictions f ( X i ) based on some user-defined criteria (typically the model's detection confidence). Unlike in standard classification or regression, Y i will contain a differing number of labels depending on the number of objects in the scene. Furthermore, the number of extracted pseudo-labels f ( X i ) will generally not be equal to the number of scene ground-truth labels Y i due to false positive/negative detections. Therefore it makes sense to express the doubly robust loss function in terms of the individual box labels as opposed to the scene-level labels. We define the doubly robust object detection loss as follows:

where M is the total number of pseudo-label boxes from the unlabeled split, N is the total number of labeled boxes, X ′ i is the scene with pseudo-label boxes from the labeled split, and N ps is the total number of pseudo-label boxes from the labeled split. We note that the last two terms now contain summations over a differing number of boxes, a consequence of the discrepancy between the number of manually labeled boxes and pseudo-labeled boxes. Both components of the object detection loss (localization/classification) adopt this form of doubly robust loss.

Dataset and setting. To evaluate doubly robust self-training in the autonomous driving setting, we perform experiments on the large-scale 3D detection dataset nuScenes (Caesar et al., 2020). The nuScenes dataset is comprised of 1000 scenes (700 training, 150 validation and 150 test) with each frame containing sensor information from RGB camera, LiDAR, and radar scans. Box annotations are comprised of 10 classes, with the class instance distribution following a long-tailed distribution, allowing us to investigate our self-training approach for both common and rare classes. The main 3D detection metrics for nuScenes are mean Average Precision (mAP) and the nuScenes Detection Score (NDS), a dataset-specific metric consisting of a weighted average of mAP and five other true-positive metrics. For the sake of simplicity, we train object detection models using only LiDAR sensor information.

Results. After semi-supervised training, we evaluate our student model performance on the nuScenes val set. We compare three settings: training the student model with only the available labeled data

(i.e., equivalent to teacher training), training the student model on the combination of labeled/teacherlabeled data using the naive self-training loss, and training the student model on the combination of labeled/teacher-labeled data using our proposed doubly robust loss. We report results for training with 1/24, 1/16, and 1/4 of the total labels in Table 2. We find that the doubly robust loss improves both mAP and NDS over using only labeled data and the naive baseline in the lower label regime, whereas performance is slightly degraded when more labels are available. Furthermore, we also show a per-class performance breakdown in Table 3. We find that the doubly robust loss consistently improves performance for both common (car, pedestrian) and rare classes. Notably, the doubly robust loss is even able to improve upon the teacher in classes for which pseudo-label training decreases performance when using the naive training (e.g., barriers and traffic cones).

4 Conclusions

We have proposed a novel doubly robust loss for self-training. Theoretically, we analyzed the double-robustness property of the proposed loss, demonstrating its statistical efficiency when the pseudo-labels are accurate. Empirically, we showed that large improvements can be obtained in both image classification and 3D object detection.

As a direction for future work, it would be interesting to understand how the doubly robust loss might be applied to other domains that have a missing-data aspect, including model distillation, transfer learning, and continual learning. It is also important to find practical and efficient algorithms when the labeled and unlabeled data do not match in distribution.

References

N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic. Prediction-powered inference. arXiv preprint arXiv:2301.09633 , 2023.

Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2020.

Azriel, L. D. Brown, M. Sklar, R. Berk, A. Buja, and L. Zhao. Semi-supervised linear regression. Journal of the American Statistical Association , 117(540):2238-2251, 2022.

Bang and J. M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics , 61(4):962-973, 2005.

Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 , 2019a.

Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv:1905.02249 , 2019b.

M. Bibby, K. V. Mardia, and J. T. Kent. Multivariate Analysis . Academic Press, 1979.

Birhanu, G. Molenberghs, C. Sotto, and M. G. Kenward. Doubly robust and multiple-imputationbased generalized estimating equations. Journal of Biopharmaceutical Statistics , 21(2):202-225, 2011.

Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters, 2018a.

Chernozhukov, W. Newey, and R. Singh. De-biased machine learning of global and local parameters using regularized Riesz representers. arXiv preprint arXiv:1802.08667 , 2018b.

De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):3366-3385, 2021.

Ding, B. Xiao, N. Codella, P. Luo, J. Wang, and L. Yuan. Davit: Dual attention vision transformers. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pages 74-92. Springer, 2022.

Ding and F. Li. Causal inference. Statistical Science , 33(2):214-237, 2018.

J. Foster and V. Syrgkanis. Orthogonal statistical learning. arXiv preprint arXiv:1901.09036 , 2019.

Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision , 129:1789-1819, 2021.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

W. Holland. Statistics and causal inference. Journal of the American Statistical Association , 81 (396):945-960, 1986.

Jeong, S. Lee, J. Kim, and N. Kwak. Consistency-based semi-supervised learning for object detection. In Advances in Neural Information Processing Systems , 2019.

M. Jiang, M. Najibi, C. R. Qi, Y. Zhou, and D. Anguelov. Improving the intra-class long-tail in 3d detection via rare example mining. In Computer Vision-ECCV 2022: 17th European Conference , pages 158-175. Springer, 2022.

D. Kang and J. L. Schafer. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science , 22:523-539, 2007.

H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12689-12697, 2019.

D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL) , 07 2013.

Li, Z. Liu, J. Hou, and D. Liang. Dds3d: Dense pseudo-labels with dynamic threshold for semisupervised 3d object detection. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , 2023.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV , pages 2980-2988, 2017.

Liu, C. Gao, F. Liu, P. Li, D. Meng, and X. Gao. Hierarchical supervision and shuffle data augmentation for 3d semi-supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Y.-C. Liu, C.-Y. Ma, Z. He, C.-W. Kuo, K. Chen, P. Zhang, B. Wu, Z. Kira, and P. Vajda. Unbiased teacher for semi-supervised object detection. In Proceedings of the International Conference on Learning Representations (ICLR) , 2021a.

Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021b.

Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

W. Marshall and I. Olkin. Multivariate chebyshev inequalities. The Annals of Mathematical Statistics , pages 1001-1014, 1960.

J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345-1359, 2010.

Park, C. Xu, Y. Zhou, M. Tomizuka, and W. Zhan. Detmatch: Two teachers are better than one for joint 2D and 3D semi-supervised object detection. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel , 2022.

R. Qi, Y. Zhou, M. Najibi, P. Sun, K. Vo, B. Deng, and D. Anguelov. Offboard 3D object detection from point cloud sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika , 70(1):41-55, 1983.

B. Rubin. Inference and missing data. Biometrika , 63(3):581-592, 1976.

B. Rubin. Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association , 74(366a):318-328, 1979.

Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision , 115:211-252, 2015.

O. Scharfstein, A. Rotnitzky, and J. M. Robins. Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association , 94(448): 1096-1120, 1999.

Semenova, M. Goldman, V. Chernozhukov, and M. Taddy. Estimation and inference on heterogeneous treatment effects in high-dimensional dynamic panels. arXiv preprint arXiv:1712.09988 , 2017.

N. Smith and N. Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications , volume 11006, page 1100612. International Society for Optics and Photonics, 2019.

Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07685 , 2020a.

Sohn, Z. Zhang, C.-L. Li, H. Zhang, C.-Y. Lee, and T. Pfister. A simple semi-supervised learning framework for object detection. In arXiv:2005.04757 , 2020b.

Tang, W. Chen, Y. Luo, and Y. Zhang. Humble teachers teach better students for semi-supervised object detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3131-3140, 2021.

Wang, Y. Cong, O. Litany, Y. Gao, and L. J. Guibas. 3dioumatch: Leveraging iou prediction for semi-supervised 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14615-14624, 2021.

Weiss, T. M. Khoshgoftaar, and D. Wang. A survey of transfer learning. Journal of Big Data , 3 (1):1-40, 2016.

Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

Yin, X. Zhou, and P. Krähenbühl. Center-based 3D object detection and tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Zhang, L. D. Brown, and T. T. Cai. Semi-supervised inference: General theory and estimation of means. Annals of Statistics , 47:2538-2566, 2019.

Zhou, Z. Ge, S. Liu, W. Mao, Z. Li, H. Yu, and J. Sun. Dense teacher: Dense pseudo-labels for semi-supervised object detection. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision - ECCV 2022 , pages 35-50, 2022.

Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.09492 , 2019.

J. Zhu. Semi-supervised learning literature survey. 2005.

A Implementation Details for Image Classification

We evaluate our doubly robust self-training method on the ImageNet100 and mini-ImageNet100 datasets, which are subsets of ImageNet-1k from ImageNet Large Scale Visual Recognition Challenge 2012 (Russakovsky et al., 2015). Two models are evaluated: (1) DaViT-T (Ding et al., 2022), a state-of-the-art 12-layer vision transformer architecture with a patch size of 4, a window size of 7, and an embedding dim of 768, and (2) ResNet50 (He et al., 2016), a classic and powerful convolutional network with 50 layers and embedding dim 2048. We evaluate all the models on the same ImageNet100 validation set (50 samples per class). For the training, we use the same data augmentation and regularization strategies following the common practice in Liu et al. (2021b); Lin et al. (2017); Ding et al. (2022). We train all the models with a batch size of 1024 on 8 Tesla V100 GPUs (the batch size is reduced to 64 if the number of training data is less than 1000). We use AdamW (Loshchilov and Hutter, 2017) optimizer and a simple triangular learning rate schedule (Smith and Topin, 2019). The weight decay is set to 0 . 05 and the maximal gradient norm is clipped to 1 . 0 . The stochastic depth drop rates are set to 0 . 1 for all models. During training, we crop images randomly to 224 × 224 , while a center crop is used during evaluation on the validation set. We use a curriculum setting where the α t grows linearly or quadratically from 0 to 1 throughout the training. To show the effectiveness of our method, we also compare model training with different curriculum learning settings and varying numbers of epochs.

B Additional Experiments in Image Classification

Ablation study on curriculum settings. There are three options for the curriculum setting: 1) α t = 1 throughout the whole training, 2) grows linearly with training iterations α t = t/T , 3) grows quadratically with training iterations α t = ( t/T ) 2 . From results in Table 4, we see: the first option achieves comparable performance with the 'Naive Labeled + Pseudo' baseline. Both the linear and quadratic strategies show significant performance improvements: the linear one works better when more labeled data is available, e.g., 70% and 90%, while the quadratic one prefers less labeled data, e.g. 30% and 50%.

Ablation Study on the Number of Epochs. We conduct experiments on different training epochs. The results are shown in Table 5. Our model is consistently superior to the baselines. And we can observe the gain is larger when the number of training epochs is relatively small, e.g. 20 and 50.

Fully trained results (300 epochs) on ImageNet-100. In our original experiments, we mostly focus on a teacher model that is not super accurate, since our method reduces to the original pseudo-labeling when the teacher model is completely correct for all labels. In this experiment, we fully train the teacher model with 300 epochs on ImageNet-100, leading to the accuracy of the teacher model as high as 88.0%. From Figure 2, we show that even in this case, our method outperforms the original pseudo-labeling baseline.

20

50

80

100

Labeled Data Fraction (%)

65

70

75

80

85

90

Top-1 Acc (%)

63.6

81.2

85.5

88.0

64.9

82.9

87.0

89.4

67.2

85.5

87.6

90.6

Labeled Only

Labeled + Pseudo

Doubly-Robust (Ours)

Comparisons with previous SOTAs on CIFAR-10 and CiFAR-100. We compare with another 11 baselines in terms of error rate on CIFAR-10-4K and CIFAR-100-10K under the same settings (i.e., Wide ResNet-28-2 for CIFAR-10 and WRN-28-8 for CIFAR-100). We show that our method is only 0.04 inferior to the best method Meta Pseudo Labels for CIFAR-10-4K, and achieves the best performance for CIFAR-100-10K.

C Implementation Details of 3D Object Detection

Our experiments follow the standard approach for semi-supervised detection: we first initialize two detectors, the teacher (i.e., labeler) and the student. First, a random split of varying sizes is selected from the nuScenes training set. We pre-train the teacher network using the ground-truth annotations in this split. Following this, we freeze the weights in the teacher model and then use it to generate pseudo-labels on the entire training set. The student network is then trained on a combination of the pseudo-labels and ground-truth labels originating from the original split. In all of our semi-supervised experiments, we use CenterPoint with a PointPillars backbone as our 3D detection model (Yin et al., 2021; Lang et al., 2019). The teacher pre-training and student training are both conducted for 10 epochs on 3 NVIDIA RTX A6000 GPUs. We follow the standard nuScenes training setting outlined in Zhu et al. (2019), with the exception of disabling ground-truth paste augmentation during training to prevent data leakage from the labeled split. To select the pseudo-labels to be used in training the student, we simply filter the teacher predictions by detection confidence, using all detections above a chosen threshold. We use a threshold of 0.3 for all classes, as in Park et al. (2022). In order to conduct training in a batch-wise manner, we compute the loss over only the samples contained within

the batch. We construct each batch to have a consistent ratio of labeled/unlabeled samples to ensure the loss is well-defined for the batch.

D Additional Experiments in 3D Object Detection

Comparison with Semi-Supervised Baseline We compare our approach to another semi-supervised baseline on the 3D object detection task, Pseudo-labeling and confirmation bias Arazo et al. (2020). We shows that in multiple settings, our approach surpasses the baseline performance.

Ablation on Pseudo-label Confidence Threshold To demonstrate that appropriate quality pseudolabels are used to train the student detector, we performance ablation experiments varying the detection threshold used to extract pseudo-labels from the teacher model predictions. We show that training with a threshold of τ = 0 . 3 outperforms training with a more stringent threshold, and is the appropriate experimental setting for our main experiments.

E Considerations when ˆ f is trained from labeled data

In Theorem 2, we analyzed the double robustness of the proposed loss function when the predictor ˆ f is pre-existing and not trained from the labeled dataset. In practice, one may only have access to the labeled and unlabeled datasets without a pre-existing teacher model. In this case, one may choose to split the labeled samples D 2 into two parts. The last n/ 2 samples are used to train ˆ f , and the first n/ 2 samples are used in the doubly robust loss:

Since ˆ f is independent of all samples used in the above loss, the result in Theorem 2 continues to hold. Asymptotically, such a doubly robust estimator is never worse than the estimator trained only on the labeled data.

F Proof of Proposition 1

For the labeled-only estimator ˆ θ TL , we have

For the self-training loss, we have

For the doubly robust loss, on one hand, we have

On the other hand, we have

The proof is done by taking the minimum of the two upper bounds.

G Proof of Theorem 2

Proof. We know that

From the multi-dimensional Chebyshev inequality (Bibby et al., 1979; Marshall and Olkin, 1960), we have that with probability at least 1 -δ/ 2 , for some universal constant C ,

Similarly, we also have that with probability at least 1 -δ/ 2 ,

Furthermore, note that

Here we use Assumption 1 and Assumption 2 to ensure that the expectation and differentiation are interchangeable. Thus we have that with probability at least 1 -δ ,

On the other hand, we can also write the difference as

Here the last inequality uses the multi-dimensional Chebyshev inequality and it holds with probability at least 1 -δ . This finishes the proof.

H Proof of Proposition 3

Proof. We have

In the first case when π ( x ) ≡ P X ( x ) Q X ( x ) , we have

In the second case when ℓ θ ( x, ˆ f ( x )) = E Y ∼ P Y | X = x [ ℓ θ ( x, Y )] , we have

This finishes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a very simple approach to semi-supervised learning that utilizes both labeled and unlabeled datasets. When there is a large amount of unlabeled data available, following the same distribution as the labeled dataset, the most effective method to leverage this unlabeled data for training is self-training, where pseudo labels are generated and used for training. The main limitation of self-training is that the performance can degrade when the pseudo-labels produced by the predictor for the unlabeled set are not accurate. \nIn this paper, the authors propose a very simple method to overcome this limitation by replacing the conventional loss for the labeled dataset and the unlabeled dataset in self-training. This modification allows the model to be trained effectively in both cases, whether the pseudo-labels are correct or not, leading to improved performance in all scenarios. The effectiveness of this approach is experimentally demonstrated on image classification benchmarks and 3D object detection benchmarks.\n\nSTRENGTHS:\nOverall, the method is very simple, intuitive, and quite novel. The paper is well-written and it provides a well-derived explanation using equations for both cases where the pseudo labels are accurate and when they are not. Additionally, the method is thoroughly analyzed from a theoretical perspective.\n\nWEAKNESSES:\nAlthough the theoretical derivation demonstrates the soundness of the method, there is doubt regarding its effectiveness in experiments.\n\n1. Especially, in the experiments (sec. 3.1), the authors use curriculum-based loss in each epoch. With $\\alpha_t  < 1 $, the proposed method will behave like self-training (exactly when $\\alpha_t = \\frac{n}{m+n}$) and the behavior/effectiveness of the proposed method could not be well represented in this case.  \n\n2. Image classification experiments were conducted using ImageNet-100. However, when the labeled set ratio is 100%, the top-1 accuracy of DaVIT and ResNet50 is remarkably lower at 47.8% and 46.7%, respectively, compared to the reported top-1 accuracy in other papers. Following the paper of DaVit and ResNet, the top-1 accuracy on the more complex task of ImageNet1k, compared to ImageNet-100, using DaVit-tiny and ResNet is 82.8% and 79.26%, respectively. Following [1], the top-1 accuracy of various resnet50-based methods for ImageNet-100 consistently surpasses 70%. It appears that the baseline has not been sufficiently well-trained.\n\n3. There is a lack of comparison with other semi-supervised methods. While this method compares with the basic self-training loss, it is necessary to compare it with various methods that utilize pseudo-labels for self-training [2], [3], [4]. Most semi-supervised image classification methods have performed experiments on benchmark datasets such as ImageNet1k and CIFAR100. These methods achieve a top-1 accuracy of 72% or higher when using a labeled dataset of 10% on ImageNet1k. Therefore, it is necessary to compare the effectiveness of this method on these benchmark datasets.\n\n4. Similarly, in semi-supervised 3D object detection, it is necessary to compare the results with existing baseline methods [5], [6].\n\n5. Although this paper proposes a method for the distribution mismatch case in Section 2.4, estimating the probability values of each sample, p(x) and q(x), for each distribution is not easy. Practical methods for this issue have not been provided, and experiments on this aspect are also lacking. Practical algorithms and experiments need to be provided in order to address this concern.\n\n[1] Zelin Zang, et al. DLME: Deep Local-flatness Manifold Embedding. ECCV 2022\n\n[2] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 2020\n\n[3] Kihyuk Sohn, David Berthelot, Zizhao Li, Chun-Liang Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In IEEE Conference on Computer Vision and Pattern Recognition, 2020\n\n[4]  Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le. Meta pseudo labels. In IEEE Conference on Computer Vision and Pattern Recognition, 2021\n\n[5] He Wang, Yezhen Cong, Or Litany, Yue Gao and Leonidas J. Guibas. 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D \nObject Detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2021\n\n[6] Na Zhao, Tat-Seng Chua and Gim Hee Lee. SESS: Self-Ensembling Semi-Supervised 3D Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2020\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nFuture work is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper proposed a doubly robust loss for self-training. The proposed loss is analysed and shown to have preferable theoretical properties.

STRENGTHS:
1. The idea is interesting: a simple change from 1/(m+n) to 1/n (line 51 - 53) lead to a doubly robust loss function for self-training. 
2. The writing is clear and easy to follow

WEAKNESSES:
1. While the proposed doubly robust loss for self-training enjoys theoretical advantages, directly minimizing the loss during network training leads to instability. The actual loss used in line 219 is very different especially in the early epochs. 

2. Lack of comparison to other stronger semi-supervised learning baselines. For example, the authors discuss MixMatch and FixMatch as related work but did not compare with them in experiments.

minor:
1. I don't think the description in line 79-87 is precise. MixMatch/FixMatch does not pre-trained a teacher model on 'labeled' data.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a very simple yet effective modification to the original loss for self-training by re-weighting terms of the loss function making. This change effectively balances between using the pseudo-labels when the predictor is strong and learning to not use it when it is unreliable, making it doubly robust. They provide a sound theoretical analysis and  empirical evaluation on classification and object detection tasks substantiating their claims.\n\nSTRENGTHS:\n- Strengths\n    - The paper is well written and easy to follow with sufficient background and motivated examples given to present the chain of reasoning well.\n    - For linear predictor, the proposed loss is unbiased with lower variance which is strictly better than self-training\n    - The results on both classification and 3D object detection highlight clear improvement over standard self distillation\n    - The technical novelty in the paper is limited with just a small modification to the overall loss. But the theoretical insights including guarantees for general loss and positive experimental results make it a meaningful contribution. The simplicity of the modification also make it much more likely to be adopted and have higher impact.\n\nWEAKNESSES:\n- Weaknesses\n    - The analysis provided is for very simplistic settings of linear predictor or mean-predictions and it’s unclear how much of it translates to realistic settings of over-parameterized deep nets trained on SGD.\n    - For image classification, the baselines used in the paper are meaningful but thorough comparison with other state of the art self-imitation methods like noisy-student etc is lacking.\n    - The evaluation is restricted to lower data regimes as the model shines when the training data is limited, making their evaluation a bit more contrived as compared to real world scenarios. It seems the proposed loss should shine more when unlabelled data scales, so some experimentation on the impact on performance as the unlabelled data scales would also be interesting to see\n    - One important use of self-distillation is using unlabelled data to show domain adaptation to that domain. I would like to see experiments for that as well.\n\nQUESTIONS:\n* In Section 2.1, the intuitive interpretation for when the predictor is bad is not super clear and paper would benefit from some elaboration. If just m goes to infinity, how would the two terms cancel out\n\n* In the case of distribution mismatch, what’s the motivation for changing the first term to average of (n) sample instead of (m+n)? Also how are $\\pi (x)$ ie importance weights chosen?\n\n* Did the authors explore tuning the predictor $\\hat{f}$ on the labelled examples before generating the pseudo-examples to train the student?\n\nTypos and possible errors\n- Inconsistent notation : line 155 should be $\\hat{\\theta}$ instead of $\\theta^{*}$\n- In line 161, I believe the upper bound should be $6/n(var[Y] ..) $ instead of $4/n$. Please verify!\n- in equation in line 171, it should be $\\theta_{SL}$ instead of $\\theta_{DR}$\n\nLIMITATIONS:\nThe authors have adequately addressed some limitations. Other suggestions are listed above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pseudo-labeling approach that balances out the supervised signal between the labeled and incorrect pseudo-labeled datapoints during the training process. The aim is to only account for the pseudo-labels when they are correctly labeled, which may happen when the covariate distribution of the unlabeled dataset and the labeled dataset matches. They show some analysis of the proposed loss and results in ImageNet100 (a subset with 100 random classes from ImageNet-1k) and mini-ImageNet100  and nuScenes dataset.\n\nSTRENGTHS:\nThe proposed method is clearly written, and the paper can easily be understood.\nThe paper with all necessary details and deep explanations for experimental results.\nComprehensive algorithmic analysis and clear motivation.\nThe proposed method effectively improves over vanilla pseudo-labeling [1]\n\n[1] D.-H. Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop : Challenges in Representation Learning (WREPL), 07 2013.\n\nWEAKNESSES:\nNovelty and missing prior work: Pseudo-labeling is the defacto method for entropy regularization techniques in semi-supervised learning problems. This is a paper that tackles the semi-supervised learning problem. Recent work has explored different ways to mitigate the error propagation from the teacher model and confirmation bias present in pseudo-labeling approaches. For example, [2,3,4] investigate the thresholding effect via fixed and curriculum based approaches, with flexible thresholds that are dynamically adjusted for each class according to the current learning status. With such prior exploration and no comparison with any of that work, it's difficult to assess the importance and impact of this work, which seems limited and incomplete.\n\nThe paper in its current state also fails to provide technical details to validate fair comparisons in the main text. Furthermore, no ablations for any of the technical selections are conducted.\nThe alternative loss proposed for distribution mismatch (Section 2.4) is only shown in the method but not in the empirical section.\n\n[2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In IJCNN, pages 1–8, 2020.\n\n[3] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6912–6920, 2021.\n\n[4] Zhang B, Wang Y, Hou W, Wu H, Wang J, Okumura M, Shinozaki T. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems. 2021 Dec 6;34:18408-19.\n\nQUESTIONS:\nPrior literature [2,3,4,5] has shown that predefined threshold values impact the overall performance in pseudo-labeling. Using 0.3 as the threshold seems to be too permissive, allowing too many noisy pseudo-labels. How sensitive is this threshold in your setup?\n\n[5] Oliver A, Odena A, Raffel CA, Cubuk ED, Goodfellow I. Realistic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems. 2018;31.\n\nLIMITATIONS:\nNo limitation section is provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}