Title: InstanT: Semi-supervised Learning with Instance-dependent Thresholds
Venue: NeurIPS 2023
Accepted: yes
Keywords: semi-supervised learning

pseudo-labeling
Abstract: Semi-supervised learning (SSL) has been a fundamental challenge in machine learning for decades. The primary family of SSL algorithms, known as pseudo-labeling, involves assigning pseudo-labels to confident unlabeled instances and incorporating them into the training set. Therefore, the selection criteria of confident instances are crucial to the success of SSL. Recently, there has been growing interest in the development of SSL methods that use dynamic or adaptive thresholds. Yet, these methods typically apply the same threshold to all samples, or use class-dependent thresholds for instances belonging to a certain class, while neglecting instance-level information. In this paper, we propose the study of instance-dependent thresholds, which has the highest degree of freedom compared with existing methods. Specifically, we devise a novel instance-dependent threshold function for all unlabeled instances by utilizing their instance-level ambiguity and the instance-dependent error rates of pseudo-labels, so instances that are more likely to have incorrect pseudo-labels will have higher thresholds. Furthermore, we demonstrate that our instance-dependent threshold function provides a bounded probabilistic guarantee for the correctness of the pseudo-labels it assigns.

InstanT: Semi-supervised Learning with Instance-dependent Thresholds

Abstract

Semi-supervised learning (SSL) has been a fundamental challenge in machine learning for decades. The primary family of SSL algorithms, known as pseudo-labeling, involves assigning pseudo-labels to confident unlabeled instances and incorporating them into the training set. Therefore, the selection criteria of confident instances are crucial to the success of SSL. Recently, there has been growing interest in the development of SSL methods that use dynamic or adaptive thresholds. Yet, these methods typically apply the same threshold to all samples, or use class-dependent thresholds for instances belonging to a certain class, while neglecting instance-level information. In this paper, we propose the study of instance-dependent thresholds, which has the highest degree of freedom compared with existing methods. Specifically, we devise a novel instance-dependent threshold function for all unlabeled instances by utilizing their instance-level ambiguity and the instance-dependent error rates of pseudo-labels, so instances that are more likely to have incorrect pseudo-labels will have higher thresholds. Furthermore, we demonstrate that our instance-dependent threshold function provides a bounded probabilistic guarantee for the correctness of the pseudo-labels it assigns. Our implementation is available at https://github.com/tmllab/2023_NeurIPS_InstanT .

1 Introduction

In recent years, machine learning algorithms trained on abundant accurately labeled data have demonstrated unparalleled performance across different domains. However, in practice, it is often financially infeasible to collect reliable labels for all samples in large-scale datasets. A more practical solution is to select a subset of the data and use expert annotations to assign labels to them [39]. This scenario, where the majority of the data is unlabeled and only a portion has reliable labels, is known as semi-supervised learning (SSL). In SSL, our aim is usually to learn a classifier that has comparable performance to the one trained in a fully supervised manner. To achieve this, the majority of existing approaches adopt a training strategy named pseudo-labeling. More specifically, a model will be trained on a small set of labeled data first, and this model will then be applied to the larger unlabeled set to assign predicted pseudo-labels to them. If the model's confidence in an instance exceeds a certain threshold, then this instance along with its predicted pseudo-label will be added to the training set for this iteration. Thus by iteratively expanding the training set, if the expanded instances are indeed assigned with correct labels, then the classification error will be gradually reduced [43].

However, since the model is bound to make mistakes, the newly added training samples are not always assigned with correct labels, which generates label noise [5, 46, 44]. Under the influence of label noise, the model will gradually over-fits to noisy supervision, hence accumulating generalization error. Since the model makes predictions based on instance features, it is usually assumed that for

T（C2）

口

口

T(△)

△

△

△

△

△

△

△

△

T(O)

△

T(c1)

"hard" examples, the classifier is more likely to give incorrect predictions to them [55], making the label noise pattern instance-dependent [11, 45, 55].

Consequently, selecting an appropriate confidence threshold for pseudo-label assignment becomes a key factor that determines the performances of SSL methods. For predictions that are more likely to have incorrect labels, we want to assign higher thresholds to avoid selecting them prematurely. And for predictions that are more likely to be correct, we want to assign lower thresholds to encourage adding them to the training set at an earlier stage. Conventionally, such a threshold is usually determined by empirical experience in an ad-hoc manner, where only a few of the existing works consider the theoretical implication of threshold selection. Recently, increasing research focused on the investigation of dynamic or adaptive thresholds [9, 17, 41, 48, 54], where such confidence threshold is conditional on some external factors, such as the current training progress. Notably, the vast majority of those methods consider a single threshold for all instances and overlook their instance-level information, such as the instance-level label noise rate. As illustrated in Figure 1, the instance-dependent threshold is the most flexible among existing threshold types.

Motivated by this challenge, in this paper, we propose a new SSL algorithm named Instan cedependent T hresholding ( InstanT ). Since the learned model is subject to the influence of instancedependent label error, we aim to quantify and reduce such error by estimating an instance-dependent confidence threshold function based on the potential label noise level and instance ambiguity (clean class posterior). In addition, we can derive a lower-bounded probability, for samples that satisfy the thresholds will be assigned to a correct label. From our main theorem, we show that as the training progresses, this probability lower-bound will asymptotically increase towards one, hence guaranteeing the pseudo-label quality for instances that satisfies our proposed thresholds.

We summarize our contributions as follows: (1) We propose an SSL algorithm named InstanT, which assigns thresholds to individual unlabeled samples based on the instance-dependent label noise level and prediction confidence. (2) We prove InstanT has a bounded probability to be correct, which vouch for the reliability of the pseudo-labels it assigns with a theoretical guarantee. (3) To the best of our knowledge, this is the first attempt to estimate the instance-dependent thresholds in SSL, which has the highest degree of freedom compared with existing methods. (4) Through extensive experiments, we show that our proposed method is able to surpass state-of-the-art (SOTA) SSL methods across multiple commonly used benchmark datasets.

Related work. As we mentioned, some existing works have already been considered to improve pseudo-labeling by leveraging "dynamic" or "adaptive" thresholds. More specifically, Dash [48] considers a monotonically decreasing dynamic threshold, based on the intuition that as training progresses, the model at a later stage will provide more reliable predictions than at early stages. FlexMatch [54] further introduced class-dependent learning status in addition to the dynamic threshold. Adsh [17] employs adaptive threshold under class imbalanced scenario by optimizing the number of pseudo-labels per class. As for adaptive thresholds, AdaMatch [9] considers using a pre-defined threshold factor multiplied by the averaged top-1 prediction confidence, making it adaptive to the

model's current confidence level. FreeMatch [41] considers a combination of adaptive global and local thresholds, which combines the training progress and class-specific information.

In addition to the aforementioned more recent methods, most of the classical methods use pre-defined fixed thresholds, which remain constant throughout the entire training process [8, 18, 19, 29, 36]. Such thresholds are usually set to be high enough to prevent the occurrence of incorrect pseudolabels. Notably, other approaches such as reweighting [10, 22], distribution alignment [20, 31, 42], contrastive learning [24, 49], consistency regularization [1, 34] were also employed to improve SSL.

2 Preliminaries

2.1 Notation and Problem Setting

We consider the general setting from SSL, namely, we have a small group of labeled instances X l := { x 1 , ..., x n } with their corresponding labels Y l := { y 1 , ..., y n } . And there is an unlabeled instance group X u := { x n +1 , ..., x n + m } with their latent labels Y u := { y n +1 , ..., y n + m } , usually, we have m>>n . Furthermore, we assume that the labeled instances and unlabeled instances are independent and identically distributed (i.i.d), i.e. X := { X l , X u } ∈ X , Y := { Y l , Y u } ∈ Y .

We term the model trained with clean and pseudo-labels as ˆ f , parameterized by θ . Since ˆ f θ is bound to make mistakes, its generated pseudo-label ˆ Y u will contain label errors. Hence the softmax predictions of ˆ f θ are actually approximating the noisy class posterior. We fix the notation for the real noisy class posterior as P ( ˆ Y | X ) t at t -th iteration, whereas our approximated noise class posterior is ˆ P ( ˆ Y | X ) t , which can be obtained from the softmax predictions of the model ˆ f θ at t -th iteration.

As the clean label for the unlabeled set is unknown, we instead wish to recover the Bayes optimal label for all the unlabeled instances, which is the label class that maximizes the clean class posterior [50, 56]. The Bayes optimal label is generated by the hypothesis that minimizes the risk within the hypothesis space, i.e. the Bayes optimal classifier, which we denote as h ∗ . And we can denote the Bayes optimal label for x as h ∗ ( x ) .

2.2 Instance-dependent Label Noise in SSL

As mentioned in the previous section, label noise is an inevitable challenge in SSL. Moreover, since P ( ˆ Y | X ) is dependent on X , we say the label noise is instance-dependent [12, 45]. Concretely, we have P ( ˆ Y | X = x ) = T ( x ) P ( Y | X = x ) [32, 35], where T ( x u ) is the instance-dependent transition matrix , which models the generation of label noise. P ( Y | X = x ) represents the latent clean class posterior for unlabeled samples. We can define the ij -th entry of T ( x ) as:

which means the ij -th entry of T ( x ) models the probability that x with clean label y = i will be predicted as ˆ y = j .

2.3 Tsybakov Margin Condition

We assume our learned classifier satisfies Tsybakov Margin Condition [38], which essentially restrained the level of complexities of the classification problem by assuming the data are separable enough. It is a commonly used assumption for various Machine Learning problems [2, 6, 56] including SSL [48]. Without the loss of generality, we will directly present the multi-class Tsybakov Margin Condition [2, 56], whereas the binary case will be provided in the Appendix B.

Assumption 1 (Multi-class Tsybakov Margin Condition) . For some finite constant C, α > 0 , and δ 0 ∈ (0 , 1] , the Tsybakov Margin Condition holds if ∀ δ ∈ (0 , δ 0 ] , we have

Where P ( Y = γ | X = x ) and P ( Y = s | X = x ) are the largest and second-largest clean class posterior probability for x .

2.4 Quality-Quantity Trade-off in SSL

To understand the choice of pseudo-label thresholds in SSL, it is essential to consider the trade-off between the quality and the quantity of the pseudo-labels [10, 41]. Typically, a higher threshold is presumed to yield superior label quality, as it assigns pseudo-labels solely to instances where the classifier exhibits a high level of confidence. Conversely, elevating the threshold diminishes the quantity of pseudo-labels, as it leads to the exclusion of instances where the classifier lacks sufficient confidence. Hence causing the dilemma of quality-quantity trade-off.

One possible approach to address the quantity-quality trade-off in SSL is to apply a dynamic threshold, which is dependent on the training progress. This is because the evaluation metric for confident samples is usually not stationary throughout the training process of neural networks. As the model's predicted probability is subject to the level of over-fitting [16, 30], designing a thresholding function that is dependent on the training process is crucial to the success of SSL algorithms[9, 41]. Intuitively, in the early stage of the training process, where the model has just started to fitting, the threshold should not be set too high to filter out too many samples, and as the model has already well-fitted to the training data, the threshold should be elevated to combat falsely confident instances caused by confirmation bias [4].

3 Instance-dependent thresholds - A theoretical perspective

In this section, we focus on answering the specific research question: For SSL, can we derive instance-dependent pseudo-label assignment thresholds with theoretical guarantees?

The answer to this question is affirmative. We will present our main theorem and the proof to show that for samples that satisfy our instance-dependent threshold function, their likelihood of being correct is lower-bounded.

3.1 Lower-bound Probability of Correctness

For a classifier trained with sufficient accurately labeled data, its prediction γ will be made if P ( Y = γ | X = x ) > P ( Y = s | X = x ) . However, under the existence of label errors, the fidelity of the classifier can no longer be guaranteed. Recently, Zheng et al. [56] showed that the classifier influenced by the label noise can still have a bounded error rate. We show that in SSL tasks, where the label noise is instance-dependent, we can establish a lower-bound probability for the predictions to be correct through similar derivations.

̸

Recall Assumption 1, we have γ := arg max i P ( Y = i | X = x u ) , s := arg max i = k P ( Y = i | X = x u ) , k := arg max i ˆ P ( ˆ Y = i | X = x u ) = ˆ y u . And let ϵ θ be the estimation error between our estimated noisy class posterior ˆ P ( ˆ Y | X ) and true noisy class posterior P ( ˆ Y | X ) . We can then formulate our main theorem:

Theorem 1. Assume estimated noisy class posterior ˆ P ( ˆ Y | X ) satisfies Assumption 1 with C, α > 0 , δ 0 ∈ (0 , 1] , and ϵ θ ≤ δ 0 min i T i,i ( x u ) , we have:

where τ ( x u ) is the instance-dependent threshold function:

̸

The proof of Theorem 1 is provided in Appendix C. Theorem 1 substantiates that, as our model over-fits to the label error (as ϵ θ decreases), and the transition matrix can be successfully estimated, the probability lower-bound of the assigned pseudo-labels are correct will increase asymptotically. This guarantees the quality of the pseudo-labels τ ( x u ) assigned. Note that, τ ( x u ) requires the clean class posterior of unlabeled instances, which can be provably inferred upon the successful estimation of T ( x ) and the noisy class posterior [26, 32].

To gain a deeper understanding of Theorems 1, we will try to intuitively understands the outputs of τ ( x u ) . Our aim is to examine the behavior of τ ( x u ) in relation to the unlabeled sample x u , which

Classifier

Classifier

P(Y|X =αu) · T(αu)-1

P(Y|X = ci)

0.7

X1

fe(x)

0.10.80.1

1X

fe(x)

0.60.40.2

×

0.4

=T(xu)

↑

0.2

Yi

L(0')=

=-log(0.6)

Yi

↑

0.50.40.1

0.70.20.1

nX

0.30.60.1

f(α)

0.40.50.1

0.10.10.8

0.20.20.6

T(xi)

T(cu)

NoiseTransitionEstimator

NoiseTransitionEstimator

has a predetermined clean class posterior. If we observe an increase in the sum of column k in the transition matrix, it will result in a corresponding increase in the threshold τ ( x u ) . This implies that when x u is highly likely to be assigned with an incorrect pseudo-label, τ ( x u ) will assign a higher threshold in order to prevent its premature inclusion of { x u , ˆ y u } in the training set. Conversely, when x u is less likely to be assigned with a incorrect pseudo-label, τ ( x u ) will assign a lower threshold to encourage adding { x u , ˆ y u } to the training set.

3.2 Identification of Instance-dependent Transition Matrix in SSL

Theorem 1 builds upon the assumption that T ( x ) can be successfully identified. Yet, the identification of T ( x ) has been a long-standing issue in the field of label noise learning. Fortunately, in SSL, we argue that T ( x ) can be provably identified under mild conditions. We will first justify our argument from a theoretical perspective based on the Theorems from Liu et al. [27]. Subsequently, we will also propose the empirical methods for estimating T ( x ) from an intuitively understandable aspect in the following section.

We will start by defining the identifiability, we use Ω to represent an observation space and use Θ to represent a general parametric space. Then, for a distribution with parameter θ ′ , such that θ ′ ∈ Θ , a model P θ ′ on Ω can be defined [3, 50]. We further define its identifiability as:

̸

Definition 1 (Identifiability) . Parameter θ ′ is said to be identifiable if P θ ′ = P θ ′′ , ∀ θ ′ = θ ′′ .

̸

In our case, θ ′ ( x ) := { T ( x ) , P ( Y | X = x ) } , i.e., P θ ′ is a distribution defined by the transition probability and clean class posterior. To determine the identifiability of T ( x ) , we also need to define the term informative noisy labels . Specifically, we have:

Definition 2 (Informative noisy labels) . For a given sample ( x , y ) i , its noisy label ˆ y i is said to be informative if rank ( T ( x i )) = | Y | .

Then, we can make the following theorem:

Theorem 2. With i.i.d ( x , y ) i pairs, three informative noisy labels ˆ y i are sufficient and necessary for the identification of T ( x i ) .

The Theorem 2 is based on Kruskal's Identifiability Theorem, the proof is provided in the Appendix D. From Theorem 2 and Definition 2, we can conclude that, in SSL, if we have more than three labeled samples per class, transition matrix T ( x ) is identifiable, more discussions on this can also be found in Appendix D.

4 InstanT: Instance-dependent thresholds for pseudo-label assignment

4.1 Instance-dependent Threshold Estimation

While Theorem 1 provides a theoretical guarantee for the correctness of τ ( x u ) , as discussed in section 2.4, SSL algorithms can benefit from non-constant thresholds [9, 41]. Here we define the dynamic threshold κ t at iteration t from the relative confidence threshold (RT) in AdaMatch [9]:

where n is the number of labeled samples, and β is a fixed discount factor. Subsequently, we present the instance-dependent threshold function at iteration t :

̸

Note that this estimation process requires the clean class posterior, which can be estimated or approximated using a transition matrix via importance re-weighting [26, 46] or loss correction [32, 45], details will be introduced in the following section. We also remark here that a less strict instance-dependent version of the threshold function can be used with the class-dependent transition matrix [32, 46, 52], which can lead to better empirical results under certain cases, it is also important to note that even under class-dependent transition matrix, the threshold function is still instancedependent, as the noise class posterior are instance-dependent.

4.2 Modelling Instance-dependent Label Noise

Now we will introduce a piratical method for estimating T ( x ) in SSL. Our approach involves approximating the label noise transition pattern using a Deep Neural Network. As we recall from Equation 1, T ( x ) can be described as a mapping function T : X → R | Y |×| Y | , since we already know that, in general cases, T ( x ) is identifiable in SSL, we only need to find the correct mapping function. In addition, since we have clean label Y l for X l , and we also have the noisy pseudo-labels ˆ Y l for X l , this enables us to directly fit a model to approximate T ( x ) . Therefore, we want our transition matrix estimator, parameterized by θ ′ , to approximate the label noise transition process at every iteration t :

The above approximation holds since label noise is assumed to be i.i.d, therefore both labeled and unlabeled samples share the same noise transition process. As we can estimate the noisy class posterior for labeled samples, we can therefore collect distribution D := { X l , Y l , ˆ P ( ˆ Y | X = x l ) } . Observing Equation 6, we can design following objective for θ ′ to minimize:

Where y l is the one-hot vector for the label of labeled samples, thus y l · ˆ T ( x l ) is equivalent to finding the class-conditional noise class posterior ˆ P ( ˆ Y l = j | Y l = i, X l = x l ; θ ′ ) . ˆ y l is the one-hot noisy pseudo-label of x l , minimizing L D ( θ ′ ) will result ˆ T ( x l ; θ ′ ) approximate T ( x ) . As shown in Figure 2, ˆ f θ ′ will be trained in parallel to the main classifier and updated in every epoch.

Notably, the estimator for the transition matrix in InstanT is not constrained to one specific method, there has been a wide versatile of T estimator in the field of label noise learning [25, 45, 51], we have incorporated some of them into the implementation of InstanT, indicating the good extensibility of InstanT.

4.3 Distribution Alignment

When the prediction of the classifier becomes imbalanced, Distribution Alignment (DA) is used to modulate the prediction so that the poorly predicted class will not be overlooked completely [7, 9, 20, 42]. This is achieved by estimating the prior of the pseudo-label P ( ˆ Y ) during training, and setting a clean class prior ˆ P ( Y ) , which is usually assumed to be uniformly distributed. The distribution alignment process can be described as ˆ P ( ˆ Y = j | X = x u ) = Norm( ˆ P ( ˆ Y = j | X = x u ) P ( Y = j ) /P ( ˆ Y = j )) , where Norm is a total-sum scaling. Therefore, if the distribution of ˆ Y is severely imbalanced, DA will force the prediction for the minority class to be scaled up, and the prediction for the over-populated class to be scaled down.

4.4 Loss Correction

While the classifier trained with incorrect pseudo-labels is subject to the influence of noisy supervision, in order to satisfy Theorem 1, the estimated softmax predictions from ˆ f θ must approximate the clean class posterior [56]. As we have already demonstrated that the transition matrix can be identified in standard SSL setting, in this part, we will show how to infer clean class posterior from noisy predictions and transition matrix, by utilizing forward correction [32]. Specifically, let ℓ ψ be a proper composite loss [33] (e.g. softmax cross-entropy loss), forward correction is defined as:

where ψ is an invertible link function. It has been proven that minimizing the forward loss with an accurately estimated transition matrix is equivalent to minimizing the loss defined over clean latent pseudo-labels [32]. Therefore, minimizing ℓ → ψ will let the softmax prediction of classifier ˆ f θ approximate the clean class posterior.

4.5 Training with Consistency Regularization Loss

Lastly, we provide an overview of the training objective of InstanT. We employ the widely adopted consistency regularization loss [36, 47], which assumes a well-learned and robust model should generate consistent predictions for random perturbations of a given instance. Previous methods have commonly employed augmentation techniques to introduce perturbations. The training loss of classifier ˆ f θ consists of two parts: supervised loss L D s ( θ ) and unsupervised loss L D u ( θ ) . L D s ( θ ) is calculated by the labeled samples, specifically, we have:

where n is the number of labeled samples, y l is the one-hot label for x l , and W is a weak augmentation function [36, 47], and ℓ is the cross-entropy loss function. Unsupervised loss L D u ( θ ) is calculated by the unlabeled samples X u and their predicted pseudo-labels ˆ Y u , which can be defined as:

where m is the number of unlabeled instances, ˆ y u is the one-hot pseudo-label for x u , and S is a strong augmentation function [7, 14, 15]. ✶ is an indicator function that filters unlabeled instances using τ ( x u ) t . Combining the supervised and unsupervised loss, we then have the overall training objective:

where λ is used to control the influence of the unsupervised loss.

5 Experiments

5.1 Experiment Setup

We use three benchmark datasets for evaluating the performances of InstanT, they are: CIFAR-10, CIFAR-100 [21], and STL-10 [13]. CIFAR-10 has 10 classes and 6,000 samples per class. CIFAR100 has 100 classes and 600 samples per class. STL-10 has 10 classes and 500 labeled samples per class, 100,000 unlabeled instances in total. For each dataset, we set varying numbers of labeled samples to create different levels of difficulty. Following recently more challenging settings [10, 41], where the labeled samples could be extremely limited, we set the number of labeled samples per class on CIFAR-10 as { 1 , 4 , 25 } , for CIFAR-100, we set as { 2 , 4 , 25 } , for STL-10, we set as { 1 , 4 , 10 } .

To ensure fair comparison between our method and all baselines, and to allow simple reproduction of our experimental results, we implemented InstanT and conducted all experiments within USB (Unified SSL Benchmark) framework [40]. To improve the training efficiency, e.g. faster convergence, we use pre-trained ViT as the backbone model for all baselines with the same hyper-parameters in part I. We use AdamW [28] as the default optimizer, where the learning rate is set as 5 e -4 for CIFAR-10/100, 1 e -4 for STL-10 [40]. The total training iterations K are set as 204,800 for all datasets. The training batch size is set as 8 for all datasets. For a more comprehensive evaluation of our proposed method, we also conduct experiments with Wide ResNet [53] trained from scratch, the detailed settings are aligned with existing works [7, 9, 36, 47, 48, 54], these results are summarized in part II. We select a range of popular baseline methods to evaluate against InstanT, which includes Pseudo-Label (PL) [23], MeanTeacher (MT) [37], VAT [29], MixMatch [8], UDA [47], FixMatch [36], Dash [48], FlexMatch [54] and AdaMatch [9]. More comprehensive setting details, including full details of hyper-parameters, can be found in Appendix A.

5.2 Main results

Part I. Our main results will be divided into two parts: part I and part II. For training efficiency, we will use a pre-trained Vision Transformer as the backbone model in part I, enabling us to compare a more diverse collection of baselines over a broader range of benchmark settings. In Part II, we will select baselines with better performance from Part I and train them from scratch for a more comprehensive evaluation.

Experimental results from part I are summarized in Table 1, where we can observe that, overall, InstanT achieves the best performances in most of the settings. More specifically, in the setting where label amount is

Table 2: Running time on STL-10(40) 2 .

extremely limited, InstanT exhibits the most significant improvement over all datasets, including an average 2.17% increase in accuracy on CIFAR-10 (10) over SOTA. While the identifiability of the transition matrix cannot be guaranteed with extremely limited labels, this could be remedied by including highly confident instances and their pseudo-labels to the labeled set and using them to better estimate the transition matrix. However, as the number of labeled samples increases, the

dominance of InstanT becomes less pronounced. We hypothesize that this can be attributed to the classifier ˆ f θ performing better with more labeled samples, resulting in fewer label errors. Since InstanT primarily aims to increase the threshold for instances likely to have noisy pseudo-labels, the impact of InstanT becomes less significant in the presence of reduced label noise. Moreover, comparing the performance gap between InstanT and its closest counterpart, AdaMatch, it becomes apparent that InstanT consistently outperforms AdaMatch in nearly all cases. This observation underscores the non-trivial improvement and contribution brought forth by InstanT.

Part II. In this part, we will present the experiment results of part II, which are summarized in Table 3. Specifically, we will compare InstanT against other popular baseline methods on CIFAR-10/100 trained from scratch. For all methods, we use WRN-28-2 as default backbone model, trained with 2 20 iterations. We use SGD as the default optimizer, with momentum set as 0.9, initial learning rate as 0.03 and a cosine learning rate scheduler.

Observing from experiment results, we can summarize similar patterns as the results from part I. Overall, InstanT brings most significant performance increases when the number of labeled samples are limited, e.g., when there is only 4 labeled samples per-class. Notably, for CIFAR-100 (400), InstanT exhibits an increase in accuracy for nearly 2%, which can be view as a significant improvement over SOTA baseline. While on other cases such as CIFAR-10 (250) and CIFAR-100 (10000), InstanT outperformed by other baseline methods, we emphasize that no single baseline consistently outperforms InstanT in all cases.

Another question of interest is whether InstanT will be as efficient as other methods in training time. While learning transition matrix estimator does sacrifice time and space complexity to some extent, we show that InstanT can still maintain high efficiency. As shown in Table 2, compared with FixMatch and AdaMatch, InstanT almost does not introduce further significant computational burden.

To gain a better understanding of the key differences between InstanT and existing methods, we now focus on the visualizations of InstanT compared to representative baselines. Specifically, we will display the classification accuracy, unlabeled sample utilization ratio, and pseudo-label accuracy of InstanT, FreeMatch [41], AdaMatch [9], and FixMatch [36] in Figure 3. These models are trained from scratch on CIFAR-100 with 400 labeled samples, following commonly used settings [36, 41].

Examining the classification accuracy in Figure 3(a), we observe that while InstanT does not converge as rapidly as FreeMatch, it ultimately achieves superior accuracy in the later stages of training. This is attributed to InstanT's ability to find a better balance between the quantity and quality of pseudo-labels. As depicted in Figure 3(b) and 3(c), although InstanT utilizes fewer unlabeled samples compared to FreeMatch, it demonstrates significantly improved accuracy on pseudo-labels. On the other hand, AdaMatch exhibits similar pseudo-label accuracy to InstanT but utilizes a smaller proportion of unlabeled instances. The key distinction lies in the instance-dependent threshold function employed by InstanT, which allows us to set a lower κ t for InstanT compared to AdaMatch, as InstanT mitigates the negative impact by increasing the instance-dependent thresholds for instances more prone to label noise, whereas AdaMatch will incorporate too many instances with incorrect pseudo-labels.

5.3 Ablation study

0

100k

200k

300k

400k

500k

Training Iter.

0.1

0.2

0.3

0.4

0.5

Acc.

InstanT

FreeMatch

AdaMatch

FixMatch

(a) Top-1 Accuracy

0

100k

200k

300k

400k

500k

Training Iter.

0.0

0.2

0.4

0.6

0.8

Unlabeled data util. ratio

InstanT

FreeMatch

AdaMatch

FixMatch

(b) Utilization ratio

0

100k

200k

300k

400k

500k

Training Iter.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Pseudo-label Acc.

InstanT

FreeMatch

AdaMatch

FixMatch

In this section, we will be verifying the effects of each component of InstanT, specifically, we will focus on evaluating the parts that we adapted from existing works, and determine how many performance improvements are attributed to our instance-dependent thresholds. Results of this ablation study are summarized in Table 4, where all experiments are averaged over three random seeds and ran on STL-10(40) using the setting of part I. First, we observe that InstanT-I removed both Distribution Alignment (DA) and relative thresholds (RT), which makes it equivalent

to adding τ ( x u ) on top of the fixed threshold of FixMatch. Notably, this also brings a significant improvement over FixMatch for over 2%, which further verifies the effectiveness of our proposed instance-dependent thresholds. Simply removing RT or DA will also cause a performance drop, which is aligned with the results from AdaMatch [9].

6 Conclusion

In this paper, we introduce a novel approach to thresholding techniques in SSL called instancedependent confidence threshold. This approach offers the highest level of flexibility among existing methods, providing significant potential for further advancements. We then present InstanT, a theoretically guided method designed under the concept of instance-dependent threshold. InstanT assigns unique confidence thresholds to each unlabeled instance, considering their individual likelihood of having incorrect labels. Additionally, we demonstrate that our proposed method ensures a bounded probability of assigning correct pseudo-labels, a characteristic rarely offered by existing SSL approaches. Through extensive experiments, we demonstrate the competitive performance of InstanT when compared to SOTA baselines.

References

Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou. Adaptive consistency regularization for semi-supervised transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6923-6932, 2021.

Alekh Agarwal. Selective sampling algorithms for cost-sensitive multiclass prediction. In International Conference on Machine Learning , pages 1220-1228. PMLR, 2013.

Elizabeth S Allman, Catherine Matias, and John A Rhodes. Identifiability of parameters in latent structure models with many observed variables. 2009.

Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2020.

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems , 34:24392-24403, 2021.

Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems , 31, 2018.

David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785 , 2019.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106.04732 , 2021.

H Chen, R Tao, Yue Fan, Y Wang, M Savvides, J Wang, B Raj, X Xie, and Bernt Schiele. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In Eleventh International Conference on Learning Representations . OpenReview. net, 2023.

Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11442-11450, 2021.

Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In International Conference on Machine Learnin , pages 1789-1799. PMLR, 2020.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 702-703, 2020.

Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning , pages 8082-8094. PMLR, 2022.

Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. Advances in Neural Information Processing Systems , 34:26714-26725, 2021.

Zhuo Huang, Jian Yang, and Chen Gong. They are not completely useless: Towards recycling transferable unlabeled data for class-mismatched semi-supervised learning. IEEE Transactions on Multimedia , 2022.

Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems , 33:14567-14579, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah. Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data. In International Conference on Machine Learning , pages 11828-11843. PMLR, 2022.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896, 2013.

Doyup Lee, Sungwoong Kim, Ildoo Kim, Yeongjae Cheon, Minsu Cho, and Wook-Shin Han. Contrastive regularization for semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3911-3920, 2022.

Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International conference on machine learning , pages 6403-6413. PMLR, 2021.

Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence , 38(3):447-461, 2015.

Yang Liu. Identifiability of label noise transition matrix. arXiv preprint arXiv:2202.02016 , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence , 41(8):1979-1993, 2018.

Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems , 33:15288-15299, 2020.

Youngtaek Oh, Dong-Jin Kim, and In So Kweon. Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9786-9796, 2022.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Mark D Reid and Robert C Williamson. Composite binary losses. The Journal of Machine Learning Research , 11:2387-2422, 2010.

Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. arXiv preprint arXiv:2105.14148 , 2021.

Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Conference on learning theory , pages 489-511. PMLR, 2013.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics , 32(1):135-166, 2004.

Xudong Wang, Long Lian, and Stella X Yu. Unsupervised selective labeling for more effective semi-supervised learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX , pages 427-445. Springer, 2022.

Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems , 35:3938-3961, 2022.

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang. Freematch: Self-adaptive thresholding for semisupervised learning. arXiv preprint arXiv:2205.07246 , 2022.

Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A classrebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10857-10866, 2021.

Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations .

Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, and Gang Niu. Class2simi: A noise reduction perspective on learning with noisy labels. In International Conference on Machine Learning , pages 11285-11295. PMLR, 2021.

Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems , 33:7597-7610, 2020.

Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in neural information processing systems , 32, 2019.

Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 6256-6268, 2020.

Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning , pages 11525-11536. PMLR, 2021.

Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14421-14430, 2022.

Shuo Yang, Erkun Yang, Bo Han, Yang Liu, Min Xu, Gang Niu, and Tongliang Liu. Estimating instance-dependent bayes-label transition matrix using a deep neural network. In International Conference on Machine Learning , pages 25302-25312. PMLR, 2022.

Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. Advances in neural information processing systems , 33:7260-7271, 2020.

LIN Yong, Renjie Pi, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. A holistic view of label noise transition matrix in deep learning and beyond. In The Eleventh International Conference on Learning Representations , 2022.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In Ninth International Conference on Learning Representations , volume 9, 2021.

Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learning , pages 11447-11457. PMLR, 2020.

A Comprehensive settings

For all methods, we apply temperature scaling for probability calibration by default, and the scaling factor is set as 0.5. For FixMatch, Dash, FlexMatch and AdaMatch, τ is set as 0.95. For UDA, τ is set as 0.8, temperature scaling factor is set as 0.4, according to the recommendation of the original paper. For Dash, γ is set as 1.27, C is set as 1.0001, ρ is set as 0.05, warm-up iteration is set as 5120 iterations. For InstanT, base τ is set as 0.9 by default.

B Tsybakov Margin Condition (Binary)

We present the binary case of Tsybakov Margin Condition here. It can be defined as:

Assumption 2 (Binary Tsybakov Margin Condition) . For some finite constant C, α > 0 , and δ 0 ∈ ( 0 , 1 2 ] , the Tsybakov Margin Condition holds if ∀ δ ∈ (0 , δ 0 ] , we have

Through similar derivations, we can provide our main theorem in binary case:

Theorem 3. Assume estimated noisy class posterior ˆ P ( ˆ Y | X ) satisfies Assumption 2 with C, α > 0 , δ 0 ∈ ( 0 , 1 2 ] , and ϵ θ ≤ δ 0 (1 -T 1 , 0( x ) -T 0 , 1 ( x )) ) , we have:

where τ ( x u ) is the instance-dependent threshold function under binary case:

̸

C Proof of Theorem 1

Definition 3. For multi-class classification, the relationship between noisy class posterior P ( ˆ Y | X ) and clean class posterior P ( Y | X ) can be concluded as:

Proof of Theorem 1

For simplicity, we will denote the clean class posterior P ( Y = k | X = x ) as η k ( x ) , noise class posterior P ( ˆ Y = k | X = x ) as η ˆ k ( x ) , estimated noise class posterior ˆ P ( ˆ Y = k | X = x ) as ˆ η ˆ k ( x ) .

Proof.

Based on Definition 3, we can substitute η ˆ k ( x u ) with ∑ | Y | i =1 T i,k ( x u ) η i ( x u ) , and since ∥ η ˆ k -ˆ η ˆ k ∥ ∞ ≤ ϵ θ ( ϵ -close), we can continue with:

̸

̸

Note that, since ϵ θ ≤ t 0 min k T k,k ( x u ) , the Multi-class Tsybakov Condition can be applied here:

Which gives us:

Hence proved P [ k = h ∗ ( x u ) , ˆ η ˆ k ( x u ) ≥ τ ( x u ) ] > 1 -C [ O ( ϵ θ )] α .

D Proof of Theorem 2

In this section, we provide the proof from Liu et al. for the necessary and sufficient condition of the identification of instance-dependent transition matrix T ( x ) . First, we provide the Kruskal's Identifiability Theorem (or Kruskal rank Theorem):

Theorem 4. The parameters M i , i = 1 , ..., p are identifiable, up to label permutation, if

Proof. Then we follow the proof from Liu et al. to first prove the sufficiency. Since we cannot observe the true label for unlabeled instances X u , we denote them as hidden variable Z for simplicity. P ( Z = i ) is the prior for the latent true label.

Each ˆ Y = i, ∀ i = 1 , ..., p corresponds to the observation O i . | ˆ Y | is the cardinality of the pseudo-label space. Without loss of generality, we will fix the number of classes as three during the analysis.

Each ˆ Y i corresponds to an observation matrix M i :

Therefore, by definition of M 1 , M 2 , M 3 and T ( x ) , they all equal to T ( x ) : M i ≡ T ( X ) , i = 1 , 2 , 3 . When T ( x ) has full rank, we know immediately that all rows in M 1 , M 2 , M 3 are independent. Therefore, the Kruskal ranks satisfy

Checking the condition in Theorem 4, we easily verify

Where Theorem 4 proves the sufficiency for three labeled samples per-class is sufficient for the identification of T ( x )

Subsequently, Liu et al. also provides the proof for necessity, without the loss of generality, the analysis for this part will be conducted in binary case only. More specifically, the label error generation in SSL can be described by a binary transition matrix:

In order to prove necessity, we must show that less than three informative labels per-class will not give us a unique solution for T ( x ) . If we were given two unlabeled instances with their pseudo-label ˆ Y 1 and ˆ Y 2 . Specifically, with two unlabeled instance and pseudo-label pairs, we can conclude the following probabilities can derive all potential probabilities we can possibly get from them:

Posterior: P ( ˆ Y 1 = +1 | X )

Positive Consensus: P ( ˆ Y 1 = ˆ Y 2 = +1 | X )

Negative Consensus: P ( ˆ Y 1 = ˆ Y 2 = -1 | X )

This is because other probability distributions are able to be derived from these three quantities:

But P ( ˆ Y 2 = +1 | X ) = P ( ˆ Y 1 = +1 | X ) , since noisy labels are i.i.d. The above three quantities led to three equations that depend on e + , e -. This gives us the following system of equations:

Where:

Given Y , ˆ Y 1 , ˆ Y 2 are conditional independent, which gives us:

Conversely, for P ( ˆ Y 1 = ˆ Y 2 = -1 | X ) , the derivation can follow similar procedure. We can now assert that e + , e -are not identifiable. Where a simple counter example can show that two sets of unique e + , e -both satisfies above derivation:

Which proves that two informative noisy labels are insufficient to identify T ( x u ) .

Peer reviews:
comments: SUMMARY:
This paper introduces a new thresholding methodology for Semi-Suervised Learning. This paper proposes InstanT, which uses an instance-dependent threshold for each unlabeled data (Fig1). This algorithm shows the improvement on multiple semi-supervised benchmark datasets.

STRENGTHS:
Pros. 
- This paper is well-written and easy to follow.
- This paper is well-motivated (Fig 1).
- This paper proposes InstanT, and it is very well derived.
- This paper shows extensive experiments (Especially it seems to train all other semi-supervised learning baselines using ViT models) and shows the effectiveness of this method.

WEAKNESSES:
Cons.
- (Minor) This paper uses a pre-trained ViT model. In pre-training, I think the classes in Cifar10/Cifar100 could be the subset of the pre-training dataset. If These classes are already trained, they could be already well clustered and easy to train with a very small labeled dataset. (It also shows the performance improvement in the original training setting in the Appendix)

- (Minor) It would be great if it reports the supervised learning results (full labeled dataset)

- (Major) This paper evaluates their algorithm in very small datasets (Cifar10/Cifar100/STL-10). Therefore, I don't know how it robustly works in large datasets such as ImageNet. Nowadays, most of the papers evaluate their algorithms on ImageNet. (In this case, it seems to train the model from scratch. (not using pre-trained ViT model)

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
Please see the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on semi-supervised learning (SSL) and proposes the study of instance-dependent thresholds to make incorrect pseudo-labels have higher thresholds.\n\nSTRENGTHS:\nThis paper studies an important problem in SSL, and tries to propose a dynamic and adaptive threshold to guarantee the pseudo-label quality for instances. Also, it gives a theoretical analysis on estimating the threshold. The experiments are sufficient and the results surpasses SOTA. In the results, it is obvious to see that the algorithm has a considerable improvement in Top-1 accuracy. The ablation study shows that every part of this algorithm is useful.\n\nWEAKNESSES:\nTheorem 2 is proved with Bayes rule with Assumption 2. But if the dynamic value k_t > 0, Theorem 2 does not hold. It is incorrect P(X>=a+b)>=P(X>=a) for any b>0. I think there may be some mistakes in Assumption 2 and Theorem 2. The theorems and analyses based on Assumption 2 may be wrong.\nIn section 4.1 the authors do not give any theoretical reasons for the definition of k_t and \\tau(x_u). Whether k_t satisfies Assumption 2 or not? \nIn Equation 8, the authors use an approximation \\hat{T}(x) to estimate T(x). But the analysis is based on the real value of T(x). If the label noise is large or the number of samples is not enough, the error between the real value of T(x) and the approximation \\hat{T}(x) cannot be ignored. Some analysis should be provided here.\n\nQUESTIONS:\nAs mentioned in the part of weaknesses, the authors may make a mistake in Theorem 2, which makes that the paper is not convincing. The theoretical analysis part of the paper should be considered more seriously.\nThe dynamic threshold is determined by the transition matrix T(x). But in practice it is impossible to estimate it without any bias. The bias may influence on the final result. Is it possible to provide an error bound in the analysis?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new approach for selecting confident examples called InstanT, which uses instance-dependent thresholds for assigning pseudo-labels to unlabeled data. Unlike existing methods that apply the same threshold to all samples, InstanT considers the instance-level ambiguity and error rates of pseudo-labels, assigning higher thresholds to instances more likely to have incorrect pseudo-labels. The paper demonstrates that this approach provides a probabilistic guarantee for the correctness of the assigned pseudo-labels. This innovative method may offer a new perspective on SSL.\n\nSTRENGTHS:\n1.The paper finds a significant and challenging problem in semi-supervised learning (SSL) that has not been adequately tackled by existing methods. Traditional SSL methods typically use a single loss threshold to select confident examples, implicitly assuming that examples with the same loss have the same likelihood of pseudo-label correctness. However, this assumption does not always hold, as there can be hard but confident examples that have larger loss values but correct pseudo-labels. \n\n2.The innovation of this paper is noteworthy. This paper innovatively proposes to estimate the probability of examples being incorrect and applies instance-dependent thresholds based on these estimates. This approach is more nuanced and potentially more effective as it takes into account the individual characteristics of each example. It is the first to propose the estimation of instance-dependent thresholds in SSL. This novel approach represents could open up new avenues for research in the field. \n\n3.The authors also provide sufficient theoretical analysis about their proposed method. They present a theorem that shows that for samples that satisfy their instance-dependent threshold function, the likelihood of the pseudo-labels being correct is lower-bounded. This provides a solid foundation for the proposed method and helps to convince readers of its assumption and correctness.\n\nWEAKNESSES:\n1.Certain aspects of the paper could benefit from further explanation and clarification. Specifically, the relationship between the Quality-Quantity Trade-off and the effective dynamic value is not clearly articulated in the main body of the paper. These are key components of the proposed method, and their interaction could significantly impact the performance of the method.\n\n2.The paper does not sufficiently discuss the limitations of the proposed method. Every method has its limitations and potential drawbacks, and a thorough discussion of these is crucial for a balanced and comprehensive presentation of the work. \n\n3.The empirical improvement of the proposed method is marginal when the amount of labeled data increases. This suggests that the method's performance may not scale well with larger labeled datasets.\n\n4.The experimental evaluation of the method is based on only three datasets. This limited number of datasets may not provide a comprehensive evaluation of the method's performance.\n\nQUESTIONS:\n1.What might be the potential drawbacks or limitations of this method? How could these impact its applicability or performance in certain scenarios or with certain types of data?\n\n2.The experimental evaluation of the method is based on only three datasets. Could the authors explain their choice of datasets and how representative these datasets are of the types of data the method would encounter in real-world applications? Would the performance of the method vary significantly if tested on other datasets? \n\n3.The authors may need to clarify the relationship between the Quality-Quantity Trade-off and the effective dynamic value. How do these two factors interact within the context of their proposed method?\n\n4.The paper discusses the balance between the quality and quantity of pseudo-labels. Could the authors provide some intuition of how this balance can be achieved in practice?\n\n5.This paper is highly related to the problem of learning with instance-dependent label noise. To clearly demonstrate the effectiveness, can authors provide a comparison with methods for handling instance-dependent label noise?\n\nLIMITATIONS:\nThe main paper seems to lack a comprehensive discussion on the limitations of the proposed method. It would be beneficial for the authors to conduct a thorough review of their method to identify any potential limitations. I suggest including a separate section in the main paper dedicated to discussing these limitations, which would provide a more balanced and complete perspective of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a semi-supervised method with instance-dependent thresholds (InstanT), which can assign different thresholds to individual unlabeled data based on the instance-dependent label noise level and prediction confidence. Also, this paper provides a theoretical analysis of the proposed InstanT. Extensive experiments have shown the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper proposes an Instance-dependent Thresholding strategy for semi-supervised learning. Also, the proposed method can vouch for the reliability of pseudo-labels it assigns with a theoretical guarantee. \n\n2. This paper is easy to read and well organized.\n\n3. The experimental results and the proof of  Theorems in this paper seem to be solid.\n\nWEAKNESSES:\n1. This paper is devoted to addressing the SSL problem with dynamic thresholding. However, there is a lack of some SOTA methods to be compared with the proposed method. For example, \n[1] Guo et al., Class-imbalanced semi-supervised learning with adaptive thresholding, ICML, 2022. \n[2] Yang et al., Class-aware contrastive semi-supervised learning, CVPR, 2022.\n[3] Wang et al., Freematch: Self-adaptive thresholding for semi-supervised learning, ICLR, 2023. \n[4] Chen et al., Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning, ICLR, 2023.\n\n2. Most previous SSL methods have reported their performance on both datasets: SVHN and ImageNet. It is therefore suggested that the authors supplement some experiments to evaluate the generalization performance of the proposed method. \n\n3. It is suggested to omit some symbols whose meanings have been explained in the previous context, so as to increase the brevity of this paper.\n\n4. The proposed method is motivated by the learning with noisy labels. Therefore, it is necessary to review some related methods to give readers a whole picture.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nThe limitations of this paper should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Assumption for instance-dependent threshold setting and theoretical proof based on it\n- Transition matrix modeling (estimator design) to reduce label error through instance-dependent threshold function\n- This paper shows good performance on various datasets (CIFAR10, 100, STL-10)\n\nSTRENGTHS:\n1) A new approach in SSL called instance-dependent threshold setting\n2) Solid theoretical proof\n3) Excellent performance, especially in environments with little labeled data\n\nWEAKNESSES:\n1) Transition matrix modeling for threshold calculation for each instance is required (complexity is expected to increase).\n2) There is no consideration for class imbalance, which is mainly dealt with in SSL. If there is a class imbalance problem, it may not be a good idea to set different thresholds for each instance.\n3) Comparative papers (FlexMatch, Dash, AdaMatch, etc.) are papers published before 2022. Comparisons with recent papers have not been made. In addition, AdaMatch, which is mainly compared, is a paper dealing with domain adaptation issues.\n\nQUESTIONS:\n1) The running time in Table 2 was 1s lower than Fix and AdaMatch. Do you have the result in terms of the amount of calculations and the number of parameters?\n2) If the reason for applying Distribution Alignment in InstanT-II in Table 3 is to eliminate class imbance, what is the reason why performance improves when DA is applied in ablation study? (The experimental environment in the paper appears to be a situation with low imbalance using the same number of labeled data for each class).\n3) Usually SSL papers show error rates of CIFAR10/100. How does the error rate come out?\n4) What about the results on larger sized images (e.g. ImageNet)?\n5) Papers on instance dependent pseudo-labeling already exist. Is there any difference from that? For example, the paper SimMatch: Semi-supervised Learning with Similarity Matching (2022 CVPR) also uses instance dependent similarity. This paper theoretically proves the validity of using instance-level threshold, but is there any excellence or novelty compared to the similarity method used in the above paper?\n6) The results of the comparative papers presented in the proposed paper and the results of the papers below are different. Is it an accurate comparison? Is there a reason why the second paper below is not included in the result table?\n- SimMatch: Semi-supervised Learning with Similarity Matching (2022 CVPR) \n- SOFTMATCH: ADDRESSING THE QUANTITY-QUALITY TRADE-OFF IN SEMI-SUPERVISED LEARNING (2023 ICLR)\n7) Please also respond to the weaknesses pointed out.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}