SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Muyang Li $^{1}$ , Runze Wu $^{2}$ , Haoyu Liu $^{2}$ , Jun Yu $^{3}$ , Xun Yang $^{3}$ , Bo Han $^{4}$ , Tongliang Liu $^{1*}$
$^{1}$ Sydney AI Center, The University of Sydney;  $^{2}$ FUXI AI Lab, NetEase;  $^{3}$ University of Science and Technology of China;  $^{4}$ Hong Kong Baptist University
Semi-supervised learning (SSL) has been a fundamental challenge in machine learning for decades. The primary family of SSL algorithms, known as pseudo-labeling, involves assigning pseudo-labels to confident unlabeled instances and incorporating them into the training set. Therefore, the selection criteria of confident instances are crucial to the success of SSL. Recently, there has been growing interest in the development of SSL methods that use dynamic or adaptive thresholds. Yet, these methods typically apply the same threshold to all samples, or use class-dependent thresholds for instances belonging to a certain class, while neglecting instance-level information. In this paper, we propose the study of instance-dependent thresholds, which has the highest degree of freedom compared with existing methods. Specifically, we devise a novel instance-dependent threshold function for all unlabeled instances by utilizing their instance-level ambiguity and the instance-dependent error rates of pseudo-labels, so instances that are more likely to have incorrect pseudo-labels will have higher thresholds. Furthermore, we demonstrate that our instance-dependent threshold function provides a bounded probabilistic guarantee for the correctness of the pseudo-labels it assigns. Our implementation is available at https://github.com/tml1ab/2023_NeurIPS_InstanT.
In recent years, machine learning algorithms trained on abundant accurately labeled data have demonstrated unparalleled performance across different domains. However, in practice, it is often financially infeasible to collect reliable labels for all samples in large-scale datasets. A more practical solution is to select a subset of the data and use expert annotations to assign labels to them [39]. This scenario, where the majority of the data is unlabeled and only a portion has reliable labels, is known as semi-supervised learning (SSL). In SSL, our aim is usually to learn a classifier that has comparable performance to the one trained in a fully supervised manner. To achieve this, the majority of existing approaches adopt a training strategy named pseudo-labeling. More specifically, a model will be trained on a small set of labeled data first, and this model will then be applied to the larger unlabeled set to assign predicted pseudo-labels to them. If the model's confidence in an instance exceeds a certain threshold, then this instance along with its predicted pseudo-label will be added to the training set for this iteration. Thus by iteratively expanding the training set, if the expanded instances are indeed assigned with correct labels, then the classification error will be gradually reduced [43].
However, since the model is bound to make mistakes, the newly added training samples are not always assigned with correct labels, which generates label noise [5, 46, 44]. Under the influence of label noise, the model will gradually over-fits to noisy supervision, hence accumulating generalization error. Since the model makes predictions based on instance features, it is usually assumed that for
*Correspondence to Tongliang Liu (tongliang.liu@sydney.edu.au)
Figure 1: Illustration on the differences between uniform thresholds, class-dependent thresholds, and instance-dependent thresholds in SSL. The black solid lines are the decision boundaries generated by the classifier, and the colored dashed lines are the confidence thresholds, the colored square, triangle, and circle represent the unlabeled instances with their predicted class. We can observe that a uniform threshold does not depend on any factors, class-dependent thresholds only depend on the predicted pseudo-label, and instance-dependent thresholds depend on the features of unlabeled instances.
"hard" examples, the classifier is more likely to give incorrect predictions to them [55], making the label noise pattern instance-dependent [11, 45, 55].
Consequently, selecting an appropriate confidence threshold for pseudo-label assignment becomes a key factor that determines the performances of SSL methods. For predictions that are more likely to have incorrect labels, we want to assign higher thresholds to avoid selecting them prematurely. And for predictions that are more likely to be correct, we want to assign lower thresholds to encourage adding them to the training set at an earlier stage. Conventionally, such a threshold is usually determined by empirical experience in an ad-hoc manner, where only a few of the existing works consider the theoretical implication of threshold selection. Recently, increasing research focused on the investigation of dynamic or adaptive thresholds [9, 17, 41, 48, 54], where such confidence threshold is conditional on some external factors, such as the current training progress. Notably, the vast majority of those methods consider a single threshold for all instances and overlook their instance-level information, such as the instance-level label noise rate. As illustrated in Figure 1, the instance-dependent threshold is the most flexible among existing threshold types.
Motivated by this challenge, in this paper, we propose a new SSL algorithm named Instance-dependent Thresholding (InstanT). Since the learned model is subject to the influence of instance-dependent label error, we aim to quantify and reduce such error by estimating an instance-dependent confidence threshold function based on the potential label noise level and instance ambiguity (clean class posterior). In addition, we can derive a lower-bounded probability, for samples that satisfy the thresholds will be assigned to a correct label. From our main theorem, we show that as the training progresses, this probability lower-bound will asymptotically increase towards one, hence guaranteeing the pseudo-label quality for instances that satisfies our proposed thresholds.
We summarize our contributions as follows: (1) We propose an SSL algorithm named InstantT, which assigns thresholds to individual unlabeled samples based on the instance-dependent label noise level and prediction confidence. (2) We prove InstantT has a bounded probability to be correct, which vouch for the reliability of the pseudo-labels it assigns with a theoretical guarantee. (3) To the best of our knowledge, this is the first attempt to estimate the instance-dependent thresholds in SSL, which has the highest degree of freedom compared with existing methods. (4) Through extensive experiments, we show that our proposed method is able to surpass state-of-the-art (SOTA) SSL methods across multiple commonly used benchmark datasets.
Related work. As we mentioned, some existing works have already been considered to improve pseudo-labeling by leveraging "dynamic" or "adaptive" thresholds. More specifically, Dash [48] considers a monotonically decreasing dynamic threshold, based on the intuition that as training progresses, the model at a later stage will provide more reliable predictions than at early stages. FlexMatch [54] further introduced class-dependent learning status in addition to the dynamic threshold. Adsh [17] employs adaptive threshold under class imbalanced scenario by optimizing the number of pseudo-labels per class. As for adaptive thresholds, AdaMatch [9] considers using a pre-defined threshold factor multiplied by the averaged top-1 prediction confidence, making it adaptive to the
model's current confidence level. FreeMatch [41] considers a combination of adaptive global and local thresholds, which combines the training progress and class-specific information.
In addition to the aforementioned more recent methods, most of the classical methods use pre-defined fixed thresholds, which remain constant throughout the entire training process [8, 18, 19, 29, 36]. Such thresholds are usually set to be high enough to prevent the occurrence of incorrect pseudolabels. Notably, other approaches such as reweighting [10, 22], distribution alignment [20, 31, 42], contrastive learning [24, 49], consistency regularization [1, 34] were also employed to improve SSL.
2.1 Notation and Problem Setting
We consider the general setting from SSL, namely, we have a small group of labeled instances  $X_{l} \coloneqq \{x_{1},\ldots ,x_{n}\}$  with their corresponding labels  $Y_{l} \coloneqq \{y_{1},\ldots ,y_{n}\}$ . And there is an unlabeled instance group  $X_{u} \coloneqq \{x_{n + 1},\ldots ,x_{n + m}\}$  with their latent labels  $Y_{u} \coloneqq \{y_{n + 1},\ldots ,y_{n + m}\}$ , usually, we have  $m >> n$ . Furthermore, we assume that the labeled instances and unlabeled instances are independent and identically distributed (i.i.d), i.e.  $X \coloneqq \{X_{l},X_{u}\} \in \mathcal{X}, Y \coloneqq \{Y_{l},Y_{u}\} \in \mathcal{Y}$ .
We term the model trained with clean and pseudo-labels as  $\hat{f}$ , parameterized by  $\theta$ . Since  $\hat{f}_{\theta}$  is bound to make mistakes, its generated pseudo-label  $\hat{Y}_u$  will contain label errors. Hence the softmax predictions of  $\hat{f}_{\theta}$  are actually approximating the noisy class posterior. We fix the notation for the real noisy class posterior as  $P(\hat{Y} |X)_t$  at  $t$ -th iteration, whereas our approximated noise class posterior is  $\hat{P} (\hat{Y} |X)_t$ , which can be obtained from the softmax predictions of the model  $\hat{f}_{\theta}$  at  $t$ -th iteration.
As the clean label for the unlabeled set is unknown, we instead wish to recover the Bayes optimal label for all the unlabeled instances, which is the label class that maximizes the clean class posterior [50, 56]. The Bayes optimal label is generated by the hypothesis that minimizes the risk within the hypothesis space, i.e. the Bayes optimal classifier, which we denote as  $h^*$ . And we can denote the Bayes optimal label for  $x$  as  $h^*(x)$ .
2.2 Instance-dependent Label Noise in SSL
As mentioned in the previous section, label noise is an inevitable challenge in SSL. Moreover, since  $P(\hat{Y} | X)$  is dependent on  $X$ , we say the label noise is instance-dependent [12, 45]. Concretely, we have  $P(\hat{Y} | X = x) = T(x)P(Y | X = x)$  [32, 35], where  $T(x_{u})$  is the instance-dependent transition matrix, which models the generation of label noise.  $P(Y | X = x)$  represents the latent clean class posterior for unlabeled samples. We can define the  $ij$ -th entry of  $T(x)$  as:
$$
T _ {i, j} (\boldsymbol {x}) = P \left(\hat {Y} = j \mid Y = i, X = \boldsymbol {x}\right), \tag {1}
$$
which means the  $ij$ -th entry of  $T(\pmb{x})$  models the probability that  $\pmb{x}$  with clean label  $y = i$  will be predicted as  $\hat{y} = j$ .
2.3 Tsybakov Margin Condition
We assume our learned classifier satisfies Tsybakov Margin Condition [38], which essentially restrained the level of complexities of the classification problem by assuming the data are separable enough. It is a commonly used assumption for various Machine Learning problems [2, 6, 56] including SSL [48]. Without the loss of generality, we will directly present the multi-class Tsybakov Margin Condition [2, 56], whereas the binary case will be provided in the Appendix B.
Assumption 1 (Multi-class Tsybakov Margin Condition). For some finite constant  $C$ ,  $\alpha > 0$ , and  $\delta_0 \in (0,1]$ , the Tsybakov Margin Condition holds if  $\forall \delta \in (0,\delta_0]$ , we have
$$
P \left[ P (Y = \gamma | X = \boldsymbol {x}) - P (Y = s | X = \boldsymbol {x}) \leq \delta \right] \leq C \delta^ {\alpha}.
$$
Where  $P(Y = \gamma | X = x)$  and  $P(Y = s | X = x)$  are the largest and second-largest clean class posterior probability for  $x$ .
2.4 Quality-Quantity Trade-off in SSL
To understand the choice of pseudo-label thresholds in SSL, it is essential to consider the trade-off between the quality and the quantity of the pseudo-labels [10, 41]. Typically, a higher threshold is presumed to yield superior label quality, as it assigns pseudo-labels solely to instances where the classifier exhibits a high level of confidence. Conversely, elevating the threshold diminishes the quantity of pseudo-labels, as it leads to the exclusion of instances where the classifier lacks sufficient confidence. Hence causing the dilemma of quality-quantity trade-off.
One possible approach to address the quantity-quality trade-off in SSL is to apply a dynamic threshold, which is dependent on the training progress. This is because the evaluation metric for confident samples is usually not stationary throughout the training process of neural networks. As the model's predicted probability is subject to the level of over-fitting [16, 30], designing a thresholding function that is dependent on the training process is crucial to the success of SSL algorithms[9, 41]. Intuitively, in the early stage of the training process, where the model has just started to fitting, the threshold should not be set too high to filter out too many samples, and as the model has already well-fitted to the training data, the threshold should be elevated to combat falsely confident instances caused by confirmation bias [4].
3 Instance-dependent thresholds - A theoretical perspective
In this section, we focus on answering the specific research question: For SSL, can we derive instance-dependent pseudo-label assignment thresholds with theoretical guarantees?
The answer to this question is affirmative. We will present our main theorem and the proof to show that for samples that satisfy our instance-dependent threshold function, their likelihood of being correct is lower-bounded.
3.1 Lower-bound Probability of Correctness
For a classifier trained with sufficient accurately labeled data, its prediction  $\gamma$  will be made if  $P(Y = \gamma |X = x) > P(Y = s|X = x)$ . However, under the existence of label errors, the fidelity of the classifier can no longer be guaranteed. Recently, Zheng et al. [56] showed that the classifier influenced by the label noise can still have a bounded error rate. We show that in SSL tasks, where the label noise is instance-dependent, we can establish a lower-bound probability for the predictions to be correct through similar derivations.
Recall Assumption 1, we have  $\gamma \coloneqq \arg \max_{i}P(Y = i|X = \boldsymbol{x}_{\boldsymbol{u}})$ ,  $s \coloneqq \arg \max_{i\neq k}P(Y = i|X = \boldsymbol{x}_{\boldsymbol{u}})$ ,  $k \coloneqq \arg \max_{i}\hat{P} (\hat{Y} = i|X = \boldsymbol{x}_{\boldsymbol{u}}) = \hat{y}_u$ . And let  $\epsilon_{\theta}$  be the estimation error between our estimated noisy class posterior  $\hat{P} (\hat{Y} |X)$  and true noisy class posterior  $P(\hat{Y} |X)$ . We can then formulate our main theorem:
Theorem 1. Assume estimated noisy class posterior  $\hat{P} (\hat{Y} |X)$  satisfies Assumption 1 with  $C,\alpha >0$ $\delta_0\in (0,1]$ , and  $\epsilon_{\theta}\leq \delta_{0}\min_{i}T_{i,i}(\pmb{x}_{\pmb{u}})$ , we have:
$$
P \left[ k = h ^ {*} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \hat {P} (\hat {Y} = k | X = \boldsymbol {x} _ {\boldsymbol {u}}) \geq \tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \right] > 1 - C [ O \left(\epsilon_ {\theta}\right) ] ^ {\alpha}, \tag {2}
$$
where  $\tau (\pmb {x}_u)$  is the instance-dependent threshold function:
$$
\tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) = T _ {k, k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) P (Y = s | X = \boldsymbol {x} _ {\boldsymbol {u}}) + \sum_ {i \neq k} ^ {| Y |} T _ {i, k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) P (Y = i | X = \boldsymbol {x} _ {\boldsymbol {u}}). \tag {3}
$$
The proof of Theorem 1 is provided in Appendix C. Theorem 1 substantiates that, as our model over-fits to the label error (as  $\epsilon_{\theta}$  decreases), and the transition matrix can be successfully estimated, the probability lower-bound of the assigned pseudo-labels are correct will increase asymptotically. This guarantees the quality of the pseudo-labels  $\tau(\boldsymbol{x}_u)$  assigned. Note that,  $\tau(\boldsymbol{x}_u)$  requires the clean class posterior of unlabeled instances, which can be provably inferred upon the successful estimation of  $T(\boldsymbol{x})$  and the noisy class posterior [26, 32].
To gain a deeper understanding of Theorems 1, we will try to intuitively understand the outputs of  $\tau(\pmb{x}_u)$ . Our aim is to examine the behavior of  $\tau(\pmb{x}_u)$  in relation to the unlabeled sample  $\pmb{x}_u$ , which
Figure 2: Overview of InstanT. Left: training process of transition matrix estimator  $\hat{f}_{\theta'}$ . For a labeled instance  $x_{l}$ , whose label is 2, the classifier  $\hat{f}_{\theta}$  will first generate its noisy class posterior. Then, we want the second row of the transition matrix to approximate the noisy class posterior by minimizing  $L(\theta')$  from Equation 7. Right: The inference process of  $\tau(x_{u})$ . When calculating  $\tau(x_{u})$ ,  $x_{u}$  will be simultaneously passed to the classifier and transition matrix estimator. Since  $\mathbf{1}$  is the predicted class label from the model that minimizes the forward loss (Equation 8), we will use the first column from our estimated transition matrix to modulate threshold based on Theorem 1.
has a predetermined clean class posterior. If we observe an increase in the sum of column  $k$  in the transition matrix, it will result in a corresponding increase in the threshold  $\tau(\boldsymbol{x}_u)$ . This implies that when  $\boldsymbol{x}_u$  is highly likely to be assigned with an incorrect pseudo-label,  $\tau(\boldsymbol{x}_u)$  will assign a higher threshold in order to prevent its premature inclusion of  $\{\boldsymbol{x}_u, \hat{y}_u\}$  in the training set. Conversely, when  $\boldsymbol{x}_u$  is less likely to be assigned with an incorrect pseudo-label,  $\tau(\boldsymbol{x}_u)$  will assign a lower threshold to encourage adding  $\{\boldsymbol{x}_u, \hat{y}_u\}$  to the training set.
3.2 Identification of Instance-dependent Transition Matrix in SSL
Theorem 1 builds upon the assumption that  $T(\pmb{x})$  can be successfully identified. Yet, the identification of  $T(\pmb{x})$  has been a long-standing issue in the field of label noise learning. Fortunately, in SSL, we argue that  $T(\pmb{x})$  can be provably identified under mild conditions. We will first justify our argument from a theoretical perspective based on the Theorems from Liu et al. [27]. Subsequently, we will also propose the empirical methods for estimating  $T(\pmb{x})$  from an intuitively understandable aspect in the following section.
We will start by defining the identifiability, we use  $\Omega$  to represent an observation space and use  $\Theta$  to represent a general parametric space. Then, for a distribution with parameter  $\theta'$ , such that  $\theta' \in \Theta$ , a model  $P_{\theta'}$  on  $\Omega$  can be defined [3, 50]. We further define its identifiability as:
Definition 1 (Identifiability). Parameter  $\theta^{\prime}$  is said to be identifiable if  $P_{\theta^{\prime}}\neq P_{\theta^{\prime \prime}},\forall \theta^{\prime}\neq \theta^{\prime \prime}$
In our case,  $\theta^{\prime}(\pmb {x})\coloneqq \{T(\pmb {x}),P(Y|X = \pmb {x})\}$ , i.e.,  $P_{\theta^{\prime}}$  is a distribution defined by the transition probability and clean class posterior. To determine the identifiability of  $T(\pmb {x})$ , we also need to define the term informative noisy labels. Specifically, we have:
Definition 2 (Informative noisy labels). For a given sample  $(x,y)_i$ , its noisy label  $\hat{y}_i$  is said to be informative if  $\text{rank}(T(x_i)) = |Y|$ .
Then, we can make the following theorem:
Theorem 2. With i.i.d  $(\pmb{x},y)_i$  pairs, three informative noisy labels  $\hat{y}_i$  are sufficient and necessary for the identification of  $T(\pmb{x}_i)$ .
The Theorem 2 is based on Kruskal's Identifiability Theorem, the proof is provided in the Appendix D. From Theorem 2 and Definition 2, we can conclude that, in SSL, if we have more than three labeled samples per class, transition matrix  $T(\pmb{x})$  is identifiable, more discussions on this can also be found in Appendix D.
4 InstanT: Instance-dependent thresholds for pseudo-label assignment
4.1 Instance-dependent Threshold Estimation
While Theorem 1 provides a theoretical guarantee for the correctness of  $\tau (\pmb{x}_u)$ , as discussed in section 2.4, SSL algorithms can benefit from non-constant thresholds [9, 41]. Here we define the dynamic threshold  $\kappa_{t}$  at iteration  $t$  from the relative confidence threshold (RT) in AdaMatch [9]:
$$
\kappa_ {t} = \frac {1}{n} \sum_ {i = 1} ^ {n} \max  _ {j \in [ 1, \dots , | Y | ]} \hat {P} (\hat {Y} = j | X = \boldsymbol {x} _ {i}) _ {t} \cdot \beta , \tag {4}
$$
where  $n$  is the number of labeled samples, and  $\beta$  is a fixed discount factor. Subsequently, we present the instance-dependent threshold function at iteration  $t$ :
$$
\tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) _ {t} = \min  \left[ 1, \hat {T} _ {k, k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) _ {t} \hat {P} (Y = s | X = \boldsymbol {x} _ {\boldsymbol {u}}) _ {t} + \sum_ {i \neq k} ^ {| Y |} \hat {T} _ {i, k} \hat {P} (Y = i | X = \boldsymbol {x} _ {\boldsymbol {u}}) _ {t} + \kappa_ {t} \right]. \tag {5}
$$
Note that this estimation process requires the clean class posterior, which can be estimated or approximated using a transition matrix via importance re-weighting [26, 46] or loss correction [32, 45], details will be introduced in the following section. We also remark here that a less strict instance-dependent version of the threshold function can be used with the class-dependent transition matrix [32, 46, 52], which can lead to better empirical results under certain cases, it is also important to note that even under class-dependent transition matrix, the threshold function is still instance-dependent, as the noise class posterior are instance-dependent.
4.2 Modelling Instance-dependent Label Noise
Now we will introduce a piratical method for estimating  $T(\pmb{x})$  in SSL. Our approach involves approximating the label noise transition pattern using a Deep Neural Network. As we recall from Equation 1,  $T(\pmb{x})$  can be described as a mapping function  $T: X \to \mathbb{R}^{|Y| \times |Y|}$ , since we already know that, in general cases,  $T(\pmb{x})$  is identifiable in SSL, we only need to find the correct mapping function. In addition, since we have clean label  $Y_l$  for  $X_l$ , and we also have the noisy pseudo-labels  $\hat{Y}_l$  for  $X_l$ , this enables us to directly fit a model to approximate  $T(\pmb{x})$ . Therefore, we want our transition matrix estimator, parameterized by  $\theta'$ , to approximate the label noise transition process at every iteration  $t$ :
$$
\hat {T} _ {i, j} (\boldsymbol {x} _ {l}) = \hat {P} \left(\hat {Y} _ {l} = j \mid Y _ {l} = i, X _ {l} = \boldsymbol {x} _ {l}; \theta^ {\prime}\right) \approx P \left(\hat {Y} = j \mid Y = i, X = \boldsymbol {x}\right) = T (\boldsymbol {x}). \tag {6}
$$
The above approximation holds since label noise is assumed to be i.i.d, therefore both labeled and unlabeled samples share the same noise transition process. As we can estimate the noisy class posterior for labeled samples, we can therefore collect distribution  $D := \{X_{l}, Y_{l}, \hat{P}(\hat{Y} | X = x_{l})\}$ . Observing Equation 6, we can design following objective for  $\theta'$  to minimize:
$$
L _ {D} \left(\theta^ {\prime}\right) = - \frac {1}{n} \sum_ {l = 1} ^ {n} \hat {\mathbf {y}} _ {l} \log \left(\mathbf {y} _ {l} \cdot \hat {T} (\mathbf {x} _ {l})\right). \tag {7}
$$
Where  $\pmb{y}_l$  is the one-hot vector for the label of labeled samples, thus  $\pmb{y}_l \cdot \hat{T}(\pmb{x}_l)$  is equivalent to finding the class-conditional noise class posterior  $\hat{P}(\hat{Y}_l = j | Y_l = i, X_l = \pmb{x}_l; \theta')$ .  $\hat{\pmb{y}}_l$  is the one-hot noisy pseudo-label of  $\pmb{x}_l$ , minimizing  $L_D(\theta')$  will result  $\hat{T}(\pmb{x}_l; \theta')$  approximate  $T(\pmb{x})$ . As shown in Figure 2,  $\hat{f}_{\theta'}$  will be trained in parallel to the main classifier and updated in every epoch.