SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notably, the estimator for the transition matrix in InstantT is not constrained to one specific method, there has been a wide versatile of  $T$  estimator in the field of label noise learning [25, 45, 51], we have incorporated some of them into the implementation of InstantT, indicating the good extensibility of InstantT.
4.3 Distribution Alignment
When the prediction of the classifier becomes imbalanced, Distribution Alignment (DA) is used to modulate the prediction so that the poorly predicted class will not be overlooked completely [7, 9, 20, 42]. This is achieved by estimating the prior of the pseudo-label  $P(\hat{Y})$  during training, and setting a clean class prior  $\hat{P}(Y)$ , which is usually assumed to be uniformly distributed. The distribution alignment process can be described as  $\hat{P}(\hat{Y} = j | X = x_u) = \mathrm{Norm}(\hat{P}(\hat{Y} = j | X = x_u) P(Y = j) / P(\hat{Y} = j))$ , where Norm is a total-sum scaling. Therefore, if the distribution of  $\hat{Y}$  is severely imbalanced, DA will force the prediction for the minority class to be scaled up, and the prediction for the over-populated class to be scaled down.
4.4 Loss Correction
While the classifier trained with incorrect pseudo-labels is subject to the influence of noisy supervision, in order to satisfy Theorem 1, the estimated softmax predictions from  $\hat{f}_{\theta}$  must approximate the clean class posterior [56]. As we have already demonstrated that the transition matrix can be identified in standard SSL setting, in this part, we will show how to infer clean class posterior from noisy predictions and transition matrix, by utilizing forward correction [32]. Specifically, let  $\ell_{\psi}$  be a proper composite loss [33] (e.g. softmax cross-entropy loss), forward correction is defined as:
$$
\ell_ {\psi} ^ {\rightarrow} \left(\hat {\boldsymbol {y}} _ {\boldsymbol {u}}, \hat {f} _ {\theta} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right)\right) = \ell \left(\hat {\boldsymbol {y}} _ {\boldsymbol {u}}, \hat {T} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) ^ {\top} \psi^ {- 1} \hat {f} _ {\theta} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right)\right), \tag {8}
$$
where  $\psi$  is an invertible link function. It has been proven that minimizing the forward loss with an accurately estimated transition matrix is equivalent to minimizing the loss defined over clean latent pseudo-labels [32]. Therefore, minimizing  $\ell_{\psi}^{\rightarrow}$  will let the softmax prediction of classifier  $\hat{f}_{\theta}$  approximate the clean class posterior.
4.5 Training with Consistency Regularization Loss
Lastly, we provide an overview of the training objective of InstantT. We employ the widely adopted consistency regularization loss [36, 47], which assumes a well-learned and robust model should generate consistent predictions for random perturbations of a given instance. Previous methods have commonly employed augmentation techniques to introduce perturbations. The training loss of classifier  $\hat{f}_{\theta}$  consists of two parts: supervised loss  $L_{D_s}(\theta)$  and unsupervised loss  $L_{D_u}(\theta)$ .  $L_{D_s}(\theta)$  is calculated by the labeled samples, specifically, we have:
$$
L _ {D _ {s}} (\theta) = \frac {1}{n} \sum_ {l = 1} ^ {n} \ell \left(\boldsymbol {y} _ {l}, \hat {f} _ {\theta} (\mathbb {W} \left(\boldsymbol {x} _ {l}\right)), \right. \tag {9}
$$
where  $n$  is the number of labeled samples,  $y_{l}$  is the one-hot label for  $x_{l}$ , and  $\mathbb{W}$  is a weak augmentation function [36, 47], and  $\ell$  is the cross-entropy loss function. Unsupervised loss  $L_{D_u}(\theta)$  is calculated by the unlabeled samples  $X_{u}$  and their predicted pseudo-labels  $\hat{Y}_u$ , which can be defined as:
$$
L _ {D _ {u}} (\theta) = \frac {1}{m} \sum_ {u = 1} ^ {m} \mathbb {1} (\hat {P} (\hat {Y} = k | X = \mathbb {W} (\boldsymbol {x} _ {\boldsymbol {u}})) > \tau (\boldsymbol {x} _ {\boldsymbol {u}}) _ {t}) \ell_ {\psi} ^ {\rightarrow} (\hat {\boldsymbol {y}} _ {\boldsymbol {u}}, \hat {f} _ {\theta} (\mathbb {S} (\boldsymbol {x} _ {\boldsymbol {u}})), \tag {10}
$$
where  $m$  is the number of unlabeled instances,  $\hat{\pmb{y}}_u$  is the one-hot pseudo-label for  $\pmb{x}_u$ , and  $\mathbb{S}$  is a strong augmentation function [7, 14, 15].  $\mathbb{I}$  is an indicator function that filters unlabeled instances using  $\tau (\pmb{x}_u)_t$ . Combining the supervised and unsupervised loss, we then have the overall training objective:
$$
L _ {D} (\theta) = L _ {D _ {s}} (\theta) + \lambda L _ {D _ {u}} (\theta), \tag {11}
$$
where  $\lambda$  is used to control the influence of the unsupervised loss.
Table 1: Top-1 accuracy with pre-trained ViT. The best performance is bold and the second best performance is underlined. All results are averaged with three random seeds  $\{0,1,2\}$  and reported with a  $95\%$  confidence interval.
Dataset	CIFAR-10	CIFAR-100	STL-10
# Label	10	40	250	200	400	2500	10	40	100
PL	37.65±3.1	88.21±5.3	95.42±0.4	63.34±2.0	73.13±0.9	84.28±0.1	30.74±6.7	57.16±4.2	73.44±1.5
MT	64.57±4.9	87.15±2.5	95.25±0.5	59.50±0.8	69.42±0.9	82.91±0.4	42.72±7.8	66.80±3.4	77.71±1.8
MixMatch	65.04±2.6	97.16±0.9	97.95±0.1	60.36±01.3	72.26±0.1	83.84±0.2	10.68±1.1	27.58±16.2	61.85±11.3
VAT	60.07±6.3	93.33±6.6	97.67±0.2	65.89±1.8	75.33±0.4	83.42±0.4	20.57±4.4	65.18±7.0	80.94±1.0
UDA	78.76±3.6	97.92±0.2	97.96±0.1	65.49±1.6	75.85±0.6	83.81±0.2	48.37±4.3	79.67±4.9	89.46±1.0
FixMatch	66.50±15.1	97.44±0.9	97.95±0.1	65.29±1.4	75.52±0.1	83.98±0.1	40.13±3.4	77.72±4.4	88.41±1.6
FlexMatch	70.54±9.6	97.78±0.3	97.88±0.2	63.76±0.9	74.01±0.5	83.72±0.2	60.63±12.9	78.17±3.7	89.54±1.3
Dash	74.35±4.5	96.63±2.0	97.90±0.3	63.33±0.4	74.54±0.2	84.01±0.2	41.06±4.4	78.03±3.9	89.56±2.0
AdaMatch	85.15±20.4	97.94±0.1	97.92±0.1	73.61±0.1	78.59±0.4	84.49±0.1	68.17±7.7	83.50±4.2	89.25±1.5
InstanT	87.32±10.2	97.93±0.1	98.08±0.1	74.17±0.3	78.80±0.4	84.28±0.5	69.39±7.4	85.09±2.8	89.35±1.9
We use three benchmark datasets for evaluating the performances of InstantT, they are: CIFAR-10, CIFAR-100 [21], and STL-10 [13]. CIFAR-10 has 10 classes and 6,000 samples per class. CIFAR-100 has 100 classes and 600 samples per class. STL-10 has 10 classes and 500 labeled samples per class, 100,000 unlabeled instances in total. For each dataset, we set varying numbers of labeled samples to create different levels of difficulty. Following recently more challenging settings [10, 41], where the labeled samples could be extremely limited, we set the number of labeled samples per class on CIFAR-10 as \{1, 4, 25\}, for CIFAR-100, we set as \{2, 4, 25\}, for STL-10, we set as \{1, 4, 10\}.
To ensure fair comparison between our method and all baselines, and to allow simple reproduction of our experimental results, we implemented InstantT and conducted all experiments within USB (Unified SSL Benchmark) framework [40]. To improve the training efficiency, e.g. faster convergence, we use pre-trained ViT as the backbone model for all baselines with the same hyper-parameters in part I. We use AdamW [28] as the default optimizer, where the learning rate is set as  $5e - 4$  for CIFAR-10/100,  $1e - 4$  for STL-10 [40]. The total training iterations  $K$  are set as 204,800 for all datasets. The training batch size is set as 8 for all datasets. For a more comprehensive evaluation of our proposed method, we also conduct experiments with Wide ResNet [53] trained from scratch, the detailed settings are aligned with existing works [7, 9, 36, 47, 48, 54], these results are summarized in part II. We select a range of popular baseline methods to evaluate against InstantT, which includes Pseudo-Label (PL) [23], MeanTeacher (MT) [37], VAT [29], MixMatch [8], UDA [47], FixMatch [36], Dash [48], FlexMatch [54] and AdaMatch [9]. More comprehensive setting details, including full details of hyper-parameters, can be found in Appendix A.
Part I. Our main results will be divided into two parts: part I and part II. For training efficiency, we will use a pre-trained Vision Transformer as the backbone model in part I, enabling us to compare a more diverse collection of baselines over a broader range of benchmark settings. In Part II, we will select baselines with better performance from Part I and train them from scratch for a more comprehensive evaluation.
Experimental results from part I are summarized in Table 1, where we can observe that, overall, InstantT achieves the best performances in most of the settings. More specifically, in the setting where label amount is
extremely limited, InstanT exhibits the most significant improvement over all datasets, including an average  $2.17\%$  increase in accuracy on CIFAR-10 (10) over SOTA. While the identifiability of the transition matrix cannot be guaranteed with extremely limited labels, this could be remedied by including highly confident instances and their pseudo-labels to the labeled set and using them to better estimate the transition matrix. However, as the number of labeled samples increases, the
Table 2: Running time on STL-10(40) $^2$ .
Method	s/epoch
FixMatch	30.1
AdaMatch	30.1
InstanT	31.1
2Running time is tested on NVIDIA RTX 4090 GPUs.
Table 3: Top-1 accuracy with WRN-28. The best performance is bold and the second best performance is underlined. All results are averaged with three random seeds  $\{0,1,2\}$  and reported with a  $95\%$  confidence interval.
Dataset	CIFAR-10	CIFAR-100
# Label	40	250	4000	400	2500	10000
UDA	91.60±1.5	94.44±0.3	95.55±0.0	40.60±1.8	64.79±0.8	72.13±0.2
FixMatch	91.45±1.7	94.87±0.1	95.51±0.1	45.08±3.4	65.63±0.4	71.65±0.3
FlexMatch	94.53±0.4	94.85±0.1	95.62±0.1	47.81±1.4	66.11±0.4	72.48±0.2
Dash	84.67±4.3	94.78±0.3	95.54±0.1	45.26±2.6	65.51±0.1	72.10±0.3
AdaMatch	94.64±0.0	94.76±0.1	95.46±0.1	52.02±1.7	66.36±0.7	72.32±0.2
InstanT	94.83±0.1	94.72±0.2	95.57±0.0	53.94±1.8	67.09±0.0	72.30±0.4
dominance of InstantT becomes less pronounced. We hypothesize that this can be attributed to the classifier  $\hat{f}_{\theta}$  performing better with more labeled samples, resulting in fewer label errors. Since InstantT primarily aims to increase the threshold for instances likely to have noisy pseudo-labels, the impact of InstantT becomes less significant in the presence of reduced label noise. Moreover, comparing the performance gap between InstantT and its closest counterpart, AdaMatch, it becomes apparent that InstantT consistently outperforms AdaMatch in nearly all cases. This observation underscores the non-trivial improvement and contribution brought forth by InstantT.
Part II. In this part, we will present the experiment results of part II, which are summarized in Table 3. Specifically, we will compare InstantT against other popular baseline methods on CIFAR-10/100 trained from scratch. For all methods, we use WRN-28-2 as default backbone model, trained with  $2^{20}$  iterations. We use SGD as the default optimizer, with momentum set as 0.9, initial learning rate as 0.03 and a cosine learning rate scheduler.
Observing from experiment results, we can summarize similar patterns as the results from part I. Overall, InstantT brings most significant performance increases when the number of labeled samples are limited, e.g., when there is only 4 labeled samples per-class. Notably, for CIFAR-100 (400), InstantT exhibits an increase in accuracy for nearly  $2\%$ , which can be view as a significant improvement over SOTA baseline. While on other cases such as CIFAR-10 (250) and CIFAR-100 (10000), InstantT outperformed by other baseline methods, we emphasize that no single baseline consistently outperforms InstantT in all cases.
Another question of interest is whether InstantT will be as efficient as other methods in training time. While learning transition matrix estimator does sacrifice time and space complexity to some extent, we show that InstantT can still maintain high efficiency. As shown in Table 2, compared with FixMatch and AdaMatch, InstantT almost does not introduce further significant computational burden.
To gain a better understanding of the key differences between InstantT and existing methods, we now focus on the visualizations of InstantT compared to representative baselines. Specifically, we will display the classification accuracy, unlabeled sample utilization ratio, and pseudo-label accuracy of InstantT, FreeMatch [41], AdaMatch [9], and FixMatch [36] in Figure 3. These models are trained from scratch on CIFAR-100 with 400 labeled samples, following commonly used settings [36, 41].
Examining the classification accuracy in Figure 3(a), we observe that while InstantT does not converge as rapidly as FreeMatch, it ultimately achieves superior accuracy in the later stages of training. This is attributed to InstantT's ability to find a better balance between the quantity and quality of pseudo-labels. As depicted in Figure 3(b) and 3(c), although InstantT utilizes fewer unlabeled samples compared to FreeMatch, it demonstrates significantly improved accuracy on pseudo-labels. On the other hand, AdaMatch exhibits similar pseudo-label accuracy to InstantT but utilizes a smaller proportion of unlabeled instances. The key distinction lies in the instance-dependent threshold function employed by InstantT, which allows us to set a lower  $\kappa_{t}$  for InstantT compared to AdaMatch, as InstantT mitigates the negative impact by increasing the instance-dependent thresholds for instances more prone to label noise, whereas AdaMatch will incorporate too many instances with incorrect pseudo-labels.
5.3 Ablation study
(a) Top-1 Accuracy
(b) Utilization ratio
Figure 3: Results from Instant and selected baselines, trained from scratch on CIFAR-100(400).
(c) Pseudo-label Accuracy
In this section, we will be verifying the effects of each component of InstantT, specifically, we will focus on evaluating the parts that we adapted from existing works, and determine how many performance improvements are attributed to our instance-dependent thresholds. Results of this ablation study are summarized in Table 4, where all experiments are averaged over three random seeds and ran on STL-10(40) using the setting of part I. First, we observe that InstantT-I removed both Distribution Alignment (DA) and relative thresholds (RT), which makes it equivalent
Table 4: Ablation study on STL-10(40).
Method	RT	DA	Acc.
FixMatch	X	X	77.72±4.4
AdaMatch	✓	✓	83.50±4.2
InstanT-I	X	X	79.92±5.8
InstanT-II	✓	X	81.49±4.9
InstanT-III	X	✓	82.97±2.7
InstanT	✓	✓	85.09±2.8
to adding  $\tau(x_{u})$  on top of the fixed threshold of FixMatch. Notably, this also brings a significant improvement over FixMatch for over  $2\%$ , which further verifies the effectiveness of our proposed instance-dependent thresholds. Simply removing RT or DA will also cause a performance drop, which is aligned with the results from AdaMatch [9].
In this paper, we introduce a novel approach to thresholding techniques in SSL called instance-dependent confidence threshold. This approach offers the highest level of flexibility among existing methods, providing significant potential for further advancements. We then present InstantT, a theoretically guided method designed under the concept of instance-dependent threshold. InstantT assigns unique confidence thresholds to each unlabeled instance, considering their individual likelihood of having incorrect labels. Additionally, we demonstrate that our proposed method ensures a bounded probability of assigning correct pseudo-labels, a characteristic rarely offered by existing SSL approaches. Through extensive experiments, we demonstrate the competitive performance of InstantT when compared to SOTA baselines.
Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031. Muyang Li, Runze Wu and Haoyu Liu are supported by NetEase Youling Crowdsourcing Platform. This research was undertaken with the assistance of resources from the National Computational Infrastructure (NCI Australia), an NCRIS enabled capability supported by the Australian Government. The authors acknowledge the technical assistance provided by the Sydney Informatics Hub, a Core Research Facility of the University of Sydney.
[51] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. Advances in neural information processing systems, 33:7260-7271, 2020.
[52] LIN Yong, Renjie Pi, Weizhong Zhang, Xiaobo Xia, Jiahui Gao, Xiao Zhou, Tongliang Liu, and Bo Han. A holistic view of label noise transition matrix in deep learning and beyond. In The Eleventh International Conference on Learning Representations, 2022.
[53] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[54] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408-18419, 2021.
[55] Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature-dependent label noise: A progressive approach. In Ninth International Conference on Learning Representations, volume 9, 2021.
[56] Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learning, pages 11447-11457. PMLR, 2020.
A Comprehensive settings
For all methods, we apply temperature scaling for probability calibration by default, and the scaling factor is set as 0.5. For FixMatch, Dash, FlexMatch and AdaMatch,  $\tau$  is set as 0.95. For UDA,  $\tau$  is set as 0.8, temperature scaling factor is set as 0.4, according to the recommendation of the original paper. For Dash,  $\gamma$  is set as 1.27,  $C$  is set as 1.0001,  $\rho$  is set as 0.05, warm-up iteration is set as 5120 iterations. For InstantT, base  $\tau$  is set as 0.9 by default.
B Tsybakov Margin Condition (Binary)
We present the binary case of Tsybakov Margin Condition here. It can be defined as:
Assumption 2 (Binary Tsybakov Margin Condition). For some finite constant  $C, \alpha > 0$ , and  $\delta_0 \in \left(0, \frac{1}{2}\right]$ , the Tsybakov Margin Condition holds if  $\forall \delta \in (0, \delta_0]$ , we have
$$
P \left[ \left| P (Y | X = \boldsymbol {x}) - \frac {1}{2} \right| \leq \delta \right] \leq C \delta^ {\alpha}.
$$
Through similar derivations, we can provide our main theorem in binary case:
Theorem 3. Assume estimated noisy class posterior  $\hat{P}(\hat{Y} | X)$  satisfies Assumption 2 with  $C, \alpha > 0$ ,  $\delta_0 \in (0, \frac{1}{2}]$ , and  $\epsilon_\theta \leq \delta_0(1 - T_{1,0(\pmb{x}) - T_{0,1}(\pmb{x})))$ , we have:
$$
P \left[ k = h ^ {*} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \hat {P} (\hat {Y} = k | X = \boldsymbol {x} _ {\boldsymbol {u}}) \geq \tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \right] > 1 - C [ O \left(\epsilon_ {\theta}\right) ] ^ {\alpha}, \tag {12}
$$
where  $\tau (\pmb{x}_{\pmb{u}})$  is the instance-dependent threshold function under binary case:
$$
\tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) = T _ {k, k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) P (Y = s | X = \boldsymbol {x} _ {\boldsymbol {u}}) + \sum_ {i \neq k} ^ {| Y |} T _ {i, k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) P (Y = i | X = \boldsymbol {x} _ {\boldsymbol {u}}). \tag {13}
$$
Definition 3. For multi-class classification, the relationship between noisy class posterior  $P(\hat{Y} | X)$  and clean class posterior  $P(Y|X)$  can be concluded as:
$$
P (\hat {Y} = j | X = x) = \sum_ {i = 1} ^ {| Y |} T _ {i, j} (X = x) P (Y = i | X = x). \tag {14}
$$
For simplicity, we will denote the clean class posterior  $P(Y = k|X = \pmb{x})$  as  $\eta_k(\pmb{x})$ , noise class posterior  $P(\hat{Y} = k|X = \pmb{x})$  as  $\eta_{\hat{k}}(\pmb{x})$ , estimated noise class posterior  $\hat{P} (\hat{Y} = k|X = \pmb{x})$  as  $\hat{\eta}_{\hat{k}}(\pmb {x})$ .
$$
\begin{array}{l} P \left[ k = h ^ {*} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \hat {\eta} _ {\hat {k}} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \geq \tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \right] = 1 - P \left[ k = h ^ {*} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \hat {\eta} _ {\hat {k}} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) <   \tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \right] \\ = 1 - P \left[ \eta_ {k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \geq \eta_ {s} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \hat {\eta} _ {\hat {k}} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) <   \tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \right] \\ \end{array}
$$
Based on Definition 3, we can substitute  $\eta_{\hat{k}}(\pmb{x}_{\pmb{u}})$  with  $\sum_{i=1}^{|Y|} T_{i,k}(\pmb{x}_{\pmb{u}}) \eta_i(\pmb{x}_{\pmb{u}})$ , and since  $\|\eta_{\hat{k}} - \hat{\eta}_{\hat{k}}\|_{\infty} \leq \epsilon_{\theta}$  ( $\epsilon$ -close), we can continue with:
$$
\begin{array}{l} \leq 1 - P \left[ \eta_ {k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \geq \eta_ {s} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \sum_ {i = 1} ^ {| Y |} T _ {i, k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \eta_ {i} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) <   \tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) + \epsilon_ {\theta} \right] \\ = 1 - P \left[ \eta_ {k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \geq \eta_ {s} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right), \eta_ {k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) <   \frac {\tau \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) - \sum_ {i \neq k} ^ {| Y |} T _ {i , k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \eta_ {i} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) + \epsilon_ {\theta}}{T _ {k , k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right)} \right] \\ = 1 - P \left[ \eta_ {s} (\boldsymbol {x} _ {\boldsymbol {u}}) \leq \eta_ {k} (\boldsymbol {x} _ {\boldsymbol {u}}) <   \frac {\tau (\boldsymbol {x} _ {\boldsymbol {u}}) - \sum_ {i \neq k} ^ {| Y |} T _ {i , k} (\boldsymbol {x} _ {\boldsymbol {u}}) \eta_ {i} (\boldsymbol {x} _ {\boldsymbol {u}})}{T _ {k , k} (\boldsymbol {x} _ {\boldsymbol {u}})} + \frac {\epsilon_ {\theta}}{T _ {k , k} (\boldsymbol {x} _ {\boldsymbol {u}})} \right] \\ = 1 - P \left[ \eta_ {s} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) \leq \eta_ {k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) <   \eta_ {s} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right) + \frac {\epsilon_ {\theta}}{T _ {k , k} \left(\boldsymbol {x} _ {\boldsymbol {u}}\right)} \right] \\ \end{array}
$$
Note that, since  $\epsilon_{\theta} \leq t_0 \min_k T_{k,k}(\boldsymbol{x}_u)$ , the Multi-class Tsybakov Condition can be applied here:
$$
P \left[ 0 \leq \eta_ {k} (\boldsymbol {x _ {u}}) - \eta_ {s} (\boldsymbol {x _ {u}}) <   \frac {\epsilon_ {\theta}}{T _ {k , k} (\boldsymbol {x _ {u}})} \right] \leq C \left[ \frac {\epsilon_ {\theta}}{T _ {k , k} (\boldsymbol {x _ {u}})} \right] ^ {\alpha}
$$
Which gives us:
$$
1 - P \left[ 0 \leq \eta_ {k} (\boldsymbol {x} _ {\boldsymbol {u}}) - \eta_ {s} (\boldsymbol {x} _ {\boldsymbol {u}}) <   \frac {\epsilon_ {\theta}}{T _ {k , k} (\boldsymbol {x} _ {\boldsymbol {u}})} \right] > 1 - C \left[ \frac {\epsilon_ {\theta}}{T _ {k , k} (\boldsymbol {x} _ {\boldsymbol {u}})} \right] ^ {\alpha}
$$
Hence proved  $P\left[k = h^{*}(\pmb{x}_{\pmb{u}}), \hat{\eta}_{\hat{k}}(\pmb{x}_{\pmb{u}}) \geq \tau(\pmb{x}_{\pmb{u}})\right] > 1 - C[O(\epsilon_{\theta})]^{\alpha}$ .
In this section, we provide the proof from Liu et al. for the necessary and sufficient condition of the identification of instance-dependent transition matrix  $T(\pmb{x})$ . First, we provide the Kruskal's Identifiability Theorem (or Kruskal rank Theorem):
Theorem 4. The parameters  $M_{i}, i = 1, \dots, p$  are identifiable, up to label permutation, if
$$
\sum_ {i = 1} ^ {p} \left(M _ {i}\right) \geq 2 | Y | + p - 1 \tag {15}
$$