SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Then we follow the proof from Liu et al. to first prove the sufficiency. Since we cannot observe the true label for unlabeled instances  $X_{u}$ , we denote them as hidden variable  $Z$  for simplicity.  $P(Z = i)$  is the prior for the latent true label.
Each  $\hat{Y} = i, \forall i = 1, \dots, p$  corresponds to the observation  $O_i$ .  $|\hat{Y}|$  is the cardinality of the pseudo-label space. Without loss of generality, we will fix the number of classes as three during the analysis.
Each  $\hat{Y}_i$  corresponds to an observation matrix  $M_i$ :
$$
M _ {i} [ j, k ] = P \left(O _ {i} = k \mid Z = j\right) = P \left(\hat {Y} _ {i} = k \mid Y = j, X = x\right)
$$
Therefore, by definition of  $M_1, M_2, M_3$  and  $T(\pmb{x})$ , they all equal to  $T(\pmb{x})$ :  $M_i \equiv T(X), i = 1, 2, 3$ . When  $T(\pmb{x})$  has full rank, we know immediately that all rows in  $M_1, M_2, M_3$  are independent. Therefore, the Kruskal ranks satisfy
$$
(M _ {1}) = (M _ {2}) = (M _ {3}) = | \hat {Y} |
$$
Checking the condition in Theorem 4, we easily verify
$$
\left(M _ {1}\right) + \left(M _ {2}\right) + \left(M _ {3}\right) = 3 | \hat {Y} | \geq 2 | \hat {Y} | + 2
$$
Where Theorem 4 proves the sufficiency for three labeled samples per-class is sufficient for the identification of  $T(\pmb{x})$
Subsequently, Liu et al. also provides the proof for necessity, without the loss of generality, the analysis for this part will be conducted in binary case only. More specifically, the label error generation in SSL can be described by a binary transition matrix:
$$
T (\boldsymbol {x}) = \left[ \begin{array}{c c} 1 - e _ {-} (\boldsymbol {x}) & e _ {-} (\boldsymbol {x}) \\ e _ {+} (\boldsymbol {x}) & 1 - e _ {+} (\boldsymbol {x}) \end{array} \right]
$$
In order to prove necessity, we must show that less than three informative labels per-class will not give us a unique solution for  $T(\pmb{x})$ . If we were given two unlabeled instances with their pseudo-label  $\hat{Y}_1$  and  $\hat{Y}_2$ . Specifically, with two unlabeled instance and pseudo-label pairs, we can conclude the following probabilities can derive all potential probabilities we can possibly get from them:
Posterior:  $P(\hat{Y}_1 = +1|X)$
Positive Consensus:  $P(\hat{Y}_1 = \hat{Y}_2 = +1|X)$
Negative Consensus:  $P(\hat{Y}_1 = \hat{Y}_2 = -1|X)$
This is because other probability distributions are able to be derived from these three quantities:
$$
P \left(\hat {Y} _ {1} = - 1 | X\right) = 1 - P \left(\hat {Y} _ {1} = + 1 | X\right),
$$
$$
P \left(\hat {Y} _ {1} = + 1, \hat {Y} _ {2} = - 1 | X\right) = P \left(\hat {Y} _ {1} = + 1 | X\right) - P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 | X\right),
$$
$$
P \left(\hat {Y} _ {1} = - 1, \hat {Y} _ {2} = + 1 | X\right) = P \left(\hat {Y} _ {2} = + 1 | X\right) - P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 | X\right).
$$
But  $P(\hat{Y}_2 = +1|X) = P(\hat{Y}_1 = +1|X)$ , since noisy labels are i.i.d. The above three quantities led to three equations that depend on  $e_+$ ,  $e_-$ . This gives us the following system of equations:
$$
\begin{array}{l} P (\tilde {Y} = + 1 | X) = P (Y = + 1) \cdot \left(1 - e _ {+}\right) + \left(1 - P (Y = + 1)\right) \cdot e _ {-} \\ P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 | X\right) = P (Y = + 1) \cdot \left(1 - e _ {+}\right) ^ {2} + \left(1 - P (Y = + 1)\right) \cdot e _ {-} ^ {2} \\ P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = - 1 | X\right) = P (Y = + 1) \cdot e _ {+} ^ {2} + \left(1 - P (Y = + 1)\right) \cdot \left(1 - e _ {-}\right) ^ {2} \\ P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 | X\right) \\ = P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1, Y = + 1 \mid X\right) \\ + P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1, Y = - 1 | X\right) \\ = P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 \mid Y = + 1, X\right) \cdot P (Y = + 1 \mid X) \\ + P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 \mid Y = - 1, X\right) \cdot P (Y = - 1 \mid X) \\ = P (Y = + 1) \cdot \left(1 - e _ {+}\right) ^ {2} + \left(1 - P (Y = + 1)\right) \cdot e _ {-} ^ {2} \\ \end{array}
$$
Given  $Y, \hat{Y}_1, \hat{Y}_2$  are conditional independent, which gives us:
$$
P (\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 | Y = + 1, X) =
$$
$$
P \left(\hat {Y} _ {1} = + 1 | Y = + 1, X\right) \left(\hat {Y} _ {2} = + 1 | Y = + 1, X\right)
$$
$$
P \left(\hat {Y} _ {1} = \hat {Y} _ {2} = + 1 \mid Y = - 1, X\right) =
$$
$$
P (\hat {Y} _ {1} = + 1 | Y = - 1, X) (\hat {Y} _ {2} = + 1 | Y = - 1, X)
$$
Conversely, for  $P(\hat{Y}_1 = \hat{Y}_2 = -1|X)$ , the derivation can follow similar procedure. We can now assert that  $e_+$ ,  $e_-$  are not identifiable. Where a simple counter example can show that two sets of unique  $e_+$ ,  $e_-$  both satisfies above derivation:
$$
\begin{array}{l} \cdot P (Y = + 1) = 0. 7, e _ {+} = 0. 2, e _ {-} = 0. 2 \\ \cdot P (Y = + 1) = 0. 8, e _ {+} = 0. 2 4 2, e _ {-} = 0. 0 7 \\ \end{array}
$$
Which proves that two informative noisy labels are insufficient to identify  $T(\pmb{x}_u)$ .