SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhongqi Yue<sup>1</sup>, Hanwang Zhang<sup>1</sup>, Qianru Sun<sup>2</sup>
<sup>1</sup>Nanyang Technological University, <sup>2</sup>Singapore Management University yuez0003@ntu.edu.sg, hanwangzhang@ntu.edu.sg, qianrusun@smu.edu.sg
Domain Adaptation (DA) is always challenged by the spurious correlation between domain-invariant features (e.g., class identity) and domain-specific features (e.g., environment) that does not generalize to the target domain. Unfortunately, even enriched with additional unsupervised target domains, existing Unsupervised DA (UDA) methods still suffer from it. This is because the source domain supervision only considers the target domain samples as auxiliary data (e.g., by pseudo-labeling), yet the inherent distribution in the target domain—where the valuable de-correlation clues hide—is disregarded. We propose to make the U in UDA matter by giving equal status to the two domains. Specifically, we learn an invariant classifier whose prediction is simultaneously consistent with the labels in the source domain and clusters in the target domain, hence the spurious correlation inconsistent in the target domain is removed. We dub our approach "Invariant Consistency learning" (ICON). Extensive experiments show that ICON achieves the state-of-the-art performance on the classic UDA benchmarks: OFFICE-HOME and VIsDA-2017, and outperforms all the conventional methods on the challenging WILDS 2.0 benchmark. Codes are in Appendix.
Domain Adaptation (DA) is all about training a model in a labelled source domain  $S$  (e.g., an autopilot trained in daytime), and the model is expected to generalize in a target domain  $T$  (e.g., the autopilot deployed in nighttime), where  $T$  and  $S$  has a significant domain shift (e.g., day vs. night) [41]. To illustrate the shift and generalization, we introduce the classic notion of causal representation learning [20, 49]: any sample is generated by  $\mathbf{x} = \Phi (\mathbf{c},\mathbf{e})$ , where  $\Phi$  is the generator,  $\mathbf{c}$  is the causal feature determining the domain-invariant class identity (e.g., road lane), and  $\mathbf{e}$  is the environmental feature (e.g., lighting conditions). As the environment is domain-specific, the domain shift  $\mathbf{e}_S\neq \mathbf{e}_T$  results in  $\mathbf{x}_S\neq \mathbf{x}_T$ , challenging the pursuit of a domain-invariant
class model. However, if a model disentangles the causal feature by transforming each  $\mathbf{x} = \mathbf{c}$ , it can generalize under arbitrary shift in  $\mathbf{e}$ . In Figure 1, even though  $\mathbf{e}_T =$  "colorful" is significantly
Figure 1: A digit classification model trained on red digits in  $S$  generalizes to colorful digits in  $T$  by disentangling digit shape (i.e., the causal feature  $\mathbf{c}$ ), as color in  $S$  (i.e.,  $\mathbf{e}_S$ ) is not discriminative.  $\bullet$ : "0",  $\bullet$ : "1". The classification boundary is rendered blue and red.
Figure 2: Red:  $S$ , Blue:  $T$ . ●: "0",▲: "1". (a) A failure case of existing lineups. The red line denotes the classifier trained in  $S$ . (b) Classifier can be corrected (black line) by respecting the BCE loss between  $S$  (red line) and  $T$  (blue line). (c) A failure case of minimizing the combined BCE losses in  $S$  and  $T$ . Due to the low variance of  $\mathsf{e}_S$  in each class, the optimal classifier w.r.t. the BCE loss in  $S$  (red line) looks "flat", i.e., it has large weight on  $\mathsf{e}$  [39]. (d) Our ICON prevents such a failure case.
different from  $\mathbf{e}_S =$  "red", thanks to the non-discriminative "red" color in  $S$ , the model still easily disentangles  $\mathbf{c}$  from  $\Phi (\mathbf{c},\mathbf{e}_S)$  and thus generalizes to digits of any color.
Unfortunately, in general, disentangling  $\mathbf{c}$  by using only  $S$  is impossible as  $\mathbf{e}_S$  can be also discriminative, i.e., spuriously correlated with  $\mathbf{c}$ . In Figure 2a, the color  $\mathbf{e}_S$  is correlated with the shape  $\mathbf{c}$ , i.e.,  $\mathbf{c}_1 = "0"$  and  $\mathbf{c}_2 = "1"$  in  $S$  tend to have  $\mathbf{e}_1 =$  dark and  $\mathbf{e}_2 =$  light digit colors, respectively. As both  $\mathbf{c}$  and  $\mathbf{e}_S$  are discriminative, a classifier trained in  $S$  will inevitably capture both (see the red line in Figure 2a). This leads to poor generalization when the correlation between  $\mathbf{e}_T$  and  $\mathbf{c}$  is different, i.e., the colors of  $\mathbf{c}_1 = "0"$  and  $\mathbf{c}_2 = "1"$  in  $T$  tend to be  $\mathbf{e}_2 =$  light and  $\mathbf{e}_1 =$  dark instead.
To avoid spurious correlations between  $\mathbf{e}_S$  and  $\mathbf{c}$ , a practical setting called Unsupervised DA (UDA) introduces additional information about  $T$  through a set of unlabeled samples [15, 35, 12]. There are two main lineups: 1) Domain-alignment methods [58, 34, 10] learn the common feature space of  $S$  and  $T$  that minimizes the classification error in  $S$ . In Figure 2a, their goal is to unify "0"s and "1"s in  $S$  and  $T$ . However, they suffer from the well-known misalignment issue [31, 71], e.g., collapsing "0"s and "1"s in  $T$  to "1"s and "0"s in  $S$  respectively also satisfies the training objective, but it generalizes poorly to  $T$ . 2) Self-training methods [72, 38, 52] use a classifier trained in  $S$  to pseudo-label samples in  $T$ , and jointly train the model with the labels in  $S$  and confident pseudo-labels in  $T$ . Yet, the pseudo-labels can be unreliable even with expensive threshold tweaking [32, 29], e.g., the  $\pmb{x}$  areas in Figure 2a have confident but wrong pseudo-labels. In fact, the recent WILDS 2.0 benchmark [47] on real-world UDA tasks shows that both lineups even under-perform an embarrassingly naive baseline: directly train in  $S$  and test in  $T$ .
Notably, although both methods respect the idiosyncrasies in  $S$  (by minimizing the classification error), they fail to account for the inherent distribution in  $T$ , e.g., in Figure 2a, even though  $T$  is unlabelled, we can still identify the two sample clusters in  $T$  (enclosed by the blue circles), which provide additional supervision for the classifier trained in  $S$ . In particular, the classifier (red line) in Figure 2a breaks up the two clusters in  $T$ , showing that the correlation between color  $\mathbf{e}_S$  and  $\mathbf{c}$  in  $S$  is inconsistent with the clusters affected by  $\mathbf{e}_T$  in  $T$ , which implies that color  $\mathbf{e}$  is the environmental feature. Hence, to make the U in UDA matter, we aim to learn a classifier that is consistent with classification in  $S$  and clustering in  $T$ :
Consistency. We use the Binary Cross-Entropy (BCE) loss to encourage the prediction similarity of each sample pair from the same class of  $S$  or cluster of in  $T$ , and penalize the pairs from different ones. In Figure 2b, the red and blue lines denote the classifiers that minimize the BCE loss in  $S$  and  $T$ , respectively. By minimizing the combined loss, we learn the classifier consistent with the sample distribution in both domains (the upright black line), which predicts solely based on the causal feature  $\mathbf{c}$  (i.e., disentangling  $\mathbf{c}$  from  $\Phi(\mathbf{c}, \mathbf{e})$ ).
Invariance. However, enforcing consistency is a tug-of-war between  $S$  and  $T$ , and the BCE loss in one domain can dominate the other to cause failures. Without loss of generality, we consider the case where  $S$  dominates in Figure 2c. Due to the strong correlation between  $\mathbf{e}_S$  and  $\mathbf{c}$ , the classifier that minimizes the combined losses of BCE (black line) deviates from the desired upright position as in Figure 2b. To tackle this, we aim to learn an invariant classifier [1] that is simultaneously optimal w.r.t. the BCE loss in  $S$  and  $T$ . In Figure 2d, for each of the two losses, we color the candidate regions where the classifier is optimal (i.e., gradients  $\approx 0$ ) with red and blue, respectively. In particular, by
Figure 3: A training example where the pair of samples in  $S$  is from different classes and that in  $T$  is from the same cluster. The black arrows on the similarity bars denote that minimizing the BCE losses in  $S$  and  $T$  will decrease the similarity in  $S$  (as  $y_{1} \neq y_{2}$ ) and increase that in  $T$  (as  $y_{1} = y_{2}$ ).
learning an invariant classifier lying in the intersection (purple region), we successfully recover the desired one as in Figure 2b, even when one domain dominates.
Overall, we term our approach as Invariant CONSistency learning (ICON). Our contributions:
- ICON is a novel UDA method that can be plugged into different self-training baselines. It is simple to implement (Section 2), and disentangles the causal feature  $\mathbf{c}$  from  $\Phi (\mathbf{c},\mathbf{e})$  that generalizes to  $T$  with a theoretical guarantee (Section 3).
- ICON significantly improves the current state-of-the-art on classic UDA benchmarks [29, 6] with ResNet-50 [19] as the feature backbone:  $75.8\%$ $(+2.6\%)$  accuracy averaging over the 12 tasks in OFFICE-HOME [60], and  $87.4\%$ $(+3.7\%)$  mean-class accuracy in VISDA-2017 [43] (Section 5).
- Notably, as of NeurIPS 2023, ICON dominates the leaderboard of the challenging WILDS 2.0 UDA benchmark [47] (https://wilds.stanford.edu/leaderboard/), which includes 8 large-scale classification, regression and detection tasks on image, text and graph data (Section 5).
UDA aims to learn a model with the labelled training data  $\{\mathbf{x}_i, y_i\}_{i=1}^N$  in the source domain  $S$  and unlabelled  $\{\mathbf{x}_i\}_{i=1}^M$  in the target domain  $T$ , where  $\mathbf{x}_i$  denotes the feature of the  $i$ -th sample extracted by a backbone parameterized by  $\theta$  (e.g., ResNet-50 [19]), and  $y_i$  is its ground-truth label. We drop the subscript  $i$  for simplicity when the context is clear. As shown in Figure 3, our model includes the backbone  $\theta$ , a classification head  $f$ , and a cluster head  $g$ , where  $f$  and  $g$  output the softmax-normalized probability for each class and cluster, respectively. Note that  $f$  and  $g$  have the same output dimension as the class numbers are the same in  $S$  and  $T$ . As discussed in Section 1, the generalization objective of ICON is to learn an invariant  $f$  that is simultaneously consistent with the classification in  $S$ , and the clustering in  $T$ , identified by  $g$ . ICON is illustrated in Figure 3 and explained in details below.
2.1 Consistency with  $S$
We use Binary Cross-Entropy (BCE) loss to enforce the consistency of  $f$  with  $S$ , denoted as  $\mathrm{BCE}(S, f)$ . Specifically,  $\mathrm{BCE}(S, f)$  is a pair-wise loss given by:
$$
\mathbf {B C E} (S, f) = \mathbb {E} _ {\left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right) \sim S} [ b \log (\hat {y}) + (1 - b) \log (1 - \hat {y}) ], \tag {1}
$$
where  $b = \mathbb{1}(y_i = y_j)$  is a binary label indicating if the pair  $(\mathbf{x}_i, \mathbf{x}_j)$  is from the same class, and  $\hat{y} = f(\mathbf{x}_i)^{\top} f(\mathbf{x}_j)$  is the predictive similarity. Intuitively, BCE is a form of contrastive loss that clusters sample pairs with the same label (by increasing the dot-product similarity of their predictions) and pushes away pairs with different labels (by decreasing the similarity). Note that in practice, BCE loss is computed within each mini-batch, hence there is no  $N^2$  prohibitive calculation.
2.2 Consistency with  $T$
Unfortunately,  $\mathrm{BCE}(T,f)$  cannot be evaluated directly, as the binary label  $b$  in Eq. (1) is intractable given unlabeled  $T$ . To tackle this, we cluster  $T$  to capture its inherent distribution and use cluster labels to compute  $\mathrm{BCE}(T,f)$ .
Clustering  $T$ . We adopt the rank-statistics algorithm [18], as it can be easily implemented in a mini-batch sampling manner for online clustering. The algorithm trains the cluster head  $g$  to group features whose top values have the same indices (details in Appendix). Ablation on clustering algorithm is in Section 5.4. Note that our  $g$  is trained from scratch, rather than initialized from  $f$  trained in  $S$  [56, 29, 32]. Hence  $g$  captures the distribution of  $T$  without being affected by that of  $S$ .
Computing  $\mathrm{BCE}(T,f)$ . After training  $g$ , we replace  $b = \mathbb{1}(y_i = y_j)$  for  $\mathrm{BCE}(S,f)$  in Eq. (1) with  $b = \mathbb{1}$  ( $\operatorname{argmax} g(\mathbf{x}_i) = \operatorname{argmax} g(\mathbf{x}_j)$ ) for  $\mathrm{BCE}(T,f)$ , which compares the cluster labels of each sample pair. We emphasize that it is necessary to use pairwise BCE loss to enforce the consistency of  $f$  with  $T$ . This is because cross-entropy loss is not applicable when the cluster labels in  $T$  are not aligned with the class indices in  $S$ .
2.3 Invariant Consistency (ICON)
ICON simultaneously enforces the consistency of  $f$  with  $S$  (i.e., the decision boundary of  $f$  separates the classes) and  $T$  (i.e., it also separates the clusters). The objective is given by:
$$
\min  _ {\theta , f} \overbrace {\operatorname {B C E} (S , f) + \operatorname {B C E} (T , f)} ^ {\text {C o n s i s t e n c y}} + \overbrace {\operatorname {C E} (S , f) + \alpha \mathcal {L} _ {s t}} ^ {S - \text {S u p e r v i s i o n}}
$$
$$
\text {s . t .} \underbrace {f \in \operatorname {a r g m i n} _ {\bar {f}} \mathrm {B C E} (S , \bar {f}) \cap \operatorname {a r g m i n} _ {\bar {f}} \mathrm {B C E} (T , \bar {f})} _ {\text {I n v a r i a n c e}}. \tag {2}
$$
Line 1. The BCE losses train  $f$  to be consistent with the pair-wise label similarity in  $S$  and  $T$ . CE loss in  $S$  trains  $f$  to assign each sample its label (e.g., predicting a clock feature as "clock").  $\mathcal{L}_{st}$  is the self-training loss with weight  $\alpha$ , which leverages the learned invariant classifier to generate accurate pseudo-labels in  $T$  for additional supervision (details in Section 5.2).
Line 2. The constraint below requires the backbone  $\theta$  to elicit an invariant classifier  $f$  that simultaneously minimizes the BCE loss in  $S$  and  $T$ . This constraint prevents the case where one BCE loss is minimized at the cost of increasing the other to pursue a lower sum of the two (Figure 2c).
To avoid the challenging bi-level optimization, we use the following practical implementation:
$$
\min  _ {\theta , f} \operatorname {B C E} (S, f) + \operatorname {B C E} (T, f) + \operatorname {C E} (S, f) + \alpha \mathcal {L} _ {s t} + \beta \text {V a r} (\left\{\operatorname {B C E} (S, f), \operatorname {B C E} (T, f) \right\}), \tag {3}
$$
where the variance term  $\mathrm{Var}(\cdot)$  is known as the REx loss [25] implementing the invariance constraint. The self-training weight  $\alpha$  and invariance weight  $\beta$  are later studied in ablation (Section 5.4).
Training and Testing. Overall, at the start of training, the backbone  $\theta$  is initialized from a pre-trained weight (e.g., on ImageNet [46]) and  $f, g$  are randomly initialized. Then ICON is trained by Eq. (3) until convergence. Only the backbone  $\theta$  and the classifier  $f$  are used in testing.
Preliminaries. We adopt the group-theoretic definition of disentanglement [20, 62] to justify ICON. We start by introducing some basic concepts in group theory.  $\mathcal{G}$  is a group of semantics that generate data by group action, e.g., a "turn darker" element  $g\in \mathcal{G}$  transforms  $\mathbf{x}$  from white "0" to  $g\circ \mathbf{x}$  as grey "0" (bottom-left to top-left in Figure 4b). A sketch of theoretical analysis is given below and interested readers are encouraged to read the full proof in Appendix.
Definition (Generalization). Feature space  $\mathcal{X}$  generalizes under the direct product decomposition  $\mathcal{G} = \mathcal{G} / \mathcal{H} \times \mathcal{H}$ , if  $\mathcal{X}$  has a non-trivial  $\mathcal{G} / \mathcal{H}$ -representation, i.e., the action of each  $h \in \mathcal{H}$  corresponds to a trivial linear map in  $\mathcal{X}$  (i.e., identity map), and the action of each  $g \in \mathcal{G} / \mathcal{H}$  corresponds to a non-trivial one.
This definition formalizes the notion of transforming each sample  $\Phi (\mathbf{c},\mathbf{e})$  to the causal feature c. Subgroup  $\mathcal{H}\subset \mathcal{G}$  and quotient group  $\mathcal{G} / \mathcal{H}$  transform the environmental feature e and the causal c, respectively. If  $\mathcal{X}$  has a non-trivial  $\mathcal{G} / \mathcal{H}$ -representation, samples in a class are transformed to the same feature  $\mathbf{c}\in \mathcal{X}$  regardless of e (i.e., representation of  $\mathcal{G} / \mathcal{H}$ ), and different classes have different c (i.e., non-trivial). This definition is indeed in accordance with the common view of good features [62], i.e., they enable zero-shot generalization as c is class-unique, and they are robust to the
domain shift by discarding the environmental features  $\mathbf{e}$  from  $\Phi (\mathbf{c},\mathbf{e})$ . To achieve generalization, we need the following assumptions in ICON.
Assumption (Identificability of  $\mathcal{G} / \mathcal{H}$
1)  $\forall \mathbf{x}_i, \mathbf{x}_j \in T$ ,  $y_i = y_j$  iff  $\mathbf{x}_i \in \{h \circ \mathbf{x}_j \mid h \in \mathcal{H}\}$ ;
2) There exists no linear map  $l: \mathcal{X} \to \mathbb{R}$  such that  $l(\mathbf{x}) > l(h \circ \mathbf{x}), \forall \mathbf{x} \in S, h \circ \mathbf{x} \in T$  and  $l(g \circ \mathbf{x}) < l(gh \circ \mathbf{x}), \forall g \circ \mathbf{x} \in S, gh \circ \mathbf{x} \in T$ , where  $h \neq e \in \mathcal{H}, g \neq e \in \mathcal{G} / \mathcal{H}$ .
Assumption 1 states that classes in  $T$  are separated by clusters. This corresponds to the classic clustering assumption, a necessary assumption for learning with unlabeled data [59]. Figure 4a shows a failure case of Assumption 1, where the intra-class distance (i.e., dotted double-arrow) is much larger than the inter-class one (i.e., solid double-arrow) in  $T$ , i.e., shape  $\mathbf{c}$  is less discriminative than color  $\mathbf{e}_T$ . Hence the clusters in  $T$  are based on  $\mathbf{e}$ , causing ICON to capture the wrong invariance. To help fulfill the assumption, UDA methods (and SSL in general) commonly leverage feature pre-training and data augmentations. We also specify the number of clusters as the class numbers in  $T$ , which is a necessary condition of this assumption.
Figure 4: Red: S, Blue: T.  $\bullet$ : "0",  $\triangle$ : "1". Black lines denote the invariant classifier satisfying the ICON objective. (a) Assumption 1 is violated. (b) Assumption 2 is violated.
Assumption 2 states that  $\mathbf{c}$  is the only invariance between  $S$  and  $T$ . Figure 4b illustrates a failure case, where there exist two invariant classifiers: both the vertical one based on shape  $\mathbf{c}$  and the horizontal one based on color  $\mathbf{e}$  separate the classes in  $S$  and clusters in  $T$ . Yet as  $T$  is unlabelled, there is no way for the model to determine which one is based on  $\mathbf{c}$ . In practice, this assumption can be met by collecting more diverse unlabelled samples in  $T$  (e.g., darker "0"s and lighter "1"s in Figure 4b), or introducing additional priors on what features should correspond to the causal one.
Theorem. When the above assumptions hold, ICON (optimizing Eq. (3)) learns a backbone  $\theta$  mapping to a feature space  $\mathcal{X}$  that generalizes under  $\mathcal{G} / \mathcal{H} \times \mathcal{H}$ . In particular, the learned  $f$  is the optimal classifier in  $S$  and  $T$ .
By transforming each sample  $\Phi (\mathbf{c},\mathbf{e})$  in  $S$  and  $T$  to  $\mathbf{c}\in \mathcal{X}$ , the classifier easily associate each  $\mathbf{c}$  to its corresponding class label using the labelled  $S$  to reach optimal.
Domain-Alignment methods [13, 4, 57] map samples in  $S$  and  $T$  to a common feature space where they become indistinguishable. Existing works either minimize the discrepancy between the sample distributions of  $S$  and  $T$  [36, 2, 27], or learn features that deceive a domain discriminator [69, 63, 33]. However, they suffers from the misalignment issue under large domain shift from  $S$  to  $T$  [71, 31], such as shift in the support of the sample distribution [23] (e.g., Figure 2a).
Self-Training methods [17, 3, 53] are the mainstream in semi-supervised learning (SSL), and recently became a promising alternative in UDA, which focuses on generating accurate  $T$  pseudo-labels. This lineup explores different classifier design, such as k-NN [51], a teacher-student network [11] or an exponential moving average model [30]. Yet the pseudo-labels are error-prune due to the spurious correlations in  $S$ , hence the model performance is still inconsistent [32].
Alleviating Spurious Correlation has been studied in several existing works of UDA [56, 32, 29]. They train a  $T$ -specific classifier to generate pseudo-labels [56, 29]. However, their  $T$ -classifiers are still initialized based on a classifier trained in  $S$ , hence the spurious correlation in  $S$  are introduced to the  $T$ -classifiers right in the beginning. On the contrary, our ICON learns a cluster head in  $T$  without supervision in  $S$ , which is solely for discovering the sample clusters in  $T$ . Hence, the spurious correlation in  $S$  inconsistent with the clusters in  $T$  can be removed by ICON (Section 5).
Graph-based SSL methods [16, 42, 22] share a similar spirit with ICON, where they learn a classifier using the labeled samples that is consistent with the unlabelled sample similarities. However, they lack the invariant constraint in ICON, which can cause the failure cases (e.g., Figure 2c).
Dataset	OFFICE-HOME	VisDA-2017	iWILDCAM	CAMELYON17	FMOW	POVERTYMAP	GLOBALWHEAT	OGB-MOLPCBA	CIVILCOMMENTS	AMAZON
Sample x	object image	object image	camera trap photo	tissue slide	satellite image	satellite image	wheat image	molecular graph	online comment	product review
Label y	65 categories	12 categories	182 species	tumor/not	62 land uses	asset wealth	wheat bbox	bioassays	toxic/not	5 review scores
Task	classification	classification	classification	classification	classification	regression	detection	classification	classification	classification
Source S	various*	synthetic images	photos from 243 traps	slides from hospital 1-3	images from 2002-2013	images in 14 countries	images in Europe	44,930 scaffold groups	online articles*	1,252 reviewers
Example S	Super easy to put together. Very well built.
#Samples S	average 3,875	152,397	129,809	302,436	76,863	~10,000	2,943	350,343	269,038	245,502
Target T	various*	real photos	photos from 3215 traps	slides from hospital 5	images from 2016-2018	images in 5 countries	images across the world	43,793 scaffold groups	online articles*	1,334 reviewers
Example T	As a Christian, I will not be patronizing any of those businesses.	I am disappointed in the quality of these.
#Samples T	average 3,875	55,388	819,120	600,030	173,208	261,396	42,445	517,048	1,551,515	268,761
Evaluation	average acc.	mean-class accuracy	macro-F1	acc.	worst-region acc.*	Pearson correlation* %	acc.	average precision	worst-group acc.*	10th percentile acc.
Existing Methods	GVB [7]	Empirical Risk Minimization (ERM)
70.4	75.3	47.0 / 32.2	90.6 / 82.0	60.6 / 34.8	65 / 48	77.8 / 51.0	- / 28.3	89.8 / 66.6	72.0 / 54.2
TCM [61]	CORAL [51]
70.7	75.8	40.5 / 27.9	90.4 / 77.9	58.9 / 34.1	54 / 36	-/-	- / 26.6	-/-	71.7 / 53.3
SENTRY [41]	DANN [13]
72.0	76.7	48.5 / 31.9	86.9 / 68.4	57.9 / 34.6	50 / 33	-/-	- / 20.4	-/-	71.7 / 53.3
CST [29]	Pseudo-Label [24]
72.2	80.6	47.3 / 30.3	91.3 / 67.7	60.9 / 33.7	-/-	73.3 / 42.9	- / 19.7	90.3 / 66.9	71.6 / 52.3
ToAlign [59]	MDD [63]	Noisy Student [60]
72.7	77.8	47.5 / 32.1	93.2 / 86.7	61.3 / 37.8	61 / 42	78.1 / 46.8	- / 27.5	-/-	-/-
FixBI [37]	MT+16augs [11]	FixMatch [50]	Masked LM [9]
73.0	82.8	46.3 / 31.0	91.3 / 71.0	58.6 / 32.1	54 / 30	-/-	-/-	89.4 / 65.7	71.9 / 53.9
ATDOC [26]	MCC+NWD [6]	SwAV [5]	ERM (labelled T)
73.2	83.7	47.3 / 29.0	92.3 / 91.4	61.8 / 36.3	60 / 45	-/-	-/-	89.9 / 69.4	73.6 / 56.4
ICON	75.8	87.4	50.6 / 34.5	95.6 / 93.8	62.2 / 39.9	65 / 49	78.6 / 52.3	- / 28.3	89.7 / 68.8	71.9 / 54.7
+2.6	+3.7	+2.3	+2.4	+2.1	+1	+1.3	+0.0	+1.9	+0.5