SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To further testify the performance of FlatMatch on a large-scale datasets, we first conduct experiments on ImageNet30 dataset which is a subset from the original ImageNet dataset and contains 30000 training examples with resolution  $256 \times 256$  from 30 classes. Moreover, we also test the performance on the original ImageNet dataset. As Vision Transformer (ViT) [21] has manifested great power on classification tasks, we also adopt ViT as our backbone to validate the performance of FlatMatch.
The experiments on ImageNet30 are more time-consuming which normally takes 5 days to finish, much more than CIFAR10 dataset which takes 2 days. We vary the number of labeled data as 1500 and 3000 and show the comparison in Table 6. We observe the effectiveness of FlatMatch over all other baseline methods in both two settings, which again validates the superiority of our method and its effective performance on large-scale datasets.
Additionally, we have conducted the experiments on the original ImageNet dataset by choosing only 100 labels per each class, and provide the test error results of FlatMatch, FlatMatch-e, FixMatch,
Table 6: Comparison on ImageNet30.
Dataset	ImageNet30
# label	1500	3000
FixMatch	12.48±0.67	8.25±0.54
Dash	13.29±1.26	8.79±0.42
FlexMatch	11.48±0.52	8.04±0.75
SoftMatch	10.81±0.40	7.78±0.61
FreeMatch	10.34±0.46	7.21±0.19
FlatMatch	9.71±1.55	6.77±1.27
Table 7: Comparison on ImageNet using Wide-ResNet-28-2 and Vision Transformer.
Dataset	Architecture	FlatMatch	FlatMatch-e	FreeMatch	FlexMatch	FixMatch
ImageNet	Wide-ResNet-28-2	38.70	39.92	40.57	41.95	43.66
ViT-Base (86M)	21.57	22.07	23.55	23.78	25.52
FlexMatch, and FreeMatch as shown in Table 7. We can see that on large-scale dataset such as ImageNet, the performances of both FlatMatch and FlatMatch-e are still superior to other components. Moreover, we can still observe the effectiveness of the two of our methods on ViT. Therefore, it is reasonable to conclude that the performance of FlatMatch is extendable to large-scale datasets and sophisticated architectures.
Figure 8: Comparison of sharpness between various SSL methods during training.
To further evaluate the flatness of different SSL models during training, we leverage a validation set to compute the sharpness. The sharpness is measured by the increase of loss within a  $\ell_2$  bounded neighbor, which is formally defined as Sharpness :=  $\mathcal{L}(\theta + \epsilon^*(\theta)) - \mathcal{L}(\theta)$ , where  $\epsilon^*(\theta) = \arg \max_{\|\epsilon\|_2 \leq \rho} \mathcal{L}(\theta + \epsilon)$ . Specifically, we compare the proposed FlatMatch with FixMatch, Dash, and FreeMatch, and use fully supervised learning as a baseline method. The experiments are conducted on CIFAR10 and SVHN datasets whose results are shown in Figure 8. First, we observe that FlatMatch achieves the lowest sharpness curve during training on both two datasets, which indicates the SSL model learned by FlatMatch is more robust to perturbations and would not oscillate significantly when facing changes in parameter space. Moreover, we find that fully supervised learning does not improve the flatness as the training proceeds, while all SSL methods can decrease the sharpness to some extent, which demonstrates that training with unlabeled data can help improve the flatness of SSL models.
Figure 9: Analysis on changing the number of fixed labels.
Furthermore, as shown in the main paper, we find that FlatMatch has limited performance on extremely scare labeled settings. However, this limitation can be addressed by introducing some unlabeled data with fixed labels to improve the computation of cross-sharpness. Hence, here we investigate the effect of changing the number of fixed on the performance of FlatMatch. Specifically, we conduct experiments on CIFAR10 and SVHN datasets and fixing different numbers of labels as 0 ("w/o fix label"), 250, 500, 1000, 2000,  $4000^4$ . The results are shown in Figure 9. We find that both too few fixed labels, i.e., 250 labels and too many fixed labels, i.e., 4000 labels in CIFAR10 and 2000 labels in SVHN, would show a performance drop compared to the optimal number, 500 fixed labels. This is because if the number of fixed labels is too small, the gradient computation would be inaccurate, further limiting the learning results. On the other hand, too many fixed labels would introduce noisy labeled unlabeled data, which would largely mislead the SSL and show serious performance degradation.
Figure 10: Convergence of accuracy analysis between FlatMatch and FreeMatch.
D Convergence Analysis
In this section, we conduct analyses regarding the test accuracy and training loss to validate the convergence property of FlatMatch.
For convergence of accuracy, here we compare our FlatMatch with FreeMatch, which has the best performance among most SSL baseline methods. The accuracy curve is shown in Figure 10. We
4The 4000 fixed labels setting is not conducted on SVHN as the performance of 2000 fixed labels setting already shows significantly performance degradation.
observe that the performances of two methods are almost comparable in early stages, but FlatMatch continues to improve the training performance in the middle and later stages and finally achieves better accuracy than FreeMatch on the final point. Therefore, we can conclude that our method can converge to a superior performance than FreeMatch.
Figure 11: Convergence of loss values of FlatMatch and FixMatch.
Moreover, to illustrate the convergence of loss curves, we show the loss values of labeled data and unlabeled data from both FlatMatch and FixMatch during training. Moreover, we subtract the loss value of unlabeled data with loss of labeled data to compute a loss difference, which gives an illustration about the performance gap between both two datasets.
Loss curves of FixMatch
Loss curves of FlatMatch
Figure 12: Loss landscapes of labeled data and unlabeled data obtained simultaneously from training using FlatMatch and FixMatch on CIFAR10 with 250 labels per class. The results are generated from the last model checkpoint. The first column and second column show the 2D loss contours of labeled data and unlabeled data, respectively, and the last column shows the 1D loss curves.
There are two findings: 1) The loss value of labeled data quickly converges to zero and is significantly smaller than unlabeled data. Such phenomenon occurs in two methods which supports our claim that the learning on labeled data converges faster than unlabeled data. 2) The loss difference between two datasets of FlatMatch is significantly smaller than FixMatch, which indicates that our FlatMatch can alleviate the unmatched convergence speed of two datasets and helps decrease the loss gap between two datasets.
E More Visualizations
To show how the loss curves appear in the later stage of training, we generate the loss landscape of FixMatch and FlatMatch on the last training epoch  $(2^{20})$ . We can see that FlatMatch generates a wider loss landscape than FixMatch. Moreover, the loss curve of labeled data from FlatMatch is smoother than that of FixMatch. Therefore, we can again conclude that FlatMatch can benefit the generalization result.
F Summary and Future Work
In this paper, we propose a novel FlatMatch approach that minimizes the cross-sharpsness measure to improve the generalization performance of SSL. Through extensive quantitative and qualitative experiments, we have thoroughly evaluated the performance of FlatMatch and demonstrated its superiority to other compared methods. Thanks to the generalization improvement of FlatMatch, the classification accuracy on many scenarios have even passed the fully-supervised baseline.
However, the learning performance of SSL still largely depends on the careful selection of labeled data. Specifically, in the barely-supervised learning scenario, if the selected scarce labeled data deviate from the cluster center, the learning performance of many existing SSL methods would be significantly affected. This is due to the generalization performance between labeled data and unlabeled data being largely mismatched. Under this scenario, the performance of FlatMatch should be further evaluated.