FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning
Authors: Zhuo Huang
Li Shen
Jun Yu
Bo Han
Tongliang Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Semi-Supervised Learning
TL;DR: We connects the labled data and unlabeled data through the parameter space to enhance consistency training.

Abstract: Semi-Supervised Learning (SSL) has been an effective way to leverage abundant unlabeled data with extremely scarce labeled data. However, most SSL methods are commonly based on instance-wise consistency between different data transformations. Therefore, the label guidance on labeled data is hard to be propagated to unlabeled data. Consequently, the learning process on labeled data is much faster than on unlabeled data which is likely to fall into a local minima that does not favor unlabeled data, leading to sub-optimal generalization performance. In this paper, we propose FlatMatch which minimizes a cross-sharpness measure to ensure consistent learning performance between the two datasets. Specifically, we increase the empirical risk on labeled data to obtain a worst-case model which is a failure case needing to be enhanced. Then, by leveraging the richness of unlabeled data, we penalize the prediction difference (i.e., cross-sharpness) between the worst-case model and the original model so that the learning direction is beneficial to generalization on unlabeled data. Therefore, we can calibrate the learning process without being limited to insufficient label information. As a result, the mismatched learning performance can be mitigated, further enabling the effective exploitation of unlabeled data and improving SSL performance. Through comprehensive validation, we show FlatMatch achieves state-of-the-art results in many SSL settings.

FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning
Zhuo Huang', Li Shen', Jun Yu', Bo Han", Tongliang Liu
SSydney Centre, The University of Sydney; ²JD Explore Academy; Department of Automation, University of Science and Technology of China; Department of Computer Science, Hong Kong Baplist University http:/uuiiinynnddnnn sydney. edu, au, mathshenli@gmaii com, jarryyun@ustc.om edw.. bhanml bhanml@comp hkbu. edu. hk, tongliang liu@sydney. edu. au
zhua0420@uni.
Abstract
Semi-Supervised Learning (SSL) has been an effective way to leverage abundant unlabeled data with extremely scarce labeled data. However, most SSL methods are commonly based on instance-wise consistency between different data transfor- mations. Therefore, the label guidance on labeled data is hard be propagated unlabeled data. Consequently, the learning process on labeled data is much faster than on unlabeled data which is likely to fall into local minima that does not favor unlabeled data, leading to sub-optimal generalization performance. In this paper, we propose FlatMatch which minimizes cross-sharpness measure ensure consistent learning performance between the two datasets. Specifically, we increase the empirical risk on labeled data obtain a worst-case model which is a failure case that needs to be enhanced. Then, by leveraging the richness of unlabeled data, we penalize the prediction difference (i.e., cross-sharpness) between the worst-case model and the original model so that the learning direction is beneficial to gen- eralization on unlabeled data. Therefore, we can calibrate the learning process without being limited to insufficient label information. As result, the mismatched learning performance can be mitigated, further enabling the effective exploitation of unlabeled data and improving SSL performance. Through comprehensive vali- dation, we show FlatMatch achieves state-of-the-art results in many SSL settings. Our code is available at https//giihub.com//lllabb// _ FlatMatch.
Introduction
Training deep models [31, 45] not only requires countless data but also hungers for label supervision which commonly consumes huge human laboring and unaffordable monetary costs. To case the dependency on labeled data, semi-supervised learning (SSL) [6, 12, 27] has been one of the most effective strategies exploit abundant unlabeled data by leveraging scarce labeled data. Traditional SSL commonly leverages unlabeled data by analyzing their manifold information. For example, semi-supervised support vector machine [6. 13] finds classifier that crosses the low-density region based on large-margin theorem and label-propagation [60, 84) computes an affinity matrix in the input space to propagate the labels their neighbor unlabeled data. Due to the computational burden of exploring manifold knowledge, recent advances in SSL benefit from sophisticated data augmentation techniques [4, 19, 20. 74, 80], they usually enforce predictive consistency between the original inputs and their augmented variants [75, 8, 7, 57, 66. 71, 76, 79], meanwhile using pseudo labels [46, 52, 53, 2] to guide the learning on unlabeled data.
*Corresponding to Tongliang Liu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2D contours of labeled data
2D contours of unlabeled data
ID loss curves
1.00
15
Accuracy
LOGS
e>
20
-55
040
CN
123
12-0 25 75--500--555
0.00
0.25
0:50
0.75
1.50
6.0
16
Accuracy
Lass
20
Figure 1:
Loss landscapes of labeled data and unlabeled data obtained simultaneously from training using FixMatch [57] on CIFARIO with 250 labels per class. The first row and second row show the results obtained from epoch 60 and epoch 150, respectively. The first column and second column show the 2D loss contours of labeled data and unlabeled data, respectively, and the last column shows the ID loss curves.
However, as most of the existing methods only apply instance-wise consistency on each example with its own transformation, labeled data and unlabeled data are disconnected during training. Hence, the label guidance cannot be sufficiently propagated from the labeled set to the unlabeled set, causing the learning process labeled data is much faster than unlabeled data. As result, the learning model could be misled to a local minima that cannot favor unlabeled data, leading to non-negligible generalization mismatch. As generalization performance is closely related to the flatness of loss landscape [14, 24, 41, 51], we plot the loss landscapes of labeled data and unlabeled data using FixMatch [57] in Fig. 12. We have two major findings: 1) The loss landscape of labeled data is extremely sharp, but the loss curve of unlabeled data is quite flat. 2) Moreover, the optimal loss value of labeled data is much lower than that of unlabeled data. Such two intriguing discoveries reveal the essence of SSL: The learning on scarce labeled data convergences faster with lower errors than on unlabeled data, but it is vulnerable perturbations and has unstable loss curvature which rapidly increases as parameters slightly vary. Therefore, the abundant unlabeled data are leveraged so that SSL models are fitted to wider space, thus producing a flatter loss landscape and generalizing better than labeled data. Based on the analysis, existing methods that use instance-wist consistency with mismatched generalization performance could have two non-trivial pitfalls:
The label supervision might not be sufficiently leveraged to guide the learning process of unlabeled data.
The richness brought by unlabeled data remains to be fully exploited to enhance the general- ization performance of labeled data.
As seeking the connection in input space is quite prohibitive, we resort to exploring the parameter space and propose FlatMatch which targets to encourage consistent learning performance between labeled data and unlabeled data, such that SSL models can obtain strong generalization ability without being limited by insufficient label information. Specifically, as the loss landscape on labeled data is quite unstable, we aim to mitigate this problem through applying an adversarial perturbation [24, 43, 69, 82, 83] to the model parameters that the worst failure case can be found. Further, we enforce the worst-case model and the original model to achieve an agreement on unlabeled data via computing their prediction differences which are dubbed pross-sharpness. By minimizing such a cross-sharpness measure, the richness of unlabeled data can be fully utilized to calibrate the learning direction. In turn, the label information can be activated for producing accurate pseudo labels, thus successfully bridging the two datasets to enable improved SSL performance.
2 NNote that the loss curves cannot be directly generated using the raw data from training. because networks can fit all examples perfectly without showing any generalization evidence. Therefore. we load the data for plotting using slight data augmentation, f.e., Random Rotation with 90", which is very commonly used in SSL.

---

Furthermore, the proposed FlatMatch is easy to implement and can be adapted to complement many popular SSL methods [10. 15, 16, 28]. Although computing the proposed tross-sharpness normally requires an additional back-propagation step, we propose an efficient strategy so that our FlatMatch can be processed without extra computational burden. Through extensive experiments and analyses on various benchmark datasets, we show that our FlatMatch achieves huge performance improvement compared to some of the most powerful approaches. Specifically, on CIFARIOO dataset with 2500 labeled examples, our FlatMatch surpasses the foremost competitor so far with 1.1 1119 % improvement. In general, our contribution can be summarized into three points:
We identify a generalization mismatch of SSL due to the disconnection between labeled data and unlabeled data, which leads to two critical flaws that remain be solved (Section 1).
We propose FlatMatch which addresses these problems by penalizing novel cross-sharpness that helps improve the generalization performance of SSL (Section 4.2).
We reduce the computational cost of FlatMatch by designing an efficient implementation (Section 4.3).
Extensive experiments are conducted to fully validate the performance of FlatMatch, which achieves state-of-the-art results in several scenarios (Section 5).
2
Related Works
SSL has been developed rapidly since the booming trend of deep learning [31, 45, 25]. We roughly introduce it through three stages, namely traditional SSL, data-augmented SSL, and open-set SSL.
Traditional SSL: One of the first SSL methods is Pseudo Label [46] which has been very effective for training neural networks in a semi-supervised manner, which relies on the gradually improved learning performance select confident samples [73, 72]. Another classical method is Co-training [9] which utilizes two separate networks to enforce consistent predictions on unlabeled data so that the classification can be more accurate. Inspired by co-training, many SSL approaches are motivated to conduct consistency regularization. Temporal ensembling [44] proposes to leverage historical model predictions to ensemble an accurate learning target that can be used to guide the learning of the current model. Mean Teacher [59] further suggests such an ensemble can be operated on the model weights to produce a teacher model that can provide improved learning targets. Instead of improving the teacher model, Noisy student [75] finds that adding noise to the student model can also improve SSL.
Data-Augmented SSL: Thanks to the development of data augmentation techniques and self- supervised learning [30, 34, 68, 67], further improvements SSL are achieved by enforeing consis- tency between the original data and their augmented variants. VAT [49] proposes to add adversarial perturbations [25, 26] to unlabeled data. By using the pseudo-label guidance to leverage such ad- versarial unlabeled data, the model can be more robust to strong input noises, thus showing better performance on the test set. MixMatch [8] employs the MixUp [80] technique to interpolate between the randomly shuffled examples which largely smooths the input space as well as the label space, further enhancing the generalization result. Moreover, FixMatch [57] is based on strong augmentation such as Auto.ugment [19] and RandAugment [20] to utilize the predictions of weakly augmented data to guide the learning of strongly augmented data, which can achieve near supervised learning performance on CIFAR10. 10. Recently, Dash [76], FlexMatch [79], and FreeMatch [66] advanced SSL through investigating the threshold strategy for pscudo labels. They analyze the training dynamic of SSL and design various approaches to enhance the selection of unlabeled data, which effectively boosts the SSL performance to a new level.
Open-Set SSL: Moreover, another branch of SSL studies the realistic scenario where unlabeled data
contains duito--distribution (OOD) data [37, 38, 32, 33, 63, 64, 62]. The common goal to alleviate the influence of OOD such that the learning process of SSL will not be misled. Several studies propose many techniques that can achieve this goal, such as distillation [17], meta-learning [29], large-margin regularization [10], content-style disentanglement [36), and consistency regularization can also help [56]. But in this paper, we mainly focus on the common assumption that labeled data and unlabeled are sampled independently and identically.
3

---

3 Preliminary: Improving Generalization via Penalizing Sharpness
Sharpness-Aware Minimization (SAM) [24, 43, 47, 69, 81] which focuses on optimizing the sharp points from parameter space that the training model can produce a flat loss landscape. Specifically, given a set of fully-labeled data D' = {(r), v))= containing n data points (at,, y²), SAM seeks to optimize a model parameterized by 0 E @ fit the training dataset D' so that 0 can generalize well test set D drawn independently and identically as D'. Such optimization process is normally conducted via minimizing the empirical risk (i = Σi= fee (0; go yt) which is realized by the cross-entropy loss tee computed between the label prediction s = go (x.) and the class label y/. To find the sharp points that maximally increase the empirical risk, SAM applies a weight perturbation O to the model parameters O and conducts the following inner maximization:
(0) = arg max (0 + c) arg arg mrgmaxx T VoLi(0) P= VoLi(0) (eCC((0)12
(I)
Sp
1el,
where denotes the optimal perturbation which can be estimated using the gradient information of mini-batch inputs, p restricts the perturbation magnitude of c within a & bp-ball, and the approximation given by the dual norm problem [24].
By perturbing the original model 9 with c², we can obtain the sharpness measure from SAM: sharpmess == Li(0+ model perturbation e". To combine the sharpness term with empirical risk minimization (ERM), the SAM objective can be differentiated as follows:
L() which indicates the how quickly loss changes under the worst-case
V 1 SAM
Ve [C + E0) -
~ VoCi(0+<'(0)
d(0 (0) VoLi(O))tt>(0) = Vo VoLi
+ o(0),
de
where the last second-order term o(0) can be discarded without significantly influencing the ap- proximation. For detailed derivation and analysis, we refer to the original paper and related stud- ies [1, 24, 82]. Intuitively, the gradient of SAM is computed on the worst-case point + e", then such a gradient is utilized to update the original parameter f. As a result, the model can produce more flat loss landscape than ERM which would not change drastically, and being robust along the sharpest direction e" among its bp norro neighbor.
Since the obtained flatness property has been demonstrated to have many benefits for generalization, such as heing resistant to distribution shift [11, 39, 61]. robustness against adversarial attack [69, 58), and effectiveness under label noise 3, 40, 70, 77], SAM has inspired many research progresses. However, SAM is only conducted under a fully-supervised setting, and whether or how can it be leveraged improve learning with unlabeled data is still undiscovered. Next, we carefully introduce the proposed FlatMatch which improves the generalization performance of SSL by bridging labeled data and unlabeled data with a novel srass-sharpsess.
4 FlatMatch: Semi-Supervised Learning with Cross-Sharpnesss
In this section, we carefully introduce FlatMatch. First, we describe the SSL setting in generalized way. Then, we demonstrate a novel regularization from FlatMatch which is dubbed cross-sharpness. Finally, explain its optimization and design an efficient implementation.
4.1 General Semi-Supervised Learning
In SSL, we commonly given small set of labeled data D' = (1//, 1/ containing labeled examples (z/, u/) and a large unlabeled data D = {x |m containing m (m >> n) unlabeled examples xi which are drawn independently and identically. Similar to previous notations, we aim to optimize a deep model 0 E θ in semi-supervised manner so that 0 can perform well on i.i.d sampled test set Die_ The general objective for SSL is as follows:
T2
m
min Lest = min Li + C.
min 8
lae (o go (o))))
T(u > T)da((; go (A(x))
72
m
(3)

---

x
11
mat
x
cross-sharpness
Figure 2: Illustration of FlatMatch. The blue arrows and red arrows denote the learning flows related to labeled data and unlabcled data, respectively; and the black arrows indicate the computation of the worst-case model.
where the first term is similar to Eq. 1 and denotes the empirical risk under the label supervision, the second term indicates the semi-supervised regularization for exploiting unlabeled data, the function (a(-) denotes the loss criterion for unlabeled data, such as KI KI--iiergence or cross-entropy loss, A(-) stands for an augmentation function that transforms the original input æ; to its augmented variants, Vi represents the unsupervised learning target which can be a pseudo label or a guiding signal from teacher model, and an indexing function II()) is commonly used to select the confident unlabeled data if their learning target surpasses predefined threshold r. Our FlatMatch replaces the second term with cross-sharpness regularization as described below.
4.2
The Cross-Sharpness Regularization
The computation of cross-sharpness can be briefly illustrated in Fig. 2. Particularly, to solve the sharp loss landscape problem, as demonstrated in Section I, we penalize the shossess regularization on each unlabeled example using a worst-case model derived from maximizing the loss on labeled data. Formally, such optimization shown as follows:
min Rx-sharp = min 1 6 m7
Σ E"E
(( where Ō 0=6 0 + arg max Lis Sp
in which (-(-) indicates the forward propagation using the parameter 0 whose prediction result is f (-).), and 9 stands for the worst-case model that maximally increases the empirical risk on labeled data. Here we drop the subscript from each datum for simplicity. By meaning "cross", the sharpness is not obtained by revisiting labeled data as SAM does, instead, we compute the sharpness by leveraging abundant unlabeled data to take full advantage of their generalization potential.
In detail, we leverage the original model to conduct a first propagation step as f' go ge());; (o(x", where f and f are label predictions from labeled data and unlabeled data, respectively.
Then, the empirical risk on labeled data is computed Li E (o: go (a'), y')) Following
Eq. 1, we maximize the empirical risk Li to obtain the weight perturbation c" := P (0)1l
V.C,(0)
further having the worst-case model 0 + c" Moreover, we conduct second propagation step by passing the unlabeled data to 0 and have f ga ()."). The second unlabeled data prediction f'r is combined with the first prediction f obtained before to compute the cross cooss--sappness in Eq. 43.
Why does crossing the sharpness work? As shown by many theoretical signs of progress, the generalization error of SSL is largely dependent on the number of labels [5, 48). Especially in recently proposed barely-supervised learning [57], the generalization error would be enormous. Although such a result may be pessimistic, unlabeled data can still be helpful. It is shown that when two distinct hypotheses on labeled data are co-regularized achieve an agreement on unlabeled data, the generalization eтror can be reduced, as we quote: "The reduction is proportional to the difference between the representations of the labeled data in the two different views, where the measure of difference is determined by the unlabeled data" [54]. Intuitively. disconnecting labeled data and unlabeled data during training may be sub-optimal. To achieve better theoretical performance, the SSL model should focus on learning something from unlabeled data that are not contained by labeled
NNoee that there is slight difference between the sharpness from SAM and our sharssshrppsss:: The former one measured by the change of cross-entropy loss under model perturbation. But in SSL, the pseudo labels of unlabeled data is not accurate enough compared to labeled data, so we use difference between label predictions of 0 and 0 to measure cross-sharpness.
5

---

Algorithm FlatMatch and FlatMatch-e
Input: Labeled dataset D', unlabeled datasct D model 9, memory buffer M for storing historical gradients, and EMA factor a for updating buffer.
I: for E 0, ,,...,? 7 do
2:
if Efficient training then
FlatMatch-e
3:
Compute optimal perturbation as c" = PIMM""la*
4:
else
FlatMatch > First propagation step
5:
Compute gradient as V VaLi(0) - Vo Ex'epi las (o Falilo)
6:
Compute optimal perturbation as P P
7:
Obtain worst-case model as 8 = +";;;
8:
Compute cross-sharpness via Eq. 4;
9:
Optimizing 0 via Eq. 5, meanwhile save the gradient VoLi(0):
Second propagation step
10:
Update memory buffer M (1 a) x M' VoCi());
data. In our FlatMatch, we can achieve this goal by enforcing @ and 0, which produces the maximal loss difference on labeled data, be consistent in classifying unlabeled data. Therefore, FlatMatch can be more effective than other instance-wise consistent methods by bridging the knowledge of labeled data and unlabeled data for training.
4.3 Optimization and Efficient Implementation
To this end, we solve the optimization of FlatMatch by substituting Lu in Eq. with Rx-shorp in Eq.
4:
= arg min (i) shupp (0 c² c²)) VoLi Rxxxx (
Rabier 0 AA
(5)
Particularly, the second gradient obtained on the worst-case point 0 1 then it is applied on the original model 0 for gradient descent. As back-propagation on labeled data, which doubles the computational complexity. Fortunately, our FlatMatch can be implemented efficiently with extra computational burden than the baseline stochastic gradient descent (SGD) optimizer, which is dubbed FlatMatch-e. The implementation is summarized in Algorithm 1.
V.C,(0)
see, such optimization requires additional
Although many efficient methods for implementing SAM have already been proposed [22, 23, 47, 83], they commonly need extra regularization or gradient decomposition, which are not harmonious to SSI. and would complicate our method. our scenario, FlatMatch has critical difference from SAM regarding the two propagation steps: The gradients from the first step and second step not computed on the same batch of data, i.e., they are crossed from labeled data to unlabeled data. Since in SSL, each labeled batch and unlabeled batch are randomly coupled, gradient computed from the last batch labeled data to obtain the cross-sharpness from the current batch of unlabeled data. Even better, to stabilize the gradient so that our cross-sharpness can be computed accurately. Next, we conduct extensive experiments to carefully validate approach.
allowed to the
use exponential moving average (EMA) [30, 44, 59]
5 Experiments
In this section, we conduct extensive comparisons and analyses to evaluate the proposed FlatMatch method. We first describe the experimental setup and implementation details. Then we compare FlatMatch with many edge-cutting SSL approaches to show the effectiveness of our method. Further, we conduct an ablation study to justify our design of FlatMatch. Finally, we demonstrate the efficiency, stability, and sensitivity of FlatMatch through various analytical studies.
5.1 Experimental Setup and Details
We follow the most common semi-supervised image classification protocols by using CI-
FAR 10/100 [42] SVHN [50], and STL10 [18] datasets where a various number of labeled data are equally sampled from each class. Following Wang et [66], We choose Wide ResNet-28-2 [78]
for CIFARIO and SVHN, Wide ResNet-28-8 for CIFAR ResNet-37-2 [31] for STL10. All

---

Table 1: Error rates on CIFARIO/100, SVHN, and STL10 datasets. The fully suly-suue results of STL10 are unavailable since we do not have label information for its unlabeled data. The best results are highlighted with Bold and the second-best results highlighted with underline.
CHARRI
CHEARIOO
10
250
2500
250
1000
Ti-Madel (44)
79.
11
14. S4
46 Me:
56. So
36 6
67.48
13 1.
3.16
14 a4.
11.78ee
Pieudo Label
90.21 11:55
74.61.02
46.491122
15.00
$7.45 1a35
57.74183
36.55 1024
64.01
15.59
9,401e32
74.68
52.641671
VAT (49)
79.81 13:17
74 66666.12
41.05
10.51111
35.2011.40
15.54187
32. 14.0.0
74.75123
4.33
74.74 4.11:020
57.55 37.55111
Moin Teaaber [59]
76.37 44.4
70.0
57. 57.46440
10
$1.11
45.17amm
56.09
3.5
3.27
71.7244.0
13.00am
MisManch 1S]
65.76. mm
36. 19ase
13,63am
5.66
67: a7.5
39.7644:0
27.70 00.30
50
4.56a02
3.60
54.93
21. 70.00
ReMisMacch 171
20.77
9.38.u
6.30uemd
4.84
42.75
26.03
20.02
1027
24.04
6.36
5.16
12.12
6.7400
UDA (14)
10632.m
4.20
46.19a1
27:752mm
5.17
188
17.42ax44
6.
FisMarch (57)
7.47am
4.96amm
4.21
46 46-4
28.002014
3.81e1
202an
15.07a311
6.5
(78)
27 19:29.11.39
893131
5.16182
4.30.001
44:82
27.15
2.19
1.57.091
34.521438
6,39
MPL 152]
23.55.601
6.02
5.761824
46.26.1.04
27:711815
21.04:00
9
2,28
35:761450
[20]
13.85a123
8.19
6.591229
6.72800
SofMaank
4.52
4.04400
17.10
26.664325
22.03.0m:
21.42
5,73
FreeMarch 1867
4,07.424
4.10uo02
1756
26.47
21.65.08:
1997
197
1.96100
15.56aa55
5.lana
FlatManchh
15.23
5.58
4,22
18 18/76a1.0
25,38s
g0.0
1 3.46
148gmm
1.41
16:00sr
482g11
FlaMaasbb
15,89
5.63110
4.53
3,57
25.62 48.9841.9
19.78
2.66a02
147
1.46
16.32
PlarMinch (n kont
7.
4.891174
3,90 17
36,97
25. c
19.01.n
2.14
1.4gpm
1.41
14.6
4822111
Fully-Sapervial
4.62
19.30
2.13
SSL approaches are implemented using Pytorch framework, and the computation is based on 12GB
NVIDIA 3090 GPU. We compare our method with well-known approaches, including 1I-model [44], Pseudo Label [46], VAT [49]. Mean Teacher [59], MixMatch [8], ReMixMatch [7]. UDA [74],
FixMatch [57], Dash [76], MPL [52], FlexMatch [79], SoftMatch [16], and FreeMatch [66].
To optimize all methods, we use SGD with a momentum of 0.9 with an initial learning rate of 0.03. We set the batch size as 64 for all datasets. Moreover, we set the weight decay value, pseudo label threshold т. unlabeled batch ratio μa, and trade-off for SSL regularization as the same for all compared methods. To implement our FlatMatch, we set the perturbation magnitude p = 0.1, and EMA factor a = 0.999 which is the same for Mean Teacher and other methods that require EMA. Moreover, in line 9 from Alg. 1, we need to separate the gradients regardin labeled data and unlabeled data in one back-propagatee "functorch. make_ make_functii aareioonllllliteelee cacalllwtt__uuuffeee buffers" " in Pytorch, which can efficiently compute the gradients for each sample. For more details, please see the appendix.
regarding step. Such an operation can be done using
5.2 Quantitative Comparison
The main comparison results are shown in Table 1, we can see that our FlatMatch achieves state-of- the-art results on many scenarios. For example, on CIFARI00 with 2500 labels, FlatMatch achieves 25.38% test errors, surpassing the second-best method, FreeMatch, for 1. 1.09%; and on CIFARIOS with 10000 labels, FlatMatch achieves 19.01% performance, further improving the second-best result, ReMixMatch, for 1.01 %, which even passes the fully-supervised baseline result for 0.29%. Moreover, FlatMatch reaches extremely low error rates in many settings. For instance, in CIFAR10 100 with 250 labels and 4000 labels, FlatMatch only has 4.22% and 3.61 % error rates; and in SVHN with 250 labels and 1000 labels, FlatMatch can also surpass the sully-supervised learning by producing 1.43% and 1.41% error rates, respectively. Additionally, FlatMatch breaks the record on the STL dataset with 1000 labels by reaching 4.82% performance.
Moreover, we also show the performance of FlatMatch-e, an efficient variant of FlatMatch, which can also surpass many existing sfute-o--th--ase methods. Such as in CIFAR 100 with 10000 labels, FlatMatch-e can beat the second-best method, SoftMatch, with 2.04% improvement. More importantly, FlatMatch-e only shows slight performance drops from Flat- Match but significantly reduces the computational burden,
Gradlent Angle
which is demonstrated in Section 5.4.
tabels
labels
1000
Despite its effective performance, we also noticed that in some scenarios where the labels are extremely limited, such as CIFARIO with 10 labels, CIFAR100 with 400 labels, etc, our FlatMatch is slightly worse than a recently proposed method FreeMatch [66]. This is because the gradient esti- mated by the extremely scarce labeled data cannot align with the general training direction. To intuitively demonstrate this issue, we show the gradient angle between the cross-entropy loss from labeled data and the total SSL loss in Fig. 3. We can see that the gradient angle oscillates stronger as
Itteration
Figure 3: Gradient angle between cross- entropy loss C, and general SSL loss Lest-
7

---

Algorithm/ 2 FlatMatch with fixed labels for Stabilizing Cross-Sharpness.
Input: Labeled dataset D', unlabeled dataset D", model 8, number of fixed labels # fix, number of pre-train epochs #pre_traat
I: for E 0, 1....,. T do if E 0, 1,..., #pre brrin then
2:
3:
SSL pre-trannggg by optimizing Eq. 3:
4:
i #rrr_ lrrin then
5:
Select Top-# f fix confident unlabeled data au and assign fixed labels g. := argmax(go
6:
else
7:
Using the augmented labeled data to compute cross-sharpness via Eq. 4;
8: Using the original D' and DD'h to optimize O via Eq. 5;
the number of labels gets smaller. When the label number reduces to only 10, the angle becomes extremely unstable and could surpass 90" As result, such phenomenon could affect the worst-case perturbation computed by FlatMatch, which would introduce negligible noise to the training process.
However, this drawback can be properly solved by slightly augmenting the number of labels with pseudo-labeled unlabeled data. Specifically, before we start minimizing cross-sharpness, pre-train the backbone model with a common SSL method, such as FixMatch or FreeMatch, for 16 epochs to increase the confidence on unlabeled data. Then, we choose the most confident, i.e. high softmax probability, unlabeled data to be pseudo-labeled as labeled data. Different from other unlabeled data that are also trained with pseudo labels, the selected unlabeled data are fixed with their labels and act as labeled data when computing the sharpness. In this way, the cross-sharpness would be computed accurately by using both the original labeled data and augmented labeled data. Note that the augmentation with fixed labels is only used in the sharpness computation, when conducitng the second propagation step, all unlabeled data are treated similar to general SSL strategy, such process is summarized in Algorithm 2. Under this training strategy, we dub our method as "FlatMatch (Fix label)" and show its result in Table 1. The number of fixed laheled data in CIFARIO, CIFAR 100, SVHM, and STL1O is set to 500, 2500, 500, and 1000, respectively. If the number of the original labeled dataset contains enough labels, we do not add more fixed labeled data. We can see that in this scenario, our method has the best performance in all settings including the ones with I or 4 labels in each class, which demonstrates that using more labels can largely stabilize sharpness computation and further benefits the performance of FlatMatch, thus achieving superior results on all settings.
first
5.3 Ablation Study
To carefully justify the design of our method, we compare FlatMatch with "sharp. on labeled data D and "sharp. on unlabeled data D where the former one denotes both the worst-case model 0 and sharpness is computed on labeled data and the latter one denotes Ô and sharpness are calculated on unlabeled data. Additionally, we compute the sharpness on the full dataset, as denoted by "sharp. on unlabeled data D U D"",, Moreover, we also analyze the effect of EMA smoothing on the performance of FlatMatch-e. Specifically, we compare FlatMatch-e with setting "w/o EMA" that just uses gradient from the last batch of labeled data to calculate our worst-case model. The ablation study is conducted using CIFAR100 with a varied number of labels, which is shown in Table 2.
First, we can see that both two choices of our method are effective. Particularly, computing sharpness only on labeled data D' shows smaller performance degradation than computing sharp- ness on unlaheled data Du, and even shows better results than FixMatch. Hence, we know that the sharpness of labeled data can indeed im- prove the performance of SSL, only having lim- ited performance because the number of labeled data too scarce. On the other hand, when sharp- ness is completely based on unlabeled data, the performance significantly drops by nearly 10% compared to "sharpness on " This is because the training process on unlabeled data contains too much noise which causes erroneous gradient computation that would hinder the effectiveness of penalizing sharpness. Furthermore, we find that "w/o EMA" shows slightly inferior performance
Table 2: Ablation study on CIFARI00.
Dataset
CIFAR100
Label
400
2500
10000
sharp. on D
42.63
26.85.10.5
21.79.024
sharp. on D
49.45 36.30.221
27.05
sharp. on D' U D
43.88 1.64 27.26.11.2
23.42.1.77
FlatMatch w/o EMA FlatMatch-e
38.76 1.62 25.38
19.01
40.64=0.7 29.44+1156
23,23++1.28
38.98=1.53 25.62++088 19.78+0..9
8

---

2D contours of labeled data
2D contours of unlabeled data
ID loss curves
1.0
Accuracy
LOGS
20
640
OZ
26
ON
24
EN
2N
120
10-00
25-0350--55
0.25 0:50
0.75
1.00
15
Accuracy
LOSs
629
4.4
20
Figure 4: Loss landscapes of labeled data and unlabeled data obtained simultaneously from training using FlatMatch CIFARIO with 250 labels per class. The first row and second row show the results obtained from epoch 60 and epoch 150, respectively. The first column and second column show the 2D loss contours of labeled data and unlabeled data, respectively, and the last column shows the ID loss curves.
to FlatMatch-e. Such degradation is consistent with the findings from Liu al. [47] that the last gradient direction is distinct from the current one. As gradient descent has been conducted in the previous batch, reusing the gradient perturb the current model might not find the perfect worst-case model on the current labeled data. Based on our observation, using EMA can stabilize the gradient can lead to accurate sharpness calculation.
Parameter Sensitivity
Training Stability
Training Efficiency
0.5
Farttakk
Foottack
traniin (ee./iter)
test error (%))
gradient norm
0.4
BOC (76)
0,3
0.2
0.1
20
0.0
battorbation magnitude P
11.5
8.25
training epoch
0.0
Flathhach
FlatMtnche
FreaMatch
FlexMatch
PreMatch
Figure 5: Parameter sensitiv- ity analysis regarding perturbation magnitude p on CIFARI00
Figure 6: Stability analysis on
Figure 7: Training efficiency compari- son CIFAR100 with 10000 labels.
gradient norm from training CI- FAR
5.4 Analytical Study
In this section, we analyze the performance of FlatMatch by considering visualization, parameter sensitivity, training stability, and effficiecy..
Loss visualization: To show that FlatMatch can properly solve the sharp loss problem of labeled data, we train FlatMatch under the same setting the ones demonstrated in Section and plot the 2D loss contour as well as the ID loss curve in Fig. 4. We can see that our FlatMatch can produce a much flatter loss landscape than FixMatch does in Fig. 1, where the jagged curve has been eliminated and become very smooth. Therefore, by minimizing cross-sharpness, the generalization performance on labeled data can be largely improved.
Sensitivity analysis: Our FlatMatch requires hyperparameter P which controls the perturbation magnitude to obtain the worst-case model. To analyze the performance sensitivity of varying we show the result on CIFAR 100 Fig. 5. We observe that small o values show little impact on the test performance. However, when p increases to more than 0.25, the performance would largely degrade. Moreover, among three settings with varied label numbers, we find that more numbers labels could enhance the model sensitivity against changing of p. For example, when changing the from 0.

---

to 0.25, the performance difference on 400 labels, 2500 labels, and 10000 labels are 0.05%, 1.2%, 3.42%. Generally, the optimal value for p 0.1.
Training stability: To validate the training stability of FlatMatch, we show the gradient norm which an important criterion to present gradient variation in Fig. 6. We observe that the gradient norm of FlatMatch is significantly smaller than FreeMatch during the whole training phase. Therefore, minimizing the cross-sharpness can indeed improve training stability.
Training efficiency: We compare the training time and test accuracy of FlatMatch, FlatMatch-e, FreeMatch, FlexMatch, and Fix.Match to validate the efficiency property of method. As shown in Fig. 7, we find that despite the superior accuracy performance, FlatMatch requires more time for each iteration than the other three methods. However, the efficient variant FlatMatch-e can largely reduce the computational cost without losing too much learning performance. Hence, we can conclude that leveraging EMA for computing cross-sharpness is both effective and efficient.
6 Conclusion, Limitation and Broader Impact
In this paper, we propose a novel SSL method dubbed FlatMatch to address the mismatched gen- eralization performance between labeled data and unlabeled data. By minimizing the proposed cross-sharpness regularization, FlatMatch can leverage the richness of unlabeled data to improve the generalization performance on labeled data. As result, the supervised knowledge from labeled data can better guide SSL, and achieve improved test performance than most existing methods. Moreover, we propose an efficient implementation to reduce the computation cost. We conduct comprehensive experiments to validate our method regarding effectiveness, sensitivity, stability, and efficiency. Additionally. FlatMatch is slightly limited under supervied learning due to the requirement of enough labeled data. Our method shows that SSL can be further improved by exploring generalization, which could potential direction future research.
7 Acknowledgements
Li Shen was partially supported by STI 2030-Major Projects (20212D0201405) Zhuo Huang was supported by JD Technology Scholarship for Postgraduate Research in Artificial Intelligence (SC4103). Jun Yu was supported by the Natural Science Foundation of China (62276242), Na- tional Aviation Science Foundation (2022Z07 1078001), CAAI-Huawei MindSpore Open Fund Anhui Province Key Research and Develop-
CCAAIXSJLJJ-2211 2001-0001-..B ment Program (202 144a05020007, USTC-IAT Application Sci. & Tech. Achievement Cultivation Program (JL06521001Y) Y), and Sci. & Tech. Innovation Special Zone Bo Han was supported by the NSFC Young Scientists Fund (62006202), NSFC General Program (62376235), and Guangdong Basic and Applied Basic Research Foundation (2022A15155) 1652). Tongliang Liu was partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
1-01-------00000
References
[1] Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-awee aware minimization. In /CML, pages 639-668. PMLR, 2022.
[2] Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In ICCV, pages 9312-9321, 2021.
[3] Yingbin Bai, Erkun Yang. Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In NeurIPS, volume 34, pages 24392-24403, 2021.
[4] Yingbin Bai, Erkun Yang, Zhaoqing Wang, Yuxuan Du, Bo Han, Cheng Deng, Dadong Wang, and Tongliang Liu. Rsa: Reducing semantic shift from aggressive augmentations for self- supervised learning. In NeurlPS, volume 35, pages 21128-21 21141, 2022.
[5] Shai Ben-David, Tyler Lu, and Dávid Pál. Does unlabeled data provably help? worst-case analysis of the sample complexity of semi-supervised learning. In COLT, pages 33 33-44, 2008.
10

---

[6] Kristin P Bennett and Ayhan Demiriz. Semi-supervised support vector machines. In NeurlPS, pages 368-374, 1999.
[7] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervisee learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:/911.0973 2019.
[8] David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. In NeurIPS, pages 5049-5059, 2019.
[9] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In COLT, pages 92-100, 1998.
[10] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In ICLR, 2022.
[11] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In NeurIPS, volume 34, pages 22405-22418, 2021.
[12] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 20(3):542 542, 2009.
[13] Olivier Chapelle, Vikas Sindhwani, and Sathiya S Keerthi. Optimization techniques for semi- supervised support vector machines. Journal of Machine Learning Research, 9(),, 2008
[14] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124188 2019.
[15] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In NeurIPS, 2022.
[16] Hao Chen, Ran Tao, Yue Fan, Yidong Wang. Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi- supervised learning. In ICLR, 2023.
[17] Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong, Semi-supervised learning under class distribution mismatch. In AAAI, 2020.
[18] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in un- supervised feature learning. In AISTATS, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[19] Ekin Dogus Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaug- ment: Learning augmentation policies from data. In CVPR, 2018.
[20] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In NeurIPS, volume 33, pages 18613-18624, 2020.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins
arXiv:2010.11990 19299 2020.
[22] Jiawei Du, Hanshu Yan, Jiashi Feng, Jocy Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. ICLR, 2022.
[23] Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In NewrIPS, volume 35, pages 23439-23451 2022.

---

[24] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. In ICLR, 2020.
[25] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, pages 1672-2680, 2014.
[26] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[27] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NeurIPS, pages 529-536, 2005.
(28) Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In ICML,, pages 8082-8094. PMLR, 2022.
[29] Lan-Zhe Guo, Zhen-Yo Yu Zhang, Yuan Jiang. Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICML, 2020.
[30] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages
2020.
[31] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[32] Rundong He, Zhongyi Han, Yang Yang, and Yilong Yin. Not all parameters should be treated equally: Deep safe semi-supervised learning under class distribution mismatch. In AAAI, pages 6883. AAAI Press, 2022.
[33] Rundong He, Zhongyi Han, and Yilong Yin. Towards safe and robust weakly-supervised anomaly detection under subpopulation shift. Knowledge-Based Systems, 250: 109088, 2022.
[34] Ziming Hong, Shiming Chen, Guo-Sen Xie, Wenhan Yang. Jian Zhao, Yuanjie Shao, Qinmu Peng, and Xinge You. Semantic compression embedding for generative zero-shot learning. In IJCAI, volume 7, pages 956-963, 2022.
[35] Huaxi Huang, Hui Kang, Sheng Liu, Olivier Salvado, Thierry Rakotoarivelo, Dadong Wang, and Tongliang Liu. Paddles: Phase-amplitude spectrum disentangled early stopping for learning with noisy labels. In ICCV, pages 167 197700 2023.
[36] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distribution examples via augmenting content and style. In ICLR, 2023.
[37] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurlPS, volume 34, 2021.
[38] Zhuo Huang, Jian Yang, and Chen Gong. They are not completely useless: Towards recycling transferable unlabeled data for Glass-mismatched semi- cemi-supervieed learning. IEEE Transactions on Multimedia, 2022.
[39] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Averaging weights leads to wider optima and better generalization. arXiv preprint 1rXiv:1803.05407, 2018.
[40] Hui Kang, Sheng Liu, Huaxi Huang. Jun Yu, Bo Han, Dadong Wang, and Tongliang Liu. Unleashing the potential of regularization strategies in learning with noisy labels. arXiv preprint arXiv:2307.05025 2023.
[41] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

[43] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness aware minimization for scale-invariant learning of deep neural networks. In ICML. pages 5905-5914. PMLR, 2021.
[44] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2016.
[45] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444, 2015.
[46] Dong-Hyur Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, 2013.
[47] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In CVPR, pages 12360-12370, 2022.
[48] Yury Maximov, Massih-Rezz Reza Amini, and Zaid Harchaoui. Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm. Journal of Artificial Intelligence Research, 61:761-786, 2.
[49] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and supervsueee learning. IEEE Transactions on Pattem Analysis and Machine Intelligence (TPAMI), (1(8):19799199 1993,, 2018.
[50] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop. 2011.
[51] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In NeurlPS, volume 30, 2017.
[52] Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoe V Le. Meta pseudo labels. arXiv preprint arXiv:0003. 10580, 2020.
[53] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-awar pseudo-label selection framework for semi-supervised learning. In ICLR, 2021.
[54] David S Rosenberg and Peter L Bartlett. The rademacher complexity of co-regularized kernel classes. In Artificial Intelligence and Statistics, pages 396-403. PMLR, 2007.
[55] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 111(3)211-222, 2015.
[56) Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. In NeurlPS, volume 34, 2021.
[57] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang. Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.0766 2020.
[58] David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. ICCV, pages 7807-7817, 2021.
[59] Antti Tarvainen and Harri Valpola. Meanteachers are better role models: Weight-averaged consistency targets improve supe-supe deep learning results. In NeurlPS, pages 1195- 1204, 2017.
[60] Fei Wang and Changshui Zhang, Label propagation through linear neighborhoods. IEEE Transactions on Knowledge and Data Engineering (TKDE), 20(1):55-67, 2007.
[61] Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. arXiv preprint 103.::233333333.. 2023.
13

---

[62] Qizhou Wang. Zhen Fang, Yonggang Zhang, Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of aut-of-distribution detection. In NeurIPS, 2023.
[63] Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In NeurIPS, 2022.
[64] Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Mareus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In ICLR, 2023.
[65] Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang. Zhi Zhou, Lan-Zhe Guo, et al. Usb: A unified semi-supervised learning benchmark for classification. In NeurIPS, volume 35, pages 3938-3961, 2022.
[66] Yidong Wang, Hao Chen, Qiang Heng. Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang. Freematch: Self-adaptive thresholding for semi- supervised learning. In ICLR, 2023.
[67] Zhaoqing Wang. Ziyu Chen, Yaqian Li, Yandong Guo, Jun Yu. Mingming Gong, and Tongliang Liu. Mosaic representation learning for self-supervised visual pre-training. In ICLR, 2022.
[68] Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learning. In CVPR, pages (6590-16599) 2022.
[69] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In NeurlPS, volume 33, pages 2958-2968899 2988-296,,45 2020.
[70] Yuhao Wu, Xiaobo Xia, Jun Yu, Bo Han, Gang Niu, Masashi Sugiyama, and Tongliang Liu. Making binary classification from multiple unlabeled datasets almost free of supervision. arXiv preprint 23Xiv:2306.07036, 2023.
[71] Zhengning Wu, Tianyu He, Xiaobo Xia, Jun Yu, Xu Shen, and Tongliang Liu. Conditional con- sistency regularization for cemi-supereised multi-label image classification. JEEE Transactions on Multimedia, 2023.
[72] Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu. Combating noisy labels with sample selection by mining high-discrepancy examples. In ICCV, pages 1833-1843, 2023.
[73] Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Sample selection with uncertainty of losses for learning with noisy labels. arXiv preprint arXivv21066.0045,, 2021.
[74] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. In NeurIPS, 2020.
[75] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe Le. Self-training with noisy student improves imagenet classification. In CVPR, pages 10687-10698, 2020.
[76] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In ICML, pages 1 11525-11536. PMLR, 2021.
[77] Suqin Yuan, Lei Feng, and Tongliang Liu. Late stopping: Avoiding confidently learning from mislabeled examples. In ICCV, pages 16079-16088, 2023.
[78] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[79] Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In NewrIPS, volume 34, pages 18408-18419, 2021.
[80] Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2017.
14

---

[81] Zhiyuan Zhang, Ruixuan Luo, Qi Su, and Xu Sun. Ga-sam: Gradient-strength based adaptive sharpness-awart minimization for improved generalization. arXiv preprint arXiv:2210.06895, 2022.
[82] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. ICML, 2022.
[83] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Ss-sam: Stochastic scheduled sharpness-aware minimization for efficiently training deep neural networks. arXiv preprint arXiv:2203.09962. 2022.
[84] Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NeurIPS, pages 321-328, 2004.
15

---

Supplementary Material for "FlatMatth: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning"
In this Appendix, we provide additional details and experimental results to complement the proposed method. First, describe supplementary experimental details in Section A. Then, provide extra quantitative results, including employing FlatMatch to other SSL methods, comparisons on ImageNet30 and ImageNet [55] datasets, performance on Vision Transformer in Section B. Further, we show more empirical results to qualitatively validate FlatMatch in Section C. Then, we conduct convergence study on test aceuracy and training loss curves in Section Moreover, we provide extra visualizations to show the loss landscape on later state of training in Section E. Finally, we summarize this paper and make discussion on prospective research in Section
A Supplementary Details
The experimental setting of this paper follows Wang et al. [65]. Specifically, the hyper-parameters are composed of Aggorithm-dependen parameters and shown in Table 3 and Table 4, respectively. For algorithn-dependent parameters of FlatMatch, we use the same unlabeled data and labeled data ratio FreeMatch [66] well methods to sample data into mini-batch. The perturbation magnitude is based on the results from pyper-parameter sensitivity analysis the main paper and is chosen as 0.05 for all experiments. For updating the historical gradient using memory buffer, use EMA with factor a to ensemble the gradient result. Moreover, we choose the thresholding strategy from FreeMatch and use an EMA decay. Note that for combining the cross-sharpness regularization from FlatMatch with empirical risk, we find that there is no need to introduce another weight to trade off the two loss functions, hence the weight for cross-sharpness is just set to for all experiments. For algoritmm-independent hyper- parameters, we have listed the important model setting, optimizer parameters, and data sampling setting below. Note that all baseline methods follow the implementation of USB [65] and are trained with EMA decay with 0.999 to smooth the parameter updating.
parameters, which are
other baseline
Table 3: Algorithm-dependent hyper-parameters.
Algorithm
FlatMatch
Unlabeled Data to Labeled Data Ratio (CIFAR-I0/100, STL-10, SVHN)
Unlabeled Data to Labeled Data Ratio (ImageNet30)
Perturbation magnitude for all experiments
0.05
EMA factor for updating gradient
0.999
Thresholding EMA decay for all experiments
0.999
Trade-off weight Ax-shap for cross-sharpness
B Additional Quantitative Results
In this section, we conduct additional experiments CIFAR10 and ImageNet30 [55] datasets to compare the performance between some of the most edge-cutting methods, including FixMatch [57], Dash [76], FlexMatch [79], FreeMatch [66], SoftMatch [16], and our FlatMatch.
16

---

Table Algorithm-independee hyper-parameteryy
Dataset
CIFAR-10
CIFAR-100
STL-10
SVHN
ImageNet30
Model
WRN-28-2
WRN-28-8
WRN-37-2
WRN-28-2
ResNet-50
Weight decay
5e-4
le-3
5e-4
5e-4
3e-4
Batch size
64
128
Learning rate
0.03
SGD momentum
0.9
EMA decay
0.999
Table 5: Performance on boosting other SSL methods using FlatMatch with the fixed number of labels.
Dataset
CIFARIO
# label
40
250
4000
FixMatch
7.47
4.86:10.05
4.21
FixMatch+FlatMatch
6.50
4.27
3.92
Dash
8.93+3..11
5.16:t0.23
4.36
Dash+FlauMatch
6.73++2.49
4.45
4.02
FlexMatch
4.97+00.6
4.98
4.19
FlexMatch-FlatMacch
4.47+0.92
4.25
3.88
SoftMatch
4.91
4.82+10.09
40.02
SoftMatch+FlatMatch
4.89+1.32
3.18
1.88
FreeMatch
4.90
4.88
4. 4..00
FlatMatch (Fix label)
4.89+1.24
3.90
3.61 #0.49
B.1 Combining FlatMatch with Other Methods on CIFARIO
We choose CIFARIO dataset with the number of labeled data varied as 40, 250, and 4000, and apply the FlatMatch methodology to several recently proposed SSL methods to show the effectiveness of the proposed cross-sharpness regularization. The results are shown Table as we can see that our method can further boost the learning performance of all five methods on all three settings, which proves that the cross-sharpness method quite universal to SSL approaches and can bring non-trie non-trivial performance enhancement. Note that in the 40 labels setting, we compute our cross cross-sharpness on 500 examples with fixed labels, as demonstrated in Section 5.2 from the main paper.
B.2 Comparing FlatMatch to Other Methods on Large-Scale Datasets and Sophisticated Architecture
To further testify the performance of FlatMatch on a large-scale datasets, we first conduct experiments ImageNet30 dataset which is subset from the original ImageNet dataset and contains 30000 training examples with resolution 256x 256 from 30 classes. Moreover, we also test the performance on the original ImageNet dataset. As Vision Transformer (ViT) [21] has manifested great power on classification tasks, we also adopt ViT as our backbone to validate the performance of FlatMatch.
The experiments ImageNet30 are more time-consuming which normally takes 5 days finish, much more than CIFAR 10 dataset which takes days. We vary the number of labeled data as 1500 and 3000 and show the comparison in Table 6. We observe the effectiveness of FlatMatch over all other baseline methods in both two settings, which again validates the superiority of our method and its effective performance on large-scale datasets.
Additionally, we have conducted the experiments on the original ImageNet dataset by choosing only 100 labels per each class, and provide the test error results of FlatMatch, FlatMatch-e, FixMatch,
17

---

Table 6: Comparison on ImageNet30.
Dataset
ImageNet30
label
1500
3000
FixMatch
12.48=0.67
8.25+0.54
Dash
13.29
8.79
FlexMatch
11.48=0.52
8.04
SoftMatch
10.81
7.78
FreeMatch
10.34=0.46
#0.19
FlatMatch
9.71
6.77+1122
Table 7: Comparison on ImageNet using Wide-ResNet-28-4 and Vision Transformer.
Dataset
Architecture
FlatMatch
FlatMatch-e
FreeMatch
FlexMatch
FixMatch
Sede-ResNeet22-4
38.70 21.57
39.92
40.57 23.55
41.95 23.78
43.66
ImageNet
ViT-Base (86M)
22.07
25.52
FlexMatch, and FreeMatch as shown in Table 7. We see that large-scale dataset such as ImageNet, the performances of both FlatMatch and FlatMatch-e are still superior to other components. Moreover, still observe the effectiveness of the of methods on ViT. Therefore, is reasonable to conclude that the performance of FlatMatch is extendable to large-scale datasets and sophisticated architectures.
CIFAR10
SVHN
validation sharpness
validation sharpness
0.06
3:30
0.04
2.2
0.c0
SheMtaich
Back
FreeMatch
PlatMatth
Sepursbed
HieMarck Badh
ErveMarch
PaaMttch
Piils Superviict
0.02
5:7
400
1002
200
D00
training epoch
training epoch
Figure 8: Comparison of sharpness between various SSL methods during training.
C Additional Qualitative Results
To further evaluate the flatness of different SSL models during training, we leverage validation set to compute the sharpness. The sharpness measured by the increase of loss within & bounded neighbor, which is formally defined as Sharpness := L(0 + 6 (0)) L(0), where e (0) arg ma C( + e). Specifically, we compare the proposed FlatMatch with FixMatch, Dash, and FreeMatch, and use fully supervised learning as baseline method. The experiments are conducted on CIFAR10 and SVHN datasets whose results are shown in Figure 8, First, we observe that FlatMatch achieves the lowest sharpness eurve during training on both two datasets, which indicates the SSL model learned by FlatMatch is more robust to perturbations and would not oscillate significantly when facing changes parameter space. Moreover, we find that fully supervised learning does not improve the flatness as the training proceeds, while all SSL methods can decrease the sharpness to some extent, which demonstrates that training with unlabeled data can help improve the flatness of SSL models.
18

---

CIFARIO
SVHN
label
Bx 500 latrels
fxx 2009 labels
win fix lubel fix 250 labels
fxx 500 labels
2000 lubels
flx: 250 Inbels
Rx 1000 labels
fix 4000 lahcls
Rx 1000 labels
16
test crror.
test error.
10-
40
253
1003
250
1003
label
label
Figure 9: Analysis on changing the number of fixed labels.
Furthermore, shown in the main paper, find that FlatMatch has limited performance extremely scare labeled settings. However, this limitation can be addressed by introducing some unlabeled data with fixed labels to improve the computation of cross-sharpness. Hence, here investigate the effect of changing the number of fixed on the performance of FlatMatch. Specifically, conduct experiments
CIFAR1O and SVHN datasets and fixing different numbers of labels as O ("w/o fix label"), 250, 500, 1000, 2000, 4000'. The results are shown in Figure 9. We find that both too few fixed labels, i.e,, 250 labels and too many fixed labels, i.e., 4000 labels in CIFARIO and 2000 labels in SVHN, would show a performance drop compared to the optimal number, 500 fixed labels. This is because if the number of fixed labels is too small, the gradient computation would be inaccurate, further limiting the learning results, On the other hand, too many fixed labels would introduce noisy labeled unlabeled data, which would largely mislead the SSL and show serious performance degradation.
CIFARI00
0.7
Test Accuracy
0.6
0.4
0.3
FlaiMatch
FreeMatch
200
400
600
800
1000
Training Epoch
Figure 10: Convergence of accuracy analysis between FlatMatch and Frechatch.
D Convergence Analysis
In this section, we conduct analyses regarding the test accuracy and training loss to validate the convergence property of FlatMatch.
For convergence of accuracy, here we compare our FlatMatch with FreeMatch, which has the best performance among most SSL baseline methods. The accuracy curve is shown in Figure 10. We
'The 4000 fixed labels setting is not conducted on SVHN as the performance of 2000 fixed labels setting already shows significantly performance degradation.
19

---

observe that the performances of two methods are almost comparable in early stages, but FlatMatch continues to improve the training performance in the middle and later stages and finally achieves better accuracy than FreeMatch on the final point. Therefore, we can conclude that our method can converge to superior performance than Free
Curve FixMatch CIFARIO0
Loss Curve FlatMatch on CIFAR100
Loss Difference CIFARI00
2.0
Df Dffluaae alTobhed
135
1.50
1.0ss Value
Loss Value
Loss Value
2.09
0.5
Mull
0.9
Training Epoch
Training Epoch
Training Epoch
Figure 11: Convergence of loss values of FlatMatch and Fix Match.
Moreover, illustrate the convergence loss curves, we show the loss values of labeled data and unlabeled data from both FlatMatch and FixMatch during training. Moreover, we subtract the loss value of unlabeled data with loss of labeled data compute loss difference, which gives an illustration about the performance gap between both two datasets.
2D contours of labeled data
2D contours of unlabeled data
ID loss curves
18
10
Accuracy
Loss
-42
20
-57
0.75
Loss of FixMatch
2.2
Accuraty
LOcS
229
20
440
1.00
0.75
0.50
0.25 0.00
0.25
0.50
0.35
1.00
Loss curves FlatMatch
Figure 12:
Loss landscapes of labeled data and unlabeled data obtained simultaneously from training using FlatMatch and FixMatch CIFARI0 with 250 labels per class. The results are generated from the last model checkpoint. The first column and second column show the 2D loss contours of labeled data and unlabeled data, respectively, and the last column shows the ID loss
There are two findings: 1) The loss value of labeled data quickly converges zero and is significantly smaller than unlabeled data. Such phenomenon occurs in two methods which supports our claim that the learning on labeled data converges faster than unlabeled data. 2) The loss difference between two datasets of FlatMatch is significantly smaller than FixMatch, which indicates that our FlatMatch can alleviate the unmatched convergence speed of two datasets and helps decrease the loss gap between two datasets.
20

---

E More Visualizations
To show how the loss curves appears in the later stage of training, we generate the loss landscape of FixMatch and FlatMatch on the last training epoch (2 We can see that FlatMatch generates a wider loss landscape than FixMatch. Moreover, the loss curve of labeled data from FlatMatch is smoother than that of FixMatch. Therefore, generalization result.
again conclude that FlatMatch benefit the
Summary and Future Work
In this paper, we propose a novel FlatMatch approach that minimizes the cross-sharpness measure improve the generalization performance SSL. Through extensive quantitative and qualitative experiments, we have thoroughly evaluated the performance of FlatMatch and demonstrated its superiority other compared methods. Thanks to the generalization improvement of FlatMatch, the classification accuracy on many scenarios have even passed the fully-supervised baseline.
However, the learning performance of SSL still largely depends on the careful selection of labeled data. Specifically, in the barely-supervise learning scenario, the selected scarce labeled data deviate from the cluster center, the learning performance of many existing SSL methods would be significantly affected. This is due to the generalization performance between labeled data and unlabeled data being largely mismatched. Under this scenario, the performance of FlatMatch should be further evaluated.
21


Reviews:
[{"review_id": "247mFvcUBf", "reviewer": "Reviewer_9rMs", "summary": "This paper introduced FlatMatch, a new SSL method which encourages consistent learning performance between labeled and unlabeled data and aims to boost the performance of SSL methods without being limited by insufficient label information. The paper analyzes the loss landscapes of labeled and unlabeled data and proposes to use a worst-case model alongside the original SSL model to achieve agreement on unlabeled data through a novel cross-sharpness regularization method. The approach is tested on 4 computer vision datasets from image classification and obtains state-of-the-art results.", "questions": "For Figures 1 and 4 last column, how do the curves look at later iterations? The plots are at epochs 60 and 150 which is at the very beginning of training (e.g. CIFAR-100 trains for 9K epochs). Also, what is the dataset used for those plots? I haven’t seen it mentioned in text.\n", "limitations": "The paper states limitations. I believe the computational cost limitation should be explicitly stated here. I don't think this work has any negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of the approach is clear and concise. Overall, this paper is very-well written.\n- First, I want to say that the idea is very interesting. SSL models usually obtain perfect accuracy on labeled data very fast and it is intuitive that at this point there is not much information gain. This paper solves this drawback  in a very clever way.\n- The performance of FlatMatch is impressive, obtaining considerable improvements especially in “harder” datasets where there was room for improvement from prior approaches.\n- Additionally, extensive analyses are performed, which offer valuable insights and help further understand the approach.\n", "weaknesses": "- A notable weakness is the absence of experiments on large-scale datasets (ImageNet?). It’s clear that FlatMatch performs well on “harder” datasets, however CIFAR-100 and STL are still low-scale. I believe ImageNet experiments are vital since it’s the closest to a real-world setup and would further solidify the effectiveness of your approach. At the same time, all recent proposed SSL methods verify performance on ImageNet (100/300K). \n- (Minor) The training efficiency is not that good. This is okay, but the graph in Figure 7 is built around FixMatch as the most efficient method. I think this is a bit misleading. FlexMatch has the same training time with a higher accuracy compared to FixMatch; so I think FlexMatch should be the reference. Moreover, it would be interesting to also see a convergence analysis as done in FlexMatch (plot accuracy/iteration) since I believe this would give a better sense of the training efficiency compared to how the plot is structured here.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1**: We thank Review 9rMs for your positive opinion. We have carefully conducted experiments on ImageNet, added efficiency comparison with FlexMatch, analyzed the convergence regarding training accuracy, and showed the loss curves on the final iteration.\n\n\n*Q1*: **Result on ImageNet.**  \n*A1*: Please see the general response.\n\n*Q2*: **Efficiency comparison with FlexMatch.**  \n*A2*: Thanks for the suggestion, we have included the efficiency comparison with FlexMatch, as shown in Fig. 1 in the rebuttal pdf. \n* We want to rectify that the efficiency of FlexMatch is not the same as FixMatch, as it requires an indexing process that takes a noticeable computational cost, as testified by the empirical result. Moreover, the efficiency of FreeMatch on the CIFAR100 dataset is also inferior to FixMatch. \n* When comparing our methods with the SSL baseline methods, we find that FlatMatch achieves the best performance and FlatMatch-e can still bring non-trivial improvements without introducing too much computational cost.\n\n\n*Q3*: **Convergence analysis of accuracy.**  \n*A3*: Thanks for the advice, here we compare our FlatMatch with FreeMatch, which has the best performance among most SSL baseline methods. The accuracy curve is shown in Fig. 2 in the rebuttal pdf.  \n* We observe that the performances of the two methods are almost comparable in the early stages, but FlatMatch continues to improve the training performance in the middle and later stages and finally achieves better accuracy than FreeMatch on the final point. Therefore, we can conclude that our method can converge to superior performance than FreeMatch.\n\n\n\n*Q4*: **Loss curves of later epoch, and the used dataset.**  \n*A4*: Thanks, our loss curves are generated on CIFAR10 datasets, we have made this clear in the main paper.\n* We also showed the loss curves of FlatMatch and FixMatch on the last epoch from training in Fig. 5 in the rebuttal pdf. We can see that FlatMatch generates a wider loss landscape than FixMatch. \n* Moreover, the loss curve of labeled data from FlatMatch is smoother than that of FixMatch. Therefore, we can again conclude that FlatMatch can benefit the generalization result."}, {"review_id": "ddXNxeALzI", "reviewer": "Reviewer_Y5aq", "summary": "The authors propose a semi-supervised learning (SSL) method that applies the sharpness-aware minimization (SAM) method for consistency regularization. Unlike FixMatch, it does not apply perturbation to the unlabeled training sample but evaluates the consistency between posterior distributions of predicted labels between neural networks with original and perturbed parameters. Experimental results show that the proposed method outperforms existing SSL methods with the help of initialization pre-processing to increase labeled samples when their numbers are small.", "questions": "Equation (4) explains the proposed method. Is min missing in the second term?\nSimilarly, min is missing in the second and the third terms in equation (3).", "limitations": "The authors explain the condition where the proposed method is weak when they use it alone.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper proposes a new SSL algorithm that shows consistently improved performance.", "weaknesses": "The authors should show results of existing SSL methods that use SAM as the optimizer instead of SGD.\n\nThe equations contain errors mixing the definition of objective function and their derivatives.\nFor example, the second term in the first line of equation (2) looks for optimal \\theta that minimizes the SAM's objective, but the third term is the derivative of the objective function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R2**: We thank Reviewer Y5aq for your constructive comments. We have carefully included several typical baselines with SAM optimizer and reformulated our objectives, derivatives, and optimization.\n\n*Q1*: **Comparison to existing SSL methods with SAM optimizer.**  \n*A1*: Thanks for the advice. Here we conduct experiments by employing SAM to several SSL baseline methods such as FreeMatch, FlexMatch, and FixMatch on CIFAR100 with a varied number of labels. The results are shown below:\n| Dataset       |  CIFAR100 | CIFAR100 |  CIFAR100 |\n|---------------|:---------:|:---------:|:---------:|\n| # Label       |    400   |    2500   |  10000   |\n| FixMatch      |   46.42  |   28.03   |  22.20   |\n| FixMatch-SAM  |   47.59  |   28.62   |  22.67   |\n| FlexMatch     |   39.94   |  26.49   |   21.90  |\n| FlexMatch-SAM |   39.89   |  27.05   |   22.38  |\n| FreeMatch     |   37.98   |  26.47   |   21.68  |\n| FreeMatch-SAM |   39.02   |  27.64   |   22.55  |\n| FlatMatch     | **38.76** | **24.28** | **18.21** |\n\n* We find that such a vanilla combination of SAM and SSL methods does not bring further performance improvement. As we demonstrated in the gradient angle analysis in Fig. 3 in the main paper, the gradient could contain many noises especially when too many unlabeled data are incorporated. As a result, the SAM optimization might be misled by the uncertainty of unlabeled data, thus causing performance drops. \n* Such an issue is also found in Cha et al. [B1] where SAM is not beneficial to the generalization across datasets. \n\n[B1] Cha et al., Swad: Domain generalization by seeking flat minima, in NeurIPS 2021.\n\n\n\n*Q2&Q3*: **Mixing the objective function and derivatives. Missing the optimization notation ''min''.**  \n*A2&A3*: Thanks for pointing this out. We have rigorously improved our formulation by clarifying the differentiation and addressing the optimization goal.\n"}, {"review_id": "BxBNSFLGiK", "reviewer": "Reviewer_yYJy", "summary": "This paper focuses on Semi-supervised Learning (SSL) where the training data consists of scarce labeled data and a massive amount of unlabeled data. The authors argue that the propagation of label guidance from labeled data to unlabeled data is challenging. This can cause the learning process on labeled data to progress much faster than on unlabeled data, leading to sub-optimal generalization performance. To alleviate this issue, the authors propose FlatMatch. This method seeks to increase the empirical risk on labeled data to obtain a worst-case model and then penalize the prediction difference (i.e., cross-sharpness) between the worst-case model and the original model, which can benefits the final model.", "questions": "See weaknesses", "limitations": "The performance of FlatMatch on large-scale datasets remains unverified, and the computational complexity is high to achieve strong performance.\n\n\n\n\n", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow. The motivation is clear, and the unlabeled loss is indeed always larger than the labeled loss in the SSL setting. The idea is interesting.\n2. The experiments on CIFAR, SVHN, and STL-10 demonstrate the effectiveness of FlatMatch.\n3. The original FlatMatch algorithm is computationally complex, which makes it difficult to run FlatMatch on large-scale datasets like ImageNet. However, the authors propose an efficient version of FlatMatch to make the runtime more manageable.", "weaknesses": "1. Although FlatMatch performs well on smaller datasets like CIFAR, SVHN, and STL-10, its performance on large-scale datasets remains unknown. Experiments on ImageNet are necessary to prove the generalization capabilities of the proposed method.\n2. The authors train wide-resnets from scratch. However, many recent works have conducted SSL research using Vision Transformers[1,2], which can greatly improve the performance of SSL algorithms in many settings. The performance of FlatMatch when Vision Transformers are used as the backbone remains unknown.\n3. While FlatMatch can achieve high performance, it remains computationally complex. The running time of the efficient version of FlatMatch is manageable, but its performance cannot surpass that of SOTA SSL algorithms.\n\n\n[1] Cai, Zhaowei, et al. \"Semi-supervised vision transformers at scale.\" Advances in Neural Information Processing Systems 35 (2022): 25697-25710.\n\n[2] Wang, Yidong, et al. \"Usb: A unified semi-supervised learning benchmark for classification.\" Advances in Neural Information Processing Systems 35 (2022): 3938-3961.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R3**: We thank Reviewer yYJy for your helpful comments. Here we carefully conducted experiments on large-scale datasets (ImageNet) as well as sophisticated models (ViT) and justified the efficiency and effectiveness of FlatMatch.\n\n\n*Q1&Q2*: **Performance on ImageNet dataset and Vision Transformer architecture.**  \n*A1*: Please see the general response.\n\n\n\n*Q3*: **Limited efficiency of FlatMatch and limited effectiveness of FlatMatch-e.**  \n*A3*: We want to argue that requiring efficiency and effectiveness at the same time is quite demanding for any method.  \n* The proposed FlatMatch method shows non-trivial performance improvement in most of the settings among various datasets. Although its efficiency is one defect, we propose an efficient variant that shows significant efficiency improvement. \n    * Empirically, our FlatMatch-e takes 0.29 sec/iter, compared to 0.28 sec/iter from FreeMatch;\n    * Meanwhile, FlatMatch-e brings 2.67% performance improvement on CIFAR100 with 10000 labels, and its efficiency is much faster than FlexMatch (0.34 sec/iter).\n* About the effectiveness, we want to clarify that the proposed FlatMatch-e can still surpass existing SSL methods in most of the settings. \n    * In the main result, FlatMatch-e passes existing SSL methods in the following settings:\n\n        | Dataset  | CIFAR10 | CIFAR10 | CIFAR10 | CIFAR10 | CIFAR100 | CIFAR100 | CIFAR100 | SVHN | SVHN | SVHN | STL10 | STL10 |\n        |----------|---------|---------|---------|---------|----------|----------|----------|------|------|------|-------|-------|\n        | # labels | 10      | 40      | 250     | 4000    | 400      | 2500     | 10000    | 40   | 250  | 1000 | 40    | 1000  |\n        | Best?    |       |       | &check;     | &check;     |        | &check;      | &check;      |    | &check;  | &check;  |     | &check;   |\n    * Moreover, as the additional experiments on the ImageNet dataset and ViT backbone show, FlatMatch-e can still surpass strong baseline methods such as FreeMatch and FlexMatch."}, {"review_id": "ORdIggfjfo", "reviewer": "Reviewer_Z9Pn", "summary": "The paper focuses on the problem of semi-supervised learning (SSL). The authors first study the loss landscapes of labeled data and unlabeled data and find a generalization mismatch. Base on the findings, they propose FlatMatch to encourage consistent learning performance between labeled data and unlabeled data. Extensive experiments demonstrate the effectiveness of their methods.", "questions": "1. What is the function $l_d$ used in the experiment?\n2. For \"sharpness on $D^{l}(D^{u})$\" in Table 2, is D^{u}(D^{l}) used? If yes, how to use it? If not, it is recommended to compare the performance of using SAM on the full dataset ($D^{l} \\cup D^{u}$).\n3. The significance of this work is somewhat limited, as the focus is only on visual datasets. The effectiveness and practicality of the method would be further improved if results from more types of datasets (such as text classification) can be provided.", "limitations": "The authors point out the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, this paper is the first to introduce sharpness-aware minimization into SSL, so the work is novel.\n- The experiments are wide-ranging and cover several different datasets. The experimental analysis is also relatively comprehensive.\n- The paper is well written and easy to follow. \n", "weaknesses": "1.The analysis of loss landscape is somewhat unclear and contradictory.\n- The paper states in lines 42-43: The learning on scarce labeled data convergences faster with lower errors  than on unlabeled data. However, in Figure 1, it can only be observed that the loss landscape of labeled data is sharper at both epoch 60 and epoch 150. If want to compare the convergence speed, maybe it is more intuitive to compare the change curve of loss in the time dimension.\n- The paper states in lines 44-46: the abundant unlabeled data…generalizing better than labeled data. However, the accuray of labeled data is better than that of unlabeled data in Figure 1 (right).\n- The paper states that using slight data augmentation to plot loss landscape is common in SSL. It is recommended to add relevant references to justify the drawing method.\nBased on the above questions, the reasonable of two critical flaws proposed in the paper needs further clarification.\n\n2.The paper states in lines 8-9 that FlatMatch minimizes cross-sharpness measures to ensure consistent learning performance between the two datasets. But further verification is lacking. Does the cross-sharpness measure achieve consistent learning performance? If yes, why is it possible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R4: We thank Reviewer Z9Pn for your helpful feedback. We have carefully addressed all your concerns as follows.\n\n*Q1*: **Why faster convergence leads to sharper loss curves and how to compare the convergence speed between labeled data and unlabeled data.**  \n*A1*: Good point. \n* Here we clarify that convergence does not lead to flatness: Converged accuracy or loss can only indicate first-order property. However, flatness stands for the relative loss under parameter change, it is a second-order statistic. When the labeled data lie in sharp minima, the sharpness could be large even though the loss is small. Therefore, our claim is not a contradiction with our empirical findings.  \n* To illustrate the convergence of loss curves, we show the loss values from both FlatMatch and FixMatch during training. Moreover, we subtract the loss value of unlabeled data from the loss of labeled data to compute a loss difference, which gives an illustration of the performance difference between both two datasets.  \n    * Firstly, the loss value of labeled data quickly converges to zero and is significantly smaller than unlabeled data. Such phenomenon occurs in two methods which support our claim that the learning on labeled data converges faster than unlabeled data.\n    * Secondly, the loss difference between the two datasets of FlatMatch is significantly smaller than FixMatch, which indicates that our FlatMatch can alleviate the unmatched convergence speed of two datasets and helps decrease the loss value of unlabeled data.\n* Moreover, to understand how FlatMatch affects the sharpness curve, we compare the sharpness between labeled data and unlabeled data by using FixMatch and FlatMatch. Similarly, we also compute a sharpness difference. The results are shown in Fig. 4 in the rebuttal pdf.\n    * We can see that the sharpness of FlatMatch on both two datasets is smaller and stabler than that of FixMatch.\n    * Moreover, the sharpness difference of FlatMatch is obviously smaller than FixMatch.\n    Hence, we can know that FlatMatch can successfully alleviate the flatness gap between two datasets, further leading to consistent learning performance.\n\n\n*Q2*: **Why the better generalization performance of unlabeled data shows inferior accuracy to labeled data.**  \n*A2*: Similar to the last concern, we want to address that the generalization performance is not equivalent to training accuracy.\n* As shown in Fig. 1 in the main paper, we can see that the loss curve of unlabeled data is smoother than that of labeled data, because the number of unlabeled data is much more than labeled data, thus containing abundant knowledge which benefits the generalization of unseen data. \n* However, as the supervision is absent, it is reasonable that the model is less accurate on unlabeled data than on labeled data.\n\n\n*Q3*: **Justification of using data augmentation for generating loss landscape.**   \n*A3*: Our choice of using data augmentation to demonstrate generalization performance is well-supported by existing studies. Engstrom et al. show that when training data is augmented, the loss landscape could be significantly non-concave [D1], thus exposing the limited generalization performance on slightly changed data. Moreover, Gotmare et al. [D2] also conduct data augmentation to plot the loss curves.\n\n[D1] Engstrom et al., Exploring the Landscape of Spatial Robustness, in ICML 2019.   \n[D2] Gotmare et al., Using Mode Connectivity for Loss Landscape Analysis, in ICMLW 2018.  \n\n\n\n*Q4*: **Verification of ensuring consistent learning performance between two datasets.**  \n*A4*: There are two major findings show that our FlatMatch can benefit consistent learning performance between labeled data and unlabeled data:  \n* The flatness on labeled data has been largely improved, leading to low sharpness on both labeled and unlabeled datasets. In Fig. 4 in the main paper, we have illustrated that our FlatMatch can smoothen the loss curves of labeled data by leveraging cross-sharpness minimization. As a result, the generalization performance on labeled data is improved, which can be supported by many studies regarding flatness [D3, D4, D5].\n* The error rate and sharpness gap between the two datasets have been effectively decreased. As shown in Figs. 3 and 4 in the rebuttal pdf, our FlatMatch can benefit both loss and sharpness minimization of unlabeled data compared to FixMatch. Therefore, the performance gap between the two datasets is effectively alleviated.  \nBased on the above reasons, our FlatMatch can benefit consistent learning.\n\n[D3] Neyshabur et al., Exploring generalization in deep learning, in NeurIPS 2017.  \n[D4] Foret et al., Sharpness-Aware Minimization for Efficiently Improving Generalization, in ICLR 2021.  \n[D5] Chaudhari et al., Entropy-sgd: Biasing gradient descent into wide valleys, in J. Stat. Mech. 2019.\n\n\n*Minor*: **1) function $\\ell_d$; 2) ablation study on sharpness with full dataset; and 3) Requirements on language/text task.**  \n*A*: \n* In line 154, we have stated that ''the function $\\ell_{d}(\\cdot)$ denotes the loss criterion for unlabeled data''.\n* We did not use $D^{u}$ (or $D^{l}$) when sharpness is computed on $D^{l}$ (or $D^{u}$). Thanks to the advice, we have conducted an ablation study on the full dataset as shown in Tab. 2 in the rebuttal pdf. We find that leveraging the sharpness of the full dataset cannot surpass the performance when only using labeled data, this could be caused by the inaccurate gradient signal from unlabeled data.  \n* We follow the most common SSL learning process in the community to evaluate our method. However, adapting to the text classification could lead to extremely huge work such as reformulating the data augmentation, re-designing the loss objectives, and re-training all the baseline methods. Hence, it is too challenging to fulfill this heavy requirement in such a short time."}, {"review_id": "jpEf1BxnfG", "reviewer": "Reviewer_9MiC", "summary": "Inspired by Sharpness-aware minimization (SAM) technique, this paper proposes a new semi-supervised learning method FlatMatch. The main idea is in Eqn.(4), where FlatMatch picks both $\\theta$ (current weight parameters) and $\\tilde{\\theta}$ (the parameters close to $\\theta$, and maximize the loss on labeled dataset), and then try to optimize $\\theta$, so that both $\\theta$ and $\\tilde{\\theta}$ agree with the unlabeled data sets. \nThe authors also propose an efficient version of FlatMatch called FlatMatch-e, that saves some computation. In experiments, the authors demonstrated that their method FlatMatch with fix label trick outperforms all the existing methods. ", "questions": "I only have one question: can you improve Table 1 to include the cases for other methods with fix label trick? ", "limitations": "I think limitation is not a main issue for this paper. ", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "I think it is an interesting idea to use SAM like techniques to solve semi-supervised learning methods. The presentation of this paper is fairly good, so I find the paper very easy to follow. The experiment section is fairly comprehensive. \n\nOriginality: good, because it is my first time hearing using this technique to solve the problem. \n\nQuality: poor, as I will mention below. \n\nSignificance: fair. If the authors can improve the experiments to validate the real improvement of their algorithm, I think it seems to be a nice contribution to the semi-supervised learning. \n", "weaknesses": "I think the experimental setting, the FlatMatch (fix label) part, has serious issue. \n\nSpecifically, I think the main motivation of semi-supervised learning, is to consider the case that the labeled dataset is fairly small, and the unlabeled dataset is huge. In that case, we hope to combine both the information of labeled data and unlabeled data to improve the performance of the model. \n\nHowever, in their setting, they mentioned that they pretrain a common SSL method, such as FixMatch for the dataset, and select high-confidence unlabeled data with pesudo labels, **convert them into labeled data. After that, they will use this \"expanded labeled dataset\" for their FlatMatch training**. Interestingly, although they claim they only \"slightly augmenting the number of labels\", they in fact added a huge set of labels: 500, 2500, 500, 5000 for CIFAR10, CIFAR100, SVHN, STL10, respectively. \n\nWithout this \"fix label\" trick, their method is not as competitive as the existing methods. \n\nI personally feel that this is not a fair comparison. If \"fix label\" trick is indeed helpful, I think the authors should at least compare FlatMatch with the existing methods with fix label as well. If all the other methods does not have any improvement after using fix label trick, then FlatMatch (fix label) will indeed look promising. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "R5: We thank Reviewer 9MiC for your feedback. Here we justify that our Fix label setting is a common SSL technique which is reasonably designed in our method.\n\n*Q1*: **Whether our Fix label setting violates the motivation of SSL.**  \n*A1*: We want to stress that transforming unlabeled data to labeled data is not a violation of SSL. \n* The proposed fix label strategy is essentially a well-known SSL technique: Pseudo label, which also selects highly confident examples as labeled data.\n* So, the Fix label setting can be considered as combining FlatMatch with Pseudo label, which just takes advantage of an SSL technique to solve the SSL problem.\n\n*Q2*: **Claim of ''slightly augmenting the number of labels''.**  \n*A2*: All the chosen number of labels in all settings is less than or equal to the largest ''# Label'' value in our experimental setting. Therefore, all of the expanded labels are less than or equal to 10% of the total number of training data.  \n**Note:** Particularly, we apologize that the number for STL10 is 5000 is a typo, which is actually **1000** number of labels. \n\n*Q3*: **Limited performance of FlatMatch without fix label trick.**  \n*A3*: We want to stress that the proposed FlatMatch method shows non-trivial performance improvement on most of the settings among various datasets.  \n* For example, FlatMatch achieves the best results on most of the settings, as listed below.\n\n| Dataset  | CIFAR10 | CIFAR10 | CIFAR10 | CIFAR10 | CIFAR100 | CIFAR100 | CIFAR100 | SVHN | SVHN | SVHN | STL10 | STL10 |\n|----------|---------|---------|---------|---------|----------|----------|----------|------|------|------|-------|-------|\n| # labels | 10      | 40      | 250     | 4000    | 400      | 2500     | 10000    | 40   | 250  | 1000 | 40    | 1000  |\n| Best?    |       |       | &check;     | &check;     |        | &check;      | &check;      |    | &check;  | &check;  |     | &check;   |   \n* Additionally, we have conducted additional experiments on the ImageNet dataset and ViT backbone and found that FlatMatch can still surpass strong baseline methods such as FreeMatch and FlexMatch. For details, please see the general response.\n\n\n*Q4*: **Comparison to other SSL methods with Fix label for fairness.**   \n*A4*: Thanks for the helpful advice, we have conducted experiments on several SSL methods with the Fix label strategy on the small-scale dataset CIFAR10 and large-scale dataset ImageNet, and we show the results below:\n\n\n| Dataset               |                CIFAR10               |                CIFAR10               |                CIFAR10               |                CIFAR10               |\n|-----------------------|:------------------------------------:|:------------------------------------:|:------------------------------------:|:------------------------------------:|\n| # Label               |                  10                  |                  40                  |                  250                 |                 4000                 |\n| FlexMatch (Fix label) |    14.24&pm;0.25    |      6.02&pm;0.73     |      5.36&pm;0.34     |     4.92&pm;0.29     |\n| FreeMatch (Fix label) |     9.39&pm;3.72    |      6.35&pm;2.30     |      5.34&pm;1.21     |     4.66&pm;0.53     |\n| FlatMatch (Fix label) | **7.36**&pm;5.62 | **4.28**&pm;1.61 | **3.90**&pm;1.72 | **3.55**&pm;0.64 |\n\n\n| Dataset               |    ImageNet    |\n|-----------------------|:--------------:|\n| # Label               |  100 per class |\n| FlexMatch (Fix label) |      43.06     |\n| FreeMatch (Fix label) |      41.25     |\n| FlatMatch (Fix label) | **38.01** |\n\nNote that the Fix label setting seems to only favor FlatMatch over other SSL methods, but there are reasonable explanations: Our FlatMatch only uses expended labeled data to compute the gradient, and our risk minimization is still conducted on the original labeled dataset and unlabeled dataset. As a result, there are two benefits:\n* The gradient is diversified by more labeled data, which makes the worst-case model vulnerable to a wide range of examples. In turn, our sharpness minimization can effectively make up for the vulnerability.\n* Our risk minimization is still based on the original dataset and does not introduce too many labels in the early stage of training. As a result, we can avoid introducing noisy labels.  \n\nOn the other hand, if use the expanded labels directly for risk minimization in SSL, as shown by the baseline results, the performance would not bring any improvement.\n"}]

Author rebuttal:
***General Response***:
We thank the reviewers for their insightful and constructive reviews of our manuscript. Delightfully, we are glad that the reviewers found that:
* Our idea is **novel, interesting, and well-motivated**. (Reviewers 9rMs, Z9Pn, 9MiC)
* The presentation of our paper is **clear and easy to follow**. (Reviewers 9rMs, yYJy, Z9Pn, and 9MiC)
* Our experiments are comprehensive and show **effective or efficient** performance. (Reviewers 9rMs, Y5aq, yYJy, Z9Pn, and 9MiC)


Based on all the comments from the Reviewers, here we provide a general response to the concerns raised by multiple reviewers. The individual responses are commented on below each review.

* **Regarding extension on large-scale datasets and sophisticated architectures.**
    * Although we have conducted experiments on ImageNet30 in the initial appendix, here we follow the advice to validate the performance of FlatMatch on full ImageNet, we choose only 100 labels per class and provide the test error results of FlatMatch, FlatMatch-e, FixMatch, FlexMatch, and FreeMatch. Additionally, we consider a sophisticated architecture such as Vision Transformer (ViT) as the training backbone and compared the proposed methods with several SSL methods. The results are shown below:  

        | Dataset  | Architecture     | FlatMatch | FlatMatch-e | FreeMatch | FlexMatch | FixMatch |
        |----------|------------------|:---------:|:-----------:|:---------:|:---------:|:--------:|
        | ImageNet | Wide-ResNet-28-2 |   **38.70**  |    39.92   |   40.57  |   41.95  |  43.66  |
        | ImageNet |  ViT-Base (86M)  |   **21.57**  |    22.07   |   23.55  |   23.78  |  25.52  |

        * We can see that on large-scale datasets such as ImageNet, the performances of both FlatMatch and FlatMatch-e are still superior to other components. 
        * Moreover, we can still observe the effectiveness of the two of our methods. Therefore, it is reasonable to conclude that the performance of FlatMatch is extendable to large-scale datasets and sophisticated architectures.

* **Regarding convergence analysis on accuracy, loss, and sharpness.**
    * We have carefully conducted experiments to analyze the convergence of accuracy and loss curves. 
    * We also made a demonstration of how FlatMatch affects the sharpness of labeled data and unlabeled data during training.

* **Fairness comparisons with other SSL methods on the same optimization and experimental setting.**
    * For the optimization strategy, we have carefully compared our methods with several SSL methods implemented with SAM to justify our design choice.
    * For experimental setting fairness, we have applied the Fix label setting to several SSL methods and compared their performance with our FlatMatch.

For other detailed concerns, we have tried our best to fully address every single concern raised by each reviewer. Thanks to all reviewers for your time in reviewing this paper, your help and support have largely improved this manuscript.


Meta-review: This is a semi-supervised learning paper. It is the next generation of a series of papers that rely on one type of consistency. Here, the authors propose a more general kind that attempts to match the loss landscapes for labeled and unlabeled data. 

There's two contributions here: (1) The authors propose a simple and nice idea. (2) The results are very strong. This is especially impressive since this area has had many innovations, all of which have made improvements on the standard set of benchmarks. The idea is also fairly flexible, since improvements to sharpness methods may also enable the paper's already strong results to improve (i.e., become more computationally efficient). 

The authors have additionally done a good job of answering reviewer questions. Many of the reviewers have substantially improved their scores as a result.
Decision: Accept (poster)