(FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning
Authors: Seungjoo Lee
Thanh-Long V. Le
Jaemin Shin
Sung-Ju Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Semi-Supervised Learning
Federated Semi-Supervised Learning
Primary area: other
TL;DR: Overcoming Few Labels in Federated Semi-Supervised Learning

Abstract: Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitive data. However, most FL approaches assume that clients possess labeled data, which is often not the case in practice. Federated Semi-Supervised Learning (FSSL) addresses this label deficiency problem, targeting situations where only the server has a small amount of labeled data while clients do not. However, a significant performance gap exists between Centralized Semi-Supervised Learning (SSL) and FSSL. This gap arises from confirmation bias, which is more pronounced in FSSL due to multiple local training epochs and the separation of labeled and unlabeled data. We propose $(FL)^2$, a robust training method for unlabeled clients using sharpness-aware consistency regularization. We show that regularizing the original pseudo-labeling loss is suboptimal, and hence we carefully select unlabeled samples for regularization. We further introduce client-specific adaptive thresholding and learning status-aware aggregation to adjust the training process based on the learning progress of each client. Our experiments on three benchmark datasets demonstrate that our approach significantly improves performance and bridges the gap with SSL, particularly in scenarios with scarce labeled data.

(FL);: Overcoming Few Labels in Federated Semi-Supervised Learning
Seungjoo Lee
Thanh-Long V. Le
Jaemin Shin
Sung-Ju Lee
KAIST
Republic of Korea
(seungjoo. lee, thanhlong0780, jammin thin,profse Skaist ac.
Abstract
Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitivy data. However, most FL approaches assume that clients possess labeled data, which is often not the case practice. Federated Semi-Supervised Learning (FSSL) addresses this label deficiency problem, targeting situations where only the server has a small amount of labeled data while clients do not. However, a significant performance gap exists between Centralized Semi-Supervised Learning (SSL) and FSSL. This gap arises from confirmation bias, which is more pronounced in FSSL due to multiple local training epochs and the separation of labeled and unlabeled data. We propose (FL), a robust training method for unlabeled clients using sharpness-awar consistency regularization. We show that regularizing the original pscudo-labeling loss is suboptimal, and hence we carefully select unlabeled samples for regularization. We further introduce client-specific adaptive thresholding and learning status- aware aggregation to adjust the training process based on the learning progress of each client. Our experiments three benchmark datasets demonstrate that our approach significantly improves performance and bridges the gap with SSL, particularly in scenarios with scarce labeled data. The source code is available https: / //g thub. com/seungj oo-jooo-o/////aii D IPS24
Introduction
Federated learning (FL) [1] distributed machine learning system that trains accurate global models while preserving clients" privacy-sensitive data. Each FL client trains its local model on their device using only their data, and the server aggregates these local models into global model. As result, clients' private data protected as only the local models' weights are shared with the server.
Because of its privacy-preserving nature, FL has garnered recent attention, with efforts to make it reliable and efficient [2, 3, 4]. However, most previous FL studies assumed that clients have labeled data, which is unrealistic in practical settings for two reasons. First, clients are often reluctant or lack the motivation to label data. Second, certain data types require domain expertise during the labeling process [5, 6]. For example, labeling medical data demands specialized knowledge and expertise. Similarly, sensory data, which can have multiple dimensions, difficult for most clients to interpret accurately. Therefore, we envision a labels-at-server [7] scenario more realistic for FL., where small amount of labeled data is available only at the server while the clients' data remains unlabeled.
Various Federated Semi-Superviie Supervised Learning (FSSL) approaches [8, 7,9, 10, 11] have been developed for the labels-at-server scenario. However, there is substantial performance gap between FSSL and centralized Semi-Supervies Learning (SSL), particularly when labeled data is limited. Fig. illustrates this issue across varying amounts of labeled data on the CIFAR10 dataset [12]. When
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

100
- FreeMatch (SSU
SemiFL (FSSL)
80
Accuracy (%)
FedCon [FSSU]
60-
FedMatch [FSSL]
40
20
4000
250
Number labeled samples
Figure I: Comparison of SSL and FSSL algorithms on CIFAR-I0 with varying numbers of labeled samples, where FreeMatch [19] represents SSL, and SemiFL [8], FedCon [9], and FedMatch [7] represent FSSL.
sufficient amount of labeled data is available, the performance difference between SSL and FSSL is minimal. However, this gap widens considerably as the quantity of labeled data decreases.
We point out confirmation bias as the primary cause, where the model tends to overfit to easy-to-learn samples or incorrectly pseudo-labeled data [13]. This issue is particularly pronounced in FSSL as the training process involves multiple local epochs on clients [8, 14, 15J. This extended training on localized data accelerates the overfitting process, making the model more susceptible to confirmation bias. Moreover, labeled and unlabeled data are kept separate in a labels-at-sene server setting. Unlike in centralized SSL where labeled and unlabeled objectives could be jointly optimized, this separation in prevents effective co-optimization, further contributing to the performance gap.
We propose Few-Labels Federated semi-supervised Learning, abbreviated (FI.)2, mitigate confirmation bias in FSSL using (1) client client-specific adaptive thresholding, (2) sharpness aware consistency regularization, and (3) learning status-aware aggregation. Previous FSSL approaches [7, 8] use fixed threshold to obtain high cegh-confieecce pseudo-labels but are prone to confirmation bias as only small portion of data is utilized the early stages of training. Instead, the threshold according to the clients' learning status. In the early stages, we use a low threshold to include more data for training. As training progresses and the model becomes more confident, we increase the threshold to obtain more accurate pseudo-labels We profile the learning status of each client and determine specifcc adaptive thresholds.
adaptively change
Recently, Sharpness-Awee Aware Minimization (SAM) has demonstrated strong generalization capabilities across various tasks [16, 17, 18]. Inspired by this, hypothesized that applying SAM could effectively mitigate confirmation bias among clients. However, our findings revealed that naïvely applying SAM degrades performance. This issue occurs as SAM generalizes not only correctly pseudo-labeled samples, but also incorrectly pseudo-labeled ones. Generalization of incorrect data samples leads the propagation of errors, thereby degrading the model's performance. Therefore, we apply consistency regularization to carefully selected data samples that are highly likely correct. We also uncover that the standard SAM objective (i.e.. achieving flatter local minima) does not work well in FSSL. We thus propose novel consistency regularization between the model outputs of adversarially perturbed and original weight parameters.
Finally, we propose novel learning status-aware aggregation. In FSSL., the learning difficulty can vary across clients. Since the server can access only small labeled dataset, clients whose data closely resembles the server's data will face lower learning difficulty. In comparison, those with more distinct data will encounter higher difficulty. Additionally, due the non-iid data distribution of clients, the learning difficulty naturally differs among them. To account for different client learning difficulties, we assign higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. In contrast, previous FSSL approaches did not consider these variations in learning difficulty and relied on fixed aggregation weights.
Our main contributions are summarized as follows:
We propose client-specific adaptive threshold that adjusts the pseudo-labeling threshold accord- ing to each client's learning status. By using low threshold at the early stage training, we effectively reduce confirmation bias by utilizing more data.

---

We demonstrate that applying the SAM objective in FSSL is non-trivial and requires careful considerations. Minimizing the sharpness of incorrectly pseudo-labeled samples reduces the model performance. We also identify that the original SAM objective is ineffective in FSSL and propose a novel sharpness-awan consistency regularization that regularizes consistency between original and perturbed model outputs.
We propose learning status-aware aggregation that adjusts the weight based on the client's learning status. Clients with lower learning status receive higher aggregation weights, ensuring their updates are well reflected in the global model.
Our evaluation shows that our approach significantly outperforms existing methods across differ- ent settings, particularly when labeled data is scarce. (FL)² improves the classification accuracy of up to 23.0% compared with existing FSSL methods.
2 Related work
Semi-supervised learning (SSL. Recent SSL methods primarily stem from two key ideas: pseudo- labeling [20] and consistency regularization [21]. Pseudo-labeling artificially creates pseudo-labels and uses them as hard labels for supervised training. On the other hand, consistency regularization trains models by minimizing the variance between stochastic outputs, typically achieved through various weak or strong augmentations. FixMatch [22] generates high-quality pseudo labels via static probability thresholding and trains models to predict these labels from strongly augmented inputs. FlexMatch [23] enhances this approach by incorporating class-specific local thresholds alongside fixed global threshold, adjusting based on the model's learning status. Free:atth [19] dynamically adjusts the confidence threshold according to the model's 's learning status and introduces self-adaptive class fairesss regularization penalty to encourage diverse predictions during early training. FlatMatch [24] increases generalization by adopting sharpness-awarenes. minimization [25] into cross-sharpness measure in SSL settings ensure consistent learning performance between the labeled and unlabeled data.
Federated semi-supervised learning (FSSL) Federated Learning (FI.) enables collaborative training of a global model while ensuring data remains on the client side, thereby preserving data privacy (further discussed in Appendix D). FSSL leverages FL in scenarios where labeled data is limited. FSSL research addresses two primary settings: labels-at-clients [26, 27, 7, 28, 29] and labels-at-server [9, 7, 8J. In the labels-at-server scenario, FedMatch [7] encourages similar outputs from similar clients using inter-client consistency loss. It employs disjoint training between the server and clients to mitigate forgetting issues. FedCon [9] utilizes contrastive learning to assist clients' networks in learning embedding projections. SemiFL [8] achieves state-of of-the-art results in the label-at-server setting by introducing alternate training, which finetunes the global model with labeled data after each communication round. It generates pseudo-labels with the global model the start of every communication round, rather than the common per-batch generation.
Real-world mubels-at-eerve FL scenarios to have extremely limited labeled data as labeling data requires domain expertise and could be costly [5. 6]. However, existing FSSL approaches target scenarios with hundreds of labeled data points (> 250) the server, and their accuracy significantly deteriorates when only tens labeled data points are available (Section 5.2). In contrast, (FL)² achieves high accuracy
in extremely label-scarce settings, such as when only 10 labeled data points are available on the server, demonstrating increased usability and practicality for real-world applications.
Preliminaries
3.1 Federated learning
Federated Learning (FL) collaboratively trains a global model via coordinated communication with multiple clients. communication round f, the server selects K clients among available clients. The server transmits the current global model weights Wt to selected clients. The selected clients update the model weight W with the local dataset for E epochs, where k indicates the client index. Formally, W W V/ Colient. where Lelient denotes the objective function of clients, e.g..
3

---

cross-entropy loss for the classification task. After local training, the server aggregates the trained model weights with 3 aggregation weight of each client, which is
W
W
(I)
3.2 Federated semi-supervised learning
In Federated Semi-Supervised Learning (FSSL), especially in the labels-at-server scenario, labeled dataset D {1 y6) b [Nu]} is only available at the server, while clients have only unlabeled dataset D = {us : b E [ND], where Nr and Nv = Ek= Ný are the total number of labeled data and unlabeled data, respectively. In general, Nr < Nu. At each communication round t, the server updates its model weight W with supervised loss Leerver for E local epochs with
B
Lserver
Σ
H(ye. Pwy (y)w(x)))), W W V/ Ceerver.
where data batch (ar.. yb) is randomly drawn from Dÿ with batch size B. H((,)) refers the cross- entropy loss, w..) ) is the weak data augmentation (e.g., random horizontal flip and crop), and (-) is the output probability from model W. Clients update their model weight W using cross-entropy loss with ppeudo-labeling which can be expressed
BB
Lelient
Σ
(max(()) 7) H(gb, Q6), W W.r v/ W Selient;
μ.
where d and Qu are the abbreviations of Pw: (ulw(us)) and Pw: (y) (g)s((u)) respectively. Data batch uy randomly selected from DIj with batch size of B. The one-hot label form of qu is denoted as lb, and the ratio of data with confidence above is represented by μ.. The indicator function 1 r) is used for confidence-based thresholding. 12(-) represents strong augmentation (c.g.. RandAugment [30]).
We adopt "fine-tune global model with labeled data" and "generate pseudo-labels with global model" strategies from SemiFL [8]. In communication round f, the server distributes the current global model W K selected clients. Before training, clients generate pseudo-labels for local dataset with fixed global model Wy. The changed local objective function is
BB
Celient
1 (max (g,) >T H Qs),
BB
b=1
where oi stands for pwy (ylw(u))) Subsequently, the server aggregates trained local models with Eq 1. The server fine-tunes the aggregated model with Laeverrs yielding new global model W/9 1+1'
3.3 Sharpness-awark minimization
Sharpness-Awar Minimization (SAM) [25, 31] has been increasingly applied to various tasks [16, 17, 18] due to its ability to enhance generalization. SAM improves generalization by minimizing the sharpness of the loss landscape, which helps finding flatter minima that generalize better across different tasks and datasets. Traditional optimization methods could lead to sharp minima, resulting in poor generalization unseen data. SAM addresses this issue by incorporating weight perturbations into the optimization objective to find flatter minima. The core objective of SAM is defined as:
min max Lwtees ||e||
where e is a perturbation vector constrained within a p-ball around the model weight w. The inner maximization seeks to find the perturbation that maximizes the loss L within the specified p-ball.
To make this optimization feasible, SAM approximates the perturbation as:
VuL.
=
| Lull
(6)

---

Small labeled dataset
Server
Learning statinngtttt---aaa status-aware aggregation
Clinnt local training
W
W
W.&k
Server training
W [+]
modell
Distribute
Client
WF?
W
① Client-specific adaptive thresholding
CLIENT
Global model
W
Inn__ Prediction
Learning status profiling
Adaptive threshold
2
Sharpness-awe consistency regularization
Wesldy sugnented samples
W
High-conffdent pscuda label
Prediction
dolenn Prediction
H-!
Cross-entropy
Local unlabeled dataset
Weight perturbation
Strongly augmented samples
Ca(is))
Sharpness-aware consistency loss
dinto
Perturbed weight
Prediction
Figure 2: Overview (FL):: (1) client-specific adaptive thresholding adjusts the pseudo-labeling threshold according to each client's 's learning status, (2) sharpness-aware consislency regularization ensures consistency between the original model and the adversarially perturbed model with carefully selected high-confident pscudo labels, and (3) learning status-aud aware aggregation aggregates client models considering each client's learning progress.
This approximation simplifies the inner maximization by scaling the gradient direction to have a norm of p. The outer minimization updates the weights using the gradient evaluated the perturbed weights wo + e Specifically, the gradient used for the weight update is V whowter
4 Method
Few-Labels Federated semi-supervised Learning, abbreviated as (FI.)2, has three key components: (1) client-specffic adaptive thresholding, which leverages more unlabeled data by dynamically adjusting thresholds for pseudo-labeling, (2) sharpness-aware consistency regularization, which minimizes sharpness for carefully selected data to ensure better generalization, and (3) learning status-aware aggregation, which aggregates local models from clients while considering their learning progress. Fig. 2 overviews (FL)² and Appendix details the algorithm.
4.1 Client-specific adaptive thresholding
We use an adaptive thresholding mechanism rather than fixed threshold to incorporate more unlabeled data from the beginning of training. This approach is inspired by FreeMatch [19] that gradually increases the threshold according to the model learning status. At round 4, each client profiles its learning status during the pseudo-label generation stage using local unlabeled dataset D with global model W. Adaptive threshold Th of client & at round f is
p
max (
(7)
D
5

---

where of is qo calculated with global model W... This approach sets low initial threshold value, as the model exhibits lower confidence in the data the beginning of training. The threshold gradually increases as training progresses, allowing the model to focus on high-confidence data. Additionally, we estimate the learning status specific to each class and apply different thresholds for each class. This is achieved by utilizing the output probabilities of the global model's predictions for each class, which can be expressed as
D Σ C
1 (c)
(8)
D
We calculate client-specific adaptive thresholds for each class using of and p (c)
as
(c) (c) = MaxNorm(i?(c) - (ax :ce IC]]
+..
The unsupervised training objective La of client k with adaptive thresholding at each iteration is:
C
1(max (g() > Tr. (argmaxx(())) H(G Qo).
(10)
4.2 Sharpness-aware consistency regularization
While Sharpness-Aware Minimization (SAM) generalizes well in many tasks [16, 17, 18), it is not trivial to apply it to FSSL,, as SAM generalizes not only correctly pseudo-labeled samples but also incorrect samples. This indiscriminate generalization results in the propagation of errors, thereby degrading the model's performance (Section 5.4). To tackle this issue, we apply consistency regularization to carefully curated subset of data samples with high confidence of correctness. While we use client-specific adaptive threshold (Section 4.1), we use a high fixed threshold to get high-confidence data samples. (FL)² adversarially perturbs the weight parameters that maximize loss calculated with high-confidence data samples and regularizes consistency using the perturbed weight.
Adversarial weight perturbation When a client k trains its local model W with mini- batch, the model weight is perturbed with that increases 2 the most, where and C are defined as
aB
c
1 (max(gg) TT) H(G Q6),
(11)
BB b=
V Wx
argmax
Ck Srr
W W + E..
(12)
I
V
where p stands for perturbation strength. We use large fixed threshold Ty get a high-confidence pseudo-label.
Consistency regulation With the perturbed weight W we calculate Qi, which is the output probability of a strongly augmented sample for Wik Unlike traditional SAM objective that takes Vw Lp. we measure the difference of model outputs between the original and the perturbed models (Section 5.5). Formally,
ch = Ea(ii Q6), where Q = Pw (y)(()))(
(13)
in which (a(;,)) measures the distance (e.g., L.2 distance or KL divergence). Finally, local training objective of client & with client-specific adaptive thresholding (Section 4.1) and sharpness-aware consistency regularization is
C Celient C. + Wes C.
(14)
with Wa and Wes being the loss weights. (FI.)2 effectively leverages both low-confidence data with client-speciicc adaptive threshold and high-confidence data with sharpness-aware consistency regularization to minimize the confirmation bias of clients.

---

4.3 Learning status-aware aggregation
After the local training of the selected K clients, the server aggregates the trained local models using weights 8h, as shown in Eq. 1. While existing FSSL approaches use uniform weights (@* 1 K) we propose a learning status-aware aggregation that adjusts the aggregation weight based on the client's s learning status. For a client with low learning status, indicated by a low adaptive threshold T we increase the aggregation weight so that the local learning is better reflected in the global model. We calculate the aggregation weight as
(15)
D 7()
Our aggregation method complements the client- -peccfci adaptive thresholds (Section 4.1). this scheme, we use lower thresholds for clients with a lower learning status to enable more extensive learning from their data. By extending this notion to the client level, clients with lower thresholds, which indicate more valuable learning updates, given greater influence the global model. This ensures that the most informative updates are prioritized.
5 Experiments
5.1 Setup
Data setup We evaluate (FL)² in three public datasets: CIFARIO, CIFARIO0 [12], and SVHN [32]. We test method under balanced IID and unbalanced non-IID -ID data distribution settings. Each client receives an equal amount of unlabeled data in the balanced IID setting. We sample data using Dirichlet distribution Dir(a) for the unbalanced non-III setting. Each client receives different number of data samples and samples per class. As a + oo, the distribution approaches IID. We set {0.1, 0.3) in our experiments. The number of labeled data samples the server (NVr, set to {10, for CIFARIO, {100, 4003 for CIFARI 100, and {40, 250} for SVHN, following widely-used evaluation settings for SSL [19, 24].
Learning setup In our experiments, we use 100 clients, with a participation ratio of 0.1 per communication round (K 10). We adopt the WideResNet [33] NedeRessNer28x2 for the CIFAR10 and SVHN datasets, and Nede for the CIFAR100 dataset. Both the server and clients optimize their local datasets for five local epochs, with 800 communication rounds. We employ the momentum SGD optimizer with a learning rate of 0.03, momentum of 0.9, and weight decay of 5e-4, following previous work [8]. For sharpness-aware consistency regularization (Section 4.2), we use the KL-divergence loss function for fa(;)). ). For adversarial weight perturbation (Eq. 12). we use ASAM [31], which implements scale invariance on standard SAM [25]. Based on grid search, the perturbation strength is set to 0.1 for the CIFARI0 10 and SVHN datasets and 1.0 for the CIFAR100 dataset. For strong data augmentation, we use Rand/Augeet [30]. We also adopt the static Batch Normalization (sBN) [34] strategy, as utilized in SemiFL. Further details on sBN are in Appendix We used RTX3090 GPUs throughout the experiment. Additional details are in Appendix C.
backbone, employing
5.2 Performance comparison with FSSL algorithms
We evaluate (FL.) against existing FSSL methods: FedMatch [7], FedCon [9], and SemiFL. [8]. Table shows that (FL,)2 consistently delivers the best or nearly the best performance across all settings. For instance, although SemiFL performs the best in the non-IID-0.3 setting of CIFAR100 with 100 labels, it struggles to generalize to other scenarios. SemiFL achieves only around 10% accuracy in CIFARIO with 10 labels and about 43% accuracy in SVHN with 250 labels. In contrast, (FL)2 consistently maintains high accuracy across all tasks. The performance gap compared with the berf-performing algorithm (SemiFL) in non- non-III--0.3 ///.IIAAAAOOOOOOOOO00 that, (FL)² consistently outperforms the baseline methods across all other settings. Additionally, (FL)² demonstrates substantial improvement over existing methods, achieving 20.3% higher performance in non- 3//3////////--/beelllll and 23.0% % higher performance in IID/SVHN/SS00labeet These findings indicate that (FL)2 effectively mitigates confirmation bias among clients, leading to robust generalization even with limited data across different settings.
is only 0.3%. Except for
7

---

Table I: Evaluation of (FL)² compared with existing FSSL methods. We report the average aceu- racy(%) and standard deviation across three runs with different random seeds. (FL)² shows significant performance improvements over existing methods across different settings. Bold indicates the best result and underline indicates the second-best result.
Dataset
CITARIO
SVHN
CIFAR100
# of labeled data sumples (N₂)
10
40
40
250
100
400
FedMatch
16.0((.3)
25.6(2.2)
20.7(2.7)
70.1(222)
6.30.3)
10.0(1.8)
Unhalanced Non-IID,
FedCon
16.6(2.1)
25.4(2.3)
20.5(1.4) ())
73.1(2.0)
4.0(0.4)
8:2(0.6)
1)
SemiFL
10.0K0.0)
19.9(7.5)
18.0(2.))
82.3(18)
9.8(2.4)
13,5(5.0)
(FL)²
19.2(5.7)
36.4(1.44
21.3(3.3)
88.0(1.0)
10.4(L.3)
23.5(1.2)
FedMatch
15.3(1.3)
25.2(3.5)
22.3(0.7)
72.3(30)
5.5(1.5)
9.8(1.1)
Unbalanced Non-IID, Dir(0.3)
FedCon
16.9(2.4)
26.5(2.1)
21.6(1.7)
68.7(2.7)
5.8(0.)
13.3(0.9)
SemiFL
10.0(0.0)
38.0(2.7)
26.3(2.5)
42.7(40.1)
12.4(1.2)
18,9(9.7)
(FL)²
24.3(4.5)
43.5(7.5)
31.0(4.)
92.6(0.5)
12.1(1.1)
25.4(1.0)
FodMatch
16.2(1.9)
25.4(2.8)
18.4(4.7)
66.2(0.8)
6.4(0.6)
10.0(1.7)
FedCon
16.7(2.0)
23.3(6.2)
20.3(1.0)
71.6(1.5)
5.7(0.6)
12.4(1.6)
Balanced IID
Semil'L
10.0K0.0)
75.3(2.8)
53.4(13.))
43.3(41.0)
13.9(3.3)
27.9(6.1)
(FL)
38.9(11.1)
81.5(7.4)
75.3(2.4)
94.6(1.1)
14.4(2.3)
28.1(2.2)
We emphasize that (FL.) significantly outperforms other methods when labeled data is extremely limited: by 22.2% % on the IID setting of CIFARIO with 10 labels and by 21.9% on the IID setting of SVHN with 40 labels. This substantial margin highlights (FL)²'s exceptional ability leverage scarce labeled data, making it practical for real-world federated learning applications. Additional experiments are provided in Appendix B.
Table 2: Contribution of each component of (FL.² on the SVHN dataset (N, = 40, balanced IID). By applying Client-specific Adaptive Thresholding (CAT) and Sharpness- Aware Consistency Regulariza- tion (SACR) to the baseline (FixMatch + FedAvg), performance is boosted. The combination of CAT and SACR further improves the accuracy. Incorporating Learning Status-Aware Aggregation (LSAA) leads to the best performance, finally achieving (FL)². The result demonstrates the importance of each component (FL)².
Algorithm
Accuracy
FixMatch + FedAvg
50.2
SACR + FixMatch + FedAvg
60.9
CAT FedAvg
68.2
CAT SACR FedAvg
71.7
(FL):: CAT SACR + LSAA
73.2
Significance of each component of (FL)2 We assess the contribution of each component of (FL)²: Client-specific Adaptive Thresholding (CAT), Sharpness- Aware Consistency Regularization (SACR), and Learning Status-Aware Aggregation (LSAA) in Table 2. The accuracy improvements provided by each component are evaluated using the SVHN dataset with 40 labeled data points and balanced IID setting. We use FixMatch + FedAvg as the baseline, where FixMatch [22] employs a fixed threshold for pscudo-labeling. Our results indicate that both SACR and CAT significantly enhance the performance over the baseline. Combining SACR and CAT yields further accuracy improvements. Finally, integrating LSAA for model aggregation, equivalent to (FL)², achieves the highest accuracy. These findings demonstrate that each component of (FL)² contributes uniquely and complementarily to the overall performance.
5.3 Effect of (FI.)2 on confirmation bias
Since incorrect pseudo-labels usually lead to confirmation bias [35], we evaluated pseudo-label accuracy, label ratio, correct label ratio, wrong label ratio, and C/W ratio in addition to test accuracy. We compared (FL)² against bascline methods using the SVHN datasct with 40 labels in balanced
8

---

80
100
70
70
80
50
80
40
"PUS LAAR
Istel
40
20
10
100
200
300
400
500
600
700
800
100
200
300
400
900
400
700
800
100
200
300
400
500
600
700
no0
Communkotim
Communtratio
Communiaaiion muni
(a) Test accuracy.
(b) Pseudo label accuracy.
(c) Pseudo label ratio.
70
Conest labe adee 99
60
40
C/W
80
10
10
200
800 Communictiion murd
600
500
700
300
600
800
200 300
600 700
Communication mondd
Communication mund
(d) Correct label ratio.
(e) Wrong label ratio.
(l) C/W ratio.
Figure 3: Comparison of SemiFL. (FL)2, and its variants on the SVHN dataset (N₂ = 40, balanced IID). Pseudo-label accuracy measures the percentage of correct pscudo-labels. The label ratio is the proportion of pseudo-labeled samples among all unlabeled data. Correct and wrong label ratios indicate the percentages of correctly and incorrectly labeled samples, respectively. The C/W ratio shows the number of correct labels relative to wrong labels. All subgraphs share the legend of Fig. 3a.
IID setting, reported in Fig. 3. A high pseudo-label accuracy indicates that the method produces reliable pscudo-labels. A high correct label ratio suggests that the method supplies the model with more accurate labels. Conversely, low wrong label ratio indicates that the model encounters fewer incorrect labels, which crucial for minimizing confirmation bias [35]. Lastly, a high C/W ratio signifies that the model is exposed to more correct labels than incorrect ones, further helping to reduce confirmation bias.
We observed that (FL.)2 consistently outperforms SemiFl. across all metrics. While SemiFL. generates more incorrect labels (C/W ratio < (F1.)2 produces twice as many correct labels than incorrect ones (Fig. 3f). Additionally, the wrong label ratio for (FL)² is approximately 30%, significantly lower than SemiFL's s 45% (Fig. 3e). These results suggest that (FL)² effectively reduces incorrect pseudo-labels while increasing correct ones, thereby mitigating confirmation bias. Furthermore, observe the effectiveness
each component of (FL)", which are CAT, SACR, and LSAA. Using CAT and SACR alone delivers better performance than the baseline for all metrics. If SACR, pseudo label accuracy increases, correct label ratio increases, and wrong label ratio decreases, which means reduce the confirmation bias. When LSAA is added, which is (FL)², it achieves the best performance across all metrics. This suggests that the synergistic effect of CAT, SACR, and I.AA effectively reduces confirmation bias.
CAT +
5.4 Impact of incorrect pseudo-labels on sharpness-aware consistency regularization
We investigate the impact of incorrect pscudo-labeles data on Sharpness-Awar Consistency Reg- ularization (SACR). We compare the performance of SACR in two scenarios: when applied only to correctly pseudo-labeled data assuming that we know the ground truth labels to assess the upper bound of SACR, and when applied all lseude-labeled data, including incorrectly pseudo-labelae samples. We examine when Client-specific Adaptive Thresholding (CAT) is used in both scenarios.
Fig. 4 reports the test accuracy and pseudo-label accuracy for the following cases: CAT alone, CAT+SACR (all data), and CÁT+SACR (only correct pseudo-lahels). CAT+SACR (only correct pscudo-labels) achieves high pseudo-label accuracy, indicating that SACR can effectively reduce

---

80
accurccy (40
80
Text acp.rtt B0
60
40
+0
CAT+SACR (AI cata)
CAT-SACR (AII doaa
20
CATTSACK jOnly cornect pacuco bood
20
CAT-SACR (Drll correct pooude lebel CAT-SAM IO coneec coneec preuco--boel
CAT+SAM iOnly correct poe peuld-abbll)
100 200
300
400
500
600
700
803
100
230
300
400
500
600 700 800
Communiccaion round
Communication round
(a) Test accuracy.
(b) Pscudo-label accuracy.
Figure 4: Test accuracy and pseudo-label accuracy on the CIFAR10 dataset with 40 labels, bal- anced IID setting. Client-specific Adaptive Thresholding (CAT) is used the baseline. Applying Sharpness-aware Consistency Regularization (SACR) to all data, including wrongly pseudo-labeled data, degrades performance than using only CAT, while applying SACR to correctly labeled data improves performance. SACR also outperforms the standard SAM objective (CAT+SAM).
confirmation bias when applied correctly pseudo-labeled data. Conversely, when SACR is applied to all data, including wrongly pseudo-labeled samples, the performance significantly decreases and shows worse performance than using only CAT. This emphasizes the importance of applying SACR exclusively to carefully selected data samples that are highly likely to be correct.
5.5 Comparison with the standard SAM objective
We compare the proposed Sharpness-aware Consistency Regularization (SACR) to the standard Sharpness-Awarn Minimization (SAM) objective. Both SAM and SACR perturb the model to maximize the given loss function. However, in SACR, the distance between the model outputs of perturbed and original model weights is minimized, while SAM takes the gradient of the given loss function at the perturbed weights.
Fig. 4 shows the test and pseudo-label accuracy using the standard SAM objective versus SACR. We examine the effects of SAM and SACR when applied only to correctly labeled samples in conjunction with Client-specific Adaptive Thresholding (CAT). Although SAM improves the performance over standalone CAT. SACR outperforms the standard SAM in convergence speed and final accuracy. The effectiveness of SACR can be attributed to the fundamental differences between SAM and SACR. SAM explores the given loss landscape in search of flat local minima. In contrast, SACR changes the loss landscape by explicitly incorporating an additional consistency regularization term.
Discussion and conclusion
We introduced novel federated learning algorithm, Few-Labels Federated semi-supervised Learning, (FLJ², that addresses the challenge of few-labels settings in Federated Semi-Supervised Learn- ing (FSSL) for unlabeled clients. (FL)² effectively reduces the confirmation bias through three key strategies: (1) client-specific adaptive thresholding, which adjusts the pseudo-labeling threshold based on each client's learning status; (2) sharpness-aware consistency regularization, which ensures consistency between the original and the adversarially perturbed models with carefully selected aggregation, which incorporates each client's 's learning progress into the aggregation of client models. (FL)² closes the performance gap between SSL. and FSSI,, making FSSL. an effective solution for practical scenarios.
high-confidence pseudo labels; and (3) learning
Limitations and future work Our approach introduces additional computational demands on clients, as client-specifi adaptive thresholding generates more pseudo-labels than traditional fixed threshold methods. Furthermore, sharpness-aves step with a perturbed model, increasing the computational burden. While our study is grounded in empirical findings, promising future direction is to theoretically analyze the impact of the proposed methods, particularly understanding how the generalization of incorrectly pseudo-labell labeled data affects overall performance.
consistency regularization adds an extra inference
10

---

Acknowledgments and Disclosure of Funding
This work was funded by the National Research Foundation of Korea (NRF), funded by the Ministry of Science and ICT (MSIT) under grant RS-2024-00444269 and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2024-00337007), * MSIT: Ministry of Science and ICT.
References
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication ceamumicatiooneffiieettt learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[2] Jaemin Shin, Yuanchun Li, Yunxin Liu, and Sung-Ju Lee. Fedbalancer: data and pace control for efficient federated learning on heterogeneous clients. Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, pages 436-449, 2022.
[3] Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learning via guided participant selection. In 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21), pages 19-35, 2021.
[4] Felix Sattler, Simon Wiedemann, Klaus-Rober Müller, and Wojciech Samek. Robust and communication-cffiiice federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 31(9):300-3443,, 2019.
[5] Dong Yang, Ziyue Xu, Wenqi Li, Andriy Myronenko, Holger R Roth, Stephanie Harmon, Sheng Xu, Baris Turkbey, Evrim Turkbey, Xiaosong Wang, et al. Federated semi-supervised learning for covid region segmentation in chest et using multi-national data from china, italy, japan. Medical image analysis, 70: 101992, 2021.
[6] Ahmet M Elbir, Burak Soner, Sinem Çöleri. Deniz Gündüz, and Mehdi Bennis. Federated learning in vehicular networks. In 2022 IEEE International Mediterranean Conference on Communications and Networking (MeditCom), pages 72-77. IEEE, 2022.
[7] Wonyong Jeong, Jachong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency disjoint learning. In International Conference on Learning Representations, 2021.
[8] Enmao Diao, Jie Ding, and Vahid Tarokh. Semifl: Semi-supervised federated learning for unlabeled clients with alternate training. Advances in Neural Information Processing Systems, 35: 18771-188888, 2022.
[9] Zewei Long. Jiaqi Wang, Yaqing Wang, Houping Xiao, and Fenglong Ma. Fedcon: A contrastive framework for federated semi-supervised learning, arXiv preprint arXiv:2109, 100X::2009904533, 2021.
[10] Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. In 2021 IEEE International Conference on Big Data (Big Data), pages 1114-122... IEEE, 2021.
[11] Woojung Kim, Keondo Park, Kihyuk Sohn, Raphael Shu, and Hyung- Siuun--iin&o Kim. Federated semi-supervised learning with prototypical networks. arXiv preprint arXiv:2205. 13921, 2022.
[12] Krizhevsky Alex. Learning multiple layers of features from tiny images. https://www. cs. toronto.
20000/99 pdf. 2009.
[13] Khanh-Binh Nguyen and Joon-Sung Yang. Boosting semi-supervised learning by bridging high and low-confidence predictions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1028-1038, 2023.
[14] Xinchi Qiu, Yan Gao, Lorenzo Sani, Heng Pan, Wanru Zhao, Pedro PB Gusmao, Mina Alibeigi, Alex Iacob, and Nicholas D Lane. Fedanchor: Enhancing federated semi- supervied learning with label contrastive loss for unlabeled clients. arXiv preprint arXiv:2402. 10191, 2024.
[15] Gihun Lee, Minchan Jeong, Sangmook Kim, Jaehoon Oh, and Se- GouYoung Yun. Fedsol: Stabilized orthogonal learning with proximal restrictions in federated learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 12512 12522, June 2024.
11

---

[16] Momin Abbas, Quan Xiao, Lisha Chen, Pin- Yu Chen, and Tianyi Chen. Sharp-mamll Sharpness- aware model-agnostic meta learning. In International conference on machine learning, pages 10-32. PMLR, 2022.
[17] Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3769-3778 2023.
(18) Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12360, 2022.
[19] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang. Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. et al. Freematch: Self-adaptive thresh- olding for semi supi--sueee learning. In The Eleventh International Conference on Learning Representations, 2023.
[20] Dong-Hyun Lee. Pseudo-label: The simple and efficient comi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
[21] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural information processing systems, 27, 2014.
[22] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596 308, 2020.
[23] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 18408-18419 2021.
[24] Zhuo Huang, Li Shen, Jun Yu, Bo Han, and Tongliang Liu. Flatmatch: Bridging labeled data and unlabeled data with cross- sharpnssss for semi coni-suprrvise learning. Advances in Neural Information Processing Systems, 36, 2024.
[25] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awe aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[26] Xiaoxiao Liang. Yiqun Lin, Huazhu Fu, Lei Zhu, and Xiaomeng Li. Rsefed: Random sampling consensus federated semi-supervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10154-10163, 2022.
[27] Chenyou Fan, Junjie Hu, and Jianwei Huang. Private semi-supervised federated learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 2009-2015. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.
(28) Ming Li, Qingli Li, and Yan Wang. Class balanced adaptive pseudo labeling for federated supervupe learning. In 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16292-16301, 2023.
[29] Yonggang Zhang, Zhiqin Yang, Xinmei Tian, Nannan Wang. Tongliang Liu, and Bo Han. Robust training of federated models with extremely label deficiency. In The Twelfth International Conference Learning Representations, 2024.
[30] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition workshops, pages 702-703, 2020.
[31] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Muchine Learning, pages 5905-5914. PMLR, 2021.
[32] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
12

---

[33] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:1605.07146. 2016.
arXiv preprint
[34] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication effi cient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2021.
[35] Eric Arazo, Diego Ortego, Paul Albert, Noel E O' O'Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervine learning. In 2020 International joint conference neural networks (ICCNN), pages 1-8. IEEE, 2020.
[36] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for henchmarking machine learning algorithms. arXiv preprint arXiv: 1708.07747 2017.
[37] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[38] Hao Chen, Ran Tao, Yue Fan, Yidong Wang. Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj. and Marios Savvides. Softmatch: Addressing the quantity-quality tradeoff in semi- supervised learning. In The Eleventh International Conference on Learning Representations, 2023.
[39] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018
[40] Sashank Reddi, Zachary Charles. Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.0023 00095, 2020.
[41] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[42] Chamath Palihawadana, Nirmalic Wiratunga, Anjana Wijekoon, and Harsha Kalutarage. Fedsim: Similarity guided model aggregation for federated learning. Neurocomputing, 483:432-445, 2022.
[43] Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
[44] Ulyanov. Instance normalization: The missing ingredient for fast stylization. arXiv preprint 16Xi: 2016.
[45] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[46]
Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv:1607.064500 2016
13

---

(FL):: Overcoming Few Labels in Federated Semi-Supervised Learning Appendix
A
Algorithm
Algorithm (FL)²: Few-Labels Federated semi- semi-supervised Learning
I: Input: Small labeled dataset DJ. {( vo) b E [Nu]] at server. Unlabeled dataset Dij = {us E [N[]], m E [M] distributed over clients. is fixed threshold. Be client batch size. w--) indicates weak data augmentation, and 92(+) indicates strong data augmentation. :(. indicates cross-entropy loss. La(:,-) -) is KL-divergence loss.
2: Initialize global model weight W
3: for each communication round do
4: W ServerUpdate D
Supervised server update with D
5: Update sBN statisties
6: Server samples clients K E [M]
7: Server broadcasts W7 to selected K clients
8:
for each client i E [K] do parallel
9:
T Adaptive ThanptiivTTrresood(WWW.. Dt)
10:
W < W
11:
for each local step do
Sample B, sized batch us from D
13:
of Pwy (y (w)))
14:
T - OneHot(gg)
15:
Q - Pw: (y)s(ub))
Lk uBe ExEl (max(()) > T (argmax((x)) H(E. Q6)
C uB. Ebzn 1 (max (1)) T₁) H(G. Qτ)
Vw c
18:
W + wh + 11 we Cillaa
Q - Pw:- (v))((u))
20:
C Ea(Qr, Q6) W + w v we (Wes Lh - w w.
21:
22:
end for
end for
24: Clients uploads Wh, Th server
25: BK
26: W > W
27:
end for
B More experiment results
We conducted additional experiments on the Fashion-MNIST [36] and AGNews [37] datasets, and the result is shown in Table 3. For FasHion-MNIST, we used the WideResNet28x2 architecture, consistent with the SVHN and CIFAR-10 experiments. We compared (FL)² with the previous state-of-the-art SemiFL. When trained with only 40 labeled samples, SemiFL failed in all three runs under the
14

---

Table More evaluation results of (FL)² compared with SemiFL on Fashion-MNIST and AGNews dataset. We report the average accuracy(%) and standard deviation across three runs with different random seeds.
Dataset
Fashion-MNIST
AGNews
# of labeled data samples (Nt.
40
20
Unbalanced Non-IID,
SemiFL.
12.8(4.8)
59.1(13.7)
Dir(0.3)
(FL)²
63.2(0.5)
73.6(3.7)
SemiFL (FL)²
10.0(0.0) 49.8(34.5)
47.4(14.3)
Balanced IID
87.0(0.6)
One run failed, resulting in only 10% accuracy. while the other two runs achieved accuracies of 69.0% and 70.4%.
balanced IID setting and in two out of three runs under the non-IID-0.3 setting, resulting in accuracies around 10%. In the single successful run under non- Non--II-O.3, SemiFL achieved an accuracy of 18.4%. In contrast, (FI.)2 successfully trained in all three runs under non-IIID-0.3 and in out of three runs under balanced IID. In the one failed balanced III run, the accuracy dropped to around 10%, while in the successful runs, it reached 69% and 70.4%. On average, (PL.)² achieved 63.2% accuracy under the non-IID-O.3 setting, demonstrating its robustness and effectiveness even with minimal labeled data.
For the AGNews dataset, we randomly sampled 12,500 training samples per class from a total of 50,000 samples and applied muck-translation for strong data augmentation, following the Soft- Match [38] approach. We used the bast-base-uccased [39] model as the backbone, freezing the BERT parameters and training only the linear classifier for 20 epochs. Since the mixup loss cannot be applied to NL.P datasets, we compared (F1.)2 to SemiFL. without the mixup loss. (F1.)² significantly outperformed the baseline, with a 39.6% accuracy improvement under the balanced IID setting and a 14.5% improvement under the non-IID-0.3 setting. With only 20 labeled samples, SemiFL showed substantial performance variability, with standard deviations of 14.3 and 13.7 for the IID and non-IID-0.3 settings, respectively. In contrast, (FL)² delivered more consistent results, achieving standard deviations of 0.6 for IID and 3.7 for non-IID-0.3.
C Details of learning setup
All experimental results for FedMatch and FedCon were reproduced using the official PyTorch implementation of FedCon wwweer-lonn//ceco-ptttooooo which is included in our repository. For SemiFL and (FL)² results, we implemented our own pipeline based on the GitHub repository for SemiFL
Sinlabeled-Clintte
-raining-
In Table 4, we list the hyperparameters used in the experiments. We utilized SGD as our optimizer and cosine learning rate decay as our scheduler. Additionally, we adapted the principles of adaptive federated optimization [40] into our FedAvg algorithm by introducing a FedAvg optimizer. Instead of simply aggregating the local models' weights from clients and using this as the new global model's weights, as done in FedAvg, we calculated the difference between the aggregated local models' weights and the global model's weights. This difference was treated as the gradient of the global model's weights, which was then used to optimize the global model through the FedAvg optimizer. We set 3 = 0.9 for the local optimizer and 3, = 0.5 for the FedAvg optimizer.
For training with labeled data at the server, we used the standard supervised loss. For local training at unlabeled clients, our objective function was a weighted sum of the unsupervised loss, the fairness loss (from client-specific adaptive thresholding), and the consistency loss (from sharpness-award regularization), with loss weights of wa = 1, and wes regularization, we used p 0.5 and Ty = 0.95. We also used an unlabeled batch size of 32, except for SemiFl., where training became unstable with this batch size, so we opted for batch size of 10 as in the original paper.
respectively. For sharpness-aware
15

---

Table 4: Hyperparameters in our experiments
Method
FedMatch [7]
FedCon
[9]
SemiFL. [8] (F1.)
Batch size
10
Epoch
5
Optimizer
SGD
Server
Learning rate
Weight decay
0.0005
Momentum
0.9
Nesterov
Epoch
Optimizer
SGD
Learning rate
0.03
Weight decay
0.0005
Momentum
0.9
Client
Nesterov
Batch size
32
32
10
32
Unsupervised loss weight (w%,)
N/A
N/A
N/A
1.0
Consistency loss weight (Wess
N/A
N/A
N/A
1.0
N/A
N/A
N/A
0.1, 1.0
Ty Communication round
N/A
N/A
N/A
0.95
800
Global
FedAvg momentum
0.5
Scheduler
Cosine Annealing
D Federated learning (FL)
FL enables collaborative learning by sharing model updates while maintaining data privacy and distribution across clients. A widely used FL algorithm is FedAvg [1], which creates a global model by aggregating parameters from randomly selected clients, achieving convergence after multiple communication rounds. FedProx [41] enhances the stability of FedAvg in non-llD settings by averaging local updates uniformly and incorporating proximal regularization against the global weights. FedOpt [40] improves performance over FedAvg by introducing federated versions of adaptive optimizers. FedSim [42] uses similarity-guider approach, which clusters clients with similar gradients to enable local aggregations, However, most FL methods assume that labeled data available to the client, which is impractical.
E
Static batch normalization (sBN)
Following HeteroFL. [34] and SemiFL [8], we adopt the Static Batch Normalization (sBN) into our weight aggregation algorithm the server. This approach is specifically designed for federated learning (FL) settings and has been shown to accelerates convergence and improve the performance of the trained model compared to naive adoption other normalization method for centralized setting such as Batch Normalization (BN) [43], InstanceNorm [44], GroupNorm [45], and LayerNorm [46].
In detail, unlike the normal training phase where each client tracks its own running statisties and affine parameters of the BN layer to send to the server for aggregation, sBN disables the tracking of rumning statistics during local training at clients. At the beginning of each communication round, before local training begins, the server sequentially sends the model to all active clients. At each client, running statistics tracking temporarily enabled (without momentum), and all training data fed into the global model to cumulatively compute the mean and variance for the BN layers in the model.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state our scope and contribution in abstract and last part of introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitation in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experiment details in Section 5.1 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We open-sourced the code, which can be found in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.o cc/ httpe//guieee//oeeeee
oessionnolcyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct httpa//ggiiieee/eeeuuuubiiiiiiii
/CodeSubmie sswonnoolll. onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experiment details in Section 5.1 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report standard deviation across 3 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the corresponding information in Section 5.1.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eurrpp.ee/ee//ee cc wwtpppcccc/eaeccccccccccccccceet Ethi ittiiccieeedieieeeee
Answer: [Yes]
Justification:
We use public datasets, and we do not violate other criteria.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
We discuss about potential positive societal impacts in introduction and conclusion.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA].
Justification:
We do not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly mentioned when we use existing assets throughout the paper, and in Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide code and documentation in the provided link in abstract.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "PQ96hfwwja", "reviewer": "Reviewer_LSrU", "summary": "The paper proposes a new method for federated semi-supervised learning tasks where only the server has a small amount of labeled data. The paper combines 3 different methods to tackle the problem and claims to reduce the confirmation bias issue with the method proposed.", "questions": "- Section 5.3: I find it hard to understand why to compare with ‘correctly pseudo-labeled data’. First, under the semi-supervised case, we won’t know the label, hence there is no point to compare test only on the correct pseudo-label data. This can only induce bias. Second, if we compare with CAT and CAT+SACR(all data), CAT only performs consistently better. This negate the idea of SACR, isn’t it?", "limitations": "Same as above", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and gives established motivations.", "weaknesses": "- The method seems to be just a combination of FreeMatch and FlatMatch under the FL case.\n- It is better to give more motivation to the LSSA method\n- Only on 2 datasets while most of the existing SSL /FSSL methods use three datasets. Based on a lot of existing semi-supervised learning literature, it is quite common to at least also include the CIFAR100 dataset. It will be better to also incorporate other type of dataset (not only image) to show the robustness of the method\n- Authors claim that the new method can effectively reduce the confirmation bias, so it would be better to have experiments specially design to show this part. It would be better to show that the method can successfully label the hard data compared to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:\n\n### Novelty of $(FL)^2$\nWe appreciate your question regarding the difference between $(FL)^2$ and related works (FreeMatch and FlatMatch). We would like to clarify $(FL)^2$’s novelty as follows:\n\nDifferent motivation with FlatMatch\n- The primary goal of FlatMatch is to bridge the gap between labeled and unlabeled data by addressing the differences in their loss landscapes. However, since $(FL)^2$ is designed for the labels-at-server scenario, we cannot utilize both unlabeled and labeled data at the same moment, FlatMatch’s methodology cannot be applied.\n\nUse of client-specific threshold (CAT) unlike FreeMatch\n- $(FL)^2$ can deal with non-iid settings by calculating specific adaptive thresholds for each client, while FreeMatch calculates a single global learning status. Because clients’ data distributions vary in non-iid settings, each client’s learning status can be different, making it necessary to estimate client-specific thresholds. \n- $(FL)^2$ mitigates the risk of overfitting by evaluating the learning status based on the entire dataset at a fixed point in time. Since the local model repeatedly encounters the same data across multiple local epochs, using a running batch as FreeMatch for estimating learning status might reinforce incorrect labels. \n\nIntroducing Learning Status-Aware Aggregation (LSAA)\n- We newly introduced LSAA, which adjusts aggregation weights of client local models. \n\nUtilization of SAM under FSSL\n- We developed a novel SAM objective specific to FSSL setting. We selectively apply the SAM objective to a small subset of high-confidence pseudo-labeled data, while using an adaptive threshold to incorporate a larger portion of unlabeled data.\n\n### Motivation of LSSA method\n- The learning difficulty can vary across clients. First, since the server has access to only a small labeled dataset, clients whose data closely resembles the server’s data will face lower learning difficulty, while those with more distinct data will encounter higher difficulty. Second, due to the non-iid distribution of data across clients. \n\n- We propose LSAA where LSAA assigns higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. Figure 1 in the global response PDF highlights the effectiveness of LSAA. Not only does it achieve higher test accuracy compared to the fixed aggregation weights (CAT + SACR), but it also demonstrates higher pseudo-label accuracy. Additionally, LSAA consistently achieves the highest correct label ratio, indicating the percentage of correct pseudo labels among all unlabeled data, throughout the experiment. After 600 training rounds, LSAA also records the lowest wrong label ratio, representing the percentage of incorrect pseudo labels among all unlabeled data. These findings suggest that LSAA effectively reduces incorrect pseudo labels while increasing correct ones, thereby mitigating confirmation bias [1].\n\n### Effect of $(FL)^2$ on confirmation bias\n\nThank you for your feedback. Since the wrong pseudo-labels usually lead to confirmation bias [1], we evaluated pseudo-label accuracy, label ratio, correct label ratio, wrong label ratio and C/W ratio in addition to test accuracy. We compared $(FL)^2$ against baseline methods using the SVHN dataset with 40 labels in a balanced IID setting, as illustrated in Figure 1 of the global response PDF.\n\nA high pseudo-label accuracy indicates that the method produces reliable pseudo labels. A high correct label ratio suggests that the method supplies the model with a higher number of accurate labels. Conversely, a low wrong label ratio indicates that the model encounters fewer incorrect labels, which is crucial for minimizing confirmation bias [1]. Lastly, a high C/W ratio signifies that the model is exposed to more correct labels than incorrect ones, further helping to reduce confirmation bias.\n\nWe observed that $(FL)^2$ consistently outperforms the baseline SemiFL across all metrics. Notably, while SemiFL generates more incorrect labels (C/W ratio < 1), $(FL)^2$ produces twice as many correct labels compared to incorrect ones. Additionally, the wrong label ratio for $(FL)^2$ is approximately 30%, significantly lower than SemiFL’s 45%. These results suggest that $(FL)^2$ effectively reduces the number of incorrect pseudo-labels while increasing the number of correct ones, thereby mitigating confirmation bias.\n\nFurthermore, we can observe the effectiveness of each component of $(FL)^2$, which are CAT, SACR, LSAA. Using CAT and SACR alone delivers better performance compared to the baseline in terms of all metrics. If we use CAT + SACR, pseudo label accuracy increases, correct label ratio increases, and wrong label ratio decreases, which means we can reduce the confirmation bias. When LSAA is added, which is $(FL)^2$, it achieves the best performance across all metrics. This suggests that the synergistic effect of CAT, SACR, and LSAA is reducing confirmation bias effectively.\n\n### Questions:\n\nWe want to clarify terms used in the manuscript.\n- CAT + SACR in Table 3 (Section 5.2): Our proposed method in Section 4.2. We apply SACR to only high-confidence samples.\n- CAT + SACR (all data) in Figure 3 (Section 5.3): SACR is applied to all of the pseudo labels generated by CAT. This is an ablation study to show why SACR should not be applied to all of the pseudo labels by CAT.\n- CAT + SACR (only correct pseudo-label) in Figure 3 (Section 5.3): SACR is only applied to correctly pseudo-labeled samples. This is for testing upper bound of our proposed method assuming we know the ground truth labels.\n\n[1] Arazo, Eric, et al. \"Pseudo-labeling and confirmation bias in deep semi-supervised learning.\" 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YGuKM46MWA", "reviewer": "Reviewer_dbte", "summary": "This paper studies the federated semi-supervised learning (FSSL) problem. A significant gap between the centralized semi-supervised learning and FSSL is found due to the confirmation bias. To address this issue, the current paper proposes a new FSSL algorithm, by incorporating three new ideas, namely client-specific adaptive threshold, sharpness-aware consistency regularization, and learning status-aware aggregation. Experimental results show that the proposed method significantly improves the performance of existing FSSL algorithms.", "questions": "+ How does the proposed method perform in Figure 1? Does it perform comparably to the centralized SSL method?", "limitations": "The paper only briefly mentions the lack of theoretical formulation as one limitation of the work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ This paper is very well-written. The proposed method and the underlying idea are clearly explained.\n\n+ The idea of using client-specific adaptive thresholds is very neat and well-motivated by mitigating the confirmation bias. \n\n+ The paper proposes the sharpness-aware consistency regularization to address the issue of generalizing to wrongly labeled data points when using sharpness-aware minimization.\n\n+ The numerical experiments show that the proposed method outperforms the existing methods by a significant margin consistently. \n\n+ The paper also numerically assesses the contribution of each component and studies the impact of incorrect pseudo-labels, providing further insight into the success of the proposed method.", "weaknesses": "+ The proposed method is based on heuristic reasoning and lacks theoretical justification. It would be nice if the authors could provide some theoretical justification for some components of the proposed method. \n\n+ The numerical studies are not that extensive; only two public datasets are used for benchmarking. It would be more convincing if the authors could provide more thorough comparison studies over different datasets across different settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:\n\n### Theoretical justification\n\nThank you for your feedback. We added more experiments to show that our method is robust in different settings. We plan to prove our methodology in terms of theoratical formulation in future works.\n\n### More experiments\n\nThank you for suggesting additional experiments on other public datasets and settings. In response, we conducted further experiments using the CIFAR-100, Fashion-MNIST, and AGNews datasets, and also introduced non-iid-0.1 settings for the CIFAR-10 and SVHN datasets. For detailed information and experimental results, please refer to the global response.\n\n### Questions:\n\nThank you for your feedback. Our accuracies are 38.9%, 81.5%, 83.2%, and 92.2% for 10, 40, 250, and 4000 labels, respectively. While the centralized FreeMatch method achieves 91.9% accuracy, our results, though still limited, represent a significant improvement. Specifically, our method outperforms the previous best-performing approach by a factor of 2.3X. This demonstrates that our approach narrows the gap between centralized and federated settings.\n\n### Limitations of $(FL)^2$:\n\nThank you for the feedback on the limitation. Our limitations are as follows, and we will address them in the camera-ready version.\n- CAT generates more pseudo-labels compared to fixed threshold methods, which in turn increases the time required for client training.\n- In SACR, additional computation is required due to the need for inference on unlabeled samples using perturbed models.\n- Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WS1m0r8CS", "reviewer": "Reviewer_YhMN", "summary": "This paper focuses on the federated semi-supervised learning (FSSL) scenario, which is a more challenging problem in FL. There are two different scenarios in FL, labels-at-server and labels-at-clients and this paper tackles the former issues. The author found the gap between SSL and FSSL is confirmation bias. To diminish this bias, this paper proposes client-specific adaptive threshold, modified SAM, and learning status-aware aggregation for a new aggregation scheme. The experiments of $(FL)^2$  demonstrate improvement on two benchmark datasets.", "questions": "- I go through the source code and find that implementing FedMatch may lack some components, e.g., parameter decomposition for disjoint learning and K-Dimensional Tree for helper selection. So the incomplete baseline is not convincing to compare with the proposed methods.\n- Why the performance of SemiFL under SVHN dataset with Balanced IID, 250 labeled samples is lower than 40 samples which is counterintuitive.\n- The performance of SemiFL under CIFAR10 dataset with Unbalanced Non-IID,  and Balanced IID is 10.0. This seems to be a training problem.", "limitations": "- The dataset is limited, there are also CIFAR-100 and FMNIST which are most used in FL research.\n- The limitations of this work are not well-discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well-organized and easy to follow with clear contribution.", "weaknesses": "- Lack of motivation and insights. It is not clear how the performance is influenced by confirmation bias Moreover, while the technologies discussed in this paper are not new, the argument that their particular combination can mitigate confirmation bias lacks persuasiveness.\n- The related work about labels-at-clients is outdated. Some new works should be included, e.g., [R1][R2].\n- The experiments of this paper are insufficient. E.g., How $(FL)^2$ tackles the confirmation bias is not mentioned. This paper should add more ablation studies to prove the bias is decreased rather than only the performance. There are many reasons to get the improvement.\n\nReference:\\\n[R1] Li, Ming, et al. Class balanced adaptive pseudo labeling for federated semi-supervised learning. In CVPR, 2023.\\\n[R2] Zhang, Yonggang, et al. Robust Training of Federated Models with Extremely Label Deficiency. In ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:\n\n###  Effect of $(FL)^2$ on confirmation bias\n\nThank you for your feedback. Since the wrong pseudo-labels usually lead to confirmation bias[1], we evaluated pseudo-label accuracy, label ratio, correct label ratio, wrong label ratio and C/W ratio in addition to test accuracy. We compared $(FL)^2$ against baseline methods using the SVHN dataset with 40 labels in a balanced IID setting, as illustrated in Figure 1 of the global response PDF.\n\nA high pseudo-label accuracy indicates that the method produces reliable pseudo labels. A high correct label ratio suggests that the method supplies the model with a higher number of accurate labels. Conversely, a low wrong label ratio indicates that the model encounters fewer incorrect labels, which is crucial for minimizing confirmation bias [1]. Lastly, a high C/W ratio signifies that the model is exposed to more correct labels than incorrect ones, further helping to reduce confirmation bias.\n\nWe observed that $(FL)^2$ consistently outperforms the baseline SemiFL across all metrics. Notably, while SemiFL generates more incorrect labels (C/W ratio < 1), $(FL)^2$ produces twice as many correct labels compared to incorrect ones. Additionally, the wrong label ratio for $(FL)^2$ is approximately 30%, significantly lower than SemiFL’s 45%. These results suggest that $(FL)^2$ effectively reduces the number of incorrect pseudo-labels while increasing the number of correct ones, thereby mitigating confirmation bias.\n\nFurthermore, we can observe the effectiveness of each component of $(FL)^2$, which are CAT, SACR, LSAA. Using CAT and SACR alone delivers better performance compared to the baseline in terms of all metrics. If we use CAT + SACR, pseudo label accuracy increases, correct label ratio increases, and wrong label ratio decreases, which means we can reduce the confirmation bias. When LSAA is added, which is $(FL)^2$, it achieves the best performance across all metrics. This suggests that the synergistic effect of CAT, SACR, and LSAA is reducing confirmation bias effectively.\n\n\n### Novelty of $(FL)^2$\nWe appreciate your question regarding the technical novelty of $(FL)^2$. We would like to clarify $(FL)^2$’s novelty as follows:\n\nUse of client-specific threshold (CAT) unlike FreeMatch\n- $(FL)^2$ can deal with non-iid settings by calculating specific adaptive thresholds for each client, while FreeMatch calculates a single global learning status. Because clients’ data distributions vary in non-iid settings, each client’s learning status can be different, making it necessary to estimate client-specific thresholds. \n- $(FL)^2$ mitigates the risk of overfitting by evaluating the learning status based on the entire dataset at a fixed point in time. Since the local model repeatedly encounters the same data across multiple local epochs, using a running batch as FreeMatch for estimating learning status might reinforce incorrect labels. \n\nIntroducing Learning Status-Aware Aggregation (LSAA)\n- We newly introduced LSAA, which adjusts aggregation weights of client local models. While CAT gives lower thresholds to the clients with low learning status to learn more from abundant unlabeled data, such information is not effectively reflected in the global model because of using fixed model aggregation weights. LSAA gives more weight to the client with low learning status, thus such information can be reflected more in the global model.\n\nUtilization of SAM under FSSL\n- We developed a novel SAM objective specific to FSSL setting. We selectively apply the SAM objective to a small subset of high-confidence pseudo-labeled data, while using an adaptive threshold to incorporate a larger portion of unlabeled data.\n\n\n### Questions:\n- Thank you for your thorough review. We have re-implemented the missing components of FedMatch and conducted an evaluation on the updated implementation. The updated performance results, presented in Table 1 of the global response PDF, demonstrate that $(FL)^2$ consistently outperforms FedMatch across all settings.\n- Thank you for your thorough review of the experimental results. Our observations indicate that SemiFL is sensitive to the labeled dataset when the number of labeled data is small. For instance, in the case of SVHN with 250 labels, two out of three runs failed after training 700 rounds, resulting in a low accuracy of around 20%. However, the one successful run achieved an accuracy of 90.6%. When given 40 labels on SVHN, we could not observe the training failure and the average accuracy was 53.4%. Similarly, when testing CIFAR10 with just 10 labels, all runs of SemiFL failed completely, yielding an accuracy of only 10%. We also attempted to replicate these experiments using the official SemiFL repository with 10 labels and observed the same outcomes. \n- We truly appreciate your thoughtful suggestions. We will certainly update the related work on labels-at-clients in our future manuscript. \n\n### Limitations of $(FL)^2$:\n- CAT generates more pseudo-labels compared to fixed threshold methods, which in turn increases the time required for client training.\n- In SACR, additional computation is required due to the need for inference on unlabeled samples using perturbed models.\nOur methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.\n- Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.\n\n[1] Arazo, Eric, et al. \"Pseudo-labeling and confirmation bias in deep semi-supervised learning.\" 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D6w6y3UjTg", "reviewer": "Reviewer_ytwd", "summary": "This work addresses a very practical challenge against successful FL deployments, which is of unlabeled data at FL clients. Furthermore, the problem is set in the regime of low count of labeled samples at server. The proposed solution to train a model in semi-supervised manner includes having an adaptive confidence threshold for each client to get pseudo-labels of more samples in the initial stages of training, then updating the model by perturbing the weights and training them on high-confidence pseudo labels; and lastly by aggregating the model weights through a learning status aware hyperparameter.\n\nThe results show superiority of the proposed method against existing state-of-the-art methods for federated self-supervised learning, and shows why naive application of centralized self-supervised methods are not cut for the disjoint nature of server and clients in FL.", "questions": "1. In Line 68, \"Clients with lower learning status (e.g., whose models are less certain about their predictions) receive higher aggregation weights, ensuring their updates are more significantly reflected in the global model.\"\n\nIt's unclear why do we need the above? Shouldn't low confidence show bad generalizability of the trained model? If yes, I didn't catch how the authors are preventing the model from learning on \"wrong\" input-output pairs.\n\n2. Related to Question #1, in what cases would the learning status be low? And what was the intuition behind giving high importance to those low status clients during the aggregation? What if this leads to some other clients getting low learning status then?\n\n3. Why does $(FL)^2$ need strongly augmented samples for SACR? What happens if we just use weakly-augmented samples instead?\n\n4. A minor suggestion: Figure 2 can benefit from numbers to show the flow of what happens after what. It was difficult to figure out where to start reading the diagram from.", "limitations": "I do not see a limitations section. Although the authors mention that the limitations are mentioned in Conclusion, I am not sure what they are referring to (not having theoretical analysis does not sound like a limitation of the proposed method, more like potential future work).", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is written very well. The flow of logic is mostly clear.\n2. The issue this paper is tackling is very important (especially low labeled sample count at the server), and the solution proposed is elegant (clearly states why and how the existing centralized semi-supervised methods are not enough, which brings a unique solution for the FL setting).\n3. Strong results with comprehensive experiments.", "weaknesses": "1. As expanded in Questions, some parts of the methodology is unclear. \n\nAs an example, why do we need Eq 10 (unsupervised training objective) at the stage of adaptive thresholding? I thought we are just getting the confidence threshold for pseudo-labels and sending it to the server.\n\nAnother example where the methodology was slightly fuzzy is around line 193, \"While we use client-specific adaptive threshold, we use a high fixed threshold to get high-confidence data samples\". What if that fixed threshold is not set correctly? The goal of dynamic client threshold was to avoid the sub-optimal results of a fixed threshold, wouldn't the same issues arise for this adversarial perturbations?\n\n2. The authors have tried two datasets: CIFAR10 and SVHN. I wonder how this translates to harder classification problems with more classes (where a few classes might not have representative samples at server and many clients just do not have samples related to those classes at all) or predicting classes or next words on natural language datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your thoughtful comments and feedback. In our revised manuscript, we intend to address these points as follows:\n\n### Clarification of methodology\n\nAbout Eq10\n- At each communication round, the selected clients individually calculate their adaptive thresholds based on their own unlabeled data using Eq 9. Once the adaptive threshold is determined, each client trains its local model according to Eq 10, which incorporates pseudo-labeling and consistency regularization using strongly augmented samples.\n\nAbout fixed threshold in SACR\n- In Section 5.3, the ablation study reveals that applying an adaptive threshold for SACR can actually degrade performance, as seen in the CAT+SACR (All data) scenario. This occurs because applying SACR to wrongly pseudo-labeled samples leads to the generalization of erroneous samples. To mitigate this, we opted for a high fixed threshold to ensure that only high-confidence data samples, which are more likely to be correct, are utilized. Since we already employ an adaptive thresholding scheme (CAT) to address the limitations of a fixed threshold, we believe it is safe to use a fixed threshold specifically for SACR.\n\n### Motivation of LSSA method (Related to questions 1, 2)\n- In a centralized SSL, methods like FlexMatch and FreeMatch introduced the use of different thresholds for each class within a dataset and showed their effectiveness. The rationale is that different classes pose varying levels of learning difficulty, so lower thresholds are assigned to more challenging classes.\n- In the context of FSSL, the learning difficulty can vary across clients. This variation arises for two main reasons. First, since the server has access to only a small labeled dataset, clients whose data closely resembles the server’s data will face lower learning difficulty, while those with more distinct data will encounter higher difficulty. Second, due to the non-iid distribution of data across clients, the learning difficulty naturally differs among them.\n- We propose LSAA to take account of different learning difficulties of clients, where LSAA assigns higher aggregation weights to clients with higher learning difficulty, enabling the global model to learn more effectively from these clients. Figure 1 in the global response PDF highlights the effectiveness of LSAA. Not only does it achieve higher test accuracy compared to the fixed aggregation weights (CAT + SACR), but it also demonstrates higher pseudo-label accuracy. Additionally, LSAA consistently achieves the highest correct label ratio, indicating the percentage of correct pseudo labels among all unlabeled data, throughout the experiment. After 600 training rounds, LSAA also records the lowest wrong label ratio, representing the percentage of incorrect pseudo labels among all unlabeled data. These findings suggest that LSAA effectively reduces incorrect pseudo labels while increasing correct ones, thereby mitigating confirmation bias [1].\n\n### Questions\nWhy does $(FL)^2$ need strongly augmented samples for SACR? What happens if we just use weakly-augmented samples instead?\n- Thank you for your question regarding SACR. Building on previous work in centralized SSL and SemiFL, our method trains the client's local model using pseudo-labeling combined with consistency regularization (as detailed in Eq 10). In this approach, the pseudo-label, which is generated from the confidence of weakly augmented data, guides the training of its strongly augmented counterpart. This ensures the model produces consistent predictions across various perturbations of the same unlabeled data. Following this scheme, we aim to provide additional consistency regularization with SACR , thus we used strongly augmented samples to calculate the loss.\n\nA minor suggestion: Figure 2 can benefit from numbers to show the flow of what happens after what. It was difficult to figure out where to start reading the diagram from.\n- Thank you for your valuable suggestion to enhance the readability of our manuscript. We will make sure to update Figure 2 accordingly in our camera-ready version\n\n### Limitations of $(FL)^2$:\nThank you for the feedback on the limitation. Our limitations are as follows, and we will address them in the camera-ready version.\n- CAT generates more pseudo-labels compared to fixed threshold methods, which in turn increases the time required for client training.\n- In SACR, additional computation is required due to the need for inference on unlabeled samples using perturbed models.\n- Our methodology relies on strong data augmentation, which may not be feasible for datasets such as sensor data.\n \n[1] Arazo, Eric, et al. \"Pseudo-labeling and confirmation bias in deep semi-supervised learning.\" 2020 International joint conference on neural networks (IJCNN). IEEE, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## More experiments

We conducted additional experiments using the CIFAR-100, Fashion-MNIST, and AGNews datasets, and introduced a non-iid-0.1 setting for the SVHN and CIFAR-10 datasets. Additionally, we performed an ablation study to determine the optimal rho value for Adaptive Sharpness-Aware Minimization (ASAM [1]). Through grid search, we identified rho=0.1 as best performing for both the SVHN and CIFAR-10 datasets, leading to revisions in the main table (Table 1 in the global response PDF).  Thanks to reviewer YhMN's feedback, we discovered and addressed missing parts in the FedMatch implementation, re-implementing it and re-conducted the experiments.

The results for CIFAR-100 are presented in Table 1, while the results for Fashion-MNIST and AGNews are detailed in Table 2, both of which can be found in the PDF file of the global response.

CIFAR-100
For CIFAR-100, we used WideResNet-28x8 as in previous works [2,3]. $(FL)^2$ consistently outperforms the baselines across most configurations. Notably, in the 400-label, non-iid-0.3 setting, $(FL)^2$ achieved a 6.4% improvement over the baselines. We will add experiments on a non-iid-0.1 setting in our camera-ready version.

Fashion-MNIST
For Fashion-MNIST, we used WideResNet-28x2 as in SVHN and CIFAR-10 datasets. We compared our method with SemiFL, the previous state-of-the-art. Because of the time constraint, we only compare with SemiFL, but we will add other baselines for the camera-ready version. With only 40 labeled samples, SemiFL failed in 3 out of 3 runs in iid setting and 2 out of 3 runs in non-iid-0.3 setting, resulting in ~10% accuracy. In one unfailed run in a non-iid-0.3 setting, SemiFL achieved 18.4% accuracy. In contrast, $(FL)^2$ successfully trains all 3 runs in non-iid-0.3 setting and 2 out of 3 runs in iid setting. In one failed run in iid setting, it yields ~10% accuracy. However, in the remaining iid runs, $(FL)^2$ achieved accuracies of 69% and 70.4%. In the non-iid-0.3 setting, $(FL)^2$ achieved an average accuracy of 63.2%. The results demonsrate that $(FL)^2$ is robust and effective even with minimal labeled data, particularly in a few-labels setting.

AGNews
We randomly sampled 12,500 training samples per class (out of total  50,000 samples) and applied back-translation for strong augmentation, following the methodology outlined in the SoftMatch [4] paper.
We used bert-base-uncased as a backbone model. We froze the bert parameters and trained only the linear classifier's parameters, running the training for 20 epochs. Since the mixup loss cannot be applied to an NLP dataset, we conducted our comparison using SemiFL without the mixup loss.
$(FL)^2$ significantly outperforms the baseline, achieving a 39.6% accuracy improvement in the iid setting and a 14.5% improvement in the non-iid setting. With only 20 labels provided, SemiFL exhibited considerable performance variance across experiments, where STD is 14.3 and 13.7 for iid and non-iid-0.3 setting, respectively. In contrast, $(FL)^2$ consistently delivered stable results, where STD is 0.6 for iid and 3.7 for non-iid-0.3. 

More experiments on previously used datasets (CIFAR-10 and SVHN)
We also introduced a new non-IID-0.1 setting to our previous experiments on the CIFAR-10 and SVHN datasets. In this non-IID-0.1 setting, $(FL)^2$ consistently outperformed all baseline methods across the datasets, showing 10% higher accuracy over the best-performing baseline on CIFAR-10 with 40 labels.

[1] Kwon, Jungmin, et al. "Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks." International Conference on Machine Learning. PMLR, 2021.

[2] Wang, Yidong, et al. "FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning." The Eleventh International Conference on Learning Representations. 2022.

[3] Diao, Enmao, Jie Ding, and Vahid Tarokh. "Semifl: Semi-supervised federated learning for unlabeled clients with alternate training." Advances in Neural Information Processing Systems 35 (2022): 17871-17884.

[4] Chen, Hao, et al. "SoftMatch: Addressing the Quantity-Quality Tradeoff in Semi-supervised Learning." The Eleventh International Conference on Learning Representations.

Meta-review: The paper proposes a novel heuristic algorithm for labels-at-server FL that empirically can improve significantly over previous state-of-the-art. The reviewers generally find the paper to be well written, but especially reviewer LSrU raises concerns about limited novelty and insufficient motivation for the methods, although these are rather unspecific. The latter of these is also addressed by the authors at length in their rebuttal.

Based on my own reading, I also find the paper well-written and easy to follow. The proposed method can provide significant boost especially in classification problems with fewer classes such as CIFAR-10 and SVHN, while its benefit in CIFAR-100 seems more limited. Still, I believe the paper would be a worthy addition to the literature and in line with the majority of the reviewers, recommend its acceptance.
Decision: Accept (poster)