BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
Authors: Haozhao Wang
Shengyu Wang
Jiaming Li
Hao Ren
XINGSHUO HAN
Wenchao Xu
Shangwei Guo
Tianwei Zhang
Ruixuan Li
Venue: ICML 2025 poster (2025)
Keywords: Semi-supervised Federated Learning; Bayesian Approach
Primary area: optimization->large_scale_parallel_and_distributed
TL;DR: A novel Bayesian approach that solves the FL unlabel problem of data in clients

Abstract: Semi-supervised Federated Learning (SSFL) is a promising approach that allows clients to collaboratively train a global model in the absence of their local data labels. The key step of SSFL is the re-labeling where each client adopts two types of available models, namely global and local models, to re-label the local data. While various technologies such as using the global model or the average of two models have been proposed to conduct the re-labeling step, little literature delves deeply into the performance dominance and limitations of the two models. In this paper, we first theoretically and empirically demonstrate that the local model achieves higher re-labeling accuracy over local data while the global model can progressively improve the re-labeling performance by introducing the extra data knowledge of other clients. Based on these findings, we propose BSemiFL which re-labels the local data through the collaboration between the local and global model in a Bayesian approach. Specifically, to re-label any given local sample, BSemiFL first uses Bayesian inference to assess the closeness of the local/global model to the sample. Then, it applies a weighted combination of their pseudo labels, using the closeness as the weights. Theoretical analysis shows that the labeling error of our method is smaller than that of simply using the global model, the local model, or their simple average. Experimental results show that BSemiFL improves the performance by up to $9.8\%$ as compared to state-of-the-art methods.

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
Haozhao Wang' Shengyu Wang?
Jiaming Li"
Hao Ren Xingshuo Han' Wenchao Xu² Shangwei Guo
Tianwei Zhang
Ruixuan Li" 1. Introduction
Abstract
Semi-supervised Federated Learning (SSFL) al- lows clients to collaboratively train a global model in the absence of their local data labels. The key step of SSFL is the re-labeling where each client adopts two types of available mod- els. namely global and local models, to re-label the local data. While various technologies such as using the global model or the average of two models have been proposed to conduct the re-labeling step. little literature delves deeply into the performance dominance and limitations of the two models. This paper first theoreti- cally and empirically demonstrate that the local model achieves higher re-labeling accuracy over local data while the global model can progres- sively improve the re-labeling performance by introducing the extra knowledge of other clients. Based on these, we propose BSemiFL which re-labels the local data via the collaboration be- tween the local and global model in a Bayesian approach. Specifically, to re-label any given local sample, BSemi FL first uses Bayesian inference to assess the closeness of the local/global model to the sample. Then, it applies a weighted com- bination of their pseudo labels, using the close- ness as the weights. Theoretical analysis shows that the labeling error of our method is smaller than that of simply using the global model, the lo- cal model, or their simple average. Experiments show that BSemiFL improves the performance by up to 9.8% as compared to existing methods.
Federated learning (FL) enables the collaboration of mul- tiple clients train global model without sharing their private dataset (Hu et al., 2024a; Huang et al., 2024a; Qi & et al., 2025; Liu et al., 2024), gaining much attention recently and being applied to wide range of applications such as traffic prediction(Zhang et al., 2024a), recommen- dation (Yang et al., 2024), and IoT (Liu et al., 2021a). How- ever, typical FL (McMahan et al., 2017) presumes that each client possesses labels for their data, requirement which may not align with practical realities in numerous applica- tions, especially when most clients may not be experts in the task of interest or may not be well motivated to label their data. To tackle this challenge, Semi-supervised FL (SSFL) (Diao et al., 2022; Jeong et al., 2021; Zhuang et al., 2021) comes to the rescue, by allowing clients to have ac- cess to purely unlabeled data while maintaining small amount of labeled data in the server.
SSFL typically runs the re-labeling and local training steps in each client and then conducts the aggregation step in the server, as shown in Figure 1. Although the process is easy to implement, the re-labeling error of the local data is in- evitable, thus degrading the performance. To improve the SSFL performance, many approaches have been proposed from the re-labeling step (Diao et al., 2022), local training step (Malaviya et al., 2023; Lee et al., 2024), and aggrega- tion step (Wang et al., 2023b) separately. In this paper, we mainly focus on the re-labeling step by considering that it can directly reduce the error of pseudo labels.
The representative re-labeling technique (Diao et al., 2022) is to adopt the global model to re-label the local data. While easy to implement, the upper bound of its performance is inevitably limited by adaptation error of global model, which arises from the substantial gap between the global and the local data distribution (Hu et al., 2024b; Huang et al., 2024b; Qi et al., 2023; Wang et al., 2023a; Tang et al., 2022). To reduce the adaptation error, a natural solution to adopt the local model re-label the local data, where the local model is trained with the local data and adapts better than the global model. Yet, we find that merely utilizing the local model leads to a less optimal global model than using the global model. We identify that the underlying principle behind this is that the data knowledge embodied by the local model is inherently solidified to its own small
'School of Computer Science and Technology. Huazhong Uni- versity of Science and Technology, Wuhan, China Division of In- tegrative Systems and Design, Hong Kong University of Science and Technology, Hongkong, China School of Cyher Science and Engineering, Sichuan University, Chengdu, China *College of Computer Science and Technology, Namjing University of Acro- nautics and Astronautics, Nanjing, China 'College of Computer Science, Chongqing University, Chongqing. China "College of Computing and Data Science, Nanyang Technological Univer- sity, Singapore, Singapore. Correspondence to: Ruixuan Li <xxli @ hust.edu.en>.
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
the closeness of the global and local model each local sample, and then we apply a weighted ensemble of the global and local model for labeling the local data where the weights are the obtained confidence.
Aggregation
Fine-tune
Server
Laboled data
Global
Local model
Local model
Global
@Re-label
DRe-label
We establish the theories for re-labeling performance of the proposed method. Our results show that BSemiFl FL theoretically reduces the labeling error compared to the maximum error of merely adopting either the global or local model. Besides, we also theoretically show that our Bayesian- weighted ensemble achieves a smaller loss than naively averaging.
global
global
data
local
Habel
label
local
Client
global
global model
model
Train
model
model
2
Train
We conduct extensive experiments across diverse array of datasets and experimental configurations. The out- come robustly substantiates the superiority of our pro- posed method, demonstrating an improvement in model accuracy that reaches up to 9.8% compared with the cur- rent state-of-te of-the-art methodologies.
Figure 1. The framework of SSFL.
local dataset, which cannot be optimized by referring extra knowledge from other clients.
Considering the above limitations, we propose BSemiFI which uses a Bayesian- based approach to jointly utilize the ensemble of the global and local model to label the lo- cal data to harness both their benefits. Instead of adopt- ing naively averaging ensemble where the correctness of pseudo labels heavily depends on the prediction of both models, BSemiFL performs the ensemble in a weighted manner where the weights are adaptively calculated for each sample using Bayesian approach. More specifically, BSemiFi FL adopts the Bayesian approach to infer the close- ness of the local sample to the training data distribution of the local or global model and sets the weights based on the closeness. The principle behind this is that the model tends to correctly predict the label of a sample when the given sample is closer to its training data. Therefore, BSemi FL can assign higher weight to the global/local model when the given sample is closer to the global/local training data, thus being able to reduce the impact of wrong pseudo la- bels. Extensive experiments show that the BSemiFL sig- nificantly improves the model accuracy as compared to state-of-the-art a algorithms. Our contributions are:
2. Related Work
2.1. Semi-supervised Federated Learning
Existing SSFL frameworks can be categorized into two types according to the location of the labeled data, i.e., "la- bels at client" (Zhang et al., 2024b; Bai et al., 2024; Zhu et al., 2024) and "labels at server". SSFL of "labels client" considers labeled data are available at local clients (Kim et al., 2023; Itahara et al., 2023; Xu et al., 2024; Zhang et al., 2023a;b; Cho et al., 2023). The first work of this type FedMatch (Jeong et al., 2021) assumes that each client contains both labeled and unlabeled data and em- ploys a consistency method from semi semi-supervised learning, enhancing cross-model intra-client consistency by select- ing proxy helpers among the clients. FedAC (Jiang et al., 2024) identifies that using the models of similar and dissim- ilar clients to label the unlabeled data has different advan- tages, thus proposing alternatively leveraging the models of similar and dissimilar clients to label the unlabeled data, which demonstrates great effectiveness. Considering the discrepancy of clients that some clients may not be able label the data while other clients available, some works (Lin et al., 2021; Liang et al., 2022; Liu et al., 2021b) also focus on the setting where clients are fully labeled while some clients only contain unlabeled samples.
As far as we know, this is the first work that theoretically reveuls why merely adopting the global or local model is not optimal for labeling local data, which are also vali- dated by our empirical evaluations. Specifically, the main causes are that the distribution gap between the global data and local data limits the performance of the global model over the local data and the error of the local model can not be corrected by is local data.
In this paper, we focus on the second type of SSFL, i.e., "labels at server", by considering that it is generally prac- tical in real-world FL scenarios, e.g., in the healthcare field where the labels are mostly assigned by experts and users usually are not able to label their health monitoring data (Lee et al., 2024; Malaviya et al., 2023; Yang Xu, 2023; Yang et al., 2023). SSFL of "labels at server" as- sumes that clients have purely unlabeled data with a limited amount of labels residing at the central server ((Lin al.,
We propose novel method called BSemiFL which jointly leverages the global and local model to label the local data via a Bayesian approach. In particular, we employ Bayesian- based approach to adaptively discern
2

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
2021; He et al., 2021; Zhang al., 2021)); The represen- tative work SemiFL. (Diao et al., 2022) innovates with alternative training regimen that re-trains the global model in the server and labels the local data with the global model. Being different from SemiFI., we leverage an Bayesian- based collaboration between the global model and the lo- cal model to re-label local data, A subsequent work (FL)* (Lee et al., 2024) utilizes the idea of contrastive learning to improve the local training process. Considering that the client may be resource-constrained, pFedKnow (Wang et al., 2023b) proposes compressing the local models to re- duce the computation and communication cost of SSFL, Different from these works, this paper focuses on the re- labeling step, which is orthogonal to them.
global model Wt. local model w²", be their combina-
tions. To construct high-confidence dataset Sm each client uses threshold to filter out labels with low confi- dence values like FixMatch (Sohn et al., 2020):
S. = {(=="",, ,IO"") with max(f.") 2 T), (2)
where I(-) is to transform the vector y."h into the one-hot label vector with only remaining the maximum probabil- ity. 0 < < is threshold pre-selected by all clients, max (it")) denotes the maximum confidence among all K classes. With the constructed labeled dataset, as illustrated in ② of Figure 1, client m trains its local model for E local epochs to obtain the local model:
Lm = ([(TT)) I(F"))
W w 17 V win Lin-
2.2. Unsupervised Federated Learning
Unsupervised federated learning (UFL) does not assume the existence of labeled data in either the server or clients during the training process. Since contrastive learning is the main technology for learning in unlabeled data (Li et al., 2021; Zheng et al., 2022), many works (Zhuang et al., 2021; 2022) propose applying the contrastive learn- ing methods over the scenario of UFL. For example, FedU (Zhuang et al., 2021) allows each client to adopt a conservative learning loss during the local training process. Orchestra (Lubana et al., 2022) addresses the challenge of NonIID data by adopting client clustering approach, thereby mitigating the adverse effects caused by data het- erogeneity among clients. These works typically focus on Step 2 as illustrated in Figure 1, which is orthogonal to this work that seeks to improve Step 1.
After that, as presented by 3 in Figure 1, each client m uploads its local model w to the server and server aggre- gates them to obtain the t + 1-th round global model
M
S,
Wi-1 Σ
w."
(4)
St
m= Sm and Sm denotes the size of Sm- Finnally, corresponding to 4 in Figure I, the server fine- tunes the global model Wt+1 with the loss Lg for E epochs:
where St = Σ Mr.
L. L-=1(f(m;, (:), Wr+1 - We+1 V/ W. Las (5)
where represents the loss function (e.g.. Cross Entropy loss) and 17 denotes the learning rate.
3. Problem Formulation and Preliminaries
4. Motivation
Problem Formulation. Semi-supervised federated learn- ing is collectively train a global model w that adapts to the global data distributed in M clients. In SSFL, the server has access to a labeled dataset $ = {2x, ) with N₂ samples. Each client m has access to a distinct unlabeled dataset U,, = {-;" J=m with Nm. samples where the true label y/" does not exist. We consider a K-class classification task in the SSFL and denote [ar, w) by the K di-dimenioonal soft prediction output by the model w us- ing the softmax function.
This section specifies the limitations and benefits of the global model and local model for re-labeling the local data. For simplicity, we below denote the SSFL method of re- labeling with only the global model by 'global-only' and with only the local model by 'local-only'. In the follow- ing, we conduct motivation experiments by using the Wide ResNet28 model and CIFAR-10 dataset.
4.1. Limitations and benefits of the global model
Limitations of the global model. Specifically, the global model inherently has generalization errors on local data, which stem from the differences between the global data distribution and the local data distribution due to statisti- cal heterogeneity. To validate this, we run the global-only method over CIFAR-10 dataset to compare the average ac- curacy of the global model over the local data of all clients to that of the local models over their corresponding local data, where the results are shown in Figure 2a. Noting that the test data for the global model is the global data, while
Basic workflow of SSFL. For simplicity and clarity, we here specify the basic four steps of SSFL, as illustrated in Figure 1. In each round I, after receiving the global model W. from the server, corresponding ! in Figure 1, each selected client m starts generate pseudo labels 0,"h for each local sample z; by using the model wiTh
D = f(x"" w
(1)
where the adopted re-labeling model can either be the
3

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
70
sp M/velyy Ma poa
onsyl global ony
-5
60
86
Ioal roddd
gRoul mocel
100
200
300
400
500
0000
700
900
120
202
302
400
702
500
100
200
705
030
roundd
rounds
(a) Average local accuracy of the global and (h) Global accuracy of the global model ob- (c) Local accuracy of each local model on local model on local data separately
tained by local-only& & global-only method its corresponding local data
Figure 2. Motivation experiments. (a) The average accuracy of testing the global and local model on local data of all clients separately when using the global-only method. (b) Comparison hetween the local-only and global-only methods. The accuracy is to test the global model on the global test data. (c) The accuracy of the local model on its corresponding local data when using the local-only method.
the test data for the local model is the local data correspond- ing to its respective client. As can be seen, the accuracy of the global model over local data is much lower than the local model. Theoretically, we show that a substantial per- formance gap exists when only using the global model to label local data.
global dataset over CIFARIO. As shown in Figure 2b, the test accuracy of the global-only method is nearly 15% higher than the local-only method, where the global model is tested using two different annotation methods, and the test data used is the global data.
4.2. Limitations and benefits of the local model
Theorem 4.1. Denote the data distribution of each client m by Pms the empirical distribution of the global dataset by p. and the real distribution of the global data by p. By using d=az(:;-) to measure the distance between two dis- tributions. given constants 0- < 5 S 1 and >> O, with the probability at least 1 - 5, the expected labeling error pm (ft) of the global hypothesis fo over the local data distribution Pm is bound by:
The limitation of the local model. Specifically, re- labeling with the local model suffers from the local knowl- edge solidification, where in each round, the client trains the local model based only on the aseudo-labelee data from the previous round, leading to the updated model merely inheriting and retaining the previous model's knowledge without correcting erroneous labels. To validate this, we record the accuracy of the local model of each client on its local data across rounds. We test the local model using its corresponding local data. As shown in Figure 2c, we ran- domly present the results of 9 clients for clarity of presen- tation. As can be seen, there no change in the accuracy after the initial stage, indicating that the knowledge of the local model has been fixed and its error cannot be gradu- ally corrected by itself. Besides, training the local model in the local-only method relies on the limited local data in terms of size, causing a high generalization error. These also verify that why the test accuracy of the global model obtained by the local-only method nearly 15% less than the global-only method in Figure 2b. A theoretical verifi- cation is present below.
log 8
pm (Ft) 5 L
2 _dFAr(Pm p) Ag: (6)
25
M of the global dataset, and >, [p.[a) + Lym (Ja) with f= arg ger + Lp.. (f).
where S -m Sm + N₂ denotes the total data size
The proof is deferred to Appendix A. Theorem 4.1 indi- cates that the labeling performance of the global model is substantially limited by the gap dFar(Pm p) + Ag be- tween the global distribution p and local distribution Pm- As a consequence, merely adopting the global model can- not fully unleash the performance potential of SSFL.
Benefits of the global model. The global model is trained by aggregating local models from all clients, where the generalization error of re-labeling local data can be pro- gressively reduced with the training process by bringing extra knowledge. As can be seen from Figure 2a, the re- labeling accuracy of the global model gradually approaches the local model. Further, we compare the test accuracy between the local-only and global-only method over the
Theorem 4.2. Let the data distribution of client m be pm and its empirical distribution be Pm Given constants 0 < 8 < 1 and a > O, with the probability at least 1 1, 8, the expected labeling error Lpn. (fp.) ) of the local hypothesis Jm.. over the local data distribution Pm is bound by:
log
T.p.. (Jom) S. T.p... (Jo.. + V
(7)
25
4

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
where S. denotes the data size of the labeled local dataset.
bel k. Then, the ELBO is defined as:
N,
N. K
p(r: (k) (8)
(zg
Σ
1z(k)z;)oog
Theorem 4.2 can be directly derived from Hoeffding in- equality and we ignore its proofs. Comparing Theorem 4.2 to Theorem 4.1, we can find that the generalization error log of the global-only method due to the limited size of the local data.
(k(())
By fixing the posterior probability fz(k)zzz), and consider- ing facts
1og
of local-only metho is larger than
2S
25
K f(x;) Σ
N, (ailkk) = 1, (9)
Benefits of the local model. The local model adapts bet- ter to the distribution of the local data than the global model due to the eliminated distribution gap of training data. This can be observed by comparing Theorem 4.2 to Theorem 4.1, where the error items {d===(pm..) p) + X, in Theorem 4.1 does not exists in Theorem 4.2. Figure 2a also presents that the accuracy of the local model on its lo- cal data converges faster and higher than the global model. Therefore, how to harness both the benefits of the small dis- tribution gap of the local-only method and the small gener- alization error of the global method is the key.
Mas, k),
k=1
minimizing (8) obtains the optimum of p(xz(k) which sat-
isfies:
fo(k)
p(a,(k) = ΣN, E (k((2)))
(10)
We parameterize fs((k)rz)) by the global model w, which is fine-tuned on the global dataset S with empirical distri- bution
p. i.e., (p),,,, wt). Then, when considering any specific local sample x:th in the client m. the corresponding closeness of the global model wt conditioned on the class k is approximately
5. Methodology
In this section, we specify the proposed method which jointly leverages the global and local model to label the lo- cal data. The framework of our method is illustrated in Fig- ure and Figure 3. Our method only refines the re-labeling process, corresponding to ! in Figure 1, which is orthogo- nal most recent advancements. For clarity of expression, here mainly specify the labeling process in this tion. Overall, the labeling process of 3SemiFL consists of two stages, i.e., calculating the closeness of the global/local model on each local sample and then using the confidence values to apply a weighted ensemble of the pseudo labels generated by the two models. The algorithm workflow can be found in Appendix B.
folk|a wt) falkla wo), ((=71k)- Ja(K/aTT w.) + Q
(11)
Further, considering the empirical distribution of the la- beled dataset $ in the server as pe and approximating the class prior probability of the server dataset to the global dataset, i.e., ((k) Rs (k(k), we have
K
p(aT") Σ (aTTik)p
(12)
this end, we can obtain the closeness p(x:") of the global model w, on any local sample xy. Similarly, we can also obtain the closeness pm (x:) of the local model w on any local sample v;
5.1. Bayesian-basee Closeness
Based on the domain adaptation theory in (Ben-David al.. 2010), a model tends to achieve low generalization error when the test data sampled from the space close to its training domain. Therefore, we adopt the closeness of the sample to its training data to measure the correct- ness. Specifically, we utilize the sampling probability R(x) to represent the closeness of the given sample x to the global distribution p. Obviously, a larger p(x) indicates that there are more similar samples around the area of the sample x, demonstrating that x closer to the global data. To estimate the probability p(x)), we leverage the Varia- tional Bayesian Inference to build a negated evidence lower bound (ELBO). Without causing confusion, we leverage (k(a)) as the posterior probability of classifying the sam- ple x into the class k. Further, we denote p(z(k) by the probability of generating the sample x conditioned on la-
K
Pm
Σ km
Pm (k))m
K
fpm (k]| w
fpm (k) (o wi')) ) + Q Pm (k))
A=1
Sm
where
Q
fom (k)aa w (?"
(13)
5.2. Weighted Ensemble of Global and Local Model
After obtaining the closeness p(x,") of the global model and pm (x?")) of the local model on the local sample xt
5

---

BSemiFL: Semi-supervised Federated Learning via Bayesian Approach
Probability vector Q,
Local dutaset "Sal-lainet
Saheleeeee diinecd in client
E (12)
Sum
Weighted Trseebbe
Global model
F8 (4t
Q
P(x"")
S
Local model w
Local Model W!l
fam
@t Am
Normalize
1
Pscudo label
model
Eq.13)
Client m
Labeled dataset server
Figure 3. Framework of BSemiFL with Bayesian saassd Ensemble.
the client employs a weighted ensemble of them generate pseudo labels. Specifically, each client first normalizes the weights:
less than the maximum individual loss of either the global local model, i.e.,
L(aef + (1- (1-m) max LL(ff),((fe.. (16)
p(=
Pm (x!")
a. p(a.")) + pm())
1 a; p(x,71 ) + Pm (x
Proof is in Appendix A. Theorem 6.1 indicates that jointly leveraging the global and local model will inherit the bene- fits of the better one and thus prevent its performance from becoming the same as the worse one. the other side, such an approach may restrict itself from outperforming the best model, of which the loss on each sample may be larger than the better model. In other words, the prediction probability (confi- dence value) of the ensemble model on the ground-truth label is smaller than an individual model. However, it is worthwhile to note that a smaller class prediction proba- bility does not mean worse labeling performance. fact, we can accordingly adopt a small threshold (i.e., smaller specified in equation (2)) when choosing good labels. This has also been verified by evaluations shown in Figure 5c, where our optimal threshold is 0.8 while the op- timal threshold of the global-only method is 0.95 as stated in (Diao et al., 2022). Now, we show that the performance of the weighted ensemble is better than the naive average.
(14)
and then executes:
V.- = a. fp(2,"" wi) (1 a,"" pfp (2," w(") (15)
After that, each client m constructs the high-confidence dataset Sm and performs local training by conducting (2),(), and (4).
between the global model and the local
Privacy Discussion. Although our method requires more interaction terms, does actually increase the risk of privacy leakage. Compared to traditional FL meth- ods, our method only needs to additionally transmit the K- dimensional probability distribution vector Q of the server- side public dataset categories on the global model. In the SSFL scenario, the data on server side generally non- private. Therefore, in practice, our method does not result additional client privacy leakage compared to traditional FL methods.
Theorem 6.2. Consider the same notations of distribution and hypothesis defined in previous theorems. For each lo- cal sample
6. Theoretical Analysis
Pm with the ground-trume -ruth label y k. considering the Cross-Entropy loss function is adopted, then under the condition that fp(k(z) > fpm. (k(x) when p(x) > pm(x) and vice versa, the loss of Bayesian-baat model ensemble smaller than the direct averaging:
We below theoretically present the benefits the Bayesian-based ensemble. We initiate by juxtaposing the performance of our method against the global-only or local-only methods. Then, we show how the weighted en- semble surpasses the straightforward averaging method.
L(afø + (1- (1-a)fful 5 fo fpm.
(17)
Theorem 6.1. Denote the hypothesis trained the empir- ical distribution of the local datuset pm by Jt.. and the hy- pothesis trained on the empirical distribution of the global dataset p by fo- For any local data sample x ~ Pms consid- ering the Cross- Cross-Entropy loss function adopted, then the loss of the ensemble between the global and local model is
2
Proof can be found in Appendix A. Theorem 6.2 demon- strates the effectiveness of the proposed Bayesian- based approach. Although this theorem holds under some spe- cific conditions, we also note that this condition commonly
6

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
Table 1. The comparison of final test accuracy on the two datasets. The best results bolded.
Method
SVH)(?%)
CIFAR- 10(%)
CIFAR 100(%)
Shards (M,S)
(100, 2)
(100, 4)
(100, 8)
(100, 2)
(100, 4)
(100, 8)
(100, 2)
(100, 4)
(100, 8)
SomiFL
88,36#1.11
93.33 =0.48
92.88 =2.33
66.34+3.21
76.68+3.22
81.0+22.79
37.73+0.86
42.68+2.12 12
47.87=1.66
FedMatch
73.50+3.11
73.57+1.34
73.66+1.15
45,43+2.48
45.04=0.47
45.03+1.57
16.8942.97
17.05+2,94
17.05+3.45
FodU
83.78
8..5
1..6
6..51+1.03
61.6243.
65.74x1.66
30.8312.78
32.2
33.57:2.09
FedEMA
84,40=0.46
36.62=0.89
85.77=1.68
65 05.08+2.55
68.23#1.81
66, 83+8000344
29,48 29,48+2.43
32.86+2.65
33.62=3.08
Orchestra
86.57+1.78
1.70
37.59+3..20
688.7722.55
69.09+1.98
69.0342.35
39.93+2.17
38.95+1.02
38.01+2.60
(FL)
84.17=1.52
82.78
1..59:1.35
60.63+2.06
62.61=1.91
61.82+1.73
27.73+3.04 3004
28.4743.01
29. 19.11=2.93
FedFAME
86,84+0092 9:3
86, 10.39+11. 1.22
17.35+1.71
63.41+1.73
61.8660.92
62.4 03.433+0..3
25,91+1 995
26, 26.05+2.07
25. 25.50+1.00
PFodKnow
56,88+2.73 73
59, 59.80+2.06
19,58+1..3
19.93+1.75
60,03=1.52
60,57+1.36
19.55+0.75
19.77+1.03
18 1..00=1 1.60
Ours
94.32#0.71
96.32=0.93
96,53+0,73
78.31=1.32
82.36:0.91
84.4020.72
39.09#3.12
43,56+2..4
49.75:2.31
holds according to Bayes' theorem. Specifically, Bayes' Theorem states that the posterior probability P(y[z) is proportional to the generation probability P(xly), i.e.. P(y]z) & P(x))) As consequence, by considering [y]xz) and p(z:) the posterior probability and generation probability respectively, the condition of Theorem 6.2 is reasonable. In fact, our evaluation also verifies our theoret- ical analysis that our Bayesian- based approach outperforms simply averaging, as shown in Figure 5a
Implementation. We implement the whole experiment in a simulation environment based on PyTorch 2.0 and 4 NVIDIA GeForce RTX 3090 GPUs. We use 100 clients in total and randomly choose 10% each round for local train- ing. We set the local epoch to 5, batch size to 10, and learn- ing rate to 3.0e 2. We employ SGD optimizer with the momentum of 0.9 and weight decay of 5e -4 4 for all meth- ods and datasets. The number of global communication rounds is 800. Each experiment is run 3 times and we take each run's final 10 rounds' accuracy to calculate the av- erage value and standard variance. We set the threshold of our method to be 0.7. For SemiFl. and FedMatch, we adopt the same thresholds as leveraged in their original paper, i.e., 0.95 for both SemiFll and for FedMatch.
7. Experiments
7.1. Setup
Datasets and Models. We consider three popular datasets in experiments, i.e., SVHN (Netzer et al., 2011), CIFAR- 10 (Krizhevsky et al., 2009) and CIFAR-100 (Krizhevsky et al., 2009). which contains 10, 10, 100 classes respec- tively. We use Wide ResNer28x2 (Zagoruyko & Ko- modakis, 2016) as our backbone model for CIFARIO and SVHN datasets and Wide ResNet28x8 for CIFAR100 datasets throughout our experiments, by following the set- tings of existing SSFL works (Diao et al., 2022). The num- bers of labeled data at the server for CIFAR10, 100, SVHN, and CIFAR100 datasets are 500,500,2500 respectively.
7.2. Comparison with Baselines
Shards-based NonIID. Table I compares results with baselines in Shards-based Non-IID scenarios. Our pro- posed BSemiFL consistently achieves best performance in nearly all settings. For CIFAR-10 with 2 shards, BSemiFL achieves 78.31% accuracy, outperforming Or- chestra's 68.57% by 9.8%, highlighting its effectiveness with highly non-lII data. However, on CIFAR-100 with 2 shards, Orchestra outperforms BSemiFL. FL. Regarding the slightly lower average value compared to Orchestra, we be- lieve this is because Orchestra is a federated self-supervised learning method. In this specific scenario, its better perfor- mance is mainly due to the fact that the client data distri- bution is highly concentrated in a few categories. This lo- cal consistency makes it easier for local clustering to cap- ture clear category patterns. Since they focus on Step 2 of SSFL, Our method can be jointly used with them.
Data Partition. We adopt two Non-IID data partition methods: Shards (McMahan et al., 2017) and Dirichlet (Linn al., 2020). In the Shards setting, the sorted samples are shuffled into M S shards, and assigned to M clients ran- domly. Dirichlet distribution uses to characterize the de- gree of heterogeneity. We set of Dirichlet: {0.1, 1, and shards for each client: {2, 4,8).
Baselines. We compare our method against both SSFL and unsupervised FL methods. UFL includes FedU (Zhuang al., 2021), FedEMA (Zhuang al., 2022) and Orches- tra (Lubana et al., 2022) while SSFL approaches contain FedMatch (Jeong et al., 2021), (FL)² (Lee et al., 2024), PFedKnow (Wang et al., 2023b), FedFAME (Malaviya et al., 2023), and SemiFL. (Diao et al., 2022). We exclude the local-only method due to its deficiency.
Dirichlet-based NonIID. Figure 4 compares our method with baselines in Dirichlet NonIID settings. Our method outperforms most baselines, achieving 76.71% accuracy vs. SemiFL's 69.03% ( (7% lower). However, on CIFAR- 10 with a = 10, SemiFL slightly outperforms our method 19% vs. 87.27%). This is because SemiFL uses the global model to label local unlabeled data, performing bet- ter when the global model is well-trained (low NonIID).
7

---

BSemiFL: Semi-supervised Federated Learning via Bayesian Approach
100
100
103
Fackdach
Pedtacch
90
Farthil
Test Accuracy((%)
93
Accuracyy
Test Acouracy
85
85
80
75
75
85
*s
SVHN Dirichlet (a 0.1)
SVHN Diniclet (at
SVHN Dirichlet (a 10)
(a) SVHN with a=0.1.
(b) SVHN with a=1.
(c) SVHN with a=10.
Paokaaid phadkroe
Puddaai pfaditio
pFaction
Test Accuracy (%)
70
Faothol
Test Accuracy (%)
Test. Accuracy
85
60
60
60
30
45
40
CIFAR-10 Dinchlet 0.1)
CIFAR-10 Dirichlet (a
CIFAR-10 Dirichlet 10)
(d) CIFAR-10 with a=0.1.
(c) CIFAR- with a=1.
(f) CIFAR- 10 with a=10.
Figure 4. The comparison of final test accuracy the NonlID setting of Dirichlet.
88 Sincle average
waman13 15
Eroil air
Test Accurasy1e
Rastom
Maprly Voiig
71
CIFAR-10
122
800
500
700
80
(a) Comparison between naive
(b) Accuracy with different
(c) Accuracy of 3SemiFL (d) Accuracy of different beled server data size
average and Bayesian method
warm-up round in BSemiFL with different thresholds
Figure 5. Results of ablation experiments.
However, Semil-L struggles with high NonIID, while our method remains robust.
assign a pseudo-label if both the global model the local model produce the same pseudo-label for that data point, Our Bayesian method achieves the best performance, in- dicating the importance of adaptively setting weights for local and global models.
7.3. Sensitivity Analysis and Ablation Study
Impact of Ensemble Strategy. Figure shows abla- tion experiments on CIFAR-10 using shards(S=2) Non-IID, Figure 5a compares ensemble strategies: simple averaging, majority voting, random, and our Bayesian method. Simple average refers to assigning equal weights (0.5) to the results produced by the global model and the local model, and then computing a weighted sum. 'Random' refers to randomly assigning two weights to the results produced by the two models and then computing a weighted sum. 'Majority Vote' means that for given unlabeled data point, only
Impact of Warmup. Figure 5b shows results for various warm-up rounds: 10, 20, and 100. Global accuracy differs minimally among methods. Accuracy nearly identical 78% for 10 and 20 rounds, but increases about 79.5% at 100 rounds. This suggests longer warm-up period can modestly enhance performance and overall accuracy.
Impact of Threshold. Figure Sc shows the impact of dif- ferent threshold settings (0.7, 0.8, and 0.9) in BSemi for generating pseudo labels, with other parameters at default
8

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
8. Conclusion
Clknt 18
3.52
Clent 26
Re-labeling local data is crucial Semi-supervised Fed- erated Learning. This paper theoretically and empirically identifies that using only the global or local model lim- its performance due distribution gaps and local errors. Considering these limits, this paper proposes BSemi FL, method that collaborativlly re-labels local data using a weighted ensemble of both models, with weights deter- mined by a Bayesian approach. Our analysis shows that the re-labeling loss of BSemi FI. theoretically smaller than the maximum individual loss of either the global or the local model. Besides, the labeling loss of the Bayesian- based ensemble of the global and local models is also the- oretically smaller than simply averaging them. Empirically demonstrate that BSemiFI improves the performance by up to 9.8% as compared to saat--of-the-are hir-art methods.
Client 72
Cleent 94
0.46
20
100
Round
Figure 6. Value investigation of weight a among different clients.
values (e.g., 20 warm-up rounds). Results indicate minimal differences in final convergence accuracy, all around 78%. This robustness to threshold variations ensures consistent performance and reduces manual hyperparameter tuning costs, enhancing practicality for real-world use.
A Acknowledgement
Impact of Labeled Dataset Size. Figure 5d shows the im- pact of labeled data size (250, 500, 1000 samples) on test accuracy. Accuracy rises significantly with more labeled data, but gains diminish as the dataset grows. Increasing from 250 to 500 samples boosts accuracy from 68% to 78% (a 10% gain), while increasing from 500 to 1000 sam- ples only improves accuracy from 78% to 81% (a 3% gain). This indicates diminishing returns, guiding efficient resource allocation for labeling efforts.
This work is supported by the National Key Re- search and Development Program of China under grant 2024YFC3307900; the National Natural Science Foun- dation of China under grants 62302184, 62376103, 62436003, 62206102, and 62402331; Major Science and Technology Project of Hubei Province under grant 2024BAA008; Hubei Science and Technology Talent Ser- vice Project under grant 2024DIC078; and Ant Group through CCF-Ant Research Fund. This work is also sup- ported by two grants from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU15222621, PolyU15225023), and is supported by the Fundamental Research Funds for the Cen- tral Universities (Grant No. YJ202429), Besides, this work is supported by the National Research Foundation, Sin- gapore, and Cyber Security Agency of Singapore under its National Cybersecurity R&D) Programme and CyberSG R&D Cyber Research Programme Office. Any opinions, findings and conclusions or recommendations expressed in these materials are those of the author(s) and do not re- flect the views of National Research Foundation, Singa- pore, Cyber Security Agency of Singapore as well Cy- berSG R&D Programme Office, Singapore.
Robustness to Data Heterogeneity. Comparing results of different degrees of statistical heterogeneity exhibited in Table and Figure 4, we can find that BSemiFL presents greater advantages over baselines when the data is more heterogeneous. For example, our method outperforms SemiFL by 6% when the shard S 2 while achieving an improvement by 4% when the shard S 8 for SVHN dataset. Besides, it can also be observed that our method achieves similar performance under different NonIID set- tings. For example, the performance of our method ranges from 95.52% to 96.47% when a ranges from 0.1 to 10. As comparison, the performance of SemiFL ranges from 91.56% 96. 96.29%.7 The results indicate that our method is more robust to the degree of NonIlI).
Investigation of Weight a. To investigate the values of weight a during training process, we add the following ex- periment. When each client is selected for training, we record the average value of a for all data points in its lo- cal dataset. We here present the corresponding values for five randomly selected clients, as shown in Figure 6. As can be seen, in the initial rounds, the local model achieves higher accuracy on the local data compared to the global model, resulting in a relatively smaller weight a for the global model. In later rounds, the values of a gradually sta- bilize around 0.5, which aligns with the gradual improve- ment in the accuracy of the global model.
Impact Statement
Our approach enables learning from mostly unlabelled data, promoting resource efficiency. In healthcare and edu- cation, it accelerates knowledge and service improvements, offering high-quality services remote under-resourced areas, thus fostering fairesss. Yet, ensuring algorithmic faimess and transparency remains challenge to avoid in- creasing social inequalities.
9

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
References
Annual Conference on Neural Information Processing Systems, 2024b.
Bai, S., Li, S., Zhuang, W., Zhang, J., Yang, K., Hou, J., Yi, S., Zhang, S., and Gao, J. Combating data imbalances in federated semi-supervised learning with dual regula- tors. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), Thirty-Eightl AAAI Conference on Artificial In- telligence, AAAI 2024, Thirty-Sixth Conference on Inno- vative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Ar- lificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, 2024.
Itahara, S., Nishio, T., Koda, Y., Morikura, M., and Ya- mamoto, K. Distillation-based semi-supervised feder- ated learning for communication-effiieee collaborative training with non-iid private data. IEEE Trans. Mob. Comput., 22( 2023.
Jeong, W. Yoon, J., Yang, E., and Hwang, S. J. Feder- ated semi-supervised learning with inter-client consis- tency & disjoint learning. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. theory of learning from different domains. Machine. Learning, 79(1-2): 151-175, 2010.
Jiang. Z., Xu, Y. Xu, H., Wang, Z., and Qiao, C. Clients help clients: Alternating collaboration for semi- supervised federated learning. In 40th IEEE Interna- tional Conference on Data Engineering, ICDE 2024, Utrecht, The Netherlands, May 13-16, 2024, pp. 1847- 1860. IEEE, 2024.
Cho, Y. J., Joshi, G., and Dimitriadis, D. Local or global: Selective knowledge assimilation for federated learning with limited labels. ference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 17041-17050. IEEE, 2023.
IEEE/CVF International Con-
Kim, T., Lin, E., Lee, J., Lau, C., and Mugunthan, V. Navi- gating data heterogeneity in federated learning: A semi- supervised approach for object detection. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Diao, E., Ding, J., and Tarokh, V. Semifl: Semi-supervised federated learning for unlabeled clients with alternate training. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009.
He, C., Yang, Z., Mushtaq, E., Lee, S., Soltanolkotabi, M., and Avestimehr, A. S. Ssfl: Tackling label deficiency in federated learning via personalized self-supervision. ArXiv, abs/2110.004700 2021.
Lee, S., Le, T. V., Shin, J., and Lee, S. (A) Overcoming few labels in federated semi-supervised learning. CoRR, abs/2410.23227,, 2024.
Hu, M., Cao, Y., Li, A., Li, Z., Liu, C., Li, T., Chen, M., and Liu, Y. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 12528- 12537, 2024a.
Li, Xiong, C., and Hoi, S. C. Comatch: Semi-supervised learning with contrastive graph regularization. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9475-9484, 2021.
Hu, M., Zhou, P., Yue, 7.., Ling, Z., Huang, Y., Li, A., Liu, Y., Lian, X., and Chen, M. Federoss: Towards accurate federated learning via multi-mode! aross-gggre In IEEE International Conference on Data Engineering (ICDE), pp. 2137-2150. IEEE, 2024b.
Liang, X., Lin, Y., Fu, H., Zhu, L., and Li, X. Rscfed: Random sampling consensus federated semi-supervised learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10154-10163, 2022.
Huang, W., Shi, Z., Ye, M., Li, H., and Du, B. Self-driven entropy aggregation for byzantine- robust heterogeneous federated learning- In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024a.
Lin, H. Lou, J., Xiong, L., and Shahabi, C. Semifed: Semi- semi-supervised federated learning with consistency and pseudo-labeling. ArXiv, abs/2108.0941 2021.
Lin, T., Kong, L., Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion federated learning. Advances in Neural Information Processing Systems, 33: 2351-2363. 2020.
Huang, W., Ye, M., Shi, Z., Wan, G., Li, H., and Du, B. Parameter disparities dissection for backdoor defense in heterogeneous federated learning. In The Thirt--ighhth
10

---

BSemiFL: Semi-supervised Federated Learning via a Bayesian Approach
Liu, B., Cai, Y., Zhang, Z., Li, Y., Wang, L.,, Li, D., Guo, Y., and Chen, X. Distfl: Distribution-awer feder- ated learning for mobile scenarios. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 5(4):168: 2021a.
consistency and confidence. Advances in neural infor- mation processing systems, 33:596 6086, 2020.
Tang, Z.,, Zhang, Y., Shi, S., He, x., Han, B., and Chu, X. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In Chaud- huri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference Ma- chine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 21111-11111111 1-21132. PMLR, 2022.
Liu, Q. Yang. H., Dou, Q. and Heng. P.-A. Federated remi-supervised medical image classification via inter- client relation matching, In International Conference Medical Image Computing and Computer-Asisted In- tervention, pp. 325-335. Springer, 2021b.
Liu, Y., Wu, H., and Qin, J. Fedcd: Federated semi- supervised learning with class awareness balance via dual teachers. In Wooldridge, M. J., Dy, J. G., and Natarajan, S. (eds.), Thirty-Fighth AAAI Conference Artificial Intelligence, AAAI 2024, Thirty-Sixth Con- ference on Innovative Applications of Artificial Intel- ligence, IAAI 2024, Fourteenth Symposium on Educa- tional Advances Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 3837- 3845, 2024.
Wang, H.,, Li, Y., Xu, W., Li, R., Zhan, Y., and Zeng, Z. Dafkd: Domain-aware federated knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 20412-20421. IEEE, 2023a.
Wang, J., Zeng, S., Long, Z., Wang. Y., Xiao, H., and semi-supervised feder-
Ma, F. Knowledge- ated learning for aggregating heterogeneous lightweight clients in iot. In Proceedings of the 2023 SIAM In- ternational Conference on Data Mining, SDM 2023, Minneupolis-SL Paul Twin Cities, MN, USA, April 27- 29, 2023, pp. 496-504. SIAM, 2023b.
Lubana, E., Tang, C., Kawsar, F., Dick, R., and Mathur, A. Orchestra: Unsupervised federated learning via globally consistent clustering. May 2022.
Xu, Y., Wang. L.., Xu, H., Liu, J., Wang, Z.., and Huang, L.. Enhancing federated learning with server-side unlabeled data by adaptive client and data selection. IEEE Trans. Mob. Comput., 23(4):281 (111)::81132200 3--2811, 2024.
Malaviya, S., Shukla, M., Korat, P., and Lodha, S. Fed- fame: A data augmentation free framework based on model contrastive learning for federated semi-supervised learning. In Proceedings of the 38th ACM/SIGAPP Sym- posium on Applied Computing, SAC 2023, Tallinn, Esto- nia, March 27-31, 2023, pp. 1114-1121, ACM, 2023.
Yang. N., Yuan, D., Liu, Z., Deng, Y., and Bao, W. Fedil: Federated incremental learning from decentralized un- labeled data with convergence analysis. arXiv preprint arXiv:2302.1 / 1823, 2023.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and learning of deep
Arcas, B. A. Communication-efficii networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, 2017.
Yang. Z., Peng, Z., Wang, Z., Qi. J., Chen, C., Pan, W., Wen, C., Wang, C., and Fan, X. Federated graph learning for cross-domain recommendation. Advances in Neural Information Processing Systems 37: Annual Conference Neural Information Processing Systems, NeurlPS 2024, 2024.
Netzer, Y., Wang. T., Coates, A., Bissacco, A., Wu, B., and Ng. A. Y. Reading digits in natural images with unsupervised feature learning. 2011.
Qi, Zhuang, M. L. and et al. Cross- silo feature space align- ment for federated learning on clients with imbalanced data. In The 39th Annual AAAI Conference on Artificial Intelligence (AAAI-25), pp. 19986-19994, 2025.
Yang Xu, Yumming Liao, X. Z. S. L. S. C. Q. Semisfl: Split federated learning on unlabeled and non-iid data. CORR, 2023.
Zagoruyko. S. and Komodakis, N. Wide residual networks. In Procedings of the British Machine Vision Conference 2016, Jan 2016. doi: 10.5244/c.30.87. URL http: / //xx. doi. orgioorg/ia 52.40..2444.... 30.
Qi, Z., Meng, L., Chen, Z., Hu, H., Lin, H., and Meng. X. Cross-silo prototypical calibration for federated learning with non-ii data. In Proceedings of the 31st ACM In- ternational Conference on Multimedia, pp. 3099-3107, 2023.
Zhang, C., Wu, F., Yi, J., Xu, D., Yu, Y., Wang, J., Wang, Y., Xu, T., Xie, X., and Chen, E. Non-iid always bad? semi-supervised heterogeneous federated learning with local knowledge enhancement. In Proceedings of the 32nd ACM International Conference on Information
Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C. A., Cubuk, E. D., Kurakin, A., and Li, C.- L. Fixmatch: Simplifying supersupe learning with

---

BSemiFL: Semi-supervised Federated Learning via Bayesian Approach
and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pp. 3257-3267. ACM, 2023a.
Zhang. J., Ma, X., Guo, S., and Xu, W. Towards unbiased training in federated open-world semi-supervised learn- ing. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Re- search, pp. 41498-41509 PMLR, 2023b.
Zhang, Y., Wang. X., Wang. P., Wang, B., Zhou, Z., and Wang, Y. Modeling spatio-temporal mobility across data silos via personalized federated learning. IEEE Trans. Mob. Comput.. 23(12); 12899)) 2024a.
Zhang, Y., Yang. Z., Tian, X., Wang, N., Liu, T., and Han, B. Robust training of federated models with extremely label deficiency. In The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024b.
Zhang. Z., Yang, Y., Yao, Z., Yan, Y., Gonzalez, J., Ram- chandran, K., and Mahoney, M. W. Improving semi- supervised federated learning by reducing the gradient diversity of models. 2021 IEEE International Confer- ence on Big Data (Big Data), pp. 1214-1225, 2021.
Zheng, M., You, S., Huang, L., Wang, F., Qian, C., and Xu, C. Simmatch: Semi-supervised learning with similar- ity matching. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 14471-14481, 2022.
Zhu, G., Liu, X., Wu, X., Tang, S., Tang, C., Niu, J., and Su, H. Estimating before debiasing: A bayesian approach to detaching prior bias federated semi-supervised learn- ing. In Proceedings of the Thirty- Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 2625-2633. ij- cai.org, 2024.
Zhuang. W., Gan, X., Wen, Y., Zhang, S., and Yi, Collaborative unsupervised visual representation learn- ing from decentralized data. In 2021 IEEE/VVF Inter- national Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 4892- 4901, 2021.
Zhuang, W., Wen, Y., and Zhang, S. Divergence-aware federated self-supervised learning. Apr 2022.
12

---

BSemiFL: Semi-supervised Federated Learning via Bayesian Approach
A. Proofs
Preliminary of Theories. Our following analysis relies on the following theorem which bounds the transferring error by adapting the model of one domain another domain.
Proposition A.I. (Theorem in (Ben-David el al., 2010)). Considering the distributions and pt. for every hypothesis f E FF and any constant E (0, 1), with probability at least 1 - 8, there exists
(p((f) S (f) dd + 2,
(18)
where dFaF(Ps.Pi Pt) measures the distance between the distribution Ps and pv. Lp. (f) denotes the expected labeling error the hypothesis over the data distribution Pa. i.e., Lpp. (f) Exwp, Uff x), and Lp (f) has a similar meaning to Lp. (f). = Lpp (J+) + [pp((a) (Ja) with f- = arg feF Lvp, (f) + (f)-
Theorem A.2. Denote the data distribution of each client m by Pms the empirical distribution of the global dataset by P- and the real distribution of the global data by p. Given constants < S 1 and a > with the probability at least 8, the expected labeling error Lpm (ft) of the global hypothesis fg over the local data distribution pm is bound by:
log 2
Len (So) S Lo(s)) +
2 dFaF((pmsD) + Ag
(19)
25
where = Σm= Sm + N2 denotes the total data size of the global dataset, and 0g = Lp(f.) + Lpm (f=) with fa = arg min + Lp.. (f).
Proof. For convenience expression, we denote the hypothesis (ax w) by f and denote arg min [sp(J) by Sp- Ac- cording to the Hoeffding inequality, with probability at least 5, there exists
log
(pp(s)) S Lp(sa) +
(20)
25
Then, by denoting the distribution of the local data in client m as Pm: based on the Proposition we have
pm (fp) S Lp(fs) ddrar(pm p) + Ag.
(21)
Jointly considering (20) and (22) together, have
log 2 25
Lpn S
dFAA(PmmPP) Ag.
(22)
which completes the proof.
Theorem A.3. Denote the hypothesis trained the empirical distribution of the local dataset Pm by fpm. and the hypoth- esis trained the empirical distribution of the global datasel by fç. For any local data sample Pms considering the Cross-Entropy loss function is adopted, then the loss of the ensemble between the global and local model is less than the maximam individual loss of either the global or local model, i.e.,
L(af; + (1 al)-pf))m))) I max LLff)),((fpm))
(23)
Proof. Considering the convexity of the Cross- Cross-Entropp loss function in terms of the prediction, for any x ~ Pmo we have
L(afa + (1 - -a)fpm) < aL(fp) + (1 L(f
< max (L(fp), L(fe.))
(24)
13

---

BSemiFL: Semi-supervised Federated Learning via Bayesian Approach
Theorem A.4. Consider the same notations distribution and hypothesis defined in previous theorems. For each local sample Pm with the ground-truth label y k. considering the Cross-Entropy loss function is adopted, then under the condition that fo(k)z) > fo. (k]z)) when p(x) > (m) and vice versa, the loss of Bayesian-based model ensemble is smaller than the direct averaging:
Maa + (1 11-111)1ppp (...).)) S L I +p
(25)
Proof. According to the definition of the Cross-Entropy loss, we have
Maafp + (1 a.).p..
1og(a(z))f + pm (k12))
((x)
Pm (2)
log ((x) + Pm (x) fx(k(22) + p(x) + pm (x) pm (k1)z)
fp(k((z) + fpm (k(az)
(p(x) pm (x))
-f (61)z)
log
2
2(0(x) Pm (x))
((a(k(z)) + fam (M/xz)
S llog
L( fp + 1pm
(26)
2
where the last inequality is derived based on the condition that (0(x)) (F))))(()(((()) -sm. (k|z)) always positive.
B. Algorithm Workflow
The workflow is presented in Algorithm 1. In lines 4-5, the server first trains the global model on the labeled dataset and then calculates the sum of sampling probability. Next, lines 6-8, the server distributes the global model and probability sum to selected clients. Each client mainly conducts two main steps as follows.
Labeling local data (lines 14-21):
1. Each client m calculates the closeness of the global model in lines 16-17;
2. Each client calculates the closeness of the local model in lines 18-19;
3. Each client m applies an ensemble of the global and local model to construct the labeled dataset lines 20-22.
Local training (lines 24-25):
1. Each client m performs local training with labeled dataset for local epochs;
2. Each client m pushes the local model to the server.
Finally, in lines 9-11, the server aggregates the local models of all participated clients into the global model.
14

---

BSemiFL: Semi-supervised Federated Learning via Bayesian Approach
Algorithm Algorithm workflow of BSemiFL
Input :
T: round; M: client number; vy: learning rate;
Initialize the parameter Wo: In server:
for = to T do
Fine-tune the global model w, the dataset S;
Calculate the sum of sampling probability of the server dataset S for all K classes: Q =i Sa(k(o,, we);
Randomly select Mr clients;
for each selected client ra in parallel do
Send the global model w' and probability vector Qsi Receive the local model w.":
end
Aggregate local models: We+1 = DMM Sm w
8
.end
7 In client me
for each sample x²" E Um do
Calculate the output of the global model f=(";", we) and of the local model fm (N.²², Wimji
Calculate the conditional probability of the global model: p(x;" Tp(.FT w..++y
fa(lle"" w.)
for all classes k 1, ....,, K;
Calculate the closeness of the global model: p(x:") = Ek= p(x [k);
Jom (Klo"" w(T) for all K classes;
Calculate the conditional probability of the local model: Pm (x!" [k)
Σim fam (HIT).
Calculate the closeness of the local model: Pm (x." = Σ k= Pm (2i77 (k))
M(a7))
(x:7))
Normalizes the weights: a." p(x;"" Pm (xf") 1 a, (mm")) (e:²))
Apply an ensemble of the global and local model: 0 as fa(x." we) + (1 a. )fp (I w("); Constructs the dataset S., by conducting (2);
end
10 Update local model w." for E' local epochs on the labeled dataset S. w w V wi Lm:
Send the model w to the server;
15


Reviews:
[{"review_id": "a7oN7FLB0z", "reviewer": "Reviewer_wPXP", "summary": "This study focuses on the semi-supervised learning paradigm within Federated Learning (FL), focused on re-label technology.  Theoretical and empirical results demonstrate the local model has higher relabel accuracy on local data. Furthermore, this paper propose a Bayesian approach to re-label the local data by both the local and global model. Specifically,  bayesian inference to choose model and get weighted combination of pseudo labels. Theoretical analysis shows the lower labeling error and experimental also gives SOTA results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your reviewing and valuable suggestions.\n```\nQ1: How to get the p^(xi,k) in 5.1. It is better to use a consistent notation.\n```\n$p^(x_i,k)=p^(x_i,y_i=k)$ represents the joint probability distribution of the sample $x_i$ and $y_i$, i.e., the probability that the sample takes the value $x_i$ while the label simultaneously takes the value $k$. In fact, our notation is consistent throughout the paper. However, due to the relatively large number of variable types involved, it may give an impression of complexity. We will further optimize the explanation and representation of the variable notations moving forward.\n\n```\nQ2：p and p^ in eq.(20)? It is unclear how to get (20) from Hoeffding.\n```\nEq.(20) involves derivation steps that are commonly used in empirical risk analysis, so we omitted these steps. Since $L_p(f)$ is the expectation of $L_{\\hat{p}}(f)$, by applying Hoeffding's inequality, we can derive:\n$$\n\\mathbb{P}\\left( |L_{\\hat{p}}(f) - L_p(f)| \\geq \\epsilon \\right) \\leq 2 \\exp(-2S\\epsilon^2)\n$$\nHere, $ \\epsilon > 0 $ is an arbitrarily small positive number. Let:\n$$\n\\delta = 2 \\exp(-2S\\epsilon^2)\n$$\nSolving for $ \\epsilon $:\n$$\n\\epsilon = \\sqrt{\\frac{\\log\\frac{2}{\\delta}}{2S}}\n$$\nThis implies that with a probability of at least $ 1 - \\delta $, the following inequality holds:\n$$\n|L_{\\hat{p}}(f) - L_p(f)| \\leq \\sqrt{\\frac{\\log\\frac{2}{\\delta}}{2S}}.\n$$\nDecomposing the above absolute value inequality yields formula (20). We will include these in the revised version.\n\n```\nQ3：Experiments of \"Impact of Labeled Dataset Size\" lack the specific heterogeneity, the number of clients, and other settings.\n```\nThe total number of clients is 100, with 10% activated in each round. The heterogeneity among clients is set to a shard distribution with \\( k=2 \\). The threshold is 0.7.\n```\nQ4:α visualization.\n```\nWe add the following experiment. When each client is selected for training, we record the average value of α for all data points in its local dataset. We here present the corresponding values for five randomly selected clients in the table below. As can be seen, in the initial rounds, the local model achieves higher accuracy on the local data compared to the global model, resulting in a relatively smaller weight α for the global model. In later rounds, the values of α gradually stabilize around 0.5, which aligns with the gradual improvement in the accuracy of the global model.\n\n|Client ID&Round|10|20|40|60|Last|\n|-|-|-|-|-|-|\n|18|0.4531|0.4880|0.5060|0.5146|0.5287|\n|26|0.4643|0.4857|0.4951|0.5101|0.5061|\n|46|0.4477|0.4781|0.4827|0.5045|0.5069|\n|72|0.4585|0.4756|0.4926|0.5088|0.5093|\n|94|0.4602|0.4829|0.4943|0.5023|0.5303|\n\n```\nQ5:Meaning of “global model can progressively...”. The difference between a local model with a high capability and a global model with a progressively higher capability.\n```\n- This statement is relative to the local model. In Fig.2b, we observed that when labeling data using only the local model, some labeling errors occur. When these labeled data are used to train the local model, the errors are carried forward, and during the next round of labeling, the same errors persist, making it impossible for the model to self-correct. On the other hand, the global model, which incorporates knowledge from other clients, can correct these errors.\n  \n- Since the local model converges quickly while the global model converges more slowly, there is a significant difference in accuracy between the two in the early stages of training, but they gradually converge later on. However, due to the impact of heterogeneity, the error of the global model on the local dataset remains consistently higher than that of the local model. \n```\nQ6：Motivation does not clearly explain which model to train and what data are used. How are the global and local model obtained in Fig.2a?\n```\nAll motivation experiments used the Wide ResNet28*2 model. As indicated in lines 151 and 208, we used CIFAR10. All experiments were conducted to train the global model.\n\n- In all experiments, the global model is obtained on the server using FedAvg, while the local model is obtained after further local training of the global model.\n  \n- In all figures, the global model is tested using the combined local data from all clients (i.e., the global data), whereas the local model is tested using the local data corresponding to its respective client. This is explained in the captions.\n\nThus:\n- In Fig.2a, as in lines 150–154, the test data for the global model is the global data, while the test data for the local model is the local data corresponding to its respective client.\n- In Fig.2b, the global model is tested using two different annotation methods, and the test data used is the global data.\n- In Fig.2c, the local model is tested using its corresponding local data.\n```\nQ7: Lacks in some recent citations in “labels at client” [1,2].\n```\nThank you for the reminder. We will include them.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "- It is unclear how we can get the $\\hat{p}(x_i,k)$. 5.1 part, the notation are confused, it is better to use a consistent notation to represent this bayesian inference process, because this process is the same for global model and local model.\n- In the equation (20), what’s the meaning of $p$ and $\\hat{p}$? It is also unclear how to get the equation (20) based on the Hoeffding inequality.", "experimental_designs_or_analyses": "- The experiments about \"Impact of Labeled Dataset Size\" lack the specific heterogeneity of the experiment, the number of clients, and other specific settings.\n- Lacks the $\\alpha$ visualization experiments to demonstrate whether the model depends more on the global model to label as the global model improves in performance, so it is recommended to add these experiments.", "relation_to_broader_scientific_literature": "New insights about the relationship between the global and local models to relabel data in SSFL.", "essential_references_not_discussed": "The related work in ““labels at client” part lacks some recent citations e.g., [R1,R2]\n\nReferences:\n\n[R1] Zhang, Yonggang, et al. \"Robust Training of Federated Models with Extremely Label Deficiency.\" *The Twelfth International Conference on Learning Representations*.\n\n[R2] Bai, Sikai, et al. \"Combating data imbalances in federated semi-supervised learning with dual regulators.\" *Proceedings of the AAAI conference on artificial intelligence*. Vol. 38. No. 10. 2024.", "other_strengths_and_weaknesses": "Weakness:\n- What’s the meaning of “the global model can progressively improve the re-labeling performance by introducing the extra data knowledge of other clients” in the abstract part? What is the difference between a local model with a high relabel capability and a global model with a progressively higher relabel capability?\n- In the motivation part, all experiments do not clearly explain which model to train and what training data and test data are used, so it is a little difficult to follow. For example, how are the global model and local model obtained in Figure 2a, and what are their test data respectively", "other_comments_or_suggestions": "See weakness", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLCrrNBGT9", "reviewer": "Reviewer_kFQg", "summary": "This paper aims to solve the problem in Semi-supervised Federated Learning where the local data labels are absent in clients. They first theoretically and empirically demonstrate that the limitations and benefits of local model and the global model for relabeling the local data. They propose a new method which re-labels the local data through the collaboration between the local and global model in a Bayesian approach. Finally, they theoretically empirically demonstrate the effectiveness of their proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your reviewing and valuable suggestions.\n```\nQ1：more recent advanced SSL methods [1] can considered.\n[1] BEM: Balanced and Entropy-Based Mix for Long-Tailed Semi-Supervised Learning. CVPR 2024.\n```\nBEM [1] mainly proposes a novel hybrid method for rebalancing the class distribution in terms of data quantity and uncertainty. This method can be integrated into our proposed approach.\n\n```\nQ2：The proposed method requires more interaction items except of the models between the clients and the server. The risk of the privacy leakage is not discussed.\n```\nAlthough our method requires more interaction terms, it does not actually increase the risk of privacy leakage. Compared to traditional FL methods, as shown in line 681, our method only needs to additionally transmit the K-dimensional probability distribution vector $Q_s=[Q_s^1,...,Q_s^K]$ of the server-side public dataset categories on the global model. In the SSFL scenario, the data on the server side is generally non-private. Therefore, in practice, our method does not result in additional client privacy leakage compared to traditional FL methods.\n\n```\nQ3：There exists a SSFL method using Bayesian approach [1]. What’s the difference between your proposed method their method?\n[1] Estimating before Debiasing: A Bayesian Approach to Detaching Prior Bias in Federated Semi-Supervised Learning. IJCAI 2024.\n```\nFirst, the scenario of our method differs from that of the referenced article:\n\nIn our scenario, we assume that clients only have unlabeled data, while the server has a small amount of labeled data. In contrast, in [1], the scenario assumes that clients possess both supervised and unsupervised data, while the server does not have any data.\n\nSecond, regarding the application of Bayes' theorem:\n\n1. \nOur method leverages Bayes' theorem to assign pseudo-labels to local data based on the knowledge of both the local model and the global model. Specifically:\nWe first fine-tune the global model on the server using the labeled data available on the server:  \n   $ Q_s^k = \\sum_{i=1}^{N_s} f_{\\hat{p}}(k|x_i, w_t) $\n\nand then compute the probability distribution of the server-side data on the global model.\nThis parameter, along with the global model, is distributed to the clients. Subsequently, we calculate:  \n\n   $$\n   \\hat{p}(x_i^m|k) = \\frac{f_{\\hat{p}}(k|x_i^m, w_t)}{f_{\\hat{p}}(k|x_i^m,w_t) + Q_s^k}\n   $$\nFor each data point in the local client, we compute:  \n\n   $$\n   \\hat{p}(x_i^m) = \\sum_{k=1}^K \\hat{p}(x_i^m|k)\\hat{p}_s(k).\n   $$\n\nSimilarly, in the local model, we compute the empirical distribution (using pseudo-labels from the previous round) and the probability distribution of the local data:  \n\n   $ Q_m^k = \\sum_{i=1}^{S_m} f_{\\hat{p}_m}(k|x_i,w_t^m) $.  \n\n   Then, we calculate$A=\\sum_{k=1}^K f_{\\hat{p}_m}(k|x_i^m, w_t^m)\\hat{p}_m(k)$, \n\nand $B=(f_{\\hat{p}_m}(k|x_i^m, w_t^m)+Q_m^k)$\nto obtain $\\hat{p}_m(x_i^m)=A/B$\n\nNext, for each local data point, we obtain the confidence levels of the global model and the local model as:  \n   $$\n   \\alpha_i^{m} = \\frac{\\hat{p}(x_i^m)}{\\hat{p}(x_i^m)+\\hat{p}_m(x_i^m)}, \\quad 1-\\alpha_i^{m} = \\frac{\\hat{p}_m(x_i^m)}{\\hat{p}(x_i^m)+\\hat{p}_m(x_i^m)}.\n   $$\n\nFinally, by denoting $C=f_{\\hat{p}}(x_i^m, w_t)$, and $D=f_{\\hat{p}_m}(x_i^m, w_t^m)$, we derive the final probability distribution:  \n\n   $$\n   \\hat{y}_i^m =  \\alpha_i^m C+ (1-\\alpha_i^m) D\n   $$\n\n2. \nIn contrast, in FedDB, the use of Bayes' theorem occurs during local training, where the knowledge of the local model is utilized to optimize pseudo-labeling via APP-U (the Average Prediction Probability of Unlabeled Data), thereby reducing label prior bias. Specifically, it is argued that the imbalance in the local dataset introduces bias when the local model assigns pseudo-labels to unlabeled data, favoring classes with more data. To address this issue, the authors propose applying Bayes' theorem to rewrite:  \n   $$\n   p_s(y|x) = \\frac{e^{z(x)[y]}}{\\sum_{k=1}^{K} e^{z(x)[k]}}\n   $$  \n   as:  \n   $$\n   p_s(y|x) = \\frac{p_s(y) p_s(x|y)}{\\sum_{k=1}^{K} p_s(k) p_s(x|k)}.\n   $$\nTo mitigate the imbalance issue, a regularization-like term is introduced, yielding:  \n   $$\n   \\hat{p} = \\frac{p(y|x)/\\overline{p}}{\\sum_{k=1}^{K} {p(k|x)}/\\overline{p}_k},\n   $$  \n\nThus, our method primarily uses Bayes' formula to combine the knowledge of the global model and the local model through pseudo-labeling, whereas the work in [1] focuses on addressing data imbalance issues.", "claims_and_evidence": "This paper claims that using a single model (e.g., a local or global model) for relabeling the local data will lead to poor generalization or personalization. They verify this from both theoretical and empirical perspectives. Besides, they claim that Bayesian based ensemble can solve this problem. Their established theories demonstrate that the achieved performance outperforms a single global or local model or their simply average.", "methods_and_evaluation_criteria": "They propose using the weighted ensemble of the local and global model to re-label the local data, where the weights are calculated by the Bayesian approach. They evaluate the proposed method by comparing 8 baselines over both shard and Dirichlet based NonIID distribution settings, which are sufficient to me. Besides, the ablation study also verify the effectiveness of the Bayesian based ensemble.", "theoretical_claims": "Their theories can be divided into two main parts. The first part includes Theorem 4.1 and 4.2, which claim that using a single model (e.g., a local or global model) either cannot achieve generalization or cannot fill the distribution gap. The second part includes Theorem 6.1 and 6.2, which claim that their proposed method outperforms using a single model or the simple average of two models. The claims are consistent with the empirical observation and the proofs are correct.", "experimental_designs_or_analyses": "They use SVHN, CIFAR10, and CIFAR100 as the evaluation dataset and Resnets as  models, which is a generalized adopted benchmark. They compare their proposed method with 8 baselines including both SSFL and Unsupervised FL methods, which are sufficient enough to verify the effectiveness. The analysis also explains the reason for the phenomenon.", "relation_to_broader_scientific_literature": "The field of this paper focus on is de-facto a combination of the semi-supervised learning and federated learning. Although the proposed method achieves great effectiveness for SSFL, they only adopt a basic semi-supervised learning method in FL. In fact, more recent advanced SSL methods [1] can considered.\n\n[1] BEM: Balanced and Entropy-Based Mix for Long-Tailed Semi-Supervised Learning. CVPR 2024.", "essential_references_not_discussed": "The references are sufficient..", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe idea of using Bayesian based method is interesting and novel. The design of weighted ensemble to leverage both benefits of two models makes sense.\n\n2.\tThe motivation is clear. The analysis reveal the underlying principles behind the problem using a single global or local model.\n\n3.\tThe techniques are sound and the established theories are solid, which corresponds to the empirical observations.\n\n4.\tThe paper is well organized and the writing is good.\n\n5.\tThe evaluations are sufficient which many recent baselines and various settings.\n\nWeakness:\n\n1.\tThe proposed method requires more interaction items except of the models between the clients and the server. The risk of the privacy leakage is not discussed.\n\n2.\tThere exists a SSFL method using Bayesian approach [1]. What’s the difference between your proposed method their method?\n\n[1] Estimating before Debiasing: A Bayesian Approach to Detaching Prior Bias in Federated Semi-Supervised Learning. IJCAI 2024.", "other_comments_or_suggestions": "No", "questions_for_authors": "See concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQj2I6nsy0", "reviewer": "Reviewer_uy97", "summary": "This paper focus on the semi-supervised scenarios in Federated Learning. This paper delves deeply into the performance dominance and limitations of the global and local models for relabeling the local data from both theoretical and empirical perspectives. Then, they propose a novel method which re-labels the local data through the collaboration between the local and global model in a Bayesian approach. Established theories demonstrate the effectiveness of their proposed method. Experimental results also show that their method greatly improves the performance.\n\n## update after rebuttal\nAfter the Reviewer-author discussion phase, I maintain my score and explicitly support acceptance.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your reviewing and valuable suggestions.\n\n```\nQ1：The details of the designed ensemble strategies in Figure 5(a) are not specified. It seems that these methods are designed by this paper itself. However, the details are not clear. For example, what’s the ,meaning of the majority voting? In my opinion, there are only two models, which has the concept of “majority”.\n```\nWe apologize for the lack of clarity in some of our descriptions:\n\nIn Figure 5(a):\n- **Simple average** refers to assigning equal weights (0.5) to the results produced by the global model and the local model, and then computing a weighted sum.\n- **Random** refers to randomly assigning two weights to the results produced by the two models and then computing a weighted sum.\n- **Majority Vote** means that for a given unlabeled data point, we only assign a pseudo-label if both the global model and the local model produce the same pseudo-label for that data point.\n\n\n```\nQ2：I'm curious as to why your method performs worse than Orchestra when it comes to Cifar100 (100, 2)?\n```\nThank you for your feedback.\n\nFirst, our method achieves a performance of **39.09±3.12** on CIFAR100 (100, 2), while Orchestra achieves **39.93±2.17**. These results are based on multiple experiments, so the performance of our method is not inferior to that of Orchestra. Regarding the slightly lower average value compared to Orchestra, we believe this is because Orchestra is a federated self-supervised learning method. In this specific scenario, its better performance is mainly due to the fact that the client data distribution is highly concentrated in a few categories. This local consistency makes it easier for local clustering to capture clear category patterns. \n\nIn contrast, our method does not rely on clustering but instead leverages the collaboration between the global model and the local model to assign pseudo-labels. Given the higher complexity of CIFAR100 images compared to other datasets, this leads to slightly higher fluctuations in our method compared to Orchestra.", "claims_and_evidence": "This paper presents the challenge of using single model from both theoretical and empirical perspectives.", "methods_and_evaluation_criteria": "The method is well elaborated and has strong motivations. The designed evaluations are sufficient to verify that the method can solve their identified problem, meeting the criteria.", "theoretical_claims": "The theories clearly illustrate the limits of using single models and guarantee the effectiveness of the proposed method. The proofs are clear and correct.", "experimental_designs_or_analyses": "The experimental designs are comprehensive and reasonable to evaluate the proposed method. Specifically, the experimental datasets and tasks are widely adopted in this SSFL area, which meets the criteria. Besides, the compared baselines are sufficient enough to demonstrate the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "This paper focuses on the integration of semi-supervised learning and federated learning, with particular attention to the issue of inaccurate labeling caused by NonIID. Therefore, it is related to works on semi-supervised learning in centralized scenarios [1] and efforts addressing NonIID in federated learning [2]. \n[1] Z. Zhu et al. The rich get richer: Disparate impact of semi-supervised learning. ICLR 2022.\n[2] SP Karimireddy et al. Scaffold: Stochastic controlled averaging for federated learning. ICML 2020.", "essential_references_not_discussed": "The references are sufficient.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe empirical and theoretical analysis about the limits and benefits of using single models are sound and clearly present the motivations.\n\n2.\tThe proposed method of using Bayesian-based ensemble of the global and local model to relabel the local data is interesting and innovative.\n\n3.\tThe theoretical validation of the performance is solid and guarantees the effectiveness of the proposed method.\n\n4.\tThe presentation is good and clear, making the paper easy to follow.\n\n5.\tThe experiments compare their method with many baselines over different datasets and NonIID settings, which are sufficient.\n\nWeakness and Concerns:\n\n1.\tThe details of the designed ensemble strategies in Figure 5(a) are not specified. It seems that these methods are designed by this paper itself. However, the details are not clear. For example, what’s the ,meaning of the majority voting? In my opinion, there are only two models, which has the concept of “majority”.\n\n2.\tI'm curious as to why your method performs worse than Orchestra when it comes to Cifar100 (100, 2)?", "other_comments_or_suggestions": "No.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IIOx05thx0", "reviewer": "Reviewer_YYii", "summary": "1. This paper proposes BSemiFL, a federated semi-supervised learning framework. BSemiFL theoretically demonstrates why solely relying on either the global model or the local model for labeling local data is suboptimal.\n2. The authors employ a Bayesian approach to evaluate the proximity of local and global models to the samples, and then dynamically weight their contributions to pseudo-label prediction based on their inferred relevance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the reviewing.\n\n```\nQ1.FedDB[3] also employs Bayesian in FSSL. FedDB explicitly and mitigates class prior bias from imbalanced data. BSemiFL does not address internal class bias. Additionally, BSemiFL’s analysis is confined to its aggregation strategy and lacks a comparative evaluation.\n```\nAlthough [3] also employs Bayesian methods, the scenarios in which the two methods are applicable are entirely different. The scenario considered by [3] is \"labels at client,\" which may not be suitable for our \"labels at server\" scenario. The core step of [3] involves using a single model to annotate local data and suppress the prediction probabilities of the majority class to correct annotation errors of that single model.\n- However, this method requires calculating the prior probability of the majority class, which relies heavily on labeled data from the \"labels at client\" setting. When there is no labeled data locally, an initial miscalculation of the prior probability will exacerbate errors during the suppression process, and there would be no way to correct these priors based on labeled data later on. In contrast, our proposed method does not depend on local labeled datasets but instead leverages a Bayesian weighting approach between the local and global models to jointly correct prediction probabilities.\n- The fundamental idea of [3] is a weighted inference method for addressing class imbalance, which has inherent limitations. For example, in a three-class classification task, assuming class C1 is the majority class and classes C2 and C3 are minority classes with equal sample sizes, the trained model might tend to classify C2 and C3 as C1. Although [3] penalizes the majority class, due to insufficient training on minority classes, it still cannot distinguish between C2 and C3, leading to misclassification. Therefore, widely used methods in the field of class imbalance are oversampling rather than weighting. Our method adaptively assigns the labeling of the majority class to the local model and the labeling of minority classes to the global model, thereby avoiding such errors.\n\nCifar10\n\n| |dir d=0.1| shard k=2|\n|-|-|-|\n|FedDB|66.78|66.14|\n|Our|76.19|78.31|\n\nThus, [3] and our method apply to different scenarios. Indeed, combining the error correction of a single model from [3] with the correction of integrated annotations from two models in our method could potentially lead to further improvements and broader applicability across various scenarios.\n```\nQ2. Theories have limitations: the paper assumes a known local distribution. using pseudo-labels to estimate the distribution has noise.\n```\nAlthough the pseudo-label estimation distribution contains some noise, as training progresses, the model's accuracy improves, and this estimation error gradually decreases. Moreover, while the theorem has certain limitations, it still demonstrates the effectiveness of the method to a certain extent. It shows that under approximate conditions, the method outperforms using a single model or a simple average of two models.\n\n```\nQ3：Novelty is somewhat limited. The limitations of data heterogeneity have been recognized in FL. Although this work seeks to extend it to FSSL, the characteristics are not sufficiently reflected.\n```\nThis paper investigates multiple characteristics of FSSL, including but not limited to:\n- The first systematic exploration of the advantages and disadvantages of local-only and global-only labeling methods, as well as the underlying reasons, which has not been addressed in traditional FL.\n- The first proposal of an adaptive weighting integration of global and local models tailored to the labeling needs in FSSL, a unique feature in FSSL.\n- Our theoretical analysis specifically targets the accuracy of labeling in FSSL.\nAdditionally, we adopt an approximation of pseudo-label distributions and do not require prior knowledge of the local data distribution.\n\n```\nQ4. The paper ignores the cumulative amplification of pseudo-label errors across multiple rounds. The theories focus on a single re-labeling step, overlooking the long-term accumulation of noise.\n```\nOur method involves labeling in conjunction with the global model, which can be refined using the global labeled dataset, thereby reducing such noise. This is also reflected in SemiFL, where using only the global model can gradually lead to better performance without amplifying cumulative errors.\n\n```\nQ5.Many big models were used in FL. Their knowledge helps alleviate the imbalance. How to evaluate improvements of the method compared to big models in pseudo-label generation? Can theories incorporate big models?\n```\nOur method can be combined with foundational models to a certain extent. For example, we can treat the foundational model as the global model and integrate it with the local model to label local data. Considering the differences between foundational models and global models, new theoretical and experimental analyses may be required.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, it is practically meaningful. BSemiFL enables the dynamic adjustment of the contributions from local and global models in pseudo-label generation, which provides a certain contribution to addressing data heterogeneity in federated semi-supervised learning. However, its novelty is somewhat limited.", "theoretical_claims": "Starting from the empirical distributions of the global dataset and local datasets, it theoretically proven that the ensemble loss between the global model and local models is lower than the maximum individual loss of either the global model or any local model.", "experimental_designs_or_analyses": "Yes, I have checked it. The main contribution of this paper lies in its ensemble strategy. In the experimental section, the authors compare their proposed strategy with other ensemble approaches and demonstrate its superior performance.", "relation_to_broader_scientific_literature": "Previous studies in the field of FSSL have also utilized both local and global models for pseudo-label generation, such as [1,2]. However, these works did not provide theoretical analysis to demonstrate the limitations of relying on a single model.", "essential_references_not_discussed": "FedDB[3] also employs Bayesian analysis in the federated semi-supervised learning setting. \n\nCompared to this work: FedDB explicitly models class prior bias arising from imbalanced data, aiming to mitigate its impact throughout the training process. In contrast, BSemiFL focuses solely on weighting the predictions of global and local models, without addressing the internal class bias of the model or adjusting the prior distribution within the predictions. Moreover, the analysis in BSemiFL is limited to its aggregation strategy and does not provide a comparative evaluation against the debiasing strategy proposed in FedDB.", "other_strengths_and_weaknesses": "Strengths: \n1. The writing is clear and fluent. \n2. One of the main contributions of this paper lies in its ensemble strategy. In the experimental section, the authors appropriately validate this contribution by comparing their method with other ensemble strategies.\n \nWeaknesses:\n1. The theoretical analysis has certain limitations. Specifically, the paper assumes that the local data distribution is known. However, in semi-supervised settings, the local distribution is typically unknown. Although pseudo-labels can assist in estimating the distribution, they are prone to errors. The impact of pseudo-label noise is not considered in the knowledge derived from the local models.\n \n2. The novelty is somewhat limited. The limitations of local and global models under data heterogeneity have become widely recognized in federated learning. Although this work attempts to extend such insights to FSSL, the characteristics of FSSL are not sufficiently reflected—particularly due to the assumption of known local data distributions.", "other_comments_or_suggestions": "This paper ignores the cumulative amplification of pseudo-label errors across multiple rounds:The theoretical analysis focuses on a single re-labeling step, overlooking the long-term accumulation and propagation of pseudo-label noise through iterative training rounds, which can degrade model performance over time.", "questions_for_authors": "Currently, many foundation models, such as CLIP, have been introduced into FL and FSSL, where their prior knowledge helps alleviate the problem of client data imbalance. In this context, how do you evaluate the improvements brought by the method proposed in this paper compared to foundation models in pseudo-label generation? Furthermore, do you think the theoretical analysis in this paper could be extended to incorporate considerations related to foundation models?\n\n [1] Cho, Y.J., Joshi, G. and Dimitriadis, D., 2023. Local or global: Selective knowledge assimilation for federated learning with limited labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision.\n [2] Liu, Y., Wu, H. and Qin, J., 2024, March. Fedcd: Federated semi-supervised learning with class awareness balance via dual teachers. In Proceedings of the AAAI Conference on Artificial Intelligence.\n [3] Zhu, G., Liu, X., Wu, X., Tang, S., Tang, C., Niu, J. and Su, H., 2024, January. Estimating before debiasing: A Bayesian approach to detaching prior bias in federated semi-supervised learning. In Proceedings of the International Joint Conference on Artificial Intelligence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After rebuttal, all reviewers agree to accept this paper. To be specific, the proposed method in this paper has been well validated from both theoretical an empirical studies. Besides, the paper is also clearly written.
Decision: Accept (poster)