Learning Unlabeled Clients Divergence for Federated Semi-Supervised Learning via Anchor Model Aggregation
Authors: Marawan Elbatel
Hualiang Wang
Jixiang CHEN
Hao Wang
Xiaomeng Li
Venue: Accepted by TMLR ()

Abstract: Federated semi-supervised learning (FedSemi) refers to scenarios where there may be clients with fully labeled data, clients with partially labeled, and even fully unlabeled clients while preserving data privacy. However, challenges arise from client drift due to undefined heterogeneous class distributions and erroneous pseudo-labels. Existing FedSemi methods typically fail to aggregate models from unlabeled clients due to their inherent unreliability, thus overlooking unique information from their heterogeneous data distribution, leading to sub-optimal results. In this paper, we enable unlabeled client aggregation through SemiAnAgg, a novel Semi-supervised Anchor-Based federated Aggregation. SemiAnAgg learns unlabeled client contributions via an anchor model, effectively harnessing their informative value. Our key idea is that by feeding local client data to the same global model and the same consistently initialized anchor model (i.e., random model), we can measure the importance of each unlabeled client accordingly. Extensive experiments demonstrate that SemiAnAgg achieves new state-of-the-art results on four widely used FedSemi benchmarks, leading to substantial performance improvements: a 9% increase in accuracy on CIFAR-100 and a 7.6% improvement in recall on the medical dataset ISIC-18, compared with prior state-of-the-art. Code is available at: https://github.com/xmed-lab/SemiAnAgg.

Published in Transactions on Machine Learning Research (10/2024)
Learning Unlabeled Clients Divergence for Federated Semi- Supervised Learning via Anchor Model Aggregation
Marawan Elbatel The Hong Kong University of Science and Technology Hong Kong SAR, China
mkfrrellaatel@ustt.kk
Hualiang Wang
hwangfddrust..h
The Hong Kong University of Science and Technology Hong Kong SAR, China
Jixiang Chen
johenhu@ust.hl
The Hong Kong University of Science and Technology, Hong Kong SA R, China
Hao Wang
hw4880cs. edu
Rutgers University, New Jersey, USA
Xiaomeng Li
eeamli@rst.h
The Hong Kong University of Science and Technology, Hong Kong SAR, R, China
Reviewed on OpenReview: https: // openreview. net/ strmm? id- GDn6x9LIDs
Abstract
Federated semi-supervised learning (FedSemi) refers to scenarios where there may be clients with fully labeled data, clients with partially labeled, and even fully unlabeled clients while preserving data privacy. However, challenges arise from client drift due to undefined heterogeneous class distributions and erroncous pseudo-labels. Existing FedSemi methods typically fail to aggregate models from unlabeled clients due to their inherent unreliability, thus overlooking unique information from their heterogencous data distribution, leading to sub- optimal results. In this paper, we enable unlabeled client aggregation through SemiAnA novel Semi-supervised client contributions via an anchor model, effectively harnessing their informative value. Our key idea is that by feeding local client data to the same global model and the same consistently initialize anchor model (f.e., random model), we can measure the importance of each unlabeled client accordingly. Extensive experiments demonstrate that SemiAnAgg achieves new saato-of-thh-art results on four widely used FedSemi benchmarks, leading to substantial performance improvements: 9% increase in accuracy on CIFAR-100 and a 7.6% improvement in recall the medical dataset ISIC-18, compared with prior state-of-the-art. Code is available at: https: h/gt:://tt thub. cwtp:modo///eeoeeooeeooooooo lab/SemiAnAgg.
Anchor-Basee federated Aggregation. SemiAnAgg learns unlabeled
1 Introduction
Federated learning has emerged a promising solution for learning in decentralized environments, where data centralization is often infeasible due to privacy concerns. Federated learning has gained considerable attention in machine learning tasks in natural image domains (McMahan et al., 2017) well medical image domains (Liu et al., 2021; Saha et al., 2023; Jiang et al., 2023). Due to the challenges of data and label heterogeneity, multiple methods such as MOON (Li et al., 2021a), FedDisco (Ye et al., 2023), FedFed (Yang et al., 2023), and FedRoD (Chen Chao, 2022) have been developed, utilizing FedAvg (McMahan et al., 2017) as their baseline. While these methods have shown promise, they often assume that all clients have exhaustive and expert-level annotations, which is a requirement that is both time-consuming and renders them impractical in real-world cross-silo federated settings, such those found in hospitals. For
This

---

Published in Transactions on Machine Learning Research (10/2024)
instance, in a federated learning framework where multiple hospitals collaborate to develop shared model for complex tasks like lesion classification, some hospitals may provide fully labeled data for training, while others, with limited expert manpower, can only offer unlabeled or partially labeled data for model training. Therefore, developing methods that require minimal expert-lovel annotations in decentralized scttings is a crucial area of research that deserves further attention.
Federated Semi-Supervised Learning (FedSemi) arises to alleviate exhaustive expert-level annotations by utilizing unlabeled data along with labeled data to benefit the global performance in many recent works (Zhang et al., 2023; Cho et al., 2023; Zhang et al., 2023; Wang et al., 2020; Jeong et al., 2021; Lin et al., 2021; Liu et al., 2021; Saha et al., 2023; Liang ct al., 2022; Li et al., 2023; Kim et al., 2022; 2023b). Rescarchers have explored various settings for FedSemi, with the most generalizable setting considering fully unlabeled clients and other clients that can be either labeled or partially labeled, in either independent and identically distributed (TID) or non-IID settings (Liang et al., 2022; Li et al., 2023).
Despite the effectiveness of these methods, existing approaches have two crucial limitations. First, their aggregation typically follows the standard FedAvg (McMahan et al., 2017), which aggre- gates clients based on the volume of local data, failing to account for the fact. that labeled clients can generate more accurate in- formation than unlabeled clients, regardless of data volume, For
70
mmg
65
Accuracy
60
55
FedAvg
example, when the number of unlabeled clients is dominant, the global model becomes heavily influenced by the unlabeled clients, leading to limited performance primarily due to errors introduced by the pseudo-labels assigned to the unlabeled data. Second, existing methods treat the divergence of unlabeled clients, calcu- lated in the gradient space, as noise due to the unreliability of the pseudo-labels used in training. It is worth mentioning that client divergence may reflect the presence of minority classes and unique attributes within the dataset, rather than mere noise.
50-
CBAFed
FedAvg-Semi (our strong baseline)
45
SemiAnAgg (ours)
40
how
500
600
700
800
900
1000
Round
2.5
9000
2.0-
8000
Ereo (%)
Data Size
To address the first limitation, we first show that aggregation based on the local dataset size, as in FedAvg (McMahan et al., 2017), falls short in the FedSemi setting. At first glance, this may not be surprising: an unlabeled client with a large amount of data but incorrect pseudo-labels should not dominate the global opti- mization. Recall that traditional semi-supervised learning (SST.) typically seeks a balance between optimizing the empirical risks associated with both labeled and unlabeled data to avoid skewing toward potentially incorrect pseudo-labels (Tarvainen & Valpola, 2017; Sohn et al., 2020; Zhang et al., 2021; Chen et al., 2023; Wang et al., 2023). Therefore, by balancing the empirical risk of labeled and unlabeled clients during global model aggregation, we present a stronger baseline for FedSemi termed FedAvg-Semi. Figure 1 shows that our simple baseline, FedAvg-Semi, can achieve com- parable performance to SOTA of FedSemi, CBAFed (Li et al., 2023). Nevertheless, treating unlabeled clients based their local datasct size is suboptimal: an unlabcled client with small dataset size may contain underrepresented attributes and minority classes that are not present in others.
1.5
7000
1.0
6000
0.5
5000
0.0
2
4
Leave Unlabeled Client Out
Figure 1: Upper: FedAvg (McMahan et al., 2017) fails miserably. Yet, disentangling the aggregation (FedAvg-Semi) achieves perfor- mance comparable to the state-of-the-art Fed- Semi method, CBAFed (Li et al., 2023). By promoting diversity among unlabeled clients our
achieves new SOTA on four FedSemi benchmarks. Lower: Leave one unlabeled client out. The bar refers to the A Error (%), and the line refers to the Data Size. In the leave-one-out experiment selting, one unlabeled client is excluded during train- ing. The results indicate that the decrease in accuracy (represented by the bar) does not correspond proportionally with the data size (represented by the line).
The second limitation of current methodologies is their failure to account for the diverse contributions of unlabeled clients, partic- ularly in reflecting the presence of minority classes and unique attributes within majority classes. To validate the importance of unlabeled clients' diverse contributions, we conducted an extensive empirical evaluation using the popular ont-oto valuation strategy (Ghorbani & Zou, 2019). As shown in Figure 1, the leave-one-out valuation strategy reveals significant differences in
2

---

Published in Transactions on Machine Learning Research (10/2024)
how unlabeled clients contribute to the overall performance of FedSemi, allowing to draw two important conclusions. First, there is no correlation between the size of the unlabeled client's 's dataset and the value a client contributes to the system. For instance, excluding client 9 significantly degrades performance (A Error 1) compared to when client 2 or are removed, even though client 9 have lower amount of data. Second, the varying impact of removing different unlabeled clients showcases the importance of each unlabeled client. accordingly. Therefore, measuring the value of unlabeled clients irrespective of their data size challenge yet to be tackled in FedSemi.
To this end, propose novel aggregation, SemiAnAgg, which learns the importance of unlabeled clients via consistently randomly initialized model. Our key idea is that by fooding the local client data to the same global model and the same consistently initialized random anchor model, we can measure the importance of each unlabeled client. Specifically, our intuition is two-fold: 1) The optimal feature representation for client data should differ from a random representation, as random representations lack meaningful structure. Therefore, clients contributing to representations deviating from randomness are more likely to steer the model toward the global optimum. 2) Since all clients share the same global and anchor model, the inter-client. data variability can be measured consistently. In other words, the expectation of the global model on client data that is more distant indicates a more diverse distribution covering unique information. Consoquently, SemiAnAgg gives higher weights to these clients, harnessing their potential. Without requiring any direct sharing of data, features, or gradient information among the clients and maintaining the same communication costs as previous FedSemi approaches with minimal computation and storage overhead, SemiAnAgg achieves tate-of-theeart results on four widely used benchmarks with improvements up to 9% in accuracy on CIFAR- 100, 9.5% the imbalanced CIFAR-TOOTT, and a 7.65% improvement in recall on the medical dataset. ISIC-18.
Our contributions can be summarized as follows:
We provide a new insight for FedSemi that was neglected in prior work: measuring the importance of unlabeled clients.
Unlike previous FedSemi approaches that treat the divergence of unlabeled clients as noise, we propose SemiAnAgg, novel aggregation method that effectively aggregates the most informative unlabeled clients, significantly harnessing their unique information.
Our method consistently outperforms prior slate-of-the-arl methods on four widely used benchmarks, surpassing SOTA CBAFed by 9% in accuracy on CIFAR-100, 9.5% on its highly imbalanced version, CIFAR-10OLT, and achieving a 7.65% recall improvement on the medical dataset, ISIC-18.
2 Related Work
2.1
Semi-Superrised Learning
Semi-Supervised Learning (SSL) leverages large amount of unlabeled data along with labeled data to learn generalizable model. Semi-supervised methods include consistency regularization (cnsuring consistency between two distorted unlabeled images) (Li et al., 2021b; Tarvainen & Valpola, 2017), generating pseudo labels through supervised objectives (Zhang et al., 2021; Chen et al., 2023; Wang et al., 2023), or self-supervised clustering objectives (Fini et al., 2023). Prominent methods incorporating adaptive pseudo labeling are FlexMatch (Zhang et al., 2021), SoftMatch (Chen et al., 2023), and FreeMatch (Wang et al., 2023). However, these strategies assume uniform and identical class distribution of labeled and unlabeled data, which is often not the case in real-world applications.
Semi-Supervised Imbalanced Learning (SSIL) aims to learn SSL models in scenarios where there is class imbalance distribution in labeled and unlabeled data. Several works address this issue by amplifying pscudo labels for minority classes through resampling (Wei et al., 2021), re-weighting (Wei et al., 2022; Wang & Li, 2023a;b), and classifier blending (Oh el al., 2022). However, these methods assume the unlabeled class imbalanced distributions follow the labeled ones. In the non-lID FedSemi setting, a more realistic scenario ariscs where there is a class distribution mismatch between labeled and unlabeled data. This sctting, which has not been extensively explored, addressod by the state-of-tih -f-th--r-- method proposed by ACR (Wei & Gan, 2023). ACR builds upon FixMatch (Sohn et al., 2020) with dual branch and introduces adaptive
3

---

Published in Transactions on Machine Learning Research (10/2024)
consistency regularization. The intensity of logit adjustment, controlled by scaling parameter, is adaptively calculated based on the pseudo-label distribution distance to three anchor distributions: uniform, labeled distribution, and its reversed version. While ACR (Wei & Gan, 2023) showed promising results in centralized settings, the three anchor distribution is crafted and does not adhere to non-IID FedSemi. Thus, the challenge of semi-supervisel learning with heterogeneous labeled and unlabeled class distributions, particularly FL non-IID scenarios, remains unresolved.
2.2 Federated Semi- Semi-Suuer Learning
Federated Semi-Supervised Learning (FedSemi) addresses the decentralized learning of both unlabeled and labeled data while preserving privacy. FedSemi has been explored in three different settings. Tn the first two settings, labeled data is available on the server, in FedMatch (Jeong et al., 2021), Semi-FL (Diao et al., 2022), and FedLID (Psaltis et al., 2023), or clients have partially labeled data, as seen in SemiFed (Lin et al., 2021) and FedTabel (Cho et al., 2023). The third setting, which we consider more realistic, involves fully unlabeled clients, while other clients can be either fully labeled or partially labeled. For the third setting, recent works have shifted from addressing clients with independent and identically distributed (IID) data, such as FedConsist (Wang et al., 2020) and FedIRM (Liu al., 2021), to non-IID data, including RSCFed (Liang et al., 2022), IsoFed (Saha et al., 2023), and CBAFed (Li et al., 2023). This work focuses the third setting, characterized by most clients with fully unlabeled and potentially non-lID data.
2.3 Federated Model Aggregation
Federated aggregation aims to improve the global model through proper aggregation across various clients. Given the heterogeneity of client data, clients with unique information could benefit the global model more during optimization, rendering the control of clients' contributions a crucial concern. This direction has shown great promise in supervised federated learning settings (Jiang et al., 2023; Elbatel et al,, 2023; Ye et al., 2023). For instance, FedCE (Jiang et al., 2023) measures contribution in both the gradient and sample space. Using the sample space involves prediction error calculation, relying on labeled validation data within clients. FedMAS (Elbatel et al., 2023) proposes leveraging labeled data for estimating inter-client intra-class variation to emphasize the contribution of clients with minority classes. However, these hots rely on label information, making them inapplicable to unlabeled clients in FedSemi. In unlabeled clients, the non-IID distribution results in noisy pseudo-label istiton rendering the surrogation with pseudo-labels on unlabeled clients less effective for aggregation measurement.
Existing FedSemi (Liang et al., 2022; Li et al., 2023) methods regard non-robust clients or data as outliers and opt to minimize their influence. RSCFed (Liang et al., 2022) proposes to leverage unlabeled gradient divergence to eliminate non-robust (noisy) clients through average consensus inspired by RANSAC (Fischler & Bolles, 1981). However, RSCFed (Liang et al., 2022) does not consider cases when unlabeled clients are diverse due to unique information (i.e. underrepresented classes or attributes). IsoFed (Saha et al., 2023) proposes alternating model training between labeled and unlabeled clients in each round, which might result in significant catastrophic forgetting after each round. CBAFed (Li et al., 2023) introduced federated class adaptive pseudo labeling, which can be seen as auricuuuu-paced pseudo learning from a global perspective (Zhang et al., 2021). It uses hard temporal ensembling (Rasmus al., 2015; Tarvainen & Valpola, 2017; Laine & Aila, 2017) to address the stochastic variability in the global model and simply aggregates models based ou reliable data amount. These strategies overlook the informative unlabeled clients with heterogeneous data, leading to suboptimal optimization. Therefore, a more comprehensive method for quantifying unlabeled clients is needed. In this work, we fill this gap by introducing SemiAnAgg, an anchor- based aggregation strategy in FedSemi, which harnesses the heterogeneity unlabeled clients during valuation.
3 Preliminaries on Federated Semi- SupeeSuupr Learning
FedSemi Setting. Let us consider the generic Federated Semi-Supervisedd Learning (FedSemi) set- ting, which allows clients to be fully unlabeled, partially labeled, or fully labeled which is introduced
4

---

Published in Transactions on Machine Learning Research (10/2024)
in RSCFed (Liang et al., 2022), IsoFed (Saha et al., 2023), and CBAFed (Li et al., 2023). We de- note the set of clients as {C1, Cx), where each client Cs possesses local dataset represented by N" {(X)) }
|=m]. Here, N and N correspond to the number of labeled and unlabeled data instances, respectively. The objective is to derive a robust global model, Ogtab, which effectively leverages all the available data across all clients. The most rigorous setting, as defined in (Liang et al., 2022; Li ct al., 2023), is characterized by the majority of clients possessing fully unlabeled non-IID data, with N²=
Detient = {[(( )
Federated Warm-up. To ensure reliable pseudo-labels for unlabeled clients in initial stages, previous FedSemi approaches (Li et al., 2023; Liang et al., 2022) have employed warm-up phase based on labeled clients using traditional FedAvg (McMahan et al., 2017). In this paper, we follow the generic FedSemi benchmark (Liang et al., 2022; Li et al., 2023) in label-dependent federated warmup for all reported results, yet we present additional experiments in Appendix with federated self-supervised warmup (Zhang et al., 2020; Lubana et al., 2022; Zhuang al., 2022; Kim al., 2023a) highlighting their effectiveness in case of working with limited and highly imbalanced datasets (Yang & Xu, 2020; Yan et al., 2022).
Local Training. Each client C performs E local steps based on the local dataset Detient- Given our proposed aggregalion melhod is perpendicular to self-training regimes (Tarvainen & Valpola, 2017; Chen et al., 2023; Wang et al., 2023), we adopt the canonical supervised training and self-training regimes on labeled and unlabeled data, respectively. Specifically, we utilize FlexMatch (Zhang et al., 2021) as our bascline to minimize a local objective function:
C C=Aupoup + A2Zunsups
(1)
where a is equal to zero in fully unlabeled clients, and X2 is the pre-defined weight coellicient between supervised and unsupervised loss terms. We fix d2 I as traditional pseudo labelling approaches (Sohn et al., 2020; Wang et al., 2023; Zhang et al., 2021).
4 Method
4.1
FedAvg- Semi: A Strong Aggregation Baseline for Fedsemi
Previous FedSemi approaches (Cho et al., 2023; Liang et al., 2022; Li et al., 2023) show that assigning a higher weight to labeled clients on the server produces better performance than traditional FedAvg aggregation (MeMahau et al., 2017). Notably, semi-supervised learning necessitates a weighted combination of supervised and unsupervised loss (Equation 1), with prior semi-supervised regimes (Sohn et al., 2020; Wang et al., 2023; Zhang et al., 2021) equally weighting Loup and Lunsup to avoid bias towards potentially incorrect pseudo-labels. Consequently, it is crucial to re-adjust the global optimization objective in federated semi-supervised learning. Unlike existing FedSemi approaches (Cho et al., 2023; Liang et. al., 2022; Li et al., 2023) that aggregate clients on the server based on traditional FedAvg (McMahan et al., 2017), we disentangle the aggregation based on labeled and unlabeled data on the server to write a more generic form, termed FedAvg-Semi.
Given K clients, with number of labeled samples, N4, and the selected number of unlabeled samples contribut- ing for Lunsup in cach client local training, NVU Fed dynamically discntangles FedAvg (McMahan et al., 2017) to ensure global optimization objective consistent with traditional semi-supervised optimization as follows:
K Σ 1=
No NL
NU & No
K
K
B global
=
0x + Eez
Or, No = Σ N and No
Σ km
No
(2)
total!
km
Supervised
Unsupervised
where 2, + 222 = to cnsure a normalized mcan, and control the optimization of the labeled and unlabeled data consistent with semi-supervised regimes. Setting Å₁ reduces to supervised warm-up on labeled clients. For simplicity, we set o₁ ' 0.5, noting that ideally these values could be ramped during federation.
5

---

Published in Transactions on Machine Learning Research (10/2024)
4.2 SemiAnAgg: Semi- Semi-Supervised Machor-Basede Aggregation
Unlabeled clients semi-supervised learning can provide valuable and diverse information, but their local models are unreliable due to erroneous pseudo-labels. FedSemi, aggregation strategies typically aim to identify and treat unlabeled clients' local gradient divergence as noise. Nevertheless, this approach overlooks that such divergence could be due informative, underrepresented attributes and classes in the data. Therefore, it is crucial to develop aggregation strategies that leverage the full potential of unlabeled clients' data.
To this end, we propose a novel aggregation strategy to re-think the valuation of unlabeled clients in FedSemi, capturing their diversion without relying on labeled samples. Specifically, we propose to measure the diversity in the feature space based on a reference embedding from a consistently initialized anchor model across clients.
At the start of the federated process, given the locally unlabeled samples: {(x{) } store dictio- nary of feature representations as: {(+)) where Qe annch (X,), and Canch represents a Kaiming ini- tialized random encoder with the same seed across clients (He et al., 2015). The molivation for using random encoders stems from the realm of generative model evaluation, where they have been shown to provide macroscopic views on distributional discrepancies (Naeem al., 2020). The storage foot- print of the feature dictionary varies, from 968KB for client with 484 samples to 5.01MB for client with 2567 samples, totaling 15.6MB across all clients for the ISIC-18 dataset-a mere 0.006 of the dataset's 2.5GB size.
genc
C
Diversity Scalars
Inter-elient Normalization
(u,y(oos)
Q0
gene
glob
fo tc
00
Ø5
SemlAnAgr
Figure 2: Illustration of the proposed Semi-supervised Anchor-Based Aggregation. The ignored pseudo label owing to low confidence is denoted as -1.
During the federated process, the client receives the averaged global model ggob from the server up- date Orocat From this, we can extract the feature representation é glob (X.), where glob refers to the global model's encoder applied to the client data. X,.
Additionally, the pscudo label is generated as 0: = offos connected layer used for classification. Note that q and 4 for each client are computed using the same ggts and Bene, rendcring client data, Detient as the only variable. We extract the features and computc the necessary labels simultaneously, maintaining the computational complexity of O(BE) per client per round, where represents the batch count and E denotes the local epoch count.
(4.), where glob represents the global model's fully
At cach round, we build pscudo-aware running summation of the fcature similarity between q and & for pseudo-class e with M. samples as follows:
M
1
q: q |ill ||
0.
M
where q and á are normalized along the feature dimension they lie on the unit sphere. Note q represents the indexed features from the dictionary computed with the same initial weight distribution across all clients. Notably, the distance between and 4 can serve as consistent measure for assessing divergence.
A high 0e indicates that the class representation the client is close to random. Consequently, clients with lower w. are more likely to approximate the global optimum. Given that all clients adhere the same anchor and global model, client with a lower de demonstrates greater diversity in its class representation. Therefore, re uh reflects the class importance of each client. based diversity measurement, which

---

Published in Transactions on Machine Learning Research (10/2024)
should be normalized across clients ic = by summing the distances to a scalar and the final averaged model, gtob is computed as:
The final unlabeled client contribution can be computed
E
K
N N total
fk= = D fer and globel =
Σ
Y
O2
+ d2
O2
(4)
Supervised
Unsuporvised
where C' is the total number of classes. SemiAnAgg converges when unlabeled clients' pscudo labels become reliable. We discuss the convergence of SemiAnAgg in Section 5.3 and provide a detailed convergence analysis in Appendix B.
5
Experiments
5.1 Experimental Setup
Datasets, Models, and Settings. We adhere to eristing FedSemi benchmarks in all experimental settings, established by (Li et al., 2023) and (Liang et al., 2022), while additionally expanding multiple scenarios. Specifically, we utilize four datasets to assess the effectiveness of our approach: SVHN, CIFAR- 100 (Krizhevsky, 2009) (both the standard and its long-tailed imbalanced variant with an imbalance factor of 100), and the skin-lesion classification dataset, ISIC-18. Note that for imbalanced datasets, the imbalance is global, existing in both labeled and unlabeled data with class distribution mismatch between the label and unlabeled data. For all datasets, we reproduce the reported results of RSCFed (Liang et al., 2022) and CBAFed (Li et al., 2023) on the same non-IID federated partitioning publicly available, Dir(a) 0.8. While CBAFed utilize ResNet-18 18 ImageNet version detailed in (He et al., 2016) and RSCFed use simple CNN, we find their lower-bound is not comprehensive. Thus, we reproduce their results by using the ResNet-18 CIFAR version detailed (He et al., 2016) for datasets with small spatial input dimensions (SVHN, CIFAR100), and the traditional ImageNet ResNet-18 in (He et al., 2016) for ISIC-18. This results in improved reproduction of all basclines than reported in CBAFed (Li al., 2023).
To demonstrate the generalization performance, evaluate the global model on the standard balanced test. set for all datasets and report accuracy, following the conventions of previous FedSemi methods (Li al., 2023; Liang et al., 2022). It should be noted that the ISIC-18 test set exhibits imbalanced class distribution, prompting us to provide a more comprehensive evaluation for this particular dataset.
Implementation Details (A detailed version in Appendix C.) To ensure a fair comparison, we maintain consistency in the training protocol, architecture, exact federated partitioning checkpoint, and other exper- imental settings across all methods. The same warmup model is utilized for initialization in all reported tables unless otherwise stated, which is trained for 250 rounds for SVHN, 250 rounds for ISIC-18, and 500 rounds for CIFAR-100. This is followed by FedSemi learning for additional 500 rounds for SVHN, 500 rounds for ISIC-18, and 1000 rounds for CIFAR-100 until convergence. Notably, unlike RSCFed (Liang el al., 2022), and SemiAnAgg (ours), CBAFed (Li et al., 2023) benefit from temporal ensembling and a higher lower-bound. Consequently, we initialize CBAFed (Li et al., 2023) with a temporal ensembled model to maintain consistency.
Baselines. In our evaluation, we compare our results with state-of-the-art reproducible methods: RSCFed (Liang et al., 2022), IsoFed (Saha et al., 2023), and CBAFed (Li et al., 2023). To provide more competitive baseline, we present in Appendix D and Appendix E additional baselines ablating local training strategies, specifically involving ACR (Wei & Gan, 2023) and SoftMatch (Chen et al., 2023) within the FedSemi framework.
5.2 Quantitative Comparisons with State-of-the-Art Methods
We strictly follow the experimental setting of the previous FedSemi benchmark (Li al., 2023; Liang al., 2022). This rigorous setting includes a single labeled client, which holds only 5% of the global dataset,
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table I: Results on SVHN, C1 CIFAR-100-LI and ISIC 2018 datasets under heterogeneous data partition. We report the results of all compared methods implemented by ourselves. Besides, the original results in papers are also reported and marked by t. We adhere to our baselines in the most rigorous setting (single labeled client), where client contains a mere 5% of the global labels and 9 unlabeled clients. On imbalanced datasets, CIFAR-100LT and ISIC-18, use logit adjustment (Ren et al., 2020) loss for all baseline methods.
Datasct
Labeling Strategy
Method
Balanced
Imbalanecd
SVIIN
CIFAR-I00
CIFAR-I00-LT 38 40
ISIC-18
FedAvg (upper-bound)
94.77
64.75
80.78
Fully supervised
FodAvg (lowcr-bound)
75.86
29.51
14.38
66.25
RSCFedt (Liang al., 2022)
76.74
28.46
67.21
CBAF (Li et al., 2023)
88.07
30.18
68.29
Semi supervised
RSCFed (Liang et al., 2022)
81.84
31.98
15.13
69.85
IsoFed (Saha et al., 2023)
82.48
32.49
16.98
68.49
CBAFed (Li al., 2023)
91.57
40.20
14.29
69.99
SemiAnAgg (ours)
91.69
49.23
23.81
72.24
Table 2: Partially Labeled Clients.
Table 3: Effectiveness of aggregation methods.
Method
ISIC-18
ISIC-18 Dataset
Acc (%) AUC (%) B-Ace (%) AUC (%)
Agg. Methods
Aec. [%6]B (%)
CBAFed
RSCFFed
AnmlAnAgg (ours)
92.57
87.27
90.94
99.50
98.76
99.20
42.62
33.22
88.34
81.69
79.86
FedAvg-Seml baseline)
SemiAnAgg (ours)
FedAvg (baseline)
72.24
67.79
62. 16
48.77
40.21
31.73
alongside nine unlabeled clients. Additionally, we expand our experiments to include multiple scenarios and ablation studies. Table 1 reports the quantitative results on four benchmarks, including two balanced datasets SVHN and CIFAR-100, and two imbalanced datasets CIFAR-100-LT and ISIC-18 (skin-lesion).
Results on the Balanced Global Setting. As shown in Table 1, SemiAnAgg achieves the best accuracy on two balanced datasets, SVHN and CIFAR-100, performing the compared methods terms of accuracy. Specifically, SemiAnAgg surpasses RSCFed, IsoFed, and CBAFed by 9.85%, 9.21%, and 0.12%, respectively, SVHN, which is relatively simple dataset. On more challenging dataset, CIFAR-100, SemiAnAgg showcases substantial improvement of 17.25%, 16.74%, and 9.03% respectively.
Results with Imbalanced Global Setting. Previous FedSemi methods cid not consider the imbalanced global setting, which is critical in non-IID FedSemi, where class distribution mismatch between the label and unlabeled clients exists. To this end, we report the results on two imbalanced datasets (CIFAR- 100LT and ISIC-18) while all baselines adopting logit-adjustments (Ren et al., 2020) account for the dass-imbalance. Our SemiAnAgg outperforms the SOTA baseline, CBAFed (Li et al., 2023) with 9.52% and 2.25% on CIFAR-100-LT and ISIC-18 respectively, achieving the best performance.
Results with Partially Labeled Clients. In Table 2, we compare the generic FedSemi approaches in the scenario of having partially labeled clients on the SVHN and ISIC dataset, a subset of the setting, In this partially labeled setting, all clients possess 10% of their data as labeled, while the remaining 90% is considered unlabeled. Our SemiAnAgg method demonstrates higher performance over CBAFed (Li et al., 2023), achieving a 1.63% increase in accuracy on the SVHN datasct and 4.41% improvement in "B-Ace" on ISIC. Further experiments and insights are provided in Appendix C.
5.3 Ablation studies
Effectiveness of SemiAnAgg. The results of employing different aggregation strategies are presented in Table 3. Adopting Avg-SSm, which aligns with the principles of traditional semi-supervised optimization regimes, increases accuracy by 5.63% and "B-Acc" by 8.48% compared the FedAvg baseline. FedAvg- Semi disentangles the aggregation of labeled and unlabeled clients, thereby avoiding the skewing of global
8

---

Published in Transactions on Machine Learning Research (10/2024)
optimization towards unlabeled clients with potentially incorrect pseudo-labels. Unlike methods that assign weights based on the amount of data for unlabeled clients, our SemiAnAge leverages data heterogeneity for valuation, substantially surpassing our simple baseline, FedAvg-Semi, by 4.45% on accuracy and 8.56% on "B-Acc".
Effects of Random Anchor Initialization. Figure presents the results of our ablation study using different random seeds for the anchor model in SemiAnAgg compared to our simple baseline, FedArg-Semi. The upper plot illustrates the performance balanced dataset, SVHN, while the lower plot shows results on the imbalanced dataset, ISIC-18. SemiAnAgg with different anchor seeds (denoted as facch and 2n consistently outperforms FedAvg- Semi with consistent and faster convergence. Notably, the results demonstrate that the choice of anchor seed has a minimal impact on the final performance, indicating the stability of SemiAnAgy, It. is worth mentioning that the random anchor model does not introduce additional information during local client optimization.
the
Arcurag (96)
88
86
FedAvg-Semt
SemiAnAgg Panch
SemiAnAgg 9anch
500
600
700
800
900
1000
Round
70
SemiA Ag Convergence. We analyze the convergence behavior of SemiAnAgg on the SVHN dataset Figure 4. Given that SVHN is a relatively simple task, the pseudo-labels become highly reliable (* 99%) in Figure (a). Consequently, SemiAnAgg converges to almost equal contribution from all clients (1/9 0.111) as shown in Figure 4 (b) Notably, in the early rounds, SemiAnAgg values clients 9, 7, and 6 the most due to their respective distances from the anchor model. SemiAnAgg behavior is supported by the leave-one- out in Figure 4 (c) indicating these clients are the most significant (their removal results in the highest error rate). Conversely, clients 2, 3, and are weighted less in Semi,nAgg, supported by the fact that their removal does not significantly impact performance (their removal results in the lowest error rate). To this end, SemiAnAgg's novel setup of using a randomly initialized anchor model can measure not only the learned probably (most distant from the random anchor) but also consider the inter-client divergence (relative client distance). Unlike other FedSemi approaches (Liang et al., 2022; Li ct al., 2023), Semi AnAg's novel setup enables diverse unlabeled client aggregation.
65
6.0
55
50
FedAug-Semm SemiAnAgg 9anch
45
SemiAnAgg w/ 9anch,
40
500
600
700
800
900
1000
Round
Figure Ablation using FedAvg- Semi and SemiAnAgg with different random anchors. Upper: SVHN. Lower: ISTC-18.
0.1120
99
Global Contribution Value
2.5
0.1115
98
2.0
Acccuracy(8)
0.1110
97
1.5
0.1105
96
95
Unlabeled Client
4
0.1100
Unlabeled Client
1.0
0.1095
0.5
94
3
0.1090
0.0
500
600
700
800
1000
600
800
1000
3
Round
Round
Leave Unlabeled Client Out
(a)
(b)
(c)
Figure 4: SemiAnAgg convergence analysis on the SVHN dataset. Note that as training progresses, the unlabeled client has reliable pseudo labels in (a) (+ 99%) given digit classification is relatively a simple task. ScmiAnAgg converges to the clients contributing almost equally 1/9 z 0.1111. (c) Client Importance by leaving one unlabeled client out (Each bar corresponds to the performance drop in FedSemi where an unlabeled client is removed).
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table Ablation of varying numbers of clients and different Dir(a) on ISIC.
Number of Clients
10
25
30
B-Ace (%)
AUC (%)
B-Ace (%)
AUC (%)
B-Aco (36)
AUC (%)
B-Ace (%)
AUC (%6
CBAFed
52.58
86.45
41.12
86.04
33.61
$1.03
28.76
74.08
SemiAnAgg (ours)
55.41
87.85
48.77
86.98
35.45
82.14
33.32
75.77
Dirichlet Parameters
Dir(0.1)
Dir(0.5)
Dir(0.8)
Dir(2)
B Acc (%)
AUC (%6
B Acc (%6
AUC (%)
B Acc (%)
AUC (%6
B: Aoc (%)
AUC (%)
CBAFed
33.12
60.23
41.03
85.01
41.12
86.01
45.83
87.68
SemiAnAgg (ours)
39.90
65.36
46.06
86.38
48.77
86.98
48.86
88.63
Effects of Client Numbers. Expanding the experimental scenarios, we consider broader range of clients. Following the rigorous tests of (Liang et al., 2022; Li et al., 2023), conduct ablation study with 5, 10, 25, and 50 clients, where only one client is labeled and the others are unlabeled. Our SemiAnAgg outperforms the SOTA FedSemi approach, CBAFed (Li et al., 2023), with improvements of 4.56% in "B-Acc" and 1.69% in AUC on the ISIC dalaset when federating 50 clients as shown in Table 4.
Effects of Different Heterogeneous Levels. Table 4 expands the ablations with scenarios of different. heterogeneity and a broader number of unlabeled clients. Specifically, we examine the impact of client heterogeneity by employing Dirichlet distribution Dir(a) with varying values. A smaller a indicates greater heterogeneity. The results demonstrate that the SemiAnAgg method outperforms the otate-of-the-art CBAFed (Li et al., 2023), particularly under the most challenging conditions of heterogencity with a 0.1, achieving improvements of 6.78% in "B-Ace" and 5.11% in AUC on the ISIC-18 dataset.
Effects of Labeled Clients. Figure presents an ablation study on the impact of increas- ing the number of labeled clients on the ISIC dataset. Notably, all methods show improve- ments when the number of labeled clients is increased larly outperforms the state-of-the-art CBAFed (Li et al., 2023) under the two-labeled-clients setting. Detailed metrics in Appendix C reveal remarkable 12% increase in precision and a substantial 4.5% in "B-Acc".
two. Our SemiAnAgg method particu-
One Labeled
Two Labeled
77.25
77.25
Privacy Implications of FedSemi. While suscepti- bility to attacks within the FedSemi framework has not been studied previously due to pseudo-labels adding an extra layer of stochasticity, SemiAnAgg more privacy- preserving than the SOTA FedSemi, CBAFed (Li et al., 2023), given the latter sharing the pseudo-class count of each unlabeled client, while SemiAnAgg opts for sharing pseudo-diversity scalars instead. Sharing pieudo-diversity scalars adds a layer of ambiguity against attempts to deci- pher the class distribution of unlabeled clients, stemming from the pseudo-diversity scalars being influenced not only by class count but also by the presence of minority at- tributes within majority class, or conversely, majority attribute within a minority class.
75.25
15.25
(
73.25
73.25
Accuracy
71.25
71.25
69.25
69.25
67.25
lower-bound RSCFed
CBAFed
67.25
SemiAnAgg (aurs)
65.25
65.25
Figure 5: Performance improvement when the number of labeled clients is increased from one to two ISIC.
Additional Comments. SemiAnAgg communication cost is the exact same as CBAFed (Li et al., 2023), 60% of RSCFed (Liang et al., 2022).
6 Conclusion
In this paper, we provide a new insight for FedSemi- FedSemi-hhighighttng the importance of measuring the divergence for unlabeled clients, which has been neglected in prior work. We introduce SemiAnAee novel anchor-
10

---

Published in Transactions on Machine Learning Research (10/2024)
based semi-supervised aggregation method that leverages consistently initialized random anchor model across clients, allowing informative unlabeled clients to contribute more effectively during global aggregation. SemiAnAgg achieves new state-of-the-art results on four different benchmarks, offering unique insights for future research in FedSemi and semi-supervised imbalanced learning.
7 Limitations and Future Directions
In the absence of labeled data from unlabeled clients, SemiAnAgg approximates their contribution by comparing their data distribution through two consistently initialized models across clients. SemiAnAgg adds a layer of feature dictionary saving, nevertheless, it is negligible compared to the dataset size and the improvements obtained. Cosine similarity does not accurately represent the direction of the divergence given it shares only scalars. A more comprehensive approach, potentially involving multiple anchor models or feature representations, could better bound the similarity within the unit sphere. Nevertheless, sharing more independent features from clients might raise privacy concerus, which would need to carefully addressed. Despite its simple design, SemiAnAgg achieves state-of satte-of-the-ard ff-the-art on four different benchmarks in FedSemi, compared to state-of-the therr federated semi-supervised learning (FedSemi). FedSemi is widely addressed in classification, whereas extending it to imbalanced regression (Yang ct al., 2021; Wang & Wang, 2023) remains challenging future task. SemiAnAgg's ' lack of theoretical formulations is rather due to FedSemi complexity that has not been previously studied theoretically given the stochasticity of geeudo-lbbel highly dependent on initialization. We hope that our findings and insights inspire future approaches addressing the non-iid class mismatch and imbalanced problems in both centralized and decentralized settings.
Broader Impact Statement
This paper introduces novel federated learning aggregation, SemiAnAgy, whose goal is to enhance the utilization of unlabeled data across distributed networks, improving collaborative learning and ensuring data confidentiality. Uniquely, SemiAnAgg promotes the diversity of unlabeled clients, aspect previously mnxxpored, by establishing a more equitable framework. While SemiAnAgg shares model weights and distance-derived scalars, these scalars do not. reveal sensitive information due to their irreversible nature. We acknowledge the potential societal impacts, yet no specific issues must be highlighted here.
Acknowledgments
This work was partially supported by grants from the National Natural Science Foundation of China (Grant No. 62306254) and the Hetao Shenzhen-How Kong Science and Technology Innovation Cooperation Zone (Project No. HZZQB-KCZYBB-2000333)
References
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality tradcoff in semi-supervised lcarning. In ICLR, 2023. URL
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In ICLR, 2022. URL http://rrrreeeeeeeeeoomrmmddd/aa
2, 3, 5, 7, 16, 18, 22, 23
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution URL https://ojs. aaai.
mismatch. AAAI, 34(04):3569 3576, Apr. 2020. doi: org/index.phl
18
Yac Jee Cho, Gauri Joshi, and Dimitrios Dimitriadis. Local or global: Selective knowledge assimilation for federated learning with limited labels. In ICCV, pp. 77087-17097 October 2023. 2, 4,
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randangment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), NewrIPS, volume 33, pp. 18613 18624. Curran Associates, Inc., 2020. 19, 20
11

---

Published in Transactions on Machine Learning Research (10/2024)
Enmao Diao, Jie Ding, and Vahid Tarokh. Semifi: Semi-supervised federated learning for unlabeled clients with alternate training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17871-17884. 17884. Cur- ran Associates, Inc., 2022. URL
https/s......p..ppppppeeeeeeeeee httppppcccppppprreeeeeeeeeeeeeee hctpppprrfilleeppper/////// files//paper/2022//11.. TLLo551Fcccdttt022222eeaaaaaapp Coreerree pdf. 4
Marawan Elbatel, Hualiang Wang, Robert Mart, Huazhu Fu, and Xiaomeng Li. Federated model aggregation via self-supervieed priors for highly imbalanced medical image classification. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pp. 334-346. Springer, 2023. 4
Enrico Fini, Pietro Astolfi, Karteek Alahari, Xavier Alameda-Pinedee Julien Mairal, Moin Nabi, and Elisa Ricci. Semii-uupervised learning made simple with self-supervised clustering. In CVPR, pp. 3187 3197, June 2023. 3
Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381 --5-38 jun 1981. ISSN 0001-0782. doi: 10.1145/358600 238629936.6. URL
11488//91145//88699.. 358692. 4
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. Tn
International Conference on Machine Learning, pp. 2242 2251, 2019. 2, 18
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10.1109/ICCV.200 201.20051122 123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
19
Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang, Federated semi-supervised learning with inter-client consistency & disjoint learning. ICLR, 2021. URL https: http://ppaarreeeeeeeee/tttooacaa net/formmidd ce6CFXBh30h. 2,
Meirui Jiang, Holger R Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu. Fair foderated medical image sogmentation via client contribution ostimation. In CVPR, 2023. 1,
Hansol Kim, Youngjun Kwak, Minyoung Jung, Jinho Shin, Youngsung Kim, and Changick Kim. Protofl: Unsupervised federated learning via protolypical distillation. In ICCV, pp. 6470-6479, October 2023a. 5,
17
Tachyeon Kim, Fric Lin, Junu Lee, Christian Lau, and Vaikkunth Mugunthan. Navigating data heterogeneity in federated learning: A semi-supervised approach for object detection. In Shirty-seventh Conference on Newral Information Processing Systems, 2023b. URL https://openreviee wttppperrrwiee..netooormmttaaaae
2
Woojun Kim, Keondo Park, Kihyuk Sohn, Raphael Shu, and Hyung-Sin Kim. Protofssl: Federated semi-
supervised learning with prototype-based consistency regularization. 2022. 2
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32 33, 2009. 16
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. ICLR, 2017. URL bttap:://ppereviiie wttp://oooommiid/J000oooo
Ming Li, Qingli Li, and Yan Wang. Class balanced adaptive pscudo labeling for foderated semi-supervised learning. CVPR, pp. 16292 16292-16301, 2023. 2, 4, 5, 7, 8, 9, 10, 16, 17, 19, 20, 21, 22, 23
Qinbin Li, Bingshong He, and Dawn Song. Model-contrastive federated learning. In CVPR, 2021a.
Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, Lei Xing, and Pheng-Am Hong. Transforatiooncoosiittttt self-ensembling model for semisupervised medical image segmentation. IEEE Transactions on Neural Nelworks and Learning Systems, 32(2):523 534, 2021b. doi: 10.109//TNNISS.2000020
12

---

Published in Transactions on Machine Learning Research (10/2024)
Xiaoxiao Liang, Yiqun Lin, Huazhu Fu, Lei Zhu, and Xiaomeng Li. Rscfed: Random sampling consensus federated semi-supervised learning. In CVPR, 2022. 2, 4, 5, 7, 8, 9, 10, 16, 17, 19, 20, 21, 22
Haowen Lin, Jian Lou, Li Xiong, and Cyrus Shahabi. Semifed: Semi-supervised federated learning with consistency and pseudo-labeling. ArXiv, abs/2108.09412, 2021. URL http::////pp.ssmmtiiiiiitt.eeeee
htg//ooppusID:2372644003 2, 4
Quande Liu, Hongzheng Yang, Qi Dou, and Pheng-Amn Heng. Federated semi-supervised medical image classification via inter-client relation matching. In Marleen de Bruijee, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2021, pp. 325-335, Cham, 2021. Springer International Publishing. ISBN 978-3-030-87199-4. 1,
Ekdeep Lubana, Chi lan Tang, Fahim Kawsar, Robert Dick, and Akhil Mathur. Orchestra: Unsupervised federated learning via globally consistent chustering. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14461-14484. PMLR, 17-23 Jul 2022. URL https://proceeeings. mlr www.mmlrr.peeee.. wwtpppeeaee//aaeeeeeeeeeeee
html. 5, 17, 18
H. B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication efficient learning of deep networks from decentralized data. In ATSTATS, 2017. 1, 2, 5, 17, 20, 21, 23
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Hal Daumé III and Aarti Singh (eds.): Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, mlr www..peess/vvlv het////1111//eeeeee htt.//aaeemma.amm........... 6
pp. 1176-7185. PMLR, 13-18 Jul 2020. URL pattss:/pproeeedine
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011. URL http://uflee htwpl/fflll.fffffo//ooeoaaoeaelo stanford ebt/hoousemuberrs/nnp00000 housenumbers, pdf. .df..
16
Youngtaek Oh, Ding-Jin Kim, and In So Kweon. Daso: Distribution-awan semantics-oriented pseudo-labe for imbalanced semi-supervised learning. In Proccedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9786-9796, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), NewrIPS, pp. 8024-8035, Curran Associates, Inc., 2019. 19
Athanasios Psaltis, Anestis Kastellos, Charalampos Z. Patrikakis, and Petros Daras. Fedlid: Self-supervised federated learning for leveraging limited image data. In ICCV, pp. 1039-1048, October 2023.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised lcarning with ladder networks. In NewrIPS, 2015. 4
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced mela-soflmax for long-tailed visual recognition. In NewrIPS, Dec 2020. 8, 20, 21, 22
Pramit Saha, Divyanshu Mishra, and J. Alison Noble. Rethinking semi-supervised federated learning: How to co-train fully-labeled and fully-unlabeled client imaging data. In Hayit Greenspan, Anant Madablushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Sreda-Mahmood. and Russell Taylor (eds.), Medical Image Computing and Computer Assisted Intervention -ICCAI 2023, pp. 414-424, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-445955-0 1, 2, 4, 5, 7,
13

---

Published in Transactions on Machine Learning Research (10/2024)
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), NewrlPS, volume 33, pp. 596-608. 608. Curran Associates, Inc., 2020. URL https:: pdf. 2, 3, 5, 22
neurips cc cc/paper_
Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang, Invariant feature learning for generalized long-tailed classification. In ECCV, pp. 709-726, 2022. 21
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency largets improve semi-supervised deep learning results. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Newr/PS, volume 30. Curran Associates, Inc., 2017. 2,
3, 4,
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collect ion of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, 2018. 16
Haonan Wang and Xiaomeng Li. Dhe: Dual-debiased heterogeneous co-training framework for class- insasanmallneedd semi-supervisod medical image segmentation. In International Conference on Medical Image Computing and Computer-Assise Assisted Intervention, pp. 582-591. Springer, 2023a.
Haonan Wang and Xiaomeng Li. Towards generic supervise framework for volumetric medical image segmentation. In NeurIPS, 2023b. URL
Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2020. URL 2, 4
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yuc Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. In ICLR, 2023. URL /ttps://openreveew pttp://roo...ee/oommmo///popoopp A. 2, 3, 5, 22
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview. wwwparrrwwee.eeeffouurdddddodddd net:/EorummLd--cMURXRTTT.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Loddon Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. 2021 IEEE/CVL Conference on Computer Vision and Pollern Recognition (CVPR), pp-
2021.
Tong Wei and Kai Gan. Towards realistic long-tailed semi-supervised learning: Consistency is all you need. In CVPR, pp. 3469 3478, June 2023. 3, 4, 7, 16, 18, 22
Xiu-Shen Wei, He- Yang Xu, Faen Zhang, Yuxin Peng, and Wei Zhou. An embarrassingly simple approach to semi-supervised few-shot learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NewrIPS, 2022. URL https: /twp://iieeeeeeeeee/emm www//eeuutieieeeocm (astrrummd==pPPPP-e -d 3
Rui Yan, Liangqiong Qu, Qingyue Wei, Shih-Cheng Huang, Liyue Shen, D. Rubin, Lei Xing, and Yuyin Zhou. Label-efficient self-supervised federated learning for tackling data heterogeneity in medical imaging. IEEE Transactions on Medical Imaging, 42: 42:1932- 1943, 2022. 5, 17
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-ibbalanced learning. In NewrIPS, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. 5, 17
Yuzhe Yang, Kaiwen Zha, Ying- Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference Machine Learning (ICML), 2021. 11
14

---

Published in Transactions on Machine Learning Research (10/2024)
Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng, Tongliang Liu, and Bo Han. Fedfed:
Feature distillation against data heterogeneity in federated learning. In Thirly-seventh Conference Neural Information Processing Systems, 2023. URL https://openrevier: net//oorum/idrppnniin
Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang- Feddisco: federated learning with discropancy-aware collaboration. In Proceedings of the 40th International Conference
Muchine Learning, ICML'23 23. JMLR.org, 2023. 1,
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphanc Deny. Barlow twins: Sclf-supervised learning via redundancy reduction. In International Conference Machine Learning, Proceedings of Machine Learning Research, pp. 12310-12320, ML Research Press, 2021. 17, 19
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pscudo labeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), NewrIPS, 2021. URL https://openreview wett/forummiddd3//aaaBBB
2, 3, 4, 5, 16, 19, 22, 23
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. Federated unsupervised representation learning. CoRR, abs 20s/2000008982, 2020. 5,
J. Zhang, Xiaosong Ma, Song Guo, and Wenchao Xu. Towards unbiased training in federated open-world semi-supervised learning. In International Conference on Machine Learning, 2023.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In ICLR, 2022. URL https:
5, 17
15

---

Published in Transactions on Machine Learning Research (10/2024)
Appendix for "Semi-supervised Anchor-Based Aggregation in Federated Learning"
The appendix organized as follows:
In Appendix A, we provide detailed experimental analysis and discussion on federated self-supervised warmup, addressing the data dependency issues in FedSemi under limited data conditions.
In Appendix B, we provide a more in-depth discussion of for the unlabeled client model aggregation (SemiAnAgg) and present empirical convergence analysis.
In Appendix C, we discuss and present the detailed experimental settings well as additional experiments. This appendix is organized as follows:
Appendix C.1: Comprehensive implementation details.
Appendix C.2: Information on dataset splitting and pre- pre-processing.
Appendix C.3: Experiments involving more than one labeled client.
Appendix C.4: Experiments and discussion with partially labeled clients.
In Appendix D, we offer additional details on the integration of semi-supervised imbalance learning techniques, such as ACR (Wei & Gan, 2023), into the federated learning framework to establish a stronger baseline.
In Appendix E, we provide an ablation study that examines the impact of different local client's training strategies, including FlexMatch (Zhang et al., 2021) and SoftMatch (Chen et al., 2023).
License of the assets
License for the codes
We have reproduced the code for RSCFed (Liang al., 2022) and CBAFEd (Li et al., 2023) and have achieved higher KPIs. The code for both implementations is publicly available. We plan to make code publicly available upon acceptance licensed under the MIT License, which can be found at nttps://openoource org/Licenses/MIT.
License for the dataset
For CIFAR-100 (Krizhevsky, 2009): We adhere to the terms provided by the CIFAR-100 dataset, which is released under the MIT License.
For SVHN (Netzer et al., 2011): We adhere to the usage of SVHN, which consists of Google Street View images, for non-commercial purposes.
For ISIC-18 (Tschandl al., 2018): In accorrdacce with the data use agreement, we comply with the attribution requirements of the Creative Commons Non-Commercial license (CC-BY-NC).
16

---

Published in Transactions on Machine Learning Research (10/2024)
A Federated Self-Supervised Warmup
To ensure reliable pseudo-labels for unlabeled clients in initial stages, previous Federated Semi-Supervised Learning (FedSemi) approaches (Li et al., 2023; Liang et al., 2022) have employed a warm-up phase based on labeled clients using traditional FedAvg (McMahan et al., 2017). Unfortunately as depicted in Figure 6, a poorly executed supervised warm-up can lead to model degradation, especially in the case of working with limited samples (Yang & Xu, 2020; Yan et al., 2022). To address this issue, intuitive federated self-supervised learning (Zhang et al., 2020; Lubana et al., 2022; Zhuang et al., 2022; Kim et al., 2023a) can be employed solution to alleviate the warm-up problem.
(a) Label-Dependent Warmup
(b) FedBarlow (ours) Warmup
Accuracy (96)
Accuracy
myyyy 4M,
0
600
800
1000 Round
1200
1400
600
800
1000 Round
1200
1400
RSCFed
CBAFed
SemiAnAgg (ours)
Figure 6: Comparative performance of RSCFed (Liang et al., 2022) and CBAFed (Li et al., 2023), and SemiAnAgg (ours) with (a) label-dependent warm-up and (b) self-supervised warm-up on CIFAR-100 LT.
this paper, we shed light on the impact of an overlooked straightforward architecture, Barlow Twins (Zbontar et al,, 2021), which is not studied previously in the FedSelf framework. We re- FedBarlow through the
50
name this architecture minimization of Barlow-Twins redundancy objective
Accuracy
locally, while the aggregation of local clients' models is performed using the FedAvg algorithm.
20
It empirically demonstrated that FedBarlow, de- spite its simplicity, is highly effective baseline in FedSelf.
10
Orchestra
FedBarlow (ours)
0
200
400
600
800
1000
Round
In Table 5, we present comparative results on the SVHN and CIFAR-100 datasets in the context of self-supervised federated learning. We compare our approach, FedBarlow, with the clustering-based ap- proach, Orchestra (Lubana al., 2022). Our results demonstrate significant improvement over Orches- tra CIFAR-100, achieving 9.52% increase in lincar evaluation performance. This improvement is even higher with 4.02% compared to the reported results in (Lubana et al., 2022) (55.89%). Furthermore, when training a federated online classifier using the 10 labeled clients, FedBarlow showcases remarkable improvements with a substantial increase 14.6% and 10% on CIFAR-100 and SVHN, respectively. This highlights the importance utilizing the Barlow Twin objective as baseline in federated self-supervised learning, as it provides consistent objective that promotes smooth alignment and leads to significant performance gains.
Figure 7: Comparison of Orchestra (Lubana et al., 2022) and FedBarlow (ours) in Federated Self- Supervised warmup on CTFAR-100. A federated online classifier trained on the labels with a stop-gradient (SG).
17

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: Comparison the accuracy between FedBarlow and Orchestra (Lubana et al., 2022) a cross-silo 10-device setting on the CIFAR-100 and SVHN dataset. The upper bound corresponds to "Lincar" probing over the dataset. while the lower bound pertains "Linear" on 10% of the data (one labeled client). FedCLR. is online classifier learned online between all clients with stop gradient.
Method
CIFAR-100
SVHN
Acc.(R)
AUC(R)
Aec.(%)
AUC(X)
Orchestra (Upper Hound]
50.39
96.81
97.43
FedBarlow Ours (Upper Bound)
59.91
98.32
85.89
98.56
Orchestra (Lower Bound]
25.02
89.63
64.14
94.85
FedBarlow Ours (Lower Bound)
37.07
94.06
64.96
96.37
Orchestra (FedCIR)
42.81
96.49
69.03
95.08
FedBarlow Ours (FedCLR)
57.37
98.02
79.09
97.30
B SemiAnAgg Convergence
This section provides in-depth discussion and empirical convergence analysis of the unlabeled client model aggregation approach, SemiAnAgg, on two datascts, SVHN and ISIC-18. Our SemiAnAgg focuses on the valuation of whether the model is learned properly while measuring divergence. SemiAnAgy introduces a novel client weighting strategy that aims to achieve a balanced contribution from unlabeled clients based on diversity measurements.
The SemiAnAgg weighting strategy offers an alternative metric to the computationally expensive impractical leave-one-out data valuation (Ghorbani & Zou, 2019). Unlike other approaches in semi-supervised learning that rely metrics such model confidence (Chen et al., 2023) uncertainty ensembles (Chen et al., 2020), which have proven to be unreliable in imbalanced and class distribution mismatch scenarios, SSIL-CDM (as demonstrated in ACR (Wei & Gan, 2023))- SemiAnAgg takes a different perspective to address the challenges of fodcrated non-iid semi-supervised training by considering diversity measurements as a critcrion for client wnighting. SemiAnAgg as novel FedSemi adaplive weighting strategy shows competitive results to the baseline not using it as demonstrated in Figure 8.
92
70
90
65
E45.0
Accurace (%)
42.5
88
Accuracy.
660
40.0
86-
37.5
FedAvg Semii
50
FedAug Semi
FodAvg Semi
35.0
SomiAnAgg Barch
SemiAnAgg 0wach
SemiAnAgg w/ Sunch
84
SemiAnvgg 0areh.2
45
SemiAnAga Oucch,
32.5
SemiAnAgg w/ Sunch.2
600
760
860
900
1000
40
600
1060
500
760
860
1000
Round
Round
Round
(a) SVHN Ablation Accuracy.
(b) TSIC-18 Ablation Accuracy.
(c) ISIC-18 Balanced Accuracy.
Figure 8: Ablation study comparing the use of SemiAnAgg and not using it on two different datasets, SVHN and 1SIC-18. Note that ISIC-18 is highly imbalanced, so we also report the balanced accuracy.
On ISTC-18, an extremely imbalanced dataset, the global optimization can be skewed towards the majority classes, heavily biasing pseudo generation leading the model get stuck local optimum. In Figure 9, analyze the convergence of SemiAnAgg SemiAnAgg, which is supported by the leave-one-out analysis showing significant error increase when client 2 is removed Figure 9c. Interestingly, client 4, despite having the highest pscudo-label accuracy in Figure 9a, contributes moderately in SemiAnAgy, as supported by its moderate ranking in the leave-one-out analysis. Contradictions observed: client contributes substantially in SemiAnAgg, yet its removal does not result in significant performance drop. This may be attributed client containing data with sensitive attributes (e.g., color, gender) that are not well represented in the imbalanced test set. Therefore, the leave-one-out analysis does not. fully reflect its importance if such attributes are absent in the test set. Through these two
ISIC-18. In Figure 9b, client 2 contributes the most to
18

---

Published in Transactions on Machine Learning Research (10/2024)
case studies, substantial improvements, and ablations, it becomes crucial to use SemiAnAgg approach when training in federated semi-supervised learning set settings.
2500
70
Global Contribution Value
0.113
2.5
2.0
2000
Accuracy
0.112
Enror (%)
50
1.5
1500
0.111
.40
Unlabeled Client
Unlabeled Client
1.0
1000
4
0.110
30
0.5
500
3
0.109
20
0.0
700
1000
500
600
800
900
1000
Round
Round
Leave Unlabeled Client Out
(a) Pseudo Label Accuracy,
(b) SemiAnAgg Client Contribution.
(c) Leave one Unlabeled-
Figure SemiAnAgg convergence analysis on the imbalanced ISIC-18. Note that unlike SVHN which is relatively easy task. SemiAnAgg converges to balanced version considering class proportion to avoid majority class skewing.
Additional Experiments
C.1
Implementation Details
This section provides detailed explanation of the implementation details for the main section (Sec. 4.1). We implement our method using the PyTorch framework (Paszke et al., 2019). In the Federated Semi-Supervised Learning (FedSemi) framework, SGD optimizer is used with a learning rate of 0.03 for labeled clients and 0.02 for unlabeled clients, following previous FedSemi approaches such as RSCFed (Tiang et al., 2022) and CBAFed (Li et al., 2023). This difference in learning rates serves as an implicit regularization to prevent unlabeled clients from deteriorating the received global model case of erroneous pseudo labels. A batch size of 64 is utilized for all datasets. The experiments were conducted on NVIDIA RIX 3090 GPUs.
The reported results in the main table follow warm-up stage, similar to CBAFed (Li ct al., 2023) and RSCFed (Liang el al., 2022). Specifically, C1FAR-100 and its long-tailed version are pre-trained for 1000 epochs, while the other datasets are pre-trained for 500 epochs. This is followed by an additional 1000 epochs for CIFAR-100 and its long-tailed version, and an additional 500 epochs for the other datasets. The optimal hyperparameteer settings for other methods, RSCFed (Liang et al., 2022) and CBAFed (Li al., 2023), as reported and presented their respective code, are utilized. It noted that, unlike other methods that involve scaling and extensive hyperparameters, our SemiAnAgg does not contain any hyperparameters, and the parameters solely dependent the local learning method.
For the local training procedure FlexMatch (Baseline) (Zhang et al., 2021), the default setting with the default confidence thresholds presented in TorchSSL, which is the official implementation of FlexMatch, is adopted. Better results are reproduced for all FedSemi baselines compared to the originally stated results in their tables (Li et al., 2023; Liang et al., 2022). The improvement for smaller spatial resolution datasets (SVHN, CIFAR) comes from utilizing an optimal architecture, CIFAR ResNet-188 (He al., 2016), while for ISIC-18, the architecture (ImageNet ResNet-18 (He et al., 2016)) remains the same, and the improvement solely comes from utilizing RandAugment (Cubuk et al., 2020) augmentation for all datasets. For all basclines, the classifier is learned without weight decay, while the backbone has weight decay of 5c-4.
In experiments with self-suervised pre-training, the official Barlow Twins implementation is followed (Zbontar et al., 2021), but with some modifications. SGD optimizer used with a learning rate 0,008, and a scale loss of 0.01 is applied, which acts as an implicit scaling the learning rate for biases and batch norm parameters. The choice of hyperparameters is based on the small batch size of 64 used for pre-training on all datasets.
19

---

Published in Transactions on Machine Learning Research (10/2024)
C.2 Dataset pre- ve-processinn
We explain in this section the steps taken for dataset splitting and pre-processing. The same dataset splitting for CIPAR-100, SVHN, and the skin dataset is adopted as in previous FedSemi approaches, RSCFed (Liang et al., 2022) and CBAFed (Li ct al., 2023). It is ensured that the partition loading and splitting are derived from the same checkpoint for all evaluated methods, ensuring consistency.
Furthermore, the same pre-processing steps as those used in CBAFed (Li et al., 2023) and RSCFed (Liang et al., 2022) are adopted. These steps are likely described in more detail in the respective papers (Li et al., 2023; Liang et al., 2022). Additionally, consistent. data augmentation technique based RandAugment (Cubuk et al., 2020) is employed. RandAngment (Cubuk et al., 2020) has been shown to improve performance for all baseline methods.
C.3 More than One Labeled Client
In this section, we present the results of more than one labeled client setting for the ISTC-18 dataset. As expected, all methods show improved performance compared to the one labeled client setting.
In Table 6, we report the lower bounds for FedAvg (MeMahan al., 2017), which is used for the initialization of RSCFed (Liang et al., 2022) and our SemiAnAgg, and the lower bound of CBAFed (FedAvg with residual weight connection), which serves as the initialization for CBAAFd (Li et al., 2023). The lower bound of CBAFed (Li el al., 2023) is significantly higher than that of FedAvg (MeMahan et al., 2017), with an improvement of 4.9% in accuracy. However, while CBAFed (Li et al., 2023) exhibits enhanced performance in terms of accuracy (0.5% improvement) and AUC (3.0% improvement), it experiences a drop in balancod accuracy (1.8% decrease) compared to its lower bound. This observation is consistent with previous evaluations in the one-labeled client setting.
Table 6: Comparison of SemiAnAgg against CBAFed (Li et al., 2023) and RSCFed (Liang et al., 2022) on the ISIC-18 dataset. All methods are reported using logits adjustments (Ren et al., 2020) in labeled clients. Note that CBAFed benefits from an initialization with temporal ensembling while RSCFed and Ours are initialized with the same model without temporal ensembling.
Labeling Strategy
Method
Jabeled
Cliens Nun.
unlabeled
Ace. (%)
AUC. (%)
Metrics
Precision
Hecil (%)
Fully superiised
CBARsl (lower-homdd)
Fedldgg (lover bound)
2
2
60.00 96
74.84
85.61
85.35
51.38
44.42
68 $3.01
47.95
RSCPed
76.34
Secni supervised
CBAF
76. 40
89.57
54.87
45.49
SemiAnAgs (oues)
76.59
$9.69
67.17
50.05
Remarkably, our SemiAnAgg achieves the best results across all four metries, particularly demonstrating impressive 12.3% increase in precision and significant 4.5% improvement in recall (balanced accuracy) compared to state-of-the-art CBAFed (Li et al., 2023), despite being initialized from worse starting point.
When compared to RSCFed (Liang et al., 2022), which maintains consistent initialization, our SemiAnAgg achieves greater improvements of 8.28% in precision and 0.95% in recall (balanced accuracy). It is important to note that RSCFed (Liang et al., 2022) removes noisy clients through average consensus, leading to the climination of minority classes and consequently lowering prccision (by prodicting a high number of false positives, considering minority as majority) and recall (by predicting lower numbers for minority classes). In contrast, our SemiAnAgg addresses this issue by adaptively valuing unlabeled clients, implicitly assigning greater weight to clients with minority classes, thereby achieving a more balanced performance.
C.4 Partially Labeled
In the context of FedSemi, we consider a scenario where clients have partial labeling, with each client having only 10% of its samples labeled. This setting presents a relatively easier local optimization compared to the case of one fully labeled client, but global optimization becomes challenging due to the non-iid data distribution. In this section, we present results for partially labeled clients on two datasets: SVHN and ISIC-18. The results are shown in Table 7.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Comparison of SemiAnAgg against CBAFed (Li et al., 2023) and RSCFed (Liang et al., 2022) in the partially labeled setting on two datasets, SVHN and ISIC-18. For ISIC-18, all methods are reported using logits adjustments (Ren al., 2020) based on the local labeled samples distribution. Note that CBdFedd benefits from initialization with temporal ensembling while RSCFed and SemiAnAgg (Ours) are initialized with the same model without temporal ensembling.
Client Ratio
Metrics
Labeling Strategy
Method
Tabeled unlabeled
Acc. (%6
AUC. (%)
Precision (76]
Recall
Dataset
SVHN
Fully supervisod
FodAvg (upper-boand)
100%
0%
94.76
99.67
93.95
94.82
FedAAg (lower- bownr-
10%
0%
83.80
98.51
82.10
85.15
CBAFed (lowet-bound)
10%
0%
84.15
98.45
82.00
81.83
RSCFed
10%
90%
87.27
98.76
85.79
87.19
Semi supervised
CBAFed
10%
90%
90.94
99.20
90.43
90:18
SemiAnAgg (ours)
10%
90%
92.57
99.50
92.02
92.28
Dataset
ISIC-18
Fully supervised
FedAwgg (upper-bound)
100%
0%
80.78
93.53
61.55
69.22
FedAvg (lower-bound)
10%
0%
66.95
82. 17
35.39
11.62
CBAFed [lower-bound)
10%
0%
68.30
82.
36.62
40.78
RSSFFdd
10%
907
08. 20
81.69
35.36
33.22
Semi supervised
CBAFed
10%
90%
67.40
79.86
37.00
38.21
Semi AnAR (ours)
10%
90%
70.39
88.34
40.71
42.62
99.4
99.2
Accuracy (%)
99.0
88
AUC
98.8-
1
98.6
86
CBAFed
98.4-
CBAFed
SemiAnAgg
SemiAnAgg
84
98.2
700
900
1000
700
860
1000
Round
Round
(a) Accuracy on SVHN dataset.
(b) Area Under the Curve (AUC) on SVHN dataset.
Figure 10: Comparison with the state-of-the-ar. FedSemi on SVHN in the partial label setting.
We observed that the lower bound of CBAFed (Li et al., 2023) benefits from temporal ensembling, achieving 0.4% and 1.4% improvement in accuracy SVHN and ISIC-18, respectively, compared to the lower bound of FedAvg (McMahan et al., 2017). However, our SemiAnAgg surpasses CBAFed (Li et al., 2023) with a 1.6% improvement in accuracy on SVHN and a significant. 2.9% improvement in accuracy and 4.4% improvement in balanced accuracy on the ISIC-18 dataset. Notably, the results for SVHN, which is a relatively easier dataset, are higher than those obtained the one fully labeled client setting. This can be attributed to the dual environment learning in FL, which promotes invariant feature learning (Tang et al., 2022). On the other hand, for the highly imbalanced ISIC-18 dataset, it becomes challenging to learn the distribution in such heterogeneous way, resulting in lower performance compared to the one fully labeled client setting. In Figure 10, we provide an analysis of the learning behavior of SemiAnAgg and the state-of-the-art CBAFed (Li et al., 2023) on the SVHN dataset in the partial label setting.
Comparing our SemiAnAgg RSCFed (Liang et al., 2022), we outperform RSCFed (Tiang et al., 2022) by 5.3% in accuracy on SVHN and by 2.1% in accuracy and a remarkable 9.4% balanced accuracy on the ISIC-18 dataset. Additionally, SemiAnAgg achieves these improvements with less than 60% of the communication cost of RSCFed (Liang et al,, 2022). It is worth noting that RSCFed (Liang al., 2022) removes noisy clients through averaged consensus, which may exclude properly learned labeled clients that have unique classes not previously encountered. In contrast, SemiAnAgg incorporates bias learning for properly learned models in a balanced manner.
In the field of Foderated Semi-Supervised Learning (FodSemi), previous approaches have often overlooked the aspect of supervupe imbalanced learning (SSIL) in their methods. Particularly in the non-iid setting, SSTL becomes apparent depicted in Figure 11.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Client
Client
5
1
4
7
Class
Class
(a) SVHN
(b) ISIC-18 (Skin)
Figure 11: Clients Class distribution (Client 0 is labeled).
To establish competitive baseline, implemented the state-of-the-art SSIL method known ACR (Wei & Gan, 2023). Our intuition is that ACR (Wei & Can, 2023) could take into account both the imbalance resulting in the non-iid FL and the mismatch between labeled and unlabeled clients' class distributions. In the context of FL. we renamed this method as SSIL-CD.. Our previous experimental results have demonstrated that SSIL-CDM outperforms existing FedSemi methods, including RSCFed (Liang et al., 2022) and the state-of-the-art CBAFed (Li et al., 2023). However, it worth highlighting that successful design strategies employed in FL play a crucial role the ACR (Wei & Gan, 2023) method.
Table 8: Ablation of SemiAnAgg on the ISIC-18, skin. and the average per class accuracy B-Ace scnsitive to imbalance which is the macro averaged recall in multi-class classification.
Sodi Aduetivv Adj (Wel & Gan, 2023)
Dual Branch
Metries
Act. (%)
AUC (%)
Pre (%)
B-Axc (%)
SSIL-CDI, w/o VedAvg-Semil
66.60
62.47
50.33
35.88
SSIL-CDM w/ PeniAg--Smni
71.31
85.12
46.36
43.14
FedRoD
FedRoD Dual
71.84
87.81
44.20
44.62
SomiAnAgg (ours)
D Integration of Semi-Supervised Imbalance Learning Techniques
The original ACR (Wei & Gan, 2023) requires two-
90
branch network with logit adjustment (Ren et al., 2020). Without prior knowledge about the global distribution (e.g,, uniform, long-tailed, or imbalanced), one approach is to use labeled clients to adjust the logits of unlabeled clients. However, this can amplify an inaccurate distri- bution, leading to severe performance degradation (See orig-ACR FL in Figure 12). An improvement can be achieved by estimating stable pseudo-label distribution with Exponential Moving Average (EMA) locally using the global model. This modification namely SSIL-CDM achieves reasonable performance in Figure 12. All models are ialized with Fed detailed in Equation 2.
80
40
30-
SSIL-CDM FedAvg-Seml
20
500
520
580
600
Round
Figure 12: Comparison ACR (Wei Gan, 2023) with logit adjustment calculated from la- beled clients (SSIL-CDM) and ACR with logit calculated locally with global model pscudo la- bel distribution with EMA and FedAvg-Se Semi on SVHN dataset.
Fed modifies the global optimization be sim- ilar to local self-tree training optimization (Chen et al., 2023;
Zhang et al., 2021; Sohn et al., 2020; Wang et al., 2023).
22

---

Published in Transactions on Machine Learning Research (10/2024)
In Table we present the results of different strategies the ISIC-18 dataset. Using SSIL-CDM with Fed-vg-Semi achieves improvements of 4.7% and 7.3% on accuracy and balanced accuracy compared to not using FedAvg-Semi (w/o FedAvg-Semi).
Our proposed approach, SemiAnAgg, outperforms our most competitive baselines with 0.9% and 5.6% on accuracy and balanced accuracy respectively. Notably, SemiAnAg achieves this superior performance without. requiring specifically tailored architecture design the need for a dual branch architecture.
E
Client Local Training Methods
In this section, we present an ablation study to evaluate the impact of different local training methods, namely FlexMatch (Zhang al., 2021) and Mofth (Chen et al., 2023), on the ISIC-18 dataset.
In Table 9, local semi-supervised learning meth- ods show limited results due to the nature of Fe- dAvg (McMahan et al., 2017), which does not fully account for the fact that labeled clients can provide more accurate information than unlabeled clients, regardless of data volume. While contributions such CBAFed (Li et al., 2023) offer enhanced perfor- mance with 9.41% increase in balanced accuracy, this still does not fully resolve the limilation.
Table 9: Ablation study on local training methods for FL. f indicates methods specifically tailored for FL.
Method
Acc. (%)
B-Ace (%)
FedAvg
CBAFedj (Li et al., 2023)
69.99
41.12
FlexMatch (Zhang et al., 2021)
62.16
31.73
SoftMatch (Chen et al., 2023)
66.99
38.02
FedAvg-Somi [our strong baseline)
Our FedAvg-See Semi approach mitigates this limitation by balancing the empirical risk between labeled and unlabeled data, resulting enhanced performance: an 8.48% increase in balanced accuracy for Flex- Match (Zhang et al., 2021) and 8.57% increase in
FlexMatch (Zhang ct al., 2021)
67.79
40.21
Mott (Chen et al., 2023)
72.39
46.59
SemiAnAge (ours)
FlexMatch (Zhang cl al., 2021)
72.24
48.77
SoftMatch (Chen et al., 2023)
72.59
49.84
balanced accuracy for SoftMatch (Chen et al., 2023). Finally, by leveraging SemiAnAgg and considering the diversity of clients, we achieve the highest performance with a 7.65% increase in balanced accuracy for FlexMatch and 8.72% for SoftMatch compared with the state-of- of--th--rttttttoo (SOTA) FedSemi approach, CBAFed (Li et al., 2023).
23


Reviews:
[{"review_id": "09X3JaFgjk", "reviewer": "Reviewer_upWd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the idea of ​​leveraging unlabeled client's data that was previously treated as noise to improve accuracy. It's simple idear, but it seems work well.\n\nHowever, there is a lack of analysis on why the proposed technique produces better performance. Even if it is not a mathematical proof, I think that more deeper analysis is needed. And the some of model's accuracy is too low. Of course, this is not about optimization paper, so ahieving optimal accuracy is not important,but it is a bit difficult to compare the performance of models with a b-acc value of less than 50%. I am curious whether the proposed method demonstrates better performance even when a model have a meaningful level of accuracy. If not, it would be helpful to explain why it was necessary to use a model with such low accuracy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SemiAnAgg, a novel semi-supervised anchor-based aggregation method for federated learning, which effectively learns and leverages the contributions of unlabeled clients. The authors showed through experiments that the proposed technique improves accuracy compared to existing techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide a more in-depth explanation of why the proposed method achieves better performance.\n\n2. I think that the authors need to conduct experiments to show whether the proposed method still delivers improved results when applied to a model with better accuracy, instead of one with low accuracy. If it does not, please explain the reasons.", "broader_impact_concerns": "None of the concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "67db03XPi6", "reviewer": "Reviewer_kwtP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe manuscript is overall well-written and clear. The authors motivate the problem and their solution well. They explain the method concisely. \n\nThe experiments' coverage seems broad. The authors have performed a wide range of ablation studies regarding their method.\n\nThe authors have shared the experiment codes.\n\nWeaknesses:\n\nThe authors have discussed many previous papers on Federated Semi-supervised Learning. However, only two well-performing methods are used as baselines. Considering this is a fully experimental method and paper, adding more competing baselines could be better. \n\nPlease also see the \"Requested Changes\" section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors identify the two limitations of semi-supervised FL settings where clients may have all/some/no labeled data. First, existing techniques rely on the FedAvg type of aggregation, where the dataset sizes of clients are used to determine the aggregation weights. Authors claim that this may lead to suboptimal training. Second, the current methods cannot handle the diverse contributions of unlabeled clients. They are inclined to overlook and minimize those clients' effects as those clients can be regarded as noisy.  The authors propose SemiAnAgg, which resolves these issues by using anchor models to evaluate the importance of the unlabeled clients to determine the weights. The proposed method does not introduce significant computation, communication, and memory burden. The authors provide experimental evidence showing the superiority of their methods compared to the baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In the Introduction, a more concrete example can be given for the FedSemi setting instead of only unclearly mentioning a cross-silo hospital setting. (Minor)\n2. Do all previous methods use the FedAvg type of aggregation? If not, please mention some exceptions while explaining the first problem in the Introduction. \n3. The second limitation of current methodologies is their failure to account for the diverse contributions of unlabeled clients, particularly in reflecting the presence of minority classes and unique attributes within majority classes: This may require a brief further explanation. What made current methods overlook those clients? Are there no solutions to this in the literature? (Minor)\n4. The first one of two intuitions at the bottom of the second page requires more explanation. Why is it so?\n5. I have a high-level question about your method and Centralized Semi-Supervised ML: Is your method also applicable to centralized training? For example, using your method by considering unlabeled data samples as unlabeled clients. I mean, in a batch of samples to make a gradient update in the training procedure, we may weigh the importance of samples differently based on sample scores. Does a similar approach exist in centralized literature? If so, please discuss it and the connections to your work. If not, can we say that your method is also a contribution to the centralized problem?\n6. The performance of your method and comparisons to baselines should depend on how the local training is done. Did you have an ablation study on local training you used, FlexMatch?\n7. On page 6, please define  $\\theta_{glob}^{enc}$ and  $\\theta_{glob}^{fc}$  and any other variables before using them. That section requires a clarification about the variables.\n8. A question for future improvement: What if you used more than one anchor model/feature representation? Does using more independent feature information improve the training?\n9. In the second part of Equation 4., I suggest using parentheses for clarity, e.g., $\\sum\\left(\\dots\\right)\\theta_k$ (Very minor)\n\nTypos etc.:\n- A missing space in Figure 1 Caption: benchmarks.Lower:", "broader_impact_concerns": "I have no concerns about the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5qNjyxRAWL", "reviewer": "Reviewer_Wkv2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The critique of existing approaches is well-reasoned and persuasive.\n\n2. Visuals and diagrams in the paper are effectively presented, enhancing comprehension.\n\n3. Introducing a randomly initialized model as a baseline to assess deviations in client models appears to be novel to me.\n\n**Weaknesses:**\n\n1. The initial reasoning presented by the authors is largely heuristic and lacks a formal guarantee for ensuring robustness. For example, a highly complex model (with a high Lipschitz constant) may deviate significantly from a random initialization, resulting in an overfit and non-generalizable model while exhibiting high weighting score from your proposed metric.\n\n2. The paper would benefit from additional comparative baselines to fully demonstrate the empirical relevance of the proposed method [1,2,3].\n\n3. It omits the discussion of several highly relevant works [4]. \n\n4. The methodology seems to be unrelated to the core principles of semi-supervised learning, functioning independently of its specific features. This method might be more appropriate compared with other techniques that focus on client reweighting or selection.\n\n**Other Issues:**\n\nThere appears to be an inconsistency in reference management, particularly noticeable with the repeated citation of references [2]. A thorough review of the reference list is recommended.\n\n[1] Rethinking Semi-Supervised Federated Learning: How to co-train fully-labeled and fully-unlabeled client imaging data, MICCAI 2023.\n\n[2] Towards unbiased training in federated open-world semi-supervised learning, ICML 2023.\n\n[3] Local or global: Selective knowledge assimilation for federated learning with limited labels, ICCV 2023.\n\n[4] SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training, NeurIPS 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes, SemiAnAgg, in Federated Semi-supervised Learning, focusing on better integrating unlabeled clients by leveraging a randomly initialized model as an anchor to evaluate client deviations. The introduction of SemiAnAgg allows for an enhanced aggregation strategy by recognizing and valuing the diverse contributions of unlabeled clients without sharing sensitive data. Extensive experiments across multiple benchmarks demonstrate the method's effectiveness, significantly improving accuracy and recall on datasets like CIFAR-100 and ISIC-18 compared to existing state-of-the-art methods. This approach not only outperforms traditional federated learning aggregations but also provides a framework for leveraging unlabeled data in a privacy-preserving and efficient manner.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please make appropriate adjustments based on the aforementioned weaknesses.", "broader_impact_concerns": "n.a.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper argues that previous methods for federated semi-supervised learning do not take advantage of unlabeled clients but rather consider them as noise. The authors propose a new method for federated semi-supervised learning that aggregates the unlabelled clients by using a randomly initialized model as a common baseline that helps them evaluate deviations in client models. The method is shown to lead to improved accuracy and recall on a range of data sets.

All reviewers are in agreement that with the revision, the paper is suitable for TMLR and should be published. I concur.
Decision: Accept (poster)