Unlocking the Potential of Model Calibration in Federated Learning
Authors: Yun-Wei Chu
Dong-Jun Han
Seyyedali Hosseinalipour
Christopher Brinton
Venue: ICLR 2025 Poster (2025)
Keywords: Federated Learning
Model Calibration
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Over the past several years, various federated learning (FL) methodologies have been developed to improve model accuracy, a primary performance metric in machine learning. However, to utilize FL in practical decision-making scenarios, beyond considering accuracy, the trained model must also have a reliable confidence in each of its predictions, an aspect that has been largely overlooked in existing FL research. Motivated by this gap, we propose Non-Uniform Calibration for Federated Learning (NUCFL), a generic framework that integrates FL with the concept of model calibration. The inherent data heterogeneity in FL environments makes model calibration particularly difficult, as it must ensure reliability across diverse data distributions and client conditions. Our NUCFL addresses this challenge by dynamically adjusting the model calibration objectives based on statistical relationships between each client's local model and the global model in FL.  In particular, NUCFL assesses the similarity between local and global model relationships, and controls the penalty term for the calibration loss during client-side local training.  By doing so, NUCFL effectively aligns calibration needs for the global model in  heterogeneous FL settings while not sacrificing accuracy. Extensive experiments show that NUCFL offers flexibility and effectiveness across various FL algorithms, enhancing accuracy  as well as model calibration.

Published as a conference paper at ICLR 2025
UNLOCKING THE POTENTIAL OF MODEL CALIBRA- TION IN FEDERATED LEARNING
Yun- Wei Chu', Dong-Jun Han', Seyyedali Hosseinalipour', Charissoper Purdue University, YYonsei University, 'University at Buffalo-SUNY chul phur9e, edu, djheyonsei. www.seiiic...... kr. alipourebuffal falo. edu, cgbêpurdue.
Christopher G. Brinton'
ABSTRACT
Over the past several years, various federated learning (FI.) methodologies have been developed to improve model accuracy, a primary performance metric in ma- chine learning. However, to utilize FL. in practical decision-making scenarios, beyond considering accuracy, the trained model must also have reliable con- fidence each of its predictions, an aspect that has been largely overlooked Calibra-
existing FL research. Motivated by this gap, propose tion for Federated Learning (NUCFL), a generic framework that integrates FL with the concept of model calibration. The inherent data heterogeneity in FL environ- ments makes model calibration particularly difficult, as it must ensure reliability across diverse data distributions and client conditions. Our NUCFL addresses this challenge by dynamically adjusting the model calibration objectives based on sta- tistical relationships between each client's local model and the global model FL. In particular, NUCFL L assesses the similarity between local and global model relationships, and controls the penalty term for the calibration loss during client- side local training. By doing so, NUCFL effectively aligns calibration needs for the global model heterogeneous FL settings while not sacrificing accuracy. Ex- tensive experiments show that NUCFL offers flexibility and effectiveness across various FL algorithms,, enhancing accuracy as well as model calibration.
INTRODUCTION
Federated learning (FL.) has rapidly gained traction as a prominent distributed machine learning ap- proach, enabling collaborative model training across network of clients by periodically aggregating their local models at a central server (Koneený et al. 2016 McMahan et al. 2017). Its significance extends to various critical applications, including medical diagnostics, self-driving cars, and multi- lingual systems, where data privacy and decentralized training are paramount, FL has been widely studied with focus improving local model updates (Reddi al. [021 Sahu et al. (2018), refin- ing global aggregation methods (Ji et al., 2019; Wang et al.] 2020a), and enhancing communication efficiency (Sattler al. 2019 Diao et al. 2020 Parasnis et al. 2023). Most of these works in FL. consider accuracy as the main performance metric.
Motivation. However, beyond accuracy, in various fecision-making scenarios where an incorrect prediction may result in high risk (e.g., medical applications or autonomous driving), it also crucial for the users determine whether to rely on the FL model's prediction not for each decision. In particular, users should rely on the neural network's 's decision only when the prediction is likely to be correct. Otherwise, they may need consider alternatives such human decision. To achieve this, the trained FL model should have reliable confidence in each of its predictions, meaning that the confidence of the neural network matches well with its actual accuracy. centralized settings, recent studies including Guo et al. (2017) have uncovered that neural networks are often miscalibrated, indicating that the prediction confidence of the model does not accurately reflect the probability of correctness. Handling this miscalibration issue is even more important in many FL use-cases, where an overconfident global model could lead to misinformed decisions with potentially severe consequences for each client.
In centralized training settings, research generally follows two paths to address this miscalibration issue. First, train-time calibration methods (Hebbalaguppe et al. 2022b Liang et al. 2020 Liu et al.

---

Published as a conference paper at ICLR 2025
(3) Centrallzed Training without calibvation
(b) Centraltsed Traininng with DCC
(d) Non LDD FL without caliiration lon
(4) Noo- HID FL with DICA
Edl Non HDD EL with NUFFI (nars)
sap
take
Accunay
5.30
52
7.69
Domfidence
Coniidence
Confiionco
Confidence
Gonfideence
Reliability diagrams and calibration errors for centralized training and non-IlID FL (using FedAvg) trained with various calibration methods on CIFAR-100 dataset. Our method ensures well-calibrated FI., evi- denced by a notably smaller calibration error and smaller gap (red region) between confidence and accuracy.
Figure 1:
2021 Mukhoti et al.; 2020 Kumar et al., 2018 Müller al.) 2019 Lin et al. 2017) incorporate
explicit regularizers during the training process to adjust neural networks, scaling back over/under- confident predictions. On the other hand, post-hoc calibration (Guo et al.] 2017 Hendrycks & Gimpel, 2016] Hinton et al., 2015; Kull et al.) (017) transforms the network's ' output vector to align the confidence of the predicted label with the actual likelihood of that label for the sample. Post- hoc calibration methods are applied to the already trained model to improve calibration using an additional holdout dataset. However, an appropriate holdout dataset may not be available in many privacy-sensitive applications, making post-hoc calibration strategies often impractical for FI..
Goals and observations. In this paper, we aim to incorporate model calibration into the FL training process, problem that has been largely underexplored in existing research. This setting introduces new challenges to improving model calibration, as the heterogeneous data distributions across dif- ferent clients must be carefully considered. We first observe whether existing train-time calibration methods applied during client-side local training can effectively address calibration needs in FL. To gain insights, in Fig- [ we report accuracy, confidence, and the expected calibration error (ECE) of different methods in different settings. Figs. aa) to (Tb) illustrate the capability of train-time calibration methods to narrow the gap between model confidence and accuracy in centralized learn- ing. A comparison between Figs. ([a) and [[(e) reveals that FL experiences more significant model miscalibration than centralized learning. This disparity is likely due to data heterogeneity across distributed FL clients, causing each client's ss local data to have different impact on calibration per- formance. When applying auxiliary- based calibration methods, such as DCA (Liang et al.) 2020) and MDCA (Hebbalaguppe et al. (022b)- which add penalty term to the original classification loss- - to FL's local training (Fig. dd), better ECE is achieved compared to FL without calibration (Fig. (c)). However, observe that the model in Fig. dd) is still not well well-calibrated by neglecting global calibration needs in heterogeneous FL settings, resulting in overconfident predictions. Given the fact that train-time calibration methods directly modify confidence during training, potentially affecting the model's accuracy-whine which is also a critical factor- we thus pose the following question:
How can we unlock the potential of model calibration in FL. while not sacrificing accuracy?
Contributions. In this paper, we propose Non-Uniformm Calibration for Federated Learning (NUCFL), a versatile and generic model calibration framework for FL that integrates seamlessly with any existing FL algorithms, such as FedProx (Sahu et al., 2018), Scaffold (Karimireddy et al,, 2019). FedDyn (Acar et al. 202I). and FedNova (Wang et al. 2020b). Designed work with any train-time calibration auxiliary loss, NUCFL is a straightforward yet effective solution that dynam- ically adjusts calibration penalties based on the relationship between global and local distribution. Specifically, our intuition is that if each client's 's local model closely resembles the global model, it is likely that the local model and data represent the global calibration needs well. Therefore, our idea is to choose a large penalty term to improve model calibration for the clients that have a high similarity, while imposing a small penalty for other clients prioritize accuracy. This non-uniform calibration approach considering data heterogeneity across FL clients significantly minimizes calibration errors without accuracy degradation, as seen in Figure [e.). Importantly, the flexibility of NUCFL allows for continuous improvement as new FI. algorithms or calibration losses are developed. Our key contributions are summarized as follows:
Our work pioneers systematic study on model calibration in FL. We analyze whether existing calibration methods can be applied to FL algorithms and study the following ques- tion: How does model calibration impact the behavior of federated optimization methods?
2

---

Published as a conference paper at ICLR 2025
We propose NUCFL, non-uniform model calibration approach tailored to FL. Our NUCFL dynamically adjusts calibration penalties based on the nuanced relationship between each local model and the global model, effectively capturing data heterogeneity.
Extensive experiments demonstrate that NUCFL is highly flexible, seamlessly integrating with 5 different FL algorithms (e.g., FedProx, Scaffold, FedDyn) and various auxiliary- based train- time calibration methods to improve both calibration performance and accuracy.
2 RELATED WORK
Federated learning. FedAvg (McMahan et al. 2017) is the pioneering FL algorithm, with numer- ous adaptations proposed to miligate the accuracy drop experienced with non-lID data. Much of the research has focused on designing better global aggregation methods to enhance model convergence and performance. Some researchers (He et al., [200)] [Lin et al. (2200) replaced weight averaging with model ensemble and distillation, others (Yurochkin et al. 2019] Wang al., 2020a) matched local model weights before averaging, and another group (Ji et al. 2019) introduced an attention mechanism optimize global aggregation. Another significant area has concentrated on optimiz- ing local training protocols to optimize computation and data usage on clients' devices. Some re- searchers (Sahu et al. 2018; Acar et al., 202I) employed regularization toward the global model, while Karimireddy et al. (2019) utilized server statistics to correct local gradients. Additionally, various studies have addressed challenges specific FL, such as ensuring fairness across all partici- pants (Li et al. 2020; Chu et al. 2024b; 2022a b). enhancing communication efficiency to minimize bandwidth usage (Ko et al. [023 Chu et al. 2024a| Chang et al. 2024; Lan et al.) [023a by Yuan et al. 2024). and addressing privacy concerns inherent in distributed computations (Truong al., 2020 However, the main focus of existing FL research is on improving accuracy. The mismatch between confidence and actual accuracy in these works leads to overconfident FL models, making FL inapplicable in high-risk decision maaiing scenarios. Our work bridges this gap by investigating an overlooked aspect of federated learning: model calibration.
Model calibration. Model calibration techniques can be categorized into post-hoc calibration and train-time calibration methods. Post-hoc methods adjust the model using hold-out dataset after training. Common examples include Temperature Scaling (Guo et al., 2017). which adjusts logits by a scalar learned from a hold-out set before softmax, and Dirichlet Calibration (Kull al. (017), which modifies log- tr-assoonfome class probabilities with an extra network layer. However, in FL applications, the lack of an additional validation set due to privacy concerns often renders post-hoc calibration methods inapplicable. On the other hand, train-time calibration integrates calibration di- rectly within the training process. Müller et al. (2019) enhance calibration using label smoothing on soft targets, while Lin et al. (2017) use focal loss to minimize the KL divergence between predicted and target distributions. Recently, augmenting standard cross-entropy loss with additional penalty loss terms has become popular train-time calibration method (Liang et al. 2020 Hebbalaguppe ctt al. 2022b Kumar et al., (2088). This auxiliary loss penalizes the model when there is a reduction in cross-entropy loss without a corresponding increase in accuracy, often indicative of overfitting. However, as seen in Fig. 1 a direct application of these methods still faces limitations, as they are not able to meet the calibration needs considering data heterogeneity across FL clients. Our work fills this gap by taking non-uniform calibration approach considering the characteristics of FL.
Calibration and FL. Peng et al. (2024) proposed FedCal, a calibration scaler aggregated solely from local clients, and, to the best of our knowledge, is the only prior work explicitly addressing calibration error in FL. However, FedCal does not consider the interaction between global and local calibration needs, which can lead to a calibration bias towards local heterogeneity. Our calibration framework addresses this issue by considering the interaction between client and server, thereby better accommodating broader global calibration requirements. We empirically confirm these ad- vantages of our approach compared to the existing work in Section 5.2 that Zhang et al. 2022); Luo et al. (2021) also use the term "calibration" in the FL setup. However, we stress that the context of calibration in their work is different from the one in the model cali- bration literature that considers confidence of the prediction. Zhang et al. (2022) y Luo et al. (2021) focuses on calibrating the classifier to improve the accuracy itself, which can be categorized into conventional FL. works that do not consider the error between model confidence and accuracy.
also worth mentioning
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 General FL Framework
1: Input: global model local model wm for client m, local epochs E, and rounds T.
2: for each round 1. 2, 2,...? do
3: Server sends w (f-1) to all clients.
4: for each client
M do
5: Initialize local model w/ <- (t(t-1)
for each epoch e 1, 2, ...,
Each client performs local updates via: whie)) CLiietooPT(W Lm) 8: end for
9:
win (t.E) denotes the result after performing E epochs of local updates.
10: Client sends d( = w(t-1)
wmm
(1.E) to the server after local training.
11: end for
12: Server computes aggregate update 8
E MEM
Dm 8/
13: Server updates global model (t) <- ServerOP (t- ,8(1)
(((1))
14:
end for
3 PRELIMINARIES
3.1 FEDERATED LEARNING
In FL., the training data is gathered from set of clients M and each client possesses training set Dm = {I,, y4 ) Pmll, where r represents the input data and denotes the corresponding true label. We consider the following standard optimization formulation of federated training and seek to find
model parameters w that solve the problem:
(Dml [D]
min L(w)
Σ
L. (w),
MEM
where w represents the model parameter, D UmE Dm denotes the aggregated training set from all clients, and Lm measures the average loss of model on the training data of the m-th client. The objective is to find a model that fits all clients' data well on (weighted) average.
FL methods involve parallel local training at clients and global aggregation at a server over mul- tiple communication rounds to address the aforementioned problem. Most federated optimization methods fit within general framework outlined by Reddi et al. (2020). shown in Algorithm At round f, the server sends its previous global model (((11) to all clients as initialization. Each client m then performs L' epochs of local training using C1 ClientOPT depending on the FL algorithm, and subsequently produces a local model um Ft. Then each client communicates the difference sft) be-
(t,E), tween their learned local model and the server model, denoted o! ( (.), The server computes a weighted average of the client updates, denoted as 8((), and then updates the global model (1) using ServerOPT, which also depends on Fl. designs. For example, FedAvg employs standard stochastic gradient descent updates for C1 ienttPP and its ServerCFT is formulated as (t))) (t-+1) ((t)
3.2 MODEL CALIBRATION
In supervised multi-class classification, the input & E X' and label E == (1,.., K} are random variables following ground truth distribution (2, y) = ())lz)((z) Let s be the confidence score vector in gx with syy] = f(z) representing the confidence that the model f(-) predicts for class y given input æ. The predicted class ÿ is computed as: ý arg max, s(y]. with the corresponding confidence & = max, s(y]. A model is considered perfectly calibrated (Guo et al. 2017) if:
P(g y p) p, Vp [0.
That is say, we expect the confidence estimate the prediction reflect the true probability of the prediction being accurate. For example, given 100 predictions, each with confidence of 0.8, we expect that 80 should be correctly classified. One concept of miscalibration involves the difference between confidence and accuracy, which can be described as:
E y/ p) zl]-
(3)
4

---

Published as a conference paper at ICLR 2025
Note that "increasing confidence" refers to adjusting the model's predicted probabilities to higher values, but it does not necessarily imply "better model calibration." Excessively high confidence can lead to overconfident predictions, potentially misleading users over-rely the model even when its predictions are inaccurate. The primary goal of model calibration, whether via train-time or post- hoc methods, is minimize discrepancy prediction accuracy. This ensures that users can directly know the probability of correctness based on the model confidence, allowing the model be applied in various decision-makee scenarios.
ensure the model's confidence accurately reflects its
PROPOSED MODEL CALIBRATION FOR FEDERATED LEARNING
4.1 BRIDGING FEDERATED LEARNING AND MODEL CALIBRATION
Our approach aims to implement calibration each FL client's local training. Note that in conven- tional FL algorithms, the loss Lm of client m in (T) can be expressed as follows:
D
Lm (w!lee)
1
Σ
(t.e) el 3z ; Vi),
(4)
[Dmm
where Wm is the local model, the current global round, e the local iteration index, and ((.) the loss function (e.g., cross-entropy) applied to a data instance. One can also modify the loss in (4) i.e., by adding proximal term as in FedProx (Sahu et al.) 2018). Recent studies on calibration (Liang et al.) 2020 Hebbalaguppe et al. 2022b) suggest that incorporating an auxiliary loss, which captures the disparity between accuracy and confidence (as in equation (3), and penalizing the model accord- ingly, leads to superior performance compared to direct loss type adjustments (Müller et al., 2019 Brier 1950 Lin et al., 2017). These auxiliary loss methods maintain the original classification loss and seamlessly add a penalty, minimizing interference with FL performance, especially since some FL algorithms are specifically designed around their classification loss strategies. This makes the auxiliary loss methods easily applicable and effective across various FL settings.
Proposed framework. Therefore, for client m, we propose to integrate the calibration loss with the FL loss as follows:
DD
Doal (w (t.e)
(w(the) ((wltt)); J Mi) + Ame (w x,, (...v))
(5)
D
where (-ai()) is the auxiliary calibration loss function, and Bm is a scalar weight for each client m. In the centralized calibration case, this scalar weight is manually selected to refine the auxiliary calibration loss. However, in our design, we automatically determine 3m based on the relation- ship between the server and clients in a Fl. setup (without any hyperparameter tuning), taking into account the heterogeneity as detailed in Section 4.2 The auxiliary calibration loss loal can be de- rived from various calibration designs. For instance, Liang et al. (2020) directly adds the difference between confidence and accuracy (DCA) as an auxilarr loss, which can be represented as:
N
N
leal (lethSIsN, (AlhSIEN) ldea
1
(6)
N
N
where N denotes the total number of training samples, 8; represents the confidence score of the model's predicted label 0 for sample i, and c 1 if 8; equals the true label yı; otherwise, C, = O. On the other hand, Hebbalaguppe al. (022bb) provide a more detailed analysis of (6) and propose a Multi-class DCA (MDCA), which offers more granular examination by considering all classes instead of focusing solely on the predicted label. The proposed auxiliary loss can be written as:
K
Row ( (Cchhsisn.sshhee (sthsssN)
Emdea
1 K
cbl
1
Σ
s
(7)
where K is the number of classes and N the number of samples. c = 1 shows that label j is the ground truth for sample i, and c,] gives the confidence score for the j-th class of sample i.
Advantages and challenges. The differentiability of these auxiliary loss terms (6) and 77) fa- cilitates their integration with other application-specifi loss functions, allowing them enhance
5

---

Published as a conference paper at ICLR 2025
calibration without significantly impacting the primary classification loss, l. This minimizes any adverse effects on performance while correcting calibration errors. For instance, FL algorithms like FedProx (Sahu et al.) 2018) and FedNova (Wang et al., 2020b) incorporate custom loss functions within their local modeling function, C1 i entOPT (outlined in Algorithm[] D to address deviations from the global model due non-III data. Despite these enhancements, is critical to note that these advanced FL algorithms still fundamentally require a standard classification loss, which they fine-tune for specific objectives. When integrating calibration into the ClientOPT of FL algo- rithms, we will exclusively calibrate the standard classification loss, maintaining the integrity of other design elements, such as global aggregation, to ensure their original objectives are not com- promised. Choosing an appropriate 3.. tailored to each client in is the remaining challenge.
4.2 NON-UNIFORM CALIBRATION FOR FEDERATED LEARNING (NUCFL)
Motivation: Limitation of uniform penalty. In Equation (5), we promote calibration during feder- ated local training by targeting the confidence-accuracc gap in individual local models via auxiliary calibration loss. The main aim of FL is to train a global model capable of effectively handling the global data distribution. Therefore, we aim for calibration to account for the heterogeneity inherent in FL setups, marking departure from calibration approaches used in centralized learning. Direct application of auxiliary loss calibration methods without adapting to FL characteristics would result in uniform calibration weights, such that B1 B2 ... 8m. This leads to local models being calibrated solely to their respective datasets Dm, with uniformly small weights biasing calibration toward local heterogeneity, and uniformly large weights potentially neglecting accuracy improve- ments from classification. Such discrepancies can result in a global model that, while comprising well-calibrated local models, does not achieve optimal performance across the broader distribution. This fixed calibration penalty approach, therefore, risks neglecting the broader global calibration needs necessary for optimal performance across the entire global distribution, e.g., as in Fig. dd).
penalty design (NUCFL). Given that each client receives the global model from the previous round as their starting point for each new federated round, this offers an op- portunity to indirectly adjust local calibration to mirror global model characteristics. To leverage this, we propose Non- Uniform Çalibration for Federated Learning (NUCFL.), which provides dynamic calibration penalty to local models to better reflect global calibration needs. Drawing inspiration from the concept of similarity characterization in FL tech-
Non-uniform
high similarity
Servar
low similarity
pm 1
Client
Cliem
...
Ciient
Pm +
larger calibration penalty
NUCFL Cy = + fmm
smaller calibration penalty
Figure 2: Idea of proposed NUCFL.
niques (Sahu et al. 2018; Tan et al.] 2023 , we hypothesize that a local model closely resembling the global model is likely to represent global characteristics well, suggesting that the penalty appropriately reflects the calibration needs of the global model. Conversely, a dissimilarhhetrogeeeeus local model suggests focus local objectives (e.g.. to im- prove accuracy) at the expense of global alignment. By adjusting penalties at each local epoch based on the similarity between local and global models, NUCFL ensures that local training enhances both the calibration and accuracy of the global model. As illustrated in Fig. we specifically set
Bm = sim( g(t.e))
(8)
where ((t-1) is the accumulated gradient from the previous round t 1, representing the received global model, while <(t.e) = (t-1) - (t.e) calculates the gradient for client m at local epoch e during round f. The function a im(-) measures similarity between local and global models, gener- ating values between O and 1. Specifically, it can use general cosine similarity or more advanced measures like Centered Kernel Alignment (CKA) (Kornblith et al., 2019).
For instance, if the local model is similar to the global model, indicating alignment in their char- acteristics, a higher calibration penalty applied to the local model to address global calibration needs. Otherwise, if the similarity score is low, smaller penalty is applied, focusing accu- racy improvement instead. Our non-uniform penalty method tailored to FL, strategically improving model calibration and accuracy. Additionally, our NUCFL L requires extra hyperparameter tuning, as 3m is automatically determined by (8) based on the given similarity function.
Remark 1. NUCFL can integrate any auxiliary- based calibration loss and adapt any FL algorithm, independent of aggregation method or local loss function. To our knowledge, this is the early work to specifically address model calibration across various FL algorithms. Similar to most calibration

---

Published as a conference paper at ICLR 2025
research Lin et al. 2017 Müller et al. 2019 Brier 1950 Kumar et al. 2018 Mukhoti al. 2020 et 2021 Liang et al. 2020 Hebbalaguppe et 2022b a). our work is an empirical study that explores the practical impacts of our methods. This research area remains challenging, particularly in providing theoretical guarantees for calibration methods, even in centralized settings; we instead substantiate the effectiveness of our method through extensive experiments. We will demonstrate how our strategic approach effectively calibrates FL in (ection[552)
Remark (Compatibilityy with post-hoc calibration). Although our NUCFL is train-time cali- bration approach that is applied during FL training, the proposed technique complements post-hoc methods whenever holdout dataset available. For example, temperature scaling can be applied to the trained global FL model for further refinement, as empirically demonstrated in Section 5.2
5
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Dataset and model. We conduct experiments using four image classification datasets commonly
utilized in FI. research (Caldas et al. 20188) McMahan et al.] 2017, et al. 2019): MNIST (Le-)
Cun et al., 1998), FEMNIST (Cohen et al. (017)) CIFAR-IO (Krizhevsky) 2009), and CIFAR- 100 (Krizhevsky 2009). For the MNIST and FEMNIST datasets, we use CNN (LeCun & Bengio, 1998) with two 5x5 convolution layers, each followed by 2x2 max pooling, and fully connected lay-
ers with ReLU activation. We use AlexNet (Krizhevsky et al. 2012) with five convolutional layers and ResNet-34 (He et al. 2015) for the CIFAR-I0 and CIFAR- 100 datasets, respectively.
FL data distribution. In the III) setup, data samples from each class are distributed equally to M = 50 clients. To simulate non-lll) conditions across clients, we follow (Hsu et al.) 2019 Nguyen et al. 2023 Chen et al.] 2023) to partition the training set into M 50 clients using Dirichlet
distribution with a = 0.5. Results with different are reported in the Appendix B.2
Calibration baselines. We compare our method against models trained using Cross-Entroen (Un- cal.), representing training without calibration. We adapt centralized train-time calibration methods, such Focal Loss (FL) (Lin et al.) (2177). Label Smoothing (LS) (Müller et al. 2019), Brier Score (BS) (Brier 1950). MMCE (Kumar et al.] (2018), FLSD (Mukhoti et al. (2020). MrLS (Liu et al., 2021 DCA (Liagg et al. 2020), and MDCA (Hebhalaguppe et al. (002b)), to the FL setting based on 5) and (6) to evaluate their performance. We also compare our method with FedCal (Peng et al.)
2024). a state-of-the-an method designed for post-hoc calibration in FL. Detailed parameters for each calibration baseline are provided in Appendix A.I
FL algorithms and setups. We conduct model calibration on five well-known FL algorithms, in- cluding FedAvg (McMahan et al. (017), FedProx (Sahu et al. 2018), Scaffold (Karimiredyy et al.) 2019), FedDyn (Acar et al. (2021), ). and FedNova (Wang et al. 2020b). We rum each FL algorithm for 100 rounds, evaluating the final global model, with 5 epochs for each local training. We use the SGD optimizer with a learning rate of 10-3, weight decay of 10-1, and momentum of 0.9. For additional details on the training specifics of each algorithm, please see Appendix A.2
Auxiliary loss and similarity measures for NUCFL. We implement our framework using two auxiliary-based calibration losses, lden and Endea- We utilize three similarity measurements- cosine similarity (COS), linear centered kernel alignment (L-CKA), and RBF-CKA (Kornblith et al.] which output similarity score from 0 (not similar at all) (identical) for evaluating the similarity between the learned local model and the global model. We refer to configurations using our technique as "NUCFL (* + ))," where denotes auxiliary calibration losses lndaa and @ denotes similarity measurements COSIL-CKA/BBF
yielding six settings for our methods.
Evaluation metrics. To evaluate performance, we measure the accuracy (%) of the final global model on the test set and assess calibration using commonly used metrics such as Expected Cal- ibration Error (ECE%) and Static Calibration Error (SCE%). ECE (Naeini et al.) 2015) approx- imates calibration error (3) by dividing predictions into bins and calculating a weighted aver- age of ceruracy-confidenee discrepancies: ECE Σ/m B;
.N |A, C.). where N represents the total number of samples, with weighting based on the proportion of samples within each confi- dence bin/interval. Each i-th bin covers the interval (=1,+ within the confidence range, with B, indicating the number of samples in the i-th bin. A = [B EjEB, T(ū, = y3) calculates the accuracy within bin B,, and C
B. ES 8, gives the average predicted confidence
7

---

Published as a conference paper at ICLR 2025
Callbratio Methed
FedlAgg
FedPPoxx
Neatisid
Felliyn
FedNNon ECE
Acc
ECE
Asc
Ase
SCE
ECE.
SCE.1
Asc
SCEE
Lnnell
61.34
10:52
1,61
61.53
9:53
1.56
62.10
13:47
1.62
62.5 W.
12.55
1.54
63:11
11:45
5,30
Fooal Linett 7
60.59
12.38
3.24
62.07
11.45
3.79
61.29
13.28
4.38
00.21
13.22
3.81
61.15
13.21
4.71
1.5
Mins
59.35
13.61
6:38
62:23
14:37
6.34
11.59
4.05
15.19
5.58
63.05
16.01
5.89
10:15
361
11:39
60.00
61.03
11.37
12.00
3.95
61.28
58.71
11:51
1:52
59.24
10.66
00.45
13.61
5:15
64194
11.83
4.04
60.09
13:81
4.88
Mil5
Taares di 2i1211
59.52
9.42
3.55
60.59
11.37
3.84
61.38
11.99
4.00
62.90
14.95
5.50
62.93
13.66
4.00
D.A (Ling
63.25
7.09
3.24
01.33
5.74
3,20
52
9:23
1355
02.11
10.17
3.75
63:15
92"
5:3
NICTET rca T1xT
61.88
6:21
1.11
62. 16
5:15
1.35
62.
12
8.88
1.50
62.81
0.20
1.54
69:24
8.85
3.11
NOCFL (DCA)L CKA)
62.05
6.14
2
62.31
8.04
3,30
62.
8.41
3.45
62.94
9.14
352
63.27
8.52
3.43
NECEL, (2CA--BF--CAAA
61.59
6.19
3.11
61.80
6.12
1.35
8.50
3.45
60.64
9.21
3.55
63.17
8,01
MICA (Bihhalappge
61.05
61:00
K:21
3:37
0.04
10024
372
1010
ADCFLONDCAAL.CAA
62.17
7.88
3.25
9.19
3.53
63.14
NUCHL CMDC---DCC-RII-CK.
01.56
3.00
61.70
K.00
3.39
62.65
9.24
3.55
65.27
0.59
Table 1: Accuracy (%) calibration measures ECE (%). and SCE (%) of various federated optimization meth-
ods with different calibration methods under non-llD scenario on the CIFAR-IO6 dataset. Values in boldface represent the best calibration provided by our method for the auxiliary calibration method, and underlined val- ues indicate the best calibration across all methods. Averaged performances are reported; complete results with standard deviation are included in the Appendix.
Calleraiin Metheal
FedAag ECE
TelPPox ECE
Scalina ECEE
Felllyy ECE
FoalNoca BCE
SCE
Ace
SCE
Ace
SCE
Ace
SCE.
Auc
SCE
Escal
90.95
91,45
4.61
1:95
91.28
2.51
5.24
2.50
92.22
2:44
Rocal
al 300
90.63
1.56
90.10)
5:94
3.97
91.19
5:58
3.12
19
9.84
7.91
91:13
5.39
2.61
1.
91.00
3.75
1.62
90.59
4.50
1.50
91:53
5.18
3.54
09.08
5.19
3.69
92.04
4.95
2.46
91.48
5.19
2665
90.59
5:42
2,50
95.72
4.19
1.51
91.39
5.37
2,62
90.38
4.65
1.92
MMCS
Aols
90.22
4.83
2.30
90.12
4.01
1.79
91:44
5.00
2.55
02.00
4.93
2.40
91.35
5.08
2:40
HLSD
Mistro 21
90.00
4.93
2.55
90.39
4.95
2,04
92. KM
5.19
2.5%
91.17
5.01
2.54
91.21
5.25
2.63
90.52
4.27
91.49
4.70
1.94
92.87
5.99
2.50
02.00
3.44
2.75
00.39
4.61
(Tangg
91.95
3.61
1.35
92.42
93.22
4.20
92.25
NUCFL (DCA)L CKAA
91.63
92,10
3,00
1,30
92.19
4.22
1.65
92.33
NECEL, (IX (44R0F--KK)
01.36
3,69
1.40
91.00
4.79
1.18
91.34
4.15
1.63
02.91
4.54
1.75
00:41
4.11
1.68
XLCC (Seltalguipe
10220
91.64
3775
1.35
92.17
4.23
2725
92.95
4.6
1.30
4.71
1.33
92315
4.02
1.32
91.29
3.6
1.46
91.05
3.05
1.77
93.00
4.14
1.82
92.83
4.41
1.80
92.45
4.32
1.70
MUOFLOONICALL CKA)
91.97
3,28
1.28
92.20
3.88
1.51
92.77
4.20
1.80
93.11
4.31
1.75
91.99
4.28
1.67
NUCFL (MBCA-88F.CKA
91.35
1.40
92.21
4.00
1.77
93.04
93.04
4.40
1.50
92.39
Table 2:
Average performance of each algorithm under non-III scenario on the FEMNIST dataset. Complete
results with standard deviation are included in the Appendix.
for samples where By E B;- The recently proposed SCE (Nixon et al. 2019). extending ECE for multi-class settings, groups predictions into bins by class probability, computes calibration errors per bin, and averages these across all bins: SCE 1 Σ/_ Σソ= B.i K
N |As,j - Ca,gle where K represents the number of classes, and Br,, denotes the count of j-th class samples in the i-th bin.
A..j [h., Ekep., [[j yu) measures accuracy, while Ct, = [B., ExER., s% gives the average confidence for the j-th class in the i-th bin. We set the number of bins, = 20, for ECE and SCE, and evaluate the final global model. We performed all the experiments on four random seeds and reported the average performance along with the standard deviation.
5.2 EXPERIMENTAL RESULTS
Can train-time calibration apply to federated learning? We first compare train-time calibration in traditional centralized training and non-IID FL. Using the entire CIFAR-100 dataset, we train models for 100 epochs with various calibration methods, presenting the results in Table 6 of Ap- [B.d Most methods effectively maintain accuracy and reduce calibration errors in centralized settings. Yet, as shown in Table [ these methods often underperform in calibration and accuracy when applying to FL.. Notably, methods like DCA and MDCA, which include an auxiliary loss, show robustness in federated settings. They balance classification accuracy and calibration, preserv- ing confidence while penalizing miscalibration, making them stand out in FL environments.
The effects of confidence calibration on FL. Tables D and 2 show accuracy and calibration error for various calibration methods applied to FL algorithms under non-IID scenarios using C00 and FEMNIST datasets. Among these baselines, mexiliary-based methods like DCA and MDCA show superior performance, with comparable accuracy and lower error. Our method outperforms them, as evidenced by the superior performance of NUCFL (DCA//DCA @)) over DCA/MDCA, highlighting the importance of our non-uniform penalty, which is specifically tailored to address global calibration needs during local training. NUCFL's 's superior performance, utilizing various sim- ilarity measures and auxiliary losses, demonstrates its flexibility and effectiveness. Our consistent improvements across various FL algorithms also highlight its adaptability. See Appendix [B.2 for complete results with standard deviations, as well as other datasets and data distributions. Further discussion the trade-off between accuracy and reliability can be found in Appendix [B.6]
Mitigating over/under- The results in Tables I and [ show that our method improves SCE and ECE over the baselines, but they highlight whether they correct for over-confidence or
8

---

Published as a conference paper at ICLR 2025
Uncal.
DCA
MOCA
(8) NUCEL, (OCA-L-CKA)
NUCPI, (MD)CA-L-CRA)
Accuraay
ECE 10.52
Accuracy
ECE 7.69
Accuraay
ECE 7.71
Accuracy
ECE 6.14
Accurscy
ECE 6.39
Confidence
Confidence
Confidence
Confidence
Confidince
Figure 3: Reliability diagrams for non-IID FedAvg with different calibration methods using the CIFAR-100 dataset. The lower ECE and smaller gap (red region) show the effectiveness of our method.
Ge) Uneal
(b) DCA
(c) MDCA
(4) NUCFL (BCA-LLACC)
NUCFL (MAC
Oupput
Output
cutputa
OUSLD
Accuracy
9.69
Accurssy
B.45
Accurayy
8.44
Accuracy
7.06
Accurscy
Confidence
Confidence
Confidence
Confidence
Confidence
Figure 4: Reliability diagrams for non-IID FedAvg using the CIFAR- 10 dataset.
under-confidence. To clarify this, we present reliability diagrams (Degroot & Fienberg. 1983
Niculeseu-Mizil & Caruana 2005). which visually compare predicted probabilities to actual out- comes, plotting expected accuracy against confidence. If the model is perfectly calibrated (i.e. if P(j= - y" [)) = =(ū)), then the diagram should plot the identity function. Any deviation from a perfect diagonal represents miscalibration. In Fig. andd we present reliability histograms for non- IID FL using various calibration methods, where each FL method yields over-confident predictions. For example, if a model predicts the probability of a class i from (1...., K} as 0.7, it is expected to be correct 70 out of 100 cases; if the actual number of correct predictions is less than 70, the model is considered over-confident. Such over-confidence can be problematic in critical applications like medical diagnostics, potentially leading to under-treatment and serious health risks. Our method shows a smaller gap (red region) and lower ECE, effectively mitigating over-confidence. For details on how our method also tackles under-confidence, please refer to Appendix B.3
Effectiveness of our design. We explore model calibration on various FL settings, analyzing the ef- fects of different non-IID degrees a, numbers of clients M, and percentages of participating clients per round. Fig. 5 shows the accuracy and calibration results of FedAvg across various configurations on the CIFAR- 100 dataset, where we adjust one variable at a time from the default setting: M = 50, = 0.5, 100% participation. Across all settings, our method consistently shows robust calibration. Notably, NUCFL I. remains effective under challenging conditions, such with lower a values (higher non-IID disparity) or fewer participant per round. Additional results under diverse conditions, in- cluding an even smaller number of partial participants under various data distributions, are provided in Appendix B.4, highlighting the adaptability of our approach in complex FL environments.
Calibration strategy validation. Our calibration strategy assigns higher auxiliary penalty loss to local models that deviate more from the global
Non- FedAvg
Culibration Method
Acc
ECE
SCE 1
NUCFL (DCA-L-CKA)
62.05
6.14
3.07
NUCFL (MDCA=L -CC)
62.17
model, addressing global calibration needs dur- ing local training. To validate our design, we test opposite logic, applying higher penalties to clients more similar to the global model. We
Reversed NUCFL (DCA-L
60.36
13.89
4.98
Reversed NUCFI (MDCA+L-CKA)
61.77
16.11
6,87
Table 3: Destructive experiment for our method.
adjust the weight 3.. for each client m. to s sim(8(t-1),,ff 06hh-)))-11 We conducted this experiment to assess whether reversing our calibration logic affects FL. performance and to validate our original de- sign. In Table we applied reverse penalty strategy to optimal configurations of our method (i.c., NUCFL (DCA/MDCA + L-CKA)) on the CIFAR- 100 dataset. The results show that reversed logic degrades performance by biasing calibration towards local heterogeneity while neglecting global needs, even performing worse than uncalibrated FL (Table []. This confirms the soundness of our design, which strategically prioritizes global calibration needs during local training.
Comparison with CedCal (Peng et al. 2024). We now conduct comparisons with the only existing model calibration framework for FI., FedCal (Peng et al. 2024). FedCal bases its algorithm on

---

Published as a conference paper at ICLR 2025
Uncal
DCC
MDCA
NUCFI (DCA + (-CKA)
NUCHI (MDCA 1-CKA)
ECE ACC
65
02.5
00-
00.0
66
57.5
411
10
20-
10-
10-
110
0.5
0.1
50
200%
50%
25%
Data Distribution
Number Clients
Ratio Porticiponts
Figure 5: Comparison confidence calibration across different FL settings using the CIFAR- 100 dataset.
FedAvg: thus, we display the calibration results of our method alongside the bascline using FedAvg under non-IID conditions (a = 0.5) on CIFAR-I00 100 datatset in Table 4
For the scaler used in their work, we adhered to their de- sign, employing an MLP with layer sizes of 100-64-64 100. We see that our proposed NUCFL shows superior performance compared to FedCal. While using a scaler aggregated from local clients can reduce global ECE, it may neglects the interactions between global and local calibration needs. Our method considers these interactions, minimizing local biases and better meet- ing global calibration requirements. More comparisons using other datasets and data distribution are available in Appendix [B.] The superiority of our method both accuracy and calibration error shows NUCFL's effectiveness in complex FL environments.
Calibration Method
Aec T
Non-IID FedArg
BCE
SCB
Uncal FedCal Peng 2024
61.34
10.52
3.61
61.34
NUCHETDE SECECCCCC K2Y
62.05
NUCEL, NUCCL. (MDCA-RRF-CKAA
61.54
6.20
Table 4: Comparison with FedCal.
Compatibility with post-hoc calibration. Temper- ature scaling (TS) (Guo et al. 2017) is one of the most effective post-hoe calibration methods, which adjusts model's logits by scaling factor to cali- brate confidence scores without affecting accuracy. Like all post-hoe methods, TS requires hold-off dataset to tune the model and learn the calibration parameter (i.e., temperature). However, this need conflicts with the data privacy policies of FL, mak- ing TS impractical in such settings. For this analysis. we set aside these privacy concerns to compare with TS, assuming that data from each client is available at the server. Under these assumptions, we conduct non-III (a 0.5) FedAvg and sample five images from each client to create server dataset for post-hoc calibration. Table 5 compares train-time and post-hoc calibration methods using the CIFAR-100 dataset. Results show TS outperforming some train-time methods, like MDCA, while our method still provides superiority in both accuracy and calibration error. Recall from Remark 2 that our NUCFL can also be complemented by post- hoc methods when holdout dataset is available for refining the global model. We apply post-hoc calibration on the global model using TS after train-time calibration during local training. This com- bined approach shows improved performance over train-time calibration alone. Notably, integrating TS with NUCFL yields the best results, significantly reducing error.
Non-III FedAvg
Calibration Method
Acc 1
ECE.
SCE,1 J
Uncal. DCA MDCA
61.34
10.32
3.61
61.24
7.69
3.24
61.03
7.71
3.29
NUCFL (DCA+L-CKA)
62.05
6.14
3.07
NUCFL (MDCA-L.CKA)
62.17
6,25
3.11
TS (Gii al. [2017]
61.34
8.22
3.43
DCCFFTS
61.24
7,00
3.03
MDCA+TS
61.03
7.29
3.11
NUCFL CKA))+
62.05
5,91
2.97
NUCEL MIDCA+L-CKA)HTS
62.17
6.03
3.00
Table 5: Comparison and integration with post- hoc calibration. The results demonstrate our method's 's compatibility with post-hoc calibration.
6 CONCLUSION
We conducted one of the earliest systematic study on model calibration for FL and introduced a novel framework, NUCFL, designed to ensure a well-calibrated global model. NUCFL measures the similarity between client and server models and dynamically adjusts calibration penalties for each client, addressing both heterogeneity and global calibration needs effectively. Through exten- sive experiments, NUCFL demonstrated significant improvements over existing calibration methods, proving its adaptability with various types of auxiliary calibration penalties and compatibility across different FL algorithms. Our research seeks to deepen understanding of the underlying effects of model calibration on FL, enhancing the trustworthiness of standard FL systems. Future work could explore calibration for personalized FL settings with personalized layers, unsupervised FL settings with dynamically adjusted calibration weighting for domain adaptation, and the use of similarity- based weighting for other auxiliary losses to enhance FI..
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Science Foundation (NSF) under grants CNS- 2146171 and CPS-2313109, by the Office Naval Research (ONR) under grant -00014-22-1-2305, and by the Air Force Office of Scientific Research (AFOSR) under grant FA9550-24-1-008..
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. abs/2111 1..4664,,3, 2021. URLAttps:://
semant wwt.maaatt.ccccco......ooooooooa
Glenn W. Brier. Verification of expressed in terms of probabiliiy Monthl Monthly Weather Review, 78:1-3, 1950.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H. B. McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. ArXiv, abs/1812.01097, 2018. URL
https://apls sewaeeseeettticcchhhiioooo www.aaa.rrooooooooooooaoa 53:301554
Zhan-Lum Chang, Seyyedali Hosseinalipour, Mung Chiang, and Christopher G Brinton. Asyn- chronous multi-model dynamic federated learning over wireless networks: Theory, modeling, and optimization. IEEE Transactions on Cognitive Communications and Networking, 2024.
Hong- You Chen, Cheng-Har Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao. On the importance and applicability of pre-training for federated learning. In The Eleventh International Confer- ence on Learning Representations, 2023. URL hnttps://operrevit/ hwpp///ppeee....cooo/o net forum?id
ENWEvv-POx??
Yun-Wei Chu, Seyyedali Hosseinalipour, Elizabeth Tenorio, Laura Cruz, Kerrie Anna Douglas, Andrew S. Lan, and Christopher G. Brinton. Mitigating biases in student performance pre- diction via attention- based personalized federated learning. Proceedings of the 31st ACM In- lernational Conference Information & Knowledge Management, 2022a. URL https:
//api. swwanteeeet...e
//api.semanticss..... coamanttccccolar.coo oowpratdtescctllaooooooooooooooo
Yun-We Chu, Seyyedali Hosseinalipour, Elizabeth Tenorio, Laura Cruz, Kerrie Anna Douglas, An- drew S. Lan, and Christopher G. Brinton. Multi-layer personalized federated learning for mit-
igating biases in student predictive analytics. ArXiv, abs/2212.02985, 2022b. URL https:
laaiissmmaticschollla wttp:l.oro/ooooore wtt.://aaa....ooooooouoaooaa 254774422244548
Yun-Wei Chu, Dong-Jun Han, and Christopher G. Brinton. Only send what you need: Learning to communicate efficiently in federated multilingual machine translation. ArXiv, 0oos/2401.074555 2024a. URL /api..p... semant csshhlar org/ org/Corpus 2669296669
Yei Chu, Dong-Jun Han, Seyyedali Hosseinalipour, and Christopher G. Brinton. Rethinking the starting point: Enhancing performance and fairness of federated leamning via collaborative pre- training. ArXiv, 0ttp//222.0222555 2024b. URL https: /api. www.sseettitncoooooe org/
To:pusID:267404
Gregory Cohen, Saeed Afshar, Jonathan C. Tapson, and André van Schaik. Emnist: an exten- sion of mnist to handwritten letters. ArXiv, a02/1702. 2017. URL httpss://api.
semant www.tiiccchlar..o.oo Corp ID: 12507257.
Morris H. Degroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. The Statistician, 32:12-22, 22, 1983. httttsss::///pp wwwant www.tticcchoollrroorggoooooooo
109884250
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient
federated learning for heterogencous clients. ArXiv, abs/2010.01264, 2020. URL https://
api. semant insiaanticcchlarz waa.l.orrg//e wwtp:ooooooo..10222111.1
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modemn neural
networks. In International Conference on Machine Learning, 2017. URL //app. semant cschol ar. att.orrg/corrpussn226113331
11

---

Published as a conference paper at ICLR 2025
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Fed- erated learning of large cnns at the edge. arXiv: Learning, 2020. URL https://apa
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 770-778, 2015.
URL https: /ttp:/apppp/
R. Hehhalaguppe, Rishabh Patra, Tirtharaj Dash, Gautam M. Shroff, and Lovekesh Vig. Calibrating deep neural networks using explicit regularisation and dynamic data pruning. 2023 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pp. 1541-1549, 2022a. URL
https://api..enaat semant icschhlarcccccc..o..o.oo.oo..a.i
R. Hehbalaguppe, Jatin Prakash, Neelabh Madan, and Chetan Arora. stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16060-16069, 2022b. URL https:
//api. semant icscholaar org/ CorpusID: 24:
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. ArXiv, abs/1610.02136, 2016. URL. https://api:
semant www.::rrrrrrrrgaaooooooo wttp:oooops.::00000.353.3
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural net- work. ArXiv, 503.0533,,,,,, 2015. URL
orgg
CorpusID: 7200344 7200347
Tzu-Ming Harry Hsu, Qi, and Matthew Brown. Measuring the effects of non-identical data dis- tribution for federated visual classification. ArXiv, abs/1909.065) 06335, 2019. URL https:
//api. s/apitsemantioi wesemmatticccooar...ooo. Sttpg//CrrpuIID:22522000
Ming Hu, Peiheng Zhou, Zhihao Yue, Zhiwei Ling, Yihao Huang, Yang Liu, and Mingsong Chen. Federoss: Towards accurate federated learning via multi-mode gross-agrregation. 2024 IEEE 40th International Conference on Data Engineering (ICDE). pp. 2137-2150, 2022. URL
attps://ane www.nticscooola........ wttp:aroooroopp..1112217777777
Ming Hu, Zhihao Yue, Zhiwei Ling, Yihao Huang, Cheng Chen, Xian Wei, Yang Liu, and Mingsong Chen. Is aggregation the only choice? federated learning via layer-wise model recombination. In
Knowledge Discovery and Data Mining, 2023. URL https: : api semant iichhtccrc.........
Dtc//ooppulD:2251555
Shaoxiong Ji, Shirui Pan. Guodong Long, Xue Li, Jing Jiang, and Zi Huang. Learning private neural language modeling with attentive aggregation. The International Joint Conference on Neural Networks, pp. 1-8, 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated leam- ing. In International Conference on Machine Learning, 2019. URL https::/api.
Haneul Ko, Jaewook Lee, Sang Kun Seo, Sangheon Pack, and Victor C. M. Leung. Joint client selection and bandwidth allocation algorithm for federated learning. IEEE Transactions on Mobile Computing. 22:3380- 2023. URL https://////.....tt. panps://app... swmanticscholar.cooomo
Corpus 15:
Jakub Konecný, H. B. McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. ArXiv, abs/1610.05492, 2016.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neu-
ral network representattinss revisited. ArXiv, abs/1905.00411 000400 2019. URL httpss:/appi.
semanticscho wttp:/chh.......ooooooooos..ssas
10:114403299
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https
7/api semant i cacholas a wtw..rr//ooopsoD.ii.....
12

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep volutional neural networks. Communications of the ACM, 60:84 - 90, 2012. URL https:
Meelis Kull, Telmo de Menezes e Silva Filho, and Peter A. Flach. Beta calibration: a well- founded and easily implemented improvement on logistic calibration for binary classifiers. In International Conference on Artificial Intelligence and Statistics, 2017. URL https://app)
www.ant.sccola.....m wwtpaaaarrrrrrrooaureeeee 217264968
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kemel mean embeddings. In International Conference on Machine Learning, 2018. URL
org/ ottg/orruuueeeo :9:314007
Guangchen Lan, Xiao-Yame Yang Liu, Yijing Zhang, and Xiaodong Wang. Communication-effciieenn fed- erated learning for resource-constrained edge devices. IEEE Transactions on Machine Learning in Communications and Networking. 1:210-224, 2023a.
Guangehen Lan, Han Wang, James Anderson, Christopher Brinton, and Vaneet Aggarwal. Improved communication efficiency in federated natural policy gradient via ADMM-based gradient updates. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), volume 36, pp. 59877-99888,, 2023b.
Yann LeCun and Yoshua Bengio. Convolutional networks for images, speech, and time series. 1998. URL httph::t////t www.nt.ccccooar..ooooo www.:oll...rr//ooooooooooo http://aaa...r//ooooooooooos 691 6627
semanti wescccccllaaso......o
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning ap-
plied to document recognition. Proc. IEEE, 86:2278-2324 1998. URL httpps///ppi
semanticsen ietiannisccholar....om www.oollarrrrr//oooooooooooeoo /ttpg//rruuuseee ID: I 14522661
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated
learning. In International Conference Learning Representations, 2020. URL https:// openreview net htt.://oooommmneyyyeee.....
Gongbo Liang, Yu Zhang, Xiaoqin Wang. and Nathan Jacobs. Improved trainable calibra-
tion method for neural networks. ArXiv, abs/2009.0400 .44077, 2020. URL https://api,
semanticscholad http:arooo/ccrrS..2225.5555.
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. ArXiv, abs/2006.07242, 2020. URL hhttps://api.
semanticscholi wttp:ooooos:::21155333003
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2017 IEEE International Conference Computer Vision (ICCV), pp. 2999- 3007, 2017. URL htt.:tss/pp....mmaiiiccccooooooo
www.eticccooolaae........c.c. www.:o/aaa..rooooooouo..... 472529988
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and José Dolz. The devil is in the mar- gin: Margin-based label smoothing for network calibration. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 80-88, 2021. URL https://api.
semanticschole. org wrtpoooppsID::2000222 :4472244444444444
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. ArXiv,
2021. URL
hittps://apl semant icscho 1 ar.. org/ Cotp/s ID: 2353::333311 77224
H. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communicaioon-eficieeet learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. ArXiv, abs/ 0000000146.... 2019. URL https: : a/ppi. www.nticscool.eaaee..e.eccccc
5955511311
13

---

Published as a conference paper at ICLR 2025
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet Kumar Dokania. Calibrating deep neural networks using focal loss. ArXiv,
abs/2002.09437,
2020.
URL httpss//app wtt.apptsw.att..too...o...o.../e Dorpppuuaa
211 652346
Rafael Müller, Simon Korbllihh, and Geoffrey E. Hinton. When does label smoothing help? ArXiv, abs/1906.02629, 2019. URL https: hatp::://ap..eeeee semanttcccccl
Corpus ID: 1778029888
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial In- felligence. AAAI Conference Artificial Intelligence, 2015:2901-2907, 2015. URL https:
//api. cet.aaaticcccolla.ooo//rrrrrrree 62928077
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? on the impact of pre-training and initialization in federated learning. In The Eleventh International Conference Learning Reprexentations, 2023. URLhttps: httphhtts////peeeeeee http://///////eeeee///oomee//ee. wwt. forum?
iddMpa3tRJFEB
Alexandru Niculescu- Mizil and Rich Caruana. Predicting good probabilities with supervised leam- ing. Proceedings of the 22nd international conference on Machine learning, 2005. URL
https://api .semant icscho ar. www//org/ooaao :t
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mea- suring calibration deep learning. ArXiv, abs/ 1904,01685, 2019. URL httpps://apl
commantisccholar... www.aarrooooooooooaooaa. 1022046660
Rohit Parasnis, Seyyedali Hosseinalipourr, Yun-Wei Chu, Christopher G. Brinton, and Mung Chi- ang. Connectivity-awarr cemi-decentralized federated learning over time-varying d2d networks. Proceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Founda- tions, and Protocol Design for Mobile Networks and Mobile Computing, 2023. URL https:
7/api. www.stttttnnnttoeeeeeeee htt.aaar.........o.S..555555555.
Hongyi Peng, Han Yu, Xiaoli Tang, and Xiaoxiao Li. Fedcal: Achieving local and global calibration in federated learning via aggregated parameterized scaler. ArXiv, 140522055.555555 2024. URL
https://apii www.nt wwtpsscchaaaa...oeeeeeeoeeeee
Zhe Qu, Xingyu Li, Rui Duan, Yaojiang Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, 2022. URL
https://api www.:eseemeattntiieieeiie
ar. www.arr.rroooooae.eeeeeee httpg//oorpsst::2339300
Sashank J. Reddi, Zachary B. Charles, Manzil Zaheer, Zachary Garrett. Keith Rush, Jakub Konecný, Sanjiv Kumar, and H. B. McMahan. Adaptive federated optimization. ArXiv, 0tt/22033.0095 2020. Uttpstsss: //ap:
Sashank J. Reddi, Zachary B. Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konecný, Sanjiv Kumar, and H. B. McMahan. Adaptive federated optimization. The Ninth International Conference on Learning Representations,
.002990 2021.
Anit Kumar Sahu, Tian Li, Maziar Sanjabi, Manzil Zaheer, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. 2018. URL https://api.
semanticscholar, wtw.ecooo.aa..ogo/oooppsssssssss cttp:/oorrpss111:9911.6666
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication-effiieent federated learning from non-i.idd data.
IEEE Transactions on URL hhttt::///// //api.
Newral Networks and Learning Systems, 31:3400-3413, 2019.
semant www.:attccccccoolloooooooooooee
Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. ArXiv, abs/2302. 10429, 2023.
(tL.httts: /tatps://api.... petapp.semaaate semant isscool ort. ar.po///errpsss1122555553333
14

---

Published as a conference paper at ICLR 2025
Jiahao Tan, Yipeng Zhou, Gang Liu, Jessie Hui Wang. and Shui Yu. pfedsim: Similarity- aware model aggregation towards personalized federated learning. ArXiv, abs/2305.15706, 2023. URL
https://api. www.s:ssscaannnniiiii..mem wwtp/oorg.useeeo. ID: 258888023
Nguyen Binh Truong, Kai Sun, Siyao Wang. Florian Guitton, and Yike Guo. Privacy preservation in federated learning: An insightful survey from the gdpr perspective. Comput. Secur, 110:102400
2020. /ttps:://appp www.ppp...mmaaccccc.ccc.a.aaaaa. o/ wttp::/o/ooooo1111::555555550330
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni.
Federated learning with matched averaging. ArXiv, abs/2002. 06440, 2020a. URL https://
api. rremanticsche aar. crrp.rroo/orrrut:111111333333
Jianyu Wang. Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. ArXiv, abs/2007.07445 2020b. :
URL https://pppp
Liangqi Yuan, Dong-Jun Han, Su Wang, Devesh Upadhyay, and Christopher G. Brinton. Communication--ffice multimodal federated learning: Joint modality and client selec-
tion. ArXiv, 0bs/2401111688. 2024. URL
CorpusID:267320666
Mikhail Yurochkin, Mayank Agarwal, Soumya Shubhra Ghosh, Kristjan H. Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. ArXiv, abs/1905.12022 2019. BRR/attpss//apl.eenniicccoo
org/Corpus 68170092
J Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. ArXiv, 0000/209990089, 2022. URL
/stpss//aap.. semant icscholar orr./ooopppss1000233330033
15

---

Published as a conference paper at ICLR 2025
A DETAILED SETTINGS FOR CALIBRATION BASELINES AND FEDERATED ALGORITTMS
A.1 CALIBRATION BASELINES
We offer an overview of each compared calibration method, including the hyperparameter settings utilized during training. Focal loss (Lin et al. 2017) defined as FL - E pi.y.) (og(Pt..), where Pt.y, is the predicted confdence score for sample i and is a happhh-paaaaee parameter. We trained models with 7 E {1, Label Smoothing (I.S) (Müller et al. 2019) defined as L. -Ein شك Pij log(Pt.s). where Pi,j is the predicted confidence score for class j of sample i, and qi is the soft target vector for sample i, such that q., = K-I if * else 9r = (1-a). a is hyperparameter set at 0.1, following established literature settings (Hebbalaguppe et al. 2022b). For the Brier Score (BS) (Brier 1950), we train using loss defined as the squared difference between the predicted probability vector and the one-hot target vector. For Maximum Mean Calibration Error (MMCE) (Kumar et al., 2018), we use the weighted MMCE, with À selected from {2, 4). as regularization term along with cross-entropy. For FLSD (Mukhoti et al.) 2020), we use 7 3. reported as the best configuration in their paper. For Margin-based Label Smoothing (MbLS) (Liu et al. 202T). we set the margin at 10 and l = 0.1, configurations reported as optimal in their paper. For DCA (Liang et al. 2020), we train using a loss composed of cross-entropy plus B Edeas where 3 is hyperparameter selected from {1, 5, 10) demonstrated in their paper. Similarly, for MDCA (Hebbalaguppe et al. 2022b), we apply the cross-entropy loss enhanced by 3 Emdea. with also chosen from {1, 5, 10) well. For all baselines, we report results that show the lowest ECE among their hyperparameter settings in our paper.
A.2 FEDERATED ALGORITHMS
For all FL algorithms, we distributed the training data across M = 50 clients and performed 100 rounds of training. Each client conducted local training epochs per round. The batch size was set at 64, and we explored learning rates within the range [1e-2, 5e-3, le-3, 5e-4]. For FedAvg (McMahan jct al. 2017). clients perform standard stochastic gradient descent (SGD) updates as C1 Client and upload their gradients to the central server. On the server side, the global model is updated us- ing SGD as ServerOPT with the received gradients from the clients. FedProx (Sahu et al. 2018) introduces proximal term during local client training as their ClientOPT, to address data het- erogeneity issues by preventing significant drifts in client model updates. It employs a weighted average for aggregation and updates the global model using SGD, similar to ServerOPT in Fe- dAvg. We set the proximal term coefficient parameter = 1 for the proximal term coefficient in FedProx. Scaffold (Karimireddy et al.) 2019) employs client- variance reduction as its Client and corrects model drifts in local updates by introducing control variates. For ServerOP1 T, Scaf- fold incorporates the clients' gradients and control variates for aggregation, and updates the global model using SGD. FedDyn (Acar et al. (021) dynamically adjusts the local objectives by incorpo- rating two regularizers as ClientOPT to ensure that the local optimum aligns asymptotically with the stationary points of the global objective. We set the parameter = 0.1 as commonly used in the literature, and the local model is updated using SGD. Additionally, they introduce their server update as Server rere through the use of server state vectors. FedNova (Wang et al.) (020b) uses normalized stochastic gradients as ClientOPT to compensate for data imbalances across clients, and then introduces weighted rescaling process their ServerOPT. For applying NUCFL FL algorthms, we provide a detail framework of our NUCFL in Algorithm 2 We run all experiments on a 3-GPU cluster of Tesla V100 GPUs, with each GPU having 32GB of memory.
signitie
B ADDITIONAL EXPERIMENTS AND ANALYSES
B.I CALIBRATION ON CENTRALIZED LEARNING
We present train-time time calibration methods for traditional centralized training in Table 6 We train ResNet-34 using the SGD optimizer on the entire CIFAR-100 training dataset for 100 epochs with various tame-time calibration methods. We found that most of these methods effectively calibrate in centralized settings, maintaining accuracy and exhibiting lower calibration errors.
16

---

Published as a conference paper at ICLR 2025
Calirratin Method
MNIST
FEMNIST (CNN)
CIPAR-IOAAleNNEU
CIFAR- (ReSSNE)
Acc
ECE L
SCE
Acc
ECE
SCE
Acc
ECE
SCE
Acc 1
BCE L
SCE +
Lincal.
0.74
3.07
5.53
2.55
68
5.80
3,00
Rocal Lin al 2017
97.09
0.70
0.25
94.33
2.98
1.55
84.17
5.01
2.43
68.42
5.66
2.88
LS
Muuse et
2019
97.22
0.61
0.25
94.51
2.89
1.53
$4.69
5.27
2.46
68.19
5.59
2.85
RS Hneer (950
96.99
0.70
0.34
95.02
3.00
84.23
5,44
2.53
68.59
5.60
2,87
MMCE
4.87
2.35
5,44
2.55
FL SD
Muscaa ei mi 23120
97.11
0.70
0.24
94.11
2.95
1:34
84.11
2.88
1.54
68.44
3.09
2.88
MBLS
2021
97.55
0.54
0.20
94.30
2.66
1.41
84,29
3,66
2.04
68.71
4.14
1.96
(Liang et
3.71
2.05
4.29
MDCA [echounguppt ot al. 20226
94.22
0.55
0.20
94.37
2.70
1.43
84.22
3.82
2,08
68.44
4.19
1,98
Table 6: Performance of centralized training with various calibration methods across different datasets. Train- time calibration methods demonstrate improvements in calibration error compared to the uncalibrated (Uncal.) model.
NUCFL, (DCA+L-CKA)
(MDEA-L-CKN)
Curputa Cap
Dupuca
Outpus Gap
Oopera
Oirqura
Gap
Acouracy
ICF 1.89
Accuracy
ECE
1.09
Acouracy
ECE 1.33%
Acouracy
ECE 0.78
Acco Accuracy
ECE 0.82
Confidence
Confidence
Confidence
Confidence
Confidlece
Figure 6: Reliability diagrams for non-IID FedAvg (a = 0.5) with different calibration methods using the MNIST dataset. The lower ECE and smaller gap show the effectiveness of our method.
B.2 ADDITIONAL RESULTS OF CALIBRATION ON FL
This section explores the performance of each calibration method on different datasets and data distributions for various FL algorithms. To ensure consistency in our experiments, most of the configuration remains unchanged. Specifically, we set the number of clients (M) to 50 and the total number of federated rounds (T) 100, with each client performing E epochs of local training. Additionally, the hyperparameter search space and the details for each FL. algorithm are consistent, as detailed in Appendix [ Tables 7
8 9 and didispla the performance and calibration metrics of various FL algorithms on the MNIST T dataset across IID and non-IID conditions, with values of 0.5, 0.1, and 0.05, respectively. Tables II FEMNIST dataset across IID and non-Ill) conditions. Similarly, Tables 15 16 17, and presentt the performance and calibration metrics of different FL algorithms on the CIFAR- dataset under IID and IID conditions, with corresponding values of 0.5, 0. and 0.05, respectively. Finally, Tables 19 20 21 and 22 show the performance using CIFAR- 100 dataset across IID and non-III conditions. Table 26 shows the calibration results with different local epoch numbers for non-III (a = 0.5) FedAvg using CIFAR- 100 dataset. Across these experimental results, which consider various data distribution setups and different datasets, our generic framework NUCFL consistently demonstrates superiority over the baseline by achieving lower calibration error. The performance improvements of NUCFL with different configurations, such as various similarity measurements and different auxiliary calibration losses, highlight the effectiveness and flexibility of our method.
13 and 14 show the performance using 12
In addition to the widely used FL algorithms introduced in Section 5 we conducted additional ex- periments using FedSpeed (Sun et al. 2023), FedSAM (Qu et al. 2022). FedMR (Hu et al.) 2023), and FedCross (Hu al., 2022) which are considered as the SOTA FL optimization methods. We conducted our experiments using the CIFAR-100 100 dataset under a non-IID (a = 0.5) data distribu- tion, following a setup similar that in Table For the hyperparameter of FedSpeed, we selected A = 0.1 as the default value provided by the authors, and we selected a 0.99 for FedCross. In Table 25 we see that in the absence of any calibration, these advanced FL algorithms do improve accuracy compared to FedAvg (ACC: 61.34: ECE: 10.52). However, they also introduce higher calibration errors. By applying NUCFL to these advanced FL algorithms, we find that our approach effectively reduces calibration eиror and even outperforms the SOTA calibration method in terms of calibration results. These results also demonstrate the adaptability of our method, showing that can be applied to any FL. algorithm to improve calibration while maintaining accuracy.
17

---

Published as a conference paper at ICLR 2025
Colibraiin Methoel
Fodlrg
McMahan oL.
FediProx
Saln eE 2013
Scatio
Karmmnedyy
2019
Arc
SCE
Acc
RCF
SCE.
Acc
BCP
0.15
11:5
Focal
(Lin.et 2011
2.71
0.18
0.77
1.30
2.85 0.95
0.44
1.
Minter 2019
10 0.39
8.66
1.88
3.31 8 0.30
95
0.95
7.49 0.02
2.95
8:00 0.34
97.00
100
6.88 1.61
2.67
0.14
05.29
0.58
2.24
0.27
0.67
: 0.10
95.
0.33
3.71
+ 0.10
0.76
8: 0.30
95.43
0.6
2
1/4.::0.444 0.43
0.51
0.12
MMC
7016
97.61
0.62
4.88
0.20
1.14 8 0.11
97.23
0.53
5.00 6 0.35
1.15
3: 0.11
96.77
1.22
12.19 0:11
3.79
0.10
11.0
Deganne 2431
95.25 0:33
K.04
0.17
3.30
8 0.06
05.44
0.53
M.02
8 0.25
3.92
8: 0.11
95.3
0:24
10,00
8 0.25
2.08
0.14
Mhl
Lanree Hir sitrr
50
1.00
5:01
1:30
1.10
9 0.25
95.51
0.66
4.01 + 0.44
1.10
96.31
0.85
3,05 9 0.47
0.77
0:11
ICA
12501 of (3737
96.55 8 0:45
0.53
B06
0.20
7 0.06
97.00
0.50
0.AK 0.10
11:23
7 0177
97.21
1 1.02
0.101:6.11 6.11
0.2 0:7
NUCTE (Derorcos)
97.10 1 0.52
1L69
0.2
0.18
2: 0.05
96.94
0.77
0.59 2: 0.09
1.19
2: 0.06
92.00
2: 1.22
0 0.10
0.23 0.07
NUCIL (DCALL CKA)
96,90 0:41
11:4 0.21
0.16
2 0.10
97.07
0.52
0.45 2: 0.07
0.15
2: 0.11
96.84
1002
0.50
2: 0.00
0.29 0.06
96.01 0:052
11:11
0077
0.37
2 0.05
96.1
0.77
0.A3 1: 0:20
0.16
- 0.13
96.91
1.22
0,53
0.00
0.21 0.06
MDCA
[iehhsiapppe et al. 20022
90.57 0:441
[161
019
0.24
8 0.07
97.04
0.53
0.8 0.135
11:00-1
00X
97004
1002
0.92 0.1.4
9,31 0.8
07.11 0.64
0.10
0.23
0.56 6:10
I.19
96.97
1.00
0.61 0.83
0.2 0.20
NUCEL (MDCAAL.CK))
0.85
0.27
0.25
97.11
0.55 0.00
0.20
NUCEL (MECA+RBF CKA)
0.11
0.57
0:83
0.25
Fellyyn
Near 5021
Mone (1250
SCE
97.23
1:22
1:40 2: 0,50
0335
2:0.1
96.22 3: 121
0.50 00.39
0.23
Focal Linesrl
2017
97.61
0.54
4.37 1: 0.55
1.09 0.10
95:25
4: 0.92
3.42 2: 0.94
0.80
0.09
91
0.50
+:
0.20
10:43:
0.8
92.63
1.04
0.88 0.09
96
1.00
0.17
0.07
2015
0.79
96
0115
0:311
0.72
0.54
97
0.08
NUCFL (DCAL CKA)
57.53
1.22
0.17 0.09
97.13 1: 1.04
0.05
NUAIL, (XX AA-AIR
96.75
9.54
0.17
0.09
97.6 0.9
005
MUCA [eehalpppp stal
97.41
1.21
0.65
075 0.08
9700
1.04
0.0
96 91
0.25
0.51
1.00
0.25
0.00
96
90
1:11
0.2 0.22
0.26 0.10
NUCFL /MDCM CKA)
97.09
0.25
0.50
3: 0.33
0.25 0.06
97.14
4: 1.00
0.54 0.38
0.24 0.09
NUCTL (MDCA-REE CKA)
97.56
0.6
0.55
4: 0.09
0.20
0.06
97.55
4 0.52
0.60
0.23
0.25 0.10
Table 7: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimiza- tion methods with different calibration methods under IID scenario on the MNIST dataset.
FediProx
Seathold
Kanminataly
Methaal
Linall
1,80
1115
01.3
M4.55
11.3
U1//
1002
11:44
1:111:00000
Lf 2017
6.66 0.11
1:25:1 0.10
0.73
1.10
1.20
0.95
91.51-
1:22
7.00 ::
1:52
Miior 21- mar
0:30
8:00 1.38
0.95
1.02
4.50
0.58
0.27
0:
0.33
92.77
0.1
1.58 0.12
2.51
0:11
0.53
92.41
Mhls
1.30
0.66
0:25
95.55
0.5
2411
1.09 0.10
0.50
0.12
0.59
0.0
0.07
NUCT
0.05
0.45
95.70
0.55
0.00
NUCFL (DCALL
0.35
004
0:45
0.55
0.1 0.00
NECFL (DC (DCA44RF-KAA)
96.07 6 0.37
IL74 + 004
0.30 + 0.00
95.67
0.45
1,6 0.04
0.55 + 003
95.85
0.55
3,05 + 0.06
0.57 + 0.0
MDCA
RiStsipeeee di, (002h
0532 0:41
1.3
011
0.55
= 0.06
94.90
0.53
135 0
1162
3 000
94
102
1.25
5 015
1Ix 2 0:00
NECFE DRODL
35.80
2 0.40
0.8 0.5
0.42
5 0.03
98.18
0.50
1.99 2: 0.04
0.57
2 0.04
95.17
0.00
3.01
= 0.06
0.87 2 0.07
NUCFL (IDCAAL-CKA)
35.95
2 0.47
0.2
2: 0.09
0.43
2 0.02
95.61
0.50
1,4 2: 0.05
0.55
2 0.03
96
0.54
274
= 0.05
0 0.08
NUCHI, (MIXAA-RIP-CKA)
95.96
8 0:41
085
B05
0.42
8 0.01
94.55
0.58
1.67 8 0.100
0.5
9 0.10
98.08 0.75
2,83
8 0.06
0.68 0.06
Callbratoo Mothoe
Filllyn
Fralooa
Mang at pooth
Auc
ACE
Apc
Pocal II
2015
0.85
0.10
119
0.94
1.10 1 0.09
ES
0.90
5.65 1.02
2.36 0.20
96
0.99
1.31
1.94 0.111
bS Brerr 1950
93
1.04
5.81
0.44
2.93
= 0.09
92
1.00
3.55
0.17
1.96 1:
MMCL
Rumpr 2D 2015
M5.48
0.79
2.0 0.26
0.74 = 1.07
96.30
0.82
6.75
0.41
2.57 0.155
FLSD
0.39
2.49
0.31
0.74 = 1111
91.11
1:31
1.57 0.52
2.45 1 0.15
Mal' Jalrer di pilee
95.71
0.92
3.05 0.33
1.06
= 0.10
94.24
0.8
5.67
0 0.42
2:82 2.0.7
DCA Lirii cr
96.77
0.70
144 0.09
0.47
= 0.03
95.13
0.50
0.94 0.08
0.48 1: 0.05
NUCF (UCC110))
96.93
0.65
124 0.07
0.43
= EUB4
96.01
0.53
0.76 0.05
0.45
1 0.03
NUCEL (DCAsL CKA)
87.07
1 0.65
1.23 0.06
0.43
= 0UB2
96,05
0.55
0.37
2: 0.05
0.44 1 0.03
NLCFI, IDC (DCA+BBF-CKA))
97.08
0.65
0.05
0.47
0.55
MDCA [ettalapugee et a. (N022
V100 1.21
1
047
= 100
95
1:04
1. 0.18
0 0UT
NO INDDGGII
0.70
0.08
NUCFL (CDA
0.77
1.30
0.22
Table 8: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimiza- tion methods with different calibration methods under non-IlI (a = 0.5) scenario on the MNIST dataset. Underlined values indicate the best calibration across all methods.
(a) Uscal Duppits
(b) DCA
60 MBCA
(60 NUCFL (DCA & L-CKA)
(d) NUCEL (MICAAL CKA
Cirputs
Carpunn
Cay
Accuncc
5.58
Accurscyy
Accuracy
Confidence
Confidince
Confidence
Confidence
Confidence
Figure 7: Reliability diagrams for non-III FedAvg (a 0.5) using the FEMNIST dataset.
18

---

Published as a conference paper at ICLR 2025
Callediin
FrdAkng
Mc
Felliox
Kato
Scalian
Karmmniiyy
Methedl
Avc
Acc
Asc
Froal [Linzt 2013
0.68
7:00 0.23
2.58
91
0.55
1:55
1.99
0.55
89.24
1:10
1.91
0.55
1.5
90.51
0:38
4.28 2.10
2.53
0.48
90.25
0.55
5.48
+: 1.45
1.62 0.38
90.84
1.25
9.36
8 3.35
4.84
0:50
b5 Bre (1550
90.08
0.35
5.06
2 0.32
2.49
0.17
59.50
0.48
5.58
4: 0.25
1,58 0.75
10.06
1.05
9.45
9: 0.55
3.00
0.20
MMCL
Kimmretaie 2018
90.22
0:35
10.20
6 0.35
5.15
3:10
0.22
50.95
6:30
5.25
0.50
3.00 0.17
89.57
1.15
7.49
2
0.20
ILSD Nylbite 20 10:0
90.57
0.35
5.26
2 0.24
2.01
3: 0.16
53.05
0:75
0,00
0.40
1 0.30
10.37
1.00
7.65
2 0:45
2,00
2 0:24
Mhts [nirn pile)
93.00
1.20
5.22
5: 1.60
1.58
2:1
11
0:50
9.60
2 0.70
4,86
0.30
89.82
0:75
9.29 2.060
4,90 0.22
DCA Liint (illmaaar p0r0
92.99 1 0.35
2.06
0.18
0.81
2.1 0.2.2
91.95
0.60
1.62
2 0.45
1,06 1: 0.33
02.54
2 0.80
4.49 0.7.0
1.52 0.40
NUCT (DCAtos
93.00 1 0:40
1,45 5 0.20
0.70
1 0.12
92.07
0.48
2.11
2: 0.50
140 0.18
02.58
0:70
4.43 1.0.6
1.52 0.30
NUCFL (DCA/L CKA)
92.41 1
1.32 2
0.66 0.14
92 0:35
233
2 0.25
1,06 0.30
02,44
2 0.85
4.34 1.35
1.48 0.28
NECEL IDC (CCBAFFACCC
0.51
1
0.71 +
0.65
1,06
0.15
0.0 92.18
4.36 0.70
1.40 0.35
MICA
liéhhllppgee et (8022h
919E 0:433
153 0:35
039 1
21 0:20
1.74 1050
1 3 0.24
V1
0.3 1.05
155 0:00
2.00
0.40
1.50
NUCEL OMDCA&L-CKA)
0.1
92.09 0.75
92.17
1.37
Fedlign
Wang
Unsel
56
030
4.25 32
1.00
04.30
1112
205 25
0.30
2t 2117
7.38
0.65
3.61
0:13 0.55
4.80 0.13
18
Mins
80
0.90
9:00 1:70
4.50
Biter
4.23
1.19
0.35
M
10M7
MidS
2:
1:12
161
NECTL (DCALL CKA)
94.30
2,1
0.90
140 0.23
0,70
NUCFL (DCA-RSF-CKA)
05.00
0.35
2.50 +1 0.30
0.89 0.14
93.57
0.65
1.52 6 0.35
0.33 6 0.13
MIXA Hebholloguppe
b022b
0.05
91.74
1.00
277 0.422
0.21
94.37
9 0.55
2.85 6: 0.35
1.00 = 0.18
95.23
0.90
2
2 0.30
0.52 0.13
NUCEL (MDAIILCCA)
94.33
2 0.45
2.84
2: 0.28
1.01 = 0.15
94.36
0.05
2.39
2 0.25
0,50 0.14
NUTRI CULDRI.MIICA-REE (MIKA)------
94
0.60
2.51
1 0.40
0.83
= 0.23
93.68
1.05
2,4%
6.78
0,90
0.12
Table 9: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimiza- tion methods with different calibration methods under non-IID (a = 0.1) scenario on the MNIST dataset. Underlined values indicate the best calibration across all methods.
Callestion
FalArg
McMaaan di sl. 2017
FelPrrs
Schua al. 2015
Scafial
Karininaly sl. 2019
Medled
Acc
SCE
Ace
SCE
Acc
131
57.02
0.20
0.30
1.330
0.33
Faoal
[in al 2017
K7.05
1.55
10.45
0.35
3.50
+ 0.30
84 22
0.70
11:26
+ 0:95
3.73 0.25
$3.50
+ 0.85
8.37
+ 1.65
3.04
0,8
L.
M
66
1.91
11.62 6: 1.15
3.01
8 0.92
53.52
0.69
7
2 1.77
2.68 0.68
84
111
14.57
1.55
4.64
1 0.70
bS IIF 11991
53.36
0.55
5.00
2 0.50
1.72
0.35
57995
1.5
10.41
3: 1.61
3.64 1.99
18.21
1: 1.55
19.55 2 2.9
7.53
2 1
MMUU
Cimmrraa Star
1016
5859
1.65
12.18
1.43
6.05
0.77
66 32
135
11:74
0:56
5.75
0.72
56
8:
1.29
18.35
8 2:38
4.61 0.00
F
45
0:77
9.28
106
5-66
8 0:45
52
1.08
10.11
9
5,00
0:55
86
+
1:35
11
1:11
1 0
MbLs
Lil 22-20
58.96
1.81
13:27
4: 1.55
4.51
0.40
$7.65
1.19
11.59
3.75
0.89
59.75
2:
2.25
18.65
4 2.79
7,05
1.25
IC Linng ol P3-37
15.45 1 1.35
5.83
0.80
1.25
1.33
$7.30
0.59
5.58
016-000 0.35
1.75 1.25
58,09
1: 110
0.32
0.72
1.4 033
NUCTL XCCXUUS
53.00
0.71
5.38
0.25
1.69
1
58,00
1.11
5,83
0.22
1.41 1.777
58.82
1: 1.35
3.09
0.77
3.29 0.37
NULIL (DUA-L-UKA)
57.70
1.14
5
0.73
1.59 1
57.35
1.05
3,05
0.17
0.24 1.41
58.71
1: 1.13
8.1%
0.35
3 0.20
NUCEL (DCAARBF:CCAA
5.30
B8.14
1.19
0.30
1.47
89.0
MOCA [ishtsiapepe et (00
16
5.41
1.23
18 1.55
54
0.81
1.77
067
56 115
6:7.3
AESFLIRDCATCCO
1.21
0.67
1
58.64
0.29
(MDCA-L CKAI
0.32
1.49
NUCFL (MICA-RBF-CKA)
Callbratin
FalNowa
Method
$5.31
1.20
1: 0,40
2.07
0.26
39:11 1: 0.30
9:58 25 0.18
3.49 01:2
Linee
87.49
1.65
10:34-11
0.50
0.27
86.05
2.65
1.20
0.77
0.92
84.05
0.95
2.35 0.33
20
1.70
1.28
0.44
0.95
3.43 0.92
57.47
0.82
Longes
2.57
6.39
89.00
0.99
2:30
0.18
59
0.73
NUCHL (KA)
HK.1
1.25
2:21
1382
19.00
1.00
0.6
NUCFL (CCARBEFCCAI
85.00
0.99
6.23 6 0.73
2.25
0.48
98
0.66
6.7 0.72
2.45 2 0.17
MICAA Hereeshpupe etal 10122
1.25
6.36
6 0.77
2.51
0.40
88
1.25
X.12 1.08
M 0.49
NUCFL (MDERVIOS
85,90
1.29
5.91
4: 0.58
179
0.39
89.76
2: 1.29
7.51
1: 0.68
2.67 0.23
NUCFL (MDCAL CKA)
99.09
1.24
5,88
à 0.49
1.9
0.15
9000
4 1.15
7.55
8 0.66
2.64 0.37
NUCFL (MICA-RBB--CKA
85.76
0.67
3.881
+: 1.38
1.00
0.44
50.75
4 1.35
7.41
0.76
2.60 = 0.09
Table 10: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimiza- tion methods with different calibration methods under non-IlI (a = 0.05) scenario on the MNIST dataset. Underlined values indicate the best calibration across all methods.
19

---

Published as a conference paper at ICLR 2025
McMalon
Kanminoddy
Uncal
92A8 0:25
1 0.15
1:56 = 0.00
MAR 022
0.09
0.16
1.27 017
0:32
4.25 = 0:15
12::1:0:00
Hocal
Lin 2017
91.06 0:28
4.73 0.17
1.98 2: 0.11
5.18
0.14
2.55 3. 0.07
9088
0:24
5.64
2: 0.00
2.75 0.00
1.5
50.30
0:33
3.94 0.17
1.00 0.19
0.26
0.35
0:22
3.16
0.17
9061
0.6
4:21
2: [16)
1/66 0:20
0:31
2.
2
3.00
0.29
2.52: 2
0.29
2.
0.27 92.75
0.26
2.7%
5.07
Mhi's
0:25
0:25
0:355
0:11
91.55 0:34
4.22
1.4 0:000
0:211
0.22
92.10 0.34
0.14
NUCRL (DCA-L-CKA)
4
0.23
0:14
92.75
0.34
1,1 2:13
NICHL (DA,00444FF-KK
92.58 0:33
3.90 0.17
1.49 01
93.11
9.96
0.2 3,00
156 0.18
0:49 92.81
4.12 + 0.25
1 0-144
MDDA
910 0:117
X95
10
03.56
1124
185 010
9201 0:25
4:20 TTS
TAN TANTFI
92.59
4 0.28
377%
0.30
1.50
+ 0.15
93.14
0.34
4.04
0.12
163
92.02
+ 0.19
4,12 0.22
LAB 0.14
NUCHL (MDCAALLCKA)
92.57
4 0.21
3.70
0.25
1.58
+ 0.13
93.12
0.55
3,77
+ 041
160
92.13 0.24
4
= 0.20
1,61 + 0.13
NUCHL (MICA-RII--CA))
92.6
0.31
361
0.12
1,53 8 0.06
95.21
0.17
4.12
0.55
1.00
0.1.1
92.14
0:31
4 s 0.14
140-2:008
Callbratoo Method
Felllyn
FraNora
Manig et (0000
Amc
ACR.
Apc
511
Focal (Lin 2011
0.24
0.42
LS
0.28
0.17
1.19
0.53
0.11
b5 Bireer (950)
91 2 0.61
5:23
0.21
2.57
0 0.17
91.44
0.71
4.51
0.32
2:40 0.19
MMCE
93.30 0.49
7.73
0.42
3,85
= 0.14
92.44
0.6
5.56
1: 0.68
3.03 0:33
FLSD Aypi 10
92 0.27
5.03 0.35
2.32
= 0.05
91 05
1
4,94 0:333
2.44 2.119
Mhl (pupaai 20031
91 0.30
5.332:0.2
2.56
= 0.17
92.00
0.55
4.58 0.42
2.41 +-0.17
DCA Linggatiin
03000 0.27
0.10
2.50
0.13
93.06
0.25
4:25 0.11
166 0:1333
NUC CUs
03.08 5 0.35
4.38
0.229
1.67
= D.17
93.26
0.25
4.34 0.30
1.59 0.117
NUCEL (DCA+L.CKA)
02.57 0.26
4.23 0.22
1.44
= D.15
93.06
0.24
4.03 0.27
1.54 1 1.011
NICEI (ICAAABPPCCCAA)
0.19
M
0.25
0.17
0.10
0270 0.26
91R9
025
0.19
4.41# 0.14
NUCRL (KA)
0.34
Table 11: Accuracy (%), calibration measures ECE (%), and SCE (%) various FL algorithms with different calibration methods under IID scenario on the FEMNIST dataset. Underlined values indicate the best calibra- tion across all methods.
Calibration Methed
FedAvg
McMahan ctal
MOLY
FolPPox
Sala IT
2013
Scalold
Kaimindds CtaL
2019
Arc
WCB
Ace
Acc
B.E.
SCPP
Uncal
90.95 8.0.4
4.19
1:04
1:56
0.77
91
1.17
4.1
1.00
1.95 8 0.99
91
1:79
5:00
1:19
2:51
1:388
Floal
Linn al. 2017
90.63 2 0.98
4.77
1.30
1.96
0: 0.75
90000
1.29
5.39
1: 1.45
2.77
2: 1.05
91.13
2 1,66
5.38
0: 1.27
2.72
1.04
L.
91.07 1.23
3.75
1.72
1.52 6: 1.02
90.37
1.38
4.52
4: 1.14
1.90
1.04
91.35
11:332100 1.25
5.18
8: 1.08
2.64
2: 0.99
bs I1r 1120)
91.88
0.58
5.19
1.24
2.65 0: 0.53
90
1.00
5.42 6: 0.55
2.50 02 0.60
52.72
1.11
4.19 8: 1.43
1.81 1.06
MMCE
2018
90.22
1.11
4.85
1:33
2.30
= 1.03
92.12
1.33
4.01 4: 2.04
1.79
2 1.38
01.44
1.33
5.07
= 1.68
2:55
1.21
FLSD
90.02
1.37
4.03
1.28
2.95
= 0.74
90.39
1.25
4,99 4: 1.38
2.04
2: 1.06
92.88
0.84
5.19
= 1.14
2.58 0:92
Mhh
Triner
90A2
1.59
4.21
1.0%
1:99
0.80
91.49
1.50
4.79
8: 1.30
1.94
0.64
92.87
1.64
5,99
0 1.55
2,60 8 0.9
DCX
Looss et (3.0)
31.3 1.00
200
2: 032
1,32 = 0.33
91.03
1.04
4:25
1: 0.38
1.01
2: 0.20
92304
1.30
4,44
- 1.03
1 0:38
91.77
0:70
1.00
0.96
1.37
NUCS, (KA)
91.63
0:95
0:58
0.88
92.19
0:95
=
1 0.65
NLXIL
IUCCA CKA)
1.62
2.033
1:29
2.03
1
MXTTL1900
NUCEL
0.99
1.79
FullNow
Methed
Uncal
1:12
3.00
1.01
0.95
2:44
Focal (Lin 2017
1.05
2.91
1:65
1.32
0.95
1.77
1.33
1:77
90.99
1.21
1.06
4.31
NUTE (DC
91.22 1.11
4.20 1.25
1,6 - 0.91
92.25
1.04
4.15
11 0.95
6666 0.34
NUCFL (DCAFL (DCAsL-(KA)
93 0.96
4.22
1.14
145
0.69
92.33
1.22
4.02
+ 1.00
1.44 0.83
NUXTL (DCAAII CKA)
9256 2 1.61
4.34 1:35
1.75
= 1.10
92.41
1.15
4.11
4: 1.00
1.65 4 0.65
MIDTA
Bettalugge (2022
03.99
1.00
4.111034
1.33
= 1675
92
2 1.28
4.22
1.14
T 0.35
01.11
1.44
4.31
1:22
1.75
= 1.99
91
1.22
4.28
0.98
1.67 0.59
NUTR, (MICAA---KA))
91.11
1.44
4.31
1.32
1.78
= 0.99
00
1.23
4.38
0.98
1669 à 0.59
99.04
1.23
4.40
1:33
1.00
= 1.05
02.
0.95
4.17
0.95
1,67 0.88
Table 12: Accuracy (%), calibration measures ECE (%), and SCE (%) of various FL algorithms with different
calibration methods under non-III (a 0.5) scenario on the FEMNIST dataset. Values in boldface represent the best calibration provided by our method for the auxiliary calibration method, and underlined values indicate the best calibration across all methods.
20

---

Published as a conference paper at ICLR 2025
Colibrilonn
McMahan
FedFrox
Scatood
Kaimiredd
Methed
Uneal.
0.22
2.83
2.79
0.2
[Lin
0.33
3.51 2
0.52
0.57
3.34 0.17
0.17
300
0.20
3.41
0.56
2.77
57.53
0.32
0.76
0.92
90.25
1.08
3,05 0.30
1L.00
M6997 0.67
541 0.35
2.51
89.73
0.1
90.1
0.1
8 0.30
3.36
Mbl
1:00
0:77
1.59
8944
096
0:04
0:33
5905
0.77
3.39 0:30
I
10.00 0.57
4.701:023 0:21
1.66 0.14
89.01
06
4311 6:233
2:51
3 0:15
98
0.4
6388 + 0:34
1/0 0:17
90.01 0-72
4.50
0.34
1.77
+ 0.19
85
0.88
4.62 0.39
1.72
0.25
90.00
0:71
5.86
+ 0.41
2,69 1 0.39
NUCFL (DCArL CKA)
59,89 0.53
1,69
0.19
1.77
58,99
0.57
4.56 0.20
1.09
2 0.11
90.23
4: 0.00
5.61
= 0.39
246 2 0.20
NUCTL (XCAABBB-
59.7 4 061
4.58
0:25
1.71 5 0.14
89.51 0.54
4.77
4 0.22
1.55
4 0.12
90.51
641
5,70
6 0.21
246 2 0.14
MDRA mechaagggeee
MT 0:37
1142
1 11
MM 1131
2:50
2.50
015
91
1700
5,95 022
295 1.10
MEXF AFCTS
59.05 8 1.00
4:64
0:39
1.78
8 0.28
HK 01
4.77 8: 066
1.90
8 0:32
90.22
0:55
79
6 0.31
2,67 0.18
N (MICA-L--OA)
89.11 + 0:78
4.89
061
1.73
+ 0.90
88.96
0.51
461
+ 0.02
1.72
+ 0.25
90.47
0:56
5.60
+ 0.21
2 H 0.14
DUCPLLMECAA /MECX+RBF CKA)
53.99 0.48
4.87
0.21
2.35 4 0.15
89.91
0.52
4.5% 6: 0.22
1.09
2 0.13
90.84
0:53
5.67
= 0.24
2.45 à 0.14
Calibration Methed
Fedlyyn
Moar
0021
FedNova
Wong et 50506
Acc 1
SCB
Acc
Un-al.
89.03 2 0.58
0:225
3,00
= 0.17
89.56
2 0.51
5.58 0.22
2.75 0.19
Focal eLin al. 2017
99.13 2 0.30
8.48 0.22
3.45
0: 0.20
89442
0.69
6.12 0.30
3.07 0.21
LS
88.38 2 0.53
033 0.44
1.50 =: 0.23
88.63
0
7.57 0.225
3.20
83.50
0.71
7
3.90
= 0.14
88.94
0.45
7.43 0.25
361 +-0.44
MMCE
2018
68.50
1.57
7.00
1.01
3.33
= 0.49
88.94
1.05
7.11 0:7.
0.039-0.39
FLSD
85.00
0.57
7.18
0.24
3,15
1113
81.45
1.04
6:00 0.88
2.9 0.39
Mhh
10
0.80
7 0:37
86.74
0.9
5:14
2 0:79
DC
00
575
0.25
2.77 1115
MET1
3740
NUI
0.56
533 0:32
0.54
0.40
2.00
NUCFL CKA
80:37 0.77
5.07
0:23
2.55
1.97 +
0.39
0.72
(MIDCA-L-CCA)
(MECA+RDE 89.30
0.38
0.53
1.99
Table 13: Accuracy (%), calibration measures ECE (%), and SCE (%) of various FI. algorithms with different calibration methods under non-lll) (a = 0.1) scenario on the FEMNIST dataset. Underlined values indicate the best calibration across all methods.
Calibration Metharl
FedAyg McMalan 2017
FedPrrox
Salu
Scaffold
Kanmrrddy
2019
Arc
Ace
Uncal
120
3.88
K7.17
0.25
Lin
à
3.42.1
57.25
1.00
0.95
168
3.44 0.37
0.80
0.27
3.69 1:.06:0.6
9.33
0.39
038
0.57
3.08 3.01
87.25
0.44
13.35
0.32
4.52.400.0
FESID
K5997 8 0.38
9.61 x 0:55
547 8
73
081
9.20
0:20
142-8-0.11
87.01
1.70
14.56
1.46
5,07 9-072
Mbt
85055 6 0:77
9.59 = 0.66
3.51 9
86433
1.22
R82 +
0.30
86.05
1.33
10:4
9 0.73
3.81 1.11+-0.88 0.38
DCX [12001
57.84 1.24
1.99022
321 2
$7.30
1:22
511
0.33
83.25 = 1:12
000 0:44
0.52
NUCF (AC.A
57.07 1.00
5.02
0.36
0.30
58.15
1.25
5.96
2 0.33
3.01
2 0.21
87.50
:: 1.00
8.79
0.6
3.49 0.44
NUCFL CCA)
$6.39
1.24
7.54 0.71
3.27
0.33
$7.55
2:55:100 1.04
5.66
2 0.29
2.54
2 0.17
87.23
= 1.34
K,51 0.72
3 0.31
NI NLCFLIXX (XCEMBCCCAA
66.95
1.08
7,80 8 0.49
3.26
0.72
18
1:31
5
1 0:46
2,88 8 0.25
11.11
+ 1.00
R.47
0.66
3.66
0:30
MINA
57107 T 1:12
5.27
1.14
19115
1:15
528
11
1:06 0:11
1:18
1:07 11:11
1:11 11377
N.. naRX SECTSS
60.05
+ 1.10
6.20
0.41
3.36
0.27
58.74
1.14
5.1
+ 0.38
2.79 0.36
87
1.33
R
0:87
3.45 0.41
NUCEL (MDCA&L-CKA)
67.11
1.33
7.99
0.92
3.30
0.66
87.06
+ 1.42
5.54
= 0.01
2.70
0.33
47
1.00
R6l 0.76
3,66
0.41
NUCH,,MIXXA-HHI /MIXCA-IIII-KKM
$7.59
1.31
7,87
06)
3.25
0.44
5001
1.14
5.99
(1))
2.67
0.28
87.54
1.51
K:0
0.41
3.4.4 0.28
Calbraaion Methoo
Tellyg
NERE 2025
Tediowa
Wann et 202000
Apc
SCE
Tread
3k 5 10K
10.3
3.66
1121
81.56
1.02
91hh 1:10.0
3.50 2: 0L18
Rogil L1 al 2017
17.13 2:
103
11:25
0.77
3.09
01 1..9
85.002
1194
12.92
2: 11.49
4.47 2 0.35
LS Mheraaat T0TV
88 34
1.46
15
1:31
5.10 0: 1177
85.63
1195
10.77 0 0.35
3,20 8 0.25
BS Taterr 19500
$7.00
2
0.7
12.58
0.5.4
4.45 2 0.21
86.94
0.94
10:13
2 0.25
3.64 2 0.15
MMCCE
2015
$7.80
1 1.19
15.10
0.48
5:13
= 0.22
81.11
1.03
13:31
2: 0.73
4.52
- 0.49
FLSD Muninn 21 Hir
36.00
1 094
9,07
1: 0.53
3,52
= 0.34
86.45
1.12
10.29
2: 0.49
3.65 0.50
9 0.61
0.67
35.0
5,60 0:42
3
NUXTL
38.00
8.53
8
3.41
NUCFL (DCAIL-CRA)
38337
7,87 1:
=
8.36 0.47
Michillgepee al
7.55
(MDCAL-CCAA
3.58
NUCHL (MUCA-KIP CKA)
0.44
3,35
0.22
Table 14: Accuracy (%), calibration measures ECE (%), and SCE (%) of various FL algorithms with different calibration methods under non-IID (a = 0.05) scenario on the FEMNIST dataset. Underlined values indicate the best calibration across all methods.
21

---

Published as a conference paper at ICLR 2025
Callediin
FrdAkgg
Mc MoManan
MOTS
Stadindl
Sarminily
PITV
Methedl
Avc
Acc
Apc
Froal [Linetal 2013
19
10.19 0.47
5.51
12.24
1.26
0.77
3009
0.09
1.5
70
1.30
8.30
0.51
5.0
8:00 0.41
09006
1.05
9.18 0.91
3.51 0:93
79.25
1.42
9.49 0.85
1.55
0.50
b5 Brr (1550
76.90
1.32
5.27
2 0.49
1.64
3:10 0.35
77.97
1.07
5,52
0.64
3.13 0.34
78.54
2: 1.28
0.75
2.53
0.45
MMCL Kimmrreaa 2018
79
1.07
9.82 0.41
3.50
0.33
78.54
0.1
5,00
0000200:3 0.74
3.10 0:22
79.25
1.16
10.01
2 0:73
3,52
0.42
ILSD Nylbite 21 10:0
79.76
1:25
9.52
0.61
3.68
2:10 0.43
1001
1:14 1.41
11.46
2 0.32
3.79 1:58
10.11
1: 1100
945
155
Mhts [eirn p31e)
81.73 4-2
2.12
11.17
2: 0.65
3.83
2.0.0
79.30
1,68
10.01
+ 0.61
3,69
0.20
79.57
2 1224
9.11 0.0.2
1.46 0.53
DCA Liint (ilimaarr p0r00
50.35
1.09
6.37
1.91
0.3.4
$2
1.12
7.35
0.64
2.61 0.255
50.23
2 1.20
502 0:83
1.49 0.44
NUCT (DCAttos
50.35
1.39
5.75
8: 0.48
1.75
0.25
$1.65
0.55
6.71
0.50
2:50 0.39
$1.32
2 1.14
4,22 0:77
1 0.31
NUCFL (DCA/L CKA)
$1.05
1.01
5.77
2 0.50
1.75
1
7966
1.13
6.29
0.51
1,88 0.25
S099
2: 1.00
4,66 2: 0.81
1.34 0.4
NECEL IDC (DBBBFAAAA
80.48
5.64 0.65
1.71 0.20
1.12
6.54
1.20
2.28 0.33
$1.63
1.26
MICA
liéhhllapgee et (8022h
1.14
5.9 0.59
2:37
1202
1:24
7.41 0.54
297 0.1
81.11 2
6.11
2.2
1.08
7,07
0.54
81-92
NUCEL OMDCA&L-CKA)
5.38
168 0.31
1.08
0.56
81
236
Fedlig
Unsel
11:01
1:30
9.11 1: 0.005
5.47 1133
$1.00
1111
1035
3.32
2t 2117
10:12 10112:133 0:52
0.33
80.39
15:61
0..5
0.63
18
Mins at 2119
0.62
70
15.93
4.50
Biter
9.04 0.71
3.48
12.00- 0.50
11:12
M
0.65
4.51
MidS
09 0.544
14.63 0:0.5
8.66 0.9.1
3.30 0.19
NECTL (DCALL CKA)
50.89
1: 0.51
1.79
2,67
NUCFL (DCA-RSF-CKA)
$0.26 2- 1.07
6.16 9: 0.54
1.84 0.36
82.04 0.97
1.29::0.11 0.61
2.68 + 0.30
MIXA Hebholloguppe at (022b
0.161:0..8
100
81
VDT 0.777
1.48 0.3
$1.59
3: 1.15
7.20
2: 0.82
2.54
= 0.40
82.05
1111
8.31 0.53
3.29 2 0.27
NUCEL (MDCAIL-CA)
$0.91 = 0.34
6,96
2: 0.47
2.39
0.38
81.36
1.12
7,33 2:33::0..3
2.63
2 0.41
NUTRI CALTRR.(MICA--HRT (MICC--R---KK
30
1.11
7.34
9- 1.32
2,53
= 066
81.56
0166
7.72.800.5
7.94 8 0.25
Table 15: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated opti- mization methods with different calibration methods under IID scenario on the CIFAR-10 dataset. Underlined values indicate the best calibration across all methods.
Callestion Medlod
FalAAg
McMaaan di nl. 2017
FelPrro
Schu al. 2015
Scafial
Karininaly sl.
2019
Ace
UCL.
SCE
Ace
SCE
Acc
PCP
9:59
330
0:5
345
1019
115
Faoal
[in al. 2017
14.64
1.68
13.11 + 0.61
4.7
8 0.36
7600
1:11
10.34 + 0.71
3.54
0.44
77.00
1.07
11.37 8 0.54
1.56
0.35
L.
M Minse
77.00
1.10
10.04
:: 0.57
3.55
2 0.35
42
1.09
14.71
8: 0.65
4.72 0.36
77.54
111
14.44 0.51
4.71
0.22
bS IIF 11991
78
1.00
10.57
5: 0.55
3.55
2 0.34
14
1.13
10.26 2: 1.17
3.55
038
76.00
1: 1.22
8.38 1: 0.55
3,00
0.35
MMUU
Cimmrraa Star
10116
78.22 8 1.35
7.40
8 1.22
2.71
0.36
50.39
1.07
14.27
8: 1.00
4.75
1:51
76.51-
8: 1.09
9.0
8: 1.00
1.42
0.0
F
18 8
1.10
1400
1.11
4.79
9 0.88
1.80
16.79
8:040
4.75
1:36
75
0.77
7.37
0:01
2.65 0:51
MbLs
Lile 22-20
79.07
0.51
13:70
0.89
4.00
0.66
78.46
2: 1.13
11.31
3 0.66
3,56
0.4
70.15
2 1.10
13,67
0.53
4.87
0.60
IC Liang ol P3-37
18.64
1.38
1,25
- 0.35
5.1
1:44
11.11
1:34
9.10
0.54
3.45
0.34
70.12
+: 1:20
0.35
0.5
1 0.54
NUCTL DDCXCUS
79.0
1.00
7,40
:: 0.96
2.72
1.
79.13 11131:00 0.91
5.71
0.06
3.43 0.235
79.33
2: 1.14
8.74
1:741:00 0.49
3,30 0.2
NULIL (DUA-L-UKA)
78
1.13
7.06 0.66
2.60 1 0.41
79
1.12
8.66 0.06
5,29 0:33
50.54
1.15
1.00
1.11 103
NUCEL (DCAABBF:CCAA)
5.03
1.00
0.43
3,45
79.31 1
MOCA [ishtsiapepe et (00
6.44
511
19.2 1.12
5.80
0.00
0.35
1966
1.10
0.67
7.53
2.75
0.91
8.67
0.47
3,38
80.56
(MDCA-L CKAI
0.51
1.11
0.71
79.77
NUCFL (MICA-RBF-CKA)
Callbration
FelDyn
DOAT
FedNowa
Method
Unsol
10300
24.44.110 0.35
4.50 =: 11:55
15.99
1.05
15.07 2: 11,9
4,54 :: 0130
76.90
0.36
10:11
3.67
0.79
1.22
14
4.50
0.61
1.74
15.00
4.98
77.66
1.79
17.03
13.72
7950
4:
0191
11:94
3,89 0.66
NUCEL (ICA+LCCA)
5035
1.20
8A2
11.99
10.19
1,90
NUCFL (DCA-RSF-CKA)
7905 2: 1.10
8.58
2: 1.50
3,16
0.62
79.04
0.62
10.70 2 0.87
3.63 2 0.44
MIXAA [irhhologggee dt (0200)
79.55-111 1.1%
10.21
0.46
1.47
022
7010
1.2
1522
1
4.77 (.:)
IECELIMECNNIOO
79.30
2 0.1
8,33
4: 0.59
3
= 0.44
79.53
1.41
11.74
2: 1.25
3,67
2 0.66
NUCEL IMDCA CKA)
30.17
2: 1.22
3,47
4: 0.56
3.15
= 0.42
79
1.12
10.21
2 0.55
3
2: 0.34
NUCRL (MBCA--SF--CA)
7961
2: 1.00
9.88
6 0.51
3.54 = 0.60
80007
1.10
10:00
: 1.06
3,62
0.48
Table 16: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimiza- tion methods with different calibration methods under non-lII (a = 0.5) scenario on the CIFAR-10 dataset. Underlined values indicate the best calibration across all methods.
22

---

Published as a conference paper at ICLR 2025
FedAvE
McMatzn 20L7
SCaTTO
Karmineday
Methoel
Uncall
54.36
121
12.02
1:15
1.11
13:19 1: 1:35
4.65
0.35
66
1.10
4,09
0.30
[Innztal
50.08
4.81
1:30
16:30
4.99
0.83
11:02
1.08
12:02
1.07
10.90
3.00
0.88
1:71
0.21
10.61
3.56
0.82
69.09
19:42:4
1.91
2.07
10.91
1
MbL'
12.77 1.26
1:26
10.03
3,50
64.95
0.96
11:4
NUCTE MOKTCIDCTTCOS
9.94
10.74
67.47
0.89
NUCTL (DCAr (KA)
8.43
62.03
10.42
06.99
1.17
115
NUCFL (DCA+BBF-CKA)
55.00
1.39
9.99 8: 1.38
3.12 4: 0.72
67.08
1.23
9.59
2.12
3.50
1.6
66.17 4: 1.1
12.14
1.10
4.36
0.0
MODA [ishbatapigpe
[M016]
10,95 1.088
3.56
08
1:32
10.20
8:
3.54
0.66
67
102
15.77
2.15
4.49
191
MECFL (ADCATIOS)
25.25:4 1,88
9.74
1,05
3.44
3: 1.01
56.54
1.39
9.57
1.37
3.47
0.85
67.62
4: 1.45
12:00
1.11
4,40
1.37
NUCTL (MDCAAL
54.97
1.03
9.27
1.06
3.41
3: 1,03
56.91
1.06
9,01
2 1.77
533
1.24
66.94
4: 1.6
12.63
3.13
4 2.39
NUCEL (MEC/+RBF CKA)
35
1.00
10.16
4: 1.07
3.52
1:35
56.92
6 2,09
9.54
1.35
3.5
1003
67.04
4 1.19
11,06
8 2.10
38%
1.8
Calibratiaa Method
Fellyg
Near et 10C
2021)
FedNova
Wong
20205
Acc
SCE
Acc
SCE
1335
9:49
1. Mulkeretal
64 13
1.25
15.00-1
1.08
4.68 m 0.80
61:71
1:40
13.65
1.35
4.57 + 0.91
TVP-ry
63.62
1.26
19:23
1:199
0.33 =0 0.95
60.25
1.24
0.54 +: 1:41
3.58 = 0.97
MMCE
2013
63.99 09
8: 1.25
15.53
1.221
4.71 = 0.93
64.50
1.41
14:03
0: 2.35
4.50 = 2.00
FLSU
MUsn of
06.09
1:27
20:53
2:77
9.30 8: 1.95
03.54
1.25
14.00
2: 1.35
4.50 8: 1.00
Mbl
66
1:24
17:59
01778:111 1:17
7:00 = 0.91
62.64
1.25
10.09
8: 1.30
1.58 0.92
DEA Inngetaa
65.5
9: 1.25
15
4.44
= 0.6
67
1.21
11:11
- 1.26
1.64
0.91
65.94
8.9
12.10++2244
2:006
4.54 1.39
67:54
1.23
9.0
+ 1.37
3.30
1.25
NUCTL (KA))L
64.88
1: 1.52
11.39
1.28
4.07
1.35
67.41
1.36
9:15
1: 1,38
3,67
= 0.77
NUCHL (CA-L UKA)
04.77
2.32
12:27
1.48
4.36
1.00
67.20
1.18
10.19
:: 1.15
3.71
0.92
MIXA
[Hebhtllggageee et (M222
06 70
1:24
13355
1:35
4.47
100
6784
2.01
12.55
3: 1.49
3:97
06
12
1:32
06.0
10.1
1:04
1.6
IMDCAIL-CKA)
1.44
10.12
3,60
(MDCALMMD
2.30
11:36
Table 17: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimiza- tion methods with different calibration methods under non-lll) (a = 0.1) scenario on the CIFAR-10 dataset. Underlined values indicate the best calibration across all methods.
Staffol
Karimnnkay
Methall
Lincal
1:013
1100
1603
4.93
01
1135
4.01
Raoal
al
1.08
11:72-1
4118 0.96
1:11
1.10
18 mm
1.07
7.63
60.1
1:31
38.68
8:54
0.9
1:35
17:70
1.09
22.48
1.13
1:21
17:98
0.36
1.05
MMCL
1.00
13:52
4.53
1.07
11,80
Mhls
18.31
20.28
60
1.13
13.07 1.06
62.25
14.57
0.89
60.75 1 1:15
11,07
1.04
NUCTL ICCATCOST
4.19
100
4.59 0.2
62.05
1.29
1.02
NUCFL, (DCA-L
1.00
1.25
13,03
4.57
0.42
62.09
1.18
1.33
1.20
NLCFL (DCA4BBF-CKA)
50.52
1.31
10.76
1.34
5.94
0.1
62.00
1.32
13.00 + 1.11
4.00
0.6
61.99
: 1.00
10.14
1.52
4.00
111
MDCA Riilsoueee dl 1002h
50.15 9 1.71
12.99
9 1.05
4.30
1.16
1:21
15.18
1.31
4:70
091
61
3
12255
1.16
4,50
152
NOCFL
50.00
1.92
10.53
à 1.03
3.00
1.15
62.20
2 1.33
14,01
3: 1.09
466
0.90
61.37
1 1.35
12.00
1.37
4,50
1.00
NUCFL OMDCAAL CKAI
50
1.72
10,85
1.01
3
2 1.14
$2.01
1.22
13.92
à: 1.88
4.61
0.89
62.35
1 1.32
11.35
1.3
4.17
1.71
NUCFI, (RBF--KK
92.02-1
1.32
11.08
8 1.00
4.001:111
1.15
61
1 1:38
13,44
1: 1.92
4.5
9 0.76
61.69
1 1.00
10.30
8 1.12
4
1 1.00
Calliratim
Felllgn
Actral 20AD
Whng al 20205
Mathud
Acc
SCE
Apc
SCE
1315
370
Feval
20177
13
10:55
1.00
1,59
0.89
0022100
1.23
15.64
1.16
4.00 6 0.97
1.
Minerera mamm
61.26
2.32
1.32
6.96
1.13
50
1.24
17:20
+ 1.25
5.18
1.00
BS Baner (9-4))
59.62
4: 1.15
14.23
2:
1.13
4.33 0: 0.93
60.25
1.20
19:54
6: 1.20
5
1.01
MMC
2015
59.39
1.20
19.76
2: 1.67
7.000 8: 1.32
61.11
1.21
17:95 3: 1.17
5.42
1.00
FUSU MURiarr
15.61 31 1.22
22.00-1
1.12
4.33
01 1195
6015
1.25
17.0
21 1.19
5:22
1.09
Mids 11
60
1 1.21
17.37
1.42
5.34
= 1.02
62.64
1.22
20.99
2 1.16
7.58
0.95
DCA Litre 20100
6005
3: 1.32
13,00
1.29
4.00
= 1171
62.11
1.20
1539
2 1.16
4,50
2 009
60.25 2: 1.62
12
1.38
4.49
= 1U88
02.54
1.22
14.13
2 1.37
4.70
2: 1.00
NUXTL (DCALL CKA)
60.6
1: 1.24
11.55
1.21
4,30
= 1131
02.41
1.42
1309
2 1.15
4.58
0.99
NUCFL (DCA-RSF:CKA)
60.7
11.27
1.00
4.5
13.19
1.58
MIXAA [Nehhollggggpe et (3022h)
61
1.18
14
1.10
4.5
02
161
1500
1:32
487
1.04
NECPL MMECATCCO)
1.11
471
IMDCAIL (KA)
61.75
199
1411
471
(MICC--RR---KAA
Table 18: Accuracy (%). calibration measures ECE (%), and SCE (%) of various federated opti- mization methods with different calibration methods under non-lll (a 0.05) scenario on the CIFAR-10 10 dataset. Underlined values indicate the best calibration across all methods.
23

---

Published as a conference paper at ICLR 2025
Kanmindyy
Lo:31
10
1:25
7.01
1314
3:25 0.30
66.15
0,32 2: 1.01
319
U.55
56
5.24
3.37
Poga Latetal 201
61.88
1.61
11.21
1.31
3.78
1.00
1.55
9.25
1:24
3.5
1.07
50.05
10.00 6: 0.992
3.59
LS/Murraaa salr HerV
17
1:87
12.19
1.88
0:94
1:30
10.21
1:40
0.24 1:10
7:01
3.67
BSSiiter
63:00
1.71
1.24
3.38
1.00
65.19
1:20
7.11
3.04
MNCE Nuprrr
1.61
3.73 1.01
64.22
1.23
3.50
9:27
1.95
64.02
1.25
63.83
1005
0:92
1.05
1.07
1.1
0.85
66.73
6.92
NUCFL. (DCAsL.CCC CK.)
65.43
1003
0.78 2,96
66.66
1.20
0.92
0.75
NLCFL (IX A+HBF-CKA)
63.3
0.95
899
1.1%
0.00
66.17
1.06
5.58 + 1.03
2.20 + 0.78
56.37 6 0:58
6.79
1.00
2.96 8 0.8
MIDIA MARAATGhSSSHHEE 201200
63.32
1225
R25
T T75
512
0:90
61AVT
1:75
53H
110
112 10
56.11 1:35
7:0
T05
63.41
1.04
6.77
1.11
3.00
+ 0.82
63.92
+ 1.44
5.92
+ 1.24
2.95 + 0.63
$6.65
1.04
6.83
+ 1.22
2.99 + 0.91
NUCR, (MDCA&L-CKA)
65.50
1.27
6.04
1000
382
2 0:66
65.85
+: 1.03
5.1
+ 1.00
2,93 4: 0.85
56.70
1.30
6:77
1.06
2.97 à 0.83
NUCEL (MIXAARRI-CCAA
05.64
8 1:00
6.08
1:25
3,05
0.92
6..8
8: 1,11
5,82
4 1.35
2M3 6: 1.00
56.75
1.04
0,55
1.20
2.99 0.97
Caltiration Method
Fedilgn Acaretal
Fedloora Wang ml
(0000
Ace
F.
Acc
at. 2015
9:00
1.69
10.00 0:
3.70
1.14
66.89
11:52 1.32
3,83
06.71
1:21
12.01 8:
5:73
0191
BalBrier 1920
65.92
1.06
10.62 4: 0.94
3.68
0: 0.70
65.13
1.49
7.91
:: 1.34
3.30
1.00
MMC Di Mirii SOIIEELBRR RU13
65.61 1.45
9.11
2: 1.27
3.56
8: 1.04
65.92
1.52
10.55
2: 1.22
3.00 0.89
FLSE NyptonEss1
03.33 1.95
8.21
2: 1.40
3.37
2: 1.31
64.89
1.91
9.59
1.74
3.70 1.56
Mia MicStranes Serresi Pa223
65.70 8- 0.92
1091
+ 0.49
3.80
- 0.37
63.21
0:11-:00 1.04
10.25
+ 1.28
3.77
0.91
DCA Mamp MUN0
66.25 3: 1:34
5.20
0.00
3.37
8 0.83
67.39
1.49
8,08
1,08
3:31= 0.91
66.15 1: 1.23
5.04
1.03
3.29
4 0.71
67.44
2 1,02
7.81
::1::00 1.00
3.26 0.87
NUCEL (DCAIL-CAA)
66.02 1.04
5.04 5: 1.23
3.25
1.02
67.52
1,02
7.77
1.23
3.24 0.95
NUDRI (DCA-RBB-KKA)
8.01
3.36
091
67
1.25
1.04 7.79
3.36 0.8
06 122
3:34
333
6797
1:39
311
1:58
133 191
NECPL MMEX AICOO
7.06
67.73
1.09
8.0
1.33
NECFL (MDCALL-CAA)
1.33
7.00
67.92 1.31
7.85
1.22
3,27
(MUCA-RHH-CKA)
3.00
1006
1,04
Table 19: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimization
methods with different calibration methods under IID scenario on the CIFAR-100 dataset. Underlined values indicate the best calibration across all methods.
Calibration Methed
FedAvz
McMahin et a.
2017
Fellrrox
Solu CE J. 2013
Scatiad
Karmirddy d..
2019
Arc 1
DCL.
SC1.
Acc
SCE
Ncc
SCT.
Uncal
64.36 1:31
10,52
1:12
3.6-1
8 0.9
61
1:55
9037
1.31
3.58
1.05
02.05
0.9
11:42 2.05
3,67
1:55
Floal Linn al. 2017
60.59
1.42
12.58
1: 0.95
3.74
4: 1.35
62.07
1.39
11.45
4: 1.04
3.79
0.91
61.39
1: 1.15
13.25
2: 1.71
4.53
1.30
L.
5000 1
99.25 8 2.15
15.51
1: 1.27
0.25
4: 1.24
52
1.48
14.37
4: 1.35
6.14
1.55
62.15
1: 1.55
11.69
1.34
4.05
2: 1.07
Hire (120
51.20
1.58
11:32
+: 1.44
3.80-1
+1 1.10
50.43
1.04
10.15
+1 1.38
3,61
1.17
60.39 + 1.30
10.92
1:33
1,80
1.10
MMCE
Rilliee 2018
50.00
1.07
13.11
+ 2.05
4.03
4: 1.73
50.11
1.35
11:32
+: 1.22
3.77
1.39
61:03 +: 1:22
11:37
1.52
1.63
1.08
FL5D
58.71
2.50
11.51
2: 1.33
3.02
4: 1:05
59.28
1.40
10.66
4: 1.39
3.81
1.04
60.49
2: 1.35
13:61
1.51
5.15
0.99
Mhh Triner
58
2:31
9.67
1.58
3.55
0.96
N1.39
1:11
11:37
8: 1:300
5.84
1:25
61.38
8: 1.67
11:99
1.59
4.00
1.25
DC.
Lioss (3.0)
01.3 1.20
7.00
= 1.1.
3.24
2: 1325
61.33
1.37
$:4.
1:22
3,00
0.93
02:11 2: 1:30
9:25 4: 1.22
3.35
038
61.38
6:21
0.95
8.15
1.01
5.55
1.06
1.04
0.79
NUCS,
(KA)
60.05
6.14
3,07
62.31 1:31
8.04
1.19
3
62:25
1.15
1.00
NLXIL
IUC CKA)
6,19
5.17
1.04
01
1.33
61.05
7:71
1:33
3:21 1.45
337
1:20
NUCEL CKA)
1044
3.29
62.15
3,45
Callbration
Fellyy
Wang
Method
Unxed
12.5% 1:35
1.10
1.05
1:53
2017
1.00
3,81
15:21
1.34
4.71
2.08
16:03
6.59
62
1.27
1.61
12.05
12:55
3.50
11.85 1.04
1.52
13.81
1.23
4.58
62.90
95
1:41
02.91
1.95
13:65
4.80
1.35
9:27
3355
NUCW TE
62.11 + 0.91
9.70 + 0.95
1.54
- 0.67
63.24
1.29
8.85 + 1.11
3.51 + 0.2
NUCFL IDCAFL (DCA+L-CKA)
62.94 4: 1.22
9.14 + 1.22
3.52
0.84
63.27
1.30
8.52
+ 0.94
3.43 0.64
NUCIL (DCA-RBI CKA)
02.44
2
9:31
+ 1.04
335
= 1192
03.17
1.17
8,01
2: 1,08
3,30
0.8
MEXA Hahhaloguge et (0226
02
3 0.2
1024 1:35
1722
= 1139
03.30
1.05
1010
= 1:15
3.71
2 0.14
62.1
8: 1.04
9.35 8: 1.00
1.5
1.91
64.14
1.11
9.16 1.24
3.58
8 0.99
NUCFL (MDCAIM
62
4 1.95
9.89
1: 0.64
A83
0.67
69.14
155
9
1.48
3.51
8 1.17
NUCEL (MDCA -RR CKA)
62.65
1.12
9.24 4: 1,20
3.55
= 0.94
03.22
1.58
8.59
1.27
3.47
8: 1.06
Table 20: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimization
methods with different calibration methods under non-III (a 0.5) scenario the CIFAR-100 dataset. Values in boldface represent the best calibration provided by our method for the auxiliary calibration method, and underlined values indicate the best calibration across all methods.
24

---

Published as a conference paper at ICLR 2025
Colibralonn
FedAvg McMahan
Scatiod
Methed
Uneal
1.30
10.61
1.33
[Lin
14.28
1.20
11:25
4.19
1.49
12:01
3.81
14.77
4.00
1:31
57
1.24
1.47
11:72
4.17
1.21
1L.0D
11:91 1.52
3.71
16.28:5
1.30
10.00
3,89 1.15
57
1.6
0.93
Mbl
5862
..49
13:52
1.82
4.25 0.55
57.59
9:33 0.32
11:77
1414 105
59.38
8 1055
1:15 16.99
4.77
IXAA 1301
99.99
1.12
10,45
0.82
561 0.55
58.95
1:35
3.57
6.77 0.74
K65 0.45
59.10
+ 1.76
11.09
1:35
1.61
0:51
58.97 + 1.35
9.35
0.90
3.55
59.1
1.12
5.00
0.88
341 0.00
59.13
+ 1.35
10.87 0:79
3.68 0.53
NUCFL (DCArt CKA)
50.75
1.20
9.14
0.83
3.50
59.15
1 1.32
7.99
0.71
542
0:33
60099
4: 1.29
10.95
2 0:73
3.70
0.64
59.20
1.30
10.16
2 0.94
3.00
59
2: 1:24
5,09
0.62
5.41
0.10
00JIS 4 1:35
10.35
à 0.69
361
0.57
MDRA mechaagggee
1.22
10.19 1.10
3011
1115
5.01
0191
337
11339
CUNS 1 11
12107
130
4.13
0.4
MIXFFISIX AFCIS
59.34 8 1:35
9:74
8 0:95
3.59
8 0.69
58.7
1:30
8.12
0.80
3.44
8 0:57
6007 1.12
11
1.01
4.06
061
N (MICAAL--OAA)
59.64-1
1:44
9,70
1.91
3.99
1:07
59.69
1:18
8.70
0.90
1.45
0:59
59.64
+ 1.15
16.1
0:599
366
0.6
DUCPL /MECX+RBF CKA)
58.71
1.14
10.09 3: 0.89
3,61
3 0.60
58.11
1.13
5,31
a 0.91
3,45
0.53
50.10
2: 1.00
10.64
1 0.87
3.64
2 0.68
Calibratina Method
Felliyn NNar
al. 0021
FedNova
Wong ct a). 50200
Acc 7
SCB
Acc
BC11
SCB
Uncel
39.39 8: 1:30
13.03
1:30
5.04 =
60.01
1.33
13.95
1: 1.14
4.97
2 0.92
Fooil R017
57.21
4: 1.27
13:62
1.22
5.05
0: 1.07
55.15
2.05
14.61
1: 1.53
5.19 2: 1.25
LS Mule RUEM
19.34
8: 1.67
15.59
2:33
5.38
1.72
60005
2.55
16.23 1: 1.31
5,09
8 2.05
59.17
1.63
14:21
0: 1.51
4.81
= 1.02
57.44
1.24
11:79
-: 1.86
4.19
= 1.35
MMCE
2016
58.35
+ 1.27
15.33
1.58
5.35
1.24
58.25
1.55
12:05
=: 1.62
4.98
1.06
F Muttiar 10
57
2.31
12:25
1:22
4.85
1.00
00.09
1.25
16:21
1.41
5.64
1.39
MNi
19
1:21
1,62 15.35
1.90
59.91
1.17
14:05
1:84
4.75
1.00
80
12155-003
4-1
1036
11:21
1.55
U.T
59.93
11:53-0000 0.94
60.14
1062
0.72
NECFL (KCAAL-AAA))
29:29: 0.30
1.01
8061 : 0.70
3,68 0.67
NUCH, (OCABBOOAA----XXXX.
0.95
10:00
3,65
1112
NUCFL (MDCA-KBP
1000
3,63
Table 21: Accuracy (%). calibration measures ECE (%), and SCE (%) of various federated optimization methods with different calibration methods under non-III (a = 0.1) scenario on the CIFAR-100 dataset. Underlined values indicate the best calihration across all methods.
Calibration Metharl
FedAyg McMahan 2017
LedProx
Satrer
Scaffild
Korminday
2019
SCB.
Acc
Uncal
311
Lin
0.54
16,40
5.58
57.63
17.0 0:33
13.98
1.30
5.75
5601
0:77
16.66
0.55
143
16.26
0.37
0.45
$7.45
1.34
17.77 0:9.1
FESID
54.00 1.35
11.39 0:00
5.5
59.23
1.5
19.45
1.45-860.0
6.50
0:11
55.39
1.42
10.84 0:76 12.47
1,61 0.54
Mbt
55.16
1.55
14.99 + 1.08
4.00+061
56 + 1.48
6:799 15.4
5.18 0.51
55.11
1.44
9.47--00.32
4.21 0.555
DCX [12011
56.19 2
13225
11:85 6 0:2
3093003
5699
1:10
1317
4.15
1:20
5810
2 1:05
ITRP 0:33
1.85 0.37
NUCT- (1..AA
56.97 à 1.05
3.64 0.333
57.64
0:57
11.12 0.28
3,53
0.22
57.33
4 1.02
10.97
0:311
1.80 0.29
NUCFL (DCA+L-CKA) CKA)
57,03 2 1.21
10.24 2 0.42
3.52 1:222:0.11
57.95
2: 1.01
10,09
2 0.32
3.50
2 0.19
57.75
4: 1.08
9
2:5-1:00 0.29
3 0.25
$6.00
1.11
10.76 8 0.10
3.54
0:34
50.5
1.00
10.19
8 0:92
5,57
861
57.85
8 1.00
10
0:30
1.81
0.2
MIKA
57.12
1:01
1777
170
1:
$7.77
1:12
19211
1:.1
4:38
57.05
1777
1037
11:11
151 1120
N..EE naRX XECTSS)
S6.04
+ 1.23
11.34
+ 0.42
3.76
+ 0.33
57.56
1.00
13,92
+ 0.30
4.11 0.31
57.87
+ 1.00
9,86 0.30
3.3 0.27
NUCEL (MDCA+L-CKA)
57
1.33
11.30
+: 0.41
3.76
+ 0.20
$7
+ 1,6
11,89
+ 6:39
3,33
0.23
56.94
+ 1.11
10.11
+
3,38
0.31
NUCH,,MIXXA-HHI /MIXCA-KIII-CK.
56.51
1.04
11
2 0.46
336
0.36
58.11
1.06
1407
4 0.31
4.35
0.26
57.90
A 1.09
10.24 1 0:35
3,39
0.30
Calibration Method
Fedilyn
Acar
2011
FedNova
Wang et al
20200N
Acc
SCE:
Uncal
35 3 1.0
14:25
1::5-11177 133
4.59 = 11.6
35.43
1.29
11:25
11::
6.14 (1:4
Rogil et slall 2017
$7.64
3:
1.50
17.00 2: 1.05
4.50
8 1.68
50.65
1.42
14.51
4: 1.12
4.61 2: 0.74
LS Mueraaat TUT
57.62
3: 1.55
18.57
1.39
0.64 0: 1171
54.25
1.09
16:11
0 1185
4.77 8 0.99
BS Taterr 19500
37.83
1.57
16.05
2: 1.18
5:23
= 0.81
56.26
1.20
16.77
2: 0.81
4.73
- 0.52
MMCEE
2015
36.15 4: 1.46
10.27
1.08
3.54
= 0.40
56,99
1.55
17:32
2: 0.30
6.18 0.64
96.95
1.60
19
11000000 0:95
0.6
17:1
6.17 0.99
10
1.06
0.79
16
35
12.55 0.48
0.97
1533 1.39
3:21
NUKTL
38.25
0.42
0.77
14:72
NUCFL (DCAIL-CRA)
38.19
0.45
1421 1
4.56
0.48
(DCA-L-CCAA
14.25
NUCHL (MUCA-KIP CKA)
1.10
9.95
16.13
4,5
Table 22: Accuracy (%), calibration measures ECE (%), and SCE (%) of various federated optimization methods with different calibration methods under non-lID (a = 0.05) scenario on the CIFAR-100 dataset. Underlined values indicate the best calibration across all methods.
25

---

Published as a conference paper at ICLR 2025
Dutasel
Calibration Method
Ace
ECF
IDD
SCF.
Acc
Non-IID (r =
PCF
0.5)
SCI.
Ace
Now-IID in = 6..1
ECE
SCF. L
Agc
Now-IID (n
FCF
0305)
SCR
UECaL
96:20
0.38
15.022
0.65
92.68
1.04
87.31
6:22
1.85
MNIST
CUCAL
96.90
0.44
0.16
95.97
0.78
0.31
92.41
1.32
0.66
87.70
5,20
1.59
UsCal
92.45
4,10
1.06
90.95
4.17
1.66
$8..1
5.58
2,83
86.38
8.7%
3.49
FEMNIST
FodCal
92.48
3,92
1.66
90.95
4.02
1.62
88.91
5.01
2.62
86,98
8.00
3,42
NICHLIDCAAILCOO
3,49
1.77
7.84
UaCal
50.3
7,60
2.70
78.33
9.59
3.30
61.86
12.62
4.42
39.81
14:30
4.50
CIFAR-10
FedCal
80.23
7.55
2.70
78.83
8.27
3.19
64.86
11.39
4,40
59.84
14.07
4.58
MUCCR/DCA=LLCCO
81.85
5.77
1.75
78.66
7.06
2.60
64.77
8.43
3.12
60.30
10.62
3.92
USCaL
7.61
61.34
3.61
58.34
3,80
14.00
CIFAR-100
7.01
3.49
58.34
3,80
13:39
NUCFLICA--LCKA)
65.43
5,98
2.99
62.05
6.14
3.07
60.73
9.14
3,50
57.03
10.34
3.52
Table 23: Comparison with SOTA baseline (FedCal) on different datasets using FedAvg under different data distribution. The average ACC and calibration error reported. Our method outperforms the bascline in terms of calibration error and sometimes in average accuracy as well.
B.3 MITIGATING OVER/UNDER--CONIIE CONFIDENCE
In addition to the over-confidence examples discussed in Section we also observe cases of under- confidence FL algorithms. Figure and presents the reliability histograms for non-llI (a 0.5) FedAvg using the MNIST and FEMNIST datasets, where the results show gap that exceeds the diagonal line, indicating under-confidence issues. While over-confidence is often viewed as more problematic in deep neural networks for real-world applications, under-confidence can also pose challenges in specific scenarios. For example, in medical diagnosis scenario, a threshold of 0.95 for negative predictions means that any instance classified as negative with probability I 0.95 requires manual review. An under- confident model may classify an instance as negative with a probability of 0.51, suggesting true probability of correctness that is at least 0.51, yet this does clarify whether it surpasses the threshold. This leads to compulsory manual inspection regardless of the actual correctness, thus increasing the workload. Among all the figures, we that our proposed method effectively calibrates FL and exhibits smaller gap (red region), demonstrating improved performance over other approaches.
B.4 COMPARISON WITH FEDCAL.
This section presents additional experimental results and comparisons with FedCal (Peng et al. 2024) from Section 5.2 Table 23 provides complete comparisons across different datasets. The results in Table 23 indicate that our proposed NUCFL outperforms FedCal. FedCal improves local client calibration using a local scaler design and aggregates these scalers to create a global scaler for the global model. However, relying solely on an aggregated scaler from local clients can lower global ECE but may neglect crucial interactions between global and local calibration needs. Our method accounts for these interactions, mitigating biases toward local heterogeneity and more effectively meeting global calibration requirements.
B.5 MORE DIFFICULT SCENARIO
In addition to evaluating our method a cross-silo scenario where all clients participate simultane- ously, also assess its performance in cross-device scenario with partial participation. To accom- plish this, we divided the CIFAR-100 training set among M = 100 clients, each with a distinct data distribution. We then implemented FedAvg using ResNet-34, setting the client participation ratio to 10% for each round. This experiment aims to provide deeper insights into how our method performs under more practical conditions, where participation rates are typically low. Table 24 shows that in this practical scenario, our NUCFL continues to demonstrate its benefits. While the global model may not fully represent the global distribution due to the bias toward clients participating in the previous round, our approach can effectively minimize calibration bias towards current participants.
B.6 TRADE-OFF BETWEEN ACCURACY AND RELIABILITY
During this study, we observed two types of trade-offs between accuracy and reliability: the Cali- bration Method Trade-Oft and the FL Algorithm Trade-Off.
Calibration Method Off. In a centralized setup, there are two types of calibration meth- ods, train-time calibration and post-hoc calibration. Post-hoc calibration, which is often used for
26

---

Published as a conference paper at ICLR 2025
Calibration Method
HDD
Non-III (a
0.5)
Non-1ID (a.
(1.1)
Non-IID (G
0.05)
Acc
BCE
SCE
ECE
SCE
ECE
SCE 4
SCE 1
Uncal.
57.33
10:21
4.39
54.99
11.49
4,46
$1.34
14.02
4.70
49.35
18.30
5.97
DCA
57.61
8.21
3.00
55.10
9.28
3.41
51.24
11.02
3.61
49.29
14.13
4.76
MDCA
57.49
9.40
3.52
55.23
10.33
3.64
51.63
12.39
3.82
49.61
14.66
4.81
FedCal
57.33
9.31
3.49
54,99
9.49
1.46
51 11
10.93
3.60
49.33
13.91
4.74
NUCTLIDCA+L-CKAN
2.96
3.37
51.93
49.29
NUCFLIMICCAL-CCRA)
57.21
8.47
3.17
55.39
9.44
3.46
51.49
10.02
3,51
49.69
12.79
4,06
Table 24: Comparison under partial participants scenario (10% clients in each round) using FedAvg the CIFAR-I00 100 dataset, where representative calibration methods are selected for comparison. Improved calibra- tion error shows that our method significantly enhances model calibration.
Calibration Method
FedSpeed (Sun
2023
FedSAM (Qu
FedMR
(Hu
FodCross
et
Aoc
ECE
SUB4
Apc 1
BCE
SCET
Acc 7
ECB
SCE.
Acc
ECE
SCET
UnCal.
65.10
13.33
1.11
64.13
13.95
6.92
64.93
13.18
6.70
63.30
15.81
7.19
FedCal
12.95
12.08
64.95
12.85
6.49
NUCHLDCA+4LCKAN
65.17
11.15
5.92
64.52
10.22
5.55
64.95
12.11
6.47
63.33
13.01
6,53
Comparison using other FL optimization methods shows that method can adapt to any FL algorithm and improve calibration error.
Table 25:
centralized models, adjusts the calibration error without impacting accuracy. However, shown in Table 6 when we examine train-time calibration methods in a centralized context, we observe that in over 25% of cases, these methods improve calibration the cost of reduced accuracy, unlike the stable post-hoc methods. But given that train-time methods are more practical for FL due to the lack of post-hoc dataset, we take this concern into consideration. We found that existing train-time cali- bration methods tend to degrade accuracy in FL settings, although they vary in their effectiveness at reducing calibration error. This observation further guided our goal to design calibration method that could improve reliability without compromising accuracy.
FL Algorithm Trade-Off. Another trade-off we identified is between accuracy and reliability across different FL algorithms. While advanced FL algorithms, such as FedDyn and FedNova, can achieve higher accuracy than naive FedAvg, they also tend to introduce greater calibration error. For exam- ple. in Table the accuracy (ACC) for FedAvg is 61.34, while FedDyn and FedNova reach 62.39 and 63.01, respectively. However, their calibration error also increases from 10.52 in FedAvg to 12.53 in FedDyn and 11. 1..45 in FedNova. With the help of our method applied FedDyn and Fed- Nova, we improve their accuracies to 62.84 and 63.17, respectively, while significantly reducing their calibration error 10.24 and 8.01-both lower than the original FedAvg calibration error of 10.52. This trend is also found in other datasets and data distributions, as shown in our result tables in Appendix B.2 This shows our method's s effectiveness in improving reliability without sacrificing accuracy across various FL algorithms.
B.7 COMPARISON WITH FL OPTIMIZATION METHODS USING "CALIBRATION"
While searching for related work, we found that some FL studies Zhang et al. 2022): Luo et al. (22TT) use the term "calibration." Although these works do not focus on "model calibration" as discussed in this paper, we consider them advanced FI. optimization methods and evaluate their performance in terms of model calibration. Specifically, we evaluate both the uncalibrated versions of FehLC al. (2022) (with r 1) and CCVR al. (202I) (on FedAvg) as well their calibrated counterparts by integrating our proposed NUCFL. We follow the same experimental setup as described in Table utilizing the CIFAR- 100 dataset under non-IID (a 0.5) data distribution
scenario. The results are provided in Table 27
Comparing uncalibrated CCVR and FedLC with FedAvg, observe increase in accuracy along- side a rise in calibration error. This indicates that these algorithms are advanced FL. optimization methods designed primarily to improve accuracy rather than address model calibration. Since view CCVR and FedLC as advanced FL optimization methods, we can apply NUCFL to them.
local epoch =
local epoch E 5 (default) FCF.
local epoch E 10
Calibration Method
Ace f
ECF.
SCF.L
Acc t
SCF.
Acc 1
ECE 1 SCF
UnCal. NUCFLDCA+-LCKA))
59.23
8.92
3.47
61.34
10.52
3.61
62.00
11.39
3.98
59.47
6.40
3.29
62.05
6.14
3.07
62.05
6.08
3.00
Table 26: Calibration performance with different local epochs using FedAvg under non-IID conditions (a =
0.5 on CIFAR-100 dataset.
27

---

Published as a conference paper at ICLR 2025
FedAvg
CCVR
FedLC
Calibration Method
Acc
ECB
SCE 4
Acc f
ECB
SCE
Acc f
ECE 1
SCE J
UnCal.
61.34
10.52
3.61
63.01
13.69
4.99
62.73
14.58
6.11
NUCFLIDCA+L-CKA)
62.05
6.14
3.07
63.05
9.28
3,53
62.77
10.98
3.79
Table 27: Calibration performance of FL algorithms incorporating "calibration."
Comparing the original versions with their NUCFL-calibrated counterparts, observe improve- ments in both accuracy and calibration error. This demonstrates the adaptability of our method and its compatibility with any FL algorithm.
Algorithm 2 Applying NUCFL to FI.
I: Input: global model local model wm. for client m, local epochs E, and rounds T.
for each round
...,? do
3: Server sends w (t-1) to all clients.
4: for each client
M do
Initialize local model (4.0) < (t
6: for each epoch e 1, ..., E do
Get the gradient for each client: 8(t (t-
win.
Calculate similarity with previous accumulated gradient: An sim(8(t-1), ((
Decide calibration loss function lear based on DCA (6) or MDCA 7
Each client obtains calibrated local loss Cscl (wih-)) based on
11:
Win (t.e)
ClientOPTrui (t.e-1), w.c.mwww (w((:)))
end (C.E)
13:
Wm denotes the result after performing E epochs of local updates.
14:
Client sends 6 = (t-1)) (t.E) to the server after local training.
15:
end for
16: Server computes aggregate update g(t) EmEM PD (()
Server updates global model (t) <- ServerOPT 5(1)
18:
end for
28


Reviews:
[{"review_id": "TRHVB8k67I", "reviewer": "Reviewer_pbym", "summary": "The paper addresses the issue of model calibration in federated learning (FL), with a specific focus on heterogeneous scenarios, characterized by distribution shifts across clients. The paper argues that most works have focused on improving the accuracy of the proposed methods so far, but almost no attention has been given to the model's confidence in its predictions. This highlights a gap in the current literature: FL models should hava reliable confidence in their predictions to be deployed in real-world use cases. To fill this gap, this work introduces Non-Uniform Calibration for Federated Learning (NUCFL), a novel method to integrate FL with model calibration, that can be easily combined with existing approaches.\n\nThe work discusses the various benefits of NUCFL, studying its application with several calibration approaches and FL methods, showing its effectiveness in improving the model's confidence in its predictions, according to multiple metrics.", "questions": "- Could the authors provide a comparison between NUCFL and Zhang et al. [2022] and Luo et al. [2021], according to the introduced metrics?\n- Does NUCFL increase computational costs on the client-side?\n- How could NUCFL be adapted to unsupervised/semi-supervised settings?\n- How sensible if NUCFL to the hyperparam $\\beta$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper underlines a gap in the current FL research and proposes an effective method to address it\n-  NUCFL can be easily added on top of existing methods\n- The paper extensively shows the efficacy of NUCFL across various settings, model architectures and methods\n- NUCFL is extensively studied through in-depth and well-carried analyses\n- NUCFL does not increase communication costs", "weaknesses": "- Difficulties in proving NUCFL's convergence and theoretical properties, as also pointed out by the authors\n- The paper argues that works like Zhang et al. [2022] and Luo et al. [2021] misuse the term \"calibration\" and do not further analyze the comparison with those methods. However, I believe it would still be relevant to the FL community to understand how NUCFL compares to them. \n- The application of NUCFL seems limited to supervised scenarios\n\nComments (did not affect my rating):\n- I believe the paper would benefit from the outlined Algorithm of NUCFL for better understanding\n- Please modify the citation style to improve the paper's readibility (the citation are not enclosed in brackets)\n- Line 136: typo \"..\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jE6PLxd7JB", "reviewer": "Reviewer_kU7Z", "summary": "This paper presents a model calibration method for federated learning, which adaptively adjusts penalties based on the relationship between local and global models.", "questions": "Please see the weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The motivation makes sense.\n\nThe paper is well-written and easy to follow.", "weaknesses": "The contribution of this paper is limited. The proposed method seems a simple incremental work of the existing FL model calibration method.\n\nAlthough I understand the author's motivation, I need the author to explain why the proposed metric is more important than accuracy. The proposed metrics are not intuitive enough. Typically, accuracy and the proposed metric are positively correlated. The author should add some detailed experiments to validate its motivation.\n\nThe experimental results show that The proposed method is only slightly optimized for federated learning. In addition, the proposed method may cause a decrease in accuracy in some scenarios.\n\nLake of comparison with SOTA FL optimization methods [1-5]. These methods achieve better model performance by guiding the model to optimize towards the flat region. I think these methods can still reduce the measures proposed by the author. I wonder if the author can compare these methods or integrate the proposed method into them.\n\n[1] FedSpeed: Larger local interval, less communication round, and higher generalization accuracy ICLR 2023\n\n[2] Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination, KDD 2024\n\n[3] Generalized Federated Learning via Sharpness Aware Minimization, ICML 2022\n\n[4] Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape. ICML 2023\n\n[5] FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation, ICDE 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C63dMmdd46", "reviewer": "Reviewer_KxQB", "summary": "This paper aims to ensure the reliability of FL in real-world applications. It proposes Non-Uniform Calibration for Federated Learning (NUCFL) that integrates FL with model calibration to achieve the goal. NUCFL dynamically adjusts the model calibration objectives by measuring the relationships between the local and the global model. Experiments show the superiority of NUCFL compared with existing FL methods.", "questions": "N/A", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "[1] The paper takes an important step to explore FL from the perspective of model calibration, which is important to various decision-making scenarios. \n\n[2] Extensive experiments are carried out to show that NUCFL can seamlessly integrate with existing FL methods and improve the original performance of those methods.", "weaknesses": "[1] It is unclear whether there is a trade-off between accuracy and reliability. If so, please give a detailed illustration of this aspect. If not, the advantage of not sacrificing accuracy seems an irrelevant contribution. \n\n[2] The relation between model calibration and enhancing the confidence of FL outputs is weak. It requires further demonstration and analysis of why model calibration can realize that. \n\n[3] It is better to provide a further investigation into how existing centralized calibration methods can adapt to the FL setting, for example, simply combining them with FedAvg framework. Empirical studies can also involve this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BLByOgj4uz", "reviewer": "Reviewer_zKQ7", "summary": "This paper contributes to a well-established line of research that adapts standard machine learning (ML) techniques for federated settings by leveraging the similarity between global and local models to weight client contributions. Here, the ML technique under adaptation is model calibration, achieved through an auxiliary loss function. The paper’s primary contribution is a proposal to weight this auxiliary calibration loss based on a customizable similarity metric between global and local models, such as cosine similarity or other advanced measures. In this context, the “local” model refers to the locally trained replica of the global model. The authors provide an extensive empirical evaluation on standard federated learning datasets for computer vision, using LDA to generate non-IID partitions for IID datasets and some naturally heterogeneous datasets as well (FEMNIST). Their results indicate improved performance over baseline methods across all datasets, models, and levels of data heterogeneity.", "questions": "1. Can the authors provide any insight on how would the proposed approach perform under different local training conditions? For example with minimal local training (≤ 1 epoch or with very low learning rates) or extensive local training (e.g., training the local model to convergence or near convergence every round).\n2. Have the authors considered the use of auxiliary local models or personalized layers, to compute similarity? \n3. Why did the authors decide to use raw similarity/relevance values rather than a softmaxed version with a normalization factor constructed based on the entire distribution of client similarities?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Comprehensive Evaluation: Experiments are extensive and demonstrate consistent improvements across datasets, models, and non-IID settings.\n- Writing and Presentation: The paper is well-written, with clear diagrams and formatting that aid readability.\n- Practical Relevance: The method holds strong practical implications, bringing effective calibration methods to FL.", "weaknesses": "I must mention that I am much more familiar with FL than model calibration and thus focus on this side of the work. \n\n- Limited Novelty in Core Insight: Although the similarity-based weighting approach is practical, it is not particularly novel, e.g.,[1,2,3]. The authors do acknowledge the influence of prior FL works that leverage model similarity metrics. \n- Potential Sensitivity to Hyperparameters: The method might be sensitive to local training hyperparameters like learning rate and epoch count, which could impact the similarity measure’s effectiveness. In cases closer to FedSGD with only one or a few steps performed, data heterogeneity may minimally impact the model replica, potentially diminishing the relevance of the similarity function. Similarly, scenarios involving extensive local training or very high learning rates may affect the similarity measure’s utility as all models may diverge very far from the global model regardless of the degree of local data heterogeneity. \n- No Exploration of Alternative Similarity References: Using only the client’s local model replica as the similarity reference could limit the applicability; other approaches (e.g., auxiliary persistent local models or specific personalized layers) might better capture client-specific characteristics.\n\n\n\n\n[1] Tian Li, Shengyuan Hu, Ahmad Beirami, Virginia Smith:\nDitto: Fair and Robust Federated Learning Through Personalization. ICML 2021: 6357-6368\n\n[2] Tao Yu, Eugene Bagdasaryan, Vitaly Shmatikov:\nSalvaging Federated Learning by Local Adaptation. CoRR abs/2002.04758 (2020)\n\n[3] Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, Yong Zhang:\nPersonalized Cross-Silo Federated Learning on Non-IID Data. AAAI 2021: 7865-7873", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)