Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Authors: Hongyao Chen
Tianyang Xu
Xiaojun Wu
Josef Kittler
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Batch Normalisation
Distributed Learning
Data Heterogeneity
Primary area: optimization->large_scale_parallel_and_distributed

Abstract: Batch Normalisation (BN) is widely used in conventional deep neural network training to harmonise the input-output distributions for each batch of data.
However, federated learning, a distributed learning paradigm, faces the challenge of dealing with non-independent and identically distributed data among the client nodes. 
Due to the lack of a coherent methodology for updating BN statistical parameters, standard BN degrades the federated learning performance.
To this end, it is urgent to explore an alternative normalisation solution for federated learning. 
In this work, we resolve the dilemma of the BN layer in federated learning by developing a customised normalisation approach, Hybrid Batch Normalisation (HBN). 
HBN separates the update of statistical parameters (*i.e.*, means and variances used for evaluation) from that of learnable parameters (*i.e.*, parameters that require gradient updates), obtaining unbiased estimates of global statistical parameters in distributed scenarios. 
In contrast with the existing solutions, we emphasise the supportive power of global statistics for federated learning. 
The HBN layer introduces a learnable hybrid distribution factor, allowing each computing node to adaptively mix the statistical parameters of the current batch with the global statistics. 
Our HBN can serve as a powerful plugin to advance federated learning performance.
It reflects promising merits across a wide range of federated learning settings, especially for small batch sizes and heterogeneous data. 
Code is available at https://github.com/Hongyao-Chen/HybridBN.

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Hongyao Chen
Tianyang Xu
Xiao-jun Wu
Josef Kittler
Abstract
+ FedAvg-BN
=FodAvg-GN 60
+ =FedAvg+HBN
60
Batch Normalisation (BN) is widely used in con- ventional deep neural network training to har-
57
57
[irror(%))
monise the input-output distributions for each batch of data. However, federated learning, a distributed learning paradigm, faces the challenge of dealing with non-independent and identically distributed data among the client nodes. Due to the lack of a coherent methodology for updating BN statistical parameters, standard BN degrades the federated learning performance. To this end, it is urgent to explore an alternative normalisa- tion solution for federated learning. In this work. resolve the dilemma of the BN layer in fed- erated learning by developing customised nor- malisation approach, Hybrid Batch Normalisation (HBN). HBN separates the update of statistical pa- rameters (i.e,, means and variances used for eval- uation) from that learnable parameters (i.e., pa- rameters that require gradient updates), obtaining unbiased estimates of global statistical parameters in distributed scenarios. In contrast with the exist- ing solutions, we emphasise the supportive power of global statistics for federated learning. The HBN layer introduces learnable hybrid distribu- tion factor, allowing each computing node to adap- tively mix the statistical parameters of the current batch with the global statistics. Our HBN can serve as a powerful plugin advance federated learning performance. It reflects promising merits across wide range of federated learning settings, especially for small batch sizes and heterogeneous data. Code is available at https: /tttps:/ggtitii github. com/Honge Hom/Hongyaa--oo
54
51
51
48
48
IID
0.3
0.1
32
16
8
Heterogeneity (a)
Batch size
(b)
Figure 1: comparison of Hybrid Batch Normalisation
(HBN) with standard Batch Normalisation (BN) and Group Normalisation (GN) in the federated learning settings. (a) Classification error rate of Simple-CNN on CIFAR-100 vs Data heterogeneity (controlled by a Dirichlet distribu- tion coefficient). The batch size is 16. (b) Classification error rate of Simple-CNN on CIFAR R-FAR-100 vs Batch size. The Dirichlet distribution coefficient is 0.6. For other imple- mentation details, please refer Section 4.1.
1. Introduction
Federated learning (FL) is a distributed machine learning paradigm that trains a global model on a central server, while protecting the raw data of each client node (Yang et al., 2019; Bharati et al., 2022). Different from data-centralised training, in FL. data across client nodes may be non- independently and identically distributed (Non-III) (Zhao et al., 2018; Li et al., 2019; 2022), which has sparked widespread interest in how to achieve the system-centralised performance during distributed training.
To guarantee data privacy, only the model parameters, rather than data, are allowed to transferred between the central server and local clients. Specifically, the classic Feder- ated Averaging (FedAvg) family (MeMahan et al., 2017) performs multiple mini-batch stochastic gradient descent (SGD) on several local clients, and then aggregates the model parameters on the server in each communication round.
'School of Artificial Intelligence and Computer Science, Jiangnan University. Wuxi, China. "School of Computer Sci- ence and Electronic Engineering and the Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford, GU2 7XH, UK.. Correspondence to: Tianyang Xu <tianyang, xu jijannan.cdu.cn>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
harmonise the use of global model parameters in con-

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
junction with client specific data distributions, a suitable Batch Normalisation (BN) (loffe, 2015) is required. In the case of centralised training, samples from different batches are rescaled and shifted to predefined range by the BN layer, reducing potential supervision conflicts. In general, the funcciin of BN is to accelerate convergence (Bjorek et al., 2018; Karakida et al., 2019), to smooth the optimi- sation landscupe (Santurkar al., 2018; Yong et al., 2020: Peng et al., 2023), and to improve generalisation (Luo et al., 2018; Lubana et al., 2021), etc. However, (Hsieh et al., 2020) and (Du et al., 2022) found that the BN layer does not maintain its expected merits in the FL scheme. The underlying reason is that each local client works with a spe- cific data distribution, resulting in heterogeneity from the global perspective. The difference between the local statis- ties (for training) pertaining to each client and the global statistics (for evaluation) compromises the effectiveness of the standard BN layer.
the global statistical parameters. While the use of historical global statistical parameters for BN in decentralised training is precarious, we suggest using them to recognise heteroge- neous nodes instead. Our approach demonstrates that this idea leads to an efficient solution for adapting the global statistical parameters of BN, so as to mitigate the negative impact of the distribution heterogeneity.
Recognising that the negative impact produced during the update stage of statistics, where the local statistical pa- rameters and learnable parameters are jointly updated, we propose to separate the update stage into sequential steps, i.e, collecting the local statistical parameters, followed by updating the learnable parameters. In this way, our sequen- tial update strategy recovers unbiased estimates of the last round's global statisties, without explicitly changing the communication transmission process. In the subsequent nor- malisation stage, we perform Hybrid Batch Normalisation (HBN) to balance the global statistical parameters and the statistical parameters from the current batch. In Figure 1, we compare HBN with standard BN and GN configured with the FedAvg baseline. Intuitively, HBN outperforms its contenders regardless of the actual degree of heterogeneity and batch size, achieving consistent normalisation for fed- erated learning in diverse scenarios. We demonstrate that the proposed HBN can be used effectively in conjunction with several federated learning frameworks, resolving the dilemma of batch normalisation, and delivering consistently outstanding performance.
To effectively avoid collecting the statistics from each batch in FL. (Hsich et al., 2020) suggests replacing BN with Group Normalisation (GN) (Wu & He, 2018), which performs a normalisation along the channel dimension. (Du et al., 2022) suggests using LN (Ba, 2016). Besides, (Kim al., 2023; Zhang et al., 2024) identifies that Feature Normali- sation(FN) can be helpful for federated training. Similar to GN, FN and LN both use the statistical information of a single sample for normalisation, ignoring the dependency relationships among samples. Although these normalisation methods avoid batch dependence by relying on individual samples, they fail to incorporate global information, which may further exacerbate data distribution heterogeneity.
The main contributions of our work are as follows:
We address the dilemma of batch normalisation for the federated learning framework, and show how to derive unbiased global statistical parameters from inconsistent local distributions.
Many studies (Ioffe, 2017; Yao et al., 2021; Pham et al.,
2022) have found that the degree of heterogeneity and the batch size are the two dominant factors causing the global/local discrepancy for BN. As shown in Figure 1, when the heterogeneity of the data distribution is severe (with smaller Dirichlet distribution coefficient) or the batch size is tiny, BN deteriorates because of the lack of confi- dence in the local statistics. In the decentralised training scenario, global statistical information is crucial for ensur- ing local consistency and synchronisation. At this point, normalisation based on global statistics is more appropriate. However, obtaining real-time global statistics is difficult in FL. Drawing on this, FedTAN (Wang et al., 2023) proposes to correct the potentially unreliable statistical parameters of BN. However, FedTAN requires more client-server com- munications to rectify the statistics, sacrificing x3 times as many BN layers in the communication rounds, compared to FedAvg.
We develop hybrid batch normalisation (HBN) layer, harmonising local batch statistics and global statistical parameters for each local client.
We demonstrate the superiority of HBN by extensive experimental comparisons with several existing FL nor- malisation solutions. The experimental results show that HBN exhibits excellent merits under heteroge- neous and small batch sizes scenarios.
We verify the consistent compatibility of HBN with existing federated learning schemes and classic deep neural network architectures.
2. Related Work
To date, the potential of BN in handling FL tasks has not been sufficiently investigated. One challenging issue is how to gauge the client distribution heterogeneity, in relation to
Learning Model Parameters in Federated Setting, Com- pared to traditional distributed training, data heterogeneity among various local clients is fundamental challenge in
2

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
federated learning (Kairouz et al., 2021; Vahidian et al., 2023; Mendieta al., 2022). Due privacy constraints, local clients cannot address this challenge by sharing data.
SiloBN (Andreux et al., 2020) limit certain BN parameters each local client. The above approaches avoid relying server-side statistical parameters and focus exclusively on local client normalisation, which are suitable for person- alised federated learning but incapable of obtaining a global model.
Since FedAvg (McMahan et al., 2017) was proposed to aggregate model parameters from multiple local clients, setting up a baseline for federated learning, recent stud- ies typically focus on handling heterogeneous client data distributions. For instance, to alleviate oscillated aggrega- tion at the central server end across different iterations, Fe- dAvgM (Hsu et al., 2019) proposes to introduce momentum in the aggregation stage, aligning the entire model param- eters. To achieve adaptive moment estimation (Kingma, 2014), FedAdam (Reddi et al., 2020) was developed to as- sign a suitable combination ratio between the previous and current parameters. To stabilise the local training phase, Fed- Prox (Li et al., 2020) adds a proximal regularization term to local models. In terms of explicit anti-heterogeneouu mod- elling, Scaffold (Karimireddy et al., 2020) reduces the vari- ance of local updates by introducing a correction term for local gradients. In order to further collectorate the local and global models, FedCM (Xu et al., 2021) and FedSAM (Qu et al., 2022) introduce client-level momentum and client- level optimiser to enhance the generalisation power at the local end.
To achieve server-client statistical parameters interaction and aggregation within the batch scope, recent studies have explored how to optimise BN in the FL setting. FixBN (Zhong et al., 2023), exploiting straightforward strategy. proposes a two-stage training approach, where the statistical parameters are updated in the first stage and frozen to replace batch statistics during normalisation in the second stage. To involve historical statistics, Feder- ated Batch Normalisation (FBN) (Guerraoui et al., 2024) proposes to normalise batch data using rumning average. However, FixBN and FBN neglect the diversity of statistics in each local client, which is a challenging issue in Non-III scenarios. To simultancously maintain the local data char- acteristics, FedTAN (Wang et al., 2023) uploads the local means and variances layer by layer to the server. Then, the global statistical parameters are sent each client to update the local learnable parameters. Despite the improved ac- curacy in collecting statistics, FedTAN sacrifices 3x times the communication cost for each BN layer compared to FedAvg.
The above solutions harmonise the server-client parame- ters from the updating perspective of learnable parameters, achieving promising performance against heterogeneous distributions. On the contrary, in this paper, we focus on alleviating the negative impact of heterogeneity by design- ing hybrid batch normalisation module, emphasising the update of statistical parameters, which can further boost the performance accompanied by the above solutions parallelly.
Although existing attempts optimise the normalisation al- gorithms for FL., the potential value of BN statistics is not sufficiently explored during training. On the contrary, in this paper, we propose to harmonise the BN statistics with unbiased estimator at the global server, which can endow global perspective to each local training phase, unifying the task supervision thereby.
Updating Statistical Parameters in Federated Setting. Normalisation based on input statistics has been widely studied deep learning, e.g., batch normalisation, layer nor- malisation, and group normalisation. However, endowing federated learning solutions with the above normalisation ap- proaches cannot deliver the expected improvement, remain- ing a concern on how to collect and update the statistical parameters in federated learning. For instance, batch normal- isation relies on batch statistics to normalise data, which can be easily misled by the Non-IID clients' data (loffe, 2017). Drawing on this, Group Normalisation and Layer Normali- sation are used for federated learning, avoiding collecting statistics across the batch dimension. Feature Normalisa- tion alleviates the representation norm differences between federated learning clients, which is functionally similar to LN. However, GN, LN, and EN rely only on the statistics of each individual sample, representing significant difference compared global statistics, which cannot maintain their advantages in heterogeneous FL. To preserve the statisti- cal stability at the client end, FedBN (Li et al., 20) and
3. Approach
3.1. Formulation
Batch Normalisation. Batch Normalisation is widely used in deep neural networks to reduce internal covariate shift (loffe, 2015). Given input feature {a; [ of batch size B, BN calculates the means Ae and variances of of the current batch and normalises them.
B
B
1
Ph
0
B
B
1=1
1=1
In general, BN estimates global statistical parameters for evaluation by employing the exponential moving average to update the running statistics.
In the current deep learning community, the estimated sta- tistical parameters from the training phase are frozen in the evaluation phase. The vitality of these statistical parameters
3

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
holds only if the data share a similar distribution. Intu- itively, independent and identically distributed (II) data is expected. However, this assumption always conflicts with the federated learning setting, where heterogeneous data distributions are assigned to different local clients.
Algorithm 1 Workflow of Federated Learning with HBN
Input: the number of communication rounds T; the num- ber of clients K; the datasets of clients (ER) the datasize of clients [eK]] the initial global model parameters {wg, H3 _ for the server; the initial hybrid distribution factor {ax- 0} for each client k;
Federated Learning with Batch Normalisation. In the context of federated learning, handling non-III data among local clients a typical challenging issue. A direct solution is passing data, which is forbidden in the federated learning setting. Therefore, other local clients can access only model parameters to protect data privacy. Taking the classic Fe- dAvg as an example, the goal is to learn a global model that can perform well on all clients. Specifically, assuming that set D3 = {Dx } ]ε[K]] contains the datasets distributed across all K clients. Considering a deep neural network with the BN layer, the model parameters include learnable parameters w and BN's statistical parameters, which are means and variances o². FL aims to minimise the follow- ing empirical risk:
for ( 1 7'+ do for each client & in parallel do download {Wg, Hg. 02 from server;
without backpropagation
compute / (o²) according Equation (7);
I| with backpropagation
wh: of + SGD((W), Mg: o az) ax PD));
upload (uR, of ' to server:
end for
compute H4g (o²){ according to Equation (8);
if = T + 1 then
set T+1 =
++ =
else
update wt Σt= K end if end for
No wh:
K E C,(w.,, k=1
No N
min (w)) w.p.a?
Output: (Wg, Ug: ozj
where N₂ is the size of the k-th client dataset DR, N is the total size of data and (-) is the empirical risk of the k-th client. However, the above Equation (2) cannot be directly optimised because the central server cannot access the client data. For this purpose, FL iteratively aggregates the model parameters trained locally by the clients. On both global learnable parameters Wi and global statistical parameters lg og. FedAvg performs the following operations:
This method obviously cannot accurately approximate ideal global statistical parameters. We provide a more detailed derivation in Appendix A.1. Firstly, the main reason is that the calculation of local statistical parameters is not based the same global model, but on each local models, which would lead to external covariate shift (Du et al., 2022). Sec- ondly, the direct average local variance may deviate from the actual global variance, and this error may accumulate during the forward propagation of deep neural networks. Furthermore, even if we restore the ideal statistical param- eter estimation of BN, these statistical parameters cannot guarantee positive impact on federated training. During lo- cal training, they still use batch data statistical parameters to normalise the input, losing awareness of global distribution. Driven by these issues, we propose a novel BN solution, Hybrid Batch Normalisation (HBN), for federated learning.
K Nu (wk: Pk. o
(ulg Mg: o2))
Σ
N
where Wki ue and of are all updated by the k-th client local data.
The Dilemma of Batch Normalisation. Next, we focus on
analysing the statistical parameters ur, o We use
nr, 0² S(w; D),
(4)
to represent the statistical parameters μr, 0² obtained by the model parameters w on input data DD
3.2. Global Server End: Obtaining Unbiased Statistical Parameters
The dilemma of BN layer in federated learning mainly lies in our expectation of obtaining ideal global statistical param- eters igg de during the training phase, which can be used for evaluation, i.e..
To harmonise the distribution, obtaining an ideal global sta- tistical estimation is a prerequisite for our hybrid batch nor- malisation. In order to ensure that the statistical parameters are not affected by the model (learnable parameters) trans- formation during each client training process, we propose an update strategy that separates the statistical parameters from the learnable parameters. As shown in Algorithm 1, (6) after downloading the global model from the server, the local client calculates the statistical parameters of the local
igg 02 = S(wg; Dz).
(5)
Unfortunately, previous methods approximate this process by averaging the local statistical parameters, i.e.,
K
Nk N
Mg. og =
S(wp: Dx) + Slwg; Dg).
4

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
(a) Original Clusters
(b) Local Normalisation
(c) Global Normalisation
(d) Hybrid Normalisation
Cluster
2.0
Cluster
35-
03
-03
-15
-1.
20
-15
-10
2.0
Figure Normalisation methods for two toy FL clusters.
data based on the current global learnable parameters before starting training. So in the t-th round, client-k: can get:
each cluster separately using local statistics, the normalised data exhibits indistinguishable overlap compared globally normalised data. Most local normalisation methods (BN, GN, LN. etc.) cannot produce satisfactory results in such cases. Compared to local normalisation, global normalisa- tion can better maintain consistency among local clients. However, real-time global normalisation is impractical, par- ticularly in federated learning, where models undergo mul- tiple local updates before communication. This means the ideal global statisties inherently dynamic. We usually only have access to historical global statistics, whose time- liness is constrained by intermittent communication and partial client participation. Using only historical global statistics will yield a suboptimal result. But historical global statistics still retain valuable global structural information. Especially since we have addressed the dilemma of global statistical parameter bias for BN, can these ready parameters more efficiently. Therefore, we propose hybrid normalisation method that is more suitable for handling fed- erated learning tasks, coined as hybrid batch normalisation (HBN). As shown in Figure 2 (d), this hybrid normalisation combines global statistics with local statistics, which can standardise the size of two clusters while maintaining the global structure.
After obtaining the statistical parameters of the last global model, each client optimises the local learnable parameters based on standard stochastic gradient descent (SGD) with mini-batch. Usually, if local statistics are independent and randomly sampled from the entire data, then averaging them is reasonable, but in federated learning, this assumption does not hold. Drawing on this, we consider client data as a certain group of the entire population. According to distributed statistical analysis, we can use the following approach calculate unbiased estimates of the population mean and variance:
N N
NR [0 k + (1h - -m
(0
-
By separately calculating local statistical parameters, the server can obtain ideal global statistical parameters s(w(-1;2, Dg). We provide the detailed derivation in Ap- pendix A.2. It should be noted that after using this asyn- chronous strategy to update between the statistical param- eters and the learnable parameters, we need perform an additional round of statistical parameters update for syn- chronisation at the end.
Specifically, HBN adds a hybrid distribution factor in the base BN layer to adaptively learn the balance between batch and global normalisation. Typically, the hybrid distribution factor a is a learnable parameter with the same size as the number of input channels. For layer input { over a mini-batch, calculate the normalisation parameters for each input follows:
)
3.3. Local Client End: Performing Hybrid Batch Normalisation
A 1 44b +
1 1te 1
(9)
e a
In heterogeneous FL, local client reflect only their local data distribution, often differ- ing substantially from the global distribution. As shown in Figure 2, two toy clusters follow Gaussian distributions, simulating the activation output of two clients in federated learning. Intuitively, after applying local normalisation to
sampled by each
82 = 1tea of 1te 02
where Pb. of are the current batch statistics calculated ac- cording to Equation (1), and M4: o² are the global statistics stored in HBN according to Equation (8). In particular, the global statistic frozen during the gradient update process.
5

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Like other normalisation methods, HBN also learns a per- channel linear transform to compensate for the possible loss of representative power:
CIFARIO ( 0.6)
CIFARIO ( = 0.1)
400
Closs Cabe
100
y/ 7 Tj +3=
# H.N.
(10)
100
Cloos
750
200
293
where é is a small positive constant that prevents the de- nominator from being zero and y, f are trainable scale and offset.
Client
Client
Figure 3: Different Dirichlet coefficients (006, 0.1) label distribution on CIFAR-10 with 100 clients.
Due to the varying degrees of distribution differences be- tween clients and the global server, the hybrid distribution factor is configured for each local client, which does not participate in communication. To clarify this process, in Algorithm I, we distinguish the hybrid distribution factor from other learnable parameters w. In the training phase, the learnable parameters and local statistical parameters are updated separately and do not affect each other. In the eval- uation phase, normalisation parameters use global statistical parameters {/g. o²) and do not require the participation of hybrid factor.
(GN), Layer Normalisation (LN), FedEN, FixBN, and Fed- erated Batch Normalisation (FBN), For fairness, we default to combining all normalisation methods with the baseline FedAvg.
Benchmark. All experiments are conducted on the classic image classification datasets, including CIFAR-10/CIFAL 10/CIARR 100 (Krizhevsky, 2009) and Tiny-Imagede with 200 classes) (Chrabaszcz et al., 2017), which are widely adopted in FL research (Chen & Zhang. 2024). We use a customised Simple-CNN network on CIFAR-IO/CIFAR- 100, which has a representativeness in computer vision do- main. Simple-CNN mainly consists of three convolutional blocks and two fully connected layers. Each convolutional block contains convolutional layer, normalisation layer, ReI.U activation layer, and a max-pooling layer (LeCun et al., 1998; Maas et al., 2013; Ioffe, 2015). We replace the normalisation layer with different normalisation methods, except for FN, which is added before the input of the classi- fication head. We also use the classic ResNet- 18 (He et al., 2016) on the large-scale dataset Tiny-ImageNet.
(100K images
3.4. Implementation
Comparison of parameters with BN. For HBN, we can easily implement programming codes based on BN. HBN needs to store both local and global statistical parameters, but only one of them needs to participate in each communi- cation round. Local statistics can be directly calculated each client, while global statistics are obtained by aggregat- ing local statistics on the server. In addition, we need to add trainable hybrid distribution factor parameters per channel and discard the moving average momentum. In the forward passing, HBN normalises the batch input in the order of Equations (1), (9) and (10).
Similar to existing works (Wang et al., 2024; Tian et al., 2024; Zhuang et al., 2024), we use Dirichlet distribu- tion (Hsu et al., 2019) to model the data distribution, which controls the degree of statistical heterogeneity through a coefficient Φb The smaller the o, the more severe the label distribution heterogeneity across clients becomes. As shown in Figure 3, we visualise the label distribution of 100 clients the CIFAR-10 with different d(0.6, 0.1).
Communication with stragglers. In situations where there a large number of stragglers, inevitably, we unable to calculate statistical measures for all data. We empirically recommend using the moving average to update the global statistical parameters for each server stage:
2/ (1 -))p + AuG: (o²)g = (1 - -)) + \(o2))
(11)
Hyper parameters. For CIFAR- 1O/CIFAR- WWWCIIFA-OOOTii-----.. 100/Tiny- ImageNet, unless otherwise specifie, we set the total num- ber of clients K = 100/1 the activation rate of each round of clients C = 1.0. (i.e., 10 clients partic- ipate in per round), the local training epoch E 1, the optimiser is SGD optimiser with a momentum of 0.9. The communication rounds T for CIFAR-IO/CIFAR ImageNet are 0000/1000000 respectively. For all algo- rithms, we select the appropriate initial learning rate from (0.02, 0.01, 0.005, 0.002, 0.001} when batchsize is 4. The learning rates decay quadratically with 0. 0.998. In particular, we adopt the linear learning rate scaling rule (Krizhevsky,
where E (0, 1] is the moving average momentum. For more stragglers, small momentum can provide more smooth estimates which is beneficial for training.
4. Experiments
4.1. Implementation Details
Baselines. Six normalisation baselines are considered, in- cluding Batch Normalisation (BN), Group Normalisation
6

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Table I: Top-1 Test Accuracy (%) of different FL normalisation solutions on CITAR---OCCIAA---O//Tny--aaaaa with batchsize 4/4/64.
Setting (FedAvg+)
CIFAR-I0
CIFAR-
Tiny-ImageNet
Φ = 0.6
= 0.3
Φ == 0.1
= 0.6
Φ 0.3
Φ = 0.1
Φ == 0.1
Ø = 0.05
BN (loffe, 2015)
75.82
73.85
69.61
45.84
45.15
43,40
23.95
21.55
GN (Wu He, 2018)
75.74
74.10
68.76
47.11
46.33
43.38
19.71
14.89
LN (Ba, 2016)
74.08
72.50
66.22
46.90
45.65
42.08
19.72
14.84
FedFN (Kim et al., 2023)
75.51
74.30
67.96
46.29
45.15
43.80
20.05
17.30
FixBN (Zhong et al.. 2023)
77.34
74.51
70.36
46.05
45.54
42.15
24.71
23,42
FBN (Guerraoui al., 2024)
73.91
71.94
66.13
45.35
43.72
41.61
4.90
3.76
HBN (ours)
78.22
76.53
71.56
49.88
48.93
46.52
25.59
24.69
33
Accuracy
Accuracd
FedAng-UN,
Follvy (NN, Be2
Follng FIBN, 3-2
FotAvg-BN,
PolAogrEINN
BoN B-4
Padkig (NN.
FadAogeHHIN,
Fod B-16
Poddiy Gb), 19-16
PalAog 1IBN, 2-16
FolAng-BN BN,
PadAng (NN,
PadAuprtiin HiNN, 2-12
1000
Communication round
Communication round
Communication round
(a)
(b)
(c)
Figure 4: The sensitivity of different normalisation methods to batch size on CIFAR-100 with Φ = 0.6 by Simple-CNN.
Table 2: Test Accuracy (%) of different normalisation meth- ods to batch size on CIFAR-100 with Φ 0.6/0.3 by Simple-CNN.
4.2. Performance Comparison
Influence of Data Heterogeneity. Table 1 reports the test accuracy of all compared FL. normalisation methods CIFAR-10, CIFAR-100 and Tiny-lmageNet with ious heterogeneous settings. We conduct experiments in two computing resource scenarios, Simple-CNN with batch and ResNet-18 18 with batch size 64 on Tiny-ImageNer. Accordingly, although GN, LN and FN avoid dependence on global statistical parameters, they do not demonstrate outstanding performance in deal- ing with data heterogeneity. We find that traditional BN remains competitive in most scenarios. We believe that this is due to the moving average mechanism of BN, which prevents the locally obtained statistics from deviating too much from the population. FixBN and FBN employ shared global statistics for normalisation at different stages. When these global statistics are reliable, they benefit local train- ing. Conversely, when the global statistics are unreliable. they exhibit detrimental effects. Our experimental scenario in Table 1 is deliberately challenging featuring strong heterogeneity, numerous clients, restricted batchsize and frequent stragglers. In such conditions, FixBN and FBN fail to judge whether the local statistics, which are derived from diverse local models, can directly be aggregated, resulting in unreliable global statisties. Our proposed hybrid batch normalisation adaptively combines historical global statis- tics with real-time local statistics. Analogous augmenting
Ø 0.6
Ø = 0.3
BN
GN
HBN
BN
GN
HBN
B=32
49.63
45.82
51.05
48.33
44.66
49.83
B=16
48.30
46.84
50.75
49.52
45.65
49.84
size 4 on
B=8
46.86
46.33
50.06
48.54
45.39
49.79
B=4
45.84
47.11
49.88
45.15
46.33
48.93
B=2
43.89
45.72
48.86
42.97
45.32
48.75
46.90
46.36
50.12
46.90
45.46
49.42
Avg.
+2.21
40.61
20.85
22.74
20.60
20.54
2014; Goyal et al., 2019) to adapt batch size changes. For specific learning rate settings, please refer to the Appendix. The batch size B and Dirichlet distribution coefficient ó will be specified in each experiment. For the moving aver- age momentum in BN and FBN, we set it to 0.9. For GN, we set the number of groups 2. For FixBN, we set the first half of the total number of rounds the first stage and the rest the second stage. For HBN, set À to 0.01 in Equation (11).
Due to the mismatch between the statistical parameters and the learnable parameters of HBN each communication round, we test the statistical parameters of the same client subset for accuracy evaluation.

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Table 3: Experiments on CIFAR-I0 across varying client numbers with B = 4 and Φ = 0.6 by SiMple-CNN.
Table 4: Test Accuracy (%) of different advanced FL solu- tions combined with HBN. The experiments are conducted on CIFAR-10 with B and / = 0.6 by Simple-CNN,
K=100
K=200
K=500
K=1000
BN
75,82
73.88
68.94
61.25
Method
+BN/GN/-
+HBN
GN
75.74
67.93
60.31
52.24
FedAvg (McMahan et al., 2017)
75.82
78.22
LN
74.08
69.08
61.22
51.55
FedProx (Li et al., 2020)
76.08
78.44
FedFN
75.51
70.71
62.02
53.75
FedAdam (Reddi et al., 2020)
75.67
78.17
FixBN
75.65
71.67
69.07
63.08
Scafflod (Karimireddy et al., 2020)
77.63
78.65
FBN
73.91
68.21
59.22
50.73
Moon (Li et al., 2021a)
75.77
78.48
HBN
78.22
75.76
72.49
64.95
FedSAM (Qu ct al., 2022)
76.39
78.40
FedACG (Kim et al., 2024)
76.01
79.19
Fedwon (Zhuang & Lyu, 2024)
78.85
79.93
local client training with global statistics, HBN dynami- cally integrates global information into the normalisation of current batches, achieving more effective normalisation.
learning. FedSAM performs Sharpness Aware Minimisa- tion local optimiser for local training. FedACG initialises local models using global momentum. Fedwon modifies the convolution layers directly to adjust the distribution, without using the standard normalisation layer. Because the update stage of FedAdam and FedACG conflicts with the statisti- cal parameters BN, resulting in a performance crash, we use GN for these two methods. We have not equipped any normalisation layer for Fedwon. For other methods, BN is used. Table 4 lists their basic test accuracy (+BN/GN/-) of Simple-CNN on CIFAR-10 with B = 4 and 6 0.6, as well as the test accuracy combined with HBN (+HBN). HBN obtains consistent gains in performance regardless of the federated learning approaches. The results demonstrate the potential of our HBN in delivering additional boosting power for the federated learning community.
Sensitivity to Batch Sizes. We collect the experimental results in terms of batch size in Table 2. We find that GN can alleviate the impact of batch size in federated learning as expected, but its generalisation power is worse than BN in large batch sizes. A larger batch size does not necessarily deliver better performance for GN, as it reduces the number of local updates. When the batch size is small, the perfor- mance of BN is greatly affected. For example, when the batch size is reduced from 32 to 2, the accuracy of BN drops by 5.74 percentage points, whereas HBN shows a smaller drop of only 2.19 percentage points. It is worth noting that even with batch size of 2, the accuracy of HBN exceeds BN with batch size of 32 and the optimal GN. Because HBN adaptively coordinate the global normalisation and the batch normalisation, HBN not only maintains the con- vergence speed of BN at appropriate batch sizes, but also reduces the sensitivity to small batch size. We provide a more intuitive visualisation in Figure 4.
Table 5: Test accuracy (%) of different model architectures with different normalisation layers on CIFAR-100 with Φ = 0.1 and B = 64.
Different Number of Clients. When the number of clients increases, methods based on local statistics are difficult to reflect the distribution of global data, and methods based on global statistics are difficult to obtain reliable global statis- ties under communication limitations. We conduct compar- ative experiments on CIFAR-10 (B = 0.6, B 4) across varying client numbers (10 clients sampled per round). As shown in Table 3, HBN consistently outperforms baselines on all scales (K = 100 to K 1000) through adaptive bal- ance of these two factors.
Model
+BN
+GN
+HBN
MobilenetV2
40.19
41.50
44.40
ResNet-18
51.51
52.38
55.69
ResNet-50
54.97
55.37
60.74
VGG-11
62.74
59.47
63.45
VGG-19
64.09
61.39
64.56
Consistent Compatibility with Classical Deep Networks Architectures. In heterogeneous federated learning sce- narios, deeper networks are more challenging in terms of training and convergence. Moreover, for vanilla BN statisti- cal parameters, the variance obtained by directly weighted averaging is smaller than the true variance. For networks with more BN layers, this will result in more offsst in the model output. We assemble HBN into deeper networks to verify its compatibility. We conduct experiments on CIFAR- 100 with á = 0.1, adjusting the batch size to 64 while keeping all other settings the same as before. We mainly verify the compatibility of HBN in five pretrained network architectures, including MobilenetV2 (Sandler et al., 2018),
Consistent Boosting Power with Advanced FL Solutions. As HBN belongs to the normalisation paradigm, it should act as plug-in layer to other advanced FL solutions. We attempt to configure HBN on several advanced FL solu- tions, including FedProx, FedAdam, Moon, Scaffold, Fed- SAM, FedACG and Fedwon. FedAdam adds first-order and second-order moment estimates during global updates. FedProx and Scaffold alleviate local model bias by intro- ducing additional constraint terms in local updates. Moon optimises local feature representation based on contrastive
8

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
ResNet-18/50 (He et al., 2016) and VGG-11/19 (Simonyan & Zisserman, 2014). Usually, their original networks are assembled with BN. We replace BN with GN and HBN, and remove the dropout layer from the original network, which had no positive effect in our experiment. As shown in Ta- ble 5, HBN is suitable for most classic deep convolutional neural network architectures, achieving significant perfor- mance improvements in heterogeneous federated learning scenarios compared to BN and GN.
#2=0.1
#2=0.01
Accurncy (%)
80
70
C-1
0.7.7
0=0.4
C
Figure 5: Impact of the hyper-parametes À under different client activation ratios C.
CIFAR-10 with & = 0.6, using Simple-CNN with a batch size of 32. We keep other settings consistent with those in Section 4.1. When the client activation rate low, adding a moving average can alleviate the oscillation of global statis- tics and improve the performance of HBN. When the client activation rate is high, it is not sensitive to the selection of x. According to our experiments, we recommend using a smaller X, which usually performs satisfactorily.
4.3. Ablation Study
Component Analysis. Compared with the vanilla BN, HBN modifies the operation in two aspects, calculating accurate statistical parameters (Equations (7) and (8)) and achieving adaptive normalisation (Equation (9)).
Table 6: Test Accuracy (%) of different configurations for HBN's each component. The experiments are conducted on CIFAR-10 with B = 4 and Φ = 0.6 by Simple-CNN.
5. Discussion
Index
Ablation Setting
Acc (%)
Communication and Computation Overhead. For feder- ated learning, communication overhead is worth considering and is proportional to the parameter quantity. Compared to other methods, HBN only increases the communication cost for one round, which is negligible compared to the large number of communication rounds required for training. In terms of computational cost, HBN requires additional cal- culation of local statistics on the clients, but this part is much lower than the cost of model training. In addition, find that using partial data to calculate local statistics on the client produces a marginally negative impact on model performance, as the client's local data usually follows the same distribution. For example, when we compute local statistics from randomly sampled 128 images per client, the final model performance decreases by only 0. 13 percentage points.
(I)
Equation (7) f moving average
72.25
(2)
Equation (8) <- weighted average
77.90
(3)
Equation (9) <- only global statistics
76.87
(4)
Equation (9) f- only batch statistics
76.33
(5)
Equation (9) + fix ratio at 1:1
77.49
FedAAg+IIBN
78.22
Table 6 displays the performance of using other candidates to replace each of our designed components. In principle, we freeze the other parts and only modify the following for each ablation experiment: (1) update local statisties based on moving average without separation, (2) summarise local statistics based weighted average, (3) normalise only using global statistics, (4) normalise only using batch statistics, and (5) fix an equal hybridization ratio for global statistics and batch statistics.
We can see that replacing any component causes a negative impact on the performance of HBN. It is worth mention- ing that in experiment (1), we do not separate the updates of the statistical parameters and the learnable parameters, which results in HBN being unable to benefit from unstable global statistical parameters and potentially exacerbating the shift of local statistics. In experiment (4), we only use batch statistics for normalisation, which consistent with the vanilla BN. The improved performance indicates that separating the calculation of statistical parameters can com- pensate for the accuracy loss caused by the deviation of the vanilla BN statistical parameters.
6. Conclusion
In this work, we propose a targeted improvement for batch normalisation in federated learning, Hybrid Batch Nor- malisation. The Hybrid Batch Normalisation resolve the dilemma of batch normalisation statistical parameter mis- match in Federated Learning. In order to leverage the po- tential of global statistical parameters in solving Non-III problems, Hybrid Batch Normalisation combines global nor- malisation and batch normalisation adaptively, alleviating vanilla batch normalisation limitations small batch size and heterogeneous data. Extensive experiments validate the effectiveness of Hybrid Batch Normalisation. Our HBN extends the current batch normalisation scope in terms of the federated learning tasks.
Selection of Hyper-parameters. Figure 5 measures the impact of the choice of in Equation (11) under different client activation ratios C. We select C from [1, 0.7, 0.4, 0.1] and À from [1, 0.1, 0.01]. We conduct this experiment on
9

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Acknowledgements
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
This work was supported in part by the National Key Re- search and Development Program of China under Grant No.2023YFE01 16300, the National Natural Science Foun- dation of China (62020106012, 62332008, 62336004, 62106089), the Fundamental Research Funds for the Central Universities (JUSRP202504007).
Hsieh, K., Phanishayee, A., Mutlu, O., and Gibbons, The non-iid data quagmire of decentralized machine learning. In ICMI., pp- 4387-4398. PMLR, 2020.
Hsu, T.-M. H.. Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual clas- sification. arXiv preprint (0Xiv:9909.0633.0 2019.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
loffe, S. Batch normalization: Accelerating deep net-
work training by reducing internal covariate shift. arXiv preprint urXiv: 2015.
loffe, S. Batch renormalization: Towards reducing mini- batch dependence in batch-normalize models. NeurIPS, 30, 2017.
References
Andreux, M., du Terrail, J. O., Beguier, C., and Tramel, E. W. Siloed federated learning for multi-centric histopathology datasets. In Domain Adaptation and Rep- resentation Transfer, and Distributed and Collaborative Learning, pp. 139. Springer, 2020.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and trends in machine learning, (4(1-2):1-211 2021.
Ba, J. L. Layer normalization. arXiv: 1607.06450, 2016.
arXiv preprint
Karakida, R., Akaho, S., and Amari, S.-i. The normalization method for alleviating pathological sharpness in wide neural networks. NewrIPS, 32, 2019.
Bharati, S., Mondal, M., Podder, P., and Prasath, V. Feder- ated learning: Applications, challenges and future direc- tions. International Journal of Hybrid Intelligent Systems, 18(1-2):19-35, 2022.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averaging for federated learning. In ICML, pp. 5132- 5143. PMLR, 2020.
Bjorck, N., Gomes, C. P., Selman, B., and Weinberger, K. Q. Understanding batch normalization. NeurIPS, 31, 2018.
Kim, G., Kim, J., and Han, B. Communication--ffiiieenn federated learning with accelerated client gradient. In CVPR, pp. 12385-12394, 2024.
Chen, J. and Zhang, A. FedMBridge: Bridgeable multi- modal federated learning. In ICML, 2024.
Chrabaszez, P., Loshchilov, L, and Hutter, F. A downsam- pled variant of imagenet as an alternative to the cifar datasets. ArXiv, abs/1 7777008199 2017.
Kim, S., Lee, G., Oh, J., and Yun, S.-Y. Fedfn: Feature normalization for alleviating data heterogeneity problem in federated learning. In NeurIPS Workshop, 2023.
Du, 7.,, Sun, J., Li, A., Chen, P., Zhang, J., Li, H., and Chen, Y. Rethinking normalization methods in federated learning. In Distributed Machine Learning Workshop, pp. 16-22, 2022.
Kingma, P. Adam: A method for stochastic optimization.
arXiv preprint arXiv: 2014.
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's '' thesis, University of Tront, 2009.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour. arxiv 2017. arXiv preprint arXiv: /70..0 02667, 2019.
Krizhevsky, A. One weird trick for parallelizing convolu- tional neural networks. arXiv preprint arXiv:1404.599 2014.
LeCun, Y., Bottou, L.,, Bengio, Y., and Hafiner, Gradient- based learning applied document recognition. Proceed- ings of the IEEE, 86(1 11:22)))223333 1998.
Guerraoui, R., Pinot, R., Rizk, G., Stephan, J., and Taiani, F. Overcoming the challenges of batch normalization in federated learning. arXiv preprint 240.:22005.166700 2024.
Li, Q., He, B., and Song, D. contraccn federated learning. In CVPR, pp. 10713-10722, June 2021a.
10

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Li, Q. Diao, Y., Chen, Q. and He, B. Federated learning non-iid data silos: An experimental study. In ICDE, pp. 965-978. IEEE, 2022.
Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. How does batch normalization help optimization? NeurIPS, 31, 2018.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Mlsys, 2:429 450, 2020.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. arXiv preprinl arXiv: 089 2019.
Tian, C., Shi, Z.,, Xinpengqin, Li, I.., and zhong Xu, C. Ranking- -baed client imitation selection for efficient fed- erated learning. In JCML, 2024.
Li, X., Jiang, M., Zhang, X., Kamp, M., and Dou, Q. Fedbn: Federated learning non-iid features via local batch normalization. arXiv preprini arXiv:2102.07623, 2021b.
Vahidian, S., Morafah, M., Chen, C., Shah, M., and Lin, B. Rethinking data heterogeneity in federated learning: Introducing new notion and standard benchmarks. TAI, 133):136-11397, 2023.
Lubana, E. S., Dick, R., and Tanaka, H. Beyond batchnorm: Towards a unified understanding of normalization in deep learning. NewrlPS, 34:4778 4, 2021.
Wang. Y., Shi, Q. and Chang, T.-H. Why batch normaliza- tion damage federated learning on non-iid data? TNNLS, 2023.
Luo, P., Wang, X., Shao, W., and Peng. Z. Towards un- derstanding regularization in batch normalization. arXiv preprint
Wang, Y., Wang, S., Lu, S., and Chen, FADAS: Towards federated adaptive asynchronous optimization. In ICML, 2024.
00846, 2018
Maas, A. L., Hannun, A. Ng, A. Y., et al. Rectifier nonlinearities improve neural network acoustic models. In ICML, pp. 3. Atlanta, GA, 2013.
Wu, Y. and He, K. Group normalization. In ECCV, pp. 3-19, 2018
Xu, J., Wang, S., Wang, L., and Yao, A. C.-C. Fedem: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874 2021.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. Communication-efiiceent learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Yang, Q. Liu, Y., Chen, T., and Tong, Y. Federated machine learning: Concept and applications. TTTS, 10(2):1-19, 2019.
Mendieta, M., Yang, T., Wang. P., Lee, M., Ding, Z., and Chen, C. Local learning matters: Rethinking data hetero- geneity in federated learning. In CVPR, pp. 8397-8406, 2022.
Yao, Z., Cao, Y., Zheng, S., Huang, G., and Lin, S. Cross- iteration batch normalization. In CVPR, pp. 12331- 12340, 2021.
of batch normalization and beyond. TPAMI, 2023. Peng, H., Yu, Y., and Yu, S. Re-thinking the effectiveness
Yong. H., Huang, J., Hua, X., and Zhang, L. Gradient centralization: new optimization technique for deep neural networks. In ECCV, pp. 635-652. Springer, 2020.
Pham, Q. Liu, C., and Hoi. S. Continual normalization: Rethinking batch normalization for online continual leam- ing. arXiv preprint arXiv:2203. 16100, 2022.
Zhang, G., Beitollahi, M., Bie, A., and Chen, X. Under- standing the role of layer normalization in label-skewed federated learning. TMLR, 2024.
Qu, Z., Li, X., Duan, R., Liu, Y., Tang, B., and Lu, Z. Generalized federated learning via sharpness aware mini- mization. In ICML., pp. 18250-18280. PMLR, 2022.
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, Federated learning with non-iid data. arXiv preprint arXiv: / 0t0..800.000550 2018
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
Zhong, J., Chen, H.-Y., and Chao, W.-L. Making batch nor- malization great in federated deep learning, In NeurIPS Workshop, 2023.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.- Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520, 2018.
Zhuang, W. and Lyu, L. Fedwon: Triumphing multi-domain federated learning without normalization. In ICLR, 2024.

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Zhuang, W., Xu, J., Chen, C., Li, J., and Lyu, L. COALA: A practical and vision-centric federated learning platform. In ICML, 2024.
12

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
A. Problem Formulation
A.1. Statistical Bias of Vanilla BN FL
In centralised training scenarios, internal covariate shift (loffe, 2015) refers to the change in the input distribution of each layer during the training of deep neural networks due to updates in model parameters. BN can effectively solve this problem under centralised training. However, in the context of federated learning, the changes in model parameters also occurs during the global aggregation stage. This results in the statistical measures obtained locally no longer being applicable globally, phenomenon known external covariate shift (Du et al., 2022). We can deduce that this statistical bias is caused by the heterogeneity of the distribution.
Let us consider the statistical bias of BN during one server-client communication caused by FedAvg. Assuming w is the initial model of the server in round t - 1. Each client E [K] downloads the global model as the initialisation of the local model, and trains it on local dataset D,. Due the fact that the statistical parameters in vanilla BN are updated synchronously with the learnable parameters, the statistical parameters obtained in each locality are approximate estimates of w w v/VL(w D2) on DR, i.e.,
N2 xf);
14
Awk
N
(12)
N2 x =
(o
Σ
N
where f()) is the output of model w before normalising all layers of the sample x.
Then, the server aggregates the parameters of each client model, including the statistical parameters of BN. Ideally, we would like to obtain the following global statistics:
Mul ah);
N
k=1
(13)
1
(6
(f(w x1) 122)2 -
where wgg N
No
Σ w
However, there is a deviation between the average of local statistics and the ideal global statistics. We consider the difference Ex= u and the ideal mean i
between the mean uº
No N
K
N N
1/
mk
k=1
N₂
xw/
(14)
K N2
Σ
If(wt, 2/.) - -w xل:1
N
k=1 i=1
1 K Nx
1 - -n (k). fluE N
VL(wg 1-1 (2)))
N
It can be seen that the deviation of the mean is caused by the inconsistency between the local updates (VL(wt-), Dx)) and the global updates N Σk= VL(W Dz)). Only when the data of each local client follows the same distribution will the local updates be consistent with the global updates, which contradicts the non-independent and identically distributed (Non-IID) federated learning. Therefore, the greater the difference data distribution, the greater the offset of statistical parameters.
13

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
A.2. Obtain Unbiased Global Statisties
Recognising that statistical bias is produced during the update stage, where the local statistical parameters and learnable parameters are jointly updated, we have sufficient motivation to separate their updates. In the stage of calculating local statistics, we freeze the learnable parameters to get:
N₂
14
N
(15)
(o
1
Mmw ,Ihh) - n2) -.
N
Then in the server stage, we can use distributed methods aggregate these statistics obtain an unbiased estimate the population according to Equation (8):
K
Nk N
N
20))) f(w5-1 xx/)
N
N₁:
==1
K
N₂
i=
p
NA
000)
Σ
((a2) + (1h - -1
N
K
No
(1(W . m1)2 + Nu 16/91
(16)
N
1 1=
N2
1
(f - =t)/k + (1(2)) + Na((H4)2 - 24hr (xt)
N
No
No
1
(w ((w)) Σ 20( ( + No(u2)2 + NN((4)2- 2Nanhug + (N(()))))
N
1=
N
(( - 2N + Na(LE)
N
k=
ΣΣ
(f(Wl-) ( -b))
N
(0 (
1-1
Therefore, we can obtain unbiased estimates of model w on the overall data without being affected by heterogeneous data distributions.
B. More Experiments
B.1. Model architecture of Simple-CNN
As shown in Table 7, Simple-CNN mainly consists of three convolutional blocks and two fully connected layers. Each convolutional block contains a convolutional layer, a norm layer, a ReLU activation layer, and a max-pooling layer. We replace the norm layer with different normalisation methods, except for FedEN. For FedEN, we normalise the input before the fully connected layer according to (Zhang et al., 2024).
14

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
Table 7: Model architecture of Simple-CNN with different normalisation layers (BNGGWAWTTAANTTENCCUINIIN
Simple-CNN
+BN
+GN
+LN
+FixBN
+FBN
+FedFN
+HBN
Conv2D(3, 16,31111
BN
GN
LN
FiXBN
FEN
HBN
Block
ReLU
Max.2
Conv2 (1,06,,,,,11.. 32,3,111)
Block 2
BN
GN
LN
FixBN
FEN
HBN
ReLU
(on,22(32.64,3,1.
BN
GN
LN
FixBN
FEN
HBN
Block 3
ReLU
MaxPool2D(2.22)
FedEN
FC(1024,128)
Block 4
ReLU
Block 5
FC(128,n)
B.2. Datasets
We mainly evaluate algorithm three image classification datasets:
CIFAR-10: CIFAR-10 It consists of 60,000 32x32 color images, divided into 10 classes with 6,000 images per class.
CIFAR-100: CIFAR-100 consists of 60,000 32x32 color images, divided into 100 classes with 6,00 images per class.
Nett A smaller version of the ImageNet dataset. Tiny-ImageNer consists of 100,000 64x64 color images, divided into 200 classes with 5,00 images per class.
B.3. Convergence Curves of Different Model Architectures
MobileNetV2
ResNet-50
100
FGR
Communication
Communication
Communication round
VGG11
VGG-19
Loxs
120
200
300
420
300
600
700
300
500
2000
200
200
300
400
300
630
700
600
500
1000
Communication round
Communication round
Figure 6: Convergence curves HBN under different model architectures on CIFAR-100 with Φ = 0.1 and B = 64.
As shown in Figure 6, we plot the convergence curves the above three normalisation methods under different model architectures, and HBN significantly better than the other two normalisation methods in terms of convergence.
15

---

Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learning
B.4. Learning Rate Grid Search
Since different normalisation techniques operate on distinct dimensions (e.g. batch, layer, or group), they induce varying scale transformations in the data. This necessitates tuning of learning rates to match their respective normalisation. We present the learning rate grid search results on three datasets in Table 8.
Table Learning rate grid search results on three datasets
CIFAR-10 (3 = 0.6, B =
Methods
Search range
Accuracy
BN
[0.02, 0.01, 0.005, 0.001]
[75.36, 75.83, 74.29, 73.37]
GN
[0.01, 0.005, 0.002, 0.001, 0.0005]
[74.08, 75.74, 73.47, 73.10, 70.01]
LN
[0.01, 0.005, 0.002, 0.001]
[N/A, 74.55, 74.34, 74.0]
FedFN
[0.01, 0.005, 0.002, 0.001, 0.0005]
[75.33, 75.49, 75.44, 74.30]
FixBN
[0.01, 0.005, 0.002, 0.001]
[75.65, 77.34, 75.30]
FBN
[0.01, 0.005, 0.002, 0.001]
173.91 73.05, 72.44, 68.16)
HBN
[0.02, 0.01, 0.005, 0.001]
[78.08, 78.22, 77.90, 69.30]
CIFAR-100 (8 0.6, B 4)
Methods
Search range
Accuracy
BN
[0.05, 0.02, 0.01, 0.005, 0.001]
[N/A, 45.77, 45.84, 45.54, 45.13]
GN
[0.02, 0.01, 0.005, 0.002, 0.001, 0.0005]
[N/A, 44.47, 46.70, 46.99, 47.11, 39.25]
LN
[0.01, 0.005, 0.002, 0.001, 0.0005]
[N/A, 42.43, 46.90, 45.04, 31.48]
FedEN
[0.01, 0.005, 0.002, 0.001, 0.0005]
[N/A, 43.75, 46.29, 35.15)
FixBN
[0.01, 0.005, 0.002, 0.001, 0.0005]
[43.72, 46.05, 43.90, 32.47, 32.48]
FBN
[0.02, 0.01, 0.005, 0.002, 0.001]
[45.35, 43.68, 43.33, 39.11, 35.47]
HBN
[0.02, 0.01, 0.005, 0.001]
[44.93, 49.88, 49.36, 49.11]
Tiny-ImageNet (3 0.1, = 64)
Methods
Search range
Accuracy [23.45, 23.95, 21.92, 20.07, 15.39]
BN
[0.32, 0.16, 0.08, 0.032, 0.016]
GN
[0.32, 0.16, 0.08, 0.032, 0.016]
[16.56, 18.43, 19.71, 18.40, 15.71]
LN
[0.32, 0.16, 0.08, 0.032, 0.016]
[16.28, 18.07, 19.72. 18.53, 15.31]
FedFN
[0.32, 0.16, 0.08, 0.032, 0.016]
[14.92, 20.05, 19.38, 18.61, 16.17)
FixBN
[0.32, 0.16, 0.08, 0.032, 0.016]
[24.71, 24.62, 22.95, 20.38, 15.65]
FBN
[0.32, 0.16, 0.08, 0.032, 0.016]
IN/A, N/A, 4.90, 3.40, 2.32]
HBN
[0.32, 0.16, 0.08, 0.032, 0.016]
[25.59, 24.79, 23.76, 22.0, 17.21]
-9- BN
GN
- HBN
Accuracy (%)
50
46
M4
Local Epochs
Figure 7: Experiments CIFAR-100 (3 0.6, B 32) across varying local epochs by Simple-CNN.
B.5. Different local training epochs
Figure 7 shows the changes in model accuracy with increasing local epochs for three normalisation methods. HBN maintains its advantage within an appropriate range of local epochs. When the number of epochs reaches 10, different normalisation methods longer have significant impact model performance, as the primary limiting factor at this stage becomes the excessive divergence among local model parameters.
16


Reviews:
[{"review_id": "kXSwvFAjK9", "reviewer": "Reviewer_jjB4", "summary": "Due to the lack of a coherent methodology for updating BN statistical parameters, standard BN degrades the federated learning performance. This paper proposes Hybrid Batch Normalization (HBN), which separates the update of statistical parameters from learnable parameters and adaptively combines local batch statistics with global statistics. The solution aims to obtain unbiased global statistical parameters while maintaining awareness of local data distributions during training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions. \nPlease find our response below.\n\n**1. Client Number Experiment**\n\nWe conducted comparative experiments on CIFAR-10 ($\\beta$ = 0.6) across varying client numbers (10 clients sampled per round). \nAs shown in Table A, HBN consistently outperforms baselines at all scales (from K=100 to K=1000).\n\n*Table A: Experiments on CIFAR-10 (β = 0.6) across varying client numbers.*\n| Method  | K=100 | K=200 | K=500 | K=1000 |\n|---------|-------|-------|-------|--------|\n| BN      | 75.82 | 73.88 | 68.94 | 61.25  |\n| GN      | 75.74 | 67.93 | 60.31 | 52.24  |\n| LN      | 74.08 | 69.08 | 61.22 | 51.55  |\n| FixBN   | 75.65 | 71.67 | 69.07 | 63.08  |\n| FBN     | 73.91 | 68.21 | 59.22 | 50.73  |\n| FN      | 75.51 | 70.71 | 62.02 | 53.75  |\n| **HBN(Ours)**| **78.22** | **75.76** | **72.49** | **64.95** |\n\n**2. Hyperparameter Sensitivity**\n\nWe apologise for any confusion caused. \nWe clarify that $\\alpha$, $\\beta$, and $\\gamma$ are learnable parameters (not hyperparameters). \n$\\alpha$ is an adaptive hybrid distribution factor, while $\\beta$ and $\\gamma$ are learnable affine transformations. \nIn this work, they are initialised as $\\alpha=0$, $\\beta=0$, and $\\gamma=1$ respectively. \n\n**3. Algorithm Typo**\n\nThank you for this observation. \nWhat we want to express is that updating statistical parameters does not require backpropagation to calculate gradients, which can save computational costs. \nFor clarity, we will modify the algorithm annotation to 'without backpropagation' and 'using backpropagation' respectively in the final version. \n\n**4. Details of Model Architecture and Symbol Explanations**\n\nRegarding the model architecture of Simple-CNN, we provide its details in Appendix B.1. \n$\\phi$ is the Dirichlet distribution factor that controls the degree of label heterogeneity [1] (visualisation is provided in Appendix B.2), while $B$ is the batch size.\n\n[1] Hsu, Tzu-Ming Harry, Hang Qi, and Matthew Brown. \"Measuring the effects of non-identical data distribution for federated visual classification.\" arXiv preprint arXiv:1909.06335 (2019).", "claims_and_evidence": "Strengths:\n\n1. Important question: Since federated learning uses distributed data, how to use aggregation of local statistics in batch normalization in federated learning is an important question. \n\n2. Adaptive hybrid approach: The learnable hybrid distribution factor that balances global and local statistics is an elegant solution that adapts to varying degrees of data heterogeneity across clients.\n\n3. Comprehensive empirical validation: The experiments cover multiple datasets (CIFAR-10/100, Tiny-ImageNet), network architectures (Simple-CNN, ResNet-18), and varying degrees of data heterogeneity and batch sizes.\n\n4. Compatibility with existing FL methods: The authors demonstrate that HBN can be effectively combined with various federated learning approaches (FedAvg, FedProx, FedAdam, etc.), consistently boosting performance across different methods.\n\nWeakness:\n\n1. It doesn't consider the scalability of client number. The paper does not present the experiment to change the number of clients for one dataset. \n\n2. Hyperparameter sensitivity: While there is some discussion of hyperparameter selection, a more comprehensive analysis would be beneficial for practitioners, such as the selection of $\\alpha, \\beta and \\gamma$ in eq(10)", "methods_and_evaluation_criteria": "See Claims And Evidence part.", "theoretical_claims": "Since sampling is out of my research area, I did not check the correctness of proof.", "experimental_designs_or_analyses": "See Claims And Evidence part.", "relation_to_broader_scientific_literature": "Unkonw", "essential_references_not_discussed": "Unkonw", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. In the algorithm, \"// forward with gradient\". Is this a typo and should be \"// backforward with gradient\"?\n\n2. Could we explain the structure details of the Simple-CNN network? \n\n3. what is the $\\phi$ in the experiment? \n\n4. What is B in the table 3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C85bpCNXtv", "reviewer": "Reviewer_Dpdp", "summary": "This paper introduces Hybrid Batch Normalisation (HBN), a normalization technique designed to address the limitations of standard Batch Normalisation (BN) in federated learning (FL) with non-IID data. HBN separates the update of statistical parameters (means and variances) from learnable parameters, enabling unbiased global statistical estimates. It incorporates a learnable hybrid distribution factor to adaptively blend local batch statistics with global statistics. HBN outperforms BN, Group Normalisation (GN), in classification tasks across CIFAR-10, CIFAR-100, and Tiny-ImageNet, particularly under data heterogeneity and small batch sizes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful suggestions. Please find our response below.\n\n**1. FedTAN Baseline**\n\nFedTAN employs real-time communication to synchronise the use of shared global statistics. \nHowever, to obtain these global statistics, FedTAN requires three rounds of communication per BN layer during the forward propagation. \nThis requirement fails to accommodate a fundamental FL constraint: communication rounds restrictions. \nWhile the theoretical discussion of the impact of  BN in FL provides valuable insights, we have intentionally excluded FedTAN from our comparative baselines because of these  limitations. \nWe acknowledge its conceptual contributions while noting its impractical communication overhead in real-world FL applications. \nOur method adds only one round overhead compared to the original BN, making it more suitable for practical scenarios. \n\n**2. Analysis of Results**\n\nFixBN and FBN employ shared global statistics for normalisation at different stages. \nWhen these global statistics are reliable, they benefit local training. \nConversely, when the global statistics are unreliable, they exhibit detrimental effects. \nOur experimental scenario in Table 1 is deliberately challenging——featuring strong heterogeneity, numerous clients, and restricted batchsize. \nIn such conditions, FixBN and FBN fail to judge whether the local statistics, which are derived from diverse local models, can directly be aggregated, resulting in unreliable global statistics.\nIn addition, FN is a local normalisation method that normalises features using the statistics of a single sample. \nIn non-iid scenarios, FN, like GN and LN, lacks awareness of the global structure, leading to performance degradation in our setup. \n\n**3. Weight Standardisation Methods**\n\nWe further analysed the weight standardisation solution, Fedwon, which achieves promising results. \nFedwon modifies the convolution layers directly to adjust the distribution, without using the standard normalisation layer.\nTherefore, the starting point of Fedwon is different from ours.\nWhat is exciting is that when we combine HBN with Fedwon, as shown in Table B, Fedwon+HBN yields impressive performance improvements, demonstrating the complementary power of our design towards Fedwon.\nIn our work, we want to emphasise the supportive role of BN's global statistical information in FL. \nHowever, inappropriate implementations of BN in FL fail to fully activate its potential. \nOur key contribution is resolving the BN's dilemma in federated learning.\nThrough analysis and empirical validation, HBN proves both simple and effective. \n\nDue to the lack of available open-source code for [1], we are unable to replicate it within a limited time. \nWe will provide a comprehensive analysis of weight standardisation methods in the final version.\n\n*Table B: Compatibility with weight standardisation methods.*\n| Settings| Cifar10(β=0.3)| Cifar100(β=0.3)| Tiny(β=0.1)| Tiny(β=0.05)|\n|----------------|---------------|----------------|---------------------|----------------------|\n| HBN            | 76.53         | 48.93          | 25.59               | 24.69                |\n| Fedwon         | 77.87          | 49.66          | 26.97               | 24.82                |\n| Fedwon+HBN     | **78.19**     | **50.15**      | **27.90**           | **26.69**            |\n\n[1] Siomos, Vasilis, et al. \"Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration.\" arXiv, 2024. \n\n**4. Why do we need hybrid batch normalisation?**\n\nIf we rely solely on local statistics, this reduces to the standard BN. \nThe standard BN degrades performance in non-iid scenarios, as the local statistics are diverse among different clients. \nConsider a toy example: two clusters following Gaussian distributions, simulating data from two clients in FL. \nNormalising each cluster separately using local statistics would cause their distributions to overlap (please refer to the link: \nhttps://anonymous.4open.science/r/ICML_2025-F29D/adaptive_normalisation.png). \nIn contrast, global normalisation preserves the global structure of two clusters.\nHowever, real-time global normalisation is impractical, as discussed above about FedTAN. \nUsing only historical global statistics ,whose timeliness is constrained by intermittent communication, will yield a suboptimal result. \nBut historical global statistics still retain valuable global structural information.\nInspired by that, our proposed HBN adaptively combines historical global statistics with current local statistics, achieving more effective normalisation. \n\n\n**5. Amelioration of Figure 2**\n\nWe replace Figure 2 with an intuitive toy example (please refer to the link: \nhttps://anonymous.4open.science/r/ICML_2025-F29D/adaptive_normalisation.png), where hybrid normalisation can standardise the size of two clusters while maintaining the global structure. \n\n**6. Technical Details**\n\nEMA is also used in FBN, excluding other methods.", "claims_and_evidence": "- The proposed two-stage update for separating statistical and learnable parameters is simple yet effective, with solid theoretical support in the Appendix.\n\n- However, the paper does not fully explain why combining global and local statistics in local training is beneficial, and the evidence provided is limited.\n\n\n- Similarly, the paper presents two conflicting claims: \"Using global statistics for batch normalization in local training is helpful\" versus \"FixBN and FBN overlook the diversity of local client statistics, a key challenge in non-IID settings.\" Simply stating that prior works ignore local diversity feels unclear, as their main contribution is using shared global statistics to tackle data heterogeneity across clients. The ablation study (Table 4) shows the hybrid component drives performance gains, but there’s little analysis or theoretical backing for why adding local batch statistics improves local training.", "methods_and_evaluation_criteria": "The proposed HBN method, which separates statistical and learnable parameter updates and introduces a hybrid factor, is logically sound for addressing BN’s mismatch in FL’s non-IID settings. Evaluation on standard datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet) with Dirichlet-distributed heterogeneity is appropriate and aligns with FL research norms. The use of Simple-CNN and ResNet-18 as benchmarks is reasonable, though testing on a broader range of architectures could strengthen generalizability. Metrics like top-1 accuracy are standard and suitable for the classification focus.", "theoretical_claims": "The paper provides derivations for unbiased global statistics (Appendix A.2, Equations 15-16), claiming they mitigate statistical bias in FL. I checked these proofs, and they appear mathematically correct, leveraging distributed statistical analysis to aggregate local statistics accurately. The formulation of the hybrid normalization (Equation 9) is conceptually clear, though its theoretical optimality lacks deeper justification beyond empirical success.", "experimental_designs_or_analyses": "- I think FedTAN should be included as a baseline for comparison.\n\n- The main results for FixBN, FBN, and FN in Table 1 show they often perform no better than BN, which contradicts the claims and results in their original papers. If these methods are implemented correctly, the paper should discuss the reasons for this discrepancy in the main paper.", "relation_to_broader_scientific_literature": "While existing methods focus on obtaining accurate global statistics, this paper concentrates on deriving an unbiased estimator and effectively using it with local statistics. It shows that combining both is better than using global statistics alone, but it doesn’t thoroughly analyze the issues with relying solely on global statistics.", "essential_references_not_discussed": "Another line of research [1, 2] uses weight standardization instead of normalization to handle statistical differences between clients. I believe these studies are relevant to this work and should be included as baselines for comparison.\n\n[1] Siomos, Vasilis, et al. \"Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration.\" arXiv preprint arXiv:2410.02006 (2024).\n[2] Zhuang, Weiming, and Lingjuan Lyu. \"Fedwon: Triumphing multi-domain federated learning without normalization.\" ICLR 2024.", "other_strengths_and_weaknesses": "If authors provide sufficient analysis or theoretical evidence on how the hybrid normalization works, its strength would lie in effectively combining local and global statistics, offering a compelling methodology.", "other_comments_or_suggestions": "- The T-SNE results in Figure 2 look very similar across normalization techniques. It’s hard to tell which one is better by eye, and the differences seem extremely marginal.", "questions_for_authors": "- Is the statistics EMA technique used in the implementation also applied in other global statistics-based approaches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqegNfTbQx", "reviewer": "Reviewer_b8GA", "summary": "The paper introduces Hybrid Batch Normalization (HBN) as a new normalization method designed to overcome the limitations of Batch Normalization in FL. In FL, client data is Non-IID, leading to a discrepancy between local and global statistics, which degrades BN’s performance. HBN addresses this issue by adaptively combining global and local batch statistics. The experimental results demonstrate that HBN outperforms existing normalization methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable contributions to improving this paper. In response to your suggestions, please find our detailed replies below.\n\n**1. FedBN Baseline**\n\nFedBN is designed for personalised FL without obtaining a unified global model, as it keeps BN parameters client-specific, \nwhile focusing on the general FL scenarios, which train a global model for all the local clients.\nTherefore, we did not compare with FedBN in our experiments.\nHowever, we will include more detailed discussion of the techniques used in FedBN in our final version.\n\n**2. FedFN Terminology**\n\nWe thank the reviewer for this valuable suggestion. \nWe confirm that FedFN is implemented consistently with FN as described in our paper. \nTo improve the accuracy of description, we will explicitly replace FN with FedFN in the revised version to avoid any potential ambiguity.\n\n**3. Hyperparameter Search**\n\nThe use of inconsistent local update epochs would compromise the fairness of comparisons, as it primarily impact on the number of local updates. \nTo mitigate this, we fixed the epoch value to 1 in our work. \nSimilarly, the local batch size affects the communication frequency, which is why we haven't made adjustments for local update epochs.\nAs you rightly observed, different algorithms exhibit distinct optimal learning rates across datasets (please refer to the anonymous link: \nhttps://anonymous.4open.science/r/ICML_2025-F29D/grid_search.png). \nWe ensure all the methods could stably converge within the searched range. The empirical results indicate that larger learning rates prevent trainability under our small batch size constraints.\n\n**4. Combining with Classifier-Freezing Methods**\n\nIn Table C, we compare the performance of the suggested three classifier-freezing approaches, integrated with our HBN, following the original experimental setup of MobileNet on CIFAR-100 (sharding partition strategies with 10 shards per client) [1]. \nIn our implementation, we solely replace the standard BN layers in MobileNet with HBN. \nHBN achieves encouraging performance gains. \nHBN enhances activation normalisation during training by adaptively blending local and global statistics, thereby mitigating bias introduced by relying exclusively on local statistics. \nThis hybrid approach ensures more stable and representative feature distributions across heterogeneous data partitions, which is particularly critical in classifier-freezing methods.\n\n*Table C: A comparison of classifier-freezing methods with BN and HBN.*\n| Methods   | +BN   | +HBN     |\n|-----------|-------|----------|\n| FedAvg    | 34.92 | **38.86**|\n| sphereFed | 42.80 | **49.57**|\n| FedETF    | 32.30 | **45.38**|\n| FedDr+    | 47.58 | **50.22**|\n\n[1] S. Kim et al., \"FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning,\" TMLR, 2025.", "claims_and_evidence": "The proposed method is sound, and the theoretical formulation aligns well with the motivation and objectives of the paper.", "methods_and_evaluation_criteria": "here is a foundational and impactful paper, FedBN [1], that initially addressed the issue of BN in FL. The paper should introduce FedBN more explicitly and clearly highlight the differences between HBN and FedBN throughout the presentation. Additionally, further comparative experiments with FedBN would strengthen the paper’s contribution and clarify its novelty.\n\n[1] FedBN: Federated Learning on Non-IID Features via Local Batch Normalization, ICLR 2021.", "theoretical_claims": "The paper does not provide any formal theoretical analysis or proofs. As a result, there are no theoretical claims to verify.", "experimental_designs_or_analyses": "### Recommendations for Table 1  \n\nIn **Table 1**, which presents experimental comparisons of different **normalization solutions in FL**, I recommend making the following modifications and additions:  \n\n1. **Include FedBN [1] as an additional baseline**  \n   - FedBN is a foundational work that directly addresses **batch normalization in FL**. Adding its results would provide a more comprehensive evaluation of existing normalization methods.  \n\n2. **Cite FedFN [2] and replace FN with FedFN**  \n   - The paper currently references **FN**, but a more direct and relevant work that first introduced **feature normalization in FL** is **FedFN**. I recommend citing **FedFN** explicitly and renaming **FN to FedFN** for accuracy.  \n\n3. **Expand the hyperparameter search space for local training epochs and learning rates**  \n   - **Section 4** states that the grid search for the **learning rate** was conducted over **{0.01, 0.005, 0.002, 0.001}**, with **local training epochs fixed at 1**.  \n   - I recommend **extending the grid search** as follows:  \n     - **Local training epochs**: Include **{3, 5, 10, 15}** in the search space.  \n     - **Learning rate**: Add **{1.0, 0.5, 0.1, 0.05}** to the search range.  \n\n   **Reasoning:**  \n   - As stated in **FedFN [2]**,  \n     > *\"FedFN scales the gradient of **θ_cls** by dividing it by the feature vector norm. This scaling significantly impacts the gradient of **θ_cls** and, consequently, the applied learning rate.\"*  \n   - Similar to **FedFN**, the proposed method applies **feature normalization**, meaning it could exhibit **different optimal learning rates compared to baselines**.  \n   - Other FL studies that applied **scaled learning rates for feature normalization** include:  \n     - **SphereFed** [3]  \n     - **Neural Collapse Inspired FL** [4]  \n     - **FedDr+** [5]  \n\n   - Since the experimental setup appears to follow benchmarks similar to **[5]**, it is possible that the current **learning rate and local epoch settings are relatively small**. Expanding the **grid search range** would ensure a more robust evaluation.  \n\n4. **Provide detailed hyperparameter settings in the appendix**  \n   - Each **dataset and algorithm** should have a **detailed breakdown of the applied learning rate, local epoch, and other key hyperparameters** in the appendix to improve reproducibility.  \n\n---\n\n### **References**  \n[1] **FedBN** : Federated Learning on Non-IID Features via Local Batch Normalization, ICLR 2021.  \n[2] **FedFN**: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning. \n[3] **SphereFed**: Hyperspherical Federated Learning, ECCV 2022.  \n[4] **No Fear of Classifier Biases**: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier, ICCV 2023.  \n[5] **FedDr+**: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning, TMLR 2025.", "relation_to_broader_scientific_literature": "Apart from the points mentioned earlier, I have no additional comments.", "essential_references_not_discussed": "1. **Include FedBN [1] as an additional baseline**  \n   - FedBN is a foundational work that directly addresses **batch normalization in FL**. Adding its results would provide a more comprehensive evaluation of existing normalization methods.  \n\n2. **Cite FedFN [2] and replace FN with FedFN**  \n   - The paper currently references **FN**, but a more direct and relevant work that first introduced **feature normalization in FL** is **FedFN**. I recommend citing **FedFN** explicitly and renaming **FN to FedFN** for accuracy.  \n\n\n### **References**  \n[1] **FedBN** : Federated Learning on Non-IID Features via Local Batch Normalization, ICLR 2021.  \n[2] **FedFN**: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning.", "other_strengths_and_weaknesses": "Apart from the points mentioned earlier, I have no additional comments.", "other_comments_or_suggestions": "There are existing approaches **[1,2,3]** that address **data heterogeneity in FL** by **freezing the classifier and modifying the loss function**. In contrast, this study focuses on modifying the **forward pass** of the model. Given this distinction, it seems that the proposed method could be effectively combined with these prior approaches.  \n\nI am particularly interested in whether **HBN** maintains its advantage over **BN and FedBN** when integrated with the **classifier-freezing methods** from **[1,2,3]**. It would be valuable to examine whether **Simple BN vs. FedBN vs. HBN** still demonstrates performance improvements when applied alongside these approaches.  \n\n### **References**  \n[1] **SphereFed**: Hyperspherical Federated Learning, ECCV 2022.  \n[2] **No Fear of Classifier Biases**: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier, ICCV 2023.  \n[3] **FedDr+**: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning, TMLR 2025.", "questions_for_authors": "Apart from the points mentioned earlier, I have no additional comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the issue of batch normalization in federated learning, and proposes hybrid batch normalization as a method that works well. Reviewers agree that this is important, and that the experiments are mostly good / wide-ranging. 

The authors rebutted most comments provided by the reviewers, although reviewers did not explicitly respond to this (aside from Reviewer b8GA). There are some concerns about additional baseline methods (FedTAN and FedBN), which the authors rebutted effectively in my mind. The authors also added some results with more clients, which is good. I agree with reviewer b8GA that the authors should run experiments with more than 1 local epoch (as methods can perform much worse in this setting, and this could be biasing the effectiveness of the HBN method). This is especially the case as this method is presented as a plug-and-play improvement to existing methods: it is not clear to me that this will be an improvement when more local optimization steps, as is common in federated learning.
Decision: Accept (poster)