# Entropy-Regularized Federated Optimization for Non-IID Data
###### Abstract  
Federated learning (FL) struggles under non-IID client data when local models drift toward conflicting optima, impairing global convergence and performance. We introduce entropy-regularized federated optimization (ERFO), a lightweight client-side modification that augments each local objective with a Shannon entropy penalty on the per-parameter update distribution. ERFO requires no additional communication, adds a single-scalar hyperparameter \(\lambda\), and integrates seamlessly into any FedAvg-style training loop. We derive a closed-form gradient for the entropy regularizer and provide convergence guarantees: under \(\mu\)-strong convexity and \(L\)-smoothness, ERFO achieves the same \(O(1/T)\) (or linear) rates as FedAvg (with only \(O(\lambda)\) bias for fixed \(\lambda\) and exact convergence when \(\lambda_{t}\to 0\)); in the non-convex case, we prove stationary-point convergence at \(O(1/T)\). Empirically, on five-client non-IID splits of the UNSW-NB15 intrusion-detection dataset, ERFO yields a +1.6 pp gain in accuracy and +0.008 in macro-F1 over FedAvg with markedly smoother dynamics. On a three-of-five split of PneumoniaMNIST, a fixed \(\lambda\) matches or exceeds FedAvg, FedProx, and SCAFFOLD--achieving 90.3% accuracy and 0.878 macro-F1--while preserving rapid, stable learning. ERFO's gradient-only design is model-agnostic, making it broadly applicable across tasks. federated learning; entropy-regularized optimization; non-IID data; convergence guarantees; Shannon entropy +
Footnote †: journal:  
[PERSON], [PERSON]  
drift but can impose constraints on plasticity (the ability to adapt to new data) or incur additional overhead. Recent algorithms such as FedDyn (2018) dynamically regularize local objectives toward an adjusted global model to prevent drift, and FedCurv (2018, 2019) penalizes changes to important weights (analogous to elastic-weight consolidation in FL) to reduce forgetting. Other works address objective inconsistencies via normalization or adaptive steps (e.g., FedNova (2018) and FedOpt (2019)) or pursue personalized FL (2019, 2020), where each client's model is partially specialized. Despite these advances, there remains a need for simple, effective mechanisms that handle both non-IID data and continual adaptation in FL without significant communication or computational overhead. Meanwhile, techniques from continual learning (CL) have tackled the stability-plasticity dilemma in sequential training of neural networks ([PERSON] et al., 2014). The stability-plasticity dilemma refers to the trade-off between retaining past knowledge (stability) and incorporating new information (plasticity) ([PERSON] et al., 2014). Regularization-based CL methods such as Elastic Weight Consolidation (EWC) ([PERSON] et al., 2014) penalize changes to important weights to prevent catastrophic forgetting (favoring stability), whereas replay-based methods like iCaRL ([PERSON] et al., 2015) and knowledge-distillation approaches such as Learning without Forgetting (LwF) ([PERSON] et al., 2015) preserve prior knowledge by other means. However, directly applying such techniques in FL is challenging because of decentralized data and limited communication. Federated continual-learning variants have begun to emerge ([PERSON] et al., 2015; [PERSON] et al., 2015), but they often add significant complexity or require storing additional data or models. Despite these advances, statistical heterogeneity in federated settings remains a fundamental obstacle. When clients hold highly skewed or non-IID data, local updates can \"drift\" in conflicting directions, slowing convergence and degrading global accuracy. Classical FedAvg exhibits significant performance drops under extreme distribution skews, and extensions such as FedProx and SCAFFOLD only partially mitigate drift at the cost of extra hyperparameters, additional communication rounds, or storage overhead. Robust aggregation approaches likewise assume bounded or adversarial gradient models that may not hold in many real-world deployments.