# Entropy-Regularized Federated Optimization for Non-IID Data
## 3 Methodology
### Entropy-Regularized Client Objective  
In ERFO, each client augments its local objective with an entropy-based regularization term that penalizes the distribution of model updates. Let \(F_{i}(w)\) denote the original loss function on client \(i\) (e.g., the empirical risk over its local dataset) evaluated with the model parameter \(w\). In a federated round, let \(w^{t}\) be the current global model weights (broadcast from the server at the start of the round) and \(w\) be the local model being optimized on the client (initialized as \(w=w^{t}\)). We define the vector of normalized update magnitudes \(p=(p_{1},p_{2},\ldots,p_{d})\) (of dimension \(d\), the number of model parameters) as follows:  
\[p_{j}=\frac{|w_{j}-w_{j}^{t}|}{\sum_{k=1}^{d}|w_{k}-w_{k}^{t}|},\qquad j=1,2, \ldots,d, \tag{2}\]where \(w_{j}\) is the \(j\)-th component of the local model and \(w_{j}^{t}\) is the corresponding component of the initial global model. In summary, \(p_{j}\) represents the fraction of the total model change (in absolute value) that is attributable to parameter \(j\). By construction \(p_{j}\geq 0\) and \(\sum_{j}p_{j}=1\), so \(p\) can be viewed as a discrete probability distribution over the model's coordinates (or groups of parameters). We then define the entropy regularizer \(H(p)\) as the Shannon entropy of this distribution \(p\):  
\[H(p)=-\sum_{j=1}^{d}p_{j}\ln p_{j}, \tag{3}\]  
where \(0\ln 0\) is understood as \(0\) (and in practice we avoid \(p_{j}=0\) by adding a very small \(\epsilon\) inside the log if needed). The entropy \(H(p)\) is maximized when the update magnitudes are spread evenly across all parameters (a uniform \(p\)) and minimized when the update is highly concentrated in a few parameters (a peaked \(p\)). By entropy-regularized optimization, we mean that the client aims to minimize its loss while simultaneously discouraging an overly concentrated update (i.e., encouraging higher entropy in the update distribution). We introduce a non-negative regularization weight \(\lambda\) to balance these goals. Formally, the ERFO client's objective for client \(i\) is given by  
\[\bar{F}_{i}(w)=F_{i}(w)+\lambda\cdot H\big{(}p(w,w^{t})\big{)}, \tag{4}\]  
where we explicitly note that \(p=p(w,w^{t})\) is a function of the local model \(w\) (and the fixed reference point \(w^{t}\)). Equation (4) augments the ordinary local loss \(F_{i}(w)\) with the entropy term \(H(p)\) multiplied by \(\lambda\). When \(\lambda>0\), this term penalizes updates with low entropy (i.e., it adds a cost when \(p\) is too high on certain coordinates). Conversely, it effectively rewards updates that are spread broadly (high-entropy updates) since those yield a larger \(H(p)\), reducing the objective. The hyperparameter \(\lambda\) thus controls the strength of this regularization: \(\lambda=0\) recovers standard federated training, while larger \(\lambda\) values place more emphasis on distributing the update across many parameters. To optimize \(\bar{F}i(w)\), we derive the gradient of the entropy term. Let \(Z=\sum k=1^{d}|w_{k}-w_{k}^{t}|\) denote the \(L^{1}\) norm of the update (the normalization denominator in Equation (2)). Using \(Z\), we can rewrite \(p_{j}=|w_{j}-w_{j}^{t}|/Z\). The derivative of \(H(p)\) with respect to a particular weight \(w_{j}\) can be obtained via the chain rule, taking into account the dependence of all \(p_{k}\) on \(w_{j}\).