## 4 Statistical Methodology for Federated Learning
### Privacy Leakage  
Attackers can exploit the privacy of the system with different tactics. Each of these tactics can lead to different results and details. This is particularly important in the field of distributed federated learning, for instance, if the recent development of Recall ([[https://learn.microsoft](https://learn.microsoft)]([https://learn.microsoft](https://learn.microsoft)). com/en-us/windows/ai/apis/recall, accessed on 14 September 2024), an artificial intelligence model of Microsoft, which can be used to search and retrieve images using details of their content, expressed in natural language. If systems similar to this one use federated machine learning and send data to a central analyzer, privacy must absolutely be one of the core features of the system to avoid leakage of personal data of users. Gradient Inversion Attacks: Attackers can reconstruct original data from the shared gradients during the model update process. Even though raw data are not directly shared, gradients can carry enough information to reveal sensitive data points. For example, given a gradient \(\
abla L(\mathbf{w},\mathbf{x})\) computed with respect to the model weights \(\mathbf{w}\) and input data \(\mathbf{x}\), it is possible to approximate \(\mathbf{x}\) by minimizing the difference between the computed gradient and the gradient of a guessed input. Model Updates: Repeatedly sharing model updates can lead to leakage of information about the training data. Over time, these updates can accumulate enough information for an attacker to infer private data. For instance, in a typical FedAvg setup, the global model update at round \(t+1\) is computed as:  
\[w_{t+1}=w_{t}+\eta\sum_{k=1}^{K}\frac{n_{k}}{n}\Delta w_{t}^{k} \tag{3}\]  
where \(\eta\) is the learning rate, \(n_{k}\) is the number of data points at client \(k\), \(n\) is the total number of data points, and \(\Delta w_{t}^{k}\) is the local model update from client \(k\). If the model update is performed within a short period of time, the attacker can infer from the absence or the high volume of sent data the position or the details on the data of the user. Otherwise, if the model is not sufficiently protected in transit, the attacker can intercept it and try to analyze it. Side-Channel Attacks: Attackers can exploit side-channel information, such as the timing or size of the communications between clients and the server, to gain insights into the data being processed. These side channels can indirectly leak sensitive information even if the data and model updates are encrypted. Famous attacks such as Spectre and Meltdown fall in this category. Membership Inference Attacks: These attacks aim to determine whether a specific data point was part of the training dataset. This is particularly concerning in scenarios where the presence of certain data points can imply sensitive information. Given a model \(\mathcal{M}\) and a data point \(\mathbf{x}\), an attacker can train a shadow model to infer the membership status of \(\mathbf{x}\) by analyzing the output confidence scores. To mitigate these risks, several techniques have been proposed. The main and most present in literature are as follows:  
Differential Privacy: A technique that adds noise to the gradients or the model parameters to mask the contribution of individual data points [25]. By ensuring that the inclusion or exclusion of a single data point does not significantly affect the output, differential privacy provides a strong privacy guarantee. The formal definition of differential privacy is as follows:  
\[\Pr[\mathcal{M}(D)\in S]\leq e^{\epsilon}\Pr[\mathcal{M}(D^{\prime})\in S]+\delta \tag{4}\]  
where \(\mathcal{M}\) is the randomized mechanism, \(D\) and \(D^{\prime}\) are datasets differing by one element, \(S\) is a subset of possible outputs, \(\epsilon\) is the privacy budget, and \(\delta\) is a small probability. Secure Multi-Party Computation (SMPC): SMPC protocols [26] allow multiple parties to jointly compute a function over their inputs while keeping those inputs private.