# Cybersecurity for AI Systems: A Survey
## 3 AI Attacks and Defense Mechanisms
### Attack on Federated Learning  
In the federated learning scenario, each and every individual device has its own model to train, securing the privacy of the data stored in that device [47]. Federated learning algorithms are susceptible to model poisoning if the owner of the device becomes malicious. Research [111, 112] introduced a premise for federated learning, where a single adversary attacks the learning by changing the gradient updates to arbitrary values, instead of introducing the backdoor property into the model. The objective of the attacker is to obstruct the convergence of the execution of the distributed Stochastic Gradient Descent (SGD) algorithm. In a similar study, [PERSON] et al. [48] proposed a multi-agent framework, where multiple adversaries jointly conspired to replace the model during model covergence. [PERSON] et al. [41] worked on targeted misclassification by introducing a sequence of attacks  
Figure 8: Transfer Learning set up attacks [42]. Figure 7: Transfer learning using the BadNet [42]. induced by a single adversary: (1) explicit boosting, and (2) alternating minimization. The underlying algorithm is SGD. * **Explicit Boosting:** The adversary updates the boosting steps to void the global aggregated effect of the individual models locally distributed over different devices. The attack is based on running of boosting steps of SGD until the attacker obtains the parameter weight vector, starting from the global weight, to minimize the training loss over the data and the class label. This enables the adversary to obtain the initial update, which is used to determine the final adversarial update. The final update is obtained by the product of the final adversarial update and the inverse of adversarial scaling (i.e., the boosting factor), so that the server cannot identify the adversarial effect. * **Alternating Minimization:** The authors in [45] showed that, in an explicit boosting attack, the malicious updates on boosting steps could not evade the potential defense related to measuring accuracy. Alternating minimization was introduced to exploit the fact that it is updates related only to the targeted class that need to be boosted. This strategy improves adversarial attack that can bypass the defense mechanism with the goal of minimizing training loss and boosting parameter updates for the adversarial goals and achieved a high success rate. In terms of potential defense mechanisms, two typical strategies are deployed, depending on the nature of the attacks on federated learning: (1) robust aggregation methods, and (2) robust learning rate. * **Robust aggregation methods:** These methods incorporate security into federated learning by exploring different statistical metrics that could replace the average (mean) statistic, while aggregating the effects of the models, such as trimmed mean, geometric median, coordinate-median, etc [47, 111, 113, 114, 115, 116]. Introducing the new statistic while aggregating has the primary objective of staving off attacks during model convergence. [PERSON] et al. [117] proposed a sign aggregation technique on the SGD algorithm, distributed over individual machines or devices. The devices interact with the server by communicating the signs of the gradients. The server aggregates the signs and sends this to the individual machines, which use it to update their model weights. The weight update rule can be expressed by the following equation: \[w_{t+1}=w_{t}+\gamma(sgn\sum_{i\in A_{t}}sgn(\Delta_{t}^{i}))\] where \(\Delta_{t}^{i}\) is the weight update of the device \(i\) at time \(t\). \(\Delta_{t}^{i}=\ w_{t}^{k}-\ w_{t}.w_{t}\) is the weight the server sent to the set of devices A\({}_{t}\) at time \(t\) and \(\gamma\) is the server learning rate. This approach is robust against convergence attacks, but susceptible to backdoor attacks in federated learning scenarios. In a recent study, [118] the authors modified the mean estimator of the aggregate by introducing weight-cutoff and addition of noise [119] during weight update to deter backdoor attacks.