# Cybersecurity for AI Systems: A Survey
## 3 AI Attacks and Defense Mechanisms
### Outsourced Training Attack  
We want to train the parameters of a model, M. using the training data. We outsource the description of M to the trainer who sends the learned parameters back to us \(\beta_{M}\). Our trustability of the trainer depends on a privately held validation dataset, with a targeted accuracy, or on the service agreement between us and the trainer. The objective of the adversary is to return a corrupted model with backdoored trained parameters \(\beta^{{}^{\prime}}_{M}\). This is different from \(\beta_{M}\) and either should not lower the validation accuracy or decrease the model accuracy of the inputs with a backdoor trigger. Thus, the training attack can be targeted or untargeted. In a targeted attack, the adversary switches the label of the outputs for specific inputs, while in an untargeted attack, the input of the backdoored property remains misclassified to degrade the overall model accuracy. Figure 4 depicts an example of backdoor attacks where the second and third images are the original image's backdoored version, whereas Figure 5 depicts an example of BadNet attacks on traffic images. Figure 5: Badnet Example [42]. Figure 6 illustrates a special type of potential BadNet (i.e., BadNet with backdoor detector) which makes use of a parallel link to identify the backdoor trigger. It also uses a combining layer to produce misclassifications if the backdoor appears. This perturbed model would not impact the results on a cleaned dataset, so the user would not be able to identify if the model has been compromised. In terms of defense mechanisms, Backdoor attacks like BadNet happen when we use pre-trained models. So, the less pre-trained the model, the less the attack. However, today, almost all networks are built using pre-trained models. To make the models robust against backdoor attacks, [PERSON] et al. [42] proposed the following defense strategies:  
* Securely hosting and disseminating pre-trained models in virtual repositories that guarantee integrity, to preclude benevolent models from being manipulated. The security is characterized by the fact that virtual archives should have digital signatures of the trainer on the pre-trained models with the public key cryptosystem [43]. * Identifying backdoors in malevolently trained models acquired from an untrustworthy trainer by retraining or fine-tuning the untrusted model with some added computational cost [44, 46]. These researchers considered fully outsourced training attacks. Another research [107], proposed a defense mechanism with an assumption that the user has access to both clean and backdoored instances. Transfer Learning Attack  
The objective of transfer learning is to save computation time, by transferring the knowledge of an already-trained model to the target model [45]. The models are stored in online repositories from where a user can download them for an AI/ML application. If the downloaded model, M\({}_{cor}\), is a corrupted model, then, while transferring learning, the user generates his/her model and parameters based on M\({}_{cor}\). In transfer learning attacks, we assume that the newly adapted model, M\({}_{cor}\), and the uncorrupted model have the same input dimensions but differ in number of classes. Figure 7 compares a good network (left), that rightly classifies its input, to BadNet (right), that gives misclassifications but has the same architecture as the good network. Figure 8 describes the transfer learning attack setup with backdoor strengthening factor to enhance the impact of weights. In terms of potential defense mechanisms, the obvious defense strategy is to obtain pre-trained models from trusted online sources, such as Caffe Model Zoo and [PERSON] trained Model Library [108], where a secure cryptographic hashing algorithm (e.g., SHA-1) is used as a reference to verify the downloads. However, the researchers in [42] showed that downloaded BadNet from \"secure\" online model archives can still hold the backdoor property, even when the user re-trains the model to perform his/her tasks.