# Cybersecurity for AI Systems: A Survey
## 3 AI Attacks and Defense Mechanisms
### Poisoning Attack
#### 3.3.1 Dataset Poisoning Attacks
The right sub-figure shows how the decision boundary is drastically changed by tampering with one training data instance without changing the label of the instance. It was observed that the classification accuracy would be reduced by 11% by a mere 3% manipulation of the training set [81]. [PERSON] et al. [39] showed that an attacker can breach the functionality on the spam filter by poisoning the Bayesian classification model. The filter becomes inoperable under the proposed Usenet dictionary attack, wherein 36% of the messages are misclassified with 1% knowledge regarding the messages in the training set. [PERSON] et al. [38] illustrated poisoning attacks on multi-class classification problems. The authors identified two attack scenarios for the multi-class problems: (1) error-generic poisoning attacks and (2) error-specific poisoning attacks. In the first scenario, the adversary attacks the bi-level optimization problem [40, 82], where the surrogate data is segregated into training and validation sets. The model is learned on the generated surrogate training dataset with the tampered instances. The validation set measures the influence of the tampered instances on the original test set, by maximizing the binary class loss function. It is expressed in the following equation:  
\[D_{x}^{*}=argmax_{D_{x}^{\prime}}A\left(D_{x}^{\prime},\sigma\right)=L( \widehat{D_{val}},\ \widehat{\theta})\]  
\[\text{\emph{Such as}}\ \ \widehat{\theta}=\ min_{w^{*}}L(\widehat{D_{tr}}\ U \ D_{x}^{{}^{\prime}},\ \theta^{*})\]  
The surrogate data \(\widehat{D}\) is segregated into training \(\widehat{D_{tr}}\) and validation sets \(\widehat{D_{val}}\). The model is trained on \(\widehat{D_{tr}}\) along with \(D_{x}^{\prime}\) (i.e., the tampered instances). \(\widehat{D_{val}}\) is used to measure the influence of the tainted samples on the genuine data via the function \(A\left(D_{x}^{\prime},\sigma\right)\) that explains the loss function, \(L\), with respect to the validation dataset \(\widehat{D_{val}}\) and the parameters \(\widehat{\theta}\) of the surrogate model. In the multi-class scenario, the multi-class loss function is used for error-generic poisoning attacks. Figure 2: Poisoning attack changing the decision boundary. In error-specific poisoning attacks, the objective remains to change the outcome of specific instances in a multi-class scenario. The goal of desired misclassification is expressed with the equation:  
\[\left(D_{x}^{\prime},\sigma\right)=-L(\overbrace{D_{val}^{\succ},\overbrace{ \theta}^{\succ}}^{\succ})\]  
\(\widehat{D_{val}^{\succ}}\) is the same as the \(\widehat{D_{val}}\) with different labels for desired misclassified instances that the adversary chose. The attacker aims to minimize the loss of the chosen misclassified samples. In separate research, [PERSON] and [PERSON] [83] explained the adversarial attack on detection of outliers (anomalies), where the adversary is assumed to have knowledge about the algorithm and the training data. Their work introduced a finite sliding window, while updating the centre of mass iteratively for each new data instance. The objective is to accept the poisoned data instance as a valid data point, and the update on the center of mass is shifted in the direction of the tainted point, that appears to be a valid one. They show that relative displacement, d, of the center of mass under adversarial attack is lower bounded by the following inequality when the training window length is infinite:  
\[d_{i}\leq ln(1+\ \frac{i}{n})\]  
where \(i\) and \(n\) are the number of tampered points and number of training points, respectively. The intuition behind the use of anomaly detection is to sanitize the data by removing the anomalous data points, assuming the distribution of the anomalies is different from that of the normal data points. [PERSON], [PERSON], and [PERSON] [84] presented data poisoning attacks that outsmart data sanitization defenses for traditional anomaly detection, by nearest neighbors, training loss and singular value decomposition methods.