# Cybersecurity for AI Systems: A Survey
## 3 AI Attacks and Defense Mechanisms
### Types of Failures
#### 3.1.2 Categories of Intentional Failures  
The goal of the adversary is deduced from the type of failure of the model. [PERSON] et al. [70] identify four different classes of adversarial goals, based on the machine learning classifier output, which are the following: (1) confidence reduction, where the target model prediction confidence is reduced to a lower probability of classification, (2) misclassification, where the output class is altered from the original class, (3) output misclassification, which deals with input generation to fix the classifier output into a particular class, and (4) input/output misclassification, where the label of a particular input is forced to have a specific class. [PERSON] et al. [60] identified the taxonomy of intentional failures/attacks, based on the knowledge of the adversary. It deals with the extent of knowledge needed to trigger an attack for the AI/ML systems to fail. The adversary is better equipped with more knowledge [70] to perform the attack. There are three types of classified attacks based on the adversary's access to knowledge about the system. 1. **Whiteb ox Attack:** In this type of attack, the adversary has access to the parameters of the underlying architecture of the model, the algorithm used for training, weights, training data distribution, and biases [71, 72]. The adversary uses this information to find the model's vulnerable feature space. Later, the model is manipulated by modifying an input using adversarial crafting methods. An example of the whitebox attack and adversarial crafting methods are discussed in later sections. The researchers in [73, 74] showed that adversarial training of the data, filled with some adversarial instances, actually helps the model/system become robust against whitebox attacks. 2. **Blackbox Attack:** In blackbox attacks the attacker does not know anything about the ML system. The attacker has access to only two types of information. The first is the hard label, where the adversary obtained only the classifier's predicted label, and the second is confidence, where the adversary obtained the predicted label along with the confidence score. The attacker uses information about the inputs from the past to understand vulnerabilities of the model [70]. Some blackbox attacks are discussed in later sections. Blackbox attacks can further be divided into three categories:
3. **Non-Adaptive Blackbox Attack:** In this category of blackbox attack, the adversary has the knowledge of distribution of training data for a model, T. The adversary chooses a procedure, P, for a selected local model, T', and trains the model on known data distribution using P for T' to approximate the already learned T in order to trigger misclassification using whitebox strategies [53, 75]. 4. **Adaptive Blackbox Attack:** In adaptive blackbox attack the adversary has no knowledge of the training data distribution or the model architecture. Rather, the attacker approaches the target model, T, as an oracle. The attacker generates a selected dataset with a label accessed from adaptive querying of the oracle. A training process, P, is chosen with a model, T', to be trained on the labeled dataset generated by the adversary. The model T' introduces the adversarial instances using whitebox attacks to trigger misclassification by the target model T [70, 76]. 5. **Strict Blackbox Attack:** In this blackbox attack category, the adversary does not have access to the training data distribution but could have the labeled dataset (x, y) collected from the target model, T. The adversary can perturb the input to identify the changes in the output. This attack would be successful if the adversary has a large set of dataset (x,y) [70, 71]. **Grayb ox attacks:** In whitebox attacks the adversary is fully informed about the target model, i.e., the adversary has access to the model framework, data distribution, training procedure, and model parameters, while in blackbox attacks, the adversary has no knowledge about the model.