## 2 Related Work  
Adversarial attacks against DNNs have become an important research field in the last few years. For a comprehensive survey, we refer the reader to [11]. An important distinction is between 'white box' and 'black box' attacks. In white-box attacks, the attacker has knowledge of the attacked model's internal structure and parameters, and exploits that knowledge. It is commonly performed by using the model's gradients [5; 7; 12]. Although white-box methods achieved good results they usually do not represent a real-world scenario. More realistic is a black-box attack, where the attacker has no  
Figure 1: Examples of adversarial attacks generated by QuEry Attack. With higher resolution the attack becomes less visible to the naked eye. The differences between images that successfully attack the model and those that do not are subtle. (**Top row**): Imagenet (\(l_{\infty}=6/255\)). ( **Middle row**): CIFAR10 (\(l_{\infty}=6/255\)). ( **Bottom row**): MNIST (\(l_{\infty}=60/255\)). (**Left**): the original image. (**Middle**): a successful attack. (**Right**): A failed attack. knowledge of the model's structure and parameters. The attacker can only query the model--and act upon its outputs. We can further distinguish between the more common 'light black box' attacks, where the model gives the prediction's confidence (which can be exploited), and 'dark black box' attacks where the model only gives the final class prediction (e.g., [10]). The first effective black-box attack traded a priori information of the model with extensive runtime querying [10]. Using a large number of queries it builds a substitute model, and attacks it with traditional white-box methods. It uses the transferability property, namely, an attack that succeeds on one model will likely succeed on a similar--though not identical--model. Other works used the more permissive 'light black-box' scenario, which can use the prediction's confidence value. Some works estimate the gradient with this information and then use traditional gradient-based attacks [13]. All these black-box methods rely on gradients, and, thus, are sensitive to many defense methods that obscure gradients [14; 15; 16]. This has given rise to methods that do not rely on gradients at all, e.g., [17], which uses random search and is also query-efficient. Instead of randomness, one can use evolutionary methods. In _Evolutionary Algorithms_ (EAs), core concepts from evolutionary biology--inheritance, random variation, and selection--are harnessed in algorithms that are applied to complex computational problems. EA techniques have been shown to solve numerous difficult problems from widely diverse domains, and also to produce human-competitive machine intelligence [18]. EAs also present many important benefits over popular Machine Learning methods, including [19]: less reliance on the existence of a known or discoverable gradient within the search space; ability to handle design problems, where the objective is to design new entities from scratch; fewer required a priori assumptions about the problem at hand; seamless integration of human expert knowledge; ability to solve problems where human expertise is very limited; support of interpretable solution representations; and support of multiple objectives. The evolutionary method GenAttack is a _targeted_ attack (thus not directly comparable to ours, which is _untargeted_) that used a fitness function that tries to increase the probability of the target class and decrease the probability of the other classes [20]. Its fitness function ignored the distance between the images. Interestingly, GenAttack uses fitness-proportionate selection, which is employed less often nowadays due to known problems. It uses an adaptive mutation rate to balance between exploration in early phases and exploitation in later phases. Ref. [21] treated the adversarial problem as one of multi-objective optimization: minimize the class prediction's score on one hand, and the distance between the original image and a perturbed one on the other hand.