SlowGuess/NeurIPS_2023_OCR huggingface dataset:
QuadAttacK: A Quadratic Programming Approach to Learning Ordered Top-  $K$  Adversarial Attacks
Thomas Paniagua, Ryan Grainger and Tianfu Wu
Department of Electrical and Computer Engineering, NC State
{tapaniag, rpgraing, twu19}@ncsu.edu
https://thomaspaniagua.github.io/quadattack_web/
The adversarial vulnerability of Deep Neural Networks (DNNs) has been well-known and widely concerned, often under the context of learning top-1 attacks (e.g., fooling a DNN to classify a cat image as dog). This paper shows that the concern is much more serious by learning significantly more aggressive ordered top- $K$  clearbox $^{1}$  targeted attacks proposed in [Zhang and Wu, 2020]. We propose a novel and rigorous quadratic programming (QP) method of learning ordered top- $K$  attacks with low computing cost, dubbed as QuadAttac $K$ . Our QuadAttac $K$  directly solves the QP to satisfy the attack constraint in the feature embedding space (i.e., the input space to the final linear classifier), which thus exploits the semantics of the feature embedding space (i.e., the principle of class coherence). With the optimized feature embedding vector perturbation, it then computes the adversarial perturbation in the data space via the vanilla one-step back-propagation. In experiments, the proposed QuadAttac $K$  is tested in the ImageNet-1k classification using ResNet50, DenseNet-121, and Vision Transformers (ViT-B and DEiT-S). It successfully pushes the boundary of successful ordered top- $K$  attacks from  $K = 10$  up to  $K = 20$  at a cheap budget  $(1 \times 60)$  and further improves attack success rates for  $K = 5$  for all tested models, while retaining the performance for  $K = 1$ .
As the development of machine deep learning and Artificial Intelligence (AI) continues to accelerate, the need to address potential vulnerabilities of Deep Neural Networks (DNNs) becomes increasingly crucial towards building safe-enabled and trustworthy learning and AI systems. Among these vulnerabilities, adversarial attacks [Xie et al., 2017, Kos et al., 2018, Sharif et al., 2016, Ebrahimi et al., 2018, Qin et al., 2019, Lin et al., 2017, Papernot et al., 2017, Liu et al., 2017, Xie et al., 2019b, Dong et al., 2019, Goodfellow et al., 2015, Kannan et al., 2018, Madry et al., 2017, Xie et al., 2019a], that can almost arbitrarily manipulate the prediction of a trained DNN for a given testing data by learning visually imperceptible perturbations, especially under clear-box target attack settings, are of particular interest in terms of revealing the shortcut learning of discriminatively-trained DNNs [Geirhos et al., 2020]. Clear-box targeted adversarial attacks are often learned under the top-1 attack setting with the objective of causing catastrophic performance drop of the top-1 accuracy on adversarial examples against the clean counterparts.
To generalize the vanilla setting of learning top-1 attacks, much more aggressive ordered top- $K$  attacks have been proposed in [Zhang and Wu, 2020], which aim to learn adversarial examples that manipulate a trained DNN to predict any specified  $K$  attack targets in any given order as the top- $K$  predicted classes. As shown in Fig. 5, in the ImageNet-1k [Russakovsky et al., 2015] classification
This is often referred to as white/black-box attacks in the literature. We choose to adopt neutral terminology, clear/opaque-box attacks in this paper, and omit the prefix clear-box for simplicity.
Clean Image (agama)
Clean Attention
Adv. Example
Adv. Perturbation
Adv. Attention
Adv. Example
5-5
Figure 1: Adversarial examples learned by our QuadAttacK ( $K = 20$ ) for a same clean image using four different networks: two popular convolutional neural networks (ResNet-50 [He et al., 2016] and DenseNet-121 [Huang et al., 2017]), and two widely used Vision Transformers (ViT-B [Dosovitskiy et al., 2020] and DEiT-S [Touvron et al., 2021]). The ground-truth label is agama. The ordered top-20 targets (randomly sampled and kept unchanged for the four models) are: [sea cucumber, barrow, odometer, bloodhound, hen-of-the-woods, ringneck snake, snail, tiger shark, Pembroke, altar, wig, submarine, macaw, combination lock, ram, Irish wolfhound, confectionery, buckle, chime, garden spider]. For example, the top-20 classes of the clean image by DEiT-S are [agama, frilled lizard, shovel, reel, common iguana, Yorkshire terrier, coho, alligator lizard, hand blower, meerkat, ostrich, mongoose, fiddler crab, eft, wing, bustard, green lizard, whiptail, brass, American chameleon], which have more or less visual similarities. More examples are the Appendix B.
Adv. Perturbation
eennnnnne
task, consider a clean testing image of 'Agama' that can be correctly classified by some trained DNNs such as the four networks we tested, let's say a desired list of the ordered top-20 attack targets as shown in the caption. An ordered top- $K$  attack method would seek to find the adversarial perturbation to the input image that manipulates the DNN's predictions to ensure that the top-20 predicted classes of the perturbed image match the specified list in the given order.
Why to learn ordered top- $K$  attacks? They facilitate exploiting the principle of "class coherence" that reflect real-world scenarios where the relative importance or priority of certain classes is crucial to recognize the relationships or logic connecting classes within ordinal or nominal context. Unlike unordered top- $K$  or top-1 attacks, ordered top- $K$  attacks can subtly manipulate predictions while maintaining coherence within the expected context.
- An Ordinal Example. Imagine a cancer risk assessment tool that analyzes 2D medical images (e.g., mammograms) to categorize patients' cancer risk into the ordinal 7-level risk ratings ([Extremely High Risk, Very High Risk, High Risk, Moderate Risk, Low Risk, Minimal Risk, No Risk]), An oncologist could use this tool to triage patients, prioritizing those in the highest risk categories for immediate intervention. An attacker aiming to delay treatment might use an ordered top-3 adversarial attack to change a prediction for a patient initially assessed as Very High Risk. They could target the classes [Moderate, Low, Minimal], subtly downgrading the urgency without breaking the logical sequence of risk categories. An unordered attack, in contrast, might lead to a sequence like [Low, Very High, Minimal], disrupting the ordinal relationship between classes. Such a disruption could raise red flags, making the attack easier to detect.
- A Nominal Example. Traffic control systems could use deep learning to optimize flow by adjusting the timing of traffic lights based on the types of vehicles seen. Priority might be given to certain vehicle classes, such as public transit or emergency vehicles, to improve response times. Imagine a city's traffic control system, which has specific traffic light timing behavior for the nominal vehicle categories [Emergency Vehicle, Public Transit, Commercial Vehicle, Personal Car, Bicycle]. Public transit might be given slightly extended green lights during rush hours to encourage public transportation use. An attacker wanting to cause delays for personal cars without raising alarms could launch an ordered top-2 adversarial attack, targeting the sequence [Commercial Vehicle, Public Transit]. This would cause the system to interpret most personal cars as commercial vehicles during the attack, applying the extended green light times meant for public transit to lanes primarily
Figure 2: Illustration of the proposed QuadAttack  $K$  method in comparison with the prior art (e.g., the adversarial distillation (AD) method [Zhang and Wu, 2020]).
used by commercial vehicles. An unordered top-2 attack that may result in [Emergency Vehicle, Commercial Vehicle], would likely be quickly detected, as emergency vehicle priority changes are significant and could be easily noticed by traffic operators (this weakness is exacerbated in any top-1 attack or unordered attacks).
Successful ordered top-  $K$  attacks can potentially provide several advantages: enabling better controllability in learning attacks that are more difficult to detect, revealing deeper vulnerability of trained DNNs, and testing the robustness of an attack method itself, especially when  $K$  is relatively large (e.g.,  $K \geq 15$ ) and the computing budget is relatively low (e.g., 60 steps of optimization).
Learning ordered top- $K$  attacks is an extremely challenging problem. The adversarial distillation method [Zhang and Wu, 2020] is the state of the art method (see Sec. 3.1), which presents a heuristic knowledge-oriented design of the ordered top- $K$  target distribution, and then minimizes the Kullback-Leibler divergence between the designed distribution and the DNN output distribution (after softmax). It often completely fails when  $K > 10$ . In this paper, we show it is possible to learn those attacks for a variety of DNNs, including ResNet-50 [He et al., 2016], DenseNet-121 [Huang et al., 2017] and Vision Transformers [Dosovitskiy et al., 2020, Touvron et al., 2021].
The key to learning clear-box targeted attacks ( $K \geq 1$ ) lies in the objective function for optimization, which usually consists of two terms, one is the  $\ell_p$  norm (e.g.,  $\ell_2$ ) of the learned adversarial perturbation (to be as small as possible to be visually imperceptible), and the other is the surrogate loss capturing the specified attack constraints such as the top- $K$  extended C&W (hinge) loss [Carlini and Wagner, 2017] and the adversarial distillation loss proposed in [Zhang and Wu, 2020] (see Sec. 3.1). The trade-off between the two terms are often searched in optimization with respect to a certain computing budget (e.g.,  $9 \times 30$  means to test 9 different trade-off parameter assignments based on linear search in a predefined range, and to run 30 forward-backward computation iterations of the DNN per trade-off parameter search step). After the optimization, Attack Success Rates (ASR, higher is better) and some  $\ell_p$  norms (e.g.,  $\ell_1$  and  $\ell_2$ ) of learned successful perturbations (smaller is better) are used as evaluation metrics. As illustrated in Fig. 2, in this paper, we propose a novel formulation which is different from the prior art in the three aspects as follows:
- We identify that while sufficient to capture the top- $K$  attack constraint, hand-crafted surrogate losses are not necessary and often introduce inconsistency and artifacts in optimization (see Sec. 3.2). We eliminate the need of introducing surrogate losses. Instead, we keep the top- $K$  attack constraints in the vanilla form and cast the optimization problem as quadratic programming (QP). We solve the QP by leveraging a recently proposed differentiable QP layer (for PyTorch) [Amos and Kolter, 2017]. We present an efficient implement to parallelize the batched QP layer for any ordered top- $K$  targets specified individually for each instance in a batch.
- We observe that directly minimizing the  $\ell_p$  norm of learned perturbations together with the handcrafted surrogate loss could miss the chance of exploiting semantic structures of the feature embedding space (i.e., the input to the final linear classifier). Instead, we minimize the Euclidean distance between the feature embedding vectors at two consecutive iterations in the optimization. This can be understood as the latent perturbation learning versus the raw data perturbation learning (see Sec. 3.3). Our proposed latent perturbation learning enables more consistent optimization trajectories in pursuing the satisfaction of the specified top- $K$  attack constraints. The minimized Euclidean distance is then used as the loss together with the  $\ell_p$  norm of the learned perturbation in computing the adversarial perturbation via back-propagation at each iteration.
2 Related Work and Our Contributions
Adversarial Attacks have remained as a critical concern for DNNs, where imperceptible perturbations to input data can lead to significant misclassification [Xie et al., 2017, Hendrik Metzen et al., 2017, Chen et al., 2019, Liu et al., 2018]. Various approaches have been proposed to investigate the vulnerabilities of DNNs and exploit their sensitivity to non-robust features. Notable works include the seminal discovery of visually imperceptible adversarial attacks [Szegedy et al., 2013], which highlighted the need to evaluate the brittleness of DNNs and address it with explicit defense methods [Madry et al., 2017, Cohen et al., 2019, Wang et al., 2020, Wong et al., 2020]. Clear-box attacks, which assume full access to the DNN model, have been particularly effective in uncovering vulnerabilities [Madry et al., 2017]. The C&W method [Carlini and Wagner, 2017], for instance, introduced loss functions that have been widely adopted to generate adversarial examples. Momentum-based methods like [Dong et al., 2018, Lin et al., 2019] and gradient projection techniques like PGD [Madry et al., 2017] have also been successful in crafting adversarial examples.
Unordered top- $K$  Attacks aim to manipulate the top- $K$  predicted classes of a DNN without enforcing a specific order among them. A common approach is to optimize a loss function that combines multiple objectives, such as maximizing the probability of the target classes while minimizing the probabilities of other classes simultaneously. [Tursynbek et al., 2022] presents a geometry inspired method that allows taking gradient steps in favor of simultaneously maximizing all target classes while maintaining a balance between them. Other approaches, for example, may use sorting strategies [Kumano et al., 2022] to limit the set of logits involved simultaneously in a loss and target specific logits that contribute most to a misclassification. The "Superclass" attacks have been proposed in [Kumano et al., 2022], for which ordered top- $K$  attacks can be seen as a generalization. Non-targeted top- $K$  attacks are studied in [Zhang et al., 2022].
Ordered top- $K$  Attacks [Zhang and Wu, 2020] require preserving both the attack success rate and the order of the top- $K$  predicted classes. Following the intuitions from [Kumano et al., 2022] we note that adversarial sample detection and defense methods [Lee et al., 2018, Huang and Li, 2021, Aldahdooh et al., 2022] may benefit from the fact that many adversarial attacks tend to generate a nonsensical class prediction (e.g. the top- $K$  predictions may all be from a different super-class [Kumano et al., 2022]). An ordered top- $K$  adversarial attack may choose to create an attack that is semantically plausible and have a higher potential to fool detection methods and defense methods. The sorting constraint in ordered top- $K$  attacks adds a layer of complexity to the optimization problem. By enforcing a specific order among the top- $K$  predictions, the attacker must not only manipulate the logits to maximize the target classes' probabilities but also ensure that the predicted order aligns with the desired order. In [Zhang and Wu, 2020], two methods are presented through the use of semantic information in the form of class language embedding.
Constrained Optimization in DNNs, such as OptNet [Amos and Kolter, 2017] and other differentiable optimization works [Agrawal et al., 2019, Butler and Kwon, 2023], have introduced powerful formulations for integrating optimization layers within the network. These types of works have produced many DNN based problem solutions while also being able to use domain knowledge to constrain solutions and reduce data requirements [Sangalli et al., 2021] and solve problems otherwise intractable with DNNs [Wang et al., 2019, Mandi et al., 2020]. While OptNet's quadratic solver is typically used as a layer, our focus is on attacking a pre-trained model with a fixed architecture. Thus, we leverage the principles of constrained optimization to formulate an objective function that captures the constraints of ordered top- $K$  adversarial attacks. This adaptation allows us to guide the attack process to satisfy the ordering constraints while maximizing target class probabilities, which provides new insights for enhancing attack effectiveness and robustness.
Our Contributions This paper makes two main contributions to the field of learning clear-box targeted adversarial attacks:
- It presents a quadratic programming (QP) approach to learning ordered top- $K$  attacks, dubbed as QuadAttac $K$ . It eliminates hand-crafting surrogate loss functions to capture the top- $K$  attack constraint. It provides a QP based formulation to better exploit the semantics of the feature embedding space in capturing the top- $K$  attack requirement.
- It obtains state-of-the-art adversarial attack performance in ImageNet-1k classification using both ConvNets (ResNet-50 and DenseNet-121) and Transformer models (ViT-B and DEiT-S). It pushes the limit of the number of targets,  $K$  to a large number, for which the prior art completely fails.
In this section, we first define the problem of learning ordered top- $K$  attacks following [Zhang and Wu, 2020], as well as the extended top- $K$  C&W  $(\mathrm{CW}^K)$  method and the adversarial distillation (AD) method. We then present details of our proposed QP based formulation, i.e., QuadAttac  $K$ .
3.1 Learning Ordered top- $K$  Clear-Box Targeted Attacks: the Problem and the Prior Art
We consider image classification DNNs, which consist of a feature backbone and a head linear classifier. Denote by  $F: x \in \mathbb{R}^{3 \times H \times W} \to z \in \mathbb{R}^{D \times h \times w}$  the feature backbone which transforms an input image to its feature map, where an input image  $x$  is often a RGB image of the spatial height and width,  $H$  and  $W$  (e.g.,  $224 \times 224$ ), clean or perturbed, and the feature map  $z$  is in a  $D$ -dim feature space with the spatial height and width,  $h$  and  $w$  (e.g.,  $7 \times 7$ ) based on the overall spatial downsampling stride implemented in the feature backbone. Let  $\bar{z} \in \mathbb{R}^D$  be feature embedding vector for  $x$  via any spatial reduction method (e.g. global average pooling). Denote by  $f: \bar{z} \in \mathbb{R}^D \to l \in \mathbb{R}^C$  the linear head classifier, where  $C$  is the number of classes (e.g., 1000 in ImageNet-1k [Russakovsky et al., 2015]), and  $l$  is the output logit vector. We have,
$$
l = f (F (x; \theta); A, B) = A \cdot \bar {z} + B, \tag {1}
$$
where  $\theta$  collects all the model parameters of the feature backbone, and  $A\in \mathbb{R}^{C\times D}$  and  $B\in \mathbb{R}^C$  the weight and bias parameters of the head linear classifier. In learning clear-box attacks, we assume all the information of the network are available, and the parameters  $(\theta ,A,B)$  are frozen throughout.
Denote by  $(x,y)$  a pair of clean image and its ground-truth label  $(y\in \mathcal{V} = \{1,2,\dots ,C\})$  . For learning attacks, we assume  $x$  can be correctly classified by the network, i.e.,  $y = \arg \max_{i}l_{i}$  Denote by  $T$  the ordered list of attack target(s) with the cardinality  $K = |T|$  , where the groundtruth label is excluded,  $y\notin T$  , and by  $T^c = \mathcal{V}\setminus T$  the complement set. Let  $\delta (x,T;F,f)$  be the adversarial perturbation to be learned. An ordered top-  $K$  adversarial example is defined by  $\hat{x} = x + \delta (x,T;F,f)$  if the top-  $K$  prediction classes for  $\hat{x}$  equal to  $T$  based on its logits  $\hat{l}$  (Eqn. 1).
Learning ordered top- $K$  attacks is often posed as a constrained optimization problem,
$$
\underset {\delta} {\text {m i n i m i z e}} \quad | | \delta | | _ {p}, \tag {2}
$$
$$
\text {s u b j e c t} \quad \hat {l} _ {t _ {i}} > \hat {l} _ {t _ {i + 1}}, \quad i \in [ 1, K - 1 ], \quad t _ {i} \in T,
$$
$$
\hat {l} _ {t _ {K}} > \hat {l} _ {j}, \qquad t _ {K} \in T, \quad \forall j \in T ^ {c},
$$
$$
\hat {x} = x + \delta \in [ 0, 1 ] ^ {3 \times H \times W},
$$
$$
\hat {l} = f (F (\hat {x}; \theta); A, B),
$$
where the first two constraints capturing the ordered top- $K$  attack requirement. This traditional formulation leads to the challenge in optimization, even with  $K = 1$  as pointed out in the vanilla C&W method [Carlini and Wagner, 2017]. Some surrogate loss,  $\mathcal{L}(\hat{x})$ , is necessary to ensure the first two terms are satisfied when  $\hat{\mathcal{L}}(\hat{x})$  is minimized. We have,
$$
\text {m i n i m i z e} \quad \lambda \cdot \mathcal {L} (\hat {x}) + | | \delta | | _ {p}, \tag {3}
$$
$$
\text {s u b j e c t} \quad \hat {x} = x + \delta \in [ 0, 1 ] ^ {3 \times H \times W},
$$
where  $\lambda$  is the trade-off parameter between the visual imperceptibility of learned perturbations and the ASR. The remaining constraint can be addressed via a projected descent in optimization. So, the optimization can enjoy the straightforward back-propagation algorithm that is used in training the DNN on clean images.
The extended top-  $K$  C&W (hinge) loss [Zhang and Wu, 2020] is defined by,
$$
\mathcal {L} _ {C W} ^ {K} (\hat {x}) = \sum_ {i = 1} ^ {K} \max  \left(0, \max  _ {j \in \mathcal {Y} \backslash \{t _ {1}, \dots , t _ {i} \}} \hat {l} _ {j} - \min  _ {t \in \{t _ {1}, \dots , t _ {i} \}} \hat {l} _ {t}\right). \tag {4}
$$
And, the adversarial distillation loss [Zhang and Wu, 2020] is defined by,
$$
\mathcal {L} _ {A D} ^ {K} (\hat {x}) = K L (\hat {p} | | P ^ {A D}) = \sum_ {t _ {i} \in T} \hat {p} _ {t _ {i}} \left(\log \hat {p} _ {t _ {i}} - \log P _ {t _ {i}} ^ {A D}\right) + \sum_ {j \in T ^ {c}} \hat {p} _ {j} \left(\log \hat {p} _ {j} - \log P _ {j} ^ {A D}\right), \tag {5}
$$
where  $\hat{p} = \mathrm{Softmax}(\hat{l})$  and  $P^{AD}$  is the knowledge-oriented heuristically-designed adversarial distribution with the top- $K$  constraints satisfied (see [Zhang and Wu, 2020] for details).  $KL(\cdot ||\cdot)$  is the Kullback-Leibler divergence between two distributions.
3.2 Limitations of the Prior Art
From the optimization perspective, we observe there are two main drawbacks in the aforementioned formulations (Eqns. 3, 4, 5):
- The two surrogate loss formulations (Eqns. 4 and 5) are sufficient, but not necessary. They actually introduce inconsistency and artifacts in optimization. The extended top- $K$  C&W loss in Eqn. 4 is not aware of, and thus can not preserve, the subset of targets whose relative order has been satisfied. For example, consider there are 5 classes in total, and a specified ordered top-3 list of targets, [2, 3, 1]. Assume at a certain iteration, the predicted classes for  $\hat{x}$  in sort are  $[4, 2, 3, 5, 1]$ , in which the relative order of the specified 3 targets has been satisfied. The loss  $\mathcal{L}_{CW}^{3}(\hat{x}) = \sum_{i=1}^{3} (\hat{l}_{4} - \hat{l}_{i})$ , which mainly focuses on pushing down the logit  $\hat{l}_{4}$  and/or pulling up the logits,  $\hat{l}_{i}$ 's  $(i = 1, 2, 3)$ . So, at the next iteration, it may results in the sorted prediction like  $[1, 3, 2, 4, 5]$ , leading to a totally wrong relative order. The adversarial distillation loss in Eqn. 5 has similar problems, i.e., the first part,  $\sum_{i \in T} \hat{p}_{i} (\log \hat{p}_{i} - \log P_{i}^{AD})$  is not aware of some satisfied relative order. It further enforces order between non-target classes since the adversarial distribution  $P^{AD}$  needs to be specified before the optimization, as shown in the second part,  $\sum_{j \in T^{c}} \hat{p}_{j} (\log \hat{p}_{j} - \log P_{j}^{AD})$ .
- Directly minimizing  $||\delta||_p$  in Eqn. 3 (i.e., adversarial learning in the data space) may actually hinder the effectiveness of learning adversarial examples due to the fact that the  $\ell_p$  norm is totally unaware of the underlying data structures in the complex data space. Since a trained DNN is kept frozen in learning attacks, we can first perform adversarial learning in the feature embedding space (i.e., the head linear classifier's input space,  $\bar{z}$  in Eqn. 1), which has been learned to be discriminatively and/or semantically meaningful. With a learned adversarial perturbation for  $\bar{z}$ , we can easily compute the perturbation for the input data.
We address these limitations in this paper by proposing a novel QP based formulation - QuadAttack  $K$ . We present the detail of our QuadAttac  $K$  in the following sub-sections.