SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xingjian Bai
xingjian.bai@sjc.ox.ac.uk
Guangyi He
Mathematical Institute
guangyihe2002@outlook.com
Yifan Jiang
yifan.jiang@maths.ox.ac.uk
Jan Obloj*
jan.obloj@maths.ox.ac.uk
Deep neural networks are known to be vulnerable to adversarial attacks (AA). For an image recognition task, this means that a small perturbation of the original can result in the image being misclassified. Design of such attacks as well as methods of adversarial training against them are subject of intense research. We re-cast the problem using techniques of Wasserstein distributionally robust optimization (DRO) and obtain novel contributions leveraging recent insights from DRO sensitivity analysis. We consider a set of distributional threat models. Unlike the traditional pointwise attacks, which assume a uniform bound on perturbation of each input data point, distributional threat models allow attackers to perturb inputs in a nonuniform way. We link these more general attacks with questions of out-of-sample performance and Knightian uncertainty. To evaluate the distributional robustness of neural networks, we propose a first-order AA algorithm and its multistep version. Our attack algorithms include Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) as special cases. Furthermore, we provide a new asymptotic estimate of the adversarial accuracy against distributional threat models. The bound is fast to compute and first-order accurate, offering new insights even for the pointwise AA. It also naturally yields out-of-sample performance guarantees. We conduct numerical experiments on CIFAR-10, CIFAR-100, ImageNet datasets using DNNs on RobustBench to illustrate our theoretical results. Our code is available at https://github.com/JanObloj/W-DRO-Adversarial-Methods.
Model uncertainty is a ubiquitous phenomenon across different fields of science. In decision theory and economics, it is often referred to as the Knightian uncertainty (Knight, 1921), or the unknown unknowns, to distinguish it from the risk which stems from the randomness embedded by design in the scientific process, see Hansen and Marinacci (2016) for an overview. Transcribing to the context of data science, risk refers to the randomness embedded in a training by design, e.g., through random initialization, drop-outs etc., and uncertainty encompasses the extent to which the dataset is an adequate description of reality. Robustness, the ability to perform well under uncertainty, thus relates to several themes in ML including adversarial attacks, out-of-sample performance and
*Corresponding author. www.maths.ox.ac.uk/people/jan.obloj
out-of-distribution performance. In this work, we mainly focus on the former but offer a unified perspective on robustness in all of its facets.
Vulnerability of DNNs to crafted adversarial attacks (AA), diagnosed in Biggio et al. (2013), Goodfellow et al. (2015), relates to the ability of an attacker to manipulate network's outputs by changing the input images only slightly – often in ways imperceptible to a human eye. As such, AA are of key importance for security-sensitive applications and an active field of research. Most works so far have focused on attacks under pointwise  $l_{p}$ -bounded image distortions but a growing stream of research, pioneered by Staib and Jegelka (2017) and Sinha et al. (2018), frames the problem using Wasserstein distributionally robust optimization (DRO). We offer novel contributions to this literature.
Our key contributions can be summarized as follows. 1) We propose a unified approach to adversarial attacks and training based on sensitivity analysis for Wasserstein DRO. We believe this approach, leveraging results from Bartl et al. (2021), is better suited for gradient-based optimization methods than duality approach adopted in most of the works to date. We further link the adversarial accuracy to the adversarial loss, and investigate the out-of-sample performance. 2) We derive a general adversarial attack method. As a special case, this recovers the classical FGSM attack lending it a further theoretical underpinning. However, our method also allows one to carry out attacks under a distributional threat model which, we believe, has not been done before. We also propose a rectified DLR loss suitable for the distributional attacks. 3) We develop asymptotically certified bounds on adversarial accuracy, applicable to a general threat, including the classical pointwise perturbations. The bounds are first-order accurate and much faster to compute than, e.g., the AutoAttack (Croce and Hein, 2020) benchmark. The performance of our methods is documented using CIFAR-10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), ImageNet (Deng et al., 2009) datasets and neural networks from RobustBench (Croce et al., 2021).
Adversarial Attack (AA). Original research focused on the pointwise  $l_{p}$ -bounded image distortion. Numerous attack methods under this threat model have been proposed in the literature, including Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015), Projected Gradient Descent (PGD) (Madry et al., 2018), CW attack (Carlini and Wagner, 2017), etc. In these white-box attacks, the attacker has full knowledge of the neural network. There are also black-box attacks, such as Zeroth Order Optimization (ZOO) (Chen et al., 2017), Boundary Attack (Brendel et al., 2018), and Query-limited Attack (Ilyas et al., 2018). AutoAttack (Croce and Hein, 2020), an ensemble of white-box and black-box attacks, provides a useful benchmark for pointwise  $l_{p}$ -robustness of neural networks. Notably Hua et al. (2022) considered AA with  $l_{p}$  distance replaced by a proxy for human eye evaluation.
Adversarial Defense. Early works on data augmentation (Goodfellow et al., 2015, Madry et al., 2018, Tramér et al., 2018) make use of strong adversarial attacks to augment the training data with adversarial examples; more recent works (Gowal et al., 2021, Xing et al., 2022, Wang et al., 2023) focus on adding randomness to training data through generative models such as GANs and diffusion models. Zhang et al. (2019) consider the trade-off between robustness and accuracy of a neural network via TRADES, a regularized loss. Analogous research includes MART (Wang et al., 2020) and SCORE (Pang et al., 2022). Other loss regularization methods such as adversarial distributional training (Dong et al., 2020) and adversarial weight perturbation (Wu et al., 2020) have been shown to smooth the loss landscape and improve the robustness. In addition, various training techniques can be overlaid to improve robustness, including droup-out layers, early stopping and parameter fine-tuning Sehwag et al. (2020). The closest to our setting are Sinha et al. (2018), García Trillos and García Trillos (2022) which employ Wasserstein penalization and constraint respectively. However, so far, even the papers which used distributional threat models to motivate DRO-based training methods actually used classical pointwise attacks to evaluate robustness of their trained DNNs. This highlights the novelty of our distributional threat attack.
Robust Performance Bounds. Each AA method gives a particular upper bound on the adversarial accuracy of the network. In contrast, research on certified robustness aims at classifying images which are robust to all possible attacks allowed in the threat model and thus providing an attack-agnostic lower bound on the classification accuracy. To verify robustness of images, deterministic methods using off-the-shelf solvers (Tjeng et al., 2019), relaxed linear programming (Wong and Kolter, 2018,
Weng et al., 2018a) or semi-definite programming (Raghunathan et al., 2018, Dathathri et al., 2020) have been applied. Hein and Andriushchenko (2017), Weng et al. (2018b) derive Lipschitz-based metrics to characterize the maximum distortion an image can uphold; Cohen et al. (2019) constructs a certifiable classifier by adding smooth noise to the original classifier; see Li et al. (2023) for a review.
Distributionally Robust Optimization (DRO). Mathematically, it is formulated as a min-max problem
$$
\inf  _ {\theta \in \Theta} \sup  _ {Q \in \mathcal {P}} \mathbf {E} _ {Q} [ f _ {\theta} (Z) ], \tag {1}
$$
where we minimize the worst-case loss over all possible distributions  $Q \in \mathcal{P}$ . In financial economics, such criteria appear in the context of multi-prior preferences, see (Gilboa and Schmeidler, 1989, Föllmer and Weber, 2015). We refer to (Rahimian and Mehrotra, 2019) for a survey of the DRO.
We focus on the Wasserstein ambiguity set  $\mathcal{P} = B_{\delta}(P)$ , which is a ball centered at the reference distribution  $P$  with radius  $\delta$  under the Wasserstein distance. We refer to Gao and Kleywegt (2022) for a discussion of many advantages of this distance. In particular, measures close to each other can have different supports which is key in capturing data perturbations, see Sinha et al. (2018). Staib and Jegelka (2017) interpreted pointwise adversarial training as a special case of Wasserstein DRO (W-DRO). Volpi et al. (2018) utilized W-DRO to improve network performance on unseen data distributions. More recently, Bui et al. (2022) unified various classical adversarial training methods, such as PGD-AT, TRADES, and MART, under the W-DRO framework.
W-DRO, while compelling theoretically, is often numerically intractable. In the literature, two lines of research have been proposed to tackle this problem. The duality approach rewrites (1), changing the sup to a univariate inf featuring a transform of  $f_{\theta}$ . We refer to Mohajerin Esfahani and Kuhn (2018) for the data-driven case, Blanchet and Murthy (2019), Bartl et al. (2020), Gao and Kleywegt (2022) for general probability measures and Huang et al. (2022) for a further application with coresets. The second approach, which we adopt here, considers the first order approximation to the original DRO problem. This can be seen as computing the sensitivity of the value function with respect to the model uncertainty as derived in Bartl et al. (2021), see also Lam (2016), García Trillos and García Trillos (2022) for analogous results in different setups.
Image Classification Task. An image is interpreted as a tuple  $(x,y)$  where the feature vector  $x\in \mathcal{X}$  encodes the graphic information and  $y\in \mathcal{Y} = \{1,\dots ,m\}$  denotes the class, or tag, of the image. W.l.o.g., we take  $\mathcal{X} = [0,1]^n$  . A distribution of labelled images corresponds to a probability measure  $P$  on  $\mathcal{X}\times \mathcal{Y}$  . We are given the training set  $\mathcal{D}_{tr}$  and the test set  $\mathcal{D}_{tt}$  , subsets of  $\mathcal{X}\times \mathcal{Y}$  , i.i.d. sampled from  $P$  . We denote  $\hat{P}$  (resp.  $\check{P})$  the empirical measure of points in the training set (resp. test set), i.e.,  $\hat{P} = \frac{1}{|\mathcal{D}_{tr}|}\sum_{(x,y)\in \mathcal{D}_{tr}}\delta_{(x,y)}$  . A neural network is a map  $f_{\theta}:\mathcal{X}\to \mathbb{R}^{m}$
$$
f _ {\theta} (x) = f ^ {l} \circ \dots \circ f ^ {1} (x), \quad \text {w h e r e} f ^ {i} (x) = \sigma \left(w ^ {i} x + b ^ {i}\right),
$$
$\sigma$  is a nonlinear activation function, and  $\theta = \{w^i, b^i : 1 \leqslant i \leqslant l\}$  is the collection of parameters. We denote  $S$  the set of images equipped with their labels generated by  $f_{\theta}$ , i.e.,
$$
S = \Bigl \{(x, y) \in \mathcal {X} \times \mathcal {Y}: \arg \max  _ {1 \leqslant i \leqslant m} f _ {\theta} (x) _ {i} = \{y \} \Bigr \}.
$$
The aim of image classification is to find a network  $f_{\theta}$  with high (clean) prediction accuracy  $A := P(S) = \mathbf{E}_P[\mathbb{1}_S]$ . To this end,  $f_{\theta}$  is trained solving the stochastic optimization problem
$$
\inf  _ {\theta \in \Theta} \mathbf {E} _ {P} [ L (f _ {\theta} (x), y) ], \tag {2}
$$
where  $\Theta$  denotes the set of admissible parameters, and  $L$  is a (piecewise) smooth loss function, e.g., cross entropy loss $^3$ CE:  $\mathbb{R}^m \times \mathcal{V} \to \mathbb{R}$  given by
$$
\operatorname {C E} (z, y) = - (\log \circ \operatorname {s o f t m a x} (z)) _ {y}. \tag {3}
$$
In practice,  $P$  is not accessible and we use  $\hat{P}$  or  $\check{P}$  instead, e.g., in (2) we replace  $P$  with  $\hat{P}$  and then compute the clean accuracy as  $\check{P}(S)$ . In our experiments we make it clear which dataset is used.
<sup>3</sup>By convention, cross entropy is a function of two probability measures. In this case, we implicitly normalize the logit  $z$  by applying softmax, and we associate a class  $y$  with the Dirac measure  $\delta_y$ .
Wasserstein Distances. Throughout,  $(p,q)$  is a pair of conjugate indices,  $1 / p + 1 / q = 1$ , with  $1 \leqslant p \leqslant \infty$ . We consider a norm  $\| \cdot \|$  on  $\mathcal{X}$  and denote  $\| \cdot \|_*$  its dual,  $\| \tilde{x} \|_* = \sup \{\langle x, \tilde{x} \rangle : \| x \| \leqslant 1\}$ . Our main interest is in  $\| \cdot \| = \| \cdot \|_r$  the  $l_r$ -norm for which  $\| \cdot \|_* = \| \cdot \|_s$ , where  $(r,s)$  are conjugate indices,  $1 \leqslant r \leqslant \infty$ . We consider adversarial attacks which perturb the image feature  $x$  but not its label  $y$ . Accordingly, we define a pseudo distance  $d$  on  $\mathcal{X} \times \mathcal{Y}$  as
$$
d \left(\left(x _ {1}, y _ {1}\right), \left(x _ {2}, y _ {2}\right)\right) = \left\| x _ {1} - x _ {2} \right\| + \infty \mathbb {1} _ {\left\{y _ {1} \neq y _ {2} \right\}}. \tag {4}
$$
We denote  $\Pi(P, Q)$  the set of couplings between  $(x, y)$  and  $(x', y')$  whose first margin is  $P$  and second margin is  $Q$ , and  $T_{\#}P \coloneqq P \circ T^{-1}$  denotes the pushforward measure of  $P$  under a map  $T$ .
The  $p$ -Wasserstein distance,  $1 \leqslant p < \infty$ , between probability measures  $P$  and  $Q$  on  $\mathcal{X} \times \mathcal{Y}$  is
$$
\mathcal {W} _ {p} (P, Q) := \inf  \left\{\mathbf {E} _ {\pi} \left[ d ((x _ {1}, y _ {1}), (x _ {2}, y _ {2})) ^ {p} \right]: \pi \in \Pi (P, Q) \right\} ^ {1 / p}. \tag {5}
$$
The  $\infty$ -Wasserstein distance  $\mathcal{W}_{\infty}$  is given by
$$
\mathcal {W} _ {\infty} (P, Q) := \inf  \left\{\pi - \operatorname {e s s} \sup  d \left(\left(x _ {1}, y _ {1}\right), \left(x _ {2}, y _ {2}\right)\right): \pi \in \Pi (P, Q) \right\}. \tag {6}
$$
We denote the  $p$ -Wasserstein ball centered at  $P$  with radius  $\delta$  by  $B_{\delta}(P)$ . We mainly consider the cases where  $p, r \in \{2, \infty\}$ . Intuitively, we can view  $p$  as the index of image-wise flexibility and  $r$  as the index of pixel-wise flexibility. Unless  $p = 1$  is explicitly allowed,  $p > 1$  in what follows.
4 Wasserstein Distributional Robustness: adversarial attacks and training
W-DRO Formulation. The Wasserstein DRO (W-DRO) formulation of a DNN training task is given by:
$$
\inf  _ {\theta \in \Theta} \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} [ L (f _ {\theta} (x), y) ], \tag {7}
$$
where  $B_{\delta}(P)$  is the  $p$ -Wasserstein ball centered at  $P$  and  $\delta$  denotes the budget of the adversarial attack. In practice,  $P$  is not accessible and is replaced with  $\hat{P}$ . When  $p = \infty$ , the above adversarial loss coincides with the pointwise adversarial loss of Madry et al. (2018) given by
$$
\inf  _ {\theta \in \Theta} \mathbf {E} _ {P} \left[ \sup  \left\{L \left(f _ {\theta} \left(x ^ {\prime}\right), y\right): \| x ^ {\prime} - x \| \leqslant \delta \right\} \right].
$$
Recently, Bui et al. (2022) considered a more general criterion they called unified distributional robustness. It can be re-cast equivalently as an extended W-DRO formulation using couplings:
$$
\inf  _ {\theta \in \Theta} \sup  _ {\pi \in \Pi_ {\delta} (P, \cdot)} \mathbf {E} _ {\pi} \left[ J _ {\theta} \left(x, y, x ^ {\prime}, y ^ {\prime}\right) \right], \tag {8}
$$
where  $\Pi_{\delta}(P,\cdot)$  is the set of couplings between  $(x,y)$  and  $(x^{\prime},y^{\prime})$  whose first margin is  $P$  and the second margin is within a Wasserstein  $\delta$ -ball centered at  $P$ . This formulation was motivated by the observation that for  $p = \infty$ , taking  $J_{\theta}(x,y,x^{\prime},y^{\prime}) = L(f_{\theta}(x),y) + \beta L(f_{\theta}(x),f_{\theta}(x^{\prime}))$ , it retrieves the TRADES loss of (Zhang et al., 2019) given by
$$
\inf_{\theta \in \Theta}\mathbf{E}_{P}\Bigl[L(f_{\theta}(x),y) + \beta \sup_{x^{\prime}: \| x - x^{\prime}\| \leqslant \delta}L(f_{\theta}(x),f_{\theta}(x^{\prime}))\Bigr ].
$$
W-DRO Sensitivity. In practice, training using (7), let alone (8), is computationally infeasible. To back propagate  $\theta$  it is essential to understand the inner maximization problem denoted by
$$
V (\delta) = \sup  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} \left[ J _ {\theta} (x, y) \right],
$$
where we write  $J_{\theta}(x,y) = L(f_{\theta}(x),y)$ . One can view the adversarial loss  $V(\delta)$  as a certain regularization of the vanilla loss. Though we are not able to compute the exact value of  $V(\delta)$  for neural networks with sufficient expressivity, DRO sensitivity analysis results allow us to derive a numerical approximation to  $V(\delta)$  and further apply gradient-based optimization methods. This is the main novelty of our approach — previous works considering a W-DRO formulation mostly relied on duality results in the spirit of Blanchet and Murthy (2019) to rewrite (7).
Our results can be adapted to regression tasks where the class label  $y$  is continuous and sensitive to the perturbation. In such a setting a different  $d$  would be appropriate.
Assumption 4.1. We assume the map  $(x,y)\mapsto J_{\theta}(x,y)$  is L-Lipschitz under  $d$  , i.e.,
$$
\left| J _ {\theta} \left(x _ {1}, y _ {1}\right) - J _ {\theta} \left(x _ {2}, y _ {2}\right) \right| \leqslant \mathsf {L} d \left(\left(x _ {1}, y _ {1}\right), \left(x _ {2}, y _ {2}\right)\right).
$$
The above assumption is weaker than the L-smoothness assumption encountered in the literature which requires Lipschitz continuity of gradients of  $J_{\theta}$ , see for example Sinha et al. (2018)[Assumption B] and Volpi et al. (2018)[Assumptions 1 & 2]. We also remark that our assumption holds for any continuously differentiable  $J_{\theta}$  as the feature space  $\mathcal{X} = [0,1]^n$  is compact.
The following result follows readily from (Bartl et al., 2021, Theorem 2.2) and its proof.
Theorem 4.1. Under Assumption 4.1, the following first order approximations hold:
(i)  $V(\delta) = V(0) + \delta \Upsilon + o(\delta)$ , where
$$
\Upsilon = \left(\mathbf {E} _ {P} \| \nabla_ {x} J _ {\theta} (x, y) \| _ {*} ^ {q}\right) ^ {1 / q}.
$$
(ii)  $V(\delta) = \mathbf{E}_{Q_{\delta}}[J_{\theta}(x,y)] + o(\delta)$ , where
$$
Q _ {\delta} = \left[ (x, y) \mapsto \left(x + \delta h (\nabla_ {x} J _ {\theta} (x, y)) \| \Upsilon^ {- 1} \nabla_ {x} J _ {\theta} (x, y) \| _ {*} ^ {q - 1}, y\right) \right] _ {\#} P,
$$
and  $h$  is uniquely determined by  $\langle h(x),x\rangle = \| x\|_{*}$
The above holds for any probability measure, in particular with  $P$  replaced consistently by an empirical measure  $\hat{P}$  or  $\check{P}$ . In Figure 1, we illustrate the performance of our first order approximation of the adversarial loss on CIFAR-10 (Krizhevsky, 2009) under different threat models.
Figure 1: Performance of the first order approximation for the W-DRO value derived in Theorem 4.1. Left: WideResNet-28-10 (Gowal et al., 2020) under CE loss (3) and  $(\mathcal{W}_{\infty}, l_{\infty})$  threat model with  $\delta = 1/255, \ldots, 10/255$ . Right: WideResNet-28-10 (Wang et al., 2023) under ReDLR loss (10) and  $(\mathcal{W}_2, l_2)$  threat models with  $\delta = 1/16, \ldots, 10/16$ .
WD-Adversarial Accuracy. We consider an attacker with perfect knowledge of the network  $f_{\theta}$  and the data distribution  $P$ , aiming to minimize the prediction accuracy of  $f_{\theta}$  under an admissible attack. Complementing the W-DRO training formulation, Staib and Jegelka (2017), Sinha et al. (2018) proposed Wasserstein distributional threat models under which an attack is admissible if the resulting attacked distribution  $Q$  stays in the  $p$ -Wasserstein ball  $B_{\delta}(P)$ , where  $\delta$  is the attack budget, i.e., the tolerance for distributional image distortion. We define the adversarial accuracy as:
$$
A _ {\delta} := \inf  _ {Q \in B _ {\delta} (P)} Q (S) = \inf  _ {Q \in B _ {\delta} (P)} \mathbf {E} _ {Q} [ \mathbb {1} _ {S} ]. \tag {9}
$$
Note that  $A_{\delta}$  is decreasing in  $\delta$  with  $A_0 = A$ , the clean accuracy. For  $p = \infty$ , the Wasserstein distance essentially degenerates to the uniform distance between images and hence the proposed threat model coincides with the popular pointwise threat model. For  $1 \leqslant p < \infty$ , the distributional threat model is strictly stronger than the pointwise one, as observed in Staib and Jegelka (2017, Prop. 3.1). Intuitively, it is because the attacker has a greater flexibility and can perturb images close to the decision boundary only slightly while spending more of the attack budget on images farther away from the boundary. The threat is also closely related to out-of-distribution generalization, see Shen et al. (2021) for a survey.
WD-Adversarial Attack. We propose Wasserstein distributionally adversarial attack methods. As mentioned above, to date, even the papers which used distributional threat models to motivate DRO-based training methods then used classical pointwise attacks to evaluate robustness of their trained DNNs. Our contribution is novel and enabled by the explicit first-order expression for the distributional attack in Theorem 4.1(ii), which is not accessible using duality methods.
We recall the Difference of Logits Ratio (DLR) loss of Croce and Hein (2020). If we write  $z = (z_{1},\ldots ,z_{m}) = f_{\theta}(x)$  for the output of a neural network, and  $z_{(1)}\geqslant \dots \geqslant z_{(m)}$  are the order statistics of  $z$ , then DLR loss is given by
$$
\mathrm {D L R} (z, y) = \left\{ \begin{array}{l l} - \frac {z _ {y} - z _ {(2)}}{z _ {(1)} - z _ {(3)}}, & \text {i f} z _ {y} = z _ {(1)}, \\ - \frac {z _ {y} - z _ {(1)}}{z _ {(1)} - z _ {(3)}}, & \text {e l s e}. \end{array} \right.
$$
The combination of CE loss and DLR loss has been widely shown as an effective empirical attack for pointwise threat models. However, under distributional threat models, intuitively, an effective attack should perturb more aggressively images classified far from the decision boundary and leave the misclassified images unchanged. Consequently, neither CE loss nor DLR loss are appropriate — this intuition is confirmed in our numerical experiments, see Table 1 for details. To rectify this, we propose ReDLR (Rectified DLR) loss:
$$
\operatorname {R e D L R} (z, y) = - (\operatorname {D L R}) ^ {-} (z, y) = \left\{ \begin{array}{c c} - \frac {z _ {y} - z _ {(2)}}{z _ {(1)} - z _ {(3)}}, & \text {i f} z _ {y} = z _ {(1)}, \\ 0, & \text {e l s e .} \end{array} \right. \tag {10}
$$
Its key property is to leave unaffected those images that are already misclassified. Our experiments show it performs superior to CE or DLR.
An attack is performed using the test data set. For a given loss function, our proposed attack is:
$$
x ^ {t + 1} = \operatorname {p r o j} _ {\delta} \left(x ^ {t} + \alpha h \left(\nabla_ {x} J _ {\theta} \left(x ^ {t}, y\right)\right) \| \breve {\Upsilon} ^ {- 1} \nabla_ {x} J _ {\theta} \left(x ^ {t}, y\right) \| _ {*} ^ {q - 1}\right), \tag {11}
$$
where  $\alpha$  is the step size and  $\mathrm{proj}_{\delta}$  is a projection which ensures the empirical measure  $\check{P}^{t + 1} := \frac{1}{|\mathcal{D}_{tt}|}\sum_{(x,y)\in \mathcal{D}_{tt}}\delta_{(x^{t + 1},y)}$  stays inside the Wasserstein ball  $B_{\delta}(\check{P})$ . In the case  $p = r = \infty$ , one can verify  $h(x) = \operatorname{sgn}(x)$  and write (11) as
$$
x ^ {t + 1} = \operatorname {p r o j} _ {\delta} \left(x ^ {t} + \alpha \operatorname {s g n} \left(\nabla_ {x} J _ {\theta} \left(x ^ {t}, y\right)\right)\right).
$$
This gives exactly Fast Gradient Sign Method (single step) and Projected Gradient Descent (multi-step) proposed in Goodfellow et al. (2015), Madry et al. (2018) and we adopt the same labels for our more general algorithms. A pseudocode for the above attack is summarized in Appendix C.