SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {\underline {{x}}} _ {i} ^ {\xi} = \left\{ \begin{array}{l l} \mathbf {\underline {{x}}} _ {i} ^ {c} & \text {i f} \quad y _ {i} f (\mathbf {\underline {{x}}} _ {i} ^ {f}) \geq y _ {i} f _ {c} (\mathbf {\underline {{x}}} _ {i} ^ {c}) \\ \mathbf {\underline {{x}}} _ {i} ^ {f} & \text {i f} \quad y _ {i} f (\mathbf {\underline {{x}}} _ {i} ^ {c}) <   y _ {i} f _ {c} (\mathbf {\underline {{x}}} _ {i} ^ {c}) \end{array} \right.
$$
$\left.\overline{\mathbf{x}}^{\delta}\right)\cdot \cdot \cdot))))$  and  $g_0(\underline{\mathbf{x}}^\delta) = f_c(\underline{\mathbf{x}}^\delta)$  . Then, we have
$$
\left\| \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \cdot \cdot \cdot)) \right\| _ {\mathrm {F}} \leq \prod_ {l ^ {\prime} = 1} ^ {l} \left\| \underline {{\mathbf {W}}} ^ {(l ^ {\prime})} \right\| _ {\mathrm {F}} \left\| \underline {{\mathbf {x}}} _ {i} ^ {\xi} \right\| _ {\mathrm {F}} \leq \prod_ {l ^ {\prime} = 1} ^ {l} B _ {l} B _ {x, R _ {\mathrm {a}}, \xi}
$$
$$
\begin{array}{l} \left\| \sigma (\underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots)) - \sigma (\underline {{\mathbf {W}}} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots)) \right\| _ {\mathrm {F}} \\ \leq \left\| \underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots) - \underline {{\mathbf {W}}} ^ {(l + 1)} * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma (\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}) \dots)\right) \| _ {\mathrm {F}} \\ \leq \left\| \left(\underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} - \underline {{\mathbf {W}}} ^ {(l + 1)}\right) * _ {M} \sigma \left(\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma \left(\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) \dots\right) \right\| _ {\mathrm {F}} \\ \leq \left\| \underline {{\mathbf {W}}} _ {c} ^ {(l + 1)} - \underline {{\mathbf {W}}} ^ {(l + 1)} \right\| _ {\mathrm {F}} \left\| \sigma \left(\underline {{\mathbf {W}}} ^ {(l)} * _ {M} \dots * _ {M} \sigma \left(\underline {{\mathbf {W}}} ^ {(1)} * _ {M} \underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) \dots\right) \right\| _ {\mathrm {F}} \\ \leq \delta_ {l} \cdot \prod_ {l ^ {\prime} = 1} ^ {l - 1} B _ {l ^ {\prime}} B _ {x, R _ {\mathrm {a}}, \xi}. \\ \end{array}
$$
$$
\left| g _ {l} \left(\underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) - g _ {l - 1} \left(\underline {{\mathbf {x}}} _ {i} ^ {\xi}\right) \right| \leq \delta_ {l} \cdot B _ {\mathbf {w}} \prod_ {l ^ {\prime} \neq l} B _ {l ^ {\prime}} B _ {x, R _ {\mathrm {a}}, \xi} = \frac {\delta_ {l} B _ {\tilde {f}}}{B _ {l}}.
$$
Letting  $\frac{\delta_l B_{\tilde{f}}}{B_l} = \frac{\epsilon}{L}$  gives
$$
\sup  _ {\tilde {f} \in \mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}}} \inf  _ {\tilde {f} _ {c} \in \mathfrak {F} _ {c} ^ {\mathrm {a d v}}} \left\| \tilde {f} - \tilde {f} _ {c} \right\| _ {S} \leq \sum_ {l = 1} ^ {L - 1} | g _ {l} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) - g _ {l - 1} (\underline {{\mathbf {x}}} _ {i} ^ {\xi}) | \leq \epsilon .
$$
Then,  $\mathfrak{F}_c^{\mathrm{adv}}$  is an  $\epsilon$ -covering of  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$ . We further proceed by computing the  $\epsilon$ -covering number of  $\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}}$  as follows:
$$
\mathsf {N} \left(\mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}}, \| \cdot \| _ {S}, \epsilon\right) \leq | \mathfrak {F} _ {c} ^ {\mathrm {a d v}} | = \prod_ {l = 1} ^ {L} | \mathbf {C} _ {l} | \leq \prod_ {l = 1} ^ {(i)} \left(\frac {9 B _ {l}}{\delta_ {l}}\right) ^ {\operatorname {c r} _ {l} (d _ {l - 1} + d _ {l} + 1)} \leq \left(\frac {9 L B _ {\tilde {f}}}{\epsilon}\right) ^ {\sum_ {l = 1} ^ {L} c r _ {l} (d _ {l - 1} + d _ {l} + 1)},
$$
where the inequality  $(i)$  holds due to Lemma 33.
Then, we use Dudley's integral to upper bound  $\hat{R}_S(\mathfrak{F}_{\mathbf{r}}^{\mathrm{adv}})$  as follows
$$
\begin{array}{l} \hat {R} _ {S} \left(\mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}}\right) \leq \inf  _ {\delta > 0} \left(8 \delta + \frac {1 2}{\sqrt {N}} \int_ {\delta} ^ {D _ {\bar {f}} / 2} \sqrt {\log N \left(\mathfrak {F} _ {\mathbf {r}} ^ {\mathrm {a d v}} , \| \cdot \| _ {S} , \epsilon\right)} \mathrm {d} \epsilon\right) \\ \leq \inf _ {\delta > 0} \left(8 \delta + \frac {1 2}{\sqrt {N}} \int_ {\delta} ^ {D _ {\tilde {f}} / 2} \sqrt {\sum_ {l = 1} ^ {L} \mathsf {c r} _ {l} (d _ {l - 1} + d _ {l} + 1) \log \left(9 L B _ {\tilde {f}} / \epsilon\right)} \mathrm {d} \epsilon\right) \\ = \inf  _ {\delta > 0} \left(8 \delta + \frac {1 2 D _ {\tilde {f}} \sqrt {\sum_ {l = 1} ^ {L} c r _ {l} (d _ {l - 1} + d _ {l} + 1)}}{\sqrt {N}} \int_ {\delta / D _ {\tilde {f}}} ^ {1 / 2} \sqrt {\log (9 L / (2 t))} \mathrm {d} t\right), \\ \end{array}
$$
where the diameter  $D_{\tilde{f}}$  of  $\mathfrak{F}^{\mathrm{adv}}$  is given in Lemma 40 and we have  $D_{\tilde{f}} = 2B_{\tilde{f}}$ . Following Ref. [55], let  $\delta \to 0$ , and use interation by part, we obtain  $\int_0^{1 / 2}\sqrt{\log(9L / (2t))}\mathrm{d}t\leq \sqrt{\log 9L}$ . Further applying Lemma 29 completes the proof.
D Implicit bias towards low-rankness in the transformed domain
Recent research has shown that GF maximizes the margin of homogeneous networks during standard training, which leads to an implicit bias towards margin maximization [16, 30]. Moreover, it has been demonstrated that this implicit bias also extends to adversarial margin maximization during adversarial training of multi-homogeneous fully connected neural networks with exponential loss [29]. Our analysis builds on these findings by showing that this implicit bias also holds for adversarial training of t-NNs when the adversarial perturbation is scale invariant [29].
First, it is straightforward to see that any t-NN  $f \in \mathfrak{F}$  is homogeneous as follows
$$
f (\underline {{\mathbf {x}}}; a \underline {{\mathbf {W}}}) = a ^ {L + 1} f (\underline {{\mathbf {x}}}; \underline {{\mathbf {W}}}), \tag {24}
$$
for any positive constant  $a$ .
Lemma 18 (Euler's theorem on t-NNs). For any  $t$ -NN  $f \in \mathfrak{F}$ , we have
$$
\left\langle \frac {\partial f (\underline {{\mathbf {x}}} ; \underline {{\mathbf {W}}})}{\partial \underline {{\mathbf {W}}}}, \underline {{\mathbf {W}}} \right\rangle = (L + 1) f (\underline {{\mathbf {x}}} ; \underline {{\mathbf {W}}}). \tag {25}
$$
Proof. By taking derivatives with respect to  $a$  on both sides of Eq. (24), we obtain
$$
\left\langle \frac {\partial f (\underline {{\mathbf {x}}} ; a \underline {{\mathbf {W}}})}{\partial (a \underline {{\mathbf {W}}})}, \frac {\mathrm {d} (a \underline {{\mathbf {W}}})}{\mathrm {d} a} \right\rangle = (L + 1) a ^ {L} f (\underline {{\mathbf {x}}} ; \underline {{\mathbf {W}}}),
$$
which immediately results in Eq. (25).
In this section, we follow the setting of Ref. [29] where the adversarial perturbation  $\underline{\delta}_i$  is scale invariant. As Lemma 7 shows,  $l_{2}$ -FGM [34], FGSM [9],  $l_{2}$ -PGD and  $l_{\infty}$ -PGD [31] perturbations for the t-NNs are all scale invariant.
Proof of Lemma 7. Note that by taking derivatives with respect to  $\underline{\mathbf{x}}$  on both sides of Eq. (24), we have
$$
\frac {\partial f (\underline {{\mathbf {x}}} ; a \mathbf {W})}{\partial \underline {{\mathbf {x}}}} = a ^ {L + 1} \frac {\partial f (\underline {{\mathbf {x}}} ; \mathbf {W})}{\partial \underline {{\mathbf {x}}}}, \tag {26}
$$
Therefore, any  $\frac{\partial f(\underline{\mathbf{x}};\underline{\mathbf{W}})}{\partial\underline{\mathbf{x}}}$  is positive homogeneous. Then, for any non-zero  $\underline{\mathbf{z}} = \frac{\partial f(\underline{\mathbf{x}};\underline{\mathbf{W}})}{\partial\underline{\mathbf{x}}}$ , we prove Lemma 7 in the following cases:
-  $l_{2}$ -FGM perturbation [34]. The  $l_{2}$ -FGM perturbation is defined as  $\underline{\delta}_{\mathrm{FGM}}(\underline{\mathbf{W}}) = \xi y\tilde{\ell}^{\prime}\underline{\mathbf{z}}\left\| y\tilde{\ell}^{\prime}\underline{\mathbf{z}}\right\|_{\mathrm{F}}^{-1} = -\xi y\underline{\mathbf{z}}\|\underline{\mathbf{z}}\|_{\mathrm{F}}^{-1}$ , because  $\tilde{\ell}^{\prime}\leq 0$ . Using Eq. (26), we have
$$
\underline {{\boldsymbol {\delta}}} _ {\mathrm {F G M}} (a \underline {{\mathbf {W}}}) = - \frac {\xi y \cdot a ^ {L + 1} \underline {{\mathbf {z}}}}{\| a ^ {L + 1} \underline {{\mathbf {z}}} \| _ {\mathrm {F}}} = \underline {{\boldsymbol {\delta}}} _ {\mathrm {F G M}} (\underline {{\mathbf {W}}}).
$$
- FGSM perturbations [9]. The FGSM perturbation is taken as  $\underline{\delta}_{\mathrm{FSGM}}(\underline{\mathbf{W}}) = \xi \mathrm{sgn}(\xi y\tilde{\ell}'\underline{\mathbf{z}})$ . Using Eq. (26), we have
$$
\underline {{\boldsymbol {\delta}}} _ {\mathrm {F S G M}} (a \underline {{\mathbf {W}}}) = \xi_ {\mathrm {s g n}} (\xi y \tilde {\ell} ^ {\prime} a ^ {L + 1} \underline {{\mathbf {z}}}) = \underline {{\boldsymbol {\delta}}} _ {\mathrm {F S G M}} (\underline {{\mathbf {W}}}).
$$
-  $l_{2}$ -PGD perturbation [31]. The  $l_{2}$ -PGD perturbation is taken as
$$
\underline {{\boldsymbol {\delta}}} _ {\mathrm {P G D}} ^ {j + 1} (\underline {{\mathbf {W}}}) = \mathfrak {P} _ {\mathbb {B} _ {2} (0, \xi)} \left[ \underline {{\boldsymbol {\delta}}} _ {\mathrm {P G D}} ^ {j} (\underline {{\mathbf {W}}}) - \rho \frac {\xi y \underline {{\mathbf {z}}}}{\| \underline {{\mathbf {z}}} \| _ {\mathrm {F}}} \right], \tag {27}
$$
where  $j$  is the attack step,  $\mathfrak{P}_{\mathbb{B}_2(\xi)}$  is the projector onto  $l_{2}$ -norm ball of radius  $\xi$ , and  $\rho$  is the learning rate. We prove by induction. For  $j = 0$ , we have
$$
\underline {{\delta}} _ {\mathrm {P G D}} ^ {1} (a \underline {{\mathbf {W}}}) = \mathfrak {P} _ {\mathbb {B} _ {2} (0, \xi)} \left[ - \rho \frac {\xi y a ^ {L + 1} \underline {{\mathbf {z}}}}{\| a ^ {L + 1} \underline {{\mathbf {z}}} \| _ {\mathrm {F}}} \right] = \underline {{\delta}} _ {\mathrm {P G D}} ^ {1} (\underline {{\mathbf {W}}}).
$$
If we have  $\underline{\delta}_{\mathrm{PGD}}^j (a\underline{\mathbf{W}}) = \underline{\delta}_{\mathrm{PGD}}^j (\underline{\mathbf{W}})$ , then for  $j + 1$ , we have
$$
\begin{array}{l} \underline {{\boldsymbol {\delta}}} _ {\mathrm {P G D}} ^ {j + 1} (a \underline {{\mathbf {W}}}) = \mathfrak {P} _ {\mathbb {B} _ {2} (0, \xi)} \left[ \underline {{\boldsymbol {\delta}}} _ {\mathrm {P G D}} ^ {j} (a \underline {{\mathbf {W}}}) - \rho \frac {\xi y a ^ {L + 1} \underline {{\mathbf {z}}}}{\| a ^ {L + 1} \underline {{\mathbf {z}}} \| _ {\mathrm {F}}} \right] \\ = \mathfrak {P} _ {\mathbb {B} _ {2} (0, \xi)} \left[ \underline {{\boldsymbol {\delta}}} _ {\mathrm {P G D}} ^ {j} (\underline {{\mathbf {W}}}) - \rho \frac {\xi y \underline {{\mathbf {z}}}}{\| \underline {{\mathbf {z}}} \| _ {\mathrm {F}}} \right] \tag {28} \\ = \underline {{\boldsymbol {\delta}}} _ {\mathrm {P G D}} ^ {j + 1} (\underline {{\mathbf {W}}}). \\ \end{array}
$$
-  $l_{\infty}$ -PGD perturbation [31]. Since the scale invariance of this perturbation can be proved very similarly to that of  $l_{2}$ -PGD perturbations, we just omit it.
For an original example  $\underline{\mathbf{x}}_i$ , the margin for its adversarial example  $\underline{\mathbf{x}}_i + \underline{\boldsymbol{\delta}}_i(\underline{\mathbf{W}})$  is defined as  $\tilde{q}_i(\underline{\mathbf{W}}) := y_i f(\underline{\mathbf{x}}_i + \underline{\boldsymbol{\delta}}_i(\underline{\mathbf{W}}); \underline{\mathbf{W}})$ ; for sample  $S = \{(\underline{\mathbf{x}}_i, y_i)\}_{i=1}^N$ , the margin for the  $N$  corresponding examples is denoted by  $\tilde{q}_m(\underline{\mathbf{W}})$  where  $m \in \operatorname{argmin}_{m=1,\dots,N} y_i f(\underline{\mathbf{x}}_i + \underline{\boldsymbol{\delta}}_i(\underline{\mathbf{W}}); \underline{\mathbf{W}})$ .
Let  $\rho = \| \underline{\mathbf{W}}\|_{\mathrm{F}}$  for simplicity. We use the normalized parameter  $\hat{\mathbf{W}} \coloneqq \underline{\mathbf{W}} / \rho$  to denote the direction of the weights  $\underline{\mathbf{W}}$ . We introduce the normalized margin of  $(\underline{\mathbf{x}}_i, y_i)$  as  $\hat{q}_i(\underline{\mathbf{W}}) \coloneqq \tilde{q}_i(\underline{\hat{\mathbf{W}}}) = \tilde{q}_i(\underline{\mathbf{W}}) \rho^{-(L + 1)}$ , and similarly define the normalized robust margin of the sample  $S$  as  $\hat{q}_m(\underline{\mathbf{W}}) \coloneqq \tilde{q}_m(\underline{\hat{\mathbf{W}}}) = \tilde{q}_m(\underline{\mathbf{W}}) \rho^{-(L + 1)}$ .
Note that the adversarial empirical risk can be written as  $\hat{\mathcal{L}}^{\mathrm{adv}} = \frac{1}{N}\sum_{i = 1}^{N}e^{-\mathsf{f}(\tilde{q}_i(\underline{\mathbf{W}}))}$ . Motivated by [30] which uses the LogSumExp function to smoothly approximate the normalized standard margin, we define the smoothed normalized margin  $\tilde{\gamma}$  as follows.
Definition 11 (Smoothed normalized robust margin). For a loss function  $\ell$  satisfying Assumption 2, the smoothed normalized robust margin is defined as
$$
\tilde {\gamma} (\underline {{\mathbf {W}}}) := \frac {\ell^ {- 1} (N \hat {\mathcal {L}} ^ {\mathrm {a d v}})}{\rho^ {L + 1}} = \frac {\mathfrak {g} \left(\frac {1}{N \hat {\mathcal {L}} ^ {\mathrm {a d v}}}\right)}{\rho^ {L + 1}} = \frac {\mathfrak {g} \left(- \log \left(\sum_ {i = 1} ^ {N} e ^ {- \mathfrak {f} (\tilde {q} _ {i} (\underline {{\mathbf {W}}}))}\right)\right)}{\rho^ {L + 1}}. \tag {29}
$$
In this paper, the loss function satisfying Assumption 2 belongs to the class of margin-based loss function [42, Definition 2.24]. That is, although the loss function  $\ell(f(\underline{\mathbf{x}}), y)$  is a binary function of  $f(\underline{\mathbf{x}})$  and  $y$ , there is a unary function  $\mathfrak{l}(\cdot)$ , such that  $\ell(f(\underline{\mathbf{x}}), y) = \mathfrak{l}(y f(\underline{\mathbf{x}}))$ . With a slight abuse of notation, we simply use  $\ell^{-1}(\cdot)$  to denote  $\mathfrak{l}^{-1}(\cdot)$ , i.e., if  $z = \ell(f(\underline{\mathbf{x}}), y)$  then we have  $\ell^{-1}(z) = y f(\underline{\mathbf{x}})$ .
To better understand the relation between the normalized sample robust margin  $\hat{q}_m$  and the smoothed normalized robust margin  $\tilde{\gamma}$ , we provide the following lemma.
Lemma 19 (Adapted from Lemma A.5 of Ref. [30]). Under Assumption 2, we have the following properties about the robust marin  $\tilde{q}_m$ :
(a)  $\mathfrak{f}(\tilde{q}_m) - \log N\leq \log \frac{1}{N\hat{\mathcal{L}}^{\mathrm{adv}}}\leq \mathfrak{f}(\tilde{q}_m).$
(b) If  $\log \frac{1}{N\hat{\mathcal{L}}^{\mathrm{adv}}} > \mathfrak{f}(b_{\mathfrak{f}})$ , then there exists  $\xi \in (\mathfrak{f}(\tilde{q}_m) - \log N, \mathfrak{f}(\tilde{q}_m)) \cap (b_{\mathfrak{f}}, \infty)$  such that
$$
\hat {q} _ {m} - \rho^ {- (L + 1)} \mathfrak {g} ^ {\prime} (\xi) \log N \leq \tilde {\gamma} \leq \hat {q} _ {m},
$$
which shows the smoothed normalized margin  $\tilde{\gamma}$  is a rough approximation of the normalized robust margin  $\hat{q}_m$ .
(c) For a sequence  $\{\underline{\mathbf{W}}_s\mid s\in \mathbb{N}\}$ , if  $\hat{\mathcal{L}}^{\mathrm{adv}}(\underline{\mathbf{W}}_s)\to 0$ , then  $|\tilde{\gamma} (\underline{\mathbf{W}}_s) - \hat{q}_m(\underline{\mathbf{W}}_s)|\to 0$
D.1 Convergence to KKT points of Euclidean norm minimization in direction
The KKT condition for the optimization problem Eq. (14) are
$$
\underline {{\mathbf {W}}} - \sum_ {i = 1} ^ {N} \lambda_ {i} \frac {\partial \hat {q} _ {i}}{\partial \underline {{\mathbf {W}}}} = \mathbf {0}, \tag {30}
$$
$$
\lambda_ {i} (\tilde {q} _ {i} - 1) = 0, \quad i = 1, \dots , N,
$$
where the dual variables  $\lambda_{i}\geq 0$ . We define the approximate KKT point in a similar manner to Ref. [29] as follows.
Definition 12 (Approximate KKT points). The  $(\kappa, \iota)$ -approximate KKT points of the optimization problem are those feasible points which satisfy the following two conditions:
$$
\text {C o n d i t i o n (I) :} \quad \left\| \underline {{\mathbf {W}}} - \sum_ {i = 1} ^ {N} \lambda_ {i} \frac {\partial \tilde {q} _ {i}}{\partial \underline {{\mathbf {W}}}} \right\| _ {\mathrm {F}} \leq \kappa , \tag {31}
$$
$$
\text {C o n d i t i o n (I I) :} \quad \lambda_ {i} \left(\tilde {q} _ {i} - 1\right) \leq \iota , \quad i = 1, \dots , N,
$$
where  $\kappa, \iota > 0$  and  $\lambda_i \geq 0$ .
Proof of Lemma 9. Let  $\underline{\mathbf{W}} := \underline{\mathbf{W}} / \tilde{q}_m^{\frac{1}{L + 1}}$  denote the scaled version of  $\underline{\mathbf{W}}(t)$  such that the sample robust margin  $\tilde{q}_m = 1$ . Thus we have  $f(\underline{\mathbf{x}}; \underline{\mathbf{W}}) = \tilde{q}_m f(\underline{\mathbf{x}}; \underline{\mathbf{W}})$  by homogeneity of t-NNs. According to Lemma 18, we further have
$$
(L + 1) f (\underline {{\mathbf {x}}}; \underline {{\mathbf {W}}}) = \left\langle \underline {{\mathbf {W}}}, \frac {\partial f (\underline {{\mathbf {x}}} ; \underline {{\mathbf {W}}})}{\partial \underline {{\mathbf {W}}}} \right\rangle , \quad (L + 1) f (\underline {{\mathbf {x}}}; \breve {\underline {{\mathbf {W}}}}) = \left\langle \breve {\underline {{\mathbf {W}}}}, \frac {\partial f (\underline {{\mathbf {x}}} ; \breve {\underline {{\mathbf {W}}}})}{\partial \breve {\underline {{\mathbf {W}}}}} \right\rangle ,
$$
$$
\left\langle \check {\mathbf {W}}, \frac {\partial f (\underline {{\mathbf {x}}} ; \check {\mathbf {W}})}{\partial \check {\mathbf {W}}} \right\rangle = \frac {1}{\hat {q} _ {m} ^ {L / (L + 1)}} \frac {\partial f (\underline {{\mathbf {x}}} ; \underline {{\mathbf {W}}})}{\partial \underline {{\mathbf {W}}}}.
$$
We will prove that  $\underline{\mathbf{W}}$  is a  $(\kappa, \iota)$ -KKT point of Problem (14) with  $(\kappa, \iota) \to \mathbf{0}$ .
Let  $\underline{\dot{\mathbf{W}}}(t) \coloneqq \frac{\mathrm{d}\mathbf{W}(t)}{\mathrm{d}t}$  for simplicity. By the chain rule and GF update rule, we have
$$
\underline {{\dot {\mathbf {W}}}} = \frac {1}{N} \sum_ {i = 1} ^ {N} e ^ {- \mathfrak {f} (\tilde {q} _ {i})} \mathfrak {f} ^ {\prime} (\tilde {q} _ {i}) \frac {\partial \tilde {q} _ {i}}{\partial \underline {{\mathbf {W}}}}.
$$
Using the homogeneity of t-NNs, we obtain
$$
\frac {1}{2} \frac {\mathrm {d} \| \underline {{\mathbf {W}}} \| _ {\mathrm {F}} ^ {2}}{\mathrm {d} t} = \left\langle \dot {\underline {{\mathbf {W}}}}, \underline {{\mathbf {W}}} \right\rangle = \left\langle \frac {1}{N} \sum_ {i = 1} ^ {N} e ^ {- \mathfrak {f} (\tilde {q} _ {i})} \mathfrak {f} ^ {\prime} (\tilde {q} _ {i}) \frac {\partial \tilde {q} _ {i}}{\partial \underline {{\mathbf {W}}}}, \underline {{\mathbf {W}}} \right\rangle = (L + 1) \cdot \frac {1}{N} \sum_ {i = 1} ^ {N} e ^ {- \mathfrak {f} (\tilde {q} _ {i})} \mathfrak {f} ^ {\prime} (\tilde {q} _ {i}) \tilde {q} _ {i}.
$$
By letting  $\nu(t) = \sum_{i=1}^{N} e^{-\mathfrak{f}(\tilde{q}_i)} \mathfrak{f}'(\tilde{q}_i) \tilde{q}_i$ , we obtain  $\left\langle \underline{\dot{\mathbf{W}}}, \underline{\mathbf{W}} \right\rangle = (L + 1)\nu / N$ .
We construct the dual variables  $\lambda_{i}$  in Problem (31) in terms of  $\dot{\mathbf{W}}$  as follows
$$
\lambda_ {i} (t) := \frac {1}{N} \tilde {q} _ {m} ^ {1 - 2 / (L + 1)} \cdot \rho e ^ {- \mathfrak {f} (\tilde {q} _ {i})} \mathfrak {f} ^ {\prime} (\tilde {q} _ {i}) \left\| \underline {{\dot {\mathbf {W}}}} \right\| _ {\mathrm {F}} ^ {- 1}. \tag {32}
$$
To prove  $\tilde{\mathbf{W}}$  is a  $(\kappa, \iota)$ -KKT point of Problem (14), we need to check the conditions in Problem (31).
Step 1: Check Condition (I) of the  $(\kappa, \iota)$ -approximate KKT conditions. We check the Condition (I) in Problem (31) for all  $t > t_0$  as follows
$$
\begin{array}{l} \left\| \underline {{\check {\mathbf {W}}}} - \sum_ {i = 1} ^ {N} \lambda_ {i} \frac {\partial \tilde {q} _ {i} (\underline {{\check {\mathbf {W}}}})}{\partial \underline {{\check {\mathbf {W}}}}} \right\| _ {\mathrm {F}} ^ {2} \overset {(i)} {=} \left\| \frac {\underline {{\mathbf {W}}}}{\bar {q} _ {m} ^ {1 / (L + 1)}} - \frac {\dot {\mathbf {W}}}{\left\| \dot {\underline {{\mathbf {W}}}} \right\| _ {\mathrm {F}}} \cdot \frac {\rho}{\bar {q} _ {m} ^ {1 / (L + 1)}} \right\| _ {\mathrm {F}} ^ {2} \\ = \frac {\rho^ {2}}{\dot {q} _ {m} ^ {2 / (L + 1)}} \left\| \frac {\underline {{\mathbf {W}}}}{\| \underline {{\mathbf {W}}} \| _ {\mathrm {F}}} - \frac {\dot {\mathbf {W}}}{\left\| \dot {\underline {{\mathbf {W}}}} \right\| _ {\mathrm {F}}} \right\| _ {\mathrm {F}} ^ {2} \\ \stackrel {(i i)} {=} \frac {1}{\tilde {\gamma} ^ {2 / (L + 1)}} \left(2 - 2 \left\langle \frac {\underline {{\mathbf {W}}}}{\| \underline {{\mathbf {W}}} \| _ {\mathrm {F}}}, \frac {\dot {\underline {{\mathbf {W}}}}}{\left\| \dot {\underline {{\mathbf {W}}}} \right\| _ {\mathrm {F}}} \right\rangle\right) \tag {33} \\ \stackrel {(i i i)} {\leq} \frac {2}{\tilde {\gamma} (t _ {0}) ^ {2 / (L + 1)}} \left(1 - \left\langle \frac {\underline {{\mathbf {W}}}}{\| \underline {{\mathbf {W}}} \| _ {\mathrm {F}}}, \frac {\dot {\underline {{\mathbf {W}}}}}{\left\| \dot {\underline {{\mathbf {W}}}} \right\| _ {\mathrm {F}}} \right\rangle\right) \\ := \kappa^ {2} (t), \\ \end{array}
$$
where equality (i) is obtained by using the definition that  $\underline{\mathbf{W}} = \underline{\mathbf{W}} / \tilde{q}_m^{\frac{1}{L + 1}}$ , the fact  $\tilde{q}_i(\underline{\mathbf{W}}) = y_i f(\underline{\mathbf{x}}_i + \underline{\boldsymbol{\delta}}_i(\underline{\mathbf{W}}); \underline{\mathbf{W}}) = y_i \cdot \tilde{q}_m f(\underline{\mathbf{x}}_i + \underline{\boldsymbol{\delta}}_i(\underline{\mathbf{W}}); \underline{\mathbf{W}}) = \tilde{q}_i(\underline{\mathbf{W}})$  due to the scale invariance of the adversarial perturbation  $\underline{\boldsymbol{\delta}}_i(\underline{\mathbf{W}})$  and the homogeneity of t-NN, and the chain rule in computing  $\frac{\partial \tilde{q}_i}{\partial \underline{\mathbf{W}}}$  as follows
$$
\sum_ {i = 1} ^ {N} \lambda_ {i} \frac {\partial \tilde {q} _ {i} (\breve {\mathbf {W}})}{\partial \breve {\mathbf {W}}} = \sum_ {i = 1} ^ {N} \frac {\lambda_ {i}}{\hat {q} _ {m} ^ {\underline {{L}} + 1}} \frac {\partial \tilde {q} _ {i}}{\partial \breve {\mathbf {W}}} = \left\| \breve {\mathbf {W}} \right\| _ {\mathrm {F}} ^ {- 1} \frac {\rho}{\tilde {q} _ {m} ^ {\underline {{L}} + 1}} \cdot \frac {1}{N} \sum_ {i = 1} ^ {N} e ^ {- \mathfrak {f} (\tilde {q} _ {i})} \mathfrak {f} ^ {\prime} (\tilde {q} _ {i}) \frac {\partial \tilde {q} _ {i}}{\partial \breve {\mathbf {W}}} = \frac {\dot {\mathbf {W}}}{\left\| \breve {\mathbf {W}} \right\| _ {\mathrm {F}}} \frac {\rho}{\tilde {q} _ {m} ^ {\underline {{L}} + 1}}.
$$
In Eq. (33), equality  $(ii)$  holds by property  $(b)$  in Lemma 19;  $(iii)$  holds by the non-decreasing property of  $\tilde{\gamma}(t)$  for all  $t \in [t_0, \infty)$  in Lemma 22.
Note that Eq. (33) indicates that  $\kappa(t)$  is in terms of the cosine of the angle between  $\underline{\mathbf{W}}(t)$  and  $\dot{\underline{\mathbf{W}}}(t)$ . We can further obtain that  $\kappa(t) \to 0$  as  $t \to \infty$  by showing the angle between  $\underline{\mathbf{W}}(t)$  and  $\dot{\underline{\mathbf{W}}}(t)$  approximates 0 which was originally observed by Ref. [30] for standard training<sup>10</sup> on a fixed training sample  $S$ .
Lemma 20 (Adapted from Lemma C.12 in Ref. [30]). Under Assumption 2 and Assumption 8 for  $t$ -NNs, the angle between  $\underline{\mathbf{W}}(t)$  and  $\dot{\underline{\mathbf{W}}}(t)$  approximates 0 as  $t \to \infty$  along the trajectory of adversarial training with scale invariant adversarial perturbations, i.e.,
$$
\lim  _ {t \rightarrow \infty} \left\langle \frac {\underline {{\mathbf {W}}} (t)}{\| \underline {{\mathbf {W}}} (t) \| _ {\mathrm {F}}}, \frac {\dot {\underline {{\mathbf {W}}}} (t)}{\left\| \dot {\underline {{\mathbf {W}}}} (t) \right\| _ {\mathrm {F}}} \right\rangle\rightarrow 1.
$$