SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For simplicity of the notation, in our analysis of the PGD attack, we drop the subscripts denoting the iterates. That is, at the  $t$ -th iterate of the outer loop, the weight matrix and the sample point is denoted by  $\mathrm{W}$  and  $\mathbf{x}, y$ , instead of  $\mathrm{W}_t$  and  $\mathbf{x}_t, y_t$ . We can then use the same variable  $t$  to measure the progress of the attack in the inner loop. PGD updates are therefore given by  $\Pi_{\Delta(\mathbf{x})}[\delta_t + \eta \nabla \ell_-(y f_{\mathrm{W}}(\delta_t))]$ .
Proof of Theorem 3.2. We first bound the norm of the gradient of the iterates as follows:
$$
\begin{array}{l} \| \nabla f _ {\mathrm {W}} (\delta_ {t}) \| = \| \sum_ {r = 1} ^ {m} a _ {r} \sigma^ {\prime} \left(\mathrm {w} _ {r} \cdot \delta_ {t}\right) \mathrm {w} _ {r} \| \\ \leq \sum_ {r = 1} ^ {m} \left\| a _ {r} \sigma^ {\prime} \left(\mathrm {w} _ {r} \cdot \delta_ {t}\right) \mathrm {w} _ {r} \right\| \\ \leq \kappa \sum_ {r = 1} ^ {m} \left\| \mathrm {w} _ {r} \right\| \\ \leq \kappa \sqrt {m} \| W \| _ {F} =: G \\ \end{array}
$$
We analyze the distance of iterates from  $\delta_{*}$ :
$$
\begin{array}{l} \left\| \delta_ {t + 1} - \delta_ {*} \right\| ^ {2} - \left\| \delta_ {t} - \delta_ {*} \right\| ^ {2} = \left\| \Pi \left[ \delta_ {t} + \eta_ {\text {a t t}} \nabla \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \right] - \delta_ {*} \right\| ^ {2} - \left\| \delta_ {t} - \delta_ {*} \right\| ^ {2} \\ \leq \left\| \delta_ {t} + \eta_ {\mathrm {a t t}} \nabla \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) - \delta_ {*} \right\| ^ {2} - \left\| \delta_ {t} - \delta_ {*} \right\| ^ {2} \\ = 2 \eta_ {\text {a t t}} \left\langle \nabla \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right), \delta_ {t} - \delta_ {*} \right\rangle + \eta_ {\text {a t t}} ^ {2} \left\| \nabla \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \right\| ^ {2} \\ = 2 \eta_ {\mathrm {a t t}} \ell_ {-} ^ {\prime} (y f _ {\mathrm {W}} (\delta_ {t})) y \langle \nabla f _ {\mathrm {W}} (\delta_ {t}), \delta_ {t} - \delta_ {*} \rangle + \eta_ {\mathrm {a t t}} ^ {2} \ell_ {-} ^ {\prime} (y f _ {\mathrm {W}} (\delta_ {t})) ^ {2} \| \nabla f _ {\mathrm {W}} (\delta_ {t}) \| ^ {2} \\ \leq 2 \eta_ {\mathrm {a t t}} \ell_ {-} ^ {\prime} (y f _ {\mathrm {W}} (\delta_ {t})) y \left\langle \nabla f _ {\mathrm {W}} (\delta_ {t}), \delta_ {t} - \delta_ {*} \right\rangle - \eta_ {\mathrm {a t t}} ^ {2} \ell_ {-} (y f _ {\mathrm {W}} (\delta_ {t})) G ^ {2} \\ \left(\left| \ell_ {-} ^ {\prime} (\cdot) \right| \leq \max  \{1, - \ell_ {-} (\cdot) \}\right) \\ \leq 2 \eta_ {\mathrm {a t t}} \ell_ {-} ^ {\prime} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right) - y f _ {\mathrm {W}, t} \left(\delta_ {*}\right)\right) - \eta_ {\mathrm {a t t}} \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \\ \left(\eta_ {\mathrm {a t t}} \leq 1 / G ^ {2}\right) \\ \leq 2 \eta_ {\text {a t t}} \left(\ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) - \ell_ {-} \left(y f _ {\mathrm {W}, t} \left(\delta_ {*}\right)\right)\right) - \eta_ {\text {a t t}} \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \\ (\text {c o n c a v i t y}) \\ \leq \eta_ {\mathrm {a t t}} \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) - 2 \eta_ {\mathrm {a t t}} \ell_ {-} \left(y f _ {\mathrm {W}, t} \left(\delta_ {*}\right)\right) \\ \end{array}
$$
where  $f_{\mathrm{W},t}(\delta_{*}) \coloneqq \langle \nabla f_{\mathrm{W}}(\delta_t), \delta_* \rangle$ . Averaging over iterates, rearranging, and cancelling telescopic terms, we arrive at:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} - \ell_ {-} (y f _ {\mathrm {W}} (\delta_ {t})) \leq \sum_ {t = 1} ^ {T} \frac {\| \delta_ {t} - \delta_ {*} \| ^ {2} - \| \delta_ {t + 1} - \delta_ {*} \| ^ {2}}{\eta_ {\mathrm {a t t}} T} - \frac {2}{T} \sum_ {t = 1} ^ {T} \ell_ {-} (y f _ {\mathrm {W}, t} (\delta_ {*})) \\ \leq \frac {\left\| \delta_ {1} - \delta_ {*} \right\| ^ {2}}{\eta_ {\mathrm {a t t}} T} - 2 \min  _ {t} \ell_ {-} \left(y f _ {\mathrm {W}, t} \left(\delta_ {*}\right)\right) \quad \text {(T e l e s c o p i c s u m)} \\ \leq \frac {\nu^ {2}}{\eta_ {\mathrm {a t t}} T} - 2 \min  _ {t} \ell_ {-} (y f _ {\mathrm {W}, t} (\delta_ {*})) \quad (\delta_ {1} = \mathrm {x}, \delta_ {*} \in \Delta (\mathrm {x})) \\ \Longrightarrow - \ell_ {-} (y f _ {\mathrm {W}} (\delta_ {\mathrm {a t t}})) \leq \frac {\nu^ {2}}{\eta_ {\mathrm {a t t}} T} - 2 \min _ {t} \ell_ {-} (y f _ {\mathrm {W}, t} (\delta_ {*})) \\ \end{array}
$$
Next, we show that if  $\delta_{*}$  is an effective attack on  $f_{\mathrm{W}}$ , then it's also effective on  $f_{\mathrm{W},t}$ . In other words, we have that:
$$
\begin{array}{l} y f _ {\mathrm {W}, t} \left(\delta_ {*}\right) = y \left\langle \nabla f _ {\mathrm {W}} \left(\delta_ {t}\right), \delta_ {*} \right\rangle \\ = y \left\langle \nabla f _ {\mathrm {W}} \left(\delta_ {*}\right), \delta_ {*} \right\rangle + y \left\langle \nabla f _ {\mathrm {W}} \left(\delta_ {t}\right) - \nabla f _ {\mathrm {W}} \left(\delta_ {*}\right), \delta_ {*} \right\rangle \\ = y f _ {\mathrm {W}} \left(\delta_ {*}\right) + y \left\langle \nabla f _ {\mathrm {W}} \left(\delta_ {t}\right) - \nabla f _ {\mathrm {W}} \left(\delta_ {*}\right), \delta_ {*} \right\rangle \\ \end{array}
$$
Let  $\mathcal{I}_t\coloneqq \{j\in [m]|\sigma '(\mathrm{w}_j\cdot \delta_t)\neq \sigma '(\mathrm{w}_j\cdot \delta_*)\}$  be the set of nodes whose pre-activation sign at time  $t$  is different from the optimal  $\delta_{*}$ . The second term can be bounded as follows:
$$
\begin{array}{l} y \langle \nabla f _ {\mathrm {W}} (\delta_ {t}) - \nabla f _ {\mathrm {W}} (\delta_ {*}), \delta_ {*} \rangle = y \sum_ {j = 1} ^ {m} a _ {j} \left(\sigma^ {\prime} \left(\mathrm {w} _ {j} \cdot \delta_ {t}\right) - \sigma^ {\prime} \left(\mathrm {w} _ {j} \cdot \delta_ {*}\right)\right) \mathrm {w} _ {j} \cdot \delta_ {*} \\ \leq | y | \sum_ {j = 1} ^ {m} \left| a _ {j} \left(\sigma^ {\prime} \left(\mathrm {w} _ {j} \cdot \delta_ {t}\right) - \sigma^ {\prime} \left(\mathrm {w} _ {j} \cdot \delta_ {*}\right)\right) \mathrm {w} _ {j} \cdot \delta_ {*} \right| \\ = \kappa \sum_ {j \notin \mathcal {I} _ {t}} | \sigma^ {\prime} (\mathrm {w} _ {j} \cdot \delta_ {t}) - \sigma^ {\prime} (\mathrm {w} _ {j} \cdot \delta_ {*}) | | \mathrm {w} _ {j} \cdot \delta_ {*} | \\ \leq (1 - \alpha) \kappa \sum_ {j \notin \mathcal {I}} | w _ {j} \cdot \delta_ {*} - w _ {j} \cdot \delta_ {t} | \quad (j \notin \mathcal {I} _ {t}) \\ \leq (1 - \alpha) \kappa \sum_ {j \notin \mathcal {I}} \| w _ {j} \| \| \delta_ {*} - \delta_ {t} \| \quad (\text {C a u c h y - S c h w a r z}) \\ \leq (1 - \alpha) \kappa \nu \sqrt {m} \| W \| _ {F} \quad \text {(C a u c h y - S c h w a r z)} \\ \end{array}
$$
Given that there exist a  $\beta$ -effective, i.e. there exist  $\delta_*$  such that  $yf_{\mathrm{W}}(\delta_*) \leq -\beta \| \mathrm{W} \|_F$ , we have:
$$
\begin{array}{l} y f _ {\mathrm {W}, t} \left(\delta_ {*}\right) \leq y f _ {\mathrm {W}} \left(\delta_ {*}\right) + (1 - \alpha) \kappa \nu \sqrt {m} \| \mathrm {W} \| _ {F} \\ \leq y f _ {\mathrm {W}} \left(\delta_ {*}\right) + \frac {\beta}{2} \| \mathrm {W} \| _ {F} \quad (\nu \leq \frac {\beta}{2 (1 - \alpha) \kappa \sqrt {m}}) \\ \leq \frac {1}{2} y f _ {\mathrm {W}} \left(\delta_ {*}\right). \\ \end{array}
$$
Together with the previous inequality on the average of instantaneous loss, we arrive at:
$$
\begin{array}{l} - \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {\mathrm {a t t}}\right)\right) = \min  _ {t \in [ T ]} - \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} - \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {t}\right)\right) \\ \leq \frac {\nu^ {2}}{\eta_ {\mathrm {a t t}} T} - 2 \ell_ {-} (y f _ {\mathrm {W}} (\delta_ {*}) / 2). \\ \end{array}
$$
$$
2 \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {*} / 2\right)\right) \leq \ell_ {-} \left(y f _ {\mathrm {W}} \left(\delta_ {\mathrm {a t t}}\right)\right) + \frac {\nu^ {2}}{\eta_ {\mathrm {a t t}} T}. \tag {3}
$$
Next, we show that for any  $z,z^{\prime}$ , and  $\epsilon >0$ , the inequality  $2\ell_{-}(z / 2)\leq \ell_{-}(z^{\prime}) + \epsilon$  implies that  $-\ell^{\prime}(z)\leq -2\ell^{\prime}(z^{\prime}) + 4\epsilon$ . The following inequalities hold true:
$$
\begin{array}{l} 2 \ell_ {-} (z / 2) \leq \ell_ {-} \left(z ^ {\prime}\right) + \epsilon^ {\prime} \\ \Longrightarrow - 2 \log \left(1 + e ^ {z / 2}\right) \leq - \log \left(1 + e ^ {z ^ {\prime}}\right) + \epsilon^ {\prime} \\ \Rightarrow 2 \log \left(\frac {1}{1 + e ^ {z / 2}}\right) \leq \log \left(\frac {e ^ {\epsilon^ {\prime}}}{1 + e ^ {z ^ {\prime}}}\right) \\ \Longrightarrow \frac {1}{1 + e ^ {z} + 2 e ^ {z / 2}} \leq \frac {e ^ {\epsilon^ {\prime}}}{1 + e ^ {z ^ {\prime}}} \\ \Rightarrow \frac {e ^ {- z}}{1 + e ^ {- z} + 2 e ^ {- z / 2}} \leq \frac {e ^ {- z ^ {\prime}}}{1 + e ^ {- z ^ {\prime}}} \cdot e ^ {\epsilon^ {\prime}} \\ \Rightarrow \frac {1}{2} \frac {e ^ {- z}}{1 + e ^ {- z}} \leq \frac {e ^ {- z ^ {\prime}}}{1 + e ^ {- z ^ {\prime}}} \cdot e ^ {\epsilon^ {\prime}} \quad (2 e ^ {- z / 2} \leq 1 + e ^ {- z}) \\ \Rightarrow - \ell^ {\prime} (z) \leq - \ell^ {\prime} \left(z ^ {\prime}\right) \cdot 2 e ^ {\epsilon^ {\prime}} \quad (\text {d e f i n i t i o n} \ell^ {\prime} (\cdot)) \\ \Longrightarrow - \ell^ {\prime} (z) \leq - \ell^ {\prime} \left(z ^ {\prime}\right) 2 \left(1 + 2 \epsilon^ {\prime}\right) \quad \left(e ^ {z} \leq 1 + 2 z \text {f o r a l l} z \in [ 0, 1 ]\right) \\ \Longrightarrow - \ell^ {\prime} (z) \leq - 2 \ell^ {\prime} \left(z ^ {\prime}\right) + 4 \epsilon^ {\prime} \quad (- \ell^ {\prime} (\cdot) \leq 1) \\ \end{array}
$$
Let  $z_{*} \coloneqq yf_{\bar{\mathrm{W}}}(\delta_{*})$ ,  $z_{\mathrm{att}} \coloneqq yf_{\bar{\mathrm{W}}}(\delta_{\mathrm{att}})$  and  $\epsilon' \coloneqq \frac{\nu^2}{\eta_{\mathrm{att}}T}$ . Using the above inequality, sub-optimality in terms of  $\ell_{-}(\cdot)$ , as given in Equation (3), implies sub-optimality in terms of  $-\ell'(\cdot)$ :
$$
2 \ell_ {-} \left(z _ {*} / 2\right) \leq \ell_ {-} \left(z _ {\text {a t t}}\right) + \epsilon^ {\prime} \Longrightarrow - \ell^ {\prime} \left(z _ {*}\right) \leq - 2 \ell^ {\prime} \left(z _ {\text {a t t}}\right) + 4 \epsilon^ {\prime}
$$
That is, for any  $(\mathbf{x},y)\sim \mathcal{D}$  , it holds with probability one that
$$
- \ell^ {\prime} \left(y f _ {\bar {W}} \left(\delta_ {*}\right)\right) \leq - 2 \ell^ {\prime} \left(y f _ {\bar {W}} \left(\delta_ {\text {a t t}}\right)\right) + \frac {4 \nu^ {2}}{\eta_ {\text {a t t}} T}. \tag {4}
$$
Let  $\mathbb{E}_{\mathrm{AdvTr}}[\cdot]$  denote the expectation over a random draw of samples  $(\mathrm{x}_i, y_i)_{i=1}^t$  for Adversarial Training given in Algorithm 2. Let  $\hat{H}_t \coloneqq \langle \mathrm{W}_t, \mathrm{V}_\ast \rangle$ ,  $\hat{G}_t^2 = \| \mathrm{W}_t\|_F^2$ , and let  $H_t \coloneqq \mathbb{E}_{\mathrm{AdvTr}}[\hat{H}_t]$ ,  $G_t^2 \coloneqq \mathbb{E}_{\mathrm{AdvTr}}[\hat{G}_t^2]$  be their corresponding population version.
Proof of Lemma 3.7. The proof heavily builds on the prior works of Frei et al. [2021]; we include it here for completeness. Let  $\mathrm{V}_{*} \in \mathbb{R}^{m \times d}$  be such that  $\mathrm{v}_r = \frac{1}{\sqrt{m}} \operatorname{sgn}(a_r) \mathrm{v}_*$ . We define the correlation between the iterates and  $\mathrm{V}_{*}$  as follows  $\hat{H}_t \coloneqq \langle \mathrm{W}_t, \mathrm{V}_* \rangle$ . This correlation evolves as:
$$
\begin{array}{l} \hat {H} _ {t + 1} = \left\langle \mathrm {W} _ {t + 1}, \mathrm {V} _ {*} \right\rangle \\ = \left\langle W _ {t} - \eta_ {\mathrm {t r}} \nabla \ell \left(y _ {t} f _ {\delta_ {t}} \left(W _ {t}\right)\right), V _ {*} \right\rangle \\ = \hat {H} _ {t} - \eta_ {\mathrm {t r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) y _ {t} \left\langle \nabla_ {\mathrm {W}} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right), \mathrm {V} _ {*} \right\rangle \\ \end{array}
$$
Recall that  $\frac{\partial}{\partial\mathrm{w}_r} f_{\delta}(\mathrm{W}) = a_r\sigma '(w_r\cdot \delta)\delta$  . Therefore, we have that:
$$
\begin{array}{l} y _ {t} \langle \nabla_ {\mathrm {W}} f _ {\delta_ {t}} (\mathrm {W} _ {t}), \mathrm {V} _ {*} \rangle = y _ {t} \sum_ {r = 1} ^ {m} \langle a _ {r} \sigma^ {\prime} (\mathrm {w} _ {r} \cdot \delta) \delta , \frac {1}{\sqrt {m}} \mathrm {s g n} (a _ {r}) \mathrm {v} _ {*} \rangle \\ = \frac {\kappa}{\sqrt {m}} \sum_ {r = 1} ^ {m} \sigma^ {\prime} \left(\mathrm {w} _ {r} \cdot \delta\right) y _ {t} \langle \delta , \mathrm {v} _ {*} \rangle \\ \end{array}
$$
$$
\begin{array}{l} y _ {t} \langle \delta , v _ {*} \rangle = y _ {t} \langle x, v _ {*} \rangle + y _ {t} \langle \delta - x, v _ {*} \rangle \\ \geq \gamma - \left| y _ {t} \left\langle \delta - \mathrm {x}, \mathrm {v} _ {*} \right\rangle \right| \\ \geq \gamma - \| \delta - x \| \| v _ {*} \| \\ \geq \gamma - \nu \\ \end{array}
$$
On the other hand, for leaky ReLU, it holds that  $\sigma'(\cdot) \geq \alpha$ . Therefore, we arrive at
$$
\begin{array}{l} y _ {t} \left\langle \nabla_ {\mathrm {W}} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right), \mathrm {V} _ {*} \right\rangle \geq \frac {\kappa}{\sqrt {m}} \sum_ {r = 1} ^ {m} \alpha (\gamma - \nu) \\ = \kappa \alpha \sqrt {m} (\gamma - \nu) \\ \end{array}
$$
$$
\begin{array}{l} \hat {H} _ {t + 1} \geq \hat {H} _ {t} - \eta_ {\mathrm {t r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \alpha \kappa \sqrt {m} (\gamma - \nu) \\ \Rightarrow H _ {T + 1} \geq H _ {1} - \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathrm {A d v T r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \quad (t a k i n g \text {e x p e c t i o n} \mathbb {E} _ {\mathrm {A d v T r}} [ \cdot ]) \\ \end{array}
$$
The gradient norm is bounded as follows:
$$
\| \nabla_ {\mathrm {W}} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right) \| ^ {2} = \sum_ {j = 1} ^ {m} \| a _ {j} \sigma^ {\prime} \left(\mathrm {w} _ {j, t} \cdot \delta_ {t}\right) \delta_ {t} \| ^ {2} \leq m \kappa^ {2} (R + \nu) ^ {2}
$$
Next, we analyze the norm of the iterates, i.e.,  $\hat{G}_t^2 = \| \mathrm{W}_t\| _F^2$ . It is also easy to verify that  $-\ell^{\prime}(z)z = \frac{ze^{-z}}{1 + e^{-z}} = \frac{z}{1 + e^{z}}\leq 1$ . We have:
$$
\begin{array}{l} \hat {G} _ {t + 1} ^ {2} = \left\| \mathrm {W} _ {t} - \eta_ {\mathrm {t r}} \nabla \ell \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \right\| ^ {2} \\ = \| \mathrm {W} _ {t} \| _ {F} ^ {2} + \eta_ {\mathrm {t r}} ^ {2} \| \nabla \ell \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \| ^ {2} - 2 \eta_ {\mathrm {t r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) y _ {t} \nabla_ {\mathrm {W}} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right) \cdot \mathrm {W} _ {t} \\ = \hat {G} _ {t} ^ {2} + \eta_ {\mathrm {t r}} ^ {2} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) ^ {2} \left\| \nabla_ {\mathrm {W}} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \| ^ {2} - 2 \eta_ {\mathrm {t r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right) \\ \leq \hat {G} _ {t} ^ {2} + \eta_ {\mathrm {t r}} ^ {2} m \kappa^ {2} (R + \nu) ^ {2} + 2 \eta_ {\mathrm {t r}} \quad (- \ell^ {\prime} (z) z \leq 1) \\ \leq \hat {G} _ {t} ^ {2} + 3 \eta_ {\mathrm {t r}} \quad (\eta_ {\mathrm {t r}} \leq m ^ {- 1} \kappa^ {- 2} (R + \nu) ^ {- 2}) \\ \end{array}
$$
Therefore, taking expectation  $\mathbb{E}_{\mathrm{AdvTr}}$  on both sides, we have that
$$
G _ {T + 1} ^ {2} \leq G _ {1} ^ {2} + 3 \eta_ {\mathrm {t r}} T, \tag {5}
$$
and using  $\sqrt{a + b} \leq \sqrt{a} + \sqrt{b}$ , we have  $G_{T} \leq G_{1} + \sqrt{3\eta_{\mathrm{tr}}T}$ . Also, we have that  $H_{t}^{2} = (\mathbb{E}_{\mathrm{AdvTr}}\langle \mathrm{W}_{t}, \mathrm{V}_{*}\rangle)^{2} \leq \mathbb{E}_{\mathrm{AdvTr}}\| \mathrm{W}_{t}\|_{F}^{2}\| \mathrm{V}_{*}\|_{F}^{2} \leq G_{t}^{2}$  so that  $|H_{t}| \leq G_{t}$ . Putting all together, we get:
$$
\begin{array}{l} - G _ {1} - \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathrm {A d v T r}} \ell^ {\prime} (y _ {t} f _ {\delta_ {t}} (\mathrm {W} _ {t})) \leq H _ {0} - \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathrm {A d v T r}} \ell^ {\prime} (y _ {t} f _ {\delta_ {t}} (\mathrm {W} _ {t})) \\ \leq H _ {T} \\ \leq G _ {T} \\ \leq G _ {1} + \sqrt {3 \eta_ {\mathrm {t r}} T} \\ \end{array}
$$
$$
- \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathrm {A d v T r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \leq 2 G _ {1} + 2 \sqrt {\eta_ {\mathrm {t r}}} T
$$
We now argue that for any  $\epsilon$ , there exist an iterate  $t$  such that  $-\mathbb{E}_{\mathrm{AdvTr}}\ell'(y_t f_{\delta_t}(\mathrm{W}_t)) \leq \epsilon$ . Assume otherwise, then we get:
$$
\begin{array}{l} \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \epsilon T \leq - \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathrm {A d v T r}} \ell^ {\prime} \left(y _ {t} f _ {\delta_ {t}} \left(\mathrm {W} _ {t}\right)\right) \\ \leq 2 G _ {1} + 2 \sqrt {\eta_ {\mathrm {t r}} T} \\ \Longrightarrow \eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \epsilon \tau^ {2} - 2 \sqrt {\eta_ {\mathrm {t r}}} \tau - 2 G _ {1} \leq 0 \\ \Rightarrow \tau \leq \frac {\sqrt {\eta_ {\mathrm {t r}}} + \sqrt {\eta_ {\mathrm {t r}} + 2 G _ {1} \eta_ {\mathrm {t r}} \gamma \alpha \kappa \sqrt {m} \epsilon}}{\eta_ {\mathrm {t r}} (\gamma - \nu) \alpha \kappa \sqrt {m} \epsilon} \\ \Rightarrow T \leq \frac {4 (1 + G _ {1} \gamma \alpha \kappa \sqrt {m} \epsilon)}{\eta_ {\mathrm {t r}} (\gamma - \nu) ^ {2} \alpha^ {2} \kappa^ {2} m \epsilon^ {2}} \\ \end{array}
$$
Therefore, we have that  $-\mathbb{E}_{\mathrm{AdvTr}}\ell '(y_tf_{\delta_t}(\mathrm{W}_t)) = -\mathbb{E}_{\mathrm{AdvTr}}\ell '(y_tf_{\mathrm{W}_t}(\delta_{\mathrm{att}}(\mathrm{x}_t)))\leq \epsilon$  . Moreover,
$$
\mathbb {E} _ {\text {A d v T r}} \left[ - \ell^ {\prime} \left(y _ {t} f _ {\mathrm {W} _ {t}} \left(\delta_ {\text {a t t}} \left(\mathrm {x} _ {t}\right)\right)\right) \right] = \mathbb {E} _ {S _ {t} \sim \mathcal {D} ^ {t}} \left[ - \ell^ {\prime} \left(y _ {t} f _ {\mathrm {W} _ {t}} \left(\delta_ {\text {a t t}} \left(\mathrm {x} _ {t}\right)\right)\right) \right]
$$
(independence from future samples)
$$
= \mathbb {E} _ {\mathcal {S} _ {t - 1} \sim \mathcal {D} ^ {t - 1}} \mathbb {E} _ {\left(\mathrm {x} _ {t}, y _ {t}\right) \sim \mathcal {D}} \left[ - \ell^ {\prime} \left(y _ {t} f _ {\mathrm {W} _ {t}} \left(\delta_ {\mathrm {a t t}} \left(\mathrm {x} _ {t}\right)\right)\right) \mid \mathcal {S} _ {t - 1} \right]
$$
(Smoothing property of the conditional expectation)
$$
= \mathbb {E} _ {\mathcal {S}} \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ - \ell^ {\prime} (y f _ {W _ {t}} (\delta_ {\mathrm {a t t}} (x))) ]
$$
$\left(\mathrm{W}_t\right)$  independent of  $(\mathbf{x}_t,y_t)$  given  $\mathcal{S}_{t - 1}$