SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\mathrm {a d v}} \left(\boldsymbol {x} ^ {\mathrm {i n}}\right) = \max  _ {\| \boldsymbol {\eta} \| _ {p} \leq \epsilon} \left\| \boldsymbol {J} \left(\boldsymbol {x} ^ {\mathrm {i n}}\right) \boldsymbol {\eta} \right\| = \epsilon \left\| \boldsymbol {J} \left(\boldsymbol {x} ^ {\mathrm {i n}}\right) \right\| _ {p, q}. \tag {A107}
$$
Proposition G.2. Suppose that the perturbation constraint  $\epsilon$  is sufficiently small. For any  $\pmb{x}^{\mathrm{in}}\in \mathbb{R}^{d}$  and  $(p,q) = (1,1)$ ,  $(1,2)$ ,  $(2,\infty)$ , and  $(\infty ,\infty)$ , the following equality holds:
$$
\mathcal {L} _ {\mathrm {a d v}} \left(\boldsymbol {x} ^ {\text {i n}}\right) = \epsilon \beta_ {p, q} \omega^ {L / 2}. \tag {A108}
$$
Proof. As shown in Eq. (A107), the adversarial loss (Eq. (3)) is equal to the  $(p,q)$ -operator norm of  $J(\boldsymbol{x}^{\mathrm{in}})$  under this assumption. For  $(p,q) = (1,1)$ ,  $(1,2)$ ,  $(2,\infty)$ , and  $(\infty ,\infty)$ , we can obtain the equalities of the  $(p,q)$ -operator norms (cf. the proof of Thm 5.1). Thus, the claim is established. Note that, for  $(p,q) = (1,\infty)$  and  $(2,2)$ , we can derive only upper bounds (cf. the proof of Thm 5.1).
Proposition G.3. Suppose that the perturbation constraint  $\epsilon$  is sufficiently small and the output dimension is one. For any  $\pmb{x}^{\mathrm{in}}\in \mathbb{R}^{d}$ , and  $(p,q) = (2,\infty)$  and  $(\infty ,\infty)$ , the following equality holds:
$$
\mathcal {L} _ {\mathrm {a d v}} \left(\boldsymbol {x} ^ {\text {i n}}\right) = \epsilon \beta_ {p, q} \omega^ {L / 2}. \tag {A109}
$$
In addition, for any  $\pmb{x}^{\mathrm{in}}\in \mathbb{R}^{d}$  and  $(p,q) = (2,2)$ , the following equality holds:
$$
\mathcal {L} _ {\mathrm {a d v}} \left(\boldsymbol {x} ^ {\text {i n}}\right) = \epsilon \omega^ {L / 2}. \tag {A110}
$$
Proof. Similar to Prop G.2, the upper bounds for  $(p,q) = (2,\infty)$  and  $(\infty ,\infty)$  become equalities. The upper bounds for  $(p,q) = (1,1)$  and  $(1,2)$  require sufficiently large  $K$ , and thus, they do not become equalities. In addition, as Tab. A3 shows, the  $(2,2)$ -operator norm is a maximum singular value. When  $K = 1$ , the maximum singular value is identical to the Frobenius norm  $(\ell_2$  norm), which is calculated as follows:
$$
\begin{array}{l} \left\| \boldsymbol {J} \left(\boldsymbol {x} ^ {\text {i n}}\right) \right\| _ {\mathrm {F}} = \sqrt {\sum_ {j = 1} ^ {d} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {1 j} ^ {2}} (A111) \\ = \sqrt {d \frac {1}{d} \sum_ {j = 1} ^ {d} J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {1 j} ^ {2}} (A112) \\ = \sqrt {d \mathbb {E} \left[ J \left(\boldsymbol {x} ^ {\text {i n}}\right) _ {1 j} ^ {2} \right]} (A113) \\ = \omega^ {L / 2}. (A114) \\ \end{array}
$$
Thus, in contrast to the proof of Thm 5.1, we can obtain the equality of the  $(2,2)$ -operator norm instead of the inequality (upper bound).
G.2 Derivation of the theorems in Sec. 5.2
In the following, we set  $\mathcal{L}_{\mathrm{adv}} \coloneqq \epsilon \beta_{p,q} \omega^{L/2}$ . As a preliminary, we state the following two lemmas. These are used to solve the differential equation derived from gradient flow.
Lemma G.4. For  $t, a, b \in \mathbb{R}, x: \mathbb{R} \to \mathbb{R}$ , and  $x(0) \coloneqq x_0$ , the following holds:
$$
\frac {\mathrm {d} x (t)}{\mathrm {d} t} = - a x (t) ^ {b} \Leftrightarrow x (t) = \left(a (b - 1) t + x _ {0} ^ {- (b - 1)}\right) ^ {- 1 / (b - 1)}. \tag {A115}
$$
$$
\frac {\mathrm {d} x (t)}{\mathrm {d} t} = - a x (t) ^ {b} \tag {A116}
$$
$$
\int x ^ {- b} \mathrm {d} x = - a \int \mathrm {d} t \tag {A117}
$$
$$
\frac {1}{- (b - 1)} x ^ {- (b - 1)} = - a t + C _ {1} \tag {A118}
$$
$$
x (t) = \left(a (b - 1) t + C _ {2}\right) ^ {- 1 / (b - 1)}. \tag {A119}
$$
By the initial value,
$$
x (0) := x _ {0} = C _ {2} ^ {- 1 / (b - 1)} \tag {A120}
$$
$$
C _ {2} = x _ {0} ^ {- (b - 1)}. \tag {A121}
$$
$$
x (t) = \left(a (b - 1) t + x _ {0} ^ {- (b - 1)}\right) ^ {- 1 / (b - 1)}. \tag {A122}
$$
Lemma G.5. For  $t, c \in \mathbb{R}, a, b \geq 0, x: \mathbb{R} \to \mathbb{R}, x(0) := x_0$ , and  $0 \leq bx(0) \ll 1$ , the following holds:
$$
\frac {\mathrm {d} x (t)}{\mathrm {d} t} = - a \left(1 + b x (t)\right) ^ {c} x (t) \Leftrightarrow x (t) = \frac {x _ {0}}{\left(1 + b c x _ {0}\right) \exp (a t) - b c x _ {0}}. \tag {A123}
$$
Proof. By  $-a(1 + bx(t))^c x(t) \leq 0$ ,  $x(t)$  does not increase with  $t$ . That is,  $bx(t) \leq bx(0) \ll 1$ . Therefore, we can approximate  $(1 + bx(t))^c$  as  $1 + bcx(t)$  using the binomial theorem. Thus, we try to solve the following differential equation:
$$
\frac {\mathrm {d} x (t)}{\mathrm {d} t} = - a (1 + b c x (t)) x (t), \tag {A124}
$$
whose solution is as follows:
$$
x (t) = \frac {C _ {1}}{b c \left(\exp (a t) - C _ {1}\right)}. \tag {A125}
$$
$$
x (0) := x _ {0} = \frac {C _ {1}}{b c \left(1 - C _ {1}\right)} \tag {A126}
$$
$$
b c \left(1 - C _ {1}\right) x _ {0} = C _ {1} \tag {A127}
$$
$$
b c x _ {0} = (1 + b c x _ {0}) C _ {1} \tag {A128}
$$
$$
C _ {1} = \frac {b c x _ {0}}{1 + b c x _ {0}}. \tag {A129}
$$
$$
\begin{array}{l} x (t) = \frac {\frac {b c x _ {0}}{1 + b c x _ {0}}}{b c \left(\exp (a t) - \frac {b c x _ {0}}{1 + b c x _ {0}}\right)} (A130) \\ = \frac {b c x _ {0}}{b c \left(\left(1 + b c x _ {0}\right) \exp (a t) - b c x _ {0}\right)} (A131) \\ = \frac {x _ {0}}{(1 + b c x _ {0}) \exp (a t) - b c x _ {0}}. (A132) \\ \end{array}
$$
Then, using the update equation of parameters (Eq. (10)), we consider the differential equation of weight variance as follows:
Lemma G.6. Suppose that Asm 5.2 holds. Let  $\mathcal{L}_1, \ldots, \mathcal{L}_n: \mathbb{R} \to \mathbb{R}$  be the  $n$  loss functions and  $\mathcal{W} := \{W_{11}^{(1)}, W_{12}^{(1)}, \ldots, W_{NN}^{(L)}\}$  be the set of all network weights. A network is trained by minimizing the sum of loss functions,  $\sum_{i=1}^{n} \mathcal{L}_i(\boldsymbol{x}^{\mathrm{in}})$ . The network parameters are updated similarly to Eq. (10). The differential equation of  $\sigma_w^2(t)$  is given by:
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - N \mathbb {E} _ {W \in \mathcal {W}} \left[ W (t) \sum_ {i = 1} ^ {n} \frac {\partial \mathcal {L} _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial W (t)} \right] \tag {A133}
$$
Proof. Since Asm 5.2 holds and the number of weights,  $LN^2$ , is sufficiently large,  $\sigma_w^2 (t + \mathrm{d}t)$  can be represented as follows:
$$
\sigma_ {w} ^ {2} (t + \mathrm {d} t) = N \frac {\sigma_ {w} ^ {2} (t + \mathrm {d} t)}{N} = N \mathbb {V} _ {W \in \mathcal {W}} [ W (t + \mathrm {d} t) ] = N \mathbb {E} _ {W \in \mathcal {W}} [ W (t + \mathrm {d} t) ^ {2} ]. \tag {A134}
$$
We can expand  $W(t + \mathrm{d}t)$  using Eq. (10) and rearrange the above equation as follows:
$$
\begin{array}{l} \sigma_ {w} ^ {2} (t + \mathrm {d} t) = N \mathbb {E} _ {W \in \mathcal {W}} \left[ \left(W (t) - \sum_ {i = 1} ^ {n} \frac {\partial \mathcal {L} _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial W (t)} \mathrm {d} t\right) ^ {2} \right] (A135) \\ = N \mathbb {E} _ {W \in \mathcal {W}} \left[ W (t) ^ {2} - W (t) \sum_ {i = 1} ^ {n} \frac {\partial \mathcal {L} _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial W (t)} \mathrm {d} t + \mathcal {O} (\mathrm {d} t ^ {2}) \right] (A136) \\ = \sigma_ {w} ^ {2} (t) - N \mathbb {E} _ {W \in \mathcal {W}} \left[ W (t) \sum_ {i = 1} ^ {n} \frac {\partial \mathcal {L} _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial W (t)} \mathrm {d} t \right] + \mathcal {O} (\mathrm {d} t ^ {2}) (A137) \\ \approx \sigma_ {w} ^ {2} (t) - N \mathbb {E} _ {W \in \mathcal {W}} \left[ W (t) \sum_ {i = 1} ^ {n} \frac {\partial \mathcal {L} _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial W (t)} \right] \mathrm {d} t. (A138) \\ \end{array}
$$
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - N \mathbb {E} _ {W \in \mathcal {W}} \left[ W (t) \sum_ {i = 1} ^ {n} \frac {\partial \mathcal {L} _ {i} \left(\boldsymbol {x} ^ {\text {i n}}\right)}{\partial W (t)} \right]. \tag {A139}
$$
Lemma G.7. For  $\mathcal{L}_{\mathrm{adv}}\coloneqq \epsilon \beta_{p,q}\omega^{L / 2}$ , the following equality holds:
$$
\frac {\partial \mathcal {L} _ {\mathrm {a d v}}}{\partial W} = \frac {\epsilon \alpha \beta_ {p , q} \omega^ {L / 2 - 1}}{N} W. \tag {A140}
$$
Proof. If the number of network weights, i.e.,  $LN^2$ , is sufficiently large, we can represent the weight variance as follows:
$$
\frac {\sigma_ {w} ^ {2}}{N} = \frac {1}{L N ^ {2}} \sum_ {W \in \mathcal {W}} W ^ {2}. \tag {A141}
$$
The derivative of  $\omega$  with respect to  $W\in \mathcal{W}$  can be calculated as follows:
$$
\frac {\partial \omega_ {\mathrm {v}}}{\partial W} = \frac {\partial \left(\alpha \sigma_ {w} ^ {2}\right)}{\partial W} = \frac {\partial \left(\alpha N \frac {\sigma_ {w} ^ {2}}{N}\right)}{\partial W} = \alpha N \frac {\partial \left(\frac {1}{L N ^ {2}} \sum_ {V \in \mathcal {W}} V ^ {2}\right)}{\partial W} = \frac {2 \alpha}{L N} W, \tag {A142}
$$
$$
\frac {\partial \omega_ {\mathrm {r}}}{\partial W} = \frac {\partial \left(1 + \alpha \sigma_ {w} ^ {2}\right)}{\partial W} = \frac {2 \alpha}{L N} W. \tag {A143}
$$
The derivative of  $\omega^{L / 2}$  with respect to  $W\in \mathcal{W}$  can be calculated as follows:
$$
\frac {\partial \omega^ {L / 2}}{\partial W} = \frac {L}{2} \omega^ {L / 2 - 1} \frac {\partial \omega}{\partial W} = \frac {L}{2} \omega^ {L / 2 - 1} \frac {2 \alpha}{L N} W = \frac {\alpha \omega^ {L / 2 - 1}}{N} W. \tag {A144}
$$
$$
\frac {\partial \mathcal {L} _ {\mathrm {a d v}}}{\partial W} = \epsilon \beta_ {p, q} \frac {\partial \omega^ {L / 2}}{\partial W} = \frac {\epsilon \alpha \beta_ {p , q} \omega^ {L / 2 - 1}}{N} W. \tag {A145}
$$
Lemma G.8. Suppose that Asm 5.2 holds. Let  $\mathcal{L}_{\mathrm{std}}: \mathbb{R}^d \to \mathbb{R}$  be the standard loss function and  $\mathcal{L}_{\mathrm{adv}}: \mathbb{R}^d \to \mathbb{R}$  be the adversarial loss function. Suppose that Asm B.1 applies to  $\mathcal{L}_{\mathrm{std}}$ . The adversarial loss function is defined as  $\mathcal{L}_{\mathrm{adv}}(\pmb{x}^{\mathrm{in}}; t) \coloneqq \epsilon \beta_{p,q} \omega(t)^{L/2}$ . A network is trained by minimizing  $\mathcal{L}_{\mathrm{std}} + \mathcal{L}_{\mathrm{adv}}$ . Network parameters are updated by Eq. (10). Then, the differential equation of  $\sigma_w^2(t)$  is given by:
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \omega (t) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t). \tag {A146}
$$
Proof. By Lemma G.6,
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - N \mathbb {E} \left[ W (t) \frac {\partial \mathcal {L} _ {\mathrm {s t d}}}{\partial W (t)} \right] - N \mathbb {E} \left[ W (t) \frac {\partial \mathcal {L} _ {\mathrm {a d v}}}{\partial W (t)} \right]. \tag {A147}
$$
Since the standard loss satisfies Asm B.1,  $\mathbb{E}\left[W(t)\frac{\partial\mathcal{L}_{\mathrm{std}}}{\partial W(t)}\right] = 0$  and
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - N \mathbb {E} \left[ W (t) \frac {\partial \mathcal {L} _ {\mathrm {a d v}}}{\partial W (t)} \right]. \tag {A148}
$$
By Lemma G.7,
$$
\begin{array}{l} \frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - N \mathbb {E} \left[ W (t) \frac {\epsilon \alpha \beta_ {p , q} \omega (t) ^ {L / 2 - 1}}{N} W (t) \right] (A149) \\ = - \epsilon \alpha \beta_ {p, q} \omega (t) ^ {L / 2 - 1} \mathbb {E} [ W (t) ^ {2} ] (A150) \\ = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \omega (t) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t). (A151) \\ \end{array}
$$
Proof. By Lemma G.8, the time evolution of weight variance in a vanilla network is represented as follows:
$$
\begin{array}{l} \frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \omega_ {v} (t) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t) (A152) \\ = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \alpha^ {L / 2 - 1} \sigma_ {w} ^ {2} (t) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t) (A153) \\ = - \frac {\epsilon \alpha^ {L / 2} \beta_ {p , q}}{N} \sigma_ {w} ^ {2} (t) ^ {L / 2}. (A154) \\ \end{array}
$$
Denote  $L' \coloneqq L / 2 - 1$ . By Lemma G.4, we can obtain
$$
\begin{array}{l} \sigma_ {w} ^ {2} (t) = \left(\frac {\epsilon \alpha^ {L / 2} \beta_ {p , q}}{N} (L / 2 - 1) t + \sigma_ {w} ^ {2} (0) ^ {- (L / 2 - 1)}\right) ^ {- 1 / (L / 2 - 1)} (A155) \\ = \left(\frac {\epsilon \alpha^ {L ^ {\prime} + 1} \beta_ {p , q} L ^ {\prime}}{N} t + \sigma_ {w} ^ {2} (0) ^ {- L ^ {\prime}}\right) ^ {- 1 / L ^ {\prime}} (A156) \\ = \left(1 + \frac {\epsilon \alpha^ {L ^ {\prime} + 1} \sigma_ {w} ^ {2} (0) ^ {L ^ {\prime}} \beta_ {p , q} L ^ {\prime}}{N} t\right) ^ {- 1 / L ^ {\prime}} \sigma_ {w} ^ {2} (0) (A157) \\ = \left(1 + \frac {\epsilon \alpha \omega_ {\mathrm {v}} (0) ^ {L ^ {\prime}} \beta_ {p , q} L ^ {\prime}}{N} t\right) ^ {- 1 / L ^ {\prime}} \sigma_ {w} ^ {2} (0). (A158) \\ \end{array}
$$
By  $t\leq T\ll N$
$$
\sigma_ {w} ^ {2} (t) \approx \left(1 - \frac {1}{L ^ {\prime}} \frac {\epsilon \alpha \beta_ {p , q} \omega_ {\mathrm {v}} (0) ^ {L ^ {\prime}} L ^ {\prime}}{N} t\right) \sigma_ {w} ^ {2} (0) = \left(1 - \frac {\epsilon \alpha \beta_ {p , q} \omega_ {\mathrm {v}} (0) ^ {L ^ {\prime}}}{N} t\right) \sigma_ {w} ^ {2} (0). \tag {A159}
$$
Theorem G.9 (Weight time evolution of a residual network in adversarial training). Suppose that Asm 5.2 holds and  $\alpha \sigma_w^2(0) \ll 1$ . The time evolution of  $\sigma_w^2$  of a residual network in adversarial training is given by:
$$
\sigma_ {w} ^ {2} (t) = \left(1 - \left(1 + \alpha L ^ {\prime} \sigma_ {w} ^ {2} (0)\right) \epsilon \alpha \beta_ {p, q} t / N\right) \sigma_ {w} ^ {2} (0). \tag {A160}
$$
Proof. By Lemma G.8, the time evolution of weight variance in a residual network is represented as follows:
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \omega (t) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t) = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \left(1 + \alpha \sigma_ {w} ^ {2} (t)\right) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t). \tag {A161}
$$
By Lemma G.5, we can obtain
$$
\begin{array}{l} \sigma_ {w} ^ {2} (t) = \frac {\sigma_ {w} ^ {2} (0)}{(1 + \alpha (L / 2 - 1) \sigma_ {w} ^ {2} (0)) \exp \left(\frac {\epsilon \alpha \beta_ {p , q}}{N} t\right) - \alpha (L / 2 - 1) \sigma_ {w} ^ {2} (0)} (A162) \\ = \left(\left(\sigma_ {w} ^ {2} (0) ^ {- 1} + \alpha L ^ {\prime}\right) \exp \left(\epsilon \alpha \beta_ {p, q} t / N\right) - \alpha L ^ {\prime}\right) ^ {- 1}. (A163) \\ \end{array}
$$
By the Maclaurin expansion of the exponential function and  $t \leq T \ll N$ ,
$$
\begin{array}{l} \sigma_ {w} ^ {2} (t) \approx \left(\left(\sigma_ {w} ^ {2} (0) ^ {- 1} + \alpha L ^ {\prime}\right) \left(1 + \frac {\epsilon \alpha \beta_ {p , q} t}{N}\right) - \alpha L ^ {\prime}\right) ^ {- 1} (A164) \\ = \left(\sigma_ {w} ^ {2} (0) ^ {- 1} + \left(\sigma_ {w} ^ {2} (0) ^ {- 1} + \alpha L ^ {\prime}\right) \frac {\epsilon \alpha \beta_ {p , q} t}{N}\right) ^ {- 1} (A165) \\ = \left(1 + \left(1 + \alpha L ^ {\prime} \sigma_ {w} ^ {2} (0)\right) \frac {\epsilon \alpha \beta_ {p , q} t}{N}\right) ^ {- 1} \sigma_ {w} ^ {2} (0). (A166) \\ \end{array}
$$
$$
\sigma_ {w} ^ {2} (t) = \left(1 - \left(1 + \alpha L ^ {\prime} \sigma_ {w} ^ {2} (0)\right) \frac {\epsilon \alpha \beta_ {p , q} t}{N}\right) \sigma_ {w} ^ {2} (0). \tag {A167}
$$
Then, we consider the time evolution of weight variance, Thms 5.4 and G.9, at initialization satisfying  $(M,m)$ -trainability condition (Lemma 5.6). Here, we assume  $\omega(0) \approx 1$  satisfying Lemma 5.6. Under this assumption, the time evolution of weight variance in vanilla networks, Eq. (11), can be rearranged as follows:
$$
\sigma_ {w} ^ {2} (t) = \left(1 - \epsilon \alpha \beta_ {p, q} \omega_ {\mathrm {v}} (0) ^ {L / 2 - 1} t / N\right) \sigma_ {w} ^ {2} (0) \xrightarrow {\omega_ {\mathrm {v}} (0) \approx 1} \left(1 - \epsilon \alpha \beta_ {p, q} t / N\right) \sigma_ {w} ^ {2} (0). \tag {A168}
$$
In addition, the time evolution of weight variance in residual networks, Eq. (A160), can be rearranged as follows:
$$
\sigma_ {w} ^ {2} (t) = \left(1 - \left(1 + \alpha L ^ {\prime} \sigma_ {w} ^ {2} (0)\right) \epsilon \alpha \beta_ {p, q} t / N\right) \sigma_ {w} ^ {2} (0) \xrightarrow {\omega_ {\mathrm {r}} (0) \approx 1} (1 - \epsilon \alpha \beta_ {p, q} t / N) \sigma_ {w} ^ {2} (0). \tag {A169}
$$
Thus, the time evolution of weight variance is consistent in vanilla and residual networks at initialization satisfying Lemma 5.6.
G.3 Derivation of the theorems in Sec. 5.3
Proof. Applying Eq. (A38) to Eqs. (2) and (A19), we can obtain the following equations:
$$
\frac {\chi^ {(l)}}{\chi^ {(l + 1)}} = \left\{ \begin{array}{l l} \sigma_ {w} ^ {2} \mathbb {E} [ \phi^ {\prime} (h ^ {(l + 1)}) ] = \alpha \sigma_ {w} ^ {2} = \omega_ {\mathrm {v}} & (\text {v a n i l l a}) \\ 1 + \sigma_ {w} ^ {2} \mathbb {E} [ \phi^ {\prime} (h ^ {(l + 1)}) ^ {2} ] = 1 + \alpha \sigma_ {w} ^ {2} = \omega_ {\mathrm {r}} & (\text {r e s i d u a l}) \end{array} . \right. \tag {A170}
$$
Then, recursively computing the above equation, we can derive the following equations:
$$
\frac {\chi^ {(0)}}{\chi^ {(L)}} = \omega^ {L} = \left\{ \begin{array}{l l} \left(\alpha \sigma_ {w} ^ {2}\right) ^ {L} & (\text {v a n i l l a}) \\ \left(1 + \alpha \sigma_ {w} ^ {2}\right) ^ {L} & (\text {r e s i d u a l}) \end{array} . \right. \tag {A171}
$$
Applying this equation to Defn 5.5,
$$
m \leq \frac {\chi^ {(0)}}{\chi^ {(L)}} \leq M \Leftrightarrow m ^ {1 / L} \leq \omega \leq M ^ {1 / L}. \tag {A172}
$$
For a vanilla network,
$$
m ^ {1 / L} \leq \alpha \sigma_ {w} ^ {2} \leq M ^ {1 / L}. \tag {A173}
$$
For a residual network,
$$
m ^ {1 / L} - 1 \leq \alpha \sigma_ {w} ^ {2} \leq M ^ {1 / L} - 1. \tag {A174}
$$
In particular, for a residual network, by  $m^{1 / L} - 1\leq 0$  and  $\alpha \sigma_w^2\geq 0$
$$
(0 \leq) \alpha \sigma_ {w} ^ {2} \leq M ^ {1 / L} - 1. \tag {A175}
$$
Proof. Let us consider  $T$  breaking the  $(M, m)$ -trainability condition. As Thm 5.4 claims,  $\sigma_w^2(t)$  decreases monotonically with  $t$ . Thus, we consider  $T$  such that  $\sigma_w^2(T)$  is less than the lower bound.
$$
m ^ {1 / L} \geq \alpha \sigma_ {w} ^ {2} (T) = \left(1 - \frac {\epsilon \alpha \beta_ {p , q} T}{N}\right) \alpha \sigma_ {w} ^ {2} (0) = 1 - \frac {\epsilon \alpha \beta_ {p , q} T}{N} \tag {A176}
$$
$$
\frac {\epsilon \alpha \beta_ {p , q} T}{N} \geq 1 - m ^ {1 / L} \tag {A177}
$$
$$
T \geq \frac {\left(1 - m ^ {1 / L}\right) N}{\epsilon \alpha \beta_ {p , q}}. \tag {A178}
$$
Proof. By Thm G.9,  $\sigma_w^2 (t)$  decreases monotonically. Thus, the following inequality holds, and, by Lemma 5.6, the  $(M,m)$ -trainability condition always holds in  $0\leq t\leq T$ :
$$
\alpha \sigma_ {w} ^ {2} (t) \leq \alpha \sigma_ {w} ^ {2} (0) \leq M ^ {1 / L} - 1. \tag {A179}
$$
Proposition G.10 (Residual networks are adversarially trainable without careful weight initialization). Suppose that Asm 5.2 holds, the network is residual, and the  $(M,m)$ -trainability condition is not satisfied at  $t = 0$ . If
$$
T \geq \frac {\left(\alpha \sigma_ {w} ^ {2} (0) - \left(M ^ {1 / L} - 1\right)\right) N}{\epsilon \alpha^ {L / 2 + 1} \beta_ {p , q} \sigma_ {w} ^ {2} (0) ^ {L / 2}}, \tag {A180}
$$
there exists  $0 < \tau \leq T$  such that the  $(M,m)$ -trainability condition hold for  $\tau \leq t \leq T$ .
Proof. Denote the time evolution of the weight variance on a vanilla network by  $\sigma_{w,\mathrm{v}}^2$  and the variance on a residual network by  $\sigma_{w,\mathrm{r}}^2$ . By Eq. (A161),
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - \frac {\epsilon \alpha \beta_ {p , q}}{N} \left(1 + \alpha \sigma_ {w} ^ {2} (t)\right) ^ {L / 2 - 1} \sigma_ {w} ^ {2} (t) \leq - \frac {\epsilon \alpha^ {L / 2} \beta_ {p , q}}{N} \sigma_ {w} ^ {2} (t) ^ {L / 2}. \tag {A181}
$$
Recall that the following differential equation represents the time evolution of weight variance on a vanilla network (cf. Thm 5.4):
$$
\frac {\mathrm {d} \sigma_ {w} ^ {2} (t)}{\mathrm {d} t} = - \frac {\epsilon \alpha^ {L / 2} \beta_ {p , q}}{N} \sigma_ {w} ^ {2} (t) ^ {L / 2}. \tag {A182}
$$
From the above,  $\sigma_{w,\mathrm{r}}^2 (t)\leq \sigma_{w,\mathrm{v}}^2 (t)$  if  $\sigma_w^2 (0)$  is the same. Here, we consider  $T$  such that  $\sigma_{w,\mathrm{v}}^2 (T)$  is less than the upper bound of the  $(M,m)$ -trainability condition. By Thm 5.4,
$$
\alpha \sigma_ {w, v} ^ {2} (T) \leq M ^ {1 / L} - 1 \tag {A183}
$$
$$
\left(1 - \frac {\epsilon \alpha \beta_ {p , q} \omega_ {\mathrm {v}} (0) ^ {L ^ {\prime}} T}{N}\right) \alpha \sigma_ {w, \mathrm {v}} ^ {2} (0) \leq M ^ {1 / L} - 1 \tag {A184}
$$
$$
1 - \frac {\epsilon \alpha \beta_ {p , q} \alpha^ {L ^ {\prime}} \sigma_ {w} ^ {2} (0) ^ {L ^ {\prime}} T}{N} \leq \frac {M ^ {1 / L} - 1}{\alpha \sigma_ {w} ^ {2} (0)} \tag {A185}
$$
$$
1 - \frac {M ^ {1 / L} - 1}{\alpha \sigma_ {w} ^ {2} (0)} \leq \frac {\epsilon \alpha \beta_ {p , q} \alpha^ {L ^ {\prime}} \sigma_ {w} ^ {2} (0) ^ {L ^ {\prime}} T}{N} \tag {A186}
$$
$$
\frac {\left(\alpha \sigma_ {w} ^ {2} (0) - M ^ {1 / L} + 1\right) N}{\alpha \sigma_ {w} ^ {2} (0) \epsilon \alpha \beta_ {p , q} \alpha^ {L ^ {\prime}} \sigma_ {w} ^ {2} (0) ^ {L ^ {\prime}}} \leq T \tag {A187}
$$
$$
\frac {\left(\alpha \sigma_ {w} ^ {2} (0) - M ^ {1 / L} + 1\right) N}{\epsilon \alpha^ {L / 2 + 1} \beta_ {p , q} \sigma_ {w} ^ {2} (0) ^ {L / 2}} \leq T. \tag {A188}
$$
G.4 Derivation of the theorems in Sec. 5.4
First, we state the following two lemmas as a preliminary.
Lemma G.11. Consider a vanilla network. Suppose that Asm B.1 is applied to the network output. For any  $l \in [L]$ , the mean squared gradient of  $f_{i}$  with respect to  $h_{j}^{(l)}$  is given by:
$$
\mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {j} ^ {(l)}}\right) ^ {2} \right] = \alpha \omega_ {\mathrm {v}} ^ {L - l} \chi^ {(L)}. \tag {A189}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {j} ^ {(l)}}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial \boldsymbol {h} ^ {(l + 1)}} \frac {\partial \boldsymbol {h} ^ {(l + 1)}}{\partial x _ {j} ^ {(l)}} \frac {\partial x _ {j} ^ {(l)}}{\partial h _ {j} ^ {(l)}}\right) ^ {2} \right] (A190) \\ = \mathbb {E} \left[ \left(\sum_ {k = 1} ^ {N} \frac {\partial f _ {i}}{\partial h _ {k} ^ {(l + 1)}} W _ {k j} ^ {(l + 1)}\right) ^ {2} \phi^ {\prime} \left(h _ {j} ^ {(l)}\right) ^ {2} \right]. (A191) \\ \end{array}
$$
By Asm B.1 and Eq. (A38),
$$
\begin{array}{l} \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {j} ^ {(l)}}\right) ^ {2} \right] = \sigma_ {w} ^ {2} \mathbb {E} [ \phi^ {\prime} \left(h _ {j} ^ {(l)}\right) ^ {2} ] \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {k} ^ {(l + 1)}}\right) ^ {2} \right] (A192) \\ = \omega_ {\mathrm {v}} \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {k} ^ {(l + 1)}}\right) ^ {2} \right] (A193) \\ = \omega_ {\mathrm {v}} ^ {L - l} \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {k} ^ {(L)}}\right) ^ {2} \right]. (A194) \\ \end{array}
$$
Moreover, by Asm B.1,
$$
\mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {j} ^ {(L)}}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial x _ {j} ^ {(L)}} \frac {\partial x _ {j} ^ {(L)}}{\partial h _ {j} ^ {(L)}}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial x _ {j} ^ {(L)}}\right) ^ {2} \phi^ {\prime} (h _ {j} ^ {(L)}) ^ {2} \right] = \alpha \chi^ {(L)}. \tag {A195}
$$
$$
\mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial h _ {i} ^ {(l)}}\right) ^ {2} \right] = \alpha \omega_ {\mathrm {v}} ^ {L - l} \chi^ {(L)}. \tag {A196}
$$
Lemma G.12. Suppose that Asm B.1 is applied to the network output. For any  $l \in [L]$ , the mean squared gradient of  $f_{i}$  with respect to a weight is given by:
$$
\mathbb {E} \left[ \left(\frac {\partial f _ {i}}{\partial W _ {j k} ^ {(l)}}\right) ^ {2} \right] = \alpha \omega^ {L - 1} \chi^ {(L)} \mathbb {E} [ (x _ {j} ^ {(0)}) ^ {2} ] + \alpha^ {2} \omega^ {L - l} \chi^ {(L)} \sigma_ {b} ^ {2} \sum_ {m = 1} ^ {l - 1} \omega^ {m - 1}. \tag {A197}
$$
Proof. Vanilla. By Asm B.1,