SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, we set training data size  $n = 20$ ,  $d = 40$  and train the NN with full batch gradient descent with a learning rate 0.1 for 1000 epochs. We train the FNN model defined in Section 3 with ReLU (or leaky-ReLU) activation function and width  $m = 10000$ . As we can observe from Figure 5,
the stable rank for the leaky ReLU network with large slopes  $\gamma$  will converge to 1 when epoch  $t\to \infty$  In comparison, the stable rank for the large-width ReLU network will not converge to 1 but to 2.
A.2 Additional Experiment on MINIST
Our focus is the training of a two-layer feed-forward neural network, as discussed in Section 3, utilizing either ReLU or leaky-ReLU activation functions. We examine different widths, specifically choosing from  $\{10,50,100,500,1000\}$ .
The network initialization process follows a Gaussian distribution, with a variance of  $\sigma_0 = 0.00001$ . Training is executed using stochastic gradient descent, a batch size of 64, and a learning rate of 0.1, for a total of 10 epochs. As discerned from Figures 6 and 7, the stable rank of networks utilizing either ReLU or leaky ReLU is weakly influenced by the width. For an exceedingly small width such as 10, the weight matrix is low rank with a correspondingly small stable rank. However, this also results in low test accuracy as the network cannot effectively learn all necessary features. As the width increases, the test accuracy and final stable rank will increase. However, for sufficiently large widths, an increase in width no longer corresponds to stable rank or test accuracy increases.
Figure 6: Stable ranks and test errors for different width across multiple runs (ReLU Activation Function). Each line represents the mean stable rank or test error for a given weight variance, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 5 runs.
Figure 7: Stable ranks and test errors for different width across multiple runs (leaky-ReLU Activation Function). Each line represents the mean stable rank or test error for a given weight variance, while the shaded regions indicate the variability of the values ( $\pm 3$  times the standard deviation) across the 5 runs.
In this section, we present some pivotal lemmas that illustrate some important properties of the data and neural network parameters at their random initialization and provide the update rule of coefficients from data-correlated decomposition.
Now turning to network initialization, the following lemma studies the inner product between a randomly initialized neural network neuron  $\mathbf{w}_{j,r}^{(0)}$  ( $j \in \{\pm 1\}$  and  $r \in [m]$ ) and the training data. The calculations characterize how the neural network at initialization randomly captures the information in training data.
Lemma B.1. Suppose that  $d = \Omega(\log(mn/\delta))$ ,  $m = \Omega(\log(1/\delta))$ . Then with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sigma_ {0} ^ {2} d / 2 \leq \| \mathbf {w} _ {j, r} ^ {(0)} \| _ {2} ^ {2} \leq 3 \sigma_ {0} ^ {2} d / 2, \\ | \langle \mathbf {w} _ {j, r} ^ {(0)}, \mathbf {x} _ {i} \rangle | \leq \sqrt {2 \log (8 m n / \delta)} \cdot \sigma_ {0} R _ {\max } \\ \end{array}
$$
Proof of Lemma B.1. First of all, the initial weights  $\mathbf{w}_{j,r}^{(0)}\sim \mathcal{N}(\mathbf{0},\sigma_0\mathbf{I})$ . By Bernstein's inequality, with probability at least  $1 - \delta /(4m)$  we have
$$
\left| \left\| \mathbf {w} _ {j, r} ^ {(0)} \right\| _ {2} ^ {2} - \sigma_ {0} ^ {2} d \right| = O \left(\sigma_ {0} ^ {2} \cdot \sqrt {d \log (8 m / \delta)}\right).
$$
Therefore, if we set appropriately  $d = \Omega(\log(m / \delta))$ , we have with probability at least  $1 - \delta / 2$ , for all  $j \in \{\pm 1\}$  and  $r \in [m]$ ,
$$
\sigma_ {0} ^ {2} d / 2 \leq \| \mathbf {w} _ {j, r} ^ {(0)} \| _ {2} ^ {2} \leq 3 \sigma_ {0} ^ {2} d / 2.
$$
Under definition, we have  $\| \mathbf{x}_i\| _2\leq R_{\max}$  for all  $i\in [n]$ . It is clear that for each  $j,r$ ,  $\langle \mathbf{w}_{j,r}^{(0)},\pmb {\mu}\rangle$  is a Gaussian random variable with mean zero and variance  $\sigma_0^2\| \mathbf{x}_i\| _2^2$ . Therefore, by Gaussian tail bound and union bound, with probability at least  $1 - \delta /2$
$$
| \langle \mathbf {w} _ {j, r} ^ {(0)}, \mathbf {x} _ {i} \rangle | \leq \sqrt {2 \log (8 m n / \delta)} \cdot \sigma_ {0} R _ {\max}.
$$
Next, we denote  $S_{i}^{(0)}$  as  $\{r \in [m] : \langle \mathbf{w}_{y_i,r}^{(0)}, \mathbf{x}_i \rangle > 0\}$ . We give a lower bound of  $|S_{i}^{(0)}|$  in the following two lemmas.
Lemma B.2. Suppose that  $\delta > 0$  and  $m \geq 50\log(2n/\delta)$ . Then with probability at least  $1 - \delta$ ,
$$
0. 4 m \leq | S _ {i} ^ {(0)} | \leq 0. 6 m, \forall i \in [ n ].
$$
Proof of Lemma B.2. Note that  $|S_i^{(0)}| = \sum_{r=1}^{m} \mathbb{1}[\langle \mathbf{w}_{y_i, r}^{(0)}, \mathbf{x}_i \rangle > 0]$  and  $P(\langle \mathbf{w}_{y_i, r}^{(0)}, \mathbf{x}_i \rangle > 0) = 1/2$ , then by Hoeffding's inequality, with probability at least  $1 - \delta/n$ , we have
$$
\left| \frac {| S _ {i} ^ {(0)} |}{m} - \frac {1}{2} \right| \leq \sqrt {\frac {\log (2 n / \delta)}{2 m}}.
$$
Therefore, as long as  $m \geq 50\log(2n/\delta)$ , by applying union bound, with probability at least  $1 - \delta$ , we have
Now we give the update rule of coefficients from data-correlated decomposition. We will begin by analyzing the coefficients in the data-correlated decomposition in Definition 5.1. The following lemma presents an iterative expression for the coefficients.
Lemma B.3. (Restatement of Lemma 5.2) The coefficients  $\overline{\rho}_{j,r,i}^{(t)},\underline{\rho}_{j,r,i}^{(t)}$  defined in Definition 5.1 satisfy the following iterative equations:
$$
\begin{array}{l} \overline {{\rho}} _ {j, r, i} ^ {(0)}, \underline {{\rho}} _ {j, r, i} ^ {(0)} = 0, \\ \overline {{\rho}} _ {j, r, i} ^ {(t + 1)} = \overline {{\rho}} _ {j, r, i} ^ {(t)} - \frac {\eta}{n m} \cdot \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = j), \\ \underline {{\rho}} _ {j, r, i} ^ {(t + 1)} = \underline {{\rho}} _ {j, r, i} ^ {(t)} + \frac {\eta}{n m} \cdot \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = - j), \\ \end{array}
$$
Proof of Lemma B.3. First, we iterate the gradient descent update rule (3.2)  $t$  times and get
$$
\mathbf {w} _ {j, r} ^ {(t + 1)} = \mathbf {w} _ {j, r} ^ {(0)} - \frac {\eta}{n m} \sum_ {s = 0} ^ {t} \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime (s)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(s)}, \mathbf {x} _ {i} \rangle) \cdot j y _ {i} \mathbf {x} _ {i}.
$$
According to the definition of  $\rho_{j,r,i}^{(t)}$ , we have
$$
\mathbf {w} _ {j, r} ^ {(t)} = \mathbf {w} _ {j, r} ^ {(0)} + \sum_ {i = 1} ^ {n} \rho_ {j, r, i} ^ {(t)} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {- 2} \cdot \mathbf {x} _ {i}.
$$
Therefore, we have the unique representation
$$
\rho_ {j, r, i} ^ {(t)} = - \frac {\eta}{n m} \sum_ {s = 0} ^ {t} \ell_ {i} ^ {\prime (s)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(s)}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \cdot j y _ {i}.
$$
Now with the notation  $\overline{\rho}_{j,r,i}^{(t)}\coloneqq \rho_{j,r,i}^{(t)}\mathbb{1}(\rho_{j,r,i}^{(t)}\geq 0)$ ,  $\underline{\rho}_{j,r,i}^{(t)}\coloneqq \rho_{j,r,i}^{(t)}\mathbb{1}(\rho_{j,r,i}^{(t)}\leq 0)$  and the fact  $\ell_i^{\prime (s)} < 0$ , we get
$$
\bar {\rho} _ {j, r, i} ^ {(t)} = - \frac {\eta}{n m} \sum_ {s = 0} ^ {t} \ell_ {i} ^ {\prime (s)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(s)}, \mathbf {x} _ {i} \right\rangle\right) \cdot \left\| \mathbf {x} _ {i} \right\| _ {2} ^ {2} \cdot \mathbb {1} \left(y _ {i} = j\right), \tag {B.1}
$$
$$
\underline {{\rho}} _ {j, r, i} ^ {(t)} = \frac {\eta}{n m} \sum_ {s = 0} ^ {t} \ell_ {i} ^ {\prime (s)} \cdot \sigma^ {\prime} \left(\left\langle \mathbf {w} _ {j, r} ^ {(s)}, \mathbf {x} _ {i} \right\rangle\right) \cdot \left\| \mathbf {x} _ {i} \right\| _ {2} ^ {2} \cdot \mathbb {1} \left(y _ {i} = - j\right). \tag {B.2}
$$
Writing out the iterative versions of (B.1) and (B.2) completes the proof.
C Coefficient Analysis of Leaky ReLU
In this section, we establish a series of results on the data-correlated decomposition for two-layer leaky ReLU network defined as
$$
\begin{array}{l} f (\mathbf {W} ^ {(t)}, \mathbf {x}) = F _ {+ 1} (\mathbf {W} _ {+ 1} ^ {(t)}, \mathbf {x}) - F _ {- 1} (\mathbf {W} _ {- 1} ^ {(t)}, \mathbf {x}) \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {+ 1, r} ^ {(t)}, \mathbf {x} \right\rangle\right) - \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {- 1, r} ^ {(t)}, \mathbf {x} \right\rangle\right), \tag {C.1} \\ \sigma (z) = \max  \{\gamma z, z \}, \gamma \in (0, 1). \\ \end{array}
$$
The results in Section C, D and G are based on Lemma B.1, which hold with high probability. Denote by  $\mathcal{E}_{\mathrm{prelim}}$  the event that Lemma B.1 in Section B holds (for a given  $\delta$ , we see  $\mathbb{P}(\mathcal{E}_{\mathrm{prelim}}) \geq 1 - \delta$ ). For simplicity and clarity, we state all the results in Section C, D and G conditional on  $\mathcal{E}_{\mathrm{prelim}}$ .
Denote  $\beta = \max_{i,j,r}\{|\langle \mathbf{w}_{j,r}^{(0)},\mathbf{x}_i\rangle |\}$ ,  $R_{\mathrm{max}} = \max_{i\in [n]}\| \mathbf{x}_i\| _2$ ,  $R_{\mathrm{min}} = \min_{i\in [n]}\| \mathbf{x}_i\| _2$ ,  $p = \max_{i\neq k}|\langle \mathbf{x}_i,\mathbf{x}_k\rangle |$  and suppose  $R = R_{\mathrm{max}} / R_{\mathrm{min}}$  is at most an absolute constant. Here we list the exact conditions for  $\eta, \sigma_0, R_{\mathrm{min}}, R_{\mathrm{max}}, p$  required by the proofs in this section.
$$
\sigma_ {0} \leq \gamma \left(C R _ {\max } \sqrt {\log (m n / \delta)}\right) ^ {- 1}, \tag {C.2}
$$
$$
\eta \leq \left(C R _ {\max } ^ {2} / n m\right) ^ {- 1}, \tag {C.3}
$$
$$
R _ {\min } ^ {2} \geq C r ^ {- 4} R ^ {2} n p, \tag {C.4}
$$
where  $C$  is a large enough constant. By Lemma B.1, we can upper bound  $\beta$  by  $2\sqrt{\log(12mn / \delta)}\cdot \sigma_0R_{\mathrm{max}}$ . Then, by (C.2) and (C.4), it is straightforward to verify the following inequality:
$$
\beta \leq c \gamma , \tag {C.5}
$$
$$
\gamma^ {- 4} R _ {\min } ^ {- 2} n p \leq c, \tag {C.6}
$$
$$
\gamma^ {- 4} R _ {\min } ^ {- 2} R ^ {2} n p \leq c, \tag {C.7}
$$
where  $c$  is a sufficiently small constant.
Suppose the conditions listed in (C.2) and (C.4) hold, we claim that for any  $t \geq 0$  the following property holds.
Lemma C.1. Under the same conditions as Theorem 4.1, for any  $t \geq 0$ , we have that
$$
\sum_ {r = 1} ^ {m} \left| \rho_ {j, r, i} ^ {(t)} \right| \geq c _ {1} \gamma^ {2} \sum_ {r = 1} ^ {m} \left| \rho_ {j ^ {\prime}, r, i ^ {\prime}} ^ {(t)} \right|, \forall j, j ^ {\prime} \in \{\pm 1 \}, \forall i, i ^ {\prime} \in [ n ], \tag {C.8}
$$
where  $c_{1}$  is a constant.
To prove Lemma C.1, we divide it into two lemmas, each addressing a specific case:  $0 \leq t \leq T_{1}$  (Lemma C.2) when the logit  $|\ell_i^{(t)}| = \Theta(1)$ , and  $t \geq T_{1}$  (Lemma C.3) when the logit  $|\ell_i^{(t)}|$  is smaller than constant order. Here,  $T_{1} = C' \eta^{-1} nm R_{\max}^{-2}$ , and  $C'$  is a constant. For each case, we apply different techniques to establish the proof.
Lemma C.2  $(0\leq t\leq T_{1})$  . Under the same conditions as Theorem 4.1, for any  $0\leq t\leq T_1 =$ $C^\prime \eta^{-1}nmR_{\mathrm{max}}^{-2}$  , where  $C^\prime$  is a constant, we have that
$$
\left| \rho_ {j, r, i} ^ {(t)} \right| \geq c _ {2} \gamma \left| \rho_ {j ^ {\prime}, r ^ {\prime}, i ^ {\prime}} ^ {(t)} \right|, \forall j, j ^ {\prime} \in \{\pm 1 \}, \forall r, r ^ {\prime} \in [ m ], \forall i, i ^ {\prime} \in [ n ], \tag {C.9}
$$
where  $c_{2}$  is a constant.
Proof of Lemma C.2. In this lemma, we first show that (C.8) holds for  $t \leq T_1 = C' \eta^{-1} nm R_{\max}^{-2}$  where  $C' = \Theta(1)$  is a constant. Recall from Lemma B.3 that
$$
\overline {{\rho}} _ {j, r, i} ^ {(t + 1)} = \overline {{\rho}} _ {j, r, i} ^ {(t)} - \frac {\eta}{n m} \cdot \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = j),
$$
$$
\underline {{\rho}} _ {j, r, i} ^ {(t + 1)} = \underline {{\rho}} _ {j, r, i} ^ {(t)} + \frac {\eta}{n m} \cdot \ell_ {i} ^ {\prime (t)} \cdot \sigma^ {\prime} (\langle \mathbf {w} _ {j, r} ^ {(t)}, \mathbf {x} _ {i} \rangle) \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \cdot \mathbb {1} (y _ {i} = - j),
$$
we can get
$$
\bar {\rho} _ {- y _ {i}, r, i} ^ {(t)}, \underline {{\rho}} _ {y _ {i}, r, i} ^ {(t)} = 0, \tag {C.10}
$$
$$
\bar {\rho} _ {y _ {i}, r, i} ^ {(t + 1)} \leq \bar {\rho} _ {y _ {i}, r, i} ^ {(t)} + \frac {\eta}{n m} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \leq \bar {\rho} _ {y _ {i}, r, i} ^ {(t)} + \frac {\eta R _ {\max } ^ {2}}{n m}, \tag {C.11}
$$
$$
\left| \underline {{\rho}} _ {- y _ {i}, r, i} ^ {(t + 1)} \right| \leq \left| \underline {{\rho}} _ {- y _ {i}, r, i} ^ {(t)} \right| + \frac {\eta}{n m} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \leq \left| \underline {{\rho}} _ {- y _ {i}, r, i} ^ {(t)} \right| + \frac {\eta R _ {\max } ^ {2}}{n m}. \tag {C.12}
$$
Therefore, we have  $\max_{j,r,i}\{\overline{\rho}_{j,r,i}^{(t)},|\underline{\rho}_{j,r,i}^{(t)}|\} = O(1)$  for any  $t\leq T_1$  and hence  $\max_i\{F_{+1}(\mathbf{W}_{+1}^{(t)},\mathbf{x}_i),F_{-1}(\mathbf{W}_{-1}^{(t)},\mathbf{x}_i)\} = O(1)$  for any  $t\leq T_1$ . Thus there exists a positive constant  $\widetilde{c}$  such that  $|\ell_i^{\prime (t)}|\geq \widetilde{c}$  for any  $t\leq T_1$ . And it follows for any  $j\in \{\pm 1\},r\in [m],i\in [n]$  that
$$
| \rho_ {j, r, i} ^ {(t + 1)} | \geq | \rho_ {j, r, i} ^ {(t)} | + \frac {\gamma \eta}{n m} \cdot | \ell_ {i} ^ {\prime (t)} | \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2} \geq | \rho_ {j, r, i} ^ {(t)} | + \frac {\widetilde {c} \gamma \eta}{n m} \cdot \| \mathbf {x} _ {i} \| _ {2} ^ {2}, \forall 0 \leq t \leq T _ {1},
$$
$$
\left| \rho_ {j, r, i} ^ {(t)} \right| \geq \frac {\widetilde {c} \gamma \eta t}{n m} \cdot \left\| \mathbf {x} _ {i} \right\| _ {2} ^ {2} \geq \frac {\widetilde {c} \gamma \eta R _ {\min } ^ {2} t}{n m}, \forall 0 \leq t \leq T _ {1}. \tag {C.13}
$$
On the other hand, by (C.10), (C.11) and (C.12), we have for any  $j' \in \{\pm 1\}$ ,  $r' \in [m]$ ,  $i' \in [n]$  that
$$
\left| \rho_ {j ^ {\prime}, r ^ {\prime}, i ^ {\prime}} ^ {(t)} \right| \leq \frac {\eta R _ {\max } ^ {2} t}{n m}, \forall 0 \leq t \leq T _ {1}. \tag {C.14}
$$
Dividing (C.14) by (C.13), we can get for any  $j, j' \in \{\pm 1\}, r, r' \in [m], i, i' \in [n]$  that
$$
| \rho_ {j, r, i} ^ {(t)} | \geq \frac {\widetilde {c} \gamma R _ {\mathrm {m i n}} ^ {2}}{R _ {\mathrm {m a x}} ^ {2}} | \rho_ {j ^ {\prime}, r ^ {\prime}, i ^ {\prime}} ^ {(t)} |,
$$
which indicates that the first bullet holds for time  $t \leq T_1$  as long as  $c_2 \leq \widetilde{c} R_{\min}^2 R_{\max}^{-2}$ .
Lemma C.3 ( $t \geq T_1$ ). Let  $T_1$  be defined in Lemma C.2. Under the same conditions as Theorem 4.1, for any  $t \geq T_1$ , we have that
$$
\sum_ {r = 1} ^ {m} \left| \rho_ {j, r, i} ^ {(t)} \right| \geq c _ {3} \gamma^ {2} \sum_ {r = 1} ^ {m} \left| \rho_ {j ^ {\prime}, r, i ^ {\prime}} ^ {(t)} \right|, \forall j, j ^ {\prime} \in \{\pm 1 \}, \forall i, i ^ {\prime} \in [ n ], \tag {C.15}
$$
where  $c_{3} = \Theta(1)$  is a constant. Moreover, we also have the following increasing rate estimation of  $|\rho_{y_i,r,i}^{(t)}|, |\rho_{-y_i,r,i}^{(t)}|$ :
$\frac{1}{m}\sum_{r = 1}^{m}\rho_{y_i,r,i}^{(t)}\leq c_4^{-1}\log \left(1 + \frac{\eta\|\mathbf{x}_i\|_2^2c_4e^{2\beta}}{nm}\cdot t\right),$
$\frac{1}{m}\sum_{r = 1}^{m}|\rho_{-y_i,r,i}^{(t)}|\leq c_5^{-1}\gamma^{-1}\log \left(1 + \frac{\gamma\eta\|\mathbf{x}_i\|_2^2c_5e^{2\beta}}{nm}\cdot t\right),$
$\frac{1}{m}\sum_{r = 1}^{m}\rho_{y_i,r,i}^{(t)}\geq c_6^{-1}\log \left(1 + \frac{\gamma\eta\|\mathbf{x}_i\|_2^2c_6e^{-(\gamma + 1)\beta}}{nm}\cdot t\right),$
$\frac{1}{m}\sum_{r = 1}^{m}|\rho_{-y_i,r,i}^{(t)}|\geq c_6^{-1}\gamma \log \left(1 + \frac{\eta\|\mathbf{x}_i\|_2^2c_6e^{-(\gamma + 1)\beta}}{nm}\cdot t\right),$
where  $c_{4}, c_{5}, c_{6}$  are constants.
Proof of Lemma C.3. We prove this lemma by induction. By Lemma C.2, we know that (C.15) holds for time  $t = T_1$  as long as  $c_3 \leq c_2$ . Suppose that there exists  $\widetilde{t} > T_1$  such that (C.15) holds for all time  $0 \leq t \leq \widetilde{t} - 1$ . We aim to prove that they also hold for  $t = \widetilde{t}$ . For any  $0 \leq t \leq \widetilde{t} - 1$ , we have
$$
\begin{array}{l} F _ {y _ {i}} \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) = \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {(t)}, \mathbf {x} _ {i} \right\rangle\right) \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \langle \mathbf {w} _ {y _ {i}, r} ^ {(t)}, \mathbf {x} _ {i} \rangle \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {(0)}, \mathbf {x} _ {i} \right\rangle + \sum_ {i ^ {\prime} = 1} ^ {n} \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)} \| \mathbf {x} _ {i ^ {\prime}} \| _ {2} ^ {- 2} \cdot \left\langle \mathbf {x} _ {i ^ {\prime}}, \mathbf {x} _ {i} \right\rangle\right) \tag {C.16} \\ \geq \frac {1}{m} \sum_ {r = 1} ^ {m} \left(\rho_ {y _ {i}, r, i} ^ {(t)} - \sum_ {i ^ {\prime} \neq i} | \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)} | R _ {\min } ^ {- 2} p\right) - \beta \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} - \sum_ {i ^ {\prime} \neq i} \left(\frac {1}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)}\right) R _ {\mathrm {m i n}} ^ {- 2} p - \beta \\ \geq \frac {1 - \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\mathrm {m i n}} ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} - \beta , \\ \end{array}
$$
where the first inequality is by  $\sigma (z)\geq z$ ; the second equality is by (5.1); the third inequality is by triangle inequality and the definition of  $\beta ,\underline{p},R_{min}$ ; the fourth inequality is by the induction hypothesis (C.15). Besides, for any  $0\leq t\leq \tilde{t} -1$  , we also have the following upper bound of
$$
F _ {y _ {i}} \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right):
$$
$$
\begin{array}{l} F _ {y _ {i}} \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) = \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {(t)}, \mathbf {x} _ {i} \right\rangle\right) \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\left\langle \mathbf {w} _ {y _ {i}, r} ^ {(0)}, \mathbf {x} _ {i} \right\rangle + \sum_ {i ^ {\prime} = 1} ^ {n} \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)} \| \mathbf {x} _ {i ^ {\prime}} \| _ {2} ^ {- 2} \cdot \left\langle \mathbf {x} _ {i ^ {\prime}}, \mathbf {x} _ {i} \right\rangle\right) \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\rho_ {y _ {i}, r, i} ^ {(t)} + \sum_ {i ^ {\prime} \neq i} \left| \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)} \right| R _ {\min } ^ {- 2} p + \beta\right) \tag {C.17} \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \left(\rho_ {y _ {i}, r, i} ^ {(t)} + \sum_ {i ^ {\prime} \neq i} \left| \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)} \right| R _ {\min } ^ {- 2} p + \beta\right) \\ = \frac {1}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} + \sum_ {i ^ {\prime} \neq i} \left(\frac {1}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i ^ {\prime}} ^ {(t)}\right) R _ {\min } ^ {- 2} p + \beta \\ \leq \frac {1 + \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\mathrm {m i n}} ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} + \beta , \\ \end{array}
$$
where the first inequality is by triangle inequality and the definition of  $\beta, p, R_{min}$ ; the second inequality is by the induction hypothesis (C.15). On the other hand, for any  $0 \leq t \leq t$ , we can give following upper and lower bounds for  $F_{-y_i}(\mathbf{W}_{-y_i}^{(t)}, \mathbf{x}_i)$  by applying similar arguments like (C.16) and (C.17):
$$
\begin{array}{l} F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) \geq \frac {\gamma}{m} \sum_ {r = 1} ^ {m} \langle \mathbf {w} _ {- y _ {i}, r} ^ {(t)}, \mathbf {x} _ {i} \rangle \\ \geq \frac {\gamma}{m} \sum_ {r = 1} ^ {m} \left(\rho_ {- y _ {i}, r, i} ^ {(t)} - \sum_ {i ^ {\prime} \neq i} | \rho_ {- y _ {i}, r, i ^ {\prime}} ^ {(t)} | R _ {\mathrm {m i n}} ^ {- 2} p - \beta\right), \\ \geq \frac {\gamma \left(1 + \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\operatorname* {m i n}} ^ {- 2} p n\right)}{m} \sum_ {r = 1} ^ {m} \rho_ {- y _ {i}, r, i} ^ {(t)} - \gamma \beta , \tag {C.18} \\ \end{array}
$$
$$
\begin{array}{l} F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \sigma \left(\rho_ {- y _ {i}, r, i} ^ {(t)} + \sum_ {i ^ {\prime} \neq i} | \rho_ {- y _ {i}, r, i ^ {\prime}} ^ {(t)} | R _ {\min } ^ {- 2} p + \beta\right) \\ \leq \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \sigma \left(\rho_ {- y _ {i}, r, i} ^ {(t)}\right) + \sigma \left(\sum_ {i ^ {\prime} \neq i} | \rho_ {- y _ {i}, r, i ^ {\prime}} ^ {(t)} | R _ {\min } ^ {- 2} p\right) + \sigma (\beta) \right] \\ = \frac {\gamma}{m} \sum_ {r = 1} ^ {m} \rho_ {- y _ {i}, r, i} ^ {(t)} + \sum_ {i ^ {\prime} \neq i} \left(\frac {1}{m} \sum_ {r = 1} ^ {m} | \rho_ {- y _ {i}, r, i} ^ {(t)} |\right) + \beta \\ = \frac {\gamma \left(1 - \gamma^ {- 3} c _ {3} ^ {- 1} R _ {\operatorname* {m i n}} ^ {- 2} p n\right)}{m} \sum_ {r = 1} ^ {m} \rho_ {- y _ {i}, r, i} ^ {(t)} + \beta , \tag {C.19} \\ \end{array}
$$
where the second inequality is by a property of leaky ReLU function that  $\sigma(a + b) \leq \sigma(a) + \sigma(b), \forall a, b \in \mathbb{R}$ .
Next, we can bound  $|\ell_i^{\prime (t)}|$  for  $0\leq t\leq \widetilde{t} -1$
$$
\begin{array}{l} | \ell_ {i} ^ {\prime (t)} | \\ = \frac {1}{1 + \exp \{F _ {y _ {i}} (\mathbf {W} _ {y _ {i}} ^ {(t)} , \mathbf {x} _ {i}) - F _ {- y _ {i}} (\mathbf {W} _ {- y _ {i}} ^ {(t)} , \mathbf {x} _ {i}) \}} \\ \leq \exp \left\{- F _ {y _ {i}} \left(\mathbf {W} _ {y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) + F _ {- y _ {i}} \left(\mathbf {W} _ {- y _ {i}} ^ {(t)}, \mathbf {x} _ {i}\right) \right\} \\ \leq \exp \left\{- \frac {1 - \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\min } ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} + \frac {\gamma (1 - \gamma^ {- 3} c _ {3} ^ {- 1} R _ {\min } ^ {- 2} p n)}{m} \sum_ {r = 1} ^ {m} \rho_ {- y _ {i}, r, i} ^ {(t)} + 2 \beta \right\}, \tag {C.20} \\ \end{array}
$$
where the second inequality is by (C.16) and (C.19). And
$$
\begin{array}{l} | \ell_ {i} ^ {\prime (t)} | \\ = \frac {1}{1 + \exp \{F _ {y _ {i}} (\mathbf {W} _ {y _ {i}} ^ {(t)} , \mathbf {x} _ {i}) - F _ {- y _ {i}} (\mathbf {W} _ {- y _ {i}} ^ {(t)} , \mathbf {x} _ {i}) \}} \\ \geq \frac {1}{1 + \exp \left\{\frac {1 + \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\operatorname* {m i n}} ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i} , r , i} ^ {(t)} - \frac {\gamma (1 + \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\operatorname* {m i n}} ^ {- 2} p n)}{m} \sum_ {r = 1} ^ {m} \rho_ {- y _ {i} , r , i} ^ {(t)} + (\gamma + 1) \beta \right\}} \\ \geq \frac {1}{2} \exp \left\{- \frac {1 + \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\min } ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} + \frac {\gamma (1 + \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\min } ^ {- 2} p n)}{m} \sum_ {r = 1} ^ {m} \rho_ {- y _ {i}, r, i} ^ {(t)} - (\gamma + 1) \beta \right\}, \tag {C.21} \\ \end{array}
$$
where the first inequality is by (C.17) and (C.18); the last inequality is by  $1 / (1 + \exp(z)) \geq \exp(-z)/2$  if  $z \geq 0$ . By (C.20), we can get for  $0 \leq t \leq \widetilde{t} - 1$  that
$$
\left| \ell_ {i} ^ {\prime (t)} \right| \leq \exp \left\{- \frac {1 - \gamma^ {- 2} c _ {3} ^ {- 1} R _ {\operatorname* {m i n}} ^ {- 2} p n}{m} \sum_ {r = 1} ^ {m} \rho_ {y _ {i}, r, i} ^ {(t)} + 2 \beta \right\}, \tag {C.22}
$$