SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training data. We conduct experiments on a synthetic single data point  $(x, y)$ , where  $x = e_1 = (1, 0, \ldots, 0) \in \mathbb{R}^d$  and  $y \in \mathbb{R}$ . Throughout the experiments in this subsection, we keep the data dimension fixed at  $d = 10$ , the data label at  $y = 1$ , and the step size at  $\eta = 0.01$ .
The effect of initialization scale. We investigate the impact of the initialization scale  $\alpha$  while keeping other hyperparameters fixed. We vary the initialization scale across  $\{0.5, 1.0, 2.0, 4.0\}$ . Specifically, in Figure 5, we train a 3-layer fully connected neural network with tanh activation. We observe that smaller initialization scales ( $\alpha = 0.5$ , 1.0) lead to trajectories in the gradient flow regime, while larger initialization scales ( $\alpha = 2.0$ , 4.0) result in trajectories in the EoS regime. This behavior is primarily due to the initial value of  $q$  being smaller than 1 for larger initialization scales, which causes the trajectory to fall into the EoS regime.
The effect of network width. We investigate the impact of network width on the trajectory alignment phenomenon. While keeping other hyperparameters fixed, we control the width  $m$  with values of  $\{64, 128, 256, 512\}$ . In Figure 6, we train 3-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 4$ . Additionally, in Figure 7, we examine the same setting but with different depth, training 10-layer fully connected neural networks.
It is commonly observed that the alignment trend becomes more pronounced as the network width increases. In Figures 6 and 7, all trajectories are in the EoS regime. However, narrower networks ( $m = 64$ , 128) do not exhibit the trajectory alignment phenomenon, while wider networks ( $m = 256$ , 512) clearly demonstrate this behavior. These results indicate that network width plays a significant role in the trajectory alignment property of GD.
The effect of loss and activation functions. The trajectory alignment phenomenon is consistently observed in various settings, including those with square root loss and different activation functions. In Figure 8, we investigate a 3-layer fully connected neural network with a width of  $m = 256$  and an initialization scale of  $\alpha = 2$ . We explore different activation functions, including tanh, ELU, and linear, and consider both log-cosh loss and square-root loss. Across all these settings, we observe the trajectory alignment phenomenon, where the GD trajectories align on a curve  $q = \frac{\ell'(p)}{p}$ .
Figure 6: GD trajectories (1000 iterations) of 3-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 4$ . The plots depict the trajectories for different network widths:  $m = 64, 128, 256, 512$ . Narrower networks ( $m = 64, 128$ ) do not exhibit trajectory alignment, while wider networks ( $m = 256, 512$ ) clearly demonstrate the trajectory alignment phenomenon.
(a)  $m = 64$
(b)  $m = 128$
(c)  $m = 256$
(d)  $m = 512$
(c)  $m = 256$
Figure 7: GD trajectories (1000 iterations) of 10-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 4$ . The plots depict the trajectories for different network widths:  $m = 64, 128, 256, 512$ . Similar to Figure 6, narrower networks ( $m = 64, 128$ ) do not exhibit trajectory alignment, while wider networks ( $m = 256, 512$ ) clearly demonstrate the trajectory alignment phenomenon.
(d)  $m = 512$
A.3 Training on multiple synthetic data Points
Training data. We consider training on a synthetic dataset consisting of  $n$  data points, denoted as  $(\pmb{x}_i, y_i)_{i=1}^n$ . The input vectors  $\pmb{x}_i$  are sampled from a standard Gaussian distribution  $\mathcal{N}(0, I)$ , where  $\pmb{x}_i \in \mathbb{R}^d$ , and the corresponding target values  $y_i$  are sampled from a Gaussian distribution  $\mathcal{N}(0, 1)$ , where  $y_i \in \mathbb{R}$ . Throughout our experiments in this subsection, we use a fixed data dimension of  $d = 10$  and a step size of  $\eta = 0.01$ .
The effect of function  $P$ . To investigate the impact of different choices of the function  $P$ , we train a 3-layer fully connected neural network with tanh activation. The network has a width of  $m = 256$  and is initialized with a scale of  $\alpha = 4$ . The training is performed on a dataset consisting of  $n = 10$  data points. Figure 9 displays the trajectories of GD trajectories under the generalized canonical reparameterization defined in Eq. (5) for various choices of the function  $P$ . These choices include the mean,  $\ell_1$  norm,  $\ell_2$  norm, and  $\ell_{\infty}$  norm.
We observe that GD trajectories exhibit alignment behavior across different choices of the function  $P$ . Notably, when  $P$  is selected as the mean, the trajectories align on the curve  $q = \frac{\ell'(p)}{p}$ . However, when  $P$  is based on vector norms, the alignment occurs on different curves. The precise characterization of these curves remains as an interesting open question for further exploration.
The effect of the number of data points. We examine how the size of the training dataset, denoted by  $n$ , influences the trajectory alignment behavior of GD. While keeping other hyperparameters constant, we vary  $n$  with values  $\{2, 4, 8, 16, 32, 64, 128, 512, 1024\}$ . In Figure 10, we train a 3-layer fully connected neural network with tanh activation, a width of  $m = 256$ , and an initialization scale
(a) log-cosh, tanh
(d) square-root, tanh
(b) log-cosh, ELU
(e) square-root, ELU
(c) log-cosh, linear
(f) square-root, linear
Figure 8: GD trajectories (1000 iterations) of 3-layer fully connected neural networks with different loss and activation functions. The networks have a width of  $m = 256$  and an initialization scale of  $\alpha = 2$ . The plots illustrate the trajectories for various combinations of activation functions (tanh, ELU, and linear) and loss functions (log-cosh and square-root loss). In all these settings, the trajectory alignment phenomenon is observed, where GD trajectories align on a curve  $q = \frac{\ell'(p)}{p}$ . Top row: trajectories with log-cosh loss. From left to right: tanh, ELU, and linear activations. Bottom row: trajectories with square-root loss. From left to right: tanh, ELU, and linear activations.
(a)  $P(\pmb {z}) = \frac{1}{n}\sum_{i = 1}^{n}z_{i}$
Figure 9: GD trajectories (500 iterations) under the generalized canonical reparameterization (Eq. (5)) for different choices of the function  $P$ . The plots depict the training of a 3-layer fully connected neural network with tanh activation, a width of  $m = 256$ , and an initialization scale of  $\alpha = 4$ . Each plot corresponds to a different parameterization, including the mean,  $\ell_1$  norm,  $\ell_2$  norm, and  $\ell_{\infty}$  norm. GD trajectories commonly exhibit alignment behavior, with the mean parameterization aligning on the curve  $q = \frac{\ell'(p)}{p}$ .
(b)  $P(z) = \| z\| _1$
(c)  $P(\pmb {z}) = \| \pmb {z}\| _2$
(d)  $P(z) = \| z\|_{\infty}$
of  $\alpha = 4$ . Additionally, in Figure 11, we investigate the same setting but with a different activation function, training ELU-activated fully connected neural networks. The GD trajectories are plotted under the generalized canonical reparameterization using the mean function  $P(z) = \frac{1}{n}\sum_{i=1}^{n}z_{i}$ .
We observe a consistent trajectory alignment phenomenon across different choices of the number of data points. Interestingly, for small values of  $n$ , the trajectories clearly align on the curve  $q = \frac{\ell'(p)}{p}$ . However, as the number of data points  $n$  increases, it seems that the trajectories no longer align on this curve but different "narrower" curves. Understanding the underlying reasons for this phenomenon poses an intriguing open question.
(a)  $n = 2$
(c)  $n = 8$
(d)  $n = 16$
(e)  $n = 32$
(f)  $n = 64$
(g)  $n = 128$
(h)  $n = 256$
(i)  $n = 512$
(j)  $n = 1024$
Figure 10: GD trajectories (500 iterations) under the generalized canonical reparameterization (Eq. (5)) for different choices of the number of data points  $n$ . The plots depict the training of a 3-layer fully connected neural network with tanh activation, a width of  $m = 256$ , and an initialization scale of  $\alpha = 4$ . The function  $P$  is chosen to be the mean  $P(z) = \frac{1}{n} \sum_{i=1}^{n} z_i$ . The trajectories exhibit alignment behavior, where the curves followed by the trajectories change depending on the value of  $n$ . For small values of  $n$ , the trajectories align on the curve  $q = \frac{\ell'(p)}{p}$ , while for large values of  $n$ , they align on a distinct curve.
(a)  $n = 2$
(d)  $n = 16$
(b)  $n = 4$
(e)  $n = 32$
(c)  $n = 8$
(f)  $n = 64$
(h)  $n = 256$
(j)  $n = 1024$
Figure 11: GD trajectories (500 iterations) under the generalized canonical reparameterization (Eq. (5)) for different choices of the number of data points  $n$ . The plots depict the training of a 3-layer fully connected neural network with ELU activation, a width of  $m = 256$ , and an initialization scale of  $\alpha = 4$ . The function  $P$  is chosen to be the mean  $P(z) = \frac{1}{n} \sum_{i=1}^{n} z_i$ . The trajectories exhibit alignment behavior, where the curves followed by the trajectories change depending on the value of  $n$ . For small values of  $n$ , the trajectories align on the curve  $q = \frac{\ell'(p)}{p}$ , while for large values of  $n$ , they align on a distinct curve.
A.4 Training on real-world dataset
Training data. In this subsection, we investigate a binary classification problem using a subset of the CIFAR-10 image classification dataset. Our dataset consists of 50 samples, with 25 samples from class 0 (airplane) and 25 samples from class 1 (automobile). We assign a label of  $+1$  to samples from class 0 and a label of  $-1$  to samples from class 1. This dataset was used in the experimental setup by Zhu et al. (2023).
Architectures. In Figure 4, we examine the training of two types of network architectures: (top row) fully-connected tanh network and (bottom row) convolutional tanh network. The PyTorch code for the fully-connected tanh network is provided as follows:
nnSequential( nnGBTen(start_dim=1,end_dim=-1), nn.Linear(3072,500,bias=True) nn.Tanh(), nn.Linear(500,500,bias=True), nn.Tanh(), nn.Linear(500,1,bias=True)
Similarly, the PyTorch code for the convolutional tanh network is as follows:
nnSequential( nnGBTen(start_dim=1，end_dim=-1), nn.Unflatten(dim=1，unflattened_size=(3，32，32))， nn.Conv2d(3,500,kernel_size=(3，3)，stride=(1，1)，padding=(1，1)，bias=True) nn.Tanh(), nn.Conv2d(500，500，kernel_size=(3，3)，stride=(1，1)，padding=(1，1)，bias=True) nn.Tanh(), nn.Flatten(start_dim=1，end_dim=-1), nn.Linear(512000，1，bias=True)
Note that we consider networks with bias. We use a step size of  $\eta = 0.01$  for the fully-connected network and  $\eta = 0.001$  for the CNN. The default PyTorch initialization (Paszke et al., 2019) is applied to all these networks.
In this subsection, we further explore the (reparameterized) GD trajectories of fully-connected networks with different activation functions, network widths, and the choice of function  $P$  in (5).
The effect of function  $P$ . We investigate the impact of different choices of the function  $P$  on the GD trajectories. We train a 3-layer fully connected neural network with ELU activation, a width of  $m = 256$ , and an initialization scale of  $\alpha = 1$ . Figure 12 illustrates the GD trajectories under the generalized canonical reparameterization defined in Eq. (5) for various choices of the function  $P$ , including the mean,  $\ell_1$  norm,  $\ell_2$  norm, and  $\ell_{\infty}$  norm.
We observe that the GD trajectories exhibit alignment behavior, which is more pronounced when  $P$  is chosen to be the mean or  $\ell_1$  norm, but less evident for the  $\ell_{\infty}$  norm. Unlike in Figure 9, the trajectories do not align on the curve  $q = \frac{\ell'(p)}{p}$  when  $P$  is selected as the mean  $P(z) = \frac{1}{n}\sum_{i=1}^{n}z_i$ .
The effect of network width. We investigate how the width of the network influences the trajectory alignment phenomenon. We vary the width  $m$  using values from \{64, 128, 256, 512\} while keeping other hyperparameters constant. In Figure 13, we train 3-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 1$ . Similarly, in Figure 14, we conduct experiments using the same configuration but with ELU activation, training ELU-activated fully connected neural networks.
Consistent with the observations from Figures 6 and 7 in the single data point setting, we commonly find that as the network width increases, the alignment trend becomes more pronounced. In both Figure 13 and Figure 14, all trajectories fall within the EoS regime. However, narrower networks
(a)  $P(\pmb {z}) = \frac{1}{n}\sum_{i = 1}^{n}z_{i}$
Figure 12: GD trajectories (1000 iterations) under the generalized canonical reparameterization (Eq. (5)) for different choices of the function  $P$  trained on a small subset of CIFAR-10 image dataset. The plots depict the training of a 3-layer fully connected neural network with ELU activation, a width of  $m = 256$ , and an initialization scale of  $\alpha = 1$ . The function  $P$  is varied to be the mean,  $\ell_1$  norm,  $\ell_2$  norm, and  $\ell_{\infty}$  norm. The alignment behavior of the trajectories is more prominent when  $P$  is chosen as the mean or  $\ell_1$  norm, but less evident for the  $\ell_{\infty}$  norm.
(c)  $P(\pmb {z}) = \| \pmb {z}\| _2$
(d)  $P(z) = \| z\|_{\infty}$
(c)  $m = 256$
(d)  $m = 512$
Figure 13: GD trajectories (1000 iterations) under the generalized canonical reparameterization (Eq. (5)) for different network widths trained on a small subset of CIFAR-10 image dataset. The plots depict the training of 3-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 1$ . The trajectories are shown for network widths of  $m = 64$ ,  $m = 128$ ,  $m = 256$ , and  $m = 512$ .
(a)  $m = 64$
Figure 14: GD trajectories (1000 iterations) under the generalized canonical reparameterization (Eq. (5)) for different network widths trained on a small subset of CIFAR-10 image dataset. The plots depict the training of 3-layer fully connected neural networks with ELU activation and an initialization scale of  $\alpha = 1$ . The trajectories are shown for network widths of  $m = 64$ ,  $m = 128$ ,  $m = 256$ , and  $m = 512$ .
$(m = 64)$  show less evidence of the trajectory alignment phenomenon, while wider networks  $(m = 256, 512)$  clearly demonstrate this behavior. These findings emphasize the significant impact of network width on the trajectory alignment property of GD.
The effect of data label. In our previous experiments, we assigned labels of  $+1$  and  $-1$  to the dataset. However, in this particular experiment, we investigate the training process on a dataset with
zero labels. This means that all samples in the dataset are labeled as zero ( $y_{i} = 0$  for all  $1 \leq i \leq n$ ). Figure 15 visualizes the training of 3-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 1$ . The network widths  $m$  are varied from  $\{256, 512, 1024\}$ . Interestingly, the GD trajectories align with the curve  $q = \frac{\ell'(p)}{p}$ , in contrast to our observations in Figures 13 and 14. These results suggest that the data label distribution also influences the alignment curve of GD trajectories. As a future research direction, it would be intriguing to investigate why setting the labels as zero leads to alignment towards the curve  $q = \frac{\ell'(p)}{p}$ , which aligns with our theoretical findings in the single data point setting.
(a)  $m = 256$
Figure 15: GD trajectories (1000 iterations) of the training for 3-layer fully connected neural networks with tanh activation and an initialization scale of  $\alpha = 1$ . The networks are trained on a small subset of the CIFAR-10 dataset, where all labels are set to zero. The network widths  $m$  are varied, including values of 256, 512, and 1024. GD trajectories exhibit alignment behavior, aligning on the curve  $q = \frac{\ell'(p)}{p}$ .
(b)  $m = 512$
(c)  $m = 1024$
B Proofs for the Two-Layer Fully-Connected Linear Network
We give the proof of Theorem 4.1, restated below for the sake of readability.
We note that in the given interval  $\left(\frac{2}{2 - \eta}, \min \left\{\frac{1}{16\eta}, \frac{r(1)}{2\eta}\right\}\right)$ , it is possible that the interval is empty, depending on the value of  $r(1)$ . However, this does not impact the correctness of the theorem.
Proof. By Proposition B.1,  $p_t$  converges to 0 as  $t \to \infty$ , and for all  $t \geq 0$ , we have
$$
q _ {0} \leq q _ {t} \leq \exp (C \eta^ {2}) q _ {0}.
$$
Since the sequence  $(q_{t})_{t = 0}^{\infty}$  is monotonic increasing and bounded, it converges. Suppose that  $q_{t}\rightarrow q^{*}$  as  $t\to \infty$  . Then, we can obtain the inequality
$$
q _ {0} \leq q ^ {*} \leq \exp (C \eta^ {2}) q _ {0},
$$
Proposition B.1. Suppose that  $\eta \in (0, \frac{2}{33})$ ,  $|p_0| \leq 1$ , and  $q_0 \in \left(\frac{2}{2 - \eta}, \min \left\{\frac{1}{16\eta}, \frac{r(1)}{2\eta}\right\}\right)$ . Then for any  $t \geq 0$ , we have
$$
\left| p _ {t} \right| \leq \left[ 1 - \min \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {t} \leq 1,
$$
$$
q _ {0} \leq q _ {t} \leq \exp \left(8 \eta^ {2} q _ {0} \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0} \leq 2 q _ {0}.
$$
Proof. We give the proof by induction; namely, if
$$
\left| p _ {t} \right| \leq \left[ 1 - \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {t}, \quad q _ {0} \leq q _ {t} \leq \exp \left(8 \eta^ {2} q _ {0} \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0} \leq 2 q _ {0}
$$
are satisfied for time steps  $0 \leq t \leq k$  for some  $k$ , then the inequalities are also satisfied for the next time step  $k + 1$ .
For the base case, the inequalities are satisfied for  $t = 0$  by assumptions. For the induction step, we assume that the inequalities hold for any  $0 \leq t \leq k$ . We will prove that the inequalities are also satisfied for  $t = k + 1$ .
By induction assumptions,  $q_0 \leq q_k \leq 2q_0$  and  $|p_k| \leq 1$ , so that we have
$$
1 - \frac {2}{q _ {0}} \leq \frac {p _ {k + 1}}{p _ {k}} = 1 - \frac {2 r (p _ {k})}{q _ {k}} + \eta^ {2} p _ {k} ^ {2} r (p _ {k}) ^ {2} \leq 1 - \frac {r (1)}{q _ {0}} + \eta^ {2},
$$
where we used  $r(1) \leq r(p_k) \leq r(0) = 1$ . Since  $q_0 \leq \frac{r(1)}{2\eta} \leq \frac{r(1)}{2\eta^2}$ , we have
$$
\begin{array}{l} \left| \frac {p _ {k + 1}}{p _ {k}} \right| \leq \max  \left\{\frac {2}{q _ {0}} - 1, 1 - \frac {r (1)}{q _ {0}} + \eta^ {2} \right\} \\ = 1 - \min \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} - \eta^ {2} \right\} \\ \end{array}
$$
$$
\leq 1 - \min  \left\{\frac {2 \left(q _ {0} - 1\right)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\}.
$$
$$
\left| p _ {k + 1} \right| \leq \left(1 - \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\}\right) \left| p _ {k} \right| \leq \left[ 1 - \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {k + 1},
$$
which is the desired bound for  $|p_{k + 1}|$ .
For any  $t \leq k$ , we also have
$$
1 - 4 \eta^ {2} p _ {t} ^ {2} q _ {0} \leq \frac {q _ {t}}{q _ {t + 1}} = 1 - \eta^ {2} p _ {t} ^ {2} r (p _ {t}) (2 q _ {t} - r (p _ {t})) \leq 1,
$$
where we used the induction assumptions to deduce  $4q_{0} \geq 2q_{t} - r(p_{t}) \geq 2q_{0} - 1 \geq \frac{4}{2 - \eta} - 1 > 0$ . This gives  $q_{k + 1} \geq q_{k} \geq q_{0}$ . Furthermore, note that  $4\eta^{2}p_{t}^{2}q_{0} \leq 4\eta^{2} \cdot \frac{1}{16\eta} \leq \frac{1}{4}$ , so the ratio  $\frac{q_t}{q_{t + 1}} \in [\frac{3}{4}, 1]$ . From this, we have
$$
\begin{array}{l} \left| \log \left(\frac {q _ {0}}{q _ {k + 1}}\right) \right| \leq \sum_ {t = 0} ^ {k} \left| \log \left(\frac {q _ {t}}{q _ {t + 1}}\right) \right| \leq 2 \sum_ {t = 0} ^ {k} \left| \frac {q _ {t}}{q _ {t + 1}} - 1 \right| \\ \leq 8 \eta^ {2} q _ {0} \sum_ {t = 0} ^ {k} p _ {t} ^ {2} \\ \leq 8 \eta^ {2} q _ {0} \sum_ {t = 0} ^ {k} \left[ 1 - \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {2 t} \\ \leq 8 \eta^ {2} q _ {0} \left[ \min \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {- 1}, \\ \end{array}
$$
where the second inequality holds since  $|\log (1 + z)|\leq 2|z|$  if  $|z|\leq \frac{1}{2}$ . Moreover, this implies that
$$
q _ {k + 1} \leq \exp \left(8 \eta^ {2} q _ {0} \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0}.
$$
Since  $q_0 \in \left(\frac{2}{2 - \eta}, \frac{r(1)}{2\eta}\right)$ , we have
$$
\min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \geq \eta .
$$
Therefore, since  $q_0 \leq \frac{1}{16\eta}$ , we can conclude that
$$
q _ {0} \leq q _ {k + 1} \leq \exp \left(8 \eta^ {2} q _ {0} \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{2 q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0} \leq \exp (8 \eta q _ {0}) q _ {0} \leq 2 q _ {0},
$$
the desired bounds for  $q_{k + 1}$
In this subsection, we prove Theorem 4.2. From here onwards, we use the following notation:
$$
s _ {t} := \frac {q _ {t}}{r (p _ {t})}.
$$
All the lemmas in this subsection are stated in the context of Theorem 4.2.
Lemma B.2. Suppose that the initialization  $(p_0, q_0)$  satisfies  $|p_0| \leq 1$  and  $q_0 \in (c_0, 1 - \delta)$ . Then for any  $t \geq 0$  such that  $q_t \leq 1$ , it holds that
$$
\left| p _ {t} \right| \leq 4, a n d q _ {t} \leq q _ {t + 1} \leq \left(1 + \mathcal {O} (\eta^ {2})\right) q _ {t}.
$$
Proof. We prove by induction. We assume that for some  $t \geq 0$ , it holds that  $|p_t| \leq 4$  and  $\frac{1}{2} \leq q_t \leq 1$ . We will prove that  $|p_{t+1}| \leq 4$  and  $\frac{1}{2} \leq q_t \leq q_{t+1} \leq (1 + \mathcal{O}(\eta^2)) q_t$ . For the base case,  $|p_0| \leq 1 \leq 4$  and  $\frac{1}{2} \leq c_0 < q_t \leq 1$  holds by the assumptions on the initialization. Now suppose that for some  $t \geq 0$ , it holds that  $|p_t| \leq 4$  and  $\frac{1}{2} \leq q_t \leq 1$ . Then for small step size  $\eta$ , we have
$$
\left| \frac {2 \ell^ {\prime} (p _ {t})}{q _ {t}} \right| \geq 2 | \ell^ {\prime} (p _ {t}) | \geq \frac {1}{2} \ell^ {\prime} (p _ {t}) ^ {2} | p _ {t} | \geq \eta^ {2} \ell^ {\prime} (p _ {t}) ^ {2} | p _ {t} |.
$$
Consequently, by Eq. (6),
$$
| p _ {t + 1} | = \left| (1 + \eta^ {2} \ell^ {\prime} (p _ {t}) ^ {2}) p _ {t} - \frac {2 \ell^ {\prime} (p _ {t})}{q _ {t}} \right| \leq \max  \left\{| p _ {t} |, \frac {2}{q _ {t}} \right\} \leq 4.
$$
where we used 1-Lipshitzness of  $\ell$ . Moreover,
$$
1 - 8 \eta^ {2} \leq 1 - 2 | p _ {t} | \eta^ {2} \leq \frac {q _ {t}}{q _ {t + 1}} = 1 - \eta^ {2} p _ {t} ^ {2} r (p _ {t}) (2 q _ {t} - r (p _ {t})) \leq 1,
$$
where we used  $q_{t} \in [\frac{1}{2}, 1]$  and  $|p_{t}r(p_{t})| = |\ell'(p_{t})| \leq 1$  from 1-Lipschitzness of  $\ell$ . Hence,  $q_{t} \leq q_{t+1} \leq (1 + \mathcal{O}(\eta^{2}))q_{t}$ , as desired.
Lemma B.2 implies that  $p_t$  is bounded by a constant throughout the iterations, and  $q_t$  monotonically increases slowly, where the increment for each step is  $\mathcal{O}(\eta^2)$ . Hence, there exists a time step  $T = \Omega(\delta \eta^{-2}) = \Omega_{\delta}(\eta^{-2})$  such that for any  $t \leq T$ , it holds that  $q_t \leq 1 - \frac{\delta}{2}$ . Through out this subsection, we focus on these  $T$  early time steps. Note that for all  $0 \leq t \leq T$ , it holds that  $q_t \in (c_0, 1 - \frac{\delta}{2})$ .