SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} - \left\langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla_ {\bar {\mathbf {w}}} L \left(w _ {t}, \bar {\mathbf {w}} _ {t}\right) \right\rangle \\ \leq - e ^ {- \gamma w _ {t}} \cdot \left\langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla G (\bar {\mathbf {w}} _ {t}) \right\rangle + \| \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*} \| _ {2} \cdot \left(e ^ {- 2 \gamma w _ {t}} \cdot G _ {\max } ^ {2} + e ^ {- \theta w _ {t}} \cdot H _ {\max }\right) \\ \leq - e ^ {- \gamma w _ {t}} \cdot \langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla G (\bar {\mathbf {w}} _ {t}) \rangle + \left(W _ {\max } + \| \mathbf {w} _ {*} \| _ {2}\right) \cdot \left(e ^ {- 2 \gamma w _ {t}} \cdot G _ {\max } ^ {2} + e ^ {- \theta w _ {t}} \cdot H _ {\max }\right) \\ \leq - e ^ {- \gamma w _ {t}} \cdot \langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla G (\bar {\mathbf {w}} _ {t}) \rangle + 2 W _ {\max} \cdot \left(e ^ {- 2 \gamma w _ {t}} \cdot G _ {\max} ^ {2} + e ^ {- \theta w _ {t}} \cdot H _ {\max}\right), \\ \end{array}
$$
where the second inequality is by Lemma B.3, and the last inequality is by Lemma B.4. Using the above and the gradient norm bound from Lemma B.9, we get that
$$
\begin{array}{l} \left\| \bar {\mathbf {w}} _ {t + 1} - \bar {\mathbf {w}} _ {*} \right\| _ {2} ^ {2} \leq \left\| \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*} \right\| _ {2} ^ {2} - 2 \eta e ^ {- \gamma w _ {t}} \cdot \langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla G (\bar {\mathbf {w}} _ {t}) \rangle \\ + 4 \eta \cdot W _ {\max } \cdot \left(e ^ {- 2 \gamma w _ {t}} \cdot G _ {\max } ^ {2} + e ^ {- \theta w _ {t}} \cdot H _ {\max }\right) \tag {28} \\ + \eta^ {2} \cdot e ^ {- 2 \gamma w _ {t}} \cdot \left(G _ {\max } + H _ {\max }\right) ^ {2}. \\ \end{array}
$$
By the convexity of  $G(\cdot)$ , we have
$$
\langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla G (\bar {\mathbf {w}} _ {t}) \rangle \geq G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*}). \tag {29}
$$
So we get
$$
\begin{array}{l} 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) \leq 2 \eta e ^ {- \gamma w _ {t}} \cdot \langle \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*}, \nabla G (\bar {\mathbf {w}} _ {t}) \rangle (29) \\ \leq \left\| \overline {{\mathbf {w}}} _ {t} - \overline {{\mathbf {w}}} _ {*} \right\| _ {2} ^ {2} - \left\| \overline {{\mathbf {w}}} _ {t + 1} - \overline {{\mathbf {w}}} _ {*} \right\| _ {2} ^ {2} \\ + 4 \eta \cdot W _ {\max } \cdot \left(e ^ {- 2 \gamma w _ {t}} \cdot G _ {\max } ^ {2} + e ^ {- \theta w _ {t}} \cdot H _ {\max }\right) \\ + \eta^ {2} \cdot e ^ {- 2 \gamma w _ {t}} \cdot \left(G _ {\max } + H _ {\max }\right) ^ {2} (28) \\ \leq \left\| \bar {\mathbf {w}} _ {t} - \bar {\mathbf {w}} _ {*} \right\| _ {2} ^ {2} - \left\| \bar {\mathbf {w}} _ {t + 1} - \bar {\mathbf {w}} _ {*} \right\| _ {2} ^ {2} \\ + 6 \eta \cdot W _ {\max } \cdot \left(G _ {\max } ^ {2} + H _ {\max } ^ {2}\right) \cdot \left(e ^ {- 2 \gamma w _ {t}} + e ^ {- \theta w _ {t}}\right), (30) \\ \end{array}
$$
where we use  $\eta \leq W_{\max} \coloneqq \max \{4n / b, \eta n^2 / b\} + \eta n$  in the last inequality.
Step 2: the sum of function values stays bounded. Observe that
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} \left(e ^ {- 2 \gamma w _ {t}} + e ^ {- \theta w _ {t}}\right) \leq \sum_ {t = 2} ^ {T} \left(\frac {2}{2 + \eta \gamma^ {2} \cdot t}\right) ^ {2} + \sum_ {t = 2} ^ {T} \left(\frac {2}{2 + \eta \gamma^ {2} \cdot t}\right) ^ {\frac {\theta}{\gamma}} \quad \text {b y} \\ \leq \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {2} \cdot \sum_ {t = 2} ^ {T} t ^ {- 2} + \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {\frac {\theta}{\gamma}} \cdot \sum_ {t = 2} ^ {T} t ^ {- \frac {\theta}{\gamma}} \\ \leq \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {2} \cdot 1 + \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {\frac {\theta}{\gamma}} \cdot \frac {1}{\theta / \gamma - 1} \\ \leq \max  \left\{\left(\frac {2}{\eta \gamma^ {2}}\right) ^ {2}, \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {\theta / \gamma} \right\} \cdot \frac {\theta}{\theta - \gamma}. \tag {31} \\ \end{array}
$$
Taking telescope summation over (30), we obtain
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) \\ \leq \left\| \bar {\mathbf {w}} _ {2} - \bar {\mathbf {w}} _ {*} \right\| _ {2} ^ {2} - \left\| \bar {\mathbf {w}} _ {T + 1} - \bar {\mathbf {w}} _ {*} \right\| _ {2} ^ {2} \\ + 6 \eta \cdot W _ {\max } \cdot \left(G _ {\max } ^ {2} + H _ {\max } ^ {2}\right) \cdot \sum_ {t = 2} ^ {T} \left(e ^ {- 2 \gamma w _ {t}} + e ^ {- \theta w _ {t}}\right) (30) \\ \leq 2 W _ {\max } + 6 \eta \cdot W _ {\max } \cdot \left(G _ {\max } ^ {2} + H _ {\max } ^ {2}\right) \cdot \max  \left\{\left(\frac {2}{\eta \gamma^ {2}}\right) ^ {2}, \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {\theta / \gamma} \right\} \cdot \frac {\theta}{\theta - \gamma} \quad \text {b y} (31) \\ = 2 W _ {\max } + 1 8 W _ {\max } \cdot c _ {0}, \\ \end{array}
$$
$$
c _ {0} := \eta \cdot \left(G _ {\max } ^ {2} + H _ {\max } ^ {2}\right) \cdot \frac {\theta}{\theta - \gamma} \cdot \max  \left\{\left(\frac {2}{\eta \gamma^ {2}}\right) ^ {2}, \left(\frac {2}{\eta \gamma^ {2}}\right) ^ {\theta / \gamma} \right\}
$$
is a constant (a polynomial on  $\{e^{\eta}, e^{n}, e^{1/b}, \frac{1}{\eta}, \frac{1}{\theta - \gamma}, \frac{1}{\gamma}, e^{\theta/\gamma}\}$  and is independent of  $t$ ) defined in Lemma B.10.
Step 3: function value decreases, approximately. For  $T \geq t \geq 1$ , we have
$$
G (\bar {\mathbf {w}} _ {T}) \leq G (\bar {\mathbf {w}} _ {t}) + c _ {0} \cdot 2 (1 + \eta) G _ {\max } \cdot \left((t - 1) ^ {- 1} + (t - 1) ^ {1 - \frac {\theta}{\gamma}}\right), \quad \text {b y}
$$
$$
\begin{array}{l} 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {T}) - G (\bar {\mathbf {w}} _ {*})\right) \\ \leq 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) + 2 \eta e ^ {- \gamma w _ {t}} \cdot c _ {0} \cdot 2 (1 + \eta) G _ {\max } \cdot \left((t - 1) ^ {- 1} + (t - 1) ^ {1 - \frac {\theta}{\gamma}}\right) \\ \leq 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) \\ + 2 \eta \cdot \frac {2}{2 + \eta \gamma^ {2} \cdot t} \cdot c _ {0} \cdot 2 (1 + \eta) G _ {\max } \cdot \left((t - 1) ^ {- 1} + (t - 1) ^ {1 - \frac {\theta}{\gamma}}\right) \quad \text {b y} \\ \leq 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) + \frac {8 (1 + \eta) c _ {0}}{\gamma^ {2}} \cdot \left((t - 1) ^ {- 2} + (t - 1) ^ {- \frac {\theta}{\gamma}}\right). \tag {32} \\ \end{array}
$$
Step 4: the last function value is small. Taking summation of (32) over  $t = 2, \ldots, T$ , we get
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {T}) - G (\bar {\mathbf {w}} _ {*})\right) \\ \leq \sum_ {t = 2} ^ {T} 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) + \frac {8 (1 + \eta) c _ {0}}{\gamma^ {2}} \cdot \sum_ {t = 2} ^ {T} \left((t - 1) ^ {- 2} + (t - 1) ^ {- \frac {\theta}{\gamma}}\right) \\ \leq \sum_ {t = 2} ^ {T} 2 \eta e ^ {- \gamma w _ {t}} \cdot \left(G (\bar {\mathbf {w}} _ {t}) - G (\bar {\mathbf {w}} _ {*})\right) + \frac {8 (1 + \eta) c _ {0}}{\gamma^ {2}} \cdot \left(2 + 1 + \frac {1}{\theta / \gamma - 1}\right) \\ \leq 2 W _ {\max } + 1 8 W _ {\max } \cdot c _ {0} + \frac {8 (1 + \eta) c _ {0}}{\gamma^ {2}} \cdot \frac {3 \theta}{\theta - \gamma}. \tag {31} \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 2} ^ {T} e ^ {- \gamma w _ {t}} \geq \frac {1}{e \eta \gamma^ {2} G _ {\max } + e \eta \gamma H _ {\max }} \cdot \sum_ {t = 2} ^ {T} \frac {1}{t + 1} \quad \text {b y} \\ \geq \frac {1}{e \eta \gamma^ {2} G _ {\max } + e \eta \gamma H _ {\max }} \cdot (\log (T + 1) - \log (3)) \\ \end{array}
$$
Putting these together, we get
$$
G (\bar {\mathbf {w}} _ {T}) - G (\bar {\mathbf {w}} _ {*}) \leq \left(2 W _ {\max } + 1 8 W _ {\max } \cdot c _ {0} + \frac {8 (1 + \eta) c _ {0}}{\gamma^ {2}} \cdot \frac {3 \theta}{\theta - \gamma}\right) \cdot \frac {e \eta \gamma^ {2} G _ {\max } + e \eta \gamma H _ {\max }}{\log (T + 1) - \log (3)},
$$
is a polynomial on  $\left\{e^{\eta}, e^{n}, e^{1/b}, \frac{1}{\eta}, \frac{1}{\theta - \gamma}, \frac{1}{\gamma}, e^{\theta/\gamma}\right\}$ . So for  $T \geq 3$ , we have
$$
G (\bar {\mathbf {w}} _ {T}) - G (\bar {\mathbf {w}} _ {*}) \leq \frac {1}{\log (T)} \cdot c _ {1},
$$
where  $c_{1}$  is a polynomial on  $\{e^{\eta}, e^{n}, e^{1 / b}, \frac{1}{\eta}, \frac{1}{\theta - \gamma}, \frac{1}{\gamma}, e^{\theta / \gamma}\}$  and is independent of  $T$ .
C Proofs Missing from the Main Paper
Proof of Theorem 4.1. Theorem 4.1 is a consequence of our analysis in Appendix B.
(C) is because of Lemma B.3.
(B) is because of Lemma B.6.
(D) is because of Lemma B.11.
(A) is because of the following:
$$
\begin{array}{l} L \left(\mathbf {w} _ {t}\right) = \sum_ {i = 1} ^ {n} \log \left(1 + \exp \left(- w _ {t} x _ {i} - \mathbf {w} _ {t} ^ {\top} \mathbf {x} _ {i}\right)\right) \\ \leq \sum_ {i = 1} ^ {n} \exp \left(- w _ {t} x _ {i} - \mathbf {w} _ {t} ^ {\top} \mathbf {x} _ {i}\right) \\ \leq \exp (- w _ {t} \cdot \gamma) \cdot \sum_ {i = 1} ^ {n} \exp (- \mathbf {w} _ {t} ^ {\top} \mathbf {x} _ {i}) \\ \leq c / \log (t), \\ \end{array}
$$
where the last inequality is because that
$$
\exp (- w _ {t} \cdot \gamma) \leq \frac {2}{2 + \eta \gamma^ {2} \cdot t}
$$
by Lemma B.6 and that  $\sum_{i=1}^{n} \exp(-\mathbf{w}_t^\top \mathbf{x}_i)$  is uniformly bounded by a constant by Definition 3.
Proof of Theorem 4.2. The GD iterates can be written as
$$
w _ {t + 1} = w _ {t} + \eta \gamma \cdot e ^ {- \gamma w _ {t}} \cdot \left(e ^ {- \bar {w} _ {t}} + e ^ {\bar {w} _ {t}}\right), \tag {33}
$$
$$
\bar {w} _ {t + 1} = \bar {w} _ {t} - \eta e ^ {- \gamma w _ {t}} \cdot \left(e ^ {- \bar {w} _ {t}} - e ^ {\bar {w} _ {t}}\right). \tag {34}
$$
We claim that: for every  $t \geq 0$ ,
1.  $w_{t}\geq 0$
2.  $|\bar{w}_t| \geq 1$ .
3.  $|\bar{w}_t| \geq 2\gamma w_t$
We prove the claim by induction. For  $t = 0$ , it holds by assumption. Now suppose that the claim holds for  $t$  and consider the case of  $t + 1$ .
1.  $w_{t + 1}\geq 0$  holds since  $w_{t + 1}\geq w_t$  by (33) and  $w_{t}\geq 0$  by the induction hypothesis.
2.  $|\bar{w}_{t + 1}|\geq 1$  holds because
$$
\begin{array}{l} \left| \bar {w} _ {t + 1} \right| \geq \eta e ^ {- \gamma w _ {t}} \cdot \left| e ^ {- \bar {w} _ {t}} - e ^ {\bar {w} _ {t}} \right| - \left| \bar {w} _ {t} \right| \quad \text {b y} (3 4) \\ \geq \eta e ^ {- \gamma w _ {t}} \cdot \frac {e ^ {| \bar {w} _ {t} |}}{2} - | \bar {w} _ {t} | \quad \text {s i n c e} | \bar {w} _ {t} | \geq 1 \text {a n d t h a t} e ^ {t} - e ^ {- t} \geq \frac {e ^ {t}}{2} \text {f o r} t \geq 1 \tag {35} \\ \geq 2 e ^ {\left| \bar {w} _ {t} \right| - \gamma w _ {t}} - \left| \bar {w} _ {t} \right| \quad \text {s i n c e} \eta \geq 4 \\ \geq 2 e ^ {| \bar {w} _ {t} | / 2} - | \bar {w} _ {t} | \quad \text {s i n c e} \frac {| \bar {w} _ {t} |}{2} \geq \gamma w _ {t} \\ \geq 1. \quad \text {s i n c e} 2 e ^ {t / 2} \geq t + 1 \text {f o r} t \in \mathbb {R} \\ \end{array}
$$
3. To prove that  $|\bar{w}_{t + 1}|\geq 2\gamma w_t$  , first observe that
$$
\begin{array}{l} w _ {t + 1} = w _ {t} + \eta \gamma \cdot e ^ {- \gamma w} \cdot \left(e ^ {- \bar {w}} + e ^ {\bar {w}}\right) \\ \leq w _ {t} + \eta \gamma \cdot e ^ {- \gamma w} \cdot 2 \cdot e ^ {| \bar {w} _ {t} |}. \tag {36} \\ \end{array}
$$
$$
\begin{array}{l} \left| \bar {w} _ {t + 1} \right| - 2 \gamma w _ {t + 1} \\ \geq \eta e ^ {- \gamma w _ {t}} \cdot \frac {e ^ {| \bar {w} _ {t} |}}{2} - | \bar {w} _ {t} | - 2 \gamma \left(w _ {t} + \eta \gamma \cdot e ^ {- \gamma w} \cdot 2 \cdot e ^ {| \bar {w} _ {t} |}\right) \quad \text {b y (3 5) a n d (3 6)} \\ = \frac {\eta}{2} \cdot (1 - 8 \gamma^ {2}) \cdot e ^ {| \bar {w} _ {t} | - \gamma w _ {t}} - | \bar {w} _ {t} | - 2 \gamma w _ {t} \\ \geq e ^ {\left| \bar {w} _ {t} \right| - \gamma w _ {t}} - \left| \bar {w} _ {t} \right| - 2 \gamma w _ {t} \quad \text {s i n c e} \eta \geq 4 \geq 2 / (1 - 8 \gamma^ {2}) \\ \geq e ^ {\left| \bar {w} _ {t} \right| - \gamma w _ {t}} - \left| \bar {w} _ {t} \right| \quad \text {s i n c e} w _ {t} \geq 0 \\ \geq e ^ {\left| \bar {w} _ {t} \right| / 2} - \left| \bar {w} _ {t} \right| \quad \text {s i n c e} \frac {\left| \bar {w} _ {t} \right|}{2} \geq \gamma w _ {t} \\ \geq 0. \quad \text {s i n c e} e ^ {t / 2} \geq t \text {f o r} t \in \mathbb {R} \\ \end{array}
$$
We have completed the induction.
Finally, we prove the claims in Theorem 4.2 using the above results.
(B) is because of
$$
w _ {t + 1} \geq w _ {t} + \eta \gamma \cdot e ^ {- \gamma w _ {t}}
$$
from (33).
We have already proved (C) by induction.
To show (D), without lose of generality, let us assume  $\bar{w}_t\geq 0$  , then
$$
\begin{array}{l} \bar {w} _ {t + 1} \leq \bar {w} _ {t} - \eta e ^ {- \gamma w _ {t}} \cdot \left| e ^ {- \bar {w} _ {t}} - e ^ {\bar {w} _ {t}} \right| \quad \text {b y} \tag {34} \\ \leq \bar {w} _ {t} - \eta e ^ {- \gamma w _ {t}} \cdot \frac {e ^ {| \bar {w} _ {t} |}}{2} \quad \text {s i n c e} | \bar {w} _ {t} | \geq 1 \text {a n d t h a t} e ^ {t} - e ^ {- t} \geq \frac {e ^ {t}}{2} \text {f o r} t \geq 1 \\ \leq \bar {w} _ {t} - 2 e ^ {| \bar {w} _ {t} | - \gamma w _ {t}} \quad \text {s i n c e} \eta \geq 4 \\ \leq \bar {w} _ {t} - 2 e ^ {\left| \bar {w} _ {t} \right| / 2} \quad \text {s i n c e} \frac {\left| \bar {w} _ {t} \right|}{2} \geq \gamma w _ {t} \\ \leq - 1. \quad \text {s i n c e} 2 e ^ {t / 2} \geq t + 1 \text {f o r} t \in \mathbb {R} \\ \end{array}
$$
We can repeat the above argument to show that  $\bar{w}_{t + 1} > 0$  if  $\bar{w}_t\leq 0$
(b) Normalized margin for homogenous network
Figure 3: The behaviors of GD for optimizing a homogenous neural network. The experiment setting follows that of Figure 1 in the main paper, except that we disable all bias parameters so that the neural network is homogenous [Lyu and Li, 2020]. The sub-figures (a) and (b) report the training loss and the margin along the GD trajectories, respectively. Here, we follow Lyu and Li [2020] and measure the normalized margin by  $\min_{(\mathbf{x},y)}\left(f(\mathbf{x};\mathbf{w}_t)[y] - \max_{y'\neq y}f(\mathbf{x};\mathbf{w}_t)[y']\right) / \| \mathbf{w}_t\|_2^L$ , where  $f(\cdot ;\cdot)$  refers to the homogenous neural network,  $L = 4$  is the depth of the homogenous neural network,  $\mathbf{w}_t$  is the weight at the  $t$ -th GD step, and  $\min_{(\mathbf{x},y)}$  is taken with respect to all training data.
To show (A), we apply  $w_{t}\to \infty$  and that  $|\bar{w}_t|\geq 2\gamma w_t$
$$
\begin{array}{l} L \left(w _ {t}, \bar {w} _ {t}\right) = e ^ {- \gamma w _ {t}} \cdot \left(e ^ {- \bar {w} _ {t}} + e ^ {\bar {w} _ {t}}\right) \\ \geq e ^ {- \gamma w _ {t}} \cdot e ^ {| \bar {w} _ {t} |} \\ \geq e ^ {\gamma w _ {t}} \rightarrow \infty . \\ \end{array}
$$
We have completed all the proofs.
Neural network experiments. We randomly sample 1,000 data from the MNIST $^3$  dataset as the training set and use the remaining data as the test set. The feature vectors are normalized such that each feature is within  $[-1,1]$ .
We use a fully connected network with the following structure
$$
7 8 4 \rightarrow \operatorname {R e L U} \rightarrow 5 0 0 \rightarrow \operatorname {R e L U} \rightarrow 5 0 0 \rightarrow \operatorname {R e L U} \rightarrow 5 0 0 \rightarrow \operatorname {R e L U} \rightarrow 1 0.
$$
The network is initialized with Kaiming initialization. We use the cross-entropy loss.
We consider constant-stepsize GD with two types of stepsizes,  $\eta = 0.1$  and  $\eta = 0.01$ .
The results are presented in Figure 1.
Logistic regression experiments. We randomly sample 1,000 data with labels "0" and "8" from the MNIST dataset as the training set. The feature vectors are normalized such that each feature is within  $[-1,1]$ .
We use a linear model without bias. So the number of parameters is 784. The model is initialized from zero. We use the binary cross-entropy loss, i.e., the logistic loss.
We consider constant-stepsize GD with various stepsizes.
The results are presented in Figure 2.
Additional experiments. We conduct additional experiments on the margin maximization effect of large stepsize GD on a homogenous neural network. Results are presented in Figure 3.
<sup>3</sup>http://yann.lecun.com/exdb/mnist/