The second point then follows from the first and third one.

For $i \in \mathcal{N}$, the neuron $i$ might not be activated along all $x_k$. However, we can use the same arguments as in the proof of Lemma 21, so that there are time $t_i$ (potentially $\infty$) and $\Delta t_i$, such that $|a_i^t|$ decreases on $[\tau_3, t_i]$ and $i$ is activated along all (or no) $x_k$ on $[t_i + \Delta t_i, \tau_4)$ with $\Delta t_i = \mathcal{O}(1)$. From the first part, it holds for any $t \in [\tau_3, t_i]$

$$|a_i^t - a_i^{\tau_3}| \le |a_i^{\tau_3}| \\ \|w_i^t - w_i^{\tau_3}\| \le \|w_i^t\| + \|w_i^{\tau_3}\| \le 2|a_i^{\tau_3}|.$$

And a Grönwall argument also yields for any $t \in [\tau_3, t_i + \Delta t_i]$, since $\Delta t_i = \mathcal{O}(1)$:

$$|a_i^t - a_i^{\tau_3}| = \mathcal{O}(|a_i^{\tau_3}|) \\ \|w_i^t - w_i^{\tau_3}\| = \mathcal{O}(|a_i^{\tau_3}|).$$

Also, we can bound the difference in the third part similarly to the case $i \in \mathcal{I}$, i.e., for any $t \in [t_i + \Delta t_i, \tau_4)$:

$$|a_i^t - a_i^{t_i+\Delta t_i}| \le a_i^{t_i+\Delta t_i} e^{\mathcal{O}(\varepsilon_3)} \\ \|w_i^t - w_i^{t_i+\Delta t_i}\| \le (e^{\mathcal{O}(\varepsilon_3)} + \mathcal{O}(\varepsilon_3)) a_i^{t_i+\Delta t_i}.$$

Combining these different inequalities then yields the last two points of Lemma 25, using the fact that $\varepsilon_3 = \mathcal{O}(1)$. $\square$

**Corollary 1.** Under Assumption 6, if $\lambda, \varepsilon_3, \varepsilon_4$ and $\delta_4$ satisfy the conditions of Lemma 7 for small enough constants $\tilde{\lambda}, \varepsilon_3^*, \varepsilon_4^*, \delta_4^*$, then $\tau_4 = \infty$.

*Proof.* First note that thanks to Lemma 6 and Lemma 25, we have for any $t \in [\tau_3, \tau_4)$, $i \in \mathcal{I}$ and $k \in [n]$:

$$\langle \mathbf{w}_i^t, x_k \rangle \ge \langle \mathbf{w}_i^{\tau_3}, x_k \rangle - \|\mathbf{w}_i^t - \mathbf{w}_i^{\tau_3}\| \|x_k\| \\ \ge \Omega(1) - \mathcal{O}(\varepsilon_3) \\ > \delta_4 \|x_k\|,$$

for a small enough choice of $\varepsilon_3^*$ and $\delta_4^*$. This implies that the second condition in the definition of $\tau_4$ does not break first. Moreover, thanks to Lemma 25,

$$\|\theta^t - \theta^{\tau_3}\|_2^2 \le \sum_{i \in \mathcal{I}} (a_i^t - a_i^{\tau_3})^2 + \|w_i^t - w_i^{\tau_3}\|^2 + \sum_{i \in \mathcal{N}} (a_i^t - a_i^{\tau_3})^2 + \|w_i^t - w_i^{\tau_3}\|^2 \\ = (e^{\mathcal{O}(\varepsilon_3)} + \mathcal{O}(\varepsilon_3) - 1)^2 \sum_{i \in \mathcal{I}} (a_i^{\tau_3})^2 + \mathcal{O}(1) \sum_{i \in \mathcal{N}} (a_i^{\tau_3})^2 \\ \le (e^{\mathcal{O}(\varepsilon_3)} + \mathcal{O}(\varepsilon_3) - 1)^2 \mathcal{O}(1) + \mathcal{O}(\lambda^{2(1-\varepsilon)}),$$

where the last inequality comes from the bounds on the sum of $(a_i^{\tau_3})^2$ from Lemmas 6 and 19. Now for any $\varepsilon_4^* = \Theta(1)$ and small enough $\varepsilon_3^*, \tilde{\lambda} = \Theta(1)$ (depending on $\varepsilon_4^*$), the previous inequality leads to

$$\|\theta^t - \theta^{\tau_3}\|_2^2 < \varepsilon_4^*.$$
This implies that the first condition in the definition of $\tau_4$ does not break first. As a consequence, neither of the conditions in the definition of $\tau_4$ break in finite time, i.e., $\tau_4 = \infty$. $\square$

**Proof of Lemma 7.** First note that we can choose constants $\varepsilon, \tilde{\lambda}, \varepsilon_2, \varepsilon_3, \varepsilon_4, \delta_4 = \Theta(1)$, such that all the lemmas of Appendix F simultaneously hold for any $\lambda \le \tilde{\lambda}$. An easy way to verify so is going backward after fixing $\varepsilon$, i.e., first fix $\varepsilon_4, \delta_4$, then fixing $\varepsilon_3$, then $\varepsilon_2$ and finally $\tilde{\lambda}$. Thanks to Lemmas 24 and 25 and Corollary 1, the parameters $\theta^t$ and their variations are both bounded on $[\tau_3, \infty)$. As a consequence, $\theta^t$ does have a limit:

$$ \lim_{t \to \infty} \theta^t = \theta_{\lambda}^{\infty}. $$

Moreover, Lemma 21 yields that

$$ \lim_{t \to \infty} R_t = 0. $$

This directly implies the second point of Lemma 7. Now denote

$$ \beta_{\lambda}^{\infty} := \sum_{\substack{i \in [m] \\ \forall k \in [m], \langle w_{\lambda, i}^{\infty}, x_k \rangle \ge 0}} a_{\lambda, i}^{\infty} w_{\lambda, i}^{\infty}. $$

The second point of Lemma 7 that we just proved implies that

$$ h_{\theta^{\infty}}(x_k) = \langle \beta_{\lambda}^{\infty}, x \rangle \quad \text{for any } x \in \mathcal{C}, $$

i.e. $\theta_{\lambda}^{\infty}$ behaves as a linear regression of parameter $\beta_{\lambda}^{\infty}$ on the (cone generated by the) convex hull of the training data. As a consequence,

$$ L(\theta_{\lambda}^{\infty}) = L_{\beta_{\lambda}^{\infty}} := \frac{1}{2n} \sum_{k=1}^{n} (\langle \beta_{\lambda}^{\infty}, x_k \rangle - y_k)^2. $$

Also, Lemma 23 implies that $L(\theta_{\lambda}^{\infty}) = L_{\beta_{\lambda}^{\infty}} \le L_{\beta^{*}}$. However, $\beta^{*}$ is the unique minimiser of the least square loss, among the set of linear regression parameters. This implies that $\beta_{\lambda}^{\infty} = \beta^{*}$, which concludes the proof of both Lemma 7 and Theorem 2.

The last point of Theorem 4 is obtained by using Lemma 20 and noticing that when $\lambda \to 0$, we can also choose the constants $\varepsilon_3$ and $\varepsilon_4$ such that they converge to 0 when $\lambda \to 0$.

## Appendix G. Further Discussion on (Glasgow, 2024)

In this section, we discuss in more details how the early alignment phenomenon is related to Glasgow (2024)'s work. Glasgow (2024) studies the convergence of SGD for XOR-type data with two-layer ReLU network towards four vectors, resulting in a 0 test loss.

We argue that the XOR setting under consideration falls within the scope of our framework. Specifically, the associated population loss exhibits exactly four extremal vectors corresponding to these four vectors reached at convergence (see details below). From this, the initial phase of the dynamics described by Glasgow (2024) aligns with the early alignment phase analyzed in our work—Theorem 1 has yet to be applied to gradient flow over the population loss, which requires extending it to the infinite data setting.
The primary additional technical challenges addressed in Glasgow (2024) involve establishing analogous results for stochastic gradient descent (SGD) rather than gradient flow, and under a finite data regime. Their analysis of early alignment, and the broader training dynamics, for SGD in the XOR setting is achieved by bounding the deviation between SGD and gradient flow on the population loss via concentration inequalities. Controlling the growth of the neurons from the early alignment phase also remains challenging, although simpler to control.

## G.1 Computation of Extremal Vectors in Glasgow 2024 Setting.

In this section, we justify that the population loss only counts four extremal vectors in the XOR setting. For that, we consider the population loss with Gaussian input.

While Glasgow (2024) consider data distributed uniformly over the hypercube, their proof of the first alignment phase relies on an approximation of the uniform distribution on the hypercube by a standard Gaussian in high dimension. In that effect, we directly assume here that the dataset is given by

$$x_k \sim \mathcal{N}(0, I_d)$$
$$y(x_k) = -\text{sign}(e_1^\top x_k)\text{sign}(e_2^\top x_k)$$

and consider the limit of infinite data. We thus denote the (population) loss of the model as

$$L(\theta) := \mathbb{E}_{x,y} [\ell(h_{\theta}(x), y(x))],$$

where $\ell(\hat{y}, y) = \ln(1 + e^{-\hat{y}y})$ is the logistic loss. In this population loss setting, the function $G(w)$ can still be defined, and its gradient is given by

$$D(w, 0) = \mathbb{E}_{x,y} [yx1_{w^\top x \ge 0}].$$

In that infinite data setting, the definition of extremal vector also extends to $D \neq 0$ such that both hold (see Boursier and Flammarion, 2024)

$$1. D = D(w, 0)$$

$$2. \frac{D}{\|D\|} = \pm \frac{w}{\|w\|}.$$

Using computations (see Appendices G.2 and G.3 below) similar to Lemma D.4 (Glasgow, 2024), we can then show that for any $w \in \mathbb{R}^d \setminus \{0\}$:

$$\begin{aligned} \text{sign}(w_2) &= -\text{sign}(\langle D(w, 0), e_1 \rangle), \\ \text{sign}(w_1) &= -\text{sign}(\langle D(w, 0), e_2 \rangle) \end{aligned} \qquad (62)$$

and$^{23}$

$$\text{sign}(\langle D(w, 0), w_{3:d} \rangle) = \text{sign}(w_1 w_2) 1_{w_{3:d} \neq 0}. \qquad (63)$$

From there, assume that $D(w, 0)$ is an extremal vector. If $w_1 = 0$, then the above imply that $D(w, 0) \perp e_j$ for any $j \ge 2$, so that it is not an extremal vector. In consequence, we necessarily have $w_1 \neq 0$ and $w_2 \neq 0$ for similar reasons.

<footnote n="23">23. In this section, $w_i$ is the $i$-th coordinate of $w$ and $w_{i:j}$ is the projection of $w$ onto $\text{Span}(e_i, \dots, e_j)$.</footnote>
From there assume w.l.o.g. that $w_1w_2 > 0$ and also assume that $w_{3:d} \neq 0$. We then have by Equations (62) and (63):

$$ \begin{aligned} \langle D(w, \mathbf{0}), w_{1:2} \rangle &< 0, \\ \langle D(w, \mathbf{0}), w_{3:d} \rangle &> 0. \end{aligned} $$

However, these two inequalities contradict the fact that $D(w, \mathbf{0}) \propto \pm w$, i.e., that it is an extremal vector. Thus, any extremal vector $D(w, \mathbf{0})$ satisfies $w_{3:d} = 0$. Necessarily, it must be of the form $D(w, \mathbf{0}) = \alpha_1 e_1 + \alpha_2 e_2$. We can now be more precise and show that (see Appendix G.4 below) for any $w$ such that $w_{3:d} = 0$,

$$ \begin{aligned} \text{if } |w_1| > |w_2| \quad \text{then} \quad & |\langle D(w, \mathbf{0}), e_2 \rangle| > |\langle D(w, \mathbf{0}), e_1 \rangle|, \\ \text{if } |w_1| < |w_2| \quad \text{then} \quad & |\langle D(w, \mathbf{0}), e_2 \rangle| < |\langle D(w, \mathbf{0}), e_1 \rangle|. \end{aligned} \qquad (64) $$

In consequence, any extremal vector must be such that $|w_1| = |w_2|$, i.e., of the form $D(w, \mathbf{0}) = \alpha(e_1 \pm e_2)$ for some $\alpha \in \mathbb{R}$. In consequence, there exist at most 4 extremal vectors and it is easy to check that for a good choice of $\alpha$ (both $> 0$ and $< 0$), this indeed yields extremal vectors.

We have thus shown that in the XOR setting with population logistic loss and Gaussian data, there are only 4 extremal vectors, which are proportional to the vectors $e_1+e_2, e_1-e_2, -e_1+e_2, e_1+e_2$.

## G.2 Proof of Equation (62).

Similarly to Glasgow (2024), we note $x = z + \xi$ where $z$ is the projection of $x$ on the first two coordinates and $\xi$ on the last $d-2$ coordinates. We also note $y(z) = -\mathrm{sign}(z_1z_2)$. Denoting by $\mathrm{swap}_1(x) = (-x_1, x_2, \dots, x_d)$ the flip operator on the first coordinate, we have using the symmetries of the distribution:

$$ \begin{aligned} \langle D(w, \mathbf{0}), e_2 \rangle &= \mathbb{E}[y(z) \mathbf{1}_{w^\top x \ge 0} x_2] \\ &= \frac{1}{2} \mathbb{E}[(\mathbf{1}_{w^\top x \ge 0} - \mathbf{1}_{w^\top \mathrm{swap}_1(x) \ge 0})y(z)x_2] \\ &= -\frac{1}{2} \mathbb{E}[(\mathbf{1}_{w^\top x \ge 0} - \mathbf{1}_{w^\top \mathrm{swap}_1(x) \ge 0})\mathrm{sign}(x_1)|x_2|]. \end{aligned} \qquad (65) $$

Moreover, note that if $w_1 \ge 0$, we necessarily have $(\mathbf{1}_{w^\top x \ge 0} - \mathbf{1}_{w^\top \mathrm{swap}_1(x) \ge 0})\mathrm{sign}(\langle x, e_1 \rangle) \ge 0$, so that

$$ w_1 \ge 0 \implies \langle D(w, \mathbf{0}), e_2 \rangle \le 0. $$

Moreover if $w_1 > 0$, the above expectation is non-zero since there is at least a non-zero measure subset of $\mathbb{R}^d$ for which $(\mathbf{1}_{w^\top x \ge 0} - \mathbf{1}_{w^\top \mathrm{swap}_1(x) \ge 0})|x_2| > 0$. Symmetric arguments allow to derive Equation (62).
### G.3 Proof of Equation (63).

Similar computations as above yield

$$ 
\begin{aligned}
\langle D(w, \mathbf{0}), w_{3:d} \rangle &= \mathbb{E}[\mathbf{1}_{w^\top x \ge 0} y(z) \langle \xi, w_{3:d} \rangle] \\
&= \frac{1}{2} \mathbb{E}[(\mathbf{1}_{w^\top (z+\xi) \ge 0} - \mathbf{1}_{w^\top (z-\xi) \ge 0}) y(z) \langle \xi, w_{3:d} \rangle] \\
&= \frac{1}{2} \mathbb{E}[y(z) \mathbf{1}_{|\xi^\top w_{3:d}| \ge |z^\top w|} |\langle \xi, w_{3:d} \rangle|] \\
&= \frac{1}{2} \mathbb{E}_{\xi} \mathbb{E}_{z}[\mathbf{1}_{y(z)=1} \mathbf{1}_{|\xi^\top w_{3:d}| \ge |z^\top w|} |\langle \xi, w_{3:d} \rangle|] \\
&\quad - \frac{1}{2} \mathbb{E}_{\xi} \mathbb{E}_{z}[\mathbf{1}_{y(z)=-1} \mathbf{1}_{|\xi^\top w_{3:d}| \ge |z^\top w|} |\langle \xi, w_{3:d} \rangle|]
\end{aligned}
$$

Assume w.l.o.g. that $w_1, w_2 \ge 0$. For a fixed $\xi$ and since $y(z) = -\text{sign}(z_1 z_2)$, the region $\{\mathbf{1}_{y(z)=-1} \mathbf{1}_{|\xi^\top w_{3:d}| \ge |z^\top w|} \ge 0\}$ is smaller (w.r.t. the Gaussian measure) than the region $\{\mathbf{1}_{y(z)=1} \mathbf{1}_{|\xi^\top w_{3:d}| \ge |z^\top w|} \ge 0\}$. It is indeed a consequence of the following observation: if $y(z) = -1$ then $|z^\top w| \ge |\text{swap}_1(z)^\top w|$, i.e.,

$$ \text{if } \mathbf{1}_{y(z)=-1} \mathbf{1}_{|\xi^\top w_{3:d}| \ge |z^\top w|} = 1 $$

$$ \text{then } \mathbf{1}_{y(\text{swap}_1(z))=1} \mathbf{1}_{|\xi^\top w_{3:d}| \ge |\text{swap}_1(z)^\top w|} = 1, $$

and $\text{swap}_1(z)$ also follows a standard Gaussian distribution.

As a consequence, if $w_1, w_2 \ge 0$, the previous inequality implies that $\langle D(w, \mathbf{0}), w_{3:d} \rangle \ge 0$. Moreover, if both $w_1, w_2 > 0$ and $w_{3:d} \ne 0$, the above difference becomes positive, so that $\langle D(w, \mathbf{0}), w_{3:d} \rangle > 0$. More generally, we have shown by symmetric argument that

$$ \text{sign}(D(w, \mathbf{0}), w_{3:d}) = \text{sign}(w_1 w_2) \mathbf{1}_{w_{3:d} \ne 0} $$

### G.4 Proof of Equation (64).

Assume that $w_{3:d} = 0$, Equation (65) then yields

$$ 
\begin{aligned}
\langle D(w, \mathbf{0}), e_2 \rangle &= -\frac{1}{2} \mathbb{E}_x[(\mathbf{1}_{w^\top x \ge 0} - \mathbf{1}_{w^\top \text{swap}_1(x) \ge 0})\text{sign}(x_1)|x_2|] \\
&= -\frac{\text{sign}(w_1)}{2} \mathbb{E}_x[\mathbf{1}_{|w_1 x_1| \ge |w_2 x_2|}\text{sign}(w_1 x_1)\text{sign}(x_1)|x_2|] \\
&= -\frac{\text{sign}(w_1)}{2} \mathbb{E}_x[\mathbf{1}_{|w_1 x_1| \ge |w_2 x_2|}|x_2|].
\end{aligned}
$$

And similarly $\langle D(w, \mathbf{0}), e_1 \rangle = -\frac{\text{sign}(w_2)}{2} \mathbb{E}_x[\mathbf{1}_{|w_2 x_2| \ge |w_1 x_1|}|x_1|]$. Now note that the transformation $\tilde{x} = (x_2, x_1, x_3, \dots, x_d)$ does not change the data distribution and thus yields the following inequalities

$$ 
\begin{aligned}
\mathbb{E}_x[\mathbf{1}_{|w_1 x_1| \ge |w_2 x_2|}|x_2|] &= \mathbb{E}_x[\mathbf{1}_{|w_1 x_2| \ge |w_2 x_1|}|x_1|] \\
&\quad \left\{ 
\begin{array}{ll}
\le \mathbb{E}_x[\mathbf{1}_{|w_2 x_2| \ge |w_1 x_1|}|x_1|] & \text{if } |w_2| \ge |w_1| \\
\ge \mathbb{E}_x[\mathbf{1}_{|w_2 x_2| \ge |w_1 x_1|}|x_1|] & \text{if } |w_2| \le |w_1|.
\end{array}
\right.
\end{aligned}
$$
Moreover, the inequalities are strict if $|w_2| > |w_1|$ or $|w_2| < |w_1|$, so that

$$|\langle D(w, \mathbf{0}), e_2 \rangle| > |\langle D(w, \mathbf{0}), e_1 \rangle| \quad \text{if } |w_1| > |w_2|,$$

$$|\langle D(w, \mathbf{0}), e_2 \rangle| < |\langle D(w, \mathbf{0}), e_1 \rangle| \quad \text{if } |w_1| < |w_2|.$$