SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dmitry Chistikov*
d.chistikov@warwick.ac.uk
Matthias Englert*
m.englert@warwick.ac.uk
Ranko Lazic*
r.s.lazic@warwick.ac.uk
We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters. By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets. Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm. Finally we perform a range of numerical experiments, which corroborate our theoretical findings.
One of the grand challenges for machine learning research is to understand how overparameterised neural networks are able to fit perfectly the training examples and simultaneously to generalise well to unseen data [Zhang, Bengio, Hardt, Recht, and Vinyals, 2021]. The double-descent phenomenon [Belkin, Hsu, Ma, and Mandal, 2019], where increasing the neural network capacity beyond the interpolation threshold can eventually reduce the test loss much further than could be achieved around the underparameterised "sweet spot", is a mystery from the standpoint of classical machine learning theory. This has been observed to happen even for training without explicit regularisers.
Implicit bias of gradient-based algorithms. A key hypothesis towards explaining the double-descent phenomenon is that the gradient-based algorithms that are used for training are implicitly biased (or implicitly regularised) [Neyshabur, Bhojanapalli, McAllester, and Srebro, 2017] to converge to solutions that in addition to fitting the training examples have certain properties which cause them to generalise well. It has attracted much attention in recent years from the research community, which has made substantial progress in uncovering implicit biases of training algorithms in many important settings [Vardi, 2023]. For example, for classification tasks, and for homogeneous networks (which is a wide class that includes ReLU networks provided they contain neither biases at levels deeper than the first nor residual connections), Lyu and Li [2020] and Ji and Telgarsky [2020] established that gradient flow is biased towards maximising the classification margin in parameter space, in the sense that once the training loss gets sufficiently small, the direction of the parameters subsequently converges to a Karush-Kuhn-Tucker point of the margin maximisation problem.
Insights gained in this foundational research direction have not only shed light on overparameterised generalisation, but have been applied to tackle other central problems, such as the susceptibility of networks trained by gradient-based algorithms to adversarial examples [Vardi, Yehudai, and Shamir, 2022] and the possibility of extracting training data from network parameters [Haim, Vardi, Yehudai, Shamir, and Irani, 2022].
Regression tasks and initialisation scale. Showing the implicit bias for regression tasks, where the loss function is commonly mean square, has turned out to be more challenging than for classification tasks, where loss functions typically have exponential tails. A major difference is that, whereas most of the results for classification do not depend on how the network parameters are initialised, the scale of the initialisation has been observed to affect decisively the implicit bias of gradient-based algorithms for regression [Woodworth, Gunasekar, Lee, Moroshko, Savarese, Golan, Soudry, and Srebro, 2020]. When it is large so that the training follows the lazy regime, we tend to have fast convergence to a global minimum of the loss, however without an implicit bias towards sparsity and with limited generalisation [Jacot, Ged, Şimsek, Hongler, and Gabriel, 2021]. The focus, albeit at the price of uncertain convergence and lengthier training, has therefore been on the rich regime where the initialisation scale is small.
Considerable advances have been achieved for linear networks. For example, Azulay, Moroshko, Nacson, Woodworth, Srebro, Globerson, and Soudry [2021] and Yun, Krishnan, and Mobahi [2021] proved that gradient flow is biased to minimise the Euclidean norm of the predictor for one-hidden layer linear networks with infinitesimally small initialisation, and that the same holds also for deeper linear networks under an additional assumption on their initialisation. A related extensive line of work is on implicit bias of gradient-based algorithms for matrix factorisation and reconstruction, which has been a fruitful test-bed for regression using multi-layer networks. For example, Gunasekar, Woodworth, Bhojanapalli, Neyshabur, and Srebro [2017] proved that, under a commutativity restriction and starting from a small initialisation, gradient flow is biased to minimise the nuclear norm of the solution matrix; they also conjectured that the restriction can be dropped, which after a number of subsequent works was refuted by Li, Luo, and Lyu [2021], leading to a detailed analysis of both underparameterised and overparameterised regimes by Jin, Li, Lyu, Du, and Lee [2023].
For non-linear networks, such as those with the popular ReLU activation, progress has been difficult. Indeed, Vardi and Shamir [2021] showed that precisely characterising the implicit bias via a non-trivial regularisation function is impossible already for single-neuron one-hidden layer ReLU networks, and Timor, Vardi, and Shamir [2023] showed that gradient flow is not biased towards low-rank parameter matrices for multiple-output ReLU networks already with one hidden layer and small training datasets.
ReLU networks and training dynamics. We suggest that, in order to further substantially our knowledge of convergence, implicit bias, and generalisation for regression tasks using non-linear networks, we need to understand more thoroughly the dynamics throughout the gradient-based training. This is because of the observed strong influence that initialisation has on solutions, but is challenging due to the highly non-convex optimisation landscape. To this end, evidence and intuition were provided by Maennel, Bousquet, and Gelly [2018], Li et al. [2021], and Jacot et al. [2021], who conjectured that, from sufficiently small initialisations, after an initial phase where the neurons get aligned to a number of directions that depend only on the dataset, training causes the parameters to pass close to a sequence of saddle points, during which their rank increases gradually but stays low.
The first comprehensive analysis in this vein was accomplished by Boursier, Pillaud-Vivien, and Flammarion [2022], who focused on orthogonal datasets (which are therefore of cardinality less than or equal to the input dimension), and established that, for one-hidden layer ReLU networks, gradient flow from an infinitesimal initialisation converges to zero loss and is implicitly biased to minimise the Euclidean norm of the network parameters. They also showed that, per sign class of the training labels (positive or negative), minimising the Euclidean norm of the interpolator networks coincides with minimising their rank.
Our contributions. We tackle the main challenge posed by Boursier et al. [2022], namely handling datasets that are not orthogonal. A major obstacle to doing so is that, whereas the analysis of the training dynamics in the orthogonal case made extensive use of an almost complete separation between a turning phase and a growth phase for all hidden neurons, non-orthogonal datasets cause
considerably more complex dynamics in which hidden neurons follow training trajectories that simultaneously evolve their directions and norms [Boursier et al., 2022, Appendix A].
To analyse this involved dynamics in a reasonably clean setting, we consider the training of one-hidden layer ReLU networks by gradient flow from a small balanced initialisation on datasets that are labelled by a teacher ReLU neuron with which all the training points are correlated. More precisely, we assume that the angles between the training points and the teacher neuron are less than  $\pi /4$ , which implies that all angles between training points are less than  $\pi /2$ . The latter restriction has featured per label class in many works in the literature (such as by Phuong and Lampert [2021] and Wang and Pilanci [2022]), and the former is satisfied for example if the training points can be obtained by summing the teacher neuron  $\pmb{v}^{*}$  with arbitrary vectors of length less than  $\| \pmb{v}^{*}\| /\sqrt{2}$ . All our other assumptions are very mild, either satisfied with probability exponentially close to 1 by any standard random initialisation, or excluding corner cases of Lebesgue measure zero.
Our contributions can be summarised as follows.
the initialisation scales are small but not nearly as small as in the theory, and the angles between the teacher neuron and the training points are distributed around  $\pi /4$ .
We further discuss related work, prove all theoretical results, and provide additional material on our experiments, in the appendix.
Notation. We write:  $[n]$  for the set  $\{1, \ldots, n\}$ ,  $\| \pmb{v} \|$  for the Euclidean length of a vector  $\pmb{v}$ ,  $\overline{\pmb{v}} \coloneqq \pmb{v} / \| \pmb{v} \|$  for the normalised vector,  $\angle (\pmb{v}, \pmb{v}') \coloneqq \arccos (\overline{\pmb{v}}^{\top} \overline{\pmb{v}}')$  for the angle between  $\pmb{v}$  and  $\pmb{v}'$ , and  $\operatorname{cone} \{\pmb{v}_1, \dots, \pmb{v}_n\} \coloneqq \{\sum_{i=1}^{n} \beta_i \pmb{v}_i \mid \beta_1, \dots, \beta_n \geq 0\}$  for the cone generated by vectors  $\pmb{v}_1, \dots, \pmb{v}_n$ .
One-hidden layer ReLU network. For an input  $\pmb{x} \in \mathbb{R}^d$ , the output of the network is
$$
h _ {\pmb {\theta}} (\pmb {x}) := \sum_ {j = 1} ^ {m} a _ {j} \sigma (\pmb {w} _ {j} ^ {\top} \pmb {x}),
$$
where  $m$  is the width, the parameters  $\pmb{\theta} = (\pmb{a},\pmb{W})\in \mathbb{R}^{m}\times \mathbb{R}^{m\times d}$  consist of last-layer weights  $\pmb{a} = [a_{1},\dots ,a_{m}]$  and hidden-layer weights  $\pmb{W}^{\top} = [\pmb{w}_{1},\dots ,\pmb{w}_{m}]$ , and  $\sigma (u)\coloneqq \max \{u,0\}$  is the ReLU function.
Balanced initialisation. For all  $j \in [m]$  let
$$
\boldsymbol {w} _ {j} ^ {0} := \lambda \boldsymbol {z} _ {j} \quad a _ {j} ^ {0} := s _ {j} \| \boldsymbol {w} _ {j} ^ {0} \|
$$
where  $\lambda > 0$  is the initialisation scale,  $z_{j} \in \mathbb{R}^{d} \setminus \{\mathbf{0}\}$ , and  $s_{j} \in \{\pm 1\}$ .
A precise upper bound on  $\lambda$  will be stated in Assumption 2.
We regard the initial unscaled hidden-layer weights  $z_{j}$  and last-layer signs  $s_{j}$  as given, without assuming any specific random distributions for them. For example, we might have that each  $z_{j}$  consists of  $d$  independent centred Gaussians with variance  $\frac{1}{dm}$  and each  $s_{j}$  is uniform over  $\{\pm 1\}$ .
We consider only initialisations for which the layers are balanced, i.e.  $|a_j^0| = \| \pmb{w}_j^0 \|$  for all  $j \in [m]$ . Since more generally each difference  $(a_j^t)^2 - \| \pmb{w}_j^t \|^2$  is constant throughout training [Du, Hu, and Lee, 2018, Theorem 2.1] and we focus on small initialisation scales that tend to zero, this restriction (which is also present in Boursier et al. [2022]) is minor but simplifies our analysis.
Neuron-labelled correlated inputs. The teacher neuron  $\boldsymbol{v}^{*} \in \mathbb{R}^{d}$  and the training dataset  $\{(\boldsymbol{x}_i, y_i)\}_{i=1}^n \subseteq (\mathbb{R}^d \setminus \{\mathbf{0}\}) \times \mathbb{R}$  are such that for all  $i$  we have
$$
y _ {i} = \sigma \left(\boldsymbol {v} ^ {* ^ {\top}} \boldsymbol {x} _ {i}\right) \quad \angle \left(\boldsymbol {v} ^ {*}, \boldsymbol {x} _ {i}\right) <   \pi / 4.
$$
In particular, since the angles between  $\pmb{v}^{*}$  and the training points  $\pmb{x}_i$  are acute, each label  $y_{i}$  is positive.
To apply our results to a network with biases in the hidden layer and to a teacher neuron with a bias, one can work in dimension  $d + 1$  and extend the training points to  $\left[ \begin{array}{c}x_{i}\\ 1 \end{array} \right]$ .
Mean square loss gradient flow. For the regression task of learning the teacher neuron by the one-hidden layer ReLU network, we use the standard mean square empirical loss
$$
L (\boldsymbol {\theta}) := \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(y _ {i} - h _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i}\right)\right) ^ {2}.
$$
Our theoretical analysis concentrates on training by gradient flow, which from an initialisation as above evolves the network parameters by descending along the gradient of the loss by infinitesimal steps in continuous time [Li, Tai, and E, 2019]. Formally, we consider any parameter trajectory  $\theta^t\colon [0,\infty)\to \mathbb{R}^m\times \mathbb{R}^{m\times d}$  that is absolutely continuous on every compact subinterval, and that satisfies the differential inclusion
$$
\mathrm {d} \boldsymbol {\theta} ^ {t} / \mathrm {d} t \in - \partial L (\boldsymbol {\theta} ^ {t}) \quad \text {f o r a l m o s t a l l} t \in [ 0, \infty),
$$
where  $\partial L$  denotes the Clarke [1975] subdifferential of the loss function (which is locally Lipschitz).
We work with the Clarke subdifferential, which is a generalisation of the gradient, because the ReLU activation is not differentiable at 0, which causes non-differentiability of the loss function [Bolte, Daniilidis, Ley, and Mazet, 2010]. Although it follows from our results that, in our setting, the derivative of the ReLU can be fixed as  $\sigma'(0) \coloneqq 0$  like in the orthogonal case [Boursier et al., 2022, Appendix D], and the gradient flow trajectories are uniquely defined, that is not a priori clear; hence we work with the unrestricted Clarke subdifferential of the ReLU. We also remark that, in other settings,  $\sigma'(0)$  cannot be fixed in this way due to gradient flow subtrajectories that correspond to gradient descent zig-zagging along a ReLU boundary (cf. e.g. Maennel et al. [2018, section 9.4]).
Basic observations. We establish the formulas for the derivatives of the last-layer weights and the hidden neurons; and that throughout the training, the signs of the last-layer weights do not change, and their absolute values track the norms of the corresponding hidden neurons. The latter property holds for all times  $t$  by continuity and enables us to focus the analysis on the hidden neurons.
Proposition 1. For all  $j \in [m]$  and almost all  $t \in [0, \infty)$  we have:
(i)  $\mathrm{da}_j^t /\mathrm{dt} = \pmb {w}_j^{t\top}\pmb {g}_j^t$  and  $\mathrm{d}\pmb {w}_j^t /\mathrm{dt} = a_j^t\pmb {g}_j^t$  where  $\pmb {g}_j^t\in \frac{1}{n}\sum_{i = 1}^n (y_i - h_{\pmb{\theta}^t}(\pmb {x}_i))\partial \sigma (\pmb {w}_j^{t\top}\pmb {x}_i)\pmb {x}_i;$
(ii)  $a_j^t = s_j\| \pmb {w}_j^t\| \neq 0.$
The definition in part (i) of the vectors  $\pmb{g}_j^t$  that govern the dynamics is a membership because the subdifferential of the ReLU at 0 is the set of all values between 0 and 1, i.e.  $\partial \sigma(0) = [0,1]$ .
3 Assumptions
To state our assumptions precisely, we introduce some additional notation. Let
$$
I _ {+} (\boldsymbol {v}) := \left\{i \in [ n ] \mid \boldsymbol {v} ^ {\top} \boldsymbol {x} _ {i} > 0 \right\} \quad I _ {0} (\boldsymbol {v}) := \left\{i \in [ n ] \mid \boldsymbol {v} ^ {\top} \boldsymbol {x} _ {i} = 0 \right\} \quad I _ {-} (\boldsymbol {v}) := \left\{i \in [ n ] \mid \boldsymbol {v} ^ {\top} \boldsymbol {x} _ {i} <   0 \right\}
$$
denote the sets of indices of training points that are, respectively, either inside or on the boundary or outside of the non-negative half-space of a vector  $\pmb{v}$ . Then let
$$
J _ {+} := \{j \in [ m ] \mid I _ {+} (z _ {j}) \neq \emptyset \wedge s _ {j} = + 1 \} \quad J _ {-} := \{j \in [ m ] \mid I _ {+} (z _ {j}) \neq \emptyset \wedge s _ {j} = - 1 \}
$$
be the sets of indices of hidden neurons that are initially active on at least one training point and whose last-layer signs are, respectively, positive or negative. Also let
$$
\boldsymbol {X} := \left[ \boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n} \right] \quad \gamma_ {I} := \frac {1}{n} \sum_ {i \in I} y _ {i} \boldsymbol {x} _ {i}
$$
denote the matrix whose columns are all the training points, and the sum of all training points whose indices are in a set  $I$ , weighted by the corresponding labels and divided by  $n$ .
Moreover we define, for each  $j \in J_{+} \cup J_{-}$ , a continuous trajectory  $\alpha_{j}^{t}$  in  $\mathbb{R}^d$  by
$$
\boldsymbol {\alpha} _ {j} ^ {0} := \boldsymbol {z} _ {j} \qquad \qquad \mathrm {d} \boldsymbol {\alpha} _ {j} ^ {t} / \mathrm {d} t := s _ {j} \| \boldsymbol {\alpha} _ {j} ^ {t} \|   \boldsymbol {\gamma} _ {I _ {+} (\boldsymbol {\alpha} _ {j} ^ {t})} \quad \text {f o r a l l} t \in (0, \infty)  .
$$
Thus, starting from the unscaled initialisation  $\mathbf{z}_j$  of the corresponding hidden neuron,  $\alpha_j^t$  follows a dynamics obtained from that of  $\mathbf{w}_j^t$  in Proposition 1 (i) and (ii) by replacing the vector  $\mathbf{g}_j^t$  by  $\gamma_{I_+(\alpha_j^t)}$ , which amounts to removing from  $\mathbf{g}_j^t$  the network output terms and the activation boundary summands. These trajectories will be useful as yardsticks in our analysis of the first phase of the training.
Assumption 1. (i)  $d > 1$ ,  $\operatorname{span}\{\pmb{x}_1, \dots, \pmb{x}_n\} = \mathbb{R}^d$ , and  $\|\pmb{v}^*\| = 1$ .
(ii)  $J_{+}\neq \emptyset ,I_{0}(\pmb {z}_{j}) = \emptyset$  for all  $j\in [m]$  , and  $\angle (\pmb {z}_j,\pmb{\gamma}_{[n]}) > 0$  for all  $j\in J_{-}$
(iii)  $\overline{\pmb{x}}_1, \dots, \overline{\pmb{x}}_n$  are distinct, the eigenvalues of  $\frac{1}{n} \pmb{X} \pmb{X}^\top$  are distinct, and  $\pmb{v}^*$  does not belong to a span of fewer than  $d$  eigenvectors of  $\frac{1}{n} \pmb{X} \pmb{X}^\top$ .
(iv)  $|I_0(\pmb{\alpha}_j^t)| \leq 1$  for all  $j \in J_{+} \cup J_{-}$  and all  $t \in [0, \infty)$ .
(v) For all  $j \in [m]$  and all  $0 \leq T < T'$ , if for all  $t \in (T, T')$  we have  $I_{+}(\boldsymbol{w}_{j}^{t}) = I_{0}(\boldsymbol{w}_{j}^{T'}) \neq \emptyset$  and  $I_{0}(\boldsymbol{w}_{j}^{t}) = I_{+}(\boldsymbol{w}_{j}^{T'}) = \emptyset$ , then for all  $t \geq T'$  we have  $\boldsymbol{w}_{j}^{t} = \boldsymbol{w}_{j}^{T'}$ .
This assumption is very mild. Part (i) excludes the trivial univariate case without biases (for univariate inputs with biases one needs  $d = 2$ ), ensures that exact learning is possible, and fixes the length of the teacher neuron to streamline the presentation. Part (ii) assumes that, initially: at least one hidden neuron with a positive last-layer weight has in its active half-space at least one training point, no training point is at a ReLU boundary, and no hidden neuron with a negative last-layer weight is perfectly aligned with the  $\gamma_{[n]}$  vector; this holds with probability at least  $1 - (3/4)^m$  for any continuous symmetric distribution of the unscaled hidden-neuron initialisations, e.g.  $z_j \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, \frac{1}{dm} I_d)$ , and the uniform distribution of the last-layer signs  $s_j \stackrel{\text{i.i.d.}}{\sim} \mathcal{U}\{\pm 1\}$ . Parts (iii) and (iv) exclude corner cases of Lebesgue measure zero; observe that  $\frac{1}{n} XX^\top$  is positive-definite, and that (iv) rules out a yardstick trajectory encountering two or more training points in its half-space boundary at exactly the same time. Part (v) excludes some unrealistic gradient flows that might otherwise be possible due to the use of the subdifferential: it specifies that, whenever a neuron deactivates (i.e. all training points exit its positive half-space), then it stays deactivated for the remainder of the training.
Before our next assumption, we define several further quantities. Let  $\eta_1 > \dots > \eta_d > 0$  denote the eigenvalues of  $\frac{1}{n} \boldsymbol{X} \boldsymbol{X}^\top$ , and let  $\boldsymbol{u}_1, \ldots, \boldsymbol{u}_d$  denote the corresponding unit-length eigenvectors such that  $\boldsymbol{v}^* = \sum_{k=1}^{d} \nu_k^* \boldsymbol{u}_k$  for some  $\nu_1^*, \ldots, \nu_d^* > 0$ . Also, for each  $j \in J_+ \cup J_-$ , let  $n_j := |I_{-s_j}(\boldsymbol{z}_j)|$  be the number of training points that should enter into or exit from the non-negative half-space along the trajectory  $\boldsymbol{\alpha}_j^t$  depending on whether the sign  $s_j$  is positive or negative (respectively), and let
$$
\varphi_ {j} ^ {t} := \angle (\boldsymbol {\alpha} _ {j} ^ {t}, \boldsymbol {\gamma} _ {I _ {+} \left(\boldsymbol {\alpha} _ {j} ^ {t}\right)}) \quad \text {f o r a l l} t \in [ 0, \infty) \text {s u c h t h a t} I _ {+} \left(\boldsymbol {\alpha} _ {j} ^ {t}\right) \neq \emptyset
$$
be the evolving angle between  $\alpha_{j}^{t}$  and the vector governing its dynamics (if any). Then the existence of the times at which the entries or the exits occur is confirmed in the following.
Proposition 2. For all  $j \in J_{+} \cup J_{-}$  there exist a unique enumeration  $i_j^1, \ldots, i_j^{n_j}$  of  $I_{-s_j}(z_j)$  and unique  $0 = \tau_j^0 < \tau_j^1 < \dots < \tau_j^{n_j}$  such that for all  $\ell \in [n_j]$ :
(i)  $I_{s_j}(\pmb{\alpha}_j^t) = I_{s_j}(\pmb{z}_j) \cup \{i_j^1, \dots, i_j^{\ell - 1}\}$  for all  $t \in (\tau_j^{\ell - 1}, \tau_j^\ell)$ ;
(ii)  $I_0(\pmb{\alpha}_j^t) = \emptyset$  for all  $t \in (\tau_j^{\ell - 1}, \tau_j^\ell)$ , and  $I_0\Big(\pmb{\alpha}_j^{\tau_j^\ell}\Big) = \{i_j^\ell\}$ .
Finally we define two measurements of the unscaled initialisation and the training dataset, which are positive thanks to Assumption 1, and which will simplify the presentation of our results.
$$
\delta := \min  \left\{ \begin{array}{c} \min  _ {i \in [ n ]} \| \boldsymbol {x} _ {i} \|, \min  _ {i, i ^ {\prime} \in [ n ]} \overline {{\boldsymbol {x}}} _ {i} ^ {\top} \overline {{\boldsymbol {x}}} _ {i ^ {\prime}}, \min  _ {k \in [ d - 1 ]} (\sqrt {\eta_ {k}} - \sqrt {\eta_ {k + 1}}) (d - 1),   \sqrt {\eta_ {d}}, \\ \min  _ {k \in [ d ]} \nu_ {k} ^ {*} \sqrt {d},   \min  _ {j \in [ m ]} \| \boldsymbol {z} _ {j} \|, \min  _ {j \in J _ {+}} \cos \varphi_ {j} ^ {0},   \min  _ {j \in J _ {-}} \sin \varphi_ {j} ^ {0}, \\ \min  \left\{\left| \overline {{\boldsymbol {\alpha}}} _ {j} ^ {t} \right. ^ {\top} \overline {{\boldsymbol {x}}} _ {i} | \left| \begin{array}{c} j \in J _ {+} \cup J _ {-} \wedge \ell \in [ n _ {j} ] \\ \wedge t \in [ \tau_ {j} ^ {\ell - 1}, \tau_ {j} ^ {\ell} ] \wedge i \in [ n ] \\ \wedge i \neq i _ {j} ^ {\ell} \wedge (\ell \neq 1 \Rightarrow i \neq i _ {j} ^ {\ell - 1}) \end{array} \right. \right\},   \min  _ {j \in J _ {-}} \overline {{\boldsymbol {\alpha}}} _ {j} ^ {0} ^ {\top} \overline {{\boldsymbol {x}}} _ {i _ {j} ^ {1}}, \\ \min  \{\tau_ {j} ^ {\ell} - \tau_ {j} ^ {\ell - 1} | j \in J _ {+} \cup J _ {-} \wedge \ell \in [ n _ {j} ] \} \end{array} \right\}
$$
$$
\Delta := \max  \left\{\max  _ {i \in [ n ]} \| \boldsymbol {x} _ {i} \|, \max  _ {j \in [ m ]} \| \boldsymbol {z} _ {j} \|, 1 \right\}.
$$
Assumption 2.  $0 < \varepsilon \leq \frac{1}{4}$  and  $\lambda \leq \left(mn^{9n\Delta^2 /\delta^3}\right)^{-3 / \varepsilon}$ .
The quantity  $\varepsilon$  introduced here has no effect on the network training, but is a parameter of our analysis, so that varying it within the assumed range tightens some of the resulting bounds while loosening others. The assumed bound on the initialisation scale  $\lambda$  is polynomial in the network width  $m$  and exponential in the dataset cardinality  $n$ . The latter is also the case in Boursier et al. [2022], where the bound was stated informally and without its dependence on parameters other than  $m$  and  $n$ .
4 First phase: alignment or deactivation
We show that, for each initially active hidden neuron, if its last-layer sign is positive then it turns to include in its active half-space all training points that were initially outside, whereas if its last-layer