Statistical Guarantees for Approximate Stationary Points of Shallow Neural Networks
Authors: Mahsa Taheri
Fang Xie
Johannes Lederer
Venue: Accepted by TMLR ()

Abstract: Since statistical guarantees for neural networks are usually restricted to global optima of intricate objective functions, it is unclear whether these theories explain the performances of actual outputs of neural network pipelines. The goal of this paper is, therefore, to bring statistical theory closer to practice. We develop statistical guarantees for shallow linear neural networks that coincide up to logarithmic factors with the global optima but apply to stationary points and the points nearby. These results support the common notion that neural networks do not necessarily need to be optimized globally from a mathematical perspective. We then extend our statistical guarantees to shallow ReLU neural networks, assuming the first layer weight matrices are nearly identical for the stationary network and the target. More generally, despite being limited to shallow neural networks for now, our theories make an important step forward in describing the practical properties of neural networks in mathematical terms.

Published in Transactions on Machine Learning Research (12/2025)
Statistical Guarantees for Approximate Stationary Points of Shallow Neural Networks
Mahsa Taheri
mahsa. haw.taaaaarrr@iannmmbuu.....m
Deportment of Mathematics University of Hamburg
Fang Xie
Cangrie@bbbrr.c edu.cc
Guangdong Provincial Key Laboratory IRADS Beijing Normal Hong Kong Boptist University
Johannes Lederer
jobemmees.ldeerrr@vuitttnmme de
Department of Mathematics University of Hamburg
Reviewed on OpenReview: https: oppnrrevee.... net/ forum? id- PNUMLLLLml
Abstract
Since statistical guarantees for neural networks are usually restricted to global optima of intricate objective functions, it is unclear whether these theories explain the performances of actual outputs of neural network pipelines. The goal of this paper is, therefore, to bring statistical theory closer to practice. We develop statistical guarantees for shallow linear neural networks that coincide up to logarithmic factors with the global optima but apply to stationary points and the points nearby. These results support the common notion that neural networks do not necessarily need to be optimized globally from mathematical perspective. We then extend our statistical guarantees to shallow ReLU neural networks, assuming the first layer weight matrices are nearly identical for the stationary network and the target. More generally, despite being limited to shallow neural networks for now, our theories make an important step forward describing the practical properties of neural networks in mathematical terms.
1 Introduction
Statistical thcorics for deep learning usually apply exact, global optima of cortain objective functions [Bartleti 1998 Bauer & Kohler 2019
Kohler & Langer 2021 Lederer 2022a Schmidd-HHieber 2020 Mohades & Lederer 2025: Golestaneh al 2025 But those objective functions cannot be solved explicitly and are highly non-convex, so that in practice, exact, global optimization is least to date-an an open rescarch question, and we can currently expect only approximate stationary points from current (general) algorithms (see Figure I In other words, it unclear whether the known theories have any meaning for the outputs of actual deep-learning pipelines.
Also other parts of machine learning face optimization problems that are challenging to optimize globally and to full precision. Accordingly, some statistical insights have already been established. For example, Bien et al. (2018) 2019) solve non-convex problem in linear regression in "convex" way and develop statistical thcories for their solution. Some insights on the statistical theory of stationary points for (simple) non-convex objectives have already been presented: Loh & Wainwright
(2015) Theorems 1,2) extract statistical guarantees for stationary points of non-convex objectives (allowing for non-convexity in both loss and penalty functions) in a regression-type settings, under a so-called "restricted--strong convexity" condition over the empirical
Corresponding author

---

Published in Transactions on Machine Learning Research (12/2025)
approoet atationary
Objective Fauction
Parameter
Since objective functions in deep learning are usually highly non-convex and cannot be solved explicitly, we only expect approximate stationary points from practical algorithms.
Figure 1:
loss (sce their Display (4)). Loh 2017 Theorem 1) studics the behavior of stationary points of penalized robust estimators in a linear-regression setting. They prove that under local "restricted-strrngg convexity' condition, stationary points within the region of restricted curvature target. Also Elsener & van de Geer (2018 Theorem 1) derive sharp oracle inequalities for stationary points of general non-convex objectives made by non-conver loss plus convex penalty, under restrictive condition called "two point marginal condition" on the theoretical loss. They exemplify their bounds for simple models like robust regression and binary classification. Their condition is kinda similar to the restricted-strong convexity but on the theoretical loss (and not on the empirical loss).
statistically consistent with the
However, it remains unclear how to extend these insights to deep learning, they either apply only to simple models like linear regression rely on specific curvature assumptions, while it is not clear if such curvature properties hold even for simple networks.
This paper develops statistical guarantees for the stationary points of shallow neural networks and for the points in the vicinity them. Strikingly, our statistical rates match the rates of global optimizers up log-terms (Taheri al., [2021] Lederer 2022a Golestaneh al. 2025). Thus, our results establish a mathematical proof of the "empirical fact" that global optimization not necessary in deep learning. This
complements and contrasts studies about the existence non-existence of spurious local minima and saddle
points in both linear and non-linear networks (Zhou & Liang 2018 Fukumizu & Amari
2000 Safran &
Shamir 2018
Lederer
2020
Liu 2022
One of the main challenges in the prools is the complexity, intricacy, and ambiguity of the parameter space of neural networks. To address this challenge, we introduce scaling tricks (Taheri et al., 2021 and use particular arguments from empirical-process theory for regularized objectives. Morcover, in strong contrast to most theory papers, we focus on regression, which is more general and mathematically more challenging than classification. For example, unbounded losses like least-squares cannot be treated (at least not directly) with standard techniques like McDiarmid's inequality (McDiarmid 1989 Lemma. 3.3) Rademacher complexities (Mohri et al. 2018 Chapter 3). Thus, our work also contributes considerably on the technical aspects of deep learning.
Paper contribution The three main technical contribution of this paper are as follows:
We show that every (reasonable) stationary point of regularized shallow linear neural networks and the points nearby generalize essentially as well as the global optima (Theorem and Theorem 2
2. We extend our theories to shallow ReLU neural networks for specific stationary points (Theorem
3. We determine the optimal rates for the tuning parameter across different networks and noise distributions (Theorem
Of course, our thcoretical framework still far from the extremely complex pipelines of modern deep learning. But our paper makes considerable progress in closing the gap between our theoretical understanding and practical experiences. In particular, it (i) st.rengthens the statistical foundations of deep learning and
2

---

Published in Transactions on Machine Learning Research (12/2025)
(ii) gives first mathematically rigorous proof of the empirical finding that (ii.A) approximate and (i..B) local
optimization of neural networks is usually sufficient. in practice.
Paper outline Section states the statistical guarantees for the stationary points of the shallow linear neural network (Theorem and the points nearby (Theorem 2). We extend our theories to shallow ReLU
networks in Section (Thhoorem 3). We support our theories with numerical observations in Section 4 Section provides an overview of related works. We represent some of our technical results in Soction 0 and
exlend our theory for heavy-tailed noise in Sections
7 We conclude our paper in Section B More technical
results, detailed proofs, and discussion different assumptions are given in the Appendix.
Notations We use vec(y, θ) to generate a vector of length Rw-wd from vector E IR and matrix θ E gexd (for generating the vector, we first push the elements of y and then elements of è row by row). We collect first-order partial derivatives (and subdifferentials for RcLU networks) of prodiction risk risk [7, @] and population risk risk [y, θ with respect to the 3 == vec(y, θ) in the gradient vectors Vrisk (7. O E Rw+wod and Vrisk(y, 0] I. wert respectively. We use the notation |. for general vector norm and || for a general matrix norm. We also define Ir/li := ΣØ= hys and |[0|]] := Σ Ei= 10 To reduce the amount of notations, we use some notation slightly differently depending on whether we treat linear or ReLU networks.
2 Statistical guarantees for shallow linear neural networks
Consider inputs 2ı...., In Rdd and corresponding outputs y1.....ya E R that are connected via
Vi f(æ.,] + us
(1)
for unknown target function f Rd - R and unknown stochastic noise Ui...., R. Deep learning is
about using the available data to approximate the unknown target function f by a neural network. We first. focus on linear neural networks, well-accepted toy model for more general deep learning pipelines (Saxe
ct al. 2013 : hence, we consider
x + 7 TOa,
where
(%, E B = O) R" x Rwxd].
We extend this setup to ReLU activation in the following section.
avoid unnecessary digression here, we impose three mild assumptions. The assumptions are by no means necessary and relaxed in the following sections.
Assumption 1 (Model Assumptions).
We assume that:
The target function can be approzimated by such a neural network in the first place: there a pair (y", O*)) E such that 1r" Vlogn n and [(xx]] = T x for all Rd.
2. The x, 's are independent and centered Sab-Gaussian random veclors with independent coordinates.
3. The u, 's independent centered Gaussion random variables with standard deviation and are independent of n, 's.
The first part of Assumption ensures sharp focus statistical guarantees rather than the approximation properties of neural networks, assume that the target function is itself a neural network with reasonably small parameters. A detailed description of the assumption is provided in Scction E of the Appendix; the assumption is relaxed in Theorem 5) Note that the parametrization of neural welworks is ambiguous: there are infinitely many pairs (y*,0) E B that satisfy those conditions citiuttioonscomprreeeeeee to Taheri et al. (2021) Proposition 1); for further reference, we define 3 = vec(>", O*) for a fixed but arbitrary such pair of parameters. The second part of the assumption on the input simplifies our theoretical analysis. Although it may not always strictly hold in practice, it is widely adopted in the literature convenient. modeling device;
3

---

Published in Transactions on Machine Learning Research (12/2025)
see, for example Vershynin 20188: Wainwright 2019 van de Geer (2000). Intuitively, each coordinate of a can be interpreted as feature extracted from the raw data, and independence is then simplifying assumption. Moreover, the mean-zero sub- Gab-Gassiin property is often assumed to capture concentration behavior of feature vectors in high-dimensional models and can be achieved through data preprocessing. The third part of the assumption, once more, simplifies the presentation here; extensions to other types of noise, including sub-Gaussias and sub-cxponential noise are provided in Section
We assume our regression setup (y: R) rather than a classification setup (ui {0, yi (I....,,k) k)) because the unbounded outputs make regression considerably more challenging to analyze mathematically. In other words, our regression results transfer readily to classification. The usual loss function in regression is least squares. In deep-learning practice, however, least squares (and similarly logistic loss in classification) is complemented with dropout (Srivastava et al. 2014 Salehinejad & Valaee 2019), batch normalization (loffe & Szegedy 2015), low-rank approximation (Denil et al., 2013), and so forth, which yield implicit regularization, or least squares is even complemented with explicit regularization directly (Alvarez & Salzmann 2016 Lemhadri et al. 2021 Hebiri et al. 2025). It. is well understood that implicit regularization is related to explicit regularization Lütke Schwienhorst et al. 2024]. Thus, to mimic beep-learning practice, we consider least-squares complemented by (elementwise) l E-regularrizaton:
(v. yT Tox))² + rlIvec(y,, ()1()))))))) })
(5, A) arg (x,0)
(2)
-=1
where r E [0, oo) is a tuning parameter to be calibrated (sce Sardy et al. (2020) for some theory insights). Such estimators are standard in machine learning and statistics [Lederer] 2022b Eldar & Kutyniok] 2012). Despite
f--orm non-smooth, it often poses very little problems in terms of computations (see Friedman al. (2010) Also recently, the nnomm has been effectively used to promote sparsity in neural networks (Lemhadri et al. 2021
As usual, we measure the jin-sample-pprediction risk by
risk br, 0] = 1 Σ (vi > Taar)
with X = (#1...., In)T E Rad and the generalization risk by
risk O] == (x (w y 0x)
with the expectation over a new sample (x.y) ) (that has the same distribution as æ1,..., Bn and yi.... We call B vec(S, θ) stationary point of the objective in equation 2 if it satisfies (erttsckas
vn)
1997
Page 194): (Isener & van de Geer 2018 Equation 6):(Loh & Wainwright 2015 Equation 5)
(Vriskx17, 61) (8 B) + r (B 3) 2 A v vec(y, with (7) B
(3)
for appropriate & E 0133 (where 013 is the subdiffercntial of the regularizer at 3). For an interior point B, our definition of stationary points in equation reduces to the usual vero-subgradient condition.
We call a stationary point / reasonable once |17,, |0]] S Vlogn again to avoid unnecessary complication (we refer to the Appendix Section F for a detailed description of the reasonability assumption). Due to the ambiguity of neural networks, there are infinitely many equivalent stationary and reasonable stationary points; importantly, our guarantees hold for every (reasonable) stationary point and target fr.
We say that a network indexed by (7, (7,0) θ) generalizes well if
risk Ô] 2 riskiy",
that is, the network generalizes essentially well as the best network. In the following, show that not only the "statistical" network indexed by (6, Q) but also every "practical" network indexed by reasonable stationary point (5, @) of the objective function in equation generalizes well.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Moreover, we call the total number of parameters in the network p == w + w d the problem's effective dimension and
log(np)
Fore (logn)
(4)
the oracle tuning parameter, where E (0, 00) is constant that depends only on the distributions of the inputs and noise. It has been shown that Fore is indeed optimal tuning parameter of equation 2 in some sense (Taheri et al. 2021).
We then get the following result for a new sample pair (ær with the same distribution as 31,-. Dn and V1....1 Vn-
Theorem 1 (Statistical Guarantees for Reasonable Stationary Points of Shallow Linear Networks). Under
the Assumption ! any reasonable stationary point (5. θ) of the objective function in equation with 2 Fore satisfies the risk bound
risk(Fr, O] S riskly", + 5r /lylog
(5)
with probability at least 1 - 1/2n. 1f Toro, the bound becomes
risk 6] s risk(y", v(log 3 log(np)
(6)
Theorem proves the fact that for properly chosen tuning paramcter and large enough sample sizes, any reasonable stationary point of equation 2 generalizes essentially well as B". Our results essentially have the same rates as the ones in the literature (Taheri et al., 2021 Theorem 3); Lederer 2022a Proposition 3), who prove that the prediction risk is at most of order (LL/2) 2/ 1/2- log(p) log(m)/ /n)) for l--reuulrried neural networks with depth L and p parameters. However, in stark contrast to previous results, our theorics apply to all reasonable stationary points (including saddle points) rather than to the global optimum of the objective function only, Although works like Kawaguchi (2016) and Zhou & Liang (2018) argue about the absence of local minima in lincar networks, saddle points still exist in linear ncural networks (see Zhou & Liang) (2018) Theorem 2)). Furthermore, saddle points continue to pose challenges: Lee et al. (2019) demonstrate that gradient-based algorithms can escape strict saddle points, but non-strict saddle points are problematic and also exist in linear neural networks in general (Zhou & Liang] 2018 Paragraph following their Theorem 2). We refer to our illustrative Example | (in the Appendix) to clearly illustrate the presence of sub-optimal critical points in our considered setup. Also, a recent study by Acchou et al. (2024) demonstrates that for shallow linear neural networks with least squares loss, all saddle points are strict under some assumptions (see Achour al. 22024, Assumption 1).
To emphasize the significance of using regularized objectives, it's worth mentioning that the rate of ordinary least-squares in linear regression is Oddnn, where d gives the number of parameters and n the number of data examples (Lederer 2022b Equation 1.5). But for higl-dimensional settings with d > n, least-squares are prone to overfitting, so regularization can be employed for improvement. For example, lasso with sufficiently large tuning paramoter (in lincar regression) gives predictions bounds at most boundod by vlog(d)/n (Lederer 2022b, Page 174). Also, a different prediction bound for lasso called "power-twe bound' is presented in Lederer (2022b, Page 188) that holds under strong conditions but it is far from the context of this paper. Overfitting is even more problematic for complex models like neural networks with huge number of parameters p. The focus has just shifted to networks involving sparsity to improve prediction bounds from p/n to /log(p)/m, which also appears in our results (see equation| 6 for example).
Note that in finite time, stationary points can be computed just approximately using gradient- based algorithms. Now; we exlend our results Theorem | to the points that are close but not necessarily equal to a stationary or points. We define pair (5, 0) as T approxiiaa stationary point if it satisfies
rrikk 15. θ + r r1 riskx 17, θ 33r-11111
-113111 S
(7)
for J E [0, o0). Our definition of approximate stationary points in equation 7 is closely related to the typical definitions in the literature that impose some bounds the norm of the gradient. vectors (see Appendix
5

---

Published in Transactions on Machine Learning Research (12/2025)
Section G for detailed description). Employyng gradient-based algorithms (in finite time), we expect to get close to stationary point in the sense that B = B (Ghadimi & Lan 2013 Lei et al. 2019). Then also || ~ [| which means that an approximation of reasonable stationary point is also reasonable once T is small enough. Then, we extract statistical guarantees for every practical network indexed by an approximate-reasonui
stationary as follows:
Theorem 2 (Statistical Guarantees for Approximate Stationary Points of Shallow Linear Networks). Suppose that (5, Q) is ap-poorrimate stationary point and that the conditions of Theorem 1 are satisfied. Then, have
risk f,, O I risk(y", L6e + 8r /loloog +
(8)
with probability at least 1 n. If = Tore, the bound becomes
log(np)
risk O] < riskly", O*1 + (logn)²
+T.
(9)
The bounds match the earlier ones with only two small differences: 1. summand is added to our statistical bounds and 2. the factor in equation [ is replaced by factor of 8 in equation g Let's note that gradient-ban based algorithms with sufficiently many steps O(n²) ensure that t < 1/vn (Ghadimi & Lan 2013 Theorem 2.1). We refer to our Appendix Section G for more details regarding the dynamical accessibility of approximate stationary points. Theorem 2
might look like a simple extension of Theorem 1 but the fact. that equation 7 involves the inn-sample-prrddiction risk and the sparsity factors makes the proof considerably more involved.
3 Statistical guarantees for shallow ReLU neural networks
This section generalizes our theories in Section
to shallow ReLU neural networks of the form
7J (Ox))
R x RRud). The activation function a-)) corresponds to the well-known ReLU defined as o(=)= = (0 ........ max(0, zw)) for 2 E R", which its efficacy has been extensively studiod (Pan & Srikumar 2016 Raghu et al. 2017). We then approximate the unknown target function f in equation employing shallow ReLU neural networks. For simplifying the proofs, we assume in this section that d = w that implies matrix θ to be squared. We then consider least-squares complemented by L Urregularization for shallow ReLU neural networks:
for (7, Q) E B ((,,))
(2, 0) E arg min
" (y: - -To(O))))))) + rrveecly, 011 (7,)) 1=1
(10)
Assumption 2 (Model Assumptions (ReLU)).. We assuine that the largel function can be approcimated by such a neural network, that is, there is a pair (y", (r)= E B such that |Y 1|0*1] I Vlogn n and active rows in @" are approrimately perpendicular to each other and that = TT (f+xx) for all Rd.
The term active rows in a matrix O are approximately perpendicular to each other in assumption above means, for any two distinct active rows θ,,- and Oy (where ,,,., * 0), their inner product is negligible, that is (o).., Qy..) x 0.0. Assumption
2 stipulates that the target function is itself shallow ReLU neural network with reasonably small parameters and that the active rows of the first layer are approximately orthogonal. Versions of these assumptions are very common in the literature (Hardt & Ma. 2016 Bartlett et al. 2018b); we discuss this assumption further in the paragraph following Theorem We then define the cnt-samplle-pprddictii and generalization risk for shallow ReLU noural networks as (we employ the same notation as used in the linear case)
riskxl7,,0) = 1 Σ (y:= 7 (0x)))

---

Published in Transactions on Machine Learning Research (12/2025)
and
risk 0] = -() [1v (T
((x)))
We then get the following result for new sample pair y) with the same distribution 21,..., En and M1...... Mn-
Theorem (Statistical Guarantees for Reasonable Stationary Points of Shallow ReLU Networks). Under the second and third parts of Assumption | and Assumption & any reasonable stationary point (5, Q) of the objective function in equation 10 where active rows of é are approzimately perpendicular to each other and that off-diagonal elements of 00* and Ô*ÔT are approzimately zero (1(00) ~ [(0 3,8وl 2 0.0 for
j # j') with r 2 Tore satisfies the risk bound
risk(7,6) & [akk(n",']++ [0r]+5r5/00 Vlonn
(11)
with probability at least 1 - 11/2n.
Note that Theorem 3 is an extension of our Theorem [orr shallow ReLU neural networks under the assumption that the active rows of the first layer weight matrix (for stationary point and the target) being approximately orthogonal (one simple example is near-identity matrices). Orthogonal weight matrices is needed mainly for studying Hessian behavior for shallow ReLU network's (proof of Proposition 2 Step 2). Employing this assumption, we can prove that for cases with small correlation between rows and with Gaussian input, can find a closed-form solution (approximately) for ( than it seems as previous works have studied variants of this assumption for neural networks from different perspectives: for example, HHartt & Ma (2016) shows that certain net have a global minimum close to the identity parameterization. They study the expressiveness of Residual Networks under the assumption that enough neurons are available (Hardt Ma 2016 Thcorem 3.2). Interesting is that, since our rates grow just in logp. our framework is perfectly fit for such wide networks. Additionally, Bartlett et al. (2018a) explore the representation of smooth functions as compositions of near-identity functions, highlighting implications for deep network optimization. Bartlett et al. (2018b) prove the rate of convergence of gradient- optimization under identity initialization for deep linear networks. Li & Yuan (2017) analyze the convergence of stochastic gradient descent for shallow ReLU networks, with nearly identity initialization; they state that "(ReLU) networks with small average spectral norm already have good performance." Altogether, we believe that our assumption makes sense not only from an expressivily standpoint (Hardt & Ma 2016 Theorem 3.2) but also regarding the optimization landscape (Li & Yuan 2017). Yet, of course, it would be interesting to study the subtletics even further. While studies demonstrate the existence of local minima and saddle points
Our Assumption
2 is weaker
in ReLU networks (Fukumiizu & Amari 2000 Safran & Shamir 2018 Yun et al. 2019) we argue that some of those suboptimals still yield satisfactory results. Fassentially, Theorem 3 suggests that for a sulficiently large tuning parameter, the optimization explores locally well-curved network spaces in the vicinity of specific stationary points, such that any stationary point gencralizes as effectively as a global minimum. In fact, our work concerns local curvature around the ground truth in neural networks, which we believe valuable given the infinite number of such ground truths in neural networks, while globally favorable curvature is far from practical reality in deep learning. We employ our result in Proposition and Remark [ proving our Theorem g Also, an extension of Theorem 2 to shallow ReLU networks can be obtained by combining Theorem 3 with additional machinery from empirical process theory, following the same line of reasoning
in the proof of Theorem 2 However, we omit this extension here to avoid redundancy. Also, we conjecture that our main theories can be extended to deep neural networks (see our sinulations in Section D), provided that suitable local curvature properties of the corresponding networks can be established. This presents an intriguing direction for future rescarch.
Further discussion of our Assumption 2 Our results suggest that low correlation between the rows of the first-layer weight matrix θ is desirable, it leads a well-conditioned Hessian and better generalization. This observation is closely related the benefits of random initialization: for large d, random Gaussian weights yield nearly orthogonal rows with high probability (see Vershynin| (2018) Remark 3.2.5)). However, orthogonality is not only needed at initialization but also for the estimator θ after training, which motivates arguments ensuring that training preserves this structure. Related studies show that fixing the first layer at.
7

---

Published in Transactions on Machine Learning Research (12/2025)
its random initialization while only training the last layer can still achieve good generalization [Rosenfeld] & Tsotsss 2019), suggesting that there exist network configurations where the first-layer rows form an approximately orthogonal system, leading to favorable error bounds. Finally, while some of the literature attributes low-rank structure in shallow networks to strong correlations among rows (Kou et al., 2023), in our norm-one regularized set setting low rank instead arises through sparsily: many rows might become inactive, while the surviving rows remain diverse and nearly orthogonal, which is enough for our results to hold. One can also considers group lasso to offer an alternative means of promoting structured sparsity. This alternative path to low-rank structure avoids redundancy, preserves conditioning, and further explains why such solutions generalize well.
4 Numerical observations
We provide here some numerical observations to clarify theories of Section and Section 3 We minimize a least-squares complemented by f1-regnlarization for shallow neural networks with linear and ReLU activation functions. We set our tuning parameter on the order of log(np)/ /vn based on our experiments. We consider neural networks with d= = w = 10, that are trained over 500 and tested over 300 data sample generated from a standard normal distribution and labeled by a sparse-target network (having the same structure as the considered model) plus a Gaussian noise. Note that here, we train the networks in a finite time, that means, trained networks are just an approximation of a stationary point (due to the non-convexity). We report the relative training error and the relative test error for potential global optimum, an approximale stationary point, and randomly generated network (a network with randomly assigned weights) for linear and ReLU networks in Table [ that is, the training (test) error of the "approximate stationary point" divided by the training (test) error of the "potential global optimum" (for the corresponding network). Potential global optimum and approximate stationary point (for each setting, linear or ReI.U) are reached over multiple times of training on a fixed data set and assigned by the trained networks with the lowest and highest training error, respectivcly. More precisely, we do the optimization (solving equation and equation I0p from multiple, diverse initial points (1000 times). We use Py'forch's 's default initialization, where weights are drawn from uniform distribution in [-1/Vp.1/ /pl, with p denoting the number of input features to the layer (sce our results with different initialization methods in our Section D). This helps explore different regions of the search space and increases the chances of finding different local and global optimum. Note that there are infinitely many critical points for neural networks in view of the network's rescaling properties. We use stochastic gradient descent with a small convergence threshold to ensure that the optimization process does not stop early. We analyze the distribution of the reached training errors (over the 1000 different optimization runs with random initialization). For this, we divide the training errors into two clusters via k-means. Then, we do t-test over the training errors in the two classcs. The t-test reveals statistically significant difference between the training errors in two groups (Pvalue < 0.0001), which supports the claim that the "potential global optimum" and "approximate stationary points" differ, that is, the approximate stationary points are not just other global optima. We then report the parameters that lead to the lowest training error as a "potential global optimum" " and the parameters that lead to the highest training error as "approximate stationary point". We reference to Figure 3 in the Appendix Section D for a graphical view of convergence in training. Results reveal that the test crror for a potential global optimum and an approximate stationary point are very close in both linear and ReLU networks (relative errors for approxinate stationary points are close to one for both linear and ReLU networks). Also note that the reported numbers in Table are just relative errors to compare between training and test performance of specific network so, comparison between the performance of linear and ReLU networks here is not meaningful.
These observations reveal that: First, global optimization for neural networks is far reaching even for very simple neural networks. Sccond, very practical outputs in deep learning (approximato stationary points) can still generalize well-for for linear networks and beyond. We provide the similar result for a larger network in Table 2 and more detailed experiment explanations in Appendix Section D
8

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Relative training error and test error for trained shallow neural networks (with = 10, w = 10) with linear and ReLU activations in a potential global optimum, an approximate stationary point, and randomly generated network.
Linear
ReLU
Training Error
Test Error
Training Error
Test Error
Potential Global Optimum Approximate Stationary Point
1.000
1.000
1.000
1.000
1.001
1.001
1.003
1.004
Randomly Generated Network
79618.240
58198.240
2120.060
1980.060
5 Related literature
Another interesting direction is studying optimization landscape of non-convex objectives in doep learn-
ing (Eltehhari, 2020; Hardt & Ma. 2016 Lederer 2020: Zhou & Liang) 2018 Zhang et al. 2016; Bah et. al., 2022. Trager al.) 2020). Yun al. (2017) study the optimization landscape of deep and linear neural networks. They extract necessary and sufficient conditions for critical point to be the global optima of the least-squares loss under some assumptions (input dimensions upper bounded by the number of data examples, XXT and YXT have full rank). Kawaguchi (2016) Theorem 2.3) proves that for deep and linear neural networks and under some assumptions (XX and XY have full rank), every local minimum is a global minimum and every critical point that is not global minimum is saddle point. They also prove that the same results hold for nonlinear-neure networks but under unrealistic assumptions (Kawaguchi 2016 Corollary 3.2). Zhou & Liang 2018 Theorem 2) also prove that lincar neural networks with least-squares loss have no spurious local minimum. Haeffele & Vidal (2017) established sufficient conditions ensuring that any local minimum of a non non-convex factorization problem is also global minimum. Moreover, they showed that when the factorization is parameterized with sufficiently large factors, one can always reach global minimizer from any feasible initialization using purely local descent methods. Nguyen & Hein (2017) studiod the loss surface of deep and wide neural networks and proved that, under mild overparameterrvaalin conditions, every local minimum is also global minimum. Nguyyen & Hein (2018) analyzed the optimization landscape and expressivity of deep convolutional neural networks. They established conditions under which all local minima are globally optimal and characterized how network depth and architecture affect the expressivity of the network. But in general, the absence of spurious local minima is rejected for non-linear networks (Fukumizu Safran & Shamir 2018).
& Amari 2000
More broadly, non-convexity and computational problems of neural networks have widely been studied in recent years from different perspectives, including optimization algorithms (Lovas et al. 2020 Bach & Chizat,
2021 theory of overparameterizee networks (Chizat & Bach 2018), and hyperparaaeeter calibration (Yang) et al. 2021). Liang et al. (2018) studied modified neuron activation for arbitrary deep neural network in binary classification proving that no bad local-min exists (see also Sun et al. (2020)). ). Choromanska et al.
(2015) studicd the loss landscape of neural networks from a statistical physics perspective, establishing a connection ween neural networks and spin-glass models.
6 Technical results
This section provides technical results needed for proving our main theories. All the proofs as well more related auxiliary results are deferred to the Appendix.
Additional notations For vectors 8 = vec(y, Q) E IR" and a = (ai...., aw) E R with a, * 0 for all j (1....,.), wh, we define Ba == vec(%.: θ₂) E Rp as a rescaled version of / with (ya), == 7j oy and (0a)js = / for all j E {l...., w) and E {1.....dl d}. We tabulate the second order partial derivatives (subdifferentials) of riskly, θ) with respect the B = vee(y, Q) a matrix called Virisk θ] E R We use emiul-] to generate the smallest eigenvalue of a matrix. We use the notation to generate vector of zeros.
9

---

Published in Transactions on Machine Learning Research (12/2025)
6.1 Technical results for shallow linear neural networks
Here, we provide technical results that are essential for proving our main theories for shallow linear networks but might also be of interest by themselves. We first study the behavior of the Hessian matrix for shallow linear networks in a rescaled network as follows:
Proposition 1 (Hessian Behavior for Shallow Linear Network). Suppose Assumption 1 is verified and that (7, ) E with 00T invertible. Let a = [(aa) TT E RP be vector with ||l| = 1, a E R", and a² Rwd.d If a or a² = 0, we have for all R ((0
a V' Oala 2 0;
Otherwise, above inequality holds for all a = (1/c, 1/c) e
R with oo) such that
è 2 a ? 4 aaa
a² |2
2
2
Emin θ0
aa aa
Note that if a = 0 or a² 0, the quadratic product on the Hessian matrix (in rescaled network with parameters (Ya Oa)) non-negative for all a, otherwise, it is non-negative just for with large enough Proposition 1 employed for the proof of Theorem
Lemma (Empirical Processes). Under the Assumption it holds for each reasonable stationary point 3 = vec(F, Q) of the objective function in equation that
[Vriskx 6- Vrisk @1) (8--3) 3) <ToeelB -
Fore
2n
with probability at least 1 - 1/2m, where Fore is the oracle tuning parameter defined in equation 4
The result above establishes a bound for the absolute difference between Vriskx 15, + and Vrisk[fy, θ] for every reasonable stationary point (7, +) B for shallow linear networks (a similar result can also be reached
for shallow ReLU networks; see Remark 1 We employ Lemma choosing the optimal tuning parameter for the objective function equation
6.2 Technical results for shallow ReLU neural networks
Now, we study the behavior of the Hessian matrix for shallow ReLU networks in a rescaled network. Since ReLU networks are non-difeeentiaale at zero, we employ subdifferentials in this section (instead of partial derivatives) using the same notation as used for linear nelworks. We suppose that with (Qæ), 0, where {L...., w) then we have
Proposition (Hessian Behavior for Shallow ReLU Networks). Suppose Assumption and the second and third parts of Assumption 7 are verified, and that (7, 0) B with active rows of O being approzimately perpendicular. Let = [[a)] T (TT)TTT E Rp be vector with |al2 a' R", and a² Rwdd If a² 0, we have for all a E T \ (0)
a T rirkksk/ Oaja 0;
Otherwise, above inequality holds for all a = (1/c, 1/c) R with E [1, oo) large enough.
Note that Proposition 2 is an extension of our Propostion for ReLU networks, that holds under an extra assumption over the first layer weight matrix.
Remark 1 (Empirical Processes for Shallow ReLU Neural Networks). Under the Assumption 2 and the second and third parts of the Assumption 1 almost the same bound (up to a conslant and log factor) stated in Lemma can hold for each reasonable stationary point / = vec(āi, θ) of the objective function in equation 10
As stated in Remark [ the tuning parameter for ReLU networks can be calibrated similarly to linear networks (although there's potential for improvement, omit that to avoid unnecessary complication.)
10

---

Published in Transactions on Machine Learning Research (12/2025)
7 Heavy-tailed noise
This section puts a focus on heavy-tailed noise. We limit ourselves to linear networks for simplicity, but the same techniques also work in the ReLU case. More generally, this section illustrates the much larger generality- and technical difficulty of our regression setup as compared to the common classification setups, which are bounded by design.
Definition 1 (Tails). Let I + R be an increasing function. random variable z if
The function I caplares the right toil of the
P( 1) z exp(-I((), AI (0,
In this section, we assume that noise is heavy-tailed, having a right tail as defined in Definition with (t) = cat //a for ca E (0, oc) (for example a = for sub-gaussian noise and a = for sub-exponential noise). We also define
Fore,a == mllogn)3/2 (log(np))"
(12)
Vn
where E (2, oo) and v,c E (0, x)) are constants depending the distributions of inputs and noise. Now, we extend our results in Theorem for heavy-tailed noise.
Theorem (Statistical Guarantees for Reasonable Stationary Points for Heavy-tailed Noise). Under the first two parts of Assumption 1 any reasonable stationary point (5, θ) of the objective function in equation with r 2 Tore,a satisfies the risk bound
risk θ] s risk(y" "00))"",0+]++ 5r 5lglog
(13)
with probability at least 1 - 1/n. If Fore,a, the bound becomes
risk [R, d] s riskly", 001 v(log n)² (log(1p))*
(14)
The above results show that our theories still hold under heavy tails; the bounds and the optimal tuning parameter (see Theorem [] then simply entail a power a (depending on the noise) for log(np). This an important step forward, as usual inputs to neural networks (images, text, ...) are often very noisy.
8 Discussion
We have established statistical guarantees for approximate stationary points of regularized shallow linear neural networks. We have then extended our theories to shallow ReLU neural networks under the assumption over the first layer weight matrix. Despite being limited to shallow networks, our theory is a large step forward in four ways: 1. Several papers consider the existence or non-cxistence of critical points that are not global iml in linear neural networks under certain assumptions. In contrast, our theories apply regardless of whether such local minima or saddle points exist in the objective under consideration. to ReLU neural networks not only provide thcorctical insights but also highlight the importance effective initialization, such near-identity initialization, for ReLU networks (Hardı Ma 2016). 3. While works like Bach & Chizat (2021) consider convergence of specific optimization algorithms in deep learning, our results are agnostic to the optimization algorithm and do not require infinite-width networks, making findings more general. 4. And finally, our new statistical approach inspired by high-dimensional statistics is expected to spark further progress in the mathematical understanding of deep learning.
Our extensions
Acknowledgments
J. Lederer and M. Taheri are grateful for partial funding by the Deutsche Forschugggemmcnschaft (DFG, German Research Foundation) under project numbers 541176257 and 520388526 (TRR399).. The authors thank Ali Mohades for constructive discussions at various stages of the project. F. Xie is supported in part by the Guangdong Basic and Applied Basic Rescarch Foundation (No. 2023A1515110469), the Guangdong Provincial Key Laboratory IRADS (No. 202281212010006, and the grant of Higher Education Enhancement Plan of "Rushing to the Top, Making Up Shortcomings and Strengthening Special Features" (No. 2025KTSCX186;
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
M. Achour, F. Malgouyres, and S. Gerchinovitz. The loss landscape of deep linear neural networks: second-order analysis. J. Mach. Learn. Res., 25(242):1 76, 2024.
J. Alvarez and M. Salzmann. Learning the number of neurons in deep networks. In Proc. NIPS, pp. 2270-2278, 2016.
Y. Arjevani, Y. Carmon, J. Duchi, D. Foster, N. Srebro, and B. Woodworth. Lower bounds for non-convex stochastic optimization, Math. Program., pp. 1-50, 2022.
F. Bach and L. Chizat. Gradient descent on infinitely wide neural networks: Global convergence and generalization. arXiv:2110.0805, 2021.
B. Bah, H. Rauhut, U. Terstiege, and M. Westdickenberg, Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers. Inf. Inference, 11(1):307-3533 2022.
M. Bakhshizadeh, A. Maleki, and V. de la Pena. Sharp concentration results for heavy-tailed distributions. 0rXiv:2003.13819 2020.
P. Bartlett. The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network. IEEE Trans. Inform. Theory, 44(2):525-536, 1998.
P. Bartlett, S. Evans, and P. Long. Representing smooth functions as compositions of near-identity functions with implications for deep network optimization. arXiv:1804.00111 2018a.
P. Bartlett, D. Helmbold, and Long. Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks. In Proc. ICML, pp. 521-530. PMLR, 2018b.
B. Bauer and M. Kohler. On deep learning a remedy for the curse of dimensionality in nonparametric regression. Ann. Statist., 22222 2285, 2019.
Bertsckas. Nonlincar programming. J. Oper. Res. Soc., (8(3):334-3344 1997.
D. Bertsekas, A. Nedic, and A. Ozdaglar. Convex analysis and optimization, volume 1. Athena Scientific, 2003.
J. Bien, I. Gaynanova, J. Lederer, and C. Müller. Non-convex global minimization and false discovery rate control for the trex. Comput. Gruph. Statist., 27(1):23 33, 2018.
J. Bien, I. Gaynanova, J. Lederer, and C. Müller. Prediction error bounds for linear regression with the trex. Test, 28(2):451-4P5 474, 2019.
P. Bühhmaan and S. Van De Geer. Statistics for high-dimennsional data: methods, theory and applications. Springer Science Business Media, 2011.
Y. Carmon, J. Duchi, O. Hinder, and A. Sidford. Accolerated methods for nonconvex optimization. SIAM J. Optim, 28(2):1751-177 2018.
L. Chizat and F. Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. In Proc. NIPS, volume 31, 2018.
A. Choromanska, M. Henaff, M. Mathieu, G. Arous, and Y. LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. PMLR, 2015.
M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. De Freitas. Predicting parameters in deep learning, In Proc. NIPS, pp. 115821156, 2013.
Y. Drori and O. Shamir. The complexity of finding stationary points with stochastic gradient descent. In Proc. ICML, pp. 2658 2667, 2020.
12

---

Published in Transactions on Machine Learning Research (12/2025)
A. Eftekhari. Training linear neural networks: non-local convergence and complexity results. In Proc. ICML, pp. 2020.
Y. Eldar and G. Kutyniok. Compressed sensing: theory and applications. Cambridge Univ. Press, 2012.
A. Elsener and S. van de Geer. Sharp oracle inequalities for stationary points of nonconvex penalized M-ostimators. IEEE Trans. Inform. Theory, 65(3):1452-1472, 2018.
J. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw., 33(1):1, 2010.
K. Fukumizu and S. Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural nelworks, 13(3):317-32) 2000.
S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim, 23(4):2341 2368, 2013.
P. Golestaneh, M. Taheri, and Lederer. How many samples are needed to train a deep-relu neural network?
In Proc. ICLR, 2025.
B. Haeffele and R. Vidal. Global optimality in neural network training. In Proc. CVPR, pp. 7331 7339, 2017.
M. Hardt and T. Ma. Identity matters in deep learning, crXie:1611.04200 2016.
M. Hebiri, J. Loderer, and M. Taheri. Layer sparsity in neural networks. J. Statist. Plann. Inference, 234: 106195, 2025. ISSN 0378-3758.
S. Tofle and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. 1CML, pp. 448 456, 2015.
K. Kawaguchi. Deep learning without poor local minima. arXit:1605.07110, 2016.
M. Kohler and S. Langer. On the rate of convergence of fully connected deep neural network regression estimates. Ann. Statist., 29(4))22312 2021.
Y. Kou, Z. Chen, and Q Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks on orttoy-orthogonal data. In Proc. NIPS, volume 36, pp. 30167-30221, 2023.
J. Lederer. No spurious local minima: on the optimization landscapes of wide and deep neural networks. 2020.
J. Lederer. Statistical guarantees for sparse deep learning. arain:2212.05442 2022a
J. Lederer. Fundamentals of bingh-Dieensionll Statistics: with exercises and R labs. Springer Texts in Statisties, 2022b.
J. Lee, I. Panageas, G. Piliouras, M. Sinchowitz, M. Jordan, and B. Recht. First-order methods almost. always avoid strict saddle points. Math. Program., 176(1):311 337, 2019.
Y. Lei, T. HU, G. Li, and K. Tang. Stochastic gradient descent for nonconvex learning without bounded gradient assumptions. IEEE Trans. Neural Netw. Learn. Syst., 31(10):4394 4400, 2019.
I. Lemhadri, F. Ruan, T.. Abraham, and R. Tibshirani. Lassonet: A neural network with feature sparsity. J. Mach. Learn. Res., 22(127):1 29, 2021.
Y. Li and Y. Yuan. Convergence analysis of two-layer neural networks with relu activation. In Proc. NIPS, volume 30, 2017.
S. Liang. R. Sun, J. Lee, and Srikant. Adding one neuron can climinate all bad local minima. Proc. NIPS, 31, 2018.
13

---

Published in Transactions on Machine Learning Research (12/2025)
B Liu. Spurious local minima common for deep neural networks with piecewise linear activations. 1EEE Trans. Neural Netw. Learn. Syst., 2022.
P. Loh. Statistical consistency and asymplotic normality for high-dimeensional robust. M-estimators. Ann. Statist., 45(2):866 896, 2017.
P. Loh and M. Wainwright. Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima. J. Mach. Learn. Res., 16(1):559 616, 2015.
Lovas, I. Lytras, M. Rásonyi, and S. Sabanis. Taming neural networks with tusla: non-convex learning via adaptive stochastic gradient langevin algorithms. arXiv:2006. 14514, 2020.
B. Lütke Schwienhorst, L. Kock, N. Klein, and D. Nott. Dropout regularization in extended generalized lincar models based on double exponential familics. In ECML PKDD, pp. 320-336, Springer, 2024.
C. McDiarmid. On the method of bounded differences. Surv. Comb., 141(1):148-18) 188, 1989.
A. Mohades and J. Lederer. Cardinality sparsity: Applications in matrix-matio multiplications and machine learning. TMLR, 2025.
M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning. MTT press, 2018
Q. Nguyen and M. Hein. The loss surface of deep and wide neural networks. In Proc. ICML, pp. PMLR, 2017.
Q. Nguyen and M. Hein. Optimization landscape and expressivity of deep cmns. In Proc. ICML, pp. 8730-3733 PMLR, 2018.
X. Pan and V. Srikumar. Expressiveness of rectifier networks. In Proc. ICML, pp. 1427-2433 PMTR, 2016.
M. Raghu, B. Poole, Kleinberg, S. Ganguli, and J. Sohl-Dickstein. On the expressive power of deep neural networks. In Proc. ICML, pp. 2847 2854. PMLR, 2017.
A. Rosenfeld and J. Tsotsos. Intriguing properties of randomly weighted networks: Generalizing while learning next to nothing. In Proc. CRV, pp. 9-16, 2019.
I. Safran and O. Shamir. Spurious local minima are common in two-layer relu neural networks. In Proc. ICML, pp. 4433 4441. PMLR, 2018.
H. Salehinejad and S. Valaee. drongud; regularization method for training and compression of deep neural networks. In ICASSP, pp. 3602-3606. IEEE, 2019.
S. Sardy, N. Hengartner, N. Bonenko, and Y. Lin. What needles do sparse neural networks find in nonlinear haystacks. arXin:2006.0444 2020.
Saxe, J. McClelland, and S. Ganguli. Exact solutions the nonlinear dynamics of learning deep linear neural networks. arXic:1312.611 61200 2013.
J. Schmidt-Hiiber. Nonparametric regression using deep neural networks with relu activation function. Ann. Statist., 48(4):1875 1897, 2020.
N. Srivastava, G. Hinton, A. Krizhevsky, 1. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1958 2014.
R. Sun, D. Li, S. Liang, T. Ding, and R. Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37(5):95-108, 2020.
M. Taheri, F. Xie, and J. Lederer. Statistical guarantees for regularized neural networks. Neural Networks, 142:148 2021.
M. Trager, K. Kohn, and Bruna. Pure and spurious critical points: a geometric study of linear networks. Proc. ICIR, 2020.
14

---

Published in Transactions on Machine Learning Research (12/2025)
S. van de Geer. Empirical processes in M-estimation. Cambridge Univ. Press, 2000.
S. van de Geer. Estimation and testing under sparsity. Springer, 2016.
R. Vershynin. High-dimensional probability: an introduction with applications in data science. Cambridge Univ. Press, 2018.
M. Vladimirova, S. Girard, H. Nguyen, and J. Arbel. Sub-weibull distributions: Generalizing sub-gaussian and sub-exponential properties to heavier tailed distributions. Stal, 9(1):e318, 2020.
M. Wainwright. High-dimensional statistics: non-asymptotic viewpoint, volume 48. Cambridge Univ. Press, 2019.
W. Wang and N. Srebro. Stochastic nonconvex optimization with large minibatches. In Algorithmic Learning Theory, pp. 857 857-882, 2019.
G. Yang, E. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Proc. NTPS, volume 34, pp- 17084-17097, 2021.
C. Yun, S. Sra, and A. Jadbabaie. Global optimality conditions for deep neural networks. arXic:1707.0044 2017.
C. Yun, S. Sra, and A. Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. Proc. ICLR, 2019.
Y. Zhang, Lee, and M. Jordan. L f-rreulaaried neural networks are improperly learnable in polynomial time. In Proc. ICML, pp. 993 993-1001, 2016.
Y. Zhou and Y. Liang. Critical points of linear neural networks: Analytical forms and landscape properties. In Proc. ICLR, 2018
A Appendix: Example and auxiliary results
Here we provide an illustrative and simple example to clearly show the existence of sub-optimal critical points for the regularized objective functions (equation 2 and equation I0] with linear and ReLU activations.
Example 1 (Existence of sub-optimal critical points for regularized shallow networks). Let consider a toy linear shallow neural network with just two neurons (a1, (2), and consider the loss function f(a),a)) (X) Ei=(a - 2 /2++ |an]+|a2l- Then, we suppose two trwining samples (I1 2, = 2) and (x2 4, 922 1) non-conver, including local and global minimum
that makes the objective function min((a,ee)) f(an,a)) (a) and saddle point. One can confirm that A = (a1 O, 02 =0) is a local min with fA = 2.5, while A' = (a) R 0.55, 02 2 0.55) is a global min with JA R 2.1 (see the left panel of Pigure 2). This simple exomple illustrates that there are critical points even for simple regularized linear neural networks that are not global optima in our considered setup. Note that if the optimization algorithum (for example gradient descent) starts with weight initiolization close to zero, is high likely that we stuck in the vicinity of the local min (0, A similar example also holds for ReL. networks (see the right panel of Figure!
Here we provide more technical results that are used prove our main theorems.
First, we derive uniform bound on the absolute difference between Vrisk (7, O] and Vrisk(y, θ] for linear shallow networks. We use the notation ||0| mI...(),.....) Σd= 10,ll. Lemma 2 (Uniform Bound on the Difference Between Vrisk (n, O] and Vrisk(y, Q] for Linear Networks).
Under the Assumption 1 it holds for each t, n. c E (0, oc) and B E Cy == (B vec(y, θ) RP | B11
n and To T 7" TT 07, S {) that
(Vriskx by, 0] Vriskly, 01) (8" -0--)a) |3) s 2tm(n + max( loo, 110 c)
sup BEC..
15

---

Published in Transactions on Machine Learning Research (12/2025)
Figure Non-convex objective function min .2) a(an,a) [o= - + |ai|+ + |az| for two training samples {æ] 5/1 = 2) and (x2 4, 1/2 1) includes critical points that are not global optima. The left panel illustrates the objective for linear activation function, and the right panel shows the objective for the ReLU.
32)
with probability at least 1 4dpp exp(-kn min(t///2tt/)) with constants v, K E (0, oo) depending only on the distributions of the inpuls and noise.
The set Cp. contains all parameters in a neighborhood of 8h; in particular, the bound applies to 8 voc(y", O*) wishlf-without any further assumption on 8r. The lemma the main ingredient of our proof for Lemma
We also derive uniform bound the absolute difference between riskx/y: (y, θ] and risk(y, O] (for linear shallow networks.)
Lemma 3 (Uniform Bound on the Difference Between riskxly. θ] and riskly, θ] for Linear Networks). Suppose Assumption 1 is verified and that ()) 1 9* x 1 I e' for an e' E (0). Then, we have for each t E [0, xo) that
risk (7, θ| risk(y, θ] S t(1 + +4Ve)
sup
(x,0)E
min with probability at least 1 18d monnxxmmin((t//// the distributions of the inputs and noise.
with constants v,K E (0, 0o) depending only on Lemma 3 is the main ingredient of our proof of Theorem
Then, we derive a lemma studying the invertibility of the line segment between two matrices. This lemma is employed in the proof of Theorem
Lemma (Invertibility of the Line Segment Betwoen Two Matrices). Let's define H (A+))))++
for A, C g xd' with w' d' and t E (0, 1), where has full (row) rank. Then, H(t) is not invertible at most in finitely many E (0, 1).
Here, differentiate the empirical risk risk O with respect to the parameters 8 voc(y, 9). We use the indices j, k for the first-order partial derivatives and indices j', k²' for the second-order partial derivatives. We use the notation 1(-) as an indicator function.
Lemma 5 (First- and Second- Order Partial Derivatives of the Empirical Risk for Linear Networks). It. holds for each j.j E [1.....w) ] and k, W E {1,...,d} that
0 riskx (y, 0] = - Σ (Tx,->T0xx))ex))))
"
07,
n i=1
0 riskx (y, 0] -- = Σ
(vs >r Teex))
i=l
16

---

Published in Transactions on Machine Learning Research (12/2025)
and
02
n ((x))))((0x))))
on 1isk/,,0]-
==1
0²
riskx/7 x 0 Ys Σ (z))k
00 y00k.00jk
i=1
Morcover, if J, it holds that
00
2 0 riskxl7; x
risk
(e - (yi -v² Ocx,) -Txx, (2x:)w)
00 jk 0k
i=1
and
0
" risk x 0), Σ
(5(())) - -w: - -o000)))00
07,00
and jf 4 j, it holds that
0
2
riskx (0,xxly,e 0 = 75 Σ (ax))((21))
Oyg 00
and
8²
2
80 80 riskxl7, 0] ==73"
-73 Σ (o.)) (Ox))
These derivatives are basic tools for us given that we work with stationary points.
The next result is essentially a population version of the partial derivatives Lemma that is, sums are replaced by expectations.
Lemma 6 (First- and Second- Order Partial Derivatives of the Population Risk for Linear Networks). It holds for each j, fl,..., #) and k. K E (1....,d) that
0
riskly, 0] 21 (a,)) [2y > (ex)),
07
0
riskly, 0] 22 2 [(y > (x)));
08,k jk
and
02
risk(y, 0 (. (Ox)
0y 0y
0²
risk/y, 0] = 20 ((s))
80 00gx
it holds that = =' Moreover, if
0²
risk 0
(2 [1)(())) (u - (om) (2))())))))
09 07
and
02
risk/y, 0] 2E (7))) (y -u-71 -xx) (z)
0,,00
and if tj. it holds that
82
risk (y, 0 [())
oy 00
and
0²
risk (y, 0] = (x)
08 ,0 07
17

---

Published in Transactions on Machine Learning Research (12/2025)
We use these results in the proofs of Theorem 1 and Proposition
Lemma 7 (First- and Second-Order Subdifferentials of the Empirical Risk for ReLU Networks). It holds for each j.j E {I,..., w) and k, k' = {1.....d d) that
risk (7, 07 -- Σ (wi- - (00(()))))) 87j
1=1
0²
n
riskx 17,
((((0
θy
1 i=1
And
0
fiskxl7.00 () -
n
(() y1 - (00))
08,k
n i=1
with
k(x;;)) = 1((@x:)) 0}, if (Ow:)) + 0. otherwise.
10,
J j and lis E (I....,n) n with (Oa:)) 0 then, &riskx (7, 01/00 doesn't caists otherwise,
02
2
risk x (y, 0 - '5
00 00,000000....
For j j
0² risk riskiiissh.....kk -- ( (yi -iu--"a 7 (2:))
2 n
Σ
a0 j0
and if j
82
2 ((m)))(0)))
risk 0|= _,, ,
1=1
The next result is essentially population version of the subdifferentials in Lemma that is, sums are replaced by expectations.
Lemma 8 (Second-Order Subdifferentials of the Population Risk for ReLU Networks). It holds for each j.j' E w1, and k. W E (1.....d)
0²
On riskly, θ] = 2E ((0)
> 0, (Ow); O}]
If = j and æ with (Oz) then, rriskly, 0/00 00 00 doesn't erists otherwise,
0² risk 0 - E. (x)) (1)
00 00,
where
[1((Om)) 0}, if (Ow); #
x(x,j)=
[0.11.
otherwise.
For j' j. it holds that
0
08 0r risk (y,, θ) = 2I ( (y - -->>" (=((
and 3 #5. j, holds that
82
risk (r, [7,6] = 2 Puy.EEl((an (
08 0'
18

---

Published in Transactions on Machine Learning Research (12/2025)
Lemma (Expected Value of the Joint Density Over Two Half Spaces). For two meon-zero Gaussion mndom wariahles Z and Z' with unit variance and with small enough p R[[Z"] (pl - 0.2) we have
B Z 0/1(Z' > 0}}
1 2 2
Tp 1
3pº
The proof is based on computing the integral using the joint density. We then apply change of variables and switch to polar coordinates, evaluate the radial integral, and approximate the angular integral via a binomial expansion under the assumption that the correlation is small. We skip the detailed proof as it just. involves linear algebra.
B Appendix: Proofs for shallow linear networks
Here, we provide the proofs of our main claims for linear networks.
B.1 Proof of Theorem i
Proof. The proof approach is based on Taylor's theorem and the definition of stationary points.
Let's introduce some notations: We use the notation T
IT, to generate an extended network indexed by (7, (A) with T == (x ;T E Rdtw-1, having the same distribution æ, and A = [01.... 1 w= with v1...., Uw-1 E R is matrix whose columns are basis of R such vw-1=
that Y V _.... = -.------------ 0. It means, the input's dimension of the network is extended from d to d+ w - and so the inner-layer matrix need also to be extended from E Roxd to [9, E [=x(d+=-1), We also use the notation 0a. to make an extended network that is also rescaled across the layers by a suitable a. Note that the notation Oa.A is equivalent with (04)a: both mcans we rescale matrix O R=x(d+=-1) with vector a R (see more details about rescaled networks in Section 6). Using the above definitions, it is easy to see that 7 Aa. =y Ox, which means, the output of the extended and rescaled network is the same the original network (using the definition of A and rescaled weights). In other words, we have network that is first extended and then rescaled while the output of the network is still the same as the original one. We use the notation risk (7a, 0a,A) = Ba 7a ax)) to compute the population risk in an extended and rescaled network. We also define p' == w w (d w - 1) as the effective dimension of the extended network.
Now, let's start the proof by writing second-order Taylor expansion of risk(y' a. O' a. (the risk in extended and rescaled version of the target with 9 a,,'' = vec(y' O' a'ar IR'') around an exlended and rescaled version of reasonable stationary $a,A = vec(Fa, ①a,a) E R with suitable a E I and A, A' E 1=xw-1 (we see later how to assign suitable value for a) to get
risk/y" a O al risk|Fav Oa Vrisk[Taa "T.ATT(B8 a. - -B,A)
+ 2 (0
(0 a. - -Ba,A) V'risk|Fa + t(r" -a), θa, + t(O O.N' - -a.A)l
(8' aB' - Ba.a)
for some / (0, Bertsekas et al. 2003 Proposition 1.1.13.a), where we use the notation Vrisk[fa. 00,
T and g2risk|Fa. Oa.s] E R to collect the first and second order partial derivatives of risk Oa,s) with respect to the Ba.A, respectively (note that we have no assumption on (r" a.A) nor (Fa Pa.A) have bounded norms).
Then, we employ the property of extended and rescaled networks that risk risk Sa.1l = risk (sk θ and risk(yy* a. O+ a.AA = risk(y", O*), and use the shorthand notation
m = (3* ".A' - Ba.1) 2 + t(r a - 7a), + 1(0' a.A - (.,1)( a. - Ba.1)
to obtain
riskly", 07 risk ē
Vrisk Ta.1l ( a. - Ba.a) + m.
19

---

Published in Transactions on Machine Learning Research (12/2025)
@ Oa a. - -a,,a) Vrisk[;,@] -8- To do so, we use 1. our Lemma G (for an extended and rescaled network), 2. the property of extended and rescaled networks, 3. linearity of expectations, 4. our assumption on I, 5. some rewriting, 6. linearity of expectations, 7. our assumption on ø (lot's recall that z = (xx Φ _ E Rd+w-1 with & having the same distribution as x and independent of æ)) that makes the second expectation zero, and some rewriting to obtain that
Now, we motivated to show that Vrisk[aa:
- B).
risk/Ta: da,Al = (2)( (y Ña 7a (
0 8(%a)
Tu-7T6e (6 (0a.1z) :]
= (a
y -_
((a,. /
+ 2F (2.w)
(5 (TTTx))aa..) (0),
--2E(a,v)) ((Oa2)) |
+ +v
(~ ( (Ga.Az))
d+w-1
(aB(m)))) ((@az) ,|
+ 2E
(5 Ow)
(Ga.) (z))
d
= 2E (((o)))) .]
+ ()
(5 Ow)
(a.n) 1) 5K (3))
k=1
d+w-1
+ 2
(5 (xTxx)
Σ
(G (T)
k=d+1
d (5 0xx
= 22 5( ((az)))) ,]
+ 22 (
[(Ga) (6a) (x (0))
k=1 Ozx) (Gazr.) ,,]
= (a ((aaz) .]
+ 2F (x.)))
Then, we 1. imply our result above for all j E (I...., w), 2. use the definition of rescaled parameters and linearity of expectations to cancel a's, and use our results in Lemma i to obtain
0 risk (as 0anll T - -a)== 2 E(aw)) y [( + B(aww) (TT @w) (6mzt)) (r"a - -aa)
-7---a)
07a
2 - B( [V(Gar),] ] + + (7 (GT))) ((m))) (7- - 7)
-(0 05
8 risk ris @1) (7--5))
Implying a similar argument as above for all partial derivatives, we conclude that Vrisk( (B a. 1 = Vrisk|f, ²) (8* B3 (we omit the detailed proof). Tabulating this observation in the earlier display we obtain
1
risk 0*] risk o) Vrisk (8* + m
Rearranging the display above we obtain
1 Vrisk(7, (8* - (3) = risk [7, θ] riskly", 00) +
2 m.
Now, let's recall the definition of stationary points in equation 3 which implies
Vrisk T - (3) + +r T(8 -î) 2 0.
20

---

Published in Transactions on Machine Learning Research (12/2025)
We 1. rearrange above inequality and expand the bracket, 2. use Hölder's inequality and the fact that -1811 (recall that E 11 and 3. use | S to obtain
2 T B
-Vriskx (B r 8 r 8
s 1r| - r11h
s 1 -11311
which rearranging implies
Vriskx . ( - r1 1 0.
The display above demonstrates the positivity of the terms on its left-hand side, enabling to obtain
Vriskk/y, T - -B) -Vriskisk -] ( - B) Vriskx|7. - 3) + + -
that is,
(B
- (Vriskx 6| Vrisk|7, -
+
- 1-118111-10
Now, let's use our display earlier (obtained by Taylor expansion) to rewrite the left-hand side of the display above as
risk θ] ,0""""""""""" m S (Vriskx 17, Vrisk ō1) (8" + rr10011 -3) r1
Rearranging the display above obtain
1 m.
risk O) < risk(>" + + (Vriskx (). Vrisk θ1) "3* -r]111111
For the right-hand side of the inequality above we 1. get absolute value of the third term, 2. add zero-valued factor, 3. use triangle inequality, and 4. use our results in Lemma | to obtain
risk (5. 6) < (r" 0-] - r|( + ( rrisk Fr. θj Vrisk Ø1) (8 -3) rr1/111
1
2 mm
=risk(>", - (Vriskx (7, Vrisk θ)) '(8" 3) r(1i + -1881)
1
-
S [ask("",0]=== - 11 + (Vriskx
(rriskx[7,0) [7, Vrisk θ)) (Br B) - -r|g 31
2 2m
< risklr>", 0*] 2r |" 1 Fore 1 -B11
Tore
r1| - m
2n
with probability at least 1 1/2n.
The third and fifth terms in the last inequality above can be canceled if we choose the tuning parameter large enough. Hence, we obtain
risk @] < risk(y"" 0*] + 2r +
Fore 2n 2
for r 2 Tore-
The rest of the proof analyzing the behavior of m. Lot's rewrite m 118 a. Ba.Al3 m' with
m' (B'..'' - Ba..A v'risk/Ta + t(r' a -a), θa. + f(0 a. Sa.all
(3' a Ba.A)
13. a''''''' Ba.Al2
| 1Baa. Ba.Al2
Now, we are motivated employ our results in Proposition invertibility of the matrix (0 1(@" v ( + 1(0",, x' @A)) Using the definition of the extended
To do so, we need to make sure about the
21

---

Published in Transactions on Machine Learning Research (12/2025)
networks, it is easy to see that A and e* A' have full row rank. Then, using Lemma 4 obtain that the line segment, between two matrices A and @* is not invertible at most in finitely many 1. It means, if we shift by a tiny value Re 0 then, we can make sure that in the new point t' = t - the corresponding matrix is invertible, that is,
(3r a. Ba.A) V'risklia + (t- + s)(r'a - -a)) 6a, (t- 5 <)(O" a.A' - Ga.A)l
m'
10
a. a.,' Ba,Al2
(3' a.A' - Ba,A))
18 a. - -Ba,A 12
(3" A, A)
VrrisklTa + (t ()-r) -a)), + - c)(O" a. - -a.N)l
18 a, Bo,A12
(3' aB'' - (B.A))
18* a.'' - Ba,A
(8" a, Ba,1) Vrrisklaa + - -aa), Ga, + +' a, - -a.aa) (B' - Ba,1)
18 a. Ba,12
13a,,1' Ba,Al2
where the second equation is reached by assuming is very close to and terms. Then, we have x
ignore the remaining invertible matrix.
+ (' N' ( + l'(0" A ^A))T
Implying Proposition 1 (with of the input and the effective dimension, respectively) obtain that m' [0, oc) for appropriate a, that is, a with large enough c). The observation that m' E (0, together with the definition of m implies that m E [0, oc) as well.
(8' a. Ba, A/118" a.A' Aa,Ale and + and p' as the dimension
Tabulating this observation to the display earlier together with our assumption on 8r (188)1 = 1>"111 + |6'|/ I 2yTog n) and the fact that 1/2n < Vlog we obtain for all Fore that
Torc 1
risk o < risk/y" 0* Br +
2 m
2n
< risk(y"; 0-| + |r B*
+
Fore
2n
< riskly" [7 + 5r log
with probability at least - 1/2n.
The second claim is a trivial consequence of the first claim by using r = Yore and 2, absorbing the constant. in and simplifying to obtain
log(np) log(np)
risk θ) s risk (isk 0*] + vlog n) 3/2
n 5 5v
= risk/y", @*e + vlog n)
n
with probability at least 1/2n, which completes the proof.
B.2 Proof of Theorem 2
Proof. The main ingredients of the proof are the definition of approxiintee stationary point and our Lemma 8
We start the proof using the definition of approxiiiete stationary point in equation 7 that implies
risk x 54 O +r < risk (r, 3 r|1B], +
22

---

Published in Transactions on Machine Learning Research (12/2025)
We add zero-valued terms to the both sides of the inequality above to obtain
riskx| 6] - risk 6] ris r| I riskx 17, 6) risk 6] + risk(7, 6) + r13/1 +
Then, we 1. rearrange the terms, get an absolute value of the two terms, and use the propertics of absolute values, 2. get a supremum over the reasonable parameter space Bies using our assumptions that (5 Q), (7, )) e Bres == 6) E B I7 ||@|1 I vlogn) (we use our assumption that the stationary is reasonable and our argument in the paragraph above Theorem to reach that (7, Q) is reasonable as well), 3. simplify, and 4. leave a negative term to obtain
risk R c s risk [7, @] + risk x (7, 6) risk 6| + risk x [7, 6) - risk 6) + r1111 r11B1 +
risk 0 + sup
1riskx/7, 0] riskly, θ] + sup
riskxly, 0] - riskly, θ]
(7,
(T.
+ +r|681- - 1 11 +
= risk [7, O 2
sup (),
rrisk 1 0 - risk (,. 0] + r|]3|| -r133 +
risk (7,
0 + 2
sup (
rrskk (y, 0 - risk y, 0] + r r| +
Then, we use 1. our result above, 2. Lemma 3 bounding the second term with t t= "V/loo32ndd ) /kn and B Bres (with probability at least 1 1/2m), 3. the definition of Byes to replace ())(,, ,
O
y T co I B.. 2 ||01 <. 2(logn)² =: é', 4. our Theorem I upper bounding the first term (for 2 Fore with probability at least 1 - 1/2n), 5. our assumption that stationary is reasonable, 6. simplifying,
7. an assumption that n 2 3 (just for simplifying the terms), and 8. the assumption that r 2 Fore and the
definition of Forc (note that for simplicity, we absorb all the constants in v)) to obtain
risk 1r, 6) S risk θ + sup (risk(11 0] risk(7, θ]| + r| +
(7,
log(32ndd)
< risk(7i x + 2
4e' +4ve) + r r|| +
Iog(32nd²)
< risk|7 θ6 + 2
(1 + n) 8lgg) +r13312 +
An
log( (og(32nd2)
< riskly", 0-] + 5r / log n + 2
(1 + 8(lo n)2 8logn) +r1BB111 + 5
Kn
log (02ndd)
I risk/y" 07 + 5r / logn n+ + 2
(1 +1+8(logg)) n² 8logn) 2r Vlogn T
kn
log (32nd2)
= risk(y", + 7r logn n + 2
(1 + 8(log n² + logg) +
Kn
log (og(3)nd2)
s risk/y" 01 + 7r logn + 34
(log n)² +
Kn
x risk/y", + 8r log n +T
with probability at least. 1- (1 + 1)/2n, which is obtained by the fact that if a < 21 2.2
P(aa S a + (2) 2 P(z1 + 52a
1 P( + 22 ci
> 1 (P(z) c1) + P(a2>
> 0z))
where a, 21, 22 are random variables and 02 are constants, desired.
23

---

Published in Transactions on Machine Learning Research (12/2025)
The second claim is trivial consequence of the first claim by 1. using Fore and 2. absorbing the constant 8 in to obtain
log(np) (8 (8vlogn) +
risk| ô] riskly", v 8)222
log(np)
risk(y"; @*) elog n
+T,
n
with probability at least. 1 - 1/n, which completes the proof.
B.3 Proof of Proposition
Proof. The proof is based on basic algebra and property of scaling weights across the layers in neural networks. Without loss of generality, we assuine that æ; (O laxa) (the proof for independent and centered sub-Gaussian random vectors with independent coordinates is the same, just some constants may change, which doesn't affect the main results).
Let's consider all the network parameters as a vector of length p (recall that p = w + w d). Then, we can tabulate the second-order partial derivatives of risk/y, Q] in a matrix called Viriskly, O RpxP (for notational simplicity, focus Virisk(y, 0] for the moment and then we move to Virisk 2a) at the end of the proof) of the form
A ] B
Vriiskly, θ]
D
with R=xE, E (drd)x C' 1=x(wd), and D (R(x))))(xxdd)) where
02
Ay =
risk fy, O].
0²
B = 00 i
risk (7, O],
ÿ²
Cÿ.b-1144 = 00
risk 0], 20 jk
0²
Doy-1)jtR..--DH4E 20 30
risk(y, O]
for j,3' E {I,...,w) and k, k' E d}.
Applying the block-wise structure of Viriskly, θ], we are motivated to analyze the behavior of
AA a V'risk OJa (a')T Aa + (a') + (a²) Ba + (a?) Da?
Note that C = B (see Lemma 6), so, we are left to analyze the behavior of
(2, (T')T Aa + 2((a)) ( Ca² + (a3) Da?
for all Rº with |a|2 =
We do the proof in steps: We start by going through the three terms on the right-hand side of the display above separately, to write them in mathematically nice formulation (Steps 1:3). In Step 4, we sum up the results calculated in Steps 1:3. Finally in Step 5, we use our results in Steps 1:4 to prove the main claims of the proposition.
Step We show that for a² E Rwd and D d)
(a T Da =
E(>
k=1
24

---

Published in Transactions on Machine Learning Research (12/2025)
where we denote (a²)* ((a?)), (a² )dtk: (w 1)=1k)+ E R" (as a sub-vector of a²) for each k e (I,
We start by writing matrix product in the form of sums and fill the entries of matrix D with the corresponding values from the definition to get
(a²)Taa² Da
(a")
riskly, 00) (a (a)) 115-))ak))
00
y=1 M=
By Lemma i we have
0² y jk
risk(y, 0) = , [<)))
which using assumption (identity covariance matrix) implies
82 risk 6 23
for k == k and zero herwiee (for k * k)) We use 1. our display earlier, 2. the result above, 3. the linearity of sums, 4. some rewriting (using multinomial theorem), and 5.implying our notation (a?)& for writing the sum in the form of product to obtain
0²
(a²) Da Σ
(a²) G'r-(1)++ 00,
riskly, 01 ( (-11d+k
(a ( 114+477777)7)) ) a² (--1dd+k
2
d
= 2
(aa (3' (1--1) (-1))++)
k= ==1
2
= 2 Σ
Σ )
k=1
= 2 Σ
> (a?)))²
k=1
Slep 2: We prove that for a E R and A E Ruxw
(a') Aa = ((0.))
where O, denotes the k-th column O.
For each j. E (l...., w) we use 1. the result of Lemma 6 2. the definition of covariance, 3. the fact that Cov Cov(O.)) @Cov(z)0T 4. the assumption on (identity covariance), and 5. rewriting to obtain
0²
riskly, θ (m.)) ((xx))((rr))
2(Cov((z)) fjj
2((Cov((z)01 yj
2((0T) jrj
2
0 0rn 120k
1=1
25

---

Published in Transactions on Machine Learning Research (12/2025)
We use 1. the definition of sub-matrix A to write the matrix product in the form of sum, 2. tabulating above result and using the linearity of sums, 3. some rewriting (using the multinomial theorem), and 4. writing the sum in the form of product to obtain
O²
Aa Σ
(a')s
risk (i(a)),
d
2(a (a²)),b0 )yn0su(a");
k=
y'
Σ ()x(a)),)
= 2
Σ Ta)))))aa)
Step 3: We show that for a E R", a² E Rudd and C' E Rwx(wd),
(a²) =2 Ca² E (TT
((0..)
+ 2 Σ
((7 + T0), - - (
(aa)))
k=
Expanding (al)T Ca² yields
w ((a))
8²
(a') ΣΣ Σ
risk 00 (a")
k=1 00,
Now, we need to consider two different cases:
Case 1: (i
We use 1. the result of Lemma 2. writing matrix product in the form of sum, 3. lincarity of sums and expectations, and 4. our assumption to get for each j.j (I...., w) and l E {I....,d} with j # j that
8
riskly, θ) = 2n (a.y) (0x))))))
00,
-E
(x) Σ (0)))()))))
k=1
273
1==1
= 27500'k
Case 2: ji=
We 1. the result of Lemma 0 2. lincarity of expoctations, 3. lincarity of expectations and our assumption æ (same argument above), 4. linearity of expectations, 5. linearity of expectations and our assumption 6. adding zero-valued term, and 7. again linearity of expectations, our assumption on and rearranging to obtain
82
risk (y, 0) (. (Om)) - -y--Ter (x))
0y,00,x
26

---

Published in Transactions on Machine Learning Research (12/2025)
(a.w) (7 [7)(()) + (m)))) (x)))))))))))) x(())x
2y,,05 + (my)) [(> (x)) - b(z)x)
27,0,x + 2 (2 [(7 [z)) () [b(())
-21,00h + +2 00) ())))))) [1(2)s
= -2,00s + +2( - (2 (y 7- - -
-2 + 2 ~0') 21 - T)==)
Now, we 1. use our carlier expansion, 2. separate the innermost sum in two cases, 3. use the result above (Case and Case 2), 4. rearranging, 5. use linearity of sums and some rewriting, and 6. write suns in the form of vector products and rearranging to obtain
( Ca2
(a "
@² 00,
riskly, 61 (a")
Σ
w
(a');'
0
riskly: 80 (a²) 170+
00,
1==
m
8²
(a');
riskly, 0)((")
07,00,t jk
Σ
(a'))))0)"((a") (a² (a)))
|'=,'
(a + (7 - " (a²) (1)))=)))
(a
+2 2
(a+) (> To - T0))) - -E ((==-=====>))) =0)=)(=))x)
11
Σ
Σ aa 0 Σ (1((a)))
(2(a)) (aa)))(a (a(())
10
+ 2 Σ
(7 3' 00"), -Ro (
7 T (a3)+) (a'))
2
k=1
Σ
-E x (a))*
Step 4: We prove that for any a = ( E R' and (7, E B, it holds that
a T Vriskkly θ|a = 2 d
Σ ((a)) (a)) (a3))) + +4 d
(a))
d
4
E (())))))) ( (o
27

---

Published in Transactions on Machine Learning Research (12/2025)
We use 1. the block-wise structure of the Hessian matrix and rearranging, 2. our results in Steps 1:3, and 3. multinomial theorem to obtain
Vriskly,, 0)a 1000=aa0) (aa) Aa' +
AAa
Da + (a²)
22(a))
7 (a2)))² +2 1 (0 (T) + (TTT()))))(
2
+4 >
((7 -T BV - (a))
k=
( (T a' + (a)())) +
2
k=1
(m (00->~1r>\>>>>
Step 5: Now, we employ results in Steps 1-4 to prove the main claims of the proposition.
Claim 1: (a' = 0 and a + 0)
We use 1. the block-wise structure of the Hessian, 2. the assumption that a² = 0, 3. our result in Step 1, and 4. the fact that sum of non-negative terms also non non-negative to obtain
a T riskkly θla = (a') Aa' +2(aa)
Ca² + CCaa (a²) Da²
= (a?) Da
2 Σ ( (a?))))
2
The above display can also reveal that for all a E R " (0) (moving a scaled version of the parameters)
Vrrisk(aa Oaja = 2 d Σ (ra) (a2)t)) 2 0,
as desired.
Claim 2: (al + 0 and a² 0)
The proof is similar to Claim 1 so we omit the proof.
Claim 3: (a= 4 0 and a² 0)
We use our results in Step together with getting an absolute value of the two last terms to obtain
d
a V Oaakkyy,aa-00
Σ
O ) at (a?)t))
+
TT") (a2)
d
4
E
(a²)*
k=1
> 2
(0 k) a + + (a)))))
Σ
To - -00--"T00) (T)
k=1
4
Σ
E E(an)) (y - (x) (a
28

---

Published in Transactions on Machine Learning Research (12/2025)
First, let's concentrate the second term of display above and 1. use the triangle inequality and properties of absolute values, 2. use Hölder inequality, 3. get a factor |a out of the summation, 4. use Cauchy-Schwarz Schwarz inequality, and 5. some rewriting to obtain
4
( 0 -----111 (0=), (a')
< Σ
- (T)
d
(7 0 x To)) ||
-
k=1
=4|aa'| 1l
1 1 |1(2))112
k=1
s <4|1a') 12
Σ
I 1ul7 -00
(7 0 -
Σ 1
4|la aa
Then, we use 1. our assumption that y ~0 + u, 2. independence of and æ, and 3. assumption
that [[z] = 0 (also we have E[u] = 0) to obtain
E (o. w) [( - [(==))))))aaaa (a²)
B( (a'))
E (m))) B [<+) (at) (a²)*
0
Tabulating observations above in the previous display we obtain
d
a (Prisk(y,0)a3 2 2 Σ (). T + +> - 111aa1/1 T T
Now, let's define for each k E {l...., d) that A (T..)Taa) B = V (a?)*, and using the fact (A₃+ Bx) 2
1 (Ax)² (Ba)² to obtain
(Ax + 2+))== - Tl - 1
k=1
d
d
(A6) - > (Bx) - 1 - -
km
kml
Now, analyze the first two terms on the right-hand side of the last inequality above. We use 1. the definition of Ax, 2. some rewritings, 3. the linearity of sums, 4. the definition of matrix product, 5. property of cigenvalues (emin [) denotes the smallest eigenvalue of TOT), and 6. the norm definition to obtain
Σ (Ax) Σ aa))
= Te
k=1
29

---

Published in Transactions on Machine Learning Research (12/2025)
d = (a')
Σ
a²
Ta')ToTTa
Emin
emiul
Also, using 1. the definition of BR, 2. the Cauchy-Schwarz inequality, 3. the linearity of sums, and 4. the definition norms we obtain
2 (Ba) (TTT)))) 22 -1H
- |(a?))12 -211>/1/a111
2/712
k=1
Collecting two displays above together with the earlier one we obtain
rirkkly Ola 2 emin [00 |a 12 || 4|la" "=la"""ll""0-->""
Now, it is time to concentrate on the Hessian behavior of ririk//7a Oal (and not Verisk(7; θ)). We use the known fact in neural networks that weights can be rescaled across the layers once activations are nonnegalive- homogeneous. It says for neural network parameterized by (7. θ), there another network with the same objective value such that the covariates of y are multiplied by the covariates of and the covariates in each column of θ are divided by the covariates of We use this fact with a, = 1/c for all j {1,..., w), which E 01)) together with the above result to analyze the behavior of Hessian in (Ya: Oa) and get
a T V'risk(7a Oala 2 Cmin Pata |||a 12 2|| 2 "
4||a" 1|2l|a" 1|222211 Ta Ta --a TTaaaa
00al22
cd c [00 |a 2 2111 4
4||a² llllla aalll T
where for the last line we use factorizing and the definition of scaled parameters. Using above display, can guarantee positive semidefinite Hessian once is selected large enough because, the first term dominate the other two terms. So, we use E 20) and our assumption on θ0 to obtain that for
2 ||> a
4||a|1,
a²
TlllyTe
- y- 0 0
Emin -0
|a-11
we can guarantee positive semidefinite Hessian, as desired.
B.4 Proof of Lemma
Proof. The proof idea inspired by Elsener & van de Geer 2018 Lemma 14) and main ingredients are our Lemma 2 and union bounds.
Let's define F(t) 2t for E (0, 00); se,., (n + max | (1 + c), which is basically defined by parameters and of C,,e (recall that Cy.. {8 vecly, E IR ||8" - B| S n and TY - T T+ < E)), and 2(8,8* as function of two vectors B and 3r (with B = vec(7,, 9)) defined as
Z(3 : (Vriskx (
θ] Vrisk/y, 01) ( - 3) -3)
Using Lemma and notations above and with assuming 3 E C,,e (specific values of e and v) be assigned the end of the proof) we obtain for each t (0, oo) that
p(Z(B, 3r) 2 mf(()cc...) < sup ((, 2 mf(())c...)
BEC...
< 4d exp(- Kn min min(f// (f//2,t/2)))
30

---

Published in Transactions on Machine Learning Research (12/2025)
with V, K E (0, oo) constants depending only the distributions of the inputs and noise.
We assume without loss of generality that 1/n S 17 and continue the proof in two different cases:
Case 1: (18 11 1/n)
In this case, we use 1. the fact that 1 1" 2 0, 2. our assumption that 1/n I oj and the definition
of s: and 3. our assumption that | - 1- I 1/n, which implics that B E Ci/n.e and our argument above to obtain for each & E (0, o0) that
P ((3,8')) B*) 2 213 (*)) + F(t)
f(t)
S P Z(B,8)) 2
SCn.
SCn.
n
n
F(()
$ P 7(B,8") 2
8C1/mic
n
1 4d expppxpp(pp exp(-xm min (12 (1/1121//)) .1/13).
Case 2: (1/n < 1/ - 11 S 1)
= 1 2. lower bound of ||3 - 11, 3. the fact that F(1)sc,,c/n 2 0 and removing the lower bound, 4. the fact that
In this case, use 1. the fact that for mutually exclusive events Hi,..., H H,)
2++1 // 17, and 5. the fact that B E 1 and our earlier argument to obtain for each & (0, oo) that
P 2(3. 9r) 2 21B3 ()))
f(t) for
1 133-
n S
S 1)
7
Dogg (nn)]-1
F(()
2
2
P 8") 2 213 - 8 (1)((t)
n S for
< 13 Br 1
n
[lgga [mm)]-1
P ((B,83)) 2 r(t)sc,,. + F(()
21+1
2
2
Scn for < 1 1/1
2=
[ogaa (mn)]-1 -1
2i+1
2
P (2(,,3))
2 7(t) for 13 - 3311 S
n
[loga (nm) -1
21+1 F(L)8cgu41/m. for 13 - B¹11 s
2
TP (*)
2
n
n
4 [oog, (nn) d exp( (- kn min (t2 /v2 /t²///)).
We collect all pieces of the proof (Case and Case 2), set = 1 VV/log (8nd2p) log, (n17) (we use the notation log as natural logarithum), and use the union bounds to obtain (we also need to assume n is large enough to get rid of the min operator)
P 2(3,B*) 2 2113 - 38 (1i5(u/1og (Sndyylogg (1o1)1/ (wS)) sc...
F (w log (Snd2pllog, (1n)))/((n))
+
SCn.
n
< 4[log2 (mn) exp( log(8nd [log (nn)
1
2n
31

---

Published in Transactions on Machine Learning Research (12/2025)
Now, we use the results above and the definitions of 8*) and f(t) to obtain
[[Vrissk Vrisk(R, (88" - -3] Ausc... 1/ 19111
log (8nd2 ploogp (n7)
km
log (8ndPp logz (nn)))
2vSC
kn 3
1
2n
assumption that the stationary point (7, is reasonable to obtain: 1 y T I ly 67 + 10" < 1/11001 + 11"1110*l S logn (using triangle inequality, Hölder's incquality, and our assumption on reasonable target and stationary) and | - 8 - |@|| + 18 1R|1 + ||O| + Iy + ||0'|] & 4vlog n (using triangle inequality, our definition of norm, and our assumption on reasonable target and stationary), which means we can assign 2 log n and r = 4 /lg (for n 2 2 we can make sure that 1/n I r is satisfied).
Then,
Now, we plug the values of e 21og n, n 4ylog,, and sc,,, (n + max lox" 1 11 )1 + e) s (5vlogn)(1 + lo < 15(log (for n 2 2) lo conclude that
P (Vriskx 17, G] - Vrisk( 61) (B* -33
60
v1B 8" (1lgg)))3 3/2 / /lol//ooonnn log(8nd"p[log, (4n (4n
kn
30
Van (logn) 3/2
log(8ndpp/oog, (4n n)))
2n
Then, we use the fact that d < p and simplifying display above to obtain
P (Vriskx (7, 0
Vrisk θ1) T ((8 -B]
180
v1 3r 12 n 3/222 1////3///////////// log(np)) + (logn)33/ (lnggn)33////g//me
90
V V
2n
We finally absorb all the constants (180/ and use the definition of Fore complete the proof.
B.5 Proof of Lemma 2
Proof. We start the proof with Hölder's inequality and the definition of Ca,es which implies 1|8 3/1 S " for all B Cn to obtain
- (Vriskx 0] Vrisk(y, 01) (0e T - :9)
sup ())
(/) |/Viskxx 0] Vrisk -l) B11)
sup B=vedly
17 sup )=vec(y, ()C
Vrisk by, 0] Vrisk(y,
32

---

Published in Transactions on Machine Learning Research (12/2025)
The rest of the proof is using Lemma and Bernstein's incquality (Vershynin 2018
Corollary 2.8.3) to find an upper bound for S V (,, 0 Vrisk (7, l Note that for simplifying the notation, we use E[-] as shorthand notation of (1 (....((.....111 throughout this proof.
We use 1. our result in Lemma 5 and iid. assumption on the data, 2. equation I and our assumption that [[a] = T O'x,, æ, zero-mean noise, linearity of expectations, and factorizing, the definition of sup-norm, triangle inequality, and Hölder's inequality, 4. the definition of Cy.e: which implies T - - T S e, 5. adding a zero-valued term and rewriting, and 6. the triangle inequality and the definition of Cr.es which implies 1 - 71 18 11 I n,, to obtain for each E {l...., w) and & {l....,d) d} that
1 risk 17, 0]
a
risk (7, 61
08,₁ jk
j jk
- n2
Σ
Vy - 7 Oz; (2;)1e +
2 Σ (Us > (0n))
i=1
2/71| n
u4 x3 )) (7 x T - > T (2: (x))a())a
- T 0011 ト
Σ [R| [=))))))))) (a) 1
"
s 22/>/>> 00
nz (a( )
(11 n
Σ Uz (=(=)
+ Σ [B(x;(x;)) - x, (2:)
s 22
(1
n
= =21y - 7" + 700
+
Σ
i=1
s 2(n + 17'l
1 uxxx))
Σ
n
+
1 n
Σ
We continue to work on the absolute value and sup-norm term the last inequality above separately. For each E {I,...,n} n. and & E [l,...,d}, we use our assumptions on æ; and u, to obtain that z; == u.(3))) are independent and sub-exponential random variables with zero-mean (Vershynin 2018 Lemma 2.7.7) and so, we can employ Bernstein's 's inequality in Vershynin (2018. Corollary 2.8.3) to obtain for each t E [0, oo) that
P
(x:)
2 t)
< 2 eexp(-x min (/n (1/22t//\m))
1=1
with x E (0, oo) an absolute constant and y ( (x,)))) (0, oo) a constant that depends the distributions of æ and u (for sub-exponential random variable 2, we define | == inf E (0, c0): E Eexp(((=l/4) < 2)).
Now we study the behavior of the sup-norm term in the last inequality of the earlier display. Let's rewrite the sup-norm in the form of a max as
1 (a,():) (x.())
||_
1
Σ (a. (()))((()))) - B[(x;)) (2x)
n
max Me(l....))
Following the same argument as carlier and for cach i E {1....... n) and k K {l...., d), we use our assumption on as to obtain that z/ (a:))<(a)) B[(x:))=(<;:d are independent sub-exponential random variables with zero-mean and again can employ Bernstein's 's inequality (Vershynin 2018 Corollary 2.8.3)
to obtain for each t' E [0, oo) that
P
1
(cx:) (xx)a B((a))
2 e²) S 2 exxp(-k' /in
7
33

---

Published in Transactions on Machine Learning Research (12/2025)
with k' E (0, oo) an absolute constant and /' == (I..g(l.....)))) (z,)) - E(( (0)
constant that depends on the distribution of at
Then, we use our result above together with the fact that if [(6]:] we also have P(max;e (1,...p) 16el 2 t) $ pa to obtain
N t) s holds for all i E {l.... ph, then
1 Σ (a)))) (x( (x:)xl) n 1=1
P max (e(
2 1'
S 2dexp(-w min(t?
Collecting all pieces above together with considering t = t', we obtain for cach j E fl...., w) and E (1..... that
0 risk x 17, 01 min
0
risk y, 01| - 2t(n + 11
ō with probability at least - min (1//u2 1/u)m1) using the fact that
f0,k
2dexp(-x' min (1///22
min /////////// (/)")n), which is obtained
bD S 1++b)== = P b+ 2 1- P(A > 1) P(D 1
for any E oo) and t E I.
Then, we follow the same argument as earlier and use 1. our result in Lemma and i.i.d. assumption on the data, 2. the properties of absolute values and linearity of expectations, 3. some rewriting, 4. Hölder's inequality, 5. equation and our assumptions that f[xx] = > @" æ, zero-mean noise, and definition of sup-norm, 6. triangle inequality, compatible norms (for a matrix A E Raxdd we define || = max.e( E |A 7. adding zero-valued term, 8. the triangle
and the definition of Cz,.e, which implics T - T inequality and the definition of Crye, which implies ||0 Or||| S B1 S v) to obtain for each j E (1....,w) w]
that
0 θ7j
0 risky x iy, 0] risk (y, 0] 07j
2
((yv - > Oz (zx)) + 2 Σ ((v) y (xx)))))))))))
n
n
2
((y 7 00x,) - E(( > Ox:) ((x;),)))
(wy - Tar,)ar TO,, El(ys > Tax,)or; TO,,.)
2
((v - > 0xx))x - -E((y; > Oxx TJ) 10
<
n1
(u,ar] T + To* x [0) T Elasaa; TD)
< | 00
< 2|10| 00
Σ U,::;
+
1
a T - Elar,:; TD
i=1
< <2|10 - 0- + + loc
u
+ c
( T E[I,,, TD)
i=1
(
u,a0 |
m
< 2
+
(
n
Then, we use the same argument as earlier treat the sup-norm terms above (we use our assumptions on x; and and application of Bernstein's inequality) to obtain that
0
0
risk x (7, risk |r, 0] s 21(1 + 101)))((1 +
07i
34

---

Published in Transactions on Machine Learning Research (12/2025)
with probability at least 2d exp((x min (tt t/v3))) 2d exp(-n' min(tt/ /v2 (t/////// constants depending only on the distributions of the inputs and the noise).
.t///m) (n, v, kt, v' are
Collecting all the pieces above, we obtain that for each i {I....,p] the corresponding gradient difference is bounded (I(Vriskx|y (r. θ] Vrisk 7, 01)a I 2t + max max(1>>"la. 1 l 1)(()+++ + c)) with probability at least 1-4d exp(-Ku.z min min(t (M,,,))), //V with Vy, = max(u, (v, v/ and Fu, = min k²} (Vu,m and Ku, are constants depending only on the distributions of the inputs and noise).
Now we use 1. the definition of sup-norm and 2. our results above together with earlier argument about implying max operator (note that the gradient vector is of dimension p) to obtain for each t E [0, oo) that
Vriskx (r, θ) Vrisk 7, 0l
sup v B)EC,
(Vriskx Vrisk 0)),
sup (= ic (l.....))
max
< 2 - max Mo, (1
with probability at least 1 - 4d exp( - Ku,, min (t (V=,,)) "tti.attt//uue )
exp(-
Collecting all pieces of the proof, we obtain for each /. E [0, x)) that
(Vriskxly, 0e Vriskly, 0]) (9- 3)
sup A B-
Vrisk (7, 6] Vriskly, 0)
sup
B OCC 21m(1) + max ( 110" bxx (1 +)
with probability at least 1 - pdppex min (2 ) v/V In), where for the ease of notations we
replace Ku, and Vu,æ with v and x (constants depending only on the distributions of the inputs and noise) in the statement of the lemma.
B.6 Proof of Lemma 3
Proof. The main ingredients of the proof are symmetrization of probabilities (van de Geer 2016 Lemma 16.1) and Bernstein's 's inequality (Vershynin) 2018] Corollary 2.8.3).
We note that for simplifying the notations, use E[|] as shorthand notation of E wn) w (w. []] throughout this proof.
Let's start the proof and use 1. the definition of riskxly, 0] and risk/y, 0), 2. the i.i.d. assumption on the data and that yi = > O' x; + u,, 3. expanding the squared-terms and rearranging, and 4. the triangle inequality to obtain
sup |risklyy. 0] riskly, O|| (r,0)eB
(y: - =om))²²) - E( (w y Tmm)²]]
sup (),8)
n
(r T 0rx; xx u4 7 TTOxx:))) 2
-F E ( 0 x, + u4 1u--7 7 (exx)())))
sup (7,))BB
(> T 00x;; az 7 Txx)) 2 O
T0xx; - Tow:)))) ((m:)))
sup
+ 2 (7>
-0'; T -7 Torx) u - TO'
T x Txxx ) + (u.² - Eluu²))
< sup (7.0)BB)
To'x; - - Texx)
2-2-B((s T
0 - > Texx)) 9?)
35

---

Published in Transactions on Machine Learning Research (12/2025)
1
O'x;; T7"=a" --0m))u: (ann))--B(( -B((r*T 7 (m:))ii)
+ sup (r,0)EB) n i=1
1
+
113 2 Elu;
Now, we continue to work on each term in the last inequality above separately in steps:
Step 1: Using Vershynin 2018 Corollary 2.8.3) together with our assumption on noise, which implics the squared of Gaussian noise is sub-exponential, we obtain for each 7 E xc) that
(ue - E(u,)))
2 7)
s 2 ein (i//-mm/mm/////////00 (/um),
where K, V/ E (0, o0) are constants depending only the distribution of the noise (our constants and may change from line to line in this proof, but they constantly depend just on the distribution of the inputs or noise or both).
Step 2: We now prepare the application of van de Geer 2016 Hölder's inequality and factorizing to obtain
Lemma 16.1). Let's 1. define R² and 2. use
R²
sup
E (7
2000aa x; - 7 Txx)
(T, n
11 E [1l0:l1b0
< sup (x,0)8
We also employ some linear algebra together with compatible norms (for a matrix A E Rdxd, we define |A,, == maxke (I....d)
Σx =1 (A),)))) to obtain
s T6'x; - 110-TTen TTx,)) n
T - 7 - (xx:)ci(7 T0'z; - "0',,-0-00 T
n
1
(x 0- ~ - 0) T (7-
-T=)
< 0- - x ~ ~T)
<æ
00,1
Then, we use 1. symeetrization of probabilities (van de Geer 2016
Lemma 16.1) with R as defined earlier,
2. the display above, 3. our assumption that sup( ( O* -y-""n 1) 0c I e' and rearranging, 4. the definition of norm for a matrix above, 5. the fact that if P(b 2 t) < a holds for all i E {1....d} d),
then we also have P(max [b,] 2 () S da (for E (1,.....d) d)), 6. the fact that for a vector a E Rd. POE |a,1 2 t) S d dmaxse(1,....ff PP P(aa 2 t), and 7. our assumption on x (to get rid of max term) together with Vershynin 20188 Corollary 2.8.3) to obtain for each E [0, oo) that
(6 To'x x TTxx)² - E B - - TOxx))²))
22 4R
sup (7.0)EB
2
n 1=1
n
4P
sup
c TT' - TTaT-TTTue Txx:)
2 R1.
21
(7:0)E n
4P
sup (7,0)
|| -
T(= Te)² oc Σ 2,
21 R
x,1
R
2t
< 4P
2
e
00,1
36

---

Published in Transactions on Machine Learning Research (12/2025)
Σ
d
n
R
2t
4P
max
2
kE (l...))
e'
k
R
21
4dP
2
V
c
n
R
2t
< 4d
max
P
V
1"
kE ()....)
n
s ed expl - (in 1=0(
where K, V.C. E (0, oc) are constants depending only on the distribution of the inputs. Collecting results above, we obtain for each t" E [0, oo) that
( - Txx.) 2 E T - -xx)) 2 sé't"
P
I 8d exp(- (t2 ("" //"//)n)).
Step 3: Let's define (R')² and use Hölder's inequality to obtain
(R')² = sup (x,0)E8
E ((>
- Ox;)uus) ']
sup (Y,6jEB
T -
θ
2
n
E 1
n
i=1
Lemma 16.1) with R' defined as above, > 2 c', the fact that for vector a E Rd, P([aaala 2 t) < dmaxsg().....l P(Ja.] 2 t) S d'I(ja, 2 t), and the assumption on inputs (for E (1....,d) d)), and 4. Vershynin| (0018 Corollary 2.8.3) together with our assumptions on the input and noise to obtain for each t' E (0) that
Then, we use symmetrization of probabilities van de Geer 2016
2. Hölder's inequality, 3. our assumption that sup(y,,)) 9) (
0
P sup
Σ i=1
Tora: -
- (T.x)))). u: E T To - - TTOas) - )us))
1)
2
4R' v
21'
(7,8)EB
21
S 4P
sup
Ci (1 T - (Txxs) us
2 R', R/V
(7,0)EB
n2
i=1
< 4P
sup
T - Tollo
1 Σ S:0,2s
> R'
2t'
(1,
n
2t' =: t
< 4af P
R',
e'n
8ff exp(=x (1? //" /ujn),
where K, E (0, oc) are constants depending only on the distributions of the inputs and noise. Collecting results above we obtain that
Tx O' 7- TOmr))u (F(")" -00 - (...).))) 2 Aver"))
TP
S 8f exp(-x min min ("r"//,"""/u (u]n),
where K. U E (0, oc) are constants depending only on the distributions of the inputs and noise.
37

---

Published in Transactions on Machine Learning Research (12/2025)
Collecting all the pieces of the proof in steps 1:3, obtain for each E [0, oo) that
sup (7.9)EB
riskx x (,; 6) risk r7, 01| s t(1 + 4c +4V0)
with probability at least 1 - (2 + Sef + sd2) exp(=x t/v/n) or by rewriting - 18d exn(=xmi(((/p², (1t///// (using the assumption that d > 1), where consider t = i 1" 1"
and K. E (0, o0) are constants depending only on the distributions of the inputs and noise.
]
B.7 Proof of Lemma 4
Proof. The proof follows just basic linear algebra.
Since H(t) is invertible exactly when (A + tC'T has full (column) rank, we are left to study the rank of AT +CC To do so, we employ the Singular Value Decomposition (SVD) of AT E Rd''=w' xw that xw" that U, V are semi-orthogonal matrices and D has the same rank as A, in this case, full rank. Now, we are motivated to make squared matrix as
is, AT UDVT with U Rd''w'' V Il''x'' and D E Rw'xw'
(A -U = D D = tD(
+ D
+--
where used the SVD form of matrix A, orthogonal property of V, and some rewriting. Since matrices U and V have rank for studying the rank of A + +C it is enough to study determinant of U (AT UT +VC We then use our display above, properties of determinants for squared matrices, and characteristic polynomials to obtain
du (AT +1CT)V) det((D(( -D
det(tD) det La + D--UCCTT
r"' det dott
Since, DD) # and t * 0, then the t which Ht) is singular are the roots of p2(-t-1) where Z D-UUCTVV Since the roots of pz the eigenvalues of Z, we have found that the only t for which H)() fails to be invertible are the negative reciprocals of the (nonzero) cigenvalues of Z. Since, any w' w matrix has at most w' distinct eigenvalues, there are just finitely many i such that H(I) is not invertible, desired.
B.8 Proof of Lemma 5
Proof.
The proof consists of basic algebra.
Claim We use the definition of risk 17, θ), 2. the chain rule, and 3. taking the derivatives to obtain
0 risk (v, 0]
1
Σ y/ y =Oxx)²))
0y
n
i=1
2
2
Σ
(yi -. TTx:)) O7g
(< TOxx))
1=1
2
2
Σ
n
(v: -- :en:) ?om:)()m))),,
n
i=1
desired.
38

---

Published in Transactions on Machine Learning Research (12/2025)
Claim 2: We use 1. the definition of riskx (7, 0), 2. the chain rule, and 3. taking the derivatives obtain
0 (isk\7,,6) =
(1)
Σ (Us y Tax.))
n
08ys
2
Σ6
"
(vss 7² -Texx)) θ (r TTe)))
y0
1=1
Σ (y: > -T=xx));;(0t)) (((=)a))
as desired.
Claim 3: We 1. use Claim 1 and 2. remove the term with zero derivatives and use the chain rule to obtain
8²
a
risk 0]
Σ i=1
(y: - T
(Tm:)
2 a
((a:)))(O2:))))
n
desired.
Claim 4: We 1. use Claim 2, 2. remove the term with zero derivatives, and 3. compute the derivative of the bracket, and 4. rearranging to obtain
riskx by, 0]
-
Σ
(v: - -em,)
00 y0 00
00
i=1
0
(2n,)
Σ
2
(7 (xx) (2::)x)
7j
n
2
7s
(m)))
1=1
as desired.
8² Claims 5 and 6: We only show the results for a0 obtained using the same arguments.
risk riskxl7. θ]. The results for 0n,1005 risk (y, θ] can be
.aa
We consider two cases:
Case j' = j, we use Claim 1, 2. the chain rule, and 3. taking the derivatives and simplifying to obtain
02
0
2
risk riskx(y,0]
Σ (y: - (7Txx:)((xx))
08 ,0
00
n
iml
2 n
(Oxx)) 00 (y: x Txxx)) + 1811- x TTxi)) θ (0xx))
00,
n
2
(2:())) (Gw,)) (ye > TOx, (x;)))))))1)) (2::)ss)
1=1
39

---

Published in Transactions on Machine Learning Research (12/2025)
Case 2: if j' + j, we use 1. Claim 1, 2. the chain rule, and 3. taking the derivatives and rearranging to obtain
8² 08 8 37g
a
risk x 67, 6)
(vs - - -xxx))
08 j
i=1
a
Ô (Qw:);
Σ
(Daz))s
(yi - O + (y: r Om:) yly
2 73' Σ
(
n
desired.
B.9 Proof of Lemma
Proof. The proof for this lemma follows the same steps as in Lemma 5 just sums are replaced by expectations and so we omit. the proof.
C Appendix: Proofs for shallow ReLU networks
C.1 Proof of Theorem 3
Proof. The proof approach follows almost the same line as in Theorem
We use the notation ( to make a rescaled networks using a suitable a (see more details about rescaled networks in Section 6 Using the above definitions, it is easy to see that (Ia(Oa = 7 (Px), that means, the output of the rescaled network is the same the original network (using the definition of rescaled weights and Lipschitz property of ReLU networks with Lipschitz constant one).
Now, let's start the proof by writing second-order Taylor expansion of risk/y" a. O" (the risk in rescaled version of the target with B = vec(y" a. O'a)) E IR?) around a rescaled version of a reasonable stationary Ba vec(Fa. ²a) H2 with suitable a E _ to get
risk(y" an or al = risk/Ta 0al + Vrisk Cal (B" - Ba)
1 1 - 3a) rirk + 1l7' - Ta), + + -6.)(38 - -3a)
-3)))
for some E (0, 1) [Bertsekas et al.
2003
Proposition 1.1.13.a).
Then, we employ the property of rescaled networks that is risk 0.] risk|5 d and risk(y' a. O'a] risk(y", 0*), and use the shorthand notation
m = (B' - BBa) riskkl/aa + - Ta), - - -Ba)
3a]](B'
to obtain
risk/y", risk F7, θ Vrisk/Far Tal (3' -Ba) + =..
-a)
It also straightforward show that Vrisk T Pa) Vrisk(fr T (3 (we omit the detailed proof). Tabulating this observation in the earlier display we obtain
1
risk hr" 0*] risk Vr B) + 2 -.
Rearranging the display above obtain
1
-rissk(y, - -3) risk θ risk(y", "akk""""']]]" + m.
2
40

---

Published in Transactions on Machine Learning Research (12/2025)
Now, let's recall the definition of stationary points in equation that implies Vriskx|5; 6] (3" - () + (3" B) 0.
We rearrange the above incquality and expand the bracket, use Hölder's inequality and the fact that 2 3 1/81 (recall that 01 and use |0 I to obtain
-risk 1,, [ - 3) S r 8r r
s
r1
s 1 - -11311
which rearranging implies
Vriskx [ [,,0]T(88)) -8" +
- - 0.
Display above reveals the positiveness of the terms on its left-hand side and we can obtain
(3* - () S - -risk((f, (3* B) + Vrisk Vriskx|5,66) O5 - 3) + r1 - -11811 -rrisk((, TTT(")
that is,
[risk(7,,)]T - 3) s (Vriskx 6) Vrisk [7, θ1) T + -
1 -
Now, let's use display earlier (obtained by Taylor expansion) to rewrite the left-hand side of the display above
risk θ] riskly", ,0 2m S (Vriskx 17, Ō] Vrisk(7, Ø1) (B" (3) +r1 r|:11
Rearranging the display above we obtain
risk |r. 6) < risk(r", "aak(>""0"*==+" + + (Vriskx (7. 6 Vrisk θ1) (B" - (3) - -r13) 1
2 m.
For the right-hand side the inequality above we 1. get an absolute value of the third term, 2. add a zero-valued factor, 3. use triangle inequality, and 4. Remark !
obtain
risk 0] S risk( - + (Vriskx Fr. θ] Vrisk θ1) (8"-33) 3) r| 2 2m
1
riskll"","7 -r) (Vrisk Dankkx((i: (7, é] Vrisk [5, θ)) (88- B) r(17]) + -11811)
1
2 m
< risk/y", - 1 + (Vriskx [7, [7, Vrisk θ)) (8r B) - -r|g-- B1
2 m
rrisk(>", 3-| Fore 1/9 - -B11 +
Tore
r113" - 31 2 m
2n
with probability at least 1 1/2n. The third and fifth terms in the last inequality above can be canceled if we choose the tuning parameter large enough. Hence, we obtain
risk θ < riikky",, 0*] +0]++ + 2r + Tore 1
m
2n 2
for r 2 Fore (sce Romark
The rest of the proof is analyzing the behavior of m. Let's rewrite m -18- - -Ba13 m' with
(B'a - -Ba) v rirkk 1(T" a 7a); G. + 110" -Ga)l (B'a a BBa)
m'
18aa Bal2
1 Ba 12
41

---

Published in Transactions on Machine Learning Research (12/2025)
Now, we are motivated to employ our results in Proposition 2 To do so, need to make sure about matrix (θ + 1(0" - ))) to verify our required condition, namely, active rows being approximately perpendicular. ~ Employing our assumption that the stationary point θ and O* have approximately perpendieular (active) rows and they have negligible cross-alignment (off-diagonal elements of 60*T and @*ÔT are approximately zero), we can show that the line-segment between the two endpoints also verifies the assumption of Proposition 2 (active rows are approxinately perpendicular) and so ensures the Hessian exhibits well behavior. To be more precise, (0 + t( ( + t(e - = = (1 t)) + -0 + + - () + O*ÔT) will be approximately diagonal, assuming two end-points having approximately perpendicular rows and that off-diagonal elements of 00* and Q are approximately zero.
Implying Proposition 2 (with (8 a 3a)/18*> Bal2) we obtain that m' E [0, x)) for appropriate a, that is, a with large enough c. The observation that m' E o0) together with the definition of m implies that m E [0, oc) as well.
Tabulating this observation to the display earlier together with our assumption on 8r (188), = 1>*ll + 10 s 2) and the fact that 1/2n I vlogn, we obtain for all Tore that
Forc 1
risk risk θ v risk risk(r". @*] + 2r +
m
2n
2
+ Torc
~ risk(7"' O*| + 2r
2m
S riskly" θ*] + 5r Vlg
with probability at least 1 1/2n, which completes the proof.
C.2 Proof of Proposition
Proof. The proof is based on basic algebra and the property of scaling weights across the layers in neural networks. Without loss of generality, we assume that æ; E (O,, Taxa) (the proof for independent, and centered sub-Gaussian random vectors æ with independent coordinates is the same, just some constants may change, which doesn't 't affect the main results).
Let's consider all the network parameters as a vector of length p (recall that = w + w- d). Then, we can tabulate the second order subdifferentials of risk(y, θ] in a matrix called ririkk θ] E Rexx (for notational simplicity, we focus on V²risk (7, 0] for the moment and then we move to rriskk (o Oa] the end of the proof) of the form
A C
Vriskly Q
B
D
with E R B E (w-d))x C (wx(w and D IR
(ww-d)x(w-d)) where
82
Ay On
risk (7, 0].
0²
By--1)d-Wa =
risk (7. 0].
0A i
8
Cÿ.d--na+ =
risk θ],
8y 00
D DGG--1)4A((6-1444 = a0 00,
risk θ]
for j.j E {l,..., w and k E dl. Applying the block-wise structure of V'risk(yy, θ], we are motivated to analyze the behavior of
V (7, θJa = (at)T Aat + (at)T Ca² + (a²)T Bat Baa + (a²) Da?
Note that C B (by symmetry), so, we are left to analyze the behavior of
T T O|a (at)T Aat + 2(at) Caa + +aa) Da² Da²
42

---

Published in Transactions on Machine Learning Research (12/2025)
for all E R' with |a|2 1.
We do the proof in steps: We start by going through the three terms on the right-hand side of display above separately, to write them in a mathematically nice formulation (Steps 1:3). In Step 4, we sum up the results computed in Steps 1:3 to prove the main claims of the proposition.
Step 1: On a high level, we prove that the entries of the matrix D are a function of 7.
Employing our results in Lemma g the symmetry over the input, and our assumption over θ for k = kv' and 8²
0² risk (7, θ = "y7,// /2, and for k = k and j = we obtain 30,200, risk (7, 6) = 7, j + J', we obtain 80, 00, For other cases (A # k') we use 1. our results in Lemma 2. cauchy-schwarz inequality, and 3. our assumption the input (symmetry) to obtain
8²
risk(7: (,k 0] = 22577 Ex
(x) (x)) > 0, (Ox) > 0}]
00 00,m
2/751175/y Em [()) > 0})3 ((xx) 1 > 0})
- h
Step 2: We prove that for at R= and R
Ta')TAa Aa ~ (1 (
(a'))) 2 { VT
Σ
For ReLU networks and according to Lemma
we have
(at)TAd Aa = Σ a AAyya' jz
in which Asy- = [(xr)((0)) (0x), > 0, (Ox); > 0}}. Employing some basic linear algebra implics
(a Aa
Σ aa (aa))) A
Σ j' #1
Σ
at j Ayya''s 3
j=1
10
> 1=1
(at) , [Oxx) (Ox), ((0x), 03
+ +
a y Ex (Ox) 1 (0x)) > 0, (Oz), > 03)ay.
j=1 j mm 1.
2 >
(a') 2 ((Ox), - Ex (1( 1(x)) > 0}]
1=1
+ 2
a y Fu (()) > (Om) > 0)]ayy
j=1 /==1 1 #1
Σ ;) F [(Om) EE ((0m)))))
w
a' a Eu (Ox) (((0)) > 0, (Ox), > a
j=1 j 1.3 tj
1
(a (00 )a3 +2 +2 80
Σ
Σ
a EE ( (0x), > 0, (Ox), > >
j=1
43

---

Published in Transactions on Machine Learning Research (12/2025)
We can prove that for cases with small |pyr| (roughly about |ørr|| S 0.2), where Pjs' is the correlation between the (Oa:) and (Qx), with Gaussian æ, can approximate
Ex
0, (Ox); > 0}}
1 2x
+
Pij
3p
|0,110,1
4
4m
To be more specific, we can reach above result from scaling properties of Gaussian distributions and the homogeneity of the ReLU function together with Lemma g
gr |
E=
10,10
10,
10
2 Dj' |6,1
|0,|
1 19l 1|
a
|
Then, we have
w
d
W
Σ
(a
Σ
83k 2
+
Σ
a' E ((z)) ((0x)) > 0, (0x); > a'y
-=1
k=1
j=1 j j'= X+)
3p3
(a (al)) , +
a a'ja" / 110
Pig
2T
4m
1 Σ (a 100,, + 1 M(aa) (a ,
> (a') 10
π
(
Paj' 3p
Σ Σ
at j ,0,
2T
o'=
10
2
Σ (a) ) (1 +
Σ
(a ,,11,1111
VT
1=1
1=1
3p3d
+
a' y 10,110, + y'=''4
/
Pis'
2
2π
In the last equality above, the first two terms are our desired terms, while the last term still needs care. But we can argue that for small correlation values, we can ignore this term as it is a function of Pjr' employing our assumption (rows are approximately perpendicular). Also note that for inactive rows, we are already good, since related factors will disappear from bounds.
Step 3: On high level, we prove that the entries of the matrix C' are function of the product over O and y. Expanding (a (a')TCa² Ca yields
0 0 riskly, 01) (a (a²) (o--aat)
(a') Ca²
(a'))
Now, we need to consider two different cases:
Case 1: (j f')
We use 1. Lemma g 2. rewriting the ReLU function, 3. rewriting the product in the form of sum, 4. linearity of expectations, 5. again lincarity of expectation and rewriting, 6. using the assumption over the input, and 7. the same argument as above to obtain,
risk (7, 0] 22 ( (2))
00 00,
44

---

Published in Transactions on Machine Learning Research (12/2025)
(x)) (O)) 0)1[((x), > t3l
2yy:
(I))
())xts) 1[(0)) > (]1{ 0]
d -27
Σ
Ex (2)) (0) [((x)) > 0}1[(0x), > 0}}
k=1
-2 Le (x ) 11((xx) ()1((0x); 0f
2713'
Br 1 ((x)) 0) ((x); > 0}]
1=1
1
2 55 + 2773'
Σ
Og o 1 (((x)) > 0]1[(Ex); =0}
Ex 24 (()x)) > 0}1{(0x), > 07]
1
25 2
+
7/3'
0z
4
Case 2: ji
We use 1. the result of Lemma 8 2. lincarity of expectations, almost the same proof above for simplifying the first term, replacing with its definition, and the assumption over noise to obtain
8²
risk(y, 0] 2E (1)()) (Ox) - (y - (z)
00 0y
1
5k +
133
2x
k= 1,gfft
(>7 ( - ((00)))))0)0)))))))|( > >0]
2.
Then, we use the lincarity of expectations to obtain
En ((7 T(@)) - =0) > 0)]
=R (1) 03} E (()) > >0]
and
" En (7 T (()))) > >0}}
Σ
Ex
> 0]]
y'=1
Σ
k=1,k
The same argument. can also hold for the other term. Looking at the extracted entries of the matrix C above, it is clear that the entries are a function of the product over parameters of the first and second layers.
Step & Collecting the results from Steps 1-3, we can casily approve the first claim. For the second claim, we realize that by employing the same scaling trick as in the linear case, that is considering parameters of the first layer large enough (by selecting large enough) and dividing 7 by the same value, the result from Step (that the squared of the scaling parameter will appear in the front) can dominate all the other terms. According to Step 1, the entries of the matrix are a function of y and also according to Step 3, matrix C involves product of first and last layer parameters, which in this case cancel out the scaling parameter and
45

---

Published in Transactions on Machine Learning Research (12/2025)
so, the result from Step can dominate all other parts, as long as is selected large enough. To be more precise we have
a T V'risk(Ya Oala = (a') Aa +2(a") + (a²) Da
2 (al)* Aa +aa) T Da 2 |2|[[|2|a||
m (al) T Aaa -1a' 1112 - 2
2 Σ V (a' ,)0
(1 a 12022
|| 12 2||
for all a E R' with |a]l2 = 1. For large enough 0, the first term in the last incquality above can dominate the last two terms, which involve the product of parameters that cancel out the scaling constant or they are just dependent over 7. For the special case of a² 0, if we consider large enough 0, the entries the matrix D can go zero (so implying its norm || going zero) and so we can reach our desired results.
C.3 Proof of Lemma
Proof. The proof consists of basic linear algebra.
Claim We use the definition of risk x 17, 0), 2. the chain rule, and 3. differentiating to obtain
0
risk x br, 0
y) 7² ((x:)))*
07j
i=1
2
07j (vi y (0x:)) (r (T(o)))))
i=1
2
n
(y: y ((0m:)))((aa))
n
desired. Clain 2: We 1. use Clain and 2. remove the term with zero derivative and use the chain rule to obtain
a (-2
risk x
01
θ7g n Σ (
(T((())))
01
(((0))))
as desired.
Claim 3: We use the definition of risk 17, θ), 2. the chain rule, and 3. differentiating to obtain
0 riskx ir. 0]
0
(
Σ (y: TTo (o(())))))))
n
00,k
00,,
n
i=1
2
(v x" o(Ox:))
00,r (< ((xx)))
i=1
2 n
(yi - To(a:)) (t.)))))))
Claim 4: We 1. Claim 3 and 2. differentiate the bracket to obtain for
46

---

Published in Transactions on Machine Learning Research (12/2025)
0
risk 1ookx|7,,00 0]
2
Σ
(v: -
To (00(
00 000
11
(275
Σ
(7 (000((0))))
3 yk
i=1
2 7
(U(
08
We obtain then for j' # j that
0²
risk x 07
2 7
Σ
08 00
i=1
2
7573"
and for j with (Qx:); + 0 for all i E {1,....
0
0
2
risk (r, 07,
7
00 00
06 y0
n
0
2
(vi(mi)an(asss)))
95
08
257 (x; )e (2:))n Dr(2,,,)
otherwise, the corresponding subdifferential doesn't exist, as desired.
Claims 5 and 6: We only show the results for 8² obtained using the same arguments.
risk risk (77, 0). The result for 5 risk riskxly, O] can be 00
d²
We consider two cases:
Case 1: for j j use 1. Claim 1, 2. the chain rule, and 3. differentiating and simplifying to obtain
0² risk riskxl1,00 0) =
0 2
(vi x (((())))))(mmm)))
00
00,k
i=1
n Σg
((0m)))
(v: - y ((xx)) + (v: -T (0()))) 00,
(0x:)))
00,M
2
n (2(()))
(y: - (z:z))
Case 2: For j' + we use 1. Claim 1, 2. the chain rule, and 3. differentiating to obtain
2 Σ (v: ((00
riskx (7, 0] = 50,
00
n iml
2 n
Σ
((xx))
(yr - > TT(x))))) 20,
8
n
2
78
(x: of (Ox:))(0
47

---

Published in Transactions on Machine Learning Research (12/2025)
A similar approach can give
8 risk ()
0 (-2
2 Σ
(vs -1- (
For j= j' we obtain
82 07y.00j
risk risk 07, (-- Σ
i=
2
(
+
==1
And for j j we have
02
2
risk x 17, 0)
(y: - Y To(@x:))) (m::))
0yr.00f.
71 ==1
n
(1)z:))7(((i)))n(
n
as desired.
C.4 Proof of Remark [
Proof. The proof can be followed almost in the same line as in Lemma 2 and Lemma so we just provide a high-level proof here. The only difference with linear case is how to treat the ReL.U function in subdifferentials To do so, we study here the behavior of the absolute difference between the subdifferentials of the in-sample risk and population risk for ReLU networks, showing that they almost behave the same as linear networks despite minor changes in the constants and some log terms. First, we use the definition and employ some lincar algebra to obtain
a j0 jk
riskx (r,
risk 6]
ä
2 n
U/ y or (oxx)) ((0x::))((())))M((x.))) + +E
v > o(00)))))y(()))))(()
< 2/yr|
Σ
Uu: r T (00' y To(e))))
i=1
E((r** a(
Ta(O x;)
x To ((x))) (2:)
Σ uc (x:) + | (r ((0))( -o(r)))T0((TT"0>>> (0)))=)))
2/
1 M (Y σr (0xx,) (o:)) B[(TT ((Oz))) (
The first term in the last inequality above was already treated in Lemma So, we continue with the second term. We use 1. Hölder's inequality, 2. symmetrization (Bühlmann & Van De Geer 2011 Theorem 14.3)
with Si as Rademacher random variables, and 3. an extension of contraction principle to obtain
( (00x)))) (z) -E((r* To (00' (z:: 1o
48

---

Published in Transactions on Machine Learning Research (12/2025)
s hh' (((0"))(a))))() (Brr::)))))(((mmmm) ((o(0")))())))))m;:.)))
/
1l
==1
s 4 (o [o
Then we consider 2; a0"',; (2:)) as independent and mean-zero sub-exponential random vectors and the proof can be followed same line by the proof of Lemma Also for jöriskx (7, 01/07; Örisk(y, 01/0y, we obtain
θ 0r
a risk 7 6]
risk x 07,
2
(ye 7 T (( - B[[( - ((0a))))
(us + 7x o(0'x;; - -y a(0x;))) ( - -(((a)(((''a,)
0(0'x,) - (0Toa=))
2 Σ6 ((()))))) - 7 T ((Oxx)))
2 (Ox;)) 1) +
i=1
E[(x"] T o(0"x:;) - 7 T o )
4 (
(Oa:))
+
n
4 Σ (r" ((0
4 (r ((xx)))
VA
ue (ex,))
+
+
Treating the last two terms: use Hölder's inequality to obtain
( [(T(0r)))(00)))) s |r1
4
((0))))(0m))
where 2; = ( are, mean-zero and independent sub-exponential random vectors (again can be followed as Lemma 2
The same also true for
4
To ((0':))) s 1r
Σ x ( (0))))))
with z; = o ((0))))))(0))))) again as independent with zero mean sub-exponential random vectors.
as desired.
Appendix: Complementary simulations
We show the log-training error for shallow linear and shallow ReLU neural networks in Figure 3 To extend the simulations Section 4 we show the relative error and test error for different setting (with d = 100, = 20) in Table 2 Moreover, we run our experiments in the numerical observations section 200 times (each time run 100 runs to compute the potential global optimum and approximate stationary point.) to reach the
49

---

Published in Transactions on Machine Learning Research (12/2025)
Shallow-Lnes Liner Network
Shallow-RelJ Network
-075
paniy uno.
Tahirn
1.7
-00-
-223
10
15
20
25
30
200
6:0
650
80c
1000
Re-voon GRaIs
Eorataa Court
Figure 3: Log-training error for neural networks (with d w = 10) with linear (left panel) and ReLU (right panel) activations 10 different runs (allocated with different colors). Due to the non-convexity of neural networks, optimization algorithms may end up in different approximate stationary points.
mean and standard deviation of the relative error for the approximate stationary point. For the network with d= w 10 and linear activation function, we reach the relative training error 1.0013 = 0.0003 and relative test error 1.0011 + 0.0003. For the ReLU activation function, reach the relative training error 1.004 + 0.001 and relative test error 1.005 0.001. The same experiment for the larger network (d= 100, w 20), concludes 1.04 1.04 1.03 + 0.008, 1.89 + 0.07, and 1.40 0.08 for the relative training and test error of linear and ReLU activations, respectively. These results show that our empirical observations are stable. All the simulations were executed on a local computer (Apple M2, 16GB memory), with an average run time of less than 10 minutes per individual run in Python. For optimization, we employed SGD with the learning rate 0.02.
Table 2: relative training error and lest error for trained neural networks (with d = 100, w 20) with linear and ReLU activations in potential global optimum, an approximate stationary point, and randomly generated network.
Linear
ReLU
Training Error
Test Error
Training Error
Test Error
Potential Global Optimum Approximate Stationary Point
1.00
1.00
1.00
1.00
1.04
1.03
1.85
1.10
Randomly Generated Network 1146373.94
1095543.69
5062.83
3626.28
In Tables 3 and we repeat the experiment from Section 4 this time employing different initialization strategies namely the random Gaussian initialization and its scaled variant. For the random Gaussian initialization, weights are drawn independently from a standard normal distribution. In the scaled version, the weights are subsequently rescaled so that the 4--orm of each layer individually satiss [[], S Vlog n. Results in Table 4 perfectly match our previous observations in Section 4 Thanks to our initialization technique, we now expect the weight matrices to also satisfy the required assumption for ReLU networks (see further discussion following Theorem 3 that random Gaussian weights yield nearly orthogonal rows with high probability). Our results Table show that, since the weights are not scaled and their norm bounds are large, the behavior of approximate stationary points does not closely match that of the global minimum. This clearly indicates that initializing weights with small values significantly aids the optimization (supporting the need for our reasonability assumption). We also conducted experiments with a larger tuning rate, namely of the order log log(np)/n1/4 as shown in Table 5 Comparing these results with Table clearly remonstrates the optimality of the tuning rale log(np)/ V (vs log(mp)/n1/4) supporting our proposed oracle tuning in equation 4 We also examined the relative error of the regularized estimator (same setting as Scction 4 across a range of tuning parameters, obtained by multiplying a base tuning value by different. factors, as shown in Figure D for both linear and ReLU networks. The results clearly illustrate a bias varianee trade-off when the tuning parameter either too large too small.
50

---

Published in Transactions on Machine Learning Research (12/2025)
#estie lnt onr Ralatve Tra
1045
-30
1.00
1.002
1N0M Erno
2.00
110
1002
1.02
Figure 4: Relative error versus tuning parameter for shallow networks. Left: linear activation; right: ReLU activation. The results clearly illustrate large or too small.
trade-off when the tuning parameter is either too
Table 3: Relative training error and test error for trained shallow neural networks (with d = 10, w 10) with linear and ReLU activations in a potential global optimum, an approximate stationary point, and a randomly generated network employing random Gaussian initialization.
Linear
ReLU
Training Error
Test Error
Training Error
Test Error
Potential Global Optimum
1.00
1.00
1.00
1.00
Approximate Stationary Point
1.13
1.15
3.69
3.61
Randomly Generated Network 22576.49
17970.63
4157.68
2835.71
Beyond SGD: For the sake of completeness, we have now included further sirulattins to assess the impact. of changing the optimization method. Specifically, we replaced SGD with Adam, using a learning rate of 0.005, to analyze its effect the simulation outcomes in Table 1 Our results are reported in Table 6 These results show that the performance of SGD appears to be more aligned with our case (compare results in Table t with Table T which is high likely due to the verification of our assumptions for the corresponding approximate stationary point, but in general, approximate sub-optimal solutions remain still satisfactory.
Conjecture for deep newral networks: We have now extended our simulations in Table I employing neural networks with layers. Our numerical observations make this conjecture that our theory can also hold for deep networks (with possibly minor different rates), given we reached the results in Table 7
Conjecture beyond regression: We have now extended our simulations by employing more complex networks and testing beyond our regression simulated data. We applied our method to the MNIST, fashion-MNIST, and K-MNIST dataset using cross-entropy loss, with a neural network consisting of 10-layer weight matrices and ReLU activations, with network width 50. Our results continue to support the same conclusion we aim to demonstrate for approximate sub-optimal in Table S This observation can support the conjecture that our results can be extended for classification scttings and even for deep neural networks in further studies.
E Appendix: Relaxing the nn-mrrm bound
In fact, the bound vlogni ii is merely for convenience: it can be replaced by any fixed constant or another function that increasing slowly in the sample size n. It basically means that fo-norm bound can be replaced by cylogn (with e e (0, oc) an arbitrary constant) or q(n) that the function q(-) is just mildly increasing in the sample size n. What we end up by moving to these bounds that our rates change ((log n)² (log(pn))//)) or (((g(n)))) (llg(m)))/n)) respectively that makes sense once e and q(n) are mild.
51

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Relative training error and test error for trained shallow neural networks (with = 10, w = 10) with linear and ReLU activations in a potential global optimum, an approximate stationary point, and randomly generated network employing scaled random Gaussian initialization.
Linear
ReLU
Training Error
Test Error
Training Error
Test Error
Potential Global Optimum Approximate Stationary Point
1.000
1.0000
1.00
1.00
1.001
1.0003
1.02
1.05
Randomly Generated Network 79618.240
240
4130.04
3650.06
Table Relative training error and test error for trained shallow neural networks (with d' = 10, w = 10) with linear and ReLU activations in a potential global optimum, an approximate stationary point, and a randomly generated network employing larger tuning parameter (log(np)/n//4)
Linear
ReLU
Training Error
Test Error
Training Error
Test Error
Potential Global Optimum Approximate Stationary Point
1.00
1.00
1.00
1.00
1.01
1.009
1.10
1.06
Randomly Generated Network
11498.68
8271.43
4157.68
2835.71
More explicitly, let's define
log(np)
Torc.q = e' (q(n)))
(15)
the oracle tuning parameter, where e' E (0) is constant that depends only on the distributions of the inputs and noise. Then, we get the following result:
Theorem 5 (Statistical Guarantees for Norm-Bounder Stationary Points of Shallow Linear Networks). Suppose that the second and the third part of Assumption ! are satisfied and that Iy 10 S q(n)
for a fired function (n) E 00). Then, any reasonable stationary point (7. Q) of the objective function in equalion with r 2 Fore.g salisfies the risk bound
risk(y, Ō] S "risky", +5g(())
(16)
with probability at least 1 - 1122n.
In the theorem above, 1. (y", O*) pair that approximates the target function and by reasonable stationary, we mean that 17/2, || I q(n). The proof of this theorem follows the same steps as our Theorem and so we omit the proof.
Another interesting and practical point in the training process of deep learning is that neural network weights are usually initialized by near-zero values. For example, PyTorch by default initializes weights uniform(-1/ 1/vD) (p refers to the number of parameters in the network), that means the - of the matrix and vector weights are very small. Then, in the training process, the optimization algorithm looks for a stationary point around the initialized network (and not too far from this space). So, it is more likely that the computed (approximate) stationary point has a small norm, while there might also exist other stationcrics with larger norms. This argument shows that even from practical point of view, the reasonability assumption on stationary points and the points nearby makes sense.
F Appendix: On the reasonability assumption on the stationary points and the points nearby
It stated in the text that the reasonability assumption on the stationary points makes sense. Here, we prove that claim by showing that the reasonability assumption on the target also implics reasonability on the stationary points.
52

---

Published in Transactions on Machine Learning Research (12/2025)
Table Relative training error and test error for trained shallow neural networks (with = 10, w 10) with linear and ReLU activations in an approximate stationary point employing Adam.
Linear
ReLU
Training Error Test Error
Training Error
Test Error
Approximate Stationary Point
1.0007
1.003
1.20
1.27
Table Relative training error and test error for trained neural networks (with d 10, w 10, and depth 4) with linear and ReLU activations in an approximate stationary point.
Linear
ReLU
Training Error
Test Error
Training Error
Test Error
Approximate Stationary Point
1.002
1.004
1.16
1.21
Following the same lines in the proof of Theorem have
risk|R: θ < risk(y", 0*] +*]++|13*1
(Vriskx (5, θ] Vrisk "
11 (8--3) B)|
--11
1
1
2 - 2 zm
= rriskl"", 0-] + --
(Vriskx [7, - Vrisk([, θ)) (8* ]
1
r1111 m
< <riskly", 11
3
(Vriskx 17, H] Vrisk|f. Q1) (8r -B}
2 r -311
1
2
1
1
1r11111 2 z
2
S riskkl"", 5r18"
3
2
+ rore - -B11 +
Tore
2n
1 - - -B111
2
-
1
2 m.
1
Moreover,
1 risk/7. θ] +
r1/:: 2
3 risk(r",
2 3-1,8%1 + -ooeel." -B111 +
Fore
2
1
r|
BB1
2 m.
1
2n
Then, by considering r 2 2ro we have
1 risk θ] 1/1001 S risk(y", "00]i"",']]) + 2-111"
3 0*]
m.
Following the same argument for as in the proof of Theorem we obtain
risk(f,, θ0 | < "00l"""0'") + =22"""" rrll8" +
Tore 2n
and
3 2 Torc 1||1 S risk(yy 0* +
2 Fore |
Tore
2n
Finally, by assuming a small variance in the noise and reasonability assumptions on the larget, we can conclude (for large n) that
1 < 41811 < 4olo.
| x 31
The above display revcals that having a reasonability assumption on the target can also imply reasonability on the stationary points as well, once tuning is selected large enough, which also implies reasonability on the points nearby.
53

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Relative training error and test error for trained neural networks (with w 50 and depth 10) with
ReLU activations in an approximate stationary point.
ReLU
Training Error
Test Error
A Approximate Stationary Point (MNIST) A Approximate Stationary Point (Fashion-MNISST) Approximate Stationary Point (K-MNIST)
1.0004
1.39
1.00005
1.40
1.00003
1.18
G Appendix: Dynamical accessibility of approximate stationary points
In this section, we argue that r-approximate stationary points can be reached in practice (in a reasonable time) once gradient- based algorithms iterate sufficiently.
For non-convex and differentiable objectives ((3) with gradient-based methods, dynamical accessibility of approximate stationaries 3 E B (points with small gradients [V((3] < 7' that 7' (0, x)) have widely been
studied (Ghadimi & Lan 2013 Carmon et al. 2018 Wang & Srebro 2019 Lei et al. 2019 Drori & Shamir
2020
Arjevani et al. 2022
Here, we provide some results from Ghadimi & Lan 2013 and Lei et al. main results, we impose some assumptions:
2019
Before going through the
[=,, = VE(B),
3 og E (0, 0o) E. (1(,
ve(B)] < og
(17)
where g(B,2) an estimator of VE(B) computed using a subsets of samples called 2. And
A Lg E (0, oo): E(B(()) - inf ((3) < A
IVE(3)- - ve(''\ - Lglß I1l 03, E EB,
(18)
BEB
where (()(0)) is the value of the objective function in the initialized step. Then, Ghadimi & Lan 2013 Theorem 2.1) prove that SGD finds an estimator such that ((IVE(((()) []] < r' for randomly selected
R E (1....,T) ,) (according to certain probability distribution, see Ghadimi & Lan (2013) Equation 2.3)), where the expectation is taken over R and the randomness of SGD, using O(ALgo)) ( oracle queries. Above result also imply minee(n.....r [[[V[((())] < r' using O ((ALgo//(1)) (+")** oracle queries.
We can argue that Assumptions equation 17 and equation T8 can hold the setting our paper: for Assumption equation 17 and the first part of Assumption equation 18 (objective has bounded initial suboptimalityy), we can use the reasonability assumption over the parameter space. For twice-differentiabl objectives, the second part of Assumption equation 18 means that the eigenvalues of the objective's Hessian are bounded above by Lg which is typically a reasonable assumption.
Important here is that E[[VE(((())] < and our definition of approximate stationary points in equation are in sense similar. Using 1. the definition of the objective function, 2. first order Taylor expansion of (()) around ((B) (with 3 = ) 3. Hölder's inequality, 4 our definition of B, result above, and the reasonability of approximate stationary and exact stationary we obtain
riskx 15, θ] + r|13||, risk x 17, oj r||31|, = ((3) ((3)
(ve(3) T
(g 3)
vE 3
S c Vog
for a constant c E (0, oc). It means that having a small norm on the gradients of approximate stationary can also imply small difference between the objective function of the approximate stationary and exact
stationary. The results of Ghadimi & Lan (2013) imply that gradient- based algorithms with sufficiently many steps, let's say O(n?), can guarantee small (11 nn.
54

---

Published in Transactions on Machine Learning Research (12/2025)
Lei al. 2019
Theorem 3) prove that for differentiable loss functions with a-Hölder continuous gradients:
3 Lgaa (0) [V((8) VE((3') S Lg.a 8/1 vB, e B
(19)
where E (0, 1] and Lg.a o0), SGD gets
E [[ S C s (e(I....)
min
n =:
where is a constant independent of t, Nh are stepsizes satisfying Ei 1 +o < oo, and the expectation is taken over the randomness of SGD. Lei et al. (2019 Theorem 3) reveal a rate of convergence 1/T for the smallest gradient. As a comparison, the convergence rate in Lei et al. (2019. Theorem 3) only holds for the minimum of the first T iterates, while the convergence rate in Ghadimi & Lan (2013) Theorem 2.1) holds for ([[VE(3(R))]] that is more practical (we also used Ghadimi & Lan 2013
Theorem 2.1)).
H Appendix: Heavier-tailed noise
In this section, motivated to provide materials proving our Theorem
First, we present an adapted version of the result in Bakhshizadeh et al. 2020 Corollary 2):
Lemma 10 (Empirical Processes for Heavy-Tailed Data). Suppose 21,.,,,2 centered i.i.d. random variables whose laill is caplured by (() cata/a for some E oo) and ca E (0, Moreover, assume E[ & 0)] = (ax)² < co. Then, for all E (o) we have
Σ 22
>1
Gnexp(-cminfan (n1))/a)),
(20)
where a conslant depending on the distribulion of zs-
Proof of Lemmal 10 The lemma is just an adapted version of Bakhshizadeh et 2020 Corollary 2) and reached in three steps:
Step 1: We use the result in Bakhshizadeh et al. (0020 Corollary 2) that gives
nt?
P
Σ 2, > i=1
< exp
+
+ mexp(-ca(nt)///) (21)
2ū(nt, B)
where 3 E 1) is arbitrary, a E (0,1) constant depending on n and t, and
( = (o.)² + I(2a + 1) ((1 -1-33 3)c)) 2a
+ (nt) BcaI + 1) 3((1 - 3)c,)) 3a
Step Since the factors & (0, 1) and ū(nt, 3) depend on and I, we need remove this dependence, otherwise we are in trouble. We can easily remove the constant o from equation 21 because there is max function there. Also, the factor ö(nt, f3) in the rate above is basically bounded from above. For example, for large enough (t 1/n) and specific 1/2 have
M(2a + 1)
Co T(3a +1)
b(nt,3) < vo = oa? 2+
+
2²a
30
where Cj E (0, 00) is a constant. Then, we reach
< 3n expe(cmmmm (nt))
2i > 1
1=1
55

---

Published in Transactions on Machine Learning Research (12/2025)
where min(1/2va. ca/4,
Step 3: We use the symmetry of random variables 2, moving to two-sided tail by paying factor of two as desired.
Using the above lemma, we derive a uniform bound on the absolute difference between risk x (7, O and risk(y,, θ] for heavier-tailed noise.
Lemma 11 (Difference Between Vriskx (7, Q] and Vrisk(y, θ for Heavier-tailed Noise). Under the first two parts of Assumption 7 it holds for each n, E (0) and B Cy.e = {3 vec(yy. Q) E R2 || B11 S that
and T 7" 0e {
(Vrisky b,, 0] Vrisklyy, 0]) (8* (8--3)) -B) - 2tn(n + max lo, l0))
sup BEC...
with probability at least 1 - 12d2pn pn exp(-cmin (infmmmnnn ntf, (n)) with constants c E (0, and a E (2, depending only on the distributions of the inputs and noise.
The proof follows almost the same steps as the proof of Lemma The only difference is handling the empirical processes parts.
Proof of Lemma 17
We start the proof with Hölder's incquality and the definition of Cr,e, which implics 1 -B12 17 for all B E Cy.e to obtain
(Vriskx
θ1) (B" - -9)
sup (=),,
V/ 0] Vriskly, 01 B11)
sup
1) sup (=)
Vriskx (1, Vrisk/y, x
The rest of the proof employs our Lemma (U) Vriskk by. 0] Vrisk 0]| Note that for simplifying the notation, we use E[] as a shorthand notation of E ( ((...p))...cc........... throughout this proof.
5 and Lemma 10 to find an upper bound for
We use 1. our result in Lemma 5 and i..d. assumption on the data, 2. equation [ and our assumption that [[a] = T O' x, zero-mean noise, lincarity of expectations, and factorizing, the definition of sup-norm, triangle inequality, and Hölder's inequality, 4. the definition of Cy,er which implies ly - - TOlI S e, 5. adding zero-valued term and rewriting, and 6. the triangle inequality and the definition of Cr,es which implies 1
7111 |B - 11 s n,, to obtain for each E {l...., w) and k {I,...,d} that
0 risk (7, 0
a
riskly, 6]
00, 1k
00,k
2
2
(v:- - 7" Oxx yr, (2:)4 + E
(u. >
(2)))
1=1
1=1
= 2/70|
u (x(x))) x + (7 T - >
- -(p(())))))))))
+ -- 1 Σ EE (2))()))) (x)))))))) 1
< 2//7/00
u u )
1
221/7
ui (m::))
+ [((;(()))))] - (2)))))
(11) 2 (xxx))
= -2/ - . + lo
+ Σ (x:)a - (2:))))
56

---

Published in Transactions on Machine Learning Research (12/2025)
( " us (r:))
1 n
ar (2;)) (ar (
s 2n + by lo0)
+
We continue to work on the absolute value and sup-norm term in the last inequality above separately. For each i E [I, and & (l,...,d,, we use assumptions E; and u, to obtain that (z;)), are i.i.d random variables with zero-mean and their tail is captured by ca (t) for some a E [2, oo) and Ca E (0, x), depending on the noise and input distributions. We are using the fact that the product of two random variables with tail parameters a1 and ag has the tail parameter 01 + ag (Vladimirova et al. 2020
Proposition 2.3). And since we are assuming heavier-tailed noise implies 2; be at least sub-exponential with a 2 (recall that we assumed x; are aab-gaussian). Employing Lemma 10 we obtain for each E [0. oo) that
P
(x;x))
2
t)
6n exi- (nt)
Now, we study the behavior of the sup-norm term in the last incquality of the carlier display. Let's rewrite the sup-norm in the form of max
Σ ( (Elo((
1
Σ (c:)) - Bl(ass)
n
max (E()....)))
Following the same argument as earlier and for each E (1,..., nn] and k W' E {I,...,d}, we can employ Lemma 10 with 23 = (z))) obtain for each E oo) that
7
(a))) - (((aa)))) 2 1²) S 6n exp(xp min(nt? (ntt)
P
for some a' E o0) and e' E (0, 0o), depending on the input distribution. Then, we use our result above together with the fact that if IP((6)) 2 t) s a holds for all i E {1....p) ph, then we also have P(max;E(11...)) |bal 2 t) I pa to obtain
Σ (x)))(()) (((x))) 2 t' x)= 6dn exp(-d min
P
WE(L
1=1
Collecting all pieces above together with considering t 1',, we obtain for each j E {1,..., w and & (1....,d} that
i risk 0]
30 _ (,, 61| < 2t(n + 1o0) +e)
risk
00,x
with probability at least 1 1-6nexp((ccminff (nt) 1/a)) 6dn exp( c min (nt') 1/03)) }), which is obtainod using the fact that
P(A+bD
=1 1 - > b+ 2 1 1 P(A > - PDD > 1)
for any b (0, and t E R.
Then, we follow the same argument as earlier and use 1. our result in Lemma 5 and i.id. assumption on the data, 2. the properties of absolute values and linearity of expectations, some rewriting, 4. Hölder's inequality, 5. equation and our assumptions that f[x] = = T æ. zero-mean noise, and definition of sup-norm, triangle inequality, compatible norms (for a matrix E ddxd we define ||A|| := max Eren |A ()), and the definition of Cn,e, which implics To - Tel1 S 7. adding a zero-valued term, 8. the triangle inequality and the definition of Cr,es which implies ||0 @*) S 31 & v to obtain for each j E (1....,w) w
that
8 riskx x [7,0] 0] Oy,
0 risk (y, θ
0%
57

---

Published in Transactions on Machine Learning Research (12/2025)
2
Σ ((ye - 7 (0:n--7mmmm0)) (0:))) + +
2 n
((y: y TTo (mm)))
n
- -0==0;; - B( 7 Oz: TTxx; (0x)))))))))))
1=1
((y - - 1x, To,, El(ys - 7 T @x,)2x To).1)
2
(yi - 7 (x;))x T
-E((y y Axx; TJ) 10
U T + T0 - 7 ( T- Bla,a, TD)
Σ U
(2 T B| "D
+
00,1
n
s -210-60 - -- + 10
+
-
( T 13 20,00 TD
80
n
00,1
< + (
- Elxra
Then, we use the same argument as earlier to treat the sup-norm terms above (we use our assumptions on æ; and u, and application of Lemma 10 to obtain that
θ risk (7, O] riskly, 21(1 100 +10++ +e)
θ 07j
07,
with probability at least 1 6dnexp(-cminnnud (ml))/aj) 6d min (nl))
Collecting all the pieces above, obtain that for each i E {1, , p} the corresponding gradient difference is bounded (I(Vriskxly, θ] - Vriskly, θ)),l I 2t(n + max boo, ||O lx )) +c)+ with probability at least - 12d 1-12pnexp(-d exp(-c' mit( (nl))/a')) }) for some a' E o0) and e' E oo), depending on the distributions of inputs and noise.
Now we use 1. the definition of sup-norm and 2. our results above together with our earlier argument about implying max operator (note that the gradient vector of dimension p) to obtain for each t [o, oo) that
Vriskxly, θ] - Vrisk 7,, 0i||
sup 9= B)EC,,
max (7, 0] Vrisk 01),
sup
(=ve ie (e......pp
< (1 - (1 10'1l=))))))))11111))))))))))))) +c)
with probability at least 1 12ddppm pm exp(-cmin(nt? (nt) where for the ease of notations we replace d' and a' with e and a (constants depending only on the distributions of the inputs and noise).
Collecting all pieces of the proof, we obtain for each f. E [0, 2)) that
(Vriskx 0 Vriskly, θ)) (99 -B)
sup A ECC,
S sup ()
Vrisk 0] - Vrisk 1)
< 2t + max(l>"'lao.... ||0 (1 + (1
with probability at least 1- 12d pn exp(-emin (nt? (nt) for some a E [2, 0o) and c (0, oc), depending the distributions of inputs and noise.
58

---

Published in Transactions on Machine Learning Research (12/2025)
Now, are ready to our Lemma 11 for extending Lemma 2 for tailed noise. First, recall
Torc,o = (loogn) 3/2 (log(np)) "
(22)
VTi
where a E [2, oo) and v.C. (0, x)) are constants depending on the distributions of inputs and noise. obtain
Then,
Lemma 12 (Empirical Processes for Heavier-tailed Noise). Under the first two parts of Assumption it holds for each reasomable stationary point 3 vec(?, 6) of the objective function in equation that
(Vriskx177 6] Vrisk([, θ1) (8" - -B) S Fore,al33* - B11 + Torc,a
2n
with probability at least 1 - 1/2n.
Proof of Lemma 12 The proof follows almost the same steps in the proof of Lemma The only difference is
employing Lemma 11 and the assignment of t (log 2 plogg (mn) / /n with different constants. 0
59


Reviews:
[{"review_id": "HvA4MjUrRd", "reviewer": "Reviewer_cSSm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In terms of novelty: prior to this work, theoretical guarantees in statistical learning for neural networks almost always concerned either (i) global minima of the loss or (ii) specific solutions found by algorithms under idealized conditions (often in over-parameterized regimes where global minima are reachable). In contrast, this paper is the first to provide generalization guarantees that hold for any stationary point of the training objective in a neural network setting. This is a noteworthy advancemnet in our theoretical understanding of non-convex learning problems. It is an important step in bridgings a gap between optimization and generalization. I have gone through most of the theorem statements, the statements are clearly presented with relevant discussions of the assumptions and the impact of the theorem result. the paper was an easy read because of this. i did not find any particular problems in the proofs I went through, though i did not go through all the proofs.\n\n\nTHe paper does a good job of placing the results in context of exisiting works. e.g. It acknowledges Loh, Elsener for stationary points in simpler settings; Kawaguchi, Zhou & Liang for linear networks; Hardt & Ma, Bartlett et al., Li & Yuan for assumptions similar to theirs.etc. It covers classic results (Bartlett 1998 for neural network generalization bounds, etc.) as well as very recent ones (JMLR 2024 article by Achour et al., and even a 2025 reference on layer sparsity). The authors compare their rates with prior work: for example, they note that Taheri et al. (2021) and Lederer (2022a) had derived similar risk bounds for global optima of shallow networks. Those works did not consider stationary points, so this paper extends them. The authors properly credit those sources for the rates and highlight their own novelty that they apply to all stationary points, not just the minimizer. They also mention a recent work (Taheri et al. ) that provided theory for “approximate solutions in a convex setting” – essentially an analog of Theorem 2 but for convex problems – indicating that their notion of analyzing approximate optimization error is in line with current trends.\nThere are some additional references I would suggest adding to further improve the disucssion:\n\nHaeffele & Vidal (2017) \"Global Optimality in Neural Network Training\" -- This work provided sufficient conditions under which every local minimum of a neural network is global\n\nNguyen & Hein (2018) –“Adding one neuron can eliminate all bad local minima” by Nguyen, Hein, etc. and related works by the same authors studied conditions to avoid bad local minima by slightly modifying network architectures.\n\nChoromanska et al -- \"The Loss Surfaces of Multilayer Networks\"\n\nThere are some typos, please proof read. e.g.: \n\nemprical -> empirical \n\"considerably progress\"\n\"objection function\"", "claims_and_evidence_yn": "Yes", "claims_explanation": "the assumptions, theorem statemetns, proofs and discussions are clearly stated.", "audience_yn": "Yes", "audience_explanation": "Significant result in theoretical ML.", "requested_changes": "addition of some disucssions on some relevant papers. minor typo fixes", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pvvfCCp2KT", "reviewer": "Reviewer_8T5S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides statistical guarantees for any (reasonable) stationary point of the shallow neural networks. The performance of these stationary points and the points within their vicinity is proved to coincide with the global optimal in excess population risk. This directly addresses the gap between optimization practice and theory. The guarantee is also proved for shallow ReLU nets under some assumptions. Small synthetic experiments show test error at an approximate stationary point is nearly identical to those at a potential global optimum.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The proofs appear correct.", "audience_yn": "Yes", "audience_explanation": "The paper studies the gap between theory and practice for neural network statistical guarantees. The results in this paper show compatible performance of the stationary points compared with the global optimum in shallow neural networks. This is of certain interest to the audience of TMLR.", "requested_changes": "(1) The assumptions on ReLU are not clear. Quantify \"approximately perpendicular..\", \"appxomately orthogonal\", etc.. in the section after Assumption 2.\n\n(2) Does the $\\tau$-appxmate guarantee work on ReLU? The use and connections of this are not very clear to me from the writing of this section.\n\n(3) The experiments are very small. Though it shows the expected performance of the approximated stationary points and optimum. The assumptions and settings of the theoretical results are not reflected, for example, the \"reasonability\" assumption and how tuning oracle r can affect the result, since all the bounds involve r.", "broader_impact_concerns": "no concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EMU7OmDkJe", "reviewer": "Reviewer_ZckR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops statistical guarantees for stationary points (and their neighborhoods) in shallow linear neural networks on regression problems, showing that they generalize essentially as well as global optima (up to logarithmic factors). The work then extends these guarantees to shallow ReLU networks under near-identity initialization assumptions, and further characterizes optimal rates for tuning parameters across different network and noise settings. The contributions are thus both conceptual—demonstrating that global optimization is not necessary for strong generalization—and technical, in providing precise theorems (Theorems 1–4) with rate guarantees.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. In the paragraph after Assumption 1 on page 3, there are comments and rationales for the assumptions. It would be helpful to provide more intuition on why assumptions 1 and 2 make sense. I hope to clarify with the following:\n(1) In Assumption 1, the authors assume $\\||\\gamma^{\\star}\\||_1 \\leq\\sqrt{\\log n}$ and  $\\|||\\Theta^{\\star}\\|||_1 \\leq\\sqrt{\\log n}$. It is mentioned in Appendix E that L1 norms should increase mildly in the sample size $n$. Could the author elaborate on \"mildly\"?\n(2) The authors mention that Assumption 2 is not necessarily true in practice but is a common one in the literature. Could you add citations here? It would also help if some intuition were provided here. For example, we can interpret each coordinate of $x$ as a feature extracted from the raw data and thus they are independent; $x$'s are centered sub-Gaussian random vectors due to the construction and training of the feature extractor, with some references added after the intuition.\n\n2. In equation 2, the authors use L1 norm for regularization. Is L2 norm more popular and easier to analyze? If so, why is L1 norm preferred over L2 norm?\n\n3. Theorem 3 limits the discussed reasonable stationary points to those with ly perpendicu rows. It would be helpful to elaborate on why this assumption (approximately) holds in practice and how this assumption simplifies the theoretical analyses.", "audience_yn": "Yes", "audience_explanation": "Yes. While the results are restricted to shallow networks and regression problems, the community has a strong interest in theoretical insights that bring statistical learning theory closer to empirical successes. The fact that the guarantees hold for stationary points rather than just global optima makes the results more practically relevant, and the extension to ReLU networks with specific initialization conditions touches on a widely used architecture. This work is likely to appeal particularly to the subset of researchers working at the intersection of optimization, learning theory, and deep learning practice.", "requested_changes": "1. Please refer to the answer to \"Are the claims made in the submission supported by accurate, convincing and clear evidence\" which calls for elaboration on the rationales for the assumptions.\n2. Please leverage grammar tools to improve preposition usage. For example, the paragraph name \"further discussion over our assumptions\" on page 7 is not standard.\n3. In the introduction, a more concrete and clear summary of prior work + their limitations + this work's contributions would be helpful. The authors write that \"But it is currently unclear how to extend these insights to deep learning—if at all possible\" as the limitation of prior work, but readers might not understand why prior results are hard to extend to deep learning. For example, they might wonder if this work does not cover deep neural networks with a large number of layers as well, then how is this work fundamentally more deep-learning-amenable than others?", "broader_impact_concerns": "The paper is primarily theoretical and does not present immediate risks of harmful applications. Its contributions lie in improving the mathematical understanding of neural network training and generalization for shallow architectures.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)