Should Under-parameterized Student Networks Copy or Average Teacher Weights?
Authors: Berfin Simsek
Amire Bendjeddou
Wulfram Gerstner
Johanni Brea
Venue: NeurIPS 2023 poster (2023)
Keywords: shallow neural networks
non-convex optimization
approximation error
loss landscape
TL;DR: The optimal solution of an under-parameterized student network with $n$ neurons is conjectured to be composed of copying $n-1$ teacher neurons and averaging the remaining teacher neurons up to a certain width.

Abstract: Any continuous function $f^*$ can be approximated arbitrarily well by a neural network with sufficiently many neurons $k$. We consider the case when $f^*$ itself is a neural network with one hidden layer and $k$ neurons. Approximating $f^*$ with a neural network with $n< k$ neurons can thus be seen as fitting an under-parameterized "student" network with $n$ neurons to a "teacher" network with $k$ neurons. As the student has fewer neurons than the teacher, it is unclear, whether each of the $n$ student neurons should copy one of the teacher neurons or rather average a group of teacher neurons. For shallow neural networks with erf activation function and for the standard Gaussian input distribution, we prove that "copy-average" configurations are critical points if the teacher's incoming vectors are orthonormal and its outgoing weights are unitary. Moreover, the optimum among such configurations is reached when $n-1$ student neurons each copy one teacher neuron and the $n$-th student neuron averages the remaining $k-n+1$ teacher neurons. For the student network with $n=1$ neuron, we provide additionally a closed-form solution of the non-trivial critical point(s) for commonly used activation functions through solving an equivalent constrained optimization problem. Empirically, we find for the erf activation function that gradient flow converges either to the optimal copy-average critical point or to another point where each student neuron approximately copies a different teacher neuron. Finally, we find similar results for the ReLU activation function, suggesting that the optimal solution of underparameterized networks has a universal structure.

Should Under-parameterized Student Networks Copy or Average Teacher Weights?
Berfin Şimşek
Amire Bendjeddou EPFL
NYU
bs37360nyu.edu
amire. condjeddou@epfl.. ch
Wulfram Gerstner EPFL sulfram. nulfraa.gerrstmereeppl..dd
Johanni Brea
EPFL
johanni brea@epf1.com
Abstract
Any continuous function fr be approximated arbitrarily well by neural net- work with sufficiently many neurons k. We consider the case when I itself is a neural network with hidden layer and k neurons. Approximating f with ral network with n < k neurons can thus be seen as fitting an pareter-rraeeeeizeeed "student" network with n neurons to a "teacher" network with neurons. As the student has fewer neurons than the teacher, it is unclear, whether each of the n student neurons should copy one the teacher neurons or rather average group of teacher neurons. For shallow neural networks with erf activation function and for the standard Gaussian input distribution, we prove that "copy-average" configu- rations are critical points if the teacher's incoming vectors are orthonormal and its outgoing weights are unitary. Moreover, the optimum among such configurations is reached when
student neurons each copy one teacher neuron and the nith student neuron averages the remaining h network with = 1 neuron, we provide additionally closed-form solution of the non non-trivial critical point(s) for commonly used activation functions through solving equivalent constrained optimization problem. Empirically, we find for the erf activation function that gradient flow converges either to the optimal copy-average critical point another point where each student neuron approximately copies different teacher neuron. Finally, we find similar results for the ReLU activation function, suggesting that the optimal solution of underparameterized networks has a universal structure.
teacher neurons. For the student
1 Introduction
shallow neural network with a single hidden layer of large number neurons can approximate any continuous function f arbitrarily well on compact subset of the input space [I] We consider related problem, where the function f" itself is a neural network with a large number and its approximation is smaller network with < k neurons. In other words, we fit an under- parameterized "student" network with n neurons student has fewer neurons than the teacher, it cannot perfectly match the teacher. In the configuration with the lowest loss, where the approximation error is smallest, one may expect that the incoming and outgoing weights of student neuron either identical to those of teacher neuron that they are aligned with the weights of a group teacher neurons, but it is unclear what the optimal configuration is.
neurons,
"teacher" network with & neurons. As
Previous address: EPFL.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To answer the question of whether student neurons should "copy" or "average" teacher neurons, and more generally to shed light on the loss landscape of parameearaeeterizeed neural networks, we study the theoretically tractable setup with standard Gaussian input data and teacher networks with orthogonal incoming vectors. First, we re-parameterize the loss in terms of interactions between pairs of neurons, similar to 2 3], and we re-formulate the original optimization problem as a constrained optimization problem. The interactions between neurons can be written as a function expressed in terms of the standard deviation and correlation of two Gaussian random variables, with explicit formulas for the erf and ReLU activation functions [4] Next, we prove several properties of the most extremely under- parameterized student network with single neuron = 1, extending thus the important work of [5 For many commonly used activation functions, we prove for the network with single hidden neuron that the optimal solution is the only non-trivial critical point of the loss function up to symmetries and is achieved when the incoming vector of the one-neuron student reaches configuration that can be interpreted
damped average of all incoming teacher weights.
The proof relies on identifying the critical points of the constrained optimization problem and showing that the common activation functions satisfy the assumptions. We rely in particular on the derivative rule of the interaction function which comes as a pleasant consequence of Stein's ' Lemma [8] instead of the Hermite basis expansion which is commonly used technique 24 For the erf and ReLU activation functions we derive additionally a closed-form solution of the optimization problem for 1. Next, we investigate cropy-average configurations of students with >> 1 neurons, where some student neurons copy teacher neurons and other student neurons average sub-groups of teacher neurons, in the sense that they are at the optimal one-neuron solution for the given sub-grou sub-group of teacher neurons. Our particular contributions are:
We propose constrained optimization formulation of the standard minimization problem in the weight-space terms of the interaction conectiioonnntioo function (Section 3 The interaction function natural generalization of the dual activation [15].
Applying the constrained optimization formulation for n = 1, we prove that the incoming vector of the student lies in the span of the incoming vectors of an orthogonal teacher network (Proposition 4.T). For a broad class of activation functions, we prove that the incoming vector aligns with the average of the teacher's incoming vectors for the "unit- orthonormal" teacher network (Thcorem 5.1). Using the derivative rule of the interaction function (Lemma F.T] we show that common activation functions such as erf, softplus, tanh, and ReLU satisfy this property (Lemma F2 and Corollary G.5).
Assuming a untt-orthonormal teacher network and erf activation function, we prove that the concatenation of critical points of single neurons (of the student network) each approximating a teacher subnetwork is a copy-average critical point (Theorem 4.2).
Assuming untt-orthonormal teacher network and erf activation function, prove that the optimal copy-average (CA) configuration is such that n - 1 student neurons each copy a teacher neuron and the n-th student neuron approximates optimally the sum of the remaining teacher neurons (Theorem 5.5) see also Fig. | top row). Empirically, we find that the gradient flow converges to an optimal-CA point for all seeds when n < Yik l with a fixed ?1 near 0.46 (Figure 4.
Surprisingly, we find empirically three regimes of training via gradient flow (GF or under- parameterized networks 4): (i) for n < ki GF converges to an optimal-Ca -CA point for all seeds, (ii) for n > 722 with a fixed 722 near 0.6, GF converges to a point that we call perturbed-n-copp for all seeds, (ii) for 71k < < k2 GF converges to either an optimal- CA point or perturbed-n-copy point. Therefore, as the uuder-parameterizer network grows larger, the solution found with gradient flow where the weights are initialized randomly with a fixed standard deviation changes qualitatively. The code to reproduce these findings is available on GitHub and we refer to Appendix C for details.
1.1 Related Work
The reacher-studen setup has been extensively used in the literature to study the evolution of gradient
flow trajectories and of the generalization error 2 3 27 This series of work gives insight into
We use numerical ODE solver for multi layer networks [6 to simulate the gradient flow in this paper. All "solutions", which are the points which gradient flow converges, have gradient norm of at most 5 10

---

1.00
1.00
6.75
0.75
6.50
0.50
0.25
0.25
0.00
6.00
-025
-6.23
6.50
-0.50
0.73
-0.75
+1.00
u1
u2
neuron1
neuron2
-1.00
Erf (top)
ReLU (bottom)
140
1.00
0.75
0.75
0.50
0.59
0.25
0.25
0.00
-0.2
-0.50
u1
02
neuron1
neuren2
02 u3
U4
n1
02
n3
n4
A1 Gradient flow (Weight space)
A2 Order param's (n=2, k=3)
Order param's (n=4, k=8)
The gradient flow converges to the copy-average optimum point for erf activation (top). or nearby for ReLU activation (bottom): the first n neurons copy teacher neuron each; the n-th neuron takes an average of the remaining teacher neurons. The teacher network is unit-orthonormal. i.c. P(=)=Ez=a((>
Figure 1:
where 1/ *R are orthonormal, and d = 1. A1 The gradient flow trajectory is shown in the weight space for = 2, k =3 the positions of the circles (red and green) represent incoming vector w; projected down the span of 1/, 02 U3 and the sizes of the circles represent outgoing weights a, The blue circle represents the one-neuron solution (the position shows w", the size shows a"). A2 Same setting, the weight-space parameters at convergence are mapped to the order-parameter space; u; (Mr,,..., Usa) where thj represents the normalized dot product between wi and fy and r; = ||w]]]- Order parameters shown convergence for n= 8. For erf (top) the point at convergence is exactly an (n - l--cop----averppp point, whereas for ReI.U, it is perturbed away from this configuration. Neurons are reordered for clarity.
the solution found convergence, however, they rely on numerically integrating the equations of dynamics. Tian [S gives convergence guarantees for ReLU activation function, however, their method only works for one student and one teacher neuron. Xu and Du [21] recently gave the convergence rates for multiple student neurons for the case of one teacher neuron prototypical setup for overparameteriiation. These convergence guarantees were extended broad input distributions 7 [22] and finite training data 6 23
We give the analytical formula of the optimal solution and its generalization error for one student neuron and unit-orthonormal teacher network with multiple neurons for erf and ReLU activation functions and partial characterization for a broader class of activation functions without relying on the analytic formula of the loss.
The studies cited above showed positive results for a single-neuron teacher teacher. However, even for settings where the teacher has only a few neurons, hard teachers can be constructed in the sense that the student fails to find zero-loss solution for a certain fraction of medium-scall problems, gradient flow often converges to 'non-zero loss" solutions (27] Arjevani and Field 30) characterized some families of local
unit-orthonormal
random initializations 26). Moreover,
minima using symmetries, for the ReLU activation function and unit-orthonormal teacher network. In this paper, similarly characterize, for the case that the student has smaller size than the teacher, large family of 'copy-average' critical points, but for the erf activation function. Our approach focuses on the important regime of under-paraen parameterized networks which is relevant for the superposition of features [3 and for the distillation of large networks into smaller ones 32 33
There is large history of approximation theory of neural networks that give universal guarantees on the approximation error, e.g. [ 36 However, these works focus on rates of convergence and provide neither a formula nor approximation for the error. In this paper, we make a conjecture for the exact formula the approximation error of under-parammteerized student networks which we support both thcoretically and numerically.

---

2
Setup
Neural network: Consider a two-layer (student) network function f:R² -> with n neurons
f(z) = Σ 0,0 (w; x)
(I)
where Wy IR is the incoming vector, 0g
the outgoing weight of neuron i, and the activation function 0 is twice differentiable unless it is specified to be ReLU, i.. (elu(2) = max(0, x), and the dot marks the scalar product. n + 1) is the number of parameters.
Parameter vector: The parameter vector is represented as
(W1, a1) ... ① (W,, (n)) RRP
where @ denotes the concatenation of vectors into one vector. We use the notation @ since the network function can be seen as sum of its hidden neurons. Sometimes 0 is written explicitly in the network function [xx -
We assume that the input distribution is a standard d-dimensional Gaussian = (0, La). The target function is denoted by f IR R. Using the square cost, the loss function : R + IR (also known as the risk or the generalization error) is defined as
Loss function:
((0) = En-D [(] -
Orthogonal teacher network: We assume that the target function is a neural network (also known as the teacher network or multi-index model)
(x)
Σ byo(1 j=1
(4)
where its outgoing weights non-zero and its incoming vectors vi,..., Rd are orthogonal to each other, that is, 2/ B = 0 for i + j. This implies that the input dimension satisfies 2 Following [27 30 37), we particularly focus the unil-orthonormal teacher network where the outgoing weights are all one, that is by 1, and the incoming vectors have unit norm, i.e. 0; Uy = d.j-
Optimal loss: We study the optimal solution(s) of the following non-convex optimization problem
Σ awo x) - -bb x) x))²
L (STi w.)) EOND
(5)
for under-paree sateerr-rramterzzeedd (student) networks, i.e. n < k. and orthogonal teachers. For n 2 k neurons, the network can copy all teacher neurons and set the outgoing weights of the remaining neurons to zero, therefore the optimal loss is trivially zero. If the teacher unit-orthonormal. then all of its neurons contribute equally; hence the optimal loss is determined by and & only and denoted by L L"(n, k). student neural network has one neuron we use the notation L (k) = L* (1,
3 Foundations & Constrained Optimization Formulation
In this section, we introduce a constrained optimization problem that is reformulation of the min- imization problem in Eq. [] This formulation allows us show that the incoming vector of any non-trivial critical point of the one-neuron network is in the span of the teacher's s k orthogonal (or po-
tentially even non-ortie orthogonnl,, see Appendix Remark [D.I) incoming vectors (see Proposition 4.T] We give the exact solution in the case of unit-orthonormal teacher (see Corollary 5.2 and Corollary G.5).
Using the linearity of expectation, the loss function in Eq. 5 can be expanded weighted sum of the following Gaussian integral terms
E, Ex-plo(Vi 1)1(V3-3 x]]

---

where V1 and V2 represent two arbitrary vectors of student and teacher networks such as (w;, w,) or (wy, 0j). As both V1 x and 1/2 are centered Gaussian random variables, the above expectation can be expressed in terms of the covariance of the two-dimensional Gaussian
(V1 V1)2) x)² (Vi (V22-2
(V1 Vi.zz)( (V2 x)) (V/2 x)²
rf
T1724 T2
ENDD
where ri = [[V]|| for 1,2 the f--nrrm and, assuming rV > O, u Vi V2/(ring is correlation. The covariance entries QN = r Q12 r1724 have been used to study the gradient flow trajectories [2] [4]] We prefer the parametrization with r and as enables us to make the positive definiteness constraint explicit, i..
V1 V/2
<
(6)
7,72
due to the Cauchy- Seehhy-Schwarziii inequality. We introduce the interaction function funetton 9o
IR2 [-1,1]
u
To (r1, r2, u) (o
with >
11,22 0,
(7)
1
to express the Gaussian integral terms. Note that is not well-defined one of the norms zero. Extending the formula above, for the case w.l.o.g. 72 O, we define
(ri, 0, u) := B,-N((, ) (m(ra)))00))))
for all u E [-1, In this paper, we consider the activation functions satisfying the following.
Assumption 3.1. For all r1.72 > 0 and E (-1,1), we assume that the interaction function satisfies either the first or both of the following properties
d2
(i) du 9o (r1, r2, u) > 0,
(ii)
du (r1, 1, 1,u < du
(r,,
(8)
To check whether a specific activation function satisfies the above properties, we mainly rely on Lemma F.1 which gives the rule for the partial derivative of go with respect to the correlation
d du Jo (11, 72, u) ringlElo' (riæ)o' (r29)).
Hence, if a is monotonic (increasing decreasing 1 the integrand on the right-hand side is positive; satisfying Assumption 3.1 (i). The ReLU activation function d Treta (I) = max(0, æ) also satisfies
because of the known analytical expression of the interaction 4 27
Pretu (r1, u) r172h((u) where h(u) 2T
(V1- 1 x²
ar)cos(u))u)
Checking Assumption 3.1 (ii) for given activation function is delicate. We rely it in Section
Using the interaction function, the loss function can expressed terms of the order parameters:
norms of the incoming vectors the student r; = ||
correlations between the incoming vectors of the student and teacher Ui = w; u. (r:l/r///lll) correlations between the incoming vectors of the student Par wy wy (re/(r
where we assumed r; > for all i E [n]. The constrained optimization formulation possible for general non-orthon -othoootlao teacher networks (see Remark in the Appendix). For the sake of simplicity, we formulate here the constrained optimization problem for the case of orthogonal teachers and reformulate the objective in Eq. 5las
n
minimize
Σ
ge (ri, Fi; 1) +2 Σ agay Ga (r', TV, Par) E
Σ>
az (ri, 11
1=
subject to || S 1, r 2 0, for all i e [n].
Pi ui < 1 12 VI
for all i i' E In]:
(10)
"Inrreasing (or decreasing) strictly increasing (or decreasing) everywhere in this paper.

---

B
B+
B
B
P23
R(6)
x001
P12
<igg
P22
neuron
neuron 2 neuron 3
nownon
ncuron 2
ncuran
norm space
neuron i- neuron
A Weight-space
B1 Primary domain of the order parameters
B2 Secondary domain
Figure 2: Cartoon representation of the mapping of student with three neurons from the weight space A Rnd to order parameter space B1-B2. The mapping between the outgoing weights is an identity mapping hence not shown. A Each axis shows the direction of weights of one teacher neuron (k 2 3). B1 Each incoming vector wy E Rd is first transformed into (ri, Wi wi/r:) and then
wi/r; is projected onto the span of the teacher's incoming vectors, yielding the teceee correlation correlation vector u4 = (#.,...., u.a)) B2 The student-student correlations PiP are general free U. + / |1/l3 /1 | 2|2 hence the box constraint. An E gk-1, gives a vanishing + term for the interval of correlation Pi, for all # 1, hence they are no longer free (shown red). the case d d= k, all u, are on the hypersphere due to the problem geometry, hence the correlations Pir are fixed and not free (see Appendix D.I.
parameters bounded in between
activated constraint, w.l.o.g.
where u, = (U.1 Uzk) and C = EE<D than simply bounding correlations with the help of Eq. 6 See Appendix D for the derivation of the constraints and Fig. 2 for a schematic.
[f If [f (z The constraints in Eq. 10 give tighter bounds
The objective above is exact for n = 2, in the sense that its optimal solution is equivalent to the optimal solution the weight space, since the mapping from the weight-space to the order space is invertible. However, it is relaxation for 2 3, since there order-parameter configurations in the domain (see Figure 2) that do not correspond to any weight-space configuration (see Appendix D.3 for construction). It seems possible overcome this gap by considering the geometry of the angles between n 2 3 incoming vectors to tighten the constraints between student-student correlations.
Copy-Average Critical Points
In this section, we identify new family of critical points by 'combining' critical points of one-neuron networks for the unit-orthonormal teacher and the erf activation function. We first show that network with the incoming vector w is in the span of the teacher's s incoming vectors (Proposition 4.1). Applying this proposition to the special case of the erf activation function, show that the concatenation of such critical points also a critical point for multi mellin networks (Theorem (4.)..
student neuron, for any "non-trivial" critical point. that is w" * 0 and a" & 0.
Proposition 4.1. Assume that 1s is an orthogonal teacher network (Eq.44 of width k. If activation function satisfies Assumption 3.1 (i), any non-trivial critical point f = [w" aa"), iee. V... (0*) 0,
||w"||| * 0, a" f O, satisfies that w² is in the span the teacher's 's incoming vectors.
The proof uses the constrained optimization formulation for n = 1 (see Appendix G. In short, a critical point mapped to the order parameter space satisfies either ||u-] 1 or d. go (r. D U1y)
for all j E [A:]. Under the Assumption 3.1 (i) these partial derivatives are non-zero, hence ||u|| || = 1.
In recent work [38]. the incoming vectors of the student network are also shown to converge the span the vectors of the multi-index model using weight-decay.
Finding the optimal solution for the multi multi-neuron network is challenging. Natural candidates are concatenation of student neurons where each one of them a critical point of the loss function LT.4 where E; is the number of the subgroup of teacher neurons. More precisely, let us pick partition + ... + l, < such that l, 2 1, and define 8m = > Em= E for m S n and 8p O. We denote one-neuron critical point by 0 (w., a.), when learning from a part of the teacher network
&:
(x)
Σ
olu x))
(11)
Since f is unit-orthonn orttnorthonommal teacher, w is in the span of 1a;-1+1,... Vs; due to Proposition 4.1

---

We use the term copy-average (CA) point or configuration to refer to the concatenation of such one-neuron critical points in the student network with n neurons: é; 1, the student neuron copies one of the teacher neurons (vj, 1); if & > 1, it averages group of teacher neurons in the sense of approximating their sum with one neuron. For odd activation functions, the one-neuron network problems decouple from each other, as the cross-terms Elorww 22)0 =)] vanish for w1 1 w2- For the specific case of erf, we prove that all the copy-avere configurations are critical points.
dt. We pick copy-average parameter Theorem 4.2. Assume that (xx) - - = / So e e-
=- a1) ..... @ (w o7))
(12)
where (w, a.) is a non-trivial critical point when learning from a unit-orthoormall teacher fī with the incoming vectors Va: 1+1.... Us, shown in Eq. 77 Then 6 is critical point of the loss function L where the target function (x) Σÿ= (v3 x)).
In particular, all neurons equivalent to each other in nntt-orthonormal teacher network. Therefore, the copy- avpyaeeerge configurations where n 1 student neurons each copy a distinct teacher neuron and the n-th student neuron takes an average are also equivalent and called (n- 10- --vveee-e or (n-1)-C-11- in short. Another interesting configuration is where n student neurons each copy a distinct teacher neuron, which is called n-copy, n-C in short.
For general activation functions the copy-average parameter vectors are not critical points (see Eq. 56). Nevertheless, numerically find that the gradient flow converges to similar configurations for the ReLU activation function (see Figure I see Appendix C.3 for more experiments).
Approximation Error of Underparameterized Networks
The target function is assumed to be a unit-orthonrrmal teacher network in this section. In Subsec- tion 5.1 we show for the one-neuron network that there unique non-trivial critical point up to symmetries, which is necessarily the global minimum (Theorem 5.1) Furthermore, we give the ana- lytic expression of the optimal solution and its loss for erf (Corollary (.2) and ReLU (Corollary G.5) activation functions. In Subsection [.2] we provide for the uuder-parameterizer student with > neurons the exact loss copy-average critical points for the erf activation function and show that the (n -oopy----aeeae configurations reach the lowest loss among CA-critical points (see also Appendix E. for the combinatorial number of the equivalent copy-average configurations related to the landscape complexity calculations (39]..
5.1 One-Neuron Network
Using the constrained optimization formulation in 10 we first prove that at any non-trivial critical point of the one-neuron network, the incoming vector aligns equally with all teacher's incoming vectors for unit-orthonormal teachers for activation functions satisfying Assumption 3.1 (see Theo- rem and Solla for erf activation and Tian for ReLU activation (see Appendix B for detailed comparison). Our proof works for broad class of activation functions and does not use the analytic expression of the interaction function.
rem5..1 5. This is related to the symmetric solution visited during the learning plateaus studied in Saad
Theorem 5.1. Assume that the activation function satisfies Assumptions 3.1 (i) and (ū). At any non-trivial critical point (w" ) of the loss L,L.k for the unil-orthnorrmal teacher network, the incoming vector satisfies
w
(13)
where u² is either 1. VE.
or
VE
Proof Sketch. There is no student-student interaction term since we have a single neuron; therefore we write uj instead of Uij and the constrained optimization problem in T0 simplifies to
minimize a² go (r, 1) 2a Σ Jo (r, 1, 4n) + const, subject to ||]|| S 1, 2 0. (14)
7

---

true lunction
network function
true function
network function
ert
tanh
sig
Teacher width (k)
A erf
B softplus
Cone-neuron approx. error
Figure 3: One-neuron network solutions. A Network output (color coded) function of input in d = 2 for (left) unit-orthoormal network with k = 2 neurons (incoming vectors and U2 are shown as black dots) and (right) the student network function generated by the optimal solution (incoming vector shown in red) for the erf activation function. B Same for the softplus activation function. Approximation error of student with n neurons function of the number of k teacher neurons. For large k; the approximation error for = 1 grows near-linearly for the differentiable activation functions studied this paper (erf, sigmoid, tanh, and softplus with 3 = 1); however the growth is quadratic for ReLU (see Appendix Corollary G.5
From Proposition 4.1 we have that ||w|| = 1 for any non-trivial critical point. Therefore, the constraint of 14 on the correlations u (ui, ... ug) is satisfied. The mapping of any non-trivial critical point to the order-parameter space is critical point of the Lagrangian loss (see Appendix Lemma G.2)
Hence every u satisfies
d
2a
9o (r. 1, u2) + 2Xu, duj
(15)
for fixed (r, a). Assumption 3.1 End of Proof Sketch.
(ii) implies that 1 " 0.gg (r, u, is one-to-one hence all uy are equal.
We show in Lemma F2 that the interactions of the common activation functions such erf, tanh, sigmoid, and softplus (respectively)
(ar(x)=
e - dt, (z)
1-
(zg(z)
of (x)
log(e 9z + 1).
vT
1
I+e
with 8 E (0, 2] satisfy Assumption 3.1 (ii). The interaction of the ReLU activation function, i.e.
Tretu (I) max(0. (0. 2) also satisfies Assumptions 3.1 (with slight modification in the domain for (ii);
see the proof of Corollary G.5).
Thanks to Theorem 5.1 the loss in Eq. 14 can be reduced to two-dimensional loss in and which can be solved explicitly for ReLU (Corolary G.5) and erf.
Corollary 5.2. Assume that the activation function is Cerf. The optimal solution (w", a²) is given by
1
w"
||w0||
2k a= = k;
Vis
or equivalently, by (-w", -a"). The optimal loss is then given by
(k) = arcsiin ()) 222 arcsin
1
(16)
2k:
The proof of Corollary dimensional loss does not admit an analytical expression. For this case, from the partial derivatives, obtain fixed-point equation in which solve numerically for the activation functions listed above (see Appendix Section G.2.2). For softplus, specifically, we prove in addition the following
5.2 presented in Section G.3 For general activation functions, the
Theorem 5.3. Assume that the activation function is sop (x) with < 2. The optimal solution (w" a²)) satisfies ||w"|| < 1/V4 and a* 2 k.
We use the FKG inequality to prove Theorem 5.3 (see Appendix G.5 Although the proof requires specific properties of softplus, we show that the above bounds hold also for tanh and sigmoid by numerically solving the fixed point equation (Figure also F Note that the incoming vector norm is bounded above by 1/ /vk, hence w" is damped average of the teacher incoming vectors.

---

ne3
ne2
23550
31330
19.3
11500
00030
Loss
1:500
12
8, = 16
8, 14
8, 11
witth
B1 In seeds, GF B2 seeds, GF
B3 seeds, GF
B4 In ail seeds, GF
A Solutions of Gradient flow (GF)
finds CA CA-optimal
finds CA-aptimal
finds Pin solution
finds Pin salution
Figure 4: Under-parameterizee student networks width with erf activation function learning (via gradient flow) from unit-orthnormal teacher network of width k. Each dot is the mean error convergence for 20 seeds of random initializations; black-dashed lines the theory predictions (ar(k-n+1) see Eq. 20) Standard deviations do not show on the figure as they are too small. We identify four regimes indicated by colors ggrenn-gray-bue-ree
the type of solution found by gradient flow (GF). In the green regime, GF converges to an optimal (n- (0--)-C---AA solution for all 20 initializations (Fig. B.B.). In the gray regime, GF converges either (n - 1)-C-I-A solution or to a "Perturbation of the all-copy solution" that we call P-n-C (Fig. B2). In the blue and red regimes, for "22 where n 8, 12, 16 the gradient flow converges all seeds (Fig. 4 B3). Moreover, in the red regime, for > risk where 8, 12, 16 and is near 0.75, the P-n-C solutions achieve lower loss than the (n 1)-C-1-A solutions (Fig. 4 B1-B4 Examples of loss at convergence (vertical axis) for all 20 different initialization seeds (horizontal axis); theory is shown by the red-dashed horizontal line. Insets show examples of correlation matrices Wy (k lines, n columns) between student and teacher incoming vectors convergence after reordering neurons. the gray regime (for ex. B2) the gradient flow converges to either one of the two types of minima with correlations shown in the inset; in the other regimes, it consistently converges to the same minimum up permutations.
depending
P-n-C solution from
in literature: (i) incoming vector w constrained to be on the unit sphere 1T 40), (ii) the outgoing weight is constrained be one 17 [8 20). An important step in analysis is related the norm the incoming vector which we discuss in Appendix Section G.2.2
Remark 5.4. We do not impose either of the reductions that
5.2 Multi-Neuron Network
In this subsection, we assume that the activation function is erf such that CA-configurations are critical points (see Theorem (42). For student network with and partition (f1, Eg) we can decompose the loss of CA critical point as
(ar(C1) [an(e2) Lor (&1 + E2)
(17)
where (o) := EzwDlfī (x)²] is the error made by student with vanishing output when representing reduced untt-orthonormal teacher network with neurons. This decomposition is possible because the cross-terms between orthogonal vectors are zero for odd activation functions. Furthermore, for the erf activation function, we show that
(e1 + (0, (o) > (in(e
(18)
(see the proof of Lemmal among the partitions with E + = k. Among such partitions, Lemma shows that the optimum CA-point has the partition (1, k - 1). In words, the optimum is 1-copy-1-average point.
E.2 Therefore, we should search for the minimum loss configuration
For general n, using Lemma E.2 and small trick, we prove the following.
Theorem 5.5. Consider a untt-orthonormal teacher network f*(x Σi= ou, x) and the erf activation function. For madde--prameterizee student network with neurons, the minimum-loss copy-average configuration up to permutations (of the student and teacher neurons) is
= =(E1U15E1) @ O (...e(eee.lo-e_o.....
@ W,, an)
(19)
where E {+1) and (wh, an) is given by Corollary 5.2 after substituting k with k k-n+

---

See Appendix E.3 for the proof. Because average critical points are not necessarily the only critical points of the loss function for students with n > we investigate in simulations, if they are found by gradient flow where the weights are initialized as Gaussian with fixed standard deviation (see Fig. 4
Interestingly, gradient flow converges to the CA-optimal solution for all random seeds in a broad
regime of under--arametterizationn (green in Fig. 4. Only when >> 7 for = 8, 12, 16 and 71 ~ 0.46, gradient flow converged in some seeds to points close to the n-copy critical point. We call these newly found points "perturbed n-copy" (P-n-C) points. In gray-blue regimes, the P-n-C points have higher loss than the optimal CA critical point (Fig.
However, this is not always the case: when the student width is close to the teacher width (low compression regime), the P-n-C point has a slightly lower loss the lowest amongst the CA critical points (red in Fig. 4 When k is fixed, we found that the (n 1)-C-1-A solution turns from minimum for small n. to a saddle for large n (see App. Fig. 3 which explains why the gradient flow escapes it in this regime and converges another minimum a lower loss.
Finally, based on our theory and experiments, we conjecture that there exists a 7o E (0, 73) such that when n < Yokk and when the activation function is erf, the global optimum of the non-convex loss in Eq. 5 is a (n (n-1)-C-1) 1)--C1-A configuration. Therefore, if our conjecture holds, the exact approximation error, i.e. the optimal loss, is identical to that of me-neuron network approximating teacher with k 8-MM n+1 neurons and is given by
(n, R:) (r --- 1).
(20)
6 Conclusion & Future Directions
We studied the learning of under-pareed parameterized student networks from orthogonal teacher networks for standard Gaussian input data and vanishing thresholds. For erf activation function, we introduced a new family of critical points that arise from the decoupling of the problem into one-neuron networks that can be solved separately. Moreover, the exact parameters of copy-average (CA) critical points are given which can be used to study escape behavior near saddles and to determine convergence of first and second-order optimization algorithms [6 41
Furthermore, we showed that the optimal CA point is that n - 1 neurons copy teacher neurons and the 7--hh neuron averages the remaining k -- 1 neurons. In simulations, gradient flow converges to a CA-optimal solution for n < . k where 71 is near 0.46. However, for n > 72k where 72 is near 0.6, we observe another phase where the gradient flow finds perturbed copy solution. For the ReLU activation function and the onset of under-pareee parameterization, gradient flow converges to qualitatively similar solutions; however, the crossing point from under-parameterizatioo to over- parameterization (i.e. n = k), gradient flow is known to get stuck in spurious local minima [27). On another note, determining the CA-optimal solution of the two-neuron network plays a critical role in our analysis. Still, there is only little literature on two-neuron networks 5 42 compared to
the well-studied one-neuron case 7 22 23). The two-neuron network is possibly the simplest model with interactions between neurons, hence it is important to understand the global minimum and gradient flow dynamics of this challenging problem.
On the practical side, our analysis of under-parameterrize networks gives a recipe for how to warm- start smaller neural networks for distilling unit-orthonormal teacher networks. If one desires low compression (n > 72,)), then we recommend initializing the student network in a configuration where each neuron copies different teacher neuron, to be close to a P-n-C point. However, for higher compression, we recommend initializing the student network in configuration where n 1 neurons are each copied and the n-th neuron is initialized as an average neuron to be close a (n - 1) cooy---- 1-average point. It remains an open question whether this recipe applies to non-idealized scenarios such as non-isotropic input distribution, teacher networks with orthoothogonal incoming vectors, or non-unit outgoing weights. More generally, it is natural to expect that the optimal distillation strategy changes from low compression levels to high compression levels. How exactly and where this change happens is very intriguing question of theory and practice.
10

---

Acknowledgements
The authors thank Lenka Zdeborová for the discussions and encouragement at the beginning of this project and Clément Hongler for many discussions and valuable feedback. This work was supported by the Swiss National Science Foundation (no. 200020_2074200
References
[1] Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3):183-19)) 1989.
[2] David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52(4))4225, 1995.
[3] Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-studen setup. Advances neural information processing systems, 32, 2019.
[4] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
[5] Yuandong Tian. An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis. In International conference machine learning, pages 3404-341 PMLR, 2017.
[6] Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics, 46(6A):2747-2774, 2018.
[7] Gilad Yehudai and Shamir Ohad. Learning single neuron with gradient methods. In Conference on Learning Theory, pages 3756-3786. PMLR, 2020.
[8] Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pages 1135-1151, 1981.
[9] Rishabh Dudeja and Daniel Hsu. Learning single-index models gaussian space. In Conference On Learning Theory, pages 1887-1930. PMLR, 2018
[10] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research, 22(1):4788-435 4838, 2021.
[11] Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. Advances Neural Information Processing Systems, 35:9768- 9783, 2022.
[12] Raphaël Berthier, Andrea Montanari. and Kangjie Zhou. Learning time-scales in two-layers neural networks. arXiv preprint arXiv:2303.00053. 2023.
[13] Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. Learning two-layer neural networks, one (giant) step at time. arXiv preprint arXiv:2305.18270, 2023.
[14] Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. arXiv preprint arXiv:2305. 10633, 2023.
[15] Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and dual view expressivity. Advances in neural information processing systems, 29, 2016.
[16] Johanni Brea, Flavio Martinelli, Berfin Şimşek, and Wulfram Gerstner. MIpgradientflww: going with the flow of multilayer perceptrons (and finding minima fast and accurately). arXiv preprint 2301.2301..063 2023.

---

[17] Michael Biehl and Holm Schwarze, Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995.
[18] David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337, 1995.
[19] Peter Riegler and Michael Biehl. On-line backpropagation in two-layered neural networks. Journal of Physics A: Mathematical and General, 28(20):L507, 1995.
[20] Rodrigo Veiga, Ludovic Stephan, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks. arXiv preprint arXiv:2202.00299 00293,,,0022.. 2022.
[21] Weihang Xu and Simon Du Over- wweer-arrmtterization exponentially slows down gradient descent for learning single neuron. In The Thirty Sixth Annual Conference on Learning Theory, pages 1 1155-1198. PMI.R, 2023.
[22] Gal Vardi, Gilad Yehudai, and Ohad Shamir. Learning a single neuron with bias using gradient descent. Advances in Neural Information Processing Systems, 34:28690-28700, 2021.
[23] Lei Wu. Learning single neuron for non-monotonic activation functions. In International Conference on Artificial Intelligence and Statistics, pages 4178-4197. PMLR, 2022.
[24] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High- High-dimensional limit theorems for sgd: Effective dynamics and critical scaling, arXiv preprint arXiv:2206. 04000, 2022.
[25] Flavio Martinelli, Berfin Şimşek, Johanni Brea, and Wulfram Gerstner. Expand-and and---usserrr Exact parameter recovery of neural networks. arXiv preprint arXiv:2304.12794, 2023.
[26] Alberto Bietti, Joan Bruna, and Loucas Pillaud- Vivien. On learning gaussian multi- models with gradient flow, arXiv preprint arXiv:2310. 19793, 2023.
[27] Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International Conference on Machine Learning pages 1433-4441 PMLR, 2018
(28) Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the op- timization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2):742-769, 2018.
[29] Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparameterizaatoo:: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 1(1):84-100 2020.
[30] Yossi Arjevani and Michael Field. Analytic study of families of spurious minima in two-layer relu neural networks: a tale of symmetry ii. Advances in Neural Information Processing Systems,
34:15162-15174, 2021.
[31] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209. 10652, 2022.
[32] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 2015.
arXiv preprint
[33] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129:1789-1819, 2021.
[34] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-31) 1989.
[35] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359 -366, 1989.
[36] Andrew Barron. Universal approximation bounds for superpositions of sigmoidal function.
IEEE Transactions on Information theory, 39(3):930-945, 1993.
12

---

[37] Itay M Safran, Gilad Yehudai, and Ohad Shamir. The effects of mild over parameterization on the optimization landscape shallow relu neural networks. In Conference on Learning Theory, pages 3889-3934, PMLR, R, 2021.
[38] Alireza Mousavi-Hosseini, Sejun Park. Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn low-dimensional representations with sgd. arXiv preprint 2rxXfv:2209.1488.. 2022.
[39] Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning,
pages 9722-9732. PMLR, 2021.
[40] Joan Bruna, Loucas Pillaud- Vivien, and Aaron Zweig. On single index models beyond gaussian data. arXiv preprint arXiv:2307. 2023.
[41] Charles Frye, James Simon, Neha Wadia, Andrew Ligeralde, Michael DeWeese, and Kristofer E Bouchard. Critical point-finding methods reveal gradient-flat regions of deep network losses. Neural computation, 3366):146911497, 2021.
[42] Chenwei Wu, Jiajun Luo, and Jason D Lee. No spurious local minima in a two hidden unit relu network. Workshop track ICLR, 2018.
[43] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings. 2010.
[44] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
13

---

Summary of Results
14
Further Comparison to Literature
14
Further Experiments
16
One-Neuron Network
16
Erf Experiments
16
ReLU Experiments
18
Constrained Optimization Formulation
19
D.I Equality Constraints in the Case d k
20
D.2 Binary-Equality Constraints in the Case d = k
20
D.3 Three-Neuron Network
20
Copy-Average Critical Points
21
E.1 Number of CA Critical Points
24
General Properties the Interaction Function
24
The One-Neuron Network
27
Any Nrivirrvial Critical Point Satisfies the Lagrangian Condition
27
2
General Activation Functions
28
Hlssed-Form Solution for Eri Activation
31
Closed-Fome Solution for ReLU Activation
32
G.5
Bounds on Incoming Vector Norm and Outgoing Weight for Softplus
A Summary of Results
Table 1: Summary Results
columns=conditions:
Orthogonal UO & erf
UO satisfying do assumptions
lines-results:
average is the optimal solution
n.i.g.
yes*
yes
n > CA points are critical points
n.i.g.
yes
n.i.g maybe for odd
(n L)-C-1-A is the optimal-CA solution
n.i.g
yes
n.i.g. maybe for some odd yes*
n = w is in the span of {U1,...,
yes
yes*
In the table above, UO means unit-orthonormal, n.i.g. stands for 'not in general" and yes* follows as a special case from the results with yes on the same row.
B Further Comparison to Literature
In this section, we compare the symmetric solutions found in erf [2] and ReLU networks to our one-neuron solution (n = 1). The main difference is that both earlier studies constrain the search space to the symmetric subspace whereas we first prove that the non-trivial critical points are
contained in this subspace in Theorem 5.] for a broad class of activation functions, including erf and ReLU. Solving the oiv-dimeniional loss, we recover the same solution for ReL.U and erf as 2
for unit-orthonormal teachers.
14

---

Symmetric Solution of Saad and Solla (21 for erf activation. The authors focus on the 'symmetric subspace' parameterized as
Qa=r-= Q, Qi = pejriry =C R = WinTs = R.
(21)
In this case, the loss parameterized by three values, that is Q, R, hence can be expressed
analytically in terms of these values. Solving the fixed point equations, they find the following critical/fixed point (their Eq.
Q = 2k
R
(22)
1(2k
which implics r, = 1/vK-1 and P.j = in our paramcterization. This selection of parameters forces all student vectors to be equal therefore reducing the system to one-neuron network. There are two main improvements in our analysis
1. We prove that student-teacher correlations lij are equal to each other at a non-trivial critical point, and give necessary conditions on the activation function (Assumption (3T] to satisfy this property. We show in Lemma F.2 that not only erf but a large class of common activation functions satisfy Assumption 3.1
2. Our student network has flexible outgoing weight (shallow neural network) opposed to a fixed outgoing weight + (soft-committe machine) in Saad and Solla [2] is instructive compare the generalization errors of the one-neuron network
k
2
(soft-committee machine) Lat sont (k:)
k.2 aresin
3
π
2 (shallow network) (ir(k) カー aresin
2
1 )
2 k²
aresin(
R k(
2h
which are identical since arcsin(0.5). π 6 (Saad and Solla 2 uses cy (A()) 1 L son (k)
that's why there is factor 0.5 difference with respect their Eq. (23)). However,
we set teacher outgoing weights to say a,, the shallow network adapts and reaches the generalization error of (in(()) (k) but the enror of the soft-committee machine is
Ti soft (I:) = k2
/2k 1 V2k- 1) 2k 2k²ang(
V2k- 1 1, Vk-1 ) 1,,
= =(aa) + a 1
which has worse coefficient compared to as expected.
Symmetric Solution of Tian (5) for ReLU activation. The authors foeus particular two- dimensional subspace (x. y) that allows the specialization of student neurons, namely
rV; +
Σ U/
(23)
37
In particular, they consider the 'symmetric subspace' = which the case when all student neurons collapse to neuron, and show that the dynamics converge to the following fixed point
r (Vi-1 - arccos( T).
(24)
Summing over I neurons then produces the following one-neuron due to the positive homogeneity
w" = (VA - arccos( 1
Dj-
VK
j=1
Our formula (Corollary w a
G.5 gives the identical result due
b
1 D -Vk
-
+ π
Uz.
h(1)
In this case, there is no difference between the optimal solution of the soft-committee machine and the shallow network since ReLU is positie-hhomogeneen expected.
15

---

theery (HeLLU)
magniudde
norm
Sammoid
PY
Tank
teacher (K)
of teacher neurons
Figure 5: Structure of the optimal solution of the one-neuron network for various activation functions. We trained 20 seeds of one-neuron students learning from the mnt--orthonorml teacher networks with & 2.... 10 neurons. All students converge to the same optimal solution up symmetries (that is, positive-scaling symmetry for ReLU and sign symmetry for odd activation functions such as tanh and erf). A For ReLUU, the magnitude ||w*||a" exactly matches with the result of Corollary G.5 For softplus, the magnitude very close v Vl; for sigmoid, tanh, and erf, it is below Vi. B The norm of the incoming vector is smaller than 1/v/ for softplus, sigmoid, tanh, and erf. C The outgoing weight is larger than k for softplus and tanh, and it is virtually &o for sigmoid and erf.
C Further Experiments
All experiments in this paper implemented using the gradient flow package implemented by Brea et al. [16] which is particularly suited to studying gradient flow on the population loss. For activation functions for which there is an analytic formula, it is already implemented the package; for the others, we used the approximator option for speed-up compared to the numerical integration option. This method uses neural network in the background fitted approximate Gaussian integrals. We trained for 10 ode iterations for erf and relu experiments; 10 ode iterations for softplus, tanh, and sigmoid. For erf experiments, all seeds converged to configurations with gradient norm below 5 10-8, For ReLU experiments, fraction of seeds failed to converge (large gradient norm at the end of training). Appendix C.3 we report among the seeds that succeeded in converging. Weights initialized Gaussians with 7ero mean and standard deviation 0.1 or with Glorot initialization [4]] This is contrast with Saad and Solla (2) Tian where (order) parameters are initialized with positive values (as opposed to the rotationally symmetric initializations done in practice). For each (n, k) pair, we implemented 10 or 20 seeds of random initializations.
C.1 One-Neuron Network
Empirically, gradient flow converges to the point where all student-teacher correlations are
We know from Theorem 5.1 that at the non-trivial critical point all correlations are equal at correlation 1/VR and there is possibly another critical point at correlation -I/VT. Depending on the activation function, the point where correlations are -I/VE might be
either equivalent of the optimum solution (for odd activation functions),
saddle (for ReLU),
* or does not exist (for softplus).
The details can be found in the proofs for individual cases.
C.2 Erf Experiments
In this section, first numerically investigate whether the CA-optimal critical point is saddle or minimum in Figure 6, Surprisingly, we find that the point turns from a saddle point to a minimum
'For odd activation functions, there two solutions that are sign-symmetric:: the first one where all correlations 1 and its cquivalent where all correlations are 1. The first solution plotted in Fig. for fine comparison the positive scale.
VE
VE-
16

---

k
00030
k=
elg.
0.22
00030
00075
T-n+1
0.8
0.005
kmn+3
25 20
Figure 6: The minimum eigenvalue of the Hessian al optimal-CA point. We numerically investigate whether a CA-optimal critical point is a strict saddle (min. eig. of the Hessian is negative) or minimum (min. eig. of the Hessian is non-negative). Interestingly, the minimum eigenvalue turns from positive to negative as n grows for regime, the Co-opti cannot be the optimal solution of the non-convex problem for large For >> n, for example for k = n, 2n, 3n (right panel), the min. cigenvalue is positive and approaches zero increases.
n + h fixed h 1, 2, 3 (left panel). Therefore in this
250-
0305
2.08
8.35
025
cusgpping weigghh
14.
12-
core
1tr
CAsCgo 21g usigtte
12-
6.-
(017
Tosses
66-
3
85
-20
Figure 7: Evolution of order parameters during convergence to (n - 1)-C-1-A 1-A solution: top 2, middle n 4, bottom n 8; and l 64; representative seeds. We can distinguish phases: before iteration 20, after iteration 20, and beyond iteration 40. We observe that in the first phase of training (less than iterations), the student neurons not specialize into teacher neurons but approach the one-neuron solution. For = (top row), in the second phase, we observe that the first neuron implements an average teacher neurons and the second neuron implements copy of the remaining teacher neuron. In the second phase, in general, each (or its negative equivalent) and the 7ith neuron splits its correlations into two groups: those that correspond to the teacher neurons being matched become negative and the others collapse each other. Finally, the third phase, negative correlations converge to zero correlation, decoupling the student neurons from each other. All student neuron correlation signs be flipped as long as the corresponding outgoing weight signs are flipped since the erf activation is odd. These examples illustrate the green regime (see Fig. in the main text).
neurons specialize to match one teacher neuron
point in some regimes of (n, k), despite that the configuration has the same structure of copying teacher neurons and taking an average the remaining teacher neurons. When l: n fixed, and for large Figure explains why gradient flow does not converge to the CA-optimal point.
17

---

144
outgpingg mangtte
64
Damms
cam
64
Figure 8: Evolution of order parameters; P-n-C solution: for 8 and k 9. We observe that all student neurons match teacher neuron in this case, however not perfectly the end of training. This is an example of the red regime (see Fig. in the main text), where the students converge to perturbation of the n-copy configuration.
0.75
637
050
697
360
-625
-335
01 12
RAUTONT Rouran2
Meuzon1 neuronz
A1 2
2,
A3
2. K 20
1.3
0.5
075
0.2
-02
-052
-550
-133
01 12 6 M
nd
0107 u0
n2 03
B1
B2
4,
B3
4,
1.00
8.50
1.25
025
0.25
-150
-075
-1 00
64
-3.00
C1
12, A
16
C2
12, 1: 24
C3
12,
32
Figure 9: Optimal configuration found by gradient flow for ReLU activation function for n 2 (top); n = 4 (middle); n = 12 (bottom). The last row of the correlation matrices represents
|u,
which is zero for all student neurons in all cases. Top row, copy and an average neuron: the copy neuron very close to one of the teacher neurons, and the average neuron is close the average of the remaining teacher neurons while negatively correlating with the copied teacher neuron. The negative correlation increases in magnitude as I increases. Middle row,
2; the optimal point is composed of
4; for l: = 5, the optimal point is perturbation of the all-copy configuration; for k 8, 16, it is close to the (n - 1)--coy---averrt 16, 24, the optimal point is a perturbation of the all-copy configuration; for k 32, it is close to the (n 1) cc-ppyy-1---- -laverae configuration. In all regimes, the norms and outgoing weights of all student neurons close to each other (for the bottom row only the first four neurons shown).
configuration. Bottom row; n = 12; for
We show some representative trajectories of gradient flow, the regime when the CA-optimal critical point is minimum in Figure and in the regime when it is saddle point in Figure
C.3 ReLU Experiments
In this subsection, we will present structure minimum loss configuration found by gradient flow. In the regime n & At the minimum loss configuration is qualitatively similar to the optimal-CA solution but without perfect decoupling. For that is slightly bigger than n, gradient flow finds an "all-copy" configuration for 4, 12. The overall trend of the minimum loss configuration is
18

---

similar to the case of erf activation; however, as we do not have the analytic formula of correlations, theoretical prediction for the optimal solution is left for future work.
D Constrained Optimization Formulation
Expressing the loss function in terms of the order parameters yields the 'projected' loss function
L nikk proj
(T 1) +2
a:ay Go (ri, TV', Pit)
a:bigo (Ti, |1yll mey) +C
student magnilude term
student-siune student interactiin
student-teacher interaction
(25)
where the constant term C = Ex-Dlf((x) Lproj has n(k + 2 (2) parameters, that is output weights and n(k + 1) + (2) order parameters, instead of r2dd + 1) parameters of the original loss function. For d h: + "=1 Tpoj has significantly less number of parameters.
In the special case d = k; the incoming vectors can be expressed as a linear combination of the teacher's incoming vectors, hence the correlations between them are not free (see Appendix D.I]
Each normalized incoming vector can be expressed as a sum of projection on the span of the teacher's incoming vectors and an orthogonal component
W Ti
Uj + ot,
2 Σ j=1
uij
1,
(26)
and the inequality constraint pops up since ||u, |12 = correlations between the incoming vectors, get
|| 2|2 where u; = (Ua, U4g)). For
Par uy + of u7:
(27)
which yields the second set of constraints on the optimization problem
pir Us uy v 11.11,112
uup 2
Wi [n].
(28)
since have |ut u// S ||o!! |1/0:/11 due to the Cauchy-Schward inequality.
note all incoming vectors are in the span of the teacher's incoming vectors, we have that
||ul|| = 1. As result, the second set of inequalities in Eq. 28 collapse onto equalities, hence the secondary constraints are in fact equality constraints (see Appendix Eq. (34). We show that this indeed the case for the non-trivial critical points of the one-neuron network in Section
In general, the constrained optimization formulation possible for not-orthogonal teacher networks.
Remark D.1. (Non-orthogonal teacher network) can relax the assumption of orthogonality between ....... Up to linear independence. Let collect the incoming vectors into matrix ,...... Uk] [kdkk, The expansion in Eq. 26 can be rewritten as
wi
2o + + = VT, +
(29)
where T, (Tn.... 7ak) R and us Uy foror all j [k]]. The normalized vector has unit
norm, hence we have
+ VT, + 1
(30)
The correlation vector be written u; VTVT; R2 which yields the following constraint
TT <1
(31)
19

---

D.1 Equality Constraints in the Case d = h
In this case, the teacher incoming vectors ..,..., 2y span the input domain R4. Each incoming vector (of the student) can be expressed as linear combination of the teacher's incoming vectors
W
Uij
S =1
(32)
and the equality constraint pops up since the normalized vector has unit & norm. The correlations between the incoming vectors are then expressed terms of the student-teacher correlations
Par = "4 uer
(33)
Therefore the optimization problem is equivalent to
k
n
min Σ a.go (ra, 1) 1)+ Σ
agaggo (Ta,re,
Σ
Ui ug -
a;bygo (ri, [1lll
j=1
j=1
subject to 2 1, r. 22 for all E [n].
(34)
Since (*) student-student correlations terms not free, the problem has only n(k + 2) free parameters and k equality constraints, yielding n(k 1) effective parameters, which is the same number as the number of parameters of the original problem in the weight-space.
D.2 Binary-Equality Constraints in the Case d = k +
In this case, there is only one direction orthogonal to the span of the teacher's incoming vectors (i.e. oi || v.)). ). Therefore the general incquality constraint Par reduces to
Pay the + V1 ||u 12
(355)
D.3 Three-Neuron Network
We present a case for the three-neuron network where the optimal solution of the constrained optimization problem may be projected back to the weight space. particular, let us consider positive and monotonic activation function, i.c. axx) > 0. This implies that 9., >> 0 and that is increasing correlation. It natural to expect that all outgoing weights are positive since this would bring the network function closer to the target function. If the network is overparameterized, some outgoing weights may zero or even negative (balanced by positive outgoing weight corresponding to the same incoming vector). Let us pick three positive outgoing weights aı, a2 ag > 0 and three corresponding student-student interaction terms
minimize (1a T2, P12) + ala3go (1I, r3, P1a) + a2a3go (12, r3, P23) + ....
subject Ipw' u, uyl
/1 |4 2 /1 up 12.
(36)
Note that each Pir is decoupled from each other. Since go is increasing in correlation, the minimum of each term above is achieved when
| 2 /1 || 1|2. This implies that the inequality is tight and therefore wit || 4 moreover,
= uyy
(37)
1 o- and
uy 17
(38)
up to a sign flip. However, it not possible that the three vectors all have pairwise flipped directions to each other would have (-) (-) (+). If the optimal solution of the constrained optimization problem verifies ||u < 1, then we conclude that it cannot mapped back to the weight space; in
this case, the optimal Пooo would only give lower bound the optimal loss of the weight-space.
20

---

E Copy-Average Critical Points
Theorem E.1. Assume that a is the erf activation function. We pick a copy-average parameter vector
g (wi aï) ... + (w o,))
(39)
where [w,, a)) is non-trivial critical point when learning from a unit-orthonrmal teacher fü with the incoming vectors Us.-1tl.... Us, shown in Eq. 17 Then 6* is critical point of the loss function L where the target function (x) E (v..
Proof. Let us write down the partial derivatives with respect to the outgoing weights and incoming vectors
d da;
n
T, (0t) 2E x){
Σ
ajo(us x) (x)),,
j=1
d dw;
L = En (wt 2)z(
Σ ajo(us -))
(40)
We will show that they are equivalent to the following
da; (0*) = 2E <)ato(u x) ff())),.
d (00) Ba- (wf z)æ(ajo(w} x) f7 (x)),,
(41)
dw;
which implies that the partial derivatives are zero, since (w: a.) is critical point of the loss
L Ee-Dl(aa(w x) - f (z))
(42)
Since (w at) ) is the optimal solution of the teacher network generated by 1 Us from
Theorem 5.1 we have that w is in the span of Us +1.++ Us,- We have that
w w 0 and w U2 = for je [k) [s:-1 +1
(43)
since the two incoming vectors are in the span of two orthogonal subspaces respectively and w is orthogonal all other teacher incoming vectors that are outside the span of t Us For two orthogonal vectors say wt and v, we have that
Ez- (w) 2)02(v
=)] = En- (w) (ou]:xx]BBaaa =))Ez~D(o2(() x)]
(44)
which is zero at least one of dn and d2 is odd. This implies the first equation in 41 for the partial derivatives with respect the outgoing weights. In order show the second equation for the partial derivatives with respect to the incoming vectors, let us define Sj = =19--1 + 1,
W7, = (eowolo'(wj x) x) x)
(uk x])]
(45)
KES
and note that it suffices to show that W = 0 for all + j. This is true if and only if W.,; 0g =
where (ū.....;d)) form an orthogonal basis of Rd. Let us choose 0j = U1 .... 0g = 12 and that Uk+1. ..., Ga completes the basis if d > k. One can observe that Wa, 0 0 for k + 1 < < d since x Bg is an independent Gaussian from the others. Hence, the expectation, i.c. W.j Uz, factorizes with factor of Efxx ūz] which is zero.
remains to check W., 1n for all sy's. We split the analysis into two cases. ot $ Sy, then ti is independent from w and from x for is E Sj. Hence W., splits into
alu x))] = 0, Wiy by Er jo'(00 x)x vn EEWD (aato x)) Σ KES,
(46)
21

---

where the second term in the product is zero because wj 2 and Up 2 are centered Gaussian and is odd. For the second case, where 1/ Sj. using the fäct that x by independent from w z and from Uh for 1 * k, we have
Wiy v Ezzolo' (w} x) for mmjjo(w) x) Σ ovv - x))]
(47)
kE9,
= Ez~ Bo io ulaao((w) x) o xx)]
(48)
Ezwplo' (wf x)] (Ezwolajo(us z)æ ui Ez-plo(1 Z)x -)) vil.
(49)
Applying Stein's Lemma to both terms on the right we have
Way Ez~plo' (w, x)] x)] (Baajrr ( Elo
(50)
where standard Gaussian and r V 2k-1. u² = V /. az (parameters of erf). Hence we show that
0 u Ero')) (2))
(51)
To show this, we use the following relation
2
2
y²u
gar(r, r; u) = (ro(rr))(
aresin
(52)
,2 12+1
Differentiating with respect to the correlation we have
d du gen u) = (lo''rzz (rv)) 2
1
(53)
(r2
1
w²
PI-
In particular, at correlation zero, we get
Elo (rx) =
(54)
V +1
Therefore, have
2 2k 1
E Blo (rJaa)
Elo (x:) V π
(55)
2k:
which implies 51 and the proof is complete.
For general activation functions, using the substitution Eq 41 the first partial derivatives in Eq. reduce to
40
d da,
()) = Σ a (1 11. 2(k - (x)go (1100 1,
(56)
Vigta"
which is in general non-zero if a is not odd. Lemma E.2. Assume l2 E1 We have that
(2 + 1) 12d((2) < (b + 1) - (1))
(57)
Proof. We first show that the function x² aresin 2z Using the Taylor expansion of aresin, we have that
is increasing for and convex for 0.
f(x) x² aresin
-
1-3
(58)
2r 2
2.3
23
2
254
where the higher-order terms all have positive coefficients. The first derivative is
1.3.3
f(x) 2- 2
(59)
2.4.5 25
22

---

which is positive for z 2 1 since we have
1 1
1-3-3 3
1
1
(60)
2
2
2.4-5 5 2
2²
2
4
42
43
3
The second derivative
1
1.3.3
f" =
+
+
(61)
3 23
2.4.
25
which is positive for positive
First, let us show that Eq. 18 holds. Plugging in the analytic expressions for (0, and I.ar(), is equivalent to
2 to aresin π
(5) to π
2
2 π aresin (2) - ((4+ aresin 2((++2)) lg arcsin 1 )
(62)
28
Since f(x) is increasing for x 2 1, the second term inside the parenthesis is positive, hence the inequality holds.
We will now prove the statement of the Lemma. It suffices to show the following for all lg 2
(e + 1) (e)(e) (e) (e
(63)
since then continue decrease by one, i.e. l 1-1, 1, equivalent to
2,... until we reach 61. Eq. 63 is
(e + -1 + (e 1) < <0
(64)
that is the second-order finite difference, similar the The proof is completed by observing that (.) is discrete-concut function since its continuous interpolation
of continuous function.
(x)(()) = x arcsin( ) x²² arcsin( 2x.
1
(65)
is concave for since it can be written (x) oz f(x) where is convex for x >> 0.
Lemma E.2 tells us that we add one neuron the teacher, then it is better to approximate it by the student neuron that already approximates many teacher neurons. Applying Lemma E.2 iteratively, we get
Lir( (k- + Lur(1) < (k- Lir(2) < (f2 L(n() < Lir(b2) + Len +1)
(()(
where bu 1== 2 = if l is even and = 11
if - k-3
odd.
Theorem E.3. Consider unit-orthonormal teacher network = o(13 x)) and the erf activation function. For an copy-average configuration up to permutations (of the student and teacher neurons)
student network with n neurons, the minimum-loss
= (E1V1, 61) ..... 2 (en- 11n-1Un-1,,--- @ (6 kn))
(66)
where e {1 and (w a,))) is given by Corollary 5.2 after substituting i with k-
Proof. We will conclude with a simple argument that the minimum-loss CA configuration for multi-neuron network with neurons is (n-1)-C-1-A In particular, if there are two averaging neurons inside the student network, we can redistribute the teacher neurons shared between these two to a lower-loss CA configuration by ensuring that one student neuron copies and the other student neuron averages (see Lemma E. 2 The minimum-loss CA point then achieved among CA configurations where at least n- neurons each copy single teacher neuron (of the A possible ones). The remaining student neuron can be treated as a single-neuron network learning from teacher with k --n+1 1 neurons for which we know the optimal solution is to average (Theorem 5.1
D
23

---

E.1
Number of CA Critical Points
There is combinatorial number of (fi.... CACC critical points, that is
(
k
(k-
(61 + Ln)
cll1.... (n)
(67)
(1)
fn
where On ;= c(li,..., fn) counts distinguishable permutations between the neurons of the student network, and the binomial coefficients stand for grouping teacher neurons into n non-empty buckets.
lf = = a, permutation between the student neurons already counted when distributing the teacher neurons, hence C2 = 1. If all f,...,, en distinct from each other, have that en n! since we swap all pairs of student neurons after assigning groups of teacher neurons. In general, let e, denote the number of i's among f,,..., e, for all i 1,..;: k;; the formula for the permutation- is given by
n!
dl1......)) = Ci ci!...c!!
(68)
F General Properties of the Interaction Function
In this Section, we introduce some general properties of the interactions. We use these only for the one-neuron network this paper (see Section G). however, these properties are likely to play a role in studying the networks with two more neurons.
We first present the partial derivative of general interaction function, i.c. activation functions may be different, for example, if the student activation function does not match the teacher, with respect to the correlation in a simple expression Lemma In the second part, we present property of the activation function sufficient for Assumption (ii,, and show that the differentiable activation functions studied in this paper satisfy this property in Lemma
Lemma F.1. Assume that functions σn and 02 are differentiable. The partial derivative of the following Gaussian integral term Bla1(11)) with respect to the correlation E[xyy] = u
d Blo (rix)o2((2)) du
(69)
We apply the Lemma for σ1 02 in the main text in Eq.
Proof. We compute the derivative of Bla (riæ))2 by making the correlation explicit. Denote u' = "V=vv->> 2²d and y = un + u'z. After the computation, we use Stein's lemma to reach the desired
formula.
8 E. o1 (ri (222))(222) = r2 (ro))))) T2" B[a (ra)))
(70)
u'
where x and are independent standard Gaussians. Here is reminder for Stein's Lemma for standard Gaussian
[[((2)2] (2)). To remove in the first term, apply Stein's formula for v(x) a ( yielding
(71)
(1r2E(((((1 + (3u
(72)
To remove z in the second term, we apply Stein's formula for u(2) = o' (rx by considering fixed which yields
- (o
(73)
Summing up the two terms completes the proof.
n
For softplus that increasing and convex, using Lemma F.I for =1=02= twice, we infer that the interaction g is also increasing and convex in u. Hence, for u < O, Assumption (ii) softplus. However, for the other activation functions, using second-order derivatives does not suffice to show the assumption. We will propose new property of the activation function that implies that the interaction satisfies Assumption 3.1 (ii) and prove that softplus with 2, sigmoid, tanh, and erf satisfy this property.
(ii) holds for
24

---

Lemma F2. If the activation function a is thrice-differeee thrice-differentiable and it satisfies
d' (z) zo" + (=) > 0,
(74)
then its interaction satisfies Assumption 3.7 for all u E 1-1,1). Softplus with 3 E (0, 2}. sigmoid, tanh, and erf activation functions satisfy the above inequality.
Proof. Let us first write out Assumption 3.1 (ii) explicitly using Lemma F.1
ri
(v) ((r
(75)
where (xx o' (x). Using Stein's ' Lemma for v(z) = (()r)) get
Blo(r1z)0 (lo'(m + [(a(r)))"))
(76)
The desired inequality is equivalent to
(lo(ria) o' xu)) 5" )
(77)
Let us introduce [(x) (a) - 2σ (x) + (s)) For y = ur + n'z where v' = v1 u², we have
the conditional average of y fixing x (we drop conditioning the right-hand terms for convenience)
[[f(y)aa] = [v) Elyo' (v)) + B[o"(v)
Ela u.:Booo (v)) [w))
[v)( - uxxE|o' [v)] (w')" EloWww)
=v]( - (x))(o + (v)]
(78)
where second last equality comes from Stein's Lemma for u(2) o' (uz + n'z)). Hence the desired inequality is equivalent to
((v) > 0.
(79)
By straightforward calculus, we will show that f(x) 0, or that f(x) 2 0 and f(z) = if and only if x O. In the latter case, the expectation in Eq. 79 is positive since f(y) 0 for some y values of the integrand. First, for the sigmoid and tanh activation functions, for which we have
ea
ea (1 e*) d"(x)=
(e 4e +1)
(x)
d'(x)
(80)
(e +1)2
(6a +1}3
(c + 1)4
Hence, we can explicitly write as
e"
- x ea (1 6*)
+ c5(2"r -6 +1)
f(x) = (e 1)2
(81)
(er + 1)3
+ +14
6²
(ee 11 (le + +1) x(1 e") + 1) + (e20 - 4e + 1)).
(82)
Therefore showing f(x) > 0 is equivalent to showing that the factor the right, that is,
2e 1) - r(1- 2
(83)
is positive. For x < O, we have ex < 1 which implies -I(1 - c²n) > 0 and (1 - E*)e" < 1/4 due to the inequality of arithmetic and geometrie means hence the first term is upper bounded by 112 and since we have -2, the whole term is positive. For x 2 0, we have ex 2 1, hence we can rewrite the inequality as a sum of non negatietee terms
(e 1) + 2 x(e2 -1) O.
(84)
Let us now handle the case of erf. Its first three derivatives given by
a(x)
2 d'(x)
2 re vT
2² 22122""
(x)) V
2
(85)
Hence, we can explicitly write f as
f(x)
2
- 12
x +
x²
1)
4
x 12 z
(86)
V
V
25

---

that is non non-negative for all x and zero iff x = 0. Finally, for the softplus activation function with B (0, we have the following derivatives
Be
82 =
(②) (e + 1)
o'(x)
(e + 1)2 d'(x)
(87)
(eRx 1)a
Plugging in the function f, we get
Bxx
BeBx
32 (00)
[(x)
(e +1
x (e + +12
(e + +133
(88)
(eR + 1)3 ((e
8r + +1)² - xx((e + +1)
32 -00))
(89)
Therefore showing f() equivalent to showing that the factor on the right, that
2B2 + 20 (2 xB ( + x3 B
(90)
positive. For S have that xx and 32 N -2 since 3 hence it sufficient to show that the following is positive
22 8z + + 8 (e 1)² + 32
(91)
which is sum of squares. For 0, in the rest of the proof we will show that
e 3r +2 xB 5²) + x,3 + 3 > 0,
(92)
for 3 E (0, 2). Using ex 2 (3x))/22 + 3x + 1, it suffices to show that
e0x ((xx)²// 2 +1
82 0. + x5+
(93)
If (8x))/2+3 - 82 > I, then the first term is bigger than 3r + hence the above term is positive. The remaining possibility is that we have
x2
2
< 1
(94)
2
32
8 < 2 implies < 1 and x² > implies 3 {> V2. Hence have -x8 + 882 0 since >
Therefore, if we have (Br)2/2 /22 + 3²
2 0, Eq.
92
is positive. Assuming the opposite, we get
x²
3
(95)
2 32
< implics x <1/V2 and x² implies 3> v3.
Going back to Eq. 92. what remains show that is positive in the domain 2<1//22.B E (V3,2). It suffices to show that e@ + 2 xz 3 > 0. Assuming the contrary implies e0x x:3 2 since
B < 2. We can then deduce that x/ < c = 1.2 since otherwise we would have
Br 1+ 8x + (3z))
(Bx))
(96)
2!
3!
²²
21 8x +
+... + Br + (e - c- 11 +1
(97)
21
3!
which implies a contradiction. c can be chosen smaller but this will be enough for our purposes.
Assuming 8z +2 x3 - 32 < 0, let us expand Eq. 92
( +2 xB 82) 1 x3 + $2 2 (using Bxx < <Bxx + 2)
(98)
(x.3 + 2z + (x + () x3 8²) + x;3 + 32
(99)
(z.3 + 2) - (x.3²² (1+ +5 52 > (using axx > Br +1)
(100)
3- + (2 3) 3 2n 0
(101)
where in the last inequality we used 2;33 1.2. We note that this inequality holds for slightly larger using the same technique, however, for significantly larger 3, the property breaks down.
26

---

G
The One-Neuron Network
We study the critical points of the following loss function
& Ei x)].
L (w, a) = BB-Dlaaow x)
(102)
in particular, the optimal solution. For = 1, all configurations of order parameters are realizable in the weight space, therefore, the optimal solution of the following loss (repeating Eq. T4)
1.k L proj
a² a²go (r, r. 2a Σ bygo (r; |1yll, u,) + const, subject Eu 0, (103)
is equivalent to the optimal solution in the weight space. Let us denote the unit ball by B (u),..., un) |u +... + u < 1.. Its interior is denoted by int B and its boundary is denoted by OB.
We will present the results for the one-neuron network in five parts
1. In Subsection G.I
we give proof of Proposition 4.1] any non-trivial critical point (w, a) of L L satisfies that w is in the span of the teacher's incoming vectors the activation function satisfies Assumption 3.1 (i). Moreover, we show in Lemma G.2 that the corresponding order parameters should satisfy Lagrangian condition (Eq. [05p.
2. In Subsection G.2] characterize the topology of the loss landscape in terms of its critical points for the activation functions studied in this paper and for the unit-orthonormal teacher. Our results for the one-neuron network strong in the sense that it gives all possible critical points of the loss landscape.
(a) In Subsection G.2.1 we give a proof of Theorem 5.1 for general activation functions satisfying Assumption 3.1 any non-trivial critical point of the one-neuron network attains equal correlations that are either 1/vTE or 1///..
(b) In Subsection G.2.2] we study the two-dimensional loss obtained after applying Theo- rem 5.1 From its derivative constraints, needs to be satisfied by the incoming vector norm r at any non con-trivial critical point with equal correlations
get fixed point equation (Eq. 120] that
Numerically, we show that there is unique solution of the fixed point equation for u. > O for differentiable activation functions studied in this paper (Fig. 10 Finally, we give some sufficient conditions in Eq. 12T to prove uniqueness based on log-concavity; numerically, these are shown to be satisfied by softplus and sigmoid activation functions.
3. In Subsection G.3] we give a proof of Corollary 5.2 by solving the two-dimensional loss for the erf activation function. Moreover, from the proof, we conclude that there are exactly two non-trivial critical points identical up to the mirror symmetry of erf (since it is odd); and these are the optimal solutions for the loss landscape.
4. In Subsection G.4 we present and prove Corollary G.5 solving the two-dimensional loss for the ReLU activation function. We find that there are two non-trivial critical points of the loss function: a saddle 'point' at correlation -1/VK and the optimal 'solution' correlation 1/Vk. Due to the positive homogeneity of ReLU, these are not two points but two equal-loss hyperbolas in the loss landscape.
5. In Subsection and give a proof of Theorem 5.3 Absence of analytical expression for the Gaussian integral terms make the problem challenging;
G.5 we study the two- two-dimensional loss for the softplus activation function
use several non-trivial steps in the proof. The proof shows that there is no critical point at correlation -I/VK; and a non-trivial critical point (w* a²) correlation 1/VT satisfies the following bounds: a" > k and ||w""| || < 1/VE. Numerically, we find that these bounds hold for other activation functions studied this paper (tanh and sigmoid; Fig. 5)
G.1 Any Non-Trivial Critical Point Satisfies the Lagrangian Condition
We add a reminder here for the definition of the non-trivial critical point (wr, a): is a critical point of the loss function that satisfies a ef 0 and ||M||| * O.
27

---

Proposition G.1. Assume that f is an orthogonal teacher network of width k. If the activation
function satisfies Assumption 3.1 3.1 (i), any non-trivial critical point 0 (w", a²), ), L.e V (8*) 0. satisfies that w" is in the span of the teacher's 's incoming vectors.
Proof. We will prove by contradiction, Let us assume that w is outside of the span of the teacher's
incoming vectors. We will show that (w, a) is not a critical point for any of 0. Mapping (w, a) to the order parameter space, we get that (r, u, a) where u = (ui,..., ux) E int which implies Mi E (-1,1). Since u E int B and r > O, we have that (r, u, a) is a critical point of pup since the boundaries are not seen near the neighborhood of this point. Therefore the partial derivatives of 4 are all zero including
pcoj
bj duy go (r, ||l Uz) = 0.
(104)
(i), we have that Ouga (ri, 12, u) > 0 for u E (-1,1) which yields a contradic- tion. Thus, each critical point of the projected loss is on the boundary, i.e. u E 0B, which implies that the incoming vector is in the span of the teacher's incoming vectors.
From Assumption 3.1
We will next show that any non non-trivial critical point satisfies a Lagrangian condition since it is on the boundary of a constrained optimization problem.
Lemma G.2. Let 0- (w, a) be non-trivial critical point of L. Then the corresponding order parameters = (r, u, a) satisfy the following Lagrangian condition
by (r, [1slll uz) = Auj for all j E [M:].
(105) )
Proof. We will first show that for any differentiable path (r; (1), a) on the boundary such that (()) E 0B for t E (-c, e) for some >> O and 7(0) = u, the following holds
L dt proj (c(t)) = V T. (p) 7(()) = 0.
(106)
Let us assume the contrary. We construct the corresponding following path in the weight space
((t)=
uy (t)oy + Tu a
(107)
Thanks to the equivalence of the losses along the path, we have that
d _ r. 1.' (0(O))0 dt
d
(108) )
dt pmns (7(1)) 0,
since ((0) A is critical point in the weight space. Therefore, Eq. 106 holds for any differentiable
path on the boundary and implies that (p) is orthogonal to all 7 (0). The vector that is orthogonal
to all 7 /(0) is the gradient of the surface, that is 2(u1, .... Ug.). Hence we get V., [(p) || which is
written explicitly as the Lagrangian condition in Eq. [05] This is equivalent to setting the partial
derivatives of the Lagrangian loss with respect új zero where the Lagrangian loss is given by
C(p, 2a
bigo (r, lesll. uz) + X
uz 1).
(109)
We set x = N/o in Eq. 105
G.2 General Activation Functions
Before we present our results, let us take a detour to check the applicability of the convex optimization framework, For convex and lifeeddifferentiable activation function such as softplus, applying Lemma F.1 twice implies that the interaction 9- (r F2, -) is convex function of the correlation E (- 1, for T1, 12 > O. Let us consider fixed a < and r r>0 0 and consider the loss parameterized by U,ys s. It is convex since its Hessian is a diagonal matrix with entries
d²
d²
T.
Bo (r, ||0y|1: uz)) > 0. 2a du
(110)
du
28

---

Since the constraint on the correlations (Eq. 14) is also convex, we get a convex optimization problem that has a unique global minimum (see Boyd al. 44 Section 4.2). Swapping a pair of u, does not change the loss, thus it is permutation symmetric. If any two uj were distinct from each other at the minimum, then permutation would also be a minimum which would violate the unicity. We conclude that at the unique minimum point, the correlations are equal to each other. However, for the case a O, and for other activation functions, the objective not convex.
We instead use Lagrange multipliers for proving Theorem 3.1
G.2.1 Proof of Theorem 5.1
Theorem G.3. Assume that the activation function satisfies Assumption 3.1 At any non-trivial critical point (w", ,a") of the loss 1,2. for the unit-orthonormal leacher network, the incoming vector satisfies
k
w
(111)
[w0)
where u is either 1T or 11/ER.
Proof. From Proposition 4.1 and Lemma G.2 we get that any non-trivial critical point should satisfy
the Lagrangian condition in Eq. 105 In particular for unit-orthonormal teacher, setting ||uy| = 1 and
by 1, we get the following Lagrangian condition
o.go (r, 1,,, Xuj Vj
E
[k],
Σ 3 =1
(112)
If 4 0, we get Ougo (r, 1, 0) = which is not possible since (r, 1, u) is increasing due to Assumption (i) Hence we have
Ougo (r, 1,,1, X.
(113)
Uj
Let us observe that (ng=(r, 1, ul/u decreasing for u E (-1, (0) if and only if
du (2 d du 90 (r, (o
d
1 d
1 d
du 90 (r, 1,
9o (r. 1, u) < 0,
(114)
u² du
which is equivalent Assumption 3.1 (ii) for (-1, 1)\ (we included (ii) for simpler statement which is already implied from Assumption 3.1 (i) at u = 0).
On Assumption 3.1
Taken together, conclude that 0.gg, (r, 1, u)/u is one-to-one in
(-1, \ {0}. We need to consider the remaining case u, E (), For k 2 2, necessarily, we have u, = 0 for j / which is possible have shown. For k = 1, u, E (-1, is the only option that satisfies the boundary condition. For 2 2, Eq. T13 jimllie that all correlations are equal. Combining it with the boundary condition, we get U1
Ug
with ku
which completes the proof.
G.2.2 Two-Dimmensional Loss, The Derivative Constraints, Uniqueness
At non-trivial critical point, we proved in Theorem 5.1 that all correlations are equal and denoted by that is either 1/Vk or - -l/Vk. The projected loss at a critical point reduces to
a² gr (r, 1) 2kagæ (r, 1, C.
(115)
Moreover, at critical point, the partial derivatives with respect the outgoing weight and norm should also be zero which gives the following two constraints
O.L 20go r, 2kg. (r, u) = 0,
0 = (r. 2kad, go (r, 1,4) 0.
(116)
29

---

11-
Agpeadi
16 20h
04
02
14
02
02
60
25
1.3
1.5
00
2.5
1.0
15
00
05
1.0
1.5
00
05
1.0
15
gr 2r (2 9o (r, 1ogg 1, u]) for activation functions erf. softplus with 3 = 1, sigmoid, tanh, and gelu, respectively. Zero crossings of are shown in red. For softplus and sigmoid, we observe that f negative for = 0, E (0, positive for r 1, E (0, 1), and increasing in E [0. 1] for any fixed u, thus satisfying the sufficient conditions in Eq. 121 However, for tanh and erf, f shows non-monotonic behavior r when u close to 1. observe that f does For the GeLU activation function oxx) (x)), which is non-monotonic,
Figure 10: The graph of fr, u)
not cross zero for any (r, u) pair the plotted domain. approaches zero from below when - oo thus showing a very different behavior from the other activation functions.
which can be rearranged into the following (assuming (r r, + and (rg.(r, r,,
a
(r(r, 1, 1, = (r,
(1,
(117)
k 90 (r)
The second equality between the two ratios of Gaussian integral terms gives fixed point equation the norm r. Writing the interactions in Eq. 117 exxpliiitly and rearranging the ratios, we get
f(r, u) = (ro((rz)o
((o((r=)((v)))) 0, ([o(r=)a
(118)
(ra))
where æ and y are standard Gaussians with correlation E[ry] = u. Let us define the following helper functions
G(r) = E (((())))(rz
B[a((x)) 2 dr log( log(E(((r=))),
[(a(r=)a(y)] dr G(u,r) (()((r)) log(E(((rz))
(119)
which yields
u(r,u)= G G(u,r) d
B|o(r=)
log Elo(rr)o(v)]
0.
(120)
Let us consider the case u 0. We want to show that for any given u E (0, 1] there is a unique E (0, 1] such that (r, u) 0. Under the assumption o00) o 0. if the following three conditions are satisfied for all u E (0, 1].
(0) (y)(y) 0,
(0) [v]
Elo (z))(())) (lo((z)o(y)))
[x)
E[o(z)o(v)
d log
Blo(rx)
(iii)
0,
(121)
2
Bla(r=)((v))
then have unique solving Eq. 120
explain next. Note that the first two conditions > U. respectively. The tricky part is the third condition which
equivalent to f00 u) < 0 and f(1, 2)
is equivalent to showing that
((((r))
(122)
E (r(()))
30

---

is log- cogcccnavee in r. We note that marginalization properties of log- conccee functions may be helpful here. In this paper, we were not able to prove the sufficient conditions listed above for general activation functions that do not admit analytic formula of the interaction, even for softplus which we studied in detail (see Subsection (.5). Instead, we present the numerical integration results, which show that for any given E (0, 1]. there is unique E (0, 1] such that f = 0 (see Fig. 10 Once shown to be unique, then the matching outgoing weight a follows from Eq. 117
G.3 Closed-Form Solution for Erf Activation
Corollary G.4. Assume that the activation function is Gerf. The optimal solution (w", a²*) is given by
1
w
|| V 2k a- k.
|
Vk
or, equivalently, by (-w", -a²). The optimal loss given by
2 k arcsinn () 122 arcsin( 1 2h ).
(k) π
(123)
Proof. Since erf is an odd activation function, it suffices to find parameters of the non-trivial critical points satisfying u 2 O. For any such critical point (w", a²). its mirror symmetry (-w", -a") is an equivalent critical point due to Eq. 41
For 0, we have that Jer (r. 1, u) which implies Gar(r; r, 1) (ro ? due the first derivative constraint in Eq. 116 which holds if and only if = 0. This gives possible trivial critical point yielding the zero predictor function.
For given и Vt > 0, from Fig. 10. we observe that there is a unique r E (0, 1] satisfying the fixed point equation in Eq. 120 For uniqueness, we rely on numerical integration. We will find one solution to the derivative constraints given below
agen(1, r, 1) kgea(r, 1, u). ad.ger((, 1) (rn 1, u)
(124)
for a given k and equivalently u - > 0; and due to uniqueness, conclude that it is the only non-trivial critical point up symmetries.
In particular, we will use the analytic formula for the interaction function
2
T172
Jed Ori, u2, arcsin
(125)
V
Let us find r where we have
(r u) = (r r,
that is satisfied we have that the arguments arcsin match, which happens at
2
u²
1
V
+1
V 2-
22k- -
(126)
Interestingly, this value of r, we also have
gor 1, Orgen (r, r,
which can be seen by inserting the guessed values in the following equation
2
20,
2 arcsin'
V 0
aresin' ,
Setting a in Eq. 124 completes the order parameters the non-trivial critical point. Finally, let us compute the loss at r 1/ = 1/vk, = k;
L (k) a² ('nnlrr r. 1) 2akgad (r, 1, (1, k2 (Tn r, 1) kgan 1, 1),
1 (baaesin()) ) k22 arcsin 2k 1
(127)
31

---

G.4 Closed-Form Solution for ReLU Activation
Corollary G.5. Assume that the activation function is retu Any optimal solution (w" ,a") satisfies
K: h(()
w²
| I|a"
11
(128)
i=1
forming an equal-loss hyperbola. The optimal loss given by
Te (k) = 1.2 (m(00)
11 )
+ A(h(1)) -(0)).
(129)
h(1) Vk"
We will first show that the interaction of ReLU satisfies
(i) h' = for uE(-1,1),
(ii) 1/ ((u)u<hh(( for u (-1,
h hwww)) (((() for m (uo, 1);
(iii)
(130)
Ug
where Un = 1/v2. Note that property (1) is equivalent to Assumption 3.1 (i), and property (ii) is
almost equivalent Assumption 3.1 (ii) except that it holds in the interval (-1, un property (iii)
covers up for the missing piece of the interval in the property (ii).
ReLU interaction satisfies Properties 130 Proof. Let us write the first two derivatives of h:
arccos(u)
(u)))=
h²(u)
(131)
2T
2T 2T u2
Property (i) easily comes from noting that the derivative of h is positive for E (-1,1). Property (ii) holds for u E (-1,0) 0] since both the first and second derivatives are positive. Let us show that Property (ii) holds for E (0, mo). that is equivalent to
< क arccos(u) = + arcsin(u).
(132)
V1 22
Let us note that the left-hand side is smaller than 1 since
u² u 11 1.
Note that arcsin(u) > 0 for u > 0: and 7/2 > 1. This completes the proof of Property (ii).
For Property (iii), we first show that h' hh(u)/u is convex in u (0, 1). The first two derivatives
[((w)) k"(u)
h'(w)
d²
[((w)
("(
2h"(w)
2h (u)
du
u2
du
Thus, it is equivalent to showing
- + 2h'(u)
u
2
2 aarcsin (w)
+
> 0.
(1 u 2/23/2
(1 12)1/2
Using the Taylor series of aresin and u >> O, we have that arcsin(u) u. Hence, suffices to show
(1 - 1/2 ( (-3 + 1- + 2(1 - u² 22)1/2) 2 0:
(133)
where dropped the positive term z which holds due to the inequality of arithmetic and geometric means
+ +1 201-wh u2)1/2 + (1 22) 2 3. u2
Let us assume the contrary of Property (iii). that there exists E (uo. 1) such that
h/ (uo)
h' h((w)
(134)
M0
u
32

---

Note that hu > (1) because x(1 uo) arccos( (u)) tu(u)/u / left-continuous u there exists such that
> holds at 1/v2. Since
((uo)
h' -c)
(135)
240
1 -
Finally, there exists a e (0, 1) such that u = a(1 c) + (1-o)u which gives due the convexity of h (u)/u u the following
(111 -) (1- 1
1 (wo)
h'(u)
(136)
24
This yields a contradiction since the left-hand side is strictly smaller than h (uo)/ uo hence the proof of Property (iii) complete. ReLU interaction satisfies Properties [30 End of Proof.
Proof. First, we replicate the proof steps of Theorem 5. to show that any non-trivial critical point must be on the boundary and attain equal correlations. From Property 130 (i), we get that there is no non-trivial critical point in int B. For k = 1, this implies that 11 or t4 = 1.
For general k, let us recall that we get the Lagrangian condition for non-trivial critical points
rh' (uy) Xu, Vj [k],
(137)
which is equivalent to Eq. 112 for ReLU activation function. Vj = 0 is not possible since we have (0( 0. Hence, we get
N'(u)) A
Vj E [+].
(138)
Property 130 (ii) implies that f(u) = 11'( (u)/u is decreasing for u E up) ((0.. Moreover, f is negative for and positive for 0.
If A/r < O, we get that are equal and negative, hence they are equal -1/Vk due to the boundary condition.
2. If A/r = O, we get uj = -1 for all j which implies that k 1 which is already covered above.
3. If A/r > 0, Property 130 (iii) gives that f(ug) > f(u) for u (up, 1). Since is decreasing have also f(u)= > (ua) for (0, uo); hence fuy) are equal only when all uj ug > up; however, the latter case is not possible for lo 2 2 since it breaks the ball constraint,
i.c. of + w²
Hence, we get that u, (0, uo] and are equal since decreasing in this interval. This completes the proof of replica of Theorem 3.1 for the ReLU activation function.
For the ReLU activation function, there is least non-differentiable critical point at 0 or r = 0. The careful analysis of this critical point is beyond the scope of this work. For any such 'trivial" point, the error of curo-fnnntion is equivalent
B(C ( =)) kh(()) k(k ())
(139)
We will next show that (-1/ (-1/VR) and (1/Vk)j are the global minimum and the global maxi- mum of the following loss function
Muy), subject to Σ n3 SI.
(140)
33

---

Due to the Lagrange condition, there is no other critical point, hence these are the only two critical
points of the constrained objective in Eq. 140 The objective then reduces kh(u) which is minimized at - -l/VE and maximized at l/VE.
Next, will give the closed-form solution of the remaining order parameters. Plugging in the correlation in the loss and using the factorization of the interaction in Eq. [4 we get
L = a222 (11) - 2kar h(u) + C.
Let us set = or. The loss is a second-order polynomial in
(a2
h(u) h(1)
h())) + k k(k 1) h(1)
L = h(1)
2
where we made the constant explicit. Since the coefficient of the leading term is positive, there is minimizer and it the only critical point. Taking the derivative, the minimum attained
lu(u) a. (1))
(141)
Finally, plugging a. we get
L(u) = -k² h(u) - h(1)
+ kh(1) k(k - 1)A(0).
(142)
For u = 1/Vl and = -1/vkk huu) is non-zero; hence I(u) is smaller than the loss of the zero function (trivial critical points). The smallest loss is attained at = 1/vI which is, therefore, the optimal solution. We conclude that the critical point at -I/VE saddle point since it is maximum in and minimum in à.
G.5 Bounds Incoming Vector Norm and Outgoing Weight for Softplus
Unlike ReLU and erf, the interaction function does not have a known analytic expression for softplus, hence the proof involves some techniques to compare ratios of Gaussian integral terms.
FKG Inequality. We will use a special case of the FKG inequality repeatedly, that is,
[[[()))(x)) > [[((())](((()))
(143)
both f. are increasing (or decreasing) implying that and g are positively correlated. The inequality changes direction if f is increasing and is decreasing (or vice versa) implying that f and g are negatively correlated.
We will rely on some specific properties of the softplus family that are developed in Section G.5.4 Unfortunately, some of these properties do apply other activation functions. As first example of managing interactions that do not have an analytic formula, the proof may inspire generalizations other activation functions. Below we present the proof sketch for Theorem 5.3 In the following
Subsections G.5.I G.5.2 G.5.3 and G.5.4] the components of the proof are presented in detail.
Proof Sketch. We want to characterize the zero(s) of f introduced in Section G.2.2 that is
G(r) -(u,r) (loa(rz))ara))t Elo(r=)2
(144)
(la(rz)
For r E [0, 1]. there is a unique correlation u E [0, 1] such that f(r, u) Denoting this correlation by h(r), we have map h [0, 1] + [0, 1] with boundary conditions h((0) 0 and h(1) = 1. For r > 1, there no solution of f. As consequence, no r 2 0 solves f(r, u) for negative u, hence there is no non-trivial critical point at u 1l//k.
In Section G.5.2 we prove the inequality ((r) 2 which gives us the upper bound on the norm since we have that the correlation at non-trivial critical point is h(r) 1/v.. Using this inequality and Stein's ss Lemma, we give a lower bound on the outgoing weight, that is 2 k. in Section G.5.3 In summary, any non-trivial critical point of the loss has equal correlations that are u = 1/vk, the norm satisfies < u, and the lower bound on the outgoing weight follows. End of Proof Sketch.
34

---

G.5.1 Constraining the Zeros of f
In this subsection, we will describe all zero-crossings of f: [0, oo) x [--1,1] +RR R. We need to check four cases (i) r = 0. (ii) = 1, (iii) r > 1, and (iv) E (0, 1).
(i) r 0: Note that G((0) ((0,0) 0. Since G is increasing in correlation for u E [0, 1] and (iu, 0) G(O, 0) for < (Lemma G.6). the only solution is u O.
(ii) r 1: Note that (11) ( 1) since y = z due to correlation one Eq. 119 Since G is increasing in correlation for E [0, 1] and G(u,, 1) G(O, 1) for < 0 (Lemma G.6.. the only solution is = 1.
(iii) 1: We will show that there no zero in this case. Let first show that G(r) G(1,r) for r > 1, which is equivalent to
Elo' (1((mr))((ra)))))(( Bo (oora))))>>
(145)
Changing the measure of æ from the standard Gaussian p(x) to p(x) = p(x))((z) /Blo(r=)))) we get the following equivalent inequality
o (rar)ar
(2x) (rx)
o d'(x) (x)) a(rx) a(rx)
E Exs
E Bgup
E Exng
(146)
o (ræ)
From the property (iv) of Lemma + rz. We need to show (xx) (rz) is decreasing in for 1. We take the derivative
G.8 we have that o' (((r))x/// (rz) is increasing after a substitution
d (z) ((())(r)) dr (ræ)
(147)
arxx)²
Since o' o(xx2//(xx) is increasing Vx E. have
d'(x)x o' d'(r))r for x O, and d' d'(x
d'(r))r for (xx) o(rz)
(xx) o(rz)
which yields o' (rz) < ox)o'
hence we conclude that o is decreasing. Thanks to the FKG inequality, o (rxx))x (ra) and o (x)//((xx) are negatively correlated which completes the argument. Since from Lemma G.6 G is increasing in correlation and ((0,r) G(u,r) for O, we have G(1,r) r(,,r) for all u E [-1, 1), therefore there no solution of f.
(iv) E (0, 1): We want to show that E (0, 1), there a unique u E (0, 1) such that f(r; u) It suffices to show
G(,,r) < G(r) G(L,r),
since G is continuous and increasing in correlation for u E [0, 1] (Lemma G.6,. it then crosses G(r) at unique E (0, 1).
First inequality; ((0,r) r) < G(r). this case, x and y are Gaussians with zero correlation, hence independent. We can expand ((0,r) r) by factorizing the integrals
E (r [o(y)] Ō(0,r) r) = E B(o(r)) E [y)) B|a(r=)]
E [o
We want to show
E lo (rx)a) [rx)) < E [o (rz)o E[(r))
(148)
which is equivalent to the following inequality after changing the measure from standard Gaussian p(x) p(x:) (rr))/(aa
(rx)z a (rx))
d'(rz)z
Ezn0
E Eo
Esn
a(rz) (rx)
(149)
This follows from the FKG inequality since we have that both a' (1(x)///((x) and (xr are increasing from the properties (iv) and (i) of softplus (Lemma G.8
35

---

Second inequality; G(r) < r)., This is equivalent to the Ineq. 145
but the direction is reversed since in this case r 1. We showed that (22)/o(rzx) is decreasing for all r' 1, therefore its
reciprocal a ((r)))//(z)) is increasing in æ. Substituting +- where = 1/r' < 1, we get that
for 1. This yields a positive correlation between o' (a( and ar (z)) (ræ) from the FKG inequality and completes the argument.
(a))/r(r)) is increasing
Overall, showed that there are no zeros of for > For E [0, 1], there is unique correlation
u, that we will denote by h(r), such that f(r, h(r)) = O. Furthermore, h [0, 1] - [0, 1] satisfies the following
i. (()) = 0 and h(1) 1,
ii. for E (0, 1), we have h(r) E (0,
G.S.2 Bound on the Norm
In this subsection, we will show that h(r) for all E (0, 1). Let us assume the contrary, which implies
((h()), < G(r,r)
due Lemma G.6 suffices to show that for all (0, 1), we have
G(r,r) $ G(r).
(150)
which yields a contradiction since G(r) = (((()) r). Showing this is equivalent to
(rx) + r')) E (r=))² < E lo (rz)o(ro)a) E (a<r))o((z + +'2)]
(151)
where r' vI- After a change of measure from standard Gaussian p(x) to
Bl((r + +222) (
p(x) p(x)
E (r + (
this is equivalent to the following inequality
T (ræ))r)
(rxx
[o((r)) o(rx)
Ezup (roo(r++r' + r<))(x]
s ExwD
(152)
(rzx)
o(rx) Eio(ro + +<2)]x]]
What remains to show is that
+
(rr))
is non-increasing in x since then we can conclude by the FKG inequality, Since r > 0 we can drop
up to change in the standard deviation of æ. We want to show that its derivative is non-positive:
(xx)
Eio(x + +r22)
(l)) + [=] < o' (s
+ +/2))a]]
<
(153)
('(x) E[o'(x +
From the property (iii) of softplus (Lemma G.8 we have that R(x) = a (x) convex. Applying Jensen, get
(xx o'xx)
o
< E
0' + r2
What remains to show is that
(a +
E
d'<
x
oo (x + r22)
< E [a +r/))/]
(154)
Note that (x + r/2 [xx] is increasing in æ since is increasing, Moreover, the function
a + r/2)
E
a'xx (x++)'z
is increasing in since its integrand R is increasing from the property (ii) of softplus (Lemma G.8 Then we conclude by the FKG inequality that Eq. 154hholds Therefore, for a solution (r, u) of the fixed point Eq. 117 we have - +.
36

---

G.5.3 Bounding the Outgoing Weight
To get a bound on let us analyze the ratio of interactions in Eq. 117
9o (r. 1. a
(155)
9o (r,r,
k
Using the convexity of softplus (property (i) Lemma G.8 we get
E (rz(ro) u/2)]
(ro())))((z BBoo (ro) r)x ('z)o'(rz)
[ra()
E (rz))
=1+ (u r) E (rx) (ra))
(156)
{ra()))
can transform the numerator using Stein's lemma with v(x) = o (ra)o'(ra)
(rz)) = rr(o'(rx)² (rz))
(157)
which is positive since softplus is positive, increasing, and convex. Combining with 2 r, we get that the ratio is bounded below by 1 which yields 2 k.
G.5.4 Helper Lemmas
In this subsection, we provide helper lemmas used the proof of Theorem 5.3 We present Lemma G.6 which shows that G is increasing in correlation and Lemma G.7 used the proof of the former. Finally, we present several properties of the softplus family in Lemma G.8 that are used throughout the proof.
Lemma G.6. The following function is increasing [0,
((o((rz)a
G r) [lo(rz)o(y))
(158)
for any r 2 0, where x and y are standard Gaussians with correlation E[ay) u. Moreover foru for O.
G(u,r) G(0,r)
Proof. Let assume 0 I u2 S 1. For the first part of the statement, want show
(r
(lo((r))o
(159)
([o(r))
(rz)
where E[zy]] = i, and E[zya] U2. Changing the measure from the standard Gaussian p(z))
(rz) E(o
p(x) p(x)
E (rz)o
we get the following equivalent inequality
a' d'(r) E (M)
(rx)x
(lo [x] E (322
E
IR
(160)
a (rx) E E(a
(ra))
Thanks to the property (iv) of softplus (Lemma G.8 after a substitution hence increasing. We will next show that (all integrations are w.r.t hereafter, hence we drop the conditioning on 2:)
we have that a ('(z)a///t (rz) is increasing in for For r the function reduces y22 with some 7 >>
E(o(v:)]
(161)
E[o(12)
is decreasing in Computing the derivative w.r.t æ, want to show that it is negative
Elo'(v) (yi)]u (}
Elo'(yiJuu
B[o'(y2)]
(162)
E (v2)
E(((1)
E Ela
37

---

Note that this is equivalent to showing
d Elo' [l)'(
d² log(E(o(v))) >
du B(o(y)) dudæ
for all u E [0, 1) and x E R. Changing the order of derivatives, it is sufficient to show
d (E)o'
Elu' (y)]
( u Bl((v]
>> 0.
(163)
dx E[o(v)]
The first function
Elo'(u))
$1
(164)
Bja(y)]
shown to be increasing in x in Lemma G.7 where we need substitute + zu1 for #1 and 0 hence it is increasing. The remaining part is to
for = O, we have $1 (z) = ' for some show that the second function
E Elu'zo'(u)
82(2)
(165)
Bya(y)
decreasing.
We will consider + u/= and x + ur what follows. We have
d 21200(x 2)]
d Elo (x + 2) dx B (x + =)]
> 0
da ([ (r + +)]
due to first applying Stein's 's Lemma to the numerator and then inverting the ratio. Using the chain rule, it is sufficient to show that
f1 = Bao( + 2)]
Elo'(x =) [x and f2(2)
(166)
Elo"(x (xr + +2]
are increasing, since both functions are positive.
Interestingly, fi is increasing in if is a log-ccncave function. Because its derivative is positive
d
Bfo(x + =(o
dr fx)
Ei 2]
if E[o(x + 2)] is log-concave. This is the case since centered Gaussian distribution p(2) is log- concave, therefore o(x + =))(2) is jointly log-coneave, and marginalization preserves log-concavity.
Similarly, fz is increasing since is also log-concave due to property (v) of softplus (Lemma G.8).
Hence showed that
Ey)(
r(u)
(167)
E[r(y)
increasing for [0, 1). The derivative of r explodes at however, can conclude by contradiction that r(1) > r(u) for < 1: if r(u) 2 r(1) for some 0 < u < 1, then there exists up E (u, where the function is decreasing. Therefore, is increasing for u 1]. We can conclude the first part of the proof by the FKG inequality a <((r))//o(rz) and ((o(v)))/E((((>)))) are negatively correlated.
For the second part of the statement, we need to show
(rx()rr + =(=))]][[a(rz]] < WEo (r(((r))) + u'z)]
(168)
for < 0. Changing the measure from standard Gaussian p(z)
(rxx
x(x) = p(x)
(169)
E(a())))
the above inequality is equivalent to
o' (rz)æ
o (rx)æ
a (ur + u' u'z)
< Exan
E En (ux + u/=)].
(170)
(rx)
(rx))
This holds since a' (rw)) (r/////o/o (rx) increasing in æ, however, ouurr u is decreasing since is negative which implics negative correlation due to the FKG incquality.
38

---

Lemma G.7. The following function
[R(o (a: + =)| E B + 2]
increasing in x where the inlegrations w.r.t a centered Gaussian
Proof. Since all integrals are w.r.t 2, we drop the conditioning with respect to x in the proof. Taking the derivative w.r.t 2, and arranging the terms, suffices to show
(1 + =)]x Elo'(x + +]] +1) B[o(x + Eio'(x x 2)|x
(171)
which is equivalent to the following due to the property d²(z) Bo (z) d')))
E| [1o'(x + ) Blo'(x +
Bxx
+1 E[(o(x + =)] > Blo' (x + =)|ar.
(172)
In the case 2 0, the LHS is bigger than Bfor( + 2)] since a' (-) is upper bounded by 1. Moreover, since o (x) > x and from the convexity softplus, we get B|a + 2)] > z. This yields the above inequality by again noting that Efo'(x =]] is upper bounded by 1.
In the case x < O, we need another strategy. We have thanks to Cauchy-Schward
Elo'(x z)
2 Elo' (x + +)]
(173)
E (x +)]
thus it suffices to show
(ox - 3x.Elo' (xr + 2)] + E(j + =)] > Elo'(x : + =)]
(174)
We will now show the following
Elo' o'
(175)
for which it suffices to show that == o to' =x) 2 20'(x) (x) for all z since the centered Gaussian measure p(2) is even and the integration can be done over the integrand v(2). We have
v'(z) = ox - o"(x
(176)
that is zero iff either x + =====: x or 2 x + where the latter is not possible since x < 0.
Hence get that critical point of v(z) at which is minimizer since v²(0) 2a" (x)
for x < O. Hence u(2) 2 v(0) = 2xx(x) for all = which completes the argument.
Finally, it remains to show
Bx
+ +1
Blo(x + 2)] Elo((z 2)]æ.
(177)
Bx +1
we have that (ar > dx))) which in combination with the following trivial observation for all x 0 (note that +1 is not needed for the following to hold)
From the proof of Lemma G.8
fa
+1> Br
(178)
e +1
shows that Eq. 177 holds, hence the proof is complete.
Lemma G.8. The softplus family has the following properties
(x) is increasing and convex,
ii (x) is log-concave (equivalently, (x))/((x) is increasing),
ii. (x:)/ (' is convex, iv. ( is increasing.
of (x) is log-concave.
39

---

Proof. For the property (i), see the formulas of a' and o" in the proof of Lemma F.2 We next prove each one of the properties one after the other. Let us start with property (ii). First note that (xx) is log-concave if and only if o (x)) (a))/o((a (a:) is increasing since
d ox dz ['
= (x)) >0 + d'(x)² > a(x))"(x)
(179)
d'(x)²
where the second inequality is a characterization of log-concavity. We will prove that R(x) (=))/o(xx) increasing.
==
Let us write out the ratio explicitly
R(x) B [og(es0 +1) +
log(e + +1)
(180)
e02
The first derivative of is given by
(x(x) = (x)) d'(x) - (x(
2x Ba (2)
(181)
Since log is concave, expanding it around 1 we get log + 1) < for all > 0. Substituting exxx, we get that the numerator of I'' is positive, thus R is increasing. This completes the proof of property (ii). Computing the second derivative of R. we get
Rd (x)x o" (=))(
230' (x) + (o(x)) = = 3x
(-0 + Bo(
(182)
ear
What remains to show is that (x) > (z). Using the fundamental theorem of calculus, we get
log(y + 1)
/-" 1+1
1
dt
(183)
y
y+1
since 1/(y + 1) is a lower bound of the integrand which completes the proof of the property (iii). Let us prove the property (iv) by taking the derivative of the function of interest
d d'(x)² (o" + d'))))((x) - d'x)²²x dr (ιx)
(184)
o(x)²
Using d'xx) = (z) - ox)) and dropping the positive term of (æ), the numerator of the derivative is
+ 1)0(x) - d()) = ( 3x ((1 - (
e
ci +1 (xx)
@i + x
(185)
1
(Bx + e
+ 1))(()) -x)
(186)
e +1
For the case x 2 O, have (x)) > x and (Bx 1)/e the case < O, we want to show
0, hence the derivative is positive. For
log(e Sa
+11
Gr + 3x +
> Br.
(187)
eBx
If ex + Br + 1 > O, it is done since the LHS is positive. If eee + Bx 0, we have
log(e +1)
log(e + +1)
(e + +3 1)
2 (e +,3x + 1) sup
(188)
8xx
eB2
since log + +1)/e is positive. We will next show that log(e + 1)/e is a decreasing function therefore its supremum is achieved at x -00 From the integral expression in Eq. 183 we deduce that log(y + 1)/y is decreasing function since adding smaller terms in the average decreases it. Thus the following limit gives us the supremum using L' L'Hôpital's 's rule
log(y + 1)
1
lim y
lim
1.
(189)
y
p y +1
40

---

Combining it with the Eq. 188 after the substitution ex we get the desired Ineq. 187 which implies that the derivative positive this case too. This completes the proof of property (iv). For the property (v), we first give a formula for the third derivative of softplus
(<)(x) - -1
(190)
o' is log-coneave if and only if we have
o ""({))(()<< < (z))o'((x) + (x))o(((x)
(1)
(191)
which is equivalent to
(1 - ez)) log + +) < et.
(192)
This is equivalent (1 y) log(y + log(y 1) y where y =0 0; the second inequality holds due to y + < eV.
41


Reviews:
[{"review_id": "PGZ7CeWlCf", "reviewer": "Reviewer_Cg9m", "summary": "The goal of the paper is to understand the following approximation problem: given a student network and teacher network, how should one construct the student network such that the approximation error is minimized. The paper considers the underparameterized regime where the student network has fewer parameters than the teacher network. The first half of the paper considers a constrained optimization problem which is equivalent to the original approximation problem if the student network consists of one neuron (otherwise, the constrained optimization problem is a relaxation of the original approximation problem). The second half provides a characterization of a subset of critical points for the approximation problem. Given additional assumptions on the activation function used in the networks, they also provide an analytical formula for non-trivial critical points of the approximation problem and some empirical verification of the paper's claims.", "questions": "In the context of section 5.2 multi-neuron network, is there a nice way to characterize which of the n - 1 teacher neurons will be copied other than trying every (n - 1) copy configurations and computing their approximation error?\n\nFor figure 1.A2, it is not clear which pair of student and teacher neurons are being used to compute the correlation (perhaps label the y-axis with the teacher neurons v1, ..., vk). Furthermore, I don't really see what insight the plots of r_i and a_i give (perhaps add corresponding teacher r'_i and a'_i ? )\n\nDoes lemma 5.4 hold for other activation functions? This seems to be the major workhorse for characterizing critical points for student networks for multiple neurons and thus it would be nice to address if a similar lemma holds for common activation functions.", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is mostly well written. In addition, to the best of my knowledge, the proof (in restricted settings) and the empirical observations (in more general settings) of the copy average of phenomenon seems to be novel. Furthermore, the constrained optimization problem that serves as a relaxation of the original approximation problem is interesting. In addition, the paper provides a nice family of critical points for the approximation problem that can be constructed by concatenating critical points of smaller approximation problems.", "weaknesses": "Many of the results of the paper requires that the student network consist of one neuron. While section 5.2 gives a characterization of critical points for the multi-neuron case, it requires that the activation function is erf, an uncommon activation function. In addition, the input data is assumed to be gaussian.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and appreciating the novelty of the copy-average phenomenon. Below, we answer every question raised by the reviewer. We appreciate follow-up questions, if anything is unclear in the responses or if the reviewer has new questions.\n\n**Which of the $n-1$ teacher neurons will be copied?** In the teacher model used in the paper and in the literature, all outgoing weights are one and the incoming vectors are orthogonal (i.e. unit-orthonormal teacher). Because of this symmetric arrangement, all copy-average optima, where one of the $n-1$ teacher neurons is copied, have the same loss. It is therefore impossible to predict which teacher neurons will be copied. \nThis symmetry is broken, if the outgoing weights are arbitrary or the incoming vectors are not orthonormal. In this case, copying one teacher neuron may have a lower loss than copying other teacher neurons. Although this is beyond the scope of this paper, one can probably use the tools developed here to tackle this difficult question. First, one needs to compute the approximation error of the one-neuron network. This requires solving Eq. 15, where the first summand would have an additional $b_j$ factor. $u_j$’s will no longer be equal but they will solve a similar analytic equation. Once $L^*_\\text{erf}(b_1, …, b_k)$ is obtained, we need a more general version of Lemma 5.4. This is an interesting question! \n\n**(Regarding Figure 1) Furthermore, I don't really see what insight…:** See the common response above for Figure 1. The teacher is unit-orthonormal, i.e. $a_i’=1, r_i’=1$, we added it in the caption. For erf (top row), $a_i$ and $r_i$ exactly match the theory ( bars-empirics and dashed lines-theory overlap). For ReLU (bottom row), we think it is interesting to see (1) $r_i$ and $a_i$ are very close to each other (aka balancedness condition, though we did not force balancedness at initialization by small initialization or brute force) (2) all student neurons have similar `magnitudes’, that is all $r_i$ and $a_i$ are also very close to each other. We think these observations will be helpful in the future for completing the theory for ReLU. \n\n**Does Lemma 5.4 hold for other activation functions?** Lemma 5.4 follows from the concavity of the optimal one-neuron loss, that is $L^*_\\text{erf}(k)$. It is possible to compute it for other activation functions. For odd monomial activation functions $\\sigma(x)=x^b$ (b odd), the interaction function $g$ admits an analytic formula. Following the same recipe, one can simply check whether $L^*_b(k)$ is concave. However, this class of activation functions is uncommon in practice. If the reviewer thinks this setting is interesting/helpful/useful, we are happy to include it in our paper. We already have the optimal one-neuron loss for ReLU, Eq. 116, page 28 in Appendix, however, ReLU is not an odd activation function (i.e. Theorem 4.2 does not hold).\nMore in detail, for ReLU, the copy-average phenomenon is tricky because (1) the concatenation of critical points is not a new critical point (see Eq 44 in the Appendix), and (2) the off-diagonal correlations are non-zero (see Figure 1). A challenge here is computing the off-diagonal perturbations to the copy-average configurations. \n\n**Erf is uncommon…:** We see how the assumption on the activation function might feel rather limiting. However, we believe, the distillation of this phenomenon into the analytically tractable setting of erf, is a success. We show empirically that the copy-average phenomenon kicks in at least for ReLU (see Appendix B.3, Figure 9) and expect that it is a universal phenomenon. This universality hypothesis is very interesting! It seems that the relevant scaling is when both n and k go to infinity with a linear ratio, however, the theoretical analysis of this setting likely needs a combination of tools from random matrix theory and dynamical systems. We aim to study this challenging but intriguing problem in the future. \n\nMoreover, observe that for ReLU (Figure 9 in Appendix B.3), when both n and k are large, the off-diagonal correlations are small as indicated by lighter colors. This observation supports universality in the limit when both n and k are large which in turn implies that the analysis of erf is in fact generic. Finally, we invite the reviewer to see further literature review in Appendix A, Table 1. All these influential works considered Gaussian data and fixed activation functions to be either erf or ReLU because both cases require distinct analyses. We are able to develop a unifying framework via interaction functions (see Assumptions 3.1) for the analysis of the one-neuron network (see the discussion in Appendix A). \n\n**Input data is standard Gaussian:** see the common response above.\n"}, {"review_id": "2GSIs2my10", "reviewer": "Reviewer_6p1W", "summary": "his paper proposes a constrained optimization approach to learning\nstudent networks $g(x) = \\sum_{j=1}^n a_j \\sigma (v_j \\cdot x)$ that\nhave fewer neurons than a two-layer teacher network $f^* (x) =\n\\sum_{i=k} b_i \\sigma( w_i \\cdot x )$. (with $ n \\le k $ as the student network has\nfewer neurons than the teacher )\n\nStarting with the squared-loss between the student and teacher, the authors\npropose a relaxed constrained optimization problem in terms of the student parameter norms\nand correlations to themselves and the teacher parameters.\nThe authors show that in the case\nof general $b$ and odd-activations, for any $n$-partition of the $k$\nteacher neurons, if $(a_j^* , v_j^* )$ are a critical point for a\none-neuron student network, then $v_j^*$ lies in the span of the\nteacher weights $w_j$ corresponding to the partition. They further\nshow that for a multi-neuron student network, concatenating the\ncritical points from one-neuron student networks will produce a\ncritical point for the overall multi-neuron network problem.\n\nThen the authors jump to unit-norm teacher networks, where $b_i = 1$,\nwhere the critical points are unique, and closed-form solutions exist\nfor the erf and ReLU activations.", "questions": "See points 1, 2, 3 in the weaknesses section above. Overall, I like this paper, but I feel that the presentation makes it hard to parse the results and some statements seem contradictory.", "limitations": "Could be discussed in more detail. It's merged with the conclusions section as a monolithic paragraph. The gap between the relaxation and original problem are not discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The stated problem is interesting and I find the results compelling.\n\n- There are several non-trivial aspects to the paper. The authors\n  first convert the student-teacher minimization problem to a relaxed\n  version where they try to optimize variables corresponding to: (a)\n  student parameter norms, (b) correlations between student vectors,\n  and (c) correlations between student and teacher vectors. As the\n  input distribution is Gaussian, the pairwise correlations can be\n  analyzed for a wide class of activation functions.  While some of\n  the techniques are standard (like odd activation functions to\n  decouple neurons in a multi-neuron network), they are compiled well.\n\n- The analysis is fairly slick and the assumptions are reasonable.", "weaknesses": "- The constrained optimization is a relaxation of the original\n  problem of learning a student network, and the critical points of\n  the relaxed problem are analyzed. There is no discussion about the\n  gap between the original and relaxed problem. There is some\n  discussion about small-scale and medium scale problems in the\n  conclusion -- are these related to the gap in the relaxation?\n\n- The teacher network has orthogonal weights, and hence is in a regime\n  where the number of neurons is smaller than the input dimension. I'm\n  not familiar with the literature on learning student networks -- is\n  this standard?\n\n- The authors say that \"for larger n that is close to k, we observe a\n  transient phase where the gradient flow finds a perturbed copy\n  solution that has a lower loss than the optimal CA point.\" How is\n  this possible if your program is a relaxation of the gradient flow\n  objective, and why does it not contradict your claim that \"For\n  shallow neural networks with odd activation function, we prove that\n  “copy-average” configurations are critical points, and the optimum\n  is reached when $n-1$ student neurons with erf activation each copy\n  one teacher neuron and the n-th student neuron averages the\n  remaining $k -n +1$ teacher neurons\"?  I've complained about this elsewhere, but \n  having a Theorem statement for section 5.2 would help alleviate this concern.\n  Also, please maintain consistency between erf and odd-activations, as the sentence I \n  quoted switches between them.\n\n- The writing can be slightly confusing from time to time because of a\n  lot of context switching between one-neuron student networks and\n  multi-neuron odd activation function networks, and then to unit-norm\n  teacher networks, where the outgoing weights are $b_i = 1$.\n\n- Figure 1 is very hard to parse. How should a reader understand that\n  the results in Figure A2 implies convergence to a copy-average of\n  the true solution? And why are the blocks for the\n  pairwise-correlation $u_1$ and $u_2$ weird rectangular blocks?\n  Wouldn't the pairwise correlations just be $(1, \\rho, \\rho, 1)$?\n  Overall, A2, A3, A4 are supposed to explain different settings for\n  $n, k$, but in their current form, they are all equally\n  incomprehensible. The caption is also half a page. There's too much\n  going on, it needs to be simplified.\n\n- Why is there no theorem statement for section 5.2? It seems like a\n  strange choice to not include it given that it's your main result.\n  The lemma currently stated is for a two-neuron network.\n\n- Minor:\n\n   - The first line of the abstract and introduction says that any\n     continuous function can be approximated arbitrarily well given\n     enough neurons in the teacher network, which is true. However,\n     all the results in this paper can only hold when the number of\n     neurons ($k$) is atmost the dimension of the input ($d$), which\n     strongly contradicts the first statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback. Thanks for liking the results too! Below, we aim to address each one of the concerns. We are confident that the writing can be improved with minor updates on the text, following the suggestions of the reviewers. This would be a minor change overall since the mathematics is stable (i.e. nothing is contradictory). For the discussion and the conclusion parts, we will structure them better using the extra page. \n\n**The gap between the constrained optimization and the original problem:** First, the constrained optim. problem is in fact exactly equivalent to the original for $n=1$ and $n=2$. Because the mapping from the weight space to the order-parameter space is invertible in this case. We exploit this to prove that the global optimum is an ‘average’ neuron for $n=1$. However, for $n \\geq 3$, due to the geometry of the problem, the mapping is no longer invertible. In particular, we gave an example configuration in the order parameter space that is not in the image of this map in Appendix C.3 (on page 18). Overall, at the moment, we are exploiting the constraints for pairwise angles by Cauchy-Schwartz, however, it may be possible to tighten these constraints (hence closing the gap between the two optimization problems) by exploiting the geometrical constraints of angle constellations between triplets or quadruples of vectors. We will add this discussion on the gap between the two problems in the paper.  \n\n**Regarding the perturbed-n-copy solution that has a lower loss than the optimal CA point:** Let us break down the sentence that the reviewer has quoted “For a shallow network with odd activation…”. Here we are referring to the ‘copy-average optimum’. More precisely, the statement is: among the copy-average configurations, for a unit-orthonormal teacher, for the erf activation function, the optimum (copy-average) configuration is an $(n-1)$-copy-$1$-average point. This is our main result. The question is then: is the CA optimum also the global optimum? Empirically, gradient flow converges to the CA-optimum for $n \\ll k$ and we conjecture that the answer is positive in this regime. However, for $n \\gtrsim 0.8k$, gradient flow converges to another critical point with lower loss, which we call perturbed-n-copy. This is an intriguing phenomenon: the system seems to exhibit abrupt changes in behavior (green-grey-blue-red regimes) as we increase n. Taken together, proving the exact parameters of the global optimum of under-parameterized shallow networks might be a very challenging optimization problem and it is beyond the scope of this paper. \n\n**Orthogonal teacher/Relevant literature:** See the common response above. \n\n**The writing can be slightly confusing:** In Section 3, we present a constrained optimization formulation of the classic teacher-student problem. In Section 4, we focus on odd activation functions and orthogonal teachers and prove that concatenating critical points of the smaller networks yields new critical points. In Section 5, we focus on the unit-orthonormal teacher and we have two subsections. In Section 5.1, we have some properties of the exact parameters of the global optimum for $n=1$ for general activation functions, and a corollary (5.2) where we give the exact parameters for erf. In Section 5.2, we focus on the erf activation function and prove that the optimum CA-point is an $(n-1)$-copy-$1$-average point. To help the reader, we will add an overview with a bullet point list of contributions that highlights the assumptions of the main results.\n\n**Why is there no Theorem in Section 5.2?** We wanted to present the proof in a constructive way and that’s why we put Lemma 5.4 in the main. This constitutes the big chunk of the proof (of the statement that the optimum CA-point is an $(n-1)$-copy-$1$-average) and the remaining argument is written in lines 240-246. We understand how this might be confusing. Therefore, we will instead include the main result as a theorem here as suggested by the reviewer, and send the Lemma to the Appendix for the interested reader. \n\n**Regarding Figure 1:** See the common response above. In addition, in A2-top (for erf), we have $u_1=(1, 0, 0)$ and $u_2=(0, 1/\\sqrt{2}, 1/\\sqrt{2})$ which shows that (1) the two student vectors are orthogonal to each other and (2) the first neuron is a copy-neuron and the second neuron is an average neuron. For ReLU (A2-bottom), there is no perfect decoupling (see the non-zero off-diagonals), however, the correlation matrix is similar to erf. We removed B to simplify it and made the caption shorter. Please see the updated figure in the pdf above. Is it understandable now? If not, we’d greatly appreciate follow-up feedback. We’d like to make it easily digestible for the general ML community.\n \n**Minor Comment:** Thank you for raising this point. We will update the abstract and the first sentence of the introduction accordingly and remove the reference to the universal approximation theorem. \n"}, {"review_id": "T3vxVqDWvk", "reviewer": "Reviewer_ThaK", "summary": "The paper studies a shallow teacher-student setting in which a student network being trained attempts to recover the orthogonal weights of a teacher network, assuming normally distributed data. Specifically, the setting being studied is where the number of student neurons $n$ is far smaller than that of the teacher, $k$. An analysis of the optimization landscape is given for the case of a single student neuron, and assuming an odd activation function, it is shown that among the family of critical that are combinations of averages of the teacher's neurons (CA points), the optimal such point is the one where $n-1$ student neurons copy $n-1$ teacher neurons, and the last student neuron averages the remaining teacher neurons.\n\nWhile I'm overall impressed by the technical results in the paper and how well it is written, I find the results a bit limited. On the other hand, this paper studies a technical and challenging problem, and with this in mind, I do believe its merits outweigh its drawbacks.\n\n\nPost-rebuttal:\n\nI have decided to increase my score following the rebuttal and my discussion with the authors. My overall opinion of the paper remains positive and I'm leaning more towards acceptance.", "questions": "- Throughout the paper, in the experimental parts, it is mentioned that gradient flow converges to certain CA or close to CA points. How can you simulate gradient flow? Did you happen to use a very small step size gradient descent? If so please clarify this, since this doesn't imply that gradient flow will necessarily exhibit the same behavior.\n\n\n\nMinor comment:\n\nLine 266: \"span of teacher's\" -> \"span of the teacher's\"", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well structured and written.\n\n- The problem studied is very technical and making interesting observations on it is challenging.\n\n- The technical idea behind the paper (reformulation as a constrained optimization problem) seems interesting and innovative.", "weaknesses": "- The results provided in the paper only partially solve the problem it addresses. For example, the optimality of the copy $n-1$ and average $k-n+1$ teacher neurons point is only established within the family of CA points, and it's possible that there are other non CA points which achieve an even better loss. While a more thorough understanding of the single neuron case is provided, it is of course still limited since it's just a single neuron.\n\n- The analysis only applies to the under parameterized regime where potentially the class of student networks is not even capable of expressing a remotely useful solution. It is far more interesting to study the regime where there are local minima that can achieve small (but not necessarily zero) loss, since this regime might have stronger practical implications. Moreover, the orthogonality assumption which implies $k\\leq d$ further restricts this setting to $n\\ll k\\leq d$, which needless to say does not capture a setting similar to how neural networks are trained in practice in terms of the magnitudes of these quantities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the actionable feedback. Thanks also for acknowledging the fundamental difficulty of the teacher-student problem. Below, we respond to each feedback. Some of the results cover the full under-parameterization regime (n<k) which is much more general than what is understood by the reviewer ($n \\ll k$). We appreciate follow-up questions, if there is anything unclear or any new questions arise.\n \n**I find the results a bit limited:** The unit-orthonormal teacher and standard Gaussian input data are common assumptions (see the common response above). Whereas the ‘realizable’ case when $n=k$ is well-studied for example by Saad&Solla 1995 and Safran&Shamir 2018, the under-parameterized regime $n<k$ is not studied so far. Importantly, the optimal CA-point is a $(n-1)$-copy-$1$-average critical point in the full under-parameterized regime (for any $n<k$).\n \nWhat is special about the $n \\ll k$ regime, is the empirical observation that gradient flow converges to the optimal CA critical point (see Figure 4). However, this is not the end of the story as we show in Figure 4: gradient flow converges to another critical point with lower loss in the red regime when n is close to k. Based on this, we conjecture that the global optimum is in fact the CA-optimum when $n \\ll k$ and propose a constrained optimization formulation. We believe the different regimes found here are intriguing and one may need to develop new proofs to find the global optimum in the full under-parameterization regime. **The results only partially solve the problem…** The system shows an abrupt change in behavior (akin to a phase transition) at about $n \\sim 0.8 k$ (red regime in Figure 4): the gradient flow converges to the perturbed-$n$-copy configuration for all $20$ seeds. Such systems are known to be challenging, for example, for the Ising model, the mathematical development of even the ‘simple’ models has taken 100 years. No one knows at this point how difficult it is to complete the theory of shallow teacher-student learning.\n\n**Why the problems with and without local minimum are *both* interesting:** While we agree with the reviewer that problems with various local minima show richer phenomena, so far a general mathematical treatment of such problems was not possible for neural networks. The problems without local minima are quite interesting as well as we argue next. *Problems without local minimum:* Let us consider linear networks as a case study. Without explicit regularization, it is well known that there is no local minimum [S1]. However, the training dynamics is non-trivial and this is still an active field of research [S2,S3,S4,S5,S6]. *Problems with local minima:* The empirics show that there is a regime of underparameterization (see Figure 4-B2, gray regime) where the gradient flow converges to either $(n-1)$-copy-$1$-average minimum or (ii) perturbed-$n$-copy minimum. \n\n**...further restricts this setting to $ n \\ll k \\leq d $:** To put it simply, we hypothesize that this is the simplest but generic problem in shallow learning and it is an essential step towards more realistic modes. Let us break down this hypothesis.  \n\n*(1) How to drop the orthogonality assumption?* See Remark C.1, line 475 in the Appendix. In short, the constrained optimization trick applies to non-orthogonal teacher networks as well, up to a change in the constraints by writing them on the basis of the teacher’s incoming vectors. However, one needs linear independence which still imposes k≤d. The analysis of linearly dependent teacher incoming vectors may invoke yet additional constraints in the formulation. \n\n*(2) Are the results restricted to n≪k?* Once again, no. The optimal copy-average point is $(n-1)$-copy-$1$-average for all $n<k$ due to Lemma 5.4 (and the paragraph after). However, as written above and in Figure 4, there are other minima found by gradient flow when $n$ is slightly bigger than $0.5 k$. Hence, the conjecture that the global optimum is the CA-optimum is stated for the limited case $n \\ll k$ and proven for $n=1$. With more effort and clever methods, we believe the constrained optimization problem can be solved for $n=2$ where it is exact (it is a relaxation starting from $n=3$, see Appendix C.3).  \n\n**How did we simulate gradient flow?** We used the MLPGradientFlow package implemented by [6] (https://github.com/jbrea/MLPGradientFlow.jl). We used the default option in the package that calls the KenCarp58 ordinary differential equation (ODE) solver from Julia, which is different from implementing gradient descent with a small step size. The package features analytic formulas of the integrals for erf and ReLU for accurate computations of gradients and Hessians. Training is done for 1e5 ode iterations. For erf, all seeds converged to configurations with gradient norm below 1e-10 (see Appendix B lines 416-428, and [6] for more details).\n\n[S1] Baldi, Pierre, and Kurt Hornik. \"Neural networks and principal component analysis: Learning from examples without local minima.\" Neural networks 2.1 (1989): 53-58.\n[S2] Li, Zhiyuan, Yuping Luo, and Kaifeng Lyu. \"Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning.\" arXiv preprint arXiv:2012.09839 (2020).\n[S3] Jacot, Arthur, et al. \"Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity.\" arXiv preprint arXiv:2106.15933 (2021).\n[S4] Pesme, Scott, Loucas Pillaud-Vivien, and Nicolas Flammarion. \"Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity.\" Advances in Neural Information Processing Systems 34 (2021): 29218-29230.\n[S5] Even, Mathieu, et al. \"(S) GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability.\" arXiv preprint arXiv:2302.08982 (2023).\n[S6] Pesme, Scott, and Nicolas Flammarion. \"Saddle-to-Saddle Dynamics in Diagonal Linear Networks.\" arXiv preprint arXiv:2304.00488 (2023).\n"}, {"review_id": "3sYjGguA5R", "reviewer": "Reviewer_Zuqd", "summary": "This paper is a theoretical study of whether an under-parameterized student network should copy or average teacher weights, there teacher is a one-hidden-layer network with k hidden units and student is the same but with n<k hidden units. The paper derived the optimal “copy-average” configuration under a set of conditions, and also provided a closed-form solution for the case of n=1. ", "questions": "Why are the theoretical results presented important when there are so many unrealistic assumptions? \n\nWhy would one “expect that the incoming and outgoing weights of a student neuron are either identical to those of a 29 teacher neuron or that they are closely aligned with the weights of a group of teacher neurons”? What’s the rationale of this expectation?\n\nCould you elaborate how “the loss function in Eq. 5 can be expanded as a weighted sum of 109 the following Gaussian integral terms”, and further expressing “the integral in terms of the covariance of the two-dimensional Gaussian” \n", "limitations": "practical significance of the results", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "derived some theoretical results in terms of whether smaller two-layer student network should copy or average the weight of a larger two-layer teacher network under certain conditions\n", "weaknesses": "The conditions for the results to hold seems to be very strict, far from practical. I struggle to understand the practical significance of this theory\n\nintuition is not well explained \n\nmathematical derivation is presented in a straightforward way\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Below, we address each point. We hope that the reviewer will find the results more intuitive. From a bird-eye view, our main contribution is the copy-average phenomenon that kicks in for underparameterized networks. On practical relevance, the results suggest a novel and applicable method to distill a large (teacher) neural network into a smaller (student) network for budget-limited scenarios which we discuss below. \n\n**Why is the under-parametized regime interesting in practice?** While it is true that the theory of deep learning community has focused on overparameterized regime in recent years, we do not think that this is the only regime of interest. Two reasons: (1) the global optimum of the under-parameterized networks creates symmetry-induced critical points in overparameterized networks [8], shaping their loss landscape and learning dynamics, (2) how to optimally compress a large (teacher) network into a smaller (student) network is an open question and we give a recipe to approach this problem. While it is true that our theorems hold for a limited setting as usual for a theory paper, one may expect that the copy-average phenomenon holds more generally. For distillation, it is advisable to initialize the (student) network by copying $n-1$ teacher neurons & averaging the remaining teacher neurons, and then training it further. The copy-average insight might be the key to finding the optimal solution and help improve the distillation methods. We will add a discussion on distillation in the conclusion. \n\n**Intuition is not well explained:** We think this comment is unwarranted. Recall the main theorem first: in the under-parameterized regime with $n \\ll k$, the loss landscape exhibits various non-global critical points, and the global optimum is achieved when $n-1$ student neurons copy $n-1$ teacher neurons each and the last student neuron implements an average of the remaining teacher neurons. We prove this by using the analytic expression of the optimal loss of one-neuron network denoted by $L^*_\\text{erf}(k)$ (Eq. 16). In particular, because $L^*_\\text{erf}(k)$ is (discrete) concave, we get Lemma 5.4 which yields the copy-average optimality of the $(n-1)$-copy-$1$-average configuration. This is all to say that this result is non-trivial and comes from algebraic properties. We even take a further step to say that our results are not necessarily intuitive, but this is not negative by any means. New results that are unintuitive at the time they are published in fact might push the boundaries of fundamental sciences. Figure 1 is simplified as well (see the common response to above).\n\n**Assumptions are far from practice:**\n* *Input data is standard Gaussian:* See the common response above.\n* *The teacher network is orthonormal:* See the common response above.\n\n**...the rationale for either copy or average groups of teacher neurons:** The optimal solution of the one-neuron (student) network that approximates a multi-neuron teacher network is to ‘average’ all teacher neurons as we prove in Section 5.1. Once this is known, one might expect that the student neurons either exactly –copy– teacher neurons, or –average– groups of teacher neurons as done by the optimal one-neuron network. Even if the reader did not have this expectation, we suggest forming such an expectation early on to warm up the reader to the results of the paper. \n\n**Weighted sum of 109 via the Gaussian integral term/expressing it in terms of covariance:** Let us break this down. Eq 5. can be expanded as a sum of squares (since it is a square of a sum of neurons) in terms of summands \n\n * $a_1 a_2 \\mathbb{E}[\\sigma(w_1 x) \\sigma(w_2 x)] $ (student-student term)\n * $a_1 b_2 \\mathbb{E}[\\sigma(w_1 x) \\sigma(v_2 x)] $ (student-teacher term)\n* $ b_1 b_2 \\mathbb{E}[\\sigma(v_1 x) \\sigma(v_2 x)] $ (teacher-teacher term).\n\nOne such term is written in Eq. 109 and $w_1$ and $w_2$ denote arbitrary vectors here. We understand it might be confusing here as $w_1$ and $w_2$ denote also student incoming vectors, hence we will write $V_1$ and $V_2$ instead in the revised version for two arbitrary vectors that represent (a) $(w_1, w_2)$, or (b) $(w_1, v_2)$, or (c) $(v_1, v_2)$.\n \nThe Gaussian integral term of Eq 109 is of the form $\\int f(x, y) p(x, y) dx dy$ where $p$ is a two-dimensional Gaussian density function. One can always reparameterize this integration after integrating out $x$ and $y$ in terms of the mean and covariance of the two-dimensional Gaussian (equivalently, the parameters of the density function $p$). Therefore, it is possible to express it in terms of the mean (which is zero) and $2 \\times 2$ covariance (shown in Eq in line 111). We are using a specific parameterization of the covariance in terms of $r_1$ ($\\ell_2$-norm of $w_1$), $r_2$ ($\\ell_2$-norm of $w_2$), and $u$ (normalized dot product of $w_1$ and $w_2$). This yields the so-called interaction function given in Eq 7. We’ll update the text here by changing to slack variables and adding explanations on expressing the integral with covariance parameters. \n"}]

Author rebuttal:
**Regarding Figure 1 (response to ```Cg9m```  and ``` 6p1W``` ):** $u_{ij}$ is the vector of correlations between the student incoming vector $w_i$ and the teacher incoming vectors $v_j$, i.e. $u_i = (u_{i1}, …, u_{ik})$. This notation is introduced in line 138 and added in the new caption (see the pdf attached). The last row of the correlation matrix (rectangle) in A2 was gray (in fact, effectively zero). This shows that $w_i$ is in the span of the teacher's incoming vectors $v_1, …, v_k$. We realized too much was going on in this figure. Hence we removed panel B and the last gray row and also simplified the caption (see the pdf attached for the updated Figure 1).

**On the practical relevance of the Gaussian input data (response to ```Cg9m```  and ``` Zuqd```):** For general data distributions, there is no theory framework to do a fine-grained analysis of neural network learning apart from special cases such as the infinite-width limit where the neural tangent kernel or mean-field analysis applies. This is a commonly used assumption (see [1,13,17,20,22,23,25] cited in the paper) and it is natural in high dimensions due to universality (that is, a generic data distribution can be replaced by a Gaussian distribution without affecting measurable quantities of interest) [S1, S2, S3, S4].

**Orthogonal teacher/Relevant literature (response to ```6p1W```, ```ThaK```, and ```Zuqd``` ):** This assumption is standard in the theoretical analysis of the teacher-student setup and we refer the reviewers to Table 1 (Appendix A, page 12). Two options: either the input dimension approaches infinity (in statistical mechanics or in high-dimensional probability approaches) [2,13,20] or the teacher network is assumed to be unit-orthonormal [1,22,23]. Then the incoming vectors of the teacher are either orthogonal or ‘approximately’ orthogonal due to the fact that random vectors are effectively orthogonal in high dimensions (i.e. $d \to \infty$). 

Following the literature, we started by analyzing the unit-orthonormal teacher network that yields a very symmetric problem. Different from the literature, our approach is generalizable to general teachers (we can choose arbitrary $b_i$ as done in Proposition 4.1) and it is possible to write the constrained optimization formulation for non-orthogonal incoming vectors (see Remark C.1, line 475 in Appendix). Overall, (1) unit-orthonormal teacher is a commonly studied data-generating model and shows intriguing and non-trivial properties (see Figure 4) (2) the constrained optimization formulation is generalizable to more general teachers, however, the measurable properties of the system will likely be more complicated. 

[S1] El Karoui, Noureddine. "The spectrum of kernel random matrices." (2010): 1-50.

[S2] Hu, Hong, and Yue M. Lu. "Universality laws for high-dimensional learning with random features." IEEE Transactions on Information Theory 69.3 (2022): 1932-1964.

[S3] Montanari, Andrea, and Basil N. Saeed. "Universality of empirical risk minimization." Conference on Learning Theory. PMLR, 2022.

[S4] Dandi, Yatin, et al. "Universality laws for Gaussian mixtures in generalized linear models." arXiv preprint arXiv:2302.08933 (2023).

Meta-review: This paper studies theoretical and empirically studies network training in the setting where labels come from a teacher of higher width; theoretically it is shown that for an orthogonal teacher and gaussian data, weights do a copy-average procedure, and this is verified empirically in more general settings. Reviewers are overall positive, with some concerns about the strength of the assumptions; we urge the authors to use the new camera ready space to polish their writing and help convey their story.
Decision: Accept (poster)