Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron
Authors: Christian Schmid
James M Murray
Venue: NeurIPS 2024 poster (2024)
Keywords: Learning Dynamics
non-linear perceptron
supervised learning
reinforcement learning
Primary area: learning_theory
TL;DR: We derive learning dynamics for a non-linear perceptron performing a binary Gaussian classification task.

Abstract: The ability of a brain or a neural network to efficiently learn depends crucially on both the task structure and the learning rule.
Previous works have analyzed the dynamical equations describing learning in the relatively simplified context of the perceptron under assumptions of a student-teacher framework or a linearized output. 
While these assumptions have facilitated theoretical understanding, they have precluded a detailed understanding of the roles of the nonlinearity and input-data distribution in determining the learning dynamics, limiting the applicability of the theories to real biological or artificial neural networks.
Here, we use a stochastic-process approach to derive flow equations describing learning, applying this framework to the case of a nonlinear perceptron performing binary classification. 
We characterize the effects of the learning rule (supervised or reinforcement learning, SL/RL) and input-data distribution on the perceptron's learning curve and the forgetting curve as subsequent tasks are learned.
In particular, we find that the input-data noise differently affects the learning speed under SL vs. RL, as well as determines how quickly learning of a task is overwritten by subsequent learning. Additionally, we verify our approach with real data using the MNIST dataset.
This approach points a way toward analyzing learning dynamics for more-complex circuit architectures.

Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron
Christian Schmid Institute of Neuroscience University of Oregon cschmid9@uureegnn edu
James M. Murray Institute of Neuroscience University of Oregon mmuraay@uurrggo.. edu
Abstract
The ability of a brain or a neural network efficiently learn depends crucially on both the task structure and the learning rule. Previous works have analyzed the dynamical equations describing learning in the relatively simplified context of the perceptron under assumptions of a student-teacher framework linearized output. While these assumptions have facilitated theoretical understanding, they have precluded a detailed understanding of the roles of the nonlinearity and input- data distribution determining the learning dynamics, limiting the applicability of the theories to real biological or artificial neural networks. Here, we use stochastic-proces: approach to derive flow equations describing learning, applying this framework to the case of a nonlinear perceptron performing binary classifica- tion. We characterize the effects of the learning rule (supervised or reinforcement learning. SL/RL) and input-data distribution on the perceptron's 's learning curve and the forgetting curve as subsequent tasks are learned. In particular, we find that the input-data noise differently affects the learning speed under SL vs. RL, well determines how quickly learning of a task is overwritten by subsequent learning. Additionally, we verify our approach with real data using the MNIST dataset. This approach points a way toward analyzing learning dynamics for more-complex circuit architectures.
Introduction
Learning, which is typically implemented in both biological and artificial neural networks with iterative update rules that are noisy due to the noisiness of input data and possibly of the update rule itself, is characterized by stochastic dynamics. Understanding these dynamics and how they are affected by task structure, learning rule, and neural-circuit architecture is an important goal for designing efficient artificial neural networks (ANNs), well for gaining insight into the means by which the brain's 's neural circuits implement learning.
As step toward developing full mathematical characterization of the dynamics of learning for multilayer ANNs solving complex tasks, recent work has made progress by making simplifying assumptions about the task structure and/or neural-cireuit architecture. One fruitful approach has been to study learning dynamics in what is perhaps the simplest non-trivial ANN architecture: the individual perceptron. Even with this simplification, however, fully characterizing the mathematics of learning has been challenging for complex tasks, and further simplifications have been required.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A
B
C j
P Df(,0) 0)
W
p(xly
+1) I4
(w) p(w,t)
y
X
pxx|y- = 1)
Figure 1: Learning dynamics in the nonlinear perceptron. A: The perceptron, parametrized by weights w, maps an input x to the output ý. B: The inputs are drawn from two multivariate normal distributions with labels +1. The weight vector w is orthogonal to the classification boundary. C: Due to the stochasticity inherent in the update equations, the weights are described by the flow of a probability distribution in weight space.
One approach has been to analyze learning in the student-teacher framework [Gardner and Derrida 1989 Seung et al. [992]. in which student perceptron learns to produce an input-to-outpee mapping that approximates that of a teacher perceptron. This has led to insights about the differences in
learning dynamics between different types of learning rules (e.g., supervised and reinforcement learning) [Werfel et al. 2003 Züge et al. 2023 Patel et al. 2023). Such insights are highly relevant for neuroscience, where longstanding goal has been to infer the learning mechanisms that are used
in the brain [Lim et al.] 20155] Nayebi et al. 2020 Portes et al. 2022 Humphreys et al. 2022 Mehta
ct al. 2023 Payeur et al. [023]] However, by construction, the student-teacher setup in the perceptron only applies to input-output mappings that are linearly separable, which is seldom the case in practice, Another approach has been study learning dynamics in the linearized perceptron (Wertel et al.)
2003 Mignacco et al. 2020 Bordelon and Pehlevan [022]. which enables exact solutions even for structured input data distributions that are not linearly separable. However, the dynamics of learning in nonlinear neural networks- even very simple ones perfforinn classification tasks are not fully understood. Further, whether and how the dynamics of learning might differ under different learning rules in such settings has not been investigated.
Here, we take secchastic-prceess approach (similar to Yaida 2018 and Muuray and Escola (0200)
to derive flow equations describing learning in the finiteddimensional nonlinear perceptron trained in binary classification task (Fig. D]. These results are compared for two different online learning rules: supervised learning (SL, which corresponds to logistic regression) and reinforcement learning (RL). We characterize the effects of the input-data distribution on the learning curve, finding that, for SL but not for RL. noise along the coding direction slows down learning, while noise orthogonal to the coding direction speeds up learning. In addition, we verify our approach by training a nonlinear perceptron on the MNIST dataset. Finally, applying the approach to continual learning, we quantify how the input noise and learning rule affect the rate at which old classifications are forgotten as new ones are learned. Together, these results establish the validity of the approach in a simplified context and provide a path toward analyzing learning dynamics for more-complex tasks and architectures.
Stochastic-process approach for describing weight evolution
We consider a general iterative update rule of the form
w
1+5t 5t
! nf:((w),,
(I)
where w' E R for arbitrary n >> O, and n is the learning rate. The stochastic update term fr on the right-hand side is drawn from a probability distribution-itt depends on the weights themselves, as well as the input to the network and, potentially, output noise. Starting from this update equation, our goal is to derive an expression characterizing the evolution the probability distribution of the weights, p(w,t) t) (cf. Fig. IC.. We assume that fw( does not explicitly depend on 7), and that all the moments (J()),, k- = 1, 1, where (-),. denotes average over the noise in the update equation ①T (including the input distribution as well as, potentially, output noise), exist as smooth functions of
2

---

Given the stochastic process defined by I the probability distribution at time + öt given the distribution at time
pw, t öt) = dw p(w, ót ttp(ww' .t).
(2)
Denoting the weight update dw
w', the integrand in this equation can be written
p(w, + öt bt/w ())(w', t) p( w + 8w 6w, + 6t|w- gw, t)p(w - ów,t).
(3)
Changing the integration variable ow and performing Taylor expansion in ów, the right-hand side of (2) yields
I dw' plppp(,, + 5t/ww, ()((w',))
① (w:(w)((w,)))
0²
(4)
Ow; p(w,t) - E
[8: [B.(w))(w,,) 0) O(ów"),
Ow,Owy
where
as(w)= = / ddwwdmpp(w + 8w, St/w,1 t)
(5)
and
ff dów + ów, 6t)w.t).
Biy Bsy(w) (w)
(6)
Here, we assumed that the probability distribution describing the weight updates f has bounded derivatives with respect to
Although 2) only defined for discrete time steps, we assume continuous probability density p(w, interpolates between the updates and exists smooth function for all values of L. We can then expand the left-hand side to obtain
p(w, t 8t) p(w,t) + 5t 2 at p(,, O
For the iterative update rules that will consider, order to take a continuous-time limit, we let 17 == ôt and take the limit öt - For the general learning rule (I), the coefficients in (4) have the form
have 5w n, where is learning rate. In
0e (w) (fe) Bi((w) (fifs),,
where ( denotes an average over the noise in the update equation ! (including the input distribu- tion well as, potentially, output noise). Thus, we find
Op 01
(w, t) = 11 Σ Ow; (p(w, ()
Finding the p(w, t) that solves this equation cannot in general done exactly when f; is nonlinear. However, by multiplying (9) with powers of w and integrating, well expanding in (w). where (-) denotes the average with respect to p(w, l). we can derive a system of equations for the moments of p(w, [Risken| 1996). As derive in the appendix, this gives the following expressions for the first two moments up to O(ww (w))3):
dt (mx) = 1 2
Σ
Cov(we, u1)0k0) (())((W)),
(10)
kil
d Cov(ws, wz)
Σ [Cov(w,, wN (O) Cov(u,, wx) ((w))
(11)
Together, these equations characterize the flow of p(w, 1) for a general iterative learning algorithm in general ANN architecture.

---

A
SL
15.0
12.5
0.9
10.0
[WN[[w]!
[(w]
7.5
0.7
E
0.1 (theory)
SL, RL
0.6
(theory)
2.5
0.1 (sim.)
SL
(sim.)
2.5
0.0
-2
0
2
0.4
10
20
30
40
B
14 (w)
D
RL
RL
15.0
1.0
1.0
[[w)/[([]]
0.8
[(w]|
10 1.0
7.5
0.7
5.0
0= 0.1 (theory)
(theary)
2.5
0.5
0.1 (sim.)
==1 (sim.)
0.0
10
20
30
40
[4- (w)
Figure 2: Learning dynamics in a perceptron classification task. A, B: Flow fields determining the weight dynamics with trajectories for different initial conditions for SL (A) and RL (B). C, D: Learning dynamics from simulations closely follow the analytical results for SL (C) and RL (D). E: Dependence of the asymptotic weight norm on the regularization parameter A.
Learning dynamics in the nonlinear perceptron
While the above approach general and could applied to any iterative learning algorithm for any ANN architecture, for the remainder of this work we will focus on its application to the nonlinear perceptron (Fig. LAD. a one-layer neural network that receives an input x RV multiplics it with weight vector E IR,,, and produces output ÿ. The task study is binary Gaussian classification task, in which the model is presented with samples x drawn from two distributions p(xly) with labels y +1, where pyy
+1)== Each p(x(y) is given by multivariate normal IB.. We analyze both the case of SL with deterministic output, for which ý o(w x), as well as RL, for which the stochastic output is given by л (
distribution with x ~ N(p", Σ²) (Fig.
+1)= = o(tw x), where & is the logistic sigmoid function. The goal of the model is to output label y that closely matches the ground truth y when given an input x.
Derivation of the flow equations
The supervised learning rule we consider is regularized stochastic gradient descent for binary cross-entropy loss, which results in the weight update rule
f(w); (ū ú)æ; Awa,
(12)
where 5 = ( E (0, the shifted input label, and À is the regularization hyperparameter. This learning rule describes online logistic regression.
For reinforcement learning, we use the REINFORCE policy-gradient rule with reward baseline Williams 1992 Sutton and Barto 2018]:
f(w): υōØ( -ww x)x: Aw;-
(13)
Here d = yÿ (ug) is the reward prediction error, and is the stochastic output of the perceptron with probability ( =1) o(tw x). To facilitate mathematical feasibility, we replace the perceptron activation function by shifted error function o(2)
Erf
We first derive the learning dynamics for stochastic gradient descent. We assume that the initial condition is uniquely specified, with p(w, 0) 5(w -w). In this case, the weight covariance will

---

be zero, and the flow equations (10) simply reduce to
dt (wx) (fr) (.)).//=z (
(14)
To make the formulas more concise, we set À = O. It can be reintroduced by simply adding the term -ww. We then get
(f))((W) = ((ū - <(w x))x
((1 6(w - (owxx))aag x)) 2t> (Ø(w x) E0)
2/4
1
1 1- +b
a+
1 (E+
211439
(15)
v + b4
2m
+ 12
a2
a
Σ
w)
2(1+02
1 82
22x
+ 12
Here, (22) (1 (1+ Erf (z/v2) 002 v8/T) π) is the cumulative distribution function of the standard normal distribution. To simplify notation, we have introduced w w V /8888 as well as the quantities
ay μ
(16)
by
v Vw T Σ. w.
(17)
To aid interpretation of these results, we assume that f I and Σ 021. Then (15) implies
d
+ 0² w ))
w
1
2 F w
(H:W)) 2(14 No
(n .w) = 1 1 1-$ +b
(18)
dt
2
0² w|
w=(w)
as well d
d kw) dt
22 1 ob
0² [w/
2(1+
(19)
VL+0=|W12 [w|2
VI+o² w//2
An interpretation of 18) is that the first term pushes the weight vector in the decoding direction, while the second term acts regularization, whereby the cross-entropy loss penalizes misclassifications more as w increases. An increase in the input noise leads to a higher overlap of the distributions, which means that even the Bayes-optimal classifier will make more mistakes.
For RI,, we need to calculate
(f() = (jōØ( ÿw- x)xx))xx.x)
(Ø(-w x)@( x) x)0x)x~N(n*
E (Ø(-w x)o(w x)
2
(I+ w): 201482)
- 2Φ
0++
+2utT
0+
2TT / + 62
+ 62
1 + 26
+ b
/ + 26
(20)
( .w);
a
a_
1 - 2Φ
2
22 V 1 1+
V1+ 62
1 262
1+ 11+22 62
V 1+ 26
Here, (( is Owen's T function:
T(h,a) J
dx.
(21)
+ x2
As for supervised learning, we can simplify this expression for isotropic distributions with means = and get
di (n- w) =/
11
W
20t W V
(22)
20 μ
2 1
11
Erf
2 /
2 4 W
w=(w)
5

---

A
B
C
SL, RL
o
1
+
SL, RL
SL RL
4.0
SL
1
Time to learm
leam
3.5
Time
2.5
2.0
2.5
0.50 0.75
-0.
-0.25 0.00 0.25 0.50
Noise
Anisotropy
Figure 3: Relationship between input noise and time learn the task. A: The time required for the alignment ja (w)) to reach 80% depends on the noise a of the isotropic input distributions. B: To characterize anisotropic input noise, the total input variance is split into noise component of parallel to and a component of orthogonal to the decoding direction. C: Shifting the input noise into the decoding direction slows down learning.
and
d dt
(w) 2 =8w LT
1
+ 0 2'
/1 20 w//
(23)
1 20 w|
2 10
1
Erf
2
VI+
2 40 w
(w (w)
As show in the appendix, and demonstrated in Fig. 2 the flow equations for both SL and RL have a unique, globally stable fixed point whenever À > or the input noise o > (Fig. (Fig.2A..B) [2A The solutions of (15) and (20) exhibit agreement with learning curves obtained by direct simulation of (I) (Fig. (C.D). where the small remaining discrepancy arises from the fact that, for the simulation, used standard logistic sigmoid function instead of the error function sigmoid curve used for the analytical calculations. We also see that the asymptotic weight norm decreases approximately linearly with In (Fig. 2E Of particular note is the observation that, perhaps counter-intuitively, higher levels of noise appear to lead to faster learning for SL, though the effect is more ambiguous in the case of RL. This will be analyzed in more detail in the following section.
Impact of noise on learning time
We next investigate the effect different types of input noise on the dynamics of learning and whether differences arise for the supervised and reinforcement algorithms. We begin with the case of isotropic imput noise, with = o²l and means =p with |a| = In this case, the optimal alignment [-(w) of 1 is always reached asymptotically, we focus on how quickly this value is approached as Tww]
function of the input noise.
In the case SL, analytically analyzing the logarithmic derivative the alignment between 4f and (w) yields flow equation of the form
d J (w) dt [w)
log
Fiso (1,, w) o2 w) O
(24)
where Diso and hiso do depend The analogous relationship for RI. is indeterminate, such that input noise may either speed up or
Thus, the higher the input noise, the faster the task is learned.
slow down learning in this case, depending the parameters. As is illustrated in Fig. (3A, numerical
integration of the flow equations reveals qualitatively distinct trends for the dependence of learning speed on noise.
Anisotropic input distributions
To analyze the case of anisotropic input noise, we divide the total noise into two components: component of 1 + in the direction of f and the noise of 1 orthogonal to it, while keeping

---

A
SL
RL
0.20
0.20
a
0=1
tr(Cov(w))
0.15
0
tr(Cov(w))
0.15
0.10
0.10
0.05
0.05
0.00
0.00
10
20
40
0
10
20
30
40
Figure 4: Dynamics of the total variance of w for isotropic input noise. Higher noise leads to a faster decay in tr (Cov(w)) for supervised learning (A) and for reinforcement learning (B).
For both SL and RL, we find that learning slows down when the noise is shifted to the decoding direction and speeds up when it is shifted to orthogonal directions (Fig. (3)) To confirm this analysis analytically, we calculate the logarithmic derivative of the alignment between and (w) and find
the total noise oi + of fixed (Fig. 3
11 (w) dt [w)]
log
[an w) + chan (1,, w) + (c)2)
(25)
where Ian and han are independent of s. From this expression, we see that, at least to leading order in c, noise anisotropy orthogonal to the decoding direction tends to increase the speed of learning, while anisotropy along the decoding direction tends decrease the speed of learning. This is in apparent contrast to recent study in two-layer networks, where input variance along the task-relevant dimension was found to increase the speed of learning [Saxe et al., 2019). The reason for these seemingly opposite results is because, in the the task studied in that work, variance along the coding direction a signal that facilitates learning, while, in our case of binary classification, variance along the coding direction is noise that impairs learning.
Input noise covariance
So far, have assumed that the initial weight distribution, which be thought of characterizing an ensemble of networks with different initializations, is specified deterninistically. i.e. p(w, 0) = 6(w w In this case, according to (TI), the covariance of will remain zero later times. If training is instead initiated with a distribution p(w, 0) having nonzero covariance, then we can ask how this covariance evolves with training-in particular, whether the covariance of this distribution diverges, converges to O, or approaches finite value as t + co.
This calculation can be easily performed in the limit - where the inputs are just x =u. Then simply becomes
15)
(f))(()) ie S(u w))
Awa,
(26)
and
implies that
(Cov(w)) W// n² Cov(w) 2Atr (Cov(w)). dt
(27)
Since Cov(w) is positive semidefinite, both terms on the right-hand side of (27) are always nonposi- tive for À > 0 and lead to exponential decay of ir (Cov(w)), so the eigenvalues of Cov(w) approach zero. Thus, the covariance of the distribution p( w. 1) vanishes as - oo (Fig. 4A
The same calculation can be performed for the RL algorithm, again with the result that tr (Cov(w)) + O as t oo whenever to zero upon including input noise (in the + 0 limit we are working in), with the decay speeding up as the noise increased.
(Fig. 4B). As can be seen in Fig. 4 the total variance continues to decay
Application to real tasks
In order test whether the theoretical equations derived above apply realistic input data, we next train perceptron with stochastic gradient descent to perform binary classification with cross-
7

---

A
B
C
1.0
0.75
Accuracy
Cor((w),, .
0.8
LSNE;
0.50
0.25
0.6
Theory
Theory
Empirical
Empirical
0.00
5000 10000 Samples
0
5000
10000
tSNE1
Samples
Figure 5: Comparison of the theory with training on MNIST. A: A nonlinear perceptron is trained to classify the digits 0 and in the MNIST dataset. B: Comparison of the empirical test classification accuracy with the thcoretical prediction. C: Even after the task has been learned, the theory accurately captures non-trivial ongoing learning dynamics.
A
SL
B
SL, RL
1.0
SL, o == 10
SL, 0.1
(we][([]]]
0.6
10-2
RL, a- 0.1
1 0.2
0.2
Mo
0
2
2
Tasks
Figure 6: Forgetting curves. A: Learning curves for multi-task learning, where w are trained on Task 1 (f = μi) after training to 80% on Task 0 (f+ M2). B: The alignment of (w) with Ho after training on additional tasks I....,! 9.
entropy loss on the MNIST dataset (Fig. 5A). To obtain suitable input representations, the images corresponding the digits O and 1 are first convolved with set of 1440 Gabor filters [Haghighat jet al. 2015). (In the appendix, we perform the same analysis on the raw MNIST data without the Gabor convolution and obtain similar results.) We then model these two input classes as multivariate
Gaussians with covariances Σn, and means 1o01 (or 44 after translation). The evolution of the weight vector during training is found by numerically integrating 155). To quantify the test accuracy during training, an approximation of the expected error at each time step is derived by integrating the Gaussian approximations
the two input distributions up to the hyperplane orthogonal to the weight vector. As can be seen in Fig. 5B this theoretically derived learning curve closely matches the actual generalization performance of the trained classifier on the hold-out set.
To further illustrate that the flow equations capture non-trivg non-trivial aspects of the learning dynamics, Fig. 5C shows the alignment of with μa, which continues to evolve after the task has been leamedd The close aligmment of the experimental results with the analytical predictions shows that the flow equations can capture learning dynamics in a realistic task with input data distributions that necessarily Gaussian.
Continual learning
In addition to describing the dynamics of learning a single task, the flow equations derived above can also describe the learning and forgetting of multiple tasks. In continual learning, natural and artificial agents struggle with catastrophic forgetting, which causes older learning be lost is overwritten with newer learning [Hadsell et al., 20200 Kudithipudi et al.) 2022 Flesch et al. 22023] Here, we ask how the number of tasks that can be remembered by the perceptron depends on the level of noise and the learning algorithm. The weights are first trained on Task 0, with input distribution o²T, until the alignment of with Po has reached 80%. We then
defined by l = Ho and train subsequent tasks μ M1 .2..... This yields forgetting curve that decays exponentially
8

---

with the number of tasks, as shown in the simulation results in Fig. 6 The decay constant does not significantly depend on the learning algorithm being used, but we observe that higher imput noise leads to faster forgetting. Together with the results in the preceding subsections, this hints toward trade-off between the learning speed and forgetting of previously learned tasks as the amount of input noise is varied.
Discussion
In this work, we have used a nochastie-process framework to derive the dynamical equations describing learning in the nonlinear perceptron performing binary classification. We have quantified how the input noise and learning rule affect the speed of learning and forgetting, in particular finding that greater input noise leads to faster learning for SL but not for RL. Finally, we have verified that our approach captures learning dynamics in an MNIST task that has a more-complex input data distribution. Together, the results characterize ways in which task structure, learning rule, and neural-cireut architecture significantly impact learning dynamics and forgetting rates.
One limitation of our approach is the assumption that the input distributions are multivariate Gaussians, which may not be the case for real datasets. While the agreement between the theoretical and empirical results applied to the MNIST data in Fig. is encouraging in this regard, there may be greater discrepancies in cases where the input distributions are more complex. Indeed, recent work on the nonlinear perceptron has shown that, while the first- and second-order cumulants of the input distribution are learned early in training, later stages of training involve learning beyond-second-ord (i.e. non-Gaussian) statistical structure in the input data (Refinetti et al.] 2023], suggesting that our theory's ' ability to describe late-stage training in complex datasets may be somewhat limited. Another limitation is the choice to neglect higher-order terms in (w) (Equations (10). (T)) and 77 (Equation (.) This may limit the ability to characterize instabilitics and noise effects induced by non-infinitesimal learning rates. Future work will be needed to assess these effects.
While other work has approached SGD learning in neural networks within a
framework, most of these works have not derived the noise statistics from the noisy update rule (as done here and in Yaidal [2018) and Murray and Escola (2200), but rather have added Gaussian noise to the mean update (e.g. [He et al. 2019 Li et al. 2019 2021)). While the results for the flow of the weights' mean (w) are the same under both approaches, the approach that we take enables us to additionally derive the flow of the weight covariance. Further, it allows for the possibility of describing effects arising from finite learning rate by including higher-order terms in 17 from the topic that we will address in upcoming publication.
expansion of
In our results on continual learning, we found that only a few tasks could be remembered by the perceptron before being overwritten. This is perhaps somewhat surprising given recent work [Murray and Escola, 2020) showing that the binary perceptron can recall ( individual random patterns in continual- continual-learning setup. This difference may arise in part from the fact that that work used more efficient, margin-based supervised learning rule [Crammer et al., 2006) rather than the stochastic gradient descent rule used here, as well as the fact that input noise and weight regularization were not included. This difference suggests that there is likely room for significant improvements in continual- learning performance with the setup studied here. This would be another interesting direction for future work, given that recent work has found that nonlincarity can drastically increase the amount of catastrophic forgetting in continual learning [Domiéé et al. 2023).
Finally, we speculate that qualitative differences between learning rules such as that shown in Fig. 3 may provide a path for designing experiments to distinguish between learning rules implemented in the brain. More work will be needed, however, to formulate testable experimental predictions for more-realistic learning rules and network architectures. More generally, the approach developed here paves the way for analyzing numerous questions about learning dynamics in more-complex circuit architectures and diverse task structures.
Acknowledgements
We are grateful to Elliott Abe for early collaboration related to this project. Support for this work was provided by NIH-BRAIX award RF1-NS131193.

---

References
Blake Rordelon and Cengiz. Pehlevan. Learning curves for SGD on structured features. International Conference on Learning Representations. ArXiv preprini arXiv:2106.02711 02713,, 2022.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman Miilee, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL httt::t/ttttt///e com google/jaa
htttp://ggttbbb.ccmmgggg////////
Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, Yoram Singer, and Manfred K Warmuth. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7(3). 2006.
Clémentine CJ Dominé, Lukas Braun, James E Fitzgerald, and Andrew Saxe. Exact learning dynamics of deep lincar networks with prior knowledge. Journal of Statistical Mechanies: Theory and Experiment, 2023 (11):114004, 2023.
Timo Flesch, Andrew Saxe, and Christopher Summerfield. Continual task learning in natural and artificial agents. Trends in Neurosciences, 46(3):199-22 210, 2023.
Elizabeth Gardner and Bernard Derrida. Three unfinished works on the optimal storage capacity networks. Journal of Physics A: Mathematical and General, 22(12):1983, 1989.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24 1028--1000 2020.
Mohammad Haghighat, Saman Zonouz, and Mohamed Abdel-Mottallt Cloudid: Trustworthy doud-based and cross-enterprise biometric identification. Expert Systems with Applications, 42((1))7905- 7916, 2015. URL https:///git //gthbb www com/mhaghi wwa.haagggghhhhhgo ghat/ gabor BSD 2-clause license.
Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. Advances in neural information processing systems, 32, 2019.
Peter C Humphreys, Kayvon Daie, Karel Svoboda, Matthew Botvinick, Timothy P Lillicrap. Bei learning phenomena can be explained by gradient-bant based optimization. bioRxiv, pages 2022 12, 2022.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learningg Advances in Neural Information Processing Systems, 13:17176-17186 2020.
Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196 210, 2022.
Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research. 20(40): 2019.
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling sgd with stochastic differential equations (sdes). Advances in Neural Information Processing Systems, 34: 14:112722-12725, 2021.
Sukbin Lim, Jillian McKee, Luke Woloszyn, Yali Amit, David Freedman, David L Sheinherg, and Nicolas Brunel. Inferring learning rules from distributions of firing rates in cortical neurons. Nature neuroscience, 18 (12):1804-1810 2015.
Yash Mehta, Danil Tyulmankov, Adithya E Rajagopalan, Glenn C Turner, James E Fitzgerald, and Jan Funke. Model based inference of synaptic plasticity rules. bioRxiv, pages 2023-12, 2023.
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 009540095500 2020.
James M Murray and G Sean Escola. Remembrance of things practiced with fast and slow learning in cortical and subcortical pathways. Nature Communications, 11(1):112,, 2020.
Aran Nayebi, Sanjana Srivastava, Surya Ganguli, and Daniel Yamins. Identifying learning rules from neural network observables. Advances in Neural Information Processing Systems, 33:2639-2650, 2020.
Donald Bruce Owen. A table of normal integrals. Communications in Statistics- Simulation and Computation, 9 (4):389 419, 1980.
10

---

Nishil Patel, Sebastian Lee, Stefano Sarao Mannelli, Sebastian Goldt, and Andrew M Saxe. The rl perceptron: Dynamics of policy learning in high dimensions. In ICLR 2023 Workshop on Physics for Machine Learning. 2023.
Alexandre Payeur, Amy Orsborn, and Guillaume Lajoie. Neural manifolds and learning regimes in neural- interface tasks. bioRxiv, pages 2023-03, 2023.
Jacob Portes, Christian Schmid, and James M Murray. Distinguishing learning rules with brain machine interfaces. Advances neural information processing systems, 35:25937-253 -25550,, 2022.
Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distributions of increasing complexity. International Conference on Machine Learning, pages 28843-28863, PMLR.
2023.
Hannes Risken. The Fokker-Planck Equation. Springer, 1996.
Andrew M Saxe, James L McClelland, and Surya Ganguli. mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 153-23): 2019.
Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review 45(8):0056, 1992.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde. Josef Perktold, Robert Cimrman, Ian Henriksen. E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:261 272, 2020. doi: 10.108///4199 019-0686-2.
Justin Werfel, Xiaohui Xie, and Seung. Learning curves for stochastic gradient descent in linear feedforward networks. Advances in neural information processing systems, 16, 2003.
Ronald Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8:229 256,, 1992.
Sho Yaida. Fluctuation-dissiaatio relations for stochastic gradient descent. In International Conference Learning Representations, 2018
Paul Zügge, Christian Klos, and Raoul-Martin Memmesheimer. Weight versus node perturbation learning in temporally extended tasks: Weight perturbation often performs similarly or better. Physical Review X. 13(2): 021006, 2023.

---

Derivation of the general evolution equations
In this section, we derive equations multiply both sides by wy and integrate over w. The left-hand side simply becomes
TT)
and
from 9
We start with
Op 0t
w. 1)
Σ
(p(w, w, t)(
(28)
Dw.z
(w))
(29)
dt
For the right-hand side, we can use integration by parts to get
- -f dw wt E Bwy (p(w, ()f))) = dw p(w, ((f))
(30)
To evaluate this expectation value, introduce the mean-zero weight w w - (w), which describes the fluctuations of w around its mean. we expand (ft) to second order in w. (30) becomes
+ ((fi)(W)) (U)))(())) + Σ (i))(())))(())))
1 2
Σ
j.k
-(fle((w)) ((s))w
Σ
O1 (31)
j.k
= (fil 1 Ecov(uk
j.k
The derivation of 11 follows analogously.
Derivation of explicit SL and RL flow equations
In order to analyze 10 and
we must evaluate the following expectation values:
(ø(w x N()x
and (o2(w- (wWxx)x)x=AA(x...
(32)
with ((z) = (1 Erf
Erf v
Without loss of generality, we will calculate these integrals in
coordinate system where w wi el. We can then factorize p(x1, .2....) Pm(x) (x) (2 ... |21). The marginal distribution is Gaussian with pm μu and Σm. Σ11. and the conditional distribution also normal with (Me), = as + En En((r - n1), and (E.)iz = Σij Σn En:Eig-
Furthermore, to simplify notations, introduce w v 4/88 well as the quantities
p W,
(33)
b == V VWTTE Ew.
(34)
Let's first calculate (ø(w x)xxx=))
(o(w x))xxN((x = ( (0(w12)) X-)()
/
doie - =xx )(w1)1)
2 /
1
/ duc duc-11126 (wimm) + u uVE))))
(35)
22 JR
V2+b2
where (xx) = 1 + Erf(x// 22) is the cumulative distribution function of the standard normal distribution. evaluate the last line of this and the following integrals, we used the reference (Owen 1980].
12

---

For the integral (ó(w x)xx)xx-V(11 we first do the calculation for = 1:
(o(w x)x1)x~((H.)) = (0(w)1) )x1)x~ 100000--1)///11 ((W(T))))
doge
V2
1
due du/-0//2
(wi(i) + uv / E11) (u1 uv Eni)
(36)
2t
Σι 2(1485).
11
VI++3>
2t vI 11+10 ++
For i + 1, we get
(o(w. N(..)) (0(w)))) (as)
1
drie - (01--1111 /En ((w1)1)
(4 (u+
Els (x) m1)
V2T
Eل
1 En w1 21100
-4
V1+13/
2t v1 + b
(37)
Thus, for general i and w we can write
Σw 22T VI b
(ø(w
11
204
(38)
+1+
We next calculate
(0 (w= x)xxx = (w1Z))))
drip - ((01- / (wIxz)
V2
(39)
-2T
VI+
VI + 62
VI + 26
where 7 stands for Owen's 7 function.
Analogously, we can calculate
(0 (w x)II))xNN(1133
2(Ew), T
(40)
pe p (w x))
2(++22)
/ + VI+BV1 21
Fixed point analysis
In this section, we analyze the fixed points of the systems of equations (88) (99 for SL and (22)
(23) for RL. We will first show the intuitive result that any fixed point (w") ) is maximally aligned with
μ. i.c. (w*)- μ |4|| ((w")), as long as O. For simplicity, we set the regularization parameter
0. Note that for both SL and RL, the flow equations take the form
d
(w 1 |p|2 (H, W. 11 (W) w,
dl
(41)
kw)l²= = pa. (w) (w)
dt
for some functions fi and f2 0. Also, it's easy that (w*) 1 0. Thus, fixed point (w") satisfies
1 (w*)
F2 14
(42)
fi (W
fz 1 (w*)
13

---

Thus, setting these equal to one another, we find that (w*) > | ((w") reduce to a single equation for (w")l-
and the two equations
Assume without loss of generality that [14| = 1. For supervised learning,
then implies that
((w))
(ww)
o² (w")
11 (43)
Erfe
10+2ca2 (w")1²
/16 2TT (w*))²
V1+ o2 (w*)) 1
where Erfc(z)
Erf(=) the complementary error function.
We can factor out the common exponential asymptotics of both terms to get
[(w*) 3 16 16 3/
1( 1( 12 16
1( V
[w=
Erfe
2
V16 16 2 (ww)
(44)
/ 1 + 0² (w (w
Of the two terms in the parentheses, the first term starts 2 for (ww)) | decreasing to the value _el Erfe( _11 20 0² + O()))) while the second term starts at O and increases the larger value V Thus, there is unique fixed point. For RL, uniqueness of the fixed point can shown using an analogous argument.
0 and is monotonically
/2a
This fixed point (w*) is stable, because for SL, (4 implies that de (w) W where L is the cross-entropy loss and (-)z is the average over the input distribution. Thus, (C) is a Lyapunov function for the dynamical system. For RL, the same argument holds by the policy-gradiens theorem, where L additionally includes an average over the output noise.
Note that, for À 0, although (w)) diverges, calculation of d. log (1) [[] shows that the
alignment between fa and (w) still converges to This result consistent with those of and Telgarsky 2020)
MNIST details
A
B
1.0
1.00
Accuracy
0.8
Cor((w)).u)
0.75
0.50
0.6
Theory
0.25
Theory Empirical
Empirical
0.00
5000 10000
10000
Samples
Samples
Figure 7:
Comparison of the theory with training on raw MNIST. A: Comparison of the empirical test classification accuracy with the theoretical prediction. B: Just like for the Gabor-filtered inputs, the theory accurately captures non-trivial ongoing learning dynamics.
When applying our theory to the MNIST dataset, compare SGD applied actual data (orange curves) with calculated SGD curves from our theory (blue curves). In the main text, we preprocess the data by convolving raw pixel values with bank of Gabor filters approximate more realistic scenario where the binary classifier appears at the end of convolutional neural network. For the plots in Figure 7 we use raw pixel values demonstrate that our evolution equations (15) work in general settings without relying on Gabor filter representation. In both cases, we globally translate all input vectors such that the dataset's mean zero. We directly evaluate test set accuracy (Figures
5B and and the correlation of w at each SGD step with the mean f of digit "1' (Figures 5C and ). calculate the theoretical accuracy curve, we first numerically solve the differential equations for the mean fe and covariances Eo,1 obtained from the empirical dataset to derive w(t). We

---

then integrate two multivariate normal distributions with these l and Σ values in the half-spaces bounded by w(t) (expressible as an error function) and plot the result as the theoretical aceuracy curve in Figures 5B and 7A The tSNE embedding in Figure 5A is included for illustrative purposes only and not used in calculations.
Experiments
The numerical code implementing the model and performing the analyses mostly written in JAX Bradbbry et al. 2018], as well as Wolfram Mathematica and SciPy [Virtanen et al.) 2020]. For plotted for the limit of zero input noise and regularization parameter of 0.1. The learning curves are plotted for À = O and Σ I²I, with o = 0.1 and a = 1. As in all other figures besides Fig. 5 we set |p 1. Each experiment
Fig. 2 the flow fields
repeated for 10 runs with different random seeds, with the standard deviation indicated as a (barely visible) shaded region. The curve of [(w>>| is plotted for the limit of zero input noise. For Fig- we numerically integrated (TO) with 0. For the total variance plotted in Fig. we set 0. and integrate the differential equations numerically. For Fig. 5 the Gabor-filtere inputs were created using (Haghighat et al., 2015 with default parameters. The perceptron was trained using SGD with a logistic sigmoid output, and the orange curve in panel B shows data. We set À = 1. For the forgetting curves in Fig. 6 we set X = 10 and the learning rate to 7) 10-2 The curves shown are averages over 50 different random initializations each. The input dimension is set to N = 500. For all other simulations, the learning rate was set to 10-3, The computations were performed on NVIDIA Titan Xp GPU, with runtimes most few minutes.
accuracy. The training was repeated 10 times with shuffled
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As we claim the abstract, we study the learning dynamics nonlinear perceptron analytically, and apply the results to better understand the effects of the input distibution, learning rules and an actual perceptron training on MNIST.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included paragraph the Discussion section that describes the most-significant limitations of our approach.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: All of our mathematical results are based on clearly stated assumptions, and the steps for obtaining the results described sufficiently thoroughly that reader should able to reproduce them.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The mathematical equations on which our empirical results are based are provided in the text, and all parameters used for producing the results are stated in the figure captions or appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All the source code files can be found in the supplemental materials, which should allow a reader to straightforwardly recreate all experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nipss wwtpcc/pcpppcpcceeeooeoooop /Codeeeee/oomdbiiiccccc wwwtanssssonppoo swiniisssiiiiicyyy Policyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All parameters used for producing the results are stated the figure captions appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The empirical curves in all relevant figures include shading to represent standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The modest compute resources required for experiments are described in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp/////eerrppp..c/cppp//////a
Answer: [Yes]
Justification: respects.
The research conducted in this paper conforms with the Code of Ethics in all
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work foundational research, and we not foresee any direct socictal impacts that could potentially arise from it.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The only non-standard external code used is (Haghighat al. 2015 for the extraction of Gabor filters. As mentioned in the citation, the code is released under BSD clause-2 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "3U4KICvDtF", "reviewer": "Reviewer_WrR8", "summary": "This paper analyzes the weight dynamics of single layer neural networks with nonlinear output functions.", "questions": "Why is the probabilistic output is considered as reinforcement learning? It should be driven by reward signal.", "limitations": "It is not clear hoe this extends to multi-layer cases.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The model of stochastic weight evolution dynamics is quite general.", "weaknesses": "The theory is tested only in binary classification task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, but we disagree with their conclusion that our paper should not be accepted based on the fact that our theory is applied only to binary classification in a single-layer system. Indeed, according to the other reviewers, our work “addresses an important open question” (m7rp), and “This paper’s strengths are its impact and novelty… Because nonlinear perceptrons comprise the building blocks of many deep learning algorithms, a derivation of learning dynamics for the nonlinear perceptron is, in my view, rather valuable” (SUcs).\n\nWe recognize that there is a tradeoff in developing theories of simple systems, where relatively thorough understanding of mechanisms can be obtained, vs. complex systems, where obtaining such understanding is difficult without making drastic approximations. In the past, NeurIPS has recognized the value of both approaches. As an example of work in a relatively simple setting similar to the one that we employ, we cite the work of Mignacco et al (NeurIPS, 2020), who derived a theory of SGD dynamics in a single-layer network with a linear or quadratic activation function performing binary classification.\n\nIf the reviewer has any specific critiques of our paper’s soundness, presentation, or contribution, we would be happy to discuss and address them.\n\nFinally, regarding the question that the reviewer asked, the learning is indeed driven by the reward signal in our reinforcement-learning setup, where the reward is $\\pm 1$ depending on whether the binary output is correct or incorrect (see below Eq. (13)). As usual in RL, the stochastic output may lead to greater or less reward, and the parameters are updated to maximize this reward.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "shHXAaumKm", "reviewer": "Reviewer_SUcs", "summary": "This paper provides what seems to be the first derivation of equations for the dynamics of SGD-style learning in a nonlinear perceptron outside of the standard student-teacher paradigm. Specifically, a Fokker-Planck-like PDE is derived for the temporal dynamics of the weight distribution and from this ODEs are derived for the first and second moments of the weights. The equations can be applied generally to different learning rules; in the paper supervised learning of logistic regression and REINFORCE are studied. Through numerical solving and analysis of the dynamical equations several interesting results are found. In particular, it was observed that increasing noise has opposite effects on supervised learning compared with REINFORCE, and that noise orthogonal to the classification boundary speeds learning in both REINFORCE and supervised learning.", "questions": "I am arranging this section in bullet points for ease of reading. Properly addressing these questions/comments could lead to me improving my rating of the paper during the rebuttal. I have divided these questions/comments based on whether I perceive them to be highly important (Primary), or regularly important (Secondary). I have also included a section with basic writing related comments at the end (Syntax). When a question/comment corresponds with a line (or lines) of the paper I list that line at the start of the comment.\n\n# Primary\n- Previous work studying SGD dynamics has had trouble deriving a continuum limit that meaningfully takes into account noise (i.e. that doesn’t become gradient dynamics in the limit – see e.g. section 2.3.3 in Yaida 2019, a reference in your paper). It could be nice to discuss a little (beyond the comment on line 70) in the text how your work is able to derive a continuous time F-P equation and avoid this issue\n- Related to the above: it would be nice to discuss the assumptions necessary for the continuum limit in the main derivation. If there are key assumptions, this could be listed in the limitations in the discussion section\n- 78 (equation 9): this is of course basically a Fokker-Planck equation but with a particularly chosen drift term and no diffusion term. Could you elaborate on the similarities and differences with the Fokker-Planck Equation (FPE) please?\n- 80: I am not sure why the definition $\\delta w = \\langle w \\rangle - w$ can be made. Is this a different variable than $\\delta\\mathbf{w}$ (with $w$ boldface) from before?\n- 80-81: Risken is referenced as the means of deriving the moment equations. Given that the PDE studied is a very particular version of the FPE, and that NeurIPS represents a fairly general audience, it would be nice to include derivations of the moment equations. These could be included in the appendix. It would also be nice to have page numbers for where the referenced derivations in Risken appear\n- 125-126: it is claimed that there is a unique globally stable fixed point, but this is not immediately clear to me from the equations. Perhaps the analysis of the equations leading to this conclusion could be listed in the appendix?\n\n# Secondary\n- It could be nice to discuss how your work compares with the related work on SGD dynamics: *Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties* – Paquette et al. 2022.\n- 59: might be nice to remind the reader here that $w \\in \\mathbb{R}^n$ for any $n$ (unless I am mistaken). This is an important aspect of the paper and deserves to be highlighted.\n- 67: would suggest changing “Performing a Taylor expansion” to a sentence that describes that you are changing the variable of integration from $w’$ to $\\delta w$ and then Taylor expanding w.r.t. $\\delta w$. This doesn’t take much extra space and is helpful for the reader.\n- 67 (equation 4): it would be good to include the smoothness assumptions on $p$ that are required to interchange the order of integration and differentiation in this equation\n- 131: “particularly for SL” => “only for SL”.\n- 175: is the convolution of the input data with Gabor filters necessary? If so, a little elaboration on why would be useful\n- Figure 6: not immediately clear which of the two plots are derived from the flow equations and which are from numerical simulations. Could this be explained?\n- 223-224: it is suggested that this approach would allow for effects due to finite learning rate, but the learning rate is taken to zero for the continuum limit, no? Does this future direction involve simply working in discrete time?\n- 321: why was $\\lambda$ set to $10$ for the forgetting task? This seems abnormally high\n\n# Syntax\n- Fig 1.B: might suggest replacing multinormal => multivariate normal as the latter seems more conventional\n- 34-39: long sentence. Suggest ending at “... (e.g., supervised and reinforcement 36 learning)” <period>. And starting a second sentence. For the second sentence, avoid saying both “important goal” and “longstanding goal”, for more concise writing\n- 67: “becomes” => “yields” or similar\n- 110 (equation 15 line 1): parentheses is on right side of $x_i$ when it should be on left\n- 77: would suggest defining the average w.r.t. $p(w,t)$ later (e.g. line 83) as it doesn’t show up in the equations yet.", "limitations": "In this section I will primarily discuss issues related to the NeurIPS checklist. I will do so in point form in the order of appearance of the topics in the checklist. Below, I only include bullet points where I have concerns about what the authors list. Addressing these issues could also lead me to improve my rating of the paper.\n\n- Theory assumptions and proofs: \n  - I disagree with the justification for this section. Certain assumptions are not included (e.g. assumptions necessary to interchange differentiation and integration mentioned above) and extra insight could be provided on derivations (see “clarity” mentioned above).\n- Broader impact: \n  - I would argue that, even with theoretical work, one should consider downstream interests. E.g., is it ethical to study general algorithms that can easily be employed for harm (e.g. in weapons industry) without taking steps to support policy that restrict the negative application of such innovations?\n  - Safegaurds: this should be “No” instead of “N/A”, for the above reasons.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "In my view this paper’s strengths are its impact and novelty. As far as I can tell there are no other works that derive the learning dynamics for nonlinear perceptrons except for in a student-teacher regime, making this work novel. Because nonlinear perceptrons comprise the building blocks of many deep learning algorithms, a derivation of learning dynamics for the nonlinear perceptron is, in my view, rather valuable (to the point that it is surprising that this has not been done already!). The paper also goes beyond just deriving equations, but studies them and finds some interesting results related to how noise impacts learning.", "weaknesses": "In my view the paper’s primary weaknesses is its clarity. There are many points in the paper where equations are derived and the sub-steps are not thoroughly explained. This might be less of a problem in a physics or mathematics publication, but for the general audience encountered at NeurIPS I believe that there should be more “holding of the reader’s hand” w.r.t. derivations. There are also several sections where it could be useful to explain assumptions a little more thoroughly (e.g. around the continuum limit). If it wasn't for this point regarding assumptions I would have given the paper a soundness rating of 4. If it wasn't for the issues of clarity I would have given a higher presentation rating.\n\n# Notes on my review\nIn part because of this issue of clarity–and in part because of lack of time on my part and a large reviewing load for this conference–I have been unable to verify many of the equations in the paper. Specifically, I have only verified the math up to and including Equation 9. Thus, it is difficult for me to comment on the soundness of many of the results, leading to my low confidence score. Critically, my score for the paper assumes that the math that I have not verified is sound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Primary questions and comments\n\n- Flow equation derivation. In deriving our continuous-time flow equation, we don’t make a priori assumptions about the nature of the noise, but rather derive the FPE-like equation directly from the weight-update equation. Although this is not yet realized in the current work, a motivation for this approach is the non-Gaussian noise inherent in non-linear models even for Gaussian inputs, as well as in features of deep neural networks. While, at leading order in $\\eta$, the evolution equation for the mean of the weights that we arrive at is equivalent to gradient flow, the method by which we arrive at this equation is different from most other works in the literature. In addition, our approach yields a flow equation for the weight covariance, which is not present in gradient-flow approaches. \n\n- Assumptions for the continuum limit. While the equations for finite learning rate assume that the discrete process $\\mathbf{w}$ can be interpolated smoothly between updates, with constraints on the uniform convergence properties of the series expansion, these assumptions are easily satisfied in the $\\eta\\to 0$ limit as long as the process $f_i(w)$ in the weight update equation behaves nicely in the limit. We assume that the moments of $f$ are finite and smooth, and we have explicitly added this assumption in the paper.\n\n- Relationship to the Fokker-Planck equation. At the order to which we are working, the reviewer is correct that our Eq. (9) is an FPE with no diffusion term. Under certain mathematical assumptions, one can work at the next order in $\\eta$. Then the right hand side of the equation would be identical to that of an FPE, while the left hand side would contain a second-order time derivative. This approach provides an alternative to previous approaches to analyzing the effects of nonzero learning rate and is the subject of our forthcoming paper.\n\n- Definition of $\\delta w$. Thanks to the reviewer for pointing out the potential point of confusion about the variable $\\delta w$. In the revised version, we have not defined this variable in the main text, and, where we use it in the appendix, we define it instead as $\\hat{w}$ in order to avoid confusion with the variable $\\delta \\mathbf{w}$.\n\n- Existence of a unique, stable fixed point. In response to the reviewer’s suggestion, we have added a new appendix in which we explicitly prove the existence of a unique, stable fixed point in our flow equations.\n\n# Secondary questions and comments\n\n- In Paquette et al., the authors study regularized least-squares regression and prove that they can approximate the loss function for finite learning rates by the loss function of the solution of a stochastic differential equation in the limit of high input dimensions. We thank the reviewer for pointing out this reference. In the present work, we study a somewhat different setup and don’t include finite-$\\eta$ effects, but the mentioned result can serve as a valuable point of comparison for the theory with finite $\\eta$ in an upcoming publication.\n\n- Interchanging the order of differentiation and integration. We thank the reviewer for pointing this out. We added technical requirements on $p(\\mathbf{w}+\\delta \\mathbf{w}, t+\\delta t|\\mathbf{w},t)$ to apply the dominated convergence theorem. For the update equations and noise distributions considered in this paper, this requirement is satisfied.\n\n- Preprocessing of the MNIST data with Gabor filters. Please see the explanation and figure in the main rebuttal.\n\n- Effects of nonzero learning rate. Please see our response to the question above about the relationship of our approach to the Fokker-Planck equation.\n\n- Value of $\\lambda$ in the forgetting task.\nThe (logarithmic) range of input noise levels between the level where noise is negligible to the point where forgetting happens too fast to show on the plot is rather small. Since higher values of lambda lead to faster convergence, and we wanted to include a high number of runs to average over, we decided to set $\\lambda=10$ for this plot to increase the speed of our simulations. To address the reviewer's question, we have repeated this procedure for $\\lambda=1$ and can produce a figure very close to Figure 6B for the noise values $\\sigma=0.2$ and $10^{-2}$.\n\nAll of the other suggestions made in the reviewer’s list of Secondary Questions and Comments have been incorporated into the revised version of our paper.\n\n# Syntax suggestions\n\nThanks to the reviewer for these helpful suggestions. They have all been implemented in the updated version of our manuscript.\n\n# Limitations\n\n- Statements of assumptions and details of derivations. We believe that our revisions, including the new appendices, described above have addressed all of the reviewer’s comments on these points.\n\n- Broader impact and safeguards. While we appreciate the reviewer’s conscientiousness about the broader impacts of basic research and agree that such issues are important, the Checklist guidelines seem to state clearly that including a statement about broader impacts and safeguards in papers focused on basic research questions is not appropriate for the type of work that we have submitted. The example given is the following: “It is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster”. While we recognize that reasonable people could differ on this point, our view is that adding a boilerplate statement on AI safety to every single paper, regardless of how relevant or irrelevant the point may be to the work, would detract from attention being paid to such statements in papers where broader impacts and safeguards are actually directly relevant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B6vX3HKDRK", "reviewer": "Reviewer_m7rp", "summary": "In this work, the authors derive a set of equations describing gradient flow in a non-linear finite-dimensional perceptron under the assumptions that the data is multinormally distributed, there is a small learning rate, and the task is binary classification. They develop the equations both for the case of a supervised learning rule and for a reinforcement learning rule. From these equations and simple simulations, they study the impact of noise on learning time, the influence of anisotropy in input distributions, input noise covariance and continual learning. They furthermore use a preprocessed MNIST dataset to asses their derived equations beyond toy datasets.", "questions": "1)\tAre the insights following the derivations of the gradient flow equations (i.e., sections impact noise on training time, Anisotropic input distributions, etc), also based on the assumption that the covariance of the weights is zero (L107)? If so, please elaborate on to what degree this assumption limits the applicability of your results in practical settings. \n\n2)\tHow does your work differ from the theory developed in [5]? \n\n3)\tProvided I understand [5] and your work correctly, it seems that the dynamics of learning in non-linear perceptrons when the input data is Guassian is equivalent to the dynamics of a linear model. As such, your theory would not really capture the effects of the non-linearity. Please elaborate; if this is correct, please indicate how you would rephrase the paper to reflect this fact. \n\n4)\tPlease compare your insights to the analyis of Saxe [4]. Specifically, they mention an analysis of the following: we find that perceptual correlations can either speed up or slow down learning, depending on whether they are aligned or misaligned with the input dimensions governing task-relevant semantic distinctions.\n\n5)\tPlease compare your insights on continual learning with the work of [3]\n\n6)\tPlease explain better how you used/modeled the MNIST dataset, cfr L177. \n\n7)\tHow would you expand the related works to include more relevant papers? \n\nIn general: if you agree something would be a useful contribution to your work, please explicitly state how you would update the current manuscript.\n\n1)\tJi, Z., & Telgarsky, M. (2020). Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33, 17176–17186.\n2)\tSaxe, A. M., Sodhani, S., & Lewallen, S. (n.d.). The Neural Race Reduction: Dynamics of Abstraction in Gated Networks.\n3)\tJ Dominé, C. C., Braun, L., Fitzgerald, J. E., & Saxe, A. M. (2023). Exact learning dynamics of deep linear networks with prior knowledge *. Journal of Statistical Mechanics: Theory and Experiment, 2023(11), 114004. https://doi.org/10.1088/1742-5468/ad01b8\n4)\tSaxe, A. M., McClelland, J. L., & Ganguli, S. (2019). A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23), 11537–11546. https://doi.org/10.1073/pnas.1820226116\n5)\tRefinetti, M., Ingrosso, A., & Goldt, S. (n.d.). Neural networks trained with SGD learn distributions of increasing complexity.\n6)\tPinson, H., Lenaerts, J., & Ginis, V. (2023). Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequencies. International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, 202, 27876–27906. https://proceedings.mlr.press/v202/pinson23a.html", "limitations": "See questions above; no ethical considerations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper adresses an important open question, namely how we should study or understand the learning dynamics of non-linear neural networks. The mathematical derivations seem sound and correct, the paper is well-written, and the carefully designed figures not only illustrate the results but also aid in understanding the approach (cfr figure 1). The theory has been developed for both the supervised and reinforcement learning case. The authors have made an excellent effort to demonstrate the insights that can be gained from their mathematical derivations, and have extended their experiments beyond toy datasets to illustrate the vailidity of their approach further.", "weaknesses": "The work is seemingly presented as a general theory of learning in non-linear perceptrons; however, the assumptions are very strong and far from practical settings. The assumptions might even render the learning dynamics equivalent to those of a linear model (see questions). This should be more clear in abstract and/or introduction.  \n\nIt is hard to assess the novelty of the results, as the work seems to lack references to closely related works that derive equations in similar settings and/or draw similar conclusions. Notably a comparison to the work by Refinetti [5] would be beneficial (see questions). \n\nFar more relevant related work exists on the dynamics of learning in complex and/or non-linear networks. Currently, the introduction presents the related work as merely falling into two approaches: the student-teacher setup and the linearized perceptron. However, several works exists for far more complex and even non-linear setups exist. References below are just a few of the possible additions. \n\nThe MNIST dataset is heavily preprocessed and it is mentioned that the two classes used are modfied in the following way: (L177) ‘We then model these two input classes as multivariate Gaussians with covariances Σ0, and means μ0,1 178 (or °æμ after a translation).’ It is unclear how this dataset is still a deviation from the multinormal assumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Questions\n\n- Setting the weight covariance to zero.\n\nWe found that Cov(w) approaches 0, which is equivalent to saying that all initial conditions $w^0$ converge to the same fixed point. Because the results mentioned above do not depend qualitatively on the initial condition, including Cov(w) > 0 would effectively average over an ensemble of trajectories w(t) with different initial conditions, which would give the same qualitative behaviors that we showed since all (or almost all) of these trajectories exhibit qualitatively similar behavior.\n\n- Relationship to Refinetti et al (2023). \n\nThanks for pointing us to this very interesting paper. In that work, the authors perform a Taylor expansion for a nonlinear perceptron performing binary classification and show theoretically that, iff the nonlinear terms in the expansion are included, the value of the weights once learning has converged depends on beyond-second-order cumulants of the input distribution. \n\nHence, the linear perceptron is unable to fit higher-order (i.e. non-Gaussian) statistical structure in the input data. \nOur finding that the fixed point of the system depends on the nonlinearity of the activation function is perfectly consistent with these results. The reviewer’s concern that, because we only include the first two cumulants of the input-data distribution in our model, the nonlinearity of the activation function is not captured by our model, does not follow from their results. It is true, however, that our choice to include only the first two cumulants of the input-data distribution in our model is a significant limitation in applying the results of our theory. Indeed, this is the first limitation that we mention in our Discussion section. However, our results on the MNIST data show that, at least for this dataset, our theory does well at capturing the dynamics of learning for non-Gaussian input-data distributions.\n\nApart from the issue of Gaussianity and how it relates to non-linearity of the activation function, our approach is overall quite distinct from that of Refinetti et al in the following ways: (i) they analyzed only SGD, whereas we compare RL vs. SGD; (ii) their theoretical results were obtained perturbatively, whereas our calculations are non-perturbative and fully capture nonlinear effects at all orders; and (iii) their theory focused on the properties of the fixed point that learning converges to, whereas our work additionally focuses on the dynamics of learning and how these are affected by the properties of the input distributions. In particular, the result on how input noise along or orthogonal to the coding direction impacts learning speed crucially depends on the saturating nonlinearity of the activation function, which is not captured by a perturbative calculation.\n\nIn response to the reviewer’s suggestion, we have included the following statement in our Discussion: “Indeed, recent work on the nonlinear perceptron has shown that, the first- and second-order cumulants of the input distribution are learned early in training, later stages of training involve learning beyond-second-order (i.e. non-Gaussian) statistical structure in the input data (Refinetti et al, 2023), suggesting that our theory's ability to describe late-stage training in complex datasets may be somewhat limited.”\n\n- Relationship to Saxe et al (2019).\n\n Thanks for pointing out the connection between this work and ours. In that paper, “perceptual correlations” refers to the anisotropy of the input distribution. In their Supplemental Material, Saxe et al found that, in a two-layer linear network trained with student-teacher learning to perform a linear mapping $y = \\mathbf{w}^* \\cdot \\mathbf{x}$, training converged more quickly when the input data had large variance along $\\mathbf{w}^*$. This is interesting to compare it with our results, since we found that increased input variance along the coding direction *decreases* the speed of learning. \n\n\nIn the updated version, we have included the following remark below Eq. (25) to account for the difference:\n“This is in apparent contrast to student-teacher learning in two-layer networks, where input variance along the task-relevant dimension tends to *increase* the speed of learning (Saxe et al, 2019). The reason for these seemingly opposite results is because, in the student-teacher case, variance along the coding direction is a signal that facilitates learning, while, in our case of binary classification, variance along the coding direction is noise that impairs learning.”\n\n- Relationship to Domine et al (2023). \n\nThe approach of Domine et al (2023) differs from ours in that their work analyzes two-layer networks in the student-teacher setup, whereas ours analyzes a single-layer nonlinear network performing binary classification. While continual learning was not the main focus of that work, they analyze a continual-learning setup within their framework, with the main finding being that nonlinearity in the network increases the amount of catastrophic forgetting. Comparing linear vs. nonlinear versions of our model to see whether this result also holds in a single-layer classifier could be interesting, but this is not a direction that we pursued.\n\nIn response to the reviewer’s suggestion, we have added the following to our Discussion: ``Recent work has found that nonlinearity can drastically increase the amount of catastrophic forgetting in continual learning (Domine et al, 2023), so the nonlinearity in our classifier may play a significant role here as well.”\n\n- Relationship to Ji et al (2020).\n\nWe thank the reviewer for pointing us to this great reference. In this work, the authors prove for a large class of (generally non-linear) models that weights converge in direction, even in situations where their size diverges. Motivated by this work, we showed that for our model, this result explicitly follows from the flow field equations and cited the paper.\n\n- MNIST approach: See main rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their comments and suggestions on our work. We were encouraged by their recognition of the work’s novelty and impact, as well as its soundness and exposition. We have made numerous revisions to our paper in order to address the reviewers’ critiques, the three most significant of which we summarize here.

The first main substantive issue that came up in the reviewers’ critiques was the need to provide additional details on intermediate steps in our mathematical derivations. In response, we have added new appendices to the paper that fill in the details of most of our calculations. In the first section of the appendix, we derive the moment equations (10) and (11). In the second part, we derive the flow equations (15) and (20) by explicitly calculating the involved integrals. Furthermore, we analyze the convergence of these equations and provide a detailed proof of the existence of a unique, stable fixed point. We thank the reviewers for encouraging us to provide these additional details, and we expect that the addition of these appendices will make it much more straightforward for future readers to follow and reproduce our results.

The second main substantive issue brought up by the reviewers was the need to be more explicit about certain assumptions in our empirical results and mathematical derivations. In applying our theory to the MNIST dataset, we did not properly explain our motivation for preprocessing the input data with a bank of convolutional Gabor filters, leading to the reasonable suspicion that our theory, which models the input-data distributions as Gaussian, could not be successfully applied to realistic datasets. Our motivation for initially passing the MNIST data through a layer of untrained, convolutional Gabor filters was that, in practice, a binary classifier such as the one that we model would often appear at the output end of a neural network rather than being applied to raw input data, so that preprocessing the data in this way brings our model closer to a realistic use case. 

Thus, our motivation in performing this preprocessing was not to make the input data more Gaussian, though it may have had that effect. In response to the reviewers questions, we have rerun this experiment using the raw MNIST input data, without the convolutional preprocessing. The results are nearly identical to the ones we showed and are included in the appendix, as well as attached to this rebuttal. This provides further evidence that our approach in practice successfully characterizes the dynamics of learning in cases where the input data is non-Gaussian.

To be as transparent as possible, here is a detailed explanation of how the MNIST Figure 5 was produced:

“When applying our theory to the MNIST dataset, we make the comparison between SGD applied to the actual data without any modeling (the orange curves) and the calculated SGD curves obtained from our theory (the blue curves). For the empirical curves, we perform SGD on either the raw pixel values for the plots in the appendix, or on a representation obtained by convolving the raw pixel values with a bank of Gabor filters in the main text, with the only modification being a global translation of all input vectors to zero the global mean of the dataset. We then simply evaluate the accuracy on the test set for Figure 5B, and the correlation of $\mathbf{w}$ at each SGD step with the mean $\mathbf{\mu}$ of the digit ‘1’ for Figure 5C. To calculate the theoretical accuracy curve, we first numerically solve the differential equations (15) for the mean $\mathbf{\mu}$ and covariances $\Sigma_{0,1}$ calculated from the empirical dataset to obtain $\mathbf{w}(t)$. We then integrate two multivariate normal distributions with these values of $\mu$ and $\Sigma$ in the half-spaces bounded by $\mathbf{w}(t)$ (which can be explicitly expressed as an error function) and plot the result as the theoretical accuracy curve in Figure 5B and find a perfect agreement with the empirical SGD curve. The tSNE embedding in Figure 5A is not used for any calculations, but only drawn for illustrative purposes.”

In addition to these changes, as described in detail below, we have added numerous minor clarifications to the text to clarify the assumptions and reasoning in our mathematical derivations, with the main restriction being the existence of all higher moments of the weight update term $\langle f_i^k\rangle_L$, $k=1,2,\ldots$ as smooth functions of $\mathbf{w}$. All these assumptions are satisfied for the non-linear perceptron we apply our results to.

Finally, the reviewers pointed out connections of our work to other recently published work on learning dynamics. Of particular note was a concern that, because we model the input-data distributions as Gaussian, the learning in our nonlinear model might be equivalent to that in a linear model. In our response below, we have provided a detailed explanation of why this is not the case. Overall, we do not believe that the existence of any of these other works detracts significantly from our work’s novelty, and we are grateful to the reviewers for giving us the opportunity to draw connections with relevant work from the literature that had escaped our attention.

In the responses that follow, we provide detailed responses to the reviewers’ questions and comments. We believe that their suggestions have been completely addressed in our revised version.

Meta-review: The paper provides a theoretical study of the behavior of a nonlinear perception on a simple problem. 

Reviewers recommended Accept, Accept and Borderline reject. While the settings are restricted, the contribution seems valuable. Reviewers appreciated the authors' modifications in response to reviews.

Overall I am willing to accept the paper.
Decision: Accept (poster)