Free Probability for predicting the performance of feed-forward fully connected neural networks
Authors: Reda CHHAIBI
Tariq Daouda
Ezechiel Kahn
Venue: NeurIPS 2022 Accept (2022)
Keywords: Infinite width regime
Lyapounov exponents
Singular spectrum of Jacobians of neural networks
Stability
Free Probability Theory
Numerical Methods
Newton-Raphson method

Abstract: Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT).

We present a reliable and very fast method for computing the associated spectral densities, for given architecture and initialization. This method has a controlled and proven convergence. Our technique is based on an homotopy method: it is an adaptative Newton-Raphson scheme which chains basins of attraction. We find contiguous lilypad-like basins and step from one to the next, heading towards the objective.

In order to demonstrate the relevance of our method we show that the relevant FPT metrics computed before training are highly correlated to final test losses – up to 85%. We also give evidence that a very desirable feature for neural networks is the hyperbolicity of their Jacobian at initialization, while remaining at the edge of chaos.

Free Probability for predicting the performance of feed-forward fully connected neural networks
Reda CHHAIBI'
Tariq DAOUDA? Ezéchiel KAHN3
Institut de Mathématiques de Toulouse, Université Paul Sabatier, Toulouse, France.
2 Institute of Biological Sciences, Mohammed VI Polytechnic University, Benguerir, Morocco. CERMICS, Ecole des Ponts, INRTA, Marn--l-VValle France.
reda. Aahaibb@mmah univ-toulouse... tariq. daouda@un6p ma, ezechiel kaaeheep
Abstract
Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians are modeled using free multiplicative convolutions from Free Probability Theory (FPT). We present a reliable and very fast method for computing the associated spectral densities, for given architecture and initialization. This method has controlled and proven convergence. Our technique is based on an homotopy method: it is an adaptative Newton- Raphson scheme which chains basins of attraction. In order demonstrate the relevance of our method we show that the relevant FPT metrics computed before training are highly correlated final test accuracies up to 85%. We also nuance the idea that learning happens at the edge of chaos by giving evidence that very desirable feature for neural networks is the hyperbolicity of their Jacobian at initialization.
Introduction
Neural network training and tuning can be wasteful in human and energy resources. For example 23 Table 1] show that a single GPU training can have high energetic costs. Often, this nothing compared to architecture search. In this context, obvious moonshot is estimating the performance of an architecture before training. The goal of this paper more realistic: providing a fast and reliable computational method for estimating stability before training and stability is good proxy for performance [1].
Framework: Consider feed-forward network of depth L E N, with L full-connected layers. For each depth 21, L}. the layer has activation vector Ie E R,,, where N₂ is the current width. The vector Ho No takes in the neural network's input, while æy E RN gives the output. The vector of widths is written N (No, N...... NE) and will appear in superscript to indicate the dependence in any of the Ne's. The following recurrence relation holds between 1
layers Zt = Φe (W(N) + +(()) where Φe is a choice of non-linearity applied entry-wise.
W(N) E MN,,NN-1 (R) is weight matrix and ((N) Ne is the vector of biases. We write he = ( W(N))((11+ (( for the pre-activations.
xe-1
The Jacobian computed during paok-propag be written explicitly by using the chain rule: dEL-1 Ox
J(N) = Ozo
0r 0r
DN W.,, (1.1)
dz1-1 0Z
Ozo
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

where Dy's are the diagonal matrices given by
[ 1. 2e ([helk)
(1.2)
Technically, step of gradient descent updates weights and biases following dc
(w/N) 6, (W: (.)))) O(W (N) (
(1.3)
for each L 1,.... L. Here > the learning rate and L is the loss minibatch. If the minibatch has size B E N. and corresponds a small sample ((X;, Yi); ; i= 1,..., B) of the dataset, we have L 3/ Σi (X,), Y,) Here d is real-valued distance or similarity function, the X,'s are the input vectors while the Y,'s are the output vectors (e.g. labels the case of classifier, Y; X, in the case of autoencoder etc...).
The chain rule dictates:
OC
OC dIL
0re+1
dz a(w (N) .b (N))
Oc
The a(w
dxL J (N
(1.4)
a(w (N) by
(())
i dEL-1 Oxe
((Ni)
where
OC
1
Σ (xi) (R),
(1.5)
d1 B
(N) = W DIN W 2+1
( E MN (R).
(1.6)
Therefore, for the sake of simplicity, we shall focus on the Jacobian J(N) given in Eq. (1.] since has exactly the same form the n² given in Eq. (T6)). The issue is that a large product of (even larger) matrices can easily become unstable. If many singular values are < 1 we have gradient
vanishing. If many singular values are >> 1, we have gradient explosion. Such a transition can be referred to as the edge of chaos 266
This instability is easily understood thanks to the naive analogy with the one-dimensional case. Indeed, the geometric progression g" with - oo is the archetype of a long product and converges extremely fast, to either 0 if (4l < 0 or to co if [g] 1.
Intuition.
less naive intuition consists in observing that mini-batch sampling in Eq. 1.5 very noisy. It fair to assume that 80g of the Central Limit Theorem if B is large enough or after time averaging, because of the mixing properties of SGD 6 g. Therefore, each gradient step "(w? M!NT
0c has Gaussian behavior with covariance proportional to IN, either because
BC in Eq. (.3) is approximately
Gaussian vector with covariance proportional to:
The B(NT)
(JNNN) T J(N)
The
a²
((W(N) (N)
a(w(N) NN) (.(N)))
Simplifying further, see the importance of the spectrum of (J!N)) J(N) for stability. Basically,
eigenvectors of (J(N)) T (N) are the directions along which the mne-dimensiona intuition applies.
Randomness. Starting from the pioneering works of Glorot and Bengio g on random initializations, was suggested that the spectral properties of (N) are an excellent indicator for stability and learning performance. In particular, an appropriate random initialization was suggested and since implemented in all modern ML frameworks [T
We make classical choices random initializations. The biases ((N) are taken as random vectors which entries are centered i..d. Gaussian random variables with standard deviation 0b.- For the

---

weights, we will consider the following matrix ensembles: the [W lis are drawn from i.i.d. centered random variables with variance ww and finite fourth moment as in [7].
Modeling spectrum thanks to Free Probability Theory. Now, following the works of Pennington et al. [20], the tools Free Probability Theory (FPT) can be used to quantitatively analyze the singular values of ((N) in the large width limit. The large width limit is particularly attractive when studying large deep networks, especially because free probability appears at relatively small sizes because of strong concentration properties [14], Indeed, random matrices of size 100 exhibit freeness.
For the purposes of this paragraph, we restrict ourselves to square matrices and assume N₂ = for all & = 1...., L. In fact, FPT is concerned with the behavior of spectral measures as N + oo. For any diagonalizable AN E MM((R), the associated spectral measure on the real line is:
MACNO (dx) = N Σ AN (dx)
with the 0; values written VA(N) == "(AcN))" ACNy. A fundamental assumption for invoking tools from Free
(N)'s 's being the eigenvalues of Av. For ease of notation, the spectrum of (squared) singular
Probability Theory, is the assumption of asymptotic freeness. Without defining the notion, which can be found in [16], let us describe the important computation it allows, discovered in the seminal work of Voiculescu [25]. Given two sequences of square matrices A B(N) in My (R), with converging spectral measures:
lim MAIN) = VA lim VBIN) = VB N
N
we have that, under the assumption of asymptotic freeness limN-soo V (A(N) = =A & vB where X is deterministic operation between measures called multiplicative free convolution. The X will be detailed in Section 2 The letter A (as well as B) does not correspond to a limiting matrix but to an abstract operator, with associated spectral measure μ.d. and measure of squared singular values VA- For such limiting operators, we drop the superscript (N).
Under suitable assumptions which motivated and detailed later following the works of [20 II] for all = 1,..., I,, the measures Vw:N) and V/ D will respectively converge to vw,, and VDD.. Again the Wy's and D2's 's abstract operators which only make sense in the infinite width regime. In the limit, asymptotic freeness will also hold. Therefore, we will see that the measure of interest is:
18 17
lim VJJN = My == VD2 X Iw, X ... X MVD, +00
VDD X VW1
(1.7)
The goals of this paper (1) To give a very fast and stable computation of y., in the more general setup of rectangular matrices (2) Empirically demonstrate that FTP metrics computed from v do correlate to the final test accuracy.
1.1 Contributions
We aim at streamlining the approach of Pennington et by providing the tools for systematic use of FPT. The contributions of this paper can be categorized as:
Theoretical: In Pennington et al., a constant width is assumed. We generalize the model to allow for varying width profiles, which is more inline with design practices. This requires us to develop rectangular multiplicative free convolution.
Then we propose a computational scheme for computing spectral densities, named "Newton lilypads". The method relies on adaptative inversions of S-transforms using the Newton-Raphson algorithm. If the Newton-Raphsol scheme is only local, we achieve a global resolution by chaining basins of attractions, thanks doubling strategies. As such, we have theoretical guarantees for the convergence.
Interestingly, even in the FPT community, inverting S S-transforms has been considered impossible to realize practice. As we shall see, an S-transform a holomorphic map with multi-valued inverse. In the words of (5) p.2 "the operations which are necessary in this method (the inversion of certain functions, the extension of certain analytic functions) are almost always impossible to realise practically."
3

---

This misconception led to the use of combinatorial methods based moments, or fixed-point algorithms via the subordination method [3 24]. In the ML community, Pennington et al. pioneered the application of FPT to the theoretical foundations of Machine Learning and did not shy away from inverting S-transforms. Their (0 Algorithm
Numerical:
based on generic root finding procedure, and choosing the root closest to the one found for the problem with one less layer. A major drawback this method is that there no guarantee to find the correct value, unlike our chaining which always chooses the correct branch.
Not only Newton lilypads has theoretical guarantces convergence, but it is also an order of magnitude faster (Fig. [.1p. A few standard Cython optimizations allow to gain another order of magnitude, although this can certainly be refined.
Compucttoon one)
Hypads
Hlseed cythen
pontil poittrrt
munic carb
34 16.
25
25
20 36
70
56
700
402
62.0
820
30.00
17.00
31.00
1.000
1900
Number of leyers [L]
Ramberr of dens points
Figure 1.1: Computation time (in ms) for the density of of w.r.t depth L (left) and number of density points (right). Vertical axis is log-scale. The benchmarked methods are Newton lilypads in pure Python (blue), Newton lilypads with Cython optimizations (orange). Pennington et al. Algorithm using a native root solver (green), Monte-Cario (red). While of a different nature, Monte-Carlo is given for indication.
Empirical: We analyze the correlation between the test accuracy of several randomly generated MLP architectures and the quantiles of v.,, after fixed number of epochs. The same architectures were independently trained on the MNIST, FashionMNIST and CIFAR10 datasets. We find that accuracy is strongly correlated to FTP quantiles of vs (see Table 1.1). Remarkably, the correlation is almost entirely captured by the higher quantiles see Table T for individual I factors. Scatter plots showing the distribution of test accuracy, log10 of the 90th quantile and number of parameters can be seen in Fig. 1.2 Interestingly, smaller networks can perform better than larger one provided they have more spread-out PJ distribution. This suggests that spread-out spectral distributions Fy are more desirable, provided of course we avoid the vanishing and explosive regimes. In the language of dynamical systems, we say that the Jacobian needs be hyperbolic i.e. with both contracting and expanding directions. This considerably nuances the idea that learning happens the edge of chaos. A similar point was made in the conclusion of [6], using the language of hyperbolic attractors.
For reproducibility of the numerical and empirical aspects, complete implementation is provided in a Github repository
1.2 Structure of the paper
start in Section 2 by stating facts from Free Probability Theory. Most of available in the literature, except the definition of the product of rectangular free matrices. To the best of our knowledge, this is novel. There, we establish in the rectangular setting an analogue of Eq. (I.7) in
Theorem 2.3

---

Regression factors
Corr. indicator
Dataset
MNIST
FaShionMNIST
CIFARIO
Spearman (p-value)
0.83 (3e-49)
0.79
(6e-45)
0.73 (4c-18)
90th quantile only
Pearson (p-value)
0.84 (9e-54)
0.81 (le-48)
0.74 (3e-19)
R factor
0.84
0.81
0.75
All quantiles of v/
R factor
0.89
0.87
0.83
Table 1.1: Correlation between test accuracy and FPT metrics. 200 randomly generated MLP
architectures were trained on 3 datasets, We performed linear regressions of test accuracy against quantiles of иJ. Although correlation is stronger considering all quantiles, higher quantiles are individually the most correlated (see Table T.2). Hence, the computation of Spearman, Pearson and R correlation factors between aceuracy and 90th quantile. The last row shows the correlation of accuracy against all the quantiles 10, 20, ..., 80, 90% during a multivariate regression.
Quantile of yJ
MNIST
FashionMNIST
CIFARIO
10%
0.058
0.035
-0.016
20%
0.084
0.055
0.005
30%
0.176
0.134
0.063
40%
0.204
0.156
0.089
50%
0.234
0.186
0.123
60%
0.301
0.244
0.196
70%
0.407
0.342
0.304
80%
0.597
0.539
0.498
90%
0.845
0.807
0.755
Table 1.2: Accuracy vs (logro of) a single quantile of yJ- For each dataset and each quantile, the table reports the R factor in bivariate linear regression.
In Section we explain in detail the FPT model for random neural networks. Then thanks to the results of 18 and our rectangular setting, we show that the spectral measure of the Jacobian ([N) converges to v/j and encode the limit explicit generating series Theorem 3.1 how My can theoretically be recovered.
This gives
Section AatioonFpresse presents the numerical resolution which inverts the (multi-valued) generating series. By chaining different (local) basins attractions, we obtain global resolution method. Our algorithm is detailed in Algorithm [ and Theorem 4.1 states the theoretical guarantees.
Finally
the experiment leading to Table [.] More details are given in Appendix including comments on the benchmark of Fig. [.]
2
Free Probability
2.1 Definitions and notations
Free Probability Theory provides framework to analyze eigenvalues and singular values of large random matrices. We now introduce various complex-analytic generating series which encode the measures and the basic operations on them. First, the Cauchy-Stion Stieltjes transform of μ. probability measure 14 is:
G
C+
Z
JR,
p(dv)
2-1
where C+ = {a E C 3z 0} measure can be recovered thanks to:
The transform Gy, encodes the measure μ and reciprocally, the
Lemma 2.1 Saachy-Stieltjer inversion formula Theorem 6 in [6]). We have the weak convergence of probability measures:
lim (G + ky)dr = p(dx).
5

---

MNIST: Accuracy vs (log of) 9ach FPT quantile
Log18 of paran Count
95
acy
-1.5
-3.5
8.5
1.5
2
Legi0 of 90th quentile
FashionNNIST: Accuracy (lag of) quantile
Log18 Paran
acy
-1.5
-3.5
0.5
1.5
Legi0 of 90th quantale
C1HAR16: Accuracy (log of) 90th quantile
Log18 Paran Count
racy
39
-1.5
-9.6
0.6
1.6
Leg10 of 90th quantile
Figure 1.2: Accuracy versus (logio of) 90th quantile for various datasets. From top to bottom, datasets are MNIST, FahsionMNIST, CIFARIO.
The moment generating function is
+00
mp(
My(=)= =G)))==G,())) - 1
(2.1)
zk
where for all l N, m4(n) == Je x² p(dx) is the k-th moment of μ. For μ # 8g. M2, is invertible in the neighborhood of co and the inverse is denoted by M²-") The S-transform of is defined 1+m as 5,(m) mM/)"(((m)) and is analytic in a neighborhood of m O. Furthermore, the variable
will always denote an element of C+, while the variables g and m will denote elements in the image of Gμ and Mμ. For diagonalizable matrix A(N) E .M we write SA(M = S Spa00 GA(N) == G2 " MA := Mp M
landmark result the field introduces free multiplicative convolution in a natural way, and shows that this operation is linearized by the t-transform:
Theorem 2.2 (Voiculescu, (25) Consider sequences of positive matrices, each element in M (IR)
MM LIM,NY N21), (B(M; N21)
such that:
lim (A(N) MA, N
lim PNAN) PB N+0

---

Under the assumption of asymptotic freeness for A and B(N), there exists deterministic prob- ability measure μ.a μn such that limx- B(N) PA 2 Pa. The operation is the multiplicative free convolution. Moreover
SAB(m) = (a(m)SB(m)
(2.2)
This convergence akin law of large numbers is the key ingredient which allows build the deterministic model for the haco-propagation of gradients in Eq. (1.7
2.2 Product of rectangular free matrices
As generalization of Eq. 2.2 to rectangular matrices, state:
Theorem 2.3. Let (pN)N21 (gN N221, (TN) be three sequences of integers satisfying
FN
PN. QN PN
c> 0.
N
ON
N
Consider for all N 2 1 let A B be random matrices of respective sizes pN 9N and 9N TN such that the (squared) singular laws of A B(N) converge weakly. Assuming that B(N) (B(NI) and (A(M)) T. A are asymptotically free, we have that in the limit N + oc:
S(AB)T AB()) = SAT A (cm) (m))
Proof. See the appendix, Subsection C.1
Implicitly this defines rectangular multiplicative free convolution, which could be denoted R in the spirit of the rectangular free additive convolution 4.. But, the current setting, this is not good idea. Indeed, if one defines μn Me #2 the measure whose S-transform is Syn (c-) then a quick computation shows that X is not associative, i.e., for triplet (#1, (2, μa) of probability measures and pair (c1, c2) R x R4, we generically have:
#1 Me (#2 ye p3) + (#1 Ne p2) Beg
better idea is to treat the dimension ratio e as part of the data via central extension: Definition 2.4. On the of pairs (1,,c) c) such that μs probability measure R4 and e Rt. define the operation X as:
(Mi, c1) & (Mz 02) (v,c102)
where v is the unique probability measure such that S, Sun (C2 -)S This extends the classical definition as the usual free convolution is recovered with (pi, 1) & (122 1) = (M1 p2, 1).
Such operation is associative and will allow a neat formulation of the measure of interest in the upcoming Theorem 3.1 entirely analogous to Eq. (I.7)
Theoretical resolution of the model
Width profile: Pennington et al. 20 consider N, = N for t= =,21=12, 1,2,.... N. Here, consider that the width of layers is not constant across layers, which is mostly the case in practice. Indeed, modern architectures typically have very sophisticated topologies with layers varying in widths.
Let us assume that we are in the infinite width regime in the sense that Ne - cc, for all 0, L with: N convention Ag =
de
0. And let us denote As = lim No N Nr = I 1 d with the
Nr N
2 FPT limits: Let N be a standard Gaussian random variable on the real line P (N E dz) =
/2z dx.
Here (NN is diagonal with entries ((ee))))) (see Eq. (1.2] and the pre-activations
(N) h w.,N x + ( w de + (2))
7

---

clearly depend on the previous layers. Because of this lack of independence, the standard results of FPT cannot be applied directly i.e. asymptotic freeness does obviously hold. This an important subtlety that is addressed in the upcoming Theorem 3.1 Based on an information propagation argument, the papers [21, [2 argue that the entries he behave as the i.i.d. samples of a Gaussian
distribution with zero mean and variance q²_ A basic law of large numbers applied to Eq. (1.2) gives
a limit for the empirical measure JAD, = -iimn- nomNe-tooop MD = &2 (VFN) the variance is:
Also the recurrence for
21 = fe = ow, Φe (Vo--NN)
+ obe
(3.1)
oiv No Σ (wt)) + to
with initial condition o²
Recently Pastur et al. completed this heuristic thanks to a swapping trick see [8] Lemma 3.3] and [17 Remark 3.4]. They proved that, regarding the asymptotical spectral properties of J(N), one can replace each D(ND by diagonal matrix with independent Gaussian entries V independent from the rest. In that setting, one can apply the results on products of asymptotically free matrices which were given in Section
Theorem 3.1. In terms of the rectangular multiplicative free convolution, the measure of (squared) singular values of J[N converges to
vJ (VDL: 1) (vWz, AL) (VDI, 1) (IW1 A1)
(3.2)
Moreover, the S-transform of JJ in the infinite width regime verifies
L
) = II pa (Nem) SwEw, (Ne-1m))
(3.3)
t=1
In particular, under the assumption that the entries of We are i.i.d.
I ( = II
S (Aem)
111 )
M)F)(()
m +
ow, (1 + Aem)
ow. 1 Aem)
m
S (Aem)
f=1
Proof. See the appendix, Subsection C.2
Master equation: In the end, we only need to fix width ratios and non-linearities to form M and get the master equation which we solve numerically thanks to an adaptive Raphonn scheme. The non-linearities ReLU, Hard Tanh and Hard Sine yield explicit formulas, which can be found in Table B.I of the appendix. If We has i.i.d. entries, one finds the explicit master equation:
JTJ (m).
+ 1 MS- (m)
m II Low (ce + Agm)
(3.4)
where c₁ = } when oe is ReLU,, = =Ct = OS 1 Vo
if te is Hard Tanh and = 1 if dz
Hard Sine.
4 Numerical resolution
Here we describe the numerical scheme aimed at computing the spectral density of JT J in Eq. 1.1 We use the following steps to compute the spectral density at a fixed æ E II:,:
Because of the Cauchy- StuchhySttilljee inversion formula given in Lemma 2.1 pick a small y > 0 in order (2 (z= = x + iy) The smaller the better, and in practice our method works for up to = 10-9. Figure A. shows the same target distribution but convolved with various Cauchy distributions ye where y E 10-10-1 10-1 =1 3G ēy) for different y's.
to compute: 3G )
10-4). -J. This corresponds to computing the density
8

---

Because of Eq. 22.1 we equivalently need to compute (z)..
Mfry (m) is available thanks the master equation Theorem 3.1 Therefore, we need to invert
M JTJ (m). This step the crucial part: M M is multi-valued and needs to choose the correct branch.
Algorithm Newton lilypads, chaining basins of attraction
Name: NEWTON LINLYLD
Input: Image value: Zobjectivn
E C++ (Optional) Proxy: mo) E C+ C.
Output: M (=oojeective)
Find proxy (2o, mg 0) using doubling strategy, if None given if' (zo, mo) is None then
< Zobjective
while not IS IN BASIN(Z, m) do
2 - + (2) = R(2) + (2) # Double imaginary part
end while
< NEWTON CIANTON_RAPHSONCC Guess = m)
else
(2) < (z0,mo)
end
# Starts heading towards Sobjective using dichotomy while |Zobjective 2| > 0 do
A2 Robjective
while not IS IS_IN_ [N_BASIN(== + Az, m) do
A: t- 0.5 Az
end while
+ A=
m <
Guess = m)
end while
return
4.1
Initial setup
We first use the classical Newton- Rapthnn scheme invert the equation : = f(m) where E C+ is fixed and ff is rational. A neat trick which leverages the fact that is rational and that 2 E C+ is to define:
(m(m) == P(m)/ (/1m//20 Q(z).
(4.1)
As such, we have fmm P(m)
Q(m) = (m = 0. There are several advantages of doing
that: (1) Inversion recast into finding the zero of a polynomial function. (2) Since we have
lim (x) = 0, if is large in modulus, m a natural starting point for the algorithm when
is large.
is well-known that the Newton-Raphes scheme fails unless the initial guess my E C belongs to basin of attraction for the method. And, provided such a guarantee, the Newton- Raphsso scheme is exceptionally fast with a quadratic convergence speed. Kantorovich's seminal work 1948 provides such a guarantee locally. For the reader's convenience, we give in Appendix the pseudo-code for the Neowton-Rahssee algorithm (Algorithm 2h. as well as a reference for optimal form of the Kantorovich criterion (Theorem D.I.
Therefore, we assume that we have at our disposal a function (z, m) + IS =S__N_BASII(2, m) which indicates if the Kantorovich criterion is satisfied for yp at any m E C. It is particularly easy to program with i polynomial.
4.2 Newton lilypads: Doubling strategies and chaining
Now we have all the (local) ingredients order to describe global strategy which solves in m E the equation ys (m) 0.

---

First, one has to notice that this problem is part of a family parametrized by E C+. And the solution m for large. Therefore. one can find a proxy solution for E C+ high enough. This is done thanks to doubling strategy until a basin of attraction is reached.
Second, if proxy (z, m) is available, can the Newton-Raphson algorithm to find solution (z + Az, m + Am) starting from m. To do so, we need A small enough. This on the other hand is done by dichotomy.
Tying the pieces together allows to chain the different basins of attraction and leads to Algorithm [] Notice that in the description of the algorithm, chose to make implicit the dependence in the function f. since it is only passed along as a parameter. Technically, f is parameter for all three functions NEWTON RAPHSON, IS IN BASIN, NEWTON LILYPADS.
The discussion leading to this algorithm, combined with the Kantorovich criterion yields:
Theorem 4.1. Given f m + M( (m) and E C+, Algorithmm | has guaranteed convergence. Moreover it returns (z) Lee. the (unique) holomorphic extension of the inverse of f in the neighborhood 0.
On the experiment
To leverage the numerical scheme, we designed the following experiment whose results are reported on Table[[.I] Consider a classical Multi-Layer Perceptron (MLP) with L = 4 layers, feed-forward and fully-connected with ReLU non-linearities. The MLP's architecture is determined by the vector À (Au,>1, , ÀL) while the gains at initialization are determined by the vector = .2.............
By randomly sampling the vector A, we explore the space of possible architectures. In other to have balanced architectures, we chose independent X, 's with (A₂) 1. Likewise, we also sample different possible gains. Hence we find ourselves with several MLPs architectures each with it's unique initialization. The spectral distributions are computed thanks to our Algorithm [
As shown by the parameter count in Fig. 1.2 some architectures are quite large for task at hand. To guarantee convergence despite the changing architectures, we intentionally chose low learning rate and did 50 epochs of training. For 200 MLPs, the experiment takes 10 hours of compute on consumer GPU (RTX 2080Ti). This is be contrasted with less than one minute of CPU compute for the spectral measure V/J.
To control for any stochastic variability, also trained multiple instances for each MLP, and the shown results are averages between a few runs. Finally we calculate the correlations between the accuracy the test set and percentiles of the spectral distribution y.. Notice that, posteriori, the FPT regime justified because the results are coherent despite various width scales.
Conclusion
In summary, this paper developed FPT in the rectangular setup which is the main tool for a metamodel for the stability of neural network. Then we gave a method for the numerical resolution both fast and with theoretical guarantees. Finally, confirming the initial claim that stability is good proxy for performance, we empirically demonstrated that the accuracy of a -erd-fd neural network after training highly correlated to higher quantiles of the theoretically computed spectral distribution using this method.
Regarding the importance of hyperbolicity, surmise that few large singular values allow the SGD to escape local minima without compromising overall stability.
A challenging problem would be to accommodate for skip-connections. In this case the chain rule used for back-propan propppropagttion changes fundamental way. The Jacobian cannot approximated by a simple product of free matrices as the same free variable will appear at multiple locations.
7
Acknowledgements
R.C. recognizes the support of French ANR STARS held by Guillaume Cébron as well as support from the ANR-3IA Artificial and Natural Intelligence Toulouse Institute.
10

---

References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Shemr Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.
[2] Octavio Arizmendi, Pierre Tarrago, Carlos Vargas, et al. Subordination methods for free deconvolution. In Annales de I'I l'Insiiuu Henri Poincaré, Probabilités et Statistiques, volume 56, pages 2565-2594. Institut Henri Poincaré, 2020.
[3] Serban T Belinschi and Mireille Capitaine. Spectral properties of polynomials in independent wigner and deterministic matrices. Journal of Functional Analysis, 273(12):301-3363 2017.
[4] Florent Theory and Related Fields, 144(3-4):471-5555 2009.
Rectangular random matrices, related convolution. Probability
[5] Florent Sennach--Georger and Mérouane Debbah. Free deconvolution: from theory to practice. Paradigms for moalogccally-nspireed Autonomic Networks and Services, pages 201-224, 2008.
[6] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
[7] Benoit Collins and Tomohiro Hayase. Asymptotic freeness of layerwise jacobians caused by invariance of multilayer perceptron: The haar orthogonal case. arXiv preprint arX1:22103.1366, 2021.
[8] Jonas Geiping, Micah Goldblum, Phillip E Pope, Michael Moeller, and Tom Goldstein. Stochas- tic training is not necessary for generalization. arXiv preprint arXiv:2109.141 14119, 2021.
[9] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256, 2010.
[10] WB Gragg and RA Tapia. Optimal error bounds for the newton-kantorovich theorem. SIAM Journal Numerical Analysis, 11(1):10-13, 1974.
[11] Boris Hanin and Mihai Nica. Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, pages 1-36, 2019.
[12] Nicholas J Higham. Accuracy and stability of numerical algorithms. SIAM, 2002.
[13] LV Kantorovich. On newton's 's method for functional equations. In Dokl. Akad. Nauk SSSR, volume 59, pages 1237-1240, 1948.
[14] Cosme Louart, Zhenyu Liao, Romain Couillet, al. A random matrix approach to neural networks. The Annals of Applied Probability, 28(2): 190-1248, 2018
[15] Mylène Maida, Tien Dat Nguyen, Thanh Mai Pham Ngoc, Vincent Rivoirard, and Viet Chi Tran. Statistical deconvolution of the free fokker-planck equation at fixed time, arXiv preprint arXiv:2006.1 1899, 2020.
[16] James A Mingo and Roland Speicher. Free probability and random matrices, volume 35. Springer, 2017.
[17] L. Pastur and V. Slavin. On random matrices arising in deep neural networks: General i.i.d. case, 2020.
[18] Leonid Pastur. random matrices arising in deep neural networks. gaussian case. arXiv preprint arXiv:2001.06188, 2020.

---

[19] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Viiioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[20] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pages 1924-1932, 2018.
[21] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. arXiv preprint 1rXiv:1606.05340, 2016.
[22] Samuel Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep informa- tion propagation. arXiv preprint arXiv:1611.01232 2016.
[23] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 02243, 2019.
[24] Pierre Tarrago. Spectral deconvolution of unitarily invariant matrix models. arXiv preprint Stt1:20006093550 2020.
[25] Dan Voiculeseu. Multiplication of certain non-commuting random variables. pages 223-235. JSTOR, 1987.
[26] Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. Advances neural information processing systems, 30, 2017.
[27] Lin Zhang, Ling Feng, Kan Chen, and Choy Heng Lai. Edge of chaos as guiding principle for modern neural network training. arXiv preprint arXiv:2107.09437. 2021.
12


Reviews:
[{"review_id": "vXEpEDXBWn", "reviewer": "Reviewer_GQUV", "summary": "This work extends the free probability theory in the rectangular setup and gives a new method for computing the spectral density of the Jacobian of a general feed-forward neural network with theoretical guarantees. In addition, it also empirically demonstrates that the accuracy of a feed-forward neural network after training is highly correlated to higher quantiles of the theoretically computed spectral distribution.", "questions": "- How is the 90th quantile selected? What about other quantiles or other summary statistics? Is a single quantile alone sufficient to quantify the model performance?\n- In Line 167, it states that \"$N_{l-1}/N_l \\rightarrow \\lambda_l \\ge 0$\". Should this be \"$\\ge$ 0\" or \"> 0\"?", "limitations": "The authors addressed the limitation in the final paragraph of the manuscript. Regarding the potential societal impact of this work, I do not see there is any.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work is well-motivated: focusing on understand the spectrum of the Jacobian for varying architectures and initializations, providing an effective computation method, and preventing unnecessary tuning of neural networks based on these results.\n- On the theoretical side, the authors generalize previous result to rectangular matrices and provide an associate multiplicative convolution operation, which appears novel.\n- On the computation side, they present a new method for computing the spectrum leveraging the Newton lilypads, which not only providing theoretical guarantees of always choosing the correct branch but also much faster comparing to existing methods.\n- To demonstrate the usefulness of the proposed method, the authors show that the accuracy of a feed-forward neural network after training is highly correlated to higher quantiles of the theoretically computed spectral distribution.\n\nWeaknesses: the main weakness of this paper is the lack of sufficient evidence to support the claims of solving the neural network tuning problem. \n- As suggested in the title and the Intro, a main focus of this work is trying to \"estimate the performance of an architecture before training\", thus providing \"a solution to the problem of tuning neural networks\". However, there is only indirect evidence that the correlations between the accuracy on the test set and the 90th percentile of the spectral distribution is considerably high for various 4-layer MLPs trained on the MNIST benchmark. This is indeed intriguing, but it is far from a solution to network tuning. The choice of 90th percentile seems arbitrary. Is the 90th quantile the best summary statistics of the spectral density for estimating model performance? What about other quantiles? Or this particular quantile alone is sufficient to quantify the model performance?\n- There is lack of discussion of how to apply this metric, 90th percentile of the theoretically computed spectral distribution for example, in practice. Also, how to avoid the vanishing and explosive regimes? It would also be nicer to provide more results on multiple benchmarks.\n- In Section 1.1, the authors argued that \"smaller networks with more spread-out $\\nu_J$ perform better than larger counterparts\". But I could not find any supporting evidence for this. In addition, the quantitative results are only concerned with the 90th quantile with little discussion/summarization about the spread-ness of the whole spectral distribution. \n- There is existing work discussing the relationship between trainability and generalization, [1] for example. It argues that \"trainability and generalization are distinct notions that are, at least in this case, at odds with one another. Indeed, good conditioning of the NTK (which is a necessary condition for training) seems necessarily to lead to poor generalization performance. It will be interesting to see whether these results carry over in shallower and narrower networks.\" The last question seems can be answered from this work. It would be interesting if the authors could discuss more on this.\n\nTypo:\n- In Line 186, it should be $M_{J^TJ}$ instead of $M_J$.\n\n[1] L. Xiao, J. Pennington, and S. Schoenholz. Disentangling trainability and generalization in deep neural networks. ICML 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rK6X4DqhQ21", "reviewer": "Reviewer_PUn2", "summary": "UPDATED SCORE AFTER REBUTTAL.\n\nThe pioneering work of Pennington et al. used free probability to predict the spectrum of the input-output Jacobian. The conditioning of this matrix is known to be important for training stability. The authors generalize those results to sequences of width ratios that need not be identical. Despite knowing the self-consistent equation for the spectrum, obtaining numerical predictions are known to be numerically challenging. The authors propose a new method, which is guaranteed to converge, to numerically solve for the spectrum. Experiments are also performed to show that the 90th percentile of the spectrum is highly correlated with the loss of a fully connected neural network trained on MNIST.", "questions": "- Why is there a line of points in Figure 2 at $x\\approx-1.6$?\n- Correlation for a random sample of architectures is interesting, however, there is still a question of whether this metric could be used iteratively in some kind of hyperparameter search.\n- There have been other works using free probability to study ML. Can this algorithm be used for the equations in any of those cases?\n- Were other functionals of the spectrum than the 90th percentile considered?", "limitations": "No concerns about negative societal impact.\n\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- To my knowledge, the lilypad algorithm is an original contribution even in the free probability community and not merely free probability applied to ML. The speed-ups provided over the algorithm of Pennington et al. are significant. Combined with the convergence guarantee the lilypad algorithm appears preferable.\n- The paper is clear and easy to read. In particular, the overview of free probability is nice and an accessible introduction.\n- It's great to see open sourced code released with the paper.\n\nWeaknesses\n- The paper's significance is hurt by only considering the MNIST dataset in the experiments. It should be easy to try CIFAR10 for example to check whether the correlation still holds. This is important given how easy MNIST is as a benchmark.\n- Similarly only FC architectures are considered. Even the activation function is unchanged.\n- The main contributions are interesting mathematically but they might not be of great interest to the NeurIPS community, given the limitations above in the more practical experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QUnmChp4I1x", "reviewer": "Reviewer_whcH", "summary": "In this paper, the authors considered the problem of computing the limiting eigenvalue/singular value distribution of the input-output Jacobian matrix of a fully connected \"deep\" neural network model having random weights, in the limit where the network widths go to infinity.\n\nThe contribution of this paper is threefold:\n1. extend the analysis proposed by Pennington et al. in a sequence of papers to the setting of non-square weight matrices by introducing the tool of \"rectangular multiplicative free convolution\"; and\n2. propose a more accurate and faster computational scheme to solve the limiting distribution (see Figure 1), with theoretical guarantees; and \n3. apply these theoretical and computational results to empirically show that the above eigenvalue distribution is closely connected to the accuracy of networks, at least in some cases (see Figure 2).\n\n", "questions": "Some minor issues/remarks:\n* The claim \"stability is a good proxy for performance\" in the introduction should be supported by theoretical, empirical, and/or related previous efforts\n* Please provide some references for the term \"edge of chaos\" in line 44, which may not be familiar to all the readers.\n* Please provide some references for the discussions in the paragraph AFTER line 47.\n* line 74: I do NOT understand the claim \"In the limit, asymptotic freeness will also hold\": this is in fact an ASSUMPTION, right? And the notion of \"asymptotic freeness\" only makes sense asymptotically? Or did I misunderstand something?\n* Just a side remark, but I believe it is of interest to mention after Equation (7) whether the operator of multiplicative free convolution is \"exchangeable\".\n* The authors constantly mentioned in the paper \"Pennington et al.\", but I personally believe it is more clear and more precise to cite the exact paper and the sequence of papers that this work is building upon.\n* I wonder if the conclusion drawn from Figure 2 remains valid on more involved ML problems such as CIFAR10/100 and ImageNet image classification.\n* line 155: is not associative, i.e., for ...\n* line 188: I believe ReLU is more commonly used than ReLu.\n* Please provide a few references for the paragraph from line 212 to 217 for better readability.", "limitations": "The authors have adequately addressed the technical limitations of their approach.\nThis work is mainly theoretical and I do not envision any potential negative social impact of this work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: Despite having some knowledge of the theory of large dimensional random matrices, I am not an expert in free probability theory and it is less evident for me to evaluate the contribution of this work.\n\nThe first contribution of this work is of significant practical interest, and the second contribution of this work is, to the best of my knowledge, long believed to be challenging in the field, and the proposed result is worth publishing.\n\nWith that said, I strongly recommend that:\n1. the Area Chairs and Senior Area Chairs could find an external reviewer for this paper (I can propose a few names if that helps) to better evaluate the (theoretical and computational) contribution of this work; and\n2. the authors submit a longer and more detailed version of this paper to a stats/applied math journal since the proposed approaches are of independent interest.\n\n**Weaknesses**: The paper, in its current form, is NOT easily accessible to a large AI/ML audience and should be carefully revised and better organized: for instance, most of the contents in the introduction are in fact \"preliminaries\", and to introduce the concept of multiplicative free convolution the authors discussed the case of square weight matrices but the scenario considered in this paper is in fact \"varying width profiles\". In short, the paper can be reorganized to be made clear.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JI176pdpoWH", "reviewer": "Reviewer_1DWD", "summary": "The paper presents an extension of the work by Pennington et al. 2018 for the spectrum of the Jacobian in deep fully connected neural networks at initialization. The result presented in the paper requires the numerical computation of the S-transform and the authors presented a method that out-performs the proposals from Pennington et al. 2018. ", "questions": "Please see the section \"Strengths And Weaknesses\"", "limitations": "The limitations have not been addressed. I commented on this point in the section \"Strengths And Weaknesses\".", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper presents an improvement over the results of Pennington et al. 2018 to more realistic settings.\n\nWeaknesses:\n* The improvement is slight. Although I agree with the authors that the settings of Pennington et al. are far from the DNN used in applications, the same critique applies to their setup: they only consider dense layers. \n* I couldn't understand whether they found situations where their generalisation leads to architectures that outperform the one identifiable using Pennington's result. \n* The section on the empirical results shows tests on the MNIST dataset. However, on this dataset even a single-layer network can achieve more than 90% accuracy, I honestly find that section not significant unless they show results on simple but more complex datasets (for instance fashion MNIST). \n* Overall I found the paper hard to read because it follows a rather unconventional structure. I would suggest shuffling and re-adapting some subsections of section 1 in order to be closer to other papers in ML conferences. For instance, without the intension of forcing this order in particular, the following order seems more natural to me:\n   Introduction ->  Randomness -> Modelling -> Contributions -> Structure of the paper -> (maybe this part could go in section 3) Framework -> intuition (this doesn't seem necessary) \n* It would be good to add a \"related work\" section. The work from Pennington is from 2018, I believe progress has been made since then and probably they should be mentioned. \n* Connected with the assumption made in section 3 on the equivalence to Gaussian random variables, I would point out the work below in supports \"Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mezard, Lenka Zdeborova Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, PMLR 145:426-471, 2022.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-kco8550W7j", "reviewer": "Reviewer_Qq6Q", "summary": "This paper deals with the multiplicative case of the free rectangular convolution and its application to neural networks. \n\nThe multiplicative case of the free rectangular convolution, contrarily to the additive one, comes down, as it is noticed i Section 2 of the paper, to the square case (ie to the case of square matrices). The numerical computation algorithm the authors give was a missing pice in this pipeline and it is valuable.\n\nI thus recommend acceptation.", "questions": ".", "limitations": ".", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper deals with the multiplicative case of the free rectangular convolution and its application to neural networks. \n\nThe multiplicative case of the free rectangular convolution, contrarily to the additive one, comes down, as it is noticed i Section 2 of the paper, to the square case (ie to the case of square matrices). The numerical computation algorithm the authors give was a missing pice in this pipeline and it is valuable.\n\nI thus recommend acceptation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept