When are smooth-ReLUs ReLU-like?
Authors: Ermal Rrapaj
Luca Celotti
Qiyao Wei
Martin Magill
Venue: Submitted to ICLR 2023 (2023)
Keywords: ReLU
SWISH
GeLU
Critical Initialization
Fully Connected Neural Networks
Deep Networks

Abstract: ReLU is one of the most popular activations in deep learning, especially thanks to its stabilizing effect on training. However, because it is non-differentiable at the origin, it complicates the use of analysis methods that examine derivatives, such as the Neural Tangent Kernel (NTK). Many smooth relaxations try to retain the practical benefits of ReLU while increasing network regularity. Although their success has ranged widely, some notable architectures (e.g., the BERT family) do utilize them. We present a theoretical characterization of smooth-ReLUs within fully-connected feed-forward neural networks. In addition to the well-known SWISH and GeLU, we introduce  GumbelLU, AlgebraicLU, and GudermanLU, as new relaxations. All these activations can be characterized by a positive temperature parameter which we can lower to  continuously improve the approximation. By studying the interplay of initialization schemes with temperature, we confirm that when these relaxations converge uniformly to ReLU, the statistical properties of the corresponding neural networks at initialization also converge to those of ReLU networks. Moreover, we derive temperature-dependent critical initialization schemes with which networks based on these activations exhibit stable ReLU-like behavior at any temperature. Finally, we empirically study both classes of networks on MNIST and CIFAR-10 in the full-batch training regime. We show that, while all networks exhibit very similar train loss trajectories at criticality, smooth-ReLU networks feature differentiable NTKs throughout training, whereas ReLU networks exhibit stochastic NTK fluctuations. Our results clarify how smooth-ReLU relaxations reproduce the practical benefits of ReLU in everywhere-smooth neural networks.

WHEN ARE SMOOTH-RELUS RELU-LIKE?
A Anonymous authors
Paper under double-blind review
ABSTRACT
ReLU is one of the most popular activations in deep learning, especially thanks to its stabilizing effect on training. However, because it is non- differentiable at the origin, it complicates the use of analysis methods that examine derivatives, such as the Neural Tangent Kernel (NTK). Many smooth relaxations try to retain the practical benefits of ReLU while in- creasing network regularity. Although their success has ranged widely, some notable architectures (c.g., the BERT family) do utilize them. We present a theoretical characterization of smooth-ReLUs within fully-connected feed- forward neural networks. In addition to the well-known SWISH and GeLU, we introduce GumbelLU, AlgebraicLU,, and GudermanI.U, as new relax- ations. All these activations can be characterized by a positive temperature parameter which we can lower to continuously improve the approximation. By studying the interplay of initialization schemes with temperature, we confirm that when these relaxations converge to ReLU, the statistical prop- ertics of the corresponding neural networks at initialization also converge to those of ReLU networks. Moreover, we derive temperature-dependeer critical initialization schemes with which networks based on these activa- tions exhibit stable ReLU-like behavior at any temperature. Finally, we empirically study both classes of networks on MNIST and CIFAR-10 in the full-batch training regime. We observe faster training dynamics of smooth- ReLU networks with our proposed initialization instead of the standard one. While all networks exhibit very similar train loss trajectories at criticality, smooth-ReLU networks feature differentiable NTKs throughout training, whereas ReLU networks exhibit stochastic NTK fluctuations. Our results clarify how smooth-Roo ReLU relaxations reproduce the practical benefits of ReLU in everywhere-smooth neural networks.
INTRODUCTION
In recent decades, deep learning has shown tremendous success e.g. computer vision, natural language processing, and drug discovery (LeCun et al., 2015). For instance, Effi- cientNet (Tan and Le, 2019) achieved state-of-the-at performance in image classification on CIFAR100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009). In natural language processing, GPT (Radford et al., 2018) and its newer versions (Radford et al., 2019; Brown et al., 2020) were capable of producing human-like responses to various reading compre- hension tasks. As the name suggests, deep learning neural architectures are composed of many layers sequentially applied. The most basic architecture, the fully connected feed- forward network (FFN), consists of an alternating sequence of linear layers and non-linear layers called activations. One of the most popular activations is ReLU (Jarrett et al., 2009; Nair and Hinton, 2010), given its computational simplicity, gradient stability, and expres- sive power (Nair and Hinton, 2010; Raghu al., 2017). Beyond ReLU, EfficcinttNt and others use SWISH (Ramachandran et al., 2017; Elfwing et al., 2018; Alcaide, 2018; Chieng et al., 2018; Howard et al., 2019). GPT-2 and ALBERT (Lan et al., 2020) employ the GeLU activation (Hendrycks and Gimpel, 2016). Both are smooth approximations to ReLU, with the advantage of the existence of higher-order derivatives to study their propertics (Hanin and Nica, 2020b; Li et al., 2021). Activation smoothness is required to ensure smoothness of the network's input-outpi mapping: this is turn necessary for certain applications (e.g., physics-informed neural networks and neural network methods to solving differential

---

equations require the network output to be smooth function of its imput, (Raissi al., 2019)), certain architectures (e.g., the existence of neural ODEs is typically proven assuming smooth activations (Chen et al., 2018)), and for certain theoretical analysis techniques (e.g., differential Neural Tangent Kernel (NTK) approaches Roberts et al. (2022)).
Our focus is to understand the properties of smooth ReLU relaxations, that call smooth- ReLUs, of the type Ør(z) = za(z/T), with T a positive temperature parameter and a(-) any sigmoid function ranging from to 1. As the temperature is lowered, the smooth function a(z/T) converges to the Heaviside function (x)) pointwise. Different choices of a(-) correspond different smooth approximations to ReLU. The well known SWISH and GeLU can be represented this way, and we introduce new activations with this property. Inspired by recent. works based on renormalization group and field theory (Roberts al,, 2022), we study the stability of information propagation through the layers of FFNs with smooth-ReLU variants. In particular, we study statistical properties over random network initializations, compare initialization and training dynamics of various smooth- smooth-ReLUs, and analyze to what extent they echo ReLU. Initializing weights and biases by sampling a standard normal distribution results in the exponential explosion of the variance of the representations. Weight initializations whose variance is proportional to the inverse the width can train deeper linear networks, more reliably and rapidly, by keeping the variance of representations and gradients constant with depth. However, for non-linear activations, an additional proportionality factor, that is activation dependent, is necessary to obtain depth stability (He et al., 2015; Roberts et al., 2022). Based on this observation, we quantify how similar different smooth-ReLU functions are to ReLU by comparing the variance of network representations at initialization. We also compare the behaviour of training loss during full-batch training. To gain further insight into training dynamics, we also use the NTK theory (Jacot et al., 2018). In recent years, kernel methods have found wide use to theoretically understand neural networks' performance (Allen-Zhu et al., 2019; Zou et al., 2020; Liu et al., 2022; 2020; Yang, 2020). A flurry of works have extended the NTK analysis to all kinds of neural networks, with Arora et al. (2019) extending NTK to convolutional networks, and more recently Feng and Kolter (2020) investigating the NTK analysis for large-depth limits in Deep Equilibrium Models Bai et al. (2019).
Our work aims to improve understanding of the expressivity and training of fully- connected deep smooth-ReLU networks and devise simple prescriptions to improve them. Specifically, our contributions are as follows:
Derive conditions of existence of stable initialization schemes for smooth-Rel.e ReLUs at any temperature, and provide code for computing the necessary hyper-parameters. Introduce GumbelLU, AlgebraicLU and GudermanL We note that, in certain aspocts of the analysis, GudermanLU is closer to ReLU than the other smooth- ReLUs analysed, including SWISH and GeLU.
Demonstrate experimentally that representation variance and NTK of smooth- ReLUs resemble those of ReLU under the stable initialization scheme we provide, for any lemperature. Also, the same applies for very low lemperatures with standard initialization.
Show that smooth-ReLUs typically have faster training dynamics with our pro- posed initialization, and stable and continuous NTK updates. ReLU's NTK shows stochasticity instead. We observe that the smooth-ReLU NTK updates under our stable initialization are similar for different temperatures.
2 PRELIMINARIES
2.1 INITIALIZATION COVARIANCE
To better undorstand the trainability of FFN, we pay attention to how the magnitude of the input propagates through the layers. Let's denote the layer preactivation as,
Σ
W ij
(
o
(1)
2

---

where i.j denote neuron indices, 0% data indices, and and n denote depth and width respectively. The activation function denoted by o. We assume the network weights and biases are drawn from normal distributions with means zero and variances E (000040
03
8, and (w()) w Bujz CW /m--1- For simplicity, we assume all layers to have f1n1
equal width ny n, possibly different from the input size no, and the same initialization paper-parameters, i.c. c. C, and CW Cw.
We are interested in the magnitude of the second order correlation metric defined as
Σ j=1
(1) (1) j:a jjaz/
(2)
12
where (.) denotes ensemble averaging with respect to the initializations of the weights of the architecture, and a denote different imput data samples for different In simple terms, one has to sample the network parameters from the normal distribution multiple times and then compute population statistics. Intuitively, this metric measures magnitude change per layer of the layer output, or representation. Given the progressive transformations through layers, the following recursion is satisfied
k((+1) = Cw (o(20, (o K= k
+ Cb
(3)
as shown in Theorem 2.2 in (Hanin, 2022). The subscript. K indicates integration with Gaussian distribution of zero mean and K variance, see App. A. For clarity, we omit input subscript, unless required. When this metric remains constant magnitude with depth, we call the architecture stable, and the initialization as critical.
As an illustration, we take into consideration linear networks, where the activation the identity function. In these networks the metric behaves like a scalar dynamic system. The resulting representation variance is (([+1) Ow K!) C.. Tf Cw > the metric grows exponentially with depth. Instead, if Cw < 1, the metric vanishes exponentially to zero with depth, eliminating any data-dependence in the outputs. Finally, if Cw = (Glorot and Bengio, 2010), it prevents both, the exponential explosion and vanishing of the variance. In other words, it critical initialization. For the same reason, in a non-linear neural network, if the metric increases or decreases exponentially, the variance either diverges to infinity or converges to zero. Both outcomes are undesirable they lead to exploding or vanishing representations and gradients, and make training very difficult.
In addition, one may be concerned with higher order correlations of the activation and how they grow with depth. For instance, in the infiitte-depth-and-vittt limit for fully connected networks with ReLU activation, the statistics is characterized by the log-Gaussian distribu- tion (Hanin and Nica, 2020a). Here we focus on the l/n & 1 regime with / >>> 1, where we can safely assume that the quadratic moment dominates the distribution (Roberts et al., 2022). In App. H we compute higher moments and confirm this assumption experimentally.
2.2 NEURAL TANGENT KERNELS
The Neural Tangent Kernel (NTK) theory is widely used to study the dynamics of infinitely- wide deep neural networks under gradient descent (Jacot et al., 2018; Seleznova and Ku- tyniok, 2020). Here we focus on the finite width version defined as,
Oians,an = - 200.000 Fisay Is Fjz0
(4)
where à represents the parameters of the network, and we sum over all the intermediate layer parameters. For the samples collected we compute the ensemble average,
θ (1 Σ (0hanshol)
(5)
3

---

Critical initialization dynamics of reasonably large neural networks can be well- approximated by results corresponding to the NTK regime (Roberts et al., 2022). Although the true optimal initialization schemes will differ from those derived based on the infinite- width approximation, the corrections will generally be of order (l/n). For networks with aspect ratios I/n < 1, the critical initialization schemes derived in this work will be very good approximations, as demonstrated in the following sections. Introducing a new index t to indicate training iteration, we define the NTK update as
A -
(6)
In general, during training, there is an initial phase of rapid decrease in the loss function followed by a longer phase of very small updales. Based on this observation, and since A related to the parameters updale, we expect it to first increase in the initial epochs and proceed to decrease in magnitude learning improves. As such, analysing this 'observable' gives another useful indication of whether training converges, in addition to the loss function curve. For a very small learning rate, we have dO/dt s AO,, and we expect it to be continuous for smooth functions, but not so for ReLU which is discontinuous at zero. As our main focus is the study of the initilization and training of fully connected networks with finite width and depth, our NTK is dynamical and dependent the training iteration. In other words, it does not only apply to initialization statistics as can be seen from Setion 4.3.2. In addition, the differential of the NTK and the Fisher Information Matrix (FIM) (Fisher, 1920) can be useful in developing new training methods (Amari, 1998). They would further display the finite size effects of width and depth but are beyond the scope of this first work.
THEORY FOR SMOOTH-RELU CRITICAL INTTIALIZATION
As mentioned, we consider the initialization to be critical when it allows the network to keep the variance constant with depth, i.e. K(({2)) K!0 K* K being the seedtpoint& value (Glorot and Bengio, 2010; He et al., 2015; Roberts et al., 2022). Here, we are interested in keeping the diagonal components of K-19) constant. In App. and App. C, we prove the following Lemma on the conditions that have to be satisfied for a network with this type of activations to admit critical Gaussian initializations:
Lemma (Criticality Condition). A FFN nelwork with aclivation a(z) za(z) admils crilical imilialization scheme if and only if the following condilions are salisfied
((za(2) = ==22((2)0""2));
22(222 and a(=)(a(2) + zal(2)) are sub-Gaussia (lim,-+to f(2)e==212k = 0)
Notice that even if the criticality exists, it might have undesirable K O, it is the case for a(z) = arctan(>) function, see App. F. Therefore:
Lemma 2 (Zero Criticality). For neural nelworks with activations of the form za(z/T), the condition that a(z/T) > H(z) as T - 0 is not sufficient. to ensure the exislence of critical initialization scheme with a non-zero fized point K* # #0.
We prove that the critical covariance and hyper-parameters can be uniquely determined for all temperatures, if they are known for a specific temperature, which we chose to be T = for reference:
Lemma 3 (Temperature Criticality Smooth ReLU). Tf the activation ((2) 20(2) is critical at K* for (Ct,, C+)), then the activation Ør(2) = za(z/T) is critical at K₃ = K-T2 for (Cw.ry CB.r) (Cw, T)C)).
A simple consequence is that both the critical representation covariance K* and bias variance C₁ tend to zero as we decrease the temperature:
Corollary 1. limy- Kj- = limy-40 Ch, = 0.
This is an interesting outcome if we keep in mind that in ReLU networks any value of K serves as fixed point when critically initialized with Cw 2 and C. = 0 (He et al., 2015),
4

---

known as He initialization. For smooth-ReLU networks the critical variance tends to zero as the temperature approaches zero and the activation becomes closer in shape to ReLU. However, we show in the next section in Fig. 1, the scaled layer update, AK/K*, shows the same dependence at both T = 0.1 and T = 1. In other words, the scaled quantities in this figure have temperature dependence. Since K is proportional to T is the layer update proportional to T As such, the AK tends to zero at low temperatures, effectively making every K behave like fixed point at =
In Table 1 we show several differentiable relaxations of ReLU, in addition to the well known SWISH, GeLU and Mish (Misra, 2019), together with their critical variances and hyper- parameter initializations. These values were obtained for infinitesimal variations as shown in App. A. App. we compute analytically the critical point for GeLU. We introduce GumbelLU, AlgebraicI,U, and GudermanL.U, by simply observing that there are many func- tions that can serve as smooth relaxations of the Heaviside function with a temperature parameter. Strictly speaking, these activation functions are different from ReLU, judged for example by Cir, which for ReLU has an exact value of Ct = 2, while it is very close to 2 but never exactly so for its relaxations. ELU (Clevert al., 2015), SoftPlus (Dugas et al., 2000), and the exponential activations cannot be tuned to criticality, since they do not satisfy the first condition of Lemma 1, and thus, are not present. in the table. However, the formulas we use to calculate Cw and Cg are based on first-order perturbation theory, and the inclusion of higher orders could bring Civ 2. As we show in this work, the difference in Cw is rather small for practical purposes. The last column comments on the stability of the critical point, and semi-stable refers the fixed point being stable only for perturbations to the left or right, but not both, see App. B.
Activation
a(u)
K
C
Cw
Stability
SWISH
(1 + (xxp(-u)
14.320
0.555
1.988
Semi-stable
GeLU
+ Erf( (
1/4 (3 + v17)
0.173
1.983
Semi-stable
GumbelLU
exp (- exp (-u))
21.123
0.606
1.988
Semi-stable
AlgebraicLU
Vu)++ +1
20.210
0.334
2.005
Unstable
GudermanLU
Tan (Lanh (u)
3.154
0.103
1.990
Semi-stable
Mish
tanh(In(1 1 + exp(u)))
1.670
0.094
2.013
Unstable
ReLU (He)
Heaviside(u)
2
Stable
Table 1: Critical hyper-parameterr for different smooth-ReLU activations. The more similar
C and Cw are to their ReLU counterpart, the more ReLU-like the activation can be considered to be in the low temperature limit or when critically initialized, see App. J.
4 NUMERICAL RESULTS
4.1 METRIC LAYER UPDATE
To better understand how the input magnitude changes with depth, we define the layer update of the metric as AK = K'-1 K' where Cit and C are tuned to criticality for given temperature. The experimental results in this and the following subsection were computed from 20 ensembles of 1000 samples each. The input data was sampled from a standard normal distribution and kept the same across all architecture samples, that resulted in K spanning values below K up to 100K" In Fig. we compare the experimental layer update, denoted by points with one standard deviation uncertainty bands, and theoretical prediction in dashed lines, see App. A, for SWISH and GeLU, T = 0.1, 1, and depth = 10. The horizontal axis denotes the magnitudes of the output of the last layer. As the figure confirus, the theoretical and experimental re-scaled update match, for all temperatures in the entire domain. As shown in the App. B, very near K* there another numerically critical point, except for AlgebraicLU, and both points scale together with temperature. There is no qualitative difference between various temperatures and only at T = 0 these two fixed points both equal to zero. As a reminder, for ReLU, once we set Cw 2, Cn 0,
5

---

any value of K serves fixed point since there is variance magnitude change between layers. For all the activations in Table 1, AK & K for K > K or K < K" so the update remains exponential in nature, but the exponent is rather small value. In practice, deep, but not terribly deep, neural architecture, properly initialized, should be both expressive and trainable. As additional confirmation, initialized at the critical parameters of the two other activations. It is remarkable how well matches them throughout the domain, with the biggest discrepancies at small values of K", in the insct.
also compute the layer update for ReLU
THAI
0.0
Witth 320
RALU (G4LU)
AK K'
Betl (SWIS1))
-0.2
SWISH
GEL
0.05
2.5
5.0
10.0
10
1b
40
K / K
Figure 1: Layer update experimental results, for == (0.1, 1), critical initializations, for the SWISH and GeLU activation functions. The thcorctical results in dashed lines match surprisingly well the experimental values. For GumbelLU see App. I.
To compare critical and He initializations in practice, in Fig. we depict the layer update for He, at depth / = 10 and width n = 320, at the two temperatures from the previous figure. As a reminder, at they have their natural shape. At T 0.1 we recover the stable critical behavior of ReLU for all three activations. The layer update is essentially zero within the uncertainty band, which can be further reduced by increasing sampling size. At T I the two activations are very different from ReLU. For both temperatures, the layer update is negative, which indicates that the magnitude decreases with depth. In summary, ReLU networks initialized with smooth-Rae U critical parameersssssers behave like them except at very low values of K. Also, mmooth-ReLUs. initialized with ReLU critical pyper-parameters behave like ReLU only at low temperatures. Usooth-ReLUs initialized with their respective hyper-parameters are markedly different from ReLU with He initialization, but the magnitude of the layer update, AK, , is very small with respect to K. We have confirmed this empirically for GeLU and SWISH in Fig. and Fig. 2.
0.1 the activations look similar to ReLU, while
SW (461
Sode 2h0 Aetliilee
0.0
-0.2
GeLD that
Bell CIR)
T=1/10
T=1
10
20
30
40
30
0.0
10
20
30
40
30
K1
(a) T = 0.1
(b) T
Figure 2: Experimental results on the layer update for T 0.1, He and critical initializa- lions, at depth L 10 and width N = 320. Figure (a) shows how the two relaxations converge to ReLU behavior at low temperatures for the He initialization. In figure (b) we plot them for T 1, and notice sharp differences between each other and ReLU, Both cases confirm that at low temperatures (T' < 1) architectures with continuous relaxations are both differentiable throughout the domain and behave like ReLU; see App. I. for GumbelLUU.

---

(width
44-
depth
SWASH
width
2 E
deyth
ReLU
Gew (T-0.1)
40
10-
0
38
31
50
100
150
200
250
300
200
37
30
100
150
200
250
500
350
width
width
(a) Critical SWISH and GeLU
(b) ReLU, SWISH and GeLU (He)
Figure 3: Experimental results for the NTK at T = 0.1 and [ = 5 as function of width. Figure (a) shows the SWISH and GeLU activations initialized at criticality, and figure (b) compares ReLU with SWISH and GELU with He initialization. Both cases confirm that at low temperatures architectures with continuous relaxations are differentiable throughout the domain and behave like ReLU when initialized in the same way.
4.2 NTK AT INITIALIZATION
In sub-figure 3 (a) we plot ensemble averaged NTK for critical initializations of SWISH and GeLU, as function of width, for a fixed depth = As it can be seen, (/(n"*) is approximately a constant with some deviation for smaller width values, irrespective of temperature. Then, in sub-figure (b) we compare them at T = 0.1, to ReLU under He initialization. We fit θ/n to confirm the constant behavior in the previous plot. We do notice a sub-lcading ~ n 112 which quickly disappears as width increases. is remarkable that in both cases we observe a constant θ/n. Not only the magnitude of the output, K, but also the gradicnts with respect to the paramcters, O, are very similar between FNN with study, for low temperatures. We have numerically
ReLU and the smooth approximations confirmed the same behaviour to be true for all the functions listed in table 1
4.3 TRAINING DYNAMICS
The previous section confirmed that smooth-ReLUs are ReLU-like at initialization, if the network is tuned to criticality. Here we study their training properties. We focus on the CIFARIO (Krizhevsky et al.) and MNIST (LeCun and Cortes, 2010) datasets. To remove mini-batch stochasticity and better resolve the dynamics, we train with full-batch gradient descent at fixed learning rate of 10 Due to computational memory limitations, we select the first 100 images for each category for training, and 10 samples per category for testing. To understand the impact of initialization, we compare the learning curves for He (critical for ReL.U), and the SWISH critical initialization, for wide range of temperatures. The trends we observe are the same across various seeds for the random number generator. For clarity of presentation, all the plots displayed in this section were obtained with seed equal to 1. We tested with various depths from 4 to 16 and for the presentation that follows we display results for 1 8, 12. The widths were set equal to the number of pixels of the input, 784 for MNIST and 1024 for CIFAR10. The cost function
the negative log-likelihoode Here we show the training curves for SWISH and ReLU on CIFAR10, and other activations can be found in App. J, where we also show results for the MNIST dataset.
4.3.1 TRAINING LOSS FUNCTION
Fig. shows the training curves on the CIFAR10 dataset, for SWISH and ReLU activations. When using He initialization, training worsens as the temperature is increased. In other words, as the activation becomes increasingly different from ReLU, training takes longer to converge. As the depth is increased, learning becomes harder for these high-temppraature He-initialized Moooth-ReLU networks. This finding agrees qualitatively with what we ob- served in Fig. 2: at low temperatures SWISH behaves like ReLU (and training is stable) while at higher temperatures it becomes less stable (slower learning made worse by greater depths). When SWISH is critically initialized, all loss curves are closer to each other, for

---

1-0000
T-483
1=1
T-583
200
107
NL1 Loss
310
NLL Loss
30"1
1000
200y
2003
4200
3003
1003
2000
3300
1000
5000
Epoch
Epoch
(a) He, depth =
(b) He, depth =12
10P
201
131
NLL Lo5s
Loss
131
10-1
10-0
10-1
1200
2002
105
1003
12-1
1603
2000
1000
0000
Epoch
Epoch
(c) Critical, depth =
(d) Critical, depth =12
Figure 4: Loss function for Cifar10 training with SWISH activation, He and critical initial- ization.
all temperatures and for ReLU. In both initialization schemes we notice that the deeper networks have a steeper decrease of loss in the first 1000 epochs region. In addition, critical initialization leads to faster training. In Fig. 4 compare the training curves in the first row versus the second row for the same temperatures. At T = 1, for instance, the critically tuned networks reach loss of 10-1 approximately five times faster with respect to the He initialization case. In addition, we trained the small Transformer (Vaswani et al., 2017) the 1M English-German translation (Bojar et al., 2014) and the BO EfficientNet (Tan and Le, 2019) on CIFAR100 with GudermanL.U and SWISH for the He initialization, App. K. The GudermanI.U's loss stays closer to ReLU, expected, given that both Cw and C₁ are closer to ReLU's values. Also, lowering the temperature moves both training curves closer to ReLU.
4.3.2 NEURAL TANGENT KERNEL
The one step update of the empirical NTK, Aô, during training plotted in Fig. Net- works with ReLU activations have highly irregular NTK variations over time, such that the differentials are not well-defined. In contrast, networks with smooth activations have smoothly varying NTKs throughout training, and this is crucial for theoretical understand- ing of all stages of training. Any training theory based on the NTK and its differentiability should be applicable to Amooth-ReLU networks at every non-zero temperature. Under- standing the convergence of these networks to ReLU networks would allows us to extend such understanding
ReLU networks as well. The Δ for smooth-ReL.ss ReLUs first increases and then proceeds to decrease, the task is learned and convergence achieved. For He there is a clear temperature dependence and the NTK update gets larger with increasing T and I, while for the critical initialization the various curves are closer to each other and less sensitive to depth. This result a further confirmation our observations throughout this work. Remarkably, very low temperature SWISH, T = 0.01, starts showing some level of stochasticity in the Δθ, as well. As the ReLU NTK shows Fig. 10, we apply a Savitzky-Gole with window of 80 epochs and polynomial order of 3, to improve the clarity of the plot. We tested several epochs and polynomial orders for the window filtering with similar results. After using the window filter, in the initial faster training phase, the ReLU follows
clear trend, in App. filter (Savitzky and Golay, 1964; Virtanen et al., 2020)
8

---

2013
IA 6
2006
5000
Tosso 12505 15005
37500
20500
2500
5006
7500
15500
12500
19550
11966 25000
Epoch
Epoch
(a) He depth = 8
(b) He, depth =12
|4 6.
J 6
1070
2505
5000
1000
T0020 12505 1000 17500 2020o
10%
3000
5000
12500 12100
19500
17505
32000
Epoch
Epoch
(c) Critical, depth = 8
(d) Critical, depth =12
Figure 5: NTK update for Cifar10 training with SWISH activation, He and critical initial- ization.
closely the low temperature SWISH behavior, while on the later stages it seems to resemble stochastic noise.
5 SUMMARY AND CONCLUSION
In this work we study differentiable approximations to ReLU, fully-connected feedfor- ward neural networks and restrict ourselves to temperature interpolation of the form dr(z) (r(=))=ozz/[)) We study approximations where a(z/T) approaches the Heaviside func- tion at low temperatures. Inspired by recent field theory treatments, study the signal magnitude propagation with depth subject to small perturbations. We develop several the- orems explain their behavior and discuss how to make information propagation stable through critical initialization. We find that for small temperal ures these activations remain close ReLU in both initialization statistics and training dynamics. For temperatures of order one or higher, typical initializations (e.g., He) hinder information propagation through exponential decay of signal magnitude. On the other hand, the critical initialization sta- bilizes the network statistics for all temperatures and results in training curves similar to ReLU. Despite the activations never strictly becoming ReLU, they show ReLU-like prop- erties in practice, while retaining differentiability. This is further confirmed by our neural tangent. kernel analysis. At initialization, these activations display NTK values very close to ReLU, with 6/n approximately constant. During training they retain a well-defined (continuous) NTK update which helps explain learning and convergence. In other words, emooth-ReLU networks have continuous updates during training, while ReLU networks have noisy updates. In future work, residual networks. It may be of interest to apply this analysis to neural ODE's, where differentiable activations play a major role. In addition, it would be interesting to analyse data normalization in conjunction with critical initialization.
would like to extend our analysis to convolutional and
REFERENCES
E. Alcaide. E-swish: Adjusting activations to different network depths. arXiv preprint arxio:1800 180100740, 2018.
9

---

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
S.-I. Amari. Natural gradient works elficiently learning. Neural compulation, 10(2): 251-276, 1998.
S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
S. Bai, J. Z. Kolter, and V. Koltun. Deep equilibrium models. Advances in Neural Infor- malion Processing Systems, 32, 2019.
O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Kochn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. s. Tamchyna. Findings of the 2014 workshop on statistical machine translation, In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12 5-5 Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http:://ww..ch/e aclueb. org/anthology w/W14/W14-3302.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Nee- lakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Serbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess,
Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learn- ers. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Newral Information Processing Systems, volume 33, pages 1877-1901. Cur- ran Associates, Inc., 2020. URL httpps:/pproced/// wwwrsss.errppccco.ee httppippcccppppp/////////// 1457
1ac.e//74441bbbbaaaa444466aa Paper. pdf.
R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018
H. H. Chieng, N. Wahid, P. Ong, and R. K. Perla. Flatten-t swish: thresholded relu- swish-like activation function for deep learning. arXiv preprint arXiv:1812. 06247, 2018.
D.-. Clevert, T. Unterthiner, and Hochreiter. Fast and accurale deep network learning by exponential linear units (elus). ar ar preprinl orXiv: 1.177999,,, 2015.
Deng, W. Dong, Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenct: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pottern recogniliin,, pages 248 255. Ieee, 2009.
C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau, and R. Garcia. Incorporating second-order functional knowledge for better option pricing. Advances in newral information processing systems, 13, 2000.
S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weiihhee linear units for neural network function approximation in reinforcement learning. Newral Networks, 107:3 11, 2018.
Z. Feng and Z. Kolter. On the neural langent kernel of equilibrium models. 2020.
R. Fisher. A mathematical examination of the methods of determining the accuracy of an observation etc monthly notices roy. Monthly Notices of the Royal Astronomical Society, 80:758-770, 1920.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Y. W. Teh and M. Titterington, editors, Proceedings of the Thirteenth International Conference of Machine Learning Research, pages 249 20-, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMI.R URL. https://proceed: http:emww....eeeeeeeeeeeeeeeeeee preas/vv/glorot10aa httpg/looota.aaaaa. html.
Artificial Intelligence and Statistics, volume 9 of Proceedings
mlr
10

---

B. Hanin. Correlation functions in random fully connected neural networks at finite width. arXiv preprint arXiv:2244.0105: 2022.
B. Hanin and M. Nica. Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, 376(1):287-322, May 2020a. ISSN 1432-0916. doi: 10.1007/00220-01990032--- URL attps://doi..o/
800220-019- 019-2-22--22
B. Hanin and M. Nica. Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, 376(1):287-385) 2020b.
K. He, X. Zhang, S. Ren, and Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-10344 2015.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint or arxiv:16000 Nxw:1606.8844 08006.0841,,.. 2016.
A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVE Inter- national Conference on Computer Vision, pages 1314 1324, 2019.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object. recognition? In 2009 TEEE 12th internalional conference computer vision, pages 2146-2153. TEFE, 2009.
A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research). URL http:// http://wwwwcc.c toronto....ooooooo edu/-kriz/cin ceitrrii/iaaar.e. .html.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL ITR.hhttps://openreevee
het/forrum?d--itel////.
Y. LeCun and C. Cortes. MNIST handwritten digit database. 2010. URL http://yann.
lecun. com/ ///ed////miis/////////////m
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521((553)):36 444, 2015.
M. Li, M. Nica, and D. Roy. The future is log-gaussian: Resnets and their infinite-dept--andd width limit at initialization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and
J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7852-7864 Curran Associates, Inc., 2021. URL https: https://proceedingg: neurips. Pdf.r..df..
C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33: 15954-15964 2020.
C. Liu, L. Zhu, and M. Belkin. Loss landscapes and optimization in over- pverpparameterized non-linear systems and neural networks. Applied and Compulational Harmonic Analysis, 2022.
D. Misra. Mish: self regularized non-monotonic activation function. arXiv preprint arXio: 08681, 2019.
V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Teml, 2010.
A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
11

---

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 188):9, 2019.
M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, On the expressive power of deep neural networks. In international conference on machine learning, pages 2847 2854. PMLR, 2017.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informet neural networks: A deep learning framework for solving forward and inverse problems involving nonlincar partial differential equations. Journal of Computational physics, 378:686 707, 2019.
P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arx preprint arXiv: 710. 1, 2017.
D. A. Roberts, S. Yaida, and B. Hanin. The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neurol Networks. Cambridge University Press, 2022.
A. Savitzky and M. J. Golay. Smoothing and differentiation of data by simplified least squares procedures. Analylical chemistry, 36(8):1677-163) 1964.
M. Seleznova and G. Kutyniok. Analyzing finite neural networks: Can we trust neural tangent kernel theory? CoRR, Abs/2012.04477 2020. URL https://arxiv. org/ abs/
0012.04477.
S. B. Shlosman. Signs of the Ising model Ursell functions. Communications in Mathematical Physics, 102(4):679 686, 1986. doi: cmp/1104114545. URL https://doi.ors org/
M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105 6114. PMLR, 2019.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pe- dregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Al- gorithms for Scientific Computing in Python. Nature Methods, 17:261- 272, 2020. doi: 10. 111108//41559-009--0605
G. Yang. Tensor programs i:: Neural tangent kernel for any architecture, 2020. URL
attps://arxie wrw. amggabs/2006.... 14548.
D. Zou, Y. Cao, D. Zhou, and Q. Gu. Gradient descent optimizes over-parameterizeed deep relu networks. Machine Learning, 109(3):467 492, 2020.
12


Reviews:
[{"review_id": "UFtJ-GD65ep", "reviewer": "Reviewer_S8pa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies temperature-dependent differentiable approximations to the ReLU activation function. The paper derives the critical initialization for this approximation at all temperatures and shows empirically that these smooth activations have similar NTK to ReLU under their proposed initialization.\n", "summary_of_the_review": "While it is useful to know the critical initializations of relaxations to ReLUs being used in practice, the paper does not show that these initializations are practically useful compared to standard initializations. The nicer properties of these activations under their proposed initialization are interesting, but again do not give us much theoretical or practical insight.", "strength_and_weaknesses": "Strengths:  \n1. The paper derives critical initializations for many commonly used relaxations of ReLUs like SWISH and GeLU. The derivation seems correct.  \n2. The proposed initializations train better than He et. al. initialization at all temperatures in the experiments with MLPs.   \n\nWeaknesses:  \n1. Experiments: It would have been useful to see how these initializations perform for realistic tasks compared to standard initializations used in practice. Activations like Swish are being used for transformers widely and it would be great if a theoretically-motivated initialization can provide improved training dynamics. I understand that residual connections and normalizations add further complexities to the derivation, but it could be useful to see if the initialization matters in the practical settings anyway.  \n2. While it is interesting that the NTK updates look smoother for these neural networks, could the authors comment on why we should care about this quantity for practical settings? Is there an analogous quantity outside the NTK regime that could be useful to look at?   \n3. I find the impact of this paper to be incremental. Theoretical impact: While it is nice that these activation functions are smooth and have nicer NTK properties, I am not sure if these highlighted properties are the theoretical reason these activation functions are preferable to ReLUs in practice. Practical impact: the proposed initializations could potentially lead to faster training dynamics to He et. al., but that has not been demonstrated in this paper i.e. they show that He et. al. behaves differently for different temperatures, but the overal dynamics are not faster than He et. al. The other remaining reason this paper could be impactful is if it develops new techniques in their derivations, but I am not well versed with the literature enough to know if so.\n\nMinor comments. \n1. Would help readability if you can relabel axes in figures with what the quantity indicates. For eg: In Figure 3, replace n with ‘width’\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QORIeb2JL6r", "reviewer": "Reviewer_Tjrr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers several smooth relaxations of ReLU. The smoothness of these networks is controlled by a parameter called “temperature”. The authors theoretically analyze fully-connected networks with these activations and derive temperature-dependent critical initialization schemes. Namely, initializations such that the values in different layers in a deep network do not explode or vanish. The authors also demonstrate empirically that representation variance and NTK of smooth ReLUs resemble those of ReLU under the stable initialization scheme we provide, for any temperature, and show that smooth-ReLUs typically have stable and continuous NTK updates during training, while ReLU shows stochasticity instead. ", "summary_of_the_review": "I recommend weak-reject with low confidence.", "strength_and_weaknesses": "The subject of understanding which initialization schemes work well for smooth relaxations of ReLU is well-motivated and interesting.\n\nI read the paper, but unfortunately, I didn’t understand it (starting from Section 3). There are two possible reasons:\n\n(1) It is my fault, namely, I am not sufficiently familiar with this subject and/or did not devote enough time to understand the paper.\n\n(2) The paper is not well-written.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CkcXL0_TWT", "reviewer": "Reviewer_AkRp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper compares the ReLU activation function to its many smooth variants that converge to it in the limit of low temperatures. The paper proposes to investigate the role of these two sub-classes of activation functions by looking at their gradient propagation at initialization time. In particular, prior work has shown that there are some initializations that produce the known phenomenon of exploding/vanishing gradients. In addition, there are specific initializations where the magnitude of gradients stays constant on average throughout the network. When the latter is possible the paper defines it as a criticality. In this work this property is studied for a number of variants of smoothed ReLUs and how it changes with temperature. The paper proves the sufficient and necessary conditions for a smoothed ReLU kind of function to have a critical initialization scheme, and then examines how it manifests for the various activations, including whether this property is stable or not (stable for small pertubations). The paper continues with examining the various activation function through the lens of Neural Tanget Kernel, and use that to argue when smooth ReLU follows similar properties as the non-smooth ReLU. Finally, the paper examines how these activation functions affect the training dynamics, showing a very different behavior of the smooth vs. the non-smooth case.", "summary_of_the_review": "Due to the severe clarity issues, I cannot recommend acceptance. It could be that a reviewer that is more versed in the prior works this work builds upon could evaluate the merits of this submission independently of its presentation, but I am (sadly) unable to do so. I hope the authors will take this criticism and revise their manuscript to be accessible to a broader readership, as what I did understand I liked.", "strength_and_weaknesses": "Strengths:\n* With an evergrowing zoo of activation functions, often very similar to each other, there is a growing need for a crisp theory to understand their inherent properties, and when one should prefer a specific one. The family of ReLU-like activation functions is vast, and this paper proposes a method to study a large set of them if they can be represented as $f(z, T) = z \\cdot a(\\frac{z}{T})$ for some sigmoid function $a(z)$ that converges to the Heaviside function for $T \\to 0$. Such a general result could have a major impact on how activation functions are used and how new ones could be designed.\n* The paper also examines 3 novel kinds of smooth ReLUs based on this generic definition, and examines their properties. This demonstrates the breadth of the theoretical framework.\n* As a side note, I suggest the authors to consider adding to their table the Softplus function (the \"original\" smooth alternative to ReLU). It too can be put into the studied form by rewriting it as $Softplus(z) = z \\cdot a(z/T)$ for $a(x) = \\frac{\\ln(1+\\exp(x))}{x}$, and it can be shown that $a(x)$ converges to the Heaviside function as the temperature goes to zero, but $a(x)$ is not a sigmoid function. It is unclear in your analysis if $a(x)$ being a sigmoid is critical to your results or not, and having a function that does not follow this pattern could shed more light.\n\nWeaknesses:\n* The paper has serious clarity issues, starting with not giving a clear definition of the core expressions in its analysis. For example, a very loose definition of $< \\cdot >$ is given in the text, and the definition for $< \\cdot >_K$ is completely absent. One has to read one of the cited works [1] to understand what it means, and the paper doesn't even bother directing the reader to do so. Moreover, after examining the other paper that defines these terms, it raised even more questions. This is because only at the end of section 2.1 is it apparent that the prior definitions were for the limit of infinite width/depth. A much more in-depth background is needed to explain how this limit is taken. Similarly, NTK is poorly introduced, giving no insight into its definitions and why the reader should care. This paper essentially requires you to be well versed in the literature on NTK and the works on the limit of infinite NN as Gaussian processes.\n* Given this point, I had a hard time understanding and verifying the correctness of this work. Therefore, I cannot judge whether the analysis is accurate or evaluate its limitations.\n* Furthermore, due to the various clarity issues, many of the results are essentially left unexplained and unmotivated. This is too bad, as beyond the point on criticality, which I understood, I cannot really say I understood the point of the later sections that are based on NTK, which is a shame as I'd like to believe there might be interesting insights there.\n* Of a minor note, it appears that the style file was modified. I'm not taking this into account in my rating, but simply pointing this out for the AC for their consideration.\n\n[1] - B. Hanin. Correlation functions in random fully connected neural networks at finite width. arXiv preprint arXiv:2204.01058, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject