Expected Gradients of Maxout Networks and Consequences to Parameter Initialization
Authors: Hanna Tseran
Guido Montufar
Venue: Submitted to ICLR 2023 (2023)
Keywords: maxout unit
input-output Jacobian
parameter initialization
expressivity
linear regions
curve distortion
NTK

Abstract: We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks. Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks. In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK. 

Under review as a conference paper at ICLR 2023
EXPECTED GRADIENTS OF MAXOUT NETWORKS AND CONSEQUENCES TO PARAMETER INITIALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks. Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks. In addition, we obtain refined bounds the expected number of linear regions, results on the expected curve length distortion, and results on the NTK.
INTRODUCTION
We study the gradients of maxout networks and derive several implications for training stability, parameter initialization, and expressivity. Concretely, compute stochastic order bounds and bounds on the moments depending on the parameter distribution and the network architecture. The analysis is based on the input-output Jacobian of maxout networks. We discover that, in contrast to ReLU networks, when initialized with a zero-mean Gaussian distribution, the distribution of the input-output Jacobian of maxout network depends on the network input, which may lead to unstable gradients and training difficulties. Nonetheless, we can obtain a rigorous parameter initialization recommendation for wide networks. The analysis of gradients also allows refine previous bounds on the expected number of linear regions of maxout networks at initialization and derive new results on the length distortion and the NTK.
Maxout networks A rank-K maxout unit, introduced by Goodfellow al. (203)), computes the maximum of K real-valued parametric affine functions. Concretely, a rank-K maxout unit with n inputs implements function I. IR; x + max (WW x) + bx where Wa E R and by R. l E [K] := {1...., K K). trainable weights and biases. The K arguments of the maximum are called the pre-activation features of the maxout unit. This may be regarded as a multi-argument generalization of ReLU, which computes the maximum of real-valued affine function and zero. Goodfellow et al. (2013) demonstrated that maxout networks could perform better than ReLU networks under similar circumstances. Additionally, maxout networks have been shown to be useful for combating catastrophic forgetting in neural networks (Goodfellow et al. 2015). On the other hand, Castaneda et al. al.] (2019) evaluated the performance of maxout networks in big data setting and observed that increasing the width of ReLU networks is more effective in improving performance than replacing ReLUs with maxout units and that ReLU networks converge faster than maxout networks. We observe that proper initialization strategies for maxout networks have not been studied in the same level of detail as for ReLU networks and that this might resolve some of the problems encountered in previous maxout network applications.
Parameter initialization work of Hochreiter (1991). It makes choosing an appropriate learning rate harder and slows training Sun 2019)) Common approaches to address this difficulty include the choice of specific architectures, e.g. LSTMs (Hochreiter 1991) or ResNets (He et al.] 2016). and normalization methods such as batch normalization (loffe & Szegedy, 2015) explicit control of the gradient magnitude with gradient clipping (Pascanu et al. 2013). We will focus on approaches based on parameter initialization that control the activation length and parameter gradients (LeCun al. 2012 Glorot & Bengio 2010
The vanishing and exploding gradient problem has been known since the

---

Under review as a conference paper at ICLR 2023
He et al. 2015 Gurbuzbalaban & Hu 2021 Zhang et al. 2019
Bachlechner et al. 2021). He et al. (2013) studied forward and backward passes to obtain initialization recommendations for ReLU. A more rigorous analysis of the gradients was performed by Hanin & Rolnick (2018), Hanin (2018).
who also considered higher-order moments and derived recommendations on the network architecture. Sun et al. (2018) derived a corresponding strategy for rank K 2 maxout networks. For higher maxout ranks, [Teran & Montúfar (202I) considered balancing the forward pass, assuming Gaussian or uniform distribution on the pre-activation features of each layer. However, this assumption is not fully justified. We will analyze maxout network gradients. including the higher order moments, and give rigorous justification for the initialization suggested by Tseran & Montúfar (202T).
Expected number linear regions Neural networks with piecewise linear activation functions subdivide their input space into linear regions, i.c., regions over which the computed function is (affine) linear. The number of linear regions serves as a complexity measure to differentiate network architectures (Pascanu al. 2014 Montufar et al. 2014 Telgarsky 2015 2016). The first results on the expected number of linear regions were obtained by Hanin & Rolnick (2019a b) for
ReLU networks, showing that it can be much smaller than the maximum possible number. [Terran & Montúfar (202I) obtained corresponding results for maxout networks. An important factor controlling the bounds in these works is a constant depending on the gradient of the neuron activations with respect to the network input. By studying the input-ouiput Jacobian of maxout networks, we obtain a refined bound for this constant and, consequently, the expected number of linear regions.
Expected curve distortion Another complexity measure is the distortion of the length of an input curve as it passes through network. [Poole et al. (2016) studied the propagation of Riemannian curvature through wide neural networks using a mean- -ield approach, and later, a related notion of "trajectory length" considered by Raghu et al. (2017). It was demonstrated that these measures can grow exponentially with the network depth, which was linked to the ability of deep networks to "disentangle" complex representations. Based on these notions, Muuray al. (2022) studies how to avoid rapid convergence of pairwise input correlations, vanishing and exploding gradients. However, Hanin et al. (202T) proved that for ReLU network with He initialization the length of the curve does not grow with the depth and even shrinks slightly. We establish similar results for maxout networks.
NTK It is known that the Neural Tangent Kernel (NTK) of finite network can be approximated by its expectation (Jacot et al. (208)). However, for ReLU networks Hanin & Nica (2020a) showed that if both the depth and width tend to infinity, the NTK does not converge to constant in probability. By studying the expectation of the gradients, we show that similarly to ReLU, the NTK of maxout networks does not converge to constant when both width and depth are sent to infinity.
Contributions Our contributions can be summarized follows.
For expected gradients, derive stochastic order bounds for the directional derivative of the input-output map of a deep fully-connected maxout network (Theorem D as well as bounds for the moments (Corollary 2 Additionally,
derive equality in distribution for the directional derivatives (Theorem 3) based on which we also discuss the moments (Remark 4 in wide networks. We further derive the moments of the activation length of fully-connected maxout network (Corollary 5p.
We rigorously derive parameter initialization guidelines for wide maxout networks preventing vanishing and exploding gradients and formulate architecture recommendations. We experimentally demonstrate that they make it possible to train standard- deep fully-connected and convo- lutional maxout networks using simple procedures (such as SGD with momentum and Adam), yielding higher accuracy than other initializations ReLU networks on image classification tasks.
We derive several implications refining previous bounds on the expected number of linear regions (Corollary 6), and new results on length distortion (Corollary 7 and the NTK (Corollary) 9
PRELIMINARIES
Architecture We consider feedforward fully- colnecconnected maxout neural networks with no inputs, L hidden layers of widths ny...., n2,,11 and a linear output layer, which implement functions of the form N 1 -- 001. The 1-th hidden layer is function di B - Rd with components
2

---

Under review as a conference paper at ICLR 2023
i E [ni] := (1.....m.) given by the maximum of K 2 2 trainable affine functions Φ1,t R - R; xl--1) + max (W (W
1,4. (1xx(-11) + +b,2),
i.k where W 2 E IR=--1, b E R. Here x ((1 E P
Xi,k = w _, (1-1) - -b to denote the kth pre-activation of the ith neuron in the Ith layer. Finally g:: R + IR is a linear output layer. We will write @ = {W, b) for the parameters. Unless stated otherwise, we assume that for each layer, the weights and biases are initialized as i..d. samples from a Gaussian distribution with mean 0 and variance e/n4-1. where is a positive constant. For the linear output layer, the variance is set as 1/nn-1. We shall study appropriate choices of c. We will use || to denote the lg vector norm. We recall that real valued random variable X is said to be smaller than Y in the stochastic order, denoted by X Sat Y, if Pr(X x) S Pr(Y > x) for all E R. In Appendix A we review basic notions about maxout networks and random variables that we will use in our results.
denotes the output of the (1 1)th layer and x( = x. We will write (1)
i.k
Input-output Jacobian and activation length We are concerned with the gradients of the outputs with respect to the inputs, (x V.N, and with respect to the parameters, (.)(@) = VeN, In our notation, the argument indicates the variables with respect to which we are taking the derivatives. To study these gradients, we consider the input-output Jacobian (x(() \V (x),... .... V.nng (x))" To see the connection to the gradient with respect to the network parameters, consider any loss function L: R - R. A short calculation shows that, for a fixed input x E "o the derivative of the loss with respect to one of the weights W:(. i,K' of maxout unit is (VL(N (x)) (x x (-1) if K' argmax, {x and zero otherwise, i.e.
OL(x) C(x, W) JJ (x()) ull x (111)
Ow (1)
(I)
where C(x, w):= (x (VC((()))) J (x and = R A similar decompo- sition of the derivative was used by Hanin (2018); Hanin & Rolnick (018) for ReLU networks. By (1) the fluctuation of the gradient norm around its mean is captured by the joint distribution of the squared norm of the directional derivative [| [x]]2 and the normalized activation length A = ||x"||//m We also observe that ||J (x) ull is related to the singular values of the input- output Jacobian, which is of interest since spectrum concentrated around one initialization can speed up training (Saxe et al. 2014; Pennington et al. 2017: 2018): First, the sum of singular values is tr(JN' (x)" JN (x) Σn (JN (x) JN (x) u. u.) E || (x) u,ll? where the vectors u, form an orthonormal basis. Second, using the Stieltjes transform, one can show that singular
values of the Jacobian depend on the even moments of the entries of J.v (Hanin 2018 Section 3.1).
3 RESULTS
3.1 BOUNDS ON THE INPUT-OUTPUT JACOBIAN
Theorem 1 (Bounds on [|Jx(x) Consider a maxout network with the settings of Section 2 Assume that the biases are independent of the weights but otherwise initialized using any approach. Let u E Tno be a fixed unit vector. Then, almost surely with respect to the parameter initialization, for any input into the network x E IR"", the following stochastic order bounds hold:
L-1
II 1=1
n2
K) Sac [Jx(()
Set
X2 710
1
L-
II
M
XUL
TH
where Ex, K) and En:(2, K) are respectively the smallest and largest order statistic in sample of size K of chi-squared random variables with 1 degree of freedom, independent of euch other and of the vectors and x.
The proof is in Appendix B It is based on appropriate modifications to the ReLU discussion of Hanin & Nica (2020b); Hanin et al. (20TT) and proceeds by writing the Jacobian norm as the product of the layer norms and bounding them with minke (W(()) i,e; u (())))² and max { (w 1,k, u (1-)))²). Since the product of a Gaussian vector with a unit vector is always Gaussian, the lower and upper bounds are distributed the smallest and largest order statistics in sample of size K of chi-squared
3

---

Under review as a conference paper at ICLR 2023
Maxout, rank K =
ReLU 5 hidden layers
hidden layer
3 hidden layers
5 hidden layers
10 hidden layers
50
R
300 -0
Figure I: Expectation of the directional derivative of the input-output map (x)) for width-2 fully-connected networks with inputs in R². For maxout networks, this expectation depends on the input, while for ReLU networks, it does not. Input points x were generated as grid of 100 x 100 points in |-103, 103 and u was a fixed vector sampled from the unit sphere. The expectation was estimated based on 10,000 initializations with weights and biases sampled from ((0 1).
random variables with degree of freedom. In contrast to ReLU networks, found that for maxout networks, is impossible to write equality in distribution involving only independent random variables because of the dependency of the distribution of (x) on the network input and the direction vector u (see Figure DT We discuss this in more detail in Section 3.2
Corollary 2 (Bounds on the moments of [|]x(x))l]]) Consider a maxout network with the settings of Section 2 Assume that the biases are independent of the weights but otherwise initialized using any approach. Let u E IR be a fixed unit vector and x E Wno be any input into the network, Then
(i) " (c8) < (x]])(())|()
nL (cl)) L-1
n10
)}
(ii) Var [1]])())
2(1-1)
12(1-1) exp
4
S2(L-1)
no
niK
n2
nL } te
(ii) E (x) 20]
nr. no
(cK)
exp 42
niKK
where the expectation is taken with respect the distribution of the network weights. The constants 8 and & depend on K and denote the means of the smallest and the largest order statistic in a sample of K chi-squared random variables. For K = 2...., 10, S E [0.02, 0.4] and & E [1.6, 4). See Table in Appendix for the exact values.
Notice that for 2 2, the tth moments of the input-output Jacobian depend on the architecture of the network, but the mean does not (Corollary?2 2). similarly their behavior in ReLU networks Hanin (2018). We also observe that the upper bound on the /th moments can grow exponentially with the network depth depending on the maxout rank. However, the upper bound the moments can be tightened provided corresponding bounds for the largest order statistics of the chi-squared distribution.
3.2 DISTRIBUTION OF THE INPUT-OUTPUT JACOBIAN
Here we present the equality in distribution for the input-output Jacobian. It contains dependent variables for the individual layers and thus cannot be readily used obtain bounds on the moments, but it is particularly helpful for studying the behavior of wide maxout networks.
Theorem 3 (Equality in distribution for (x)ul() Consider maxout network with the settings of Section 2 Let Ro be a fixed unit vector and R x * be any input into the network. Then, almost surely, with respect to the parameter initialization, [] (x)ul2 equals in distribution
1 110
X
cos2 2 71 1) u(-) + (N, K) cos (00-)),
where Yr and (N 1), K) are independent, v, (0, 1). (N, 1), is the largest order statistic in a sample of K standard Gaussian random variables. Here 7500 w(th denotes the
4

---

Under review as a conference paper at ICLR 2023
connecco network
Convolutional network
Width = 2
Width = 50
Width = 2
Width = 50
109
1019
10%
3019
Value of
1013
3074
D:35
101
122
10P
16
055555
80-00
10-
10-13
12
100
Depth
Depth
Dopth
Figure 2: Expected value and interquartile range of the squared gradients no /o as function of depth. Weights are sampled from (0, e/fan-in) in fully-connected networks and N NO, (k?2...... fan-in)), where k is the kernel size, in CNNs. Biases are zero, and the maxout rank K 5. The gradient stable in wide fully- aully-connected and convolutional networks with c 0.55555 (red line), the value suggested in Section 4 The dark and light blue lines represent the bounds from Corollary 2 and equal 1/L = 0.36 and 1/8- = 12. The yellow line corresponds to the ReL.U-H initialization. We compute the mean and quartiles from 100 network initializations and a fixed input. The same color lines that are close to each other correspond to 3 different unit-norm network inputs.
Uni 0) in RR:+11 where n w "-"))/ / (((-1)| when W (h' (t11) # 0 and 0 otherwise, and u
angle between z(0) (xl) (1)
1 1) and u()) (u 11
(1)
The matrices W
consist of rows W!l) = i,k" TR==-1, where k' = argmax4e[K] (W!4) (1) x (1-1) + 1.2))
This statement is proved in Appendix D The main strategy is to construct an orthonormal basis B (bI...., ), where b1 := ((1)/) x= on the angle between y!6) and u())
II, which allows to express the layer gradient depending
Remark (Wide networks). By Theorem 3 in a maxout network the distribution of (x)u]] depends the cos "0 ((-1)), which changes as the network gets wider or deeper. Since inde- pendent and isotropic random vectors in high-dimensional spaces tend to be almost orthogonal, we expect that the cosine will be close to zero for the earlier layers of wide networks, and individual units will behave similarly to squared standard Gaussians. In wide and deep networks, if the network parameters are sampled from N(0, c00, c/mi-1). c 1/M, and K > 3, we expect that cos Ty40 a ss 1 for the later layers of deep networks and individual units will behave more as the squared largest order statistics. Here M is the second moment of the largest order statistic in a sample of size K of standard Gaussian random variables. Based on this, for deep and wide networks, we can expect that
(x)
RV nāp
Th (cM) 1-1
nL
no
This intuition is discussed in more detail in Appendix D According to 2 we expect that the expected gradient magnitude will be stable with depth when an appropriate initialization is used, See Figure for numerical evaluation of the effects of the width and depth on the gradients.
3.3 ACTIVATION LENGTH
To have a full picture of the derivatives in (T., we consider the activation length. The full version and proof of Corollary are in Appendix E The proof is based on Theorem 3 replacing u with z/ [|=]]-
Corollary 5 (Moments of the normalized activation length). Consider maxout network with the settings of Section [ Let x w be any input into the network. Then, for the moments of the normalized activation length Al)) of the 1'th layer we have
Mean:
E [[a00] = 11 1 (cM) +
Σ
(11
(c2M)
no
j=2
1'
Moments of order t 2 2:
G1 ((N))"))
< {A(())))
< G2
(akk)" exp
Σ
mik
5

---

Under review as a conference paper at ICLR 2023
Table 1: Recommended values for the constant e for different maxout ranks K based on Section
K
2 3
4
5
6
7
S
10
1 0.78391 0.64461
0.55555
0.49462
0.45039
0.41675 0.39023
0.36872
The expectation is taken with respect to the distribution of the network weights and biases, and M a constant depending on K that can be computed approximately, see Table for the values for ... 10. See Appendix for the variance bounds and details on functions G1, G2-
K = .....,
We could obtain an exact expression for the mean activation length for a finitely wide maxout network since its distribution only depends the norm of the input, while this is not the case for the input- output Jacobian (Sections 3.1 and 3.2). We observe that the variance and the /th moments, 2 2, have exponential dependence on the network architecture, including the maxout rank, whereas the mean does not, similarly to the input- Jacobian (Corollary)) 2 Such behavior also occurs for ReLU networks (Hanin & Rolnick 2018). See Figure 8 in Appendix E for an evaluation of the result.
4 IMPLICATIONS TO INITIALIZATION AND NETWORK ARCHITECTURE
We now aim to find initialization approaches and architectures that can avoid exploding and vanishing gradients. We take the annealed exploding and vanishing gradients definition from Hanin (2018) as starting point for such investigation for maxout networks. Formally, we require
=0(1), Var ( ( LK
2t
xl(x) aw!
@L(x)
E
-0(1,, sup E 121
Vt
3,
aw!!. i,k.
where the expectation is with respect the weights and biases. Based on (1 these conditions be attained by ensuring that similar conditions hold for [xJx(x))u|] and A
Initialization recommendations Based on Corollary the mean of [x)(x)]||| can be stabilized for some e E 1/8]. However, Theorem 3 shows that [x)]( depends on the input into the network. Hence, we expect that there is no value of stabilizing input-output Jacobian for every input simultaneously. Nevertheless, based on Remark 4 for wide and deep maxout networks, []]]J>((x)l)
n if 1/M, and the mean becomes stable. While Remark 4 does not include maxout rank K = 2. the same recommendation can be obtained for it using the approach from He et al. (2015), see (Sun et al. (2018). Moreover, according Corollary 5 the mean of the normalized activation length remains stable for different network depths if c 1/M. Hence, we recommend c 1/M as an appropriate value for initialization. See Table [or the numerical value of с for K = .,..., 10. We call this type of initialization, when the parameters are sampled from (0, c/ fnnn-inn.. c 1/M, "maxout initialization". We note that this matches the previous recommendation from Tseran & Montúfar (2021), which we now derived rigorously.
Architecture recommendations In Corollaries 2 and the upper bound on the moments 2 2 of [[]]|] and 1 = ||x(0] //n can grow exponentially with the depth depending on the values of (cK)) F. and 11=1 L-1 1/(n,K). Hence, we recommend choosing the widths such that Σt-1 K) < which holds, e.g., if nı 2 L/ K, K,K, VI 1..... L 1, and choosing a moderate value of the maxout rank K. However, the upper bound can still tend to infinity for the high-order moments. From Remark 4 follows that for K 2 3 to have a stable initialization independent of the network input, maxout network has to be deep and wide. Experimentally, we observe that for 100-neuron wide networks with = 3, the absolute value of the cosine that determines the initialization stability converges to around 60 layers, and for K 4, at around 30 layers. See Figure Appendix To sum up, we recommend working with deep and wide maxout networks with widths satisfying El K) & 1, and choosing the maxout-rank not too small nor too large, c.g., K = 5.
IMPLICATIONS TO EXPRESSIVITY AND NTK
With Theorems I and 3 in place, we can now obtain maxout versions of the several types of results that previously have been derived only for ReLU networks.

---

Under review as a conference paper at ICLR 2023
5.1 EXPECTED NUMBER OF LINEAR REGIONS OF MAXOUT NETWORKS
For piece-wise linear function ʃ: R - R, linear region is defined as a maximal connected subset of Rh on which f has a constant gradient. Tseran & Montúfar (202I) and Hanin & Rolnick (2019b) established upper bounds the expected number of linear regions of maxout and ReLU networks, respectively. One of the key factors controlling these bounds is C Grad, which is any upper bound on (suPxER<0 V (x)]I for any 1 E N and z = 1, I,..., N. Here Ç2, is the kth pre-activation feature of the 2th unit in the network, N is the total number of units, and the gradient is with respect to the network input. Using Corollary [ we obtain value for Cerad for maxout networks,
which remained an open problem in the work seran Montúfar (202I). The proof of Corollary and the resulting refined bound on the expected number of linear regions are in Appendix F
Assume that the biases are independent of the weights but otherwise initialized using any approach. Consider the pre-activation feature Çz,K of a unit 2 = =,..., N. Then, for any E N.
Corollary 6 (Value for Cgrad). Consider a maxout network with the settings of Section 2
sup E [IVGGAGOD)) S no max {1, (cK) - xER"0
(t 2
Σ m +1)
exp
The value of Cgra given in Corollary 6 grows as (((c)) exp El= 1/(m The first factor grows exponentially with the network depth if ck > 1. This is the case when the network is initialized as Section However, since K usually small constant and e S 1, cK 2 1 is small constant. The second factor grows exponentially with the depth if L 1= 1/(n/K) 2 1. Hence, the exponential growth can be avoided if m4 2 (L 1)/K, AA = 1, L-1.
5.2 EXPECTED CURVE LENGTHI DISTORTION
Let M be a smooth 1-dimensional curve in R0 of length len(M) and (M) C R the image of M under the map x + (xx). We are interested the length distortion of M, defined as Ien(N((M))/len)) /len(())))) M) Using the results from Section 3.1 observing that the input-output Jacobian of maxout networks is well defined almost everywhere, and following Hanin et al. (2021). we obtain the following corollary. The proof is in Appendix G
Corollary 7 (Expected curve length distortion). Consider a maxout network with the settings of Section
2 Assume that the biases are independent of the weights but otherwise initialized using any approach. Let M be a smooth 1 1-dimensional curve of unit length in Rho, Then, the following upper bounds on the moments of
hold:
E [len(N(MM)
<
nL
(cl)
-
Var ((en(N((M))
S no
AL (cC)
no
[len(N(MM)))
<
(cK)
11
exp
t
2
ni
K
n
where L is a constant depending on K, see Table 4 in Appendix C for values for K 2,...; 10.
Remark 8 (Expected curve length distortion in wide maxout networks). If the network is initialized according to Section 4 using Remark 4 and repeating the steps of the proof of Corollary we get [[ee((((M))) S (nz/ng) and Var [len(N(MM)) ss nI/no-
Hence, similarly to ReLU networks, wide maxout networks, if initialized to keep the gradients stable, have low expected curve length distortion at initialization. However, we cannot conclude whether the curve length shrinks. For narrow networks, the upper bound does not exclude the possibility that the expected distortion grows exponentially with the network depth, depending on the initialization.
5.3 ON-DIAGONAL NTK
We denote the on-diagonal NTK with KAT(x, x) = (x000)) In Appendix H we show:
Corollary 9 (On-diagonal NTK). Consider a maxout network with the settings of Section 2 Assume
that nt. 1 and that the biases are initialized to zero and trained. Assume that 8 < c < C.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Accuracy on the test set for networks trained using SGD with Nesterov momentum. Observe that maxout networks initialized with the maxout max- max-poniing initialization perform significantly better than the ones initialized with other initializations and better or comparably ReLU networks.
value
Max-poolligg init OectionOOuurr 0.55 8 0.27
Maxout Section Chuss 0,50555
Naive
VALUE OF
ReL.U Hi
0.1
2
JULLY
MNIST
11.3520.00
97.8 97.8 91 21.6723.77 73
53. 22.222
07-43
Inis
30. 2..0000..00
62, 12.52-1.93
91.67 91.67=3.18
CONVOLUTIONAL
MNIST
11.35 11.35 0.000
99. 00.03
99.59 0.
98.02
99.49 w...9t....
CIFAR-10
10.00 80
91.70011
01.21
44. 44.8420.80
90.12 12200.22. 25
CIFAR 100
1.0040.00
65.33 a7
65 $5.39-
12.0240.6
59.5940
MNIST
10.00
(0.65*0000 +- 18
43.41
$1.56
20.12
SVHN
19.09
97.3
97.78
50.97
96.74
where the constants 8, & are as specified in Table
Then,
(c8) L-2
- 10 (ct)
(M
00
(x x)]
P.
n1
110
10
1--1
xx) 2PP (cK) (c
n
exp
ngK
+4 {
where P = = my Pw Et=o 11 and M is specified in Table 4
By Corollary E Kx(x,, (E x)])²
is in (((Pw/P)C exp 1/(n:K))) where C' depends on &, M and K. Hence, if widths A1,..., f and depth L tend to infinity, this upper bound does not converge
constant, suggesting that the NTK might not converge to a constant in
probability. This is in line with previous results for ReLU networks by Hanin & Nical (2020a).
EXPERIMENTS
We check how the initialization proposed in Section 4 affects the network training. This initialization was first proposed heuristically by Tseran & Montúfar (2021), where it was tested for 10-layer fully- connected networks with an MNIST experiment. We consider both fully-connected and convolutional neural networks and run experiments for MNIST (LeCun & Cortes 2010). Iris (Fisher 1936), Fashion
MNIST (Xiao et al., (017), SVHN (Netzer et al.) 2011). CIFAR- and CIFAR- 100 ( (Knihhvssy al. 2209). Fully connected networks have layers and CNNs have VGG-19-Ii like architecture
(Simonyan & Zisserman, 2015) with 20 or 16 layers depending on the input size, all with maxout rank Weights are sampled from (O, c/ effan-in) in fully-connected networks and N(O,, c/(h2 fan-in)) in CNNs of kernel size k. The biases are initialized to zero. We report the mean and std of 4 runs.
We use plain deep networks without any kind of modifications or pre-training. We do not use normalization techniques, such batch normalization (Toffe & Szegedy) 2015), since this would obscure the effects of the initialization. Because of this, our results are not necessarily state-of-the the-art. More details the experiments given in Appendix [I and the implementation is made available at https://anonymous.ooo http://woss.oooeeeeccceeee/eaeae science/r/maxout expected erteirrrraaggniii 68BD
Max-pooling initialization To account for the maximum in max- max-pooling layers, a maxout layer appearing after max-pooling layer is initialized as if its maxout rank was K x m² where m² is the max-pooling window size. For example, we used K = 5 and m 4, resulting in == 0.26573 for such maxout layers. All other layers initialized according to Section 4 We observe that max-pooling initialization often leads to slightly higher accuracy.
Results for SGD with momentum Table 2 reports test accuracy for networks trained using SGD with Nesterov momentum. We compare ReLU and maxout networks with different initializations: maxout, max-pooling, small value c = 0.1, and He c 2. We observe that maxout and max- pooling initializations allow training deep maxout networks and obtaining better accuracy than ReL.U
8

---

Under review as a conference paper at ICLR 2023
Table Accuracy on the test set for the networks trained with Adam. Observe that maxout networks initialized with the maxout or max- max-pooling initialization perform better or comparably to ReLU networks, while maxout networks initialized with ReLU-He converge slower and perform worse.
MAXOUT
RRLU
Max-pooling Sectical (lurs) 0.55 2.22
VALUE OF
Scction Ours)
(uurs) 0.50555
ReLU He
2
CUNLYCONEECTED
epochs
97. 18
97.40200.30
96.724
2/10 epochs
98. 1040..0
97.1720.12
97.54
spochs
98.12=0.20 98.18#0000
97.37#0008
CONVOLUTIONAL
1/10 epochs
99.06 155
96. 08.59=0.00
98.54200.22 5.252
99. 14 20.32
MNIST
2/10 epochs
99.394.0.15
98,51=0.25
99.1740.11
99.41
opochs
99.53
99.47=0.07
99.4770.04
99.40
1/10 epochs
92.04 2:
92.35 02.35=0:12
95220.33
92. 11.45-20 20000..0 41
Fishion MINIST
2/30 epochs
92.61 +0.22
92.85 92.85+0.22
90.30 90.30+00.8 4.0.8
92271 #-0
All epochs
98.57 +0.17
93.45 0.
91.63 +.0.8
92.98 +00.1
1/10 epachs
88,
8..31
74.37
85.95
CIFAR-ID
2/ 10 epochs
88, 88.7920..2 72
87.96=0.75
81.94220.04 2.424
87, 87.12
spochs
91.33
91.06=0.21
85.28+-0.20
87.70
epochs
50.30-29.04
43 oe
19:22200.5
50.39:200.8
C00
2/10 epochs opochs
57.54
57.65 400.07
33.21 4.001
51.34+0.00
+1220
61.96 00.88
37.58 +..28+0..2
62.95++0.30 +00.0
networks, whereas performance is significantly worse or training does not progress for maxout networks with other initializations.
Results for Adam Table 3 reports test accuracy for networks trained using Adam (Kingma & Ba 2015). We compare ReLU and maxout networks with the following initializations: maxout, max-pooling, and He 2. We observe that, compared to He initialization, maxout and max-pooling initializations lead to faster convergence and better test accuracy. Compared to ReLU networks, maxout networks have better comparable accuracy if maxout max-pooling initialization is used.
7 DISCUSSION
We study the gradients of maxout networks with respect to the parameters and the inputs by analyzing a directional derivative of the input-output map. We observe that the distribution of the input-output Jacobian of maxout networks depends the network input (in contrast to ReLU networks), which can complicate the stable initialization of maxout networks. Based on bounds on the moments, we derive an initialization that provably avoids vanishing and exploding gradients in wide networks. Experimentally, we show that, compared to other initializations, the suggested approach leads to better performance for fully connected and convolutional deep networks of standard width trained with SGD or Adam and better or similar performance compared to ReLU networks. Additionally, we refine previous upper bounds on the expected number of linear regions. We also derive results for the expected curve length distortion, observing that it does not grow exponentially with the depth in wide networks. Furthermore, we obtain bounds on the maxout NTK, suggesting that might not converge to constant when both the width and depth are large. These contributions enhance the applicability of maxout networks and add to the theoretical exploration of activation functions beyond ReLU.
Limitations Even though our proposed initialization is in a sense optimal, our results are applicable only when the weights are sampled from N(,, c/fan-in) for some Further, we derived theoretical results only for fully-connected networks. Our experiments indicate that they also hold for CNNs: Figure 2 demonstrates that gradients behave according to the theory for fully connected and convolu- tional networks, and Tables and show improvement in CNNs performance under the initialization suggested in Section 4 However, we have yet conduct the theoretical analysis of CNNs.
Future work In future work, we would like to obtain more general results in settings involving multi-argument functions, such as aggregation functions in graph neural networks, and investigate the effects that initialization strategies stabilizing the initial gradients have later stages of training.

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement
To ensure reproducibility. we make the code public GitHub at https
//anonymous, //anorymouu..oeen.cecer/////a/// wwt.seceeeeneeaouttttttt expected gradient-69BB 63BD, and Section 6 and Appendix provide detailed description of the experimental settings. Proofs are in Appendices B
REFERENCES
T. W. Anderson, An introduction to multivariate statistical analysis. Wiley series in probability and statistics. Wiley-Interscienee, 3rd ed edition, 2003.
Howard Anton and Chris Rorres. Elementary linear algebra: applications version. John Wiley Sons, 2013,
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, Rezero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, pp. 1352- 1361. PMLR, 2021.
Gabriel Castaneda, Paul Morris, and Taghi M. Khoshgoftaar. Evaluation of maxout activations in deep learning across several big data domains. Journal of Big Data, 6(1):72, 2019.
Lisandro D Dalcin, Rodrigo R Paz, Pablo A Kler, and Alejandro Cosimo. Parallel distributed computing using python. Advances Water Resources, 34(9):1224-113 2011.
Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2):179-18) 1936.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, pp. 256-256. JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, David Warde-Farley. Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International Conference on Machine Learning, pp. 1319 1327. PMLR, 2013.
Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting gradient-based neural networks. arXiv:1312.6211 [cx, stat]. 2015.
Mert Gurbuzbalaban and Yuanhan Hu. Fractional moment preseviir initialization schemes for training deep neural networks. Proceedings of The 24th International Conference on Artificial Intelligence and Statistics. pp. 2233-2241. PMLR, 2021.
Boris Hanin. Which neural net architcctures give rise to exploding and vanishing gradients? Advances in Neural Information Processing Systems, 31, 2018.
Boris Hanin and Mihai Nica. Finite depth and width corrections the neural tangent kernel. In International Conference on Learning Representations, 2020a.
Boris Hanin and Mihai Nica. Products of many large random matrices and gradients in deep neural networks. Communications in Mathematical Physics, 376(1):287-322 2020b.
Boris Hanin and David Rolnick. How start training: The effect initialization and architocture, Advances Neural Information Processing Systems, 31, 2018.
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference on Machine Learning, pp. 2596-2604, PMLR, 2019a.
Boris Hanin and David Rolnick. Deep ReLU networks have surprisingly few activation patterns. Advances Neural Information Processing Systems, 32, 2019b.
Boris Hanin, Ryan Jeong, and David Rolnick. Deep ReLU networks preserve expected length. arX1v:2102.10492 [es, stat]. 2021.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt. Ralf Gommers. Pauli Virtanen, David Cournaeeau, Eric Wieser. Julian Taylor. Sebastian Berg, Nathaniel Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peter- son, Pierre Sérard-Marchann Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy, Nature, 585(7825):357330 362, 2020.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human- Level Performance ImageNet Classification. In 2015 IEEF International Conference on Computer Vision (ICCV), pp. 1926-1034, Santiago, Chile, 2015. IEEE.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770-778. IEBE, 2016.
Sepp Hochreiter. Untersuchungen dynamischen neuronalen netzen. Diploma, Technische Universität München, 91(1), 1991.
John D Hunter. Matplotlib: A 2D graphics environment. Computing in science & engineering, 9033)900-95 2007.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Muchine Learning, pp. 448-456. PMLR, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alex Krizhevsky, Geoffrey Hinton, and others. Learning multiple layers of features from tiny images. Citeseer, 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database, 2010.
Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient BackProp. In Neural Networks: Tricks of the Trade: Second Edition, pp. 9-48. Springer Berlin Heidelherg, 2012.
Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg. Martin Wicke. Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in Neural Information Processing Systems, 27, 2014.
M. Murray, V. Abrol, and J. Tanner. Activation function design for deep networks: linearity and effective initialisation. Applied and Computational Harmonic Analysis, 59:117-154, 2022.
Alfred Müller and Dietrich Stoyan. Comparison methods for stochastic models and risks, volume 389. Wiley, 2002.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310-1318. PMLR, 2013.
Razvan Pascanu, Guido Montúfar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv: 131660098288811 [cs], 2014.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid deep learning through dynamical isometry: theory and practice. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, pp. 1924-1932. PMLR, 2018 8.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems, 29, 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. International Conference on Machine Learning, pp. 2847-2854, PMLR, 2017.
Andrew M. Saxe, James I.. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014.
11

---

Under review as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Steven H Strogatz. Nonlinear dynamics and chaos: with applications physics, biology. chemistry, and engineering. CRC press, 2018.
Ruoyu Sun. Optimization for deep learning: theory and algorithms. arXiv:1912.08957 [cs, math, stat], 2019.
Weichen Sun, Fei Su, and Leiquan Wang. Improving deep neural networks with multi-layer maxout networks and novel initialization method. Neurocomputingg 278:34 40, 2018.
Matus Telgarsky. Representation benefits of deep feedforward networks. arXiv:1509.0810/ 2015.
Matus Telgarsky. Benefits of depth in neural networks. In 29th Anmual Conference on Learning Theory, volume 49 of Proceedings of Machine Learning Research, pp. 1517-1539. PMLR, 2016.
Hanna Tseran and Guido Montúfar. On the expected complexity of maxout networks. Advances in Neural Information Processing Systems, 34, 2021.
Roman Vershymin. High-dimensional probability: An introduction with applications in data science, volume 47.
Cambridge university press, 2018.
Wolfram Research, Inc. Mathematica, Version 13.1, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST. a Novel Image Dataset for Benchmarking Machine Learning Algorithms, 2017.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual Iearning without normalization. arXiv preprint arXiv: 190:::99000090000 09321, 2019.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
The appendix is organized follows.
Appendix
Basics
Appendix
Bounds for the input-output Jacobian norm [x]x[
Appendix
Moments of the input-output Jacobian norm [x)]]]
Appendix
Equality distribution for the input-output Jacobian norm and wide network results
Appendix
Activation length
Appendix
Expected number of linear regions
Appendix
Expected curve length distortion
Appendix
NTK
Appendix
Eperiments
A BASICS
A.1 BASICS ON MAXOUT NETWORKS
As mentioned in the introduction, rank-K maxout unit computes the maximum of K real-valued affine functions. Concretely, rank-K K maxout unit with n inputs implements a function
R; x x+
max ((W,, + bb KE[K]
where Wx JR' and by K K == {I...., K). trainable weights and biases. The K arguments the maximum are called the pre-activation features of the maxout unit. A rank-K maxout unit be regarded layer corresponds to parallel computation of several such units. For instance a layer with n inputs and maxout units computes functions of the form
composition of affine map with K outputs and maximum gate.
[ (W 1.k x) +b
(1)
bikk
R" RT; XH
maxce[K] ( (WW() ()) m.k x) + b (.
where now (1) and b 1 are the weights and biases of the kth pre-activation feature the i maxout unit the first layer. The situation is illustrated in Figure 3 for the case of network with two inputs, one layer with two maxout units of rank three, and one output layer with single output unit.
input
affine
max
affine
(
(2(2)
Figure 3: Illustaaioon of simple maxout network with two input units, one hidden layer consisting of two maxout units of rank 3, and an affine output layer with single output unit.
13

---

Under review as a conference paper at ICLR 2023
A.2 BASIC NOTIONS OF PROBABILITY
We ought to remind several probability theory notions that we use state our results. Firstly, recall that if Vi...., D2 are independent, univariate standard normal random variables, then the sum their squares, Lim1 of is distributed according to the chi-squared distribution with & degrees of freedom. We will denote such a random variable with x
Secondly, the largest order statistic is random variable defined the maximum of random sample, and the smallest order statistic is the minimum of sample. And finally, a real valued random variable X is said be smaller than in the stochastic order, denoted by X Sat Y, if Pr(X > a)) < Pr(Y > a) for all I: E R. We will also denote with & equality in distribution (meaning the edfs are the same). With this, we start with the results for the squared norm of the input-output Jacobian [x)]])
A.3 DETAILS ON THE EQUATION
In I
are investigating magnitude of 0L(x) aw
The reason we focus on the Jacobian norm rather
than on C' is as follows. We have
aL(x)
IV
aw!)) ik's
-(VWE(N(S)) JN) (1-1)
=(VN JN J (-11
-C(x, Dull 1-1)
Note that C(x, W) (VNC(N(x)), v) with JA (x(") n/||JA (x(²) ul|. ||v||
Hence
C'x,, W) < (XVxxx) [[VxC(N( The latter term does not directly depend on the specific parametrization nor the specific architecture of the network but only on the loss function and ((-1)
the prediction. In view of the description of ac(x) the variance depends the square of x)
aw
Similarly, the variance of the gradient V Won C(x) = ac(x) ) depends on (( = (xx) (1-1)), and
L.
thus depends ||(2 This is how activation length appears in ①
B BOUNDS FOR THE INPUT-OUTPUT JACOBIAN NORM [x]||])
B.1 PRELIMINARIES
We start by presenting several well-known results we will need for further discussion.
Product of Gaussian matrix and unit vector
Lemma 10. Suppose is an n. x n' matrix with i.i.d. Gaussian entries and u is a random unit vector in Rnn that is independent of W but otherwise has any distribution. Then
Wu is independent of u and is equal in distribution to Wv where v is any fixed unit vector in R"""
2. If the entries of W are sampled ii.dd from Nu,, then for all i =,..., n W.u N(,, and independent of
3. If the entries of W are sampled i..dd from N N00, o²), then the squared E2 norm [|Wu|2 x2x2 where Xa is chi-squared random variable with degrees of freedom that independent of
Proof. Statement was proved in, e.g., Hanin et al. 2021 Lemma C.3) by considering directly the joint distribution of W u and
14

---

Under review as a conference paper at ICLR 2023
Statement 2 follows from Statement if we pick C1-
To prove Statement 3, recall that by definition of the l norm, | = (Wa))² By Statement 2, for all i = -,..... n, Wau are Gaussian random variables independent of u with mean zero and variance o². Since any Gaussian random variable sampled from N N(,, o²) can be written as μ + oU, where squared distribution, Σi_ oद is chi-squared random variable with degrees of freedom denoted with X which leads to the desired result.
(0, 1), we can write Σ (W₂u)² = o² Ei= uz. By definition of the chi-
Stochastic order We recall the definition of a stochastic order. A real-valued random variable X is said to be smaller than Y in the stochastic order, denoted by X Sa Y, Pr x) S Pr(Y x) for all x E IR.
Remark 11 (Stochastic ordering for functions). Consider two functions f : X + R and
X + R that satisfy (x) I g(x)) for all E X. Then, for a random variable X, f(X) Sst (.)). To see this, observe that for any y E R, Pr(f(X) > y) = Pr(X E fx: f(x) > w}) and
Pr(g(X) > y) Pr(X fx:: g(x) > y}). Since f(x)) S g(x) for all X, {x: f(x) > y} S
{æ: g(z) > y). Hence, Pr(f(X) > 8) S Pr(g(()= > y). and f(X) Sa (X)).
Remark 12 (Stochastic order and equality in distribution). Consider real-valued random variables X, Y and Ý. If X S.t Y and Y d Ý, then X Set Ý. Since Y and Ý have the same edfs by definition of equality in distribution, for any y R. Pr(X > 8) S Pr(Y > y) Pr(Y y)-
B.2 EXPRESSION FOR [x)
Before proceeding to the proof of the main statement, given in Theorem 1 we present Proposition 13 Firstly, in Proposition 13 below, we prove an equality that holds almost surely for an input-output Jacobian under our assumptions. In this particular statement the reasoning closely follows follows[Hani Hanin et al. (202T Proposition C.2). The modifications are due to the fact that a maxout network Jacobian is a product of matrices consisting of the rows of weights that are selected based on which pre-activation feature attains maximum, while in a ReLU network, the rows in these matrices are either the neuron weights or zeros.
Proposition 13 (Equality for [|Jr(x))l((),, Let be a cullte-co feed-forward neural network with maxout units of rank K and a linear last layer. Let the network have L layers of widths ni, m1....,0I nI and no inputs. Assume that the weights are continuous random variables (that have a density) and that the biases are independent of the weights but otherwise initialized using any approach. Let u E n be a fixed unit vector. Then, for any input into the network, x E W almost surely with respect to the parameter initialization the Jacobian with respect to the input satisfies
I-1 I
[JA(x))||2 = (L 12
(3)
where vectors u!!), 1- 1=11=1.... 1,...,L- -- are defined recursively as ④ (()) u 1n-1)/
u 11
when ( (1-1) # 0 and 0 otherwise, and ( = u. The matrices W (1) consist of rows W (1)
w:ll) E R == 1,...,n, nn where k' argmax (xx(xx(111))) b,kk x!1) is the output of the lth layer, and (0) = x.
Proof. The Jacobian J (x) a network N (x): R + Rd.. can be written as a product of matrices
w!!) (!] I= 1,...,,. L, depending on the activation region of the input x. The matrix w consists of rows ((() w:ll) R where k' argmax (w(0) ((-1) + b, for i 1, 1..... ny, and (1- is the /th layer's 's input. For the last layer, which is linear, we have W W(L), Thus,
[ (I (1-1
(())
Further we denote u with ( and assume
W
O)) A O. To see that this holds almost surely, note
that for fixed unit vector u (0),, the probability of W (1) heing such that || [n0)]] 0 is 0. This is
15

---

Under review as a conference paper at ICLR 2023
indeed the case since to satisfy |W | linear equations and this system is regular when * 0, so the solution set has positive co-dimension and hence zero measure. Multiplying and dividing
= 0. the weights must be a solution to a system of n1
4 by ||W
1?
[x = ((
w
W
(0)
W (2) 10
| W
/ W (1)
u 10
/| !1 Repeating this procedure layer-by-layer we get where (2) =1) u (0)
| (z- 13 12 W (T. 112 || u 10
(S)
By definition of the &2 norm, for any layer I, | O 11111)11== E ( n 1 Substituting this into 5 we get the desired statement.
B.3 STOCHASTIC ORDERING FOR [J^((x)u|2
Now prove the result for the stochastic ordering of the input-output Jacobian in finite-width maxout network.
Theorem (Bounds on [|Jv(x)u Consider a maxout network with the settings of Section 2 Assume that the biases are independent of the weights but otherwise initialized using any approach. Let u E "R be a fixed unit vector. Then, almost surely with respect to the parameter initialization, for any imput into the network 18"0, the following stochastic order bounds hold:
2 XTL
I
II 1-1
a Σ
G.(xi.K) S.. [JAr(x)]] St.
1 2 XAL
II
C - i=1
TÀg
74g
where (x, K) and Ei, K) are respectively the smallest and largest order statistic in a sample of size K of chi-squared random variables with 1 degree of freedom, independent of each other and of the vectors and
Proof. From Proposition 13 have the following equality
L-1 12
[J [
II 1=1
(6)
where vectors u²), 1 O....,L 1 are defined recursively as 1 W (1-1)) W u (1-1)
and (0) u. Matrices W consist of rows w whth E R 1,... ny where k' argmax (w(0)x(-1) (1-1) 112) and () the Ith layer's input, (0) x.
We assumed that weights in the last layer sampled from Gaussian distribution with mean zero and variance 1/nt.11 Then, by Lemma 10 item 3, (4--))) - (1/m1-1)x² and is independent of (L4-1), In equation (6), using this observation and then multiplying and dividing the summands by c/n;-1 and rearranging we obtain
|J [xx())]
2
u (1)1))
X
1L
1=
i=
1
11-1
((-1)))))
X
n0
Now we focus on / (W.))
((-))). Since we have assumed that the weights are sampled from Gaussian distribution with zero and variance c/14-1, any weight ((())
ikjj 1...:71-11
16

---

Under review as a conference paper at ICLR 2023
can be written v Ve/mi-uukh. where (1)
(1)
vik.j standard Gaussian random variable. We also write
Wik ve/mi-iVie 1V iik where
((0) 1, is an 1--1--11mmm dime-nsnono standard Gaussian random vector. Observe
that for any W' E [K], (, u I ( u 11 and ? u 1111)22 N
minke]k] ((W t,k 33) Therefore.
(v { 1.b. - (V
( u 1 =
(V22,u0-1 21
min Ri-1 kE[K]
max 11-1 kE[R]
Notice that vectors u (1-1) are unit vectors by their definition. By Lemma 10 the inner product of a standard Gaussian vector and a unit vector is standard Gaussian random variable independent of the given unit vector.
By definition, squared standard Gaussian random variable is distributed as x². a chi-squared random variable with 1 degree of freedom. Hence, [ { [V
i,k ud-1)y?> is distributed as the largest order statistic in a sample of size of chi-squared random variables with 1 degree of freedom. We will denote such random variable with (x, K). Likewise, minseguy (Vi.e ( is distributed as the smallest order statistic in sample of size K of chi-squared random variables with 1 degree of freedom, denoted with Et, (x): K).
Combining results for each layer, we obtain the following bounds
[J)r(x) -
X
L 1
n1
max KE(K]
(V:2 i.bb u -
2
d
1
L
R2 E (x.
711
730
1=1
|Ja(()u||
XnL
2
L-1 1
min
(vil) 21 L,k: (
d
1
2 XnL
L
G,(xi,KK K).
kE(K]
74p
[=1
1=1
Then, by Remarks
and
12 the following stochastic ordering holds
1
1
XnL
2
Ex1(2i K) S. [J
Set
Xmy
2
E K),
70
1=
which concludes the proof.
C MOMENTS OF THE INPUT-OUTPUT JACOBIAN NORM
In the proof on the bounds of the moments we use approach similar to Hanin et al. 2222 1 for upper bounding the moments of the chi-squared distribution.
Corollary 2 (Bounds on the moments of [| [x)lll) Consider a maxout network with the settings of Section 2 Assume that the biases are independent of the weights but otherwise initialized using any approach. Let u E T0 be fixed unit vector and x E Rnn be any input into the network, Then
(i) nL (c8) <
[B[]] (x]
TL (cL)2-1
n0
170
Σ n nL [
(ii) Var [|]N(x)]]]]]
n2
(1
(2(11-1) exp 4
82(1-11)
no
nt. } LEN. N.
(ii) E (1)
n2 no
(cK)
exp
F2
E
KiK
where the expectation is taken with respect to the distribution of the network weights. The constants
and E depend on K and denote the means of the smallest and the largest order statistic in sample of
K chi-squared random variables. For K = .,, 10, 8 E [0.02, 0.4] and E E [1.6, 4). See Table in Appendix for the exact values.
Proof. We first prove results for the mean, then for the moments of order >> 1, and finish with the proof for the variance.
17

---

Under review as a conference paper at ICLR 2023
Using mutual independence of the variables in the bounds in Theorem 1 and that if two non-negative univariate random variables and are such that X Sat Y then E all (Müller & Stoyan
Mean
[1 S E for
2002 Theorem 1.2.12),
L-1 II 1=1
L-1
E
[xms
EE
no
1=1
where we used čt. and Ei as shorthands for (x, f K) and (x). K). Using the formulas for the largest and the smallest order statistic pdfs from Remark 14 the largest order statistic mean equals
K-1
K
erf
2/2 C,,
2 Jo
and the smallest order statistic mean equals
E [G.1]
2 L, (1-orf
(V2
1
K-1
x 1/2 c 212 dx = 8.
Here denoted the right hand-sides with L and 8, which constants depending K. and can be computed exactly for = 2 and K = 3, and approximately for higher K-s, see Table It known that E [x²x] n-. Combining, we get
- (c8) < [X]J Y10
nL
aL (cC) 710
Moments of order t 1>1 As above, using mutual independence of the variables in the bounds in Theorem 1 and that if two non non-negative univariate random variables X and Y are such that X Set Y then IE [ I E for all n 2 (Müller & Stoyan, 2002. Theorem 1.2.12),
Jx]Ju( 21 < IE
XnL
E
710
1=1
L-1
nrr.
E
(Xnz)
2
II
E
Σ
E
ng
n2L
n
1=1
Upper-bounding the maximum of chi-squared variables with a sum,
1
ni )=)
mi
TIL ΣΣ
E
Σ
E
E
(x
my
where we used that sum of nyK chi-squared variables with one degree of freedom is chi-squared variable with K degrees of freedom. Using the formula for noncentral moments of the chi-squared distribution and the inequality 1+ See
( mz
E [(xmk))]=
- (())) (miK) (niK +2) (niK + 2)
-t (1+
niKK
2 2 )... (1+ (..((1++ 2t- -2)
nyK
< Kt
exp
Σ
t-1
R
2i
S
c' exp
|nKK f
t²
where we used the formula for calculating the sum of consecutive numbers Σ = t(t - 1)/2. Similarly,
12
E (x2xe) S exp
nt
Combining, upper bound 7 with
+ mLL }
ni
(c
(cK)
(L--))
exp
$2
Σ
ng
niK
1=1
18

---

Under review as a conference paper at ICLR 2023
Variance Combining the upper bound the second moment and the lower bound on the mean, get the following upper bound on the variance
Var [[
n
2 ((1-11 (xall-))exp {
{
1-- Σ n K
}
82(1-1))
which concludes the proof.
Remark 14 (Computing the constants). Here we provide the derivations necessary to compute the constants equal to the moments of the largest and the smallest order statistics appearing in the results. Firstly, the edf the largest order statistic of independent univariate random variables y1..... VK with edf Fx)) and pdf f(x) is
(N ykk
K
Pr
max { x Pr KE[K]
<
<x)
II Pr (yk < <) = (F(x))
Hence, the pdf is K ((F(x)) -1 f(x). For the smallest order statistic, the edf is
K
Pr
min {m))} < = kE(K]
Pr (uk 2 x) = - F(x))
Thus, the pdf is K (1 F(x)) K-1 f(x).
Now we obtain pdfs for the distributions that are used in the results.
de- Chi-squared distribution The edf of a chi-squared random variable xt with gree of freedom is F(x) = (T(#/2))
= erf()a/2), and the pdf is [(x) = (24/TT((/2))- -// 00 12m) x/2 Here we used that r(1/2) = /'r and (11/,xx/2) = v erf( 22) Therefore, the pdf of the largest order statistic in sample of K chi-squared random variables with degree of freedom (x, K) is
K (orf
(v=))
K-1
1
r
2
The pdf of the smallest order statistic a sample of K chi-squared random variables with degree of freedom (x,(x,, is
K-1
K (1 (1-orf erf
Le 2.
T
2m
Standard Gaussian distribution Recall that the edf of standard Gaussian random variable is F(x) = 1/2(1 + (rf(2//2)) and the pdf is f(x) = 1/V2m exp(-xx (222/2). Then, for the pdf of the largest order statistic in sample of K standard Gaussian random variables EN, (0, 1), we get
K
(1+ erf
1
K-1
2K 2K-1-27
2T
2
Constants Now we obtain formulas for the constants. For the mean of the smallest order statistic sample of K chi-squared random variables with degree of freedom (x, K), we get
K
x (1-exf erf (V²)
K-1
8
dr.
2T
The mean of the largest order statistic in sample K chi-squared random variables with degree of freedom Ex, K) is
K-1
K
00
L
x²
erf
dx.
2T
J0
19

---

Under review as a conference paper at ICLR 2023
The second moment of the largest order statistic in a sample of K standard Gaussian random variables (N, 1), K) equals
K
x² (1+
I
K-1
M 2K
V
2T
v2
e
da.
These constants can be evaluated using numerical computation software. The values estimated for K ....., 10 using Mathematica (Wolfram Research, Inc 2022) are in Table
Table 4: Constants and denote the means of the largest and the smallest order statistics a sample of size K of chi-squared random variables with degree of freedom. Constant M denotes the second moment of the largest order statistic in a sample of size K of standard Gaussian random variables. See Remark [4 for the explanation of how these constants are computed.
MAXOUT RANK
8
M
2
1.63662
0.36338
2.10266
0.1928
1.27566
2.47021
0.1207
1.55133
2.77375
0.08308
1.80002
6
3.03236
0.06083
2.02174
3.25771
0.04655
2.2203
3.45743
0.0368
2.39954
3.63681
0.02984
2.56262
10
3.79962
0.0247
2.7121
D EQUALITY IN DISTRIBUTION FOR THE INPUT-OUTPUT JACOBIAN NORM AND WIDE NETWORK RESULTS
We will use the following theorem from Anderson (2003). We reference it here without proof, but remark that it based on the well-known result that uncorrelated jointly Gaussian random variables are independent.
Here we prove results from Section 3.2
Theorem 15 (Anderson 2003 Theorem 3.3.1). Suppose X X..... XN are independent, where X is distributed according to ((p,, E). Let C' = (Cor) be an N x N orthogonal matrix. Then Y.
EN CaBX8 g is distributed according N(va, E), where Va 23=1 3 a= l..... and Y. YN independent.
Remark 16. We will use Theorem 15 in the following way. Notice that is possible to consider vector v with entries sampled i.i.d. from N(O, a²) in Theorem 15 and treat entries of v as a set of 1-dimensional vectors X,, XI...., XN- Then we can obtain that products of the columns of the orthogonal matrix C' and the vector v, Y3- >mm CagVa. are distributed according to N o²) and are mutually independent.
Theorem (Equality in distribution for [x)ll)) ( of Section 2 Let Ro be fixed unit vector and R x * 0 be any input into the network. Then, almost surely, with respect to the parameter initialization, [J [xx(x)u| equals distribution
Consider maxout network with the settings
Xun
cos2
Ne
+ E, K)) cos -00-1)
140
E=
where Vt and (N, 1), K) are independent, U, (0, (N, K), is the largest order statistic in a sample of K standard Gaussian random variables. Here Ty0)) u() denotes the
angle between z!!) = (x)..... ....,.,, x 1)
and u()) = (u
(1) 1 IR where u!))
/1 (1-1))| when W (1-1) (11)) # 0 and 0 otherwise, and u ( u. The matrices W consist rows W! (!l)) TR where k' argmax (W:l) (1-1) + b..)
W
20

---

Under review as a conference paper at ICLR 2023
Proof. By Proposition 13 almost surely with respect to the parameter initialization,
L-1
[J || (L- (( 11 11
II 1=
u 1)
(8)
where vectors u 1 0, O,..., L 1 are defined recursively as u " "n /1-11//| || u
and ( Matrices w consist of rows ( w E R 1, ,,... Ry, where k' ar[] (W(fx((-1) b and x(I-1) is the /th layer's input, x x0
We assumed that weights in the last layer are sampled from Gaussian distribution with mean zero and variance 1/nt-11 Then, by Lemma 10 (|4(2)( - (1/11-1)x?,, and is independent of u(--1), We use this observation in the equation ( multiply and divide the summands in the expression by c/n-11 and rearrange to obtain that
L-1
R2 Σ
|Jr(x)u||2
X
n4-1 (W WW.uu 1
nL-1
i=1
(9)
T--1
1
(W
(1) u (1 1)
XnL
14g
T1
[=1
We define r(1-1) = (x) 11 (1-1) E R and u (t-1) = (ul²
( ..... u 0) E
R"--+t1, ||u|| = 1. We append the vectors of biases to the weight matrices and denote obtained
matrices with 2)(() e TR (m=+)mm+)1) Then ( equals
Tig
u 11)))))))
Xng
14
1)) Since we have assumed that the weights and biases are sampled from the Gaussian distribution with zero mean and variance c/n-11 any weight ((()) j = 1,..., 74-1 (or bias), can be written as Ve/ni-1u,k where (1) Therefore,
Now we focus on / (
i,k.'
vik,j standard Gaussian.
Th 1111) 1))-
(10)
where 27 dimensional standard Gaussian random vectors.
200 Ro--1+1, argmax (2019)
2ff are (m-1 1)- x (1-1))) k
We construct an orthonormal basis B = (bj...., 1+1) of ---1-1 where we set b ((--))/[<<((-> and choose the other vectors be unit vectors orthogonal to b, The change of basis matrix from the standard basis / to the basis is given by BT, see, e.g., Anton Rorres 2013 Theorem 6.6.4). Then, any row 2 an be expressed as
0 ck. .... CR. 1 bm-itt
where Ck.j (20 by). j 1,..., n1-1 +
The coordinate vector of ((1-1) relative to B is (11x o,..,, Vector (I-1) in B has the coordinate vector (11 bi)..., (1 bn)++t)). This coordinate vector has norm since the change of basis between two orthonormal bases does not change the &2 norm; see, e.g., Anton & Rorres (2013) Theorem 6.3.2).
For the maximum, using the representation of the vectors in the basis B, we get
(2))) I (1-1)) max (0) K] k
r
(111)))
max kE K
|| (1-))
max {CR,1.. (11)
KE
kE[K]
21

---

Under review as a conference paper at ICLR 2023
[cosy]
Maxout rank
K 5
0.0
20
40
60 100 0.1
20
60
80
100
0.0
20
60
80
100
Layer
Layer
Layer
Width
Width 10
Width 100
The plots show that (y (()) grows with the network depth and eventually con- verges to for wide networks and maxout rank K > The results were averaged over 1000 parameter initializations, and both weights and biases were sampled from N0, c/fan-in), - (-IEE((E(N()1)) 1), K)) as discussed in Section Vectors and sampled from N (0, I).
Figure 4:
Therefore, in the basis B, (1)
has components (max {cx.1) : c.......: CK', Ck'n)-1+1) By Theo- rem 15 for all & = ...... K,
I..... nw-1 + 1, the coefficients Ck.j are mutually independent standard Gaussian random variables that are also independent of vectors b = I,..., 11-1, by Lemma and of ((t-)).
R1-1+1
[
1111)
max {CH,1} (1(1-1),bb)
Σ
Ch Ch
(1 ba)
kE[K]
(12)
m-
bi) +
Σ
Uy (y( bj),
where (O, K) K)
the largest order statistic in a sample of K standard Gaussian random vari- ables, and 1j ~ (O, 1). Since we have simply written equality in distribution for maxke [ (cx,1) and Cv the variables Ei,; (0, 1), K) K) and Vz. J = 2.... 14-1 are also mutually independent, and independent of vectors b = 1...., R4-1, and of u (!-1), In the following we will use Ei, shorthand for (N, 1), K).
linear combination Σ.=1 a, of Gaussian random variables V1y.... Uni Uj N(u o3).
j 1, ,,...,,. n with coefficients d1,..., 0n distributed according to N(E 0:t1i; Di=1 aFo?). -41+1 Uy (1-())), by) ~ (O0, Eise" (((())))))))) Since EØ=
Hence, Σ Σド=
14-111 (n (,,1)),b)) b,)
(u 11 1), b1) = cos22 Yell 1..0 1) we get
M--1+1
Σ (((, (t)( b1)
(13)
0/ / cos? 700-1, + Es,s cos 100-),001))
where NO. 1). Notice that 1% cos 2 1) u 1) and Fi cos yu x (t are stochasti- cally independent because v and E, are independent and multiplying random variables by constants does not affect stochastic independence.
Remark 17. The result in TheoremBalt Baaso holds when the biases are initialized to zero. The proof simplified this case. There is no need to define additional vectors r((-1) and (((-1) and when constructing the basis, the first vector is defined as b := ((-1) 1/I- [|| The rest of the proof remains the same.
Remark 18 (Effeetts of the width and depth maxout network). According to Theorem 3 the behavior [x]u(x) a maxout network depends on the cos "f0--) which changes as the network gets wider or deeper: Figure demonstrates how the width and depth affect [x)](x)
Wide shallow networks Since independent and isotropic random vectors in high-dimensional spaces tend to be almost orthogonal (Vershynin 2018 Remark 2.3.5), coS 7x00) (()) will be close
22

---

Under review as a conference paper at ICLR 2023
Maxout rank
1009V
[cosy:
0.4
5
0.0
20
40
60
100
0.0
20
60
100
0.0
20
60
100
Layer
Layer
Layer
c 0.3, c 1/R
C
[E
c= 10, > 1/E
Figure 5: The plots show that cos Yy does not converge for 1/E[E*) and converges for e 2 ) The network had 100 neurons at each layer, and both weights and biases were sampled from (0, c/fan-in). The results were averaged over 1000 parameter initializations. Vectors x and u were sampled from ((0, I).
Tele> Vt CooMEE
20 sample (maxout rank)
100
Figure 6: Second moment of (0, 1), for different sample sizes K. It increases with K for for K 2.
any K, 2 < K < 100, and (NVO,,,,
to with high probability for wide networks if the entries of the vectors x and u are i.i.d, standard Gaussian (or i.i.d. from an isotropic distribution). Hence, we expect that the cosine will be around zero for the earlier layers of wide networks and individual units will behave more as the squared standard Gaussians.
Wide deep networks Consider wide and deep networks, where the layers | = 0, L are approximately of the same width n,, so n4, 1/ O..... 1. Assume that c = 1/M 1/E[ (0, 1), J. We will demonstrate that under these conditions. | cos ((()),()) s= for the later layers for K 100. Thus, individual units behave the squared largest order statistics. To see this, we need to estimate cos Yel)) from Theorem 3 which is defined as
(r(0, (1) cos ())) = Pyu [[ 1 (| |||l
(r O)
Ri-1 (x(()) x(!))
(I)
78-- (1
where denoted cos 7,0, with Prus and with ( a before the normalization. Firstly, for (()) we get
()
n
2
14-1
(1
11-1 ni
max
20!!!
+1
n1
RE K
= -cll
n2
max KE[K]
(7((1)
i,k
[| })
(2-1))
2
+
(|| (14)
n--1
71:
n4-1
11]((11)1127))
where in the second line we used that (7 i,k
/
cc/n
20!2
(7(h
N 1).j = 1,..., 1--1-
In the third line, E, E(N(0, 1), K) is the largest order statistic in sample of K standard
23

---

Under review as a conference paper at ICLR 2023
Width 1
Width 2
Width
Width 20
Width 50
layer
100
100
200
22
14
3.0
12
144
16
3.6
200
3.0
1.
14
36
96 3.
1.0
13
35
15
16
32
3A
15
116
150
150
190
150
2 layers
54
50
503
922
12
900
903
5 layers
100
100
190
120
10 layers
100
16
12 3A 14
20
30 12
144
1.6
1.0
300 12
1.4
36
15
06
18
20
124
150
150
150
359
30 layers
54
50
10
53 20 12
922 32
900
28
02
Figure 7: Shown the expectation value the square norm of the directional derivative of the input output map of maxout network for fixed random direction with respect the weights, plotted as a function of the input. Weights and biases are sampled from N No,, 1 1/fan-in) and biases are zero. Inputs are standard Gaussian vectors. Vector the same for one setup. We sampled 1000 inputs and 1000 initializations for each input. The left end corresponds the second moments of the Gaussian distribution and the right end the second moment of the largest order statistic, Observe that for wide and deep networks the mean is closer to the second moment of the largest order statistic.
one-hot vector with 1 random position, and
24

---

Under review as a conference paper at ICLR 2023
(l111111////11)) / /11x are mutually independent standard Gaussian random variables. When the network width is large, 1/mz Di= Si, approximates the second moment of the largest order statistic, and ni-1/n; ss 1 when the layer widths are approximately the same. Then
Gaussians, since by Lemma 10. 2 i -
R4-1 1110 c 011
[=]]
Now we will show that
1/||="111)1 ~ O. Firstly, by the same reasoning as above,
11
Σ
max
20 1,k
+1
kE(K]
d
||
1
CA--1 1 74-2
- +1.
no
j=i-2 Ti
11-1
Since we assumed that the layer widths are large and approximately the same,
1-2
11=) ~ |0 (<E("")) L1 + cE + 1 = |1 (cB(E3)) + Σ
(EE
-=
Using the assumption that c = 1/ we obtain that ||= ~ |0:00 + (1 - 1) and goes to infinity with the network depth. Hence, 1 (1| =
and
n4-1 |1 (1|x(111)) 1E
Now consider T Using the reasoning from Theorem see equations
112 and 13
(E
(1-1) 1s (pfu 11)), I...., ny. Us (0, 1). Then in wide network
27
11
cnu
Eppu (1-1) + vy
11 Pru
(1-1)
(15)
Note that the random variable equations (14 and (15) is the same based on the derivations in Theorem 3 to see this, compare equations
(I1 and (22
Similarly, for the dot product (r()
u
(
in wide network obtain that
11
Cn
E
=
Eppu""
(1-1)
+ u 1 - 1))=
74-1
Hence, we have the following recursive map for Peu
(1)
E
E
3psu
(1-1)
+ v
Pzu (pll
(1)
Pyu
Eppn (1-1) + +U 1 1((phu
(pzu (1-1))² 2
E
(1-1)
(1-1)
B
Pgu
/ E2
(ptn
(E[22)
where we used independence of and E. see Theorem and that E[u] 0 and E 1. This map has fixed points p² = 11, which can he confirmed by direct calculation. To check if these fixed points are stable, need to consider the values of the derivative 0pp0 /pppu at them. We obtain
(1-1)
(1)
0prn
(0// (L-1))
(B[= 1) +1)
0p ([-1))
25

---

Under review as a conference paper at ICLR 2023
(1-1) 1 this partial derivative equals 1/ < 1 for K > 2, since E|=²] see Table When Pru
for K = .., 10 and Figure for K 2. 2...................... 100. Hence, the fixed points are stable (Strogatz 2018 Chapter 10.1). Note that for K 2. 1/ = I, and this analysis is inconclusive. Therefore. if the network parameters are sampled from .N (0, c/m--),, c 1/M = 1 1/E| (0, 1), K) we expect that cos cy(7) ,y((),,0>> = for the later layers of deep networks and individual units will behave more as the squared largest order statistics. Figure 4 demonstrates convergence of cos (x()),.()) to
with the depth for wide networks, and Figure 5 shows that there no convergence for c < 1/
and that the cosine still converges for c 1/E[=2)
Remark 19 (Expectation of [x]][x) in a wide and deep network). According to Remark 18
for deep and wide networks, we can expect that 1 (y(t-1), 1) R if 1/M, which allows
obtaining an approximate equality for the expectation of [x]x] Hence, using Theorem 3
L-1
[x)(()
XML
(16)
720
T2.
Then, using mutual independence of the variables in equation 16
E [
E L
E
(V))
no
Th
Since M = EE[ 1), K))J) see Table
and 1/M, we get
B||l (x)J)(x))
TiL (cM))-1 no
T4n
Remark 20 (Lower bound on the moments in a wide and deep network). Using (6 and taking into account the mutual independence of the variables,
1
n2
E||l [
X
(NO, 1).
140
T4
m4
L
E
(XmL.)
E
Σ
(N,,
no
T1
L-1
E
(X
Σ
E (E..
n
(17)
where in the last inequality, used linearity of expectation and Jensen's inequality since taking the tth power for 2 1 is a convex function for non-negative arguments. Using the formula for noncentral moments of the chi-squared distribution and the inequality In I 1/x, V/x > 0,
meaning that r = cxp(Inx)) 2} 2 expf1 1/2x,, we get
1 F
1 m4
2i nr
E
(nL) (nz + + ... (nr 21 -
II 1=U
1+
(18)
2i
t 1
exp
2 exp
nyy + 2i
2n,
where in the last inequality, we used that /1z + 21) 2 2/(nz + 2 1/(2nz) for all i, 2 1.
Using that (E,, 1), K))J] M, see Table 4 and combing this with [1 and 18
E J 21
exp
t t-
(cM))((--)
(19)
11o
2nz.
The bound in (19 can be tightened if tighter lower bound on the moments of the sum of the squared largest order statistics in a sample of K standard Gaussians is known. To derive lower bound on the moments 2 2 for the general case in Corollary 2] it is necessary obtain a non-trivial lower bound on the moments of the sum of the smallest order statistics a sample of K chi-squared random variables with 1 degree of freedom.
26

---

Under review as a conference paper at ICLR 2023
E ACTIVATION LENGTH
Figure 8 demonstrates close match between the estimated normalized activation length and the behavior predicted in Corollary 21 and
Here prove the results from Subsection 3.3
Corollary 21 (Distribution of the normalized activation length). Consider a maxout network with the _ Then, almost surely with respect the parameter initialization, for any imput
settings of Section into the network Ro and l' 1,..., the normalized activation length A is equal in distribution to
1 730
n (MO..)))
R2
(N (N. 1))))
2
14 Σ
+
n
where (00) = (x1...., Xngy 1) EN, 1), K) is the largest order statistic in sample of K standard Gaussian random variables, and Ei, (0, 1), K) are stochastically independent. Notice that variables (VV K) with the same indices are the same random variables.
Proof. Define z(0) (x1,..., X 1) E IR Append the bias columns to the weight ma-
trices and denote obtained matrices with 27(1) E R=xx(m-++1) Denote 20(h) E 1n--tt,,
k' = argmaxx[[K]] (2 ((( with 27 Under this notation, 1| = (1)x + 1),
i,k+
[x(1 = 1 (| Then ||² equals
1)
10x = 20
11
20
1]11
(1'
2)
21
21)
||
-2]|
1)
(0)
21
|
where we multiplied and divided |200
11 2 by (1r(-1))]
1| at each step. Using the approach from
Theorem 3 more specifically equations TO cos Yy(0 ( = 1,
and with (t() /11/||[(()] implying that
ni
10"
x 112
d
|
1
Et.
71
Thy
n--1
7111
j=2
1012
110
where E, = E (N K) is the largest order statistic in a sample of K standard Gaussian random variables, and stochastic independence of variables Et, follows from Theorem
Corollary
5 (Moments of the activation length). Consider a maxout network with the settings of Let x R be any input into the network. Then, for the moments of the normalized activation length, the following results hold.
Section
Mean:
1' [A9]] = =11= (cM))
Σ (cM1)" (+00)--+11) -tt1)
+
n
j=2
27

---

Under review as a conference paper at ICLR 2023
Normalized sarmallleddd.cm activation length
value Equality in distributiio
10
Formula
10
20
25 Layer
30
35
40
Figure Comparison the normalized activation length with the equality in distribution result from Corollary [1 and the formula for the mean from Corollary Plotted are means and stds estimated with respect to the distribution of parameters / random variables (N, 1), K) averaged over 100, 000 initializations, and numerically evaluated formula for the mean from Corollary 5 All layers had 10 neurons. The lines for the mean and areas for the std overlap. Note that there is no std for the formula in the plot.
Variance:
11.00011 n
Var[A
21
2r
Σ
exp
niK
Moments of the order t 2 2:
11 1
E (A(1))
5 2
2 K'l
exp
n
KeK
1)
(cK)
12
+ (2(1' - 1)))
mik
n/-1
exp
(Au3))
|1 122 no
(eM)
(cM))((-++11
n
j=2
where expectation is taken with respect to the distribution of the network weights and biases, and M a constant depending on K that can be computed approximately, see Table the values for K ==2 2,..., 10.
Proof. Mean
Taking expectation in Corollary 21 and using independence of (N 1),
||5
100
(cM))
(cM))-st11
(20)
no
n
where M is the second moment of (N, 1), K), Table 4 for its values for K = .,...,10
Moments of the order t 2 2 Using Corollary 21
we get
E (1:))))
(21)
n
=E
no
TN
m
Upper bound First, we derive an upper bound on 21p. Notice that all arguments in 21 are positive except for measure set of E Eh ni According to the power inequality, for any
28

---

Under review as a conference paper at ICLR 2023
F1....;2r En 21,....... > 0 and any R, 1, (21 +... +.... I nn' (2) +.... x
Using the power mean inequality first on the whole expression and then on the second summand,
!)]
72
S 2
E
TO
11
(22)
+
(1' - 1)
E
1=2
-=1
Using independence of Er,s. (22)
equals
1)
|
22
2
E
Σ
n
7i
1=1
i=1
1))
(
+ (2(1' - 1))
T
ni
Upper-bounding the largest order statistic with the sum of squared standard Gaussian random variables, we get that Σ E S Xnyx- Hence,
E (AV1))
o S 2
1 1
1'
[(xnx)]]
- E
T
t=1
(23)
p'
1'
(2(1' - 1)) +
e nf
j=2
Using the formula for noncentral moments of the chi-squared distribution and 1 1 S E", Vx E R.
c¹
c²
E
(x
(miK) (n:K + ... (n;K + 2t -2) 2)
ni
ck (1 (+
n )
2
2
... 1+
2t 1--2 2 niK
S K² exp
Σ niKK
t-1 2i
exp
|n(K] S
t²
where we used the formula for calculating the sum of consecutive numbers Σ/=1 t(t 1)/2. Using this result in 23 we get the final upper bound
E (ar)))]
1r
(U)
12
< 2
1 Kal
exp
n6
niK
1=1
(cK))((' (cK))((-i+11 -j+1)
t²
+ (2(1' - 1))²
n/-1
K
Lower bound Using that arguments in (T)) are non-negative and / > 1, we can lower bound the power of the sum with the sum of the powers and get,
11)
(AV)))
(0) 2
E
Σ
ni
=
R1
12
n2
71
(0)
12
72
d'
Σ
E
Σ
E
n
n
nf
1=1
[
29

---

Under review as a conference paper at ICLR 2023
where used the linearity of expectation in both expressions and Jensen's inequality in the last line. Using that (EV,,,,) -)) = M, see Table we get
E (a))t))
|| 11 n
(
(cM)" "
(24)
nj nÿ-1
Variance We can use an upper bound on the second moment as an upper bound on the variance.
Remark 22 (Zero bias). Similar results can be obtained for the zero bias case and would result
in the same bounds without the second summand. For the proof one would work directly with the vectors x(), without defining the vectors y(!,, and to obtain the equality in distribution one would use
Remark 17
F EXPECTED NUMBER OF LINEAR REGIONS
Here we prove the result from Section 5.1
Corollary 6 (Value for Ggná). Consider a maxout network with the settings of Section 2 Assume that the biases are independent of the weights but otherwise initialized using any approach. Consider the pre-aee aeeaactivation feature Sz,k of a unit 2 I....,,N N. Then, for any E N.
miK +1)}
sup E [[VVGA(()))) xER"D
S <no" max (cK) =2") exp 2
Proof. Distribution of VE, & is the same as the distribution of the gradient with respect to the network input V.N (x) in a maxout network that has single linear output unit and L = L(z) layers, where I(=) is the depth of unit 2. Therefore, will consider (supxeR""
(x) that since b 1, V (x) = (x)) = (xx u for a 1-dimensional vector u = (1). Hence,
Notice
[ ||u|| [lu|==,,ERR*,
sup || [x) || [x)x
(25)
where the matrix norm is the spectral norm. Using that matrix and its transpose have the same spectral norm, (25) equals
xx)x)]
|| (x)ull
sup |
Therefore, we need to upper bound
'))
)1)
26
sup
sup [|u - L
sup E
sup 1,
[xx(())
XER"0
xER"O
||u|||
where we used Jensen's inequality.
Now we can use an upper bound on [[[]] x (x)u|22 | from Corollary Fo
which holds for any x, E we get
Jull 1, and thus holds for the suprema. Recalling that nj
+ }
(1)) (131(()) s nou
(cK)
exp
12
Σ
n
Hence,
L-1
[ (x)ul(21)) 1 s no - (cK) E
exp
Σ
+
2
n
1ml
Taking the maximum over L E (l,..., ...,.,,,,, L). the final upper bound is
}1}
ni = max (cK)) 6-' exp
Σ TuK
30

---

Under review as a conference paper at ICLR 2023
Now provide updated upper bound on the number of p-paatial activation regions from T seran & Montúfar (2021) Theorem 9). In this bound, the case r = 0 corresponds to the number of linear regions. For detailed discussion of the activation regions of maxout networks and their differences from linear regions, see I'seran & Montúfar (2021). Since the proof of Tseran & Montúfar (2021 Theorem 9) only uses Carad for S nop, obtain the following statement.
Theorem 23 (Upper bound on the expected number of partial activation regions). Consider maxout
network with the settings of Section with N maxout units. Assume that the biases are independent of the weights and initialized so that:
Every collection of biases has a conditional density with respect to Lebesgue measure given the values of all other weights and biases.
There exists Coias so that for any pre-activation features Ci...., Se from any neurons, the conditional density of their biases Pb.....be by given all the other weights and biases satisfies
sup Pb (bi...., ,.....,,)))) I Ohins BRER
r {0,..., no)- Let Cgrad = Th and T 2 Craa Then, there exists 6g - 1 1/(2 Cbias) such that for all cubes C TR with side length ó > 80 we have
1/2 max (1, (cK)(L-1)/2) exp(no 1/(n/K 1)}
10 p-pariill activation regions of N in C7
(r
(< (M)
N N no
vol(C)
(TKN)" 20g
N n1
(2K))ng!
Here the expectation is taken with respect to the distribution of weights and biases in N. Of particular interest is the case r 0, which corresponds to the number of linear regions.
G EXPECTED CURVE LENGTH DISTORTION
In this section we prove the result from Section 5.2
Let M be smooth 1-dimensional curve in Io Fix smooth unit speed parameterization of M = ((0, IJ) with 7 P - "o" T(T) (71 (T), .... Tmo (7)). Then, parametrization of the curve
(MM) given by mapping == N y T: R R" Thus, the length of M(M) is
(en(N((M)) = f. [T)(T)]
Notice that the input-output Jacobian of maxout networks well defined almost everywhere because for any neuron, using that the biases are independent from weights, and the weights are initialized from a continuous distribution, P(k' argmax (1-1) ((() b.
argmax&e(K] (W!)2 (t-1) + (()) = =,.=1,...... - we used the chain rule, and can employ the following lemma from Hanin et al. (202T). We state
# Hence, ['(r)= = J (
where
here without proof which uses Tonelli's theorem, power mean inequality and chain rule.
Lemma 24 (Connection between the length of the curve and [x)u|, Hanin et al. 2021
Lemma C.I). For any integer 2 O,
[ [[))((())))/)) di = (x)]]))
[len(N(M)))
where u E Ie is a unit vector.
Now we are ready to proof Corollary
Corollary 7 (Expected curve length distortion). Consider a maxout network with the settings of Section Assume that the biases independent of the weights but otherwise initialized using any
31

---

Under review as a conference paper at ICLR 2023
approach. Let M be smooth 1-dimensional curve of unit length in R Then, the following upper
bounds on the moments of len(N(MM) hold:
E [llen(N(MM)
nL
(cl)
Var [len(N
- aL
(cL)L-1
E [ln)(V(M)))
<
n no
(cK)
exp
t
1
2
n
aL
where & is a constant depending on K, see Table in Appendix C
for values for K = .2....10. 10.
Proof. By Lemma 24
(MM))) < E [IJx(x)ull') [B]]](())
where used Jensen's 's inequality to obtain the last upper bound. Hence, using Corollary get the following upper bounds on the moments on the length of the curve.
Mean
n2 1 (cl) =²
E [[en(V(MM) & (E [x)[(x)))
no
Variance
Var [len(N((M)) < E [len(V((M))
I (cC)
n
7i0
Moments the order t 2 3
E [ln(NV(M)) < (EE [1]xx(x)u|111])
n2
2/2
(cK)
1(h-11
exp
f2
Σ
1-1
2
niK
H NTK
Here we prove the results from Section 5.3
Corollary 9 OOn-diagonal NTK). Consider maxout network with the settings of Section 2 Assume that nu = 1 and that the biases are initialized to zero and are not trained. Assume that S < c < L, where the constants S, L are as specified in Table 4 Then,
||5 10 (08) L-2)
P EX x)) S ||. 1012 (LC)L-2MLL-
P.
ng
no
( 2PP (P (ck))((-22)))) 2(4-2) ( 10) n
exp
t
+4
ni
K
j=l
where = Et--1 nu Pw = Et mim-1, and M as specified in Table 4
Proof. Under the assumption that biases not trained, on-diagonal NTK of maxout network is
I K ni-1
ON awly i.ki
KN((x,)
(x)
j=1
Since in maxout network for all k-s except k k' = argina (W!l) x111) + b.k) the deriva-
(1)
tives with respect to the weights and biases are zero, on-diagonal NTK equals
mi ni-1 ΣΣΣ
ON
2
Kx(x, x)
(x)
j=1
Notice that since we assumed a continuous distribution over the network weights and the biases are zero, the partial derivatives are defined everywhere except for the set of measure
32

---

Under review as a conference paper at ICLR 2023
Part I. Kernel mean (x,,((x, x)] the network weight is
Firstly, using the chain rule, partial derivative with respect to
aN
aN
aw (1) (x)
x))x (1-1) = JN(x Jeix )exxx
0x
Recall that we assumed nt 1. Therefore, we need to consider (aN(x)oW .k,'))
([)x(()) (x( where u = c₁- Combining Theorem and Corollary 21 in combina- tion with Remark 22 for the zero-bias case and using the independence of the random variables in the expressions,
E
|J (xl (xx (
27
80 E
XnI.
= Exx K)
T2
(26)
(N 1), K)3 ]
E
(1)
7o
n i=1
j=l
where we treat the (1 (1-1)dt 1)th layer as if has one unit when we use the normalized activation length result. Then, using Corollaries 2 and
[J)((()) (x (x
< || 1100 cl ch-1-1nc!-
E
non
Taking the sum, we get
L ng 14-1
ON
2
B[Ky(x, = [ikw]==> E
(x)
aw())
11 14-1
11
L -b- s 11 (c
||
P.
non
n2
where p= Et ny denotes the number of neurons in the network up the last layer, but including the input neurons. Here we used that for K 2 2, both L,M 2 1, see Table Similarly,
Ri Bi-1
1 || 10 (cS) 1-
K[Kw]
2
non
n10
Here used that for K 2 2, and M 1, see Table
Part II. Second moment
[x, Using equation
26 with Corollaries
and
17
ON
(xg (z) 47
(x)
(w i
27
T_- 1
n2
c n2
Ss0
X
Ex7 K))
(0) no
II
Σ
E
EN 1), K)²
n
n
j=1
1=1
j=1
<2(cK)2(I-22)
11 1|
exp
4
Σ
1
+1
(27)
nf
ny K K
ni
33

---

Under review as a conference paper at ICLR 2023
Notice that all summands are non-negative. Then, using AM-QM inequality,
L
aN
BKKa(( x) E
(x)
aw 1
)
b
ni-1
ON
- Pw
E
(x) aw
1=1 i=1
4 ny A1-1
11 (0)
4-1
+11
+11}
S Pw
ΣΣ 2(cK)2(L-2)
ni 0
2
EXP
(
4
ngK
niK
1=1 1= 1=1
|
S (cK)
exp
ngK
where Pw- -ti 1 denotes the number of all weights in the network.
I EXPERIMENTS
I.1 EXPERIMENTS WITH SGD AND ADAM FROM SECTION
In this subsection we provide more details on the experiments presented in Section 6 The implemen- tation of the key routines is available at https://anonymous. htwp:/onwmnouuuooooooooooooo
expected gradient-68BD Experiments were implemented in Python using TensorFlow (Martín Abadi et al. 2015), numpy (Harris et al. 2020) and mpi4py (Dalcin et al. (01I). The plots were created using matplotlib (Hunter 2007). We conducted all training experiments from Section on a GPU cluster with nodes having 4 Nvidia A A100 GPUs with 40 GB of memory. The most extensive experiments were rumning for one day on one GPU. Experiment in Figure 2 was run on a CPU cluster that uses Intel Xeon lake.akeSP processors (Platinum 8360Y) with 72 cores per node and 256 GB RAM. All other experiments were executed on the laptop ThinkPad T470 with Intel Core i5-7200U CPU with 16 GB RAM.
Training experiments Now we discuss the training experiments. We use MNIST LeCun & Cortes 2010), Iris (Fisher 1936). Fashion MNIST (Xiao et al. (017)). SVHN (Netzer et al. 2011). CIFAR-I0
and CIFAR-100 datasets (Krizhevsky et al. 2009). All maxout networks have the maxout rank K 5.
Weights are sampled from C(OO./fann- c/fan-in in fully mully-connected networks and N (0, c/(k² fan-in)),
where A the kernel size, in CNNs. The biases are initialized to zero. ReLU networks are initialized using He approach (He et al. 2015). meaning that c = 2. All results are averaged over 4 runs. We do not use any weight normalization techniques, such as batch normalization (Toffe & Szegedy 2015). We performed the dataset split into training, validation and test dataset and report the accuracy on the test set, while the validation set was used only for picking the hyper-parameters and was not used in training. The mini-batch size in all experiments is 32. The number of training epochs was picked by observing the training set loss and choosing the number of epochs for which the loss has converged. The exception is the SVHN dataset, for which we observe the double descent phenomenon and stop training after 150 epochs.
Network architecture
Fully connected networks have 21 layers. Specifically, their architecture is
[5xfc64, 5xfc32, 5xfc16, 5xfc8, out),
where "5xfe64" means that there are 5 fully-connected layers with 64 neurons, and "out" stands for the output layer that has the number of neurons equal to the number of classes in a dataset. CNNs have a VGG-19-lie architecture (Simonyan Zisserman 2015) with 20 or 16 layers, depending the input size. The 20-layer architecture is
cxcon964, mp, excony128, mp, 4xconv256, mp, 4xconv512, mp, 4xconv512, mp, 2xfc4096, fc1000, out],
34

---

Under review as a conference paper at ICLR 2023
where "conv64" stands for a convolutional layer with 64 neurons and "mm"" for max-pooling layer. The kernel size in all convolutional layers is 3 x 3. Max-pooling uses 2 x 2 pooling windows with stride 2. Such architecture is used for datasets with the images that have the side length greater or equal to 32: CIFAR-10, CIFAR-100 and SVHN. The 16-layer architecture is used for images with the smaller image size: MNIST and Fashion MNIST: This architecture does not have the last convolutional block of the 20-layer version. Concretely, it has the following layers:
[2xconv64, mp, 2xconv129 128, mp, 4xconv256, mp, 4 4xconv512, mp, 2xfc4096, fc1000, out],
Max-pooling initialization To account for the maximum in max max-pooling layers, maxout layer appearing after a max-pooling layer initialized as its maxout rank was K x m²² where m the max-pooling window size. The reason for this is that the outputs of computational block consisting of a max poollioo window and a maxout layer are taking maxima over K x m² linear functions, max W1 max(x1,.... X,n + b b...... W K max X,n? } + bx} where the fi are Km² affine functions. Therefore, we initialize the layers that follow max- maox-polling layers using the criterion for maxout rank m² x K instead of K. In our experiments, K = 5,
max f1, ................................ fKm")
2, and m2 x K = 20. Hence, for such layers, we use the constant c = 1/M = 0.26573, where M is computed for K 20 using the formula from Remark [4] in Appendix C All other layers that do not follow max-pooling layers are initialized as suggested in Section We observe that max-pooling
initialization often leads slightly higher accuracy.
Data augmentation There no data augmentation for fully connected networks. For convolutional networks, for MNIST, Fashion MNIST and SVHN datasets we perform random translation, rotation and zoom of the input images. For CIFAR-10 and CIFAR-100, we additionally apply a random horizontal flip.
Learning rate decay In all experiments, initial learning rate for all network and initialization types based on their accuracy on the validation dataset. The learning rate was halved every rith epoch. For SVHN, 10, and for all other datasets, 100.
the learning rate decay and choose the optimal
SGD with momentum We use SGD with Nesterov momentum, with the momentum value of 0.9. Specific dataset settings are the following.
MNIST (fully-connected networks). Networks are trained for 600 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.002, maxout networks with e = 0.1: 0.002, maxout networks with e = 2: x 10-7, ReLU networks: 0.002.
Iris. Networks are trained for 500 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.01, maxout networks with c= 0.1: 0.01, maxout networks with 2: 4 10-8 ReLU networks: 0.005.
MNIST (convolutional networks). Networks are trained for 800 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.009, maxout networks with max-pooling initialization: 0.009, maxout networks with 0.1: 0.009, maxout networks with c 2: 8 x 10-6, ReLU networks: 0.01.
Fashion MNIST. Networks trained for 800 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.004, maxout networks with max-pooling initialization: 0.006, maxout networks with c 0.1: 0.4, maxout networks with c = 2: 5 x 10-6, ReLU networks: 0.01.
Networks are trained for 1000 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.004, maxout networks with max- max-pooling initialization: 0.005, maxout networks with c 0.1: 0.5. maxout networks with
CIFAR-10.
10-5 ReLU networks: 0.009.
CIFAR- 100. Networks are trained for 1000 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.002, maxout networks with max- max-pooling initialization: 0.002, maxout networks with c = 0.1: 0.002, maxout networks with 2: 8 10-5 ReLU networks: 0.006.
35

---

Under review as a conference paper at ICLR 2023
Table Accuracy on the test for maxout networks trained using SGD with Nesterov momentum for values of e less than or equal to c = 0.55555, the value suggested in Section
VALUE OF
0.01
0.07
0.05
0.1
0.2
0.3
0.4
0.5 0.55655
FULLN-CONNECTED
MNIST
11.30 +1.00
11.35
+0.00
11.35
11.35=0..0
+0000
11.30 11.30+0.00
+0.00
11.35 +0.00
97.63=0.10
un 97.82 07.1
97, 17.2200.1
Iís
30.00000
30 30.0000
31 31.67
50.00.00
30
60.83 30.50
10.67
92. 5.1.....4444
90.8 t3
CONVOLUTIONAL
MNSST
11.3540.00 00
11.3540.00
11.35-0.00
11.35
90.56-2.00
20.53=0.02
90.6
90.56
90.6740.07
CIFAR-10
10.00.00
1000.00
10+00000
1000.00
10.0.00
90.97=0.11 91.15=0.07 p1agtoo.1s 91.4+0.22
SVHN. Networks are trained for 150 epochs. The learning rate is halved every 10 epochs. Learning rates: maxout networks with maxout initialization: 0.005, maxout networks with max- moxliooling initialization: 0.005, maxout networks with 0.1: 0.005, maxout networks with c = 2: x 10 ReLU networks: 0.005.
Adam We use Adam optimizer Kinigma Ba (2015) with default TensorFlow parameters $1 0.9, B2 0.999. Specific dataset settings are the following.
MNIST (fully-connected networks). Networks are trained for 600 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.0008, maxout networks with c = 2: 0.0007, ReLU networks: 0.0008.
MNIST (convolutional networks). Networks are trained for 800 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.0001, maxout networks with max-pooling initialization: 0.00006, maxout networks with 2: 0.00004, ReLU networks: 0.00009.
Fashion MNIST. Networks are trained for 1000 epochs. The learning rate is halved every 100 epochs. Learning rales: maxout networks with maxout initialization: 0.00007, maxout networks with max-pooll nax-pooling initialization: 0.00008, maxout networks with c 2: 0.00005,
ReLU networks: 0.0002.
CIFAR-10. Networks trained for 1000 epochs. The learning rate is halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.00009, maxout networks with max-pooling initialization: 0.00009, maxout networks with c 2: 0.00005,
ReLU networks: 0.0001.
CIFAR-100. Networks are trained for 1000 epochs. The learning rate halved every 100 epochs. Learning rates: maxout networks with maxout initialization: 0.00008, maxout networks with max-poolie max-pooling initialization: 0.00009, maxout networks with c = 2: 0.00005,
ReLU networks: 0.00009.
1.2 ABLATION ANALYSIS
Tables 5|and show the results of the additional experiments that use SGD with Nesterov momentum for more values of and K 5. From this, we see that the recommended value of e from Section closely matches the empirical optimum value of Note that here we have fixed the learning rate across choices of c. More specifically, the following learning rates were used for the experiments with different datasets. MNIST with fully cullecconnected networks: 0.002: Iris: 0.01; MNIST with convolutional networks: 0.009; CIFAR-10: 0.004.
1.3 BATCH NORMALIZATION
Tables and X report test accuracy for maxout networks with batch normalization trained using SGD with Nesterov momentum for various values of c. The implementation of the experiments is similar to that described in Section I.I except for the following differences: The networks use batch normalization after each layer with activations; The width of the last fully connected layer is 100, and all other layers of the convolutional networks are 8 times narrower; The learning rate is fixed at 0.01 for all experiments. We use the default batch normalization parameters from TensorFlow. Specifically, the momentum equals 0.99 and = 0.001. We observe that our initialization strategy is still beneficial when training with batch normalization.
36

---

Under review as a conference paper at ICLR 2023
Table 6: Accuracy on the for maxout networks trained using SGD with Nesterov momentum for values of greater than or equal to e = 0.55555, the value suggested in Section
VALUR OF
0.55555
0.6
0.7
0.8
0.9
1.5
2
10
FULLN-CONNECEDD
MNIST
00
77.77
97.91
-0.11
70.82=38.12
78.94-38.18
y7. 00.19
9.8 4.000.00
9.8 9.8
g.8
liis
90.83 73.633
00.83
90.00000
90
3.00000
30 70.00
30 3.00000
30
30.00000 30 t0.00
CONVOLUTIONAL
MNIST
99.57 00.07
99.59#00.02
64.694-44.50
44.9
9.840.00
9.840 00
9.8 9.840.00
9.8 9.840.00
9.840
9.840.00 00
+1.4+ #0.25 40..5
1000000
10.0000
10.0000
10
10.00000 10
Table 7: Accuracy the test set for maxout networks with batch normalization trained using SGD with Nesterov momentum for values less than or equal c = 0.55555, the value suggested in Section 4 Observe that the recommended value of from our theory closely matches the empirical optimum value of C.
VALUE OF
10
10
10
0,001
D.55653
CONVOLUTIONAL
MNIST CIFAR 10
1.351..000 oo
11.35:0.00 oo
33.33200008
99.35-2.05
99.32 0..2220.00
99 20
09.41 40.07
10=0.00
10=0.00
1000.00
74.85
75.72#4.04
77.16
77.6841. 1.07
Table Accuracy on the test set for maxout networks with batch normalization trained using SGD with Nesterov momentum for values of greater than equal to 0.55555, the value suggested in Section 4 Observe that the recommended value from our theory closely matches the empirical optimum value of c.
VALUE OF
0.55555
100
1000
10a
10
CONVOLUTIONAL.
MNIST CIFAR-10
99.41 +.0.7
90.39 +.000
99.35 99.35
+0.02
98.83
97.60 97.69 00.31
96.11+1.40
98.09 1. 39.27
87.63+1180
77.68
79. 200 re
75.6241.000 1.05
66.23
50.97-2.28
21.8
40.27
I.4 COMPARISON OF MAXOUT AND RELU NETWORKS IN TERMS OF THE NUMBER PARAMETERS
We should point out that what is a fair comparison is not as straightforward as matching the parameter count. particular, wider networks have the advantage of having a higher dimensional representation. A fully connected network will not necessarily perform as well as convolutional network with the same number of parameters, and a deep and narrow network will not necessarily perform as well as wider and shallower network with the same number of parameters.
Nevertheless, to add more details to the results, we perform experiments using ReI.U networks that have many parameters as maxout networks. See Tables and TQ) for results. We modify network architectures described in Section [.] these experiments the following way. In the first experi- ment, we use fully connected ReLU networks 5 times wider than maxout networks. For convolutional networks, however, the resulting CNNs with ReLU activations would be extremely wide, so we only made it 2 times wider. In our setup, 5 times wider CNN network would need to be trained for longer than 24 hours, which is difficult in our experiment environment. Maxout networks only required much shorter time of around 10 hours, which indicates possible benefits in some cases. In the second experiment, we consider ReI.U networks that are times deeper than maxout networks. More specifically. they have the following architecture: [25xfe64, 25xfc32, fc16, 25xfe8, out). As expected, wider networks do better. On the other hand, deeper ReLU networks of the same width do much worse than maxout networks.
37

---

Under review as a conference paper at ICLR 2023
Table 9: Accuracy the test set for networks trained using SGD with Nesterov momentum. Fully- connected ReLU networks are 5 times wider than fully-connected maxout networks. Convolutional ReLU networks 2 times wider than convolutional maxout networks. All networks have the same number of layers.
MAXOUT 0.58555
RELU
FULLY-CONNECTED
87.15
20.02
Ini
91.67=3.73
92.5-2.70
CONVOLUTIONAL
MNIST
99. 59 0.004
30.54-0.02
Table 10: Accuracy the test set for networks trained using SGD with Nesterov momentum. Fully- connected ReLU networks are 5 times deeper than fully-connected maxout networks but have the same width.
Maxout Init
He init
VALUE OF
0.55305
FULLY
MNIST
97. 07.8490 15
63. 63.474333.2
38


Reviews:
[{"review_id": "r16HF8iKNZ6", "reviewer": "Reviewer_uU8d", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the gradients of a maxout network with respect to inputs and parameters. Based on their results, they also obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK. With their theoretical understanding, they justify a parameter initialization strategy that avoid vanishing and exploding gradients in maxout networks. They also empirically verify the success of their initialization strategy on multiple datasets.", "summary_of_the_review": "Overall I believe the paper is slightly above the acceptance level, and I would like to raise my score if the authors address the above concerns.", "strength_and_weaknesses": "Pros:\n1. Interesting theoretical studies from various perspectives, including bounds for directional derivative, moments, distribution of input-output Jacobian. \n2. Good implications that refines previous bounds on the expected number of linear regions, and new results on length distortion and the NTK.\n3. Justification for the parameter initialization strategy and hyperparameter selection are reasonable.\n\nCons:\n1. Empirical ablation studies are needed to verify the selection strategy of hyperparameters. The authors compute a range of combinations of K and c given $c = 1 / M$. It would be great to test if the selected combination works better than others. For example, you can fix K=5 and vary a range of c to test the performance.\n2. Is there any difference between the proposed initialization and the original one proposed by Tseran & Montúfar (2021)? It would good to see if the original strategy can be improved by the new theoretical understanding.\n3. About max-pooling initialization, why initialize some layers as $K \\times m^2$? Some motivations are needed since the authors report it achieves better performance.\n4. More baseline strategies and architectures are needed to verify the effectiveness of the proposed initialization strategy. Currently the authors use VGG and He Initialization as a comparison, but He Init is designed for ResNet and it could be more effective under ResNet. So it would be great to compare maxout and Relu under ResNet, which is more like an apple-to-apple comparison. In addition, since the authors aim for a batch normalization-free network setting, comparing the proposed strategy with recent initialization methods targeting on this setting would be more fair, such as Fixup, Rezero.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RmhpOVobs3a", "reviewer": "Reviewer_RjF3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper derives multiple theoretical insights into the properties of maxout neural networks at initialization. In particular, they derive results for the distribution of Jacobian-vector product norms, number of linear regions, NTK, and curve length distortion. They also provide recommendations for initialization of maxout networks based on their analysis, leading to significantly improved performance when trained with SGD and Adam compared to naive He initialization.", "summary_of_the_review": "Overall, this is an interesting and dense paper deriving multiple new results for maxout networks. I don't have the background to fully appreciate the contributions of the paper, as I indicated above. I am currently voting for a weak accept, but with a low confidence score.\n\n## References\n\n[1] [_On the Expected Complexity of Maxout Networks_](https://arxiv.org/abs/2107.00379);\nHanna Tseran, Guido Montúfar", "strength_and_weaknesses": "## Strengths\n\nThe paper provides a serious study with multiple results on various properties of the maxout networks, including some practically relevant recommendations. The results of the analysis are presented relatively clearly, and the implications are discussed when possible.\n\nThe analysis also suggests an initialization strategy that the authors demonstrate works better than naive initialization.\n\n## Weaknesses\n\n**W1**: I don't fully understand why the terms in Eq. (1) are decomposed the way they are, and why the authors focus on the analysis of the term $\\|J_{\\mathcal{N}}(x^{(l)}) u\\|$ and $\\|x^{(l)}\\| / n_l$.\n\n**W2**: As the authors mention, the particular initialization strategy derived was already proposed for maxout networks in [1]. \n\n**W3**: The significance of the empirical results is not obvious, as the authors remove batch normalization and consequently do not achieve particularly outstanding results for the respective benchmarks.\n\n**Discalimer.** I am not an expert in this area, and not familiar with other works. The paper is very technical and I cannot judge the significance of most of the derived theoretical results. Please assign a low weight to my review.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LLU45OETnGf", "reviewer": "Reviewer_3PQc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at maxout networks which are randomly initialized with parameters drawn from a Gaussian distribution. It then looks at the directional derivative (i.e., the input-output Jacobian times a fixed vector). The authors derive a variety of statistics for this derivative (bounds, distribution, expected moments). Based on their findings, they suggest a good parameter initialization for wide maxout networks. They experimentally validate that their proposed parameter initialization works well.", "summary_of_the_review": "Outside of my domain of expertise, but looks like a well-written paper with a significant technical contribution (although quite directly building on previous work done for ReLU units).", "strength_and_weaknesses": "Large parts of this paper are outside of my domain of expertise, so I am unable to judge certain aspects confidently.\n\nThis looks like a strong paper. The theoretical results seem extensions to existing proofs for ReLU networks (e.g., Hanin), but the extensions seem non-trivial. The theoretical results are validated thoroughly using numerical experiments, and it is nice to see that concrete and simple recommendations are made for practitioners in section 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HK5C-2o9Jvi", "reviewer": "Reviewer_pCnC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study theoretical questions regarding the training of randomly initialized fully-connected neural networks with **maxout** nonlinear activations. In particular, they derive the moments of the input-output Jacobian of the network, which motivates a parameter initialization strategy that could avoid the exploding and vanishing gradient problem in wide networks. They validate their results by experiments, and also derive some other theoretical properties, such as bounds on the expected number of linear regions, the expected curve length distortion, and neural tangent kernels.", "summary_of_the_review": "I think the paper contains original theoretical results which contribute to people's understanding of maxout networks. However, the theoretical and practical impact of this work may be limited, and the technical questions and proof techniques have appeared in previous work. Therefore, I tend to argue the paper is marginally below the bar of the NeurIPS venue.\n\n—————\n\nUpdate:\n\nAfter reading the authors’ rebuttal, I would like to keep my suggestion that the paper is marginally below NeurIPS’s standard, due to my personal judgment on its potential impact, and I am raising my confidence score from 2 to 4.\n\nThat said, I very much appreciate the authors’ discussion and clarification during the rebuttal phase.", "strength_and_weaknesses": "Strength:\n- Intuitively, maxout networks are more difficult to analyze. Therefore compared with the analysis of ReLU network, it is a step forward.\n- The paper is well-organized and the non-technical part is easy to follow.\n\nWeakness:\n- It seems that all the questions (Jacobian moments, proper initialization scale, number of linear regions, curve length distortion, NTK) and techniques to answer them have appeared in previous work. The originality of the contributions and the theoretical impact would both be limited.\n- Maxout networks (especially the fully-connected version) are not widely used in practice, to the best of my knowledge. Therefore the practical interest of this work would be limited.\n\nSome comments on the experiment section:\n- [1] reports a simple linear kernel achieved 1.2% test error on MNIST, which is better that all the fully-connected network performance reported in this paper.\n- As a rank-5 maxout network is 5 times as large as a ReLU network with the same architecture, I wonder if their results are directly comparable. Maybe a fair comparison should try to equal the number of parameters of different networks or else indicate their different sizes.\n\n[1] Understanding Deep Learning (Still) Requires Rethinking Generalization. https://dl.acm.org/doi/pdf/10.1145/3446776", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "15qZBas0Nl-", "reviewer": "Reviewer_Fz1h", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, authors analyze the gradients of a maxout network with respect to inputs and parameters. In consequence, the authors improve the training of deep maxout networks in the case of fully-connected and convolutional networks.", "summary_of_the_review": "Most of the results from the paper are interesting, but the article's presentation and construction of the article caused the paper to be very hard and difficult to read. The paper is better suited to a journal, where we do not have a limit of pages and can describe everything clearly. ", "strength_and_weaknesses": "1. The introduction paragraph, \"Maxout networks,\" is not clear.\n2. The introduction section is long and describes simple concepts, like Parameter initialization, and does not specify details of Maxout networks.\n3. The contributions of the paper are long and need to be more convincing. After reading the introduction, it is challenging to specify the paper's main result.\n4. Also, in section 2, the paragraph Architecture needs to be clarified. Some figures may help.\n5. Theoretical results from section 3 Results are interesting but nontrivial to follow. Authors try to show too many things in very short lines.\n6. The paper is longer than 9 pages.\n7. The experimental section confirms the results from the theoretical part. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject