Large-width functional asymptotics for deep Gaussian neural networks
Authors: Daniele Bracale
Stefano Favaro
Sandra Fortini
Stefano Peluchetti
Venue: ICLR 2021 Poster (2021)
Keywords: deep learning theory
infinitely wide neural network
Gaussian process
stochastic process

Abstract: In this paper, we consider fully connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions. Extending previous results (Matthews et al., 2018a;b;Yang, 2019)  we adopt a function-space perspective, i.e. we look at neural networks as infinite-dimensional random elements on the input space $\mathbb{R}^I$. Under suitable assumptions on the activation function we show that: i) a network defines a continuous Gaussian process on the input space $\mathbb{R}^I$; ii) a network with re-scaled weights converges weakly to a continuous Gaussian process in the large-width limit; iii) the limiting Gaussian process has almost surely locally $\gamma$-Hölder continuous paths, for $0 < \gamma <1$. Our results contribute to recent theoretical studies on the interplay between infinitely wide deep neural networks and Gaussian processes by establishing weak convergence in function-space with respect to a stronger metric.

Published as a conference paper at ICLR 2021
LARGE-WIDTH FUNCTIONAL ASYMPTOTICS FOR DEEP GAUSSIAN NEURAL NETWORKS
Daniele Bracale', Stefano Favaro Sannrra
Sandra Fortini', Stefano Peluchetti'
1 University of Torino, 2 Collegio Carlo Alberto, Bocconi University, 4 Cogent Labs
ABSTRACT
In this paper, we consider fully-connected feed-forward deep neural networks where weights and biases are independent and identically distributed according Gaussian distributions. Extending previous results (Matthews et al., 201 8a;;; Yang, 2019) we adopt a function-space perspective, i.e. we look at neural networks infiniteddimensionn random elements on the input space R Under suitable assumptions on the activation function we show that: i) a network defines a continuous stochastic process on the input space R network with re-scaled weights converges weakly to a continuous Gaussian Process in the large-width limit; iii the limiting Gaussian Process has almost surely locally HöHder continuous paths, for 0 < < 1. Our results contribute to recent theoretical studies on the interplay between infinitely-wer wide deep neural networks and Gaussian Processes by establishing weak convergence in function-space with respect to stronger metric.
INTRODUCTION
The interplay between infinitely-wide deep neural networks and classes of Gaussian Processes has its origins in the seminal work of Neal (1995), and it has been the subject of several theoretical studies. See, e.g., Der & Lee (2006). Lee et al. (2018), Matthews et al. (2018a:b), Yang (2019) and references therein. Let consider a fully-connected feed-forward neural network with re-scaled weights composed of T. 2 1 layers of widths 11,- ,,,, i.e.
(z)
Σ
w
(1) iif 2j + 6
i i= 1, 1,...,m.
1=1
(x)
Σ
(1-1) (x)) 1= 2.....,, L, i 1= 1= m w (f
where is non-linearity and E R' is real-valued input of dimension I N. Neal (1995) considered the case T. = 2, finite number l E N of fixed distinct inputs each (x))) RR, and weights w/) and biases ((4) independently and identically distributed (iid) as Gaussian distributions. Under appropriate assumptions on the activation Neal (1995) showed converges
with
that: i) for a fixed unit i, the k-dimensional random vector (2) (xx(1), (2. f! in distribution, the width nı goes to infinity, k-dimensional Gaussian random vector; ii) the large-width convergence holds jointly over finite collections of i's and the limiting k-dimensional Gaussian random vectors are independent across the index These results concerns neural networks with single hidden layer, but Neal (1995) also includes preliminary considerations infinitely-wide deep neural networks. More recent works, such as Lee al. (2018), established convergence results corresponding to Neal (1995) results i) and ii) for deep neural networks under the assumption that widths 7 n, go to infinity sequentially over network layers. Matthews et al. (2018az 8a;b) extended the work of Neal (1995); Lee al. (2018) by assuming that the width grows to infinity jointly over network layers, instead of sequentially, and by establishing joint convergence over all and countable distinct inputs. The joint growth over the layers is certainly more realistic than the sequential growth, since the infinite Gaussian limit is considered as an approximation of a very wide network. We operate in the same setting of Matthews al. (2018b), hence from here onward n 2 1 denotes the
(zx(k))

---

Published as a conference paper at ICLR 2021
common layer width, i.e. R1..... RR n. Finally, similar large- width limits have been established for a great variety of neural network architectures, see for instance Yang (2019).
The assumption of countable number of fixed distinct inputs is the common trait of the literature on large-width asymptotics for deep neural networks. Under this assumption, the large- width limit of network boils down to the study of the large-width asymptotic behavior of the k-dimensional
random vector (f!²) (z("))..... y! (z (8) over 2 1 for finite k. Such limiting fiite-dimensional distributions describe the large-width width distribution of neural network priori over any dataset, which is finite by definition. When the limiting distribution is Gaussian, as it often is, this immediately paves the way Bayesian inference for the limiting network. Such an approach is competitive with the more standard stochastic gradient descent training for the fully-connected architectures object of our study (Lee al., 2020). However, knowledge of the limiting finiteddimensional distributions is not enough to infer properties of the limiting neural network which are inherently uncountable such as the continuity of the limiting neural network, or the distribution of its maximum over a bounded interval. Results in this direction give more complete understanding of the assumptions being made a priori, and hence whether a given model is appropriate for a specific application. For instance, Van Der Vaart& & Van Zanten (2011) shows that for Gaussian Processes the function smoothness under the prior should match the smoothness of the target function for satisfactory inference performance.
In this paper we thus consider novel, and more natural, perspective to the study of large-width limits of deep neural networks. This is an anfinieediimensional perspective where, instead of fixing a countable number of distinet inputs, we look at / (Ir, n) as a stochastic process over the input space R. Under this perspective, establishing large-width limits requires considerable care and, in addition, requires to show the existence of both the stochastic process induced by the neural network and its large-width limit. We start by proving the existence of i) continuous stochastic process, indexed by the network width n, corresponding to the fully-connected feed-forward deep neural network; ii) a continuous Gaussian Process corresponding to the infinitely-wide limit of the deep neural network. Then, we prove that the stochastic process i) converges weakly, as the width n goes to infinity, to the Gaussian Process ii) jointly over all units i. As by-product of our results, we show that the limiting Gaussian Process has almost surely locally 7-Hölder continuous paths, for 0 < 1. To make the exposition self-contained we include alternative proof of the main result of Matthews et al. (2018a;b), i.e. the mmite-dimensiona. limit for full-connected neural networks. The major difference between our proof and that of Matthews al. (2018b) is due to the use of the characteristic function to establish convergence in distribution, instead of relying on a CLT (Blum et al., 1958) for exchangeable sequences.
The paper is structured as follows. In Section 2 we introduce the setting under which we operate, whereas in Section 3 we present a high-level overview of the approach taken establish our results. Section 4 contains the core arguments of the proof of our aarge-width functional limit for deep Gaussian neural networks, which are spelled out in detail in the supplementary material (SM). We conclude in Section 5.
2 SETTING
Let (92, P) be the probability space which all random elements of interest are defined. Further- more, let N(u, o²) denote a Gaussian distribution with mean E R and strictly positive variance o² R., and let N₂ (m, Σ) be k-dimensional Gaussian distribution with mean m E RR and covariance matrix Σ E Bk In particular, R is equipped with || l|R=, the euclidean norm induced by the inner product (.. )R. and Fx = x R is equipped with || lla= the norm induced by the distance d(a. b) = Eis1 1 E(la, by 122 for a, b E 8e (Theorem 3.38 of Aliprantis & Border (2006)), where {(t) = (1 t) for all real values t 2 0. Note that (R. ||]) and (R || || ) are Polish spaces, i.e. separable and complete metric spaces (Corollary 3.39 of Aliprantis & Border (2006)). We choose dis since it generates a topology that coincides with the product topology (line 5 of the proof of Theorem 3.36 of Aliprantis & Border (2006)). The space (S, will indicate a generic Polish space such as IR or R with the associated distance. We indicate with SR! the space of functions from IRd into S and C(RI; ; :S C ga the space of continuous functions from Rd into S.
2

---

Published as a conference paper at ICLR 2021
Let [ (1) be the random weights of the 1-th layer, and assume that they are iid as ((0 o.), i.e.
Pugy (t) = EE [t (9] = =00 the characteristic function of wig for 2 1.j = 1, ...... and 2 Let 6!"h be the random biases of the 1-th layer, and assume that they are iid as (0, of)) i..
P000 (t) Eleln0
(3)
is the characteristic function of b. for i 2 and 1 2 1. Weights w are independent of biases (1))
for any 2 1, = -,...,, It and / 2 1. Let Φ : IR + R denote continuous non-linearity. For the finite-dimensional limit we will assume the polynomial envelop condition
[ø(s) - a + b|s|""; TM
(4)
for any E IR and some real values b > and 2 1. For the functional limit we will use stronger assumption on o, assuming Φ to be Lipschitz on IR with Lipschitz constant L
Let Z be a stochastic process on Re,, i.e. for each x E TR. (xx) defined on (9, H. P) and it takes values in S. For any & E N and .1,-..,.4 T. E IR/, let P2 P........ = (P)((z)) A,,..., ,1r.)) E Ag).
with A....., A& E B(S). Then, the family of finit-dimenniional distributions of (xx) is defined as the family of distributions {P "i....... II...... 2x E R and E NJ. See, e.g.. Billingsley (1995). In Definition 1 and Definition 2 we look at the deep neural network (1) as a stochastic process on input space R that is a stochastic process whose dinieedimmnsional distributions are determined by a finite number & E N of fixed distinct inputs (zx(1), , (k)),, with each 2x(r) E R.. The existence of the stochastic processes of Definition I and Definition 2 will be thoroughly discussed in Section 3.
Definition 1. For any fixed 1 2 2 and 2 1, let (f(")(n))n (n))n21 be a sequence of stochastic processes on TR. That is, _!0) (n) IR! - RR with x + l (xx, n), is a stochastic process on .R. whose finite- dimensional distributions are the laws, for any k E N and x(1) ((k) E R², of the k-dimensional random vectors
1
(200, 1 f( = (z)),,)
n)...................
Σ j=1
w L +b501 (5)
(x n).... f( (2 n))²
1/ (X, n)) + b (6)
Vn
where x = =(.,(")...., (b)) E BRik is I x k imput matrix of l distinct imputs .r!)) E TR. 1 denotes
a vector of dimension x 1 of I's, xj denotes the j-th row of the input matrix and 6 X is the
element-wise application of & to the matrix X. Let p!) (X, n) = 1². ((0) n. = f(0) (x(r), n) denote
the r-th component of the 1 vector f(a) (X,n), being 1, a vector of dimension 1 with in the r-the entry and 0 elsewhere.
Remark: in contrast to (I), we have defined (5)-(6) over an infinite number of units i > over each layer I, but the dependency on each previous layer remains limited to the first n components. Definition 2. For any fixed 2 2, let (F()) (n))22 be a sequence of stochastic processes on RR., That is, (n)) R. + Rxx with x + F(() (x, n), is a stochastic process on Ik! whose fiite-dimensional distributions are the laws, for any k E N and (11) vectors
....,,(()) (16 E Ik!, of the k-dimensional random
F(2)
[1(0) (X). (1) ( 1"
F (X, n) = [1 (X, n), (2 (X, .....".. T
Remark: for k inputs, the vector F(() (X) is an oo x k array, and for single input (r),, F(() (zf(1) m) can be written as (x(()()))))) (w("), n), (0) (x(r),n)..... .' E x We define F(D (X, n) F()) (2((r), ,))
the r-th column of F() (X. When we write (F(L-1) (z. (x,,,)) n), F(l n)) (see (8)) we treat F(0) F"(2,, n) and ((O) (y. n) as elements in I2" and not in 10000 iee we consider only the first n compo- nents of (()) (x., n) and p()) (y..))
3

---

Published as a conference paper at ICLR 2021
3 PLAN SKETCH
We start by recalling the notion of convergence in law, also referred to convergence in distribution weak convergence, for a sequence of stochastic processes. See Billingsley (1995) for comprehensive account.
Definition (convergence in distribution). Suppose that and (f(n)) are random elements in topological space C. Then, (f(n))a21 is said to converge in distribution to J. if ([])) - [[]( as n + 50 for every bounded and continuous function h C -> R. In that case we write f(n) f
In this paper, we deal with continuous and real-valued stochastic processes. More precisely, we consider random elements defined on C(R:; with (S, d) Polish space. Our aim is to study in C(R!; S) the convergence distribution as the width n goes to infinity for:
i) the sequence (f. (n))m2 for a fixed 2 2 and i 2 1 with (S. = (R.. D. i.c. the neural network process for single unit;
ii) the sequence (F'*) (2))n21 for fixed 2 2 with (8,d) (TR || i.e. the neural network process for all units.
Since applying Definition 3 in a function space is not easy, we need, proved in SM the following proposition.
Proposition (convergence in distribution in C(R::S), (S, d) Polish). Suppose that f and (f(n))n21 are random elements in C(R;;S) with (S, d) Polish space. Then, f(n) 4 f if: f(n) f f and ii) the sequence (f(n))22 is uniformly tight.
We denoted with 4 the convergence in law of the dinite-dmmessonall distributions of sequence of stochastic processes. The notion of tightness formalizes the concept that the probability mass is not allowed "escape at infinity": single random element in a topological space C is said to be tight for each e > 0 there exists a compact T C C such that [ E C 7 < e. If metric space (C, p) is Polish any random element on the Borel o-algebra of C is tight. A sequence of random elements (f(n))m21 in a topological space C' is said to be uniformly tight' if for every >> there exists a compact T C' such that (n) E C T for all n.
According to Proposition 1, to achieve convergence in distribution in function spaces we need the following Steps A-D:
Step A) to establish the existence of the finite- mnite-dimensional weak-limit f on R', We will rely on Theorem 5.3 of Kallenberg (2002), known as Levy theorem.
Step B) to establish the existence of the stochastic processes f and (f(n)) as elements sa the space of function from IR! into S. We make use of Daniell-Kolmogeenn criterion (Kallenberg, 2002, Theorem 6.16): given a family of multivariate distributions (P2 probability measure on diim(IZ
dimenssonal (a( ...., (x))) (R) AEN} there exists stochastic process with (Pr} Pr(),ī C C
distributions {Pz} satisfies the projective property: P- x Ryzz)
(z()).... (k)) ) (R KEN. That is, it is required consistency with respect to the marginaliza- tion arbitrary components. In this step also suppose, for moment, that the stochastic processes (f(n))n21 and j belong to C(R:: :) and we establish the existence of such stochastic processes in C S) endowed with a o-algebra and a probability measure that will be defined.
Step C) to show that the stochastic processes (f (m))))) and belong to C(R':S) C SR', With regards to (f()))21 this is direct consequence of (5)-(6) and the continuity of ø. With regards to the limiting process f. with an additional Lipschitz assumption on é, we rely on the following Kolmogorov-Chentso criterion (Kallenberg, 2002, Theorem 3.23):
Kallenheer (2002) uses the same term "rightness" for both cases of a single random element and of sequences of random clements; we find that the introduction of "uniform tightness" brings more clarity.
4

---

Published as a conference paper at ICLR 2021
Proposition 2 (continuous version and local-Höllee Heca--Höldrrianity, (S, d) complete). Let be a process R' with values in a complete metric space (S, d), and assume that there exist a, b, H. >> such that,
[ld(f()), - H|| - yll y/1(+b)
2, E IR'
Then f has continuous version (i.e. f belongs to C(R!; 5)), and the latter is a.s. locally Hölder continuous with exponent for any (0, 1/o).
Step D) the uniform tightness of (f (f(n)) nn1 C(R!; We We on an extension of the Kolmogorov- Chentsov criterion (Kallenberg, 2002, Corollary 16.9), which is stated in the following proposition. Proposition 3 (uniform tightness in C(R:; :), (S, d) Polish). Suppose that (f(n))22 are random elements C(R:; S) with (S, d) Polish space. Assume that f(0gr, n)n21 (i.e. f(n) evaluated the origin) is uniformly tight in S and that there exist a, b, H.S >> 0 such that,
[E(([ (z, (y, m))") & H|I yl| (1+)) I,Y E E.EER, .EE uniformly in n. Then (J()))n21 is uniformly tight in C)R
4 LARGE-WIDTH FUNCTIONAL LIMITS
4.1 LIMIT ON C(R:; S), WITH (8, d) (R. 1), FOR A FIXED UNIT i AND LAYER
Lemma 1 (finitediimensional limit). I satisfies (4) then there exists stochastic process f! R/ + R such that (5( (n))n21 4 f!
0c.
Proof. Fix 1 2 2 and i 1. Fixed k inputs X [r((1) ,...,, ,(1]], we show that as +00
pl (X, n) 4 No (()),
where (() denotes the k x k covariance matrix, which can be computed through the recursion: E(1);; = of + o (z )R, E();, = of + o
where ((t-1)
No (0, Σ( 1)). By means of (2), (3), (5) and (6),
(X) - (o. E()),, E(1), = o (2(000) x /3)
1 (x)())) (0, n),, for 1 2 2,
2(4,) o + -
X n)), (0 F
We prove (7) using Levy's theorem. that is the point-wise convergence of the sequence of characteristic functions of (8). We defer to SM A for the complete proof.
Lemma proves Step A. This proof gives an alternative and self-contained proof of the main result of Matthews et al. (201 8b), under the more general assumption that the activation function 6 satisfies the polynomial envelop (4). Now we prove Step B, i.e. the existence of the stochastic processes f( (n) and f(") on the space R for each layer | 2 1, unit 2 1 and n E N. In SM E. we show that the fimite-dimensional distributions of f! (n) satisfies Danicll-Kolmopee (Kallenberg, 2002, Theorem 6.16), and hence the stochastic process f!²) (n) exists. In SM E.2 prove a similar result for the -imen-iammniional distributions of the limiting process f(() In SM E.3 we prove that, if these stochastic processes are continuous, they are naturally defined in C(RR; ([R;;R) RR) In order to prove the continuity, i.e. Step C note that f(²)(() = "/=11()) by construction, thus by induction on if f(( (n) continuous for each 1 and n, then (x, = E functions. For the limiting process 1!0) assume be Lipschitz with Lipschitz constant La- In particular we have the following:
criterion
(1) z + b(2) is continuous
(xx,n) (x,, + ((") is continuous being composition of continuous
Lemma 2 (continuity). If o is Lipschitz on R then (1), (2),.. are P-a.s. Lipschitz on RR, while the limiting process p!!) is P-a.s. continuous on IR! and locally 7 Hölder continuous for each 0 7 1.
5

---

Published as a conference paper at ICLR 2021
Proof. Here we present sketch of the proof, and we defer to SM B.1 and SM B.2 for the complete proof. For (1!) (n)) is trivial show that for each n
1 1 (z. o (v. m)l - H (9) (n)llz vllus, Z. E P 0.8.
(9)
where (n) denotes a suitable random variable, which is defined by the following recursion over /
(m) = E'= | (n) Le vn
Ez= | 1 2 (n)
(10)
To establish the continuity of the limiting process ,!") rely on Proposition 2. Take two inputs
I, E R., From (7) we get that [f (z), f. (y)] ~ (0, (() where
|a (a,, ViR!
(z. v)
Σ(1) = og
+ ow
Σ(1) = of
1
1
o
/ [d(u)) (
M())(()) (Ø(v)
q q dv),
1
1
where (1-1) (0, E(l 1)). Defining a = [1, -1]. from (7) we know that _!0 (v) (10
(x)
N(a 0, a ((())())).hh Thus
lf (v) f.0 (z) Iv a (O,))N((0
We proceed by induction over the layers. For = 1,
E [ir (v) (w) 11- Cola" E(1)a)
ColoEllvlle 20
x)21 -
o |
((Iv)) - 2(y, x)2 - || )
Co(())) - xxlla
where Co = (0, ") By hypothesis induction there exists constant (1 > 0 such that Iu 11 du) - H( 21.22- Then,
11 (v) (1)) (x) 20 (N (a )()aa)
(, 2d (o2 / [10(cw) -2(())(()(( + (1
du))
IN(0.1)) / /lu-wl v0-=v220 (1 (022))) [yy x1
where used [o(w))2 20(u)((o) (0()) -b(()) 2(v)) < Lzlu w12 and the Jensen inequality. Thus,
E (1 (0) < H Fly 1
(11)
where the constant H can be explicitly derived by solving the following system
( (
H 12( H(-1)
(12)
is easy to get HD) = CL)))) (1 1)8 Observe that H does not depend on í (this will
be helpful in establishing the uniformly tightness of (1(0) (n))m²1 and the continuity of F()). By Proposition 2, setting a 20, and 20 I (since B needs to be positive, it sufficient to
choose > I/2) we get that ((!)) has continuous version and the latter is P-a.s locally H-ööle
continuous for every < < I for each 1/2. Taking the limit as f - too we conclude the proof.

---

Published as a conference paper at ICLR 2021
Lemma 3 (uniform tightness). If d is Lipschitz on R then (Js (n))n21 is uniformly tight in RR).
Proof. We defer to SM B.3 for details. Fix i 2 1,12 1. We apply Proposition 3 to show the uniform tightness of the sequence (f1") (n)) m211 in C(RY; By Lemma 2 1:0h (1), l: (2),... are random elements in C(R'; :R). Since (R. 1. 1) is Polish, every probability measure is tight, then ff f(Orr, n) is tight in I for every n. Moreover, by Lemma 1 (: n))21 4 / (0x:), therefore by (Dudley, 2002, Theorem 11.5.3), [(0g, n)121 is uniformly tight R.
remains to show that there exist two values a > and 3 > 0, and a constant H > 0 such that
IB II, (y, - (z, (z, m] S H
x x1|g
1+8
EyERR,nEN
uniformly in n. Take two points z E R. From (8) we know that fL n)) 1 (O, n)) and p! (z, (M) N(0, (2 n)) with joint distribution (2 El, n)). where
(1) [())
o2(1,) )). n)), (z((,n)
21) (1)a
201 =
(1)
with,
(1)
3 + + |
(1(1) = dg + +
E(1),.g = of +
DRR,,
(l( = o +
Σ 6 f ((-1) (z, n).
oz(1,n) = o + - 00 f (1- (y,
n).. = of + + -= ~1, (1-1) (z. n)) (( ( (g.n))
Defining aT = [1, -1] we have that (v,,)) (1--1) - p (x.,)))([ ((-1)) is distributed as (aa a ((((,)))) where a (), (l,,,) -2E(1,n)xx + (l. Consider a -29 with 0 integer. Thus
1 (v. m) m (1 m)) (1-1)
20 ~ || EII n).N(O 1))20 ~ (a T E(I n)a) (N 1) 1)(29.
As in previous theorem, for l = we get E [1120 (v, 1f (cc m|)0
where
C. 1) Set H (o and by hypothesis induction suppose that for every j 2 1
E Eli5 "1 (g. - 13 f(1111 (2,n)) s H 1
By hypothesis is Lipschitz, then
E B(1,:' ((,) 520(x, m)
Co (o nz) 2) +02
2 Co
3 o / (1-11) (y, n) - 6o 11 30
((- 1) (v, - 81t (x,
Co
(o )
p (1-1)(y,m) - - (cmm)))
Co
n
S Co
13)
((-11)( (y.n) - -,,101) -10-1)
20
n
7

---

Published as a conference paper at ICLR 2021
Using the induction hypothesis
T [ (y.n) (c,
= [,, (9)
(v.n)
1, (or, 12) |
C (o2
|13 (1 (v. m) 15 (x, 12)
S 22) H H x
We can get the constant H() by solving the same system as (12), obtaining 11 C)(o2) (L3) which does not depend on n. By Proposition 3 setting a = 20 and / = 29
since B must be a positive constant, sufficient to take 0 > / 1/2 and this concludes the proof. 0
Note that Lemma 3 provides the last Step D that allows prove the desired result which is explained in the theorem that follows:
Theorem (functional limit). If ó is Lipschitz on R then f. (n) 4 f: R(
Proof. We apply Proposition to (n))m21 By Lemma we have that f. (f. (1)) belong 1(R); From Lemma I have the convergence of the ditee-dimensional distributions of (J!") (n))m21, and form Lemma we have the uniform tightness of (J (1))m21
4.2 LIMIT ON C(R!;S), WITH (8, d) (R || [|R=), FOR A FIXED LAYER
As in the previous section we prove Steps A-D for the sequence (n))21. Remark that each stochastic process (((), F(() FI), (2).... defines on C(L1; joint measure whose i-th marginal is the measure induced respectively by _(" (5("" (n), 1 (n),... (see SM E.1 -SM E.4).
Let (()) 1=1 p!)) where X denotes the product measure.
Lemma 4 dimine-dimeesiionll limit), If satisfies (4) then (m)) fg p() + co.
Proof. The proof follows by Lemma and Cramér-Wold theorem for fintte-dimensional projection of (n)) it is sufficient to establish the large asymptotic of linear combinations of the f. n)'s for E L C N. In particular, we show that for any choice of inputs elements X, as n - 400
p()) (X n)
(0, E()),
(13)
where E() is defined in (7). The proof is reported in SM C.
Lemma 5 (continuity), I is Lipschitz on R then F(), (F(4(2)))2 belong to C(B (R)). More precisely (()(1)), (1), (1) (2).... are IPaass. Lipschitz on RR, while the limiting process F(l) is PPa.ss. continuous on R and locally Hööler continuous for each O < <. 1.
Proof. It derives immediately from Lemma 2. We defer to SM D. and SM D.2 for details. The continuity of the sequence process immediately follows from the Lipschitzianity of each component in (9) while the continuity of the limiting process F(!) is proved by applying Proposition 2. Take two inputs x, y E ! and fix a Z even integer. Since E(t) < for all f 2 0, and by Jensen incquality
20
( (v))
(x)
(x) -1 (()
( (x) f!¹ (v))
Thus, by applying monotone convergence theorem to the positive increasing sequence g(N) (1) ((() (v)) (which allows exchange E and Exe get
8

---

Published as a conference paper at ICLR 2021
N
E (x).
SE
Σ 1 - (0))
lim
E
1
11)
(z) f: (w))
-000
2
1=1
1 (1))
Σ 2
(1) ||
yll = H 11 e11
where we used (11) and the fact that H does not depend on i (see (12)). Therefore, by Proposition 2. for each a > I, setting 3 = = (since 3 needs to be positive, it is sufficient to choose > 1) F
has a continuous version F (())()) which is F-a.s locally y-Hölder continuous for every 0 < y < 1 L
Letting a - oo we conclude.
Theorem 2 (functional limit). If & is Lipschitz on R then (F") 4 F as C(R(R>>)
on
Proof. This is Proposition applied to (F))
(n)) n21- From Lemma 4 and Lemma 5 it remains to
show the uniform tightness of the sequence (F()) (n) )221 in C(R'; 1.)) Let e > 0 and let (4:)22
be positive sequence such that Ei= e/2 We have established the uniform tightness of each component (Lemma 3). Therefore for each
N there exists compact K; C R)
such that P(J!" (n) E C(R);R) \ K,J < e, for each n E (such compact depends on e,). Set K x x K; which is compact by Tychonoff theorem. Note that this is a compact on the product space x - C(RR;;R) :R) with associated product topology, and this is also compact on C(R:; R
(see SM E.4). Then p p(p"(n) (n) E C(R';
K = P (n) E C(RR; \ K}} S
Σ p p( (m) C(R \ K, < E e, < which concludes the proof.
5 DISCUSSION
We looked at deep Gaussian neural networks as stochastic processes, i.c. unfinite-dimensional random elements, on the input space R', and we showed that: i) a network defines stochastic process on the input space K'; ii) under suitable assumptions on the activation function, network with re-scaled weights converges weakly to Gaussian Process in the large-width limit. These results extend previous works (Neal, 1995; Der Lee, 2006; Lee et al., 2018; Matthews al., 201 8a;b; Yang, 2019) that investigate the limiting distribution of neural network over a countable number of distinct inputs. From the point of view of applications, the convergence in distribution is the starting point for the convergence of expectations. Let consider continuous function g C(R:: RP) + R. By the continuous function mapping theorem (Billingsley, 1999, Theorem 2.7),
we have g (FF (n)) 4 g(F()) as n + +o,, and under uniform integrability (Billingsley, 1999, Section 3), we have (Billingsley, 1999, Theorem 3.5) (lg((()) (n)) + Blg(F()) as n > +00. See also Dudley (2002) and references therein.
As by-product of our results we showed that, under a Lipschitz activation function, the limiting Gaussian Process has almost surely locally 7-öölder continuous paths, for 0 < < < 1. This raises the question on whether it is possible to strengthen our results to cover the case = 1, or even the case of local Lipschitzianity of the paths of the limiting process. In addition, if the activation function is differentiable, does this property transfer to the limiting process? We leave these questions to future research. Finally, while fully-connected deep neural networks represent an ideal starting point for theoretical analysis, modern neural network architectures are composed of a much richer class of layers which includes convolutional, residual, recurrent and attention components. The technical arguments followed in this paper are amenable to extensions more complex network architectures. Providing a mathematical formulation of network's architectures and convergence results in a way that it allows for extensions to arbitrary architectures, instead of providing ad-hoc proof for each specific case, is a fundamental research problem. Greg Yang's 's work on Tensor Programs (Yang, 2019) constitutes an important step in this direction.

---

Published as a conference paper at ICLR 2021
REFERENCES
Charalambos D. Aliprantis and Kim Border. Infinite Dimensional Analysis: A Hitchhiker's 's Guide. Springer-Verlage Berlin and Heidelberg GmbH & Company KG, 2006.
Patrick Billingsley. Probability and Measure. John Wiley & Sons, 3rd edition, 1995.
Patrick Billingsley. Convergence of Probability Measures. Wiley-Interscience. 2nd edition, 1999,
JR Blum, H Chernoff, M Rosenblatt, and Teicher. Central Limit Theorems for Interchangeable Processes. Canadian Journal of Mathematics, 10:222-229, 1958.
Ricky Der and Daniel D Lee. Beyond Gaussian Processes: On the Distributions of Infinite Networks. In Advances in Neural Information Processing Systems, pp. 275-282, 2006.
RM Dudley, Real Analysis and Probability. Cambridge University Press, 2002.
Olav Kallenberg, Foundations of Modern Probability. Springer Science & Business Media, 2nd edition, 2002.
Jachoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep Neural Networks Gaussian Processes. In International Conference on Learning Representations, 2018.
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein, Finite Versus Infinite Neural Networks: an Empirical Study. volume 33, 2020.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian Process Behaviour Wide Deep Neural Networks. In International Conference on Learning Representations, 2018a.
Alexander G. de Matthews, Mark Rowland, Jiri Hron, Richard Turner, and Zoubin Ghahramani. Gaussian Process Behaviour in Wide Deep Neural Networks, 2018b.
Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
Aad Van Der Vaart and Harry Van Zanten. Information Rates of Nonparametric Gaussian Process Methods. Journal of Machine Learning Research, 12(6), 2011.
Greg Yang. Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. Advances in Neural Information Processing Systems, volume 32, 2019.
10


Reviews:
[{"review_id": "cyauPNHe3IA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: \nThe paper is based on recent research about the limit of neural networks to Gaussian processes. The main result is the proof of the limit from a function perspective: neural networks are considered as random processes with the input space being infinite-dimensional. \n\nIn the beginning, I didn’t see the motivation for doing another proof for the limit. However, in the discussion section, the authors pointed to the possible convergence of expectations where their approach can be used. Also I think the setting is different from the setting of the previous result: their NNs are of the width the same for each layer instead of strictly increasing (Matthews et al, 2018). I didn’t notice flaws in the proofs so if I'm not mistaken the authors provided proof for a bit more general statement.  But I'm still not sure about its significance. \n\nOverall, I find the paper well written and not difficult to follow. There are some inaccuracies which are easy to fix.\n\n**Comments**: \n- “Neal (1995) showed that for each $l \\ge 2$ and a fixed unit $i$, the $k$-dimensional random vector $(f^{(l)}_i(x^{1}), \\dots, f^{(l)}_i(x^{k}))$ converges in distribution, as the widths $n_1, \\dots , n_L$ go to infinity sequentially over network layers.“\nI think that Neal showed that under certain conditions random neural networks with *one* hidden layer converge to a Gaussian process. This statement was proved by Lee et al. (2018). \n\n- “We operate in the same setting, hence from here onward n 1 denotes the common layer width, i.e. $n_1, \\dots, n_L = n$.”\nActually, it is not the same setting as Matthews et al. (2018). Their theorem states the existence of a *strictly increasing* width function such that the NN distribution convergence to a GP. \n\n**Minor**: \n- Infinitely wide -> infinitely-wide\n- full/fully connected -> fully-connected \n- Finite dimensional - finite-dimensional \n- Network’s layers -> network layers\n- “Of a neural networks”\n- “Such an approach is competitive with the more standard stochastic gradient descent training for the fully- connected architectures object of our study”  \n- “Shows that the for Gaussian Processes…”\n- “We start by proving the existence of: i)” I think the colon is unnecessary \n- “the limiting Gaussian process have”\n- Section 2, “k-dimensional Gaussian distribution with mean \\mu…” \\mu should be bold because before (\\mu was 1-dimensional). Moreover, I think it would be better to use m instead because \\mu is used as a measure notation later\n- “the characteristic function of b_i , for for”\n- Definition 1, eq. (6): first equality is between identical expressions; “a vector of dimension k \\times 1 of 1s” - should be of “1’s”; “x_j denote” -> denotes; IS the element-wise… \n- Definition 2: the expressional after “where” was not used in the equations before. I think there should be firstly everything after the word “vector” and then the rest.\n- “Since applying the Definition 3” - should be without “the”\n- “It remain to show”\n- “Finally, while fully connected deep neural networks represents” \n- References: the same authors have different names and “Gaussian” should be capital \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_KBAqS_SfE9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper nicely completes the picture of distributional results for infinite width limits of DNN. More specifically, the common framework in this literature proves limits for a finite or countable number of inputs (Neal 1995, Matthews et al 2018), while this paper addresses the question from a stochastic process viewpoint over the input space (in a sense, this corresponds to proving a large width result that holds jointly over an infinite number of inputs).\n\nThe paper is well-written and clear, despite its rather technical nature. I think it is original. The significance seems more debatable to me: actually, all the known results over a finite or countable number of inputs carry over to the infinite-dimensional joint number of inputs. This is somehow expected, and I'm unsure about how these new results are going to benefit the community. But to me, this slight downside does not necessarily detract from the relevance of the paper and overall aesthetics of the proofs.\n\nSpecific comments:\n- $I$ is never defined as the input space dimension\n- p2: \"shows that the for Gaussian Processes \"\n- there are several instances of the sentence \"proving the existence of: i) a continuous Gaussian process, indexed by the network's width n, corresponding to the fully connected feed-forward deep neural network\" (with variations, in Abstract, Introduction and Discussion). This is slightly ambiguous to me, when I read it, I understand that you prove that to any NN with fixed-width n, there corresponds a GP. However, I do not think this is what you prove (ie  a fixed-width NN is not a GP).\n- could you please discuss the d(a,b)_\\infty distance and associated norm in more detail? I do not know it, and it does not look like trivial that is satisfies the triangle inequality. This embeds the infinite-dimensional space R^\\infty with a norm. If all distances are equivalent in finite dim spaces, this is not the case here where R^\\infty is infinite dim. What sort of properties should a distance/norm possess so that your results still hold true?\n- the envelope condition (4) does not prevent the non-linearity to be identically constant (eg zero); just to be sure, are all of your results compatible with this possibility (I think they are thanks to the presence of the bias terms)? \n- unless there is a strong motivation for it, I would not include the proof of Proposition 1 in the main text, in order to focus more closely to ML related topics of higher interest to ICLR readership.\n- p5: \"for any for any\"\n- middle of p6: \\theta should be \\theta_n or vice-versa.\n- top of p7: should specify that \\Sigma(l) matrix corresponds to the two inputs (x,y) so that it is a 2x2 matrix. Of course, this can be understood with previous notations, but this is ambiguous as is.\n- p7, two line above eq (12): $L_\\phi$ should be $L_\\phi^2$, I think.\n- p7, just before Lemma 3: \"it is sufficient to choose $\\theta>I$; why not choosing $\\theta>I/2$, which seems ok as well?\n- the assumption that all widths n_1,...,n_L are equal is unrealistic and only made for technical reasons. Could you discuss how this could be relaxed?\n- the discussion section provides some insights on the implication of Lemma 5 and the \\gamma-Hölderianity of the limiting process GP paths, for 0 < \u0000\\gamma < 1. Could you discuss potential implications of this type of smoothness, and also what it would provide to obtain a stronger result covering the \\gamma=1 case?\n- the bibliography could be tidied (capitals to gaussian)", "extra_scores": {}, "extra_text": {}}, {"review_id": "HhMOiNpgNvE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper makes a deep analysis on the large-width functional asymptotics of the fully connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions.  Actually, it proves that this kind of network tends to be a continuous Gaussian process as the width of the layer increases to infinity. Some more specific convergence properties are also obtained along this direction. It is clear that these results are theoretically significant. But I cannot find out how they can be applied in the field of learning theory and algorithms.  If a network can model a Gaussian process with a specific setting of the wights and biases, it is significant. However,  these weights and biases are assumed to be random as noises,  the network will not be learned with any task and cannot contribute to machine learning. Therefore, I cannot recommend this paper for the conference. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "9BXMFsswkU9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POST-REBUTTAL\n\nThank you for your answers and incorporated modifications! I think you've succeeded in addressing my major concerns, so I'm raising my score and recommending accept as promised.\n\n---\n\nThis paper is a theoretical investigation of the asymptotic behaviour of deep fully connected neural networks (FCNs) as the number of hidden units in each layer goes to infinity (simultaneously in all layers). Its goal is to extend the existing results of Neal (1995), Matthews et al. (2018), Lee & Bahri et al. (2018), and others (see Yang, 2019, for an overview), from countable to uncountable index sets (input space). This is typically done by proving convergence of finite marginals, and establishing relative compactness of the sequence of processes induced here by the finite networks. As the former was already done by Matthews et al., Yang, and others, the main contribution of this paper is the study of relative compactness. The authors approach this problem by recognising that with additional Lipschitz assumption on the nonlinearity, both the finite network induced processes and the asymptotic GP limit have a version with paths in a space of continuous mapping between two Polish spaces (at least in most of typical use cases). This is then helpful because the space of functions between a compact separable space and a Polish space is itself Polish (under the sup-norm topology), and the links between weakly convergent and uniformly tight sequences on Polish spaces.\n\nOverall, I think the topic of this paper is a good fit for ICLR (most of the new work in this direction was presented at ICLR in the last three years), and that the paper is trying to address an important open question. The reason I have assigned a relatively low score for now is because I’m unsure about the validity of some steps in the proofs, and because I think the structure of the paper could improve. However, I am hopeful that the authors may be able to satisfactorily address (at least) my “major comments” during the rebuttal period, in which case I would improve my score to 7 or 8.\n\n\nMAJOR COMMENTS\n\n* In the proof of Theorem 2, you construct K as the Cartesian product of compact sets for each of the infinitely many hidden unit marginals (nit: may want to mention this is compact by Tychnoff’s theorem). I am not sure whether I’m misunderstanding the notation or something else, but the penultimate line of the proof looks a bit suspect. IIUC, F^\\ell (n) will be in K^c if **at least one** (not necessarily all) of the countably many units f_i^\\ell (n) is outside of its K_i, so one should be taking union over the {f_i^\\ell(n) \\in K_i^c} events, not an intersection?! The rest of the proof doesn’t work when this replacement is made though. Could you please clarify? \n\n* It might be useful to add some discussion (e.g., into the appendix) of the involved sigma-algebras. In particular, you cite theorem 6.16 from Kallenberg (Kolmogorov extension theorem) which only guarantees existence of the given stochastic process on the product sigma-algebra. On the other hand, the theorems from chapter 16 of Kallenberg AFAICT apply to stochastic processes on the Borel sigma-algebra generated on C(R^I; S) by the sup-norm; I think that if you replaced R^I by a separable compact set, these two sigma-algebras will coincide, but I’m not sure this is true with R^I itself?! Isn’t the Borel sigma-algebra generated by the sup norm strictly larger with the whole R^I? If so, could you please clarify why this is not an issue? \n\n* On page 4, you say C(R^I; S) is a subspace of C(K; S) where K is a (compact) subset of R^I, which you need to invoke proposition 16.4 of Kallenberg. I suppose I could see how C(R^I; S) is isomorphic to a subspace of C(K; S), but that doesn’t seem to be what you mean here? If it is, could you please provide more details on how this would be sufficient for the conclusion of Proposition 1? If not, could you clarify please? \n\n* Since “convergence on uncountable index set = convergence on finite sets + relative compactness”, I am not sure if the decision to spend ~2 out of the 8 pages on rederiving a known result by Matthews et al. is the best use of space. It’s absolutely ok (even commendable) to desire to be self-contained, but the space in the main text may be better spent explaining the novel material on relative compactness/tightness, and the Holder continuity (the proofs & discussion of which are largely relegated to the appendix), and moving the alternative derivation of Matthews et al.’s results into the appendix instead. \n\n* Relatedly, in the introduction you say “we include a compact proof of the main result of Matthews et al.” implying their proof is significantly longer/more complicated. However, the two proofs share many steps, as the main difference is that the weak convergence of Cramer-Wald projections is proved via Levy’s continuity theorem in your case, and via the exchangeable CLT (Blum et al., 1958) by Matthews et al. Each of these two methods requires checking three conditions (your L1-L3; a-c in Lemma 10 in Matthews et al.) which itself involves many of the similar steps (uniform integrability via uniform control over moments, inductive propagation of weak convergence through layers, etc.), and without the additional commentary injected by Matthews et al., approximately the same amount of space. I would suggest replacing the word “compact” with “alternative” in the above statement. Furthermore, I think it would benefit the reader if you acknowledged both in the main tex  and in the appendix the parts of your proof based on/inspired by Matthews et al.: \n  - Exploiting exchangeability is one of the main ideas of Matthews et al., albeit they apply the exchangeable CLT whereas you use the Levy’s continuity theorem (which is nice!). Matthews et al. also propagate weak convergence and uniform integrability across the layers in the proof of their proposition 14 (cf. your Lemma 2 in the appendix). \n  - Your construction in Equations (5) and (6) is the ‘infinite width, finite fan-out, network’ construction from Matthews et al. \n  - The use of Cramer-Wold theorem is also in Matthews et al. (which you mention in the appendix, but not on p. 8). \n  - The polynomial envelope condition is a straightforward generalisation of Matthews et al. “linear envelope”, and the proof of Matthews would work with it as well (the only change would be in propagating uniform boundedness of higher than 8th moments in their lemma 20). Also, your proof of Lemma 1 in the appendix is quite similar to Matthews et al.’s lemma 20 (combined with lemma 19). As an aside, your Lipschitzness assumption implies |phi(x)| <= |phi(0)| + L |x|, i.e., your later developments confine the discussion to a subset of the “linear envelope” satisfying activations from Matthews et al. \n  - Relatedly, Novak & Xiao & Lee et al. (2018) and also Yang (2019; 2020) have already shown requiring polynomially bounded nonlinearity is sufficient which should be mentioned as well. \n\n* It has been a while since I read Radford Neal’s PhD thesis, but I cannot recall a proof of the \\ell > 2 “sequential limit” you ascribe to him on page 1 (in particular, I do not think he studied theoretically single-hidden layer FCNs with inputs drawn from a GP). AFAIK this work was done over several papers by Hazan & Jaakola, Lee & Bahri et al., and others. Could you please clarify? \n\n\nMINOR COMMENTS\n\n* I am wondering about the extension to NTK which can---at least in the limit---be viewed as a linear transformation of the prior distribution (if f is a sample from the NNGP prior, and one assumes MSE loss, a sample from NTK evaluated at x is equal in distribution to f(x) + Theta(x, X) (y - f(X)), where (X, y) is the training set, and Theta is the neural tangent kernel). Do you think your results translate also to gradient descent trained FCNs? Relatedly, there is a recent paper called “Exact posterior distributions of wide Bayesian neural networks” which you may want to cite; it shows that when the prior converges, posterior converges too. The proof is quite short and should apply to your setup (i.e., uncountable index set) without changes. \n\n* Am I right that the assumption of the same width of each hidden layer is not necessary? \n\n* I think the proof of L1 requires some discussion of uniform integrability. Overall, it might be best to have a separate lemma establishing uniform integrability of relevant random variables as currently the reader looks at L1-L3, and thinks that your proof is incorrect (convergence in probability generally doesn’t imply convergence of expectation without uniform integrability). Uniform integrability (mentioned only in passing within the proof of L1) is also utilised in SM A.1.4 which should be highlighted. \n\n* Can you comment on measurability of the event Y_s from Lemma 1? Are you working with the Borel sigma-algebra of the weak* topology, or is this meant as more of a shorthand?! \n\n* In the proof of L3 (p.14, appendix), is the supremum over n supposed to be there? \n\n* On p.4, step B), I think you only establish that both stochastic processes have versions in C(R^I; S) here, but rather only in step C). \n\n* On p.6, the first display after L3 (the 2-line expression for the characteristic function), it would help readability if the parenthesis around the expression raised to the nth power were braces (or at least brackets), and larger. \n\n* I am somewhat confused by the sentence “convergence in distribution is the starting point for the convergence of expectations” in your conclusion. I didn’t check your proofs carefully enough to be 100% sure, but AFAICT you can relatively easily combine the established uniform integrability and weak convergence to get convergence of (at least finite-marginal) mean and covariance (as you seem to be aware). Do you mean expectations of other non-bounded or non-continuous functions of the output? \n\n* In several places, including the abstract and the conclusion, you seem to say that each **finite** network defines a **Gaussian** process which converges in distribution to another Gaussian process as n->infinity. Did you mean to say that the networks define a **stochastic** instead of specifically a Gaussian process? (I don’t think that finite networks generally induce a Gaussian process, nor was this proved.) \n\n* In several places you claim that Matthews et al. and Neal only proved convergence on finite spaces, and only sometimes mention that the results were proved also for countable index sets. I think the reader would benefit if you always said countable, as not everyone knows that the extension to countable spaces is relatively straightforward consequence (in the same way as weak convergence on all or R^I is a relatively straightforward consequence of convergence on all compact subsets---proposition 16.6 in Kallenberg). \n\n* In the last sentence of your conclusion, you mention as a future direction development of a “mathematical formulation of network’s architectures and convergence results in a way that it allows for extensions to arbitrary architectures”. You are right that this framework is missing for the exchangeability based proof techniques, but the reader would benefit from contextualising this statement with respect to Greg Yang’s work on Tensor Programs. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article provides an analysis of feedforward neural network with iid Gaussian weights and biases in the infinite-width limit. The paper  complements earlier work on this topic by taking a function-space approach, considering neural networks as infinite-dimensional random elements on the input space. This is a well-written and rigorous theoretical paper. Although, as noted by a reviewer, there are no direct practical implications, the result is interesting in itself, highly relevant to the ICLR audience, and likely to lead to further exploration of the connections between Gaussian processes and neural networks. 

There were a few questions regarding the proofs that have been answered satisfactorily by the authors. 

I recommend acceptance. 
Decision: Accept (Poster)