Title: BAYESIAN DEEP CONVOLUTIONAL NETWORKS WITH MANY CHANNELS ARE GAUSSIAN PROCESSES

Abstract: There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical. As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit -a qualitative difference between the two regimes that is not present in the FCN case. We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

1. INTRODUCTION:

Neural networks (NNs) demonstrate remarkable performance (He et al., 2016; Oord et al., 2016; Silver et al., 2017; Vaswani et al., 2017) , but are still only poorly understood from a theoretical perspective (Goodfellow et al., 2015; Choromanska et al., 2015; Pascanu et al., 2014; Zhang et al., 2017) . NN performance is often motivated in terms of model architectures, initializations, and training procedures together specifying biases, constraints, or implicit priors over the class of functions learned by a network. This induced structure in learned functions is believed to be well matched to structure inherent in many practical machine learning tasks, and in many real-world datasets. For instance, properties of NNs which are believed to make them well suited to modeling the world include: hierarchy and compositionality (Lin et al., 2017; Poggio et al., 2017) , Markovian dynamics (Tiňo et al., 2004; 2007) , and equivariances in time and space for RNNs (Werbos, 1988) and CNNs (Fukushima & Miyake, 1982; Rumelhart et al., 1985) respectively.

The recent discovery of an equivalence between deep neural networks and GPs (Lee et al., 2018; Matthews et al., 2018b) allow us to express an analytic form for the prior over functions encoded by deep NN architectures and initializations. This transforms an implicit prior over functions into an explicit prior, which can be analytically interrogated and reasoned about. 1 Previous work studying these Neural Network-equivalent Gaussian Processes (NN-GPs) has established the correspondence only for fully connected networks (FCNs). Additionally, previous work has not used analysis of NN-GPs to gain specific insights into the equivalent NNs. SGD-trained finite model, allowing for a significant improvement. (d) Finally, invariance enabled by weight sharing and pooling allows for the best performance. Due to computational limitations, the performance of CNN-GPs with pooling -which also possess the property of invariance -remains an open question. Values are reported for 8-layer ReLU models. See §G.6 for experimental details, Figure 7 , and Table 1 for more model comparisons.

In the present work, we extend the equivalence between NNs and NN-GPs to deep Convolutional Neural Networks (CNNs), both with and without pooling. CNNs are a particularly interesting architecture for study, since they are frequently held forth as a success of motivating NN design based on invariances and equivariances of the physical world (Cohen & Welling, 2016 ) -specifically, designing a NN to respect translation equivariance (Fukushima & Miyake, 1982; Rumelhart et al., 1985) . As we will see in this work, absent pooling, this quality of equivariance has no impact in the Bayesian treatment of the infinite channel (number of convolutional filters) limit (Figure 1 ).

The specific novel contributions of the present work are:

1. We show analytically that CNNs with many channels, trained in a fully Bayesian fashion, correspond to an NN-GP ( §2, §3). We show this for CNNs both with and without pooling, with arbitrary convolutional striding, and with both same and valid padding. We prove convergence as the number of channels in hidden layers approach infinity simultaneously (i.e. min n 1 , . . . , n L → ∞, see §E.4 for details), extending the result of Matthews et al. (2018a) under mild conditions on the nonlinearity derivative. Our results also provide a rigorous proof of the assumption made in Xiao et al. (2018) that pre-activations in hidden layers are i.i.d. Gaussian.

2. We show that in the absence of pooling, the NN-GP for a CNN and a Locally Connected Network (LCN) are identical (Figure 1 , §5.1). An LCN has the same local connectivity pattern as a CNN, but without weight sharing or translation equivariance.

3. We experimentally compare trained CNNs and LCNs and find that under certain conditions both perform similarly to the respective NN-GP (Figure 6 , b, c). Moreover, both architectures tend to perform better with increased channel count, suggesting that similarly to FCNs (Neyshabur et al., 2015; Novak et al., 2018) CNNs benefit from overparameterization (Figure 6 , a, b), corroborating a similar trend observed in Canziani et al. (2016, Figure 2) . However, we also show that careful tuning of hyperparameters allows finite CNNs trained with SGD to outperform their corresponding NN-GPs by a significant margin. We ex-perimentally disentangle and quantify the contributions stemming from local connectivity, equivariance, and invariance in a convolutional model in one such setting (Figure 1 ).

4. We introduce a Monte Carlo method to compute NN-GP kernels for situations (such as CNNs with pooling) where evaluating the NN-GP is otherwise computationally infeasible ( §4).

We stress that we do not evaluate finite width Bayesian networks nor do we make any claims about their performance relative to the infinite width GP studied here or finite width SGD-trained networks. While this is an interesting subject to pursue (see Matthews et al. (2018a) ; Neal (1994)), it is outside of the scope of this paper.

1.1. RELATED WORK:

In early work on neural network priors, Neal (1994) demonstrated that, in a fully connected network with a single hidden layer, certain natural priors over network parameters give rise to a Gaussian process prior over functions when the number of hidden units is taken to be infinite. Follow-up work extended the conditions under which this correspondence applied (Williams, 1997; Le Roux & Bengio, 2007; Hazan & Jaakkola, 2015) . An exactly analogous correspondence for infinite width, finite depth deep fully connected networks was developed recently in Lee et al. (2018) ; Matthews et al. (2018b) , with Matthews et al. (2018a) extending the convergence guarantees from ReLU to any linearly bounded nonlinearities and monotonic width growth rates. In this work we further relax the conditions to absolutely continuous nonlinearities with exponentially bounded derivative and any width growth rates.

The line of work examining signal propagation in random deep networks (Poole et al., 2016; Schoenholz et al., 2017; Yang & Schoenholz, 2017; 2018; Hanin & Rolnick, 2018; Chen et al., 2018; Yang et al., 2018) is related to the construction of the GPs we consider. They apply a mean field approximation in which the pre-activation signal is replaced with a Gaussian, and the derivation of the covariance function with depth is the same as for the kernel function of a corresponding GP. Recently, Xiao et al. (2017b; 2018) extended this to convolutional architectures without pooling. Xiao et al. (2018) also analyzed properties of the convolutional kernel at large depths to construct a phase diagram which will be relevant to NN-GP performance, as discussed in §B.

Compositional kernels coming from wide convolutional and fully connected layers also appeared outside of the GP context. Cho & Saul (2009) derived closed-form compositional kernels for rectified polynomial activations (including ReLU). Daniely et al. (2016) proved approximation guarantees between a network and its corresponding kernel, and show that empirical kernels will converge as the number of channels increases.

There is a line of work considering stacking of GPs, such as deep GPs (Lawrence & Moore, 2007; Damianou & Lawrence, 2013) . These no longer correspond to GPs, though they can describe a rich class of probabilistic models beyond GPs. Alternatively, deep kernel learning (Wilson et al., 2016b; a; Bradshaw et al., 2017) utilizes GPs with base kernels which take in features produced by a deep neural network (often a CNN), and train the resulting model end-to-end. Finally, van der Wilk et al. (2017) incorporates convolutional structure into GP kernels, with follow-up work stacking multiple such GPs (Kumar et al., 2018; Blomqvist et al., 2018) to produce a deep convolutional GP (which is no longer a GP). Our work differs from all of these in that our GP corresponds exactly to a fully Bayesian CNN in the infinite channel limit, when all layers are taken to be of infinite size. We remark that while alternative models, such as deep GPs, do include infinite-sized layers in their construction, they do not treat all layers in this way -for instance, through insertion of bottleneck layers which are kept finite. While it remains to be seen exactly which limit is applicable for understanding realistic CNN architectures in practice, the limit we consider is natural for a large class of CNNs, namely those for which all layers sizes are large and rather comparable in size.

Deep GPs, on the other hand, correspond to a potentially richer class of models, but are difficult to analytically characterize and suffer from higher inference cost. Borovykh (2018) analyzes the convergence of CNN outputs at different spatial locations (or different timepoints for a temporal CNN) to a GP for a single input example. Thus, while they also consider a GP limit (and perform an experiment comparing posterior GP predictions to an SGD-trained CNN), they do not address the dependence of network outputs on multiple input examples, and thus their model is unable to generate predictions on a test set consisting of new input examples.

In concurrent work, Garriga-Alonso et al. ( 2018) derive an NN-GP kernel equivalent to one of the kernels considered in our work. In addition to explicitly specifying kernels corresponding to pooling and vectorizing, we also compare the NN-GP performance to finite width SGD-trained CNNs and analyze the differences between the two models.

2 MANY-CHANNEL BAYESIAN CNNS ARE GAUSSIAN PROCESSES 2.1 PRELIMINARIES General setup. For simplicity of presentation we consider 1D convolutional networks with circularly-padded activations (identically to Xiao et al. (2018) ). Unless specified otherwise, no pooling anywhere in the network is used. If a model (NN or GP) is mentioned explicitly as "with pooling", it always corresponds to a single global average pooling layer at the top. Our analysis is straightforward to extend to higher dimensions, using zero (same) or no (valid) padding, strided convolutions, and pooling in intermediary layers ( §C). We consider a series of L + 1 convolutional layers, l = 0, . . . , L.

Random weights and biases. The parameters of the network are the convolutional filters and biases, ω l ij,β and b l i , respectively, with outgoing (incoming) channel index i (j) and filter relative spatial location β ∈ [±k] ≡ {-k, . . . , 0, . . . , k}. 2 Assume a Gaussian prior on both the filter weights and biases,

EQUATION

The weight and bias variances are σ 2 ω , σ 2 b , respectively. n l is the number of channels (filters) in layer l, 2k + 1 is the filter size, and v β is the fraction of the receptive field variance at location β (with β v β = 1). In experiments we utilize uniform v β = 1/(2k + 1), but nonuniform v β = 1/(2k + 1) should enable kernel properties that are better suited for ultra-deep networks, as in Xiao et al. (2018) .

Inputs, pre-activations, and activations. Let X denote a set of input images (training set, validation set, or both). The network has activations y l (x) and pre-activations z l (x) for each input image

x ∈ X ⊂ R n 0 d , with input channel count n 0 ∈ N, number of pixels d ∈ N, where

y l i,α (x) ≡ x i,α l = 0 φ z l-1 i,α (x) l > 0 , z l i,α (x) ≡ n l j=1 k β=-k ω l ij,β y l j,α+β (x) + b l i . (2)

We emphasize the dependence of y l i,α (x) and z l i,α (x) on the input x. φ : R → R is a nonlinearity (with elementwise application to higher-dimensional inputs). Similarly to Xiao et al. (2018) , y l is assumed to be circularly-padded and the spatial size d hence remains constant throughout the network in the main text (a condition relaxed in §C and Remark E.3). See Figures 2 and 3 for a visual depiction of our notation.

Activation covariance. A recurring quantity in this work will be the empirical uncentered covariance matrix K l of the activations y l , defined as

EQUATION

)

K l is a random variable indexed by two inputs x, x and two spatial locations α, α (the dependence on layer widths n 1 , . . . , n l , as well as weights and biases, is implied and by default not stated explicitly). K 0 , the empirical uncentered covariance of inputs, is deterministic.

Shapes and indexing. Whenever an index is omitted, the variable is assumed to contain all possible entries along the respective dimension. For example, y 0 is a vector of size |X | n 0 d, K l α,α is a matrix of shape |X | × |X |, and z l j is a vector of size |X | d.

EQUATION

)

d 1 = (6 ,6 ) d 2 = (4 ,4 ) ϕ Figure 2: A sample 2D CNN classifier annotated according to notation in §2.1, §3. The network transforms n 0 × d 0 = 3 × 8 × 8-dimensional inputs y 0 (x) = x ∈ X into n3 = 10-dimensional logits z2 (x).

Model has two convolutional layers with k = (3, 3)-shaped filters, nonlinearity φ, and a fully connected layer at the top (y 2 (x) → z2 (x), §3.1). Hidden (pre-)activations have n 1 = n 2 = 12 filters. As min n 1 , n 2 → ∞, the prior of this CNN will approach that of a GP indexed by inputs x and target class indices from 1 to n3 = 10. The covariance of such GP can be computed as

σ 2 ω 6×6 d 2 α=(1,1) (C • A) 2 K 0 α,α + σ 2 b ⊗ I n3

, where the sum is over the {1, . . . , 4} 

d 0 = d 1 = d 2 = d is assumed

in the text, yet our formalism easily extends to the model displayed ( §C). Further, while displayed (pre-)activations have 3D shapes, in the text we treat them as 1D vectors ( §2.1).

Our work concerns proving that the top-layer pre-activations z L converge in distribution to an |X | n L+1 d-variate normal random vector with a particular covariance matrix of shape

|X | n L+1 d × |X | n L+1 d as min n 1 , . . . , n L → ∞.

We emphasize that only the channels in hidden layers are taken to infinity, and n L+1 , the number of channels in the top-layer pre-activations z L , remains fixed. For convergence proofs, we always consider z l , y l , as well as any of their indexed subsets like z l j , y l i,α to be 1D vector random variables, while K l , as well as any of its indexed subsets (when applicable, e.g. K l α,α , K l (x, x )) to be 2D matrix random variables.

2.2. CORRESPONDENCE BETWEEN GAUSSIAN PROCESSES AND BAYESIAN DEEP CNNS WITH INFINITELY MANY CHANNELS:

We next consider the prior over outputs z L computed by a CNN in the limit of infinitely many channels in the hidden (excluding input and output) layers, min n 1 , . . . , n L → ∞, and derive its equivalence to a GP with a compositional kernel. This section outlines an argument showing that z L is normally distributed conditioned on previous layer activation covariance K L , which itself becomes deterministic in the infinite limit. This allows to conclude convergence in distribution of the outputs to a Gaussian with the respective deterministic covariance limit. This section omits many technical details elaborated in §E.4.

2.2.1. A SINGLE CONVOLUTIONAL LAYER IS A GP CONDITIONED ON THE UNCENTERED COVARIANCE MATRIX OF THE PREVIOUS LAYER'S ACTIVATIONS:

As can be seen in Equation 2, the pre-activations z l are a linear transformation of the multivariate Gaussian ω l , b l , specified by the previous layer's activations y l . A linear transformation of a multivariate Gaussian is itself a Gaussian with a covariance matrix that can be derived straightforwardly. Specifically,

EQUATION

where I n l+1 is an n l+1 × n l+1 identity matrix, and A K l is the covariance of the pre-activations z l i and is derived in Xiao et al. (2018) . Precisely, A : PSD |X |d → PSD |X |d is an affine transformation (a cross-correlation operator followed by a shifting operator) on the space of positive semi-definite |X | d × |X | d matrices defined as follows:

EQUATION

A preserves positive semi-definiteness due to Equation 4. Notice that the covariance matrix in Equation 4 is block diagonal due to the fact that separate channels z l i n l+1

i=1 are i.i.d. conditioned on y l , due to i.i.d. weights and biases ω l i , b l i n l+1

i=1 . We further remark that per Equation 4 the normal distribution of z l |y l only depends on K l , hence the random variable z l |K l has the same distribution by the law of total expectation:

z l |K l ∼ N 0, A K l ⊗ I n l+1 . (6) 2.2.2 ACTIVATION COVARIANCE MATRIX BECOMES DETERMINISTIC WITH INCREASING CHANNEL COUNT

It follows from Equation 6that the summands in Equation 3 are i.i.d. conditioned on fixed K l-1 . Subject to weak restrictions on the nonlinearity φ, we can apply the weak law of large numbers and conclude that the covariance matrix K l becomes deterministic in the infinite channel limit in layer l (note that pre-activations z l remain stochastic). Precisely,

EQUATION

)

where C is defined for any

EQUATION

The decoupling of the kernel "propagation" into C and A is highly convenient since A is a simple affine transformation of the kernel (see Equation 5), and C is a well-studied map in literature (see §G.4), and for nonlinearities such as ReLU (Nair & Hinton, 2010) and the error function (erf) C can be computed in closed form as derived in Cho & Saul (2009) and Williams (1997) respectively. We refer the reader to Xiao et al. (2018, Lemma A.1) for complete derivation of the limiting value in Equation 7.

A less obvious result is that, under slightly stronger assumptions on φ, the top-layer activation covariance K L becomes unconditionally (dependence on observed deterministic inputs y 0 is implied) deterministic as channels in all hidden layers grow to infinity simultaneously:

EQUATION

i.e. K L ∞ is (C • A) applied L times to K 0 , the deterministic input covariance. We prove this in §E.4 (Theorem E.5). See Figure 3 for a depiction of the correspondence between neural networks and their infinite width limit covariances K l ∞ .

2.2.3. A CONDITIONALLY NORMAL RANDOM VARIABLE BECOMES NORMAL IF ITS:

COVARIANCE BECOMES DETERMINISTIC §2.2.1 established that z L |K L is Gaussian, and §2.2.2 established that its covariance matrix

A K L ⊗ I n L+1 converges in probability to a deterministic A K L ∞ ⊗ I n l+1 in the infinite channel limit (since K L P → K L ∞ , and A (•) ⊗ I n L+1 : R |X |d×|X |d → R n L+1 |X |d×n L+1

|X |d is continuous). As we establish in §E.4 (Theorem E.6), this is sufficient to conclude with the following result.

Result. If φ : R → R is absolutely continuous and has an exponentially bounded derivative, i.e. ∃ a, b ∈ R : |φ (x)| ≤ a exp (bx) a.e. (almost everywhere), then the following convergence in distribution holds:

EQUATION

)

For more intuition behind Equation 10and an informal proof please consult §E.3.

Under review as a conference paper at ICLR 2019 Figure 1 : Different dimensionality collapsing strategies described in §3. Validation accuracy of an MC-CNN-GP with pooling ( §3.2.1) is consistently better than other models due to translation invariance of the kernel. CNN-GP with zero padding ( §3.1) outperforms an analogous CNN-GP without padding as depth increases. At depth 15 the spatial dimension of the output without padding is reduced to 1 ⇥ 1, making the CNN-GP without padding equivalent to the center pixel selection strategy ( §3.2.2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed (Xiao et al., 2018) . CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subsets of CIFAR10. See §A.7.3 for experimental details.

1.. Global average pooling: take:

EQUATION

This approach corresponds to applying global average pooling right after the last convolutional layer. 8 This approach takes all pixel-pixel covariances into consideration and makes the kernel translation invariant. However, it requires O |X | 2 d 2 memory to compute the sample-sample covariance of the GP (or O d 2 per covariance entry in an iterative or distributed setting). It is impractical to use this method to analytically evaluate the GP, and we propose to use a Monte Carlo approach (see §4).

2. Subsampling one particular pixel: take h = e ↵ ,

EQUATION

)

This approach makes use of only one pixel-pixel covariance, and requires the same amount • CNN-GP exhibits similar behavior. In from the phase boundary, the covari all different inputs becomes asympto constant and makes learning infeasib

3.. Different NN-GP architectures:

• Many different NN architectures converge to a GP.

• Some admit a tractable analytic form. E.g. the covariance of a CNN-GP with no pooling can be computed using O(#samples^2 x #pixels) resources.

• For other architectures we use a Monte Carlo approach.

Sampling finite random networks of a given architecture and empirically computing the output covariance allows to approximate the CNN-GP covariance. In our experiments this (biased) estimate converges to the true CNN-GP covariance in #(instantiated networks) and #channels, both in terms of covariance Frobenius distance and the GP accuracy.

(a) SGD-trained finite (b) Performance of SG CNN-GP with increasi (c) However, the best significant margin.

• Various CNN architectural decisions like pooling / vectorizing / subsampling, zero or no padding have a corresponding NN-GP.

• Pooling enforces translation-invariant predictions in both CNNs and CNN-GPs and allows for the best performance.

• Shallow CNN models can perform worse than fully-connected alternatives due to failing to capture non-linear interactions between distant pixels. NN 

7.. Disentangl:

n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ 4x4 covariance per class y 0 z 0 z 1 y 2 = ϕ ( z 1 ) z2 4 3 2 1 10 classes per input 4 inputs, 3 features 4 3 2 1 4 3 2 1 ⊗ I10 K 0 K 0 K 1 K 2 K 1 K 2 σ 2 ω K 0 + σ 2 b σ 2 ω K 1 + σ 2 b ( K 1 ) ( K 0 )

K 0 K 1 K 2 ( K 1 ) ( K 0 )

affine transform affine transform affine transform convolution convolution matrix-multiply nonlinearity nonlinearity 

y 2 = ϕ ( z 1 ) y 1 = ϕ ( z 0 ) n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ C (K) = E x⇠N (0,K) h (x) (x) T i < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X

EQUATION

Under review as a conference paper at ICLR 2019 MC-CNN-GP Figure 2 : Validation accuracy (left) of an MC-CNN-GP increases with n ⇥ M (i.e. channel count times number of samples) and approaches that of the exact CNN-GP (not shown), while the distance (right) to the exact kernel decreases. The dark band in the left plot corresponds to ill-conditioning of K L+1 n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 ⇥ 8. See Figure 7 for similar results with other architectures and §A.7.2 for experimental details.

random finite width networks and use the empirical uncentered covariances of activations to estimate the Monte Carlo-GP (MC-GP) kernel,

EQUATION

(#NN instantiations)

• Outputs of a CNN converge in distribution to a Gaussian process with a compositional kernel (under mild conditions on the weights and biases distribution and nonlinearity).

(#channels in hidden layers)

(convergence in distribution)

(an affine transformation) (inputs covariance)

(computes covariance of post-nonlinearity activations given normal pre-activations)

outputs inputs

D ! min{n 1 ,...,n L }!1 N ⇣ 0, A (C A) L K 0 ⌦ I n L+1 ⌘ < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0 I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A

N c e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d

+ / d 3 + w / e H h s d W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0

I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A

N c e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d

+ / d 3 + w / e H h s d W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0

I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A

N c e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d

+ / d 3 + w / e H h s d W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0

I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A

N c e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d

+ / d 3 + w / e H h s d W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > (#channels in the outputs)

(not always necessary to track the whole 4x4x10x10 covariance.)

(attaching a fully-connected layer only to the center pixel of the output)

n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ FCN activations FCN-GP covariance K 0 < l a t e x i t s h a _ b a s e = " d j K o C a j Z g k r w u K p B m p = " > A A A B + i c b V C S g N B F L b X F R S t B o N g F W a E Y M g g E c D k j X M T m a T I T O y y s E J Z g q + Q D q x R v B L H R w k F p p M L h n H s l x M k g m u D a e T W h c W l J r p r x u b W X t n Z q O U V Z l c Y i V o A a C Z x K q G G E a i W J E B o L V g / l K / M K V H N Z Q c J S b o R D z k l x k q / e X S j i E p A z R P v h x T P z Z P T h V t q F r Y n p q l k k a G C a N c G L j C j D q W B D t V q l h D a J W t D Q i k m k / m w R A d W a A w V n Y i g y b q u M S K H M r C b k p i e n v X G r e E M i Z a B O e + h m P k t S w i E T w Q g E N x E a j D F a N G D C w h V H H P K I o g g t i X t u L N d j B P a k c l i E b C x f A F T G E P u E Q P D i B M l x B B a p A o Q u P A T P z t A Z O S / O Q / z c M I f O G / f T W X v g = = < / l a t e x i t > C < l a t e x i t s h a 1 _ b a s e 6 4 = " B p 3 s 0 N X a z I 8 d Q f K 7 i 5 e d A + N 1 h F s = " > A A A C A 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E Y M p r G M Y B 6 Q L G F 2 M k m G z M w u M 7 N C W F L 6 C b Z a W N q J r b W f I N b + i L N J C k 0 8 c O F w z r 3 c w w l j z r T x v C 8 n t 7 S 8 s r q W X 3 c 3 N r e 2 d w q 7 e 3 U d J Y r Q G o l 4 p J o h 1 p Q z S W u G G U 6 b s a J Y h J w 2 w m E l 8 x t 3 V G k W y V s z i m k g c F + y H i P Y W K n V F t g M C O Z p Z d w p F L 2 S N w F a J P 6 M F C 8 / 3 I v 4 + d O t d g r f 7 W 5 E E k G l I R x r 3 f K 9 2 A Q p V o Y R T s d u O 9 E 0 x m S I + 7 R l q c S C 6 i C d R B 6 j I 6 t 0 U S 9 S d q R B E / X 3 R Y q F 1 i M R 2 s 0 s o p 7 3 M v F f L w z F 3 G v T O w 9 S J u P E U E m m n 3 s J R y Z C W S G o y x Q l h o 8 s w U Q x G x 6 R A V a Y G F u b a 1 v x 5 z t Y J P W T k n 9 a 8 m 6 8 Y v k K p s j D A R z C M f h w B m W 4 h i r U g E A E D / A I T 8 6 9 8 + K 8 O m / T 1 Z w z u 9 m H P 3 D e f w A 8 Y 5 u m < / l a t e x i t > K 1 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " X v v O k S j q l J 9 2 P o c D U d m z M o T Y 5 m 8 = " > A A A C B H i c b V C 7 S g N B F L 0 b X z G + o p Y 2 Q 0 R I F X a 1 i G X Q R r C J Y B 6 Q X c P s Z D Y Z M j u 7 z M w K y 5 L W T 7 C x 0 A + w E 1 t 7 P 0 G 0 8 0 e c P A p N P H D h c M 6 9 n M v x Y 8 6 U t u 0 P K 7 e 0 v L K 6 l l 8 v b G x u b e 8 U d / e a K k o k o Q 0 S 8 U i 2 f a w o Z 4 I 2 N N O c t m N J c e h z 2 v K H 5 2 O / d U u l Y p G 4 1 m l M v R D 3 B Q s Y w d p I 7 u W N 0 8 1 c J g K d j r r F Q 7 t i T 4 A W i T M j h 7 V S + e u z + v 5 Q 7 x a / 3 V 5 E k p A K T T h W q u P Y s f Y y L D U j n I 4 K b q J o j M k Q 9 2 n H U I F D q r x s 8 v M I H R m l h 4 J I m h E a T d T f F x k O l U p D 3 2 y G W A / U v D c W / / V 8 P 5 y L 1 s G p l z E R J 5 o K M k 0 O E o 5 0 h M a N o B 6 T l G i e G o K J Z O Z 5 R A Z Y Y q J N b w X T i j P f w S J p H l e c k 4 p 9 Z e o 5 g y n y c A A l K I M D V a j B B d S h A Q R i u I d H e L L u r G f r x X q d r u a s 2 c 0 + / I H 1 9 g O q A 5 y R < / l a t e x i t > K 2 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " O U e U K e + l I o b d s 6 D 6 x E X 6 i k d e j z A = " > A A A C B H i c b V C 7 S g N B F J 2 N r x h f U U u b I U F I F X Z j E c u g j W A T w T w g G 8 P s Z D Y Z M j u 7 z N w V l i W t n 2 B j o R 9 g J 7 b 2 f o J o 5 4 8 4 e R S a e O D C 4 Z x 7 O Z f j R Y J r s O 0 P K 7 O y u r a + k d 3 M b W 3 v 7 O 7 l 9 w + a O o w V Z Q 0 a i l C 1 P a K Z 4 J I 1 g I N g 7 U g x E n i C t b z R + c R v 3 T K l e S i v I Y l Y N y A D y X 1 O C R j J v b y p 9 F K X S x + S c S 9 f t M v 2 F H i Z O H N S r B V K X 5 / V 9 4 d 6 L / / t 9 k M a B 0 w C F U T r j m N H 0 E 2 J A k 4 F G + f c W L O I 0 B E Z s I 6 h k g R M d 9 P p z 2 N 8 b J Q + 9 k N l R g K e q r 8 v U h J o n Q S e 2 Q w I D P W i N x H / 9 T w v W I g G / 7 S b c h n F w C S d J f u x w B D i S S O 4 z x W j I B J D C F X c P I / p k C h C w f S W M 6 0 4 i x 0 s k 2 a l 7 J y U 7 S t T z x m a I Y u O U A G V k I O q q I Y u U B 0 1 E E U R

u k e P 6 M m 6 s 5 6 t F + t 1 t p q x 5 j e H 6 A + s t x + r o Z y S < / l a t e x i t > ) is consistently better than other models due to translation invariance of the kernel. CNN-GP with zero padding ( §3.1) outperforms an analogous CNN-GP without padding as depth increases. At depth 15 the spatial dimension of the output without padding is reduced to 1 ⇥ 1, making the CNN-GP without padding equivalent to the center pixel selection strategy ( §3.2.2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed (Xiao et al., 2018) . CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subsets of CIFAR10. See §A.7.3 for experimental details.

K vec 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 K 3 m C N D g h m e g 3 o p A x p s E W l s n F s k = " > A A A C H X i c b V C 7 S g N B F J 2 N r x h f U U u b R R G s w q 4 W s Q z a C D Y R T B T c G G Y n d 3 V w Z n a Z u R s M y 3 6 A 3 2 D h J 9 j q B 1 g I Y i u i n T / i J L H Q x A M D h 3 P O 5 d 4 5 Y S K 4 Q c 9 7 d w o T k 1 P T M 8 X Z 0 t z 8 w u J S e X m l a e J U M 2 i w W M T 6 N K Q G B F f Q Q I 4 C T h M N V I Y C T s K r / b 5 / 0 g V t e K y O s Z d A S 9 I L x S P O K F q p X d 4 I J M V L R k V 2 m L e z g K s I e / l 5 F i B c o 5 Z Z F 1 i e 2 5 R X 8 Q Z w x 4 n / Q z Z q 6 1 u f H 9 X n 2 3 q 7 / B V 0 Y p Z K U M g E N e b M 9 x J s Z V Q j Z w L y U p A a S C i 7 o h d w Z q m i E k w r G 3 w m d z e t 0 n G j W N u n 0 B 2 o v y c y K o 3 p y d A m + 6 e b U a 8 v / u u F o R x Z j d F u K + M q S R E U G 2 6 O U u F i 7 P a r c j t c A 0 P R s 4 Q y z e 3 x L r u k m j K 0 h Z Z s K / 5 o B + O k u V 3 x d y r e k a 1 n j w x R J G t k n W w R n 1 R J j R y Q O m k Q R m 7 I P X k g j 8 6 d 8 + S 8 O K / D a M H 5 m V k l f + C 8 f Q N y + K f T < / l a t e x i t > C < l a t e x i t s h a 1 _ b a s e 6 4 = " B p 3 s 0 N X a z I 8 d Q f K 7 i 5 e d A + N 1 h F s = " > A A A C A 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E Y M p r G M Y B 6 Q L G F 2 M k m G z M w u M 7 N C W F L 6 C b Z a W N q J r b W f I N b + i L N J C k 0 8 c O F w z r 3 c w w l j z r T x v C 8 n t 7 S 8 s r q W X 3 c 3 N r e 2 d w q 7 e 3 U d J Y r Q G o l 4 p J o h 1 p Q z S W u G G U 6 b s a J Y h J w 2 w m E l 8 x t 3 V G k W y V s z i m k g c F + y H i P Y W K n V F t g M C O Z p Z d w p F L 2 S N w F a J P 6 M F C 8 / 3 I v 4 + d O t d g r f 7 W 5 E E k G l I R x r 3 f K 9 2 A Q p V o Y R T s d u O 9 E 0 x m S I + 7 R l q c S C 6 i C d R B 6 j I 6 t 0 U S 9 S d q R B E / X 3 R Y q F 1 i M R 2 s 0 s o p 7 3 M v F f L w z F 3 G v T O w 9 S J u P E U E m m n 3 s J R y Z C W S G o y x Q l h o 8 s w U Q x G x 6 R A V a Y G F u b a 1 v x 5 z t Y J P W T k n 9 a 8 m 6 8 Y v k K p s j D A R z C M f h w B m W 4 h i r U g E A E D / A I T 8 6 9 8 + K 8 O m / T 1 Z w z u 9 m H P 3 D e f w A 8 Y 5 u m < / l a t e x i t > 2 ! K 0 + 2 b < l a t e x i t s h a 1 _ b a s e 6 4 = " G M B / 3 b m l A t k y M c o 7 q w g 6 6 s 6 G n K I = " > A A A C H H i c b V D L S s N A F J 3 U V 6 2 v q E s 3 Q 4 t Q E E p S F 3 V Z d C O 4 q W A f 0 K R l M p 2 m Q 2 e S M D M R S s j e f x D 8 B L f 6 A W 5 E 3 A q i O 3 / E 6 Q P U 1 g M X D u f c y 7 3 3 e B G j U l n W u 5 F Z W l 5 Z X c u u 5 z Y 2 t 7 Z 3 z N 2 9 h g x j g U k d h y w U L Q 9 J w m h A 6 o o q R l q R I I h 7 j D S 9 4 d n Y b 1 4 T I W k Y X K l R R F y O / I D 2 K U Z K S 1 0 z 7 0 j q c 9 Q p d x M n 5 M R H K b z o W E c / q p d 2 z Y J V s i a A i 8 S e k U I 1 X / z 8 q D z f 1 r r m l 9 M L c c x J o D B D U r Z t K 1 J u g o S i m J E 0 5 8 S S R A g P k U / a m g a I E + k m k 1 9 S e K i V H u y H Q l e g 4 E T 9 P Z E g L u W I e 7 q T I z W Q 8 9 5 Y / N f z P D 6 3 W v V P 3 I Q G U a x I g K e b + z G D K o T j p G C P C o I V G 2 m C s K D 6 e I g H S C C s d J 4 5 n Y o 9 n 8 E i a Z R L 9 n H J u t T x n I I p s u A A 5 E E R 2 K A C q u A c 1 E A d Y H A D 7 s E D e D T u j C f j x X i d t m a M 2 c w + + A P j 7 R u A R q V 6 < / l a t e x i t > 2 ! K 1 1 + 2 b < l a t e x i t s h a 1 _ b a s e 6 4 = " F Q 8 u o E k 4 e C S 4 T j i q X D n M F w j N C X g = " > A A A C J X i c b V D L S g M x F M 3 U V 6 2 v U Z d u Q o t Q E M p M X d R l 0 Y 3 g p o J 9 Q F 9 k 0 k w b m m S G J C O U Y T 7 C t R / g J 7 j V p Q t 3 I o i C C 3 / E 9 L G o r Q c u n J x z L z f 3 e C G j S j v O h 5 V a W V 1 b 3 0 h v Z r a 2 d 3 b 3 7 P 2 D m g o i i U k V B y y Q D Q 8 p w q g g V U 0 1 I 4 1 Q E s Q 9 R u r e 8 G L s 1 2 + J V D Q Q N 3 o U k j Z H f U F 9 i p E 2 U t c + a S n a 5 6 h T 7 M a t g J M + S u B V x z U P K n w 9 S u Z s L + n a O a f g T A C X i T s j u X I 2 / / V Z e r m v d O 2 f V i / A E S d C Y 4 a U a r p O q N s x k p p i R p J M K 1 I k R H i I + q R p q E C c q H Y 8 O S q B x 0 b p Q T + Q p o S G E 3 V + I k Z c q R H 3 T C d H e q A W v b H 4 r + d 5 f G G 1 9 s / a M R V h p I n A 0 8 1 + x K A O 4 D g y 2 K O S Y M 1 G h i A s q f k 8 x A M k E d Y m 2 I x J x V 3 M Y J n U i g X 3 t O B c m 3 j O w R R p c A S y I A 9 c U A J l c A k q o A o w u A O P 4 A k 8 W w / W q / V m v U 9 b U 9 Z s 5 h D 8 g f X 9 C 3 W o q b I = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 9 7 h A c 1 X q 6 o + h n y b y N 3 E F c 8 G O U A = " > A A A B / H i c b V C 7 S g N B F L 3 r M 8 Z X 1 F K R w S B Y h V 0 t t A z a W C Z g H p A s Y X Y y m w y Z m V 1 m Z o W w x M 7 W V j / A T m w l v 2 J t 6 U 8 4 m 6 T Q x A M X D u f c y 7 m c I O Z M G 9 f 9 d J a W V 1 b X 1 n M b + c 2 t 7 Z 3 d w t 5 + X U e J I r R G I h 6 p Z o A 1 5 U z S m m G G 0 2 a s K B Y B p 4 1 g c J P 5 j X u q N I v k n R n G 1 B e 4 J 1 n I C D a Z 1 I 7 7 r F M o u i V 3 A r R I v B k p l o / G 1 e / H 4 3 G l U / h q d y O S C C o N 4 V j r l u f G x k + x M o x w O s q 3 E 0 1 j T A a 4 R 1 u W S i y o 9 t P J r y N 0 a p U u C i N l R x o 0 U X 9 f p F h o P R S B 3 R T Y 9 P W 8 l 4 n / e k E g 5 q J N e O W n T M a J o Z J M k 8 O E I x O h r A n U Z Y o S w 4 e W Y K K Y f R 6 R P l a Y G N t X 3 r b i z X e w S O r n J e + i 5 F Z t P d c w R Q 4 O 4 Q T O w I N L K M M t V K A G B P r w B M / w 4 j w 4 r 8 6 b 8 z 5 d X X J m N w f w B 8 7 H D 0 B 7 m P Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 9 7 h A c 1 X q 6 o + h n y b y N 3 E F c 8 G O U A = " > A A A B / H i c b V C 7 S g N B F L 3 r M 8 Z X 1 F K R w S B Y h V 0 t t A z a W C Z g H p A s Y X Y y m w y Z m V 1 m Z o W w x M 7 W V j / A T m w l v 2 J t 6 U 8 4 m 6 T Q x A M X D u f c y 7 m c I O Z M G 9 f 9 d J a W V 1 b X 1 n M b + c 2 t 7 Z 3 d w t 5 + X U e J I r R G I h 6 p Z o A 1 5 U z S m m G G 0 2 a s K B Y B p 4 1 g c J P 5 j X u q N I v k n R n G 1 B e 4 J 1 n I C D a Z 1 I 7 7 r F M o u i V 3 A r R I v B k p l o / G 1 e / H 4 3 G l U / h q d y O S C C o N 4 V j r l u f G x k + x M o x w O s q 3 E 0 1 j T A a 4 R 1 u W S i y o 9 t P J r y N 0 a p U u C i N l R x o 0 U X 9 f p F h o P R S B 3 R T Y 9 P W 8 l 4 n / e k E g 5 q J N e O W n T M a J o Z J M k 8 O E I x O h r A n U Z Y o S w 4 e W Y K K Y f R 6 R P l a Y G N t X 3 r b i z X e w S O r n J e + i 5 F Z t P d c w R Q 4 O 4 Q T O w I N L K M M t V K A G B P r w B M / w 4 j w 4 r 8 6 b 8 z 5 d X X J m N w f w B 8 7 H D 0 B 7 m P Y = < / l a t e x i t > X = y 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " w 7 f O K h z B 6 3 i 5 9 c d F S 6 X E C o b + p D w = " > A A A C C X i c b V C 7 S g N B F L 3 r M 8 Z H V i 0 V G Q y C V d j V Q h s h a G O Z g H l A s o b Z y W w y Z G Z 3 m Z k V l i W l l Z 9 g q x 9 g F 2 z 9 C m t L f 8 L J o 9 D E A x c O 5 9 z L P R w / 5 k x p x / m 0 l p Z X V t f W c x v 5 z a 3 t n Y K 9 u 1 d X U S I J r Z G I R 7 L p Y 0 U 5 C 2 l N M 8 1 p M 5 Y U C 5 / T h j + 4 G f u N B y o V i 8 I 7 n c b U E 7 g X s o A R r I 3 U s Q t t g X W f Y J 4 1 h 1 f p v d O x i 0 7 J m Q A t E n d G i u X D U f X 7 8 W h U 6 d h f 7 W 5 E E k F D T T h W q u U 6 s f Y y L D U j n A 7 z 7 U T R G J M B 7 t G W o S E W V H n Z J P g Q n R i l i 4 J I m g k 1 m q i / L z I s l E q F b z b H M d W 8 N x b / 9 X x f z L 3 W w a W X s T B O N A 3 J 9 H O Q c K Q j N K 4 F d Z m k R P P U E E w k M + E R 6 W O J i T b l 5 U 0 r 7 n w H i 6 R + V n L P S 0 7 V 1 H M N U + T g A I 7 h F F y 4 g D L c Q g V q Q C C B Z 3 i B V + v J e r N G 1 v t 0 d c m a 3 e z D H 1 g f P x U H n c I = < / l a t e x i t > z 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " U 1 l 7 D t q 3 3 F g r R s 5 4 b T L L s j N b C y Q = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F W a 1 0 E Y M 2 l h G N A 9 I 1 j A 7 m U 2 G z O w u M 7 N C X P I J t v o B 6 c T W b / A T x N o f c Z J Y a O K B C 4 d z 7 u V c T p A I r g 3 G n 0 5 u Y X F p e S W / 6 q 6 t b 2 x u F b Z 3 a j p O F W V V G o t Y N Q K i m e A R q x p u B G s k i h E Z C F Y P + p d j v 3 7 P l O Z x d G s G C f M l 6 U Y 8 5 J Q Y K 9 0 8 3 O F 2 o Y h L e A I 0 T 7 w f U j x / d 8 + S 0 Y d b a R e + W p 2 Y p p J F h g q i d d P D i f E z o g y n g g 3 d V q p Z Q m i f d F n T 0 o h I p v 1 s 8 u o Q H V i l g 8 J Y 2 Y k M m q i / L z I i t R 7 I w G 5 K Y n p 6 1 h u L / 3 p B I G e i T X j q Z z x K U s M i O k 0 O U 4 F M j M Z F o A 5 X j B o x s I R Q x e 3 z i P a I I t T Y u l z b i j f b w T y p H Z W 8 4 x K + x s X y B U y R h z 3 Y h 0 P w 4 A T K c A U V q A K F L j z C E z w 7 Q 2 f k v D i v 0 9 W c 8 3 O z C 3 / g v H 0 D m i 6 X 7 Q = = < / l a t e x i t > z 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " V 7 A S t E 6 F L E t g S P N 8 7 a p V g z m L N D U = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E Y M 2 l h G N A 9 I 1 j A 7 m U 2 G z M w u M 7 N C X P I J t v o B 6 c T W b / A T x N o f c Z J Y a O K B C 4 d z 7 u V c T p h w p o 3 n f T q 5 h c W l 5 Z X 8 q r u 2 v r G 5 V d j e q e k 4 V Y R W S c x j 1 Q i x p p x J W j X M c N p I F M U i 5 L Q e 9 i / H f v 2 e K s 1 i e W s G C Q 0 E 7 k o W M Y K N l W 4 e 7 v x 2 o e i V v A n Q P P F / S P H 8 3 T 1 L R h 9 u p V 3 4 a n V i k g o q D e F Y 6 6 b v J S b I s D K M c D p 0 W 6 m m C S Z 9 3 K V N S y U W V A f Z 5 N U h O r B K B 0 W x s i M N m q i / L z I s t B 6 I 0 G 4 K b H p 6 1 h u L / 3 p h K G a i T X Q a Z E w m q a G S T J O j l C M T o 3 E R q M M U J Y Y P L M F E M f s 8 I j 2 s M D G 2 L t e 2 4 s 9 2 M E 9 q R y X / u O R d e 8 X y B U y R h z 3 Y h 0 P w 4 Q T K c A U V q A K B L j z C E z w 7 Q 2 f k v D i v 0 9 W c 8 3 O z C 3 / g v H 0 D m 8 O X 7 g = = < / l a t e x i t > y 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 Y 9 j t I A c X c 6 W D 6 d G 1 4 l q r S Z U M k Y = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X z G + o p a K D A b B K u x q o W X Q x j J B 8 4 B k D b O T S T J k Z n a Z m R W W J a W l r X 6 A n d g K + R V r S 3 / C y a P Q x A M X D u f c y 7 m c I O J M G 9 f 9 d D J L y y u r a 9 n 1 3 M b m 1 v Z O f n e v p s N Y E V o l I Q 9 V I 8 C a c i Z p 1 T D D a S N S F I u A 0 3 o w u B 7 7 9 Q e q N A v l n U k i 6 g v c k 6 z L C D Z W u k 3 u v X a + 4 B b d C d A i 8 W a k U D o c V b 4 f j 0 b l d v 6 r 1 Q l J L K g 0 h G O t m 5 4 b G T / F y j D C 6 T D X i j W N M B n g H m 1 a K r G g 2 k 8 n r w 7 R i V U 6 q B s q O 9 K g i f r 7 I s V C 6 0 Q E d l N g 0 9 f z 3 l j 8 1 w s C M R d t u p d + y m Q U G y r J N L k b c 2 R C N C 4 C d Z i i x P D E E k w U s 8 8 j 0 s c K E 2 P r y t l W v P k O F k n t r O i d F 9 2 K r e c K p s j C A R z D K X h w A S W 4 g T J U g U A P n u A Z X p y h 8 + q 8 O e / T 1 Y w z u 9 m H P 3 A + f g A u r p h X < / l a t e x i t > y 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 N z n f h X n M F o s L H 4 H X e 5 0 7 v F G X w = " > A A A B + 3 i c b V C 7 S g N B F L 3 r M 8 Z X 1 F K R w S B Y h d 1 Y a B m 0 s U z Q P C B Z w + x k k g y Z m V 1 m Z o V l S W l p q x 9 g J 7 Z C f s X a 0 p 9 w 8 i g 0 8 c C F w z n 3 c i 4 n i D j T x n U / n a X l l d W 1 9 c x G d n N r e 2 c 3 t 7 d f 0 2 G s C K 2 S k I e q E W B N O Z O 0 a p j h t B E p i k X A a T 0 Y X I / 9 + g N V m o X y z i Q R 9 Q X u S d Z l B B s r 3 S b 3 x X Y u 7 x b c C d A i 8 W Y k X z o a V b 4 f j 0 f l d u 6 r 1 Q l J L K g 0 h G O t m 5 4 b G T / F y j D C 6 T D b i j W N M B n g H m 1 a K r G g 2 k 8 n r w 7 R q V U 6 q B s q O 9 K g i f r 7 I s V C 6 0 Q E d l N g 0 9 f z 3 l j 8 1 w s C M R d t u p d + y m Q U G y r J N L k b c 2 R C N C 4 C d Z i i x P D E E k w U s 8 8 j 0 s c K E 2 P r y t p W v P k O F k m t W P D O C 2 7 F 1 n M F U 2 T g E E 7 g D D y 4 g B L c Q B m q Q K A H T / A M L 8 7 Q e X X e n P f p 6 p I z u z m A P 3 A + f g A w Q 5 h Y < / l a t e x i t > z2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 w e G o m N j p m J m W Z L 3 O 9 4 Y 1 O s Z s 9 o = " > A A A C A H i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X Z j o Y 0 Y t L G M Y B 6 Q r G F m M p s M m Z 1 d Z m a F G N L 4 C b b a 2 d i J r T / g J 4 i 1 P + L k U W j i g Q u H c + 7 l X A 5 J B N f G 8 7 6 c z M L i 0 v J K d t V d W 9 / Y 3 M p t 7 1 R 1 n C r K K j Q W s a o T r J n g k l U M N 4 L V E 8 V w R A S r k d 7 F y K / d M q V 5 L K 9 N P 2 F B h D u S h 5 x i Y 6 V 6 k 2 C F 7 m 6 K r V z e K 3 h j o H n i T 0 n + 7 M M 9 T Z 4 / 3 X I r 9 9 1 s x z S N m D R U Y K 0 b v p e Y Y I C V 4 V S w o d t M N U s w 7 e E O a 1 g q c c R 0 M B j / O 0 Q H V m m j M F Z 2 p E F j 9 f f F A E d a 9 y N i N y N s u n r W G 4 n / e o R E M 9 E m P A k G X C a p Y Z J O k s N U I B O j U R u o z R W j R v Q t w V R x + z y i X a w w N b Y z 1 7 b i z 3 Y w T 6 r F g n 9 U 8 K 6 8 f O k c J s j C H u z D I f h w D C W 4 h D J U g I K A B 3 i E J + f e e X F

EQUATION

)

This approach corresponds to applying global average pooling right after the last convolutional layer. 8 This approach takes all pixel-pixel covariances into consideration and makes the kernel translation invariant. However, it requires O |X | 2 d 2 memory to compute the sample-sample covariance of the GP (or O d 2 per covariance entry in an iterative or distributed setting). It is impractical to use this method to analytically evaluate the GP, and we propose to use a Monte Carlo approach (see §4).

2. Subsampling one particular pixel: take h = e ↵ ,

EQUATION

This approach makes use of only one pixel-pixel covariance, and requires the same amount of memory as vectorization ( §3.1) to compute.

We compare the performance of presented strategies in Figure 1 . Note that all described strategies admit stacking additional FC layers on top while retaining the GP equivalence, using a derivation analogous to §2 (Lee et al., 2018; Matthews et al., 2018b) .

4. MONTE CARLO EVALUATION OF INTRACTABLE GP KERNELS:

We introduce a Monte Carlo estimation method for NN-GP kernels which are computationally impractical to compute analytically, or for which we do not know the analytic form. Similar in spirit to traditional random feature methods (Rahimi & Recht, 2007) , the core idea is to instantiate many 8 Spatially local average pooling in intermediary layers can be constructed in a similar fashion ( §A.3). We • CNN-GP exhibits similar behavio from the phase boundary, the cov all different inputs becomes asym constant and makes learning infe

• Many different NN architectures converge to a GP.

• Some admit a tractable analytic form. E.g. the covariance of a CNN-GP with no pooling can be computed using O(#samples^2 x #pixels) resources.

• For other architectures we use a Monte Carlo approach.

Sampling finite random networks of a given architecture and empirically computing the output covariance allows to approximate the CNN-GP covariance. In our experiments this (biased) estimate converges to the true CNN-GP covariance in #(instantiated networks) and #channels, both in terms of covariance Frobenius distance and the GP accuracy.

(a) SGD-trained finit (b) Performance of S CNN-GP with increa (c) However, the bes significant margin.

• Various CNN architectural decisions like pooling / vectorizing / subsampling, zero or no padding have a corresponding NN-GP.

• Pooling enforces translation-invariant predictions in both CNNs and CNN-GPs and allows for the best performance.

• Shallow CNN models can perform worse than fully-connected alternatives due to failing to capture non-linear interactions between distant pixels. 

7.. Disentang:

EQUATION

⊗ I10 K 0 K 0 K 1 K 2 K 1 K 2 σ 2 ω K 0 + σ 2 b σ 2 ω K 1 + σ 2 b ( K 1 ) ( K 0 )

K 0 K 1 K 2 ( K 1 ) ( K 0 )

affine transform affine transform affine transform convolution convolution matrix-multiply nonlinearity nonlinearity 

y 2 = ϕ ( z 1 ) y 1 = ϕ ( z 0 ) n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ C (K) = E x⇠N (0,K) h (x) (x) T i < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X

EQUATION

where ✓ consists of M draws of the weights and biases from their prior distribution, ✓ m ⇠ p (✓), and n is the width or number of channels in hidden layers. The MC-GP kernel converges to the analytic kernel with increasing width, lim n!1 K l n,M = K l 1 in probability. For finite width networks, the uncertainty in K l n,M is Var

⇥ K l n,M ⇤ = Var ✓ ⇥ K l n (✓) ⇤ /M . From Daniely et al. (2016), we know that Var ✓ ⇥ K l n (✓) ⇤ / 1 n , which leads to Var ✓ [K l n,M ] / 1 Mn . For finite n, K l n,M is also a biased estimate of K l 1 ,

where the bias depends solely on network width. We do not currently have an analytic form for this bias, but we can see in Figures 2 and 7 that for the hyperparameters we probe it is small relative to the variance. In particular,

K l n,M (✓) K L 1 2 F

is nearly constant for constant Mn. We thus treat Mn as the effective sample size for the Monte Carlo kernel estimate. Increasing M and reducing n can reduce memory cost, though potentially at the expense of increased compute time and bias.

Under review as a conference paper at ICLR 2019 MC-CNN-GP Figure 2 : Validation accuracy (left) of an MC-CNN-GP increases with n ⇥ M (i.e. channel count times number of samples) and approaches that of the exact CNN-GP (not shown), while the distance (right) to the exact kernel decreases. The dark band in the left plot corresponds to ill-conditioning of K L+1 n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 ⇥ 8. See Figure 7 for similar results with other architectures and §A.7.2 for experimental details. random finite width networks and use the empirical uncentered covariances of activations to estimate the Monte Carlo-GP (MC-GP) kernel,

EQUATION

where ✓ consists of M draws of the weights and biases from their prior distribution, ✓ m ⇠ p (✓), and n is the width or number of channels in hidden layers. The MC-GP kernel converges to the analytic kernel with increasing width, lim n!1 K l n,M = K l 1 in probability. For finite width networks, the uncertainty in K l n,M is Var

⇥ K l n,M ⇤ = Var ✓ ⇥ K l n (✓) ⇤ /M . From Daniely et al. (2016), we know that Var ✓ ⇥ K l n (✓) ⇤ / 1 n , which leads to Var ✓ [K l n,M ] / 1 Mn . For finite n, K l n,M is also a biased estimate of K l 1 ,

where the bias depends solely on network width. We do not currently have an analytic form for this bias, but we can see in Figures 2 and 7 that for the hyperparameters we probe it is small relative to the variance. In particular,

K l n,M (✓) K L 1 2 F

is nearly constant for constant Mn. We thus treat Mn as the effective sample size for the Monte Carlo kernel estimate. Increasing M and reducing n can reduce memory cost, though potentially at the expense of increased compute time and bias.

(#NN instantiations)

(not always necessary to track the whole 4x4x10x10 covariance.)

(attaching a fully-connected layer only to the center pixel of the output)

n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞

CNN (1D) activations:



CNN-GP covariance:

Under review as a conference paper at ICLR 2019 2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed (Xiao et al., 2018) . CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subsets of CIFAR10. See §A.7.3 for experimental details.

EQUATION

This approach corresponds to applying global average pooling right after the last convolutional layer. 8 This approach takes all pixel-pixel covariances into consideration and makes the kernel translation invariant. • CNN-GP exhibits similar behavio from the phase boundary, the cov all different inputs becomes asym constant and makes learning infe

• Many different NN architectures converge to a GP.

• Some admit a tractable analytic form. E.g. the covariance of a CNN-GP with no pooling can be computed using O(#samples^2 x #pixels) resources.

• For other architectures we use a Monte Carlo approach.

Sampling finite random networks of a given architecture and empirically computing the output covariance allows to approximate the CNN-GP covariance. In our experiments this (biased) estimate converges to the true CNN-GP covariance in #(instantiated networks) and #channels, both in terms of covariance Frobenius distance and the GP accuracy.

(a) SGD-trained finit (b) Performance of S CNN-GP with increa (c) However, the bes significant margin.

• Various CNN architectural decisions like pooling / vectorizing / subsampling, zero or no padding have a corresponding NN-GP.

• Pooling enforces translation-invariant predictions in both CNNs and CNN-GPs and allows for the best performance.

• Shallow CNN models can perform worse than fully-connected alternatives due to failing to capture non-linear interactions between distant pixels. N 

n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ 4x4 covariance per class y 0 z 0 z 1 y 2 = ϕ ( z 1 ) z2 4 3 2 1 10 classes per input 4 inputs, 3 features 4 3 2 1 4 3 2 1 ⊗ I10 K 0 K 0 K 1 K 2 K 1 K 2 σ 2 ω K 0 + σ 2 b σ 2 ω K 1 + σ 2 b ( K 1 ) ( K 0

K 0 K 1 K 2 ( K 1 ) ( K 0 )

affine transform affine transform affine transform convolution convolution matrix-multiply nonlinearity nonlinearity

y 2 = ϕ ( z 1 ) y 1 = ϕ ( z 0 ) n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ C (K) = E x⇠N (0,K) h (x) (x) T i < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c

+ X e z u u H u F 1 X X K n j X 0 p 9 z 6 F 6 y i w S Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L K S S 7 i N m 7 W J o + g 4 n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 ⇥ 8. See Figure 7 for similar results with other architectures and §A.7.2 for experimental details.

I + B j N B l P b a w o = " > A A A C e H i c b V F d S 8 M w F E 3 r 9 / y a H 2 + + B K e o I N L u R V 8 E U Q R B E A W n w l p H k q V b M G l L c i u O M v + n z / o r f D J b q + j m h c D h n H O 5 N + f S V A o D n v f m u B O T U 9 M z s 3 O V + Y X F p e X q y u q d S T L N e I M l M t E P l B g u R c w b I E D y h 1 R z o q j k 9 / T p b K D f P 3 N t R B L f Q i / l o S K d W E S C E b B U q / q 6 F S g C X U Z k f t Y P J I 9 g 9 z L Q o t O F P X y M h x q l + X m / l b 8 E R q g f 8 1 V p 9 v a / 7 Q X R D N K u K K S X U h h n H m 8 L E G 6 1 q j X v w B s W H g d + C W q o r O t W 9 T 1 o J y x T P A Y m i T F N 3 0 s h z I k G w S T v V 4 L M 8 J S w J 9 L h T Q t j o r g J 8 2 F O f b x t m T a O E m 1 f D H j I / u 7 I i T K m p 6 h 1 D v 5 p R r U B + a 9 G q R o Z D d F R m I s 4 z Y D H r J g c Z R J D g g d X w G 2 h O Q P Z s 4 A w L e z y m H W J J g z s r S o 2 F X 8 0 g 3 F w V z / w L b 6 p 1 0 5 O y 3 x m 0 Q b a R L v I R 4 f o B F 2 g a 9 R A D H 0 4 F W f N W X c + X e z u u H u F 1 X X K n j X

(#NN instantiations)

• Outputs of a CNN converge in distribution to a Gaussian process with a compositional kernel (under mild conditions on the weights and biases distribution and nonlinearity). 

outputs inputs D ! min{n 1 ,...,n L }!1 N ⇣ 0, A (C A) L K 0 ⌦ I n L+1 ⌘ < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0 I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A N c

W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0 

I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A N c

W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0 

I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A N c

W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " r S e f Q 0 

I B g T G o B S F h X r b E 0 x A 5 J b k = " > A A A C 4 n i c b V J N b 9 Q w E H X C V 1 k + u s C R i 8 U W q Y h V l f Q C x 0 I 5 g K i q I r F t p X V 2 5 X i d X a u O H d k T 6 C q Y H 8 A N c

W k Y H z E t t T l N q e V S K D 4 C A Z K f F o b T P J X 8 J D 3 b r / M n n 7 i x Q q u P s C x 4 k t O 5 E p l g F D w 1 7 f / Z I p J n s E 2 A n 4 P J K 1 1 C U Y L F J B X z L 1 i o + s c R I + Y L e E b O m 5 M a o z + P S S 5 U U 0 o q N Y m H Z K b B D t X k o N U S R 0 A T o T J Y J h X J K S w Y l d U b 5 y 7 w o W u N o y G + 4 F 4 5 w o R h X U c r d r 9 j L 6 n a f r x Z o 3 w / i V Y d a h A 5 t / j d 1 D d V H T y P 3 U q 7 N e 0 P o p 2 o C X w V x B 0 Y o C 6 O p v 3 f / k 6 s z L k C J q m 1 4 z g q I K m o A c E k d z 1 S W l 5 Q d k b n f O y h o t 4 4 q Z q N O P z U M z O c a e M / B b h h L 1 d U N L d 2 m a d e W V / L r u d q 8 r + 5 N M 3 X r C F 7 m V T N o r h i r X N W S g w a 1 / v G M 2 E 4 A 7 n 0 g D I j f P O Y L a i h D P y r 6 P m p x O s z u A q O d 3 d i j z / s D v Z e d / P Z Q I / R E 7 S N Y v Q C 7 a G 3 6 A i N E A s O A w h c 8 D W c h d / C 7 + G P V h o G X

c 0 j 9 E + E P / 8 C 5 J r t n g = = < / l a t e x i t > (#channels in the outputs)

(not always necessary to track the whole 4x4x10x10 covariance.)

(attaching a fully-connected layer only to the center pixel of the output)

n 1 → ∞ n 1 → ∞ n 2 → ∞ n 2 → ∞ K 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " d j K o C 8 a 3 j 3 Z g k 4 r w u 2 K p B m 0 0 p 3 8 = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F W a 1 0 E Y M 2 g g 2 E c 0 D k j X M T m a T I T O 7 y 8 y s E J Z 8 g q 1 + Q D q x 9 R v 8 B L H 2 R 5 w k F p p 4 4 M L h n H s 5 l x M k g m u D 8 a e T W 1 h c W l 7 J r 7 p r 6 x u b W 4 X t n Z q O U 0 V Z l c Y i V o 2 A a C Z 4 x K q G G 8 E a i W J E B o L V g / 7 l 2 K 8 / M K V 5 H N 2 Z Q c J 8 S b o R D z k l x k q 3 1 / e 4 X S j i E p 4 A z R P v h x T P 3 9 2 z Z P T h V t q F r 1 Y n p q l k k a G C a N 3 0 c G L 8 j C j D q W B D t 5 V q l h D a J 1 3 W t D Q i k m k / m 7 w 6 R A d W 6 a A w V n Y i g y b q 7 4 u M S K 0 H M r C b k p i e n v X G 4 r 9 e E M i Z a B O e + h m P k t S w i E 6 T w 1 Q g E 6 N x E a j D F a N G D C w h V H H 7 P K I 9 o g g 1 t i 7 X t u L N d j B P a k c l 7 7 i E b 3 C x f A F T 5 G E P 9 u E Q P D i B M l x B B a p A o Q u P 8 A T P z t A Z O S / O 6 3 Q 1 5 / z c 7 M I f O G / f T 3 W X v g = = < / l a t e x i t > A < l a t e x i t s h a 1 _ b a s e 6 4 = " O 6 6 A T y 1 a w 1 t Y L Y U p v p J 8 F g N J C H 8 = " > A A A C A 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E a M 2 l h G M A 9 I l j A 7 m S R D Z m e W m V k h L C n 9 B F s t L O 3 E 1 t p P E G t / x N k k h S Y e u H A 4 5 1 7 u 4 Y Q x Z 9 p 4 3 p e T W 1 h c W l 7 J r 7 p r 6 x u b W 4 X t n Z q W i S K 0 S i S X q h F i T T k T t G q Y 4 b Q R K 4 q j k N N 6 O L j K / P o d V Z p J c W u G M Q 0 i 3 B O s y w g 2 V m q 2 I m z 6 B P P 0 Y t Q u F L 2 S N w a a J / 6 U F M 8 / 3 L P 4 + d O t t A v f r Y 4 k S U S F I R x r 3 f S 9 2 A Q p V o Y R T k d u K 9 E 0 x m S A e 7 R p q c A R 1 U E 6 j j x C B 1 b p o K 5 U d o R B Y / X 3 R Y o j r Y d R a D e z i H r W y 8 R / v T C M Z l 6 b 7 m m Q M h E n h g o y + d x N O D I S Z Y W g D l O U G D 6 0 B B P F b H h E + l h h Y m x t r m 3 F n + 1 g n t S O S v 5 x y b v x i u V L m C A P e 7 A P h + D D C Z T h G i p Q B Q I S H u A R n p x 7 5 8 V 5 d d 4 m q z l n e r M L f + C 8 / w A 5 N 5 u k < / l a t e x i t > C < l a t e x i t s h a 1 _ b a s e 6 4 = " B p 3 s 0 N X a z I 8 d Q f K 7 i 5 e d A + N 1 h F s = " > A A A C A 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E Y M p r G M Y B 6 Q L G F 2 M k m G z M w u M 7 N C W F L 6 C b Z a W N q J r b W f I N b + i L N J C k 0 8 c O F w z r 3 c w w l j z r T x v C 8 n t 7 S 8 s r q W X 3 c 3 N r e 2 d w q 7 e 3 U d J Y r Q G o l 4 p J o h 1 p Q z S W u G G U 6 b s a J Y h J w 2 w m E l 8 x t 3 V G k W y V s z i m k g c F + y H i P Y W K n V F t g M C O Z p Z d w p F L 2 S N w F a J P 6 M F C 8 / 3 I v 4 + d O t d g r f 7 W 5 E E k G l I R x r 3 f K 9 2 A Q p V o Y R T s d u O 9 E 0 x m S I + 7 R l q c S C 6 i C d R B 6 j I 6 t 0 U S 9 S d q R B E / X 3 R Y q F 1 i M R 2 s 0 s o p 7 3 M v F f L w z F 3 G v T O w 9 S J u P E U E m m n 3 s J R y Z C W S G o y x Q l h o 8 s w U Q x G x 6 R A V a Y G F u b a 1 v x 5 z t Y J P W T k n 9 a 8 m 6 8 Y v k K p s j D A R z C M f h w B m W 4 h i r U g E A E D / A I T 8 6 9 8 + K 8 O m / T 1 Z w z u 9 m H P 3 D e f w A 8 Y 5 u m < / l a t e x i t > K 1 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " X v v O k S j q l J 9 2 P o c D U d m z M o T Y 5 m 8 = " > A A A C B H i c b V C 7 S g N B F L 0 b X z G + o p Y 2 Q 0 R I F X a 1 i G X Q R r C J Y B 6 Q X c P s Z D Y Z M j u 7 z M w K y 5 L W T 7 C x 0 A + w E 1 t 7 P 0 G 0 8 0 e c P A p N P H D h c M 6 9 n M v x Y 8 6 U t u 0 P K 7 e 0 v L K 6 l l 8 v b G x u b e 8 U d / e a K k o k o Q 0 S 8 U i 2 f a w o Z 4 I 2 N N O c t m N J c e h z 2 v K H 5 2 O / d U u l Y p G 4 1 m l M v R D 3 B Q s Y w d p I 7 u W N 0 8 1 c J g K d j r r F Q 7 t i T 4 A W i T M j h 7 V S + e u z + v 5 Q 7 x a / 3 V 5 E k p A K T T h W q u P Y s f Y y L D U j n I 4 K b q J o j M k Q 9 2 n H U I F D q r x s 8 v M I H R m l h 4 J I m h E a T d T f F x k O l U p D 3 2 y G W A / U v D c W / / V 8 P 5 y L 1 s G p l z E R J 5 o K M k 0 O E o 5 0 h M a N o B 6 T l G i e G o K J Z O Z 5 R A Z Y Y q J N b w X T i j P f w S J p H l e c k 4 p 9 Z e o 5 g y n y c A A l K I M D V a j B B d S h A Q R i u I d H e L L u r G f r x X q d r u a s 2 c 0 + / I H 1 9 g O q A 5 y R < / l a t e x i t > K 2 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " O U e U K e + l I o b d s 6 D 6 x E X 6 i k d e j z A = " > A A A C B H i c b V C 7 S g N B F J 2 N r x h f U U u b I U F I F X Z j E c u g j W A T w T w g G 8 P s Z D Y Z M j u 7 z N w V l i W t n 2 B j o R 9 g J 7 b 2 f o J o 5 4 8 4 e R S a e O D C 4 Z x 7 O Z f j R Y J r s O 0 P K 7 O y u r a + k d 3 M b W 3 v 7 O 7 l 9 w + a O o w V Z Q 0 a i l C 1 P a K Z 4 J I 1 g I N g 7 U g x E n i C t b z R + c R v 3 T K l e S i v I Y l Y N y A D y X 1 O C R j J v b y p 9 F K X S x + S c S 9 f t M v 2 F H i Z O H N S r B V K X 5 / V 9 4 d 6 L / / t 9 k M a B 0 w C F U T r j m N H 0 E 2 J A k 4 F G + f c W L O I 0 B E Z s I 6 h k g R M d 9 P p z 2 N 8 b J Q + 9 k N l R g K e q r 8 v U h J o n Q S e 2 Q w I D P W i N x H / 9 T w v W I g G / 7 S b c h n F w C S d J f u x w B D i S S O 4 z x W j I B J D C F X c P I / p k C h C w f S W M 6 0 4 i x 0 s k 2 a l 7 J y U 7 S t T z x m a I Y u O U A G V k I O q q I Y u U B 0 1 E E U R

u k e P 6 M m 6 s 5 6 t F + t 1 t p q x 5 j e H 6 A + s t x + r o Z y S < / l a t e x i t > 

K vec 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 K 3 m C N D g h m e g 3 o p A x p s E W l s n F s k = " > A A A C H X i c b V C 7 S g N B F J 2 N r x h f U U u b R R G s w q 4 W s Q z a C D Y R T B T c G G Y n d 3 V w Z n a Z u R s M y 3 6 A 3 2 D h J 9 j q B 1 g I Y i u i n T / i J L H Q x A M D h 3 P O 5 d 4 5 Y S K 4 Q c 9 7 d w o T k 1 P T M 8 X Z 0 t z 8 w u J S e X m l a e J U M 2 i w W M T 6 N K Q G B F f Q Q I 4 C T h M N V I Y C T s K r / b 5 / 0 g V t e K y O s Z d A S 9 I L x S P O K F q p X d 4 I J M V L R k V 2 m L e z g K s I e / l 5 F i B c o 5 Z Z F 1 i e 2 5 R X 8 Q Z w x 4 n / Q z Z q 6 1 u f H 9 X n 2 3 q 7 / B V 0 Y p Z K U M g E N e b M 9 x J s Z V Q j Z w L y U p A a S C i 7 o h d w Z q m i E k w r G 3 w m d z e t 0 n G j W N u n 0 B 2 o v y c y K o 3 p y d A m + 6 e b U a 8 v / u u F o R x Z j d F u K + M q S R E U G 2 6 O U u F i 7 P a r c j t c A 0 P R s 4 Q y z e 3 x L r u k m j K 0 h Z Z s K / 5 o B + O k u V 3 x d y r e k a 1 n j w x R J G t k n W w R n 1 R J j R y Q O m k Q R m 7 I P X k g j 8 6 d 8 + S 8 O K / D a M H 5 m V k l f + C 8 f Q N y + K f T < / l a t e x i t > A K 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " h 4 C + k v b R l W T U q o d P H M 9 c 9 E a / p z A = " > A A A C F X i c b V C 7 T g J B F J 3 F F + J r 1 Z J m I j G B h u x q o S V q Y 2 K D i T w S F s n s M A s T Z h + Z u W u C G w q / w F K / Q F v 9 A D t j a 2 3 t j z g L F A q e 5 C Y n 5 9 y b e + 9 x I 8 E V W N a X k V l Y X F p e y a 7 m 1 t Y 3 N r f M 7 Z 2 6 C m N J W Y 2 G I p R N l y g m e M B q w E G w Z i Q Z 8 V 3 B G u 7 g L P U b N 0 w q H g Z X M I x Y 2 y e 9 g H u c E t B S x 8 w 7 P o E + J S I 5 G T m C e V C 8 u L Y c y X t 9 K H X M g l W 2 x s D z x J 6 S Q q V 0 f x t l n x 6 r H f P b 6 Y Y 0 9 l k A V B C l W r Y V Q T s h E j g V b J R z Y s U i Q g e k x 1 q a B s R n q p 2 M n x j h f a 1 0 s R d K X Q H g s f p 7 I i G + U k P f 1 Z 3 p y W r W S 8 V / P d f 1 Z 1 a D d 9 x O e B D F w A I 6 2 e z F A k O I 0 4 h w l 0 t G Q Q w 1 I V R y f T y m f S I J B R 1 k T q d i z 2 Y w T + o H Z f u w b F 3 q e E 7 R B F m U R 3 u o i G x 0 h C r o H F V R D V F 0 h 5 7 R C 3 o 1 H o w 3 4 9 3 4 m L R m j O n M L v o D 4 / M H S 8 C i B w = = < / l a t e x i t > A < l a t e x i t s h a 1 _ b a s e 6 4 = " O 6 6 A T y 1 a w 1 t Y L Y U p v p J 8 F g N J C H 8 = " > A A A C A 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E a M 2 l h G M A 9 I l j A 7 m S R D Z m e W m V k h L C n 9 B F s t L O 3 E 1 t p P E G t / x N k k h S Y e u H A 4 5 1 7 u 4 Y Q x Z 9 p 4 3 p e T W 1 h c W l 7 J r 7 p r 6 x u b W 4 X t n Z q W i S K 0 S i S X q h F i T T k T t G q Y 4 b Q R K 4 q j k N N 6 O L j K / P o d V Z p J c W u G M Q 0 i 3 B O s y w g 2 V m q 2 I m z 6 B P P 0 Y t Q u F L 2 S N w a a J / 6 U F M 8 / 3 L P 4 + d O t t A v f r Y 4 k S U S F I R x r 3 f S 9 2 A Q p V o Y R T k d u K 9 E 0 x m S A e 7 R p q c A R 1 U E 6 j j x C B 1 b p o K 5 U d o R B Y / X 3 R Y o j r Y d R a D e z i H r W y 8 R / v T C M Z l 6 b 7 m m Q M h E n h g o y + d x N O D I S Z Y W g D l O U G D 6 0 B B P F b H h E + l h h Y m x t r m 3 F n + 1 g n t S O S v 5 x y b v x i u V L m C A P e 7 A P h + D D C Z T h G i p Q B Q I S H u A R n p x 7 5 8 V 5 d d 4 m q z l n e r M L f + C 8 / w A 5 N 5 u k < / l a t e x i t > C < l a t e x i t s h a 1 _ b a s e 6 4 = " B p 3 s 0 N X a z I 8 d Q f K 7 i 5 e d A + N 1 h F s = " > A A A C A 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E Y M p r G M Y B 6 Q L G F 2 M k m G z M w u M 7 N C W F L 6 C b Z a W N q J r b W f I N b + i L N J C k 0 8 c O F w z r 3 c w w l j z r T x v C 8 n t 7 S 8 s r q W X 3 c 3 N r e 2 d w q 7 e 3 U d J Y r Q G o l 4 p J o h 1 p Q z S W u G G U 6 b s a J Y h J w 2 w m E l 8 x t 3 V G k W y V s z i m k g c F + y H i P Y W K n V F t g M C O Z p Z d w p F L 2 S N w F a J P 6 M F C 8 / 3 I v 4 + d O t d g r f 7 W 5 E E k G l I R x r 3 f K 9 2 A Q p V o Y R T s d u O 9 E 0 x m S I + 7 R l q c S C 6 i C d R B 6 j I 6 t 0 U S 9 S d q R B E / X 3 R Y q F 1 i M R 2 s 0 s o p 7 3 M v F f L w z F 3 G v T O w 9 S J u P E U E m m n 3 s J R y Z C W S G o y x Q l h o 8 s w U Q x G x 6 R A V a Y G F u b a 1 v x 5 z t Y J P W T k n 9 a 8 m 6 8 Y v k K p s j D A R z C M f h w B m W 4 h i r U g E A E D / A I T 8 6 9 8 + K 8 O m / T 1 Z w z u 9 m H P 3 D e f w A 8 Y 5 u m < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 9 7 h A c 1 X q 6 o + h n y b y N 3 E F c 8 G O U A = " > A A A B / H i c b V C 7 S g N B F L 3 r M 8 Z X 1 F K R w S B Y h V 0 t t A z a W C Z g H p A s Y X Y y m w y Z m V 1 m Z o W w x M 7 W V j / A T m w l v 2 J t 6 U 8 4 m 6 T Q x A M X D u f c y 7 m c I O Z M G 9 f 9 d J a W V 1 b X 1 n M b + c 2 t 7 Z 3 d w t 5 + X U e J I r R G I h 6 p Z o A 1 5 U z S m m G G 0 2 a s K B Y B p 4 1 g c J P 5 j X u q N I v k n R n G 1 B e 4 J 1 n I C D a Z 1 I 7 7 r F M o u i V 3 A r R I v B k p l o / G 1 e / H 4 3 G l U / h q d y O S C C o N 4 V j r l u f G x k + x M o x w O s q 3 E 0 1 j T A a 4 R 1 u W S i y o 9 t P J r y N 0 a p U u C i N l R x o 0 U X 9 f p F h o P R S B 3 R T Y 9 P W 8 l 4 n / e k E g 5 q J N e O W n T M a J o Z J M k 8 O E I x O h r A n U Z Y o S w 4 e W Y K K Y f R 6 R P l a Y G N t X 3 r b i z X e w S O r n J e + i 5 F Z t P d c w R Q 4 O 4 Q T O w I N L K M M t V K A G B P

o + h n y b y N 3 E F c 8 G O U A = " > A A A B / H i c b V C 7 S g N B F L 3 r M 8 Z X 1 F K R w S B Y h V 0 t t A z a W C Z g H p A s Y X Y y m w y Z m V 1 m Z o W w x M 7 W V j / A T m w l v 2 J t 6 U 8 4 m 6 T Q x A M X D u f c y 7 m c I O Z M G 9 f 9 d J a W V 1 b X 1 n M b + c 2 t 7 Z 3 d w t 5 + X U e J I r R G I h 6 p Z o A 1 5 U z S m m G G 0 2 a s K B Y B p 4 1 g c J P 5 j X u q N I v k n R n G 1 B e 4 J 1 n I C D a Z 1 I 7 7 r F M o u i V 3 A r R I v B k p l o / G 1 e / H 4 3 G l U / h q d y O S C C o N 4 V j r l u f G x k + x M o x w O s q 3 E 0 1 j T A a 4 R 1 u W S i y o 9 t P J r y N 0 a p U u C i N l R x o 0 U X 9 f p F h o P R S B 3 R T Y 9 P W 8 l 4 n / e k E g 5 q J N e O W n T M a J o Z J M k 8 O E I x O h r A n U Z Y o S w 4 e W Y K K Y f R 6 R P l a Y G N t X 3 r b i z X e w S O r n J e + i 5 F Z t P d c w R Q 4 O 4 Q T O w I N L K M M t V K A G B P r w B M / w 4 j w 4 r 8 6 b 8 z 5 d X X J m N w f w B 8 7 H D 0 B 7 m P Y = < / l a t e x i t > X = y 0 < l a t e x i t s h a _ b a s e = " w f O K h z B i c d F S X E C o b + p D w = " > A A A C C X i c b V C S g N B F L r M Z H V i V G Q y C V d j V Q h s h a G O Z g H l A s o b Z y W w y Z G Z m Z k V l i W l l Z g q x g F z C m t L f L J o D E A x c O z L P R w / k x p x / m l p Z X V t f W c x v z a t n Y K u d X U S I J r Z G I R L p Y U C l N M p M Y U C / T h j + G f u N B y o V i I n c b U E g X s o A R r I U s Q t t g X W f Y J h f p v d O x i J m Q A t E n d G i u X D U f X W h U d h f W E E k F D T T h W q u U s f Y y L D U j n A z U T R G J M B t G W o S E W V H n Z J P g Q n R i l i J I m g k m q i / L z I s l E q F b z b H M d W N x b / X x f z L W w a W X s T B O N A J H O Q c K Q j N K F d Z m k R P P U E E w k M + E R W O J i T b l U r n w H i R + V n L P S V H M N U + T g A I h F F y g D L c Q g V q Q C C B Z i B V + v J e r N G v t d c m a e z D H g f P x U H n c I = < / l a t e x i t > z 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " U 1 l 7 D t q 3 3 F g r R s 5 4 b T L L s j N b C y Q = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F W a 1 0 E Y M 2 l h G N A 9 I 1 j A 7 m U 2 G z O w u M 7 N C X P I J t v o B 6 c T W b / A T x N o f c Z J Y a O K B C 4 d z 7 u V c T p A I r g 3 G n 0 5 u Y X F p e S W / 6 q 6 t b 2 x u F b Z 3 a j p O F W V V G o t Y N Q K i m e A R q x p u B G s k i h E Z C F Y P + p d j v 3 7 P l O Z x d G s G C f M l 6 U Y 8 5 J Q Y K 9 0 8 3 O F 2 o Y h L e A I 0 T 7 w f U j x / d 8 + S 0 Y d b a R e + W p 2 Y p p J F h g q i d d P D i f E z o g y n g g 3 d V q p Z Q m i f d F n T 0 o h I p v 1 s 8 u o Q H V i l g 8 J Y 2 Y k M m q i / L z I i t R 7 I w G 5 K Y n p 6 1 h u L / 3 p B I G e i T X j q Z z x K U s M i O k 0 O U 4 F M j M Z F o A 5 X j B o x s I R Q x e 3 z i P a I I t T Y u l z b i j f b w T y p H Z W 8 4 x K + x s X y B U y R h z 3 Y h 0 P w 4 A T K c A U V q A K F L j z C E z w 7 Q 2 f k v D i v 0 9 W c 8 3 O z C 3 / g v H 0 D m i 6 X 7 Q = = < / l a t e x i t > z 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " V 7 A S t E 6 F L E t g S P N 8 7 a p V g z m L N D U = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X a 1 0 E Y M 2 l h G N A 9 I 1 j A 7 m U 2 G z M w u M 7 N C X P I J t v o B 6 c T W b / A T x N o f c Z J Y a O K B C 4 d z 7 u V c T p h w p o 3 n f T q 5 h c W l 5 Z X 8 q r u 2 v r G 5 V d j e q e k 4 V Y R W S c x j 1 Q i x p p x J W j X M c N p I F M U i 5 L Q e 9 i / H f v 2 e K s 1 i e W s G C Q 0 E 7 k o W M Y K N l W 4 e 7 v x 2 o e i V v A n Q P P F / S P H 8 3 T 1 L R h 9 u p V 3 4 a n V i k g o q D e F Y 6 6 b v J S b I s D K M c D p 0 W 6 m m C S Z 9 3 K V N S y U W V A f Z 5 N U h O r B K B 0 W x s i M N m q i / L z I s t B 6 I 0 G 4 K b H p 6 1 h u L / 3 p h K G a i T X Q a Z E w m q a G S T J O j l C M T o 3 E R q M M U J Y Y P L M F E M f s 8 I j 2 s M D G 2 L t e 2 4 s 9 2 M E 9 q R y X / u O R d e 8 X y B U y R h z 3 Y h 0 P w 4 Q T K c A U V q A K B L j z C E z w 7 Q 2 f k v D i v 0 9 W c 8 3 O z C 3 / g v H 0 D m 8 O X 7 g = = < / l a t e x i t > y 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 Y 9 j t I A c X c 6 W D 6 d G 1 4 l q r S Z U M k Y = " > A A A B + 3 i c b V C 7 S g N B F L 0 b X z G + o p a K D A b B K u x q o W X Q x j J B 8 4 B k D b O T S T J k Z n a Z m R W W J a W l r X 6 A n d g K + R V r S 3 / C y a P Q x A M X D u f c y 7 m c I O J M G 9 f 9 d D J L y y u r a 9 n 1 3 M b m 1 v Z O f n e v p s N Y E V o l I Q 9 V I 8 C a c i Z p 1 T D D a S N S F I u A 0 3 o w u B 7 7 9 Q e q N A v l n U k i 6 g v c k 6 z L C D Z W u k 3 u v X a + 4 B b d C d A i 8 W a k U D o c V b 4 f j 0 b l d v 6 r 1 Q l J L K g 0 h G O t m 5 4 b G T / F y j D C 6 T D X i j W N M B n g H m 1 a K r G g 2 k 8 n r w 7 R i V U 6 q B s q O 9 K g i f r 7 I s V C 6 0 Q E d l N g 0 9 f z 3 l j 8 1 w s C M R d t u p d + y m Q U G y r J N L k b c 2 R C N C 4 C d Z i i x P D E E k w U s 8 8 j 0 s c K E 2 P r y t l W v P k O F k n t r O i d F 9 2 K r e c K p s j C A R z D K X h w A S W 4 g T J U g U A P n u A Z X p y h 8 + q 8 O e / T 1 Y w z u 9 m H P 3 A + f g A u r p h X < / l a t e x i t > y 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " M 1 N z n f h X n M F o s L H 4 H X e 5 0 7 v F G X w = " > A A A B + 3 i c b V C 7 S g N B F L 3 r M 8 Z X 1 F K R w S B Y h d 1 Y a B m 0 s U z Q P C B Z w + x k k g y Z m V 1 m Z o V l S W l p q x 9 g J 7 Z C f s X a 0 p 9 w 8 i g 0 8 c C F w z n 3 c i 4 n i D j T x n U / n a X l l d W 1 9 c x G d n N r e 2 c 3 t 7 d f 0 2 G s C K 2 S k I e q E W B N O Z O 0 a p j h t B E p i k X A a T 0 Y X I / 9 + g N V m o X y z i Q R 9 Q X u S d Z l B B s r 3 S b 3 x X Y u 7 x b c C d A i 8 W Y k X z o a V b 4 f j 0 f l d u 6 r 1 Q l J L K g 0 h G O t m 5 4 b G T / F y j D C 6 T D b i j W N M B n g H m 1 a K r G g 2 k 8 n r w 7 R q V U 6 q B s q O 9 K g i f r 7 I s V C 6 0 Q E d l N g 0 9 f z 3 l j 8 1 w s C M R d t u p d + y m Q U G y r J N L k b c 2 R C N C 4 C d Z i i x P D E E k w U s 8 8 j 0 s c K E 2 P r y t p W v P k O F k m t W P D O C 2 7 F 1 n M F U 2 T g E E 7 g D D y 4 g B L c Q B m q Q K A H T / A M L 8 7 Q e X X e n P f p 6 p I z u z m A P 3 A + f g A w Q 5 h Y < / l a t e x i t > z2 < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 w e G o m N j p m J m W Z L 3 O 9 4 Y 1 O s Z s 9 o = " > A A A C A H i c b V C 7 S g N B F L 0 b X 3 F 9 R S 1 t B o N g F X Z j o Y 0 Y t L G M Y B 6 Q r G F m M p s M m Z 1 d Z m a F G N L 4 C b b a 2 d i J r T / g J 4 i 1 P + L k U W j i g Q u H c + 7 l X A 5 J B N f G 8 7 6 c z M L i 0 v J K d t V d W 9 / Y 3 M p t 7 1 R 1 n C r K K j Q W s a o T r J n g k l U M N 4 L V E 8 V w R A S r k d 7 F y K / d M q V 5 L K 9 N P 2 F B h D u S h 5 x i Y 6 V 6 k 2 C F 7 m 6 K r V z e K 3 h j o H n i T 0 n + 7 M M 9 T Z 4 / 3 X I r 9 9 1 s x z S N m D R U Y K 0 b v p e Y Y I C V 4 V S w o d t M N U s w 7 e E O a 1 g q c c R 0 M B j / O 0 Q H V m m j M F Z 2 p E F j 9 f f F A E d a 9 y N i N y N s u n r W G 4 n / e o R E M 9 E m P A k G X C a p Y Z J O k s N U I B O j U R u o z R W j R v Q t w V R x + z y i X a w w N b Y z 1 7 b i z 3 Y w T 6 r F g n 9 U 8 K 6 8 f O k c J s j C H u z D I f h w D C W 4 h D J U g I K A B 3 i E J + f e e X F e n b f J a s a Z 3 u z C H z j v P w x n m d I = < / l a t e x i t > A K 1 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " q K h e j S f y k + 5 4 m P 4 3 K q Y 2 N v y x d M 8 = " > A A A C H H i c b V A 9 S w N B E N 2 L X z F + R S 1 t j g Q h a c K d F l p G b Q S b C O Y D c v H Y 2 + w l S / b 2 j t 0 5 I R 7 p 7 W y t L C 1 t 9 Q f Y i a 1 g 7 R 9 x L 0 m h i Q 8 G H u / N M D P P i z h T Y F l f R m Z h c W l 5 J b u a W 1 v f 2 N z K b + 8 0 V B h L Q u s k 5 K F s e V h R z g S t A w N O W 5 G k O P A 4 b X q D s 9 R v 3 l C p W C i u Y B j R T o B 7 g v m M Y N C S m y 8 4 A Y Y + w T w 5 G T m c + l C 6 u L Z d h w k f h o 5 k v T 6 U 3 X z R q l h j m P P E n p J i t X x / G 2 W f H m p u / t v p h i Q O q A D C s V J t 2 4 q g k 2 A J j H A 6 y j m x o h E m A 9 y j b U 0 F D q j q J O N f R u a + V r q m H 0 p d A s y x + n s i w Y F S w 8 D T n e n l a t Z L x X 8 9 z w t m V o N / 3 E m Y i G K g g k w 2 + z E 3 I T T T p M w u k 5 Q A H 2 q C i W T 6 e J P 0 s c Q E d J 4 5 n Y o 9 m 8 E 8 a R x U 7 M O K d a n j O U U T Z N E e K q A S s t E R q q J z V E N 1 R N A d e k Y v 6 N V 4 N N 6 M d + N j 0 p o x p j O 7 6 A + M z x 8 3 4 K U z < / l a t e x i t >

Figure 3 : Visualization of sample finite neural networks and their infinitely wide Bayesian counterparts (NN-GPs). Presented are networks with nonlinearity φ, L = 2 hidden layers that regress n3 = 10-dimensional outputs z2 (x) for each of the 4 (1, 2, 3, 4) inputs x from the dataset X . A hierarchical computation performed by a NN corresponds to a hierarchical transformation of the respective covariance matrix (resulting in K vec ∞ ⊗ I 10 ). Top two rows: a fully connected network (FCN, above) and the respective FCN-GP (below). Bottom two rows: a 1D CNN with no (valid) padding, and the respective CNN-GP. The analogy between the two models (NN and GP) is qualitatively similar to the case of FCN, modulo additional spatial dimensions in the internal activations and the respective additional entries in intermediary covariance matrices. Note that in general d = 10 pixels would induce a 10 × 10-fold increase in the number of covariance entries (in K 0 , K 1 ∞ , . . . ) compared to the respective FCN, yet without pooling only a small fraction of them (displayed) needs be computed to obtain the top-layer GP covariance K vec ∞ ⊗ I 10 (see §3.1).

3. TRANSFORMING A GP OVER SPATIAL LOCATIONS INTO A GP OVER:



CLASSES:

In §2.2 we showed that in the infinite channel limit a deep CNN is a GP indexed by input samples, output spatial locations, and output channel indices. Further, its covariance matrix K L ∞ can be computed in closed form. Here we show that transformations to obtain class predictions that are common in CNN classifiers can be represented as either vectorization or projection (as long as we treat classification as regression (see §G), identically to Lee et al. (2018) ). Both of these operations preserve the GP equivalence and allow the computation of the covariance matrix of the respective GP (now indexed by input samples and target classes) as a simple transformation of K L ∞ . In the following we will denote nL+2 as the number of target classes, zL+1 (x) ∈ R nL+2 as the outputs of the CNN-based classifier, and ωL+1 , bL+1 as the last (number L + 1) layer weights and biases. For each class i we will denote the empirical uncentered covariance of the outputs as

EQUATION

a random variable of shape |X | × |X |. We will show that in the scenarios considered below in §3.1 and §3.2, the outputs zL+1 i |y 0 will converge in distribution to a multivariate Gaussian (i.i.d.

for every class i) as min{n 1 , . . . , n L+1 } → ∞ with covariance denoted as

K ∞ ∈ R |X |×|X | , or, jointly, zL+1 |y 0 D → N (0, K ∞ ⊗ I nL+2 ).

As in §2.2, we postpone the formal derivation until §E.4 (Theorem E.6).

3.1. VECTORIZATION:

One common readout strategy is to vectorize (flatten) the output of the last convolutional layer z L (x) ∈ R n L+1 d into a vector4 vec z L (x) ∈ R n L+1 d and stack a fully connected layer on top:

EQUATION

where the weights ωL+1 ∈ R nL+2 ×n L+1 d and biases bL+1 ∈ R nL+2 are i.i.d. Gaussian,

EQUATION

The argument of §2.2 can then be extended to conclude that in the limit of infinite width zL+1 i |y 0 converges in distribution to a multivariate Gaussian (i.i.d. for each class i) with covariance

EQUATION

A sample 2D CNN using this readout strategy is depicted in Figure 2 , and a sample correspondence between a FCN, FCN-GP, CNN, and CNN-GP is depicted in Figure 3 .

Note that as observed in Xiao et al. (2018) , to compute the summands K L+1 ∞ α,α (x, x ) in Equation 13, one needs only the corresponding terms K L ∞ α,α (x, x ). Consequently, we only need to compute K l ∞ α,α (x, x ) : x, x ∈ X , α ∈ {1 . . . d} (or O (d) per covariance entry in an iterative or distributed setting). Note that this approach ignores pixel-pixel covariances and produces a GP corresponding to a locally connected network (see §5.1).

3.2. PROJECTION:

Another readout approach is a projection to collapse the spatial dimensions. Let h ∈ R d be a deterministic projection vector, ωL+1 ij ∼ N 0, σ 2 ω /n L+1 , and bL+1 be the same as in §3.1. Define the output to be

EQUATION

Examples of this approach include 1. Global average pooling: ) is consistently better than other models due to translation invariance of the kernel. CNN-GP with zero padding ( §3.1) outperforms an analogous CNN-GP without padding as depth increases. At depth 15 the spatial dimension of the output without padding is reduced to 1 × 1, making the CNN-GP without padding equivalent to the center pixel selection strategy ( §3.2.2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed (Xiao et al., 2018) . CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subset of CIFAR10. See §G.3 for experimental details.

EQUATION

This approach corresponds to applying global average pooling right after the last convolutional layer. 5 This approach takes all pixel-pixel covariances into consideration and makes the kernel translation invariant. However, it requires O |X | 2 d 2 memory to compute the sample-sample covariance of the GP (or O d 2 per covariance entry in an iterative or distributed setting). It is impractical to use this method to analytically evaluate the GP, and we propose to use a Monte Carlo approach (see §4).

2.. Subsampling one particular pixel: take:

EQUATION

This approach makes use of only one pixel-pixel covariance, and requires the same amount of memory as vectorization ( §3.1) to compute.

We compare the performance of presented strategies in Figure 4 . Note that all described strategies admit stacking additional FC layers on top while retaining the GP equivalence, using a derivation analogous to §2.2 (Lee et al., 2018; Matthews et al., 2018b) .

We introduce a Monte Carlo estimation method for NN-GP kernels which are computationally impractical to compute analytically, or for which we do not know the analytic form. Similar in spirit to traditional random feature methods (Rahimi & Recht, 2007) , the core idea is to instantiate many random finite width networks and use the empirical uncentered covariances of activations to estimate the Monte Carlo-GP (MC-GP) kernel, where θ consists of M draws of the weights and biases from their prior distribution, θ m ∼ p (θ), and n is the width or number of channels in hidden layers. The MC-GP kernel converges to the analytic kernel with increasing width, lim n→∞ K l n,M = K l ∞ in probability. For finite width networks, the uncertainty in Daniely et al. (2016, Theorems 2, 3) , we know that for well-behaved nonlinearities (that include ReLU and erf considered in our experiments)

EQUATION

K l n,M is Var K l n,M = Var θ K l n (θ) /M . From

Var θ K l n (θ) ∝ 1 n , which leads to Var θ [K l n,M ] ∝ 1 M n . For finite n, K l n,M is also a biased estimate of K l ∞ ,

where the bias depends solely on network width. We do not currently have an analytic form for this bias, but we can see in Figures 5 and 10 that for the hyperparameters we probe it is small relative to the variance. In particular,

K l n,M (θ) -K L ∞ 2

F is nearly constant for constant M n. We thus treat M n as the effective sample size for the Monte Carlo kernel estimate. Increasing M and reducing n can reduce memory cost, though potentially at the expense of increased compute time and bias.

In a non-distributed setting, the MC-GP reduces the memory requirements to compute GP pool from

O |X | 2 d 2 to O |X | 2 + n 2 +

nd , making the evaluation of CNN-GPs with pooling practical.

5. DISCUSSION:



5.1. BAYESIAN CNNS WITH MANY CHANNELS ARE IDENTICAL TO LOCALLY CONNECTED NETWORKS, IN THE ABSENCE OF POOLING:

Locally Connected Networks (LCNs) (Fukushima, 1975; Lecun, 1989) are CNNs without weight sharing between spatial locations. LCNs preserve the connectivity pattern, and thus topology, of a CNN. However, they do not possess the equivariance property of a CNN -if an input is translated, the latent representation in an LCN will be completely different, rather than also being translated.

The CNN-GP predictions without spatial pooling in §3.1 and §3.2.2 depend only on sample-sample covariances, and do not depend on pixel-pixel covariances. LCNs destroy pixel-pixel covariances:

K L ∞ LCN α,α (x,

x ) = 0, for α = α and all x, x ∈ X and L > 0. However, LCNs preserve the covariances between input examples at every pixel:

K L ∞ LCN α,α (x, x ) = K L ∞ CNN α,α (x, x ).

As a result, in the absence of pooling, LCN-GPs and CNN-GPs are identical. Moreover, LCN-GPs with pooling are identical to CNN-GPs with vectorization of the top layer (under suitable scaling of y L+1 ). We confirm these findings experimentally in trained networks in the limit of large width in Figures 1 and 6 (b), as well as by demonstrating convergence of MC-GPs of the respective architectures to the same CNN-GP (modulo scaling of y L+1 ) in Figures 5 and 10 .

5.2. POOLING LEVERAGES EQUIVARIANCE TO PROVIDE INVARIANCE:

The only kernel leveraging pixel-pixel covariances is that of the CNN-GP with pooling. This enables the predictions of this GP and the corresponding CNN to be invariant to translations (modulo edge effects) -a beneficial quality for an image classifier. We observe strong experimental evidence supporting the benefits of invariance throughout this work (Figures 1, 4, 5, 6 (b) ; Table 1 ), in both CNNs and CNN-GPs. In the absence of pooling, the benefits of equivariance and weight sharing are more challenging to explain in terms of Bayesian priors on class predictions (since without pooling equivariance is not a property of the outputs, but only of intermediary representations). Indeed, in this work we find that the performance of finite width SGD-trained CNNs often approaches that of their CNN-GP counterpart (Figure 6 , b, c), 6 suggesting that in those cases equivariance does not play a beneficial role in SGD-trained networks.

However, as can be seen in Figures 1, 6 (c), 7, and Table 1 , the best CNN overall outperforms the best CNN-GP by a significant margin -an observation specific to CNNs and not FCNs or LCNs. 7 We observe this gap in performance especially in the case of ReLU networks trained with a large learning rate. In Figure 1 we demonstrate this large gap in performance by evaluating different models with equivalent architecure and hyperparameter settings, chosen for good SGD-trained CNN performance.

We conjecture that equivariance, a property lacking in LCNs and the Bayesian treatment of the infinite channel CNN limit, contributes to the performance of SGD-trained finite channel CNNs with the correct settings of hyperparameters. Nonetheless, more work is needed to disentangle and quantify the separate contributions of stochastic optimization and finite width effects, 8 and differences in performance between CNNs with weight sharing and their corresponding CNN-GPs.

6. CONCLUSION:

In this work we have derived a Gaussian process that corresponds to fully Bayesian multi-layer CNNs with infinitely many channels. The covariance of this GP can be efficiently computed either in closed form or by using Monte Carlo sampling, depending on the architecture.

The CNN-GP achieves state of the art results for GPs without trainable kernels on CIFAR10. It can perform competitively with CNNs (that fit the training set) of equivalent architecture and weight priors, which makes it an appealing choice for small datasets, as it eliminates all training-related hyperparameters. However, we found that the best overall performance, at least in the absence of pooling, is achieved by finite SGD-trained CNNs and not by their infinite Bayesian counterparts. We hope our work stimulates future research towards understanding the distribution over functions induced by model architecture and training approach, and what aspects of this distribution are important for model performance.

Another natural extension of our work is the study of other deep learning architectures in the infinitely wide limit. After the publication of this paper, Yang (2019) devised a unifying framework proving the GP convergence for even more models (such as ones using batch normalization, (self-)attention, LSTM) with slightly different assumptions on the nonlinearity.

6 This observation is conditioned on the respective NN fitting the training set to 100%. Underfitting breaks the correspondance to an NN-GP, since train set predictions of such a network no longer correspond to the true training labels. Properly tuned underfitting often also leads to better generalization (Table 1 ). 7 Performing an analogous large-dataset comparison between CNNs and CNN-GPs with pooling was not computationally feasible. Their relative performance remains an interesting open question for future research. 8 We remark that concerns about GPs not being able to learn hierarchical representations have been raised in the literature (Matthews et al., 2018a, Section 7) , (Neal, 1995 , Chapter 5), (MacKay, 2003, Section 45.7) However, practical impact of these assertions have not been extensively investigated empirically or theoretically, and we hope that our work stimulates research in this direction. Each line corresponds to a particular choice of architecture and initialization hyperparameters, with best learning rate and weight decay selected independently for each number of channels (x-axis).

(b): SGD-trained CNNs often approach the performance of their corresponding CNN-GP with increasing number of channels. All models have the same architecture except for pooling and weight sharing, as well as training-related hyperparameters such as learning rate, weight decay and batch size, which are selected for each number of channels (x-axis) to maximize validation performance (y-axis) of a neural network. As the number of channels grows, best validation accuracy increases and approaches accuracy of the respective GP (solid horizontal line). (c): However, the best-performing SGD-trained CNNs can outperform their corresponding CNN-GPs. Each point corresponds to the test accuracy of: (y-axis) a specific CNN-GP; (x-axis) the best (on validation) CNN with the same architectural hyper-parameters selected among the 100%-accurate models on the full training CIFAR10 dataset with different learning rates, weight decay and number of channels. While CNN-GP appears competitive against 100%-accurate CNNs (above the diagonal), the best CNNs overall outperform CNN-GPs by a significant margin (below the diagonal, right means only models that achieved 100% training accuracy were considered). CNN-GP outperforms SGD-trained models optimized with a small learning rate to 100% train accuracy. When SGD optimization is allowed to underfit the training set, there is a significant improvement in generalization. Further, when ReLU nonlinearities are paired with large learning rates, the performance of SGD-trained models again improves relative to CNN-GPs, suggesting a beneficial interplay between ReLUs and fast SGD training. These differences in performance between CNNs and CNN-GPs are not observed between FCNs and FCN-GPs, or between LCNs and LCN-GPs (Figure 1 ), suggesting that equivariance is the underlying factor responsible for the improved performance of finite SGD-trained CNNs relative to infinite Bayesian CNNs without pooling. Further comparison between SGD-trained CNNs and CNN-GPs with pooling (omitted due to computational limitations), where both models do share the property of invariance, would be an interesting direction for future research. See 

#Channels:

Figure 9 : No underfitting in small models. Training loss (vertical axis) of best (in terms of validation loss) neural networks as the number of channels increases (horizontal axis). While perfect 0 loss is not achieved (but 100% accuracy is), we observe no consistent improvement when increasing the capacity of the network (left to right). This eliminates underfitting as a possible explanation for why small models perform worse in Figure 6 

B RELATIONSHIP TO DEEP SIGNAL PROPAGATION:

The recurrence relation linking the GP kernel at layer l+1 to that of layer l following from Equation 9(i.e.

K l+1 ∞ = (C • A) K l ∞

) is precisely the covariance map examined in a series of related papers on signal propagation (Xiao et al., 2018; Poole et al., 2016; Schoenholz et al., 2017; Lee et al., 2018) (modulo notational differences; denoted as F , C or e.g. A C in Xiao et al. (2018) ). In those works, the action of this map on hidden-state covariance matrices was interpreted as defining a dynamical system whose large-depth behavior informs aspects of trainability. In particular, as

l → ∞, K l+1 ∞ = (C • A) K l ∞ ≈ K l ∞ ≡ K * ∞ , i.e.

the covariance approaches a fixed point K * ∞ . The convergence to a fixed point is problematic for learning because the hidden states no longer contain information that can distinguish different pairs of inputs. It is similarly problematic for GPs, as the kernel becomes pathological as it approaches a fixed point. Precisely, in both chaotic and ordered regimes, outputs of the GP become asymptically identically correlated. Either of these scenarios captures no information about the training data in the kernel and makes learning infeasible. This problem can be ameliorated by judicious hyperparameter selection, which can reduce the rate of exponential convergence to the fixed point. For hyperpameters chosen on a critical line separating two untrainable phases, the convergence rates slow to polynomial, and very deep networks can be trained, and inference with deep NN-GP kernels can be performed -see Figure 11 .

C STRIDED CONVOLUTIONS, AVERAGE POOLING IN INTERMEDIATE LAYERS, HIGHER DIMENSIONS:

Our analysis in the main text can easily be extended to cover average pooling and strided convolutions (applied before the pointwise nonlinearity). Recall that conditioned on K l the pre-activation z l j (x) ∈ R d1 is a zero-mean multivariate Gaussian. Let B ∈ R d2×d1 denote a linear operator. Then Bz l j (x) ∈ R d2 is a zero-mean Gaussian, and the covariance is

EQUATION

One can easily see that Bz l j K l j are i.i.d. multivariate Gaussian as well. Strided convolution. Strided convolution is equivalent to a non-strided convolution composed with subsampling. Let s ∈ N denote size of the stride. Then the strided convolution is equivalent to choosing B as follows:

B ij = δ(is -j) for i ∈ {0, 1, . . . (d 2 -1)}.

Average pooling. Average pooling with stride s and window size ws is equivalent to choosing B ij = 1/ws for i = 0, 1, . . . (d 2 -1) and j = is, . . . , (is + ws -1).

ND convolutions. Note that our analysis in the main text (1D) easily extends to higher-dimensional convolutions by replacing integer pixel indices and sizes d, α, β with tuples (see also Figure 2 ). In Equation 2 

EQUATION

EQUATION

We use the shorthand K (X , X ) to denote the 

E EQUIVALENCE BETWEEN RANDOMLY INITIALIZED NNS AND GPS:

In this section, we present two different approaches, the sequential limit ( §E.3) and simultaneous limit ( §E.4), to illustrate the relationship between many-channels Bayesian CNNs and GPs.

Sequential limit ( §E.3) involves taking the infinite channel limit in hidden layers in a sequence, starting from bottom (closest to inputs) layers and going upwards (to the outputs), i.e. n 1 → ∞, . . . , n L → ∞. Note that this approach in fact only gives intuition into construction a GP using a NN architecture to define its covariance, and does not provide guarantees on actual convergence of large but finite Bayesian CNNs to GPs (which is of most practical interest), nor does it guarantee the existence of the specified GP on a given probability space. However, it has the following benefits:

1. Weak assumptions on the NN activation function φ and on the distribution of the NN parameters. 2. The arguments can be easily extended to more complicated network architectures, e.g. architectures with max pooling, dropout, etc. 3. A straightforward and intuitive way to compute the covariance of the Gaussian process without diving into mathematical details.

Simultaneous limit ( §E.4) considers growing the number of channels in hidden layers uniformly, i.e. min n 1 , . . . , n L → ∞. This approach establishes convergence of finite channel Bayesian CNNs to GPs and is thus a more practically relevant result. However, it makes stronger assumptions, and the proof is more involved.

We highlight that the GPs obtained by the two approaches are identical.

In both sections, we only provide the arguments for CNNs. It is straightforward (and in fact simpler) to extend them to LCNs and FCNs. Indeed, an FCN is a particular case of a CNN where the inputs and filters have singular spatial dimensions (d = 1, k = 0). For LCNs, the proof goes through in an identical fashion if we replace A with A LCN defined as

A LCN (K) α,α (x, x ) ≡ δ α,α [A (K)] α,α (x, x ).

E.1 SETUP Probability space. Let P be a collection of countably many mutually independent random variables (R.V.s) defined on a probability space (Ω, F, P ), where F is a product Borel σ-algebra and P is the probability measure. Here P ≡ W ∪ B ∪ H is the collection of parameters used to define neural networks: Inputs. We will consider a fixed X ⊆ R n 0 ×d to denote the inputs, with input channel count n 0 , number of pixels d. Assume x = 0 for all x ∈ X and |X |, the cardinality of the inputs, is finite.

(i) Weights. W = l∈N W l and W l = ω l ij,β : i, j ∈ N, β ∈ [±k] , where [±k] ≡ [-k, k] ∩ Z. We assume ω l ij,

However, our results can be straightforwardly extended to a countably-infinite input indexing spaces X for certain topologies via an argument presented in Matthews et al. (2018a, section 2.2), allowing to infer weak convergence on X from convergence on any finite subset (which is the case we consider in this text; see also Billingsley (1999, page 19 ) for details). For this reason, as in Matthews et al.

(2018a), weak convergence of countably-infinite stochastic processes will be considered with respect to the topology generated by the following metric:

ν (s, s ) ≡ ∞ k=1 2 -k min (1, |s k -s k |) , ∀s, s ∈ R N .

Notation, shapes, and indexing. We adopt the notation, shape, and indexing convention similar to §2.1, which the reader is encouraged to review. We emphasize that whenever an index is omitted, the variable is assumed to contain all possible entries along the respective dimension (e.g. whole X if x is omitted, or all n l channels if the channel i is omitted).

E.2 PRELIMINARY:

We will use the following well-known theorem. Theorem E.1. Let X, {X n } n∈N be R.V.s in R m . The following are equivalent:

EQUATION

(iii) (Lévy's Continuity Theorem) The characteristic functions of X n , i.e. E e it T Xn , converge to those of X pointwise, i.e. for all t ∈ R m ,

EQUATION

where i denotes the imaginary unit.

Using the equivalence between (i) and (iii), it is straightforward to show that Theorem E.2 (Cramer-Wold, (Billingsley (1995) , Theorem 29.4)).

EQUATION

In particular,

EQUATION

E.3 SEQUENTIAL LIMIT:

In this section, we give intuition into constructing a Gaussian process using an infinite channel CNN with the limits taken sequentially. Informally, our argument amounts to showing that if the inputs z l-1 to the layer l are a multivariate Gaussian with covariance A K l ⊗ I n l , then it's outputs z l converge in distribution to a multivariate Gaussian with covariance A K l+1 ⊗ I n l+1 as n l → ∞. This "allows" us to sequentially replace z 1 , z 2 , . . . , z L with their respective limiting Gaussian R.V.s as we take n 1 → ∞, n 2 → ∞, . . . , n L → ∞. However, since each convergence only holds in distribution and does not guarantee the existence of the necessary GPs on a given probability space, what follows merely gives intuition into understanding the relationship between wide Bayesian neural networks and GPs (contrary to §E.4, which presents a rigorous convergence proof for min n 1 , . . . , n L → ∞).

Let Ψ 1 denote the space of functions with a uniformly bounded second moment, i.e. having

EQUATION

Let N * = N\{0}. We construct Gaussian processes with the following iterative in l (from 0 to L) procedure:

(i) If l > 0, define random variables (a GP) z l,∞ i i∈N * in (Ω, F, P ) as i.i.d. Gaussian with mean zero and covariance A K l ∞ , so that they are also independent from any future events, i.e. independent from the σ-algebra generated by all R.V.s with layer index greater than l. Here we implicitly assume the probability space (Ω, F, P ) has enough capacity to fit in the R.V.s generated by the above procedure, if not we will extend the sample space Ω using product measures.

(ii) For n ∈ N * , define In what follows, we use the central limit theorem to prove (iii). Proof. We proceed by induction. This is obvious in the base case l = 0, since the weights and biases are assumed to be independent Gaussian. Now we assume the theorem holds for l -1. This implies u j + q.

EQUATION

z l,n i,α (x) ≡    1 √ n 0 j∈[n 0 ] β∈[±k] √ v β ω l ij,

(33)

It is not difficult to see that u j are i.i.d. and q is Gaussian. Then we can use the central limit theorem to conclude that the above converges in distribution to a Gaussian, once we verify the second moment of u j is finite. Using the fact that ω l ij,β i,β is a collection of independent R.V.s, and integrating over these R.V.s first, we get 

EQUATION

)

where by Ā we denote the linear transformation part of A from Equation 5, i.e. A without the translation term σ 2 b :

EQUATION

To prove finiteness of the second moment in Equation 36, it is sufficient to show that all the diagonal terms of K l ∞ are finite. This easily follows from the assumption φ ∈ Ψ 1 and the definition of

K l ∞ = (C • A) l K 0 .

Together with the distribution of v (whose covariance is straightforward to compute), the joint distribution of z l,n i i∈ [m] converges weakly to a mean zero Gaussian with covariance matrix A K l ∞ ⊗ I m by Theorem E.1 (Equation 26). Remark E.1. The results of Theorem E.3 can be strengthened / extended in many directions.

(1) The same result for a countably-infinite input index set X follows immediately according to the respective remark in §E.1.

(2) The same analysis carries over (and the covariance matrix can be computed without much extra effort) if we stack a channel-wise deterministic affine transform after the convolution operator. Note that average pooling (not max pooling, which is not affine), global average pooling and the convolutional striding are particular examples of such affine tranforms. Moreover, valid padding (i.e. no padding) convolution can be regarded as a subsampling operator (a linear projection) composed with the regular circular padding.

(3) The same analysis applies to max pooling, but computing the covariance may require non-trivial effort. Let m denote the max pooling operator and assume it is applied right after the activation function.

The assumption y l i i∈[n] = φ(z l-1,∞ i ) i∈ [n] are i.i.d. implies m y l i i∈[n] are also i.i.d. Then we can proceed exactly as above except for verifying the finiteness of second moment of m(y l i ) with the following trivial estimate:

EQUATION

)

where s is the window size of the max pooling.

In general, one can stack a channel-wise deterministic operator op on y l i so long as the second moment of op y l i is finite. One can also stack a stochastic operator (e.g. dropout), so long as the outputs are still channel-wisely i.i.d. and have finite second moments.

E.4 SIMULTANEOUS LIMIT:

In this section, we present a sufficient condition on the activation function φ so that the neural networks converge to a Gaussian process as all the widths approach infinity simultaneously. Precisely, let t ∈ N and for each l ≥ 0, let n l : N → N be the width function at layer l (by convention n 0 (t) = n 0 is constant). We are interested in the simultaneous limit n l = n l (t) → ∞ as t → ∞, i.e., for any fixed L ≥ 1 min n 1 (t), . . . , n L (t) ---→ t→∞ ∞.

(39)

Define a sequence of finite channel CNNs as follows:

y l,t i,α (x) ≡     

x i,α , l = 0 φ z l-1,t i,α (x) , l > 0 , √ v β ω l ij,β y l,t j,α+β (x) + b l i , l > 0

EQUATION

)

This network induces a sequence of covariance matrices K l t (which are R.V.s): for l ≥ 0 and t ≥ 0, for x, x ∈ X K l t α,α (x, x ) ≡ 1 n l (t) n l (t) i=1 y l,t i,α (x)y l,t i,α (x ).

We make an extra assumption on the parameters.

Assumption: all R.V.s in W are Gaussian distributed. Let Ψ denotes the space of measurable functions φ with the following properties:

1. Uniformly bounded second moment: for every R ≥ 1, C 2 (φ, R) < ∞.

2. Lipschitz continuity: for every R ≥ 1, there exists β = β (φ, R) > 0 such that for all Σ, Σ ∈ PSD 2 (R),

EQUATION

3. Uniform convergence in probability: for every R ≥ 1 and every ε > 0 there exists a positive sequence ρ n (φ, ε, R) with ρ n (φ, ε, R) → 0 as n → ∞ such that for every Σ ∈ PSD 2 (R) and any {(x i , y i )} n i=1 i.i.d. ∼ N (0, Σ)

EQUATION

)

We will also use Ψ 1 , Ψ 2 and Ψ 3 to denote the spaces of measurable functions φ satisfying properties 1, 2, and 3, respectively. It is not difficult to see that for every i, Ψ i is a vector space, and so is , the exponent in the above expression is just a linear combination of independent Gaussian R.V.s ω l ij,β , b l i , which is also a Gaussian. Integrating out these

Ψ = ∩ i Ψ i .

Using the fact Eω ij,α ω i j ,α = 0 unless (ij, α) = (i j , α ) and integrating out the R.V.s of the dense layer, the characteristic function is equal to We first note that the affine transform A is σ 2 ω -Lipschitz and property 2 of Ψ implies that the C operator is β-Lipschitz (both w.r.t. w.r.t. • ∞ ). Indeed, if we consider Σ ≡

E exp      - 1 2 i∈[m] x,x ∈X a i (x)a i (x )      1 n l (t)d

[K] α,α (x, x)

[K] α,α (x, x ) [K] α ,α (x , x) [K] α ,α (x , x ) ,

then [C (K)] α,α (x, x ) = T ∞ (Σ). Thus C • A is σ 2 ω β-Lipschitz. We now prove the theorem by induction. Assume K l t P → K l ∞ as t → ∞ (obvious for l = 0). We first remark that K l ∞ ∈ PSD |X |d , since PSD |X |d E K l t → K l ∞ and PSD |X |d is closed. Moreover, due to Equation 4, A necessarily preserves positive semi-definiteness, and therefore A K l ∞ ∈ PSD |X |d as well. Now let ε > 0 be sufficiently small so that the ε 2β -neighborhood of A(K l ∞ ) is contained in PSD |X |d (R), where we take R to be large enough for K l ∞ to be an interior point of PSD |X |d (R).9 Since

EQUATION

to prove K l+1 t P → K l+1 ∞ , it suffices to show that for every δ > 0, there is a t * such that for all t > t * ,

EQUATION

By our induction assumption, there is a t l such that for all t > t l

EQUATION

)

Remark E.4. In practice the 1/n decay bound obtained by Chebyshev's inequality in Equation 100is often too weak to be useful. However, if φ is linearly bounded, then one can obtain an exponential decay bound via the following concentration inequality: Lemma E.8. If |φ(x)| ≤ a + b|x| a.e., then there is an absolute constant c > 0 and a constant κ = κ(a, b, R) > 0 such that property 3 (Equation 48) holds with 

We can apply Bernstein-type inequality (Vershynin, 2010, Lemma 5.16 ) to conclude that there is a c > 0 such that for every Σ ∈ PSD 2 (R) and any {(x i , y i )} n i=1 i.i.d. ∼ N (0, Σ)

EQUATION

It remains to prove Claim E.9. For p ≥ 1, 

E (x,

We applied Cauchy-Schwarz' inequality in the first inequality, the triangle inequality in the second one, the fact Σ 11 , Σ 22 ≤ R in the third one, absolute moments estimate of standard Gaussian in the fourth one, where c is a constant such that

EQUATION

F GLOSSARY

We use the following shorthands in this work:

1. NN -neural network; 2. CNN -convolutional neural network; 3. LCN -locally connected network, a.k.a. convolutional network without weight sharing; 4. FCN -fully connected network, a.k.a. multilayer perceptron (MLP); 5. GP -Gaussian process; 6. X-GP -a GP equivalent to a Bayesian infinitely wide neural network of architecture X ( §2).

:

We will use Roman letters to index channels and Greek letters for spatial location. We use letters i, j, i , j , etc to denote channel indices, α, α , etc to denote spatial indices and β, β , etc for filter indices.

The weak law of large numbers allows convergence in probability of individual entries of K l . However, due to the finite dimensionality of K l , joint convergence in probability follows.

Note that since per our notation described in §2.1 z L (x) is already considered a 1D vector, here this operation simply amounts to re-indexing z L (x) with one channel index j instead of two (channel i and pixel α).

Spatially local average pooling in intermediary layers can be constructed in a similar fashion ( §C). We focus on global average pooling in this work to more effectively isolate the effects of pooling from other aspects of the model like local connectivity or equivariance.

Such R always exists, because the diagonal terms of K l ∞ are always non-zero. See(Long & Sedghi, 2019, Lemma 4) for proof.

Due to time and compute limitations certain large configurations could not be evaluated. We believe this did not impact the results of this work in a qualitative way.

Figures and Tables:

Figure 1: Disentangling the role of network topology, equivariance, and invariance on test performance, for SGD-trained and infinitely wide Bayesian networks (NN-GPs). Accuracy (%) on CIFAR10 of different models of the same depth, nonlinearity, and weight and bias variances. (a) Fully connected models -FCN (fully connected network) and FCN-GP (infinitely wide Bayesian FCN) underperform compared to (b) LCNs (locally connected network, a CNN without weight sharing) and CNN-GP (infinitely wide Bayesian CNN), which have a hierarchical local topology beneficial for image recognition. As derived in §5.1: (i) weight sharing has no effect in the Bayesian treatment of an infinite width CNN (CNN-GP performs similarly to an LCN), and (ii) pooling has no effect on generalization of an LCN model (LCN and LCN with pooling perform nearly identically). (c) Local connectivity combined with equivariance (CNN) is enabled by weight sharing in an

Figure1: Different dimensionality collapsing strategies described in §3. Validation accuracy of an MC-CNN-GP with pooling ( §3.2.1) is consistently better than other models due to translation invariance of the kernel. CNN-GP with zero padding ( §3.1) outperforms an analogous CNN-GP without padding as depth increases. At depth 15 the spatial dimension of the output without padding is reduced to 1 ⇥ 1, making the CNN-GP without padding equivalent to the center pixel selection strategy ( §3.2.2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed(Xiao et al., 2018). CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subsets of CIFAR10. See §A.7.3 for experimental details.

Figure 2: Validation accuracy (left) of an MC-CNN-GP increases with n ⇥ M (i.e. channel count times number of samples) and approaches that of the exact CNN-GP (not shown), while the distance (right) to the exact kernel decreases. The dark band in the left plot corresponds to ill-conditioning of K L+1n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 ⇥ 8. See Figure7for similar results with other architectures and §A.7.2 for experimental details. random finite width networks and use the empirical uncentered covariances of activations to estimate the Monte Carlo-GP (MC-GP) kernel,

Figure1: Different dimensionality collapsing strategies described in §3. Validation accuracy of an MC-CNN-GP with pooling ( §3.2.1) is consistently better than other models due to translation invariance of the kernel. CNN-GP with zero padding ( §3.1) outperforms an analogous CNN-GP without padding as depth increases. At depth 15 the spatial dimension of the output without padding is reduced to 1 ⇥ 1, making the CNN-GP without padding equivalent to the center pixel selection strategy ( §3.2.2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed(Xiao et al., 2018). CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subsets of CIFAR10. See §A.7.3 for experimental details.

Figure 2: Validation accuracy (left) of an MC-CNN-GP increases with n ⇥ M (i.e. channel count times number of samples) and approaches that of the exact CNN-GP (not shown), while the distance (right) to the exact kernel decreases. The dark band in the left plot corresponds to ill-conditioning of K L+1n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 ⇥ 8. See Figure7for similar results with other architectures and §A.7.2 for experimental details.

of post-nonlinearity activations given normal pre-activations)

e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d + / d 3 + w / e H h s d

e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d + / d 3 + w / e H h s d

e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d + / d 3 + w / e H h s d

e W P c e b C z 8 D 5 2 K p s G S n y y 5 s 3 e u M Z p 4 U U F q L o V x B e u 3 7 j 5 q 2 N 2 7 0 7 d + / d 3 + w / e H h s d

r w B M / w 4 j w 4 r 8 6 b 8 z 5 d X X J m N w f w B 8 7 H D 0 B 7 m P Y = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " S 9 7 h A c 1 X q 6

Figure4: Different dimensionality collapsing strategies described in §3. Validation accuracy of an MC-CNN-GP with pooling ( §3.2.1) is consistently better than other models due to translation invariance of the kernel. CNN-GP with zero padding ( §3.1) outperforms an analogous CNN-GP without padding as depth increases. At depth 15 the spatial dimension of the output without padding is reduced to 1 × 1, making the CNN-GP without padding equivalent to the center pixel selection strategy ( §3.2.2) -which also performs worse than the CNN-GP (we conjecture, due to overfitting to centrally-located features) but approaches the latter (right) in the limit of large depth, as information becomes more uniformly spatially distributed(Xiao et al., 2018). CNN-GPs generally outperform FCN-GP, presumably due to the local connectivity prior, but can fail to capture nonlinear interactions between spatially-distant pixels at shallow depths (left). Values are reported on a 2K/4K train/validation subset of CIFAR10. See §G.3 for experimental details.

Figure 5: Convergence of NN-GP Monte Carlo estimates. Validation accuracy (left) of an MC-CNN-GP increases with n × M (channel count × number of samples) and approaches that of the exact CNN-GP (not shown), while the distance (right) to the exact kernel decreases. The dark band in the left plot corresponds to ill-conditioning of K L+1 n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 × 8. See Figure 10 for similar results with other architectures and §G.2 for experimental details.

FINITE CHANNEL SGD-TRAINED CNNS CAN OUTPERFORM INFINITE CHANNEL BAYESIAN CNNS, IN THE ABSENCE OF POOLING

Figure6: (a): SGD-trained CNNs often perform better with increasing number of channels. Each line corresponds to a particular choice of architecture and initialization hyperparameters, with best learning rate and weight decay selected independently for each number of channels (x-axis). (b): SGD-trained CNNs often approach the performance of their corresponding CNN-GP with increasing number of channels. All models have the same architecture except for pooling and weight sharing, as well as training-related hyperparameters such as learning rate, weight decay and batch size, which are selected for each number of channels (x-axis) to maximize validation performance (y-axis) of a neural network. As the number of channels grows, best validation accuracy increases and approaches accuracy of the respective GP (solid horizontal line). (c): However, the best-performing SGD-trained CNNs can outperform their corresponding CNN-GPs. Each point corresponds to the test accuracy of: (y-axis) a specific CNN-GP; (x-axis) the best (on validation) CNN with the same architectural hyper-parameters selected among the 100%-accurate models on the full training CIFAR10 dataset with different learning rates, weight decay and number of channels. While CNN-GP appears competitive against 100%-accurate CNNs (above the diagonal), the best CNNs overall outperform CNN-GPs by a significant margin (below the diagonal, right). For further analysis of factors leading to similar or diverging behavior between SGD-trained finite CNNs and infinite Bayesian CNNs see Figures1, 7, and Table1. Experimental details: all networks have reached 100% training accuracy on CIFAR10. Values in (b) are reported on an 0.5K/4K train/validation subset downsampled to 8 × 8 for computational reasons. See §G.5 and §G.1 for full experimental details of (a, c) and (b) plots respectively.

Figure 7: Aspects of architecture and inference influencing test performance. Test accuracy (vertical axis, %) for the best model within each model family (horizontal axis), maximizing validation accuracy over depth, width, and training and initialization hyperpameters ("No underfitting"means only models that achieved 100% training accuracy were considered). CNN-GP outperforms SGD-trained models optimized with a small learning rate to 100% train accuracy. When SGD optimization is allowed to underfit the training set, there is a significant improvement in generalization. Further, when ReLU nonlinearities are paired with large learning rates, the performance of SGD-trained models again improves relative to CNN-GPs, suggesting a beneficial interplay between ReLUs and fast SGD training. These differences in performance between CNNs and CNN-GPs are not observed between FCNs and FCN-GPs, or between LCNs and LCN-GPs (Figure1), suggesting that equivariance is the underlying factor responsible for the improved performance of finite SGD-trained CNNs relative to infinite Bayesian CNNs without pooling. Further comparison between SGD-trained CNNs and CNN-GPs with pooling (omitted due to computational limitations), where both models do share the property of invariance, would be an interesting direction for future research. See Table1for further results on other datasets, as well as comparison to GP performance in prior literature. See §G.5 for experimental details.

Figure 8: Validation loss convergence. Best validation loss (vertical axis) of trained neural networks (dashed line) as the number of channels increases (horizontal axis) approaches that of a respective (MC-)CNN-GP (solid horizontal line). See Figure 6 (b) for validation accuracy, Figure 9 for training loss and §G.1 for experimental details.No PoolingGlobal Average Pooling

Figure 9:  No underfitting in small models. Training loss (vertical axis) of best (in terms of validation loss) neural networks as the number of channels increases (horizontal axis). While perfect 0 loss is not achieved (but 100% accuracy is), we observe no consistent improvement when increasing the capacity of the network (left to right). This eliminates underfitting as a possible explanation for why small models perform worse in Figure6(b). See Figure8for validation loss and §G.1 for experimental details.

Figure 10: Convergence of NN-GP Monte Carlo estimates. As in Figure5, validation accuracy (left) of MC-GPs increases with n × M (i.e. width times number of samples), while the distance (right) to the the respective exact GP kernel (or the best available estimate in the case of CNN-GP with pooling, top row) decreases. We remark that when using shared weights, convergence is slower as smaller number of independent random parameters are being used. See §G.2 for experimental details.

β values would have to span the hypercube [±k] N = {-k, . . . , k} N in the preactivation definition. Similarly, in §3 the normalizing factor d (d 2 ) should be the product (squared) of its entries, and summations over α, β should span the [d 0 ] × • • • × [d N ] hypercube as well. The definition of the kernel propagation operator A in Equation5will remain exactly the same, so long as β is summed over the hypercube, and the variance weights remain respectively normalizedβ v β = 1.D REVIEW OF EXACT BAYESIAN REGRESSION WITH GPSOur discussion in the paper has focused on model priors. A crucial benefit we derive by mapping to a GP is that Bayesian inference is straightforward to implement and can be done exactly for regression(Rasmussen & Williams, 2006, chapter 2), requiring only simple linear algebra. Let X denote training inputs x 1 , ..., x |X | , t T = t 1 , ..., t |X | training targets, and collectively D for the training set. The integral over the posterior can be evaluated analytically to give a posterior predictive distribution on a test point y * which is Gaussian, (z * |D, y * ) ∼ N µ * , σ 2 * , with

β are i.i.d. R.V.s with mean zero and finite variance 0 < σ 2 ω < ∞ (note the lack of scaling by input dimensionality, compensated for later in Equations 29 and 42; see also similar notation used in Matthews et al. (2018a)). When l = 0, we further assume they are Gaussian distributed. (ii) Biases. B = l∈N B l and B l = b l j : j ∈ N . We assume b l j are i.i.d. Gaussian with mean zero and variance 0 ≤ σ 2 b < ∞. (iii) Place-holder. H is a place-holder to store extra (if needed) R.V.s , e.g. parameters coming from the final dense layer.

≡ {1, . . . , n} and [±k] ≡ {-k, . . . , 0, . . . k} . (30) (iii) Prove that for any finite m ≥ 1, z l,n i i∈[m] ⊆ R |X |d converges in distribution to a multivariate normal with mean zero and covariance A K l ∞ ⊗ I m as n → ∞, where K l ∞ is defined identically to Equation 9. As a consequence, per our remark in §E.1, z l,n i i∈N * converges weakly to the GP z l,∞ i i∈N * .

If φ ∈ Ψ 1 , then for every l ≥ 0 and every m ≥ 1, z l,n i i∈[m] ⊆ R |X |d converges in distribution to a multivariate normal with mean zero and covariance A K l ∞ ⊗ I m .

i.i.d. random variables. Choose a vector a ≡ [a i (x)] T i∈[m], x∈X ∈ R m|X | . Then i∈

Let PSD m denote the set of m × m positive semi-definite matrices and for R ≥ 1, definePSD m (R) ≡ {Σ ∈ PSD m : 1/R ≤ Σ α,α ≤ R for 1 ≤ α ≤ m} . (44)Further let T ∞ : PSD 2 → R be a function given byT ∞ (Σ) ≡ E (x,y)∼N (0,Σ) [φ(x)φ(y)] ,(45)and C k (φ, R) (may equal ∞) denotes the uniform upper bound for the k-th momentC k (φ, R) ≡ sup 1/R≤r≤R E x∼N (0,r) |φ(x)| k .(46)

, we say that a function f : R → R is exponentially bounded if there exist a, b > 0 such that |f (x)| ≤ ae b|x| a.e. (almost everywhere) (49)We now prove our main result presented in §2.2.3 through the following three theorems.Theorem E.4. If φ is absolutely continuous and φ is exponentially bounded then φ ∈ Ψ.Theorem E.5. If φ ∈ Ψ, then for l ≥ 0, If for l ≥ 0, K l t P → K l ∞ and m ≥ 1, the joint distribution of z l,t j j∈[m]converges in distribution to a multivariate normal distribution with mean zero and covariance A K l ∞ ⊗ I m .The proofs of Theorems E.4, E.5, and E.6 can be found in §E.7, §E.6, and §E.5 respectively. The proof of Theorem E.5 is slightly more technical and we will borrow some ideas fromDaniely et al. (2016).E.5 PROOF OF THEOREM E.6Per Equation26, it suffices to prove that for any vector [ai (x)] i∈[m], x∈X ∈ R m|X | , l (t) j∈[n l (t)] β∈[±k]√ v β ω l ij,β y l,t j,α+β (x) + b l i that conditioned on y l,t j j∈[n l (t)]

tr denotes the mean trace operator acting on the pixel by pixel matrix, i.e. the functional computing the mean of the diagonal terms of the pixel by pixel matrix. Therefore[f i ] i∈[m]converges weakly to a mean zero Gaussian with covariance σ 2 ω tr K l ∞ (x, x ) + σ 2 b x,x ∈X ⊗ I m in the case of vectorization ( §3.1).E.6 PROOF OF THEOREM E.5Proof. We recall K L t and K L ∞ to be random matrices in R d|X |×d|X | , and we will prove convergence K to• ∞ , the pointwise ∞ -norm(i.e. K ∞ = max x,x ,α,α |K α,α (x, x )|).Note that due to finite dimensionality of K L , convergence w.r.t. all other norms follows.

We postpone the proof of the following claim.Claim E.9. Assume |φ(x)| ≤ a+b|x|. Then there is a κ = κ(a, b, R) such that for all Σ ∈ PSD 2 (R) and all p ≥ 1,E (x,y)∼N (0,Σ) |φ(x)φ(y)| p 1/p ≤ κp. (102)Claim E.9 and the triangle inequality imply E (x,y)∼N (0,Σ) |φ(x)φ(y) -Eφ(x)φ(y)| p 1/p ≤ 2κp.

y)∼N (0,Σ) |φ(x)φ(y)| p 1/p ≤ E x∼N (0,Σ11) |φ(x)| 2p 1/2p E y∼N (0,Σ22) |φ(y)| 2p 1/2p (105) ≤ a + b E|x| 2p 1/2p a + b E|y| 2p 1

Presented is a CNN with stride 1 and no (valid) padding, i.e. the spatial shape of the input shrinks as it propagates through it d

Table content: <table><tr><td>2 hypercube</td></tr><tr><td>(see  §2.2,  §3.1,  §C).</td></tr></table>

See Figure7for similar results with other architectures and §A.7.2 for experimental details.random finite width networks and use the empirical uncentered covariances of activations to estimate the Monte Carlo-GP (MC-GP) kernel,

Table content: <table><tr><td>(CNN w/o weight sharing)</td></tr><tr><td>0 p 9 z 6 F 6 y i w S Y = &lt; / l a t e x i t &gt;</td></tr><tr><td>(error functio</td></tr><tr><td>(Monte Carlo estimate)</td></tr><tr><td>(Fully-connected network)</td></tr><tr><td>Under review as a conference paper at ICLR 2019</td></tr><tr><td>MC-CNN-GP</td></tr><tr><td>Figure 2: Validation accuracy (left) of an MC-CNN-GP increases with n ⇥ M (i.e. channel count times number of samples) and approaches that of the exact CNN-GP (not shown), while the distance (right) to the exact kernel decreases. The dark band in the left plot corresponds to ill-conditioning of K L+1 n,M when the number of outer products contributing to K L+1 n,M approximately equals its rank. Values reported are for a 3-layer model applied to a 2K/4K train/validation subset of CIFAR10 downsampled to 8 ⇥ 8.</td></tr></table>



Table content: <table><tr><td>CNN</td></tr><tr><td>Figure 3: (a): SGD-trai Each line corresponds to best learning rate and we (b): SGD-trained CNN with increasing number and weight sharing, as w and batch size, which ar performance (y-axis) of a racy increases and approa the best-performing SG point corresponds to the tion) CNN with the same on the full training CIFAR nels. While CNN-GP ap best CNNs overall outpe further analysis of factors and infinite Bayesian CN 100% training accuracy o set downsampled to 8 ⇥ details of (a, c) and (b) pl</td></tr><tr><td>4• CNNs suffer from exploding or va</td></tr><tr><td>gradients at large depths unless t</td></tr><tr><td>initialized correctly. The phase bo</td></tr><tr><td>order and chaos (rightmost plot f</td></tr><tr><td>nonlinearity) is the region where l</td></tr><tr><td>training is feasible.</td></tr></table>



Table content: <table><tr><td>(b)</td><td>N</td></tr><tr><td>LCN</td><td/></tr><tr><td>CNN</td><td/></tr><tr><td colspan="2">Figure 3: (a): SGD-trai Each line corresponds to best learning rate and we (b): SGD-trained CNN with increasing number and weight sharing, as w and batch size, which ar performance (y-axis) of a racy increases and approa the best-performing SG point corresponds to the tion) CNN with the same on the full training CIFAR nels. While CNN-GP ap best CNNs overall outpe further analysis of factors and infinite Bayesian CN 100% training accuracy o set downsampled to 8 ⇥ details of (a, c) and (b) pl</td></tr><tr><td>4. Computing the NN-GP covariance</td><td/></tr><tr><td colspan="2">8. Large-dep</td></tr><tr><td colspan="2">• CNNs suffer from exploding or va</td></tr><tr><td colspan="2">gradients at large depths unless t</td></tr><tr><td colspan="2">initialized correctly. The phase bo</td></tr><tr><td colspan="2">order and chaos (rightmost plot f</td></tr><tr><td colspan="2">nonlinearity) is the region where la</td></tr><tr><td>training is feasible.</td><td/></tr></table>

). For further analysis of factors leading to similar or diverging behavior between SGD-trained finite CNNs and infinite Bayesian CNNs see Figures1, 7, and Table1. Experimental details: all networks have reached 100% training accuracy on CIFAR10. Values in (b) are reported on an 0.5K/4K train/validation subset downsampled to 8 × 8 for computational reasons. See §G.5 and §G.1 for full experimental details of (a, c) and (b) plots respectively.

Table content: <table/>

Table1for further results on other datasets, as well as comparison to GP performance in prior literature. See §G.5 for experimental details. Best achieved test error for different model classes (vertical axis) and datasets (horizontal axis). Test error (%) for the best model within each model family, maximizing validation accuracy over depth, width, and training and initialization hyperpameters. Except where indicated by parentheses, all models achieve 100% training accuracy. For SGD-trained CNNs, numbers in parentheses correspond to the same model family, but without restriction on training accuracy. CNN-GP achieves state of the art results on CIFAR10 for GPs without trainable kernels and outperforms SGD-trained models optimized with a small learning rate to 100% train accuracy. See Figure7for visualization and interpretation of these results. See §G.5 for experimental details.

Table content: <table><tr><td>Model</td><td>CIFAR10</td><td colspan="2">MNIST Fashion-MNIST</td></tr><tr><td colspan="3">CNN with pooling CNN with ReLU and large learning rate CNN-GP CNN with small learning rate CNN with erf (any learning rate) Convolutional GP (van der Wilk et al., 2017) ResNet GP (Garriga-Alonso et al., 2018) Residual CNN-GP (Garriga-Alonso et al., 2018) -14.85 (15.65) -24.76 (17.64) -32.86 0.88 33.31(22.89) -33.31(22.17) -35.40 1.17 -0.84 0.96 CNN-GP (Garriga-Alonso et al., 2018) -1.03 FCN-GP 41.06 1.22 FCN-GP (Lee et al., 2018) 44.34 1.21 FCN 45.52 (44.73) -</td><td>--7.40 ------8.22 --</td></tr></table>

|X | × |X | matrix formed by evaluating the GP covariance on the training inputs, and likewise K (x

Table content: <table/>

Peer review (iclr2019):
 ****Reply to authors* rebuttal**** Dear Authors, I greatly appreciate the effort you have put into the rebuttal. The changes you have made have addressed most of my concerns and I believe that the few outstanding ones can be fixed without significantly affecting the main message of the paper. I will thus be recommending acceptance of the paper. Best wishes, Rev 3 Several remarks on the updated version: - (p.20, A.5.1) To ensure the random variables are well-defined, please state explicitly which sigma algebra is F (I am assuming the product Borel sigma-algebra + the relevant definitions of the random variables). This is important for the reader to understand what convergence in distribution on this particular space does and does not imply. Some readers might also appreciate if you used the mentioned "infinite width, finite fan-out, networks" (Matthews et al.) construction (or similar) which would ensure that the collection of random variables {z_i^l}_{i .in N*} is well-defined for any network width and l, which currently does not seem to be the case according to Eqs. (28-29). If the full countably infinite vectors of random variables are not defined for all networks in the sequence, it is not possible to prove their convergence in distribution to the relevant GPs. - (p.21, A.5.3) Thank you for clarifying the definition of elements of the sequential limit. If possible, I would further recommend first fixing the probability space and then defining the random variables (the argument just before Theorem A.2 seems somewhat circular as R.V.s should first be defined on some space, and not put on a probability space post-hoc; perhaps some product space with the product sigma-algebra would work here?!). Furthermore, if I understand correctly, there are now L sequences of neural networks (one sequence for networks with 0, ..., L-1 "infinite layers"), rather than a single sequence, and the "infinite layers" are squashed into a single "infinite layer" which is represented by z_i^.infty? In other words, all the infinite layers are replaced by iid samples from a particular GP and only the finite layers have the standard neural network structure? If I am mistaken (or not), perhaps a further explanatory footnote would help the reader. - (p.21, A.5.3 & p.23, A.5.4) Thank you for improving the discussion of joint convergence. Please clarify that proving convergence for any finite m is sufficient for proving convergence in distribution of the countably infinite vector {z_i}_{i .in N*} for the **product Borel sigma-algebra** (e.g. using an argument like the one on p.19 of Billingsley (1999)). - (p.21) "Uniformly square-integrable": to me, this phrase suggests that the collection of squares of the functions has to be uniformly integrable but the definition in Eq. (27) only states one of the conditions in definition of uniform integrability. Please clarify that "uniform square-integrability" here is not related to the standard notion of "uniform integrability" in the literature. ****Summary**** This paper extends recent results on convergence of Bayesian fully connected networks (FCNs) to Gaussian processes (GPs), to the equivalent relationship between convolutional neural networks (CNNs) and GPs. This is currently an area of high interest, with Xiao et al. (2018) examining the same relationship from a mean-field perspective, and two other concurrent papers making contributions: https://arxiv.org/abs/1808.05587 https://arxiv.org/abs/1810.10798 Thus the scope of the paper fits well within the aims of the conference. I really appreciate that the authors did not shy away from studying the effect of pooling layers, and find the connection to locally connected networks they describe intriguing and insightful. On the experimental side, the investigation of the relative importance of compositionality, equivariance and invariance on performance of CNNs is very interesting. These experiments and investigations are however based on a theoretical foundation which suffers from several issues. The main problems are an incorrect proof of convergence of the joint distribution of filters, and an improper use of convergence in probability in cases where random variables do not share a common underlying probability space. Unfortunately, either of these by itself invalidates the main theoretical claims which is why I am recommending rejection of the paper. However, I believe that the argument in (A.4.3) can potentially be rectified, and, as I detail below, is of greater interest to the community relative to the ones in (A.4.1) and (A.4.2). If this is accomplished and the proofs in (A.4.1) and (A.4.2) are either also fixed or left out (A.4.3 is sufficient to justify the claims in the main body), I am willing to significantly improve my rating of this paper and potentially recommend acceptance. For this reason, a "detailed comments" section is appended at the end of the standard review where the technical issues are described in much greater detail. ****General comments**** **Bayesian vs. infinite neural networks** The main theoretical claims concerning the relationship between Bayesian CNNs and GPs are within Section 2. Therein on top of page 4, the authors say "In Appendix A.4 we give several **alternative** derivations of the correspondence" (emphasis mine), and then progress to outline the skeleton of the argument (A.4.2) in Sections 2.2.1-2.2.3. Section 2.2.3 is concluded by statement of the main theoretical result of this paper, Eq. (10), which comes from (A.4.3) and can only be linked to the rest of Section 2 through the claim of equivalence between the "alternative derivations" (A.4.1), (A.4.2) and (A.4.3). The problem is that the equivalence claim does not hold, as explained below: The most important distinction here is between what I will call a "sequential" and a "simultaneous" limit. In the "sequential" case (A.4.1 & A.4.2, Sections 2.2.1-2.2.3), layers are taken to infinity one by one, whereas in the "simultaneous" case (A.4.3, used to obtain the result concluding Section 2.2.3) all layers are **finite** for **all** members of the sequence, growing in width simultaneously. The "simultaneous" limit (A.4.3) is in my view more interesting as it tells us that **finite** BNNs do indeed converge to GPs in distribution, i.e. that for each expectation of a continuous bounded function of the outputs of the limiting GP, there exists a BNN with a **finite** number of neurons in **each** layer for which the expectation of the same function is arbitrarily close. From a practical perspective, "simultaneous" limit tells us that inference algorithms for BNNs (which can be inaccurate and/or computationally expensive) can sometimes be replaced by exact or approximate inference algorithms for the limiting GP (cf. Section 5 in (Matthews et al., 2018, extended version)). The "sequential" limit (A.4.1 & A.4.2) on the other hand does not establish existence of finite BNNs arbitrarily close to a particular GP, or justify use of the GP limit as approximation for finite BNNs as above. This is because the width of individual layers goes to infinity in a sequence from first to last. This means that most of the networks that constitute the sequence converging to the GP will have **one or more infinitely wide layers** and thus do not correspond to the finite BNNs we usually work with. In other words, "sequential" limit can only ever establish that there exists a network with **all but the final hidden layer infinite** that is arbitrarily close to the limiting GP. The only case where "sequential" and "simultaneous" limits agree is thus in the single hidden layer case first studied by Neal (1996). I will call the networks with one or more infinite layers "infinite networks", inspired by the work of Williams (1997) and others. Notice that infinite networks cannot be described by Eqs. (1) and (2) as the weights would be zero with probability one and thus output of the network would only depend on biases. It is not immediately obvious how to formally replace Eqs. (1) and (2) in the case of infinite networks which is one of the technical issues with the approaches in (A.4.1) and (A.4.2) (see the detailed comments section for further discussion). Others may of course disagree and find "sequential" limits more interesting, but if the authors wish to keep the description of (A.4.2) in the main paper (Sections 2.2.1-2.2.3), it would be highly beneficial if readers were given the opportunity to understand the differences between the two types of limits so that they can form their own judgement. The authors should then also make clearer that the approach described in Sections 2.2.1-2.2.3 cannot be used to obtain the final result, Eq. (10). I would rather recommend reworking Sections 2.2.1-2.2.3 based on the "simultaneous" limit argument in (A.4.3) which unlike the current one can justify the result in Eq. (10) stated at the end. **Other comments** - (p.2, top) You say your results are "strengthening and extending the result of Matthews et al. (2018)" which is somewhat confusing. Matthews et al. prove a result for FCNs whereas this paper focuses on CNNs. Extension of (A.4.3) to FCNs may well be possible but is not included in this paper. Results in (A.4.1) and (A.4.2) are for the "sequential" whereas Matthews et al. study the "simultaneous" limit. Further differences: - Matthews et al. prove convergence for any countable rather than only finite input sets. - In Matthews et al.*s work, Gaussianity is obtained through use of a particular version of CLT, whereas this work exploits Gaussianity of the prior over weights and biases. Going forward, an extension to more general priors/initialisations (like uniform or any sub-Gaussian) is likely to be easier using the CLT approach. - Matthews et al.*s assumption on the activation functions is independent of the input set (p.7, Definition 1), whereas this work uses an assumption that is explicitly dependent on input (Eq. (37)) which might be potentially difficult to check. - (p.15, A.2 end) Should also mention Titsias (2009), "Variational Learning of Inducing Variables in Sparse Gaussian Processes", as a classical reference for approximate GP inference. ****Questions**** - (Section 4) Can you please provide more details on the MC approximation? Specifically, is only the last kernel approximated, or rather all of them, sequentially resampling from the Gaussian with empirical covariance in each layer? In case you tried, is there any qualitative or quantitative difference between the two approaches? - (Section 4 and Appendix A) Daniely et al. (2016) assume that the inputs to the neural network are l^2 normalised. You mention that the inputs have been normalised in the experiments (A.6). Is this assumption used in any of your proofs? Have you observed that l^2 normalisation improves empirical performance? - (p.8, Figure 6) How was "the best CNN with the same parameters" selected? If training error is zero for all, was it selected by validation accuracy? I was assuming that what is plotted is an estimate of the **expected** generalisation error, whereas the above selection procedure would be estimating supremum of the support of the generalisation error estimator which does not seem like a fair comparison. Can you please clarify? - (p.8 and A.6) Why only neural networks with zero training loss were allowed as benchmarks? How did the ones with non-zero training error fared in comparison? Can you please expand on footnote 3? - (p.8, last sentence) "an observation specific to CNNs and FCNs or LCNs": Matthews et al. (2018, extended version) observed in Section 5.2 that BNNs and their corresponding GP limits do not always perform the same even in the FCN case (cf. their Figure 8). Their paper unfortunately does not compare to equivalent FCNs trained by SGD. Have you experimented with or have an intuition for whether the cases where SGD trained models prevail coincide with the cases where BNNs+MCMC posterior inference outperform their GP limit? - (p.15, Table 3) The description says you were using erf activation (instead of the more standard ReLU): why? Have you observed any significant differences? Further, how big a proportion of the values in the image is black due to the numerical issues mentioned in A.6.4? - (p.18, just after Eq. 39) Use of PSD_{|X|d} in (A.4.3) suggests this proof assumes "same" padding is used?! Does the proof generalise to any padding/changing dimensions of filters inside the network? - (A.6) Can you comment on the pros & cons of "label regression" for classification and how does it compare with approximate inference when softmax is put on top of a GP (perhaps illustrating by a simple experiment on a toy dataset)? [end of standard review] [detailed comments] ****Technical concerns**** Notation-wise, I would strongly encourage incorporating the dependence on network width into your notation, at the very least throughout the appendix. It would greatly reduce the amount of mental book-keeping the reader currently has to do, and significantly increase clarity at several places. One of my main concerns is that the random variables and their underlying probability space are never formally set-up. This is problematic because convergence in probability is only defined for random variables sharing the same underlying space. At the moment, networks with different widths are not set-up to share a probability space. The practical implication for the approaches relying on convergence in probability of the empirical covariance matrices K is that the convergence in probability is not well-defined exactly because the empirical covariance matrices are not set-up on the same underlying probability space. A possible way to address this issue is to use an approach akin to what Matthews et al. (2018, extended version) call "infinite width, finite fan-out, networks" on page 20. This puts the networks on the same underlying space and because the empirical covariance matrices are measurable functions of thus defined random variables, they will also share the same underlying probability space. Also regarding convergence in probability, please state explicitly with respect to which metric is the convergence considered when first mentioned (A.4.3 is explicitly using l^.infty; A.4.2 perhaps l^2 or l^.infty?), and make any necessary changes (e.g. show continuity of the mapping C in A.4.2). At several places within the paper, you state that the law of large numbers (LLN) or the central limit theorem (CLT) can be applied. Apart from other concerns detailed later, these come with conditions on finiteness of certain expectations (usually the first one or two moments of the relevant random variables). Please provide proofs that these expectations are indeed finite and make any assumptions that you need explicit in the main text. Another major concern is that none of (A.4.1), (A.4.2) and (A.4.3) successfully proves joint convergence of the filters at the top layer as claimed in the main text (e.g. Eq. (10)), and instead only focuses on marginal convergence of each filter which is not sufficient (cf. the comment on joint vs. pairwise Gaussianity below). This is perhaps sufficient if a single filter is the output of the network, but insufficient otherwise, especially when proving convergence with additional layers added on top of the last convolutional layer (as in Section 3) whenever the number filters is taken to infinity. It would be nice, but not necessary for acceptance of the paper, to extend the proofs to uncountable index sets. I think you could use the same argument as described towards the end of Section 2.2 in (Matthews et al., 2018, extended version) and references therein. **Other comments** - I would strongly encourage distinguishing more clearly between probability distributions and density functions. For example, I would infer that lower case p refers to the probability distribution from Eq. (6); however, in Eqs. (8) and (9) the same notation is used for density functions (whilst integrating against the Lebesgue measure). This is quite confusing in this context as the two objects are not the same (see next two comments). I would suggest using capital P when referring to distribution, and lower case p when referring to its density. - (p.4, Eq. 6) If p is a density, it cannot be equal to a delta distribution. If it is a probability distribution then I am similarly confused - convergence in probability is a statement about behaviour of random variables, not probability distributions; in that case possibly Eq. (6) is trying to say that the empirical distribution of K^l (which is a random variable) conditioned on K^{l-1} converges weakly to the delta distribution on the RHS in probability? Please clarify. - (p.5, Eq. 10) I would recommend stating explicitly the mode of convergence. If p is the density then even assuming A.4.3 can be fixed to prove weak convergence of the **joint** distribution of filters is not enough not justify Eq. (10) - convergence in distribution does not imply pointwise convergence of the density function. If p is the distribution, then I would possibly use the more standard notation *.otimes* instead of *.prod*. - (p.17, end of A.4.2) You say "Note that addition of various layers on top (as discussed in Section 3) does not change the proof in a qualitative way". Can you please provide the formal details? At the very least, joint convergence of filters will have to be established if fully connected layers are added on top. This is the main reason why joint convergence of filters in the top layer is important. ****Specific comments & issues for individual proofs**** **Approaches suited infinite networks ("sequential limit")** As mentioned in the beginning, it is not entirely clear how to formalise infinite networks in a way analogous to Eqs. (1) and (2) in your paper. This is important because you are ultimately proving statements about random variables, like convergence in probability, and this is not possible if those random variables are not formally defined. This section only comments on technical issues with the approaches described in (A.4.1) and (A.4.2). From now on, I assume that the authors* were able to formally define all the mentioned random variables in a way that fits with (A.4.1) and (A.4.2). (i) Hazan and Jaakola type approach (A.4.1) This approach essentially iteratively applies a version of the recursion first studied by Hazan and Jaakola (2015), "Steps Toward Deep Kernel Methods from Infinite Neural Networks". - (p.16, A.4.1) Please provide reference for the claim that "pairwise independent Gaussian implies joint independent Gaussian". This seems to assume that the variables are jointly Gaussian which is, as far as I can see, not established here. - see second part of the linked answer for a nice example of three random variables with pairwise standard normal marginals, but joint not the multivariate standard normal: https://stats.stackexchange.com/questions/180708/x-i-x-j-independent-when-i%E2%89%A0j-but-x-1-x-2-x-3-dependent/180727#180727 - (p.16, A.4.1) The application of the multivariate CLT is slightly more complicated than the text suggests. Except for the necessity of proving finiteness of the relevant moments, multivariate CLT does not out-of-the-box apply to infinite dimensional random variables like {z_j^{l+1}}_{1 .leq j .leq .infty} as claimed. Hence joint convergence is not proved which will be problematic for the reasons explained earlier. (ii) Lee et al. type approach (A.4.2) This type of approach follows the technique used by Lee et al. (2018), "Deep Neural Networks as Gaussian Processes". Application of the weak law of large numbers (wLLN): As mentioned before, convergence in probability is only possible between random variables on the same underlying space. This is usually not a problem when wLLN is applied as the random variables converge to a constant random variable. Because every constant random variable generates the trivial sigma-algebra, it is measurable for any underlying probability space and thus convergence in probability is well-defined. The situation here is more complicated because the target is constant only conditionally on the previous layer, i.e. is not constant. As a side note, even the conditioning is only well-defined if all random variables live on the same space (conditioning on a random variable is technically conditioning on the sub-sigma-algebra it generates on the shared space). Assuming the problem with all K^{l, t} (t denotes the dependence on network width), for all l .in {1, ... L} and t .in {1, 2, 3, ...}, being on the same underlying probability space is solved, the next point is application of the wLLN itself. You claim "we can apply the law of large numbers and conclude that [Eq. (6)]" (p.4) which is not entirely correct here. Focusing on the application when the sizes of all the previous layers are held fixed, the two conditions that have to be checked here are: (i) the conditional expectation of the iid summands in Eq. (3) is finite; (ii) the sequence of iid variables is fixed. Please provide an explicit proof of (i). Regarding (ii), I am specifically concerned with the fact that with changing t (and thus network widths), the sequence of random variables changes (because the previous K^{l-1,t} matrix changes) which means that completely different size of the current layer may be necessary to get sufficiently close to the target (which has itself changed with t). In other words, instead of having a fixed infinite sequence of iid random variables, you currently have a sequence of growing finite sets of random variables which are iid only within the finite sets, but not between members of the sequence (different t). The direct implication is that this type of proof is not applicable to the "simultaneous limit" case as claimed in the main text (Section 2.2 says all proofs are equivalent and lead to Eq. (10) which explicitly takes the simultaneous limit), since the application would require some form of uniform convergence in probability akin to (A.4.3). I think that the approach taken in (A.4.3) is a correct way to address this issue and would thus recommend focusing on (A.4.3) and leaving (A.4.2) out. The appendix seems to acknowledge that (A.4.2) does not work for the "simultaneous limit" - please adapt the main text accordingly. A note on convergence in probability: In Eq. (3), the focus is on convergence in probability of individual entries of the K matrices. This in general does not imply convergence of all entries jointly. However, the type of convergence studied here is convergence to a constant random variable which is fortunate because simultaneous convergence of all entries in probability can be obtained for free in this case (thanks to having a **finite** number of entries of K). I think it might be potentially beneficial for the reader if this was explicitly stated as a footnote with an appropriate reference included. A note on marginal vs joint probability: As you say above Eq. (23), you are only proving convergence of a single filter marginally, instead of the full sequence {z_j^L}_{1 .leq j .leq .infty} jointly. Convergence of the marginals does not imply convergence of the joint, which will be problematic for the reasons explained earlier. **Approaches for BNNs ("simultaneous limit")** (iii) The proof in (A.4.3) My biggest concern about this approach is that it only establishes convergence of a single filter marginally, instead of the full sequence {z_j^L}_{1 .leq j .leq .infty} jointly. Convergence of the marginals does not imply convergence of the joint, which will be problematic for the reasons explained earlier. Other comments: - (p.17) You say "Using Theorem A.1 and the arguments in the above section, it is not difficult to see that a sufficient condition is that the empirical covariance converges in probability to the analytic covariance". - Can you please provide more detail as it is unclear what exactly do you have in mind? - I will be assuming from now on that you show that a particular combination of the Portmanteau theorem and convergence of K^L in probability to get pointwise convergence of the characteristic function is sufficient. - (p.18) Condition on activation function: The class .Omega(R) is dependent on the considered input set X through the constant R. This seems slightly cumbersome as it would be desirable to know whether a particular activation function can be used without any reference to the data. It would be nice (but not necessary) if you can derive a condition on .phi which would not rely on the constant R but allows ReLU. - (p.19, Eq. 48) I see where Eq. (48) is coming from, i.e. from Eq. (44) and the assumption of .bar{.varepsilon} ball around A(K_.infty^l) being in PSD(R), but it would be nicer if you could be a bit more verbose here and also write out the bound explicitly (caveat: I did not check if the definition of .bar{.varepsilon} matches up but assume a potential modification would not affect the proof in a significant way). - (p.19) The second part of the proof is a little confusing, especially after Eq. (49) - please be more verbose here. For example, just after Eq. (49), it is said that because the two random variables have the same distribution, property (3) of .Omega(R)*s definition can be applied. However the two random variables are not identical and importantly are not constructed on the same underlying probability space. Property (3) is a statement about the the set of random variables {T_n (Sigma)}_{Sigma .in PSD_2(R)} and not about the different 2x2 submatrices of K^{l+1}, but it needs to be applied to the latter. When this is clarified, the next point that could be made clearer is in the following sentence where changing t will affect the 2x2 submatrices of K^{l+1,t} as well as the bound through U(t) and V(t); it is not immediately obvious that the proof goes through as claimed so please be a bit more verbose. ****Typos and other minor remarks**** - (p.2, top) "hidden layers go to infinity uniformly": The use of word uniformly is non-standard in this context. Please clarify. - (p.3, Eq. 2) Using x for both inputs and post-activations is slightly confusing. - (p.4, Eq. 5) Should v_.beta multiply .sigma_.beta^2 ? - (p. 4) The summands in Equation (3) are iid -> "conditionally iid" (please also specify the conditioning variables/sigma-algebra). - (p.4, Eq. 4) Eq. (4) is slightly confusing given you mention that K is a 4D object on the previous page. - I only understood K is "flattened" into |X|d x |X|d matrix when I reached (A.4.3) - this should be stated in main text as otherwise the above confusion arises. - (p.5, 3 and 3.1) The introduction of "curly" K is slighlty confusing. Please provide more detail when introducing the notation, e.g. state in what space the object lives. - (p.5, before Eq. (11)) Is R^{n^(l+1)} the right space for vec(z^L) ? It seems that the meaning of z changes here as compared to the definition in Eq. (2). If z is still defined as in Eq. (2), how exactly is the vec operator defined here? Please clarify. - (p.16, A.4.2) "law of large number" -> "weak law of large numbers" - (p.17) T_n is technically not a function from PSD_2 only but also from some underlying probability space into a measurable space (i.e. can be viewed as a random variable from the product space of PSD_2 and some other measurable space). - (p.18, Eq. 38) Missing dot at the end. Also the K matrix either should or shouldn*t have the superscript "l" (now mixed); it does have the superscript in Eq. (39) so probably "should". - (p.18, Eq. 39) Slightly confusing notation. Please clarify that both K and A(K) should have diagonal within the given range. - (p.18) "squared integrable" -> "square integrable" or "square-integrable" - (p.18) Last display before Eq. (43): second inequality can be replaced by equality?! - (p.19, Eq. 47) The absolute value should be sup norm. - (p.19, Eq. 49) LHS is a scalar, RHS a 2x2 matrix (typo). - (p.19, last sentence of the proof) It does not seem the inequalities need to be strict.