Globally Gated Deep Linear Networks
Authors: Qianyi Li
Haim Sompolinsky
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Network Architectures
Kernel Methods
Deep Learning Theory
Gaussian Processes
Finite Width Networks
Gated Linear Networks

Abstract: Recently proposed Gated Linear Networks (GLNs) present a tractable nonlinear network architecture, and exhibit interesting capabilities such as learning with local error signals and reduced forgetting in sequential learning. In this work, we introduce a novel gating architecture, named Globally Gated Deep Linear Networks (GGDLNs) where gating units are shared among all processing units in each layer, thereby decoupling the architectures of the nonlinear but unlearned gating and the learned linear processing motifs. We derive exact equations for the generalization properties of Bayesian Learning in these networks in the finite-width thermodynamic limit, defined by $N, P\rightarrow\infty$ while $P/N=O(1)$ where $N$ and $P$ are the hidden layers' width and size of training data sets respectfully. We find that the statistics of the network predictor can be expressed in terms of kernels that undergo shape renormalization through a data-dependent order-parameter matrix compared to the infinite-width Gaussian Process (GP) kernels. Our theory accurately captures the behavior of finite width GGDLNs trained with gradient descent (GD) dynamics. We show that kernel shape renormalization gives rise to rich generalization properties w.r.t. network width, depth, and $L_2$ regularization amplitude. Interestingly, networks with a large number of gating units behave similarly to standard ReLU architectures. Although gating units in the model do not participate in supervised learning, we show the utility of unsupervised learning of the gating parameters. Additionally, our theory allows the evaluation of the network capacity for learning multiple tasks by incorporating task-relevant information into the gating units. In summary, our work is the first exact theoretical solution of learning in a family of nonlinear networks with finite width. The rich and diverse behavior of the GGDLNs suggests that they are helpful analytically tractable models of learning single and multiple tasks, in finite-width nonlinear deep networks. 

Globally Gated Deep Linear Networks
Qianyi Li' Haim Sompolinsky
Biophysics Graduate Program, Harvard University "Center for Brain Science, Harvard University
3 Edmond and Lily Safra Center for Brain Sciences, Hebrew University
qianyi_ _iig. harvard. edu, hsompolinsky@mch harvard. edu, haim@fiz. huj.ffiiiuuu.... www.whww............... ac.
Abstract
Recently proposed Gated Linear Networks (GLNs) present a tractable nonlinear network architecture, and exhibit interesting capabilities such as learning with local error signals and reduced forgetting in sequential learning. In this work, we introduce a novel gating architecture, named Globally Gated Deep Linear Networks (GGDLNs) where gating units are shared among all processing units in each layer, thereby decoupling the architectures of the nonlinear but unlearned gating and the learned linear processing motifs. We derive exact equations for the generalization properties of Bayesian Learning in these networks the finite-width thermodynamic limit, defined by N, P are the hidden layers' width and size of training data sets respectfully. We find that the statistics of the network predictor can be expressed in terms of kernels that undergo shape renormalization through data-dependent order parameter matrix compared to the infinitewiidth Gaussian Process (GP) kernels. Our theory accurately captures the behavior of finite width GGDLNs trained with gradient descent (GD) dynamics. We show that kernel shape renormalization gives rise rich generalization properties w.r.t. network width, depth and L2 regularization amplitude. Interestingly, networks with a large number of gating units behave similarly to standard ReLU architectures. Although gating units in the model do not participate in supervised learning, we show the utility of unsupervised learning of the gating parameters. Additionally, our theory allows the evaluation the network's 's ability for learning multiple tasks by incorporating task-relevant information into the gating units. In summary, our work is the first exact theoretical solution of learning in a family of nonlinear networks with finite width. The rich and diverse behavior of the GGDLNs suggests that they helpful analytically tractable models of learning single and multiple tasks, in finite-widhh nonlinear deep networks.
oo while P/N ((1) where N and
Introduction
Despite the recent advances in machine learning, theoretical understanding of how machine learning algorithms work is very limited. Many current theoretical approaches study infinitely wide networks 2 where the input-output relation is equivalent to a Gaussian Process (GP) function space with a covariance matrix defined by a GP kernel. However, this GP limit holds when the network width approaches infinity while the size of the training data remains finite, severely limiting its applicability realistic conditions. Another line of work focuses on finite-width deep linear neural networks (DLNNs) 4 5 M while applicable in wider regime, the generalization behavior of linear networks are very limited, and the bias contribution always remains constant with network parameters A. which fails to capture the behavior of generalization performance in general nonlinear networks. Therefore, a tractable nonlinear network architecture diverse generalization behavior of general nonlinear networks.
need for theoretically probing into
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Recently proposed Gated Linear Networks (GLNs) present tractable nonlinear network architecture with capabilities such as learning with local error signals and mitigating catastrophic forgetting in sequential learning. Inspired by these recent advances in GL.Ns, we propose Globally Gated Deep Linear Networks (GGDL.Ns) as simplified GI.N structure that preserves the nonlinear property of general GLNss the decoupling of fixed nonlinear gating from learned linear processing units, and the ability to separate the processing of multiple tasks using the gating units. Our GGDLN structure is different from previous GLNs in several ways. First, the gating units are shared across hidden layer units and different layers while in previous work each unit has its own set of gatings [O & 9 Second, we define global learning objective instead of local errors [ 9l. These simplifications allow us to obtain direct analytical expressions of memory capacity and exact generalization error of these networks for arbitrary training and testing data, providing quantitative insight into the effect of learning in nonlinear networks, as opposed to studies of generalization bounds (TO), expressivity estimates [1 and indirect quantities relevant to generalization such as the implicit bias of the network [02]. Furthermore, the kernel expression of the predictor statistics we propose in this work also allow to make qualitative explanations of the generalization and how it's related to data structure and network representation for single and multiple tasks.
First, we introduce the architecture of our GGDLNs and analyze its memory capacity. We then derive our theory for generalization properties of GGDLNs, and make qualitative connections between the generalization behavior and the relation between the renormalization matrix and task structure. Second, we apply our theory to GGDLNs performing multiple tasks, focusing on two scenarios where tasks are either defined by different input statistics different output labels on the same inputs. While the effect of kernel renormalization is different in the two cases, we find that for fixed gating functions, de-correlation between tasks always improves generalization.
2 Globally gated deep linear networks
0.4
0.04
0.09
0.6
0.0
0.4
0.4
0.
0.01
10
10 102
Figure I: Globally gated deep linear networks. (a) Structure of GGDLNs, each neuron in the hidden layer has M dendrites, each with a different input-dependent gating Im (x(x) which is fixed during training, the M gatings are shared across neurons in the hidden layer. The m-th dendritic branch of the i-th neuron in layer connects to neuron j in the previous layer with weight WiT,, (Shown in orange). (b) Training error of networks with 1-3 hidden layers in the GP limit as a function of M evaluated noisy ReLU teacher task, training error goes zero network capacity (black dashed lines). (c-e) Bias, variance and generalization eгтor of the same network and task as (b). Bias and generalization error diverges, variance generalization becomes nonzero at network capacity (black dashed line). See Appendix for detailed parameters.
In GGDLNS, the network input-output relation is defined as follows,
N
M
No VNOM Ej=1 ΣM (x)x1-11 1> 1
f(x)
Om L.,i (x), 21,1
NM
No /No 2 W1,501115
1=1
(L)
where Xg = is the input, N is the hidden layer width, M is the number of gating units in each layer, and No is the input dimension. Each neuron in every layer has M dendrites, each with an input-dependent global gating 9m. (x) shared across all neurons. The m-th dendritic branch of neuron i in the L-th hidden layer connects to neurons in the previous layer with cendrite-speciff weight vector WT, (or with readout weight vector am for the output neuron), as shown in Fig I (a). Note that although the gatings are fixed during learning, changes in the weights affect how these gatings act on the hidden layer activations, and it is interesting to understand how the learned task interacts with these gating operations. Since adding gatings at the input layer is equivalent to expanding the
2

---

input dimension and replacing z; by 2: (x), and learning does not affect how the gatings interact with the input, we do not add gatings at the input layer for simplicity.
Memory Capacity: Memory capacity refers to the maximum numher of random (or generic) input- output examples for which there exists set of parameters such that the network achieves zero training error (here we consider the mean squared error, MSE). By definition, it is irrespective of the learning algorithm. The capacity bounds of deep nonlinear networks has been extensively studied in many recent works [0, 3 E. To calculate the capacity of GGDLNs, note that the input-output relation given by Eqli can be alternatively expressed as fxx) = Em.... m..J Weff
m1. ".1.,,,,,.... .m mi." ma 9m, (x)gm, ....... 9mz (x)xz
which is a linear combination of the effective input I "m.,-......
(my 1,... Mij 1, 1..... .No,, with some effective weights W which is complicated function of a and W. Here my is the index of the gatings in the /-th layer. As the gating units are shared across layers, the effective input xf has No = independent dimensions. This combinatorial term represents the number of possible combinations of L gatings selected from M total number of gatings. Assuming N > M such that the effective weight Woeff mi..". ma.. can take any desired value in the N, dimensional whole space, the problem of finding Welf' with zero training error is equivalent to a linear regression problem with input xff and the target outputs. Therefore, the capacity is equivalent to the number of independent input dimensions, given by P < No No(*+//-1) The above capacity is verified by Fig lb), where the training error becomes nonzero above the memory capacity. The generalization behavior also changes drastically at network capacity (Fig [f c)), where generalization error and its bias contribution diverge, and the variance contribution shrinks to 0 (see detailed calculation in the next paragraph and Appendix A.3. This double descent property of the generalization error is similar to previously studied in linear and nonlinear networks. Furthermore, although the output of the network is a linear function of the effective input xef due to the multiplicative nature of the network weights and the gatings, learning in GGDLNs is highly nonlinear and the space of solution for W and a is highly nontrivial, and the network exhibit properties unique to nonlinear networks, as we will show in the following sections.
Posterior distribution of network weights: We consider a Bayesian network setup, where the network weights are random variables whose statistics are determined by the training data and network parameters, instead of deterministic variables. This probabilistic approach enables us to study the properties of the entire solution space instead of a single solution which may be heavily initialization dependent. We consider the posterior distribution of the network weights induced by learning with a Gaussian prior
[15
16
17
18).
given by
P Σ(((x" θ) - -Y 202 T
P(O)= = Z exp(
2T
where Z is the partition function Z= dd)(O). The first term in the exponent is the MSE of the network outputs on a set of P training data points x" from their target outputs Yи, and the second term is Gaussian prior the network parameters θ (W, a) with amplitude o a In this work focus on the T - 0 limit where the first term dominates. Below the network capacity, the distribution of θ concentrates onto the solution space that yields zero training error, the Gaussian prior then biases the solution space towards weights with smaller La norms. The fundamental properties of the system can be derived from the partition function. As the distribution is quadratic in the readout weights Om it is straightforward to integrate them out, which yields
Z z= / //Weepl- expl- 20
1 Tr(W W) + L Y (W) ( 'Y 1 log det (
(3)
2
where W denotes all the remaining weights in the network, and K (W) is the W dependent P X P kernel on the training data, defined as KZ (W)- (-" g(')(""T g(x")) x" (W) (W)x') (W)).
Generalization in infinitely wide GGDLNs: It is well known that in infinitely wide networks where N + oo while P remains finite (also referred to as the GP limit), K2(W) is self-averaging and does not depend on the specific realization of W. It can therefore be replaced by the GP kernel defined as [2]. For GGDLNs, the GP kernel for pair of arbitrary data x and y is given by Kxp(x,y) - ( ((()) g(y)) y), where yy(x,y) = X y. We denote the P x P kernel data matrix as K GP where KGr- = (ar(x",,')) and the input kernel matrix on training data as Ko where KDd = Kx"x", xx))
(K,(W)) w where W ~ o0)
3

---

Generalization error is measured by MSE including the bias and the variance contributions, 6g ((f(x)) - u(x))² (x), which depends on the first and second order statistics of the
biss
predictor. In the GP limit, we have
(f(x)) = kGp(x) KEpY
(Kaexxx>KKXXY_ (ōf(x))) = KGp(x, - (xpp://>)))__pppX))K//ppppr/> K
(4)
where (x) Kop(x, x"). Note that the rank of KGP is the same as the capacity of the network,
and the kernel matrix becomes singular as P approaches its capacity (the interpolation threshold), which results in nonzero training error, diverging bias and vanishing variance contribution to the generalization error (Fig (b-e). The singularity of the kernel at the interpolation threshold holds also for finite width networks, and similar diverging bias and vanishing variance are seen in our finite width theory below (Section 3) and are confirmed by simulation of networks trained with GD (see
Appendix 13 19)
3 Kernel shape renormalization theory in finite-width GLNs
We now address the finite width thermodynamic limit, where P,N - but P/N O(L) M, (1)) this limit, calculating the statistics of the network predictor requires integration over
W in Eq To do so, apply the previous method of Back-propagating Kernel Renormalization (see Appendix (A) to GGDLNs. The partition function for single hidden layer network given
by Z = exp(- H1). where the Hamiltonian H1 is given by
N
H1 = K Y + log det
log det U1 + NTr(U,)
2 K1th g(x
2
2
20
U U1g(x"))K3
(5)
Comparing the matrix K2 to KEP, we note that the GP kernel is renormalized by an an M x M matrix order parameter U1. This order paramter satisfies the self-consistent equation
U -- NM U 12 g K K.lgU// + NM U U
Ko (6)
where o denotes element-we wise multiplication. In the linear case (which corresponds to M = 1), the GP kernel is renormalized by scalar factor. In the M > 1 case, the effect of renormalization is more drastic as it changes that not only the amplitude but also the shape of the kernel. The renormalization matrix has interesting physical interpretation that relates it to the readout weights of GGDLNs,
U
am,30n
The calculation can be extended to multiple layers with a new order parameter introduced for each layer (see Appendix AI. The predictor statistics for input can be expressed in terms of the renormalized kernels, for network with L=
(f(x)) (x) K 'Y (ff(x))) K (x, - (x) (K, (x)
(8)
where (x, gx) Ulg(y))Ko(a y) denotes the renormalized kernel function for two arbitrary inputs x and y. K, denotes the P x P renormalized kernel matrix on the training data, and k,(x) is a P-dimensional vector, [x)
) = [(x, x²). The kernel renormalization in GGDLNG changes the shape of the kernel through the data dependent U1, reflecting the nonlinear property of the network, and resulting in more complex behavior of predictor statistics relative to the linear networks, shown in Section 4 Our theory describes the properties of the posterior distribution of the network weights induced at equilibrium by Langevin dynamics with the MSE cost function
and the Gaussian prior 4
20 21 (22] Simulating this dynamics agrees remarkably well with
the simulation (see Appendix B.2). Although our theoretical results do not directly describe the
solutions obtained by running gradient descent (GD) dynamics on the training error, it is interesting to ask to what extent the predicted behaviors of our theory are also exhibited by GD dynamics of the same network architectures, as GD-based learning more widely used. We will compare our theoretical results with numerics of GD dynamics throughout the paper. We consider the case where

---

the network is initialized with Gaussian i..d. weights with variance o². and the mean and variance of the predictors are evaluated across multiple mitiaaizztions (see Appendix C for details). As we will show, our theory makes accurate qualitative predictions for GD dynamics in all examples in this paper, in the sense that while the exact values may not match, the general trend of how generalization or representation varies with different parameters in different regimes are very similar.
4
Generalization
For linear networks the generalization error depends on N, o² and L through the variance only, while the mean predictor always assumes the same value as in the GP limit [ This is because the scalar kernel renormalization of k1 (x) is cancelled out in the mean predictor by the renormalization of the inverse kemnel KT- In contrast, for GGDLNs the mean predictor and hence the error bias also change with these network parameters due to the matrix nature of the kernel renormalization (Fq Below we investigate in detail how matrix renormalization of the kernel affects the generalization behavior (especially the bias term) of the network.
4.1 Networks with single hidden layer
a
b
0.16
bias
veriance
0g
teacher network
0.18
Global (b)
Local (c-e)
0.15
v(K)
0.15
-10
0.15
theory
0.5
EEE- wput
1000
3000
1000
2000
3000
Input
N bles
N
student network ((x)
x10
variance
local gatings
0.085
(x4
0.03
Umm.
110
0.025
0.06
0.08
theory
0.04
0.04
OCC
0.02
0.02
impul
1000
2000
3000
1000
2000
3000
1000
2000
3000
N
N=20
N=3000
2.5
10
04
10
0.04 10
20
20
20
30
40
40
50
40
50
20
30
40
Figure 2: Dependence of generalization eπror network width for ReLU teacher task. Ta) The ReLU teacher network, the input x is divided into subsets input dimensions, the input layer weights either assume same order of magnitude across different input dimensions (left, (b)), or assume larger amplitudes for one subset of input dimensions-the preferred inputs (right, bold connections to subset of input neurons,(c-e),, Bottom: The student network is GGDLN with one hidden layer and gatings with localized receptive fields: each gating is connected to only a subset of imput dimensions. (b) Bias, variance and generalization error decreases a function of N for regular ReLU teacher, theory agrees qualitatively well with GD dynamics. (c) Bias and generalization error increases as function of N for ReLU teacher with preferred inputs. (d) The renormalization matrix U, for different network widths for the teacher with preferred inputs. The first 10 x 10 block corresponds to the gatings with the same receptive field as the teacher's preferred inputs, and is amplified for small N. (e) The ratio of the average amplitude of the first 10 x 10 block relative to the average amplitude of the other four 10 10 diagonal blocks decreases as function of N.
Feature selection in finite-width networks: Unlike in DINs, the bias term in GGDLNs depends on N, exhibiting different dependence in different parameter regimes. This dependence also varies with
5

---

the choice of the gating functions. In Fig we consider a teacher learning task, commonly used for evaluating and understanding neural network performance 2
24
25
26
We present
results of learning a ReLU teacher task in GGDLNs with gatings that have localized receptive fields (i.e., the activation of each gating unit depends on only a subset of input dimensions, the receptive field of all gating units tile the No input dimensions, as shown in Fig 2 a) bottom), where the student GGDLN is required to learn the input-output relation of a given ReLU teacher. For a ReLU teacher with single fully connected hidden layer (Fig aa) top left), gatings with different receptive fields are of equal importance, hence the renormalization does not play beneficial functional role, and the infinitely wide network performs better than finite N. As shown in Fig 2 bb), bias, variance and generalization error all decrease with N. For a 'local' ReLU teacher with larger input weights for one subset of imput components (the preferred inputs, Fig 2 a) top right), renormalization improves task performance by the selective increase of the elements in U, that correspond to gating units whose receptive fields overlap the teacher's preferred inputs (Fig d&))). Hence, narrower networks (with a stronger renormalization) generalize better, and both the bias and the generalization error increase with N (Fig More generally, the input can represent set of fixed features of the data, and the 'local' teacher generate labels depending on a subset of the features. Therefore, networks with finite width are able select the relevant set of features by adjusting the amplitude in the renormalization matrix U, to assign the gating units with different importance for the task, while in the GP limit the network always assigns equal importance to all the gating units.
To summarize, our theory not only captures the more complex behavior of generalization (especially bias) as function of network width, but also provides qualitative explanation of how generalization is affected by the structure of the renormalization matrix in different tasks.
Effect of regularization strengths on generalization performance: Similar to the dependence on N. generalization also exhibits different behavior as a function of the regularization parameter o in different parameter regimes, with contributions from both the bias and the variance. The dependence of error bias on a also arises due to the matrix nature of the renormalization. In Fig we show parameter regimes where the bias can increase (Fig K (a-c)) or decrease (Fig 3 (d-f)) with a on MNIST dataset [19 (Appendix (+ ). Although the dependence on a is complicated and diverse, and there
lacks general rule for when the qualitative behavior changes, we found that our theory accurately
captures the qualitative behavior of results obtained from GD (Appendix Fig B. In both regimes the variance increases with as the solution space expands for weaker regularization. Specifically in Fig 3 (d-f), due to the increasing variance (e) and decreasing bias (d), there is a minimum error rate ((f,, Appendix A.3 Eq |50 for how error rate is calculated from the mean and variance of the predictor) at intermediate a,, indicating an optimal level of regularization strength as opposed to linear networks
4 where strong regularization (o
0) always results in optimal generalization
a
blas
b 1 variance
error rale
0.2
0.8
0.18
0.16
0.1
0.2
0.3
0.4
0.2
0.3
0.4
0.1
0.2
0.3
0.4
d
0.06
0.18
0.04
Decrease 0:45
0.14
0.02
0.4
0.12
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Figure 3: Generalization as function of a for GGDLNs trained on MNIST dataset predicted by our theory. (a-c) Bias (a), variance (b) and error rate (c) increase as a function of o. (b-f) Bias decreases as function of a while variance increases, leading an optimal a with minimum error rate.
GGDLNs with different choices of gatings achieve comparable performance to ReLU networks: The nonlinear operation of the gatings enables the network to learn nonlinear tasks. In Fig 4 we show that although the gatings are fixed during training, the network achieves comparable performance as fully trained nonlinear (Rel.U) network with the same hidden layer width for classifying even and odd digits in MNIST data when M is sufficiently large paverpaaraetterizationn does not lead to over-fitting

---

here, as shown also in other nonlinear networks 27 28 Furthermore, although the gatings are fixed during the supervised training of the GGDLN, they can be cleverly chosen to improve generalization performance. To demonstrate this strategy, we compared two different choices of gatings. Random gatings take the form Im Bm(x)= = ( V
possibly due to the explicit Lg prior).
b), where Vm is a No-dimensional random vector with standard Gaussian i.i.d. elements, b is a scalar threshold, and (ax) the heaviside step function. The pretrained gatings are trained on the unlabelled training dataset with unsupervised soft k-means clustering, such that the m-th gating 9m. (x) outputs the probability of assigning data x to the m-th cluster (Appendix (+)) As shown in Fig for pretrained gatings, generalization performance improves with M much faster compared to random gatings, and approaches the performance of ReL.U network at a smaller M. Our theory (Fig 4 and numerical results of GD dynamics (Appendix B33 E
x V
! agree qualitatively well. The result shows that GGDLNs can still achieve competitive performance on nonlinear tasks while remaining theoretically amenable.
b
C
bias
variance
error rate
0.3
0.02
0.25
0.015
0.2
0.5
0.15
0.005
0.1
0
50
100
0
50
100
50
100
M
Figure 4: Dependence of generalization on M for GGDLNs trained MNIST dataset predicted by our theory. Bias (a), variance (b) and error rate (c) as a function of M for random (red lines) and pretrained gatings (blue lines), and ReLU network with the same width (black dashed lines).
4.2 Kernel shape renormalization in deeper networks
We now consider the effect of the matrix renormalization on GGDLNS with more layers. We begin by analyzing the renormalization effect on the shape of the kernel in deep architectures. It is well known that the GP kernel of many nonlinear networks flattens (the kernel function goes to constant) as network depth increases 22. ultimately losing information about the input and degrading generalization performance. Here we show that kernel shape renormalization slows down flattening of kernels by incorporating data relevant information into the learned weights.
To study the shape of the kernel independent of kernel magnitude, we define the normalized kernel (x(x)y)
yx) = (x where Kr (x) denotes the renormalized kernel for GGDLN with hidden layers. This normalized kernel measures the cosine of the vectors x and y with generalized inner product defined by the kernel K2 (x. y), and therefore K,, (x, y) E [-1. 1]. For the GP kernel of GGDLNs, we have (x(x,y) y) = cos cos(g (x(), g(y))² cos(x, y). While Kr, depends on the specific choice of gatings in general, in the special case of random gatings with zero threshold Im (x) O Noo
VTx) and the number of gatings M CO, we can write Kr, analytically
function of the angle 0 between input vectors z and given by (f) (== cos( 0 E [- x]. Thus, as - oc, Kr (f) shrinks zero except for f = 0. This 'flattening" effect reflects the loss of information in deep networks, as pairs of inputs with different similarities now all have hidden representations that are orthogonal. The effect also empirically holds true for networks with finite M
(see Appendix B.4).
In Fig 5 we study the effect of kernel renormalization on the 'flattening' effect of deep GGDLNs. As shown in Fig 5 (a)-(c), the elements of the renormalized kernel shrink to zero at a much slower rate compared to the GP kernel. (Note that unlike the variance, the bias is affected only by shape changes, but not by changes in the amplitude of the kernel, in Fig d) we plot only the bias contribution to the generalization.) While mitigating the flattening of the GP kernel is general feature of our renormalized kernel for different parameters, its effect on the generalization performance (especially the bias) may be different for different network parameters. In the specific example in Fig finite width networks with less 'flattened' renormalized kernel achieve better performance than the GP limit. Both the GP limit and the finite width networks have optimal performance at T. this example.
7

---

06
0.8
0.04
grebabiity
0.03
grobablly
0.75
0.02
0.7
0.
0.01
0.6
0.5 Kixy)
0.5 K(xy)
Figure Shape renormalization slows down flattening of kernels in deep networks. (a-b) Distribution of kernel elements Kz (x. y) for the renormalized kernel (a) and GP kernel (b) for different network depth L. (c) Ratio kernel elements smaller or equal to 0.05 increases faster for GP kernel (blue line) compared to the renormalized kernel (black line), the renormalization slows down the rate at which elements in the GP kernel shrink to zero as a function of L. (d) The bias contribution to the generalization first decreases then increases function of L due to the flattening of the kernel (blue line). Finite width network with renormalized kernel performs better for L this parameter regime (black line). See Appendix C for detailed parameters.
GGDLNs for multiple tasks
In this section, we apply our theory to investigate the ability GGDLNs to perform multiple tasks. We consider two different scenarios below. First, different tasks require the network to learn input-output mappings on input data with different statistics. This scenario corresponds to real life situations where the training data distribution is non- non-stationary. The tasks be separated without any additional top-down information. In this case, the gatings are bottom-up, and are functions of the input data only. In the second case, different tasks give conflicting labels for the same inputs, corresponding to the situation where performing the two tasks require additional top-down contextual information, and the information scenarios, when the gatings are fixed and we modulate the de-correlation by changing network width and thus the strength of the kernel renormalization, find that de-correlation between tasks leads to better generalization performance.
be incorporated into the gating units in GGDLNs. In both
5.1 Bottom-up gating units
a
N=50
b
N=1000
0.03
Figure 6: GGDLNS with bottom-up gat- ing units learning multiple tasks trained on permuted MNIST. (a-b) Task-task correlation matrix C for N 50 and N 1000, different permutations are more decorrelated for larger N. (c) Er- ror rate decreases a function of N due the decorrelation.
0.02
0.01
10
4
8
10
d
0.25
rat0
errOr
(d) Ratio of the average amplitude of diagonal ele- ments versus off-diagonal elements in C increases as function of N. (e) Error rate first decreases then increases as function of gating threshold. (f) Decor- relation increases as a function gating threshold.
0.2
2.2
500
1000
500
1000
e
0.25
error rate
0.2
rati
0.15
-2
threshold
throshold
First we consider learning different tasks defined by vastly different input statisties with bottom-up gatings, using permuted MNIST as an example. Previous works have shown that GLNs mitigate catastrophic forgetting when sequentially trained permuted MNIST [8]. While theory does not address directly the dynamics sequential learning, we aim to shed light on this question by asking how the two tasks interfere with each other when they are learned simultaneously.
We introduce measure of inter-task interference by noting that after learning the mean predictor on new data Eq & is linear combination of the output labels Y of all the training data,

---

and the coefficient of this linear combination, is given by the 5-th coefficient of k(x) T K Thus, define task-task correlation matrix, via Cpg = Σg= Σ -e=ik'TK- lp 9 1, 1, nn, where we assume there are P training examples and P. test data for each task, with total of tasks. The amplitude of each element Cpg measures how much training data of task contribute to the prediction
the test data task p. Stronger diagonal elements indicates that the network separates the processing of data of different tasks (Fig 6 (b)). As we show in Fig K tune the relative strength of the diagonal elements of C smoothly by changing the network width (Fig C ())(d)) or by changing the threshold of the gating (Fig 3 (f)- In the case where the gatings are fixed and the network width is changed, an increase in the strength of the diagonal elements (Fig d)) results in better generalization (Fig c)), indicating that the network generalizes better by processing data of different tasks separately through the gating units. However, in the case where we change the activation of the gatings by adjusting the threshold, although different tasks are more de-correlated when the threshold is large due to a set of less overlapping gatings activated for each task, generalization eπrоr first decreases and then increases again. This is because for large threshold the sparsity of the gatings activated for each task limits the nonlinearity of the network, and therefore the generalization performance on this nonlinear task.
we can
5.2 Combined top-down and bottom-up gating units
Task f(s)
N:100 GD
N=:00 GD
fortidtinn gatige
permitte geting
3t (x)
400
200
03
400
200
ca
(x)
600
600
02
800
0.1
800
0.1
1200
1200
21,3
No
044
N=500
0.4
O0O OCO OCO
400
400
input Xck*
Task 2
800
0.1
800
0.1
parmitted gartnge
(K)
forbiddin gatinge
(arls]
1000
1200
200
600
1000
1000
1200
200
600
1000
d
0.2
GD
Theory
rafi
0.1.1
CD
theory
W1.9
COO Imput xcs
200
400
600
0
200
400
600
N
Figure 7: Kernel renormalization de-correlates different tasks defined by different labels on the same inputs. (a) GGDLNs performing two tasks using combined top-down and bottom-up task signal. (b) Top: Renormalized kernel calculated with Eq. M from GD dynamics. Bottom: Renormalized kernel theory. (c) Ratio of the magnitude of diagonal (blocks with white dashed lines in (b)) versus off diagonal blocks decreases as a function of N. (d) Generalization error increases with N.
We now consider learning two tasks that provide conflicting labels on the same input data. The gating units combine both top-down task signal which informs the system of which task to perform for a given input, and bottom-up signals which, as before, depend the input. In different tasks, different sets of gatings are permitted or forbidden depending on the top-down signal, then the states of the permitted gatings are further determined as function of the input x, while the forbidden gatings are set to 0, and the corresponding dendritic branches do not connect to the previous layer (a)) in this task. For a single hidden layer network, with a similar argument as in Section 2 it is straightforward to show that the number of different tasks that can be memorized is given by n S M and the number of training examples for each task needs to satisfy P < Ng. where M, is the number of permitted gating units in each task. In the limiting case where set of non-overlapping gating units are permitted in each of the n tasks, the network is equivalent to n sub-networks, each independently performing one task. In this case M, is limited by M/n, which in turn limits the capacity and the effective input-output nonlinearity for each independent task. We consider the case where the permitted gatings are chosen randomly for each task and are therefore in
neurons (Fig M
9

---

general overlapping across tasks. We then investigate how learning modifies the correlation induced by the overlapping gatings through the renormalization matrix. As an example we consider training on permuted and un permuted MNIST digits of O and I's. One task is to classify the two digits in both permuted and un un-permuted data, and the second task is to separate the permuted digits (both and 1) from the un perppmuueed digits. The labels of the two tasks are uncorrelated, while the permitted gatings of the two tasks are partially overlapping. In this case the renormalized kernel K can be written as Kpr.qe
x x x". Here P.q E {1, are the task indices, and p.v/ = 1,... P are the input indices. The kernel is therefore 2P 2P as shown in Fig bb) (P = 600); the diagonal blocks (white dashed lines) correspond to kernels of task and task 2, while the off diagonal blocks correspond to the cross kernels. In Fig b) bottom, we show the renormalized kernel with the renormalization matrix U1 calculated by solving Eq K Similar results are achieved by by numerically estimating Eq 7 with readout weights obtained from GD dynamics (Fig top).
( ( " )x")(x"))
No
The results demonstrate that stronger kernel renormalization achieved in narrower networks sup- presses more strongly the correlation between tasks, reflected by the weaker off-diagonal blocks in b) A decreasing ratio between the average amplitudes of the diagonal and off-diagonal blocks shows that the de-correlation effect diminishes for large N, leading to increasing generalization error with N (Figg C & d)).
Discussion
In this work, we proposed a novel gating network architecture, the GGDLN, amenable to theoretical analysis of the network expressivity and generalization performance. The predictor statisties of GGDLNs can be expressed in terms of kernels that undergo shape renormalization, resulting diverse behavior of the bias as a function of various network parameters. This renormalization slows down the flattening of the GP kernel in deep networks, suggesting that the loss of input information as L increases may be prevented in finite-width nonlinear networks. We also investigate the capability of GGDLNs to perform multiple tasks. While our theory is an exact description of the posterior of weight distribution induced by Langevin dynamics in Bayesian learning, it provides surprisingly well qualitative agreement with results obtained with GD dynamics for not only the generalization but also the kernel representation with matrix renormalization, largely extending its applicability. There are several limitations of our work. Our mean-field analysis is accurate in the 'finite-width' thermodynamic limit where both P and N go to infinity, but M and L remain finite. In practice, the size of the renormalization matrix increases as M hence for some moderate M, as L increases, any large but finite N might eventually get the network outside the above thermodynamic regime. The theory also focuses on the equilibrium distribution induced by learning and does not address important questions related to the learning dynamics. Finally, although we have shown qualitative correspondence of the GGDLN properties and standard DNNs with local nonlinearity, as ReLU, full theory of the thermodynamic limit of DNNs with local nonlinearity still an open challenge.
While our theory currently addresses learning in GGDLNs using global cost function, exploring the possibility of extending the formalization of the equilibrium distribution to characterize local learning dynamics is an ongoing work. Recent works have shown that multilayer perceptrons (MLPs) with learned gatings that implements spatial attention have surprisingly good performance on Natural Language Processing (NLP) and computer vision 22 Extension of our theory to learnable gatings that implements attention mechanisms remains to explored. Furthermore, incorporating convolutional architecture [30 3)
into our GGDLNs and using the gating units to encode centext-dependen modification of different feature maps is an interesting direction related to the
32
R3 RE!
fast-developing research topic of visual question-answering (VQA) (35) 36 37 where answering
different questions about the same image similar to performing multiple tasks in different contexts
with different labels the same dataset, we discussed in Section 5
We leave these exciting
research directions for future work.
Acknowledgement
We thank the anonymous reviewers for their helpful comments. This research supported by the Swartz Foundation, the NIH grant from the NINDS (No. 1U19NS and the Gatsby Charitable Foundation. We acknowledge the support of a generous gift from Amazon. This paper is dedicated to the memory of Mrs. Lily Safra, great supporter of brain research.
10

---

References
[1] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009. B.4
[2] Jachoon Lec. Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington,
and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint
arXiv:1711.0016: 2017.
4.2
[3] Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner and Zoubin Ghahramani, Gaussian process behaviour in wide deep neural networks. arXiv preprint
vrXiv:1804.1127,, 2018.
[4] Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Physical Review X, 113))033059, 2021. I
[5] Andrew M Saxe, James L McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(23):11537-11556
2019.
[6] Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprint arXiv: 10094 2018. D
[7] David Budden, Adam Marblestone, Eren Sezener, Tor Lattimore, Gregory Wayne, and Joel Veness. Gaussian gated linear networks. Advances in Neural Information Processing Systems, 33:16508-16519 2020. D
[8] Eren Sezener, Agnieszka Grabska-Barwe Barwinska, Dimitar Kostadinov, Maxime Beau, Sanjukta Krishnagopal, David Budden, Marcus Hutter, Joel Veness, Matthew Botvinick, Claudia Clopath, al. A rapid and efficient learning rule for biological neural circuits. BioRxiv, 2021. 6.1
[9] Joel Veness, Tor Lattimore, David Budden, Avishkar Bhoopchand, Christopher Mattern, Ag- nieszka Sabsska-Barwinsk.. Eren Sezener, Jianan Wang, Peter Toth, Simon Schmitt, et al. Gated linear networks, arXiv preprint arXiv:1910.01526. 2019.
[10] Jonathan Fiat, Eran Malach, and Shai Shalev-Shwartz. Decoupling gating from lincarity. arXiv preprint arXiv: 05033, 2019.
[11] Joel Veness, Tor Lattimore, Avishkar Bhoopchand, Agnieszka Grabska-Barwinska, Christo- pher Mattern, and Peter Toth. Online learning with gated linear networks. arXiv preprint
1rX1v:1712.01897, 2017. !
[12] Samuel Lippl, LF Abbott, and Yuo Chung. The implicit bias of gradient descent on generalized gated linear networks. arXiv preprint (iXi:22222.026499 2022. [
[13] Roman Vershynin. Memory capacity of neural networks with threshold and rectified linear unit activations. SIAM Journal on Mathematics of Data Science, 2(4):1004-1033 2020. Ø
[14] Masami Yamasaki. The lower bound of the capacity for neural network with multiple hidden layers. In International Conference on Artificial Neural Networks, pages 546-549. Springer, 1993. 2
[15] Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Statistical mechanics of neural networks near saturation. Annals of physics, 173(1)300667, 1987.
[16] Madhu Advani, Subhaneil Lahiri, and Surya Ganguli. Statistical mechanics of complex neural systems and high dimensional data. Journal of Statistical Mechanics: Theory and Experiment, PO03044)) 2013.
[17] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam $ Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11:501-528, 2020. 2
[18] Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001. 2

---

[19] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
[20] Gadi Naveh, Oded Ben David, Haim Sompolinsky, and Zohar Ringel. Predicting the outputs of finite deep neural networks trained with noisy gradients. Physical Review E, 104(6):66330,, 2021. K
[21] Jean Zinn-Justin. Quantum field theory and critical phenomena, volume 171. Oxford university press, 2021. E
[22] H. Risken and H. Haken. The Fokker-Planck Equation: Methods of Solution and Applications Second Edition. Springer, 1989. 3
[23] Shinji Watanabe, Takaaki Hori, Jonathan Le Roux, and John R Hershey. Student-teacher network learning with enhanced features. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5275-5279. IEEE, 2017. 4.1
[24] Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: review and new outlooks. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2021. 4.
[25] Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020.
[26] Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8):6056, 1992. 4.1
[27] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variane variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849 15854, 2019, 4.1
[28) Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4): 2020. 4.1
[29] Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204-9215, 2021.
[30] Mark Van der Wilk, Carl Edward Rasmussen, and James Hensman. Convolutional gaussian processes. Advances in Neural Information Processing Systems, 30, 2017. 6
[31] Gadi Naveh and Zohar Ringel. A self consistent theory of gaussian processes captures feature learning effects in finite cnns. Advances in Neural Information Processing Systems, 34, 2021.
[32] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 77:354 377, 2018.
[33] Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional networks as shallow gaussian processes. arXiv preprint arXiv: /808.05587 2018. g
[34] Saad Albawi, Tareq Abed Mohammed, and Saad Al-Zawi. Understanding of a convolutional neural network. In 2017 international conference engineering and technology (ICET), pages
1-6. Icee, 2017. 3
[35] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Ziticck, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425 2233,, 2015.
[36] Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding. 163:3-20, 2017. E
[37] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co- attention for visual question answering. Advances in neural information processing systems, 29, 2016.
[38] Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-315, 2007. C,5
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S and c
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proof's of all theoretical results? [Yes] See Appendix A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL.? [Yes] In supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Fig 5
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Appendix C
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "l0u7h4tdkh9", "reviewer": "Reviewer_Psan", "summary": "This paper presents theoretical insights that build upon the recently proposed Gated Linear Networks (GLNs). First, a simplified version of (GLNs) called Globally Gated Deep Linear Networks (GGDLNs) are proposed. The motivation for the proposal is to construct a model that can serve as a useful model of learning in general neural networks by being practical enough, yet amenable to theoretical analysis. The proposed model enables the authors to obtain useful theoretical characterizations of the memory capacity and generalization behavior for GGDLNs, which are confirmed via simulations.", "questions": "In Eq. 1, shouldn't there be a summation over $M$ dendrites when computing neuron outputs for the intermediate layers? How are the $M$ dendritic activations used to compute the $N$ neuron outputs otherwise?", "limitations": "Yes, the limitations are discussed in Sec. 6.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I should note that I am not familiar with the prior work in this area, and thus my opinions are only based on a few readings of this paper as a newcomer. I can not ascertain if all relevant prior work has been properly credited, and can not verify that the method of Backpropagating Kernel Renormalization (Li & Sompolinsky, 2021) has been applied correctly to the proposed model.\nNevertheless, the authors make a good case that GGDLNs are a more interesting model to study compared to deep linear networks studied in recent work, exhibiting generalization behavior that is closer to neural networks used in practice (Sec. 4). It is also shown that there is remarkable qualitative agreement between the paper's theoretical predictions regarding generalization and results obtained using gradient descent, even though technically the theory applies to posterior predictions obtained using Langevin Dynamics. This again indicates that GGDLNs are useful objects to study for predicting the behavior of neural networks in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T29lfJdsC8Q", "reviewer": "Reviewer_PKE5", "summary": "The authors propose a new architecture based on Gated Linear Networks (GLN) called Globally Gated Deep Linear Network (GGDLN). They derive equation for the generalization properties and describe the architecture theoretically. They present also several experiments comparing GGDLN and the same architecture learned by Gradient Descent.", "questions": "The formalism in Fig1 is different from the one used in the equations.\nFor example $x_i^L$ in Figure and $x_{L, i}$ in the equation\n\nThe Fig1(a) should be improved because it is not so clear. Some suggestions:\n- the number of neurons at layer L and L-1 are not explicit (it should be N for all layers right?)\n- it is not clear that $W_{i0}^m$ ...  $W_{iN}^m$ are: the connections between the m-th dendrite of i-th neuron and the neurons 0...N in the previous layer. Maybe the other connections could be depicted with dashed line and focus only on the part important for the computation in Eq.1.\n- it is necessary to define also the layer. In the equation (1), in fact, the authors use $W_{L,ij}^m$. \n- $x$ is a vector I suggest to use the proper style (e.g. bold, $\\mathbf{x}$) in the figure and throughout the paper.\n\nSince, based on my understanding, $a_{i,m}$ is the i-th element of $\\mathbf{a_m}$, maybe is more coherent with other approaches in eq.1 to use $a_{m,i}$. However the authors should use a formalism and be coherent throughout the paper\n\nEquation (1) what are the limits of j in the summation?\n\nrow 79 --> I didn't understand the formalism. What are $m_1$, $m_2$, etc. What is the meaning of the summation? Maybe I understood the sense but the formalism is not clear neither introduced before.\n\nIf I understood well, the parameters are learned maximizing the posterior probability in (3) [as described rows 143-145] but in rows 87-101 is not clearly stated this.\n\nrow 118 and Appendix C4 --> Is the network defined in Figure 1 trained using GD as comparison? \n\nWhat do you mean as ReLU Teacher task and what do the student network is supposed to do?\n\nThe caption of Figure 2 is not clear. I suggest to use bold style for (a), (b), (c), and (d) when the successive text is related to them because, as now, the caption is confusing. I suggest to review all the captions because this problem is common.\n\nrow 163-174 : What do you mean with localized receptive fields for the gatings? I didn't understand this part. Could you clarify it?\n\nI didn't understand the rationale behind the comparison of a learning using the proposed approach and the GD learning approach. \nWhy does these modalities should behave the same? Also the authors said that \"While we do not expect our theory to agree with GD dynamics quantitatively, as we will show, our theory makes accurate qualitative predictions for GD dynamics in all examples throughout this paper.\". \nWhat do author mean with qualitative? The shapes are different and maybe it is not a big deal because the modalities are different. Instead it is important to demonstrate that the proposed approach has a good performance on several tasks (not necessary outperform GD but that accomplish a given task as described in rows 185-201) but the rationale behind the experimental setting is not clear to me.\n\nSection 5 looks like a separated part that could deserve a paper itself. Maybe I remove it and devote more space to clarifications, descriptions, assumption, discussion, etc.\n\nThe Discussion section looks like a Conclusion paragraph.", "limitations": "The authors describe more the next perspectives that limitations. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength is the attempt to analyze the Gated Deep Linear Networks on the theoretical basis. Moreover the authors have made a considerable effort also to analyse the architecture by different perspectives.\n\nMy main concern is that the work is too dense. Nine pages looks too few to contain all the material authors want to present making the reading process difficult (also because the introductory figures are not so good as necessary). For this reason some details are missing, discussion is very short and some clarifications are missing.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IfnO1KS8iXE", "reviewer": "Reviewer_k3LQ", "summary": "The authors propose Globally Gated Deep Linear Networks (GGDLNs), which is a variant of Gated Linear Networks (GLNs) that is more amenable to theoretical analysis. The paper presents a wide breadth of theoretical results, which are additionally supported by simulations.  These include bias & variance analysis for single-layer networks, kernel normalization analysis for multi-layer networks, and multi-task learning.", "questions": "1- Is it possible to apply the kernel shape renormalisation theory to GGDLNs with local learning (like in GLNs) instead of backpropagation?\n\n2- Are GGDLNs universal function approximators?\n\n3- Have the authors observed the \"double descent\" with GGDLNs in the simulations?\n\n4- Figure 4 shows that decreasing generalisation error with M. However, as the capacity grows quickly with M, we would expect to see overfitting. Why is this not the case?\n\n5- Can the authors explain what the flattening effect mean in practice? And is it observed in, eg, ReLu networks?\n\n6- Error rate decreases as a function of the threshold in Figure 6.b. For even larger thresholds, eg (5, 10) we would expect to see the error rate to increase again, because the hyperplanes will not be able to separate the data. Is this the case in your simulations? If so, it would be insightful to widen the range of the x-axis.", "limitations": "Limitations are adequately addressed.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Backpropagating Kernel Renormalization (BKR) has been applied for analysing multi-layer linear networks. One of the contributions of this work is to extend the BKR analysis to multi-layer non-linear networks.  This non-linearity is achieved via gating as opposed to static non-linearities such as ReLu functions, which makes the analysis tractable. This transition from linear functions to non-linear functions is a significant theoretical progress.\n\nThe theoretical analysis is sound and insightful. Most results are intuitive/expected; however, I am confused by some of them, as I explain in Questions 3-6. I will update my evaluation after understanding some of the results better.\n\nThe pretrained gating idea is novel and shows promise for GLN-like architectures.\n\nPerhaps the biggest weakness is that the analysis is limited to the GGDLNs, and do not directly apply to popular architectures such as ReLu MLPs. However, analysis of non-linear networks is extreme difficult. So any progress is valuable. As the authors note, the analysis techniques developed in the paper might be applicable to more popular architectures in the future.\n\nThe paper is written and presented clearly. However, it can be quite dense in some places, which is expected given the quantity of results.\n\nThere are a few minor writing issues:\n- The text could benefit from a spell-check (\"processisng\",  \"analayzing\" etc).\n- The x-axis label in Figure 1 must be P instead of M I believe.\n- Panel label d is missing in Figure 7. See caption \"(d) Generalization error increases with...\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept