Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose Globally Gated Deep Linear Networks (GGDLNs), which is a variant of Gated Linear Networks (GLNs) that is more amenable to theoretical analysis. The paper presents a wide breadth of theoretical results, which are additionally supported by simulations.  These include bias & variance analysis for single-layer networks, kernel normalization analysis for multi-layer networks, and multi-task learning.
strengths_and_weaknesses: Backpropagating Kernel Renormalization (BKR) has been applied for analysing multi-layer linear networks. One of the contributions of this work is to extend the BKR analysis to multi-layer non-linear networks.  This non-linearity is achieved via gating as opposed to static non-linearities such as ReLu functions, which makes the analysis tractable. This transition from linear functions to non-linear functions is a significant theoretical progress.

The theoretical analysis is sound and insightful. Most results are intuitive/expected; however, I am confused by some of them, as I explain in Questions 3-6. I will update my evaluation after understanding some of the results better.

The pretrained gating idea is novel and shows promise for GLN-like architectures.

Perhaps the biggest weakness is that the analysis is limited to the GGDLNs, and do not directly apply to popular architectures such as ReLu MLPs. However, analysis of non-linear networks is extreme difficult. So any progress is valuable. As the authors note, the analysis techniques developed in the paper might be applicable to more popular architectures in the future.

The paper is written and presented clearly. However, it can be quite dense in some places, which is expected given the quantity of results.

There are a few minor writing issues:
- The text could benefit from a spell-check ("processisng",  "analayzing" etc).
- The x-axis label in Figure 1 must be P instead of M I believe.
- Panel label d is missing in Figure 7. See caption "(d) Generalization error increases with..."
questions: 1- Is it possible to apply the kernel shape renormalisation theory to GGDLNs with local learning (like in GLNs) instead of backpropagation?

2- Are GGDLNs universal function approximators?

3- Have the authors observed the "double descent" with GGDLNs in the simulations?

4- Figure 4 shows that decreasing generalisation error with M. However, as the capacity grows quickly with M, we would expect to see overfitting. Why is this not the case?

5- Can the authors explain what the flattening effect mean in practice? And is it observed in, eg, ReLu networks?

6- Error rate decreases as a function of the threshold in Figure 6.b. For even larger thresholds, eg (5, 10) we would expect to see the error rate to increase again, because the hyperplanes will not be able to separate the data. Is this the case in your simulations? If so, it would be insightful to widen the range of the x-axis.
limitations: Limitations are adequately addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents theoretical insights that build upon the recently proposed Gated Linear Networks (GLNs). First, a simplified version of (GLNs) called Globally Gated Deep Linear Networks (GGDLNs) are proposed. The motivation for the proposal is to construct a model that can serve as a useful model of learning in general neural networks by being practical enough, yet amenable to theoretical analysis. The proposed model enables the authors to obtain useful theoretical characterizations of the memory capacity and generalization behavior for GGDLNs, which are confirmed via simulations.
strengths_and_weaknesses: I should note that I am not familiar with the prior work in this area, and thus my opinions are only based on a few readings of this paper as a newcomer. I can not ascertain if all relevant prior work has been properly credited, and can not verify that the method of Backpropagating Kernel Renormalization (Li & Sompolinsky, 2021) has been applied correctly to the proposed model.
Nevertheless, the authors make a good case that GGDLNs are a more interesting model to study compared to deep linear networks studied in recent work, exhibiting generalization behavior that is closer to neural networks used in practice (Sec. 4). It is also shown that there is remarkable qualitative agreement between the paper's theoretical predictions regarding generalization and results obtained using gradient descent, even though technically the theory applies to posterior predictions obtained using Langevin Dynamics. This again indicates that GGDLNs are useful objects to study for predicting the behavior of neural networks in practice.
questions: In Eq. 1, shouldn't there be a summation over $M$ dendrites when computing neuron outputs for the intermediate layers? How are the $M$ dendritic activations used to compute the $N$ neuron outputs otherwise?
limitations: Yes, the limitations are discussed in Sec. 6.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new architecture based on Gated Linear Networks (GLN) called Globally Gated Deep Linear Network (GGDLN). They derive equation for the generalization properties and describe the architecture theoretically. They present also several experiments comparing GGDLN and the same architecture learned by Gradient Descent.
strengths_and_weaknesses: The main strength is the attempt to analyze the Gated Deep Linear Networks on the theoretical basis. Moreover the authors have made a considerable effort also to analyse the architecture by different perspectives.

My main concern is that the work is too dense. Nine pages looks too few to contain all the material authors want to present making the reading process difficult (also because the introductory figures are not so good as necessary). For this reason some details are missing, discussion is very short and some clarifications are missing.  
questions: The formalism in Fig1 is different from the one used in the equations.
For example $x_i^L$ in Figure and $x_{L, i}$ in the equation

The Fig1(a) should be improved because it is not so clear. Some suggestions:
- the number of neurons at layer L and L-1 are not explicit (it should be N for all layers right?)
- it is not clear that $W_{i0}^m$ ...  $W_{iN}^m$ are: the connections between the m-th dendrite of i-th neuron and the neurons 0...N in the previous layer. Maybe the other connections could be depicted with dashed line and focus only on the part important for the computation in Eq.1.
- it is necessary to define also the layer. In the equation (1), in fact, the authors use $W_{L,ij}^m$. 
- $x$ is a vector I suggest to use the proper style (e.g. bold, $\mathbf{x}$) in the figure and throughout the paper.

Since, based on my understanding, $a_{i,m}$ is the i-th element of $\mathbf{a_m}$, maybe is more coherent with other approaches in eq.1 to use $a_{m,i}$. However the authors should use a formalism and be coherent throughout the paper

Equation (1) what are the limits of j in the summation?

row 79 --> I didn't understand the formalism. What are $m_1$, $m_2$, etc. What is the meaning of the summation? Maybe I understood the sense but the formalism is not clear neither introduced before.

If I understood well, the parameters are learned maximizing the posterior probability in (3) [as described rows 143-145] but in rows 87-101 is not clearly stated this.

row 118 and Appendix C4 --> Is the network defined in Figure 1 trained using GD as comparison? 

What do you mean as ReLU Teacher task and what do the student network is supposed to do?

The caption of Figure 2 is not clear. I suggest to use bold style for (a), (b), (c), and (d) when the successive text is related to them because, as now, the caption is confusing. I suggest to review all the captions because this problem is common.

row 163-174 : What do you mean with localized receptive fields for the gatings? I didn't understand this part. Could you clarify it?

I didn't understand the rationale behind the comparison of a learning using the proposed approach and the GD learning approach. 
Why does these modalities should behave the same? Also the authors said that "While we do not expect our theory to agree with GD dynamics quantitatively, as we will show, our theory makes accurate qualitative predictions for GD dynamics in all examples throughout this paper.". 
What do author mean with qualitative? The shapes are different and maybe it is not a big deal because the modalities are different. Instead it is important to demonstrate that the proposed approach has a good performance on several tasks (not necessary outperform GD but that accomplish a given task as described in rows 185-201) but the rationale behind the experimental setting is not clear to me.

Section 5 looks like a separated part that could deserve a paper itself. Maybe I remove it and devote more space to clarifications, descriptions, assumption, discussion, etc.

The Discussion section looks like a Conclusion paragraph.
limitations: The authors describe more the next perspectives that limitations.
review_title: null

Meta-review: metareview: Three reviewers recommend accept. The reviewers praised the significant extensions of previous works, the novelty of the ideas, and found the theoretical analysis sound and insightful. The author responses to initial feedback were found to be insightful and reassured the reviewers about their recommendations. Hence I am recommending accept. I encourage the authors to take the reviewers feedback carefully into consideration when preparing the final manuscript and to work on the items promised in the discussion period.  
confidence: Less certain
Decision: Accept