Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper analyzes the learning dynamics of linearized deep networks with parallel pathways driven by the same cost functions. The analysis was done from a dynamical system perspective in the tradition of Saxe et al.  on linear deep networks in the infinite width limit. An interesting conclusion is that at large depth, different pathways seem to compete and learn sharply distinct and diversified feature representations. 
strengths_and_weaknesses: Strengths:
1. Fairly detailed and involved theoretical analysis, and the conclusion is verified by simulation results.
2. Results potentially reveal and illuminate a potential functional purpose of multiple pathways in the nervous systems. 

Weaknesses:
1. The analysis is on linearized networks, with the nonlinearity removed, so whether this analysis is relevant to deep convolution neural networks and their power is open to question.
2. At some level, the linearized network seems to be behaving like a competitive network performing principal component analysis and independent component analysis, which would also arrive at similar "conclusions."
questions: Issues related to weaknesses should be addressed if possible.
limitations: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Inspired by mammalian brain architecture the authors the impact of having parallel pathways in artificial neural networks. They derive results analytically and conduct simulations. They show that different features of the training set typically concentrate in one of the pathways arguing for diversified representation with specialized pathways. 
strengths_and_weaknesses: The paper provides both analytical results and simulation, making an interesting theoretical contribution.

To make the problem analytically tractable the authors use linear neural networks. It is not clear how the use of non-linearity would affect the results. While this might not be possible to figure out analytically, it might be possible in simulations. 

The authors discuss possible utilities of diverse pathways, but the contribution would be much stronger if they would be able to show this using experiments. 

It is not clear how different assumptions, such as initialization would affect the results. 
questions: Can you discuss more the non-linear case or perhaps conduct simulations?

How could one evaluate that diverse parallel pathways are indeed helpful in ANNs - is there some example (perhaps related to what and where streams) that could be turned into a quantifiable experiment? 
limitations: I think these are overall addressed reasonably well.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This works aims to theoretically study the learning dynamics of deep linear networks with multiple parallel pathways (similar to parallel pathways of processing in the cortex such as dorsal/ventral, magnocellular/parvocellular pathways, etc.) in the infinite width limit. Despite the expressivity of such networks being subpar to deep learning models used for practical purposes, deep linear networks, and their associated objective functions that the authors study possess learning dynamics that show interesting nonlinear behavior. Here the authors analytically show that the different parallel pathways learn to specialize in representing different features of the training set as a function of the depth of the network and demonstrate the finding with numerical simulation. This work is exciting and is also relevant to recent self-supervised learning methods that possess parallel pathways and have been shown to mimic biological visual processing. 
strengths_and_weaknesses: Strengths:
- The proposed work is an excellent extension of prior impactful work on studying the learning dynamics of deep linear networks by considering parallel pathways that are abundant in biological vision (e.g. dorsal/ventral, magno/parvocellular, etc.).
- I appreciate the well-designed figures that make it easier to grasp key concepts mentioned in the paper. I found the phase portraits in Fig. 3 to be really helpful in qualitatively interpreting the main result of the concentration of different features to different parallel pathways as a function of processing depth.
- The main results from this paper provide valuable insights into developing computational models of parallel circuits in biological neural networks. 
- It is great that the authors are planning to make the code for all numerical simulations and figures available publicly on Github.
- Overall, I find the paper to be well written and organized with well-designed and clearly labeled figures wherever necessary.

Weaknesses:
- There are quite strong assumptions in place that may confine the space of models that the proposed results apply to; e.g. absence of nonlinearities, specific initialization scheme, infinite width limit.

questions: - This is rather a comment that could be addressed by future work and not a suggested extension of the current work. The authors motivate the importance of studying the learning dynamics of deep networks with parallel pathways through their connection to biological information processing. 
- However, given that there are (potentially) more biologically plausible learning rules than gradient-based error propagation (I acknowledge this area to be debated) such as local learning and Hebbian learning, one would think that studying the learning dynamics of deep networks with (or without) parallel pathways with such learning algorithms that are potentially more aligned to learning in the brain would be extremely interesting for the neuroscience community.
limitations: Yes, the authors have clearly stated the limitations of this study (restricted to deep linear networks in the infinite width limit for a particular initialization scheme, whereas commonly used deep networks in practice don’t necessarily obey all these assumptions).
review_title: null

Meta-review: metareview: Reviewers agree that this is an sound and well presented contribution.
confidence: Certain
Decision: Accept