Learning dynamics of deep linear networks with multiple pathways
Authors: Jianghong Shi
Eric Todd SheaBrown
Michael A Buice
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep networks
linear networks
dynamical systems
theory
parallel pathways

Abstract:   Not only have deep networks become standard in machine learning, they are increasingly of interest in neuroscience as models of cortical computation that capture relationships between structural and functional properties.  In addition they are a useful target of theoretical research into the properties of network computation.  Deep networks typically have a serial or approximately serial organization across layers, and this is often mirrored in models that purport to represent computation in mammalian brains.  There are, however, multiple examples of parallel pathways in mammalian brains.  In some cases, such as the mouse, the entire visual system appears arranged in a largely parallel, rather than serial fashion.  While these pathways may be formed by differing cost functions that drive different computations, here we present a new mathematical analysis of learning dynamics in networks that have parallel computational pathways driven by the same cost function.  We use the approximation of deep linear networks with large hidden layer sizes to show that, as the depth of the parallel pathways increases, different features of the training set (defined by the singular values of the input-output correlation) will typically concentrate in one of the pathways.  This result is derived analytically and demonstrated with numerical simulation.  Thus, rather than sharing stimulus and task features across multiple pathways, parallel network architectures learn to produce sharply diversified representations with specialized and specific pathways, a mechanism which may hold important consequences for codes in both biological and artificial systems.

Learning dynamics of deep linear networks with multiple pathways
Jianghong Shi
Eric Shea-Brown
Department of Applied Mathematics University of Washington Seattle, WA 98195 jhshi@uw.ed
Department of Applied Mathematics
University of Washington Seattle, WA 98195
etab@uw.
Michael A. Buice
Allen Institute MindScope Program Seattle, WA 98109
mi micchellbuballenissttttteee org
Abstract
Not only have deep networks become standard in machine learning, they are in- creasingly of interest in neuroscience as models of cortical computation that capture relationships between structural and functional properties. In addition they a useful target of theoretical research into the properties of network computation. Deep networks typically have serial approximately serial organization across layers, and this is often mirrored in models that purport to represent computation in mammalian brains. There are, however, multiple examples of parallel pathways in mammalian brains. In some cases, such as the mouse, the entire visual system appears arranged in largely parallel, rather than serial fashion. While these path- ways may be formed by differing cost functions that drive different computations, here we present new mathematical analysis of learning dynamics networks that have parallel computational pathways driven by the same cost function. We use the approximation of deep linear networks with large hidden layer sizes to show that, as the depth of the parallel pathways increases, different features of the training set (defined by the singular values of the input-output correlation) will typically concentrate in of the pathways. This result is derived analytically and demonstrated with numerical simulation with both linear and non-linear networks. Thus, rather than sharing stimulus and task features across multiple pathways, parallel network architectures learn to produce sharply diversified representations with specialized and specific pathways, mechanism which may hold important consequences for codes both biological and artificial systems.
Introduction
Deep networks are increasingly used as models of cortical computation, particularly the visual pathway. Indeed, the earliest convolutional neural network was a model of the mammalian visual system (6) and the use of CNNs to model the visual pathway has continued [14] 21 22 3 2 These models usually serial models of the primate ventral stream that describe the computation of that system as being closely related to object recognition. However, processing of visual information in the primate associated with (at least) two parallel pathways, the "ventral" (or "what") and the "dorsal" (or "where") pathways 4 [7 [0]. In the mouse visual system, anatomical analysis also suggests that there parallel pathways g 19 20). although the specific functional role of these
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

pathways is less clear. Interestingly, recent research [ has shown that self self-supervised training in neural network architectures with parallel pathways can lead to the emergence of ventral-like and dorsal-like pathways. Another study shows that, with similar self-supervised training objective, shallower architectures with parallel pathways lead closer matches with functional data in the Allen Brain Observatory than deep single pathway architectures [].. Furthermore, an anatomically constrained deep neural network model for the mouse visual cortex has demonstrated that such parallel architectures produce more diverse representations of visual images, compared to single pathway architectures [8]
Additionally, a coincident observation has been made for the original AlexNet [ (a deep neural network designed for image processing) that also shows the emergence of functional specialization across parallel pathways, with different pathways (defined in the model because it was implemented on multiple GPUs) learning different features of the input data. Overall, in the road map for developing future artificial intelligence systems, architectures with parallel pathways promise potential solutions for multi-tasking, multi-sensory, energy-efficien computing [15] But despite the richness of information representation and the potential capabilities of the architectures with parallel pathways, there is a lack of research in understanding the fundamental learning behavior that occurs in such architectures. Here, we advance this frontier through mathematical analysis of the behaviour of deep neural networks with parallel pathways, in the tractable setting of deep linear networks [16].
Our starting point follows Saxe et al. [17] which elegantly shows how, in such a linear network, the training data and learning dynamics for a given task can be fully described via a of independent modes resulting from singular value decomposition (SVD) of the input-output correlation matrix. During gradient decent, a deep linear network will pick up the task related information from the data by acquiring knowledge from each mode [17]. Here, we use the same overall framework to study the dynamics of learning in linear networks with parallel pathways.
Our analysis and simulations yield the following main results. For linear networks with multiple parallel pathways of hidden neurons that feed into the same output representation and cost function:
1. For wide networks with random weight initialization, the learning dynamics reduces to a coupled of ordinary differential equations of dimension equal to the number of pathways for each non-zero singular vector in the input-output correlation matrix of the training set.
2. These dynamics are competitive in the sense that each pathway will compete to "explain" the singular values of the training set.
3. For deeper networks, this competition becomes more severe, resulting in more trajectories in phase space flowing extreme solutions, wherein features (defined by the singular values of the training set) are confined largely to a single pathway.
4. For the strict infinite-width limit, the system flows to state wherein features are shared evenly across the pathways. At finite width, fluctuations in the initial state put the system onto a trajectory leading to a biased solution, where the bias more severe at larger depths. Thus in large-width, large-depth networks, the system evinces finite-size induced spontaneous symmetry breaking.
We demonstrate this result in simulations and show empiricially that the main result holds in networks with non-linearities. Thus, this depth-dependent mechanism reveals dynamical link between network architecture and representation learning.
2 Mathematical framework
Network description: We study the learning dynamics under gradient descent of a system with parallel pathways with one or more hidden layers in each pathway. We consider "linear" networks in which the transformation of each layer to the next is given by a matrix multiplication, with no non-linearity (i.c.. linear transfer functions at each "node"). Given an input vector z (of dimension (,) and output vector y (of dimension ly), the output of the system with M parallel pathways is given by
M
WD
w x
y
(I)

---

where we have defined
Du W
Q
Σ S2
EII
(2)
a=1 d=11
and where pathway has Da - hidden layers and Da weight matrices. Each weight matrix W has shape No x N (where No = 1.)) In the product formula, we are assuming that matrices with a higher value of cl are to the left of matrices with smaller values of A schematic of this system given in Figure
As described below, we consider training this network using input-output vector pairs æ and y that have correlation matrix V USV", where we have used the singular value decomposition where is diagonal and U, V are orthogonal matrices. The gradient descent dynamics will lead to solutions U SV = S, which implies that in the steady state the pathway matrices Kg = U the singular values. A schematic of this decomposition is shown in Figure top row), adapted from 17.
S2 V must sum
Main question and illustration of main result: Given this description of how the network as a whole carries information about each input-output mode, we ask how this mode information is distributed across parallel pathways. We illustrate two possibilities information could be shared across multiple pathways (middle row) or (2) each mode's information could be exclusively carried individual pathways (bottom row). We show that case (2) occurs the limit of large network size and large depth due to finite-sizz size induced spontaneous symmetry breaking.
Figure 2 (I) each mode's
2.1 Learning dynamics under gradient descent
Using the same setting as in Saxe et al. {x', y } i ..2.....P with gradient descent on the squared loss
[16., train the network with set of P examples
P 1 1r'113
L
2
In the limit of small learning rate, the dynamics of gradient descent are approximated by the continuous time ordinary differential equations given below (for each pathway a):
d W (
d-1
II W
E TH
II
w
dl
where Li
W W Wh. Wh when 2 S l2 and I W when | > l2. ΣVz Elya"" and Σ=
E -.=
We assume without loss of generality that Σ I.
Define the initial conditions for the weight matrices as R (o) We take the components R be independently and identically distributed according to a normal distribution with zero mean and o² variance NZ. where Ná is the number of units in the output of that layer. We also denote the singular value decomposition of y USVT, where diagonal and U, are orthogonal matrices.
W
W
W!
0000000
00000000
W.
W!
W
W?
y
Figure I: Schematic of multi-pathway network. The first layer input feeds into multiple hidden layers in parallel, each of which a pathway through multiple hidden layers. The outputs of the final layer in each pathway linearly combine into the final output of the network.
3

---

Z
A
Grow
Propertes
Propertres
Rosts
Fly
Fly
Sarm
Modes
Potals
Petals
Canary
Satmong
Modes
moms
B
Nove
Move
Propettes
Roots
Roots
X
Leaves
Leeves
Potals
Petals
tems
Caneeyy
Salmon
Dask
Modes
soms
Grow
Move
Move
Nose
Properties
Roots
Propertres
00
Swit
Swim
Leaves
Leaves
Modes
Eont
Cannery
Salmon
Daal
Rosse
Figure 2: Example of multi-pathway learning. A) Example reconstructed from [17 showing the singular value decomposition hypothetical training set relating objects to properties. In the steady state, the network will learn the feature associations defined by the singular values. B) An example of learning in a shallow, two pathway network. In this case, the features will be arbitrarily split across the two pathways. C) An example of multi-pathwar learning in a deep network. In this case, the feature learning will typically concentrate into one of the pathways for each feature.
Gradient descent solutions for large hidden layer size
Here we construct solutions to the equations The general strategy is to choose coordinate systems for the neural representations in the vector space defined by the singular value decomposition. We then show that the solutions to the gradient descent equations (3) are diagonal in these coordinates.
3.1 Choosing orthogonal coordinates
Each hidden layer d in each pathway provides Nd dimensional representation of the input space from which the vectors r are derived. In order to construct solutions of the gradient descent equations of motion (3). we will choose coordinates on these representations that map to the space of singular values of the input-outpun correlation matrix E We construct these coordinates using the initial value matrices Rd Note that for large N'd. we must have
[R
Rd
R
which is exact Nd. xG. We the matrices U and V project the initial and final weight matrices onto the singular vectors: W! + W!V and W + -U' In addition, we use the initial state matrices R to define a coordinate system the singular value space for the intermediate representations. We define this map follows:
W! = a 2T R W.v
ifd =1
(5)
Wrd = o- -2dTT !R!)
R
T
W
...........
R
if1 <
< D.
(6)
WD. 2D. U W R
D.
R R
ifd Do
(7)

---

These coorinate transformations imply the following relations:
d-1
d-1 II
VJ
(R.)*
R
W.v
W
(8)
==1
==1
D.
D. Il
U
πI
W
R
VV
W
(9)
(=d+1
i=++1
which together imply
De EII
Ω = U
W
(10)
a F=1
In the singular value coordinate space, the gradient descent equations of motion for each a are now
d
W
II
W:
S 3-52 52
II W
(11)
di
(=d+1
Note that in the steady state we have
C2(t + 00)= =S
(12)
In the initial state we have in the large Ná limit:
W(O) I 8d W(0) D. = d
ifd < Da
(13)
ifd = Da
(14)
where 8d > the large Nod limit. So each W is diagonal (or zero) in the initial state. By construction, these matrices are all of dimension 1, x l,, where l, is the number of singular values of E
3.2 Diagonal solutions for the gradient descent equations
We propose the following ansatz for the solution:
W(1) = (1
(15)
where (t) are all diagonal matrices, with diagonal entries given by the vector (t), with compo- nents (L) a' where a is index over the singular values. The terms & + in the large Nd
limit. With this ansatz, in the N'd - oc limit, we have for each pathway a and component o:
Da
d-1
(1 = (r... S. Qa
(Tt)
(16)
i=d+1
where S,, and Ω2 are the diagonal components of S and Π respectively.
For each a, each of the (T&), for d Da have the same initial conditions, by symmetry define qaa = (ro) for all d < Da and Paa = (r to reduce these equations to
d
Jaa
Do P [Sa [.]
(17)
dt
d
Laa Da-1 - S2al -1a
T Paa
di
for each pathway and component a. The initial states of these variables are Gaa = 1 and Pao 0 M
in the large Nd limit. Moreover we have f. = EM K.a = E Paaq Pe- Importantly, these diagonal solutions are stable to off-diagonal perturbations, meaning that at finite size the small corrections 5d will decay
(see the section Simulations and Figure 4p.
Taking the ratio of each of the equations in (7 above yields the differential equation
dg
Poo
(18)
dpan
Joc
5

---

and thus these values are related by
Pea
qua
(19)
The variables Pan are thus sufficient to specify the solution and they obey the following equation:
d
(Vias + +1). Da-1
T di
SSa Stn]
(20)
with fl = = Pa= Paa (VDao
+1) Pa-1 Diffferent pathways will then be related by the relationship:
Puc + Da
dpoa
(21)
dpbo
/bba + D6-
which gives
1 Da
1
3
D3 3
Pi
2'
2
'2'
Pan
Pba
2
2 Pbo
1 Da
3 82
1 D,
F
(22)
2
2
where 2F is the hypergeometric function and we have defined the constant C. We have re-introduced the finite size initial conditions via daa. These are determined by the random matrix initializations and will vanish in the large network size limit. In the strict limit, the solutions therefore must be equal across all pathways. Finite size fluctuations in the initial state will break this symmetry.
The phase planes for a two pathway network with common depths D = D1 = D2 is plotted for various values of D in Figure 3 Each row represents the same set of 100 initial conditions drawn from zero mean normal distributions of given variance, with the larger variance in the top row. Note that the depth D increases, does the curvature of the trajectories away from zero. The sharper trajectories push the steady state solution to one where one or the other pathway dominates Ω2 = S.. so one can see the result already in the phase portrait that at larger depths the contribution Q, will come from a single pathway.
D
D
D=8
10
D=13
K
10
10 K1
K1.
Figure 3: Phase portraits for two pathway networks at different depths for a single a. S. = 10, D. Dr. Initial conditions drawn from mean 2d Gaussians with 0.1 (top) and a = 0.01 (bottom). All initial values are the same for each row. The steady state is governed by S. Kia + K2.. shown by the blue diagonal line. Notice that for deeper networks and larger initial fluctuations the trajectories tend towards more extreme values where one or the other pathway dominates for fixed initial condition.

---

Asymptotic approximation While the result is already qualitatively clear from the phase portrait, we can obtain an analytic statement. For more easily interpretable approach than analyzing hypergeometric functions (and a way of seeing the asymptotic behavior of the above equation), consider the case where Sa sufficiently greater than so that we can approximate
Paa qaa
(23)
(One can check this with a post-hoc self consistency condition. For the above relation to hold, S. will need be large.) Then we have
dpaa
Pan D2- -1
(24)
dpba
Pbo
which implies
1
1 2-
Pua
2- D.
Pho
Cabax
(25)
2
Da
2 Db
where Caba a constant that depends upon initial conditions and which further implies
2 - Da (Kan)
(Kan)2/Da- 2/
(Kba) 2/Dn
Caba
(26)
2 - Db
When Da, Do are large we have
1
-Cabac
(27)
Kac
Kba
for some C% Cabar In steady state, we have S., Kan Consider Sr = Sa Eozo,h Kea
Kan + Ktn- Then in the large depth limit we can write for Kbn
1
Cabaa
(28)
S hor
Kbo
the solutions of which are given by the quadratic equation
Ca Kba + +2 - Sa K - -S
(29)
which are
Cahn S. 2 VlC
aba S.a - 22)2+ +42%, Co
Kbo
(30)
20" aba
In the limit Clabal
+ co and in the domain
Kio E (0)))) these will be
S 2
S
Kbn + siig(CCooo)
(31)
2
and thus one pathway or the other dominates the singular value for each a, depending upon the sign of Caba. As mentioned above, the strict limit the constant Caber - leaving symmetric solutions across the pathways. In the finite size limit, fluctuations in the initial state will drive Cata larger values, biasing the network one pathway or the other. We note that real application, initial conditions are not necessarily drawn from the distributions that we assume, and we expect effectively larger initial fluctuations than our initial state would imply, e.g. by choosing the initial values of each weight matrix from N(0, o), rather than scaling with network size (see the examples Simulations). Such larger fluctuations would only enhance the effect of the symmetry breaking.
0
2
3
4
5
G
9
Figure 4: Learning dynamics leads to diagonal representations in each pathway. An example realization of learning in K. for a single pathway. In the initial state the K,, starts as a uniform random matrix and quickly becomes diagonal after few time steps.
7

---

Figure 5: Learning dynamics in two pathway networks with increasing depth. D2 D Left right, top bottom are D 2, 4, and respectively. The left side of each sub- sub-panel shows trajectories of the matrices K. and Kg in the singular value space. The right side of each sub-panel shows the final values for K, and Kg in the singular value space. Note how with larger depths, the features have a stronger tendency to concentrate on one pathway or the other. Na = No 1000 and 0.01.
Simulations
We demonstrate our results with numerical simulations of networks with two pathways and multiple depths. For these examples use the same number of layers per pathway and N₁ N2 1000. The initial state of the weight matrices is drawn from zero mean normal distribution with fixed standard deviation dimensional identity matrix. The output vectors y are 15-dimensional and are the rows of the matrix:
0.01. The input vectors æ are 8-dimensional and are the rows of the 8-
1
0
0
0
0
1
0
0
0
0
0
1
1
0
0
0
0
1
1
0
0
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
(32)
1
0
1
0
0
1
0
0
0
0
0
1
0
0
100
1
0
0
0
0
1
0
1
0
1
0
1
1
0
1
0
1
0
Gradient descent is performed over 1000 epochs with learning rate 4, 0.01. These simulations are not compute intensive and are easily performed on a standard modern desktop or laptop. All code for simulations and figures is available on GitHub at /htps:: //g: thub. wwwm/oomeeeeeeeeeeee htt.AAAlennnnsiiii tute/ Multipathway BerpaathwwyNNeurIIe _eeurruurr IPS2022 Figure shows an example of the matrix Kg for specific pathway a, which initialized as a uniform random matrix, becoming diagonal in the singular value space. Figure 5 shows an example of learning dynamics for networks with different depths. For the network of depth (for which "pathway"
an arbitrary concept, as there is only one hidden layer), singular values are more or less divided evenly across the pathways. This symmetry is already broken for networks of depths 3 and 4. Even by depth 7, the largest singular values are concentrated on one or the other pathway. Figure shows two different initializations, demonstrating stochasticity. One can see that the largest singular values are typically in one pathway or the other, but not always the same pathway.
Importantly, our analytic results apply strictly to the linearized network. Since the standard situation with deep networks is to use some form of nonlinear activation we test our results empirically using either of Tanh or ReLU. The general trend remains the same when using these nonlinearities. The results are shown in Figures 7 and 8 for ReLU and Tanh, respectively. Simulation parameters are as above with the exception that the corresponding nonlinearity is used as an activation function following each layer of each pathway and we train for longer (as shown in each Figure).

---

Figure 6: Learning dynamics from different initializations in two pathway networks. Da D3 = Na = No- = 1000 and a = 0.01.Left and right show two different seeds of the random initialization. The left side of each sub-panel shows trajectories of the matrices K, and Kg in the singular value space. The right side of each sub-panel shows the final values for K2 and K2 in the singular value space. Note that in the different initializations, the final assignment of the features to each pathway is random.
D=t
Figure 7: Learning dynamics in two pathway networks with increasing depth, using the ReLU nonlinearity. Da D, D. Left to right, top to bottom are D = 2,,344 4, and 7, respectively. The left side of each sub-panel shows trajectories of the matrices K, and K2 in the singular value space. The right side of each sub-panel shows the final values for K₂ and K, in the singular value space. Note how with larger depths, the features have a stronger tendency to concentrate on one pathway or the other. Nº N₂- 1000 and 0.01.
2=2
0=3
D-7
Figure 8: Learning dynamics in two pathway networks with increasing depth, using the Tanh nonlinearity. D2 = D2 = D. Left to right, top to bottom are D = 2, 3, 4, and 7, respectively. The left side of each sub-panel shows trajectories of the matrices K, and K2 in the singular value space. The right side of each sub-panel shows the final values for K, and K, in the singular value space. Note how with larger depths, the features have a stronger tendency concentrate one pathway or the other. N₂ = N₁ = 1000 and a = 0.01.

---

5 Discussion
Many biological neural systems have several computational pathways that are thought to mediate different aspects of sensory processing, prominant examples being the "what" and "where" pathways of the macaque visual system 4 7 [0], the "local" and "global" visual motion pathways [ and the
parallel anatomical pathways in the mouse [8] Here we have shown, using the linearized framework of (16), that depth alone can have a profound impact upon learning dynamics in systems with multiple channels by separating features in a learned data set by pathway. It is useful to recall again the observation made for the AlexNet model [9] which featured two pathways result of building the architecture around two GPUs. In that case, the authors noted that each pathway learned different large scale features of the data (one largely color, the other largely texture).
We have examined the supervised case using linear regression on the output features of the network. In this framework power iteration effect from multiple layers of computation (which can be seen in the weight matrix factors in equation 3 enhances the inherent "explaining away" effect in regression (as each pathway's 's features contribute to accounting for the input-output relationship) so that with increasing depth there is a tendency for each singular vector to concentrate on one pathway. Note that this does not imply that separate singular components will prefer different pathways in general. is interesting to consider the parallels and differences between this model and the classic Oja rule that leads to PCA [3]. In that case, the learning dynamics magnifies the largest principal components of the input space and competition is introduced via weight normalization. This leads to the weights matching the largest principal components of the input data, i.e. the singular vectors of Σg. (This happens in a deterministic fashion, although see [12] for a higher order generalization that does not lead to deterministic selection of the largest components.) The mechanism we describe here operates in the supervised case and decomposes Eyg.
While we have used very specific initialization scheme for mathematical purposes, posit that (and have provided an example in which) our results are more generally applicable. In some cases this is easy to show as extension of our work. This work is an analysis of the analytic properties of deep networks in the infinite width limit. As such, care must be taken in considering these results in the context of systems in which corrections due to finite size are large. The assumption of linearity is also important. Deep networks in practice have non-lineariin will push the system operate in a regime that maximizes the dynamic range of the response. For function like Tanh, this should be at the inflection point. If this is the case, the network should act in roughly linear fashion. This is just high-level argument, however, and nonlinearities may be responsible for deviations from the results described here, although we have demonstrated in our examples that common lincarities show the same empirical dynamics.
One can argue that learning dynamics
One obvious reason for having an architecture with multiple pathways is to create divergent represen- tations for different cost functions from common set of lower level features. In case we consider an alternative in which all pathways subserve the same cost function. If the result is to separate features, why would such an architecture be desirable? One answer may be that multiple pathways would serve as a form of regularization. Not only does this reduce the space of model complexity, but through the mechanism we have demonstrated this can also lead separation of features, and perhaps more easily learnable representations for downstream tasks. This may be a useful component of pre-training procedures. It may be that multiple pathways are source of architectural efficiency, and particularly useful when the system has few components (suggesting a reason why the mouse architecture appears more parallel than mammals with much larger brains). Quantifying these potential benefits would entail a proper model comparison across different architectures and examining performance as well as learning speed. Looking towards future work, is important to keep in mind that biological systems may be subject to different types of cost functions, and the dynamics of those may be quite different than what we see here.
As this is an analytic approach to the theory of deep neural networks from a dynamical systems perspective, anticipate no negative societal impacts.
Acknowledgments and Disclosure of Funding
We thank Gabriel Koch Ocker, Blake Richards, and Shahab Bakhtiari for helpful discussions. We thank the Allen Institute founder, Paul G. Allen, for vision, encouragement, and support.
10

---

References
[1] Shahab Bakhtiari, Patrick Mineault, Tim Lillicrap, Christopher C. Pack, and Blake A. Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. bioRxiv, 2021.
[2] Santiago A. Cadena, George H. Denfield, Edgar Y. Walker, Leon A. Gatys, Andreas S. Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLOS Computational Biology, 15(4))e100889, Apr 2019.
[3] James J. DiCarlo, Davide Zoccolan, and Nicole Rust. How does the brain solve visual object recognition? Neuron, 73(3):415 Feb 2012.
[4] D. Felleman and D C van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex, 1 1:1-47, 1991.
[5] Barrie J. Frost. A taxonomy of different forms of visual motion detection and their underlying neural mechanisms. Brain, Behavior and Evolution, 75(3):218 235, 2010.
[6] K Fukushima. Neocognitron:
self organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-200 1980.
[7] Melvyn A. Goodale and A.David Milner. Separate visual pathways for perception and action. Trends in Neurosciences, 15(1):20-25, 1992.
[8] Julie A Harris, Stefan Mihalas, Karla E Hirokawa, Jennifer D Whitesell, Hannah Choi, Amy Bernard, Phillip Bohn, Shiella Caldejon, Linzy Casal, Andrew Cho, Aaron Feiner, David Feng, Nathalie Gaudreault, Charles Gerfen, Nile Graddis, Peter Groblewski, Alex M Henry, Anh Ho, Robert Howard, Joseph E Knox, Leonard Kuan, Xiuli Kuang, Jerome Lecoq, Phil Lesnar, Yaoyao Li, Jennifer Luviano, Stephen McConoughey, Marty Mortrud, Maitham Nacemi, Lydia Ng, Seung Wook Oh, Benjamin Ouellette, Elise Shen, Staci A Sorensen, Wayne Wakeman, Quanxin Wang. Yun Wang, Ali Williford, John W Phillips, Allan R Jones, Christof Koch, and Hongkui Zeng. Hierarchical organization of cortical and thalamic connectivity. Nature, 575(7781): 2019.
[9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[10] Mortimer Mishkin, Leslie G. Ungerleider, and Kathleen A. Macko. Object vision and spatial vision: two cortical pathways. Trends in Neurosciences, 6:414-417, 1983.
[11] Aran Nayebi, Nathan C. L. Kong, Chengxu Zhuang, Justin L. Gardner, Anthony M. Norcia, and Daniel I.. K. Yamins. Unsupervised models of mouse visual cortex, bioRxiv, 2021.
[12] Gabriel Ocker and Michael Buice. Tensor decompositions of higher-order correlations by nonlinear hebbian plasticity. In Advances in Neurul Informution Processing Systems, volume 34, page 11326-11338 11339. Curran Associates, Inc., 2021.
[13] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15(3):267-273, Nov 1982.
[14] Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, 2011):001- nov 1999.
[15] Sudip Roy. Jeff Dean, Sanjay Ghemawat, Ryan Sepassi. Hyeontack Lim, Michael Isard, Paul Barham, Yonghui Wu, Laurent Shafey, Aakanksha Chowdhery, et al. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 4, 2022.
[16] Andrew M. Saxe, James L.. Mcclelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In in International Conference on Learning Representations, 2014.
11

---

[17] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory of semantic development deep neural networks. Proceedings of the National Academy of Sciences, 116123):11537-11546 2019.
[18] Jianghong Shi, Bryan Tripp, Eric Shea-Brown, Stefan Mihalas, and Michael Buice. Cnn mousenet: A biologically constrained convolutional neural network model for mouse visual cortex. bioRxiv, 2021.
[19] Quanxin Wang, Enquan Gao, and Andreas Burkhalter. Gateways of ventral and dorsal streams in mouse visual cortex. Journal of Neuroscience, (10t:/ieennee335:1119939 31(5):105-1118,, 2011.
[20] Quanxin Wang, Olaf Sporns, and Andreas Burkhalter. Network analysis of corticocortical connections reveals ventral and dorsal processing streams in mouse visual cortex. Journal of Neuroscience, 32(13):4386 43999, 2012.
[21] Daniel Yamins. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 2014.
[22] Daniel I.. K. Yamins and James J. DiCarlo. Using goal-driven deep learning models under- stand sensory cortex. Nature Neuroscience, 19(3):356-365, Mar 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] Code for simulations and figures is available at /tttts:://ggtiiie http:/gttththhoooooo.oo wal.ccoommlllllll Aww.elnneennnnt.... Multipathway,
NNeur Neur IPS2022]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
12

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "tdD5bERt_2E", "reviewer": "Reviewer_8s1V", "summary": "\nThis paper analyzes the learning dynamics of linearized deep networks with parallel pathways driven by the same cost functions. The analysis was done from a dynamical system perspective in the tradition of Saxe et al.  on linear deep networks in the infinite width limit. An interesting conclusion is that at large depth, different pathways seem to compete and learn sharply distinct and diversified feature representations. ", "questions": "Issues related to weaknesses should be addressed if possible.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Fairly detailed and involved theoretical analysis, and the conclusion is verified by simulation results.\n2. Results potentially reveal and illuminate a potential functional purpose of multiple pathways in the nervous systems. \n\nWeaknesses:\n1. The analysis is on linearized networks, with the nonlinearity removed, so whether this analysis is relevant to deep convolution neural networks and their power is open to question.\n2. At some level, the linearized network seems to be behaving like a competitive network performing principal component analysis and independent component analysis, which would also arrive at similar \"conclusions.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "McpBTkhXBk4", "reviewer": "Reviewer_ykV7", "summary": "This works aims to theoretically study the learning dynamics of deep linear networks with multiple parallel pathways (similar to parallel pathways of processing in the cortex such as dorsal/ventral, magnocellular/parvocellular pathways, etc.) in the infinite width limit. Despite the expressivity of such networks being subpar to deep learning models used for practical purposes, deep linear networks, and their associated objective functions that the authors study possess learning dynamics that show interesting nonlinear behavior. Here the authors analytically show that the different parallel pathways learn to specialize in representing different features of the training set as a function of the depth of the network and demonstrate the finding with numerical simulation. This work is exciting and is also relevant to recent self-supervised learning methods that possess parallel pathways and have been shown to mimic biological visual processing. ", "questions": "- This is rather a comment that could be addressed by future work and not a suggested extension of the current work. The authors motivate the importance of studying the learning dynamics of deep networks with parallel pathways through their connection to biological information processing. \n- However, given that there are (potentially) more biologically plausible learning rules than gradient-based error propagation (I acknowledge this area to be debated) such as local learning and Hebbian learning, one would think that studying the learning dynamics of deep networks with (or without) parallel pathways with such learning algorithms that are potentially more aligned to learning in the brain would be extremely interesting for the neuroscience community.", "limitations": "Yes, the authors have clearly stated the limitations of this study (restricted to deep linear networks in the infinite width limit for a particular initialization scheme, whereas commonly used deep networks in practice don’t necessarily obey all these assumptions).", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed work is an excellent extension of prior impactful work on studying the learning dynamics of deep linear networks by considering parallel pathways that are abundant in biological vision (e.g. dorsal/ventral, magno/parvocellular, etc.).\n- I appreciate the well-designed figures that make it easier to grasp key concepts mentioned in the paper. I found the phase portraits in Fig. 3 to be really helpful in qualitatively interpreting the main result of the concentration of different features to different parallel pathways as a function of processing depth.\n- The main results from this paper provide valuable insights into developing computational models of parallel circuits in biological neural networks. \n- It is great that the authors are planning to make the code for all numerical simulations and figures available publicly on Github.\n- Overall, I find the paper to be well written and organized with well-designed and clearly labeled figures wherever necessary.\n\nWeaknesses:\n- There are quite strong assumptions in place that may confine the space of models that the proposed results apply to; e.g. absence of nonlinearities, specific initialization scheme, infinite width limit.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IQKjx2lMZJw", "reviewer": "Reviewer_pqBi", "summary": "Inspired by mammalian brain architecture the authors the impact of having parallel pathways in artificial neural networks. They derive results analytically and conduct simulations. They show that different features of the training set typically concentrate in one of the pathways arguing for diversified representation with specialized pathways. ", "questions": "Can you discuss more the non-linear case or perhaps conduct simulations?\n\nHow could one evaluate that diverse parallel pathways are indeed helpful in ANNs - is there some example (perhaps related to what and where streams) that could be turned into a quantifiable experiment? ", "limitations": "I think these are overall addressed reasonably well. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides both analytical results and simulation, making an interesting theoretical contribution.\n\nTo make the problem analytically tractable the authors use linear neural networks. It is not clear how the use of non-linearity would affect the results. While this might not be possible to figure out analytically, it might be possible in simulations. \n\nThe authors discuss possible utilities of diverse pathways, but the contribution would be much stronger if they would be able to show this using experiments. \n\nIt is not clear how different assumptions, such as initialization would affect the results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept