A mechanistic multi-area recurrent network model of decision-making
Authors: Michael Kleinman
Chandramouli Chandrasekaran
Jonathan Kao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computational Neuroscience
Recurrent Neural Network
Interpretability
Decision-Making
Multi-Area Computation

Abstract: Recurrent neural networks (RNNs) trained on neuroscience-based tasks have been widely used as models for cortical areas performing analogous tasks. However, very few tasks involve a single cortical area, and instead require the coordination of multiple brain areas. Despite the importance of multi-area computation, there is a limited understanding of the principles underlying such computation. We propose to use multi-area RNNs with neuroscience-inspired architecture constraints to derive key features of multi-area computation. In particular, we show that incorporating multiple areas and Dale's Law is critical for biasing the networks to learn biologically plausible solutions. Additionally, we leverage the full observability of the RNNs to show that output-relevant information is preferentially propagated between areas. These results suggest that cortex uses modular computation to generate minimal sufficient representations of task information. More broadly, our results suggest that constrained multi-area RNNs can produce experimentally testable hypotheses for computations that occur within and across multiple brain areas, enabling new insights into distributed computation in neural systems.

A mechanistic multi-area recurrent network model of decision-making
Michael Kleinman' Chandramouli Chandrasekaran"* Jonathan C. Kao
UUniversity of California, Los Angeles Boston University
michael. wolkkeeinnne www.eeemennlla.....mm.. cchandr1@bu. edu kaoßseas ecla.edu
Abstract
Recurrent neural networks (RNNs) trained on neuroscience- tasks have been widely used as models for cortical areas performing analogous tasks. However, very few tasks involve single cortical area, and instead require the coordination multiple brain areas. Despite the importance of multi-area computation, there is limited understanding of the principles underlying such computation. We propose use multi-area RNNs with neuroscience-inspiree architecture constraints derive key features of multi-area computation. In particular, we show that incorpo- rating multiple areas and Dale's Law is critical for biasing the networks to lear biologically plausible solutions. Additionally, we leverage the full observability of the RNNs show that output-relevant information is preferentially propagated between areas. These results suggest that cortex uses modular computation to generate minimal sufficient representations of task information. More broadly, our results suggest that constrained multi-area RNNs can produce experimentally testable hypotheses for computations that occur within and across multiple brain areas, enabling new insights into distributed computation in neural systems.
1 Introduction
Decision Decision-making, multisensory integration, attention, motor control, and timing emerge from the coordination of multiple interconnected brain areas [1H)). While neural activity particular can contain behaviorally relevant signals, such as choices or percepts, it is often unclear if these signals originate within the neural bases these behaviors necessitates an understanding of the intra- and inter-area dynamics, that is, how neural activity evolves within and between brain areas. However, hypotheses for how distinct brain-area dynamics and connectivity relate to computation. To address this gap, we use multi-area recurrent neural networks (RNNs) model, probe, and gain insight into how behaviorally relevant computations emerge within and across multiple brain areas.
inherited from upstream brain areas (6]. Understanding the
currently lack clear
Optimized feedforward and recurrent neural networks have been used for machine learning but also emerging tools to model computations associated with visual [T. IЛ, cognitive [22] timing task inputs into behavior-related outputs through recurrent dynamics. Its artificial units often exhibit heterogeneous responses and population dynamics observed in brain and motor tasks [5] (77-20) If artificial units resemble cortical neurons, RNNs are subsequently
[ navigation
14 and motor tasks [5 16] RNNs transform experimenter-desigeee
implicated in cognitive
analyzed propose hypotheses for how local computation occurs in brain area 12 18 21. In comparison to direct experimental recordings, an important advantage of RNNs is that the activity
Joint senior author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Inputs
Multi-area RNN
Outputs
Rlght larg
Decsioo vartable
Left barg
Terg
Check
Slim
Check Targ Stim
on on
off
Figure 1: Task. RNN configuration. The RNN receives 4 inputs. Two inputs indicate the identity of the left and right targets, which can be red or green. These inputs are noiseless. The other two inputs indicate the value of the signed color coherence (proportional and negative signed color coherence (proportional to amount of green in checkerboard), We added independent Gaussian noise to these signals (see Appendix A.2 The network outputs two analog decision variables, each of which indicates evidence towards the right target (solid line) or left target
amount of red in checkerboard)
(dashed line). A decision is made the direction of whichever decision variable passes a preset threshold (0.6) first. The time at which the input passes the threshold is defined to be the reaction time.
of all artificial units and their recurrent connectivity fully observed. It is therefore possible to engineer [22] and reverse engineer [23 RNNs by analyzing their activity and recurrent connectivity, providing mechanistic insight into cortical computation
211
Traditionally, RNNs have provided insight into local computations, and there has been limited insight into multi-area computation 7 19]. To study multi-area computation, we explicitly constrained RNNs to have multiple recurrent areas, which we refer to as multi-area RNNs. We used these multi-area RNNs to study decision- mrriiinn-maknn,. cognitive process known to involve multiple areas including the prefrontal, parietal, and premotor cortex 7 22 28 Multi-area RNNs enable us to investigate several questions. Most broadly, what are the roles of within-ares dynamics and inter-area connections in mediating distributed computations? How does the dimensionality and dynamics of neural computation differ across areas? What role do inter-area feedforward and feedback connections play in propagating information and rejecting noise? How do intra-area dynamics and inter-area connections coordinate solve task?
We use multi-area RNNs to study these questions a decision-making task where premotor cortex and upstream areas are known to perform distinct computations. We trained multi-area RNNs to perform perceptual
task (Checkerboard Task) and compared their activity to monkey neuron recordings from the dorsal premotor cortex (PMd). We found that, when incorporating Dale's 's law and mmatomically-inoommee levels of feedforward inhibition into training, PMd-resembling dynamics emerged in multi-area RNNs. Specifically, the multi-area RNN's output area (1) resembled PMd in single unit statistics and neural population activity, and (2) only retained the "output relevant" signals. Inter-area connections preferentially propagated these output relevant signals while attenuating output irrelevant signals. Our models and analyses provide a framework for studying distributed computations involving multiple areas in neural systems.
2
Motivation: Decision-making involves multiple brain areas
2.1 Checkerboard Task
shown in Fig. monkey was first shown left and right targets whose color (red and green) was random on each trial. The monkey was subsequently shown a central static checkerboard composed of red and green squares. The monkey was trained to discriminate the dominant color of the static checkerboard and reach to the target matching the dominant color. Since the target colors were random each trial, this task separates the reach direction decision from the color decision 2 This task enables studying how information related to the selection of the color of the target and information related to the direction of the reach is represented.
In the "Checkerboard Task"
24
26
2.2 PMd Data during Checkerboard Task
We analyzed the activity of neurons from the dorsal premotor cortex (PMd), an area associated with somatomotor decisions, in monkeys performing the Checkerboard Task (30] Neural activity

---

PMd data
PCs
dPCA and decoding C
PSTHs
Choice probability
Varience:
Varannc wwwaoooaageo
80.25
direction
Targels
Check on
0.75
Right
1.0
0.5
PC₂
000
0
PC,
Siyned coherance
0.0
Check on
200m
RNN data
Area
Area 2
Area 3
CCA 0.38
CCA 0.55
CCA 0.73
Frecton dPAC variance explained
0.5
color
context
Fight green
0.4
0.3
Loft green
0.2
PC2
0.1
PC
77.%%
Left x Right
99.2%
94.2%
h
Area
Area
Area
g
1.0
0.9
Decode
0.7
0.8
Area
Area 2
Area
Direclion CP
0.5
Check
Chick on
Color CP
0.6
Figure 2: PMd-resembling dynamics emerge in neuroscience constrained RNNs. (a) PMd neural trajectories the top 3 PCs. Color reflects signed color coherence, with darker shades red (green) indicating more red (green) checkerboards. Right (left) reaches are denoted by solid (dotted) lines. (b) (Top) Variance captured by dPCA axes for the color decision, target configuration (context), and direction decision. (Bottom) Decode accuracy of the direction decision, color decision, and context in PMd sessions with U-probes and multiple neurons. (c) Representative PMd PSTHs aligned to checkerboard onset. (d) Direction and color choice probability (CP) for all recorded PMd units. (e) Neural trajectories in the top principal components for each RNN area. (f) Variance captured by dPCA axes for color, context, and direction. (g) Non-linear tSNE embedding of peri-movement activity in each area. Each dot is trial, with red or green denoting the color decision and '.' or 'x' denoting the direction decision. Unlike Areas and 2, Area only had two clusters separated based on the direction decision.
(h) Decode accuracy of direction, color, and context in all three areas. (i) Example PSTHs in each area. (j) Choice probabilities for units in all areas (pooled over 8 RNNs).
in PMd principally reflects the direction decision (left or right) and has minimal representations associated with the dominant color of the checkerboard (red or green) [26 30 31. To summarize this phenomenon, we show the principal components (PCs) of the PMd neural population activity
in Fig. 2b. These PC trajectories separate based the eventual reach direction (right reaches in
solid, left in dotted), but not the color (red and green largely overlapping). We identified principal axes via demixed PCA (dPCA (2)) that maximized variance related to the target configuration

---

(context), color decision, and direction decision (see Appendix B.4). The direction axes captured significant variance (26.7%) while the color and context axes captured minimal variance (0.7%, 0.5%. respectively), as shown in Fig. 2 It is possible, however, that there is direction-dependen color variance that is averaged away during marginalization when computing the dPCA variance (2) Given simultaneously recorded data, more appropriate measure of representation is the decode accuracy of direction, color and context. Across sessions where we analyzed multiple simultaneously recorded units from U-probes, the direction decision could be decoded from PMd activity significantly above chance (accuracy: 0.89, < 0.01, bootstrap), but the color decision and context decode accuracy were not significantly above chance in any session (overall accuracies: 0.52 and 0.52, respectively, Fig. Dp bottom).
Single neurons also had minimal color separation in individual PSTHs (c.g., Fig. 2.. To summarize this effect in single neurons, we computed the choice probabilities (CPs) reflecting how well the direction decision (direction CP) and color decision (color CP) could be decoded. PMd units generally had near chance color CP (0.5), but moderate to high direction CP, as shown in Fig. 2t. Together, these results demonstrate that PMd largely represents direction-related signals, but not the color decision or target configuration context. Since PMd activity minimally represents the color of the checkerboard or the target configuration, we reasoned that checkerboard and target inputs are transformed into direction signal upstream of PMd and that multiple brain areas are necessary for solving this task. Brain areas, including the dorsolateral prefrontal cortex (DLPFC), and the ventrolateral prefrontal cortex (VLPFC), have been implicated in related sensorimotor transformations
33 35
3 Multi-Area RNN Training Details
We trained RNNs of the form
x(( (() + Wl) + Win + brsc t
(1)
where r(t) = relu(x(t)). is a time-constant of the network, W E NNN defines how the artificial neurons are recurrently connected, bree R defines constant bias, Win Wn E g maps the RNN's ' inputs onto each artificial neuron, and is the recurrent noise. The output of the network is given by linear readout of the network rates, i.e.,
(t) W ott)
where W... out E NN maps the network rates onto the network outputs. For 3-area RNN, W ren: is defined through the following block matrix
W
W21
0
W rec
W12
12
W22
22
W
32
0
W23
W
where Wa; refer to the recurrent connections of area i, and feedforward connections, and Wi.-1 refer to the feedback connections. Feedforward and feedback connections were only allowed between adjacent areas. Task inputs were defined to project onto the first area, and outputs were read out from the final area. In the rest of the text, we primarily focus on 3-area RNN that had approximately 10% feedforward and 5% feedback connections between areas, based on projections between prefrontal and premotor cortex in a macaque atlas [36] The network also constrained to follow Dale's law, in Song et al. (37) The RNN processed the target context and checkerboard inputs to output decision variables reflecting accumulated evidence for left and right decision (Fig. D. Further details are discussed in Appendix A.2
the convention that W Wi,41 refer to
Results
Because the Checkerboard Task involves multiple brain areas, we reasoned that a single-area RNN would not resemble PMd recordings. We first trained traditional single-area RNNs to perform the Checkerboard Task. We found that these RNN representations mixed color and direction information, as summarized in Appendix Fig- g and therefore did not resemble PMd activity. This led us to study multi-area RNN models performing the Checkerboard Task, which turn out to accurately model PMd activity.

---

d
مخه
000000
Drection soccuraa raoo
Color accaracy
0.76
0.75
0.5
101
50
100
30
100
10
20 60
% to
E
% foedback
areas
(10)
A <(x10-))
Figure 3: PMd-resembling dynamics emerge in neuroscience constrained RNNs. (a) We trained 3-area RNNs without explicit excitatory (E) or inhibitory (I) neurons. Inputs projected onto Area I, and outputs were read out from Area 3. We varied the percentage of feedforward connections and computed the color and direction accuracy in Area 3. At 1% feedforward connections, color could still be significantly decoded above chance. Dots are the mean across networks and error bars are s.c.m. For significance, * is 0.05, ** is p 0.01, and $** is p < 0.001 (with appropriate correction for multiple comparisons). We incorporated Dale's law with 80% E, 20% neurons into subsequent sweeps, (b) We varied the percentage of feedforward E-to-I connections. Minimal representations with chance color decode accuracy emerged when the percentage of feedforward E to connections was 2% less (feedforward
was fixed 10%). (c-d) Color information was relatively robust to feedforward E-E connections and feedback connections. (e) At least areas were required for the RNN's last area resemble PMd dynamics. (f) 3-area RNNs with neurophysiological constraints had minimal representations that were generally robust to machine learning hyperparameters. The only exceptions were when the number of units was relatively small, or the learning rate was relatively large.
4.1 PMd-like representations emerge in optimized multi-area RNNs with neuroscience constraints
Given the anatomical and physiological evidence suggesting that multiple brain areas are implicated in the CB task, hypothesized that the last area of an optimized multi-area RNN would more closely resemble PMd, receiving transformed direction signals computed using the checkerboard coherence and target configuration from upstream areas. We trained multi-area RNNs perform the Checkerboard Task as described in Section
The 3-area RNN had qualitatively different population trajectories across areas, shown in Fig. 2 Area 1 had four distinet trajectory motifs corresponding to the four possible task outcomes (combinations of left vs right and red vs green decisions). PC₁ primarily varied with direction, while PC2 varied with both the target context and red versus green checkerboards. In contrast, Area 2 and Area population trajectories primarily separated on direction, not color, like in PMd. Area 3 trajectories most strongly resembled PMd trajectories (canonical correlations, r = 0.38, 0.55, 0.73 for Areas 1, 2, and 3; see Appendix B.7
We quantified the variance captured by dPCA principal axes for the context, color, and direction axis. We found that color axis variance decreased in later areas (Area 1: 5.6%, Area 2: 0. 0.13%, Area 3: 0.07%, Fig. 2h. In contrast. Area had the largest direction axis variance (Area 1: 30.9%. Area 18.2%, Area 3: 48.5%, Fig. 2). An important assumption of dPCA is that the neural activity can be decomposed as a sum of terms that depend solely on particular task variables (38. The color variance found by dPCA indicate that color, on its own, did not account for large fraction of the overall neural variance. However, it is possible there significant color variance within a reach direction that dPCA, lincar dimensionality reduction technique, does not capture.
As we are interested in whether the color information is contained in the representation, a more appealing measure is decode accuracy. If the color of the target can be decoded from the representation of neural activity, then color information present in the representation. We performed nonlinear dimensionality reduction via t-distributed stochastic neighbor embedding (tSNE) (39 shown in Fig. D These results suggest that Areas and 2 contain color information, but Area does not (color decisions overlap). We decoded the color decision and context (target configuration) from RNN units in each area (Fig. 2h, Area 1, 2, and color accuracy: 0.93, 0.76, 0.51, and Area 1, 2, and 3 context accuracy: 0.99. 0.87, 0.54). Area I and 2 had above chance context and color decode
5

---

accuracies (p < 0.01/9, 1-tailed t-test with Bonferroni correction), while Area 3 color and context decode accuracies were near chance, and most similar to PMd (Fig. Dh. color: p 0.05, context:
p 0.024). The direction decision could be decoded significantly above chance in all areas (Fig. 2h, p < 0.01/9) We also observed that Area unit PSTHs more closely resembled PMd neuron PSTHs (e.g., Fig. 2), and color CP progressively decreased in later areas (Fig. 2 Area 3, like PMd, had many neurons with moderate to high direction CP, but low color CP.
We tested how robust these results were to architecture and hyperparameter selection In particular, we quantified how well color could be decoded in the multi-area RNN's last area across several hyperparameter settings. We found that architecture impacted whether optimized multi-area RNNs had PMd-like minimal representations. In particular, we found that PMd representations emerged when we incorporated anatomical and neurophysiological constraints: Dale's 's law, empirical levels of feedforward inhibition, and at least 3 areas (Fig. Bh-e). When we varied machine learning hyperparameters, we found that our results were generally robust: multi-area RNNs had PMd-like representations in their last area over a wide range of hyperparameter settings (Fig. ET Together, this constellation of results shows that Area 3 of the multi-area RNN recapitulates key features of PMd activity, making this RNN a candidate model of multi-area recision-making in the Checkerboard Task.
In the next sections, we leverage the full observability of this biologcalll-plausibbe multi-area RNN to understand the mechanisms in different areas of the network and also how the network filters color information while propagating direction information.
4.2 Separation of the color and direction decision in Area
What are the key computational features of how the multi-area RNN represents color and direction information in the Checkerboard task? We first focused our analysis on Area 1, which uniquely
has substantial variance for both color and direction decisions (Fig. 22)) implying a central role in
computing the direction choice. We performed dPCA to identify demixed principal components related to the RNN inputs (coherence and context) and decisions (color and direction) 3p
We found
demixed components that separated information related to coherence, context, the color choice, and the direction choice (Fig. !b), consistent with these quantities being decodable from activity (Fig. Bh). We subsequently identified the context, color, and direction axes as the dPCA principal axes (unit norm, analogous to PCA eigenvectors), which combine the demixed components (analogous to PCA scores) to reconstruct neural activity
[2]
We projected RNN activity and input representations onto the principal axes for context, color, and direction (Fig. Fl). We found that the context and color axis both responded to context and color inputs, and overall trajectories represented both context and color information. This suggests that color and context information are mixed in Area 1. In contrast, the direction axis strongly represented the direction choice, but did not strongly represent context or color (Fig. t, right). Strikingly, context and color inputs had nearly zero projection on the direction axis (Fig. dh. right, opaque traces at O). Consistent with these observations, we found the color and context axes were highly overlapping (dot product: 0.93), indicating that context and checkerboard variance are mixed in Area activity. In contrast, the direction axes was closer to orthogonal to the context and color axes (overlap with color and context: 0.14 and 0.09, respectively).
These conclusions were upheld when we performed targeted dimensionality reduction (TDR), where we found (1) a direction axis separating left and right choices, with negligible input projections, and (2) that color and context representations were mixed (Appendix Fig. [4] Further, this structure was unique for PMd-like 3-area RNNs. In single-area RNNs, dPCA identified nearly orthogonal context, color, and direction axes, with trajectories that separated almost exclusively based on context, color,
and direction, respectively (Appendix Fig. 13)).
This Area representation has an important property: the direction choice is represented robustly on nearly orthogonal axis that has close to zero context and color input projections, (F) This is not trivial: counter-examples, single-area RNNs use direction axes that have context and color input projections (Fig. B)). while the direction axis of an unconstrained 3-area RNN (without anatomical
"These sweeps over different random initializations and different parameter settings consisted of the most significant computational cost, roughly requiring 500 CPU hours on AWS, with each model training in approxi- mately 1-2 hours.

---

a
dPCA context axis
Color axis
Direction axis
6.
activity
20:
Projections (au)
5
10-
Inputs
Check
Targ on
b
97.7% variance captured
dPCA axes
d
Target checkerboard inputs
right
Target couio
context
Taig
Targ on epoch
Oracc steckreeee lppat//pt
left red
directon axis
checiaanonndd
PG)
Targer contg
ctx
right red
dir
Figure 4: Separation of direction and color in Area (a) The context, color, and direction axis correspond the dPCA principal axes, which are not constrained to be orthogonal. Trajectories for different contexts and colors were separable on both the context and color axis. In contrast, the direction axis separated primarily on chosen direction. The RNN input representation had strong projections on the context and color axes, but not the direction axis. (b) Top PCs of Area 1 activity, which capture 97.7% of the Area variance. In the targets on epoch, the trajectories separate two regions corresponding to the two potential target configurations (Target config 1 in blue, and Target config in purple). The trajectories separate upon checkerboard color input, leading to four total trajectory motifs: right green, left red, right red, and left green. (c) Projection of the dPCA principal axes onto the PCs. (d) Projection of the context and color inputs onto the PCs. Context inputs are shown in pink, a green checkerboard input in green, and red checkerboard input in red. Green (red) checkerboards lead to an increase (decrease) in PC2 and the color axis, and differ in magnitude depending on the location of the trajectory in PC space. Trajectories are reduced in opacity to better visualize inputs.
connectivity constraints that did not resemble PMd) has context and color information, and also receives context and color inputs (Fig. l3p). We show the axes overlapped with the PCs in Fig 4 as well as the effect of the checkerboard and target inputs, which qualitatively shows that the inputs do project onto the direction axis.
4.3 Inter-area connections preferentially propagate output-relevant direction information
The differentiating aspect of multi-area computation is that the different areas are separated. A natural question ask how then does information propagate between areas? As defined in Section 3 we
denote the feedforward connections from Area to 2 as W W2,, and from Area 2 to 3 as W 23- We present results for feedforward connections from excitatory connections to excitatory units. Based on the hypothesis that the brain uses null and potent spaces selectively filter information 40)
evaluated the effective potent and null spaces of W 12 and W23- We defined the effective potent space be the right singular vectors corresponding to the largest singular values (see Appendix B.Y.
The effective null space corresponded to the singular vectors with the smallest singular values.
We quantified how the color and direction axis were aligned with these potent and null spaces (see Appendix B.9). The projections onto the potent space are shown Fig. 5p.b for Wi2 and W23, respectively. The null projection magnitudes are equal to one minus the potent projection. We found the direction axis was more aligned with the potent space and the color axis was more aligned with the null space. In fact, the direction axis (computed using the activity in Area 1) consistently
7

---

a
Area to
b
Area 2 to 3
random direction
random drecton
1.0
0.0
Null wwwjecttooooooo
0.8
0.6-
0.4
0.4
0.2
color ax8p
direston exs
0.0
0.0
Dimension potamt sparce
Dimensson porent spaiee
Potent Projections
Figure 5: (a) Projections onto the potent space between Areas and 2 for the color and direction axis, and random vector function of effective rank for the input area to the middle area. Regardless of the dimension of the potent space, the direction axis is preferentially aligned with the potent space, indicating the information along this axis propagates. while the color axis approximately randomly aligned. Shading indicates s.e.m. (b) Same as (a) but for projections between Areas 2 and 3. (c) Illustration depicting how the orientation of the axes affect the information that propagates.
most aligned with the top singular vector (governed by the parameters of the feedforward matrix; which do not affect the activity in Area 1). In contrast, the color axis was similarly aligned to a random vector. This alignment was robust to the dimension of the effective potent space, and was consistent across networks with varying feedforward connectivity percentages (10%, 20%, 30%, 50%, 100%). This suggests that learning in the multi-area recurrent network involved aligning the relevant information (in the activations) with the top singular vector (governed by the learned parameters of the feedforward matrix). These results indicate that direction information is preferentially propagated to subsequent areas, while color information is not. This phenomena is schematized in Fig. 55 To better understand the propagation and filtering of information in networks that had color information in the output area, we performed the same analyses on networks trained without Dale's 's law and area networks, and found that these networks had significantly reduced alignment of the direction axis with the top singular vectors (Appendix Fig. 17
These results also have implications how inter-area connections relay information between areas. Color activity has significant representation in Area (see Fig. 2). Therefore, the inter-area connections must not merely propagate the highest variance dimensions of preceding area [41]. Consistent with this reasoning, we found that while the top 2 PCs capture 97.7% excitatory unit variance, the top 2 readout dimensions of W only captured 40.0% of Area I's excitatory unit neural variance (Appendix Fig. [6). Hence, inter-area connections are not aligned with the most variable dimensions, but rather aligned to preferentially propagate certain types of information result consistent with a recent study analyzing links between activity in V1 and V2 41
4.4
Area 3, modeling PMd dynamics, is primarily input driven and implements bistable dynamics
We showed previously that Area 3 most closely resembled PMd's dynamics (Fig. 2 Our results suggest that a direction signal has been computed before Area 3 and is selectively propagated through the RNN's 's inter-area connections. We found that the input to Area 3 (through W 22 is graded direction signal that provides a directional evidence signal for left or right reaches (Fig. (n). This activity must be transformed into eventual DV outputs, which are the accumulated evidence for left or right reach. This is illustrated in Fig. 60, where we plot W2 (r? are the unit activations of Area 2), and r?.
To analyze Area 3's dynamics, we first observed that Wont's 's coefficients were sparse, with 44 out of 80 output weights being identically zero. We found that the readout led to two separate clusters of artificial units: units with non-zero coefficients for the left DV (orange) and those with non-zero coefficients for the right DV (blue). Artificial units projected either to the left or right DV outputs, but not both, suggesting that there are two clusters mediating left and right choices.
Based on this clustering, we sorted and visualized the connections of excitatory units of Area 3, which upon first glance generally has no discernible structure (Fig. be, left panel). After sorting, found that two self-excitatory pools of units emerged in Wree. the first pool in Fig. b (right)

---

b
Figure 6: Area 3 mechanism. (a) Projection of input and overall activity onto the direction axis identified through dPCA. The conventions the same as in Fig. 4 (b) Readout weights in W out are sparse, with many zero entries, and selective weights for a left or right reach. (c) The unsorted Projectin cnto andi (8n0)
Connections
Connections
Weight in War
Artiicaal neuron number
co
Conmecions from
Right units
Aneepe tea
Rigght Ouiput
1000
Corned one
Input
20
0.0
0.0
Lelt Output
Right
15gg Check
0.5
Directional exidence
Intiraary
connectivity matrix for the nonzero readout units (left panel), and the sorted connectivity matrix when the matrix was reordered based on the readout weight pools (right). (d) Average PSTHs from units for a leftward reach and (inset) rightwards reach. When one pool increases activity, the other pool decreases activity. (e) Averaged recurrent connectivity matrix. (I) Schematic of output area. (g) Psychometric curve after perturbation experiment, where 10% of inhibitory weights to the left pool (orange) and right pool (blue) were increased (doubled). Directional evidence is computed by using the signed coherence and using target configuration to identify the strength of evidence for left reach and strength of evidence for a right reach. Increasing inhibition to the left excitatory pool leads to more right choices and vice versa.
corresponding to the left DV and the second pool corresponding the right DV. In addition these two pools, we identified pool of randomly connected excitatory units and pool of inhibitory units with strong projections from and to the two pools. The full Area connectivity matrix is shown in Appendix Fig. 18 This structure is consistent with winner-take-all network, where increasing activity in one pool inhibits activity the other pool through separate inhibition pool (Fig. (d), By taking the averaged connectivity matrix, similar to 22 we confirmed that there were two excitatory pools that received similar projections from the random excitatory pool and inhibitory pool (Fig 6e).
We summarize the behavior with schematic of the area in Fig.
We subsequently applied selective perturbations W rec determine how behavioral performance was biased. We increased inhibition to either the right or the left pool by doubling the weights of 10% of the inhibitory neurons associated with each pool. We found that this biased the network towards more left or right reaches, respectively, shown in Fig. Dg. When inhibition was increased to the right excitatory pool, the network was more likely to respond left. Conversely, when inhibition was increased to the left excitatory pool, the network was more likely to respond right.
Together, these results show that the output area, modeling PMd, robustly transforms separable direction inputs to decision variable through a winner-take-all mechanism.
Discussion
Even though behavior and cognition arise from the coordinated computations of multiple brain areas, there is limited understanding of how interacting brain areas coordinate to produce cognitive behavior this study, we used multi-area RNNs to gain mechanistic insight into how the brain computes a perceptual decision in the Checkerboard Task and transmits only the direction decision to PMd. These results propose hypotheses for computations that occur upstream PMd, particularly how neural population activity representing context, color, and direction structured, and what
41 4

---

information is propagated between areas. We found that inter-area connections were preferentially aligned the direction axis, not axes of maximal variance, leading to selective propagation of direction activity and attenuation of color activity. This role for inter-area connections is consistent with null and potent spaces for filtering and propagating information between areas
4 and
communication subspaces, which are aligned with lower variance dimensions 41
Our results suggest that cortex and multi-area RNNs may share more general principle of multi- area information processing: if information becomes irrelevant for later computations, is reduced or discarded. In the Checkerboard Task, color information is necessary to compute the direction decision, but does not need to be represented after the direction decision is computed, as in PMd 4 26 3C Bi] 45]. In deep neural networks, it believed that minimal representations simplify the This idea is consistent with (1) the multi-area RNN developing minimal (little color information) but sufficient (robust direction information) representation of task inputs, and (2) Area 3, the output area, using a simple winner-take-all readout, forming two pools of neurons representing right and left decisions (Fig.
role of the output classifier 46 17
Our analysis of the multi-area RNN leads to testable hypotheses for future experiments. First, expect that neurons in cortical areas upstream of PMd should exhibit mixed selectivity for color and direction information, consistent with studies of dorsolateral prefrontal cortex (DLPFC) and ventrolateral prefrontal cortex (VLPFC) in cognitive tasks [8 50
More specifically. our model predicts the following organization of population dynamics in these areas: neural population dynamics should diverge two regions with slow dynamics based on target configuration, with largely overlapping context and color axes, but an orthogonal direction axis. Second, due to alignment of inter-area connections, direction axis activity in WFCC/L.CPFFFF should be more predictive of activity in downstream regions such as PMdr and PMd than activity in the top PCs.
10

---

Acknowledgments and Disclosure of Funding
We thank Laura Driscoll for helpful comments on the manuscript as well as Krishna V. Shenoy and William T. Newsome for helpful discussions on earlier versions of these results. We also thank Krishna V. Shenoy for kindly allowing us to use the data collected by Dr. Chandrasekaran when he was postdoc in the Shenoy Lab. MK was supported by the National Sciences and Engineering Research Council (NSERC). CC was supported by a NIH/NINDS ROO award RO0NS092972 and R01 award NS122969, the Moorman-Simon Interdisciplinary Career Development Professorship from Boston University, the Whitehall foundation, and the Young Investigator Award from the Brain and Behavior Research Foundation. JCK was supported by NSF CAREER 1943467, NIH DP2NS122037 the Hellman Foundation, and a UCLA Computational Medicine AWS grant. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
References
[1] S. M. Szezepanski, C. S. Konen, and Kastner, "Mechanisms of spatial attention control in frontal and parietal cortex," Journal of Neuroscience, vol. 30, 1, pp. 148-160) 160, 2010. [Online]. Available:
[2] J. Kalaska and D. Crammond, "Cerebral cortical mechanisms of reaching movements," " Science, vol. 255, no. 5051. pp. 1517-1523, 1992. [Online]. Available:
https:
[3] E. D. Remington, D. Narain, E. A. Hosseini, and M. Jazayeri, "Flexible sensorimotor com- putations through rapid reconfiguration of cortical dynamics," Neuron, vol. 98, no. 5, pp. 1005-1019.e5, Jun. 2018.
[4] T. Yamagata, Y. Nakayama, J. Tanji, and E. Hoshi, "Distinct information representation and processing for goal-directed behavior in the dorsolateral and ventrolateral prefrontal cortex and the dorsal premotor cortex," Journal of Neuroscience, vol. 32, no. 37, pp. 12 13934 12 949, 2012.
[5] J. M. Yau, G. C. DeAngelis, and D. E. Angelaki, "Dissecting neural circuits for multisensory integration and crossmodal processing," Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 370, no. 1677, p. 20140203, 2015. [Online]. Available:
[6] Cisek, "Making decisions through distributed consensus." Current opinion in neurobiology. vol. 22, no. 6, pp. 927-936, 2012.
[7] L. Pinto, K. Rajan, B. DePasquale, S. Y. Thiberge, D. W. Tank, and C. D. Brody, "Task- dependent changes in the large-scale dynamics and necessity of cortical regions," Neuron, vol. 104, no. 4, pp. 810-824, 2019.
[8] M. Siegel, Buschman, and E. K. Miller, "Cortical information flow during flexible sensori- motor decisions," Science, vol. 348, no. 6241, pp. 1352-1355, 2015.
[9] D. J. Freedman and J. A. Assad, "Neuronal mechanisms of visual categorization: an abstract view on decision making," Annual review of neuroscience, vol. 39, pp. 129-147, 2016.
[10] D. L. K. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo, "Performance-optimize hierarchical models predict neural responses in higher visual cortex," Proc. Natl. Acad. Sci. U. S. A., vol. 111, no. 23, pp. 8619 8624, Jun. 2014.
[11] D. L. K. Yamins and J. J. DiCarlo, "Using goal-driven deep learning models to understand sensory cortex," Nat. Neurosci,, vol. 19, no. 3, pp. 356-365, Mar. 2016.
[12] V. Mante, D. Sussillo, K. V. Shenoy, and W. T. Newsome, "Context-dependent computation by recurrent dynamics in prefrontal cortex," Nature, vol. 503, no. 7474, pp. 78-84, Nov. 2013.
[13] R. Laje and V. Buonomano, "Robust timing and motor patterns by taming chaos in recurrent neural networks," Nature neuroscience, vol. 16, no. 7, p. 925, 2013.
11

---

[14] A. Banino, C. Barry, B. Uria, C. Blundell, T. Lillicrap. P. Mirowski, A. Pritzel, M. J. Chadwick, T. Degris, Modayil et al., "Vector-based navigation using grid-like representations in artificial agents," Nature, vol. 557, no. 7705, pp. 429-44299444 4033, 2018.
[15] G. Hennequin, T. Vogels, and W. Gerstner, "Optimal control of transient dynamics in balanced networks supports generation complex movements," Neuron, vol. 82, no. 6, pp. 1394-1406, Jun. 2014.
[16] J. P. Stroud, M. A. Porter, G. Hennequin, and T. P. Vogels, "Motor primitives in space and time via targeted gain modulation in cortical networks," Nature neuroscience, vol. 21, no. 12, p. 1774, 2018.
[17] O. Barak, "Recurrent neural networks versatile tools of neuroscience research," Current Opinion in Neurobiology, vol. 46, pp. 1 6, 6, 2017, computational Neuroscience. [Online]. Available:
77000229
[18] D. Sussillo, M. M. Churchland, M. T. Kaufman, and K. V. Shenoy, "A neural network that finds a naturalistic solution for the production of muscle activity," Nat. Neurosci., vol. 18, no. 7, pp. 1025-1033, 2015.
[19] H. F. Song, G. R. Yang, and X. J. Wang, "Training excitatrryinhhiiiory recurrent neural networks for cognitive tasks: a simple and flexible framework," PLOS Comput. Biol., vol. 12,
no. 2. pp. 1-30, 2016.
[20] H. Song, G. R. Yang, and X.-J. Wang, "Reward-based training of recurrent neural networks for cognitive and value-based tasks," Elife, vol. 6, Jan. 2017.
[21] W. Chaisangmongkon, S. K. Swaminathan, D. J. Freedman, and X. J. Wang, "Computing by robust transience: how the fronto-parien parietal network performs sequential, sategory-based decisions," Neuron, vol. 93, no. 6, pp. 1504-1517.e4, 2017.
[22] F. Mastrogiuseppe and S. Ostojic, "Linking connectivity, dynamics, and computations in low-rank recurrent neural networks." Newron, vol. 99, no. 3, pp. 609 - e23, 2018. [Online]. Available: http://www.eecccccccccco.oeeeeee
[23] D. Sussillo and O. Barak, "Opening the black box: low-dimensional dynamics in high- dimensional recurrent neural networks," Neural Comput., vol. 25, no. 3, pp. 626-649, 2013.
[24] C. Chandrasekaran, D. Peixoto, W. T. Newsome, and K. V. Shenoy, "Laminar differences in decision-related neural activity in dorsal premotor cortex," " Nat. Commun., vol. 8, no. 1, p. 614, Sep. 2017.
[25] M. T. Kaufman, M. M. Churchland, S. I. Ryu, and K. V. Shenoy, "Vacillation, indecision and hesitation in moment-by-moment decoding monkey motor cortex," " Elife, vol. 4, pp. 1-21, 2015.
[26] É. Coallier, T. Michelet, and J. F. Kalaska, "Dorsal premotor cortex: neural correlates of reach target decisions based on a color-location matching rule and conflicting sensory evidence," " Journal of neurophysiology, vol. 113, no. 10, pp. 3543-3573, 2015.
[27] J. Gold and M. N. Shadlen, "The neural basis of decision making," Annual review of neuro- science, vol. 30, 2007.
[28] R. Romo and V. de Lafuente, "Conversion of sensory signals into perceptual decisions." Progress in Neurobiology, vol. 103, pp. 41 75, 2013, conversion of Sensory Signals into Perceptions. Memories and Decisions. [Online]. Available:
[29] D. Freedman and J. A. Assad, "A proposed common neural mechanism for categorization and perceptual decisions," Nature neuroscience, vol. 14, no. 2, 143, 2011.
[30] C. Chandrasekaran, D. Peixoto, W. T. Newsome, and K. V. Shenoy, "Laminar differences in decision- neural activity in dorsal premotor cortex," Nature Communications, vol. no. 1, p. 614, 2017.
12

---

[31] M. Wang, C. Montanède, C. Chandrasekaran, D. Peixoto, K. V. Shenoy, and J. F. Kalaska, "Macaque dorsal premotor cortex exhibits decision-related activity only when specific stimulus-response associations are known," Nature Communications, vol. 10, no. 1, p. 1793, 2019. [Online]. Available:
[32] D. Kobak, W. Brendel, C. Constantinidis, C. E. Feierstein, A. Kepecs, 7. F. Mainen, X.-I. Qi, R. Romo, Uchida, and C. K. Machens, "Demixed principal component analysis of neural population data," Elife, vol. 5, Apr. 2016.
[33] S. Wise and E. A. Murray, "Arbitrary associations between antecedents and actions," Trends in Neurosciences, vol. 23, no. 6, pp. 271 276, 2000. [Online]. Available:
[34] E. Hoshi, "Cortico-basal ganglia networks subserving goal-directed behavior mediated by conditional visuo-goal association," Front. Neural Circuits, vol. 7, no. October, p. 158, 2013.
[35] D. Badre, J. Hoffman, J. W. Cooney, and M. D' D'esposito, "Hierarchical cognitive control deficits following damage the human frontal lobe," Nature neuroscience, vol. 12, no. 4, p. 515, 2009.
[36] N. T. Markov, M. M. Eresey-Ravasz, A. R. Ribeiro Gomes, C. Lamy, L. Magrou, J. Ve- zoli, P. Misery, Falchier, R. Quilodran, M. Gariel, J. Sallet, Gamanut, C. Huissoud, S. Clavagnier, P. Giroud, D. Sappey-Marinier, P. Barone, C. Dehay, Z. Toroczkai, K. Knoblauch, D. C. Van Essen, and H. Kennedy, "A weighted and directed interareal connectivity matrix for macaque cerebral cortex," " Cereb. Cortex, vol. 24, no. 1, pp. 17-36, Jan. 2014.
[37] H. F. Song, G. R. Yang, and X. J. Wang, "Training excitatory-inhibice recurrent neural networks for cognitive tasks: simple and flexible framework," PLoS Comput. Biol., vol. 12, no. 2, pp. 1-30, 2016.
[38] K. W. Latimer, "Nonlinear demixed component analysis for neural population data as a low-rank kernel regression problem," Neurons, Behavior, Data Analysis, & Theory, pp. - 24, 2019.
[39] L. v. d. Maaten and G. Hinton, "Visualizing data using t-sne," Journal of machine learning research, vol. 9, no. Nov, pp. 2579-2605, 2008
[40] M. T. Kaufman, M. M. Churchland, S. I. Ryu, and K. V. Shenoy, "Cortical activity in the null space: permitting preparation without movement," Nat. Neurosci., vol. 17, no. 3, pp. 440 448, Mar. 2014.
[41] D. Semedo, A. Zandvakili, C. K. Machens, B. M. Yu, and A. Kohn, "Cortical Areas Interact through Communication Subspace," Neuron, vol. 102, no. 1, pp. 249-259.e4, apr 2019. [Online]. Available: https://doi.org/ htt.:/oogggg..10//..eeerroooo wwwo:/.nerr...211.......
[42] G. R. Yang, M. R. Joglekar, H. F. Song, W. T. Newsome, and X.-J. Wang, "Task representations neural networks trained to perform many cognitive tasks," Nature neuroscience, vol. 22, no. pp. 297-306, 2019.
[43] A. Kohn, A. Jasper, J. D. Semedo, E. Gokcen, C. K. Machens, and B. M. Yu, "Principles of corticocortical communication: Proposed schemes and design considerations," Trends in Neurosciences, vol. 43, no. 9, pp. 725-737, 2020/10/12 2020.
[44] S. D. Stavisky. J. C. Kao, S. I. Ryu, and K. V. Shenoy, "Motor cortical visuomotor feedback ac- tivity is initially isolated from downstream targets in Output-Null neural state space dimensions," Neuron, vol. 95, no. 1, pp. 195-208.e9 Jul. 2017.
[45] Cisek and J. F. Kalaska, "Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action," Neuron, vol. 45, no. 5, pp. 801-814, 2005.
[46] A. Achille and S. Soatto, "Emergence of invariance and disentanglement in deep representations," Journal of Machine Learning Research, vol. 19, no. 50, pp. 1-34, 2018. [Online]. Available:
13

---

[47] S. Soatto and A. Chiuso, "Visual representations: Defining properties and deep approximation." " Proceedings of the International Conference on Learning Representations (ICLR), June 2016.
[48] M. Rigotti, O. Barak, M. R. Warden, X.-J Wang, N. D. Daw, E. K. Miller, and S. Fusi, "The importance of mixed selectivity in complex cognitive tasks," Nature, vol. 497, no. 7451, pp. 585-590, May 2013.
[49] V. Mante, D. Sussillo, K. V. Shenoy, and W. T. Newsome, "Context-dependent computation by recurrent dynamics in prefrontal cortex," Nature, vol. 503, no. 7474, pp. 78 84, Nov. 2013.
[50] S. Fusi, E. K. Miller, and M. Rigotti, "Why neurons mix: high dimensionality for higher cognition," Current opinion in neurobiology, vol. 37, pp. 66-74, 2016.
[51] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, Nov. 2016.
[52] R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training recurrent neural networks," International Conference on Machine Learning, Feb. 2013, pp. 1310-1318.
[53] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint 1rXiv:1412.6980, Dec. 2014.
[54] B. W. Brunton, M. M. Botvinick, and C. D. Brody, "Rats and humans can optimally accumulate evidence for pocision-mmakin.. Science, vol. 340, no. 6128, pp. 95-98, 2013.
[55] R. Ratcliff, "A theory of memory retrieval." Psychological review, vol. 85, no. 2, p. 59, 1978.
[56] D. Sussillo, M. M. Churchland, M. T. Kaufman, and K. V. Shenoy, "A neural network that finds a naturalistic solution for the production of muscle activity," Nat. Neurosci, vol. 18, no. 7, pp. 1025-1033, 2015.
[57] D. Barber and F. Agakov, "The im algorithm: A variational approach information maximiza- tion," Proceedings of the 16th International Conference on Neural Information Processing Systems, ser. NIPS' 03. Cambridge, MA, USA: MIT Press, 2003, p. 201-208.
[58] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, "On variational bounds of mutual information." in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. Long Beach, California, USA: PMLR, 09-15 Jun 2019, pp. 5171-5180, [Online]. Available:
http:/ppoolll99..ht..
[59] M. Kleinman, Achille, Idnani, and J. Kao, "Usable information and evolution of optimal representations during training," in International Conference on Learning Representations, 2.21. [Online]. Available: https://peeeeeeveew..et//ouu/tdp
14


Reviews:
[{"review_id": "zcI29ZGvd7v", "reviewer": "Reviewer_EDPH", "summary": "This study examines a \"checkerboard\" task, in which subjects had to determine the dominant color of a stimulus to select one out of two possible targets, the position of which (=\"context\") varied across trials. Neurophysiological recordings in the premotor cortex of monkeys trained on this task revealed that neural activity encoded only the final decision, but not the sensory information on the color of the stimulus, or position of the targets. To investigate potential mechanisms, this study examines multi-area recurrent neural networks trained on the task. The activity in the different areas is analysed using population methods from systems neuroscience, in particular targeted dimensionality reduction and decoding.\nThe main result is that the network reproduces the main experimental observation, the fact that only decision is represented in the last area. Moreover, training experiments indicate that at least three areas, and segregation between excitation and inhibition, are needed to reproduce this observation.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This study is a nice application of a variety of methods from neuroscience  to analyse trained RNNs. The main result, the fact that only output-relevant information is present in the last area, is interesting. While the analysis is extensive, it is rather descriptive, and ultimately the underlying mechanism unfortunately remains unclear. \n\nIn particular, the main finding that at least three areas are needed is empirical, but it is unclear why two areas would not be enough to eliminate all information other than output. Indeed the analyses of the first area (Fig 4) show that direction and color are already separated along orthogonal axes at that stage. So projecting onto the direction axis would be sufficient to eliminate all other information. One additional area would be sufficient to do this projection. Why are three areas needed ?\n\nSimilarly, it is unclear why the segregation between excitation and inhibition would be needed, it seems that a network without Dale's law should be able to perform the same projection. (Detail: the text mentions the importance of feed-forward inhibition, but it is not clear what is meant by that, inhibition across areas?)\n\nFinally, the experimentally testable hypotheses announced in the abstract do not seem to be fully developed.\n\nAltogether, this study falls somewhat short of the promised mechanistic understanding.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "q80LLEvl_mY", "reviewer": "Reviewer_vT5V", "summary": "The authors train a RNN on a very simple neuroscience task that involves a binary decision based on the color content of two stimuli. The authors find that if they connect three RNNs in a similar feed-forward fashion as 3 specific brain areas involved in the task, the third RNN learns representations somewhat similar to what is found in a higher motor area (PMd). Specifically, their ‘area 3’ RNN discriminates the two choices (movement directions) but not color or context. This is only observed if 3 “areas” are trained and excitatory and inhibitory weights are separated, but not if only 1 or 2 areas were trained. They use the trained 3-area RNN to dissect the mechanisms behind the filtering of stimulus-related information while only the relevant choice information is passed on.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a pure neuroscience analysis paper, with, as far as I could see, no real take home’s for machine learners. No novel methods were developed, not even formal details of the methods used were presented in the main text. Also, the analyses themselves have, as far as I could see, no implications for the design of AI/ML algorithms, and no attempt was made by the authors to work out any such potential implications. \nPerhaps this is fine, but I guess it at least implies this paper can only be judged on grounds of whether it contributes something new to the body of knowledge in experimental neuroscience. I’m therefore not sure whether this conference is really the right fit for this type of paper, or whether a classical neuroscience journal would suit it better.\n\nPutting this question aside, even within neuroscience, I’m not sure whether plain RNN training on behavioral tasks is, methodologically, still up to the state of the art (where the focus shifted to training RNNs directly on observed data in recent years). I’m also not sure how surprising the results really are: If color and context is directly fed into the first but not into the third RNN (cave: details were not clear from the main text to me), wouldn’t it be expected a priori that the third area discriminates less well between these properties than the first? And that if one has only one area for training, one finds that it unlike the 3-area RNN discriminates between all these features (as they were directly fed in)?\nAlso, to judge the biological plausibility of their 3-area vs. 1-area models the authors solely focused on this one property of whether color/context information can be read out or not in a given area. I would think that many more features of the neural activity across all three artificial areas would need to be compared to their biological counterparts to make a more convincing case for the model.\n\nFinally, the authors stress a lot that, unlike in the biological preparation, they can dissect the RNN mechanisms in detail. But then a lot of the analyses rest on simple PCA-type projections that probably could have been done on experimental data as well. Admittedly, though, the analyses of the weights (Fig. 5,6) may have been more tricky, but perhaps correlation-based measures for deriving functional connectivity could have filled in.\n\nThe basic question is quite interesting, i.e. how neural systems learn to build parsimonious representations and to pass on to upstream areas only information needed to foster a decision. However, if that were the main question, I think it should have been addressed from a much wider angle, including more types of tasks and network designs, to allow for more generic conclusions and principles to be derived (beyond this one toy example). And a more rigorous mathematical and computational analysis would have been useful as well, given that – as the authors state themselves – a formal model of the process is available.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "WWxmUBRPoz", "reviewer": "Reviewer_jJXH", "summary": "This paper uses RNNs to study the role of PMd in a decision making task. The paper finds that representations in PMd are informative about the output (direction of movement), but not about the inputs to the task (here, the color information). As this information is necessary to solve the task, it suggests that areas prior to PMd represent this information. The paper builds of a model of this multi-area computation using stacked recurrent networks. For certain settings of architecture parameters, the final area in the network resembles PMd (in that the final area contains direction, but not color, information). The paper probes this multi-area RNN to understand how it solves the task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I thought this was a really clever use of multi-area (stacked) RNNs to shed insight onto a complex task (decision making) that involves multiple brain areas, as well as a really effective use of dimensionality reduction (particularly, dPCA) to reverse engineer a trained network.\n\nSome minor concerns/questions:\n- What about the dynamics of Area 2? What computation(s) are they responsible for? If they are not necessary, then how come you need 3 areas before the last one resembles PMd?\n- Another lens on to the dynamics of RNNs is to linearize them around fixed points of the dynamics, as in [Sussillo and Barak 2013](https://direct.mit.edu/neco/article/25/3/626/7854/Opening-the-Black-Box-Low-Dimensional-Dynamics-in). I would be curious to see if using that kind of analysis on each of the areas in your network led to any additional insight. For example, you might be able to better identify or quantify the winner-take-all dynamics in Area 3 by looking for approximate fixed points of the dynamics of the area 3 RNN.\n- If one really believes that the multi-area RNN corresponds to multiple brain areas, then this work provides strong predictions for what to expect if we were to record from these additional areas during the same task. These predictions could be more explicitly spelled out in the discussion (even if you are agnostic about what brain area(s) the RNN Areas 1 & 2 correspond to).\n- Can this multi-area RNN approach help inform future PMd experiments? For example by helping shape what kinds of tasks we should train monkeys on in order to find or highlight interesting computations in PMd?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "-jkiCkYDwGT", "reviewer": "Reviewer_L9GC", "summary": "The authors train a multi-area (modular) RNN to do a decision-making task known as the \"Checkerboard\" task and compare the RNN to recorded neurons in dorsal premotor cortex (PMd). They 1) show how the final layer of the multi-area RNN better match PMd responses compared with a standard RNN; 2) Investigate different parameter/architecture regimes of the multi-area RNN to determine what type of network is important for producing PMd-like responses; and 3) investigate the mechanisms by which information is (or isn't) transmitted through the multi-area RNN.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n-The manuscript is very clearly written.\n\n-The manuscript is extremely thorough and high quality - the authors provide a plethora of analyses in the main text and in the appendix. There is definitely enough here to be a full journal article.\n\n-Using RNNs, the paper investigates a significant question in neuroscience - how multiple regions may work together to produce behavior.\n\n\nCons:\n\n-The paper is not very novel in a machine learning sense and multi-area RNNs have been used within neuroscience (although only recently). However, I'm not overly concerned about this given that the paper uses an existing tool (multi-area RNNs) to do many interesting analyses.\n\n\nMinor comments:\n\nLine 128: \"is contained is\" -> \"is contained in\"\n\nWhat happens if you train the RNN without Dale's law? Do you still get PMd-like representations (as in Fig. 3A), but you wanted to use Dale's law to understand what parameters work for this type of network?\n\nLine 184: Fig 4e isn't a panel.\n\nYou say \"see Methods\" a couple of times, but this should be referring to the Appendix.\n\nFig. 5 legend: You say the \"color axis is more aligned with the nullspace.\" From this figure, it's clear to me that the color axis is not aligned with the potent space, but having random alignment doesn't necessarily seem like nullspace alignment to me (i.e. information is not actively being filtered out). This may all just be a matter of semantics, but I wanted to point it out.\n\nLine 195: Are your findings still the same for all connections (not just limited to E->E)? Either finding would be interesting.\n\nDoes that Area 3 mechanism just boost decoding accuracy from ~95 to ~100% ?\n-Seems like already linearly separable, so what's the point of this mechanism?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are briefly mentioned, but could be mentioned more explicitly or expanded on.\n\nPotential negative societal impacts are not mentioned in the text", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the clear presentation of the paper, the interesting observation that a three-area network can qualitatively reproduce the results of the Mante et al paper with regards to neural selectivity, and the thorough investigation of different hyper-parameters influencing network behaviour. However, and after extensive internal discussions in which all reviewers participated (even if they might not have updated their review reports), it remained unclear what the _causes_ of this observation were-- in part, the paper mostly (and in contrast to some of its claims) mostly made statements about _sufficient_ ingredients of the networks, and not about _necessary_ ones (e.g. regarding the role of Dale's law). More broadly, the study would have benefited from a clearer statement about _why_ a  three-area (and not, e.g. a two-area) network can achieve the desired behavior.  We hope that the feedback by the reviewers will allow be useful for you.
Decision: Accept (Poster)