Brain-Like Processing Pathways Form in Models With Heterogeneous Experts
Authors: Jack Cook
Danyal Akarca
Rui Ponte Costa
Jascha Achterberg
Venue: NeurIPS 2025 poster ()
Keywords: Processing pathways
Brain-inspired neural networks
Mixture-of-Experts
Modularity
Task complexity
Adaptive processing
Primary area: neuroscience_and_cognitive_science
TL;DR: Using a heterogeneous Mixture-of-Experts model architecture, we show that brain-like processing pathways form due to inductive biases on processing complexity and expert dropout

Abstract: The brain is made up of a vast set of heterogeneous regions that dynamically organize into pathways as a function of task demands. Examples of such pathways can be found in the interactions between cortical and subcortical networks during learning, or in sub-networks specializing for task characteristics such as difficulty or modality. Despite the large role these pathways play in cognition, the mechanisms through which brain regions organize into pathways remain unclear. In this work, we use an extension of the Heterogeneous Mixture-of-Experts architecture to show that heterogeneous regions do not form processing pathways by themselves, implying that the brain likely implements specific constraints which result in the reliable formation of pathways. We identify three biologically relevant inductive biases that encourage pathway formation: a routing cost imposed on the use of more complex regions, a scaling factor that reduces this cost when task performance is low, and randomized expert dropout. When comparing our resulting Mixture-of-Pathways model with the brain, we observe that the artificial pathways in our model match how the brain uses cortical and subcortical systems to learn and solve tasks of varying difficulty. In summary, we introduce a novel framework for investigating how the brain forms task-specific pathways through inductive biases, and the effects these biases have on the behavior of Mixture-of-Experts models.

Brain-Like Processing Pathways Form in Models With Heterogeneous Experts
Jack Cook!
Danyal 2 Rui Ponte Costa Jascha Achterberg
Centre for Neural Circuits and Behaviour, University of Oxford Department of Electrical and Electronic Engineering, Imperial College London
* *Joint senior authors
Abstract
The brain is made up of a vast set of heterogeneous regions that dynamically organize into pathways function of task demands. Examples of such pathways can be found in the interactions between cortical and subcortical networks during learning, in sub-networks specializing for task characteristics such as difficulty modality. Despite the large role these pathways play in cognition, the mechanisms through which brain regions organize into pathways remain unclear. In this work, we use an extension of the Heterogeneous Mixture-of-Expertt architecture to show that heterogeneous regions do not form processing pathways by themselves, implying that the brain likely implements specific constraints which result in the reliable formation of pathways. We identify three biologically relevant inductive biases that encourage pathway formation: a routing cost imposed on the use of more complex regions, a scaling factor that reduces this cost when task performance is low, and randomized expert dropout. When comparing our resulting Mixture- of of-Pathwayss model with the brain, we observe that the artificial pathways in our model match how the brain uses cortical and subcortical systems to learn and solve tasks of varying difficulty. In summary, we introduce a novel framework for investigating how the brain forms task-specific pathways through inductive biases, and the effects these biases have on the behavior of Mixture-of-Fxxe of-Expers models.
Introduction
The brain is made up of many heterogeneous regions distinguished by features such as connectivity, cell types, neurotransmitters, and functional specialization [1-4]. To support complex behavior, the mammalian brain dynamically organizes these regions into diverse networks and processing pathways [5], allowing it adapt to different inputs and task demands. This principle spans sensory systems [6-8], cognitive networks [9], emotion-related circuits [10]. and face perception [11]. Notably, pathway formation highly dynamic: regions can participate in many pathways, allowing cognitive processes to arise from the joint activations of specific groups of regions. While theoretical work has shown how heterogeneous regions and modules can develop within networks [12-14], how these organize into sarge-scale pathways remains poorly understood.
The importance of studying pathway formation and coordination extends beyond neuroscience: it is also becoming increasingly relevant in machine learning research. As models have evolved from small networks with couple of layers to large system-level architectures, achieving complex function while maintaining efficiency has become critical. One recent development toward this goal is the Mixture-of- Experts (MoE) architecture [15, 16], which contains specialized experts that are selectively activated based on the current input. This should create pathways between experts that selectively respond to inputs of varying complexity [17] make efficient of computational resources [18, 19). However, this theorized specialization of experts appears to be limited in practice [20, 21]. making it difficult for specialized www.-oooipppplllleeeee pathways form.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

These findings raise the question, how do stable and functionally relevant pathways form in networks of distributed heterogeneous experts? Do heterogeneous regions automatically group into such pathways, or are additional priors required? Once pathways develop in models, do they show the same context-aware adaptability that has been observed in the brain? To address these questions, we introduce a neural network architecture made up of heterogeneous experts to study the condi- tions under which processing pathways form, and the degree to which these pathways resemble those studied in the brain. Specifically, we adapt the Heterogeneous Mixture-of-Experr (HMoE) architecture [22, 23), in which information may be dynamically routed to computational experts, or recurrent
regions, of varying sizes. In our model, unlike prior work, each expert is implemented network that could be considered a standalone model or brain region. We study the pathway formation in this architecture while we train models to learn 82 Lime-series-based cognitive tasks of varying difficulty [24]. Through these analyses, we find:
Layers of heterogeneous experts do not automatically form recognizable pathways.
Instead, inductive biases are required for pathways to form: (i) routing cost that penalizes the model for using larger experts, (ii) scaling the routing cost based on task performance, and (iii) random expert dropout. These result in the formation of a Mixture-of off-ooffaPaass.....
The pathways that form in our new Mixture-of- oof-atwwaaaass architecture mirror the interactions between cortical and subcortical pathways in the brain during learning, and are in line with the dynamics of the brain's 's multiple-demee demand system [9, 25].
To arrive at these findings, in the following we start by analyzing the usage of experts of a baseline model with HMoE layers. We then develop specific inductive biases that encourage pathways to form, before finally comparing these pathways to observations made in the brain.
Related Work
Brain-like modularity and regional heterogeneity can be induced in neural networks through priors and training procedures to explain how such features develop in the brain [14, 26-30J. The priors that are especially relevant in the context of this work relate to metabolic cost and energy efficiency, which are crucial in determining the brain's 's circuitry and function [13, 31-34].
While the above work often focuses on starting from fully-connected networks to observe the formation of modules and regions, modeling multi-region interactions has also become possible with modern methods [35-40]. This line of work has revealed how joint computation can be implemented through interactions between independent modules [5]. However, these multi-region models generally predefine specific circuit structure with small set of regions, preventing further study on how regions come to interact in the first place. Notable work that allows for dynamic (non-fixed) interaction of multiple independent regions often assumes networks which are not able to learn tasks [41, 42], though [43] stands out with a trainable network made up of individual RNNs. Their multi-region networks can change their connectivity during learning, but cannot route information based on task context. The work most closely aligned with our goal is [44], which studies how spatial (metabolic) constraints in feed-forward networks can form visual processing streams. However, it too does not consider how regions may change their interaction as a function of context, and does not allow for solving standard time-series-based cognitive tasks.
In the context of artificial intelligence, the introduction outlined how the popular Mixture-of- -fxture-of-EEpprrts architecture [15, 16] is relevant to our question of how specialized regions dynamically organize into processing pathways. Recent efforts to build networks out of experts that vary in terms their architecture [17, 22, 23] and function [20] are especially relevant here. Moreover, work has argued that routing pathways are a powerful method for handling the complex data-flow of these otherwise efficient architectures [19, 45], but without investigating how adaptable pathways can be encouraged to form. In neuromorphic computing it has been shown to be possible to implement brain-like visual processing pathways to achieve efficient processing [46], but with predefined static architecture.
3 Methods
In this work, we aim to identify the mechanisms by which pathways form between heterogeneous regions, and how those pathways are used across diverse set of tasks. To study this computationally,

---

we need model made up of heterogeneous experts, analogous to brain regions, that work together solve many different tasks, We create such model by extending the Heterogeneous Mixture-of- Experts architecture [22] and training it on the Mod-Cog set of time-series-based cognitive tasks [24].
3.1 Model Architecture: Heterogeneous Mixture-of-Experts
Mixture-of-Experts models (MoEs) [15, 16] are characterized by their layers, which contain mul- tiple smaller models alongside router model, which decides which experts should process the input at each timestep. Specifically, the router determines the weight with which each expert contributes to the layer's final output. Experts can also be excluded, by setting an expert's weight zero. most modern MoEs, the ex- perts are large scale MLPs placed in between attention layers, which are typically activated at every timestep. In Heterogeneous Mixture-of- Expert models [22] (HMoEs), the experts can vary in terms of their sizes and activation func- tions. We extend the HMoE architecture with several significant adaptations. Namely, each layer of our model contains three experts: two GRUs with 16 and 32 neurons respectively, and one skip connection, which allows the model to choose to perform no computation for given timestep [17]. Our implementation uses GRUs with 64 neurons as routers and does not include any additional layers between HMoE layers. We use this setup as a baseline for our investigations (Figure 1). In the following sections, we will introduce additional inductive biases to this baseline architecture, resulting in our final Mixture-of of-Patwwyys model. The algorithm for this full model described in Algorithm 1.
Actions
Repeated for more leyara
Layer1
connection daciiy
expert
Complex expert
W=0.02
w+8.11
GRU di
Observations
Figure 1: Schematic of our baseline model archi- tecture. Information is passed through three layers, each of which can dynamically route information experts of different computational complexity.
Algorithm 1: Mixture-of-Pathwayy training protocol. Full details in Appendix A.I.
Data: Task dataset D, Experts E fei, c Cx }. Routers R (r1, (f1,72,..... rn)
Initialize routers and heterogeneous experts, set by to task input; for each training step do Sample batch b from D;
for each task i, timestep / b do
for each layer I with experts e, E do
Compute routing weights wh = (rttma))(((h))))))
Apply expert dropout (Not in baseline architecture; see Section 4.2);
Compute expert activations w, for each expert e, E L;
Combine outputs: hy Σ, Wyhi,.
end
Compute baseline model losses: Lax and Leesponse, ;
Compute Lrouting loss (Not in baseline architecture; see Section 4.1); Compute Taoaal = Lix + Trouting + E; Tresponee,
end
Update parameters using Schedule-Free AdamW optimizer [47];
end
3.2 Evaluation with Cognitive Tasks
To evaluate how pathways are formed and used across tasks with different characteristics, we use the Mod-Cog task set, which contains 82 time-series-basse cognitive tasks [24]. This is an expansion of
3

---

the popular NeuroGym framework [48], which contains tasks like Go-NoGo or two-stimuli integration tasks (see Appendix A.2 for task details). Importantly for us, the tasks vary in difficulty due their varied inputs, decision rules, and delay lengths. Generally, tasks range from 0.8 to 3 seconds in duration, during which we sample information timesteps 100 milliseconds apart. At each timestep, the model receives a 11 15-dimensional input made up of four components: 1-dimensional fixation input, two 16-dimensional stimuli, and an 82-dimensional one-hot encoding of the active task, which we pass through a 16-dimensional learned embedding layer. While the fixation input is active, the model should always output zero. After the fixation period, the model needs to use the observed information to output the correct choices during the response period.
We train our models over 10 epochs, each containing 1000 training steps. At each training step, models are given 128 x 350 x 115 matrix of input data, representing 128 batches of task sequences that are 350 timesteps long, with 115 features at each timestep. These task sequences contain many individual tasks: the average task is about 20 timesteps long, meaning that in each batch, models observe about 27 trials of each task. Models are trained with a cross entropy loss Tresponse, for the correct response during the response period of task i. An additional fixation loss Lix encourages the model to output zero during the fixation period. All losses used in our analyses are detailed in Appendix A.3. Training one model takes roughly hour single NVIDIA T4 GPU. Details on implementation and code access are outlined in Appendix A.I.
Once models have learned to solve each task by routing information between experts, we can study the conditions under which processing pathways form between layers. In the following we will first study the routing behavior of our baseline architecture. We will then show how the additional inductive biases described Algorithm result in the formation of pathways. Finally, we will test the degree to which these pathways resemble established processing pathways in the brain.
4 What Causes Pathways to Form?
In this section, we investigate the conditions under which pathways form between layers of heteroge- neous experts. We set three criteria to determine whether pathways have formed:
1. Pathways should be consistent with respect to tasks, meaning that when two models are trained to solve the same tasks, they should have structurally similar pathways.
2. Pathways should be self-sufficient, meaning that when experts outside of pathway are removed, then the model's overall performance should remain largely intact.
3. Pathways should be distinet, meaning that several different pathways should be used to solve groups of tasks with varying characteristics.
4.1 Pathway Consistency
To see if experts form consistent, task-driven pathways, we train 20 randomly initialized models with the same settings and compare their routing patterns on the same set of 82 cognitive tasks. This allows to examine whether models use similar sets of experts to solve the same tasks, such as whether smaller experts are reliably used to solve simpler tasks, or vice versa. We first do this with a baseline model made up of three HMoE layers, and then test each model on 50 trials of each task while recording the routing weights assigned to each expert at each timestep (w values in Figure I). these weights calculate each model's
To test whether routing is stable across training runs, Learned Pathway Complexity for each task i (LPC,) as follows:
1 T. LPC, T.
ΣΣ
Was
(1)
This metric is calculated by multiplying the weight wijit assigned by the router to each expert at each timestep by the squared size s² of expert while the model solves task i. This is then averaged across the total timesteps T; of each task to ensure that longer tasks are not biased toward having larger LPCss. This results in a LPC value for each of the 82 tasks and 20 model runs (see Appendix A.4 for an example calculation). The squaring of expert sizes is motivated by the O(s)) cost of storing each expert's 's weight matrix in memory, and we expand on the suitability of using sj as

---

measure of each expert's complexity in Appendix A.5. Skip connections are free to use. To measure pathway consistency, we can now correlate this list of LPC values across training runs. For the baseline model, we find that models are not consistent across training runs (mean pairwise correlation of 0.0324, Figure 2), suggesting that the baseline model does not form any stable and task-related processing pathways by default. Therefore, we next want to explore which specific inductive biases may result in such pathways.
Theories of metabolic optimization and cost minimization are core parts of our understanding of the brain's 's computations [33, 49, 50]. The reduction of energy consumption has been a powerful source of priors for building biain-like neural networks [13, 31, 34] and more generally achieving brain-inspired computing [46, 51]. Hence we hypothesize that regularizing the routing weights by making it more expensive to route to more complex experts might cause replicable pathways to develop, as observed the brain. We do so by incorporating the I.PC; (from Equation 1) into the model's 's loss, making it more costly for the model to activate more complex experts. Finally, to avoid convergence on the local minimum of only using the smallest experts without solving any tasks', we add normalization strategy, dividing each LPC; by Lresponse.. the cross-entropy loss measuring the model's 's performance on task E T. In addition to helping with convergence, this normalization term can also be viewed as helping our model more directly control the cognitive effort, or processing power, with which it solves a task. We discuss this further in Section 6.
Adding these additional components to the loss results in the following equation, where Lax and Liesponee are the standard task-based losses described in Section 3.2. A small value is added to ensure that if the model solves tasks perfectly, the routing loss is not oc. All loss calculations are outlined in detail in Appendix A.3 and A.4.
T
aLPC,
L == Lax
(Tresponse,
(2)
Iesponne, +
We now evaluate the routing consis- tency of models trained with this cus- tom loss function. Figure 2 shows that our expectations are confirmed: on its own, adding the LPC; for each task creates more consistent routing (mean pairwise correlation of 0.15, signif- icant over baseline with p < 0.01). Scaling this term by the model's per- formance on each task i am- plifies this effect, encouraging mod- els across training runs to converge on more consistent routing patterns (mean pairwise correlation of 0.71, significant over baseline with p < 0.001).
Baseline
+ Routing Cost
Tuss Performance
Scaling
0.25
0.00
0.25
0.50
0.75
Correlation of Routing Activity Between Rums
Figure 2: Models trained with routing cost and task-based scaling exhibit more stable routing. Correlations are calcu- lated between the routing patterns across 20 training runs of each model setup.
4.2 Self-Sufficiency of Pathways
Our second criterion measures whether formed pathways are self-sufficient, meaning that removing an expert that is not part of the currently activated pathway should only minimally impact the performance of the model. To test for sulf-sufficiency. we first evaluate whether models are still able to perform tasks when they are prevented from using experts that have been assigned low routing weights. We find that our baseline models are extremely sensitive to this deactivation: if they are prevented from using experts with < 0.025, which only contribute 2.5% or less each layer's output, average task accuracy drops from 98.2% to 16.4% This shows that while models learn replicable routing patterns, these are not yet pathways, as they rely on all the experts.
We speculate that pathway self-sufficiency can be achieved by stochastic dropout of experts with low routing weights. Dropout is especially interesting it is an established principle for achieving more
Note that routers converging to local minima is an established phenomenon in Mixture- of of-Exxerts models, as there is bias to rely on the expert that learns the task, or decreases the overall loss as in our case, first [16, 22, 52, 53J.
5

---

robust neural networks [54] and has also been linked to the stochastic nature of signal processing in neuroscience [55, 56]. We implement expert dropout by randomll deactivating experts that contribute very little to the output during training. The probability Pj with which expert is deactivated at given timestep is determined follows:
B
8 Wjs uy <>
Pj
=
7
(3)
0,
otherwise
We set 7 0.1, meaning that experts contributing 10% or more to the output of a layer are never deacti- vated. As this contribution weight decreases to zero, this probability increases linearly to B. To identify how much dropout needed to improve robust- ness, we train I1 groups of models with & values rang- ing from 0, 0.0.1,.... 0.9, 1 using Equation 3, with 10 models in each group. We evaluate each model on 50 trials per task, block- ing experts with routing weights below 11 values: 0.00225,... 0.225,0.22 0.25.
100%
0.25
Evaluation Accurcy
Dropout Threshold During Evaluation
0.20
80%
0.15
60%
0.10
40%
0.05
20%
0.00
0%
0%
20%
40%
60%
80%
100%
Maximum Dropout Probability During Training (8)
Figure 3: Model accuracy after removing weightdd experts across different training dropout levels. If models trained without dropout (B 0) are prevented from using experts that contribute very little to the output, accuracy drops precipitously, from 98.2% to 15.1% By comparison, the accuracy of models trained with maximum dropout value of B = 0.8 only drops from 86.5% to 77.7%.
Accuracy is averaged across the 10 models in each group. We find that expert dropout has a relatively minor impact on performance, while significantly improving the robustness of the pathways that form. Figure 3 shows that for models trained with maximum dropout of 80% (3 = 0.8), this drop in accuracy small (from 85.8% to 74.4%). This motivates us set 3 0.8 for our models in the remainder of this work. Note that routing consistency (Section 4.1) remains high with dropout, shown through an average pairwise correlation of 0.51, which is significant over the baseline with p < 0.0001 (see Appendix A.4).
4.3 Distinct Pathways Across and Within Tasks
For our final criterion, we want identify whether meaningful patterns of expert usage develop across tasks and timescales within our model. To do this, we record the routing patterns across 50 trials of each task for both the baseline model and our final model, which is trained using the routing cost with task performance scaling and expert dropout. To visualize how routing varies across layers, tasks, and time, we average routing patterns for each task in three phases: (i) before the stimulus is shown, (ii) while the stimulus is shown, and (iii) during the response period. We apply K-means clustering (k- 10) these matrices to identify groups of tasks that use similar pathways.
In our model, we observe a structured usage of expert pathways (Figure 4): during the pre-stimulus phase, models primarily rely on the cheap skip connections, yet. For some tasks, increasingly complex experts are activated with the onset of stimuli. However, since the model still only needs to output zero during this phase, most tasks continue to leverage the cheap 'all-skip' pathway until the response phase. During this final phase, we see very rich dynamics of pathways being differentially activated across tasks and time periods in our model. This shows how processing pathways interact over the time of trial, with different combinations of experts activated over tasks and time periods. The following sections analyze these dynamics in more detail, especially in comparison to the dynamics observed in the brain. Importantly, for the baseline model, clusters do not seem to employ very different combinations
information needs to be processed
experts across tasks. The more differentiated usage of pathways across clusters can be seen in the distribution of the numbers of tasks contained in given cluster (Figure 4, left). Here, our model shows very distinct power-law distribution of several large clusters containing 20 tasks and many small clusters capturing task- pathway

---

Numbor of Chuttex
Boseline
20
Chuster
Expert Usage
-00%
-00
Number of Chaaters
Our Model
400%
Chuster
20%
0%
20
Latitata
Tasks in Cluster
Skip
Simple Complex
Sudp
Sumple
Complex
Skip
Simple Complex
Pre-stitmlls
Stimulus
Respouse
Figure 4: Task clustering derived from expert usage patterns. Clusters are averaged over three phases each task: the pre-stimulus phase, during which task is known but no input data is presented, the stimulus phase, during which input data is presented, and the response phase. Left: Sizes clusters across training runs. Right: Average routing weights across training runs by cluster (y-axis) and task phase. In each phase, expert usages within each layer (across each layer's 's skip connection, simple expert, and complex expert) sum to 100%. The complex expert is rarely used in the first two phases, during which the model outputs zero, but has average usage high as I 11% in the most complex cluster (see Appendix A.8). LI /L/12/ L3 = Layer Layer 2 Layer 3.
usages. The baseline model, on the other hand, seems to distribute the tasks more evenly across clusters, indicating that there are much less distinct routing patterns. This can be shown quantitatively in the sizes of the largest clusters for each model run, which are significantly larger for our model than for the baseline model across 10 different random seeds (p < 0.0001). Further visualizations are provided in Appendices A.7 and A.8.
Our results so far show that wathways do automatically develop from a heterogeneous mixture of experts. Training models with a couting-comples and adding expert dropout, all encourage stable pathways to form. These three features define our Pfisture-ofPPatlwaay (MoP) model. In the following section, we investigate whether the pathways that form our MoP model mirror the pathways observed within the primate brain.
loss, scaling it based on task performance,
5 Do Artificial Pathways Behave Like the Brain's Pathways?
In the previous section, we showed how our brain-inspired architectural contributions resulted in the formation of mixture of processing pathways in our model. Now, we will evaluate the degree to which these artificial pathways resemble the behavior of established processing pathways in the brain. Our analyses primarily focus on pathways and dynamics of the brain relating task difficulty.
5.1 Solving Tasks of Varying Difficulty
When analyzing brain activations across tasks with varying levels difficulty, there is distinct group of activations in large frontoparietal network when solving complicated tasks. Since this network is activate while solving any difficult task, it was named the multiple-demant can be identified both in humans and non-human primates [4, 59]. With this in mind, test whether the selection of experts used to solve a task is indicative of task difficulty.
(MD) system [57, 58].
want to
To relate these findings from the MD system to our model, we expect that when solving tasks of increasing difficulty, our model should learn activate increasingly complex regions (schematic in Figure 5). We test this by measuring the correlation between task's 's difficulty and the learned pathway complexity (LPC) used by the model to solve the task. We quantify the difficulty of a task by the number of training steps it takes standalone GRU to learn the task (see Appendix A.9 for details and alternative ways of quantifying task difficulty). We find that our full MoP model shows positive significant relationship, whereas the baseline model does not, matching our expectations.
7

---

Baseline (r=00.01)
Our Model (r-0.51)
Reliance complex
cortical MD system
Expected finding:
100%
Accurocy Without Complex Experts
1600
800
LPC
-00%
600
1400
400
0%
10
10a
10a
101
10a
10a
Task difficulty
Task Difficulty
Task Difficulty
(a) Expected results
(b) Empirical results
Figure 5: Our model allocates less computation toward solving simpler tasks. Additionally, when the most complex experts in each layer are disabled, our model is still able to solve the simplest tasks with high accuracy.
Baseline
Our Model
1500
Hard
Tusk Diffcuuty
2000
Expected finding:
LPC
Relative reliance on...
Cortex
Hard task
1000
1500
500
Simple task
Easy
2
4
6
10
0
2
6
10
Learging duration
Training Epoch
Training Epoch
Subeortex
Bascline (D--0.48)
Our Model (I 0.31)
ALPC After Epoch
800
600
100
250
10-
10a
10a
10a
102
10a
Task Diffiulty
Task Diflicuity
(a) Expected results
(b) Empirical results
Figure 6: Our model moves complex tasks to more complex pathways at the start of training to support the learning process. (a) Conceptual schematic of
interactions. (b) Top row shows the pathway complexity over learning per task averaged across 10 training runs. Lower row shows the change pathway complexity between model initialization and the end of the first epoch as function of task difficulty. Our model specifically relies on complex pathways to learn difficult tasks, similar how the brain relies on complex pathways to support the acquisition of complex skills, even if they are gradually moved to simpler pathways later on.
Furthermore, known that patients with lesions to their MD system struggle solving difficult tasks but their ability to solve simple tasks usually is unaffected [60, 61]. We find that the same is true in our models: if the most complex expert each layer is lesioned, our model can still solve simple tasks with high accuracy, but accuracy difficulty tasks drops significantly. The same is not true of the bascline model: performance on all tasks drops to near-chance.
5.2 Learning Tasks of Varying Difficulty
A more nuanced view of pathways in the brain comes from observing how tasks of varying difficulty are learned over time, Here, an interesting distinction between complex and simple tasks observed: while simple tasks can be learned through simpler (subcortical) regions alone, complex tasks require more complex (cortical) regions for learning [62-65] (see schematic Figure 6a). However, as learning continues, even complex task skills are often "transferred" from complex to simple brain regions. This is possible despite the simple pathway not being sufficient to drive the learning process in the first place. We now want test whether this phenomenon can be observed in our model.

---

Table 1: Task accuracy and pathway metrics for modified versions of our model.
Model
Accuracy
Fig. 5 Correlation
Fig. 6b Correlation
Baseline
91.1% + 8.9%
-0.01
-0.49
Our model!
83.0% + 15.5%
0,54
0.31 1*
Without dropout
90.1% 8.9%
0,55
0.03
a = le
69.0% + 20.1%
-0,57**
-0.37
a = le 6
89.7% = 9.0%
0.62
0.18
Without task embeddings
83.0% + 14.2%
0.58
0.58
Router dim =
83.2% + 14.5%
0.46**
0.33
Router dim 128
81.9% + 16.9%
0.35
0.27
With dropout, a le 1e task embeddings, and router dim < 0.05, ** p < 0.01, *4* < 0.001
To study this effect, we track the complexities of the learned pathways across tasks over the duration of learning. Figure 6 shows these results across models: our MoP model seems to indeed learn complex tasks by first increasing their pathway complexity, but then reducing it gradually during learning. In contrast, very simple tasks do not increase in pathway complexity at all over learning. We now want to quantify this effect. To translate Figure 6a to our models, we can quantify to which degree the pathway complexity of given task is increasing or decreasing after the first training epoch. This is measure of how much the model specifically uses more complex pathway to learn a given task. Based findings from neuroscience, we would expect complex tasks to be explicitly moved to more complex pathways, relative to the random starting point, whereas this should not be necessary for simple tasks [35, 62, 66]. Figure 6b shows that this phenomenon can be observed in our model. The more difficult task is, the more its pathway complexity increases at the start of learning, with the simplest tasks immediately getting routed toward simpler pathways (r 0.31, p 0.0040). This not true in the baseline model, where we observe the opposite effect: the pathway complexity used to solve the most difficult tasks increases the least at the start of learning (r -0.48, p < 0.0001).
Upon further inspection, this happens because in an effort to minimize the routing loss, the baseline model learns to push tasks that it is unable to solve toward simpler pathways prematurely. As result, the baseline model fails to learn the most difficult tasks until the very end of the training process.
5.3 Ablations
Lastly, we investigate how changes made to our model can alter the degree to which it resembles pathways in the brain, as discussed in Sections 5.1 and 5.2. Table shows the effects of design parameters on the correlations reported in Figures 5 and 6. Notably, we found that when trained with our loss function that scales based on LPC but without dropout, our model exhibits the effect shown in Figure 5, but not the effect shown in Figure 6. This indicates that the finding in Figure 6 is due to an interaction between the LPC scaling in our loss function and dropout, and can not be explained by training with the LPC regularization alone. We speculate that expert dropout forces the model to be explicit about which pathway is used in learning, and is crucial for creating brain-like learning dynamics. This finding highlights how our model's behavior specifically results from the interplay of all three of our proposed inductive biases.
We also find that removing the task embedding layer improves the finding in Figure 6, however its removal drastically slows down training since the active task represented with 82 dimensions at each timestep rather than 16 (see Section 3.2). Changing the router's 's hidden size does not meaningfully affect our results. Increasing or decreasing the penalty for using large experts (a) naturally has a meaningful effect on the results, where too strong of penalty inhibits learning the tasks as well as general convergence of the model, and too weak a penalty does not sufficiently motivate the model to reduce its usage of complex experts. All rows in the table averaged over 10 runs.
Discussion
In this work, we adapted the heterogeneous Mixture-o-FEppert architecture to investigate how brain- like processing pathways can form between layers of heterogeneous experts. While these experts do

---

not form pathways on their own, once trained with a routing-cos loss, task resk-performance scaling, and expert dropout, we find that they create Mixture of Pathways. Our model provides an account of task-specific brain-wide pathways commonly observed in neuroscience.
These findings are relevant for neuroscience, energetic and processing complexity related priors have been key explanatory mechanisms for how the structure and function of the brain arises [13, 32, 49, 50J. We show that incentivizing models learn to prioritize simple experts drives the development of brain-like processing pathways from a heterogeneous set of expert models. Additionally, we show how stochasticity of signals is important for learning self-sufficient processing networks. Our model represents an exciting new architecture which can be expanded in the future to study additional heterogeneities present in the connectome, such as varying cell-types and neurotransmitters. Region- specific models, such as those for the hippocampus [67], could be integrated within our architecture.
The brain's implementation of the complexity-guided routing mechanism would likely be found in thalamic nuclei, which regulate information flow between cortical regions [68]. Two systems could modulate pathway selection based on metabolic costs: norepinephrine release from the locus coeruleus, which controls cognitive effort and processing power allocation [69-71], and hypocre- tin/orexin neurons in the hypothalamus, which govern metabolic resource budgets [72]. Both systems project strongly to thalamic nuclei and could influence routing decisions between simple and complex processing pathways. This suggests our model's routing-cose cost mechanism may reflect how the brain balances computational demands against metabolic constraints through neuromodulatory control of thalamocortical interactions. While mapping our router onto specific brain region might feel natural, it should be added that mechanisms like predictive coding can implement routing and filtering operation between regions without the need of an explicit router region [73].
In the context of machine learning, our work builds on the recent widespread adoption of the Mixture-of-Eaae -f--xesrts architecture for building parameter-efficient large language models [15, 74]. Recent innovations specifically aim at allowing MoE models to process queries dynamically to reduce processing costs [17]. Our small-scale simulations show how it may be possible to use heterogeneous experts alongside processing-cov cost loss function that allows the model to dynamically allocate resources to processing tokens. Finally, load balancing in MoEs prevents over-reliance on a single expert by encouraging distributed processing [15]. Our complexity loss serves as task-driven form of load balancing.
6.1 Limitations and Extensions
There are several ways to expand our investigations. On the neuroscience side, our architecture introduces new way of modeling multi-region interactions of the brain, but some key architectural characteristics are not yet taken into account. Most importantly, the primate brain has large loop structures which would allow signals to return to a region [27]. Our architecture only allows a forward progression of signal and does not allow signals to be routed back to earlier layers. At the same time, while our analyses demonstrated a link between experts and cortical and subcortical regions, we have not linked the router component of our HMoE layers to a specific component of the brain. Potential options are discussed earlier in Section 6, we do not make any explicit comparison to brain data yet. On the ML side, our training setup currently focuses on solving relatively simple tasks with a small model. To see whether our complexity-awe networks. we would need to train larger models more difficult tasks. Lastly, on the identification of pathways, we currently rely on three independent tests to see whether a model contains pathways. Future investigations would idcally identify one specific metric to quantify the degree to which Mixture-of of -f-Exxssts architecture has formed pathways.
routing and balanbannning measures scale to larger
7 Conclusion
In this paper, we introduced modified Heterogeneous Mixture-of- architecture that results in the formation of recognizable processing pathways. Analysis of these pathways during learning and problem solving revealed a similarity between these pathways and those observed in the brain. Our new Mixture-of- Siature-of-Pathways architecture serves as a new theoretical tool for neuroscience and can guide the search for future resource-efficient architectures in machine learning.
10

---

Acknowledgments and Disclosure of Funding
We thank our reviewers, the Neural & Machine Learning Group the University of Oxford, and the Al HW SW CoDesign Workstream at the Open Compute Project (OCP) for helpful feedback. This research is supported by the EPSRC (EP/X029336/1) and an ERC-UKRA Frontier Research Guarantee Starting Grant (EP/Y027841/1) awarded to R.P.C. J.C.'s work was supported by Rhodes Scholarship. J.A.'s work was partially supported by a Career Development Research Fellowship from St John's College, Oxford. We additionally thank Modal for compute credits.
References
[1] Zizhen Yao, Cindy TJ van Velthoven, Michael Kunst, Meng Zhang, Delissa McMillen, Changkyu Lee, Won Jung, Jeff Goldy, Aliya Abdelhak, Matthew Aitken, et al. A high- resolution transcriptomic and spatial atlas of cell types in the whole mouse brain. Nature, 624(7991):317-322 2023.
[2] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):111-178 2016.
[3] Stewart Shipp. Structure and function of the cerebral cortex. Current Biology, 17(12):R443 R449, 2007.
[4] Valentina Mione, Jascha Achterberg, Makoto Kusunoki, Mark J Buckley, and John Duncan. Neural dynamics of an extended frontal lobe network in goal-subgoal problem solving. bioRxiv, pages 2025-05, 2025.
[5] Maxwell A Bertolero, BT Thomas Yeo, and Mark D Esposito. The modular and integrative functional architecture of the human brain. Proceedings of the National Academy of Sciences, 1 112(49):E6798 E6807, 2015.
[6] Kalanit Grill-Spector and Rafael Malach, The human visual cortex. Annual Review of Neuro- science, 27:649-677, 2004.
[7] Edward HF de Haan and Alan Cowey. On the usefulness of 'what' and 'where' pathways in vision. Trends in cognitive sciences, 15(10):460-46, 2011.
[8] Stephen R Arnot, Malcolm A Binns, Cheryl L Grady, and Claude Alain. Assessing the auditory dual-pathway model in humans. Neuroimage, 22(1):401-408, 2004.
[9] John Duncan. Construction and use of mental models: Organizing principles for the science of brain and mind. Neuropsychologia, 207:109062, 2025.
[10] Amit Etkin, Tobias Egner, and Raffael Kalisch. Emotional processing anterior cingulate and medial prefrontal cortex. Trends in Cognitive Sciences, 15(2):85-93, 2011.
[11) Michal Bernstein and Galit Yovel. Two neural pathways of face processing: A critical evaluation of current models. Neuroscience Biobehavioral Reviews, 55:536-546, 2015.
[12] Guangyu Robert Yang. Madhura Joglekar, H Francis Song, William T Newsome, and Xiao- Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
[13] Jascha Achterberg, Danyal Akarca, D. J. Strouse, John Dunean, and Dunean E. Astle. Spatially embedded recurrent neural networks reveal widespread links between structural and functional neuroscience findings. Nature Machine Intelligence, Publishing Group.
2023. Publisher: Nature
[14] Olaf Sporns and Richard F Betzel. Modular brain networks. Annual review of psychology. 67(1):613 (74)0,6136440,1. 2016.
[15] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint wwx.x:2440.0445..0 2024.
11

---

[16] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford. Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mixtral of Experts, 2024, MiXiv2201.04088 [cs].
[17] David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mfxture-of-deptts: Dynamically allocating compute in transformer-based language models. arXiv preprint 2rXi:22404.02258, 2024.
[18] William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. arXiv preprint arXiv:2209001667, 2022.
[19] Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy, et al. Pathways: Asynchronous distributed dataflow for ml. Proceedings of Machine Learning and Systems, 4:430-449, 2022.
[20] Nikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker, and Ahmet Üstün. Nexus: Specialization meets adaptability for efficiently training mixture of experts. arXiv preprint 2rXiv:2408.15901, 2024.
[21] Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, and Vasudev Lal. Semantic specialization in moe appears with scale: A study of deepseek rl expert specialization. arXiv preprint arXiv:2502.10928, 2025.
[22] An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, JN Han, Zhanhui Kang, Di Wang, al. Hmoe: Heterogeneous mixture of experts for language modeling. arXiv preprint arXiv:2408. 10681, 2024.
[23] Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul- Mageed, Laks VS Lakshmanan, Ahmed Hassan Awadallah, Sebastien Bubeck, and Jianfeng Gao. Automoe: Heterogeneous mixture-of-expper with adaptive computation for efficient neural machine translation. arXiv preprint arXiv:2210.07535. 2022.
[24] Mikail Khona, Sarthak Chandra, Joy Ma, and Ila Fiete. Winning the lottery with neural connectivity constraints: faster learning across cognitive tasks with spatially constrained sparse RNNs, 2023. arXiv:2207.0355 [q-bio].
[25] John Duncan. The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour. Trends Cognitive Sciences, 14(4):172-17) 179, 2010.
[26] Guangyu Robert Yang and Manuel Molano-Mazón, Towards the next generation of recurrent network models for cognitive neuroscience. Current opinion in neurobiology, 70:182-199 192, 2021.
[27] Jascha Achterberg, Danyal Akarca, Moataz Assem, Moritz Heimbach, Duncan E Astle, and John Duncan. Building artificial neural circuits for domain general cognition: primer on brain-inspired systems-level architecture. arXiv preprint arXiv:2303. 13651, 2023.
(28) Nicolas Perez-Nieves, Vincent CH Leung, Pier Luigi Dragotti, and Dan FM Goodman. Neural heterogeneity promotes robust learning. Nature communications, 12(1))5791, 2021.
[29] Gabriel Béna and Dan FM Goodman. Dynamics of specialization in neural modules under resource constraints. Nature Communications, 16( 16(1):187, 2025.
[30] Cornelia Sheeran, Andrew S Ham, Duncan E Astle, Jascha Achterberg, and Danyal Akarca. Spatial embedding promotes specific form of modularity with low entropy and heterogeneous spectral dynamics. arXiv preprint arXiv:2409. /7693, 2024.
[31] Jake Patrick Stroud, Michal Wojeik, Kristopher Torp Jensen, Makoto Kusunoki, Mikiko Kado- hisa, Mark J Buckley, John Duncan, Mark Stokes, and Máté Lengyel. Effectt of noise and metabolic cost cortical task representations. eLife, 13:RP94961, 2025.
12

---

[32] Danyal Akarca, Simona Schiavi, Jascha Achterberg, Sila Gene, Derek K Jones, and Duncan E Astle. A weighted generative model of the human connectome. bioRxiv, pages 2023-06, 2023.
[33] Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245):273 278, 2015.
[34] Abdullahi Ali, Nasir Ahmad, Elgar de Groot, Marcel Antonius Johannes van Gerven, and Tim Christian Kietzmann. Predictive coding is a consequence of energy efficiency in recurrent neural networks. Patterns, 3(12), 2022.
[35] Kevin GC Mizes, Jack Lindsey, G Sean Escola, and Bence P Ölveczky. The role of motor cortex in motor sequence execution depends on demands for flexibility. Nature Neuroscience, pages
1 1-10, 2024.
[36] Yang Zhou, Matthew C Rosen, Sruthi Swaminathan, Nicolas Y Masse, Ou Zhu, and David Freedman. Distributed functions of prefrontal and parietal cortices during sequential categorical decisions. Elliee, 10:e58782, 2021.
[37] Joseph Pemberton, Paul Chadderton, and Rui Ponte Costa. Cerehellar-driven cortical dynamics can enable task acquisition, switching and consolidation. Nature Communications, 15(1):10913, 2024.
[38] Ching Fang and Kimberly Stachenfeld. Predictive auxiliary objectives in deep rl mimic learning in the brain. arXiv preprint arXiv:2310.06089, 2023.
[39] Ted Moskovitz, Kevin J Miller, Maneesh Sahani, and Matthew M Botvinick. Understanding dual process cognition via the minimum description length principle. PLOS Computational Biology, 20(10))1102333,, 2024.
[40] Samuel Liebana Garcia, Aeron Laffere, Chiara Toschi, Louisa Schilling, Jacek Podlaski, Matthias Fritsche, Peter Zatka-Haas, Yulong Li, Rafal Bogacz, Andrew Saxe, et al. Stri- atal dopamine reflects individual long-term learning trajectories. bioRviv, pages 2023-12, 2023.
[41] David G Clark and Manuel Beiran. Structure of activity in multiregion recurrent neural networks. Proceedings of the National Academy of Sciences, 122(10):e404003912,, 2025.
[42] Ulises Pereira-Ohe Obelii--biiliooiic, Sean Froudist- Walsh, and Xiao-Jing Wang. Cognitive network interactions through communication subspaces in large-scale models of the neocortex. bioRxiv, 2024.
[43] Leo Kozachkov, Michaela Ennis, and Jean-Jacques Slotine. Runs of mns: Recursive construction of stable assemblies of recurrent neural networks. Advances in neural information processing systems, 35:305 12-30527, 2022.
[44] Dawn Finzi, Eshed Margalit, Kendrick Kay. Daniel LK Yamins, and Kalanit Grill-Spector. A single computational objective drives specialization of streams visual cortex. bioRxiv, pages 2023-12, 2023.
[45] Xin He, Shunkang Zhang, Yuxin Wang, Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang. Xiaowen Chu, Ivor Tsang, and Ong Yew Soon. Expertflow: Optimized expert activation and token allocation for efficient mixture-of-expprts inference. arXiv preprint arXiv:2410. 17954444 2024.
[46] Zheyu Yang, Taoyi Wang. Yihan Lin, Yuguo Chen, Hui Zeng, Jing Pei, Jiazheng Wang, Xue Liu, Yichun Zhou, Jianqiang Zhang, et al. A vision chip with complementary pathways for open-world sensing. Nature, (2908014):0077-100
2024.
[47] Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. Advances in Neural Information Processing Systems, 37:9974-10007, 2024.
13

---

[48] Manuel Molano-Mazoon, Joao Barbosa, Jordi Eastor-Ciurana Marta Fradera. Ru- Yuan Zhang, Jeremy Forest, Jorge del Pozo Lerida, Ji-An, Christopher Cueva, Jaime de la Rocha, et al. Neurogym: An open resource for developing and sharing neuroscience tasks. OSF, 2022.
[49] Wouter Kool and Matthew Botvinick. Mental labour. Nature Human Behaviour, 2(12):899-908,
2018. Publisher: Nature Publishing Group.
[50] Ed Bullmore and Olaf Sporns. The economy of brain network organization. Nature reviews neuroscience, 13(5):336-349 2012.
[51] James B Aimone. A roadmap for reaching the potential of brain-derived computing. Advanced Intelligent Systems, 3(1):2000191, 2021.
[52] William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, 2022. arXiv:2101.0396 [cs].
[53] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed-Me MoF:: Advancing Mixture-of- Experts Inference and Training to Power Next-Generation AI Scale, 2022. arXiv:2201.055 [es].
[54] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(66):1299-1958 2014.
[55] Gustavo Deco, Edmund T. Rolls, and Ranulfo Romo. Stochastic dynamics principle of brain function. Progress in Neurobiology, 88(1):1-16, 2009.
[56] Hailiang Li, Jian Weng, Yijun Mao, Yonghua Wang, Yiju Zhan, Qingling Cai, and Wanrong Gu. Adaptive dropout method based on biological principles. IEEE Transactions on Neural Networks and Learning Systems, 32(9))4267 4276, 2021.
[57] John Duncan. Construction and use of mental models: Organizing principles for the science of brain and mind. Neuropsychologia, 207:109062 2025.
[58] Evelina Fedorenko, John Duncan, and Nancy Kanwisher. Broad domain generality in focal regions of frontal and parietal cortex. Proceedings of the National Academy of Sciences, 110(41):16616-166)) 2013.
[59] Daniel Mitchell, Andrew H Bell, Mark J Buckley, Anna Mitchell, Jerome Sallet, and John Duncan. A putative multiple-demant system in the macaque brain. Journal of Neuroscience, 36(33):857488888 2016.
[60] María Roca, Alice Parr, Russell Thompson, Alexandra Woolgar, Teresa Torralva, Nagui Antoun, Facundo Manes, and John Dunean. Executive function and fluid intelligence after frontal lobe lesions. Brain, 133(1): (33(1):234-244 247, 2010.
[61] V. Goel and J. Grafman. Are the frontal lobes implicated in "planning" functions? Interpreting data from the Tower of Hanoi. Neuropsychologia, 33(5):623-642, 1995.
[62] Y Kate Hong, Clay O Lacefield, Chris C Rodgers, and Randy M Bruno. Sensation, movement and learning the absence of barrel cortex. Nature, 561(7724):542-546 2018.
[63] Andrew J Peters, Julie MJ Fabre. Nicholas A Steinmetz, Kenneth D Harris, and Matteo Carandini. Striatal activity topographically reflects cortical activity. Nature, 59117750):420-425 2021.
[64] Steffen B. E. Wolff, Raymond Ko, and Bence P. Ölveezky. Distinet roles for motor cortical and thalamic inputs to striatum during motor skill learning and execution. Science Advances, r(abkab,0 2022.
[65] Ray J Dolan and Peter Dayan. Goals and habits in the brain. Neuron, 80(2):312-325, 2013.
14

---

[66] Risa Kawai, Timothy Markman, Rajesh Poddar, Raymond Ko, Antoniu L Fantana, Ashesh K Dhawale, Adam Kampff. and Bence Ólveczky. Motor cortex required for learning but not for executing a motor skill. Neuron, 86(3):800-812, 2015.
[67] Sarthak Chandra, Sugandha Sharma, Rishidev Chaudhuri, and Ila Fiete. Episodic and associative memory from spatial scaffolds in the hippocampus. Nature, pages 1-13, 2025.
[68] S. M. Sherman and R. W. Guillery. The role of the thalamus in the flow of information to the cortex. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 357(1428): 2002. Publisher: Royal Society.
[69] Gary Aston-Jones and Jonathan D Cohen. An integrative theory of locus coeruleus- norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. (8(1):403-450 2005.
[70] Susan J Sara and Sehastien Bouret. Orienting and reorienting: the locus coeruleus mediates cognition through arousal. Neuron, 76(1) 1111)1133- 2012.
[71] Andrew Westbrook and Todd S Braver. Cogmitive effort: ncuroeconomic approach. Cognitive, Affective, & Behavioral Neuroscience, 15(2):395-413 2015.
[72] Alexander L. Tesmer, Christine Dalla Pola, Dino Gilli, Nikola Grujic, Eva F Bracey, Tom- maso Patriarchi, Daria Peleg-Raibstein, Rafael Polania, and Denis Burdakov. Neurometabolic signaling and control of policy complexity. bioRxiv, pages 2025-02, 2025.
[73] Kaitlyn M Gabhart, Yihan Sophy Xiong, and André M Bastos. Predictive coding: a more cognitive process than we thought? Trends in Cognitive Sciences, 2023.
[74] Llama Team. The Llama 4 herd: The beginning of new era of natively multimodal AI innovation, 2025.
[75] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint
2018.
[76] Stefano Recanatesi, Serena Bradde, Vijay Balasubramanian, Nicholas A. Steinmetz, and Eric Shea-Brown. A scale-dependent measure of system dimensionality. Putlerns, 105555 2022.
[77] Ana C. Lorena, Luís Garcia, Jens Lehmann, Marcilio C. P. Souto, and Tin Ho. How complex is your classification problem? a survey on measuring classification complexity, 2020.
[78] John Duncan, Alice Parr, Alexandra Woolgar, Russell Thompson, Peter Bright, Sally Cox, Sonia Bishop, and Ian Nimmo-Smith. Goal neglect and spearman's is g: competing parts of complex task. J. Exp. Psychol. Gen., (07(())131-44) February 2008.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
The abstract and introduction accurately represent the paper's contributions without overstatement. We claim that (1) heterogeneous experts do not automatically form pathways without specific architectural priors, (2) normalized routing complexity loss and expert dropout enable pathway formation, and (3) the resulting pathways mirror brain- like processing pathways in multiple ways. Our experimental results directly support these claims through quantitative evaluations of pathway stability, self--uuficiency. and task-related functionality. The limitations of our approach are acknowledged in the Discussion section, including the lack of recurrent connectivity and the current small scale of our model.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Our paper includes dedicated "I Limiiaaiinss and Extensions" section that explicitly discusses the limitations of our work across three key areas: (1) Neuroscience limitations: We acknowledge that our architecture lacks important brain-like features such as recurrent connectivity loops, which are present in the primate brain, and that we have not explicitly linked the router networks to specific brain structures. (2) Machine learning limitations: We discuss that our current implementation focuses on relatively small models and simple tasks, raising questions about whether our approach would scale to larger networks with more complex tasks. (3) Methodology limitations: We note that our current approach for identifying pathways relies on two independent tests, rather than a single unified metric. Throughout the paper, we are also transparent about the computational requirements of our model and the number of training runs conducted. These limitations are presented alongside potential future research directions to address these constraints.
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
16

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include any theoretical proofs. It does make the theoretical assumption that the Mixture-of-Experts architecture is suitable backbone for testing the development of pathways but this assumption is made very clear given our extensive discussion of the Mixture-of-Exxe -fxpeetts architecture.
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We rely on commonly used and well defined building blocks both on the side of models and tasks that are openly available and clearly state the hyperparameters we rely on. Our code is also available for review as part of the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
17

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide detailed description on how access and run our code in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
httgg/ddde//oeeSubiiiee cemmiiissiiiccy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. http//ublcggguuuiee
CCe///ooeeemiiic wwtssSuuuiissiooooo cwtmmmissoonpee ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our Methods section describes all details of the algorithm setup needed, and the appendix provides additional implementation details, specifically with regard to the custom loss functions that use. Generally rely very established building blocks combined in a novel way, and our instructions on how we combined them and which hyperparameters we used are described in the main text.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide significance tests for all main claims in the paper. All significance tests use are two-sided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specifically describe the compute resources we use to run our experiments in the methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///neuri..peee httpr//s...ppplleecciieseettaaaa
Answer: [Yes]
19

---

Justification:
Yes, our work fully conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We conduct very small-scale simulations aimed at studying theoretical phenomenon in MoE. models and replicating features observed in the brain. The models trained here are in way powerful enough to be deployed for the contexts discussed in the guidelines this question.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We only simple simulated trials of cognitive tasks data which were generated based on open-access packages and hence there is no risk of misuse. We do not release any pretrained models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
20

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We appropriatly cite the source of the dataset package. All our simulations are implemented with PyTorch, as described the extended implementation details the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasstss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide anonymized simulation code for our paper which is released under CC BY 4.0 license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We do any crowdsourcing or human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: : http///eeuri...eee cc/ /ttp:////..///orreeeee////////// for what should or should not be described.
22

---

A Appendix
A.1
Implementation Details and Code Access
While most details on the model implementation are described in Section 3, we provide some additional details here. The GRUs in all of our layers, including routers, use the ReLU activation function and are initialized from U( (vk,, vR), where is the GRU's hidden size, as is standard in Py TyToch. All models are optimized with the Schedule-Free variant of the Adam W optimizer [47] using& learning rate of 0.01, betas of (0.9, 0.999), and no weight decay. As briefly described in Section 3, we use an additional embedding layer to transform the 82-dimensional one-hot encoding of the task identity into a 16-dimensional embedding vector, which concatenated with the task stimuli before being provided to the model. We include these details in a complete version of our training algorithm below Algorithm 2, expanding on the abbreviated algorithm introduced in Algorithm I. We provide our implementation at https: / github. com/ www.ccccccccititcccaacooo of off-athways.
Algorithm 2: Full Mfxture-of-Pathwaeee training protocol.
Data: Task dataset D, Experts E e2,, e, Routers R {r1 ( T2...., .nk
Initialize routers and heterogeneous experts, sampling weights and biases from u(-vk, vR); Process 82-dimensional one-hot task encoding with embedding layer;
Set hg to task input, consisting of a 1-dimensional fixation input, two 16-dimensional stimuli, and a 16-dimensional task embedding at each timestep: for each training step do
Sample batch b from D; for each layer I do
Compute routing weights wi, (oftmax(77(xx-1)))
Apply expert dropout (Not in baseline architecture; see Section 4.2); Compute expert activations w1,j for each expert in /
Combine outputs: h4 = E, Wt.j hi.ji
end
Compute baseline model losses: Lis and Leeeponne, ; Compute Lrouting loss (Not in baseline architecture; see Section 4.1); Compute Lioral Lix + Lrouting + >; Liesponse,
Update parameters using Schedule-Free AdamW optimizer [47];
end
A.2 Sample Task Visualizations and Descriptions
Section 3 briefly described the Mod-Cog task set [24]. Here, provide more detailed description of the tasks, alongside visualizations of sample trials.
The Mod-Cog task set consists of 82 time-series-bart
cognitive tasks that extend the original NeuroGym framework [48] through two primary modifications: integration tasks, which incorporate interval estimation based on delay periods, and sequence generation tasks, which require time-varying outputs with drifting directions. The original 20 cognitive tasks from NeuroGym serve as the foundation, with new integration-based tasks and new sequence generation tasks forming a set of 82 tasks in total. These tasks span a wide range of cognitive demands, from simple stimulus-response mappings to complex working memory and sequential decision-making challenges.
Tasks are presented as continuous time-series data, which we sample at 100-milisecond intervals. Each input consists of a I-dimensional fixation signal, two 16-dimensional stimulus channels, and an 82-dimensional one-hot task identifier that passes through learned embedding layer, as described in Appendix A. During the fixation period of each task, models must maintain an output of zero while processing incoming stimuli. During the subsequent response period, models must return task-specific output sequences. Task difficulty varies systematically according to several factors: the complexity of decision rules (from simple detection to multi-step integration), the duration of delay periods that tax working memory, the number of stimuli that must be simultaneously tracked, and whether responses require statie outputs or dynamic sequential patterns. Figure illustrates six representative tasks that demonstrate this range of complexity.
23

---

rigo
Stimulus 2
Stimulus 1
Fixation
Action
10
10
dml
dlydml
Stimulus 2
Stimulus
Fixation
Action
10
10
dm2
goseql
Stimulus 2
Stimmius
Fixation
Action
Timestep
Timestep
Figure 7: Model inputs and actions for the rtgo, go, dm1, dlydm1, dm2, and gosegr tasks. Ap- pendix A.2 provides a detailed description of each task.
24

---

RTGo task (rtgo): The model must immediately output the value presented in either input channel.
Go task (go): The model observes two input channels and must respond with the value presented in either stimulus once the fixation period has ended.
Decision Making task (dm1): The model observes brief stimulus presentations from both channels and must respond with the value in stimulus that had the highest average intensity (values in stimulus 2 can be ignored). This requires integration and comparison of sensory evidence.
Delayed Decision Making task (dlydm1): Similar to dm1, but includes a delay period between decoy stimulus and the noisy stimulus, requiring working memory to maintain stimulus information.
Dual Decision Making task (dm2): Similar to dm1, but the model must return the highest value from stimulus 2.
Sequential Decision Making task (goseql); Based on the Go task but requiring time- varying output that drifts in specific direction (leftward) over the response period, combin- ing stimulus detection with sequential motor control.
A.3 Loss Variants Used Across Model Types
In Section 4, we introduce architectural changes to the model that encourage pathways to form between layers. Here, we give an overview of the five different loss functions which are used to train our different model variants.
The loss function that we loss, as shown below in Equation 4. Equation 5 computes a mean-squared eтror for the fixation period where the correct output value is always zero. This loss function is not task-specific since it can be computed across all task inputs while the fixation input is active. Equation 6 computes a task-specific cross-entropy loss between the 16 possible output values and the model's output logits at each timestep. When used to train model, one of these response losses will be computed for each task in the set of tasks T.
train the baseline model only includes the fixation and task response
T
Lasslline
Lix +
L Iesppuse,
(4)
T
Lix T 46
(5)
Ti T: Σ Ye log(ū.)
Iresponse,
(6)
Here, ÿt represents the model's output logits at timestep / during fixation, 7 is the total number of fixation timesteps, Y/ is the true target output, Ot the model's predicted output at timestep f, and Tj represents the total number of response timesteps for task i.
In Section 4.1, using this baseline loss function, we find that the baseline model by itself does not converge on consistent pathways across model runs. This motivates us to create a new loss function, LRc, which introduces routing cost that penalizes the model for using more complex experts, as shown in Equation 7.
Lgc Lux +
Σ Lresponne,s + aLPC,)
T
LPC, Ti
Σ
Wi.j.t
(8)
25

---

In Equation 8, wiji represents the routing weight assigned to expert j at timestep t for task i, sj is the size of expert J. and denotes the total number of experts in the model (in this manuscript, all models have three layers of three experts each, so F 9). ar is hyperparameter that balances the trade-off between the model's performance on each task and the complexity of the experts used to solve that task. If a is too large, the model will reach local minimum where it is unable to solve any task, but it can reduce its expert usage to zero by using the skip connections in each layer and performing no computation. On the other hand, if a is too small, the model will solve each task to very high degree of accuracy, but not reduce the complexity of the experts used to solve each task, 10-5 balanced these priorities well,
and fail to form brain-like pathways. We found that setting and used this value for all of the experiments in this work. However, future work may investigate a better method for selecting this hyperparameter.
While the routing consistency for the model with the loss in Equation improved as shown in Figure 2, we do find that it does not converge on a fully consistent routing pattern. This motivates the addition of a scaling factor based on task performance, which reduces the effect of the routing loss when task performance is low. The resulting loss shown in Equation 9 is the final loss we use to train our MixtureeoffPPatwaaay model.
T L Lix E
aLPC
Lreqoonce ;
The normalization term Tresponsee + é uses the task-specific response loss response, to scale the routing penalty, where a small constant added to prevent division by zero when the model achieves perfect task performance.
A.4
Learned Pathway Complexity and Routing Consistency
A.4.1
Calculation Example
Here provide detailed example calculation of how the expert size penalty is calculated. This penalty is used within the calculation of the routing consistency described in Section 4.1 and also as part of the expert-usage loss from Equation 7.
In this equation, E is the set all experts, 8y the size of each expert, and way is the weight assigned to expert while solving task i. For example, imagine model with three experts: skip connection, a simple expert with 16 neurons, and a complex expert with 32 neurons. To solve a task i, imagine the model sets the weight of the skip connection to 31%, the simple expert to 43%, and the complex expert to 26%. The model's learned pathway complexity (LPC) for task would be 376.3, as follows:
L.C. wi.0%0 1i,18 + wr,2s (0)31)(0) (0.43)(16) + (32) 376.3
(10)
For simplicity, this equation shows how to calculate the model's LPC at a single timestep. To calculate the LPC for an entire task, this value should be calculated and averaged over all of the task's timesteps.
A.4.2 Routing Consistency Across Model Types
Using the calculated LPC values for each task, we determine the consistency of the routing decisions made by different models when solving tasks. In Figure 8, we show a version of Figure 2 with our final model, which includes expert dropout. After the addition of expert dropout, the mean pairwise correlations of the models' routing consistency is 0.51 (p < 0.0001). This is reduction when compared to our model trained only with our custom routing cost and task performance scaling, however, as explained in Section 4.2, the model with the additional dropout does develop self-sufficient pathways on top of the consistency criteria, so that the model including the dropout overall better matches the pathway formation criteria. Our main investigation on what we call the Mixture-of- Hixture-of-Puhhways model includes the expert dropout.
26

---

Baseline
Routing Cost
Task Performance Scaling
Expert Dropout
0.2
0.0
Correlation of Routing Activity Between Runs
Figure 8: Routing consistency as measured by correlation of routing activity across model runs. This version of Figure which also shows the model with expert dropout. Even with expert dropout the routing consistency is significantly above the baseline model (p < 0.0001), even though is slightly reduced when compared the model with routing and task performance scaling.
A.5 Effective Rank
To define the LPC, in Equation
use s² as penalty for using expert j. where sj the expert's hidden dimension, or zero in the case of a skip connection. Intuitively, this was motivated by the Os²)) cost of storing each expert's weight matrix memory, however this only roughly captures the learning capabilities of GRU with s, neurons. For example, as demonstrated by the lottery ticket hypothesis, is possible that experts with large hidden dimensions may converge on low-rank solutions more easily than experts with small hidden dimensions [75].
To ensure that were appropriately penalizing experts relative each other, we analyzed the effective rank of each expert's 's matrix, defined as the participation ratio of the squared sum of singular
values the sum of squared singular values, aaa)) over the course of training [76]. This metric ΣΦ
measures how evenly distributed the singular values are and thus how many dimensions the matrix effectively uses. These large experts is roughly double that of small experts, both before and after training, so that large experts always have a much larger effective rank than small experts (p < 0.0001). We additionally find that all experts across both our model and the baseline model decrease slightly in effective rank over training, but the differences between these models' effective ranks tends to be small and never becomes significant (p 0.05). This supports the conclusion that the hidden dimension is at least good approximation processing complexity, but we encourage future work to consider measuring this with scalar value.
shown in Figure 9. For both models, find that the effective rank of
27

---

Layer 1, Simple Expert
Layer 1, Complex Expert
11.2
Baseline
22.5
Baseline
Our Model
Our Model
Effective Rank
11.0
22.0
10.8
21.5
10.4
21.0
Layer 2, Simple Expert
Layer 2, Complex Expert
22.50
Baseline
Baseline
11.2
Model
22.25
Model
Effective Rank
22.00
11.0
21.75
10.8
21.50
10.6
21.25
Layer 3, Simple Expert
Layer 3, Complex Expert
Baseline
22.5
Baseline
11.2
Model
Our Model
Effective Rank
22.0
11.0
21.5
10.8
21.0
5
10
5
10
Epoch
Epoch
Figure 9: Changes in the effective rank of each expert's weight matrix over the course of training. Results for each configuration averaged across 20 runs.
A.6 Per-Task Accuracy Metries
In Table below, we report accuracy metrics on each task for three models: the baseline HMoE model described in Section 3.1, our model, which includes the routing cost and task- lask-performance scaling described Section 4.1 and the expert dropout described in Section 4.2, and our model without expert dropout.
Table 2: Per-task accuracy metrics.
Task
Baseline
Our Model
Without Dropout
Mean
91.1%
83.0%
90.1%
Median
93.3%
87.5%
91.1%
anti
100.0%
100.0%
99.9%
antiseql
100.0%
99.8%
99.8%
antiseqr
100.0%
100.0%
100.0%
ctxdlydm1
99.5%
99.6%
99.5%
ctxdlydmiing intl
99.8%
100.0%
99.3%
ctxdlydmiintz intr
99.6%
98.4%
99.6%
28

---

Task (cont.)
Baseline
Our Model
Without Dropout
ctxdlydmlseql
83.5%
79.9%
84.0%
ctxdlydm1 seqr
83.2%
84. 1 %
82.3%
ctxdlydm2
75.5%
71.7%
76.3%
ctxdlydm2ing
78.2%
70.9%
79.3%
ctxdlydm2intr
81.2%
82.3%
84.2%
ctxdlydm2seq
95.4%
90.2%
94.3%
ydx22ydmssqqr
90.7%
92.7%
90.6%
ctxdm1
93.0%
90.0%
93.7%
ctxdmiseq]
93.6%
88.6%
91.0%
ctxdm1sequ
82.2%
83.0%
76.4%
ctxdm2
98.7%
97.9%
99.3%
ctxdm2seql
99.5%
97.5%
98.5%
ctxdm2seqr
99.4%
92.1%
99.0%
dlyanti
99.7%
98.4%
99.5%
dlyantiintl
99.8%
39.1%
98.7%
dlyantiintr
99.6%
44.3%
94.1%
dlyantiseq]
96.2%
38.4%
98.4%
dlyantiseqr
98.7%
37.6%
93.4%
dl dlydm1 dlydm1 intl
91.8%
87.5%
88.6%
93.0%
88.5%
91.8%
dl dlydm1d intr
91.7%
89.2%
88.7%
dlydmiseql dl dlydmiseqr
92.8%
86.0%
89.3%
90.5%
90.6%
88.8%
dlydm2
90.2%
87.0%
90.5%
dlydm2intl
92.2%
84.1% 1%
86.8%
dlydm2intr
92.1%
85.1%
88.1%
dlydm2seq]
82.4%
78.0%
74.3%
dlydm2seqr
82.1% 1
78.9%
76.8%
dlygo
80.2%
59.9%
76.0%
dlygoint1
78.5%
60.8%
79.6%
dlygointr
82.6%
61.5%
84.3%
dlygoseql
85.5%
60.4%
82.0%
dlygoseqr
74.3%
62.6%
76.8%
dm1
76.3%
61.7%
75.0%
dm1seql
78.8%
57.3%
78.4%
dm1seqr
81.0%
63.4%
76.6%
dm2
100.0%
98.0%
99.9%
dm2seq]
99.7%
97.6%
99.2%
dm2seqr
100.0%
98.0%
99.9%
dmc
99.9%
95.5%
99.8%
dmcintl
99.8%
99.0%
99.4%
dmcintr
99.5%
98.0%
99.8%
dmcseq]
81.9%
75.2%
81.1%
dmcseqr
81.5%
75.0%
81.7%
dms
75.2%
66.5%
77.2%
dmsintl
73.4%
64.8%
77.3%
dmsintr
79.7%
72.8%
86.0%
dmsseq]
95.1%
87.6%
92.8%
dmsseqr
95.2%
88.6%
90.6%
dnmc
93.2%
87.3%
88.7%
dnmcintl
93.1% %
86.2%
89.7%
dnmcintr
80.8%
79.0%
77.1%
dnmcseql
98.5%
93.0%
98.9%
dnmcseqr
99.7%
94.3%
98.3%
dnms
98.8%
91.3%
98.8%
dnmsintl
99.3%
93.6%
98.1%
dnmsintr
99.9%
99.4%
100.0%
29

---

Task (cont.)
Baseline
Our Model
Without Dropout
dnmsseql
100.0%
97.4%
99.6%
dnmaseqr
100.0%
99.1% %
99,4%
go
100.0%
94.9%
99.7%
goseql
99.9%
96.7%
99.4%
goseqr
100.0%
94.0%
99.7%
multidlydm
81.8%
74.5%
81.9%
multidlydmintt]
78. %
77.3%
80.0%
multidlydmintr
74.1% %
66.7%
76.0%
multidlydmseqi
73.1% %
66.6%
71.8%
multidlydnseqr
79.7%
71.6%
79 19.4%
multidm
94.0%
90.0%
90.7%
multidmseq]
94.3%
88.9%
93.1%
multidmseqr
93.3%
85.9%
92.6%
rtanti
92.6%
86.1%
91.2%
rtantiseq]
82.2%
79.5%
81.6%
rtantiseqr
99.7%
95.2%
98.9%
rtgo
98.7%
94.9%
98.9%
rtgoseql
99.3%
91.9%
98.0%
rtgoseqr
98.1% %
93.1%
99.0%
A.7 Task-Specific Expert Usage Patterns
In Figures 10 and 11, we show two samples of tasks completed by our model. At each timestep, we plot the model's decisions in orange, which overlap with the blue ground truth values, indicating that these trials, the model always returned the correct answer. We additionally plot expert usage of the model each timestep. At each timestep, three bars shown for each layer, with their color indicating the usage of the skip connection, the simple expert, and the complex expert, in that order. In the go trial shown in Figure 10, the model primarily uses the skip connections each timestep during the fixation period, in which only needs to return zero. During the response period, the model moves its computations toward layers and 2, and the skip connection in layer In the more complicated dmcintr trial shown in Figure 11, the model switches between pathways multiple times depending its needs, which vary between working memory and computation. This shows that our model has formed distinct pathway and modes of processing information which it dynamically switches between. As result, the model opens up the possibility of analyzing the detailed dynamics of how pathways are combined over tasks with time courses that include varying computational demands, learn which principles underlie such coordination processes.
complex pathway, primarily using the simple experts in
30

---

Task: go
100%
Stimulus 2
80%
Stimulus
Fixation
17.5
Ground truth
15.0
Action prediction
60%
12.5
Expert Usage
Action
10.0
7.5
2.5
40%
Layer
20%
Layer 2
Layer 1
2
6
7
9
10
11
12
13
14
15
16
17
18
19
20
Timestop
Figure 10: the go task we see that models rely on extremely simple pathways for their decision
making until activating model complex pathways during the response period.
31

---

Task: dmccinr
100%
Staumlus
80%
Stinaulas
Fixation
17.5
Ground trith
prodiction
60%
12.5
Ropert Usage
Action
10.0
7.5
Loyer 3
Layer 2
Layer
076
15 19
20
26
27
28
20
30
31
Timestep
Figure 11: During more complex tasks such as dmeintr we see that models a very dynamic and rich set of pathways throughout the duration of trial to return the correct response.
32

---

A.8 Unclustered Expert Usage Patterns Across Tasks
In Figures 13 and 14, we show the unclustered expert usages during three phases of each task. Each task has a different number of timesteps, so in order to condense expert usage into a single figure, we averaged expert usage over three phases which are shared by each task: a pre-stimulus phase, during which the task identity is known but input data has not yet been presented, stimulus phase, during which the model observing input data, and response phase, during which the model needs to output its responses. In these figures, tasks are sorted based on the same clusters shown in Figure for visual clarity. Notably, the complex expert is rarely used in the first two phases, during which the model outputs zero, but is commonly used during the response phase of complex tasks. When analyzing the average usage of the most complex experts over the clusters shown Figure 4, we find that the cluster with the highest reliance of the most complex experts uses those with an average routing weight of 0.11. In Figure 12 we show the distribution of complex expert usage by layers of the model, over clusters derived in Figure 4, meaning each data points here is one of the 10 clusters.
3
Layer
2
0.00
0.02
0.04
0.06
0.08
0.10
A Average Usage of Complex Expert
Figure 12:
Average usage of the complex experts for each cluster derived in Figure 12, split by layer index.
33

---

Cluster
doc
10075
disoree
divantieegr
stanllerg
dinrseg)
discittrr
muiltsiydm
Chucter
donoeegg
dusint)
di denlegg
duzasor
dlyindsog
dlydn legg
stainbasg
dinbag
diggoeer
antiaegr
donocistr
demzinti
moltidmare
dosistr
Chueter
reselydmm
multidlyydnseop
muttiiinseq
duscial
dusscyl
dunsint
6078
mulidlycuiatt
Loypereeee Eopue LoogP
diggviere
dlyantiietr
alygo
cisdly-dntie
staclle
2ist
stucily
mulisdlyduuste
4018
ctacest
clydmizegl
clysati
ctslnddeeg!
dinleegl
dicantterg!
Chudter
etollydau
diydect
dbydnz2
2013
antisey)
tigo
Chudte
dumssuy
sal
dicantintt
dtsclydmisral
abygoitt
Churter
dtydmiserg!
done
Chueter
ctsdmiergg
dndsegg
dams
Chuster
manniingg
iigreng!
Camesey)
Claster 10
0%
Skip
Simple
Complex
Skip
Stmple
Complex
Ship
Simgll
Comples
Protimalaa Phaze
Stinutuu Plaze
Htepuzan Pitzan
Figure 13: Layer-wise expert usage averaged over three phases of each task: the pre-stimulus phase, during which the task is known but no input data has been given to the model, the stimulus phase, during which input data is being given to the model, and the response phase, during which the model needs to calculate and return the correct response. In each phase, expert usages sum to 100% within each layer, i.c. usages of the skip connection, simple expert, and complex expert of layer denoted by "LI", add up to 100%. Tasks are grouped into 10 clusters, shown along the left, based on similarities in their routing patterns. contrasting version of this figure for a baseline model trained without our cost-based loss and dropout is shown in Figure 14.
34

---

1005%
cirgoeegl
Cluster
dimmsiete
tomeetere
greeyy
dlyunissy
gosegl
Cluster
antisegr
dbygo
dlyautiisir
ctscmal
da22
Churter
clysaki
dasi
dmnc
staat:
dlydm2ittr
4bytmlint!
dmileng
Cluster
muliallyduiatt
etudmisse seg!
dnmrinti
dmsintl
rinntiee)
eodib-mmiatt
muhsllyyduiste
etdltydmlurr
dbsmmaati
Chuster
0075
dome
dume
muttsilydre
dusciate
LADeeoO Deport Usage
dimreey
disgointe
Chaster
dunsinte
dussegl
stedlydan3
dndsegl
4075
dleggintt
dlydnrSegg
<lyydiissgg
stadim2
dlyfeet
coodl-dm2istee
abyda2
ctedlycm1
2056
divoeeg
demoreer
ctadm2aegr
rigneny
rtadisegy
dimenggr
abddnlsor
Claster 10
stadmbleu
multihnssrr
dinlseg
abvdmzeerr
exxibldimbegrr
0%
1.3
Siasol
Complex
Simple
Campiies
Siupple
Compplox
Preetinsiue Phase
Stinutuue Plaae
Reepuren Piaaee
Figure 14: A separate version of Figure 13 for model trained without custom routing cost, loss normalization, and expert dropout. The pathways that form are much less distinct, and are also much less stable.
35

---

A.9 Task Difficulty
A.9.1 Number of Training Steps
To measure each task's difficulty, we train five recurrent neural networks (GRUs), each with 64 neurons, and record how many training steps it takes each model to solve that task to 99% accuracy. The task's difficulty is then reported as the median number of steps from these five training runs. Figure 15 shows these measurements for all 82 tasks in the Mod-Cog task suite [24].
A.9.2 Number Rules
There are many ways to characterize the difficulty of learning problems [77], but no universal complexity measure has been developed to date. We believe "training steps needed to learn the task," as discussed above Appendix A.9.1, is sensible measure because this can measured without any implicit biases, and takes into account task demands such as working memory and any other factors which make inference challenging [77]. At the same time, it remains unclear whether such complexity measure would neatly map onto what humans animals perceive to be "difficult tasks," " which is often linked to the number of rules a task [78]. However, this can be easily tested, as Mod-Cog tasks are created based on combinations of different motil's and rules. For example, Figure 7 shows that the "Delayed Decision Making" task (dlydm1) is an altered version of the standard "Decision Making" task (dm1) with the added "Delay" rule (dly).
We find that our difficulty metric fact correlated with the number rules each task (r = 0.39; p 0.0001, shown in Figure 16), and that our model exhibits an even stronger correlation for the brain-like finding in Figure when this used as the difficulty metric compared to the baseline model (r -0.09, as the actual difficulty metric has two downsides: (a) is a discrete and ordinal measurement from to 4 with less statistical power, and (b) some rules are harder to learn than others (i.e. go vs. dm1). This suggests to us that our current convergence-based complexity measure is, least in this specific task environment, difficulty metric providing better link to both the brain and GRU-based machine learning models.
0.57, p < 0.0001)
0.4288). At the same time, using "number of rules"
36

---

dlygo
ctedlydm2int
caxilydm
rtansiseql
stansiseqr
regoseqr
2tadlydi22ist
etodlydmiing
dlychaa inll
chadllydm22
dlydaal
etselma
etvllydm inal
dlydm2ttte
dic
dlydia2
rtgoseol
dlydm2tatl
lint
eixcial
dm2
dmi
demseeqr
dlyantinnt
dnmaseel
multilyymistt
damcsecl
multidlymmist
ilyantiints
Task Name
mullidlydm
dlygointi
ansisegl
dlygoseal
illygoint
dlyantiseq!
ausisequ
goweer
dlyansiseqr
dimeseqr
dlygowerr
goseal
damaiiat
duussiate dly duols
dissint- 2wwd
dimsial
ctoellydmmse
ctmdlydm seql
etacllyden 2m2e
dlydm2sec)
drociatl
dumciots
dly dlydm2eeqe
demeint!
dlydmi worg
dicciat:
dmilee]]
dm
ctxdm1seql
2txdm22eqr
ctxdm22ad
mutsidmsest
ctadml
mulsidmeed!
yemmer
10a
102
10?
Task Difficulty (training steps to 99% accuracy)
Five measurements of task difficulty made for each task in the Mod-Cog task suite [24]. Tasks are sorted by the median of the five measurements. Interestingly, the tasks form groups around similar difficulty levels, which we indicate with the color of each dot.
Figure 15:
37

---

Training Steps to 99% Accuracy
103
102
10
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Rule Count
Figure 16: The number of rules that form Mod-Cog task is correlated with the number of steps takes single RNN to learn the task (r = 0.39, < 0.0001).
38


Reviews:
[{"review_id": "0MiUFY5JDC", "reviewer": "Reviewer_WDiN", "summary": "The paper studies how task specific processing pathways form in neural nets with mixture of heterogeneous experts. Inspired by brain’s pathways among regions, the work extends the heterogeneous MoE architecture to observe conditions in which pathways form. The authors identify three inductive biases that lead to formation of such pathways:\n\n1. A routing complexity cost to penalize use of larger experts\n2. Scaling routing loss based on the task complexity, reducing the penalty for hard tasks\n3. Stochastic expert dropout during training which leads to self sufficiency\n\nThey train models on the Mod-Cog suite of 82 timeseries cognitive tasks.\nAuthors introduce the learned pathway complexity metric to quantify expert usage per task. Correlating LPC across 20 seeds and clustering tasks that use similar pathways, they show that baseline HMoEs do not form stable pathways (mean LPC correlation ≈0.03).\nAdding routing cost and scaling shows high consistency (mean LPC correlation ≈0.71). Expert dropout retains ≳75% accuracy when pruned, showing pathways become more self sufficient.\n\nThe resulting pathways behavior is inline with known brain dynamics (cortical vs. subcortical engagement for hard vs. easy tasks).", "questions": "1. Can this approach be studied for tasks other than timeseries?\n2. Can authors provide a per task accuracy table to show the effect of routing cost or dropout per task?\n3. Do different architectural choices in Appendix A.1 change pathway formations in any form?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths:\n1. The three inductive biases are chosen to map into principles studied in both neuroscience and machine learning.\n2. Comprehensive ablations experiments reporting statistical significance tests\n3. Authors provide a clean definition for LOC, with a detailed example of LPC calculation that helps readers in understanding the metric\n\nweaknesses\n1. Per task accuracies are not reported", "was_revised": true, "final_justification": "Authors have addressed my concerns and I continue to support acceptance of this work.", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our paper and sharing your ideas for improvements with us. We are excited to see that you liked our work! To address your questions:\n\n**(1: Applicability to non-time series tasks)** Our architecture is built with timeseries data in mind, but it works with static input data as well. In this case, one may want to turn experts and routers into feedforward layers as is typically done for Mixture-of-Experts models [ref.1], but our architecture would work without any changes.\n\n**(2: Task accuracy table)** We provide some summary statistics of task accuracies across models in our answer to your following question, giving some insight into the specific effects of some of our contributions. Dropout in particular is required for certain aspects of pathway formation, but impairs accuracy, as can be seen below and in our paper, on line 242. Note that the baseline model with the highest accuracy does not use any dropout. Nonetheless, all models perform well above chance, which would be 1/16 = 6.25%. The camera-ready version of the manuscript will provide full overview tables in the appendix, with a summary overview in the main text.\n\n**(3: Ablation of parameters in A.1)** We have now run additional ablation experiments, varying the alpha parameter, dropout, router size, and embedding layer. We highlight the effect of these ablations through extracting the correlations shown in Fig.5 and Fig.6, based on 10 models run for each configuration:\n\n|  | **Mean task accuracy** | **Std. dev. task accuracy** | **Median task accuracy** | **Fig.5 correlation** | **Fig.5 p-value** | **Fig.6b correlation** | **Fig.6b p-value** |\n|---|---|---|---|---|---|---|---|\n| Baseline | 91.05%  | 8.93% | 93.26% | -0.01 | 0.6576 | -0.49 | 0.0000 |\n| Our Model ($\\alpha$ = 1e-5; Router dim = 64) | 82.97% | 15.50% | 87.52% | 0.54 | 0.0000 | 0.31 | 0.0039 |\n| No dropout | 90.10% | 8.93% | 91.10% | 0.55 | 0.0000 | 0.03 | 0.7815 |\n| $\\alpha$ = 1e-4 | 69.02% | 20.13% | 69.62% | -0.57 | 0.0000 | -0.37 | 0.0004 |\n| $\\alpha$ = 1e-6 | 89.68% | 9.00% | 92.90% | **0.62** | 0.0000 | 0.18 | 0.1157 |\n| No task embedding | 83.04% | 14.16% | 87.05% | 0.58 | 0.0000 | **0.58** | 0.0000 |\n| Router dim = 32 | 83.15% | 14.52% | 87.15% | 0.46 | 0.0000 | 0.33 | 0.0023 |\n| Router dim = 128 | 81.95% | 16.95% | 85.98% | 0.35 | 0.0013 | 0.27 | 0.0125 |\n\nWe observe that our model lies at an intersection of good accuracy and strong significant correlations for our Fig.5 and Fig.6 findings. The router dimension does not seem to strongly affect the overall pattern. As expected, the correct alpha value is critical: setting it too high leads to an over-emphasis on routing cost, causing the model to give up on learning complex tasks and fail to converge properly. Smaller alpha values increase accuracy, but form weaker pathways, as there is not enough pressure to assign tasks to the simplest possible pathways. Having no dropout naturally helps task performance, as the model now gets to use all experts at all times, but it also has the interesting effect of causing the Fig.6 correlation to disappear. This suggests that the findings of brain-like learning as shown in Fig.6 develops as an interaction of the LPC loss and the dropout. Lastly, we find that removing our task embedding layer leads to the same effects as our main model, but with an even stronger correlation. Including the embedding layer increases training speed by about 250% and was hence helpful for us to conduct our experiments. We will include these results and analysis in our final manuscript.\n\nWe want to thank you again for taking the time to review our work and suggesting improvements to our manuscript. We hope our comments above could clarify all the open questions you had.\n\nReferences:\n\n[1] ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts: https://arxiv.org/abs/2410.15732"}, {"review_id": "b9twrdhx2a", "reviewer": "Reviewer_A6SL", "summary": "This paper studies when MoE architectures produce patterns of expert activation that match certain properties of information routing in the brain. The main findings are that regularizing with a task-dependent measure of expert usage leads that measure to correlate across independent runs of the model and that training with expert dropout makes the models more robust to dropout at test. These interventions also yielded more long-tailed distributions when clustering tasks by their usage patterns.", "questions": "How much do your results depend on details of the router? Inductive bias for sparsity seems especially relevant.\n\nThe denominator in eq 2 doesn’t match the text description, since L_response,i is negatively related to the model’s ability to solve task i. This discrepancy may be related to the undefined L_act,i in the text.\n\nSec 4.2: please state which interventions are done during training and which during evaluation.\n\nSec 4.3, fig 4: Your model clearly shows more structure within each cluster across phases and expert types compared to baseline (this follows unsurprisingly from the regularization in eq 2), but what is the evidence for the claim that it produces more-distinct routing patterns? Within both models, all the clusters seem to show nearly the same pattern.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nClever and novel ways of thinking about brain-like information routing in NNs\n\nDiverse set of experimental evaluations\n\nWeaknesses:\n\nThe notion of pathways needs to be more carefully defined and better justified. I didn’t understand what you meant until eq 1 (even sec 4.0 was vague and unclear). FWIW my guess until sec 4.1 was that pathways meant the experts form paths or chains: high weighting for some expert at one layer is consistently (within a model and across tasks) followed by high weighting for some other expert at the next layer. \n\nI’m also not seeing the motivation for why pathways in your sense is a desirable emergent property.\n\nUsing the same LPC as both a regularizer and a metric biases the results. The task-specific scaling in eq 2 is especially problematic: if the model is explicitly penalized for LPC more on some tasks than others then of course it will consistently produce lower LPC for those tasks.\n\nLikewise the correlation in fig 5 is unsurprising since the model was explicitly penalized for LPC on easier tasks.\n\nThe measure of task difficulty used (training steps needed by a standalone GRU) is too crude for comparison to brain patterns. What matters for patterns of brain activation is not raw difficulty (which can be high for simple tasks like detection in strong noise) but complexity of rules or working memory demands.", "was_revised": true, "final_justification": "The paper offers a novel way of thinking about brain-like information routing in NNs but falls short in several ways. The notion of pathway is too vague. The primary results are built in by the regularization scheme which encourages the model to use certain experts for certain tasks. This also makes the identified pathways only correlational; there's no evidence of synergy or coordination between experts.", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our paper! We are glad that you found our model a clever and novel way of characterizing brain-like information routing in neural networks. We hope the following addresses your concerns and questions. Please note that we are not able to update the PDF at this stage, but all clarifications will be represented in the camera-ready version of the manuscript.\n\n## Weaknesses\n\n**(1: What are pathways)** Thank you for pointing out that this was unclear. We think of pathways as groups of experts that tend to be used in tandem, where being part of one pathway does not prevent an expert from also being part of another pathway. In that sense, they are patterns of “recurring co-usage” that are dynamically activated across tasks and time periods within tasks.\n\n**(2: Why are pathways desirable)** From a neuroscience point of view, we argue that this “recurring co-usage” of regions is what takes place in the brain. There are dominant patterns in which brain regions tend to be co-activated, but any given brain region can be part of multiple different subnetworks, akin to neural reuse [ref.1]. From a large-scale AI point of view, we know that MoE models are parameter efficient, but the ad-hoc loading of experts can limit the speed of execution of these models. Predictable routing has been shown to help that [ref.2; see line 82], and we think pathways provide a good balance between diverse expert usage and predictable routing patterns in a way that resonates with the functional architecture of the brain.\n\n**(3: Bias through using LPC in regulariser and analyses)** We want to address this point using results from three different perspectives: two from our existing manuscript, and one from new investigations we include in the rebuttal.\n\n*(3.1: In relation to Figure 5)* We agree that this specific finding follows logically from the setup of our model once it has been shown to work. However, the mechanisms underlying pathway formation between heterogeneous regions were unknown beforehand: another reasonable hypothesis could have been that the baseline architecture (layers of heterogeneous experts without any inductive biases) learns to allocate computation toward more difficult tasks automatically. Figure 5 shows that this is not the case unless our specific inductive biases are included in the model architecture. We think one of the key benefits of our model is that it delivers an understandable implementation for a set of complex phenomena relating to how pathways form and how they behave in a brain-like fashion.\n\n*(3.2: In relation to other findings in the paper)* Like you said, penalizing the model with the LPC incentivizes it to reduce its LPC for each task. However, if routing decisions were only based on a task’s LPC, we might expect that routing is constant within each trial of a given task. Instead, our model generates nuanced routing dynamics across timescales within single trials of tasks, as can be seen in Figure 12. It discovers that for some groups of tasks, it can use a generalizable routing pattern, while others require a very unique usage of experts. This is shown through clusters on the y-axis of Figure 12 and discussed in Section 4.3 (please note our response to your related question below). These findings suggest that the model has learned a general strategy of allocating computational resources across the 82 tasks. With the model and its overall principles established through this paper, future work can now deliver more detailed investigations on how far this extends to completely new tasks.\n\n*(3.3: New findings from rebuttal in relation to Figure 6)* Motivated by your concerns, we ran some additional experiments that analyze specific aspects of our model (dropout, routing) on LPC and pathway formation. Notably, we found that when trained with our normalized LPC but without dropout, our model exhibits the effect shown in Figure 5, but not the effect shown in Figure 6. This indicates that the finding in Figure 6 is due to an interaction between LPC regularization and dropout, and is not explained by training with LPC regularization alone. We speculate that dropout forces the model to be explicit about which pathway is used in learning and is crucial for creating brain-like learning dynamics.\n\n**(4: Oversimplified task difficulty metric)** There are many ways to characterize the difficulty of learning problems [ref.3], but no universal complexity measure has been developed to date. We believe “training steps needed to learn the task” is a sensible measure because this can be measured without any implicit biases and takes into account task demands such as working memory and any other factor making inference problems challenging [ref.3]. At the same time, we agree it is slightly unclear whether such a complexity measure would neatly map onto what humans perceive to be “difficult tasks,” which is often linked to the number of rules in a task [ref.4], as you suggested. We find that our difficulty metric is in fact correlated with the number of rules in each task (r=0.39; p<0.0001), and that our model exhibits an even stronger correlation for the brain-like finding in Figure 5 when this is used as the difficulty metric (r=0.57, p<0.0001) compared to the baseline model (r=-0.09, p=0.4288). At the same time, using “number of rules per task” as the actual difficulty metric has two downsides: (a) it is a discrete and ordinal measurement from 1 to 4 with less statistical power, and (b) some rules are harder to learn than others (i.e. go vs. dm1). This suggests to us that our current convergence-based complexity measure is, at least in this specific task environment, a difficulty metric providing a valid link to both the brain and GRU-based ML models. \n\n## Questions\n\n**(1: Effect of details of router on results)** We have now run some additional experiments to better understand certain aspects of the router on our model's performance. We highlight the effect of these ablations through extracting the correlations shown in Fig.5 and Fig.6, based on 10 models run for each configuration:\n\n|  | Mean task accuracy | Fig.5 correlation | Fig.5 p-value | Fig.6b correlation | Fig.6b p-value |\n|---|---|---|---|---|---|\n| Baseline | **91.05%** | -0.01 | 0.6576 | -0.49 | 0.0000 |\n| Our Model ($\\alpha$ = 1e-5; Router dim = 64) | 82.97% | 0.54 | 0.0000 | 0.31 | 0.0039 |\n| No dropout | 90.10% | 0.55 | 0.0000 | 0.03 | 0.7815 |\n| $\\alpha$ = 1e-4 | 69.02% | -0.57 | 0.0000 | -0.37 | 0.0004 |\n| $\\alpha$ = 1e-6 | 89.68% | **0.62** | 0.0000 | 0.18 | 0.1157 |\n| Router dim = 32 | 83.15% | 0.46 | 0.0000 | **0.33** | 0.0023 |\n| Router dim = 128 | 81.95% | 0.35 | 0.0013 | 0.27 | 0.0125 |\n\nWe observe that our model lies at an intersection of good accuracy and strong significant correlations for our Fig.5 and Fig.6 findings. The router dimension does not seem to strongly affect the overall pattern. As expected, the correct alpha value is critical: setting it too high leads to an over-emphasis on routing cost, causing the model to give up on learning complex tasks and fail to converge properly. Smaller alpha values increase accuracy, but form weaker pathways, as there is not enough pressure to assign tasks to the simplest possible pathways. Having no dropout naturally helps task performance, as the model now gets to use all experts at all times, but it also has the interesting effect of causing the Fig.6 correlation to disappear. This suggests that the findings of brain-like learning as shown in Fig.6 develops as an interaction of the LPC loss and the dropout. We will include these results and analysis in our final manuscript.\n\n**(2: Denominator in eq.2)** Thank you for spotting this error, which we will correct. $L_{act,i}$ should be written as $L_{response,i}$ in the text, and we will update line 182 to clarify that we are dividing by the model’s loss, not accuracy, on task i.\n\n**(3: Clarify interventions during inference and training)** We apologize for the confusion regarding interventions during evaluation. Dropout is used during training but not evaluation, aligned with what is typically done in ML. The one exception to this is Figure 3, where the “evaluation accuracy” is calculated with dropout enforced according to the threshold given on the y-axis in order to study our “self-sufficiency of pathways” criterion.\n\n**(4: Evidence for more distinct clusters; Sec 4.3, fig 4)** Thank you for flagging this issue. Upon inspection, we noticed an issue with how we generated Figure 4, making our clustered pathways appear much more similar than they actually are. Unfortunately we are unable to share new figures during the rebuttal, but luckily Figure 12 and Figure 13 show more detailed (and correct) routing patterns. While in Figure 13 (baseline) the routing patterns seem very random and unstructured, Figure 12 shows much more structured routing with distinct pathways. For example, notice how clusters 2 and 3 (labeled on the left side of the figure) rely more on increased processing complexity during the response period, while clusters 5 and 6 do so during the prestimulus and stimulus phase.\n\nWe want to thank you again for taking the time to review our work and suggesting improvements to our manuscript. We hope our comments above were able to address the issues you raised in your review.\n\n## References\n\n[1] Neural reuse: A fundamental organizational principle of the brain: https://pubmed.ncbi.nlm.nih.gov/20964882/ \n\n[2] Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference: https://arxiv.org/abs/2410.17954\n\n[3] How Complex is your classification problem? A survey on measuring classification complexity: https://arxiv.org/abs/1808.03591 \n\n[4] Goal Neglect and Spearman’s g: Competing Parts of a Complex Task: https://pubmed.ncbi.nlm.nih.gov/18248133/"}, {"review_id": "YTeEFXl2zZ", "reviewer": "Reviewer_gQat", "summary": "This work explores the formation of processing pathways in mixture-of-experts (MoE) models. The heterogeneous MoE models studied consist of three layers, each with three experts, of which one is an identity connection, one is smaller, and one is larger. The authors first establish several improvements over their baseline to stabilize routing and improve their self-sufficiency. The improved model behaves in a more brain-like fashion, in that it solves simpler tasks without the complex expert more often, and it allocates the complex expert to more complex tasks early in training. These findings suggest that a simple mixture of heterogeneous experts, by itself, does not explain the formation of heterogeneous pathways in the brain, and suggest a potential improvement in MoE models in ML.", "questions": "- See also the limitation below – the work appears predicated on simulations with a single model configuration. Could you support the lack of any ablations, and/or better motivate the specific hyperparameters chosen?\n- In the introduction of the model in Algorithm 1, the routing weights appear to be indexed by the layer and expert, $w_{l,j}$. Later, in the definition of the _Learned Pathway Complexity_ in 4.1, they appear to be indexed by task, timestep, and expert, $w_{i,j,t}$. If my understanding so far is correct, then (1) what happened to the layer indexing in the LPC definition, and (2) perhaps the task could be a superscript, rather than a subscript? My motivation for asking is that I currently don’t understand where layers fit into the LPC definition, as it is currently described. \n- Could you motivate the evaluation with dropout in section 4.2? With standard (unit-wise) dropout, the standard is to downweigh all units by the probability of dropout, rather than continuing to drop out in evaluation. Did you try something of this sort?", "limitations": "The authors addressed several limitations. However, the authors conduct their experiments using a single, small model configuration. It is hard to evaluate to what extent hyperparameter choices (3 layers, 3 experts, expert sizes) influence the findings. This limitation is important because the work is primarily empirical and simulation-driven.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The authors study a problem of interest to both neuroscientists and machine learning in a well-motivated setting, using a simple model of a much more complex phenomenon. The improvements the authors build on their baseline model follow well from the limitations identified, and the analyses are presented well.\n\n**Weaknesses:** The entire set of results is predicated on a set of simulations performed with a single hyperparameter setting that is neither particularly strongly motivated nor ablated. As such, it is hard to disentangle to what extent some of these specific choices (enumerated below in the limitations) drive the observed results, vs. to what extent they are broader features of the class of models studied. Even minor ablations would go a long way in strengthening the work.\n\nOther minor notes/questions for the authors:\n- L132: Would it be correct to say that the input is a $128 \\times 350 \\times 115$ input tensor, accounting for the dimensionality of the input? If not, where does the 115 fit in the 128 by 350?\n- L181: When incorporating the LPC into the model loss, wouldn’t it suffice to add a weight to the LPC (the role $\\alpha$ appears to play)? Could you motivate dividing by the task loss, which would explore as the response loss goes to zero? \n- L188: Continuing from the previous question, this line suggests that the normalization is by $L_{act,i}$, but the equation has the normalization by the response loss. Could you clarify?\n- L265: If some of this analysis is predicated on the difference between the distribution of cluster assignments, (a) to what extent is it sensitive to the initialization of the K-means clustering procedure, and (b) did you conduct any sort of significance testing to validate that the differences are meaningful?", "was_revised": true, "final_justification": "With the added evaluations, ablations, and clarifications, I think this would make a worthwhile contribution to the proceedings, and I updated my score accordingly.", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our paper, for sharing your suggestions, and for your attention to detail. We are glad that you appreciated how our work tackles a complex phenomenon relevant to both neuroscience and ML, and we hope the following can address your concerns and open questions. Please note that we are not able to update the PDF at this stage, but all clarifications will be represented in the camera-ready version of the manuscript.\n\n## Weaknesses\n\n**(1: Ablations)** We have now run additional ablation experiments, varying the alpha parameter, dropout, router size, and embedding layer. We highlight the effects of these ablations through the findings shown in Fig.5 and Fig.6, based on 10 randomly initialized models for each configuration:\n\n|  | Accuracy | Fig.5 correlation | Fig.5 p-value | Fig.6b correlation | Fig.6b p-value |\n|---|---|---|---|---|---|\n| Baseline | **91.03%** | -0.01 | 0.6576 | -0.49 | 0.0000 |\n| Our Model ($\\alpha$ = 1e-5; Router dim = 64) | 82.95% | 0.54 | 0.0000 | 0.31 | 0.0039 |\n| No dropout | 90.29% | 0.55 | 0.0000 | 0.03 | 0.7815 |\n| $\\alpha$ = 1e-4 | 67.51% | -0.57 | 0.0000 | -0.37 | 0.0004 |\n| $\\alpha$ = 1e-6 | 90.00% | **0.62** | 0.0000 | 0.18 | 0.1157 |\n| No task embedding | 82.67% | 0.58 | 0.0000 | **0.58** | 0.0000 |\n| Router dim = 32 | 83.25% | 0.46 | 0.0000 | 0.33 | 0.0023 |\n| Router dim = 128 | 82.17% | 0.35 | 0.0013 | 0.27 | 0.0125 |\n\nThese new ablations do show some interesting new effects. Generally we observe that our model lies at a good intersection of accuracy and strong correlations for Fig.5 and Fig.6, indicating the formation of brain-like pathways. The router dimension does not seem to strongly affect the overall pattern. As expected, selecting the correct alpha value is crucial: if it is too high, the model gives up on learning complex tasks and does not properly converge. If it is too low, the model does not have enough pressure to assign tasks to the most simplest pathway possible, leading to the weak Fig.6 correlation. Having no dropout naturally helps task performance, but it also has the interesting effect of causing the Fig.6 correlation to disappear. This suggests that brain-like learning, as shown in Fig.6, develops as an interaction of the LPC loss and the dropout. Lastly, we find that the model without the embedding layer shows the same effects as our main model, with even stronger correlation. However, including the embedding layer increases training speed by about 250% and was hence helpful for us to conduct our experiments. We will include these additional results and analysis in the camera-ready version of our manuscript.\n\n**(2: Dimensionality of input tensor; L132)** Thank you for pointing this out, the dimensionality of the input tensor is 128x350x115. We will update this in the text.\n\n**(3: Incorporating LPC into the model loss; L181)** You are correct, the loss can be written in a simplified form as $L=L_{fix}+\\sum_i^\\mathcal{T}(L_{response,i}+\\frac{\\alpha LPC_i}{L_{response,i}+\\epsilon})$. This changes the equation slightly, specifically $\\epsilon$ would need to become $\\frac{\\epsilon}{T_i}$ in order to be equivalent. However, in practice, $\\epsilon$ is not sensitive to changes, as it is a very small value used simply to avoid division-by-zero errors. We will verify that it does not change our results and plan to update Equation 2 in the camera-ready version to improve clarity.\n\n**(4: Motivate division by task loss; L181)** This can be motivated from the perspective of both ML and neuroscience. *On the ML side*, it is an established phenomenon that when a loss is made up of two or more parts, one of which is significantly easier to minimize than the other, that this can result in optimization issues such as convergence on local minima, as studied in the context of multi-task optimization [ref.1]. In our case, one such local minimum would involve our model leveraging the simplest pathways possible, reducing the routing loss to zero, while failing to solve most or all tasks correctly. We found that normalization helped us avoid this. *On the neuroscience side*, we know that “cognitive effort,” or “processing power,” is affected by norepinephrine release from the locus coeruleus in the brainstem [ref.2, ref.3]. In our model, this would correspond with usage of the complex expert in each layer. It has been observed that activity in the locus coeruleus peaks during early stages of learning when there are large error signals, and weans as skills are acquired [ref.4]. Normalizing our routing loss by the task performance achieves this same effect. We will add this additional context to our paper in Section 4.1.\n\n**(5: Clarify division term; L188)** Thank you for bringing this error to our attention, $L_{act,i}$ should be $L_{response,i}$. This will be corrected in the text.\n\n**(6: Dependence of cluster assignment on K means parameter)** Thank you for pointing this out, we have now run tests with 10 different K-means initializations. In the original manuscript, we observe that our model tends to develop a couple of dominant routing patterns used for many tasks at the same time, as shown by the longer-tailed distribution. To statistically test for a difference between the clustering patterns of our model and the baseline model we measured the size of the largest observed cluster for each model and found that our model developed more dominant large clusters ($p=0.014$; see line 270). With the new additional tests included, this p-value is further reduced to $p<0.0001$.  \n\n## Questions\n\n**(1: Ablations)** We outline our new ablations above.\n\n**(2: Indexing when Algorithm 1 is introduced)** Thank you for spotting this inconsistency. We added the $l$ subscript in Algorithm 1 since this algorithm needs to explain how $w$ is calculated, whereas Equations 1 and 2 simply assume it exists, but in hindsight this is confusing. We will remove the layer index from Algorithm 1, and move the expert index $j$ to be a superscript in all mentions of $w$. We hope this makes sense, but welcome further suggestions on this point.\n\n**(3: Evaluation with dropout)** We do not enable expert dropout during evaluation, following typical procedure for evaluating standard dropout layers [ref.5]. The one exception to this is Figure 3, where the “evaluation accuracy” is calculated with dropout enabled according to the threshold shown on the y-axis. We now also provide an additional ablation without any dropout used during training, as shown in our first point of this rebuttal. We have not experimented with other forms of dropout.\n \nWe want to thank you again for taking the time to review our work and suggesting improvements to our manuscript. We hope that our comments above clarify all the open questions you had.\n\n## References\n\n[1] A Simple General Approach to Balance Task Difficulty in Multi-Task Learning: https://arxiv.org/abs/2002.04792 \n\n[2] Cognitive effort: A neuroeconomic approach: https://link.springer.com/article/10.3758/s13415-015-0334-y \n\n[3] AN INTEGRATIVE THEORY OF LOCUS COERULEUS-NOREPINEPHRINE FUNCTION: Adaptive Gain and Optimal Performance: https://www.annualreviews.org/content/journals/10.1146/annurev.neuro.28.061604.135709 \n\n[4] Orienting and Reorienting: The Locus Coeruleus Mediates Cognition through Arousal: https://www.sciencedirect.com/science/article/pii/S0896627312008197 \n\n[5] Pytorch documentation on dropout: https://docs.pytorch.org/docs/stable/generated/torch.nn.Dropout.html"}, {"review_id": "0GOKLi9ijw", "reviewer": "Reviewer_VsUx", "summary": "This work proposes a novel regularisation technique to be used on Hetrogenous Mixture of Expert models (HMoEs) which enforces stronger decoupling of experts. When these experts decouple it means that a few experts are necessarily active at any given time, forming a \"pathway\". The work posits three normative criteria used to define a pathway and show that their model - the Mixture of Pathways Model  (MoP). Importantly, these criteria also correlate to more practical metrics such as the robustness of the model to lesions and the amount of neurone used to perform more difficult tasks. Finally, the behaviour of the MoP is compared to the brain where it is show that the MoP, similar to the brain, starts learning by increasing pathway complexity but then moves over time to using less complex pathways and \"subcortical\" regions. Similarly, the model is shown to identify task difficulty and route the task appropriately from within the first epoch, similar to the brain. Baselines HMoEs are shown to display neither biological identified behaviours.", "questions": "To state my above points as questions:\n1. Is counting the number of neurons in an expert a valid proxy for network module complexity?\n2. Could the authors elaborate on some of the tasks within the Mod-Cog task set (Go-NoGo is obvious for the target community, but what does one of the more difficult tasks looks like)?\n3. Is there even an idea from the literature on how the human brain forms clear pathways and performs the resource allocation similar to section 5 (it's fine if not, I just think if there is one it would help to be explicit)?", "limitations": "Yes and the limitations mentioned show a good deal of insight in their own right.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n## Quality\nOverall the work is of a high quality. It is well motivated and grounded firstly in the literature. I appreciate the effort put into contextualising the model and honestly discussing its limitations in neuroscience and machine learning. The hypothesis of the work is clear and the normative criteria are motivated quite well. The tasks used are appropriate for the stated goal of the work and results are interpreted fairly and clearly reflect the hypothesis being evaluated. Overall I think the experimental design of the work is well done, again given the goals of the work.\n\n## Clarity\nThe work is very well written and structured appropriately. Figures are clear with helpful captions. Notation is intuitive and used sparingly. The use of the words \"consistent\", \"self-sufficient\" and \"distinct\" for describing the normative criteria is effective and makes each criteria memorable which aids the clarity of the remainder of the work.\n\n## Significance\nThis work presents some interesting results which should be broadly applicable to the machine learning and neuroscience communities. The highlight for me is the results showing the sensitivity of HMoEs to deactivations on Lines 211 and 212. This should be a point of concern for all who work on MoEs. Conceptually, the work is important and I think the general discussion linking machine learning and neuroscience should also be considered when evaluating the significance of this work. The work speaks well to both communities and the biologically motivated criteria provide principals which could be foundational to future work in machine learning. I do have some qualifications to this which I discuss below, but taking the work on its own terms and considering the paper as a whole (not just the regularisation terms or results) I think there is a lot here which can inform future work.\n\n## Originality\nThe work benefits from being inter-disciplinary here as I have not seen many works compare MoEs directly to explicit principals from neuroscience. The additional regularisation terms themselves are also interesting and the combination of different mechanisms is original (while dropout itself is obviously not). Finally, the experimental design is also original, for example by tracking the initial usage of different pathways after one epoch of training and also tracking LPC over time, which is enabled by the original set of criteria this work introduces. Overall the originality is the strongest point of this work. \n\n# Weaknesses\n## Quality\nMy only concern for quality is the exact definition of LPC, specifically, it is completely agnostic to the behaviour of the model itself. As a result, this definition relies heavily on the assumption that an expert, when used, will use all of its neurons for the given computation. Moreover, even if the model did use all of its neurons there is no guarantee that the actual computation being performed is more complex than a smaller module. For example, it has been shown that neural networks trained from small weights have an inductive bias towards low-rank solutions [1] and in the case of linear modules, the rank of the computation being performed depends entirely on the dataset [2]. While I don't think the LPC metric is wrong or unjustified here, I think some clear discussion is necessary about the implicit assumptions linking number of neurons to pathway complexity. I do not think this invalidates the results and as a whole I still find the experimental results compelling. But take the baseline in Figure 5 for example, this behaviour could likely result from wider networks having a higher propensity to better initialised pathways (like a lottery ticket argument [3]) but then these wider networks could still use fewer neurons or low-rank computations. As most results use the LPC, this does limit the quality of the work for me, and I do hope I am missing something which can be clarified in the rebuttal.\n\n## Clarity\nMy only issue on clarity is that some notation is used before it is introduced or inconsistently. For example, $L_{fix}$ and $L_{response}$ in Algorithm 1. Algorithm 1 is even referenced in the text before this notation is used (so it's more than a placement issue of the text box). Similarly, $m^2$ is not defined on Line 167. On Line 188, $L_{act}$ is being reference but I think this is meant to be $L_{response}$. But these are minor points.\n\n## Significance\nInter-disciplinary work is difficult but crucial, and this should be taken into account when considering the significance of this work. However, this work falls slightly short in speaking to either a machine learning or neuroscience community. For example, the absence of more experiments does limit the applicability of the model to machine learning communities - and even the fact that the benchmarks which are used are left entirely to the appendix, while machine learning communities would like to understand the task setup. On the neuroscience side, relatively little explanation of the potential mechanisms for the criteria in the human brain is given. The biological plausibility of dropout is mentioned and more of these sorts of statements would have helped a lot in grounding the work more clearly in the neuroscience. Similarly, the comparison of simpler and more complex pathway to sub-cortical and cortical regions does not seem to be extremely concrete beyond just the movement of computation between these regions over time. But a similar comparison to the dorsal and ventral striatum would also fit as initial deliberate learning is transitioned into more innate habitual behaviours (I am not saying this makes more sense just saying this comparison would also fit the general description of events). While the experts consist of hidden neurons, making the comparison to cortex more natural than striatum, the work would benefit from a more concrete mapping. However, I reiterate that these complaints are mitigated a lot by the well-done Discussion which pre-empts most of these points and also explains other considerations. It would be impossible to please everyone and I think the work as a whole is very insightful. But it would benefit from a few key points being made to link model computation more explicitly to biological mechanisms and a greater focus on the tasks being performed to inform future work in machine learning.\n\n[1] Huh, Minyoung, et al. \"The low-rank simplicity bias in deep networks.\" arXiv preprint arXiv:2103.10427 (2021).\\\n[2] Saxe, Andrew M., James L. McClelland, and Surya Ganguli. \"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.\" arXiv preprint arXiv:1312.6120 (2013).\\\n[3] Frankle, Jonathan, and Michael Carbin. \"The lottery ticket hypothesis: Finding sparse, trainable neural networks.\" arXiv preprint arXiv:1803.03635 (2018).", "was_revised": true, "final_justification": "My original assessment of the work was positive. My primary two concerns were the weak correlation between neuron counting and module complexity, and the lack of sufficient detail to keep either machine learners or neuroscientists happy.\n\nFrom the rebuttals the authors agreed to add some detail to the main text, which satisfies the second point to a degree. In response to the first point the authors mentioned an experiment on the GRU experts recurrent weights where they measured the participation as a better correlate of module complexity. The participation ratio of the bigger network was larger (which is to be expected and supports their claims) but this metric is not a measure of rank, rather a measure of sensitivity to input. I do acknowledge that no great measure of a network modules rank exists, but the persistence of the environment counting approach still raises questions for me about the claims.\n\nOverall I think the work can be admitted to the conference and presents results of interest to ML and neuroscience communities (I agree with the arguement in the rebuttal outlining the need for more models of neuroscience principals too). This interdisciplinary nature of the work is overall a strength for me and mitigated the lack of detail in places.", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our paper and sharing your ideas for improvements. We are excited to see that you found our work was well motivated, following a suitable experimental design, with results being interpreted fairly and the report being of high quality. We hope the following addresses your open questions. Please note that we are not able to update the PDF at this stage, but all clarifications will be represented in the camera-ready version of our manuscript.\n\n## Questions\n\n**(1: Computational complexity of network)** Thank you for bringing up your ideas surrounding the lottery ticket hypothesis (LTH) and low-rank learning. These are very interesting ideas and align well with several follow up research questions we have since been planning internally. We will address the LTH and low-rank issues separately:\n\n*(1.1: Do we only use large experts for complex tasks because the LTH makes it easier to learn complex tasks there?)* We agree that part of the reason why large experts are used is because this makes learning easier and this idea actually aligns with what we show in our paper. As we discuss in Section 5.2 and Figure 6, the brain (and our model) first learn tasks in “overly complex network configurations,” only after which they are gradually moved to simpler networks. Our model’s explanation for this phenomenon is that initially, learning tasks in overly complex pathways helps learning by avoiding local minima of optimizing routing loss before the task loss [ref.1]. Your LTH viewpoint is a related theoretical point, explaining why it is simpler to learn in the complex networks at first, but Figure 6 also shows that the network then works to gradually push tasks downwards towards the correct “minimal network complexity.” While the network complexity chosen during learning is partially reflective of what supports learning, during training tasks are then moved to pathways that better reflect the task’s actual difficulty.\n\n*(1.2: Does the number of neurons capture the actual processing complexity of each expert)* We appreciate your point that large networks can still converge on low-rank solutions, so that by just using the number of neurons, we do not know for sure whether the larger networks are actually using all of their hidden units to implement more complex functions. Your idea of considering the rank of networks was very helpful for running additional analyses on this. Specifically, we analyzed the effective ranks of our GRU experts’ recurrent weight matrices as the participation ratio of the squared sum of singular values to the sum of squared singular values, (Σσᵢ)²/(Σσᵢ²) [ref.2], which measures how evenly distributed the singular values are and thus how many dimensions the matrix effectively uses. We find that all experts across both model types decrease slightly in effective rank over training, but there is no difference between the two models in how they decrease (p>0.05). For both models, we find that the effective rank of large experts is roughly double that of small experts, both before and after training, so that large experts always have a much larger effective rank than small experts (p = 9e-25). We think these additional analyses support the conclusion that in our case the neuron number is at least a good approximation of processing complexity, and we will add these helpful results to our paper.\n\n**(2: Elaborate on details of ModCog tasks)** Figure 7 in the appendix provides an overview of 6 sample tasks. Generally what we see here and also in the task names in general, as listed on the y-axis in Figure 14 (also in the appendix), is that the 82 tasks in ModCog are created based on combinations of task motifs and task rules. As an example, Figure 7 shows that the “Delayed Decision Making” task (`dlydm1`) is an altered version of the standard “Decision Making” task (`dm1`) with the additional “Delay” (`dly`) rule. One part of what makes a task difficult is the number of rules that make up a task: we find that the number of rules contained in a task correlates well with our ‘GRU learning time’ task difficulty metric ($r=0.39$; $p<0.01$). This is reflected in Figure 14, which shows that the two most difficult tasks are `multidlydmseql` and `multidlydmseqr`. These are decision making tasks, like `dm1` in Figure 7, with three additional rules. These additional rules are `dly` (delay), `multi` (integration across two stimuli), and `seql` or `seqr` (the response needs to increase monotonically in a specific direction; left or right). We are not allowed to include additional visualizations as part of the review, but the official ModCog GitHub contains some visualizations of this if you would like to see them [ref.3]. Nonetheless, we will include this additional context in the camera-ready version of our manuscript.\n\n**(3: More detailed discussion of related neuroscience findings)**\n\n*(3.1: Resource allocation in the brain)* As we briefly mention in line 354, we would expect the routing mechanism to be implemented by thalamic nuclei [ref.4]. If we looked for a mechanism influencing the preference for simple or complex processing, then the most supported related finding would be “cognitive effort,” akin to “processing power,” which is controlled by the release of norepinephrine in the locus coeruleus [ref.5, ref.6, ref.7]. In our model, this “cognitive effort” would translate to the increased use of complex experts. The locus coeruleus projects to thalamic nuclei, presenting a means for influencing the routing to simple or complex processing pathways. An additional mechanism could be found in the hypothalamic hypocretin/orexin neurons which play a specific role in metabolic control and might govern the amount of processing power as a function of metabolic budgets [ref.8]. These hypothalamic neurons similarly have strong projections to thalamic neurons and could also influence routing. We will include this context in our final paper.\n\n*(3.2: Pathways formation in the brain)* Less empirical research exists on how pathways form to begin with. This is partially due to how they develop both within lifetime but also over evolution, making it difficult to study empirically, and making it especially important as a topic for theoretical investigation. We see this as a strong motivation for our work. One factor behind pathways and network formation that has been found across studies is the influence of both spatial metabolic costs and communication optimisation [ref.9, ref.10]. More recent investigations have shown that these metabolic constraints need to be studied alongside functional (task) optimisation to observe how modules and pathways form over time [ref.11, ref.12], suggesting strong interaction effects between structural and functional optimisation goals. We briefly discuss these principles in lines 60-63 and 174-177. The limitations of this existing work come from the fact that they (a) do not have heterogeneous submodules / regions in their models and (b) do not allow for dynamic combination of modules, but instead have static pathways, both of which are likely unrepresentative of the brain’s systems-level architecture [ref.13]. In our work, we extend the idea that pathway formation might be driven by metabolic optimisation (“avoiding complex processing where possible”), but bring it to a new type of model that allows us to capture the brain’s systems-level architecture more truthfully. With this, we were then able to identify the more specific inductive biases (beyond just metabolic cost), and show how dynamics between cortical and subcortical processing / learning arise in this new architecture.\n\n**(4: Additional point from clarity section / clarification on notation)** Thank you for pointing out these errors. $m^2$ should be written as $s_j^2$, and $L_{act,i}$ should be written as $L_{response,i}$ as you said.\n\nWe want to thank you again for taking the time to review our work and suggesting improvements to our manuscript. We hope our comments above could clarify all the open questions you had.\n\n## References\n\n[1] A Simple General Approach to Balance Task Difficulty in Multi-Task Learning: https://arxiv.org/abs/2002.04792 \n\n[2] A scale-dependent measure of system dimensionality: https://www.sciencedirect.com/science/article/pii/S266638992200160X \n\n[3] Mod-Cog tasks for multitask learning: https://github.com/mikailkhona/Mod_Cog \n\n[4] The role of the thalamus in the flow of information to the cortex https://royalsocietypublishing.org/doi/10.1098/rstb.2002.1161 \n\n[5] Cognitive effort: A neuroeconomic approach: https://link.springer.com/article/10.3758/s13415-015-0334-y \n\n[6] AN INTEGRATIVE THEORY OF LOCUS COERULEUS-NOREPINEPHRINE FUNCTION: Adaptive Gain and Optimal Performance: https://www.annualreviews.org/content/journals/10.1146/annurev.neuro.28.061604.135709 \n\n[7] Orienting and Reorienting: The Locus Coeruleus Mediates Cognition through Arousal: https://www.sciencedirect.com/science/article/pii/S0896627312008197\n\n[8] Neurometabolic signaling and control of policy complexity: https://www.biorxiv.org/content/10.1101/2025.02.24.639890v1  \n\n[9] Simple models of human brain functional networks: https://www.pnas.org/doi/abs/10.1073/pnas.1111738109 \n\n[10] A generative network model of neurodevelopmental diversity in structural brain organization https://www.nature.com/articles/s41467-021-24430-z \n\n[11] A single computational objective drives specialization of streams in visual cortex https://www.biorxiv.org/content/10.1101/2023.12.19.572460v1 \n\n[12] Spatially embedded recurrent neural networks reveal widespread links between structural and functional neuroscience findings https://www.nature.com/articles/s42256-023-00748-9 \n\n[13] Building artificial neural circuits for domain-general cognition: a primer on brain-inspired systems-level architecture https://arxiv.org/abs/2303.13651"}]

Author rebuttal:

Meta-review: This paper investigates how task-specific processing pathways form in neural nets with mixture of heterogeneous experts, identifying inductive biases leading to the formation of such pathways. The paper has been generally well received by reviewers, as a novel and well-motivated conceptualization of routing in neural networks within a neuroscience perspective, which elegantly took advantage of its interdisciplinary character to present interesting results of potential broad significance for both the machine learning and neuroscience communities. Reviewers also appreciated the comprehensive ablation experiments and clear reporting of results including statistical significance tests.
Rebuttals also address some of the initial concerns of reviewers, such as the extension of the work to tasks beyond time-series, reporting of accuracy metrics for individual tasks, and the need to run additional ablation experiments to assess the impact of different model parameters. 
Overall, reviewers are in broad agreement that this paper is ready for acceptance.
Decision: Accept (poster)