Title: Brain-Like Processing Pathways Form in Models With Heterogeneous Experts
Venue: NeurIPS 2025
Accepted: yes
Keywords: Processing pathways

Brain-inspired neural networks

Mixture-of-Experts

Modularity

Task complexity

Adaptive processing
Abstract: The brain is made up of a vast set of heterogeneous regions that dynamically organize into pathways as a function of task demands. Examples of such pathways can be found in the interactions between cortical and subcortical networks during learning, or in sub-networks specializing for task characteristics such as difficulty or modality. Despite the large role these pathways play in cognition, the mechanisms through which brain regions organize into pathways remain unclear. In this work, we use an extension of the Heterogeneous Mixture-of-Experts architecture to show that heterogeneous regions do not form processing pathways by themselves, implying that the brain likely implements specific constraints which result in the reliable formation of pathways. We identify three biologically relevant inductive biases that encourage pathway formation: a routing cost imposed on the use of more complex regions, a scaling factor that reduces this cost when task performance is low, and randomized expert dropout. When comparing our resulting Mixture-of-Pathways model with the brain, we observe that the artificial pathways in our model match how the brain uses cortical and subcortical systems to learn and solve tasks of varying difficulty. In summary, we introduce a novel framework for investigating how the brain forms task-specific pathways through inductive biases, and the effects these biases have on the behavior of Mixture-of-Experts models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel regularisation technique to be used on Hetrogenous Mixture of Expert models (HMoEs) which enforces stronger decoupling of experts. When these experts decouple it means that a few experts are necessarily active at any given time, forming a \"pathway\". The work posits three normative criteria used to define a pathway and show that their model - the Mixture of Pathways Model  (MoP). Importantly, these criteria also correlate to more practical metrics such as the robustness of the model to lesions and the amount of neurone used to perform more difficult tasks. Finally, the behaviour of the MoP is compared to the brain where it is show that the MoP, similar to the brain, starts learning by increasing pathway complexity but then moves over time to using less complex pathways and \"subcortical\" regions. Similarly, the model is shown to identify task difficulty and route the task appropriately from within the first epoch, similar to the brain. Baselines HMoEs are shown to display neither biological identified behaviours.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n## Quality\nOverall the work is of a high quality. It is well motivated and grounded firstly in the literature. I appreciate the effort put into contextualising the model and honestly discussing its limitations in neuroscience and machine learning. The hypothesis of the work is clear and the normative criteria are motivated quite well. The tasks used are appropriate for the stated goal of the work and results are interpreted fairly and clearly reflect the hypothesis being evaluated. Overall I think the experimental design of the work is well done, again given the goals of the work.\n\n## Clarity\nThe work is very well written and structured appropriately. Figures are clear with helpful captions. Notation is intuitive and used sparingly. The use of the words \"consistent\", \"self-sufficient\" and \"distinct\" for describing the normative criteria is effective and makes each criteria memorable which aids the clarity of the remainder of the work.\n\n## Significance\nThis work presents some interesting results which should be broadly applicable to the machine learning and neuroscience communities. The highlight for me is the results showing the sensitivity of HMoEs to deactivations on Lines 211 and 212. This should be a point of concern for all who work on MoEs. Conceptually, the work is important and I think the general discussion linking machine learning and neuroscience should also be considered when evaluating the significance of this work. The work speaks well to both communities and the biologically motivated criteria provide principals which could be foundational to future work in machine learning. I do have some qualifications to this which I discuss below, but taking the work on its own terms and considering the paper as a whole (not just the regularisation terms or results) I think there is a lot here which can inform future work.\n\n## Originality\nThe work benefits from being inter-disciplinary here as I have not seen many works compare MoEs directly to explicit principals from neuroscience. The additional regularisation terms themselves are also interesting and the combination of different mechanisms is original (while dropout itself is obviously not). Finally, the experimental design is also original, for example by tracking the initial usage of different pathways after one epoch of training and also tracking LPC over time, which is enabled by the original set of criteria this work introduces. Overall the originality is the strongest point of this work. \n\n# Weaknesses\n## Quality\nMy only concern for quality is the exact definition of LPC, specifically, it is completely agnostic to the behaviour of the model itself. As a result, this definition relies heavily on the assumption that an expert, when used, will use all of its neurons for the given computation. Moreover, even if the model did use all of its neurons there is no guarantee that the actual computation being performed is more complex than a smaller module. For example, it has been shown that neural networks trained from small weights have an inductive bias towards low-rank solutions [1] and in the case of linear modules, the rank of the computation being performed depends entirely on the dataset [2]. While I don't think the LPC metric is wrong or unjustified here, I think some clear discussion is necessary about the implicit assumptions linking number of neurons to pathway complexity. I do not think this invalidates the results and as a whole I still find the experimental results compelling. But take the baseline in Figure 5 for example, this behaviour could likely result from wider networks having a higher propensity to better initialised pathways (like a lottery ticket argument [3]) but then these wider networks could still use fewer neurons or low-rank computations. As most results use the LPC, this does limit the quality of the work for me, and I do hope I am missing something which can be clarified in the rebuttal.\n\n## Clarity\nMy only issue on clarity is that some notation is used before it is introduced or inconsistently. For example, $L_{fix}$ and $L_{response}$ in Algorithm 1. Algorithm 1 is even referenced in the text before this notation is used (so it's more than a placement issue of the text box). Similarly, $m^2$ is not defined on Line 167. On Line 188, $L_{act}$ is being reference but I think this is meant to be $L_{response}$. But these are minor points.\n\n## Significance\nInter-disciplinary work is difficult but crucial, and this should be taken into account when considering the significance of this work. However, this work falls slightly short in speaking to either a machine learning or neuroscience community. For example, the absence of more experiments does limit the applicability of the model to machine learning communities - and even the fact that the benchmarks which are used are left entirely to the appendix, while machine learning communities would like to understand the task setup. On the neuroscience side, relatively little explanation of the potential mechanisms for the criteria in the human brain is given. The biological plausibility of dropout is mentioned and more of these sorts of statements would have helped a lot in grounding the work more clearly in the neuroscience. Similarly, the comparison of simpler and more complex pathway to sub-cortical and cortical regions does not seem to be extremely concrete beyond just the movement of computation between these regions over time. But a similar comparison to the dorsal and ventral striatum would also fit as initial deliberate learning is transitioned into more innate habitual behaviours (I am not saying this makes more sense just saying this comparison would also fit the general description of events). While the experts consist of hidden neurons, making the comparison to cortex more natural than striatum, the work would benefit from a more concrete mapping. However, I reiterate that these complaints are mitigated a lot by the well-done Discussion which pre-empts most of these points and also explains other considerations. It would be impossible to please everyone and I think the work as a whole is very insightful. But it would benefit from a few key points being made to link model computation more explicitly to biological mechanisms and a greater focus on the tasks being performed to inform future work in machine learning.\n\n[1] Huh, Minyoung, et al. \"The low-rank simplicity bias in deep networks.\" arXiv preprint arXiv:2103.10427 (2021).\\\n[2] Saxe, Andrew M., James L. McClelland, and Surya Ganguli. \"Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.\" arXiv preprint arXiv:1312.6120 (2013).\\\n[3] Frankle, Jonathan, and Michael Carbin. \"The lottery ticket hypothesis: Finding sparse, trainable neural networks.\" arXiv preprint arXiv:1803.03635 (2018).\n\nQUESTIONS:\nTo state my above points as questions:\n1. Is counting the number of neurons in an expert a valid proxy for network module complexity?\n2. Could the authors elaborate on some of the tasks within the Mod-Cog task set (Go-NoGo is obvious for the target community, but what does one of the more difficult tasks looks like)?\n3. Is there even an idea from the literature on how the human brain forms clear pathways and performs the resource allocation similar to section 5 (it's fine if not, I just think if there is one it would help to be explicit)?\n\nLIMITATIONS:\nYes and the limitations mentioned show a good deal of insight in their own right.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores the formation of processing pathways in mixture-of-experts (MoE) models. The heterogeneous MoE models studied consist of three layers, each with three experts, of which one is an identity connection, one is smaller, and one is larger. The authors first establish several improvements over their baseline to stabilize routing and improve their self-sufficiency. The improved model behaves in a more brain-like fashion, in that it solves simpler tasks without the complex expert more often, and it allocates the complex expert to more complex tasks early in training. These findings suggest that a simple mixture of heterogeneous experts, by itself, does not explain the formation of heterogeneous pathways in the brain, and suggest a potential improvement in MoE models in ML.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The authors study a problem of interest to both neuroscientists and machine learning in a well-motivated setting, using a simple model of a much more complex phenomenon. The improvements the authors build on their baseline model follow well from the limitations identified, and the analyses are presented well.\n\n**Weaknesses:** The entire set of results is predicated on a set of simulations performed with a single hyperparameter setting that is neither particularly strongly motivated nor ablated. As such, it is hard to disentangle to what extent some of these specific choices (enumerated below in the limitations) drive the observed results, vs. to what extent they are broader features of the class of models studied. Even minor ablations would go a long way in strengthening the work.\n\nOther minor notes/questions for the authors:\n- L132: Would it be correct to say that the input is a $128 \\times 350 \\times 115$ input tensor, accounting for the dimensionality of the input? If not, where does the 115 fit in the 128 by 350?\n- L181: When incorporating the LPC into the model loss, wouldn’t it suffice to add a weight to the LPC (the role $\\alpha$ appears to play)? Could you motivate dividing by the task loss, which would explore as the response loss goes to zero? \n- L188: Continuing from the previous question, this line suggests that the normalization is by $L_{act,i}$, but the equation has the normalization by the response loss. Could you clarify?\n- L265: If some of this analysis is predicated on the difference between the distribution of cluster assignments, (a) to what extent is it sensitive to the initialization of the K-means clustering procedure, and (b) did you conduct any sort of significance testing to validate that the differences are meaningful?\n\nQUESTIONS:\n- See also the limitation below – the work appears predicated on simulations with a single model configuration. Could you support the lack of any ablations, and/or better motivate the specific hyperparameters chosen?\n- In the introduction of the model in Algorithm 1, the routing weights appear to be indexed by the layer and expert, $w_{l,j}$. Later, in the definition of the _Learned Pathway Complexity_ in 4.1, they appear to be indexed by task, timestep, and expert, $w_{i,j,t}$. If my understanding so far is correct, then (1) what happened to the layer indexing in the LPC definition, and (2) perhaps the task could be a superscript, rather than a subscript? My motivation for asking is that I currently don’t understand where layers fit into the LPC definition, as it is currently described. \n- Could you motivate the evaluation with dropout in section 4.2? With standard (unit-wise) dropout, the standard is to downweigh all units by the probability of dropout, rather than continuing to drop out in evaluation. Did you try something of this sort?\n\nLIMITATIONS:\nThe authors addressed several limitations. However, the authors conduct their experiments using a single, small model configuration. It is hard to evaluate to what extent hyperparameter choices (3 layers, 3 experts, expert sizes) influence the findings. This limitation is important because the work is primarily empirical and simulation-driven.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies when MoE architectures produce patterns of expert activation that match certain properties of information routing in the brain. The main findings are that regularizing with a task-dependent measure of expert usage leads that measure to correlate across independent runs of the model and that training with expert dropout makes the models more robust to dropout at test. These interventions also yielded more long-tailed distributions when clustering tasks by their usage patterns.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nClever and novel ways of thinking about brain-like information routing in NNs\n\nDiverse set of experimental evaluations\n\nWeaknesses:\n\nThe notion of pathways needs to be more carefully defined and better justified. I didn’t understand what you meant until eq 1 (even sec 4.0 was vague and unclear). FWIW my guess until sec 4.1 was that pathways meant the experts form paths or chains: high weighting for some expert at one layer is consistently (within a model and across tasks) followed by high weighting for some other expert at the next layer. \n\nI’m also not seeing the motivation for why pathways in your sense is a desirable emergent property.\n\nUsing the same LPC as both a regularizer and a metric biases the results. The task-specific scaling in eq 2 is especially problematic: if the model is explicitly penalized for LPC more on some tasks than others then of course it will consistently produce lower LPC for those tasks.\n\nLikewise the correlation in fig 5 is unsurprising since the model was explicitly penalized for LPC on easier tasks.\n\nThe measure of task difficulty used (training steps needed by a standalone GRU) is too crude for comparison to brain patterns. What matters for patterns of brain activation is not raw difficulty (which can be high for simple tasks like detection in strong noise) but complexity of rules or working memory demands.\n\nQUESTIONS:\nHow much do your results depend on details of the router? Inductive bias for sparsity seems especially relevant.\n\nThe denominator in eq 2 doesn’t match the text description, since L_response,i is negatively related to the model’s ability to solve task i. This discrepancy may be related to the undefined L_act,i in the text.\n\nSec 4.2: please state which interventions are done during training and which during evaluation.\n\nSec 4.3, fig 4: Your model clearly shows more structure within each cluster across phases and expert types compared to baseline (this follows unsurprisingly from the regularization in eq 2), but what is the evidence for the claim that it produces more-distinct routing patterns? Within both models, all the clusters seem to show nearly the same pattern.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how task specific processing pathways form in neural nets with mixture of heterogeneous experts. Inspired by brain’s pathways among regions, the work extends the heterogeneous MoE architecture to observe conditions in which pathways form. The authors identify three inductive biases that lead to formation of such pathways:\n\n1. A routing complexity cost to penalize use of larger experts\n2. Scaling routing loss based on the task complexity, reducing the penalty for hard tasks\n3. Stochastic expert dropout during training which leads to self sufficiency\n\nThey train models on the Mod-Cog suite of 82 timeseries cognitive tasks.\nAuthors introduce the learned pathway complexity metric to quantify expert usage per task. Correlating LPC across 20 seeds and clustering tasks that use similar pathways, they show that baseline HMoEs do not form stable pathways (mean LPC correlation ≈0.03).\nAdding routing cost and scaling shows high consistency (mean LPC correlation ≈0.71). Expert dropout retains ≳75% accuracy when pruned, showing pathways become more self sufficient.\n\nThe resulting pathways behavior is inline with known brain dynamics (cortical vs. subcortical engagement for hard vs. easy tasks).\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n1. The three inductive biases are chosen to map into principles studied in both neuroscience and machine learning.\n2. Comprehensive ablations experiments reporting statistical significance tests\n3. Authors provide a clean definition for LOC, with a detailed example of LPC calculation that helps readers in understanding the metric\n\nweaknesses\n1. Per task accuracies are not reported\n\nQUESTIONS:\n1. Can this approach be studied for tasks other than timeseries?\n2. Can authors provide a per task accuracy table to show the effect of routing cost or dropout per task?\n3. Do different architectural choices in Appendix A.1 change pathway formations in any form?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}