Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specialization
Authors: Badr AlKhamissi
C. Nicolò De Sabbata
Greta Tuckute
Zeming Chen
Martin Schrimpf
Antoine Bosselut
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture of Experts
Functional Specialization
Brain-Inspired AI
Interpretability
Behavioral Alignment
Primary area: applications to neuroscience & cognitive science
TL;DR: MiCRo is a brain-inspired modular transformer with interpretable experts that match or exceed baselines and can be steered via expert ablations.

Abstract: Human cognitive behavior arises from the interaction of specialized brain networks dedicated to distinct functions, such as language, logic, and social reasoning. Inspired by this organization, we propose Mixture of Cognitive Reasoners (MiCRo): a modular, transformer-based architecture post-trained with a curriculum that induces functional specialization across experts. Concretely, we partition the layers of a pretrained language model into four expert modules aligned with well-studied cognitive networks in the human brain. MiCRo offers three key advantages over standard language models. (1) The specialized experts are interpretable and causally meaningful---ablating a module causes substantial drops on benchmarks requiring its specialized domain. (2) MiCRo's behavior can be dynamically steered at inference time by routing tokens to particular experts (e.g., favoring social over logical reasoning), enabling fine-grained control over outputs. (3) MiCRo outperforms or matches comparable baselines on both machine-learning reasoning benchmarks (e.g., GSM8K, BBH) and alignment to human behavior (CogBench), while maintaining interpretability. Taken together, cognitively grounded functional specialization yields models that are both more human-like and more human-interpretable.

Published as a conference paper at ICLR 2026
MIXTURE OF COGNITIVE REASONERS: MODULAR REASONING WITH BRAIN-LIKE SPECIALIZATION
Badr AlKhamissi*
C. Nicolò De Sabbata'
Greta Tuckute2.3
Zeming Chen
Martin Schrimpf
Antoine Bosselut*1
EPFL
Brain and Cognitive Sciences at MIT
Kempner Institute at Harvard University
ABSTRACT
Human cognitive behavior arises from the interaction of specialized brain net- works dedicated to distinct functions, such as language, logic, and social reason- ing. Inspired by this organization, propose Mixture of Cognitive Reasoners (MiCRO): modular, transformer-based architecture post-trained with a curricu- lum that induces functional specialization across experts. Concretely, we partition the layers of a pretrained language model into four expert modules aligned with well-studied cognitive networks the human brain. MICRO offers three key advantages over standard language models. (1) The specialized experts are in- module causes substantial drops benchmarks requiring its specialized domain. (2) MiCRo's behavior be dynamically steered at inference time by routing tokens to particular experts (e.g.. favoring social over logical reasoning), enabling fine-grained control over out- puts. (3) MICRO outperforms or matches comparable baselines on both machine- learning reasoning benchmarks (e.g., GSM8K, BBH) and alignment human behavior (CogBench), while maintaining interpretability. Taken together, cogni- tively grounded functional specialization yields models that both more human- like and more buman-interpretti
terpretable and causally meaningful--abattnn
INTRODUCTION
Neuroscience research suggests that distinct brain regions support language, reasoning, social cog- nition, and other cognitive functions (Saxe & Kanwisher 2003 Kanwisher 2010 Fedorenko al., 2024). In contrast, the internal organization of Large Language Models (LLMs) is largely unstruc- tured. While certain units subnetworks show selective activation (Zhang al. 2022; 2023 Bayazit et al. 2023 AIKhamissi et al. 2025a, Wang et al. 2025), such specialization is implicit and difficult interpret or control. Motivated by this discrepancy, we propose model architec- ture that explicitly incorporates specialization. On the machine learning (ML) side, such designs hold great potential for improving interpretability and controllability; on the cognitive science side, they provide a framework toward formulating testable computational hypotheses about how the rel- ative contributions of different brain networks support complex behavior. To this end, we propose the Mixture of Cognitive Reasoners (MICRo), a class of modular language models that partition computation across brain-inspired expert modules.
The MICRO architecture partitions each layer of pretrained language model into four experts, each designed to mirror major cognitive network in the human brain: language (Fedorenko et al. 2011). logic (multiple demand; Duncan 2010), social reasoning (theory of mind: (axe & Kanwisher 2003). and world knowledge (default mode; Gusnard et al.) 2001). To provide the model with the inductive bias needed learn this partitioning and cohesively integrate these experts, we design three-stage
"Equal Supervision
'Code, data and models available at aalcomiiivve-ccaaoerrrs......d
Demo available

---

Published as a conference paper at ICLR 2026
(a) Neuroscience Inspiration
(b) Model Architecture
(c) Expert Ablations
(d) Token Routing Example
Prompt
Emiaa hides candles in box. Jack secruity eats When Emnma returns, how manty candles are in the box, and how does she reacts when she fines out?
Risger Tuturs
MATH
Benchmark
Response
Partumane
Ablent Lang
Abiate Social
Abiata World
Lung
Logic
Social
World
Ablute
Figure I: Brain-Inspired Modular Language Model. (a) Illustration of major cognitive networks in the human brain. (b) Our proposed Mixture of Cognitive Reasoners (MICRO) architecture. The MICRO architecture partitions each transformer block into four expert modules corresponding to analogous brain networks; a router assigns each token to an expert at every layer (i.e., assignments can vary across layers and tokens). (c) Illustration for causal steering via mechanistic ablations: removing module shifts behavior and degrades domain-relevant performance. (d) Token-level routing on a sample prompt shows semantically coherent expert usage.
curriculum that uses lightweight training in the first two stages to sequentially (1) specialize the experts to mirror cognitive networks, and (2) bias router to use certain experts for particular types of inputs (e.g., the logic expert for mathematics problems). The final training stage of this curriculum uses this now inductively- biiaee
biased architecture to perform large-scale supervised finetuning.
Our results demonstrate that MICRO's architecture and training procedure induce interpretable spe- cialization across these experts. This evidenced by routing patterns and their correlations with human judgments ($5.1) and by causal ablations, which show dramatic drops in performance on reasoning categories when their corresponding experts are removed (5..). Moreover, the semantic behavior of these experts parallels the specialization of brain networks: (1) functional localizers
used to recover brain-like mechanisms in LLMs (AlKhamissi et al., 2025a) identify the relevant experts in MICRO ($5.3), and (2) large MICRO models achieve high behavioral alignment scores on COGBENCH (Coda-Forno et al.] 2024), a human behavioral benchmark, relative to two critical controls trained the cialization (MoB) and (ii) a non-modular dense transformer (DENSE) ($5.4). Finally, we find that MICRO's performance matches or exceeds these baselines ($5.5), indicating that interpretable and controllable specialization can be achieved without sacrificing overall performance.
data: (i) mixture-of-expprr model without induced brain-like spe-
2
BACKGROUND & RELATED WORK
2.1 NEUROSCIENCE MOTIVATION
Our design follows evidence that human cognition emerges from interacting, specialized brain net- works. Cognitive neuroscience has mapped this modular organization by measuring how strongly different regions engage when people perform specific cognitive tasks (Kanwisher] 2010). We align MICRo's 's architecture with four cognitive networks as shown in Figure [a). We summarize the functions of these networks below and their relevance to our modeling approach.
The Language Network. The Language expert mirrors the human language network, which com- prises set of left-lateralized frontal and temporal regions that selectively respond to linguistic input over perceptually matched non- -ingiuuuistic stimuli (e.g., lists of nonwords; |eedrenkoo et al.) 2010). These regions are highly specific to language, showing minimal activation during tasks such arith- metic or music perception (Fedorenko et al., (0122) 201T). and their disruption can lead to selective language deficits (aphasia) without impairing general reasoning capabilities (Varley et al. 2005).
2

---

Published as a conference paper at ICLR 2026
The Multiple Demand Network. The Logic expert mirrors the Multiple Demand (MD) network, which spans bilateral regions and is activated across diverse cognitively demanding tasks such as difficult math problems, with stronger responses for higher difficulty levels (Duncan & Owen 2000 Fedorenko et al. 2013). It correlates with fluid intelligence (Woolgar et al. 2010).
The Theory of Mind Network. The Social expert mirrors the Theory of Mind (ToM) network, which is centered in the bilateral cumporo-parictte junction and medial prefrontal cortex. This net- work supports reasoning about beliefs, intentions, and mental states (Gallagher et al. [2000] [Saxe & Kanwisher 2003) [axe & Powell| 2006). It is robustly recruited across both verbal and non-verbal tasks involving perspective-taking and indirect communication (Koster-Hale & Saxe 2013).
The Default Mode Network. The World expert mirrors the Default Mode Network (DMN), which active during rest and internally directed thought such as self-reflection, memory recall, and men- tal simulation (Gusnard al., 2001; BBccknr et al. 2008 Buckner & DiNicola, 2019). Centered in medial prefrontal and parietal regions, the DMN integrates information over long timescales, supporting discourse- and event-level processing across sentences or episodes (Ferstl & von Cra- mon 2002 Jacoby & Fedorenko 2020), in contrast to the shorter temporal window of the language network (Blank& & Fedorenko 2020).
2.2 MODULAR LANGUAGE MODELS
In parallel with advances in cognitive neuroscience, recent years have seen growing interest in mod- ular language models as a way promote specialization, mitigate interference, and improve out-of- distribution generalization (Pfciffer et al. 2023] Zhang et al. 2025). One major line of work centers on Sparse Mixture-of of- of-Experts (MoE) architectures (Shazeer et al.
2017 with approaches ranging
from curating domain labeled datasets to train (Gururangan et al. 2022) or prompt (Si et al. 2023)
domain-specific experts, to frameworks such as ModuleFormer (Shen et al. 2023), which introduce load-balancing and concentration losses to encourage modular specialization without explicit do- main labels. Other modular approaches extend to multimodal integration (Liu et al., 2023 Swamy et al. 2023 Ye et al. (2023) to disentangling representations by domain or language for multilin- gual and domain-specifte applications (Piciffer et al.) 2020 2022] Zhong et al. 2022 Al-Maamari eet al. 2024). In contrast, MICRO is, to our knowledge, the first modular language model explicitly designed to induce brain-like specialization, aligning experts with well-studied cognitive networks.
2.3 BRAIN-INSPIRED MODELS
Recent studies have shown that some models achieve strong alignment with activity in the human
language network (Schrimpf et al., 2222]] Toneva & Wehbe 2019 Caucheteux & King 2022 Aw
et al. 2023 Tuckute al. 2024a AlKhamissi et al. [025bb). To further improve brain aligmment,
researchers have begun to integrate biologically inspired principles into model design drrwinn
from structures like the visual cortex hierarchy (Kubilius et al. 2019 Dapello al. 2020 Spoerer
et al.
2020). and the spatio-functional organization of the brain (Margalit et al. 2024 Binhuraib
al.
2025 Rathi al. 2025).
3 THE MIXTURE OF COGNITIVE REASONERS FRAMEWORK
3.1 MODEL ARCHITECTURE
To build MICRO, we begin with a pretrained transformer-basee backbone. For each layer, we clone the entire transformer block N times, where N corresponds to the number of experts intended for specialization, in similar spirit to parameter upcycling (Komatsuzaki et al.) 2023) Zhang et al.) 2024]. Then, we initialize a MLP-based router that assigns each token to a single expert. To maintain computational efficiency and comparable number of active parameters to the original model, we use top-1 routing akin to Fedus et al. (2022). We refer to this architecture as mixture-oo of--boocks (MoB), distinguishing it from the more common to FFN layers with shared attention. Importantly, we focus on MoB in the main paper because it induces clear functional specialization in all models, as reflected by lower router entropy and
(MOE), which restricts experts
3

---

Published as a conference paper at ICLR 2026
Stage-I Prime Experts
Stage-ll Calibrate Router
Stage-Ill Train End-to-End
Input Example
Prompt
Sarthh bakke cookies for
MIGRosFT Pseudo-Labols (-3K samples)
MiCRosFT (-3K semples)
Tulu3gFT (-1M samples)
fiardds but one couldnt come
How yy cook per friend now, say??
Pseudo- Law.odd---beeedddddooo Response
Sarch feel SP disappointed.
Siee could "Mor for two" with smile.
special
Language
Logic
4 Term
Social
World
Tabey
Training Curriculum for Inducing Specialization. Our brain brain-inspired Mixture of Cognitive Reasoners (MiCRo) model contains four experts per layer, each aligned with distinct cognitive network in the brain. In Stage-l, we train only the experts using a small, curated dataset MICROsFT (see example on the left), providing each expert with initial inductive bias. In Stage- II. we freeze the whole model and train the router on the same dataset to learn expert selection. In Stage-III, finetune the entire model end-to-end on large-scale instruction tuning dataset.
Figure 2:
domain-consistent routing patterns, whereas MoE does not exhibit the same effect at specific scales (see Appendix H.I.. Results for MICRO-MOE variants on reasoning benchmarks in Appendix H.2
3.2 TRAINING CURRICULUM FOR INDUCING SPECIALIZATION
We induce functional specialization in MICRO experts using a three-stage training curriculum (see Figure2). The first two stages use a small, curated dataset (MICROsFT) to provide targeted inductive biases, allowing specialization to emerge and solidify during the final full-scale training stage.
Stage Inducing Specialization. the first stage, we train the experts on small dataset of M = =055 examples (described below), each crafted to reflect the functional domain of a specific expert (Section 2.1). We denote this dataset as MICROsFT = {(TT,,ITT. 1.T, )) where cach input sequence I; contains T; tokens, and r; provides token-level routing labels. Each label ri, {1...., assigns the t-th token to one of the N experts. This stage focuses solely on training the expert parameters using next-token prediction loss. Tokens attend all preceding tokens in the sequence regardless of which expert processed them using the key and value representations produced by the same expert. However, only tokens that are assigned to the expert in question continue to be processed through the feed-forward network. The same setup is applied in the next training stages, with the only difference that the router assigns the tokens the experts.
Stage 2: Calibrating Router. Next, we freeze the whole model and train only the routers on the same dataset MICROsFT- The objective remains next-token prediction. Given the initial expert specialization from Stage 1, the router now learns to assign tokens to the most suitable expert. To encourage smoother transitions and more robust routing decisions, we use a soft mixture of the top-2 experts per token, which we found to be more effective than top-1 routing during this phase.
Stage 3: End-to-End Supervised Finetuning. Finally, we finetune the entire model end-to-end on full iirrucctionttnnnng dataset, TüLU-3 (Lambert et al. 2024), which consists of 939k examples. Even though this phase constitutes the majority of the training budget, we observe that the functional specialization seeded by the small MICROsFT dataset is largely preserved (see Appendix D). More- over, the experts continue to improve on tasks aligned with their initial domains, demonstrating that early inductive biases can lead to meaningful and lasting functional decomposition.
Constructing the MICROSFT Dataset. To build MICROSFT for inducing expert and router spe- cialization, first selected 19 existing reasoning datasets corresponding to the cognitive domains
4

---

Published as a conference paper at ICLR 2026
Language Samples
Logic Samples
Samples
Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
Social
Social
Social
World
World
World
World
CPF-5 Genemite Samples
Soreptes
Samples
fear"
pout hawa: "ho
D: Solse borg both Than vide
12 thep
spoaking
do A.Scc
sarig rien e'Mtagyyeaaoo
uder
She cin med cifrre neatty tneeea
rade
losar sonnd
D: Whad is nod nambernn wous vou
Or Worrs re gen inte trinnd
aiteet hoppe in you lows outr
aongeiotl
A Potten couriee warh trew. After conne 20
Heviinge vatter thot blinnd boond
A They Aorer berou on hast
Figure 3: Semantically Meaningful Routing Across Experts. Token routing patterns in MICRO- LLAMA-IB. Each bar indicates the proportion of tokens routed to a given expert across layers, with variance shown across sentences (n=50). The model exhibits clear Nomainn-spccife specialization consistent with the intended brain-inspired organization. For example, social cognition samples are routed to the social expert, while arithmetic tasks routed to the logic expert. We find that the language expert is consistently activated in the early layers (see Appendix C for layer-wise routing plots and results from additional models). Two random samples are shown below each subplot.
of our non-language experts, ensuring that each group of datasets spanned diverse range of func- tions known to engage the corresponding brain networks. From each of the three sets, we randomly sampled 1,000 examples and used OpenAl's s ol model (Jaech et al. 2024 to generate detailed, step- st-step responses for each input. We then pseudo-labeled each sentence in the generated reasoning chains by prompting GPT-40 (Hurst et al. 2024) to assign it to one of the four experts. The tokens within each sentence inherit the corresponding expert label, which is used for deterministic routing in Stage 1. Details of the datasets are provided in Appendix A
EXPERIMENTAL SETUP
We post-train five models of varying scales from three different families under our MICRO frame- work, in order to assess the generalizability of our method and identify the conditions under which it fails. Specifically, we use LLAMA-3, 2- [B,, 3B) (Dubey et al. 2024). SMOLLM2-(135MM
360M) (Allal et al., 2025). ). and OLMo-2-1B 1B (OLMo et al. 2024). Due space constraints, we present the results of -.A.M-3..24. (1B, 3B} in the main paper while providing the full results for the remaining models in Appendix F Each model is first finetuned for two epochs on the curated MICROsFT dataset (Stages and 2), followed by one epoch of end-to-end training using the TČLU- dataset (Lambert et al. 2024). as described in Section 3.2, We use next token prediction as the only learning objective in all training stages, with the loss masked the input tokens. We use an effective batch size of 32 and the Adam optimizer across all stages. The learning rate follows a linear schedule, warming up over the first 3% of training to peak of 2 10-5, then decays linearly for the remainder of training. This schedule is applied for each stage separately.
Reasoning Benchmarks. We evaluate on four widely used reasoning benchmarks: GSM8K (Cobbe et al. 202T). MATH (Hendrycks et al., (0211), BBH (Suzgun et al. (202). and MMLU (Hendrycks al.] (021a). Evaluation follows zero- or fewshot settings as detailed in Appendix
Behavioral Benchmarks, We evaluate alignment to human behavior using COGBENCII bench- mark (Coda-Forno et al., 2024). which provides 10 metrics from cognitive psychology experi- ments. These metrics capture how participants (or models) complete tasks that are designed to dis- entangle different behavioral strategies. Examples include Directed Exploration, Meta- Cognition, and Risk Taking. We refer readers to Coda-Forro et al. 2024 for detailed description of the tasks.
5

---

Published as a conference paper at ICLR 2026
GSMBK
MMLU
MICRo-Liama- MICRo-LIamas Pho--Llam-11
-35-
Legend
Se/inderr 29.73
Social
World
-4
AMAri 1142%
Figure 4: Expert Ablations Reveal the Causal Contributions of Specialized modules. Top and bottom panels show results for MICRO-LLAMA- 1B and -LCCO-LLLMM-3B Removing the Logic expert causes large drops on MATH and GSM8K, while removing the Social expert yields slight gains. For MMLU and BBH, results indicate that some group of subtasks rely on distinct experts, whereas others draw on overlapping contributions, Additional models in Appendix
RESULTS & ANALYSIS
Our results unfold two parts. First, we ask whether brain-like specialization emerges under our training curriculum, analyzing routing behavior, correlations with human judgments, causal abla- tions to test the functional contributions of those experts, and whether neuroscience experiments used identify brain networks also identify the corresponding experts in our models. Second, ask how this specialization influences alignment with human behavior and reasoning performance.
5.1 ROUTER PATTERNS ARE INTERPRETABLE AND CONSISTENT WITH HUMAN JUDGMENTS
Token Routing Per Expert. We first verify that our model routes tokens to the most relevant ex- pert module, analogous to how specialized brain networks are selectively engaged by specific stim- revealing clear domain-spectie
uli. Figure 3 shows the routing behavior of MICRO-LLAMAA specialization. To generate test inputs, we sampled 50 question-ansvee answer pairs using GPT-5 prompted with descriptions of the four brain networks (prompt provided in Appendix C). Results for the other MICRO variants are consistent and reported in Appendix C There, we also show routing patterns reasoning benchmarks, where tokens directed to experts consistent with the benchmark's do- main. Finally, layer-wise analyses in Figures [3] & [14] reveal hierarchical organization: earlier layers focus linguistic grounding, while deeper layers increasingly delegate domain-specific experts-an an organization that emerged without being enforced by the training procedure and that parallels evidence from cognitive neuroscience (Fedorenko al. 2024).
Correlation with Human Judgments. We evaluate model-human correspondence using 1,000 six-word sentences from Tuckute et al. (2024b). each annotated with human ratings across several behavioral dimensions (e.g., mental state content, grammaticality). These annotations were col- lected independently of our routing framework. We find that router probabilities correlate with the corresponding human judgments: for example, the social expert's selectivity aligns with ratings of mental state content (r = 0.7). Full results are provided in Appendix
5.2 EXPERTS ARE CAUSALLY MEANINGFUL.
Validation of Functional Experts via Ablations. Figure 4 illustrates how expert ablations reveal the causal contributions of specialized modules to task performance. By selectively removing indi- vidual experts, we can directly test whether their specialization is functionally necessary for different domains. For example, on MATH and GSM8K, ablating the Logic expert causes substantial drop in accuracy, confirming its central role in numerical reasoning. In contrast, removing the Social expert slightly improves performance, suggesting it plays a detrimental role in these tasks. For broader benchmarks such as MMLU, which span multiple subdomains, we report results for each

---

Published as a conference paper at ICLR 2026
(a) MiCRo-Llama-1B
(b) MiCRo-LLama-3B
Language Localizer
MD Localizer
ToM Localizer
Language Localizer
MD Localizer
ToM Localizer
12
13%
05%
136
27 1
2.5
31%
15%
26
24%
1.0
Loglo Loge
Lage
Luglo
Leght
Figure 5: Neuroscience Localizers Recover Functionally Specialized Experts. (a) MICRO- LLAMA-IB and (b) MICRO-LLAMA-3B. For each model, we apply three neuroscience-inspired localizers-LLaguaaeee Multiple Demand (MD) and Theory of Mind (ToM)-to examine the selec- tivity of localized units across experts and layers. Each plot shows the percentage of units in each expert of each layer that belongs to the top-10% selective units in the whole model.
subcategory separately. Performance drops after ablating the corresponding experts indicate that these clusters depend on distinct functional modules. Still, not all subtasks within a category align neatly with a single cognitive domain, and some require overlapping contributions, such as BBH. We show in Appendix [ the effect of removing the language expert, which causes a significant drop on all benchmarks, along with additional ablation results on other models.
Steering Model Behavior at Test- Our results demonstrate that test-time ablations can steer expert behavior, with social responses emerging when only the social expert is active and logical reasoning dominating when only the logic expert is retained. Qualitative examples in Appendix K
5.3 NEUROSCIRNCE LOCALIZERS REVEAL. FUNCTIONAL. EXPERT SPECIALIZATION
Neuroscientists rely on localizer experiments identify the brain regions associated with specific functional networks, as their precise locations can vary across individuals. This raises natural question: can we apply these established neuroscience localizers to identify the corresponding expert modules in our model? If so, this would provide further support for the hypothesis that our experts are functionally analogous to their associated brain networks.
To investigate this, we adopt the methodology of AlKhamissi et al. (2025a), which has been used to localize the language network, the multiple demand network, and the theory of mind network in LLMs. We apply these localizers to MICRo models to test whether they corresponding expert modules. 3 shows the percentage of units in each expert of each layer that belongs to the top 10% of selective units across the whole model, similar to what is done in the brain (Lipkin et al. 2022). The results show that language selectivity, as defined by the language localizer, favors the language expert early layers while favoring the world expert at later layers for both models. The multiple demand localizer successfully favors the logic expert in both models. In contrast, ToM localization is less effective at isolating units within the social experts, but improves with scale. suggesting that ToM ability must emerge before it can be localized. One other possible reason for this is the limited size of the ToM stimulus set, which includes only 10 contrastive pairs, in contrast to 240 for language and 100 for multiple demand. This small sample may lack the robustness needed to reliably localize ToM-selective units (Jamaa et al., 2025).
recover the
5.4 STRONG ALIGNMENT TO HUMAN BEHAVIOR
Having established that our MICRO models exhibit human-like specialization ($5.1) that is causally linked to task performance (5.2) we next examine whether they better align to human behavior compared to two basclines: one without brain-like specialization (MoB) and one without any mod- ularization (DENSE). Both models are post-trained on a mixture of 2x MICROsFT and 1x TüLU-3 matching the total amount of data used in the MICRO training curriculum.
7

---

Published as a conference paper at ICLR 2026
(a)
Llama-3.2-1B
Llama-3.2-3B (b) Behavioral Metrics (Llama-3.2-1B)
Node
Nomalized Store
Model
Bebovioon Aligemmen (hanc)
MiCRo
MoB
Dense
1.30
Ausdon
Normaizad
(c) Examples Tasks
[BART instructiona)
Rask Taking
(Prbbabiiittc Reszoning instructiona
Weighting & Weigghtngg
Balizn (ii infated the sa floon sines for sooa of points it eaplood
The um contain 2) and um comtine 2 il moroi hall Andt ball kow disen, probude hauit drem ey we andouriiity
rcal D mundee dedimal
d You are camendy with Belkoon oich to bulison bpp
by
Option caneo
Figure 6: Alignment with Human Behavior on COGBENCH. (a) Average similarity score (SBRE) across 10 behavioral metrics, showing that MICRO-LLAMA models achieves superior alignment scores for each metric sep- arately across the three models. (c) Example inputs from two of the seven classical psychological experiments verbalized for LLM evaluation following COGBENCH.
compared to their MoB and Dense baselines. (b) Eluman-normally
(igure/6pprseen the results on COGBENCH, evaluating alignment with human behavior. Unlike the original paper, which predicts answers via autoregressive generation, we pick the option with the highest log- bog-probability for multiple-choice tasks to avoid invalid generations. Each experiment is run with five random seeds. Metrics are normalized such that random 0 and human = 1. To quantify overall alignment, we introduce the bounded relative error similarity score (SBRE), which avoids inflation from superhuman scores. For normalized score 8; metric i, compute BRE; [8; - 11/ max(1, 8;) and aggregate as SARF 1 > == BRE, Thus, BRE; remains bounded in 1.
[0, even if
Overall, we find competitive alignment across models, with MICRO-LIAMA-I B showing superior alignment compared to its counterparts, Panel (a) reports the average similarity score (SBRE) aggre- gated across the 10 behavioral metrics for both MICRO-LLAMA-ID LO--LLMMA-(IBB 3B) models, while panel (b) breaks down the human- normallzdd scores for each metric separately across the three post-trained trained 3.2--1111 1 base model. Finally, panel (c) illustrates input examples from two of the seven classical psychological experiments included in COGBENCH, which are verbalized for LLM evaluation following the original benchmark design. More results in Appendix []
models for the LLAMA-3.2
5.5 COMPETITIVE PERFORMANCE ON REASONING BENCHMARKS
Here, whether our MICRO models incur any performance degradation relative their two baselines. /soows performance on GSM8K, MINERVA-MATH, MMLU, MMLU-PRO, and BBH, along with their average. Models are evaluated using fewshot ofain-of-thought prompting, except for GSM8K, which is evaluated under zero-shot CoT prompting. For both base models, MiCRo matches or exceeds comparable MoB baselines, while ablating the least relevant expert (i.e., the social expert for these benchmarks) further improves performance. We conduct pairwise Welch's s f-tests between models and report significance directly in the plot. Results show that some base models, such as LLAMA-3.22 - IB, benefit significantly from brain-like specialization, whereas others, such as LLAMA-3.2 3.233B only show significant differences relative to their baselines on some benchmarks. We report additional results for the other models and benchmarks in Appendix F We further show that our method is robust to different post-training pipelines, including DPO (Rafailov al. 2023) and domain-specific instruction tuning (Appendix G
8

---

Published as a conference paper at ICLR 2026
GSMBK
Minerva Math
MMLU
MMLU-Pro
BBH
Average
00.0
Llama-3.2-16 3.B-1B
25.0
12.5
22.5
Llama-3.22 -.22-33.-3BB
14
11: 1700
40.0
377.5
Model
MICRo
MoD (Ablation)
Dence
Competitive Performance. Results on GSM8K (0-shot CoT), MINERVA-MATIE MMLU, and BBH (fewshot CoT). MICRO matches or outperforms basclines, and ablating the least relevant expert (c.g., social for math benchmarks) yields further gains. For MoB (ABLATION) and MICRO (ABLATION) (on MMLU and BBH subtasks), results reflect the best performance obtained when ablating up to one expert. Significance is assessed with pairwise Welch's t-tests (shown in plot; denotes significant difference, while n.s. means not-significant), The dense model is shown as a dashed line. Results of the remaining models and on more benchmarks are provided in Appendix F
Figure 7:
6 DISCUSSION & FUTURE WORK
Extending Specialization Beyond Cognitive Domains While inspired by the brain's functional organization, our specialization framework can be applied any meaningful partition of expertise, such as technical domains or natural languages. One key question is whether the model preserves the intended specialization through the large-scale end-to-end training. Our results suggest that brain-inspired partitions provide a robust inductive bias; they persist throughout training and lead to structured, interpretable routing patterns. Supporting evidence Appendix [ shows that expert usage remains consistent across checkpoints over the course of Stage 3 training. Looking ahead, this framework could also be extended to other cognitive domains. For example, recent neuroscience findings point to a distinct brain network involved in abstract formal reasoning such as induction and deduction and another network for intuitive physics (Kean et al., 2025a b). Incorporating a corresponding module could improve the model's performance on tasks involving such capacities.
The Crucial Role of Stage-1 Pretraining Data Our experiments highlight the importance of the curated MICROsFT dataset inducing effective specialization. Notably, we used only 3,055 sam- ples in Stage-1, 1, suggesting that even minimal nomain-aligned supervision can shape expert behav- ior. This finding raises the possibility that different or more expansive data mixtures could further strengthen functional specialization and lead to additional gains in the model's behavior.
Towards Brain Alignment Beyond Language Since our model is explicitly designed to mirror distinct cognitive networks in the human brain, and given that established neuroscience localizers can identify the corresponding expert modules, an exciting direction for future work is to examine whether the internal representations of these experts align more closely with neural activity in their respective brain networks (Schrimpi et al., 2018: 2020). Prior studies have shown that language- selective units large language models correlate more strongly with activity in the human lan- guage network than randomly selected units (AlKhamissi et al., 2025a), suggesting a meaningful link between specialization in models and brains. This raises the natural question of whether sim- ilar alignment can be observed for other cognitive domains, such reasoning or social cognition. However, assessing MICRO's 's neural alignment beyond the language network is currently limited by the lack of suitable datasets. Existing fMRI benchmarks rarely engage non-language regions such as the Multiple Demand (Duncan (010)) network and often use blocked designs that preclude item-level analyses-hhigliggheee the need for experimentalists to collect new datasets that explic-

---

Published as a conference paper at ICLR 2026
itly target non-language brain regions. We believe that once suitable neural datasets exist, model can be used to instantiate specific hypotheses about how these networks and their corresponding experts-interrat interrc-nttrract
interact and exchange information.
Limitations While our approach improves interpretability without sacrificing performance, sev- eral open questions remain. Scaling beyond an 8B base model has yet to be demonstrated, and the impact of adding more experts to the current MICRO architecture still unknown. The MI- CROSFT dataset used in Stage-I (= 3,000 GPT-4o pseudo-labeled samples) has been evaluated for size sensitivity, leaving open whether increasing or reducing its scale would alter the degree of specialization downstream performance. Although GPT-40 provides high-quality pseudo-labels, as demonstrated in Appendix B using human -anaan-nd data could strengthen the inductive bias and potentially improve the final model. We expect, however, that the potential of this approach will con- tinue to grow as synthetic labellers become more accurate and reliable, enabling even stronger and more scalable specialization future versions of this class of methods. Further, our post-training pipeline currently includes only SFT and DPO (Appendix [G;; exploring additional stages such as RLVR remains an avenue for future work. Finally, our evaluation of alignment to human behavior focuses CogBench, and extending this analysis to broader set of behavioral cognitive datasets an important direction for future research.
CONCLUSION
This work presents the Mixture of Cognitive Reasoners (MICRo), a modular architecture and train- ing paradigm inspired by the functional specialization of the human brain. By aligning expert mod- ules with distinet cognitive domains- reflecting the functionality of well-studied brain networks, we show that incorporating cognitive inductive biases into transformer models can effectively promote functional specialization. This re- sults in improved behavioral alignment as measured by COGBENCH,, enhanced interpretability, all while being competitive or outperforming comparable models on reasoning benchmarks. Our staged training approach leverages a small curated dataset and enables specialization to emerge in a con- trollable and data-efficient manner. Furthermore, we show that the resulting modularity allows for targeted interventions at inference time, enabling the model to favor one mode of reasoning over an- other. These findings highlight promising path toward more transparent, steerable, and cognitively grounded language models.
logic, social reasoning, and world modeling- each
ETHICS STATEMENT
This work involves two forms of numan--erived data. First, all annotations newly introduced in this paper (Appendix)) B were performed exclusively by the authors; external annotators study par- ticipants were recruited, and no personal or sensitive data were collected. Accordingly, this does not constitute human-subject research under typical institutional guidelines, and formal ethics approval (e.g., IRB review) was not required. Second, we use public dataset of behavioral annotations (Tuckute et al.] 20224b in Appendix D which collected and released with approval from the corresponding university's 's Institutional Review Board.
ACKNOWLEDDGEENTS
We thank the members of the EPFL NLP and NeuroAl labs for their valuable feedback and insightful suggestions. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), the EPFL Center for Imaging, Sony Group Corporation, and Meta LLM Evaluation Research Grant. G.T. acknowledges support from MIT's McGovern Institute for Brain Research. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute at Harvard University.
REFERENCES
Mohammed Al-Maamari, Mehdi Ben Amor, and Michael Granitzer. Mixture modular ex- perts: Distilling knowledge from multilingual teacher into specialized modular language
10

---

Published as a conference paper at ICLR 2026
models. ArXiv, abs/2407.19610, 2024. URL https://api. http:s///pp...emmtttccccchhh.... CorpusiD:2215333:2
Badr AIKhamissi, Greta Tuckute, Antoine Bosselut, and Martin Schrimpf. The LLM language network: A neuroscientific approach for identifying causally task-relevant units. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Na- tions of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume 1: Long Papers), Pp. 10887-10911, Albuquerque, New Mex- ico, April 2025a. Association for Computational Linguisties. ISBN 779-8-89176-189-6 URL https://acl antho ogy.org/202..... naacl long. 544/
Badr AlKhamissi, Greta Tuckute, Yingtian Tang, Taha Binhuraib, Antoine Bosselut, and Martin Schrimpf. From language to cognition: How ilms outgrow the human language network. arXiv preprint arXiv:2503.01830, 2025b.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarin, Vaibhav Srivastav, et al. Smollm2: When smol goes big-data-centric training of small language model. arXiv preprint urXiv:2502.02737, 2025.
Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. Instruction-tuning aligns Ilms to the human brain. 2023.
Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. Discovering knowledge-criticee critiicll subnetworks in pretrained language models. ArXiv, abs/2310.0308,, 2023.
URL httpt: /api. www.nt ottp/ooppusDD-22333175
Taha Binhuraib, Greta Tuckute, and Nicholas Blauch. Topoformer: brain-like topographic organi- zation in transformer language models through spatial querying and reweighting, arXiv preprint arXiv:2510. 18745, 2025.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
Idan A Blank and Evelina Fedorenko. No evidence for differences among language regions in their temporal receptive windows. Neurolmage, 219: 16925, 2020.
Antoine Bosselut, Zeming Chen, Angelika Romanou, Antoine Bonnet, Alejandro Hernández- Cano. Badr Alkhamissi, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Vinitra Swamy, Alireza Sakhaeirad, Igor Krawezuk, Deniz Bayazit, Axel Marmet, Li Mi, and Martin Jaggi. Meditron: Open medical foundation models adapted for clinical practice, 2024.
Randy L. Buckner and Lauren M. DiNicola. The brain's default network: updated anatomy, physiology and evolving insights. Nature Reviews Neuroscience, 20(10):593 608, September 2019. ISSN 1471-0043 doi: 10. 10.1038441583--09-02222 1038/ 0088/441583-0199 0212-7
httt:tt:://d doi. doi.oog/1..
Randy L. Buckner, Jessica R. Andrews-Hanna, and Daniel L. Schacter. The brain's 's default network: Anatomy, function, and relevance to disease. Annals of the New York Academy of Sciences, doi: 10.1196/annals. 1440.011. URL http:
1124(I):1-38, March 2008. ISSN 1749-6632.
7/dx. doi. org/ 10. 1191196.... 1196/ annals 1440. 011
Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. Modeling empathy and distress in reaction to news stories. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 4758-4765, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/D 1/1////////111-/ 1507. URL https: http::::///alaathhooooooooo org/
218-15077/
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022.
11

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Julian Coda-Forno, Marcel Binz, Jane X Wang. and Eric Schulz. CogBench: a large language model walks into psychology lab. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 9076-9108. PMLR, 21-27 Jul 2024. URL https: /ttp:////ccceedees....oo
A.r. wwwppptwwww......ppppppp coda wwweoooaaaaooooooe forno24a html
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D. Cox, and James J. DiCarlo. Simulating a Primary Visual Cortex at the Front CNNs Improves Robustness to Image Perturbations. In Neural Information Processing Systems (NeurlPS, Spotlight), June 2020. doi: 10.1101/2020.66. 16. 154542.
URL htt.:pss//w.BB.ooo..voooo/ccccce 1101/
2020. 06. 16. 154542v1
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, and et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. URL https:
John Duncan. The multiple-demand (md) system of the primate brain: mental programs for intelli- gent behaviour. Trends in cognitive sciences, 14(4):172-17) 2010.
John Duncan and Adrian M Owen. Common regions of the human frontal lobe recruited by diverse cognitive demands. Trends in Neurosciences, 23(10):475-48 483, October 2000. ISSN 0166-2236. doi: 10.1016/s0166-22 2236(00)01633-77
10.101660166-23300000063-- URL htthhtp:://e www.xxxccc.ooooo or1/00,,111,.110 httt::://o..oi..og/110011111
S0166 2236 (00) 01633-7
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Castanen Susan L. Whirfilld-Gabrieli, and Nancy G. Kanwisher. New method for fmri investigations of language: defining rois fune-
tionally in individual subjects. Journal of neurophysiology. 104 2:1 2010. URL
dittps://apt semanticscho wwt.scccoo.....oorrropaaaeeeeeee wwtg/oorrss/i.......oo 740913
Evelina Fedorenko, Michael K Behr, and Nancy Kanwisher. Functional specificity for high-level linguistic processing in the human brain. Proceedings of the National Academy of Sciences, 108 (39):16428-16433, 2011.
Evelina Fedorenko, Josh H. McDermott, Sam Norman-Haignert and Nancy Kanwisher. Sensitivity to musical structure in the human brain. Journal of Neurophysiology. 108(12):3299-300, De- cember 2012. ISSN 1522-1598. doi: 10. 10.1152/jn.0022092200 2012. URL 10. 1 1111111//5000 Jn. 00209. 2012
Evelina Fedorenko, John Duncan, and Nancy Kanwisher. Broad domain generality in focal re- gions of frontal and parietal cortex. Proceedings of the National Academy of Sciences, 110 166116,16-1621,, September 2013. ISSN 1091-6490. doi: 10.1073/pnas. 13 1315235110. URL
http://dx doi wrg./0....0... 1073/pnas. 52335511111111111101
Evelina Fedorenko, Anna A. Ivanova, and Tamar Regev. The language network natural kind within the broader landscape of the human brain. Nature Reviews Neuroscience, 25(5):289-312 April 2024. ISSN
doi: 10..038/441533-02--000 2ttr:/00..003//111133222--000300
URL http: doi.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 1-39, 2022.
Evelyn Ferstl and Yves von Cramon. What does the frontomedian cortex contribute to language processing: coherence theory of mind? Neuroimage, (199):159-116) 2002.
12

---

Published as a conference paper at ICLR 2026
H.L Gallagher, F Happé, N Brunswick, P.C. Fletcher, U Frith, and C.D Frith. Reading the mind in
cartoons and stories: an fimi study of "theory of mind' verbal and nonverbal tasks. Neuropsy-
chologia, 38(1):11-21, January 2000. ISSN 0028-3932. doi: 10. 00.1016/0002-399220 3932(99))0005-66.
URI.http://dx http:///xxxdo...gg//11. o00. 101 6/ 0000s0002003000 3932 ( (99) 00053-6
doi.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Understanding social reasoning in language models with language models. In Thirty-sevnnth Conference
Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
//openrevi ew. net. mnet/forum??
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. httpttss::/// //ts:d:://eendd.oo zenodo. . org/ reco ro. 1266066
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. DEMix lay- ers: Disentangling domains for modular language modeling. In Marine Carpuat, Marie- Catherine de Mamnffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 5557-5576, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 29.18853/v1/2000 hww.://aaa..mammminaaaamaamaae 407. URL https: //ac::///clcc acc ant ogy oorg org/ 2022. satacclmainn 4077
Debra A. Gusnard, Erbil Akbudak, Gordon L. Shulman, and Marcus E. Raichle. Medial prefrontal cortex and self-referential mental activity: Relation to a default mode of brain function. Proceed- ings of the National Academy of Sciences, 98(7):4259-4266 March 2001. ISSN 1091-6490. doi:
/.pma7//pnas. 07 1043098. Utthttp::/// /xx. doi. org/ 10. 1073 / pnas 07 1043098
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojeiech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. Folio: Natural language reasoning with first-order logic. arXiv preprint arXiv:2209.0088 2022. URL /ectps://azzxae org/ abs/ 00840
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt, Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021a. URL httttp:://ppereeeeeeeeeee net wtrt/formmmii..
d7KB jm 3Gm),
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. A fine- grained comparison of pragmatic language understanding in humans and language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4194- 4213, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 18653v1vl 1023.acl-long2221 URL URL
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21276 2024.
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Ellen Wu, Valentina Pyatkin, Nathan Lambert, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback, 2024.
Nir Jacoby and Evelina Fedorenko. Discourse-level comprehension engages medial frontal theory of mind brain regions even for expository texts. Language, Cognition and Neuroscience, 35(6): 780-796, 2020.
13

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167100 2024.
Yassine Jamaa, Badr AlKhamissi, Satrajit Ghosh, and Martin Schrimpf. Evaluating contrast local- izer for identifying causal unitsin social & mathematical tasks in language models. arXiv preprint arXiv:2508.0m 08276, 2025.
Nancy Kanwisher. Functional specificity in the human brain: A window into the functional archi- tecture of the mind. Proceedings of the National Academy of Sciences, 107(25)11133-11177) May 2010. ISSN 1091-6490. doi: 10. 10. 10.1073/pnas. 1005062107
1005062107. URL htttp://dx. doi wow......... org/
Hope Kean, Alex Fung, Chiebuka Ohams, Jason Chen, Josh Rule, Joshua Tenenbaum, Steven Pi- antadosi, and Evelina Fedorenko. A human brain network specialized for abstract formal reason- ing. bioRxiv, 2025a. doi: 10. 10.22000............662 683445. URL https: httptttpspww.we...eeeetemopttt htwp:p//wwwwww..ooii.....o [onttent/aaall//2333////////////
21334454
Hope H. Kean, Alexander Fung, R.T. Pramod, Jessica Chomik-Morales, Nancy Kanwisher, and Evelina Fedorenko. Intuitive physical reasoning is not mediated by linguistic nor exclusively
domain-genee abstract representations. Neuropsychologia, 213:109125, July 2025b. ISSN
0028-3932. doi: 1011006/..euupppyhoooooaa......0 109125. URL http: doi wow.d/..ooo.oo/o....oooo 6/1 www.oopsycoooooiaaa..........
Hyunwoo Kim, Byeongehang Kim, and Gunhee Kim. Perspective-taking and pragmatics for gener- ating empathetic responses focused on emotion causes. In EMNLP, 2021.
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interactions. In EMNLP, 2023.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-expert from dense checkpoints. In The Eleventh International Conference on Learn- ing Representations, 2023. /ttpsttss///oeeeeviiii
net wtee:/oorumnddddSnugep
Jorie Koster-Hale and Rebecca Saxe. Theory of mind: A neural prediction problem. Neuron, 79(5):836-848, September 2013. ISSN 0896-6273.
doi: 10.10166j.neuron.220 20t.onn.003.08.02. URL
org/ 1t1.:110166..............
2013.000000..0 08. 020
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Ha Hong, Najib Majaj. Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Jonathan Prescott-Roy. Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L K Yamins, and James DiCarlo. Brain-like object recognition with high-performing shallow recurrent anns. In Advances in Neural Information Processing Systems, 2019.
Nathan Lambert, Jacob Daniel Morrison, Valentina Pyatkin, Shengyi Huang. Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hanna Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. ArXiv, abs/2411.15111 2024. URL /attps://app www.pp.w..aaa...m iommmannisccinn www.aar.rggoooooooeee wtt.aa...r..oooo...:22222222222.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023.
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video. In The Thirteenth International Conference on Learning Representations, 2025. URL htttttss://ppereeiieeeeeo Hd=SHAWMP5J4R
net net/forum? forum?
14

---

Published as a conference paper at ICLR 2026
Benjamin Lipkin, Greta Tuckute, Josef Affourtit, Hannah Small, Zachary Mineroff, Hope Kean, Olessia Jouravlev, Lara Rakocevic, Brianna Pritchett, Matthew Siegelman, Caitlyn Hoeflin, Alvincé Pongos, Idan A. Blank. Mclissa Kline Struhl, Anna Ivanova, Steven Shannon, Aalok Sathe, Malte Hoffmann, Alfonso Nieto-Castaßón, and Evelina Fedorenko. Probabilistic at- las for the language network based precision fmri data from;800 individuals. Scientific Data, 9(1), August 2022. ISSN 2052-4463, doi: 10. 10.10386-1157-02-0016655 URL htthtt: //dx.doi. org 10. 1038 000.////14550 022 01645-3
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124 2020.
Eshed Margalit, Hyodong Lee, Dawn Finzi, James J. DiCarlo, Kalanit Grill-Spector, and Daniel L.K. Yamins. A unifying framework for functional organization in early and higher ventral visual doi: 10. 10106//.neuuon 2024.04.018. URL /tth:::/// doi org/ / 10.111111010 101. neuron. 2024. 04.00...0 018
cortex. Neuron, (11(144):433-225111 July 2024. ISSN 0896-6273.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hanmaneh Hajishirzi. OLMo 2 Furious, 2024. URL https:ttt///cmm http://aaxvvvve. org abs 2500 00656
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter- Framework for Multi-Task Cross-Lingual Transfer. In Bonnie Webber, Trevor Cohn, Yulan Hc, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pp. 7654-7673, Online, November 2020. Association for Computa- tional Linguistics. doi: 10. 00.18653/v1/2222..or waammnmmmmin66ii main.617. URL http::tt//aaaatttollll. org/2000. em main. 617/
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Marine Carpuat, de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticy; Human Language Technologies. pp. 3479-3495, Seattle, United States, July 2022. Associa- tion for Computational Linguisties. doi: 10. 1655// waa.22aaaallmmai2555555 URL https:
//aclanthology.ezzg// wwweeeee...nncneeeeeeeee nww.nnaacaaaaaaa 255/
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Ponti. Modular deep learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL httphttsss//ppeeeeeiieeeee net/ Forum??.d=23EkXEfxxx Survey Certification.
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Zhengzhong Liu, Yuanzhi Li, and Pengfei Liu. 01 replication journe:: A strategic progress report 2024.
-part1 1.
com SAAp///AARR-NPP/OOOA
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh. T. Naumann. A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728 53741. Curran Associates, Inc., 2023. URL. httpss://proceeeddnggs nwwrips..p.... wwwpccpp.pppppppeppeee wtt.ff////pp////22...22 e 2111:/6866666667722ee22266666622
Neil Rathi, Johannes Mehrer, Badr AlKhamissi, Taha Osama A Binhuraib, Nicholas Blauch, and Martin Schrimpf. TopoLM: brain-like fuatiolful organization in topographic language model. In The Thirteenth International Conference on Learning Representations, 2025. URL
https::/spenreviiee net httpa//ewweeeeeeeooooooooooodoan
15

---

Published as a conference paper at ICLR 2026
R Saxe and N Kanwisher. People thinking about thinking peoplethe role of the camporo-paricee junction in "theory of mind". Neurolmage, 19(4):1835-144) August 2003. ISSN 1053- 8119. doi: 100166610033-8111000000222222220
URL http: wtw.://oo..rrg//o...../. 1016/
5 s1053 81 1 (03) 0
Rebecca Saxe and L Lindsey J. Powell. It's the thought that counts: Specifie brain regions for one component of theory of mind. Psychological Science, 17(8):692-699, August 2006. ISSN 1467- 9280. doi: 10. 10.11111 10.11114..447-9928000
URL http: doi. wtg.:oo.o..oogo.oooo
1467 9280. 2006. 01768.x
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najbb J. Majaj, Rishi Rajalingham, Elias B. Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott- Moys Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James DiCarlo. Brain-Scome -core:: Which Artificial Neural Network for OB- ject Recognition is most Brain-Like? preprint, Neuroscience, September 2018. http:
//biorxiv.org/ http://orrg//ooooooooooo//// 10. 1110/110000000
Martin Schrimpf, Jonas Kubilius, Michael Lee, N Apurva Ratan Murty, Robert Ajemian, and James DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413 423, 2020.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of lan- guage: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):21006646116 2021. doi: 20.1073/paa.2110546111 21 211.22111166611
URL
www. wtw.p/nas..rrg//.///bb//////////
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of- experts layer. In International Conference on Learning Representations, 2017. URL https: //openreview net/ net/forum?id-ee ckkMg
Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Mod- uleformer: Modularity emerges from mixture-of-expert 2023. URL https://api.
6974111
Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. Getting MoRE out of mixture language model reasoning experts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8234-8249, 8249, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18633/112023..0 fin.dii..iinns---e.... mrndings-emmlp.552.
URL hatpss//acllattcoooyyoorrs
Findings-ent 5527
Courtney Spoerer, Tim C. Kietzmann, Johannes Mehrer, Ian Charest, and Nikolaus Kriegesko- rte. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLOS Computational Biology, 16(10)xe008815, October 2020. ISSN 1553-7358. doi:
10.137/jourmall.obb..ooo 100821. URL 1008215
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gchrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether Mhain-of-thoughy can solve them. arXiv preprint 1rXiv:2210.09261,, 2022.
Vinitra Swamy, Malika Satayeva, Jibril Frej. Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, multi-task, interpretable modular networks.
and Mary-Anne Hartley, In sevethh Conference Neural Information Processing Systems, 2023. URL hotps /ppenreview. www.at/ffoomidddoeemee
4WL
Mariya Toneva and Leila Wehbe. Interpreting and improving natural- processing (in ma- chines) with natural tems, 2019. URL https: :atass//ppp /api...... peppi.semaann www.nticcccholl.o..mcmm. http::/ccc.........oooo/ooaaaaaa
(in the brain). In Neural Information Processing Sys-
16

---

Published as a conference paper at ICLR 2026
Greta Tuckute, Nancy Kanwisher, and Evelina Fedorenko. Language in brains, minds, and ma- chines. Annual Review of Neuroscience, 47, 2024a.
Greta Tuckute, Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human language network using large language models. Nature Human Behaviour, pp. 1-18, 2024b.
Rosemary A. Varley, Nicolai C. Klessinger, Charles A. Romanowski, and Michael Sie- gal. Agrammatic but numerate. Proceedings of the National Academy of Sciences, 102(9): 3519-3524, February 2005, ISSN 1091-6490, doi: 04117730p URL http:
7/dx. doi. org. 1073 /pnaspnsss.. 0407470100
Chengcheng Wang, Zhiyu Fan, Zaizhu Han, Yanchao Bi. and Jixing Li. Emergent modularity in large language models: Insights from aphasia simulations. bioRxiv, 2025. doi: 10.1
2025 2025500.22.6633 639441.... URL htttt::://///tt biorxiv
02. 639416
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet. Xin Wang. Yixuan Li, and Neel Joshi. Is picture worth a thousand words? delving into spatial reasoning for vision language models. In The Thirty-Fight Annual Conference on Neural Information Processing Systems, 2024.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjum Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Par- mar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannanch Hajishirzi, and Daniel Khashabi. Super--aatraalinsttutionsss Generalization via declarative instructions 1600+ nlp tasks, 2022. URL http::tts::aaaa org. abs http//ab///222. 2204. 07705
Alexandra Woolgar, Alice Parr, Rhodri Cusack, Russell Thompson, Ian Nimmo-Smite Teresa Tor- ralva, Maria Roca, Nagui Antoun, Facundo Manes, and John Duncan. Fluid intelligence loss linked to restricted regions of damage within frontal and parictal cortex. Proceedings of the National Academy of Sciences, 14993)914899914992, August 2010. ISSN 1091-6490. doi: 10. 10.1073/pnas. 1007928107. URL httt:://aaa......e. //xp doi. org/ / 10. 1073/ 1007 0977991177
Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2013-2018, Lishon, Portugal, September 2015. Association for Com- putational Linguistics. doi: 10.18653/w1/D15-1122 1237. URL https: http:///aaaaattoo.......o .org/
D15-1237
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration, 2023.
Danyang Zhang, Junhao Song. Ziqian Bi, Yingfang Yuan, Tianyang Wang, Joe Yeong, and Junfeng Hao. Mixture of experts in large language models. ArXiv. abs/2507.1118..1 2025. httpttss:
//api. www.ssaeaaanccchoceoooo wwt.:laarrrr//ooopucucuaao 280277225
Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Nicolaus Foerster, Phil Blunsom, Sebastian Ruder, Ahmet Üstün, and Acyr Locatelli. BAM! just like that: Simple and efficient parameter upeycling for mixture of experts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://cpenzevieer net/ forum? www.eeeeoooonddeDeeeeeee... BdD
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. MoEfica- tion: Transformer ferward layers are mixtures of experts. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 877-890, Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 10.18653/v112022 1/22:..ii/22....nninn-g... acl. URL https: /ttpa/aaannnttoy...oooooo wrw.looooorrg/222.....
Findings Sindings-ach 717
17

---

Published as a conference paper at ICLR 2026
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Emergent modularity in pre-trained trans- formers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Findings of the Association for Computational Linguistics: ACI. 2023, pp. 4066-4083, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 10.18653/v1/2023 10n18665.//022...nnnnnaaa..2222 http:///cciaaatooooooorr//////// www.inggsiiiinnnennneeeennngnneo act.gss-cc.2200 2507
URL
Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang. Yuanhao Yu, and Jin Tang. Meta-DMoe: Adapting to domain shift by meta-distillation from ofxture-of-experts. Thirty-Sixi Sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
A CONSTRUCTING THE EXPERTS DATASET
Datasets To construct the small curated datasets used in stages I and 2 of our training curriculum, we first identified existing datasets that align with the cognitive domain of each expert. Table lists these datasets, the number of examples sampled from each, the corresponding high-level cognitive skill they were chosen to represent, and whether we used O1 to generate responses or relied on the original reasoning chains provided with the dataset.
Table I: Datasets Used to Induce Specialization in Stage- Overview of datasets used to induce expert specialization during stages and 2. Each dataset is aligned with a cognitive skill targeted by specific expert. We indicate the number of examples sampled from each dataset and whether
responses were generated using 01 or taken directly from the dataset's 's original reasoning chains.
Expert
Task
Dataset
# Samples
Use O1
O1-Journey (Qin al.
2024
327
No
Math
Math (Li et al. 2023
200
No
GSM8K Cobbe et al
2021
100
Yes
Logic
Folio (Han et al. 2022)
100
Yes
Logic
LogicQA Liu et al.
2020)
100
Yes
Physics
Physics (Li et 2023
200
No
Total (Logic)
1027
Deceits (Hu et al.
2023)
20
Yes
Indirect Speech (Hu al.
2023
20
Yes
Irony (Hu et al.
2023
25
Yes
Pragmatics
Maxims (Hu et 2023
19
Yes
Metaphor (Hu et al.;
2023
20
Yes
Humor (Hu al. 20233
25
Yes
Social
Coherence (Hu et al.
2023)
40
Yes
Emotion Detection
EmoCause (Kim et al.
2021)
100
Yes
FanToM 1st Order (Kim et al.
2023
100
Yes
Theory of Mind
FanToM 2nd Order (Ki el al.
2023
100
Yes
BigToM (Gandhi et al. 2023)
128
Yes
Social Reasoning
Mixture ProlificA\/social-reasmm rlhf
531
Yes
Total (Social)
1028
Biology Li et al. 2023
100
No
Chemistry Li et al. 2023
100
No
World Knowledge
PIQA Bisk ct al. 2020)
200
Yes
WikiQA Yang et al. 2015
100
Yes
World
Spatial Reasoning
SpatialEval
WWagg al.
2024
100
Yes
Temporal Reasoning
TextTemporal
(ii al. 2025)
100
Yes
World Building
World Building architl 1/worldbuilding
200
No
Cause Effect
CoPA WWagg al. 2022
100
Yes
Total (World)
1000
Generating Reasoning Responses Since most of the existing datasets we identified (listed in Ta- T do not include reasoning steps for the final answer, we used 01 to generate them. Specifically, prompted the model with the input followed by "Let's think step by step." to elicit a longer sponse that includes intermediate reasoning before reaching the final answer. This was only done for datasets that did not already contain suitable reasoning chains.
19

---

Published as a conference paper at ICLR 2026
(a) Tokens
(b) Phrases
400000
20000
Number Tokens
Number of Phrases
15000
200000
10000
100000
5000
Language
Logic
Social
World
Language
Logic
Social
World
Expert
Expert
Distribution of Expert Assignments across Tokens and Phrases. (a) The distribution of expert assignments across tokens using the LLAMA-3.2- tokenizer. (b) The distribution of expert assignments labeled using GPT-4o for each phrase the provided response.
Figure 8:
Pseudo-Labeling Responses Finally, once obtained responses with intermediate reasoning steps for all sampled examples, we used GPT-40 to pseudo-label each phrase the response. During stage-1 training, each token in phrase pseudo-label. This labeling process was guided by the prompt shown in Figure 9 Figure [1 pro- vides examples of labeled responses, while Figure & shows the distribution of expert assignments across tokens and phrases.
then assigned to the expert identified by the
B INTER-ANNOTATOR AGREEMENT ANALYSIS OF MICROsFT
We assess the reliability of the MICROsFT pscudo-labels using set of standard agreement metrics between three human annotators and the GPT-40 labels on a subset the MICROsFT dataset. For the three human annotators, bility metric), Fleiss' к (multi-rater extension of Cohen's 's K), and pairwise Cohen's k with percent agreement. We also measure the agreement between the humans and the LLM labels both against the human majority vote and by treating the L.M as fourth annotator.
report Krippendorff's
(a shance-correcter multi-annotator relia-
B.1 HUMAN-HUMAN AGREEMENT
Table 2 summarizee agreement across the three human annotators. Both Krippendorff's a and Fleiss' indicate moderate reliability (0.517). The three-way exact agreement rate (all annotators assign the same label) is 49.4%.
Pairwise metrics (Table 3 reveal that annotators HI and H2 exhibit stronger mutual agreement (K 0.681) relative to HI-H3 and H2-H3, which show lower yet nontrivial agreement levels (x (.45)...55).
Table Agreement among human annotators.
Metrie
Value
Krippendorff's (nominal, annotators)
0.517
Fleiss' k (3 annotators)
0.517
3-way exact agreement
49.4%
B.2 HUMAN-LLM AGREEMENT
We evaluate human human-LLM agreement in two ways: (1) comparing the LLM to the majority-vote label of the three humans, and (2) treating the LLM as fourth annotator.
20

---

Published as a conference paper at ICLR 2026
GPT-4o Prompt Pseudo-labeling Responses
am training mittrre-of-expert (MoE) model that routes tokens individually (token-level routing) to specialized experts. The model includes four dislinct experts, each clearly analogous to specific cogniliv network in the human brain. Below is delailed explanation of each expert, along with examples the types tasks
each should typically handle:
LLaguagge Network (LN): Primarlly responsible for linguiatic processing, grammnttical structures, vocabulary synlax tasks involving language comprehension, fluency, construction, paraphrasing, and interpreting linguistic nuances. Exampie tasks: Completing sentences, grammar correction, paraphrasing summariznn rzing text.
coherence.
translating between languages,
(MD): Specializes analylical thinking, mathematical calculations, numerical reasoning, logical problem-solvinng This expert engages explicity in arihhmetic operations, logical deductions, comparisons, quantitaivee reasoning, and systemalic analysis.
Mulltiple
Example tasks: Performing arithmelic operations, solving logical puzzles, analyzing numerical data, interpreting
mathematical expressions, ovaluating logical arguments.
Theory of Mind Network (ToM): Dedicated to social cogniton and interpersonal reasoning. This expert interprets and predicts
social interactions, ernotional slales, intentions, desires, beliefs, molivations individuals groups.
Example tasks: Inferring person's feelings or intentions from their actions, understanding dialogue involving interpersonal relations, prodicting characters' behaviors based emotional context, interpreting subtle social cues.
Default Mode Network (DMN): Responsible for intagrating general world knowledgo, context understanding, background
information retrieval, and conceptual ressoning about everyday scenarios common-sense understanding.
Example tasks: Providing background knowledge on common scenarios, contextualizing real-world situstions, recalling general facts, understanding cause and eefeete relefionships, providing narrative context.
Given the following example of madel-generated text in response to prompt, carafully label each token sequence with the expert best suited handle it (LN, MD, ToM, DMN), Ensure each sequence assigned to expert. Outpul your clearly (sequence [_] corresponding the sequence
explicilly following
"Json
"Laeguence 13"; "[expert
"Iaequence 2)"; "texpert label)",
[sequunnc 3)";
"(expert_labe label)"
}.
#tl Prompt
(prompt]
(generation)
Figure 9: Prompt Used for Pseudo-Labeling 01 Responses The prompt used to instruct GPT-40 to label the O1 model generations given specific input prompt.
GPT-5 Prompt for Generating Export Specific Stimull
(BRAIN NETWORKS DESCRIPTION)
Using the description above and your knowledge of the mentioned brain networks, generate (N) prompt and chain of thought answer pairs
any experts. Output results only and nothing else in JSONL format. Each JSON object should be an LLM chat format the following.
only (BRAIN NETWORK) expert
("user": <prompt>, "assistant": <cot-answer>>
Expert-Specific Prompt Template Used with GPT-5 Prompt provided GPT-5 for generating expert-specific question-answer pairs. The stimuli for each expert was prompted separately using the same brain-network descriptions as in Figure
Figure 10:
21

---

Published as a conference paper at ICLR 2026
(a)
(b)
Input Prompt
Input Prompt
pairs
decide
they coharent story.
A cube with edge length of units has the same volume as a square-based pyramid with base edge lengths of 8 units and a height of $h$ units. What is the value of $h$?
The answer aptions are 1 or
Scenario:
Mary's exam was about to start. Her paime were aweety.
Options:
Incoherent
2) Coherent
step:
Response
Response
Step Read the sentences carefuly.
Lat's work through problem slap by step, keeping disse eye on each detail and explonng any uncertairies as we go. Our goal is to find the height W. based pyramid given that its valume matches that of cube with an edge longth of units.
square-
Sentence 'Mary's exam was about to start."
To start know the volume cube givan by the fomula ((text(cuee)) = gh3 1),, With V( 41) we
Senience "Her palme were eweely."
caloulate the voluma
Step Analyze logical connection sentences.
VVV. ((lexx//uuee) 4"3
Mary
about take an exam
pyramid. volume square- besed pynamid is detemmined V. [lleewllyymmmm)) Wiras (H))) Wtimes Weeeiibase area) Wimes liter((heigh))) VI. The base area for our pyramid given the base edge length Wi im 8 NJ, lo 8*2 R)."
Now.
second sentence deacriben physical reaction (eweaty palms) commonly
formula 16
nervousneet or aroiely.
Narvousnese typical
Subsituting formula, have:
Determine the sentences form
The plasical reaction the secend santenoe logicslly
So, appears the helght 1K, units, Let's pause consider what we done Initialy. ancountered mistaks with division, but aftr ravisiting our steps, we corrected il accurately isolating Vi).
There dear cause (oxam starting) and affect (sweaty
pyramid, maintaining equality of volumes, Is V. Wboked(3) I0.
height
"Answer:" 2) Cohanont
Expert Legend
Language
Logic
Social
World
Figure 11:
Examples of Pseudo-Labeled Responses using GPT-40 (a) shows response gener-
ated by O1 for prompt from the coherence subset of the PRAGMATICS dataset (Hu et al. 2023). (b) shows response taken directly from the O D1-JOURNEY dataset (Qin al. 2024). Each subfigure cludes the original prompt, the full model-generated response, and the corresponding pseudo-labels assigned each phrase.
22

---

Published as a conference paper at ICLR 2026
Table 3: Pairwise agreement between human annotators.
Annotator Pair
Percent Agreement
Cohen's g
H1-H2
76.3%
0.681
H1-H3
58.9%
0.431
H2-H3
58.5%
0.446
LLM vs. Human Majority Vote. Out of the full set, 240 items had a unique human majority label (13 items exhibited three-way ties). On this subset, the LLM achieves the performance shown Table 4 The Cohen's x between the LLM and the majority vote is 0.533, indicating substantial agreement comparable to human- human levels.
Table 4: LLM agreement with human majority-vote labels.
Metric
Value
Accuracy
0.658
Macro F1
0.666
Cohen's is (LLM vs. majority)
0.533
reliability including the LLM (Table 5). Krippendorf''s s a decreases slightly to 0.497, reflecting the LLM''s moderate alignment with the human annotators.
LLM Fourth Annotator. We also compute
Pairwise comparisons between each human annotator and the LLM (Table show agreement levels similar to those between some human pairs (particularly H1-H3 and H2 H2)H
Table 5: Agreement among humans + LLM.
Metric
Value
Krippendorff's a (nominal, 4 annotators)
0.497
Table 6: Pairwise agreement between humans and the LLM.
Annotator Pair
Percent Agreement
Cohen's k
HI-LLM
62.8%
0.489
H2-LLM
62.5%
0.492
H3 LLM
59.7%
0.448
Overall, the dataset exhibits moderate inter-annotator consistency across all metrics, with variation typical of multi-class subjective labeling tasks. The LLM aligns with human labels at levels compa- rable to Human-humal agreement, and the LLM agrees more with H1 and H2, who have higher inter-annotator agreement.
C TOKEN ROUTING PATTERNS
GPT-5 Prompt for Generating Expert Specific Stimuli Figure I0 presents the prompt used to instruct GPT-5 to generate the question-answer pairs shown in the non-benchmark token-routing pattern plots. The descriptions of the brain networks are identical to those in Figure 9 which were previously used to pseudo-label O1-generated responses for constructing the MICROseT dataset. We queried GPT-5 separately for each expert. We show the routing patterns for additional models in Figure 12 and for MOE models in Figure 17
23

---

Published as a conference paper at ICLR 2026
(a) MiCRo-Llama-3B
Language Bamples
Logic Samples
Samples
World Samples
Language
Language
Language
Logle
Logic
Logic
Logic
Social
Social
Social
Social
World
World
World
World
Neceragge
(a) MiCRo-OLMo2-1B
Language Samples
Logic Samples
Samples
World Samples
Language
Language
Language Logic Social
Logic
Logic
Logic
Social
Social
Social
World
Nemmier Melen No
recerbice teem bu
rersenace of leler
(b) MiCRo-SmoiLLM2-3BMM
Social
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
World
World
World
World
(c)
MiCRo-SmmlLLM2-1335
Language Samples
Logic Samples
Samples
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Social
Social
Social
Social
World
World
World
World
Aecuraage Siiia
Recaeiaaa Manicc
Pertertage
Figure 12:
Token Routing Patterns in Additional McCRo-Models. Percentage of tokens routed to each expert, aggregated across all layers, for additional MICRO models. Distributions are com- puted over GPT-5- -ennr-geeee question-answer pairs designed engage specific domains. Results show consistent brain-inspired specialization, with tokens preferentially assigned to the relevant ex- perts depending on the task domain. Figure [ shows the corresponding layer-wise token routing of these plots, while Figure shows the token routing on benchmark testing data.
24

---

Published as a conference paper at ICLR 2026
(a) MiCRo-Llama-3B
Language Bamples
Logic Bamples
Bamples
Samples
(b) MiCRo-Llama-1B
Language Samples
Sumples
Samples
MiCRo-OLMoZ-1B
Language Samples
Logic Samplee
Social Samples
World Samples
(d) MiCRo-SmoILMER MM-3360M
Language Camplos
Logic Bamples
Bamples
World Samples
(e) MiCRo-SmolLMM M2-1135M
Samples
Logic Samples
Social Samples
World
Figure 13: Layer-wise Token Routing in MICRO Models. Token routing distributions across lay- for five MICRO models, measured GPT-5 geee--eeeeetedd question-ansee pairs targeting specific domains. In all models, the language expert consistently engaged early layers, while domain- specific experts (logic, social, world) increasingly activated in deeper layers. This hierarchical organization parallels findings from cognitive neuroscience, where linguistic processing precedes engagement of higher-level networks.
25

---

Published as a conference paper at ICLR 2026
Layerwise Routing Patterns Figure illustrates layer-wise token routing patterns for five MI- CRO models. Surprisingly, consistent trend emerges: tokens are initially processed by the lan- guage expert before being delegated to higher-level experts depending on the task domain. This organization parallels findings in cognitive neuroscience, where the language network engaged early for virtually all linguistic input and then interfaces with other specialized networks (such as multiple-demand or social cognition systems) depending on task demands (Fedorenko et al. (024). In Figure]14] [4] we show senchmark-specifte token routing patterns across layers as well. To probe cial specialization directly, we also include evaluation on the EMPATHY benchmark (Buechel et al.) 2018), which primarily engages the social expert, further confirming the expected routing behavior generalizable across datasets.
D CORRELATION WITH HUMAN JUDGMENTS
We use dataset of 1,000 six-word sentences from Tuckute et al. (2024b), each annotated with human ratings across several behavioral dimensions, collected independently of our routing frame- work. To test correlations with human judgments, we selected features expected to align with spe- cific experts: GRAMMATICALITY and PLAUSIBILITY with the language expert, MENTAL STATES with the social expert, and PHYSICAL OBJECTS and PLACES with the world expert. The dataset does not include features relevant to the logic expert.
To analyze these relationships, we divide each model into three layer segments (early, middle, late) and averaged router probabilities within each segment. Figure [3 reports correlations between the average routing probability of each expert and human ratings for MICRO-LLAAMA-3, MICRO-LLAMA-3., 3.2-3B... For both models and layer segments, mental state ratings correlate most strongly with the social expert. Language expert probabilities correlate with GRAMMATICALITY and PLAUSIBILITY, but primarily in early layers. PHYYSCAL OBJECTS and PLACES correlate with the world expert, while the logic expert shows no positive correlations (and in most cases negative correlations) with these features. These findings suggest that router exhibits meaningful degree of correspondence with human behavioral judgments.
and
E ADDITIONAL EXPERT ABLATION RESULTS
Figure reports the effect of ablating individual experts, including the language expert, on bench- mark performance for five MICRO models. We find that the language expert essential for most tasks, while domain-specific experts-such as the logic expert for GSM8K and MINERVA MATII- are also necessary to maintain performance. Interestingly, in some cases, ablating an expert im- proves performance, suggesting that certain experts may interfere with more relevant ones, leading to performance degradation when all are active.
F BENCHMARKS
Table 7: Number of Shots and Samples Per Benchmark Used in Evaluation. Number of shots and samples used when evaluating the test-set of each benchmark. Last two row shows whether we used CoT or evaluated using bogabbbbabslities and the metric used to obtain the final accuracy.
Benchmark
GSM8K
Minerva Math
MMLU
MMLLU-Pr
BBH
HellaSwag
PIQA
ARC
ARCesteose
N-Shots
0-Sbot
4 scchots
4 4-chots
S-shoes
S Shots
0-Shot
O.Shot
0-Shot
0.Shet
Num Samples
1,319
5,000
14,042
12,032
6.11
10,02
1,838
2,376
1.172 No
Prompting
Metric
Exact Match
Exect Match
Exact Match
Exact Match
Exact Match
Ace Norm
Aoc Norm
Ace
Acc Norn
Benchmarks Description We evaluate our models on eight benchmarks using various fewshot settings, four of which are prompted to generate reasoning chain before producing the final answer. These reasoning steps are intended to more meaningfully engage the expert modules throughout the generation process, which is why we focused on them in the main paper. The other benchmarks multiple choice questions where the most likely candidate- as measured by the log- proobbbbaillstes of the model-is taken the prediction. Table lists the number of in-context examples used
26

---

Published as a conference paper at ICLR 2026
(a) MiCRo-LLama-3.221B
Minerva
Empathy
MMLU
Language
Language
Languine
Languege
Logic
Logic
World
World
World
World
necemion Oaa
maci thow obent
(b)
MICRo-Llama-332--3
Minerva Math
Empathy
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
Social
Social
Social
Worrld
World
World
World
(c) MiCRo-OLMo-2-1B
Empathy
MMLU
Language
Language
Languège
Languege
Logic
Social
Social
World
World
World
Pecestage vaaeu
nex dage Soee
Figure 14: Benchmark Token Routing Patterns. Token routing patterns for (a) MICRO-LLAMA and (c) MICRO-OLMO-2-11 evaluated on up to 1 1,000
3.2-1B, (b) MICRO-LLAMM--,22-1 samples drawn from the GSM8K, HINERVA-MATH. EMPATHY, and MMLU test sets. For each model, the top panel reports the overall percentage of tokens routed to each expert across the whole model (variance veals emergent hierarchy: earlier layers emphasize language grounding, whereas deeper layers increasingly delegate domain-relevamt relevant experts.
samples), while the bottom panel shows layer-wise routing. The latter
27

---

Published as a conference paper at ICLR 2026
(a) MiCRo-Llama-3.2-1
Early Layers: (1, 5]
Layers: [6 16-18 10]
Late Layers: 16
Aopports Itrry Lageno
jndyrert Lapers)
Conelatoe 2.dp pentt Logens
Loqgag 14ps Gelf
(b) MiCRo-Llama-3.2233
Early Layers: [1 -8
Middlo Layors: [9 -999
Late Layers: [20 - 28)
Compidoon with Hunar peprorts Zelyy Lepro
Conduutioon with Mhtrraaa (alymorr Lapert)
Cenclatiee with seyoot dute Loyes)
-311
Gold 16/4
Sold
Tarl
Correlations Between Expert Routing Probabilities and Human Ratings. Correla- tions shown for MICRO-LLAMA-3.2-1 and middle, and late layer segments. Mental state ratings correlate most strongly with the social expert, grammaticality and plausibility correlate some degree with the language expert (primarily in early layers), and physical objects and places with the world expert. The logic expert shows no positive correlations with these features.
Figure 15:
averaged across early,
28

---

Published as a conference paper at ICLR 2026
GSMBK
Minerva Math
MMLU
MMLU
MMLU
BBH
MICRo-Liaman PORR-LLaam--B
4.8%
AMeaee 25.3%
Abliiin 27.75
Ablidin 34.0
Allilees 2044
MICRo-Liamss 3B
bascc Motes 19)
11
dblttann 15.32%
Ho 35.5%
5000h
MICRo-OLMoz-1B LMM-O0M
-100
Mr
99 KMM006 J13P5
N0 -4.00 41.95
MICRo- MoORo-SmolLMnee 360M
-65
04.04ebe 1.44%
No Aoleee
MICRo M.CRR-SmolLMMe-- SmmoLL22-1555 1255
Sdeem
0.baatt Metten
0.50
-025
-0.75
No Aberes 11.97%
N6 Abertin 2L55%
ABA900 2391
kiddos 4P%
Legend
Language
Figure 16: Expert Ablation Results Across Benchmarks. Impact of ablating individual experts on benchmark performance for five MICRO models. Results are shown for GSM8K, MINERVA MATH, BBH, and MMLU, with the latter divided into its four subcategories. Removing the language expert causes substantial drops across most tasks, while domain speciffc experts (c.g., logic for math benchmarks) are critical for their respective domains. In some cases, ablating an expert improves performance, suggesting interference with more relevant experts.
29

---

Published as a conference paper at ICLR 2026
Additional Benchmark Results for MiCRo and Baselines Accuracy (%) standard eπror across reasoning and knowledge benchmarks, Results are reported for different model classes (Dense, MoB, and MiCRo) under each base model.
Table 8:
GSMBK
Minerva
MMLU
MMLUps
BBH
ARCaa
ARCoateg
HellaSwag
PIQA
Base Model
Model
Dense
SmelIM2-135M
MoB
0.5
06 0.1
219 0.3
74
=35
63.0 +
29.9
L3
43.5 0.5
67.8
MiCRo
3,9 0.5
0840..1
22.5 22.5
0.2
25
$6.0 1.0
27.5 2 1.3
41. 0.5
67.5
Dense
15.0 1: L.0
0.3
26.4
9.9 0.3
27.3 0.5
69.7
19.7.009
37.5
56.6
71.41
SmelIM2-36M
21.1
72.0
MSCRo
16.5 4:
34 0.3
26.0 0.4
10.1
10.1 0.3
27.2 0.5
69.9 0.9
38.2 2 1.4
56.7 0.5
71.7 4:
Dense
36.8 1.3
4.8
29.7 0.4
0.3
30.4
64.3 1.0
33.7 1.4
5.5
73.8
lama--32-118
MoB
40.+-13
3.7
0.4
11.0+0..3
274-05
61.7 +.00
144
71.3++11 4.2
29.8
Dense
58.0 1.4
144
4
09
=4.1=00 0.6
73.6 09
42.9
0.5
77.0 1.0
Llama-32--88
MoB
516 1.4
12.3
1..1
422
71.6 0.9
41.3+ 14
67.3+00.
770 1.0
MICRo
130++0.
45.4
19.0+04
75.0
1300+00
43.3
67.4+0.5
7664+11
and the number of samples tested for each benchmark. For the remaining benchmarks, used the default fewshot examples from the im-evaluatin -ir-vvaluattoon-aaneess on-harnees (Gao et al. 2024) repository. Specifically, used the bbh cot fewshot task for BBH, the mmlu flan cot fewshot task for MMLU, the mmlu pro for MML.U-PO,, the minerva math task for MINERVA MATH, and the bemmk_cot_zene choice benchmarks.
for the GSM8K task. We used the default tasks for the multiple-
Extended Benchmark Results Table & reports results for additional base models as well as on benchmarks beyond those presented in the main paper. Consistent with the main results, MiCRo remains comparable to Dense and MoB baselines across most tasks while being interpretable. These supplementary experiments provide further evidence that the observed trends hold across a broader range of model scales and evaluation settings.
G ROBUSTNESS ACROSS POST-TRAINING METHODS
We further assess the robustness of our method to different post-training methods by applying two variations. First, we further post-train our MICRO models using Direct Preference Optimization
(DPO) (Rafailov et al. 2023) on subset of the TüLU-2.5 preference dataset (Ivison et al. 2024) (Table 9). Second, we replace the large-scale general-purpose TÜLU-3 dataset used in stage-3 with
more domain-spa specific (medical) isstruction-tuning set used in Bosselut results show that our method is robust to different post-training pipelines, whether applying DPO or using an alternative instructoon-tuiig dataset, as shown Tables
(2024) (Table TOp. Our
and respectively.
Table 9: Performance After DPO Finetuning Comparison of MICRO models and baselines after further finetuning with DPO preference dataset. Results show average performance across the 4 benchmarks, indicating that specialization remains beneficial after DPO.
Base Model
Model
GSMSK
Minerva Math
MMLU
BBH
Average
Dense
38.1
3.9
29.4
30.3
25.4
Llama-3.2-1B
MiCRo
39.3
5.8
31.8
30.3
26.8
Dense
45.8
5.6
39.3
29.8
30.1
OLMo-2-1B
MICRO
48.1
5,8
39.8
30.4
31.0
H MIXTURE-OF-EXPERTY RESULTS
In the main paper, we report results using the mixture-of-blee olocks (MoB) architecture, where each expert is a full transformer block with its own attention mechanism. Here, contrast these results with the more standard mixture-of-expertt (MoE) architecture, where experts consist only of FFN blocks and attention is shared across experts within each layer. We first present routing patterns for
30

---

Published as a conference paper at ICLR 2026
Performance on Medical Benchmarks After Domain-Specific Instruction Tuning. Models are finetuned during Stage-3 using a medical instruction- dataset instead of TüLU-3, and evaluated on four medical benchmarks. Results show that specialization achieve competitive performance across both base models, and outperforming in the We choose the option with the highest log- prebaabbbbllt among the multiple-chiie options.
Table 10:
(OOD) setting.
Out-of-Distribution
In-Distribution
Base Model
Model
MMLU Medicine
MedQA
MedMCQA PuhMedQA
Average
Dense
26.0
34.3
33,9
73,4
41.9
Llama-3.2-1B
MICRO
28.3
35.2
33.9
71.4
42.2
Dense
35.8
34.2
35.3
74.0
44.8
OLMo-2-1B
MICRO
35.8
36.3
34.5
73.8
45.1
Results with Mixture-of-Eppers (MoE) Architectures, Accuracy (%) + standard error is reported for Dense, MoE, and MICRO-MoE models across multiple benchmarks, For each base model, the best score per benchmark highlighted in bold.
Table 11:
GSMSK
Minerva
MMLU
BBH
ARCra
ARCcsating
HelloSwag
PIQA
Base Model
Dense
2.7 0.4
0.5 0.1
0.3
24.1 0.5
62.8
1.0
29.6 1.3
43.6
67.7 = 1.I
Smo
2.8
0.6
62.9
67.6
MiCRo-MoE
4.1 + 0.5
0.4 0.1
03
0.5
62.0 1.0
29.0 1.3
43.4 0.5
61.
Dense
15.0
3.7
2.. 0.
0.
37.5 1.4
71.4 0..6
SmolIM2-360M
MoF
16.1 + 1.0
3.6
26.6
0.4
2.4
70.24-09 0.9
37.2
1.4
56.8 +.5
71.8
McCRo-MoB
16.1 1.0
4.0 0.3
26.1
04
2..0
6..7
37.5
1.4
56.7
71.4
Dense
36.8
4.8 03
0.
0.54
64.3
33.7
0.
73.8
1.2ma-3.2.18
MoF
29.1 + 1.3
4.7 +7-000 0.3
25.7 +.4
28.6++03. 0.5
64.1+-18 1.0
35.2
$7.9
72.5
McCRo-MoE
5.0
MICRO-MoE models, highlighting cases where our training curriculum fails to induce the intended specialization an issue we primarily observe in models larger than 1.5B parameters. We then report the performance of the models that did exhibit specialization on reasoning benchmarks.
H.1 MOE TOKEN ROUTING PATTERNS
Figure 17 shows routing patterns for five MICRO-MoE models on guestion-answer pairs gener- model exhibits the
ated with GPT-5 to target specific experts. The MICRO-MOF-LIAMA-- intended specialization, whereas the 3B variant does not. Within the SMOLLM2 family, the 135M and 360M models display partial specialization, though less cleanly than LLAMA-IB IB, often de- faulting to the language expert regardless the input domain. The 1.7B model fails to specialize, indicating that the MoE architecture does not reliably induce
similar MICRo-Mo MOE-LLAME the desired specialization under our training curriculum.
H.2 MoE BENCHMARK RESULTS
Table II presents results for models trained with a Mixture-of- (MoE) design, complemen- tary to the Mixture-of Blocks (MoB) results reported in the main paper. The key distinction between MoE and MoB lies in what replicated to form the experts. In standard MoE, only the feed-forward network (FFN) within each layer is cloned into multiple experts, with the self-attention module shared across all experts. In contrast, MoB duplicates the entire transformer block-includin both the attention and FFN components-so that each expert has its own attention mechanism as well as its own FFN. We find that MoB scales more effectively: under our training curriculum, specializa- tion emerges reliably in larger models (>,1B parameters) for MoB, but not for MoE. For this reason, we focus on MoB the main text and do not include the MoE variants of the other base models, as they did not exhibit the expected functional specialization.
31

---

Published as a conference paper at ICLR 2026
(a) MiCRo-MoE-Llama-1B
Samples
Logic Samples
Social Samples
World
Language
Language
Language
Language
Logic
Logic
Logic
Logic
World
World
World
World
(b)
Language Samples
Logic Samples
Social Samples
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
Social
Social
Social
World
World
World
World
(c) Pitro-MME-SSmoiMM----
Language Samples
Logic Bamples
Social Samples
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
Social
Social
Social
World
Perer of belerii
Promne hames PM
(d) MiCRo-MoE-Llamaas
Language Samples
Logic Samples
Social Samples
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
Social
World
World
World
World
140
(e) MICRO-MOE-SSOLLMM221.777
Language Samples
Logic Samples
Social Samples
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
World
Worldd
World
World
Routing Patterns in MICRO-MoE Models. Routing behavior for five MICRO-MoF models on GPT-S-generated question-answer pairs targeting specific experts. The MICRO-MOE- LLAMA-IB BB model shows the intended specialization, while larger variants (e.g., 3B, SmolLM2- 1.7B) fail to specialize. Smaller SMOLLM2 models (135M and 360M) display partial but less consistent specialization. often defaulting the language expert. These results suggest that the MoE architecture does not reliably induce brain-like like specialization under our training curriculum.
Figure 17:
32

---

Published as a conference paper at ICLR 2026
Performance of MICRO-OLMo-IB when increasing the number of active experts from Top-1 to Top-2 at test time. Enabling an additional expert leads to consistent improvements across most benchmarks, demonstrating that the model generalizes well to increased routing capacity even though SFT was performed with Top-1 routing. The benchmarks MATH, ARC-E, and ARC-C refer MANHH, ARC-EASY, and ARC-CHALLENGE respectively.
Table 12:
K
GSMBK
BBH
MMLU
MATH
HellaSwag
PIQA
ARC-E
ARC-C
Avg
1
45.5%
29.5%
37.9%
5.6% 6.8%
65.4%
75.2%
70.2%
41.3%
46.3%
2
47.7% %
30.6%
38.2%
66.4%
75.4%
71.9%
41.6%
47.3%
I ADDITIONAL BEHAVIORAL ALIGNMENT RESULTS
Figure 18 shows alignment human behavior for additional base models, comparing MICRO with corresponding MoB and DENSE baselines. We find that MICRO achieves higher average behavioral alignment on COGBENCH metries in larger models, while maintaining comparable performance in smaller models. Please refer to $5.4 for more details on how we evaluate the models.
J SPECIALIZATION REMAINS CONSISTENT THROUGHOUT TRAINING
Figure 19 illustrates token routing assignments across checkpoints during Stage 3 training of
MICRO-LLAMA-IB with checkpoint-0 representing the final weights from Stage 2. The results show that the model consistently preserves the specialization established in stages and 2, despite no explicit constraints being enforced during this phase, except for the initial weak inductive bias. This suggests that brain-like specialization may offer a robust initialization, enabling the model to maintain functionally distinct expert behaviors throughout continued end-to-end training.
K
QUALITATIVE EXAMPLES OF STEERING BEHAVIOR
show examples of how one can use MICRO steer the model's behavior by selee- tively ablating or activating certain experts. In the examples provided, we only retain the target expert along with the language expert using the MICRO models. When the social expert is ablated, the model shifts toward more analytical tone, producing response that is logically coherent but lacking in empathy.
Figures 22 25
L TEST-TIME SCALING BY INCREASING THE NUMBER OF ACTIVE EXPERTS
Table 12 reports the effect of increasing the number of active experts at test time from to 2 for MICRo-OLMo-IB. The results show that test-time compute can be scaled by enabling additional experts, and that the model generalizes well to this setting and improves performance on all bench- marks, even though the large-scale SFT stage was trained exclusively with top-1 routing. This indicates that MICRO retains robustness when the routing capacity is expanded inference time under the k=2 setting. However, with larger values of k the performance degrades slightly.
M SCALING MICRO TO LLAMA-3, 1.MMA-3.1130 3.B-8B BASE MODEL
We post-trained three Llama-3. 1-BB variants: (1) MICRO-LLAMA-BB (2) its modular baseline 1MMA88--Mob and (3) CEAMA-8B--ENNES However, due to compute constraints, we instan- tiated experts only in the last 12 layers of MICRO-LLAMA--8 and LLAMA-88 -AAMA8---MAA 1---8B--MMM- MoB, keeping the earlier layers dense. This choice motivated by our prior findings (Figures 14), which show that early layers predominantly route language experts, while non-language specializations emerge in later layers. The results confirm that MICRO-LLAME
exhibits the expected routing patterns in its last 12 layers, shown in Figure 20 Table 13 shows the results 9 benchmarks of the three LLAMA-3.. 1-8B model variants, along with the results when we remove the most detrimental ex- pert across all layers for given task for the MrCRo and MoB models. Using paired Wilcoxon
33

---

Published as a conference paper at ICLR 2026
Llama-3.2-3B
Normalized Score
Behavioral Alignment (S. (BEE
0.9
Prie Waggding
Randon Exporation
Bas Todin
Sweparn bucourtig
Normalized Score
043
SmoLM2-360M
Normalized Score
Behavioral Alignment (S [BRE)
Scnee Averye
Normalized Score
Opttedom
Prier Weightin
Randon Explrraion
Mak Takto
Tempora Doreenting
0.5
5
Normalized Score
SmoLM2-135M
Behavioral Alignment (S BRE)
0.35
0:30
Normalized Score
Opiteduu
Prter Weighting
Random Exploration
Takre
Temporal Daessnting
0.2
Iveng
Model
MiCRo
MoB
Dense
Figure 18: Behavioral Alignment on Additional Base Models. Results for LLAMA-3.2-31 SMOLLM2-360M, and SMOLLM2-135M COGBENCH. Left: average similarity to human behavior across all metrics. Right: fine-grained results for each behavioral metric. MICRO is compared with MoB and DENSE baselines, showing stronger alignment in larger models and com- parable performance in smaller ones.
34

---

Published as a conference paper at ICLR 2026
Language Samples
Checlpunn
Checkpoet1
Logic
Logic
Logic
Logic
Logic
Social
World
World
Logic Samples
Chadigaint
Checkpoent
Chedipsint
Checlpainc
Checkpolnt
Language
Langiuage
Langaage
Larriiage
Langgvage
Logic
Logit
Logic
Social
Social
Social
Social
Social
Werld
Social Samples
Chedipaint
Checippirt 8.25
Chackpolit
Chockpuine
Checkpolet
Langiage
Language
Langjage
Language
Langsage
Logic
Logic
Logic
Logic
Logic
World
World
World
World
World
World Samples
Chellpaert 8
Chestpoert 5.29
Chemlpsitt 2.00
Checkpoint 0.75
Checkpotnt 1
Language
Language
Larigwage
Lariuuage
Languaae
Social
Social
Social
Social
Social
World
World
Figure 19: Specialization Remains Consistent Throughout Training, Token routing across checkpoints during Stage training of MICRO-LLAMA- -B on the samples generated probe each corresponding expert. Checkpoint ( corresponds to the final weights from Stage 2. The plot shows that expert assignments remain stable throughout training, with minimal variation, indicating that the model retains its learned specialization despite the absence of explicit constraints.
Table 13: Benchmark Results for Llama-3.1-8P Model Variants. Performance comparison of LLAMA-3. 1-BB variants across multiple benchmarks. Ablation refers to selectively removing the least relevant expert per benchmark.
Model Name
GSMSK
BBI
MMLU
MMLU Pro
MATHI
HellaSwug
PIQA
ARC-E
ARC-C
Asg
Llama-8B Be-eemse
71.3
51.6
24.6
21.5
72.8
78.3
74.3
44.1
54.7
Llama-8F MoB
69.3
54.1
52.0
23.5
21.3
72.2
78.9
74.6
46.9
54.8
SB (Ablation)
72.4
MiCRo-Llama-SB
69.1
53.6
50.7
23.1
18.1
72.7
78.7
75.7
45.9
54.2
MiCRo-Llana-SB (Ablation)
69.1
52.7
18.2
73.1
78.7
76.3
56.2
signed-rank test across the nine benchmarks, we find statistically significant difference between MiCRo and either of the two baselines. The DENSE vs. MICRO comparison yields test statistic of 31.0 (p 0.18), and the MoB vs. MICRO comparison yields 33.5 (p = 0.10). both well above the conventional 0,05 significance threshold. However, when we ablate the most detrimental expert per task for both MiCRo and MoB we see a jump performance, as also illustrated in Figure 7 In general, the Llama-8B experiments confirm our initial results that MICRO remains competitive on a suite of benchmarks while remaining interpretable and relevant to cognitive neuroscience.
N LARGE LANGUAGE MODEL USAGE
We used large language models (LLMs) solely for editing and grammatical refinement of the manuscript. All substantive ideas, analyses, and conclusions presented in this work are our own.
35

---

Published as a conference paper at ICLR 2026
MiCRo-Llama-BB
Language Samples
Loglc Samples
Social Samples
World Samples
Language
Language
Language
Language
Logic
Logic
Logic
Logic
Social
Social
Social
Social
World
World
World
World
Token Routing Patterns for MICRO-LLAMA-SEE (Top) The percentage of tokens routed to each expert aggregated across the last 12 layers of MICRO-LLAMA-SIE The samples are GPT-5 generated duestion-answer pairs targeting specific domains. (Bottom) The corresponding layer-wise token routing patterns.
Figure 20:
MiCRo-Llama-8B
Language Localizer
MD Localizer
ToM Localizer
13%
20.0
21.5 %
3
11.4%
20
12
9 5
23%
999
23%
17.5
16.3%
11%
11.3%
15.3%
11
8.5%
15.0
11
11.2%
11
Number
13
Layer Number
16
13
22.4%
19 17 15
12.5
19 17 13
11.3%
11.4%
15
11.9%
10
12.5%
11.4%
10.0
10.3%
10.4%
-14
Layer
17
19
82%
42%
91%
24%
10.0
0.%
65%
27 23
-75
23
94%
59%
51%
-12
23
78%
53%
95%
9.5%
10:0
90%
14.0
10.%
2033%
25
7.3%
53%
23%
2.3%
23%
10:0
6.%
51%
5500
7.%%
23%
21%
51 29 2/
27
5.5
1433%
15.0%
75%
53%
51%
1.1%
4.4%
1000 %
67%
98%
7.3%
12.7%
81%
99%
52%
132%
31 29
2.5
2.%
1.%%
14.4
11.4%
106%
44%
81%
17%
32.7%
100
54%
1.4%
Lang Logic Social World
Lang Logic Social World
Lang Logic Social World
Figure 21: Neuroscience Localizers Fail to Recover Experts For Hybrid MICRo-LIama-SB Following the procedure Figure we apply three neuroscience-inspred functional localizers to MICRO-LIIAM--S which places experts only in the final 12 layers due to compute constraints. In contrast to the full MICRo variants with experts in every layer, the localizers are unable reliably recover the expected expert specializations in this hybrid configuration.
36

---

Published as a conference paper at ICLR 2026
Prompt
If rwo people disagree on how to split $10 fairty, what should they do?
All Experts
If two people disagree on how to split $10 fairly, they should try to find a compromise that satislies both parties. This could involve diacuasing the different ways splitting money and finding solution that fair both of them. It's important remember that the goal is to reach fair and equal agreement, so boih parties should be willing to work logether to find a solution.
Logic
two people clisegree on how to spiit $10 fairly, they should try to compromise that can agree on. They can such as the average of the two numbers, by trying find number that closer both numbera than either number itself.
this by trying to find number that both people
Social
two people disagree how to split $10 fairly, they should fry to compromise that satisfies both parties. are some steps they can take:
"CComunicaate openly":: Both parties should express their opinions and listen each other's perspectives. This will help them understand each other's viewcoints and find common ground.
2. ""Find common ground"*: Look for areas of sgreement or values that both partles share. This will help them find a compromise that is fair to both of them.
"Discuss reasons behind their disagreement*: t both parties why they disagree on split S10 fairly. This will help them understand each other's perspectives and find solution that sarisfies both parties.
4. "EExpoore alternative solutions"*: " If they cannot find compromise that satisfies both parties, they can explore attemative solutions, splitiing the money different way or finding different way to
money fairly.
disagreement is too difficult to resolve on own, they can seek help from a mediator, such as family member, friend, or counselor, to help them find a soi ution that is fair to both partles.
"Seek help from mediator"*;
*FFind solution that works for both parties" * Ultimately, they should find ution that works for both parties and is fair both them. This may involve compromising on cartain points finding different way to split the money fairly.
World
two people disagree on how to split $10 fairly, they should consider following options:
Mediation: One person can mediate the disagreement by suggesting compromise or fair split. This can he both parties reach agreement without further confiiit.
split by discussing try to find solution that both parties can agree on.
Negnliation: parties can negoliate
splits. They
Arbitration: the disegraement la significant and cannot be resolved through mediation negotiation, both partles seek of a
party, such
mediator or arbitrator, help them an agreement.
the di sagreement is not resolved through mediation or negotiation, both parties can consider seeking legal advioc or taking legai action to enforce their rights.
Legal action:
Consultation: Bnth parties consult with neutral third party, such financial ariissr or medietor, tn help them reach a fair spilt. This can provide a neutral perspective and help both parties reach an agreement.
Ultimately, the best course of action depends on the specific ciroumstances of the d'sagreement and the preferenoes of the parties involved.
Figure 22: Example for Steering Model Behavior by Expert Ablation. Responses of MICRO- LLAMA-3B to the given prompt when only the target expert and the language expert are retained. The differences illustrate the causal role of each expert and demonstrate how ablations steer the model's s behavior.
37

---

Published as a conference paper at ICLR 2026
Prompt
If your friend asks whether you like their new hairout, but you don't, what should you say?
Exports
You can say something like, appreciate your creativity, don't the style me."
Logic
You can say sornsthing like, "I'm glad you're happy with it, but I'm not a fan of it myself."
Social
When your friend asks if you like their new haircut, but you don't, it's important to be honest and respectful. Here's a suggestion on how to respond:
"Be Honost"": can say something liko, really appreciate your style, but this particular haircut isn't for think it's great choice for you, though."
""Compliment": If you do like something about their new lonk, make sure to compliment it. For example, love your hair color. This new cut just isn't for me."
"Express Your Feelings" If you're not a fan of the haircut, you say something like, "I just don't think this particular style works for me. I'm sure it's going to be a hit with you."
"Offeer Support"": your friend disappointed, affer your support. "I know you're disappointed, but I'm sure you'll find something you love."
Remember, it's important he kind and understanding, even you don't like hairout. It's also okay honest about your feelings.
World
can say something like, "I'm sorry, but don't really like it."
Figure 23: Example for Steering Model Behavior by Expert Ablation. Responses of MICRO- LLAMA-3B to the given prompt when only the target expert and the language expert are retained. The differences illustrate the causal role of each expert and demonstrate how ablations the model's behavior.
38

---

Published as a conference paper at ICLR 2026
Prompt
Your friend late meeting. Why might that
All Experts
Your friend might late meeting for variety of reasons. They could running late, they might have forgotten their keys, or they could be a rush to get to the meeting. It's important to remember that it's not always possible to avoid being late, and it's important to respect your friend's time and space.
Logic
Your friend might be late to meeting because they are late to work, or they late to the bus, they late to the train, or they are late to the airport...
Social
Your friend might be late to meeting for various reasons. They could be late due to personal emergency. they might be late because they are not ready to meet, or they could be late due to lack of preparation. It's important to approach this situation with empathy and understanding.
World
Your friend might be late meeting variety reasons. They could be running late, they might waiting for someone elso, or they could experiencing delay in their usual schodule. It's important remember that being late be frustrating and may not good reflection of your friend's character reliability.
Figure 24: Example for Steering Model Behavior by Expert Ablation. Responses of MICRO- LLAMA-1B 1B the given prompt when only the target expert and the language expert are retained. The differences illustrate the causal role of each expert and demonstrate how ablations can steer the model's behavior.
39

---

Published as a conference paper at ICLR 2026
Prompt
Hew could people decide who gets the last slice of pizza?
Experts
Two people could decide who the last slice of pizza using fair and unb ased method. Here few suggestions:
Random draw: One person could draw random number between and 2, and the other person would get the last siice their number
2. Coin flip: One person could flip coin, and the other person would get the last elice the coin lands on heads.
the s.m Roiling dioa: One parson could pair dioe, and other person would get the even.
Drawwin cards: One person could draw card from a deck, and other person would get the last slice the card is number betweon and 2.
Using a fair method:: One person could use a method, such coin flip or a random draw, to determine who gets the last slice of pizza.
is important to ensure thai the melhod used to determine who gets the last slice of pizza fair and unbiased, so lhail both people feel that the cecision made in good faith.
Logic
To deoide who gets the last siice of pizza, can use varieity of metheds One way to take turns picking the last silco. Another way to or random method determine who gela the Remember, It's important respact each other's choices and enjoy the pizza together.
Social
When deciding who gets last siic of pizza, It's imporfant consider the following factors:
1. "Relstionship and History"": If you have a history of sharing pizza or have close relationship with the other person, they may be likely
last slice with you.
2. "Carrent mood and energy"*: If you're foeling particularly energetic or in a good mood, you may be more likely to share the lust siice pizca with the other
3. "Physical appearance" you're physically more capable have better sense balance, you may be more likely to share the last slice of pizza with the other person.
"Emotiiona state"": you're feeling particulerly happy content, you may likely with the other person.
zza
5. "Situational context" you're particular situation or context, such being in group havving specific reason for sharing pizza, you may be more likely to share the lest slice of pizza with the other person.
6. "Nonverbal cues"": If you're using nonverbal cues, such a smile a friendly gesture, you may more likely to share the last elice of pizza with the other person.
7. "Communi cation" If you're communicatingg with the other person, you may be able to corvery your intentions desires effectively. could
more positive cutcome.
8. "Phyyscal proximity": If you're physically closer to the other person, you may be more likely to share the lest slice of with them.
9. ""P Personal values":: you have strong sense of fairness or equality, you may be more likely to share the last slice of with the other person.
""Socia norms" you're particular social group or communityy there may established expectations regarding sharing
World
meeting variety of reasons. They running late, they might someone also, or they could be experiencing a delay in their usual schoculo. It's important remember that haing lafe can be frustrating and may not be a good reflaction your friend's character or raliability
friend might
Figure 25: Example for Steering Model Behavior by Expert Ablation. Responses of MICRO- LLAMA-3B the given prompt when only the target expert and the language expert are retained. The differences illustrate the causal role of each expert and demonstrate how ablations can steer the model's s behavior.
40


Reviews:
[{"review_id": "mkKT3s4zTF", "reviewer": "Reviewer_JAhk", "summary": "The paper proposes the Mixture of Cognitive Reasoners, a modular, transformer-based architecture that built based on neuroscience background with individual language, logic, social, and world experts. The training procedure of the model is composed by the following three stages: Training the experts with a small curated dataset to provide basic inductive bias; Training the router using the same dataset, with the experts frozen; Training the whole model with large instruction-tuning dataset. Experiments shows that the model acquires experts specialization, exhibits alignment to human behaviors, and matches dense and non–brain-aligned modular baselines on several tasks.", "questions": "I do not have additional questions at this stage.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and understandable.\n- The neuroscience motivation of the architecture design is very novel.\n- The training procedure is novel.\n- The experimental results are promising, showing that the proposed Mixture of Cognitive Reasoners model lead to strong interpretability and performance gain over other architectures.", "weaknesses": "- The language, logic, social, and world experts decomposition might not be optimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZjJZ5chOZ7", "reviewer": "Reviewer_1ukn", "summary": "This paper introduces MICRO (Mixture of Cognitive Reasoners), a modular transformer architecture inspired by the functional specialization of human brain networks for language, logic, and social reasoning. The model partitions layers of a pretrained language model into four expert modules, each aligned with distinct cognitive domains, and is post-trained using a curriculum to promote specialization. MICRO offers three main advantages: (1) its expert modules are interpretable and causally meaningful, (2) it allows dynamic control at inference by routing tokens to domain-specific experts, and (3) it matches or exceeds baseline performance on reasoning benchmarks (e.g., GSM8K, BBH) and human-alignment tasks (CogBench). Overall, MICRO demonstrates that cognitively inspired modularity can enhance both the interpretability and human-likeness of large language models.", "questions": "Please discuss limitations of MICRO and how these limitations may be addressed.\n\nIt is unclear what is going on in Figs 4 and 16. Please explain the figures.\n\nWhat is the key for Figure 5?\n\nPlease discuss the possible impact of this research.\n\nPlease discuss the generalizability of this research.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The novelty of MICRO lies in its cognitively inspired modular design, where a pretrained language model is partitioned into expert modules aligned with human brain networks for language, logic, and social reasoning. This structure enables interpretable, causally meaningful reasoning and dynamic control at inference, while maintaining or improving performance on both reasoning and human-alignment benchmarks.\n\nThe research provides:\n\n•\tInterpretable modules that provide causally meaningful insights into model behavior.\n\n•\tDynamic control at inference, allowing selective routing to domain-specific experts.\n\n•\tStrong empirical performance, matching or exceeding baselines on reasoning and human-alignment benchmarks.", "weaknesses": "There are some shortcomings:\n\n•\tIncreased model complexity due to multiple expert modules and modular routing.\n\n•\tPotential scalability issues for very large models or tasks requiring many cognitive domains.\n\n•\tThe work makes a valuable contribution and builds effectively on current advances. However, including a discussion of remaining challenges and possible avenues for future research would strengthen the paper and highlight its long-term potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fb622A8hNQ", "reviewer": "Reviewer_vTG9", "summary": "This paper proposes MICRO (Mixture of Cognitive Reasoners), a novel mixture-of-blocks architecture designed to induce functional specialization across different experts. Unlike conventional MoE models that apply expert routing only to FFN layers, MICRO routes tokens through entire Transformer blocks, leading to emergent domain-specific experts (e.g., language, logic, social reasoning). The work is conceptually inspired by cognitive neuroscience and provides interesting empirical evidence for modularity in large language models.", "questions": "No.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is thought-provoking and connects cognitive science with model architecture in an elegant way. The results showing emergent specialization patterns and controllable routing behavior are intriguing. Overall, the idea of modeling “cognitive modularity” within Transformers is both fresh and potentially impactful.\n\n- The paper is generally well written, and the experiments are clearly structured. The figures and analyses are helpful for understanding how modularity emerges under the proposed mechanism.", "weaknesses": "- While the modular design is inspired by human cognition, the model enforces top-1 routing per layer, meaning each token is processed by only one expert. In contrast, human reasoning typically involves parallel activation and cooperation among multiple brain regions. This exclusive routing assumption may limit the biological and functional plausibility of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9rQbHlOK6C", "reviewer": "Reviewer_A5jV", "summary": "This paper proposes a method (MiCRo) for post-training a transformer to incorporate specialized expert blocks across four domains: language, logic, social, and world knowledge. Inspired by the functional regions observed in brain networks, the approach involves a three-stage training process: (1) duplicating the original transformer blocks to create four expert blocks, each trained on data labeled at the sentence level according to domain; (2) freezing the network parameters and training routers to direct inputs to the appropriate expert blocks; and (3) conducting large-scale supervised fine-tuning. Experimental results demonstrate that, after training, these expert blocks exhibit distinct and specialized functionalities.", "questions": "1.\tThe paper notes that small MOE models do not show expert specialization like the MiCRo counterpart. Is it due to the limitation of the model scale?\n\n2.\tIf a question is related to more than one field (e.g., a question that is both related to math and social science), what will be the behavior of the MiCRo model? Are there any experiments on this?\n\n3.\tThe paper mentions that using a small amount of data in Stage 1 and 2 suffices to elicit expert specialization behavior and that this specialization remains in the large-scale SFT stage. What is the intuition behind this phenomenon?\n\n4.\tIn some subplots in Figure 4 (e.g., MiCRo-Llama-3B on MMLU_other), ablating any of the logic, social, or world expert can improve performance. It seems weird since it implies all of these experts are detrimental to the task. Are there any explanations on this phenomenon?\n\n5.\tThe meaning of the marks “*” and “ns” in Figure 7 is not clear. One can guess that it shows the significance of the Welch’s t-tests, but it is better to explain them in the caption.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed method is well-motivated by the four modular functional regions in human brain.\n\n2.\tThe paper is clearly written and easy to follow.", "weaknesses": "1.\tThe diversity and scale of tested models can be further enriched. The paper test Llama, SmolLM, and OLMO models of up to 3B parameters. I would recommend testing on 7B scale models from other model families, such as Qwen2.5-7B-Instruct. Small models sometimes behave very different from large models, so experiments on 7B scale models would make the results more convincing.\n\n2.\tThe paper categorizes MMLU as a reasoning benchmark, which is not quite accurate. I would recommend extra experiments on MMLU-Pro (or a subset of it), which contains more reasoning-oriented questions.\n\n3.\tIn Figure 7 and Table 3, the proposed MiCRo architecture does not outperform the dense model baseline on reasoning tasks, even though it has three times more parameters. While MiCRo-Ablation (which shows the best results when up to one expert is ablated) can sometimes achieve better performance than the dense model baseline, it requires four times more compute at test time. Therefore, the improvement of MiCRo on reasoning tasks is still limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RA8ZPH5FRn", "reviewer": "Reviewer_D255", "summary": "This paper proposes Mixture of Cognitive Reasoners (MICRO), a modular transformer architecture inspired by the brain’s functional specialization into distinct cognitive networks (language, logic, social reasoning, world knowledge). The model partitions each transformer block into four “experts” corresponding to these domains, trained through a three-stage curriculum intended to induce brain-like specialization. The authors claim this architecture enhances interpretability, controllability, and alignment with human behavioral benchmarks (COGBENCH), while maintaining competitive reasoning performance on standard NLP tasks (e.g., GSM8K, MATH, MMLU, BBH).", "questions": "How do you justify the specific choice of four networks, given the ongoing debates about their boundaries and overlap in the brain?\n\nWhat validation steps were taken to ensure the O1/GPT-4o pseudo-labels correspond to human-like task domains?\n\nHow do you distinguish interpretability (as in mechanistic insight) from mere architectural labeling?\n\nWhat concrete cognitive-neuroscience hypotheses does MICRO make that could be tested empirically?\n\nGiven the small size of the MICROSFT dataset (≈3k examples), how sensitive are your results to its composition or labeling noise?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivating analogy to cognitive neuroscience is clear and interesting: the authors connect transformer modularity to the brain’s distributed but specialized functional architecture.\n\nThe paper is technically ambitious and proposes a relatively clean experimental pipeline (three-stage training) that is easy to reproduce conceptually.\n\nThe authors provide a comprehensive empirical evaluation, including behavioral alignment metrics, neuroscience “localizers,” and ablation analyses.\n\nThe inclusion of interpretable routing and causal ablation is a meaningful step toward testable hypotheses about functional decomposition in large models.", "weaknesses": "1. Conceptual clarity and motivation.\nWhile the analogy to brain modularity is compelling, the paper does not clearly articulate why such modular specialization is desirable in language models. The claimed benefits—interpretability and controllability—are asserted but not demonstrated. The architecture yields mixed performance gains, suggesting that interpretability alone may not justify the added complexity.\n\n2. Neuroscientific grounding.\nThe mapping between the four “expert” modules and the purported brain networks is overly categorical and simplified. The cognitive neuroscience literature remains divided on several of these assumptions. For example:\n- The supposed separation between language and reasoning networks remains debated;\n- The distinction between “logic” and “math” is underdefined, yet the paper treats them as part of a single module.\n- Recent work (e.g., Hope Kean et al.) suggests that logical reasoning may rely on a distinct neural network separate from the multiple-demand system invoked here.\nAs such, the neuroscience framing may be more metaphorical than mechanistic.\n\n3. Data labeling and methodology.\nThe “MICRO_SFT” dataset is central to inducing specialization, but the criteria for domain labeling are unclear. The dataset was pseudo-labeled using O1 and GPT-4o, but there is little evidence that these models’ judgments correspond to meaningful domain boundaries. No human validation or inter-rater reliability is reported. It remains uncertain whether the apparent “specialization” reflects genuine cognitive decomposition or artifacts of the labeling pipeline.\n\n4. Relevance to cognitive neuroscience.\nThe authors seem to suggest (implicitly) that MICRO could bridge AI and brain science, but the connection is speculative. The architecture may be inspired by brain modularity, yet it does not provide new neuroscientific insight—no neural data are modeled, and the alignment tests (using functional localizers) are correlational. While ablation studies show that removing experts affects performance, this is a coarse-grained effect and not obviously interpretable at the cognitive level.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)