Representational Alignment Across Model Layers and Brain Regions with Multi-Level Optimal Transport
Authors: Shaan Shah
Meenakshi Khosla
Venue: ICLR 2026 Poster (2026)
Keywords: Representation Similarity
Representational Alignment
Primary area: applications to neuroscience & cognitive science

Abstract: Standard representational similarity methods align each layer of a network to its best match in another independently, producing asymmetric results, lacking a global alignment score, and struggling with networks of different depths. These limitations arise from ignoring global activation structure and restricting mappings to rigid one-to-one layer correspondences. We propose Multi-Level Optimal Transport (MOT), a unified framework that jointly infers soft, globally consistent layer-to-layer couplings and neuron-level transport plans. MOT allows source neurons to distribute mass across multiple target layers while minimizing total transport cost under marginal constraints. This yields both a single alignment score for the entire network comparison and a soft transport plan that naturally handles depth mismatches through mass distribution. We evaluate MOT on vision models, large language models, and human visual cortex recordings. Across all domains, MOT matches or surpasses standard pairwise matching in alignment quality. Moreover, it reveals smooth, fine-grained hierarchical correspondences: early layers map to early layers, deeper layers maintain relative positions, and depth mismatches are resolved by distributing representations across multiple layers. These structured patterns emerge naturally from global optimization without being imposed, yet are absent in greedy layer-wise methods. MOT thus enables richer, more interpretable comparisons between representations, particularly when networks differ in architecture or depth. We further extend our method to a three-level MOT framework, providing a proof-of-concept alignment of two networks across their training trajectories and demonstrating that MOT uncovers checkpoint-wise correspondences missed by greedy layer-wise matching.

Published as a conference paper at ICLR 2026
REPRESENTATIONAL ALIGNMENT ACROSS MODEL LAYERS AND BRAIN REGIONS WITH MULTI-LEVEL OPTIMAL TRANSPORT
Shaan Shah University of California San Diego
Meenakshi Khosla
University of California San Diego
ABSTRACT
Standard representational similarity methods align each layer of network to its best match in another independently, producing asymmetric results, lacking a global alignment score, and struggling with networks of different depths. These limitations arise from ignoring global activation structure and restricting mappings rigid one-to-one layer correspondences. We propose Multi-Level Optimal Transport (MOT), a unified framework that jointly infers soft, globally consistent layer-to-layer couplings and neuron-level transport plans. MOT allows source neurons to distribute mass across multiple target layers while minimizing total transport cost under marginal constraints. This yields both a single alignment score for the entire network comparison and soft transport plan that naturally handles depth mismatches through mass distribution. We evaluate MOT on vision models, large language models, and human visual cortex recordings. Across all domains, MOT matches or surpasses standard pairwise matching in alignment quality. Moreover, it reveals smooth, fine-grained hierarchical correspondences: early layers map to early layers, deeper layers maintain relative positions, and depth mismatches are resolved by distributing representations across multiple layers. These structured patterns emerge naturally from global optimization without being imposed, yet are absent in greedy layer-wise methods. MOT thus enables richer, more interpretable comparisons between representations, particularly when networks differ in architecture or depth. We further extend our method level MOT framework, providing a proof-of-concept alignment of two networks across their training trajectories and demonstrating that MOT uncovers checkpoint- wise correspondences missed by greedy layer-wise matching.
three-
INTRODUCTION
Understanding high-dimensional neural activity is a shared challenge in neuroscience and artificial intelligence (AI). In neuroscience, comparing neural responses across individuals reveals which computations are universally shared versus idiosyncratic. In AI, comparing representations across models reveals how architectural choices, training objectives, and learning dynamics shape learned features, and helps identify prineiples of universality i.e. representational properties that emerge consistently across diverse network architectures and objectives. Comparing models to brains extends this logic further: while we cannot rerun biological evolution, we can simulate "evolution in silico" by training artificial networks with different constraints, inputs, and objectives. When such models converge on brain-like representations, they offer mechanistic hypotheses for why the brain may have adopted its computational strategies which is a deeply important theoretical question. These comparisons have revealed striking similarities between biological and artificial networks (Yamins al. (2014); Eickenberg et al. 2017): Güçlü and Van Gerven (2015); Cichy et al. (2016); Shaligh-Razav and Kricgeskorte/ (2014); Schrimpf et al. (2018) 2020); Storrs et al. (202) I); Kell et al. (2018)), common computational motifs across diverse architectures and objectives Huh et al. (2024); Korbblith et al. (2019); Bansal et al. (2021) Dravid et al. (2023), and other universal representational dimensions Chen and Bonner (2025) Hosseini et al. (2024).
The standard approach to representational comparison is layer-wise matching: each source layer is paired with the single best-matched target layer under some similarity measure (e.g., Representational

---

Published as a conference paper at ICLR 2026
Similarity Analysis (Kriegeskorte et al. 2008), Centered Kernel Alignment (Kornblith et al. 2019), Procrustes distance (Williams et al. 2021) or linear predictivity). Despite its widespread use, this approach has fundamental limitationn. It enforces rigid one-to-one correspondences that fail when networks differ in depth or when a source layer corresponds to features distributed across multiple target layers. It produces asymmetric layer mappings depending on the direction of comparison and yields no unified score for global network alignment. Most importantly by optimizing each match independently, it ignores the global activation structure and risks overfitting noise.
We propose Multi-Level Optimal Transport (MOT), a framework for globally consistent representa- tional alignment. Optimal transport- based methods, such as Soft- Moft-Matching distance (Khosla and Williams 2024 Khhsla et al.] 2024), have recently emerged as powerful metrics for comparing neural representations. Unlike metries such as RSA, CKA, or linear predictivity-whhic are rotation- invariant and thus unable to capture similarities in neuron-level tuning- OT--aasdd- methods are -taation-ensiiive. They explicitly match neurons based on their tuning profiles and, by relaxing hard permutation constraints into fractional couplings, can also handle layers of unequal size. This enables richer, more flexible neuron-level alignments than either rotation-invariant similarity metrics approaches. However, Soft Matching also remains limited to pairwise layer comparisons like other methods and does not capture global structure across networks. MOT fills this gap by operating across multiple levels: it simultaneously infers soft neuron-to-neuron couplings within layers and a soft, globally consistent layer-to-layer coupling across the two levels. Rather than forcing each source layer to match exactly one target layer. MOT allows source layers to distribute their representational "mass" across multiple target layers while minimizing the total transport cost under marginal constraints; that is, each source layer must distribute exactly 100% of its mass across target layers (no information is lost), and the total mass each target layer receives from all source layers must sum a balanced allocation (no target over- or uttll-ztil,zed), These conservation laws ensure a balanced alignment where every layer contributes meaningfully to the global correspondence, preventing any layer from being arbitrarily overweighted or ignored in the global matching. The result is a single network-level alignment score and soft transport plan that naturally handles depth mismatches.
or strict permutation-base
We evaluate MOT on three diverse domains: comparisons between foundation models in vision (Vision Transformers like DINOv2 and ViT-MAE), large language models of varying scales (LLaMA, Qwen), and fMRI recordings from human visual cortex across different participants. Our key contributions are:
A principled global alignment framework (theoretical): MOT jointly optimizes all layer correspondences to produce symmetric, globally consistent assignments with a single alignment score. In contrast, greedy pairwise approaches are asymmetric, can overweight certain layers while completely ignoring others, and may spuriously treat wide layers as similar every layer they are compared against, since their high dimensionality allows them to fit or partially overlap with many different representational subspaces, obscuring more meaningful correspondences.
Natural handling of depth mismatches (theoretical): By allowing soft, many-to- to-y mappings between layers, MOT aligns networks of different depths without forcing inappro- priate one-to-one correspondences.
extension (theoretical): We propose extension of MOT that incorpo- rates additional orthogonal transformations, making the framework inattionnivvriant. This ensures that correspondences can be recovered even when shared representational features are embedded in rotated subspaces, and yields consistently high-quality alignments.
Improved alignment scores (empirical): Across domains (vision models, large language models, and brain data), MOT matches or surpasses standard pairwise methods, yielding higher alignment.
Emergent hierarchical structure (empirical): Without imposing ordering constraints, MOT recovers known hierarchical organization in visual cortex data across subjects and reveals clean layer-to-layer correspondences
model-modes comparisons where early layers map early layers and deeper layers preserve their relative ordering. By contrast, greedy pairwise methods fail to reveal hierarchical structure, often leaving many layers unmatched while single or few layers dominate the mappings.
2

---

Published as a conference paper at ICLR 2026
Featural distribution across depth (empirical): MOT reveals how deeper networks spread computations across multiple layers that shallower networks compress into fewer stages. Concretely, single layer in shallower network often distributes its mass across several neighhoring layers in deeper network, an effect that greedy pairwise methods completely miss.
2 METHODS
2.1 PROBLEM SETUP AND EXISTING APPROACHES
Comparing the internal representations of neural networks often proceeds by a pairwise layer search under some similarity or distance measure. In general, let 7 denote the number of stimuli (e.g.. images, text sequences) used probe both models. We consider two neural networks:
The first network has T. layers. Layer contains units, and its activations across the T stimuli are represented by
X RR. Tx",
1= L.
The second network has layers. Layer m contains nym units, with activations
Ym E T
m I...., M.
Each row of Xę or Y,,. corresponds to the response of all units in that layer to single stimulus, while each column corresponds to the activity of a single unit across stimuli. For any chosen alignment metric S(., :) (e.g., linear predictivity, Procrustes distance, RSA, CKA, or Soft Matching), one typically does:
m (t) arg max S(Xe, Ym.
and then reports the layer-wise score S(x,, (e)( This enforces hard one-to-one mapping from each source layer to a single target layer m (L). However, when L & M, or when the set features represented in layer of network A distributed across multiple layers of network B, such rigid per-layer pairing is not well-suited: a source layer may genuinely correspond to a mixture of multiple target layers. Moreover, by optimizing each layer independently, this approach ignores the global structure of all activations and can overfit to noise in any single layer's responses.
Pairwise Optimal Transport
Multi-Level Optimal Transport
Matan
Figure 1: Left: Pairwise OT. Layers are matched independently, multiple target layers can be mapped to the same source while other sources remain unused, yielding asymmetric, unbalanced mappings. Right Multi-Level OT. MOT infers globally consistent transport plan where each source layer distributes all its mass and each target layer receives exactly one unit, ensuring balanced, symmetric alignments and handling depth mismatches.
2.2 MULVII-LEVEL OPTIMAL TRANSPORT
Our framework operates at two levels: an inner level that aligns neurons within each layer pair, and outer level that determines how layers should be coupled globally.

---

Published as a conference paper at ICLR 2026
2.2.1 INNER LEVEL: NEURON-TO- TO-NURON TRANSPORT
For each candidate layer pair (L, m). MOT computes a neuron-level alignment cost by applying the soft-matching OT (Appendix A for more details) formulation to the tuning functions in Xx and Ym- We first construct pairwise dissimilarity matrix
Cim [ c(Xe(;, Ym(:
where .) is the chosen tuning-based dissimilarity (here, correlation distance). The inner OT objective then computes
min ETneTT yielding a soft coupling Qem that specifies how strongly each neuron in layer & corresponds to neurons in layer m.
Cêm
(Cem Qtm).
Because Qem lies in the transportation polytope, its row and column marginals ensure that each source neuron distributes its representational mass uniformly and each target neuron receives its appropriate share. When the two layers have the same width (rig = n' ). the optimal solution reduces to permutation (by the Birkhoff-von Neumann theorem (von Neumann 1953 De Loera and Kim 2013)); when they differ, soft assignments naturally emerge, allowing neurons in one layer to map to mixtures of neurons in the other. The resulting inner OT cost Cim serves the layer-to-layer dissimilarity used by the outer level MOT.
2.2.2 OUTER LEVEL: LAYER-TO-LAYER TRANSPORT
The inner costs Cim form layer-to-layer cost matrix C RLXM, We solve second optimal transport problem to find the global layer coupling:
P arg min P) (LT(
where T(L, M) is the transportation polytope for layers:
P = M. Σ = -, Pim 2 0
T(L, M) = RLLM
7
The element Pêm represents the fraction layer f's representation that explained by layer m. These marginal constraints ensure mass conservation: each source layer distributes its unit mass across target layers (after normalization by I.), and the total mass received by all target layers is balanced.
When the two networks have the same number layers (L = M). we can show that the optimal solution P is a (scaled) permutation matrix: the objective PC is linear in P, and the constraints define the transportation polytope whose vertices are permutation matrices (scaled by 1/L) by the Birkhoff-von Neumann theorem. Since linear programs achieve their optima at vertices, the solution finds a one-to-one layer matching when allows source layers to distribute their mass across multiple target layers, naturally handling depth mismatches.
M. However, when L * M, the soft coupling
2.2.3 RECONSTRUCTION AND EVALUATION
Given the layer coupling /' and neuron couplings {Qem), J. we reconstruct layer
X₁ T. Σ Pm Qmm
m==1
The factor L appears because Pt. sums to 1/L across We evaluate alignment quality using the mean correlation between original and reconstructed neurons on held-out data:
Score,
p(Xel: 1:. 1], &cl..
1=1
The global MOT score the average across all layers:
Σ Score;-
MOT = L
4

---

Published as a conference paper at ICLR 2026
2.3 ROTATION- EXTENSION
Most existing metrics of representational similarity, such as RSA, CKA, and Procrustes distance, are designed to ontationinvvarian. The rationale that when comparing population codes, we often care less about the tuning of individual neurons and more about the geometry of the representational space: distances, angles, and relative positions between stimulus responses. Two networks can encode essentially the same geometry while using different coordinate bases (for example, rotated versions of one another). Requiring strict unit-to-unit correspondence in such cases would artificially inflate dissimilarity, even though the representational geometry and information content are preserved. A rotation-invaria extension of MOT (MOT + R henceforth) therefore provides a way to capture this geometric equivalence while still enforcing globally consistent layer- and neuron-level coupling. We introduce rotation matrices Rem E O(ne) for each layer pair and minimize the reconstruction error:
Cem
min Rem
[XeRem
Ym Qin
tm Ii2--
We optimize via alternating minimization:
Fix R, update Q: Solve optimal transport using correlation distance rotated features
X Rem-
Fix Q, update R: Solve orthogonal Procrustes via SVD of X, (mm)
Update P: Refresh the outer coupling using the Frobenius reconstruction costs.
Predictions incorporate the learned rotations:
M X
= L Σ TV
Pem Ym Q R
2.4 BASELINE COMPARISONS
We compare MOT against several baselines isolate the contribution of each component:
Random Layer Assignment (Perm-P): We randomly permute the rows of P, breaking the opti- mized layer correspondences while preserving the neuron-level optimal transport within each layer pair. This control is expected to perform reasonably well because it still finds optimal neuron matches for each (now randomly assigned) layer pairing- it only disrupts which layers are matched, not how well neurons align within those matches.
MOT Top-1 Layer Transport Plan (Single-Best OT): Each source layer é maps only to its highest-weight target layer
m m²(() arg max Pem-
converting the soft layer coupling to a hard one-to-one assignment while keeping the soft neuron-level transport.
Independent Pairwise OT (Pairwise Best OT): The standard greedy approach where each source layer is matched to the single target layer with minimum inner OT cost, computed independently on training data. Formally,
m (f) = arg min Cem
and layer & is reconstructed using only (m)- This baseline ignores global structure and can result in multiple source layers matching to the same target layer while leaving others unmatched.
We evaluate rotation-invarian versions of the pairwise OT baseline (henceforth, 'Pairwise Best + R'), where predictions use Y,, Q T RT with orthogonal transformations optimized via Procrustes alignment.
Rotation-aware variants:
Em.
5

---

Published as a conference paper at ICLR 2026
3 RESULTS
We evaluate representational similarity under the MOT metric across four distinct alignment setups: (i) large language models of different families and scales, (ii) fMRI responses from the visual cortex of four human subjects, (iii) pretrained transformer-based vision models spanning different families and scales. (iv) cross-domain comparisons between human visual cortex and vision transformers (Appendix Section BB) Across all settings, MOT matches or surpasses baseline reconstruction (predic- tion) scores. Importantly, the transport plans inferred by MOT naturally reveal systematic layer-wise correspondences across models and cortex, despite not being explicitly optimized for such structure; specifically, earlier layers or regions tend align with earlier counterparts, while deeper layers or regions map to progressively higher levels; patterns that greedy pairwise methods fail to capture. Furthermore, we demonstrate a proof-of-concept extension of MOT to Three-level MOT framework by aligning two networks across their training trajectories.
3.1 REPRESENTATIONAL SIMILARITY BETWEEN LARGE LANGUAGE MODELS
Experimental Setup. We extract layer-wise representations by averaging token activations across 2,552 prompts from the STSB dataset (May) 2021 Enevoldsen et al. 2025 Muennighoff et al. 2022). For each model, this yields a sequence of representation matrices X and Y. corresponding to successive layers. We evaluate models of varying sizes from the LLaMA-3.2 (Grattafiori et al. compared using MOT and baseline metrics, and the resulting transport plans are analyzed. To quantify alignment quality, we reconstruct representations on held-out validation split (20%) using the learned transport maps and report the correlation with ground-truth activations, as detailed in the Methods Section
2024) and Qwen-2.5 (Qwen et al., 2025) families. Representations
Model
Model 2
MOT Metric
Random (Perm-P)
Single-Best OT
Pairwise Best OT
Llama-3.2 IB
Llama-3,2 3B
0.558
0.510
0.502
0.505
Qwen-2.5 0.5B
Qwen-2.5 3B
0,510
0.494
0.467
0.477
Qwen- 2.5 0.5B
Llama-3.2 1B
0.522
0.500
0.502
0,511
Qwen-2.5 0.5B
Llama-3,
0.531
0.513
0.498
0.524
Llama-3. 3.2 IB
Qwen-2.5 3B
0.432
0.411
0.345
0,380
1lama-3.2 3B
Qwen-2.5 3B
0.383
0.374
0.338
0.346
Table LLM alignment performance. Comparison of MOT against baseline metries, evaluated by reconstruction correlation on held-out data.
Results. MOT consistently achieves higher reconstruction accuracy than baseline methods. As shown in Table reconstruction scores on the held-out validation set are significantly higher under MOT, indicating that the alignment plans learns are more robust and generalizable than those obtained from pairwise baselines. This improvement stems from MOT's (i) enforcement of global consistency across all layers and (ii) ability to distribute representational mass across multiple target layers, allowing it to recover alignments that remain hidden pairwise methods when features from single layer are distributed across several layers in another model. Beyond quantitative gains, MOT uncovers clear hierarchical correspondences between models. As illustrated in Figures 2 and B.I the transport plans produced by MOT exhibit strong diagonal structure: early layers in one model align with early layers in the other, while deeper layers align with deeper layers. Such structured correspondences are absent in pairwise OT, which often yields noisy mappings. Furthermore, when comparing shallower with deeper models, MOT reveals that single layers the shallower model distribute their mass across multiple consecutive layers in the deeper model. This soft many-to-man mapping reflects how additional depth refines and spreads computations, suggesting that representational stages compressed into one layer in shallower model are decomposed across multiple processing steps in deeper model- an organizational principle that greedy baselines fail to uncover.
To rule out the possibility that this effect stems from the multi- level formulation and not from OT, also perform pairwise linear predictivity and RSA analyses. As shown in Figures H.I and these methods also fail reveal hierarchial correspondence.

---

Published as a conference paper at ICLR 2026
Muiti-Level
Painwise
Multi-LLvel OT
Pairwise OT
Qwen-2.5 0.5B
Llama-3.2 1B
332
0332
3:13
3.315
19-
3.35
10 13 16 19 22
25
10 13
16
13 21 25
29
33
5
13
17
75
33
Layer
Llama-3.2
Layer
Layer
Qwen-2.5 3B
Layer
Figure 2: Transport plans for LLM alignment. Multi-Level OT (left) versus pairwise OT (right) for cross-model comparisons: (a) Qwen-2.5 0.5B LLaMA-3.2 3B and (b) LLaMA-3.2 1B + 2.5 3B. MOT reveals smooth, diagonal correspondences across layers, while pairwise OT produces noisier and less structured mappings.
3.2 REPRESENTATIONAL SIMILARITY BETWEEN VISUAL CORTEX ACROSS SUBJECTS
Experimental Setup. We analyze fMRI responses from the Natural Scenes Dataset (NSD; (Allen et al. 2022)), which contains recordings from 8 participants who each viewed up to 10,000 natural images. Of these, subjects viewed the full set of 10,000 images three times, with 1,000 images shared across all participants. We focus these 4 subjects and restrict analysis to their responses to the shared 1,000 images, ensuring that alignment is evaluated on common stimuli.
We target visual cortex regions (V1-V4, Lateral, Dorsal, Ventral), treating each region as "layer" and individual voxels as "neurons". The visual cortex provides strong testbed for multi-level alignment because it is one of the best-characterized cortical systems: early areas (V1, V2) are known encode low-level features such as orientation and contrast, while higher areas (V3, V4) encode progressively more complex shapes and object features (Hubel and Wiesel 1968 Pasupathy and Connor |001] Desimone and Schein| 1987 Desimone et al. 1984). This hierarchical progression is well-establisher across individuals, correspondences between homologous regions strongly expected. MOT is applied to align responses across subjects by generating transport maps between regions. To evaluate aligmment quality, we reconstruct held-out responses (20% validation split) and report correlations with ground-truth activity. We repeat this evaluation across 5 random train-validation splits to ensure robustness.
Model 1
Model
MOT Metric
Random (Perm-P)
Single-Bor Best OT
Pairwise Best OT
Subject
Subject B
0.244
0.135
0.244
0.245
Subject
Subject C
0.199
0.110
0.199
0.202
Subject A
Subject D
0.198
0.109
0.198
0.201
Subject
Subject C
0.212
0.126
0.212
0.212
Subject Subject D
0.197
0.112
0.197
0.199
Subject Subject D
0.201
0.121
0.201
0.204
Table 2: Visual cortex alignment performance. Comparison of MOT against baseline methods, evaluated by reconstruction correlation on held-out fMRI responses.
Results. As shown in Tables SallesZane and C.I MOT achieves reconstruction scores comparable pairwise OT, with only a marginal decrease in correlation. More importantly, Figures and C.T show that the transport maps inferred by MOT recover the expected cross-subject correspondences: cortical regions in one subject consistently align to the same regions in another. In contrast, pairwise OT does not produce such structured mappings in any subject pair. This indicates that MOT's global alignment scheme captures region-to-region correspondences that pairwise methods miss. We further assess pairwise linear predictivity and RSA, with the results as shown in Figures H. and [.] The layers selected as best-matching by these approaches fail to align corresponding regions as well, reinforcing that the multi-level alignment mechanism MOT crucial for obtaining robust and generalizable correspondences. Overall, these results demonstrate that MOT yields transport plans that are both interpretable and biologically meaningful, while maintaining reconstruction performance on par with baseline methods.
7

---

Published as a conference paper at ICLR 2026
Muiti-Level
Pairwise
Multi-Level
Pairwise
vi
V2
632
0.3
Subject A
Subject
V1
V2
V3
VI
V1
V2
V3
V2
V3
VI Layer
V1
V7
V3
VI
Layer
Subject
Layer
Subject
Layer
Figure 3: Transport plans for cross-subject brain alignment. Multi-Level OT (left) versus pairwise OT (right) for two randomly selected subject pairs: (a) Subject ++ Subject C and (b) Subject B Subject D. MOT recovers structured region-to-region correspondences that are absent in pairwise OT. Other subject pairs show similar trends (see Appendix C.I
3.3 REPRESENTATION SIMILARITY BETWEEN VISION MODELS
Experimental Setup. We extract wise representations from Vision Transformers by averaging patch activations for each input image. We use 20,000 images randomly sampled from the ImageNet validation set (Deng et al. 2009; Russakovsky et al. 2013). sampled to ensure a uniform coverage across all classes in Imagenet. For each model, this yields sequence of representation matrices that serve as inputs to the MOT framework. We evaluate two families of pretrained vision trans- formers-DINNO22 and ViT-MAE across multiple model scales. Alignment quality quantified by reconstructing representations on a held-out validation split (20%) using the learned transport plans, with reconstruction-grootee ground truth correlation as the metric.
Prior work has shown that the residual stream in Transformers lacks privileged axes and is invariant up to rotations of its basis (Khosla et al. 2024). Since MOT, like other OT-based methods, is Dotation-sensitive, we additionally evaluate rotation-augment
variant (MOT+R) and its baselines (see Methods Section 2.3). We restrict this analysis Vision Transformers, where rotational invariances are especially relevant and where the computational cost of MOT+R remains tractable. For large language models and fMRI data, added optimization over rotations was computationally prohibitive, so we report only the rotation-sensitive results in those domains. In MOT+R, the learned rotation matrices are incorporated into both transport optimization and evaluation, ensuring that geometric cquivalences induced by rotations properly captured.
Model
Model
MOT Metric
Pairwise Best OT
MOT R
Pairwise Best +
DINOv2 Small
VIT-MAE Base
0.289
0.301
0.600
0.526
DINOv2 Small
DINOv2 Large
0.353
0,340
0.778
0.394
DINOv2 Small
DINOv2 Giant
0.466
0.433
0.790
0.418
DINOv2 Small
ViT-MAE Large
0.381
0.354
0.633
0.509
DINOv2 Small
ViT-MAE Huge
0.411
0.386
0.657
0.508
VIT-MAE Basc
DINOv2 Large
0.577
0.624
0.732
0.283
ViT-MAE Base
DINOv2 Giant
0.202
0.180
0.580
0.293
ViT-MAE Basc
ViT-MAE Large
0.588
0.598
0,850
0.596
ViT-MAE Base
VIT-MAE Huge
0.149
0.417
0.788
0.571
VIT-MAE Huge
DINOv2 Giant
0.317
0.352
0.614
0.359
Table Vision model alignment performance Reconstruction accuracy under MOT and pairwise OT, reported both in the standard (rotation-sensitive)) and rotation-augmente (MOT+R) settings.
Results. Table shows that, on its own, vanilla MOT does not consistently outperform pairwise OT in reconstruction accuracy. The transport plans inferred by MOT (Figure 4and Figures D.I. D.10 only partially reveal layer-wise correspondences: in some cases, clear diagonal structure emerges, but in others the mappings are noisier. By contrast, the rotation-augmentee variant (MOT+R) yields substantially higher reconstruction scores surpassing both vanilla MOT and pairwise vanilla and rotational baselines (Tables D.I and D.2. Importantly, the transport plans produced by MOT+R consistently exhibit strong hierarchical correspondences, recovering clean layer-to-layer alignment in settings where vanilla MOT fails to do so. These findings indicate that incorporating rotation
8

---

Published as a conference paper at ICLR 2026
into the OT framework not only improves quantitative alignment quality but also produces more generalizable and interpretable mappings, particularly in domains like Vision Transformers where representations known to be itaationiinvaiiant.
Multi-Laval OT
Pairwise OT
Rotation Multi-Level OT
Rotation Pairwise OT
0033
1.0
3005
VIT-MAE Base Layee
9003
06
9005
VIT-MAE
0063
2005
25.20 37
37 25 29
13
20
Lopas
DINOv2 Giant
Layar
Layer
DINOv2 Giant
Lays
Figure 4: Transport plans for vision model alignment. ViT-MAE Base + DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R). MOT+R captures geometric equivalences induced by rotations, yielding clearer correspondences than the rotation-sensite variant.
3.4 THREE-LEVEI MOT FOR ALIGNING TRAINING TRAIECTORIES
Formulation of Three-Level MOT. We extend the two-level MOT formulation described above to an additional level over checkpoints. Let the first model have checkpoints l,..., , CA and the second d = 1,...,Ca, CB, with checkpoint-c activations x and checkpoint-d activations y
For each checkpoint pair (c, dl, treat x and yr(d) two networks and the two-level MOT procedure as above, yielding neuron- and layer-level couplings together with single scalar cost
Craikgi ;
where MOT(:;) denotes the two-level objective (inner neuron OT followed by outer layer OT). This defines a checkpoint-level cost matrix Cehkppt e RCCxxCe, To obtain a globally consistent alignment between the two training trajectories, we then solve third OT problem checkpoints, reusing the same uniform-marginal formulation as before:
R = arg
min RET(CA.CD)
(Cehkpt ; R),
and Red be interpreted soft assignment between checkpoint i the first model and checkpoint d in the second. For comparison, we also construct a simple pairwise-best checkpoint map that ignores the global OT constraint and matches each checkpoint independently its lowest-cost partner.
Experimental Setup and Results. We perform three-level OT (and the corresponding pairwise baseline) proof of concept for 154 checkpoints of Pythia- 14M models with seed I and seed 2 (Biderman et al. (2023)). where the 154 checkpoints correspond to steps (initialization), 1, 2, 4, 16, 32, 64, 128, 256, 512, 1000, and then every 1,000 subsequent steps. The layer-wise representations for each checkpoint are collected across 2.552 prompts from the STSB dataset (May) 2021 Enevoldsen et al. 2025 Muennighoff et al. 2022). We plot the checkpoint-leve transport maps generated by lhree-level MOT and by the pairwise-ber alignment in Figure MOT reveals coherent checkpoint- wise correspondence between the two training trajectories, whereas pairwise matching fails to do This demonstrates proof-of-coe cooofoof-cnceppp method for aligning entire training trajectories of two models. In principle, the MOT framework can be generalized to any number of levels, depending the type of alignment setting being studied.

---

Published as a conference paper at ICLR 2026
Multi-Level OT
Pairwise OT
235
1.0
0.006
pythia- 1m -sedd
26k
0.005
0.8
transport mass
Training Sepp
53k
0.004
62k
80K
80k
0.002
107k
0.001
116k
123k
134K
143K
256
17k
20k
35K
4nk
59k
624
71k
80k
39k
98k
107.
11.6
1256
1344
256
17k
20k
35k
4nk
534
62k
71k
80k
89k
98k
107
1164
1258
134c
Training Step
*
Training Step
*
pythia-14m-seed2
Figure 5: Transport plans for training-trajectorry alignment. Three-level MOT (left) and pairwise OT (right) for Pythia-14M-seed <> Pythia- 14M-seed2 across training checkpoints. MOT yields an, approximately diagonal checkpoint correspondence, whereas pairwise OT shows no clear trajectory- level alignment.
DISCUSSION
Multi-Level OT provides a principled framework for aligning two networks with arbitrary depths. By softly coupling all layer objectives, MOT (i) allows neuron in one layer of source network to align with soft combination units distributed across multiple layers enforces global consistency that mitigates overfitting noise in any single layer, and (iii) produces single network-level rether rather than layer-level -ere-leee--lllggment score. In our experiments, MOT achieves higher alignment scores than greedy pairwise matches and yields intuitive transport plans that reveal hierarchical correspondences both in artificial networks and in the human brain.
target network, (ii)
Limitations. Despite these strengths, several limitations remain. First, MOT computationally demanding: solving the inner OT scales as On (og n) for neurons in pair of layers, and the overall procedure is (L" log n) for networks with L layers. This makes scaling to very wide or deep models challenging without further algorithmic improvements. Second, our evaluation was limited to a subset of models and brain datasets; follow-up analyses on more models and diverse neural data will be critical to assess generality. Finally, focus here is developing principled method for quantifying representational alignment, rather than explaining why different systems converge similar representations. The latter is separate and substantially harder theoretical question. Recent ideas, such as the contravariance principle (which frames convergence as a consequence of shared task or ecological constraints) (Cao and Yamins 2024) and the Platonic representation hypothesis (which proposes that training on rich naturalistic data drives different systems toward similar underlying structure in the world) (Huh al., 2024), offer intriguing conceptual directions but remain challenging to evaluate empirically. By providing a rigorous way to characterize alignment, MOT offers the measurement foundation needed for such explanatory hypotheses to be tested in future work.
Future directions. Several extensions are natural. One is to study training-run level alignment in depth, examining how model architecture, training data order, learning rate, and other optimization choices shape the resulting alignment plan. Another is to incorporate priors on the transport plan (e.g. smoothness) guide alignment toward more interpretable solutions. Ultimately, as both biological and artificial networks grow in scale and complexity. methods like MOT that respect global structure while revealing network correspondences will be essential for understanding the universal principles governing intelligent systems.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse I. Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 16-126, 2022.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances neural information processing systems, 34:225-201 236, 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2430. PMLR, 2023.
Rosa Cao and Daniel Yamins. Explanatory models in neuroscience, part 2: Functional intelligibility and the contravariance principle. Cognitive Systems Research, 85:101200, 2024.
Zirui Chen and Michael F Bonner. Universal dimensions of visual representation. Science Advances, 11(27)eadw77697 2025.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific reports, 6(1):27755, 2016.
Jesús A De Loera and Edward D Kim. Combinatorics and geometry of transportation polytopes: update. Discrete geometry and algebraic combinatorics, 625:37-76, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. wttp://.R...09.20060000
Robert Desimone and Stanley J Schein. Visual properties of neurons in area v4 of the macaque: sensitivity to stimulus form. Journal of neurophysiology, 57(3)833-868, 1987.
Robert Desimone, Thomas D Albright, Charles G Gross, and Charles Bruce Stimulus-selective properties of inferior temporal neurons in the macaque. Journal of Neuroscience, 4(8):2051-2022. 1984.
Amil Dravid, Yossi Gandelsman, Alexei A Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1934-1943, 2023.
Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184 2017.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, Ömer Çagatan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafal Pošwiata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Bjöm Plüster, Jan Philipp Harries, Loic Magne, Isabelle Mohr, Mariya Hendriksen, Dawei Zhu, Tom Aarsen, Jan Kostkan, Konrad Wojtasik, Taemin Lee, Marek Suppa, Crystina Zhang, Roberta Rocca, Mohammed Hamdy, Andrianos Michail, John Yang. Manuel Faysse, Aleksei Vatolin, Nandan Thakur, Manan Dey, Dipam Vasani, Pranjal Chitale, Simone Tedeschi, Nguyen Tai, Artem Snegirev, Michael Günther, Mengzhou Xia, Weijia Shi, Xing Han Lå, Jordan Clive, Gayatri Krishnakumar, Anna Maksimova, Silvan Wehrli, Maria Tikhonova, Henil Panchal, Aleksandr Abramov, Malte Ostendorff,, Zheng Liu, Simon Clematide, Lester James Miranda, Alena Fenogenova, Guangyu Song, Ruqiya Bin Safi, Wen-Ding Li, Alessia Borghini, Federico Cassano, Hongjin Su, Jimmy Lin, Howard Yen, Lasse Hansen, Sara Hooker, Chenghao Xiao, Vaibhav Adlakha, Orion Weller, Siva Reddy, and Niklas Muennighoff. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv 2025. doi: 10. 10.48550/ar Xix. 2502. 13595. URLhttte /wtxxvivvvv./../////..////..//.. http://wwww../////////////////// 13595
Hippolyte
11

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori et al. The Ilama 3 herd of models, 2024. URL BtLL:ttta///axxi..or/////// 2407.21783
Umut Güçli and Marcel AJ Van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10055-10011 2015.
Eghbal Hosseini, Colton Casto, Noga Zaslavsky, Colin Conwell, Mark Richardson, and Evelina Fedorenko. Universality of representation in biological and artificial neural networks. bioRxiv, 2024.
David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):215-243, 1968.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07900 2024.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haigness and Josh H McDermott, A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals cortical processing hierarchy. Neuron, 98(3):630-644 2018.
Seyed-Mehdi Khaligh- Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLOS Computational Biology, 2014. URL https 7/doi oro.orgg/i/o 0. 1371 journa 1ccbi.1003913
Meenakshi Khosla and Alex H Williams. Soft matching distance: A metric on neural representations that captures single-neuron tuning. In Proceedings of UniReps: the First Workshop Unifying Representations in Neural Models, pages 326-341. PMLR, 2024.
Meenakshi Khosla, Alex H Williams, Josh McDermott, and Nancy Kanwisher. Privileged representa- tional axes in biological and artificial neural networks. bioRxiv, pages 2024-06, 2024.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519- 3529. PMLR, 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249 2008.
Philip May. Machine translated multilinuua sts benchmark dataset. 2021. URL htttt:t//sittttuu com/ Ptsiiipaa///ttsssmmmuiia
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316 2022.
Anitha Pasupathy and Charles Connor. Shape representation in area v4: position-specifie tuning for boundary conformation. Journal of neurophysiology. (6(5):2505-2519 2001.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li. Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL ht https: /ttp:////axivv//// / a 2412.4220 15115
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):2112553 2015. doi: 10.1007/s 1263-015-0811-11
12

---

Published as a conference paper at ICLR 2026
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj. Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Rov Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007,
Martin Schrimpf, Jonas Kubilius, Michael Lee, N Apurva Ratan Murty, Robert Ajemian, and James DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 2020. URL http:///www///a......../////////
(20)
30605-X
Katherine R Storrs, Tim C Kietzmann, Alexander Walther, Johannes Mehrer, and Nikolaus Kriegesko- rtc. Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting, Journal of cognitive neuroscience, 33(10):2044-2000 2064, 2021.
John von Neumann. Certain Zero-sum Two-person Game Equivalent to the Optimal Assignment Problem, pages 5-12. Princeton University Press, Princeton, 1953.
Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances Neural Information Processing Systems, 34:4738 4750, 2021.
Daniel L.K Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance opppimiizeee hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):819-8622, 2014.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A OPTIMAL TRANSPORT FOR REPRESENTATIONAL ALIGNMENT
Optimal Transport (OT) provides prineipled way compare two neural populations by framing each representation as probability distribution over its tuning functions and measuring the minimal "effort" required transform one representational distribution Y E RT'N,, denote two sets of tuning curves across 7 stimuli, we represent each population as uniform mixture of Dirac masses over its neurons:
another. Concretely, X E RT*N, and
N Ng
1
8 OZ
PY=
8
The cost function C₁ = c( Vi) quantifies the dissimilarity between the tuning functions of neuron i in and neuron j in Y: depending on application, c(:.) may squared Euclidean distance, correlation distance, or any appropriate tuning-base dissimilarity. The 2- 2-Wasserstein the soft-matching (Khosla and Williams (2024) distance between pxx and jy is
dr dr(X,Y)
min PET(N..N,))
EPss Cij-
where P ranges over the transportation polytope T(Na,Ny) {p RNEKN, Pig 2
E, J. Any P T(N2, Ny) is doubly stochastic coupling: it assigns a non-negative "mass" Pij to transporting representational weight from neuron in X to neu- ron j in Y, with uniform row and column marginals. Unlike rotation-invarian metries such as RSA or CKA, OT preserves single-neuron tuning structure and yields explicit, interpretable neuron-to-neuron correspondences.
P.
MOT builds directly on this OT foundation by elevating the coupling to the level of layers, simulta- neously inferring soft layer-to-layer couplings and per-neuron transport plans produce globally balanced alignment across networks of arbitrary depths (Figure
14

---

Published as a conference paper at ICLR 2026
B COMPARING LLM-LLM REPRESENTATIONS
Multi-Level OT
Pairwise OT
Multi-Level OT
Pairwise OT
Llama-3.2 10
3302
06
0.58
1001
3003
Lager
2013
2013
300%
10 25
10 22
20
10
29
21
23
32
12
33
Llama-3.2
Qwen-2.5
Pairwise
Multi-Level
Palrwise
Qwen-223 0.58
Qwen-2.5 0.58
2335
1315
040
c
211
Llama:3.2 18
LIama-3.2 3B
Multi-Lavel OT
Pairwise OT
Multi-Level OT
Pairwise OT
1.0
100%
1001
Llame-3.2
3009
06
Llama- 3.2
k13
10
615
25
Loyar
Qwen-2.5 3B
Lapur
Layer
Qwen-2.5
Figure B.1: Transport plans across LLM families and scales. Multi-Level OT (MOT) mappings
shown for six cross-model comparisons: (a) LLaMA-3.2 IB LLaMA-3.2 3B, (b) Qwen-2.5
0.5B ++ Qwen-2.5 3B, (c) Qwen-2.5 0.5B <> LLaMA-3.2 1B, (d) Qwen-2.5 0.5B <> LLaMA-3.2
3B, (e) LLaMA-3.2 1B ++ Qwen-2.5 3B, and (f) LLaMA-3.2 3B <> Qwen-2.5 3B. MOT uncovers structured, near-diagonal correspondences that persist across both intra-family (a,b) and cross-family
(c-f) alignments, illustrating its robustness compared to pairwise OT.
15

---

Published as a conference paper at ICLR 2026
C REPRESENTATION SIMILARITY BETWEEN VISION CORTEX RESPONSE
Model 1
Model
MOT Metric
Random (Perm-P)
Single-Best OT
Pairwise Best OT
Subject A
Subject B Subject C
0.244 + 0.005
0,135 = 0.015
0.244 + 0.005
0.245 + 0.004
Subject A
0.199 + 0.003
0.110 = 0.013
0.199 + 0.003
0.202 + 0.003
Subject
Subject D
0.198 0.008
0.109 0.016
0.198 0.008
0.201 0.007
Subject
Subject C
0.212 0.007
0.126 0.014
0.212 + 0.007
0.212 0.007
Subject
Subject D
0.197
0.007
0.112 = 0.016
0.197 + 0.007
0.199 = 0.006
Subject Subject D
0.201 + 0.008
0.121 = 0.014
0.201 + 0.008
0.204 + 0.007
Table C.I: Visual cortex alignment performance. Comparison of MOT against baseline methods,
evaluated by reconstruction correlation on held-out fMRI responses: mean + standard deviation across seeds.
C.I TRANSPORT PLANS FOR OT BASED MAPPING PATRS
V1
0.13
312
Subject A
0.8
Subject
306
000
304
Subject
Subject C
Layer
Multi-Level OT
Painwise OT
Multi-Level OT
Pairwise OT
Subject A
Eve
Subject
12
5/0
92
V7 V5
91
Layer
Subject
Layer 4
Layer
Subject
Layer
Multi-Level OT
Pairwise OT
Multi-Level OT
Pairwise OT
Subject
Subject
12
V2
V4
V2
V2
V3
14
V1
v4 V3
V1
V2
VS
Subject
Layer
Subject
Figure C.1: Transport plans for OT-based mapping pairs. Pairwise OT mappings across all subject pairs: (a) Subject A Subject B, (b) Subject A ++ Subject C, (c) Subject A ++ Subject D, (d) Subject ++ Subject C, (e) Subject ++ Subject D, and (f) Subject C Subject MOT recovers structured
region-to-region correspondences that are absent in pairwise OT.
16

---

Published as a conference paper at ICLR 2026
D REPRESENTATION SIMILARITY BETWEEN VISION MODELS
Model
Model
MOT Metric
Random (Perm-P)
Single-Bar Best OT
Pairwise Best OT
DINOv2 Small
VIT-MAE Base
0.289
0.259
0.289
0.301
DINOv2 Small
DINOv2 Large
0.353
0.298
0.312
0.340
DINOv2 Small
DINOv2 Giant
0,466
0.385
0.413
0.433
DINOv2 Small
ViT-MAE Large
0.381
0.348
0.350
0.354
DINOv2 Small
VIT-MAE Huge
0.411
0.372
0.359
0.386
ViT-MAE Base
DINOv2 Large
0.377
0.394
0.531
0.624
ViT-MAE Base
DINOv2 Giant
0.202
0.197
0.148
0.180
ViT-MAE Base
VIT-MAE Large
0.588
0.528
0.539
0.598
ViT-MAE Base
VIT-MAE Huge
0.149
0.116
0.181
0.417
ViT-MAE Huge
DINOv2 Giant
0.317
0.261
0.293
0.352
Table D.I: MOT Metric with its baseline comparisons.
Model 1
Model 2
MOT R
Single-Best + R
Pairwise Best + R
DINOv2 Small
VIT-MAE Base
0.600
0.600
0.526
DINOv2 Small
DINOv2 Large
0.778
0.708
0.394
DINOv2 Small
DINOv2 Giant
0.790
0.746
0.418
DINOv2 Small
VIT-MAE Large
0.633
0.599
0.509
DINOv2 Small
VIT-MAE Huge
0.657
0.614
0.508
ViT-MAE Base
DINOv2 Large
0.732
0.712
0.283
ViT-MAE Base
DINOv2 Giant
0,580
0.605
0.293
VIT-MAE Base
VIT-MAE Large
0.850
0.848
0.596
VIT-MAE Base
VIT-MAE Huge
0.788
0.760
0,571
VIT-MAE Huge
DINOv2 Giant
0.614
0.582
0.359
Table D.2: Rotation (MOT + R) with its baseline comparisons.
Multi-Level OT
Pairvise
Rotation Multi-Level OT
Rotation Pairwise
DINOY2 Small
Layte
06
DINOVZ
664
11
11
VIT-MAE Base
Layer
Layer
VIT-MAE Base
Layer
Figure D.1: Transport plans for vision model alignment. DINOv2 Small ++ VIT-MAE Base (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
DINOv2 Small
06
622
DINOvz
657
6.21
+221
10
10
22
10 13 26 Layer
19 22
13
16 10 22
20 33 30
Large
Layer
Large
Layer
Figure D.2: Transport plans for vision model alignment. DINOv2 Small + DINOY2 Large (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
17

---

Published as a conference paper at ICLR 2026
Multi-Level
Pairwise
Rotation Lulli-Level
Rotation Painwise
5093
3003
DIHOV2 Small Layte
DINOVZ Small
1073
Layer #
DINOV2 Giant
Layer
Layer
DINOV2 Giant
Layer *
Figure D.3: Transport plans for vision model alignment. DINOv2 Small <> DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level OT
Pairwise
Rotation + Multi-Level OT
Rotation Pairwise OT
DINOv2 Small Laper
DINOvZ Small
6:11
644
10
13
10
22
10
19
22
11
10
13
10
19
Layer
VIT-MAE Large
1ayer
Figure D.4: Transport plans for vision model alignment. DINOv2 Small + ViT-MAE Large (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level
Pairwise
Multi-Level
Rotation Pairwise
DINOV2 Small
DINOVZ Small
06
Liger
2.03
2013
3.05
3.05
Layer
VIT-MAE Huge
laytr
Layer
VIT-MAE Huge
Layer 7
Figure D.5: Transport plans for vision model alignment. DINOv2 Small < VIT-MAE Huge (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
VIT-MAE Base Layee
06
VIT-MAE
657
6:21
1021
10
12
10
22
10
26 19 22 Layer
13
16
10
22
20 33
30 Layer
19
Large
Layer
Large
Figure D.6: Transport plans for vision model alignment. ViT-MAE Base ++ DINOv2 Large (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
18

---

Published as a conference paper at ICLR 2026
Multi-Level
Pairwise
Rotation Multi-Level
Rotation Painwise
5093
300
VIT-MAE Base Laysr
3033
VIT-MAE
Layer
DINOV2 Giant
Layer
Layer
DINOVZ Giant
Layer *
Figure D.7: Transport plans for vision model alignment. VIT-MAE Base ++ DINOv2 Giant (a)
without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
VIT-MAE Base LAye
VIT-MAE
6277
6.01
622
10
72
10
16
19
77
13
16
16 22
30 13
16 77
Jaysr
Large
Layar
Laper
Large
Layer
Figure D.8: Transport plans for vision model aligmment. ViT-MAE Base + ViT-MAE Large (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level
Pairwise
Rotation Multi-Level
Rotation Pairwise
3003
1033
VITMAE Base
3005
06
2005
Layee
3003
VIT-MAE
13
VIT-MAE
VIT-MAE Huge
Figure D.9: Transport plans for vision model alignment. VIT-MAE Base ++ ViT-MAE Huge (a)
without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Multi-Level
Pairwise
Rotation Multi-Level
Rotation Painwise
VIT-MAE Huge
5005
2 2: LA
3055
VIT-MAE
5005
3.05
Layer
DINOv2 Giant
DINOv2 Giant
Layer 9
Figure 1.10: Transport plans for vision model alignment. ViT-MAE Huge DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
19

---

Published as a conference paper at ICLR 2026
E REPRESENTATION SIMILARITY BETWEEN VISION MODELS AND VISION CORTEX
Experimental Setup. We assess representational similarity between human visual cortex and vision transformers by comparing IMRI responses with model activations elicited by the same set of stimuli.
Specifically, we use responses from the Natural Scenes Dataset ((Allen al. 2022 focusing on 1,000 shared images viewed by participants in the fMRI experiment. The same images are presented pretrained vision transformers, and layer-wise representations are extracted by averaging patch embeddings across each input. Following the approach used in the cortex-corte analysis, we treat distinct visual areas (VI-V4) as "layers" and individual voxels as "neurons." We then compute both MOT and its volation-augmenter variant (MOT+R) to align cortical responses with model representations. This design enables a direct comparison of hierarchical organization across biological and artificial systems under matched visual input.
Model
Model 2
MOT Metric
Pairwise Best OT
MOT + R
Pairwise
Subject
DINOv2 Base
0.092
0.084
0.145
0.094
Subject
DINOv2 Giant
0.090
0.071
0.163
0.110
Subject
VIT-MAE Base
0.127
0.114
0,151
0.121
Subject
VIT-MAE Huge
0.072
0.099
0.154
0.122
Table E.1: Results on MOT metric vs. baselines (test split).
Results. Table E.1 reports reconstruction scores for MOT, MOT+R, and corresponding baselines averaged across the four subjects. Among all methods, MOT+R achieves the highest reconstruction score, indicating that incorporating rotation critical for capturing shared structure hetween cortical and model representations. Subject-specific results (Tables E.2 and E.3 further confirm this trend, with MOT+R consistently outperforming both vanilla MOT and pairwise baselines. Transport plans visualized in Figures 1 16 reveal partial but inconsistent layer-wise correspondences: some cases, early cortical regions align with early model layers and higher regions map to deeper layers, while others the mappings appear noisier.
Model
Model
MOT Metric
Random (Perm-P)
Single-Best OT
Pairwise Best OT
Subject A
DINOv2 Giant
0.085
0.087
0.046
0.063
Subject A
DINOv2 Base
0.081
0.072
0.075
0.080
Subject A
VIT-MAE Huge
0.065
0.057
0.063
0.105
Subject A
VIT-MAE Base
0.141
0.131
0.123
0.124
Subject B
DINOv2 Giant
0.087
0.089
0.054
0.065
Subject B
DINOv2 Base
0.088
0.080
0.075
0.082
Subject B
ViT-MAE Huge
0.077
0.072
0.073
0.093
Subject B
VIT-MAE Base
0.115
0.109
0.101
0.104
Subject C
DINOv2 Giant
0.115
0.124
0.072
0.092
Subject C
DINOv2 Base
0.115
0.108
0.095
0.100
Subject C
ViT-MAE Huge
0.092
0.088
0.082
0.114
Subject C
VIT-MAE Base
0.137
0.134
0.121
0.124
Subject D
DINOv2 Giant.
0.073
0.082
0.049
0.063
Subject D
DINOv2 Base
0.083
0.078
0.072
0.072
Subject D
VIT-MAF Huge
0.054
0.049
0.046
0.086
Subject D
ViT-MAE Base
0.114
0.111
0.101
0.106
Table E.2:
Results on MOT metrie baselines (test split).
20

---

Published as a conference paper at ICLR 2026
Model
Model 2
MOT + Single-Best
Pairwise Best
Subject DINOv2 Giant
0.161
0.140
0.099
Subject DINOv2 Base
0.146
0.140
0.083
Subject VIT-MAE Huge
0.165
0.151
0.127
Subject A
VIT-MAE Basc
0.157
0.151
0.124
Subject DINOv2 Giant
0.147
0.132
0.096
Subject B
DINOv2 Base
0.127
0.120
0.084
Subject
VIT-MAL Huge
0.125
0.109
0.104
Subject
ViT-MAF Basc
0.128
0.118
0.101
Subject C
DINOv2 Giant
0.205
0.183
0.147
Subject
DINOv2
0.185
0,181
0.128
Subject
VIT-MAE Huge
0.189
0.179
0.151
Subject C
ViT-MAE Base
0.186
0.180
0.152
Subject D
DINOv2 Giant
0.139
0.123
0.099
Subject D
DINOv2 Base
0.121
0.114
0.081
Subject D
VIT-MAE Huge
0.135
0.122
0.107
Subject
ViT-MAE Basc
0.134
0.129
0.109
Table E.3: Results on rotational MOT metric vs. baselines (test split).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
Subject La3
Subject
6622
11
11
Layar
Layar
Layar
Lager
Figure E.1: Subject < DINOv2 Base (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
VI
3009
3008
Subject
Subject
2013
VI
VL
Layer
DINOv2 Giant
DINOV2 Giant
Figure E.2: Subject A DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmenta- tion (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
00
Subject
V2
Subject
4227
+027
2
11
11
11
Layer
ViT-MAE
Laye
Layer
Laye
Figure E.3: Subject A ++ ViT-MAE Base (a) without rotation (MOT) and (b) with rotation augmenta- tion (MOT+R).
21

---

Published as a conference paper at ICLR 2026
Multi-Level
Pairwise
Rotation Multi-Level
Rotation Painwise
Subject A 201
Subject
1315
2011
1316
Layer
VIT-MAE Huge
Layer
Layer
VIT-MAE Huge
Layer #
Figure E.4: Subject A < ViT-MAE Huge (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
Multi-Leval OT
Pairwise
Rotation Multi-Level OT
Rotation Pairwise OT
Subject Loyer
Subject
11
11
Layer
Layer
Figure E.5: Subject ++ DINOv2 Base (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Pairwise
Multi-Level
Rotation Pairwise
V2
V2
Subject 8
300
06
Subjject
Y4-
V4-
Layer
DINOv2 Giant
Layer
Layer
DINOV2 Giant
Layer 9
Figure E.6: Subject B ++ DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmenta- tion (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
06
V2
Subject
Subject
4227
11
11
11
Layer
Laye
Layer
Laye
Figure E.7: Subject B + ViT-MAE Base (a) without rotation (MOT) and (b) with rotation augmenta- tion (MOT+R).
22

---

Published as a conference paper at ICLR 2026
Multi-Level
Pairwise
Rotation Multi-Level
Rotation Painwise
Subject 8 201
Subject
1913
1315
1011
1311
Layer
VIT-MAE Huge
Layer
Layer
VIT-MAE Huge
Layer
Figure E.8: Subject B 4+ VIT-MAE Huge (a) without rotation (MOT) and (b) with rotation augmenta- tion (MOT+R).
Multi-Leval OT
Pairwise
Rotation + Multi-Level OT
Rotation Pairwise OT
Subject Layer
Subject
012
11
11
Layer
Figure E.9: Subject C+ DINOv2 Base (a) without rotation (MOT) and (b) with rotation augmentation (MOT+R).
Pairwise
Rotation Pairwise
V2
V2
Subject
300
06
Subjject
Y4
V4-
Layer
DINOv2 Giant
Layer
Layer
DINOV2 Giant
Layer 9
Figure 10 Subject C ++ DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
06
Subject
V2
Subject
4227
2
11
11
11
Layer
Laye
Layer
Laye
Figure E.11: Subject C + ViT-MAE Base (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
23

---

Published as a conference paper at ICLR 2026
Multi-Level
Pairwise
Rotation Multi-Level
Rotation Painwise
Subject C 201
Subject
1913
1315
1011
1316
Layer
VIT-MAE Huge
Layer
Layer
VIT-MAE Huge
Layer #
Figure E.12: Subject C ++ VIT-MAE Huge (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
Multi-Leval OT
Pairwise
Rotation + Multi-Level OT
Rotation Pairwise OT
X2
Subject
Subject
012
604
11
11
Layer 11
Layer
Layer
Figure E.13: Subject + DINOv2 Base (a) without rotation (MOT) and (b) with rotation augmenta- tion (MOT+R).
Pairwise
Multi-Lavel
Rotation Pairwise
V2
V2
Subject D
300
06
Subject
101
Y4-
V4-
Layer 9
DINOv2 Giant
Layer
Layer
DINOV2 Giant
Layer 9
Figure 1. Subject D DINOv2 Giant (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
Multi-Level OT
Pairwise OT
Rotation + Multi-Level OT
Rotation Pairwise OT
06
Subject
Subject
4027
2
11
11
11
Layer
Laye
Layer
Laye
Figure E.15: Subject D + VIT-MAE Base (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
24

---

Published as a conference paper at ICLR 2026
Multi-Level
Pairwise
Rotation Muiti-Level
Rotation Painwise
Subject D Say
Subject
1315
1011
1311
V4
Layer
VIT-MAE Huge
Layer
Layer
VIT-MAE Huge
Layer *
Figure F.16: Subject D <> ViT-MAE Huge (a) without rotation (MOT) and (b) with rotation augmen- tation (MOT+R).
F GLOBAL ALIGNMENT
In this section, we report reconstruction scores on a held-out validation dataset for the vision-cortex aligmment study under global alignment formulation. In this setting, all layers from both networks are flattened so that all neurons are treated as belonging to a single layer. We compare the resulting reconstruction scores to those obtained with our MOT reconstruction scores.
Global alignment scores are computed using two alignment metrics: optimal transport and linear predictivity. Reconstruction performance is then evaluated on held-out validation data. The results for this experiment are presented in Table
Model
Model 2
Global OT
Global Linear Predictivity
MOT
Subject A
Subject
0.184
0.305
0.244
Subject A
Subject
0.177
0.295
0.199
Subject A
Subject
0.168
0.289
0.198
Subject B
Subject
0.183
0.306
0,212
Subject C
Subject D
0.158
0,306
0.197
Subject B
Subject D 0.163
0.297
0.201
Table F.1: Comparison global OT, Linear Predictivity, and MOT, evaluated by reconstruction correlation on held-out fMRI responses.
G QUANTIFYING HIERARCHY
Given layer-to- layer transport plan P R we define scalar "hierarchy" score that quantifies how strongly the transport mass is concentrated along a diagonal correspondence between source and target layers.
We first embed the layer indices into the unit interval via
1-1 It 1-1'
1
Cm
(= L, m I...., M,
M
so that the ideal diagonal correspondence between layers lies along the line r = c. For each entry (E, m) we define its normalized distance to this diagonal
dim
rt Cm [0.
We interpret the magnitudes of the transport plan discrete mass distribution,
|Ptml
Wim
8ém
Ee mt
St'm
and compute the average distance of this mass from the diagonal,
D(P)
Σ
Ween dem-
m=1
25

---

Published as a conference paper at ICLR 2026
Finally, we define the hierarchy
= H(P)=1-D(P))
By construction HP) E 00, and it depends only on the relative placement of transport mass with respect to the diagonal. Table G.1 reports H (P) for each transport plan as a summary measure of its hierarchy.
Model
Model 2
MOT
Pairwise OT
MOT+R
Pairwise Best R
DINOv2 Small
ViT-MAE Large
0.90
0.84
0.89
0.50
ViT-MAE Base
ViT-MAE Large
0.97
0.61
0.93
0.50
DINOv2 Small
ViT-MAE Base
1.00
0.50
VIT-MAE Base
DINOv2 Large
0.96
0.70
0.90
0.50
DINOv2 Small
DINOv2 Large
0.98
0.88
0.94
0.50
VIT-MAE Huge
DINOv2 Giant
0.86
0.62
0.86
0.50
VIT-MAE Base
DINOv2 Giant
0.60
0.55
0.91
0.50
DINOv2 Small
DINOv2 Giant
0.97
0.95
0.50
VIT-MAE Base
ViT-MAE Huge
0.68
0.50
0.83
0.50
DINOv2 Small
VIT-MAF Huge
0.82
0.72
0.88
0.50
Llama-3.2
Llama-3.2 3B
0.96
0.93
Qwen-2 2.5 0.5B
Llama-3.2 3B
0.86
0.63
Llama-3.2 3.2 3B
Qwen-2.5 3B
0.83
0.80
Llama-3.2 IB
Qwen-2.5 3B
0.85
0.59
Qwen-2.5 0.5B
Llama-3.2 IB
0.86
0.67
Qwen-2.5 0.5B
Qwen-2.5 3B
0.93
0.60
Subject
Subject D
1.00
0.93
Subject
Subject D
1.00
0.98
Subject
Subject C
1.00
0.93
Subject A
Subject B
1.00
0.98
Subject
Subject C
1.00
0.98
Subject
Subject D
1.00
0.98
Table G.1: MOT and pairwise (rotation where applicable) hierarchy scores for vision, language, and [MRI alignment.
We observe that, across all transport plans, the hierarchy metric highest for plans generated by MOT (or MOT+R), compared with those produced by pairwise best (or pairwise rotational) methods. This indicates that MOT and MOT+R based approaches uncover hierarchical correspondences that pairwise methods don't reveal.
H TRANSPORT PLANS FOR LINEAR PREDICTIVITY BASED MAPPING PAIRS
Model
Model
Pairwise LP Reconstmuction score
MOT Metric
Llama-3.2
Llama-3.2
0.737
0.558
Qwen-2.5 0.5B
Llama-3.2 3B
0.694
0.531
Llama-3.2 3B
Qwen-2.5 3B
0.784
0.383
Llama-3.2 1B
Qwen-2.5 3B
0.841
0.432
Qwen-2.5 0.5B
Llama-3.2 IB
0.848
0.522
Qwen-2.5 0.5B
Qwen-2.5 3B
0.848
0.510
Subject B
Subject D
0.210
0.201
Subject C
Subject D
0.230
0.197
Subject
Subject
0.199
0.212
Subject A
Subject B
0.238
0.244
Subject A
Subject C
0.179
0.199
Subject A
Subject D
0.206
0.198
Table H.l: Pairwise Linear Predictivity and MOT reconstruction scores on test splits. Top: LLM model-pair alignment. Bottom: fMRI subject-pair alignment.
26

---

Published as a conference paper at ICLR 2026
Pairwise Linear Predictivity
Pairwise Linear Predictivity
Pairwise Linear Predictivity
14
1.0
14
Liama-3.2 18
Qwen-2.5 0.58
p13
04
0.4
Llama
9.4
16
19
13
19
22
15
22
10 13 16 19
10 13
13
17
21
25
29
33
Layer 4
Layer +
Layer
Llama-3.2
Llama-3.2
Qwen-2.5
Pairwise Linear Predictivity
Pairwise Linear Predictivity
Pairwise Linear Predictivity
Llama-3.2 18
2.8
5.8
Qwen-2.5 0.5B
04
Layer
Qwen-2.5
213
5.4
16-
0.2
19-
22
22
17 21
25
29
33
13
17
21
25
29
33
Layer #
Layer
Layar #
Qwen-2.5 3B
Llama-3.2 18
Qwen-2.5 3B
Pairwise Linear Predictivity
Pairwise Linear Predictivity
Pairwise Linear Predictivity
Subject A
3
Subject
Subject
06
84
02
02
Layer
Layer
Subject
Subject
Subject
Pairwise Linear Predictivity
Pairwise Linear Predictivity
Pairwise Linear Predictivity
08
08
05
Subject B
Subject
Subject
2.4
5-
2.4
04
2
3
4
5
6
Layer
Layer #
Layer
Subject
Subject
Subject
Figure H.1: Transport plans for linear predictivity-based mappings in LLM and fMRI spaces.
Top two rows: pairwise mappings learned under linear predictivity constraints for all LLM pairs: (a)
1lama-3.2 1B ++ Llama-3.2 3B, (b) Qwen-2.5 0.5B ++ Llama-3.2 3B, (c) Llama-3.2 3B <> Qwen-2.5
3B, (d) Llama-3.2 1B Qwen-2.5 3B, (e) Qwen-2.5 0.5B <>> Llama-3.2 1B, and (f) Qwen-2.5 0.5B
++ Qwen-2.5 3B. Bottom two rows: pairwise mappings learned under lincar predictivity constraints
all IMRI subject pairs: (g) Subject A ++ Subject B, (h) Subject ++ Subject C, (i) Subject
++ Subject D, (i) Subject B ++ Subject C, (k) Subject B ++ Subject D, and (1) Subject C + Subject D. In both the LLM and fMRI settings, linear predictivity-based pairwise mappings do not recover structured layer-wo layer-wise correspondences, highlighting the importance of MOT for learning robust and generalizable mappings across architectures, model scales, and subjects.
27

---

Published as a conference paper at ICLR 2026
1
TRANSPORT PLANS FOR RSA-BASER MAPPING PAIRS
Pairwise RSA
Pairwise RSA
Pairwise RSA
Llama-3.2 18
0.0
Qwen-2.8 0.5B
0.
Llama 3..33.2 3B
0.6
g13
0.6
1.3-
316
16
13
02
19
22
15
22
7
10 13
19
22 25
7
10
13
19 22 25
13 17
21
25
29 33
Layer
Layer
Layer
Llama-3.2
Llama-3.2
Qwen-2.5
Pairwise RSA
Pairwise RSA
Pairwise RSA
1.0
'
Llama-3.2 18
Qwen-2.5 0.5B
Qwen-2.5
213
19-
22
13
21 25
15
13
17 21 25
Layer #
Layer
Layer #
Qwen-2.5 3B
Llama-3.2 1B
Qwen-2.5 3B
Pairwise RSA
Pairwise RSA
Painwise RSA
1.0
Subject A
Subject A
0.6
Subject
0.6
24
Laver
0.4
Subject
Subject
Subject
Pairwise
Painwise
Painwise
1.0
1.0
1.0
0.
2-
0.8
Subject B
Subject C
Subject
12yer
02
02
5
6
2
5
6
Layer
Layer
Layer
Subject
Subject D
Subject
Figure Ll: Transport plans for RSA-based mappings in LLM and IMRI spaces. Top two
rows: pairwise mappings learned under RSA-based constraints for all LLM pairs: (a) Llama-3.2
1B ++ Llama-3.2 3B, (b) Qwen-2.5 0.5B <> Llama-3.2 3B, (c) Llama-3.2 3B + Qwen-2.5 3B, (d) Llama-3.2 1B Qwen-2.5 3B, (e) Qwen-2.5 0.5B ++ Llama-3.2 1B, and (f) Qwen-2.5 0.5B <
Qwen-2.5 3B. Bottom two rows: pairwise mappings learned under RSA-based constraints for all (MRI subject pairs: (g) Subject A < Subject B, (h) Subject ++ Subject C. (i) Subject A <> Subject D. ()) Subject B Subject C, (k) Subject B ++ Subject D. and (1) Subject C < Subject D. In both the LLM and fMRI settings, RSA-based pairwise mappings do not recover structured layer-wise correspondences. highlighting the importance of MOT for learning robust and generalizable mappings across architectures, model scales, and subjects.
28

---

Published as a conference paper at ICLR 2026
J ROBUSTNESS OF METRICS TO SUB-SA SAMPLING
We consider pair of LLMs (Qwen-2.5 0.5B and Llama-3.2 1B) and systematically sub-sample neurons from the larger model (Llama3.2 IB) to study how this affects reconstruction of the smaller model's ' activations. For each fraction of neurons kept, we randomly select that fraction of neurons from Llama-3.2 IB, re-fit the transport plans between the two models, and evaluate the reconstruction score on held-out test data (a 20% split, as before), repeating this procedure over five different seeds. Figure J.T shows the resulting reconstruction scores for MOT and MOT+R (with their corresponding pairwise baselines).
0.7
Test mean correlation
0.6
0.5
0.4
MOT
Pairwise Rotation MOT Rotation + Pairwise OT
0.3
0.00
0.25
0.50
0.75
1.00
Fraction of source neurons kept
Figure J.1: Neuron sub-sampling in LLM alignment. Reconstruction performance when aligning Qwen-2.5 0,5B to Llama-3.2 IB as a function of the fraction Llama neurons kept. The curves show reconstruction scores for MOT and MOT+R (with their corresponding pairwise baselines), averaged over five different seeds.
We observe that reconstruction scores for both MOT/MOT+R and the corresponding baselines increase with fraction neurons kept. Importantly, sub-sampling does not cause MOT to collapse; its performance degrades smoothly. For MOT+R, keeping only 10% of neurons yields reconstruction score of nearly 0.5, indicating substantial robustness to aggressive sub-sampling. the upper range of fraction of neurons kept, MOT+R achieves substantially higher reconstruction scores than both its rotation- aaarre baseline and the corresponding pairwise OT and MOT scores. These results suggest that neuron sub- sub-sampling is practical way trade off computation and accuracy, and that the metrics remain informative even under significant sub-sampling.
K USE OF LARGE LANGUAGE MODELS
LLMs were used in this work to assist with writing tasks, specifically for ensuring grammatical correctness and enhancing the clarity of the text.
29


Reviews:
[{"review_id": "awwUvzpVM8", "reviewer": "Reviewer_1X33", "summary": "The paper proposes Hierarchical Optimal Transport (HOT), a unified framework for aligning representations across neural networks or between brains and models. Unlike standard pairwise layer-matching methods that treat each layer independently, HOT jointly infers globally consistent layer-to-layer and neuron-to-neuron transport plans under marginal constraints. This allows the transport “mass” to be distributed across multiple target layers while preserving global balance, yielding a single, symmetric alignment score for the entire hierarchy.\nThe authors apply HOT to compare representations across large language models, vision transformers, and human visual cortex recordings. They evaluate alignment using two criteria: (1) the average correlation between reconstructed and ground-truth responses, and (2) whether the inferred transport plans preserve hierarchical correspondences, such that layers at similar depths in the source and target systems align with each other", "questions": "1.\tOn methodological necessity:\n The paper argues that layer-wise alignment is too rigid and introduces a hierarchical coupling (HOT) that distributes representational mass across layers. However, if meaningful correspondences genuinely span multiple layers, why impose any hierarchical structure at all? Wouldn’t a global all-to-all alignment—such as linear predictivity or optimal transport across all neurons, irrespective of layer boundaries—capture these distributed relationships more directly? Could the authors clarify why the hierarchical formulation is necessary or advantageous compared to such an unconstrained alternative? At present, the hierarchical constraint seems imposed for interpretability rather than empirically justified. Additionally, might the layer-level mass-balancing constraint bias the model toward producing smooth, diagonal correspondences that appear hierarchical but could be artifactual? A comparison with a fully unconstrained baseline would help clarify whether HOT provides genuine advantages beyond aesthetic interpretability.\n\n\n2.\tOn missing baselines and metrics:\n Why were standard representational alignment baselines such as RSA, CKA, Procrustes distance, and layer-wise linear predictivity not included in the evaluation? These are commonly used and directly comparable to the proposed method. Similarly, why were fully global baselines (e.g., all-to-all OT or global linear mapping) not tested, given that they would directly address whether HOT’s hierarchical constraint provides empirical benefits?\n\n\n3.\tOn model and domain coverage:\n Why not also test HOT on convolutional neural networks (CNNs)? CNNs have known architectural inductive biases, and prior work (e.g., Yamins et al., 2014, PNAS) has shown that their layer activations align well with the ventral visual stream hierarchy. Including CNNs would clarify whether HOT can recover known brain-model hierarchical correspondences, not just those in transformers.\n4.\tOn the biological data scope:\nWhy was the inferotemporal (IT) cortex excluded from the visual cortex analyses? IT is an established component of the ventral visual stream, and CNN layer–IT alignment using linear predictivity is well documented. Including IT data could reveal whether HOT captures higher-level cortical correspondences and provide a more complete test of its claims about hierarchical alignment.\n\n5.\tOn representational choices:\nFor the transformer models, token activations were averaged to form layer-wise representations. Could the authors clarify what representational information this averaging preserves, and whether concatenating token embeddings or using other aggregation strategies changes the inferred alignment patterns?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important and timely problem in representational alignment—how to compare internal representations across networks and between brains and models when architectures differ in depth or structure. The proposed Hierarchical Optimal Transport (HOT) framework is an original and elegant reformulation of this problem, combining optimal transport with a hierarchical layer coupling that enforces global consistency. This formulation provides a principled way to compute both neuron-level and layer-level correspondences simultaneously, which is conceptually innovative compared to standard pairwise matching approaches.\nThe paper is generally well-written and situates the work in a broad and relevant context, bridging literature from both neuroscience and machine learning. The authors’ application of HOT across diverse domains—large language models, vision transformers, and human fMRI data—demonstrates methodological versatility and cross-domain relevance, which is a significant strength. The idea of producing a single, symmetric alignment score and a transport plan that naturally handles networks of different depths is both creative and potentially useful for future comparative studies of representation learning.", "weaknesses": "On Soundness: \nThe central claim of the paper is that Hierarchical Optimal Transport (HOT) provides a superior measure of representational alignment, with alignment “quality” quantified by the correlation between reconstructed and ground-truth neural responses. However, the empirical evidence does not convincingly support this claim. Across the three experimental domains, improvements are limited to the large language model comparisons, while results on human visual cortex and vision model alignments show marginal or no advantage. In fact, for vision models (Table 3), HOT underperforms the Pairwise Best OT baseline, and only the rotation-augmented version (HOT + R) shows improvement—suggesting that the observed gains stem primarily from the rotation component rather than the hierarchical formulation itself. Furthermore, critical baselines such as RSA, CKA, and linear predictivity are missing, and no measures of variance or statistical error are reported in Tables 1–3. As a result, the empirical support for the claimed superiority of HOT remains weak and incomplete.\n\nOn Presentation: \nPresentation and clarity could be improved. A short introductory paragraph or visual summary explaining optimal transport theory in the context of representational alignment would also help orient readers unfamiliar with OT.\nFigures and tables could be clearer and more polished. Axis ticks and labels are sometimes difficult to read. In Figure 2, the axis dimensions should be unified and limits matched across panels to make visual comparison meaningful; the overall aesthetics of the figure could also be improved. The tables listing model or subject-pair reconstruction scores might be better visualized as heatmaps, allowing all pairwise results to be compared at a glance. Additionally, in some heatmaps, it is unclear why certain axis labels are highlighted.\nIn the Results section for Experiment 3, the statement that the lack of a diagonal structure in transport plans is “consistent with” lower reconstruction scores is not well justified—this seems inconsistent with Experiment 2, where low reconstruction scores still coincided with clear hierarchical (diagonal) alignment patterns. Finally, the claims that HOT provides more generalizable and interpretable results are not clearly substantiated in the text: the paper does not explain what “generalizable” means in this context, and interpretability is equated only with hierarchical structure. However, whether layers at similar depths should align is itself an open empirical question, not an inherent criterion for interpretability.\n\nOn Contribution: \nWhile the idea of combining hierarchical structure with optimal transport is interesting, the paper’s contribution is primarily incremental in practice. The empirical analyses do not convincingly demonstrate substantial improvements over existing OT-based methods, and the hierarchical formulation seems more interpretive than functionally necessary. Several important baselines (e.g., RSA, CKA, linear predictivity) and control analyses are missing, making it difficult to assess the true scope of the contribution. Moreover, claims about improved interpretability and generalizability remain qualitative and are not rigorously supported.\n\nExpanded comment: \nThe empirical results do not convincingly support the central claims. The paper argues that layer-wise alignment is too rigid and proposes a hierarchical coupling (HOT) that distributes representational mass across layers. However, if meaningful correspondences truly span multiple layers, it is unclear why one should impose any hierarchical structure at all. A global all-to-all mapping—such as linear predictivity or optimal transport across all neurons, regardless of layer boundaries—could directly capture these distributed relationships without the additional constraint.\n\nThe authors do not justify why the hierarchical formulation is preferable or demonstrate that it yields empirical benefits beyond interpretability. In its current form, the hierarchical constraint seems imposed rather than validated, and the layer-level mass-balancing constraint may even bias the method toward producing smooth, diagonal correspondences that are visually appealing but potentially artifactual. A comparison with a fully unconstrained baseline (e.g., all-to-all linear mapping or global OT) would be critical to determine whether HOT provides genuine improvements in alignment accuracy or interpretability.\nAdditionally, incorporating statistical measures of uncertainty and comparing against established baselines such as RSA, CKA, and layer-wise linear predictivity would substantially strengthen the empirical case. Other recent OT-based methods such as Soft Matching Distance (Khosla & Williams, 2024) or model stitching (Bansal et al., 2021) are directly relevant and cited; including these would clarify whether the hierarchical layer coupling truly adds value over existing OT approaches.\n\nFinally, there is ambiguity in the interpretation of results: in Experiment 3 (vision models), the authors interpret the lack of diagonal structure in transport maps as consistent with low reconstruction scores, but in Experiment 2 (visual cortex), low reconstruction scores still yielded diagonal correspondences. This inconsistency weakens the interpretation of what “hierarchical structure” signifies. A clearer, quantitative definition of hierarchy (e.g., correlation between layer indices or mutual information across depth) would improve interpretability. Moreover, the scalability of HOT remains untested in practice: although the paper notes a computational complexity of O(L2n3log⁡n), it would benefit from empirical runtime or memory comparisons with pairwise OT and linear predictivity to demonstrate its practical feasibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ybZotlimsa", "reviewer": "Reviewer_xsQG", "summary": "The paper introduces Hierarchical Optimal Transport for representational alignment. Instead of matching each source layer to one best target layer in isolation, HOT matches neurons inside each candidate layer pair and, on top of that, a soft layer-to-layer coupling that must be globally consistent. This gives a single network-level alignment score, handles depth mismatches, and, the authors argue, reveals smooth hierarchical correspondences that greedy pairwise methods miss. The method also has a rotation-invariant variant (HOT+R) via alternating OT and Procrustes. Experiments span LLM to LLM, vision to vision, and NSD brain to brain. Vanilla HOT typically matches or beats pairwise in LLMs and brain, while HOT+R strongly improves on vision transformers, where rotation invariance is known to matter.", "questions": "Could the authors perform subsampling sensitivity analysis, as noted above?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This work addresses an important challenge that a lot of representational similarity comparison studies currently face: how do we match the layers in a principled manner? This problem has been overlooked, but requires attention. Therefore, the significance of this work is high. \n\n1. It is great that the authors introduce the algorithms for both rotation-sensitive and rotation-invariant similarity measures.\n\n2. It is simple to extend the proposed method to incorporate more hierarchies. As noted in the discussion section, this work opens up a possibility to perform interesting matching experiments that were not possible before. One additional application with the third hierarchical level would be simply matching models (which would, at the lower level, match the layers, and then, at an even lower level, match neurons).\n\n3. The authors provide reasonable baselines for proper comparisons\n\n4. The results are remarkably interpretable and biologically useful. The high clarity in the presentation helps.\n\n5. As far as I know, this work is original.", "weaknesses": "As already noted in the limitations section of the paper, HOT/HOT+R is computationally demanding. This can be alleviated by subsampling the neurons, particularly for HOT+R. I am guessing that subsampling the neurons for HOT would lead to a dramatically bad matching score, however. It would be nice if the authors could experimentally test how sensitive the matching (HOT/HOT+R) is to subsampling of neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l8qfGChjOX", "reviewer": "Reviewer_8PNK", "summary": "This paper introduces a way to compare what different neural networks (and even brains) represent across their layers, all at once, rather than matching layers and/or neurons one by one. Using a hierarchical optimal transport approach, it softly links each layer in one system to possibly several layers in another, producing a single overall alignment score and naturally handling models with different depths. Across vision models, language models, and brain imaging, the method matches or beats standard layer-by-layer matching and reveals clean, intuitive hierarchies, where early layers align with early layers and deeper with deeper, without hand crafting those rules.", "questions": "Points that require further attention: (i) Sensitivity to the choice of hyperparameters. (ii) The correspondence between the observed alignment and behavior.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Original extension of the optimal transport framework to entire hierarchies, integrating many-to-many soft matching across layers and neurons in a unified manner; the method can handle models of different depths/sizes without hand-crafted layer pairing.\n- Clearly written with conceptual claraity, and well explained.\n- Empirical evaluation spans three relevant, diverse application domains.\n- Candid discussion of limitations.\n- The problem is important yet not well understood, so the potential impact is high.", "weaknesses": "As the limitations section notes, hierarchical OT is computationally costly, which is likely to limit the method's scalability. The somewhat narrow experimental scope leaves it unclear whether the method can effectively generalize to long-context settings, multimodal models, or tasks beyond vision and language.\n\nSome typos remain  (e.g., 'and and' in line 086).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)