Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment
Authors: Yu Zhu
Chunfeng Song
Wanli Ouyang
Shan Yu
Tiejun Huang
Venue: ICML 2025 poster (2025)
Keywords: Neural Representational Consisitency
Motor Cortex
Visual Cortex
Zero-shot
Brain-Computer Interface
Primary area: applications->neuroscience_cognitive_science
TL;DR: We developed PNBA, a representation alignment framework that reveals preserved neural representations despite neural heterogeneity in multiple cortices via zero-shot validation, showcasing its practicality through zero-shot behavior decoding.

Abstract: Individual brains exhibit striking structural and physiological heterogeneity, yet neural circuits can generate remarkably consistent functional properties across individuals, an apparent paradox in neuroscience. While recent studies have observed preserved neural representations in motor cortex through manual alignment across subjects, the zero-shot validation of such preservation and its generalization to more cortices remain unexplored. Here we present PNBA (Probabilistic Neural-Behavioral Representation Alignment), a new framework that leverages probabilistic modeling to address hierarchical variability across trials, sessions, and subjects, with generative constraints preventing representation degeneration. By establishing reliable cross-modal representational alignment, PNBA reveals robust preserved neural representations in monkey primary motor cortex (M1) and dorsal premotor cortex (PMd) through zero-shot validation. We further establish similar representational preservation in mouse primary visual cortex (V1), reflecting a general neural basis. These findings resolve the paradox of neural heterogeneity by establishing zero-shot preserved neural representations across cortices and species, enriching neural coding insights and enabling zero-shot behavior decoding.

Individual brains exhibit striking structural and physiologi- cal heterogeneity in neuronal responses (Churchland al., Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment
Yu Zhu 1234
Chunfeng Song²
Wanli Ouyang²
Shan Yu 12 Tiejun Huang*
Abstract
2010) and connectivity patterns (Kasthuri et al., 2015). Yet, neural circuits consistently generate similar functional prop- erties across individuals, from orientation selectivity in vi- sual cortex (Hubel et al., 1959) to movement encoding in motor areas (Churchland et al., 2012). This functional sta- bility persists both within individual brains (Gallego et al., 2020; Stringer al., 2021) and across different individuals during matched behaviors (Safaie et al., 2023), presenting apparent paradox with the observed physiological hetero- geneity (Averbeck et al., 2006). Understanding this robust preservation despite biological variability is crucial for ad- vancing neural coding theories (DiCarlo et al., 2012) and developing generalizable praiutcomputer interfaces (BCIs) (Chaudhary et al., 2016; Sussillo et al., 2016).
Individual brains exhibit striking structural and physiological heterogeneity, yet neural circuits can generate remarkably consistent functional properties across individuals, an apparent para- dox in neuroscience. While recent studies have observed preserved neural representations in mo- tor cortex through manual alignment across sub- jects, the zero-shot validation of such preservation and its generalization to more cortices remain un- explored. Here we present PNBA (Probabilistic Representation Alignment), a new framework that leverages probabilistic model- ing to address hicrarchical variability across trials, sessions, and subjects, with generative constraints preventing representation degeneration. By es- tablishing reliable cross-modal representational alignment, PNBA reveals robust preserved neural representations in monkey primary motor cortex (MI) and dorsal premotor cortex (PMd) through zero-shot validation. We further establish similar representational preservation in mouse primary visual cortex (V1), reflecting general neural ba- sis. These findings resolve the paradox of neu- ral heterogeneity by establishing zero-shot pre- served neural representations across cortices and species, enriching neural coding insights and en- abling zero-shot behavior decoding.
Recent advances in large-scale neural recording technolo- gies (Stringer et al., 2019; Siegle et al., 2021) have enabled systematic investigation of functional invariance within di- verse cortices. Significant progress has been made in motor cortices, observing preserved low-dimensional neural pop- ulation representations (Rubin et al., 2019; Gallego et al., 2020; Safaie et al., 2023). However, rigorous validation and broader verification of such representation preserva- tion faces three fundamental challenges. First, and most critically, biological neural systems exhibit hierarchical vari- ability spanning multiple scales, from trial-to- trial fluctua- tions (Churchland et al., 2010; Renart et al., 2010) to inter- individual differences in population activity (Russo et al., 2018; Gallego et al., 2020). This fundamental variability poses a significant barrier for systematic neural analysis. Second, cortical regions serve fundamentally distinct func- tions. Visual cortex transforms sensory inputs into struc- tured representations (Walker et al., 2019), while motor cor- tex converts movement goals into coordinated population dynamics (Churchland et al., 2012). Third, these functional differences require distinct analytical frameworks. Sensory systems utilize encoding models mapping stimuli neural responses (Yamins & DiCarlo, 2016; Walker et al., 2019), while motor systems employ decoding models predicting movements from neural activity (Sussillo et al., 2016; Pan- darinath et al., 2018). These interrelated challenges make rigorous and generalizable validation of preserved neural representations particularly challenging.
1. Introduction
"This work was done during his internship Shanghai Artifi cial Intelligence Laboratory. Institute of Automation, Chinese Academy of Sciences "School of Artificial Intelligence, University of Chinese Academy of Sciences 'Shanghai Artificial Intelligence Laboratory 'Beijing Academy of Artificial Intelligence. Corre- spondence to: Chunfeng Song <songchunfeng @ @aiane@@iplllrgra pjlab.org.cn. Shan Yu <shan. nlpr.ia.ccccm>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Neural population activity has been consistently shown to operate in low-dimensional latent spaces (Churchland et al., 2012; Russo et al., 2018; Stringer et al., 2019). Leverag- ing this intrinsic cow-dimensional structure, we identify three key principles for addressing these challenges. First, explicit neural-behie becavioorl' modeling in low-dimensional spaces serves as the foundation to capture functionally rele- vant representations with biological interpretability (Yamins & DiCarlo, 2016; Walker et al., 2019). Second, neural representations analysis should be anchored in this neural- behavioral framework to preserve functional interpretabil- ity. Third, and most crucially, the framework should ad- dress the hierarchical neural response variability that en- ables zero-shot srojsttubbect validation. Current methods in now-dimensional analysis, including behavior-guider neural activity dimensionality reduction (e.g., CEBRA (Schneider et al., 2023; Chen al., 2024)) and beuvvolbehavioral fusion strategies (e.g., pi-VAr VAE (Zhou& & Wei, 2020; Gondur et al., 2023)), do not fully address these requirements. They either lack explicit neural- behavioral modeling or yield entangled representations that complicate the validation of preserved neural representations. Moreover, their reliance on post-hoc alignment (Safaie al., 2023; Schneider al., 2023) fails to address the fundamental challenge of neural variability, precluding direct validation of representation preservation across individuals.
ferent recording modalities (calcium imaging) and behav- ioral contexts. These emerged observations from multiple cortical regions and species reflect fundamental properties of the neural basis. We finally showcase its practicality through zero-shot V1-guided motion decoding, showing the potential for sensory-guided movement BCIs.
We highlight the main contributions of this work below:
A new probabilistic representation alignment framework, PNBA, that handles cross-scale neural heterogeneity while preventing degenerated representations.
Robust neural-behavioral representation alignment within multiple cortical regions (M1, PMd, V1) from different species (primates, mice).
Demonstration of preserved neural representations across trials, sessions, and subjects in distinct cortical regions and species through zero-shot verification, with practical appli- cations in zero-shot behavior decoding.
Codes are availiable at https: / /ttp://gtttttt com/ zhuyu-cs/PNBA.
2. Preliminaries
To investigate the consistency of neural representations in different cortices and species, we here introduce key ex- perimental paradigms, fundamental challenges, and recent breakthroughs in neuroscience.
Here we present PNBA (Probabilistic Representation Alignment), a unified framework for learn- ing aligned neural-be representations that enables zero-shot generalization. PNBA introduces probabilis- tic representation modeling to address neural variabil- correspondence through
Understanding neural process- ing requires investigating how neural populations trans- form sensory information into behavioral output (Musall et al., 2019; Bahl & Engert, 2020). Driven by different sub-questions in neuroscience, researchers have focused on studying individual cortical regions, e.g.. primary vi- sual cortex (V1) for sensory processing and motor areas including primary motor cortex (M1) and dorsal premotor cortex (PMd) for movement planning and control. Visual cortices studies have extensively utilized mouse models, leveraging their genetic tractability (Madisen et al., 2015) to enable large-scale two-photon calcium imaging in head- fixed preparations (Billeh et al., 2020; Stringer et al., 2021) (Figure la). Motor cortices investigations have spanned both mice and non- hummn primates, combining high-density elec- trophysiology with well-controledd behavioral paradigms (Churchland et al., 2012; Peters et al., 2014) (c.g., center-out task, Figure 1b). These experimental approaches in different model systems provide rich opportunities for investigating neural coding.
Experimental Paradigms.
ity, explicit dual-modal representation alignment, and generative con- straints to prevent degenerate representations. These prin- cipled considerations enable PNBA to achieve robust align- ment between neural activity and behavior, laying the foun- dation for further investigating neural representations.
We first validated PNBA's alignment capability using neural recordings from motor and sensory cortices across differ- ent species (Safaic et al., 2023; Turishcheva et al., 2024). The framework achieved robust neural- -ehavioral corre- spondence both primate motor areas (M1/PMd) during reaching tasks and mouse visual cortex (V1) during visual stimulation. Building on this reliable aligmment, we focused on analyzing neural representations. In motor cortices, zero- shot validation revealed consistent neural representations across trials, sessions, and subjects, extending beyond pre- vious findings requiring manual alignment (Safaie et al., 2023). Furthermore, our analysis of mouse V1 revealed preserved neural representations despite fundamentally dif-
Hierarchical Neural Variability. Neural activity exhibits remarkable variability across multiple temporal and spa- tial scales (Fig. Ic). At the finest scale, trial-to-trial vari- ability (Euia) reflects both intrinsic noise and moment-to-
'e use behavioaehvvoora" to refer to neuronal population activity and its associated hehavioral variables (c.g. visual stimuli, movement kinematics).
2

---

Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment
Experimental Paradigms
Hierarchical Neural Variability
sionality reduction and alignment, low-dimensional neural activity patterns from different subjects performing identical reaching movements show remarkable similarities (Fig. Id),
a.
"two-photon caleium imaging
t scosion
M subject N: nourons
While these findings in motor cortices are compelling, whether such preserved representations exist in other corti- cal regions like visual cortex remains an open question.
tine
Striat
day
Head fixed mouse during passive viewing
Monkey A
b.
*electoppyysooooy
3. Method
In this section, we propose a framework for neural-behavior alignment, guided by three principles detailed in Section 1, laying the foundation for neural representation analyses.
time
Monkey B
Non-human primates during reaching task
neural activity
behavior
Preserved Neural Representation Across Subjects
3.1. Probabilistic Representation Alignment
d.
PCA
PCCS
CCA
Given paired observations of neural activities x E X' C. RN and behavioral variables y E y C IR DxT, where N and D denote the number of recorded neural units and be- havioral dimensions respectively, we aim to identify aligned representations that captures neural-behavioral connections. Neural responses exhibit substantial variability across re- peated observations while maintaining complex many-to one mappings with behavior (Marder & Goaillard, 2006; Churchland et al., 2010). This intrinsic variability pre- cludes deterministic wise alignments, necessitating a distribution-level framework.
Monkey A
PCIA
PCZ2
same behavior
CC2
PCJa
Monkey B
PC2,
+CC: principal component
PC
CC: canonical component
Figure 1. Neural Foundations. (a) Two-photon caleium imaging in mouse V1 during visual stimulation. (b) Electrophysiological recordings from primate motor cortices during center-out reach- ing. (c) Hierarchical neural variability across trial (Ema)), session (Cwsssion, and subject (Cuukjea) scales. (d) Cross-subject alignment pipcline combining PCA-based dimensionality reduction and CCA- based alignment (Safaie et al., 2023).
To address this challenge, we formulate our approach through probabilistic representation alignment in shared latent space Z. Specifically, we construct two encoders: : - P(Z) for neural activities and ge : y + P(Z)) for behavioral variables, mapping inputs to probability dis- tributions (Fig.2a). These distributions are aligned using a
moment fluctuations in neural responses (Churchland et al., 2010; Ponce-Alvarez et al., 2013). At intermediate scale, session-to-session variability (Esession) emerges from changes in behavioral state and task-specific adaptation (Peters et al., 2014). The most prominent source of variation occurs at the broadest scale, inter-subject variability (Fsabjeat, stem- ming from individual differences in neural responses and connectivity patterns. These hierarchical variations lead to fundamental neuroscience question: how neural popu- lations maintain reliable function despite such multi-scale variations? Recent advances in population analysis have provided initial insights into this fundamental question.
probabilistic matching objective (Chun, 2023):
LProbMatch --------e ----iimmid( (- d(.;)) )) + b)
(1 m) sigmoid(( d(.:) b)
(1)
where m E {00 indicates matched or unmatched pairs, with learnable parameters a and b controlling the alignment sensitivity. d(.,-) measures the distributional difference
between two multivariate Gaussians:
d(.fo (x)(xx).(())) = |1 - 112 1 (x() to
o. (y) ll
(2)
Preserved Neural Representations in Motor Cortex. De- spite these complex sources of variability, recent studies have revealed remarkable consistency in neural population activity patterns. At the individual level, reliable trial-to-trial patterns suggest robust underlying mechanisms (Peters et al., 2014; Russo et al., 2018), while across sessions, preserved representations demonstrate stable neural dynamics despite daily variations (Gallego et al., 2020). Most strikingly, in motor cortex, these preservations extend beyond individual subjects (Safaie et al., 2023). Through appropriate dimen-
where μ and の² denote the mean and variance of the en- coded distributions, respectively.
However, directly optimizing the matching objective leads to degenerate solutions where encoders map diverse inputs into nearly identical representations (analysis in Supplemen- tary Material A.2):
fo(x) 22 go (y) R Zeonst:
Vx E y,y e y
(3)
Such degeneration, while achieving alignment, fails to main-
3

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Probabilistic Matching
Generative-infomeed Modeling
2017), we obtain:
b.
Neural spikes
Estimated Firing rates
Liotal ProbMatch + (ul-logp(x,))) + X2( log (o (x))))
Wi
To
+ A3 log log x))
(5)
Time
where A controls the strength of each generative constraint.
Rehaviors
Estimated Behaviors
Detailed derivations are Supplementary Material B.1.
8
Ha
Building on the bidirectional Markov chain strueture, we factorize the joint distribution in equivalent forms:
Timse
Time
Ne: Sendoon/Sabjett
((xy)))(()) (plyl)p(2//)p((>
p(x. y.z)
(6)
Figure
Prohabilistic Framework for Nehrao-Behavioral Representation Alignment. The framework comprises (a) a probabilistic matching module with modality- specific encoders fo and Do that project neural activities and hehavioral variables into a shared latent space, and (b) a gener- ative constraint module with decoders fö and d that preserves modality-speciff structure through reconstruction, facilitating ro- bust cross-seas scale alignment of behhviora representations.
This symmetric factorization, combined with the conditional independence assumptions inherent in the Markov structure, leads to a fundamental decomposition of the conditional distributions:
p(x, z/y) p(x(2)p(z()) 2(() ply ((z)z))((()))
tain meaningful representations, motivating our generative- informed approach to constrain the alignment.
3.2.2. END-TO-END OPTIMIZATION VIA ELBOS
To optimize the intractable generative constraints in Eq.5, we employ variational inference (Kingma & Welling, 2014). approximating the true posteriors p(z) (/)x) and p(zly) with variational distributions g(z)x) and g(zly). The negative log-likelihood terms are optimized by minimizing their re- spective negative Evidence Lower Bounds (ELBOs):
3.2. Generative-Informel Representation Alignment
3.2.1. CONSTRAINED OPTIMIZATION FRAMEWORK
To prevent degenerate solutions while maintaining effective alignment, we propose unified framework that simultane- ously ensures distribution matching and structural preserva- tion across modalities. Our key insight is that meaningful aligmment requires both distributional matching and genera- tive modeling of each modality (Fig.2b).
\1 - log p(x, y)) + (loog(oxx)))) + As p(y(x))
A (-EBOO;)
Motivated by dimensionality reduction principles in neural data analysis (Stringer& & Pachitariu, 2024), we assume that ceural-behavioral correlations primarily reside in a shared low-dimensional latent space. This leads bidirectional
Ai (Zx()/x.y) [log((x(z) -oogsoxxe)]KKKKlllxxxx)))ll/>
+ /g(l/x) [log p(ylz) 1-K---K-A X<(g(z)x,
- -o px(z)) XT[g(2())|>((2))]
Markov chain (x + z ++ y), where neural-behavioral connections are bridged through The joint distribution p(x,y) and conditional distributions p(y|x) and p(xly) to- gether ensure bidirectional mapping preservation.
- (a p(y)() - (I11(21)))l<((()I
Following (Johnson al., 2016), model the variational posteriors as conditionally independent Gaussian distribu- tions and define the joint approximate posterior as:
formalize this as a constrained optimization problem:
(x)))))
for x
min LProbMatch
q(z|x, y)
(9)
for y yo (o(22l)po2z)xx)
s.t. logg(x,y) S c1, logp((())) C2,
(4)
-logp(xly) c3
where neural networks with parameters 0 and & parameter- ize the neural activity encoder fo and behavioral encoder de respectively.
where C12 C22 and C3 specify upper bounds for negative log- likelihood of joint and conditional distributions. Applying the method of Lagrangian multipliers under the KKT con- ditions (Karush, 1939), with A; 2 0, c > (Higgins al.,
The reconstruction terms these ELBOs employ modality- specific likelihood functions. Specifically, Poisson negative log-likelihood for neural spikes x and Gaussian negative

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
log-likelihood, simplified with mean squared error, for con- tinuous behavioral variables y. These reconstruction objec- tives are crucial for preventing degenerate representations, as established in the following theorem:
Table I. Cross-subject Beavio-ee Alignment Perfor- mance. Quantitative evaluation using single-tria -riittrral Pearson correla- tion coefficients (R) between neural and behavioral representations across three cortical regions. Higher values indicate better align- ment. PNBA demonstrates superior performance across both train- ing and held-out test subjects. Methods marked with ($) require in- dependent training for each modality, while (*) denotes approaches necessitating per-trial optimization for individual subjects. Addi- tionally, methods designated with (*) require session-specifee or subject-speeific training procedures.
Theorem 3.1. Let Sø X + Z and d6 : + Z denote the neural and behavioral encoders. The following properties hold:
(i) WVonddegeneracy The optimization objective Lwnat pre- vents representation degeneration by ensuring:
lim
Lioual = +00, Veoon E Z
(10)
Cortical Area
Method
Correlation (R)
(x( -
Training Subjects
New Subjects
VADİ
0.0197
0,0016
(ii) [Information Preservation] The generative constraints ensure enhanced mutual information between input and representation spaces:
FA+ Procrustes?
0.3334
0,2009
PCA+CCA#
0,3520
0,2160
Motor Cortex (M1)
FA+amLDS+
0,5807
0.3627
Neuroformer'
0,5214
7j > O (II) C(f (x); I(f Crrastaas (x);x) +
MEME
0.7756
0,7060
PNBA (Ours)
0.9465
0.9302
(iii) (Representation Stability] For neural responses X, corresponding the same behavioral variable y, the learned representations maintain bounded distances:
VAB
0.0063
0.0028
FA+Procrustes
0.3605
0.2877
PCA+CCA+
0.3916
0.3397
Motor Cortex
FA+amLDSf
0.4733
0.4366
(PMd)
Neuroformer'
0.3283
0 < a S [x)) [xx
(12)
MEME
0.5279
0.5255
PNBA (Ours)
0.9248
0.9176
VAB
0,0029
-0,0009
These theoretical guarantees demonstrate that frame- work learns representations with non-degenerate mappings of distinct neural patterns, enhanced feature preservation beyond naive distributional matching, and bounded neural variability that maintains behaviorally-relewe structure.
FA+Procnutess
0.1221
0,1207
PCA+CCA+
0.1210
0.1209
Visual Cortex (V1)
FA DSF
0.1509
0.1501
Neuroformer*
0,4116
MEME
0.6357
0.5980
PNBA (Ours)
0,8830
0,8705
3.2.3. CROSS-SUBJECK NETWORK FOR VAIRABLE NEURAL ACTIVTIY
Notably, our PNBA is inherently modality- en- abling straightforward extension to neural recordings from diverse experimental paradigms, as introduced in Section 2. We validate these theoretical properties through comprehen- sive experiments with multiple neural recording modalities in the following section.
Our approach employs a single shared network across all subjects, despite variations in neural population sizes be- tween recording sessions. This cross-subject architecture processes neural activity matrices (N x T) using shared learnable projection heads at both network endpoints. The input projection transforms data through 2D convolution operations followed by AvraaePiver standard- izing the neuron dimension fixed size D (32 x 8 x T for M1 data and 128 256 T for V1 data).
4. Results
Our experimental results first show the robustness of cross- modal representation alignment, then especially reveal pre- served neural representations with zero-shot validations, and finally showcase in V1-driven movement BCIs.
This standardization strategy enables the network's 's encoder, latent representation, and decoder components to be entirely shared across subjects. The output projection mirrors this design symmetrically, employing bilinear interpolation to restore dimensions back each subject's original neuron count (N T'. By maintaining consistent internal dimen- sionality while accommodating variable input/output sizes, our architecture achieves true cross-subject sharing of all network parameters, facilitating direct comparison of neural representations across subjects. For detailed implementa- tion specifications. please refer to Supplementary Material
4.1. Experimental Settings
Datasets. We evaluated our framework on two distinct neu- ral recording datasets examine cross- moosl representa- tion alignment within different neural systems. The primary dataset consists of neural recordings from non-human pri- mates during center-out reaching tasks (Safaie et al., 2023), with recordings from primary motor cortex (MI) and dorsal premotor cortex (PMd). For MI evaluation, we utilized
C.4.
5

---

Neural Representational Consistency Emerges from Probabillstic Seural-Behaviorgy Representation Alignment
Distribution of Representation
Neurl-SStmulas Representarion A Aligoment
b.
1137
0.8725
110
0.8
0.7124
6.7707
0.8
1034
000
Comellaton
Coneltation
0.6
Cornlntion
0.6
1743
0.4
1744
1254
0.2
0.2
Netro
Gost
MM
only vAV
01-
T602
Loss Fanctions
Latent Distessions
>vantnol Repucectatiin Valkaz
Vical Stimulse Representation
Figure 4. Component analysis and parameter optimization of PNBA framework. Incremental component evaluation starting from baseline probabilistic matching (Naive), incorporating cross- modal VAE (Cross), multi-moda VAE (MM), two VAE-based modeling without matching (only VAE), to the complete PNBA framework. b, Neural-behaviora correlation versus latent dimen- sionality, with optimal performance at d = 32.
Figure 3. Neural-stimulus representation alignment in mouse V1. a, Histogram distributions demonstrating aligned representa- tional structure between neural activity and visual stimuli in latent space across two held-out mice. b, Trial-wise correlation from the held-out Mouse 9 demonstrating meaningful neural-stimulus correspondence (t= 25.99, p = 1.76 x 10 10-146).
4.2.
Representation Alignment
neural data from 12 training sessions per subject for two subjects (C, M), two validation sessions per subject, and three test sessions each from two held-out subjects (J. T). Neural spiking activities were recorded concurrently with kinematic trajectories. More details, including PMd, are provided in Supplementary Materials C.I.
Cross-Modal Alignment Performance. Quantitative anal- yses demonstrate that PNBA achieves superior neural- behavioral representation alignment across multiple cor- tical areas and subjects (Table 1). Using the mouse VI calcium imaging dataset, we performed systematic valida- tion of alignment quality through distribution- analyses in new subjects (Figure 3a). The observed distribution over- lap between neural activity and behavioral measurements indicates successful encoding of shared representational space that generalizes across subjects.
To examine the framework's applicability beyond motor cortices. we conducted analyses on calcium imaging data from mouse V1 (Turishcheva et al., 2024). The V1 dataset contains recordings from 10 mice presented with identi- cal dynamic visual stimuli within paired experiments (5 pairs total). We used mice (4 pairs) for training and val- idation, with 2 mice (I pair) for held-out testing to assess cross-subject generalization. More details are provided in Supplementary Materials C.I.
Trial-level correlation analyses (Figure 3b) reveal distinct discriminative patterns (t= 25.99, p = 1.76 x 10 -1-4), where neural representations exhibit maximal correlation with their corresponding behavioral counterparts while maintaining minimal correlation with non-corresponding pairs. This nigh-performane alignment result is consistently observed in the monkey motor cortices (Supplementary Figure 9), demonstrating that PNBA effectively captures modality- specific neural-behavioral relationships while maintaining cross-subject generalization. This robust alignment provides reliable foundation for exploring zero-shot preserved neu- ral representations within different cortices.
Evaluation Metrics and Baselines. We employed single- trial Pearson correlation coefficient (R) as the primary evalu- ation metric, as detailed in SI C.3. This provides a standard- ized measure of representational similarity within the range [-1, 1]. Given the inherent variability across trials, sessions, and subjects, as well modality-specffic characteristics, correlation values necessarily deviate from the theoretical maximum. We evaluated PNBA against three established baselines, i.c., conventional VAE (Kingma & Welling, 2014) with independent modality training, session-specific Neu- roformer (Antoniades et al., 2024) implementing InfoNCE loss (Oord et al., 2018), and MEME (Joy et al., 2021) uti- lizing mutual distribution supervision. All models were evaluated under consistent training protocols, provided in Supplementary Materials C.2. To assess the statistical signif- icance of the alignment, we conducted independent samples t-tests between matched and mismatched pairs to distinguish teskerelated neural correspondences from chance- level cor- relations.
Ablation Studies. We conducted systematic analyses to evaluate the architectural design of PNBA. Through pro- gressive model comparisons (Figure 4a), we first assessed the contribution of each framework component. Initial ex- periments with naive probabilistic representation match- ing yielded suboptimal solutions. Similarly, implementing cross-modal VAE modeling in isolation resulted in degraded representations due to insufficient constraints. The incor- poration of multi-modal VAE constraints enabled effective representation alignment (R = 0.71), while exclusive VAE modeling without probabilistic matching achieved R = 0.77. The complete framework, integrating both VAE modeling
6

---

Neural Representational Consistency Emerges from Probabillstic Ceural-Behaviorss Representation Alignment
and probabilistic matching constraints, demonstrated opti- mal alignment (R 0.87). validating the PNBA framework.
Preserved Neural Representations in M1 Across Scales
a.
Cross-trial
1.00
We further examined the impact of latent space dimension- ality on representation quality. For the VI dataset, our anal- yses indicate that a 32-dimensional latent space achieves optimal performance (R= = 0.87), as illustrated in Figure 4b. Lower dimensionality proves insufficient for captur- ing neural-behavioral complexity, while higher dimensions increase computational cost without performance improve- ment. Following analogous procedures, we determined that 4-dimensional latent representations adequately capture neu- ral dynamics in motor cortical areas (M1 and PMd)).
Correlation
0.95
1-134
1+3.7 1- to
L 6.0
p=5.8-10
P-10-100
1-12
0.90
p- 144
0.85
Monkey
Morkey J
Morkey T 201 1418-19
Monkey T 201 1013--9-09
2016-144-85
2016-24407
b.
C.
Cross-scssion
Cross-subject
1.00
1.00
Correlation
0.95
0.95
1-:3
4.3. Zero-shot Cross-subject Validation Reveals Hierarchical Preservation in Motor Cortex
0.90
9-34-32
p- 40-10
1-277-10 10
1-79
0.85
0.85
Monkey 1
Monkey
Cros-saljctt
Our systematic analysis reveals a hierarchical preservation of neural representations in motor cortex (Fig. 5). Through zero-shot validation on independent held-out subjects, we demonstrate robust preservation across multiple analytical dimensions. The preservation strength exhibits system- atic gradient with increasing temporal and individual vari- ability. Triel-wise patterns demonstrated high consistency (mean R = 0.960 = 0.011, 1, I-test, p < 0.001), while across- session comparisons revealed substantial stability (R 0.946 + 0.008, t-test, p < 0.001). Most importantly, the represen- tations maintained high correlation even across different subjects (R- 0.939 + 0.033, t-test, p < 0.001), indicating consistent representational organization beyond individual variability (Safaie et al., 2023). Similar preservation pat- terns were observed in monkey PMd, a region associated with motor planning (see Supplementary Materials D.2).
Figure S. Hierarchical preservation of neural representations in monkey primary motor cortex (M1). Correlation analysis demonstrates systematic preservation across trial, session and sub- ject dimensions. a, Within-session trial-wise correlations exhibit high consistency (mean R 0.960 + 0.011, Ru, =196, nu, =198, Tei 1135, m =208 trials). b, Neural representations maintain sta- bility across recording sessions (R 0.946 + 0.008). c, Zero-shot validation across held-out subjects confirms cross-subject represen- tational similarity (R 0.939 = 0.033). Error bars denote standard deviation.
The observation of preserved neural representations across both motor and visual cortices, despite their distinct func- tion, suggests broader preservation of neural coding struc- ture. This finding has important implications for calibration- free BCIs. In the next section, we demonstrate the practical utility of these preserved representations through a zero-shot VI-guided movement decoding example.
The robust preservation of neural representations across sessions and subjects reveals a consistent representational structure in motor cortex, despite inherent neural variability. This observation raises fundamental question regarding whether such preserved representations exist in other corti- cal areas. To address this question, we next examined the visual cortex, especially VI.
4.5. Zero-shot V1-guided Movement Decoding
We leveraged the V1 neural representations to decode mouse movement kinematics, specifically regressing running speed (R²)). The neural representations demonstrated robust cross- subject generalization capabilities across multiple BCI de- coders. Using a GRU-based decoder achieved the highest performance (R² = 0.888, p = 4.44 x 10-13), while linear decoders (R² = 0.866. p 1.7 x 10 9) and feedforward neu- ral networks (R² = 0.880, p = 1.24 x 10-11) also showed promising generalization. These results validate the practi- cality of the preserved neural representations.
4.4. Zero-shot Preserved Neural Representations Extend to Visual Cortex
Zero-shot validation in mouse VI calcium imaging data revealed that neural representation preservation extends be- yond motor cortex. Analysis demonstrated a preservation pattern similar to M1 (Fig. 6). V1 exhibited substantial trial-wise consistency (mean R- = 0.912 + 0.017) (Fig. 11 in Supplementary Material D.3) and, critically, maintained robust cross-subject preservation under zero-shot testing (R = -0.892 + 0.014, t= 25.38, p= 9.25 x 10-142, Fig. 6.c). The preserved representations in VI remained highly significant across trials and subject.
This decoding capability aligns with current understand- ing of distributed behavioral information in neural circuits, where V1 participates in early sensorimotor integration (Stringer et al., 2019) and shows modulation by behavioral states (Niell & Stryker, 2010). Motor-related signals propa- gate through various brain regions (Laboratory et al., 2023; Khilkevich et al., 2024), with behavioral information dis-
7

---

Neural Representational Consistency Emerges from Probabilistic Ceural-Behaviora Representation Alignment
Preserved Neural Representations Existed in Mouse VI
Preserved Neural Representations Enables Zero-shot BCI
a.
Cross trials
b.
Cross-subjac
0.90
1-7.24
0.980
0.950
444 10
1-6 78
p-172x 10
pm 1:24 10"
Corelasion
0,925
0,925
0.88
1-6.02
-1778 10
0.900
1-25.70
7.43- 10
0.86
0.575
2--22- 30
0.575
1+25.38
0.5.0
Mouse
Mouse 10
0.500
Cros-stiiz
0.84
C.
Cross-subject Correlation Matrix
0.82
Linear
GRU
MI.P
141
0.92
BCI decoder
1136
(Mous
1243
0.90
Conclition
Figure 7. Zero-shot movement decoding from VI neural activ- ity. As an application of preserved neural representations, we examined cross-subject movement decoding from VI recordings. The explained variance (R") between predicted and actual move- ment trajectories quantifies decoding performance. Multiple decod- ing architectures showed significant zero- shot generalization (Lin- ear: t= 6.02, p= 1.78 x 10 9; GLM: t= 6.73, p= p= 1.72 x 10 21 1.44
1103
0.88
1145
0.86
113
0.84
1365
GRU: 4 10 18; MLP: t= p=1.24 10"11).
tririal
2024) enhance temporal modeling, their high-dimensional projections conflict with low-dimensional principles and interpretability. Existing solutions using behavior-guided dimensional reduction (Schneider et al., 2023; Chen et al., 2024) and neural-behavioral fusion (Pandarinath al., 2018; Zhou & Wei, 2020; Keshtkaran et al., 2022; Zhu et al., 2022; Gondur et al., 2023) face fundamental challenges in validating preserved neural representations. Specifically, fusion methods bias representations toward behavioral vari- ables, sanavior-guided dimensional reduction approaches fail to establish direct neural- beara--be correspondence, and the required post-hoc alignment may further introduce potential ambiguity. Our PNBA framework addresses these limitations through probabilistic representation alignment, enabling robust low oow-dimensional representations while nat- urally bridging neural and behavioral domains.
Neural Represemations (Mouse 9)
Figure 6. Hierarchical preservation of neural representations in mouse primary visual cortex (V1). Correlation analysis demon- strates systematic preservation across trial and subject dimensions. a, Trial-wise correlations demonstrate high consistency (mean R 0.912 # 0.017, = 412, 7M, =413 trials). b, Cross-subject sta- bility analysis between mice presented with identical visual stimuli. Zero-shot validation confirms cross-subject preservation (R= = 0.892 0.014). Error bars denote standard deviation.
tributed across neural populations (Musall al., 2019). Our demonstration of movement decoding provides one exam- ple of how V1 preserved neural representations might be leveraged in behavioral decodings, suggesting potential ro- bust motor BCIs that integrate information from multiple cortical regions based on their preserved representations and distributed motor encoding characteristics.
Cross-modal Alignment. Cross-modal representation learning has emerged as a powerful paradigm in machine learning, represented by CLIP (Radford et al., 2021) which demonstrated that dual-encoder architectures with con- trastive learning can effectively align image and text repre- sentations. This success has inspired numerous advances (Zhai et al., 2023; Chun, 2023; Lavoie et al., 2024). How- ever, aligning neural and behavioral representations presents distinct challenges due to intrinsic neural variability and recording heterogeneity (Fig. la.b). Traditional approaches have focused on disentangling behavior-releww nents from neural activity (Sani et al., 2021; Hurwitz et al., 2021; Wang et al., 2024; Sani et al., 2024), but overlooked the importance of behavioral representation learning. Re- cent methods like Neuroformer (Antoniades et al., 2024) attempt to bridge this gap through contrastive learning, yet
5. Related Work
Neural Representation Learning. A fundamental analysis strategy in systems neuroscience relies on dimensionality reduction principles to study high-dimensional neural ac- tivity, where hidden regularities may emerge despite hier- archical heterogeneity (Stringer & Pachitariu, 2024). Clas- sical dimensionality reduction methods like PCA, jPCA (Churchland et al., 2012), t-SNE (Van der Maaten & Hinton, 2008), and UMAP (McInnes et al., 2018) have revealed such structures but struggle with temporal dynamics. While Transformer models (Ye & Pandarinath, 2021; Liu et al., 2022; Le & Shlizerman, 2022; Ye et al., 2024; Zhang et al.,
compo-
8

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
they require subject-specific optimization and assume de- terministic -eurio-behvvioral correspondences, failing to account for inherent neural variability. Our probabilistic alignment framework overcomes these limitations by jointly modeling neural variability and behavioral correlation, en- abling robust validation of preserved neural representations.
sentations. We validated PNBA through comprehensive zero-shot experiments across multiple cortical regions (M1, PMd, V1) and species (primate, mouse), revealing preserved neural representations that generalize across sessions and individuals. These findings provide new insights into the ap- parent paradox between neural heterogeneity and functional stability, demonstrating how robust neural representations can emerge and persist despite biological variability, with practical implications demonstrated through zero-shot de- coding. In general, we hope PNBA could advance broader neuroscience investigations via multimodal representation alignment, while the discovered zero-shot preserved repre- sentations may bring opportunities for stable BCI paradigms, particularly in multi-region BCIs.
Neural-Behavine Modeling, Neural-behavione relation- ships are studied through encoding models mapping stimuli to neural responses and decoding models predicting behav- ior from neural activity (Mathis et al., 2024). Deep neural networks have enhanced encoding models that mimics the cortices, mapping external stimuli and cognitive variables to neural activity patterns (Yamins & DiCarlo, 2016; Kell et al., 2018; Walker et al., 2019; Bashivan et al., 2019; Marks & Goard, 2021; Vargas al., 2024), while decoding models extract encoded information (e.g., kinematic variables, vi- sual features) from complex neural activities (Gallego et al., 2017; Yoshida & Ohki, 2020; Stringer et al., 2021). How- ever, these methods assume local neural recordings contain complete behavioral information or require subject-specific fine-tuning. Our PNBA framework addresses these limita- tions by learning aligned latent representations that capture intrinsic neural-behavioral correlations, maintaining mecha- nistic interpretability while avoiding direct modeling con- straints (Melbaum et al., 2022). The learned representations naturally bridge neural activity and behavioral variables, establishing foundations for BCIs.
Impact Statement
Our work establishes computational framework that ad- vances both fundamental neuroscience and neural interface technologies. PNBA's primary contribution is a robust ap- proach for aligning neural and behavioral representations through learned latent spaces. When combined with attri- bution analysis (Achtibat al., 2023), this enables precise characterization of neuron-specific encoding mechanisms within neural populations. Beyond single-region analysis, this approach can be extended to broader systems neuro- science investigations, from multi-region to whole-brain analyses, enabling quantitative assessment of area-specific contributions and inter-regional coordination in behavior generation (Steinmetz et al., 2019; Khilkevich et al., 2024).
6. Discussions and Conclusions
While PNBA demonstrates robust implicit cross-subject neural representation alignment for observable behaviors, extending the framework to covert neural processes remains challenging, such as cognitive processes like evidence in- tegration during ceciisionmmaking or attentional regulation that lack direct behavioral observations. Our PMd analysis provides an intermediate solution by leveraging temporally delayed behavioral readouts, where preparatory neural ac- tivity patterns are mapped to subsequent movement kine- matics. A promising future direction involves incorporating temporal dynamics modeling (Pandarinath et al., 2018) 8) into the PNBA framework. By explicitly accounting for the dynamical systems properties of neural circuits, such exten- sions could significantly enhance the current representation alignment approach. We expect that these dynamical en- hancements, combined with new experimental paradigms integrated with PNBA, will further advance our understand- ing of these covert processes.
Furthermore, given that neuroscience inherently relies on multimodal data integration, PNBA's extensible architec- ture facilitates integration across biological scales, from molecular profiles (Bugeon et al., 2022) to cellular dynam- ies (Stringer et al., 2019), neural circuit mechanisms (Pa- triarchi et al., 2020), and ultimately whole-brain analyses (Laboratory et al., 2023).
The preserved neural representations discovered through PNBA present significant potential for advancing calibration-free neural decoding systems, particularly for acute conditions like spinal cord injury (Ahuja et al., 2017) where collecting paired neural-behavioral training data is impractical. Additionally, this framework holds promise for enhancing broader brain-computer interface applications, including speech synthesis (Anumanchipalli et al., 2019; Moses et al., 2021) and motor function restoration (Hochberg et al., 2012).
Collectively, this framework advances our understanding of neural coding while offering promising directions for BCIs, with potential impacts spanning from fundamental neuroscience research to clinical applications.
In this work, we established PNBA as a new framework for robust multimodal representation alignment. Through prob- abilistic modeling with generative constraints, our approach effectively addresses hierarchical variability across trials, sessions, and subjects while preventing degenerate repre-
9

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Acknowledgement
Chen, C., Yang, Z., and Wang, X. Neural embeddings rank: Aligning 3d latent dynamics with movements. In The Tiirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
This work supported by Shanghai Artificial Intelligence Laboratory.
Chun, S. Improved probabilistic image-text representations.
References
arXiv preprint arXiv:2305.88771 2023.
Achtibat, R., Dreyer, M., Fisenbraun, I., Bosse, S., Wiegand, T., Samek, W., and Lapuschkin, From attribution maps to uumaa--understndabble explanations through concept relevance propagation. Nature Machine Intelligence, 5 (9):1006-1019, 2023.
Chun, S., Kim, W., Park, S., and Yun. S. Proba- bilistic language-image pre-training. arXiv preprint arXiv:2410. 18857, 2024.
Churchland, M. M., Yu, B. M., Cunningham, J. P., Sugrue, L. P.. Cohen, M. R., Corrado, G. S., Newsome, W. T., Clark, A. M., Hosseini, P., Scott, B. B., et al. Stimulus onset quenches neural variability: widespread cortical phenomenon. Nature neuroscience, 13(3):369-37 -388, 2010.
Ahuja, C. S., Wilson, J. R., Nori, S., Kotter, M., Druschel, C., Curt, A., and Fehlings, M. G. Traumatic spinal cord injury. Nature reviews Disease primers, 3(1):1-21, 2017.
Alemi, A. A., Fischer, I,, Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint 1121vv1612.00410, 2016.
Churchland, M. M., Cunningham, J. P., Kaufman, M. T., Foster, J. D., Nuyujukian, P., Ryu, S. I., and Shenoy, K. Neural population dynamics during reaching. Nature, 487 (7405):51 1-56, 2012.
Antoniades, A., Yu, Y., Canzano, J., Wang, W. and Smith, S. L. Neuroformer: Multimodal and multitask generative pretraining for brain data. In The Eighteenth International Conference on Learning Representations, 2024.
Cohen, M. R. and Kohn, A. Measuring and interpreting neuronal correlations. Nature neuroscience, 14(7):811- 819, 2011.
Anumanchipalli, G. K., Chartier, J., and Chang. E. F. Speech synthesis from neural decoding of spoken sentences. Na- ture, 568(7753):493 2019.
DiCarlo, J. J., Zoccolan, D., and Rust, N. C. How does the brain solve visual object recognition? Neuron, 73(3):
415 434, 2012.
Averbeck, B. B., Latham, P. E., and Pouget, A. Neural correlations, population coding and computation. Nature reviews neuroscience, 7(5):358-366, 2006.
Gallego, A., Perich, M. G., Miller, L. E., and Solla, S. A. Neural manifolds for the control of movement. Neuron, 94(5):978-984, 2017.
Gallego, J. A., Perich, M. G., Chowdhury, R. H., Solla, S. A., and Miller, L. E. Long-term stability of cortical popu- lation dynamics underlying consistent behavior. Nature neuroscience, 23(2):260-22) 270, 2020.
Bahl, A. and Engert, Neural circuits for evidence accu- mulation and decision making in larval zebrafish. Nature neuroscience, 23(1):94-100) 102, 2020.
Bashivan, P., Kar, K., and DiCarlo, J. Neural population control via deep image synthesis. Science, 364(6439); eaav9436, 2019.
Georgopoulos, A. P., Kalaska, J. F,, Caminiti, R., and Massey, J. On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. Journal of Neuroscience, 2(1 I): 1527-1537, 1982.
Billeh, Y. N., Cai, B., Gratiy, S. I.., Dai, K., Iyer, R., Gouwens, N. W., Abbasi-Asl, R., Jia, X., Siegle, J. H., Olsen, S. R., et al. Systematic integration of structural and functional data into multi-scale models of mouse primary visual cortex. Neuron, 106(3):388-))) 4008, 2020.
Gondur, R., Sikandar, U. B., Schaffer, E., Aoi, M. C., and Keeley, S. L. Multi-modal gaussian process variational au- toencoders for neural and behavioral data, arXiv preprint arXiv:2310.03/11 2023.
Bugeon, S., Duffield, J., Dipoppa, M., Ritoux, A., Prankerd, I., Nicoloutsopoulos, D., Orme, D., Shinn, M., Peng, H., Forrest, H., et al. A transcriptomic axis predicts state modulation of cortical interneurons. Nature, 607(78)8): 330-338, 2022.
Gosztolai. A., Peach, R. L.. Arnaddon, A., Barahona, M., and Vandergheynst, P. Marble: interpretable represen- tations of neural population dynamics using geometric deep learning. Nature Methods, pp. 1-9, 2025.
Chaudhary, U., Birbaumer, N., and Ramos- Murguilaur,,,,,,,y A. compute interfaces for communication and rehabilitation. Nature Reviews Neurology, 12(9):513- 525, 2016.
Herrero-Videe P., Rinberg, D., and Savin, C. Across-animal odor decoding by probabilistic manifold alignment. Ad- vances in Neural Information Processing Systems, 34: 20360-20372, 2021.
10

---

Neural Representational Consistency Emerges from Probabilistic Soural-Behaviory Representation Alignment
Higgins, I., Matthey, L., Pal, A., Burgess, C. P.,. Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In The Eighth International Conference on Learn- ing Representations, 2014.
Laboratory, B., Benson, B., Benson, J., Birman, D., Bonacchi, N., Carandini, M., Catarino, J. A., Chapuis, G. A., Churchland, A. K., Dan, Y., et al. A brain-wide map of neural activity during complex behaviour. Biorxiv, pp. 2023-07, 2023.
Hochberg, L. R., Bacher, D., Jarosiewizz, B., Masse, N. Y., Simeral, D., Vogel, J., Haddadin, S., Liu, J., Cash, S. S., Van Der Smagt, P., et al. Reach and grasp by people with tetraplegia using neurally controlled robotic arm. Nature, 485(7398):372-373 2012.
Lavoie, S., Kirichenko, P., Ibrahim, M., Assran, M., Wilson, A. G., Courville, A., and Ballas, N. Modeling caption diversity in contrastive vision-language pretraining. arXiv preprint arXiv:2405.00744 2024.
Hubel, D. H., Wiesel, T. N., et al. Receptive fields of single neurones in the cat's 's striate cortex. J physiol, 148(3): 574-591, 1959.
Hurwitz, C., Srivastava, A., Xu, K., Jude, J., Perich, M., Miller, L.., and Hennig, M. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 34:29379-2998 293922, 2021.
Le, T. and Shlizerman, E. Stndt: Modeling neural popula- tion activity with spatiotemporal transformers. Advances in Neural Information Processing Systems, 35:17926- 17939, 2022.
Johnson, M. J., Duvenaud, D. K., Wiltschko, A., Adams, R. P., and Datta, S. R. Composing graphical models with neural networks for structured representations and fast inference. Advances in Neural Information Processing Systems, 29, 2016.
Liu, R., Azabou, M., Dabagia, M., Lin, C.-H., Ghesh- laghi Azar, M., Hengen, K., Valko, M., and Dyer, Drop, swap, and generate: A self-supervised approach for generating neural activity. Advances in neural infor- mation processing systems, 34:10587-10599, 2021.
Joy, T., Shi, Y., Torr, P. H., Rainforth, T., Schmon, S. M., and Siddharth, N. Learning multimodal vaes through mutual supervision. arXiv preprint arXiv:2106. /2570, 2021.
Liu, R., Azabou, M., Dabagia, M.. Xiao, J., and Dyer, Seeing the forest and the tree: Building representations of both individual and collective dynamics with transform- ers. Advances in Neural Information Processing Systems, 35:2377-2391, 2022.
Karush, W. Minima of functions of several variables with inequalities as side constraints. M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939.
Madisen, L.., Gamner, A. R., Shimaoka, D., Chuong, A. S., Klapoetke, N. C.,, Li, L.,, Van Der Bourg, A., Niino, Y., Egolf, L., Monetti, C., et al. Transgenic mice for intersec- tional targeting of neural sensors and effectors with high specificity and performance. Neuron, 85(5):942-958, 2015.
Kasthuri, N., Hayworth, K. J., Berger, D. R., Schalek, R. L., Conchello, A., Shooles--Brleyy S., Lee, D., Vázquez- Reina, A., Kaynig, V., Jones, R., al. Saturated reconstruction of a volume of neocortex. Cell, 162(3):
648-6614 2015.
Marder, E. and Goaillard, J.-M. Variability, compensation and homeostasis in neuron and network function. Nature Reviews Neuroscience, 7())56335574 2006.
Kell, A. J., Yamins, D. L., Shook, E. N., Vorman-Haigners, S. V. and McDermott, J. H. A task-optimized neural net- work replicates human auditory behavior, predicts brain responses, and reveals cortical processing hierarchy. Neuron, (33))6300-644 2018
Marks, T. D. and Goard, M. J. Stimulus-dependen repre- sentational drift in primary visual cortex. Nature commu- nications, 12(1):5169, 2021.
Keshtkaran, M. R., Sedler, A. R., Chowdhury, R. H., Tandon, R., Basrai, D., Nguyen, S. L., Sohn, H., Jazayeri, M., Miller, L. E., and Pandarinath, C. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19 (12):1572-1577, 2022.
Mathis, M. W. Rotondo, A. P., Chang, E. F., Tolias, A. S., and Mathis, A. Decoding the brain: From neural represen- tations to mechanistic models. Cell, 187(21):58145583) 2024.
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint
Khilkevich, A., Lohse, M., Low, R., Orsolic, I., Bozic, T., Windmill, P., and Mrsic-Flogel, T. D. Brain-wide dynamics linking sensation to action during decision- making. Nature, pp. 1-11, 2024.
2018.
Melbaum, S., Russo, E.,, Eriksson, D.,, Schneider, A., Durste- witz, D., Brox, T., and Diester, Conserved structures of

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
neural activity in sensorimotor cortex of freely moving rats allow cross-subject decoding. Nature Communica- tions, 13(1):7420, 2022.
correlates of behavior without behavioral measurements. Nature communications, 10(1):4745, 2019.
Russo, A., Bittner, S. R., Perkins, M., Seely, J. S., Lon-
Moses, D. A., Metzger, L.., Liu, R., Anumanchipalli, G. K., Makin, J. G., Sun, P. F., Chartier, J., Dougherty, M. E., Liu, P. M., Abrams, G. M., al. Neuropros- thesis for decoding speech in paralyzed person with anarthria. New England Journal of Medicine, 385(3): 217-227, 2021.
don, B. M., Lara, A. H.,, Miri, A., Marshall, N. J., Kohn, A., Jessell, T. M., et al. Motor cortex embeds muscle-like commands in an untangled population response. Neuron, 97(4):953-66,, 2018.
Safaie, M., Chang, J. C., Park, J., Miller, L. E., Dudman, J. T., Perich, M. G., and Gallego, J. A. Preserved neural dynamics across animals performing similar behaviour. Nature, 623(7983):765-777 2023.
Musall, S., Kaufman, M. T., Juavinett, L., Gluf, S., and Churchland, A. K. Single-trial neural dynamics are domi- nated by richly varied movements. Nature neuroscience, 22(10):1677-168, 2019.
Sani, O. G., Abbaspourazad, H.,, Wong, T.T,, Pesaran, B., and Shanechi, M. M. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identi- fication. Nature Neuroscience, 24(1):140-149) 2021.
Niell, C. M. and Stryker, M. P. Modulation of visual re- sponses by behavioral state in mouse visual cortex. Neu- ron, 65(4):472-479 2010.
Oord, A. v. d., Li, Y., and Vinyals, Representation learn- ing with contrastive predictive coding, arXiv preprint arXiv:1807.0333 03748, 2018.
Sani, O. G., Pesaran, B., and Shanechi, M. M. Dissociative and prioritized modeling of behaviorally relevant neu- ral dynamics using recurrent neural networks. Nature neuroscience, pp. 1-13, 2024.
Pandarinath, C., O' Shea, J. Collins, J., Jozefowicz, R., Stavisky, S. D., Kao, J. C., Trautmann, E. M., Kaufman, M. T., Ryu, S. I., Hochberg, R., al. Inferring single- trial neural population dynamics using sequential auto- encoders. Nature methods, 15(10):805881) 2018.
Schneider, S., Lee, H., and Mathis, M. W. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):300336. 2023.
Siegle, J. H., Jia, X., Durand, S., Gale, S., Bennett, C., Graddis, N., Heller, G., Ramirez, T. K., Choi, H.,, Luviano, J. A., et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 592(7852).86-92, 2021.
Patriarchi, T., Mohebi, A., Sun, J., Marley, A., Liang, R., Dong, C., Puhger, K., Mizuno, G. O., Davis, M., Wilt- gen, B., et al. An expanded palette of dopamine sensors for multiplex imaging in vivo. Nature methods, 17(11): 1147-1155, 2020.
Steinmetz, N. A., Zatka- Haas, P., Carandini, M., and Harris, D. Distributed coding of choice, action and engage- ment across the mouse brain. Nature, 576(7786):266 273, 2019.
Peters, A. J.. Chen, S. X., and Komiyama, T. Emergence of reproducible spatiotemporal activity during motor learn- ing. Nature, 510(7504):263-667 2014.
Ponce-Alvarez, A., Thiele, A., Albright, D., Stoner, G. R., and Deco, G. Stimulus- dependent variability and noise correlations in cortical mt neurons. Proceedings of the National Academy of Sciences, 110632):1312--1311,, 2013.
Stringer, C. and Pachitariu, M. Analysis methods for large- scale neuronal recordings. Science, (866612))))))) 2024.
Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., and Harris, K. D. Spontaneous behaviors drive multidimensional. brainwide activity. Science, 364 (8437)xcaav77893 2019.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp.
Stringer, C.,, Michaelos, M., Tsyboulski, D., Lindo, S. E., and Pachitariu, M. High-precision coding in visual cortex. Cell, 184(10):2767-777) 2021.
PMLR, 2021.
Renart, A., De La Rocha, J., Bartho, P., Hollender, L., Parga, N., Reyes, A., and Harris, K. D. The asynchronous state in cortical circuits. science, 327(5965::87-559.. 2010.
Sussillo, D., Stavisky, S. D., Kao, J. C., Ryu, S. I., and Shenoy, K. V. Making to future neural variability. Nature communications, 7(1): 13749, 2016.
interfaces robust
Rubin, A., Sheintuch, L., Brande-Eiile N., Pinchasof, O., Rechavi, Y., Geva, N., and Ziv, Revealing neural
12

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Turishcheva, P., Fahey, G., Vystrčilová, M., Hansel, L., Froebe, R., Ponder, K., Qiu, Y., Willeke, K. F., Bashiri, M., Baikulov, R., et al. Retrospective for the dynamic sensorium competition for predicting large-scale mouse primary visual cortex activity from videos. ArXiv, pp- arXiv-2407, 2024.
Zhu, F,, Grier, H. A., Tandon, R., Cai, C., Agarwal, A., Giovannucci, A., Kaufman, M. TT, and Pandarinath, C. A deep learning framework for inference of single-trial neural population dynamics from calcium imaging with subframe temporal resolution. Nature neuroscience, 25 (12):1724-173, 2022.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vargas, A. M., Bisi, A., Chiappa, S., Versteeg, C., Miller, L. E., and Mathis, A. Task-driven neural network models predict neural dynamics of proprioception. Cell, 187(7): 1761, 2024.
Walker, Y., Sinz, F. H., Cobos, E., Muhammad, T., Froudarakis, E., Fahey, G., Ecker, A. S., Reimer, J., Pitkow, X., and Tolias, A. S. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, (2112):2060-200 2065, 2019.
Wang, Y., Li, C., Li, W., and Wu, A. Exploring behavior- relevant and disentangled neural dynamics with genera- tive diffusion models. arXiv preprint urX1v:2410.09614, 2024.
Yamins, D. L. and DiCarlo, J. J. Using goal-driven deep learning models understand sensory cortex. Nature neuroscience, 19(3):356-36,, 2016.
Ye, J. and Pandarinath, C. Representation learning for neural population activity with neural data transformers. arXiv preprint ar arXiv 100.0/2.///20 10, 2021.
Ye, J., Collinger, J., Wehbe, I.., and Gaunt, R. Neural data transformer 2: multi-context pretraining for neural spik- ing activity. Advances in Neural Information Processing Systems, 36, 2024.
Yoshida, T. and Ohki, K. Natural images are reliably repre- sented by sparse and variable populations of neurons in visual cortex. Nature communications, 11(1):872, 2020.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pp. 11975 11986, 2023.
Zhang, Y., Wang, Y., Benetó, D. J., Wang, 7.,, Azabou, M., Richards, B., Winter, O., Laboratory. T. I. B., Dyer, E., Paninski, L., et al. Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution. ArXiv, pp. arXiv-2407, 2024.
Zhou, D. and Wei, X.-X. Learning identifiable and inter- pretable latent models high-dimensional neural activity using pi-v Advances in Neural Information Processing Systems, 33:7234-7224 2020.
13

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Supplementary Materials
In the following sections, we provide a comprehensive analysis of cross-modal representation learning approaches for neural-behavioral data integration:
Section A. Analysis of Cross-modal Representation Alignment Methods.
Section B: Generatively Informed Neural-Behavior Alignment Framework. Our novel framework with theoretical
guarantees for representation stability and information preservation, including detailed ELBO derivations.
Section C: Practical implementation across three neurophysiologica datasets (M1, PMd, V1), covering dataset organization, network architecture, and training specifications.
-Hiddl Section Potential related works, including SwapVAE(Liu et al., 2021), amLDS (Herrero- Vidal et al., 2021) and MARBLE((Goszolaa al., 2025).
A. Analysis of Cross-modal Representation Alignment Methods for Neural-Behavioral Alignment
In this section, we analyze cross-modal alignment methods from artificial intelligence that are adapted to neural-behavioral representation alignment. We examine contrastive learning approaches (CLIP (Radford et al., 2021), SigLIP (Zhai et al., 2023) and probabilistic matching (Chun, 2023; Chun et al., 2024)), evaluating their capabilitics in handling the inherent variability structure neural data.
A.1. Hierarchical Variability in Neural Responses
Neural population recordings exhibit systematic variability at multiple scales (Stringer et al., 2019; Musall et al., 2019), characterized by:
0
<
Euial
X session
subject
(13)
At the trial level, variability manifests through firing rate fluctuations within sessions (Churchland et al., 2010; Cohen & Kohn, 2011);
sup E,, [[|x], x12
(14)
The session level exhibits broader spike count variability patterns (Stringer et al., 2019; 2021):
sup Es s1, P(xly,
(15)
session
The largest variation occurs at the subject level through inter-individual differences (Russo et al., 2018: Gallego et al., 2020):
Subject
sup y
P(xl
(16)
This hierarchical structure indicates that neural representations maintain relative consistency within trials and sessions while exhibiting substantial variations across subjects, requiring alignment methods capable of handling multi-scale variability.
A.2. Analysis of Existing Methods for Heural-Behavior Representation Alignment
We analyze how existing cross-modal alignment methods handle the inherent hierarchical variability in neural data. Neural populations exhibit substantial trial-to-trial variability while maintaining task selectivity (Churchland al., 2010; Cohen Kohn, 201 1), posing unique challenges for representation alignment.
Existing approaches differ in their treatment of neural variability. CLIP (Radford et al., 2021) employs temperature-scaled contrastive learning:
1
B
exp(sim((f(x).xxxx)))) == (xp(sim(f(x)).0x))))/
B
(xp(sim(((xx)..xx))))/
LEUP =
log
log
(17)
B
B
1=1 == exp(sim(f(xx).. 9(x)))
1=1
14

---

Neural Representational Consistency Emerges from Probabillitic Neural-Bchavioral Representation Alignment
SigLIP (Zhai al., 2023) improved CLIP with sigmoid function:
Sig
1
log cxp((Tsim(f(xx)))) gyx)) +b
+
Σ
log
(18)
1 exp(rsim(f(xxx))(0)))
jE(B){(
Both methods assume direct one-to-one mappings between neural activities and behavioral variables. For neural responses {x+ M.K
1= associated with behavioral variable Ya. this creates competing objectives:
-11((x))) 9(yx)1l2 = O vs. :i.j:1I((x)) f(x))
(19)
Representation Collapse the ront-to-PoonttA
b.
Nara Begeomnake
Sizada Rigrovtaiee
Degenerated Representation the Encoder only Probalilistic Alignment
d.
142
PC3
6726560 200662
Neari Reveureatee Sttmeka Reprosnnicee
2330325
Viaal Spndia
Figure Representation degeneration in naive neural- neural-hehavione behavioral alignment methods. Three-dimensional PCA projections in demonstrate the learned emhedding distributions under SigI.IP-basd alignment, where neural activity encodings f(x) (circles) exhibit severe convergence degenerate point-mass distribution while behavioral variable encodings g(y) (triangles) maintain their distributed structure. The trial-wise cosine similarity matrix in b shows each entry (i, j) represents sim(f(x.), g )) for trials T, with statistical analysis (N=825 trials, two mice) revealing no significant discriminative structure (independent samples t-test: t=0.01, p=0.987> 0.05). Under probabilistic matching, PCA visualization in reveals the distributed structure of both neural and behavioral embeddings, with trial-wise similarity matrix showing consistently high correlation scores (sim([((x)) 9(y;)) 1) no statistical difference between matched and mismatched pairs (t=0.02, p=0.981> 0.05). When incorporating information bottleneck regularization (A = 10 10-*), the PCA projection in demonstrates maintained distributional structure altered geometric relationships, while the corresponding similarity matrix in f exxibits systematic negative correlations without achieving significant discrimination (t=0.03, p=0.997> 0.05).
Our analysis (Figure 8a-b) reveals this tension leads to representation degeneration, where neural encoders map different activities to nearly identical embeddings. Such point-to-poin correspondence assumptions pose unique challenges for neural datasets, fundamentally distinct from computer vision tasks due the inherent variability of neural recordings.
15

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Probabilistic matching (Chun, 2023; Chun et al., 2024) attempts to address this by modeling distribution-level alignment:
CrrobMatch
Σ
log exp( -T d(I +b)
1
log 1 exp(T d(f(x)))
(20)
o lla However, this formulation admits degenerate solutions where ([f(x))((y)) = s()) s(y) 12 + 11 fix) where both encoders map to constant values:
(x(x) R (y) ~ Zeonsts E
(21)
Our experiments (Figure 8c-f) confirm that even with information bottleneck regularization (Alemi et al., 2016), simple probabilistic matching remains prone to degenerate solutions, where neural representations become overly similar across distinct behaviors. The analyses demonstrate existing approaches face a fundamental limitation in their inability to simultancously maintain effective alignment while preserving behavior-wise discriminability. This inherent trade-off between structural preservation and discriminability motivates our development of a generative-informe framework.
B. Generatively Informed Neural-Behavior Alignment Framework
Neural-behavior alignment requires simultaneously preserving trial-specific neural information and maintaining behavioral consistency. Here, we present theoretical framework that addresses this challenge through generative modeling approach. Our framework not only establishes rigorous mathematical guarantees for the alignment process but also reveals key characteristics of preserved neural representations that mirror the hierarchical organization of neural systems.
B.1. Derivation of the Constrained Optimization Framework
To formalize our neural-hehavioral model, we propose a constrained optimization approach that balances probabilistic matching with generative consistency. This appendix provides a detailed derivation of our objective function.
We begin by formalizing our approach as a constrained optimization problem:
min ProhMatch
logp(x,, y) S C1
(22)
logply/x) S C2
logp(xly) S C3
Here, LProtMacch represents our primary objective function that optimizes the probabilistic matching between neural activity x and behavior y. The constraints enforce generative consistency by placing upper bounds cı, C2, and c3 on the negative log- ligeliiel of the joint distribution p(x,y) y) and the conditional distributions p(y[x) and p(xly), respectively.
To solve this constrained optimization problem, we apply the method of Lagrangian multipliers under the Karush-Kull Tucker (KKT) conditions (Karush, 1939). For inequality constraints of the form (0) < Ci, the KKT conditions require:
1. Stationarity: VoL(O,
2. Primal feasibility: (0) < for all
3. Dual fcasibility: A, 2 0 for all
4. Complementary slackness: (g) - a) = for all
In our case, the constraints are:
91 (0) logp(x, y) < Ci
(0) logply/x (x) < C2
(23)
(a) = log & C3
16

---

Neural Representational Consistency Emerges from Probabilistic Ceural-Behavioral Representation Alignment
The Lagrangian function is defined as:
C(O, A)
ProobMath +
Xx(a(()) ci)
(24)
Following the approach in (Higgins al., 2017), we consider the case where X, 0 and >> 0. When the constraints active (i.e., (i) Cj), the Lagrangian becomes:
Luaall PrrotMatch (1g1(0) + (0) + (0)
(25)
LrubMatch
y)) 22( logp(y/x)) + As log
The Lagrangian multipliers X, effectively control the strength of each generative constraint, allowing us to balance the probabilistic matching objective with generative requirements.
B.2. Properties and Guarantees
The framework achieves robust neural-behavion alignment through carefully designed optimization objectives that balance representation similarity with information preservation. We establish the following theoretical guarantees:
Theorem B.1. Let fe X + Z and 96 y + Z denote the neural and behavioral encoders. The following theoretical guarantees hold:
(i) [Non-degeneracy] The optimization objective Liinaa prevents representation degeneration by ensuring:
lim Luntal
400, Viecomst 2
(26)
(i) IIffrrmation Preservation] The generative constraints ensure enhanced mutual information between impul and represen- tation spaces:
[(f 2 I I Coarsaans (x); + 1 n>0
(27)
(iii) [Representation Stability] For neural responses x,, X,,X, corresponding to the same behavioral variable y, the learned representations maintain bounded distances:
0<as (1f((x)) So 3
(28)
Proof of (i): [Non-degeneracy]
Proof. We will proceed by contradiction to establish that any encoder mapping that collapses to a constant representation must yield an unbounded loss value.
Assume there exists degenerate solution where fo (x) zconst for all x E X. Since X is compact metric space and f8 continuous by construction, this degenerate mapping collapses uncountable set of distinct neural activities single point Zoonst in the latent space.
The generative component Lgea of our total loss involves a probability normalization constraint:
J ((x )dx
(29)
For this normalization constraint to hold while mapping all points in X' Zconss the conditional probability density p(x Zeonst) for any specific neural activity x, must necessarily approach zero:
p(x, Zeonst)
(30)
This follows from the fact that the probability must be distributed across the entire domain X' while maintaining the normalization constraint. Consequently, the log- lig-likeldhood term becomes:
log p( (eunns) - DX
(31)
17

---

Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment
Since total loss Fiotal includes the negative log-likelihood term with weight X2 >> 0:
Luttl - 22 logp(x(zz) - log p(x Zeana) + +00
(32)
This directly contradicts our objective of minimizing Ctotal. Therefore, for any constant representation Zennst E 2:
lim
Lioal +00
(33)
1o (x) Zeress
Proof of (ii): [Information Preservation]
Proof. We will establish that encoder optimized with our complete objective Liotal preserves strictly more information than an encoder optimized solely with LrrobMachh
Let us decompose the mutual information difference using the entropy formulation:
[f (x); - In (x);x) = (x) H (x (x))] - [H( - H ( (x))
(34)
H [H[ (x)) - (x.x
The baseline encoder /Ceemss is optimized solely by the probabilistic matching loss LprobMatch, which enforces distributional alignment between neural and behavioral representations. Critically, this objective imposes no explicit constraints on the conditional entropy. The encoder is free to discard information about x as long as the resulting distribution matches that of the behavioral representations. Thus, there exists a lower bound on the conditional entropy:
H (xx (x) 82
(35)
In contrast, the encoder fe.m optimized with our complete objective Ltotal incorporates the evidence lower bound (ELBO) through the generative loss component:
Bog pxx|2)] [I(g(Z)))]I[(z)] 2 -ō1
(36)
The reconstruction term Eo(zlx)llggp((lz)) directly imposes an upper bound on the conditional entropy:
H (x) (x) S
(37)
Since the ELBO constraint explicitly encourages accurate reconstruction through its log-likelihood term, while LprobMach lacks such reconstruction incentives, we necessarily have:
81 < 62
(38)
This establishes the existence of positive constant 77 82 6n > 0 such that:
I(fcrm (x): - I(f (CCommnn (x): 2 17 >>
(39)
Proof of (iii): [Representation Stability]
Proof. We will establish both upper and lower bounds on the distance between representations of neural activities that correspond to the same behavioral variable.
First, for the upper bound: Since X' is compact and fe is continuous, the image fa(X') is also compact and thus bounded in Z. Let M = supxex fx()2 denote the maximum norm of any representation. Then, by the triangle inequality, for any X,, X2 E X:
[fo(x;) fo(x))12 S (x)( ()o(xx) S 2M =
(40)
18

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Aligument
For the lower bound, we proceed by contradiction. Suppose no positive lower bound exists, i.e.:
Vo 0, Xy E d' with x; X3 (f(x)) fo(x;))2
(41)
This implies the existence of sequence (xx (m) x (x) 0o variable y, such that:
n=1 of distinct neural activity pairs corresponding to the same behavioral
lim (fo(x(")) - fo((x)))) =
(42)
As + co, the representations become arbitrarily close. For any probability model satisfying the normalization constraint:
J (xx
(43)
The conditional probabilities must satisfy:
lim n-c
(igp(x("1
(osp(xx""\o((((]]]" -00
(44)
This is because the representations become identical, the probability model must distribute finite probability mass between
distinct neural patterns, forcing at least one probability to approach zero. Consequently:
Liotal 2 -A2 (logpix,")
(") [o((x ())) + log ((x))(((("XX((""
- +0 00 n
(45)
This contradicts the minimization of Lroaa Therefore, there exists a >> 0 such that for all distinct neural activities x; # xj corresponding to the same behavioral variable y:
0 <<a S (fo(x;) - fo(x,)12
(46)
Remark B.2 (Optimization Balance). The proposed framework achieves representation stability through a balance in the optimization objective. This equilibrium emerges from two complementary mechanisms:
(1) Clustering Force: LPoobMatch functions clustering force, promoting representational similarity among neural responses associated with the same behavioral variables. This ensures behavioral consistency and bounds the maximum distance between related neural representations by 3.
Lgen serves as a regularizing force through its generative constraints. By maintaining minimum distinctiveness threshold a, preserves trial-specific neural information and prevents representation degeneration. Remark B.3 (Key Properties). The theoretical guarantees in Theorem establish three fundamental properties:
(2) Regularizing Force:
(1) Robustness: The framework systematically accommodates neural variability while preserving behavior-specific representation structure.
(2) Information Preservation: Essential neural response characteristics are retained through rigorous mutual information bounds.
(3) Representation Stability: Related neural responses maintain consistent yet distinct representations through provable distance constraints.
Remark B.4 (Characteristics of Preserved Neural Representations) Property (iii) reveals the intrinsic characteristics of preserved neural representations: while maintaining maximal similarity ($ 8), these representations retain inherent distinctiveness (2 a). This mathematical characterization aligns with the hicrarchical variability in biological systems, where neural responses exhibit both consistency and inherent variations across different experimental conditions.
19

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
B.3. ELBO Derivation in the Generative modeling.
Building upon the theoretical foundations established above, we now detail the optimization framework that realizes these properties. Our approach leverages bidirectional generative model structured around two key Markov chains: x - and y --x, x, where x, y, and represent neural activities, behavioral variables, and latent representations respectively, This bidirectional structure ensures both neural information preservation and behavioral consistency.
Behavioral Variable to Neural Activity, We first derive the ELBO for the forward path, examining how behavioral variables generate corresponding neural activities. This derivation provides two key bounds: one for the joint probability p(x, y) and another for the conditional probability p(x
log y(x,y) = log p(x, y,
log
p(xlz)p((lx)pp()ddz
p(x(z)p(zly)p(() (z)(,,, y)dz
log
g(z(x,y)
(47)
p( ((x)))(Z)))p()))
2 (z)x,))) Dlo
(z)
p(zly)
(x/x,)) log + [z/x,,)) g(z y) + log p(y)
/ [ogg((((z) LL (efgz(x.y))lppe x)llp() + logp(y)
where p(y) is treated as constant during optimization since it represents the prior distribution of behavioral variables that is intractable. Furthermore, we can also have another generation constrain from the conditional probability p(xly):
/ p(x, 2xx,2))dz log / p(xlz)p(zly)ddz
logp(xly) log
log
p(x(z)p(z(x)dz (Since p(z|x) and p(zly) are expected to be highly similar)
log
/
p(x(z)p(z)
(z(l))
dz
(48)
(z(l))
p(z|x)
log (egltt (p(x/
(z
Eg (ogg(//))) - K KLlg(z)))ll((x/x)
Neural Activity to Behavioral Variable The reverse path analysis considers how neural activities encode behavioral variables, completing our bidirectional framework. Similar to the forward path, we derive bounds for both the joint probability y) and the conditional probability p(y[x).
log y(x,y) ) = log / p(x, y,
log
p p(yla))((2xx))(x)dd
p(xx1))p(2))))(() x)
log
g(z(x,y)
(49)
plyl2)p(z(x)x(x)
(2(2/xy)) Dlg ( (z)xx,))
p(z2x)
Ey(x (log:s (z)] + E Eg(2 x.y
+ log p(x)
g((xx,))
= 1 (lggpylzz) - KL[g(z|X,, x\)pzz + lx)
20

---

Neural Representational Consistency Emerges from Probabilistic Neural-Bchavioral Representation Alignment
Table Monkey MI Dataset Organization and Recording Sessions. The dataset comprises recordings from 4 monkeys performing center-out (CO) reaching task. Sessions are split into training (24 sessions). validation (4 sessions), and test sets (6 sessions), reflecting the consistent nature of neural responses in this well-esstblished behavioral paradigm.
Split
Subject
Session Identifiers
-00000031003,, C-CO-2013100 C-CO-2013121D 1-30-20150312,
Training
Monkey C
C-CO-20150700 1-00000150715, L-CO-20151 C-CO-20151117,
C-CO-20151201, C-CO-20160912, C-CO-20160929,
M-CO-20144205, M-CO-20140007,, M-CO-20140626, M-CO-20140922,
Monkey M
M-CC--20141203, M-CC-20155052,, M-CO-20150610, M-CO-201506155
M-CCO-20150617, M-C0-200150623, M-CO-201106255, M-CO-20150626
Monkey C
C-CO-20131203,, C-CO-20160921
Validation
Monkey M
M-CO-20140011, M-CO-20150616
Monkey T
T-CO-20130819, T-CO-20130909
Test
Monkey
J-CO-20160405, J-C--0010040
Similarly, in the backward path derivation, log ((x) represents the prior distribution neural activities and is also treated as a constant during optimization.
And for the conditional probability p(y[x):
/ ply, x/) log / p(x|k)p(z(x)dz
(x) = log
/ ply p(ylz)p(zly)da
ss log
(Since p(z(x) and p(zly) are expected to be highly similar)
log / plylz)p(zly) ((2(x)
(50)
(z(xx) dz
log (p( p(zly)
( x
(ggz - KLLg(z/x)]lp(zly)
Note that fundamental assumption underlying our derivation is the convergence of latent distributions p(z|x) and p(zly) in the learned representation space. This alignment is enforced through probabilistic matching objectives, ensuring that paired neural activities and behavioral variables share similar distributional properties in the latent space. Such alignment is crucial for establishing stable bidirectional mappings between neural and behavioral spaces while preserving their respective information content.
C. Implementation Details
C.1. Dataset Description
To comprehensively validate approach, conducted experiments three representative neurophysiological datasets that capture distinct neural encoding paradigms: motor cortex datasets (MI and PMd) for movement encoding, and visual cortex dataset (V1) for sensory processing.
Motor Cortex Datasets (M1 and PMd). We analyzed neural recordings from primary motor cortex (M1) and dorsal premotor cortex (PMd) of rhesus macaques performing a center-out reaching task, examining how these distinct motor areas encode planned movements. The MI dataset comprises recordings from monkeys (~-64 neurons/session)) while the PMd dataset includes 3 monkeys (~96 neurons/session), both collected using chronic multielectrode arrays. Neural activity paired with continuous 2D hand kinematics (IR:: position and velocity). As detailed in Tables 2 and structured these datasets to enable rigorous evaluation of cross-subject generalization. For M1, we used 24 training sessions and 4 validation sessions from two monkeys (C and M), with 6 test sessions from two held-out monkeys (J and T), The PMd dataset organized similarly with 20 training sessions and validation sessions from monkeys C and M, and 4 test sessions from monkeys M and T, maintaining parallel evaluation structures across both motor areas while accommodating the available recordings.
21

---

Neural Representational Consistency Emerges from Probabillstic Neural-Bchavioral Representation Alignment
Table Monkey PMd Dataset Organization and Recording Sessions. The dataset comprises recordings from monkeys (C. M, and T) performing center-out (CO) reaching task. To align with the MI experimental setup (which uses 4 monkeys), we treat two sessions from Monkey M in the training set as if they were from a separate monkey for zero-shot evaluation purposes. Sessions are split into training (20 sessions), validation (4 sessions), and test sets (4 sessions), maintaining consistency with the MI experimental paradigm while accommodating the available PMd recordings.
Split
Subject
Session Identifiers
C-CO-20160900 C-CO-20160912, C-CO-20160914, C-C0-20100111
Training
Monkey C
C-CO-20160919. C-CO-20160921, C-CO-20060029, 0.000000000.....0000000000001150
C-CO-201610077. C-CO-20161011
AACO-20140205, H-CO-20140218,, M-CO-20140304 M-CO-20140307
Monkey M
M-CO-20140929, M-C0-20141203, ACO-20150512,
CO2-000000,,,
M-CO-20150611, M-CO-20150615
Monkey C
C-CC0-20161013, C-CO-20161021
Validation
Monkey M
M-CO-20150616. M-C0-20150617
Monkey M
M-CO-20150623, M-CO-20150625
Test
Monkey T
3-C0-201 T-C0-20130903
Table 4. Dataset Organization and Cross-Mouse Video Stimulus Pairings. The experimental dataset contains recordings from 10 mice with an 8/2 train validvaliittion split. Each mouse assigned unique identifier and systematically paired with another mouse that viewed matching vidco sequences, enabling direct comparison of neural responses to identical visual stimuli across different subjects.
Split
Mouse ID
Dataset Identifier
Paired Mouse
Mouse
dynamic291 1-
Mouse 5
Mouse
d
Mouse 6
Mouse
Mouse
Mouse
Mouse &
Training
Mouse 5
dynamic29515-10- 10mamii295155-001112-ddoo------9
Mouse I
Mouse
Mouse 2
Mouse 1ymmmic29647-19-88
Mouse
Mouse
Mouse 4
Mouse dynamic29228-2-1
Video Suddec/-ff4eeeeee22b44eeelloolo-
Mouse
Validation
Mouse 10
dynamic29755-2 www.ab/bbbbllaaaa111111100000000
Mouse 9
Visual Cortex Dataset (V1). We analyzed two-photon calcium imaging data from the primary visual cortex of 10 head-fixed mice viewing naturalistic video stimuli, provided by the Sensorium 2023 Competition. This large-scale dataset captures the activity of 78,853 neurons (~88,000 neurons/mouse) during passive viewing of 36x64 pixel grayscale video sequences. As shown in Table 4, we employed an 8/2 train-validation split, with key experimental design feature: each mouse was paired with another subject that viewed identical video sequences. This systematic pairing enables direct comparison of neural representations across individuals while maintaining independent test sets for unbiased evaluation.
These datasets provide complementary perspectives neural information processing:
M1 and PMd recordings reveal how different motor areas encode planned movements, with PMd typically showing more complex preparatory dynamics
V1 recordings demonstrate how sensory information is processed across a large population of neurons, with natural variability in responses across subjects
All datasets feature carefully structured train/validation//es splits that enable rigorous assessment of cross-subject generalization
C.2. Training Details
The neural characteristics and computational requirements of motor cortex recordings (M1, PMd) and visual cortex data (V1) required distinct training protocols. Motor cortex datasets feature relatively sparse neural populations paired with
22

---

Neural Representational Consistency Emerges from Probabillitic Neural-Bchavioral Representation Alignment
Table Training configurations on three neural datasets. Our framework utilizes distinct optimization strategies for monkey motor cortex (MI and PMd) and mouse visual cortex (V1) tasks, with hyperparameters and computational resources tailored to each dataset's s characteristics.
Monkey Motor Cortex (M1 and PMd)
Mouse Visual Corlex (V1)
Optimization Settings
Optimization Settings
Optimizer
Adamw (81 0.9, & 0.999)
Optimizer
A (81 0.9, 8 = 0.999)
Weight Decay
0.05
Weight Decay
0.05
Butch Configuration
samples, 16 timebins per batch
Batch Configuration
32 sumples, 16 timebins per batch
Learning Rate Schedule
Learning Rate Schedule
Initial Learning Rate
1 x 10
Initial Learning Rate
10-1
Wa Period
First 600 iterations
Warm-up Period
First 600 iterations
Peak LR Duration
50 epochs
Peak LR Duration
200 epochs
Decay Strategy
Cosine anncaling x 10-7 (last 25 epochs)
Decay Strategy
Cosine annealing 1o1 x 10-7 (last 200 epochs)
Total Epochs
100
Total Epochs
400
Data Computing Specifications
Data & Computing Specifications
GPU Configuration
1x NVIDIA A100 (40GB)
GPU Configuration
4x NVIDIA A100 (40GB)
Training Duration
~1 hour
Training Duration
~ hours
Inference Latency
0.3ms
Inference Latency
1.4ms
Neuron Number
-4 neurons for MI, ~-96 neurons for PMd
Neuron Number
8000 neurons
Behavioral Features
2D position and 2D velocity (It)
Visual Stimulus
gray 36x64 image (R1x16664)
kinematic features, while visual cortex data comprises dense neuronal recordings coupled with nigh-dimensional visual inputs. We established comprehensive training configurations to address these dataset-specifie demands, with detailed optimization parameters, learning schedules, and computational specifications presented in Table 5,
C.3. Evaluation Metric
We employ Pearson correlation coefficient evaluate representation similarity. First, we measure cross-modal representation correlation. For neural activity representation Zn and behavioral representation Zy. both having identical dimensions d x T, the Pearson correlation coefficient is computed as follows:
dxT Lial (Zu,i Zxx (zy En)
r r(Zz, Zy)
(51)
dxT == (Zn,i _²)² dxT (Zy,i E2y)
where ž and Zy,i denote the i-th element of the flattened representations 2n and Zy, respectively, and E and By represent their respective means.
Additionally, we explore fithin-modalics representational similarity by calculating Pearson correlation coefficients between neural activity representations z and 2i. from different trials. It is important to note that during model training, utilized fixed time windows across all trials, preserving complete temporal information for model input. This time-window approach ensured that information was lost during the training phase.
For evaluation purposes, however, we needed to address the variability in temporal dimension T across trials specifically in the PMd and MI datasets. these cases, we aligned the time axes by uniformly subsampling the sequences to the minimum length among the compared trials. This subsampling procedure was not necessary for the mouse VI dataset, which exhibited consistent temporal dimensions across trials.
After obtaining representations of matching dimensions where required, we computed the Pearson correlation coefficient using the same formula:
Ek=1 (2 2 (" zz'
dxeTmse
r(Z 24))
(52)
Σ d Tran k=1 (2
22022d1 k=
dxTmmn (2)"
We determined that selectively resampling at the representation level during evaluation was preferable to full sequence truncation, it minimized information loss and reduced potential error propagation. This selective approach to temporal
23

---

Neural Representational Consistency Emerges from Probabillstic Neural-Bchavioral Representation Alignment
Table Network Architecture Specifications. Detailed hyperparameters for Monkey MI, PMd and Mouse implementations. The architecture maintains temporal coherence while processing different input modalities.
Hyperparameters
Monkey M1/PMd
Mouse VI
Projection Head Specifications
Input Channels
1
Output Channels
128 256
Pooled Dimension (D)
8
Encoder Specifications
Base Channels (ch)
32
64 (1,2,2,4)
Channel Multipliers
(1.2,2.4)
Resolution Levels
4
4
Residual Blocks per Level
2 [8,16] 0.
Attention Resolutions
[2.16]
Dropout Rate
0.
Data Dimensions
Neural Activity
N x T x
N x T 36 64 T
Behavior/Stimulus
Pre-defined Low-dimensional Latent space
Latent Space
1 x
4 32 x
Reshaped Latent Space
4 T
alignment during evaluation, combined with complete information preservation during training, enabled robust quantification of representational similaritics across different experimental conditions.
C.4. Network Architecture
Our encoder-decoder architecture builds upon UNet, leveraging its proven capability in multi-scale feature extraction while introducing critical modifications for temporal neural data processing. Based on the UNet 2 implementation, we developed a framework specifically optimized for multi-mon modal neural recordings.
The architecture processes neural activity (N T') single-channel temporal sequences, where N represents the number of neurons and T denotes the temporal dimension. To accommodate varying neural population sizes across recording sessions, we implemented learnable projection heads at both network endpoints. The input projection transforms single-channel data through 2D convolution operations (3x3 kernel, stride 1), followed by AdaptiveAverraeePoling that standardizes the neuron dimension to a fixed size D. This standardization maps the input dimensions from N x T to 32 x 8 x T for M1 data and 128 x 256 T for V1 data, as detailed in Table 6. The output projection mirrors this structure but employs bilinear interpolation for dimension restoration x N T.
The core network comprises 4-level hierarchical structure with distinet configurations for M1 and V1 datasets. The MI implementation utilizes 32 base channels, while V1 employs 64 base channels, both following identical channel multiplier patterns. Each hierarchical level incorporates two residual blocks featuring group normalization and timestep embeddings, with self-attention mechanisms strategically placed at specific resolutions to capture Long-range dependencies while maintaining computational efficiency.
To ensure consistent representation across modalities, we developed parallel encoders for behavioral and visual data. The behavioral encoder processes MI kinematic features (I1x4xT) by adapting the neural UNet architecture, removing spatial sampling operations while preserving the essential residual blocks and attention mechanisms. For V1 visual stimuli (R (x36666)XX") we extended the architecture with 3D convolutions (5x3x3 kernels) to effectively process spatiotemporal data, carefully compressing the stimulus representation to match the neural encoder's latent space.
The architecture converges all modalities into aligned feature spaces through reshaping operations, transforming the representations from e x x (c x d) x T. This unified approach yields compact latent representations of 4 x T for MI and 32 x T for V1 data, as specified in Table 6. This design significantly reduces computational complexity while preserving the temporal dimension throughout all processing stages, enabling robust capture of temporal dependencies and
hhttps://gi thub. com http:///....coooooaaaaaaaaaaaaaa
htt.://aw..ooooooaaaeeeeeeaooate s/unet.
24

---

Neural Representational Consistency Emerges from Probabilistic Soural-Bchaviora Representation Alignment
Distribution of Representation (MI. PMd)
Representation Alignment (MI)
b.
Naxal Acticie Byton
Noral Ietarie
Ncand Reprostratios
400
105
10.50
001
0.96
064
06
0.52
000
08 11
02
133
D4
155
10
D7
2nRrred Reproentaics Valao
supered Rquoocaatinn Valacs
Figure 9. Neural-behavioral representation alignment in motor cortices. Distribution amalysis showing aligned representational characteristics between neural activities and behavioral measurements in PMd during center-out reaching. b, Corresponding distribution analysis in M1 demonstrating similar alignment properties. Trial-wise correlation analysis in M1 revealing distributed similarity structure across movement directions (t= 2.17, p = 3.4 x 10-2), characteristic of continuous reaching movements.
population dynamics for neural data analysis.
C.5. Implementation Details for Verifying Preserved Neural Representations
Verification of preserved neural representations necessitates rigorous identification of matched behavioral conditions across recording sessions. For motor cortical regions (M1, PMd). we quantified behavioral similarity using Pearson correlations between kinematic trajectories, with correlation threshold of R >> 0.9 defining matched trials. Visual cortical (V1) analysis leveraged the deterministic nature of stimulus presentations, where identical visual sequences directly established matched conditions. Statistical validation of preserved representations was performed using independent t-tests comparing neural activity correlations between matched and non-matched behavioral conditions.
D. Extended Results of Cross-modal Alignment and Neural Representation Analysis
Neural representation analysis across scales provides deeper insights into preserved neural representations. Here we present more analyses complementing the main results.
D.I. Neural-Behavioral Representation Alignment Results of Monkey M1 and PMd
We present comprehensive multi-modal alignment results from monkey motor cortical regions (MI and PMd) during center- out reaching tasks (Figure 9). Distribution analysis demonstrates that PNBA effectively captures aligned representational properties between neural activities and behavioral measurements in both areas. The overlapping distributions in PMd and M1 indicate successful alignment of neural population dynamics with behavioral kinematics, despite their distinct functional roles in motor control.
The correlation structure in MI reveals a continuous representational pattern across movement directions (t = 2.17, p 3.4 10-2). This distributed similarity intrinsic to center-out reaching movements (Georgopoulos et al., 1982), where adjacent directional movements share substantial kinematic features, particularly during movement initiation (Churchland et al., 2012). The gradual transitions in neural and behavioral representations across movement directions align with established findings of continuous rotational dynamics in motor cortical populations (Churchland et al., 2012; Russo et al., 2018).
D.2. Additional Preserved Neural Representation Analyses on Monkey PMd
Following our investigation of preserved neural representations in MI, we conducted systematic analysis of representational stability in PMd across multiple organizational scales. This hierarchical analysis is particularly crucial for PMd given its central role in motor planning and preparation, where stable representations are essential for consistent movement execution.
Our analysis revealed robust preservation of neural representations at three distinct levels. At the finest scale, within-session trial-to-trial correlation analysis demonstrated high representational stability (mean R = 0.856 0.012, r', =109, nM, =95,
25

---

Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment
Preserved Neural Representations in Across Scales
a.
Cross-trial
0.90
Correltion
0.85
0.80
-.1
0.75
1-28
p=1.5
1m3.2
1-233
p-4.7x
p-7.0 10"
0.70
Monkey M 2015-06-25
Monkey 2015-06-25
Monkey 3-08--0-
Monkey 3-13--9-03
b.
C.
Cross-session
Cross-subject
0.95
0.95
Correlation
0.90
0.90
0.85
0.80
0.80
0.75
111
1-27
R0- 10
0.75
3.5
p 3,9-10
0.70
0.70
Monkey
Monkey
Cross-oubjoct
Figure 10. Hierarchical organization of preserved neural representations in monkey dorsal premotor cortex (PMd). Neural representation similarity analysis demonstrates systematic preservation across organizational levels. a, Within-session trial-to-trial correlations exhibit high consistency (mean R 0.856 = 0.012, YM, =109, =95, #1,=116, =87 trials). b, Neural representations maintain stability between recording sessions (R 0. 0.854 = 0.009). e, Zero-shot generalization analysis on held-out subjects validates cross-subject representation similarity (R = 0.856 # 0.072). Error bars denote standard deviation.
"r, =1,,116, =8, trials), indicating reliable encoding of motor plans across repeated behaviors. At the intermediate level, cross-session analysis showed remarkable consistency (R = 0.854 + 0.009), suggesting neural representation consistency underlying motor planning across different recording periods. Most critically, zero-shot generalization analysis on held-out subjects revealed significant cross-subject representational similarity (R = 0.856 0.072), demonstrating the preservation of fundamental representational features across different individuals.
D.3. Additional Preserved Neural Representation Analyses Mouse V1
To validate the robustness of preserved neural representations in VI, examined the cross-trial correlation patterns in zero- shot mice that were entirely held out during training. Figure la presents the correlation matrix for Mouse 9, demonstrating strong preservation of neural activity patterns across trials without any subject-seecific fine-tuning. The distinct block- diagonal structure indicates that PNBA successfully captures and maintains condition-specific neural representations in V1, with high correlations observed between trials of the same condition (self-correlation R = 1.0) and low correlations between trials of different conditions. Statistical analysis using independent samples t-tests between matched and mismatched trial pairs yielded p = 0., providing strong evidence for the significant preservation of neural representational structure at the sirale-trial level. Similar results were observed in Mouse 10 (Figure 1 1bb), confirming PNBA's consistent ability to generalize to completely unseen subjects. These results complement main findings by demonstrating that PNBA can effectively capture neural dynamics in novel subjects without any additional training adaptation.
E. Potential Related Works
This section addresses additional related works suggested during the review process. While these studies explore neural data analysis, they address fundamentally different research questions than our PNBA framework.
Liu et al. (2021)(Liu al., 2021) proposed SwaPVAE, self supervised approach for generating neural activity through data augmentation. SwapVAR operates only within the neural activity domain, employing augmentation, i.e., swap operation, based on trial similarity assumption without any behavioral constraints. In contrast, our PNBA framework explicitly bridges neural and behavioral domains through a CLIP-inspired multi-modal paradigm. PNBA directly aligns neural and behavioral representations via generative constraints, which enables zero-shot generalization to completely unseen subjects with varying neural population sizes- a capability that SwapVAR was not designed to address.
26

---

Neural Representational Consistency Emerges from Probabilistic Neural-Behavioral Representation Alignment
Preserved Neural Representations Existed in Mouse VI
b.
Zero-shot Mouse
Zero-shot Mouse 10
1,00
1:10
1736
1155
1234
000
Netral Repremmatioo
1142
1234
1234
1315
7256
1316
1386
1755
1387
Y251
T33
1366
T400
1815
1404
1952
140
1641
1737
Ropecacntations
Nami Reprsentarion
Figure 11. Cross-trial preserved neural representations in mouse primary visual cortex (V1). Correlation matrix for zero-shot mouse (Mouse 9) reveals robust preservation neural representational structure, with strong within-condition correlations (diagonal blocks, self-correlatio R 1.0. t=13.7715, p=0.) and weak ceneein-condition correlations). Independent samples t-tests between matched and mismatched trial pairs showed highly significant differences (p < 0.001), demonstrating PNBA's ability to maintain distinct neural patterns across experimental conditions without any training data from this subject. b. Similar preservation of condition-specffic neural representations was observed in another zero-shot mouse (Mouse 10), mean R = 1.0, t=121 4944, p=O., highlighting the model's s consistent generalization capability entirely unseen subjects.
Herrero- Vidal et al. (2021) Hirreero-Vidal al., 2021) developed an "aligned mixture of latent dynamical systems" (amLDS) for cross-animal odor decoding. Their approach explicitly maps neural recordings from different mice into a common latent manifold where neural trajectories assumed be similar across animals but distinet across odors. This method fundamentally begins with the assumption that shared neural encoding patterns exist across subjects in response to identical stimuli, and then builds alignment techniques based on this premise. In contrast, PNBA does not presuppose neural encoding similarity across cejjects-insteade our approach empirically tests whether such similarities exist by introducing behavioral constraints as the bridging element. While Herrero- Viiral et al. focus on stimulus-driven neural trajectories in olfactory processing, PNBA addresses the broader question of beurao-behavioral correspondence during complex, continuous behaviors. Interestingly, our findings provide empirical support for their underlying assumption by demonstrating that shared neural representations do indeed exist under similar behavioral conditions, but our approach arrives this conclusion without requiring starting assumption.
The recently published MARBLE method (Gosztolai et al., 2025) employs geometric deep learning to obtain interpretable latent representations from neural dynamics. MARBLE decomposes neural dynamics into local flow fields and maps them into common latent space based on user-defined labels of experimental conditions, allowing similarities between conditions to emerge. While MARBLE provides similarity metric between dynamical systems and can discover consistent representations across networks and animals, it remains fundamentally single- modal approach that works exclusively within neural space. In contrast, PNBA implements a distinetly multimodal strategy that directly incorporates behavioral data constraints in the latent space formation. This fundamental architectural domain geometric alignment versus PNBA's cross-modal neural- -behavioral Drigmment-reultts in approaches optimized for different objectives. MARBLE excels at capturing subtle changes in high-dimensiooal dynamical flows and relating them to task variables, while PNBA specifically addresses the correspondence between neural patterns and observable behaviors, providing framework to test whether neural representations preserve their behavioral meaning across subjects.
27


Reviews:
[{"review_id": "C0QYGs4aBn", "reviewer": "Reviewer_a8zJ", "summary": "This paper introduces PNBA, a framework leveraging probabilistic modeling to find robust preserved representations across different scales of neural variability: trials, sessions, and subjects. The method is evaluated on three datasets spanning M1, PMd and V1 of primates and mice, showing zero-shot preserved representations across cortices and species.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed feedbacks! We address each point as follows:\n\n```Q1. How does zero-shot generalization work with varying neuron counts across subjects?```\n\n**A1**: Our model uses convolutional and pooling layers to **standardize input activities of varying counts into a fixed-size latent space(detailed in SI Line 1088-1094)**. Based on **PNBA**, **training on multi-subject** further forces the model to learn behavior-relevant neural representations rather than individual-specific neuron characteristics, enabling cross-individual generalization without any fine-tuning on zero-shot subject.\n\n```Q2. Safaie et al, 2023 was not compared in Table 1.```\n\n**A2**: Table 1 excludes Safaie et al. because they focus on neural representation alignment requiring individual-specific optimization, whereas we evaluate cross-modal correlations on unseen subjects without fine-tuning. A direct comparison would be unfair to theirs.\n\n```Q3. Figure 3a: Was histogram from aggregated data of two mice?What do individual mouse histograms look like?```\n\n**A3**: Yes, the histogram aggregates data from 2 mice viewed identical stimuli. Individual mouse histograms show the same patterns for stimulus representation, with subtle peak variations in neural representation due to individual differences.\n\n```Q4. Figure 3a:What samples construct the histogram?How are normalized values computed and varying trial lengths/population sizes handled?```\n\n**A4**: The histogram uses latent representations z (d×T) from each trial, reshaped into one-dimensional vectors and z-score normalized. Neuron count variations are addressed by pooling neural features into a fixed-dimensional latent space (SI Line 1088-1094). Variable trial lengths are handled through sliding window (t=16) processing (SI Line 1067). All vectors across trials are then concatenated into a one-dimensional array, whose frequency distribution forms the histogram.\n\n```Q5. Figure 3b: Why is the correlation matrix not symmetric?```\n\n**A5**: The correlation matrix is asymmetric due to trial asymmetry. Specifically, corr_matrix[i,j] measures correlation between neural activity of trial i and visual stimulus of trial j, noted as pair (i,j), while corr_matrix[j,i] measures pair (j,i), a different combination, naturally yielding different values. Similarly in Figure 6c, pairs (i,j) and (j,i) represent different trial combinations.\n\n```Q6. Line 247: How are the 4 pairs chosen from the 8 mice?Shouldn't there be 8 choose 2 total number of pairs?```\n\n**A6**: We apologize for the confusion. Our experiment included 5 pairs of mice (10 mice total), with each pair viewing identical stimuli. 4 pairs (8 mice) were used for training, while the remaining pair (2 mice) served as an independent test set to evaluate zero-shot performance (see SI Table 4). We will refine this in the revised version.\n\n```Q7.  Figure 5a: is mean R calculated across all trial pairs regardless of behavior condition?How are different trial lengths handled?What is the standard deviation calculated over?```\n\n**A7**: No, we calculate mean R only between trials with matching behaviors, following Safaie et al, 2023. For trials with different lengths (d×T1 vs d×T2), e.g. T2>T1, we downsample T2 by uniformly discarding timepoints. This length handling only applies to monkey data, as all V1 trials have consistent lengths. \n\nStandard deviation is calculated across all possible same-behavior trial pairs. \n\nWe will add these clarifications in our revision.\n\n```Q8. Figure 5b and 5c: Similar question to the above.How are the mean R and standard deviation calculated for across-session and across-subject?```\n\n**A8**: We use identical strategy as **A7**, but comparing trials from different sessions or different subjects while maintaining matched behavioral conditions.\n\n```Q9. Section 4.5: How can the model handle zero-shot behavior decoding with varying neuron counts and time steps?What's the detailed training and inference procedure?```\n\n**A9**: Neural representations have the same spatial dimensions across subjects (**A1**). V1 trials have consistent lengths, but we can use sliding windows to handle varying lengths (SI Line 1067).\n\n**The full procedure:(1) train PNBA on training subjects (2) generate representations via the frozen neural encoder, validate preservation (3) train V1 BCI decoders using training subjects' representations and behavior (4) apply frozen neural encoder and BCI decoder to unseen subjects.**\n\n```Q10. Section 4.5: Could the authors also provide results of behavior decoding on the primate datasets?```\n\n**A10**: Our primate decoding results on unseen animals: M1 achieved R²=0.78; PMd achieved R²=0.71. We note that these results are expected as PNBA aligns neural-behavioral representations for these datasets. To avoid circular reasoning, we showed independent movement decoding using V1 data in Section 4.5, where the neural encoder was only trained with stimulus in PNBA.\n\n**Others**: We have fixed all typos.", "claims_and_evidence": "The paper claims to obtain robust neural-behavioral representation alignment within multiple cortical regions and from different species. This claim is convincingly supported by the Pearson correlation coefficients between neural and behavioral representations throughout the paper.\n\nThe second claim the paper made is on preserved neural representations through zero-shot validation with practical applications in zero-shot behavior decoding. However, it was not clear how this zero-shot generalization was achieved, given that each session/subject has varying number of neurons, making the application of the same encoder on the unseen session/subject without any finetuning impossible.", "methods_and_evaluation_criteria": "The choice of datasets (Safaie et al., 202, Turishcheva et al., 2024) and evaluation metrics made sense for this problem.", "theoretical_claims": "I have not carefully checked the correctness of proofs for Theorem 3.1 which was in the Appendix.", "experimental_designs_or_analyses": "The experimental designs and analyses look good generally, although some details are missing (see Questions for Authors)", "relation_to_broader_scientific_literature": "The paper made contributions toward representation learning in neuroscience, tackling the problem of identifying robust cross-modality representations across sessions and animals performing the same behavior tasks. The method has implications for calibration-free brain-computer interfaces with potential for motor function restoration.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n* The paper tackles an important problem in neuroscience, which is finding a robust representation amidst neural variability across trials, sessions and subjects.\n* Claims are supported with good experimental results.\n* The paper is well written and easy to follow.\n\nWeaknesses:\n* It is unclear how the model handles varying neural population sizes for zero-shot behavior decoding in unseen sessions/subjects.\n* The paper provides results of behavior decoding on V1 dataset but not M1 and PMd datasets.\n* The related alignment method in Safaie et al, 2023 was mentioned (Figure 1) but was not compared with the proposed method in Table 1.\n* Some details are missing that make it difficult to evaluate the soundness of the methods (see Questions for Authors)", "other_comments_or_suggestions": "Minor typo on line 381 and Figure 1c: \"trails\" instead of \"trials\"", "questions_for_authors": "1. Figure 3a: Was the histogram constructed using aggregated samples from two mice? I'm wondering what the histograms on each individual mouse look like.\n2. Figure 3a: What are the samples used to construct the histogram? Are the samples Nx1 vector at each timestep or NxT matrix of a trial/session? How are the normalized representation values computed from these vectors/matrices? How are trials of varying lengths and population sizes handled?\n3. Figure 3b: Why is the correlation matrix not symmetric?\n4. Line 247: How are the 4 pairs chosen from the 8 mice? Shouldn't there be 8 choose 2 total number of pairs?\n5. Figure 5a: Is mean R calculated across all possible pairs of trials regardless of which behavior conditions that trial belongs to? How to handle the problem that each trial can be of different lengths? What is the standard deviation calculated over?\n6. Figure 5b and 5c: Similar question to the above. How are the mean R and standard deviation calculated for across-session and across-subject?\n7. Section 4.5: How can the model be applied zero-shot for behavior decoding, given that the input to the model should be of fixed size while the number of neurons and time steps can vary across subjects? Could the authors provide the detailed step-by-step procedure of training and inference of the model?\n8. Section 4.5: Could the authors also provide results of behavior decoding on the primate datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Isu9GQrNFf", "reviewer": "Reviewer_ahz6", "summary": "This work proposed a probabilistic representation alignment framework PNBA that can be used to align neural activities and animal behaviors. The method is applied across brain regions, neural data modalities, and animal species. Authors provided extensive experimental evidence across multiple datasets, validating the robustness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback. We address each point as follows:\n\n```Q1. Line 382-384’s claim is confusing. How is the calcium imaging encoder trained?Is the zero-shot experiments still the cross-subject zero-shot experiments?Is the behavioral encoder frozen?If both encoders are, the evidence does not suggest broader preservation. ```\n\n**A1**: We appreciate the reviewer's careful examination.\n\na. Our conclusion summarizes preservation properties observed in three independent experiments (M1, PMd, V1) during zero-shot cross-subject conditions. We will clarify this in our revision.\n\nb. The calcium imaging encoder follows the same PNBA framework but with visual stimulus encoders. The zero-shot evaluations are also performed on unseen subjects without any fine-tuning or further alignment process.\n\nc.  During training, all encoders are optimized.\n\nd.  We understand the concern about preservation claims. However, our conclusion is supported by (1) consistent preservation observed in functionally different brain regions (M1, PMd, V1); (2) such validation are performed on unseen subjects without further finetuning or alignment. We believe this rigorous validation provides evidence for the preservation property.\n\n```Q2. The proposed matching objective in Section 3.1 is only one necessary. Is reversibility of encoders a sufficient condition for alignment?The theoretical guarantees seem too loose for sufficient alignment.```\n\n**A2**: We agree with the reviewer's insight. While reversible encoders would provide an optimal sufficient condition for alignment, this becomes impractical for neural data requiring dimensionality reduction. Our generative constraint approach offers a practical sufficient condition that balances theoretical guarantees with implementation feasibility, achieving effective representation alignment on multiple datasets without requiring strict encoder reversibility.\n\n```Q3. What assumptions underlie using Pearson correlation for alignment quality?(1)How is it applied to single-trial data?Is it per coordinate?Are vectors centered?(2)Did you try other metrics like L1 distance?Are all baseline methods optimized based on the given evaluation metrics? ```\n\n**A3**: The core assumption behind using Pearson correlation is that aligned neural and behavioral representations should exhibit similar distributional structures in latent space.\n\n(1) We compute correlation between complete neural-behavioral latent vectors for each trial, not coordinate-wise. Pearson correlation centers the vectors and normalizes by standard deviation, distinguishing it from cosine similarity.\n\n(2) We considered alternatives (L1, Euclidean) but selected Pearson correlation because it's established in neuroscience for representational similarity analysis (e.g., Safaie et al., Nature 2023) and effectively captures structural similarities in high-dimensional spaces. \n\nAll methods (baselines and our PNBA) were optimized using their original objective functions, not optimized the evaluation metrics.\n\nWe will include these details in our revision.\n\n```Q4. Limited details are provided in the main text regarding how to handle different neuron counts across sessions and how to transfer encoders given different input sizes. Are encoders re-trained when initialized on the data from a new animal? How do you deal with unseen neurons in new animals/sessions? ```\n\n**A4**: Thank you for highlighting this important issue. We addressed varying neuron counts through a neuron-adaptive encoder architecture incorporating projection layers with pooling layer (SI Line 1091-1093) to get unified dimensional latent space. This enables direct application to new animals with different input sizes.\n\nNo, encoders are not re-trained when applied to new animals.\n\nOur approach handles unseen neurons effectively because: (1) the neural encoder unifies the latent space dimensionality, and (2) based on PNBA, training across multiple subjects with diverse neural populations encourages extraction of behaviorally-relevant features rather than memorizing individual neuron characteristics, enabling generalization to completely new neural populations.\nWe will incorporate these details into the main text.\n\n```Q5. Suggested SwapVAE```\n\n**A5**: Thank you for suggesting this reference. While both approaches address neural representations, fundamental differences exist between SwapVAE and our PNBA framework. DropVAE operates within a single modality (neural spikes), using data augmentation and swap operations predicated on within-trial similarity, without directly incorporating behavioral constraints. In contrast, PNBA draws inspiration from CLIP's multi-modal paradigm, explicitly aligning neural and behavioral representations through generative constraints. This cross-modal approach directly establishes neural-behavioral associations, enabling zero-shot generalization to unseen subjects, as detailed in **A4**. We will add this in our related work.", "claims_and_evidence": "Line 382-384: \"The observation of preserved neural representations across both motor and visual cortices, despite their distinct functional roles and varying correlation strengths, suggests a broader preservation of neural coding structure.\" This claim is confusing, and seems like an over-claim: How is the calcium imaging encoder trained? Is the zero-shot experiments still the cross-subject zero-shot experiments? Is the behavioral encoder frozen? If both encoders are optimized based on the defined loss given by the authors, the experimental evidence does not suggest broader preservation of neural coding structure.", "methods_and_evaluation_criteria": "1. As stated in Section 3.1, the proposed matching objective between neural activities and behaviors is only one necessary condition for distributional alignment. It seems to me that one sufficient condition for alignment is that f and g both needs to be reversible. Is this true? Yet the provided theoretical guarantees seem to be very loose to provide a sufficient alignment.\n\n2. What is the underlying assumption of using Pearson correlation coefficient to evaluate the alignment quality? (1) Pearson Correlation is a method to evaluate the distributional similarity between two variables. In your case, how it is applied on single-trial data? Is it applied on each coordinate of a latent representation z? Are the latent vectors are centered (i.e. are you measuring cosine similarity)? (2) Have the authors tried other possible evaluation methods for alignment, e.g. L1 distance? Are all baseline methods optimized based on the given evaluation metrics?", "theoretical_claims": "I did not check carefully, as the theoretical bounds seem to be loose.", "experimental_designs_or_analyses": "This paper provides experimental results on three different datasets involving different modalities and brain regions. The experimental design is extensive, seem to be complete, and are of high-quality.\n\nHowever, the authors provided limited experimental details in the main text, which makes certain parts difficult to evaluate. For example, one common issue in spike neural data analysis is the difference between amount of neurons across sessions, and how to transfer encoder given different input sizes. How did the author address this issue? Are encoders re-trained when initialized on the data from a new animal? How did the authors deal with unseen neurons in new animals/sessions?", "relation_to_broader_scientific_literature": "The problem studied is a very important problem to the neuroscience community.", "essential_references_not_discussed": "Liu, Ran, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. \"Drop, swap, and generate: A self-supervised approach for generating neural activity.\" Advances in neural information processing systems 34 (2021): 10587-10599.\n\nThe proposed idea is actually very similar to the work pointed above. This work uses different encoders to encode neural activities of different animals, uses a behavioral-guided latent space, and uses generative loss to prevent latent collapse.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zPAWTTybWW", "reviewer": "Reviewer_PNyz", "summary": "Modeling shared variability across multiple animals is critical to understand universal principles of neural computation. Still, we like probabilistic tools to capture them into a singular representation. This work introduces a new probabilistic method to represent neural and behavioral variability across animals while allowing for individual variability. The authors validate their model in two different neural datasets, and compared the performance to relevant alternative models showing the applicability of their model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the positive assessment and constructive feedback. We address each point as follows:\n\n```Q1. Decoding behavior would better demonstrate captured behavioral information. Authors should discuss limitations on real-time inference and computational demands for BCI.```\n\n**A1**: Behavior decoding results are as follows: M1, R²=0.89, PMd, R²=0.83 (training subjects); M1, R²=0.78, PMd, R²=0.71 (testing subjects), indicating capture of behavioral information. These are expected as PNBA aligns neural-behavioral representations. To avoid circular reasoning, Section 4.5 presents movement decoding using V1 data, where the neural encoder was only trained with stimulus in PNBA, excluding movement data.\n\nFor BCI discussions, training takes ~1 hour (M1/PMd) and ~8 hours (V1), detailed in SI Line 1075, with inference requiring 0.3ms and 1.4ms, respectively (averaged over 1000 runs on an A100 GPU), showing potential for BCI.\n\nWe will add these discusions in our revision.\n\n```Q2. Simulations could reveal model limitations. Can the model generalize across missing behavioral conditions or recover ground truth parameters?How robust is it to latent space dimensionality, trial misalignment, or individual variability?```\n\n**A2**: Thanks for this suggestion. We agree with the usefulness of simulation. However, building simulations for neural-behavioral modeling with ground truth representation constraints is inherently difficult due to the complex, unknown nature of true neural encoding mechanisms. We therefore validated PNBA across three diverse real datasets.\n\nRegarding specific questions:\n\na. No,  PNBA cannot generalize to missing behavioral conditions in the raining. This is a current limitation, as discussed in Section 6.\n\nb. PNBA doesn't aim to recover ground truth parameters, as these are difficult to define in real neural data. PNBA is data-driven, learning effective representation alignment instead, similar to CLIP.\n\nc. We show robustness through theoretical guarantees (Theorem 3) and empirical validation. Figure 4b shows the results of varying latent space dimensions. Figures 3b,6c show effective handling of trial-to-trial variability in unseen subjects. \n\n```Q3. The authors missed a reference, Herrero-Vidal et al. NeurIPS 2021.```\n\n**A3**: Thanks for suggesting this work. \n\nHerrero-Vidal et al. assumes similar neural trajectories across individuals responding to identical stimuli so as to align **single-modal** neural activity through individual-specific optimization. In contrast, our **multimodal** neural-behavioral alignment doesn't presuppose neural encoding similarity, and can be tested across individuals. This methodological difference allows us to validate preserved neural representations in unseen subjects. \n\nWe believe our findings provide empirical support for assumptions in Herrero-Vidal et al., supporting shared neural representations under the same behavior. \n\nWe will add these discussions in our revision.\n\n```Q4. Compare with Procrustes alignment methods (Williams et al. 2021, Safaie et al. 2023).```\n\n**A4**: Thanks for this suggestion. We didn't directly compare with Williams et al. and Safaie et al. for several reasons:\n\na. Our work proposes cross-modal (neural-behavioral) alignment, while these methods focus on unimodal (neural-neural) alignment, making direct comparison potentially unfair.\n\nb. **Our method requires no fine-tuning on unseen individuals, whereas these methods require individual-specific optimization**.\n\nc. Different foundational assumptions (**A3**).\n\nWe will include these discussions in our revision.\n\n```Q5. Add a section discussing limitations.```\n\n**A5**: Limitations are currently discussed in Section 6, paragraph 1, noting PNBA cannot train with neural activity alone. We will add temporal dynamics discussion based on Reviewer ```DJVs```'s suggestion.\n\n```Q6. Compare with linear alignment methods regarding decoding performance, data demands, and computational costs for BCI applications.```\n\n**A6**: For computational costs and linear alignment comparisons, please see **A1** and **A4**. Regarding data requirements, PNBA is currently trained with 2 monkeys (M1/PMd) or 8 mice (V1), indicating moderate demands. While BCI applications aren't our current focus, our results show potential, and we plan dedicated BCI research in future work.\n\nWe will add these discussions in our revision.\n\n```Q7. How much data is needed for initial training? How does the transformation for handling different neuron counts work, and how much data is used for pre-alignment?```\n\n**A7**: Our model requires no pre-training, but trains directly on data from multiple individuals in the training set and directly tests on new individuals without fine-tuning.\n\nTo handle varying neuron counts, we use a **shared** convolutional projection head followed by pooling to only unify dimensions in our network. This requires no separate pre-alignment data or process, as it's learned directly during end-to-end training.", "claims_and_evidence": "The work is clearly motivated, supported by the theoretical proof and the presented results. The competitive performance results in comparison to alternative models further cements the relevance of the presented model. Moreover, they tested the model performance across recording modalities and species illustrating the broader applicability of their method. The author claim the relevance to BCI applications, still this requires additional constrains, including exploration of inference times, computational costs and data demands.", "methods_and_evaluation_criteria": "The methods are adequate and evaluation as a function of correlation between trajectories captures the shared variability across neural representations. The authors also showed minimal decoding performance to unseen observations. Still, since the emphasis of the work is to align to behavioral tasks, decoding with respect to behavior would provide a better sense of how much behavioral information is captured. To provide additional evidence on the use of the method for BCI, beyond just neuroscience discovery, the authors should discuss limitations on real-time inference and computational demands.", "theoretical_claims": "The theoretical proofs are clearly presented with enough detail and are correct.", "experimental_designs_or_analyses": "The authors evaluate their method on two neural datasets which amply shows the applicability of the work. Still, using simulation could further highlight uses and limitations of the model. For example, can the model generalize across missing behavioral conditions? Can the model recover ground truth parameters? How robust is the model to the dimensionality of the latent space, missing observations, trial misalignment, or individual variability? Moreover, adding behavioral decoding results across all the experiments and datasets would further show the ability of the model to extract those representations.", "relation_to_broader_scientific_literature": "The authors correctly frame their work around the relevant literature, they compared their model to alternative solutions. Still, the authors missed prior work introducing a probabilistic method for across-animal task-informed neural alignment (Herrero-Vidal et al. NeurIPS 2021). Additionally, the author could contrast their results with other simpler alignment methods based on Procrustes alignment (Williams et al NeurIPS 2021, Safaie et al. Nature 2023).", "essential_references_not_discussed": "The authors should include a reference, and potential comparison, to prior work introducing probabilistic method for across-animal task-informed neural alignment (Herrero-Vidal et al. NeurIPS 2021).", "other_strengths_and_weaknesses": "While the work is clearly presented, adding a section to discuss the limitations is needed to fully assess the impact of the contribution.", "other_comments_or_suggestions": "The authors could include comparisons to linear alignment methods and compare decoding performance, data demands, and computational costs to understand the direct applicability to BCI technology.", "questions_for_authors": "How much data is needed to train the initial model? The author mention that a linear head is needed to compensate for inconsistencies between the number of recorded neurons, how much of the alignment happens in this transformation? How much data is used for this pre-alignment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z1r2h7j46u", "reviewer": "Reviewer_DJVs", "summary": "The authors have introduced a new framework PNBA for aligning neural and behavioural distributional representations. Their approach allows learned and generalisable embeddings across subjects. Their framework uses a multimodal VAE architecture with a constrastive loss term (probabilistic matching term) to provide an alignment pressure. It provides a non-linear generalisation to existing alignment tools in neuroscience such as CCA or linear factor models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thoughtful feedback and the positive assessment. We address each point as follows:\n\n```Q1. The langrangian arguments could do with more detail.```\n\n**A1**: Thanks for this suggestion. In our revision, we will add a more comprehensive derivation of the Lagrangian multiplier method, explicitly clarify how constraints are transformed into optimization objectives, and provide mathematical explanations for key steps in the derivation process.\n\n```Q2. The authors state that the neural representations exhibit minimal correlation with non-corresponding pairs of neural data and behavioural data in figure 3b. But this doesn’t seem so? The correlations between all trials are somewhere between 0.8 and 0.95, including unmatched neural and behavioural data (from my understanding of the plot).```\n\n**A2**: Thank you for this observation. Correlations in Figure 3b are indeed generally high due to our compact latent space. Our intention was to emphasize trial discriminability, with diagonal elements forming distinguishable patterns in most cases. In our revision, we will better characterize successful cases and emphasize trial-level discriminability.\n\n```Q3.I don’t see how the method would be ‘present significant potential for advancing calibration-free neural decoding systems’ (line 464). There would still need to be trials from a new animal or session to internally learn the alignment.```\n\n**A3**: PNBA extracts preserved neural representations in zero-shot individuals (Sections 4.3-4.4), allowing us to train both PNBA models and BCI decoders on known individuals and directly apply them to new individuals without any fine-tuning or calibration (Section 4.5). Thus, we believe this demonstrates potential for calibration-free BCI applications.\n\n```Q4. The downside is that the full trial must be input to match with behaviour, hindering online decoding that is important in BCI applications.```\n\n**A4**: In experiments from Section 4.5, we can adopt a sliding window approach with most BCI decoders, especially Linear and MLP, supporting online decoding. While not our primary focus, this would address the concern. We will add this discussion in our revision.\n\n```Q5.Unclear how correlations are performed, are they on latent space embeddings?```\n\n**A5**: All correlation calculations are indeed performed on the low-dimensional latent representations z, as they have the same spatial dimensions. We will clarify this explicitly in our revision.\n\n```Q6.Recently published Gosztolai et al. MARBLE. (2025.3) paper seems relevant.```\n\n**A6**: Thank you for this recommendation. While both MARBLE and our PNBA aim to reduce dimensionality of neural activity, there are fundamental differences. MARBLE is a single-modal method focusing on geometric constraints and temporal dynamics on neural activity, while PNBA is a multimodal strategy directly incorporating behavioral data as constraints in latent space. \n\nWe'll add this related work in our revision.\n\n```Q7. I wonder whether there is some combined approach with sequential methods like LFADS or GPFA that could be proposed? Since at the moment it does not explicitly model temporal dynamics.```\n\n**A7**:We agree and believe such a combination is entirely feasible and would address the current limitation in modeling temporal dynamics. PNBA provides multimodal alignment constraints while methods like LFADS/GPFA offer complementary temporal modeling capabilities. These represent independent constraints that could be directly combined in future work.\n\nWe'll include this as limitation and state this future work in our revision.\n\n```Q8. It was unclear to me what the normalized representation values are? Are these the latent space activations? Unit vector normalized?```\n\n**A8**:'Normalized representation values' refers to z-score normalization applied independently to each trial's representation, making different modalities comparable by eliminating scale differences. We'll clarify this in our revision.\n\n```Q9. I wonder how the authors would approach such 'multiple solutions' with PNBA? Since in such a case, I would imagine that the low-dimensional neural representations wouldn’t align?```\n\n**A9**: PNBA directly addresses the \"multiple solutions\" challenge. Rather than forcing identical representation for the same behavior, our approach:\n\n1. Avoids assuming one-to-one correspondence between neural activity and behavior, instead employing distributional matching that preserves neural diversity.\n\n2. Ensures that PNBA must generate *different* neural representations for the same behavior (Theorem 1, Property 1), otherwise it would result in representation collapse.\n\n3. Establishes **a lower bound on representational similarity that ensures differences** and an upper bound to promote feasible similarity (Theorem 1, Property 3).\n\nTherefore, these multiple considerations make PNBA particularly suited for studying the many-to-one mapping between neural activity and behavior.", "claims_and_evidence": "The authors have provided a variety of compelling examples on real neural data. Showing cross-trial, -session and -subject correlation analyses in monkey centre out reaching tasks. They also show different neural modality (calcium imaging). Overall empirical evidence is quite strong. \n\nThey authors also provide a proof that the generative constraints should prevent mode collapse. They seem quite standard langrangian arguments, but could do with some more detail.\n\nThe authors state that the neural representations exhibit minimal correlation with non-corresponding pairs of neural data and behavioural data in figure 3b. But this doesn’t seem so? The correlations between all trials are somewhere between 0.8 and 0.95, including unmatched neural and behavioural data (from my understanding of the plot). This is not convincing.\n\nI don’t see how the method would be ‘present significant potential for advancing calibration-free neural decoding systems’ (line 464). There would still need to be trials from a new animal or session to internally learn the alignment. This would be a form of calibration.", "methods_and_evaluation_criteria": "The constrastive probabilistic matching loss seems well motivated given failures of point-wise alignment approaches and the trial-trial variability common in neural data. However, the downside is that the full trial must be input to match with behaviour, hindering online decoding that is important in BCI applications.\n\nHowever, neural data is inherently dynamic and the VAE approach introduced here does not explicitly model dynamics. Whilst there are a number of methods that do this, i.e., learn embeddings whilst ignoring the dynamics, it prevents forecasting.", "theoretical_claims": "The authors provide short ‘sketch’ proofs for theorems 3.1 and B1. I couldn’t find any glaring errors in the proofs and seem relatively consistent with constrained VAEs of beta VAEs.", "experimental_designs_or_analyses": "I was not entirely sure about how the correlations are being performed. I assume this is on the latent space z embeddings but I couldn't explicitly find this. This makes it difficult for me to evaluate the soundness of the experimental results/claims.", "relation_to_broader_scientific_literature": "Cross-subject neural alignment is an open neuroscience problem. Recent work has used linear factor models like CCA or even posthoc alignment. This method provides a clear improvement over these tools, taking advantage of non-linear generative models.", "essential_references_not_discussed": "A recent paper published just this month in Nature Methods seems to have some relevance for aligning neural representations using geometric deep learning and would not have been available upon submission of this article to ICML: Gosztolai et al. MARBLE.", "other_strengths_and_weaknesses": "I wonder whether there is some combined approach with sequential methods like LFADS or GPFA that could be proposed? Since at the moment it does not explicitly model temporal dynamics.", "other_comments_or_suggestions": "None.", "questions_for_authors": "It was unclear to me what the normalised representation values are? Are these the latent space activations? Unit vector normalised? I had to dig into the SI too much to understand what was being correlated in the empirical experiments.\n\nRecent work has shown that there are different ‘solutions’ for solving the same task, i.e., different neural representations. This puts into question a lot of work that uses trial averaging. I wonder how the authors would approach such a problem with PNBA? Since in such a case, I would imagine that the low-dimensional neural representations wouldn’t align?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to joint neural-behavioral alignment for latent representations. Specifically, the authors propose an objective that combines distribution matching (neural and behavioral data should map to similar latent distributions) with bounds on the joint and conditional probabilities of both data sources (similar to a symmetric information bottleneck, in which the latent must preserve information for decoding both modes, preventing mode collapse). The authors demonstrate that this objective prevents certain pathologies of distribution matching, and, in a series of experiments using multi-subject, multi-region recording data, demonstrate that representations inferred from behavior correlate well with those inferred from neural data. They then apply these same learned encoders to new subjects, finding strong zero-shot correlation in the same tests.

Reviewers agreed that this is an important problem in neuroscience and that the approach is novel. They also agreed that the model showed good performance again the selected benchmark methods. Concerns centered on a few themes:
1. Reviewers were confused by the method used to train on disparate numbers of neurons and different neuron identities across sessions. Authors clarified ll.1088-94 that this was done by a trainable session-specific projection at the encoder and decoder ends of the network. Thus, generalization to held-out subjects and sessions would still appear to require training these linear mappings, somewhat vitiating claims of zero-shot transfer.
2. Reviewers were also somewhat confused about the relationship between the proposed method and numerous other alignment methods in the literature. Authors clarified that while other methods focus on aligning, e.g., neural data across individuals for the same sensory inputs, their focus is on aligning latent representations inferred from jointly recorded neural and behavioral data. Authors would do well to clarify further in a revised manuscript. 
3. There was some question as to how these methods would generalize to realistic BCI settings with significant constraints on computation and streaming.

In all, reviewers found the experiments strong and method innovative, suggesting a valuable contribution to the alignment literature.
Decision: Accept (poster)