In vivo cell-type and brain region classification via multimodal contrastive learning
Authors: Han Yu
Hanrui Lyu
YiXun Xu
Charlie Windolf
Eric Kenji Lee
Fan Yang
Andrew M Shelton
Olivier Winter
International Brain Laboratory
Eva L Dyer
Chandramouli Chandrasekaran
Nicholas A. Steinmetz
Liam Paninski
Cole Lincoln Hurwitz
Venue: ICLR 2025 Spotlight (2025)
Keywords: contrastive learning
electrophysiology
extracellular
multimodal
neuroscience
cell type
brain region
Neuropixels
deep learning
Primary area: applications to neuroscience & cognitive science

Abstract: Current electrophysiological approaches can track the activity of many neurons, yet it is usually unknown which cell-types or brain areas are being recorded without further molecular or histological analysis. Developing accurate and scalable algorithms for identifying the cell-type and brain region of recorded neurons is thus crucial for improving our understanding of neural computation. In this work, we develop a multimodal contrastive learning approach for neural data that can be fine-tuned for different downstream tasks, including inference of cell-type and brain location. We utilize multimodal contrastive learning to jointly embed the activity autocorrelations and extracellular waveforms of individual neurons. We demonstrate that our embedding approach, Neuronal Embeddings via MultimOdal Contrastive Learning (NEMO), paired with supervised fine-tuning, achieves state-of-the-art cell-type classification for two opto-tagged datasets and brain region classification for the public International Brain Laboratory Brain-wide Map dataset. Our method represents a promising step towards accurate cell-type and brain region classification from electrophysiological recordings.

Published as a conference paper at ICLR 2025
IN VIVO CELL-TYPE AND BRAIN REGION CLASSIFICA- TION VIA MULTIMODAL CONTRASTIVE LEARNING
Han Yu', Hanrui Lyu², Ethan Yixun Xu', Charlic Windolf", Eric Kenji Lee', Fan Yang', Andrew M. Shelton', Olivier Winter International Brain Laboratory, Eva L. Dyer', Chandramouli Chandrasekaran', Nicholas A. Steinmetz', Liam Paninski', Cole Hurwitz'
CColumbia University, New York, NY, USA
NNrthwwestern University, Evanston, IL, USA
3 Boston University, Boston, MA, USA
4 *University College London, London, UK
5 Allee Institute, Seattle, WA, USA
6 'Champalimaud Foundation, Lisbon, Portugal
Georgia Institute of Technology, Atlanta, GA, USA
Unniversity of Washington, Seattle, WA, USA
(hy2562, hatruilyy222@0o..oottteeeerrrrr
(c3 fkenjilee, ccaneeleeecccc
nick 2tt.stteieeettemmmmmme com
clivier olivier.winterdee. hit.er..wwnter@@nneeerattaaa www.mmabraannnnn www.:oonaabaaaaaa...ooooomo.
evadyer@gatech.eed
ABSTRACT
Current electrophysiological approaches can track the activity of many neurons, yet usually unknown which cell-types or brain areas are being recorded with- out further molecular or histological analysis. Developing accurate and scalable algorithms for identifying the cell-type and brain region recorded neurons is understanding of neural computation. In this work, we develop a multimodal contrastive learning approach for neural data that can be fine-tuned for different downstream tasks, including inference of cell-type and brain location. We utilize this approach to jointly embed the activity autocorrela- tions and extracellular waveforms of individual neurons. We demonstrate that our embedding approach, Neuronal Embeddings via MultimOdal contrastive learning (NEMO), paired with supervised fine-tuning, achieves state-of-th the-art cell-type classification for two opto-tagged datasets and brain region classification for the public International Brain Laboratory Brain-wide Map dataset. Our method rep- resents promising step towards accurate cell-type and brain region classification from electrophysiologiaal recordings. The project page and code are available at https: nemo github.iti.. :oo]
thus crucial for improving
INTRODUCTION
High-density electrode arrays now allow for simultaneous extracellular recording from hundreds to thousands of neurons across interconnected brain regions (Jun et al. 2017 Steinmetz et al., 2021 Ye et 2023b Trautmann et al. 2023). While significant progress has been made in developing
algorithms for tracking neural activity (Hennig et al. 2019 Buccino et al.) 2020) Magland et al.
2020 Boussard al., 2023) Pachitariu et al.) 2024), identifying cell types and brain regions solely
from clectrophyssilogiical features remains an open problem.
Traditional approaches for electrophysiological cell-type classification utilize simple features the extracellular action potential (EAP) such its width peak-to-trough amplitude (Mountcastle
et al.
1969. Matthews & Lee 1991 Nowak et al. 2003 Barthh et al.) 2004: Vigneswaran et al.
2011 features of neural activity, such as the inter-spike interval distribution (Latuske al. 2015
Jouty et al. 2018). These simple features are interpretable and easy to visualize but lack discrimina-
tive power and robustness across different datasets (Weir al. 2015 Gouwens et al.
2019 Current

---

Published as a conference paper at ICLR 2025
automated featurization methods for EAPs (Lee et al. 2021 Vishnubhotla et al.) 2024) and neural activity (Schneider et al. 2023a) improve upon manual features but are limited to a single modality.
There has been a recent push to develop multimodal methods that can integrate information from both recorded EAPs and spiking activity. PhysMAP (Lee et al., 2024) is a UMAP-basek (MeInnes et al. 2a) approach that can predict cell-types using multiple physiological modalities through a weighted nearest neighbor graph. Another recently introduced method utilizes variational au- toencoders (VAEs) to embed each physiological modality separately and then combines these em- beddings before classification (Beau et al. 2025). Although both methods show promising results, PhysMAP is hard to fine-tune for downstream tasks as it is nondifferentiable, and the VAE-based method captures features that are important for reconstruction, not discrimination, impairing down- stream performance (Guo et al. 2017). Neither approach has been used to classify brain regions.
In this work, we introduce a multimodal contrastive learning method for neurophysiological data, Neuronal Embeddings via MultimOdal Contrastive Learning (NEMO), which utilizes large amounts of unlabeled paired data for pre-training and can be fine-tuned for different downstream tasks includ- ing cell-type and brain region classification. We utilize a recently developed contrastive learning framework (Radford et al. (202T) to jointly embed individual neurons' activity autocorrelations and average extracellular waveforms. The key assumption of our method is that jointly embedding differ- ent modalities into a shared latent space will capture shared information while discarding modality- specific noise (Huang et al. 2024). We evaluate NEMO on cell-type classification using optotagged Neuropixels Ultra (NP Ulra) data from the mouse visual cortex (Ye al., (023b) and optotagged Neuropixels 1 data from the mouse cerebellum (Beau et al. 2025). We evaluate NEMO on brain region classification using the International Brain Laboratory (IBL) Brain-wide Map dataset (IBL) it al. 2023). Across all datasets and tasks, NEMO outperforms current unsupervised (PhysMAP and VAEs) and supervised methods, with particularly strong performance in label-limited regimes. These results demonstrate that NEMO is significant advance towards accurate cell-type and brain region classification from electrophysiological recordings.
2
RELATED WORK
2.1
CONTRASTIVE LEARNING FOR NEURONAL DATASETS
The goal of contrastive learning is to find an embedding space where similar examples are close together while dissimilar ones are well-separated
et al. 2020). Contrastive learning has found success across a number of domains including language (Reimers & Gurevych| 2019), vision (Chen et al.) 2020), audio (Saeed et al., (021), and multimodal learning (Radford et al. [021]] Tian ct al. 2020). Contrastive learning has also been applied to neuronal morphological data (Chen et al, connectomics data (Dorkenwald et al. (2023) and preprocessed spiking activity (Azabou et al. Urzay et al. 2023; Schneider et al. 2023b Antoniades et al., 2023). In each of these appli- cations, associated downstream tasks such as 3D neuron reconstruction, cellular sub-compartment classification, or behavior prediction have shown improvement using this contrastive paradigm. One contrastive method, CEED, has been applied raw extracellular recordings to perform spike sorting and cell-type classification. In contrast to NEMO, CEED is unimodal (it ignores neural activity) and has never been applied to optotagged data or brain region classification (Vishnubhotla et al. 2024).
2022
2021
2.2 CELL-TYPE CLASSIFICATION
The goal of cell-type classification is to assign neurons to distinct classes based on their morphol- ogy, function, electrophysiological properties, and molecular markers (Masland 2004). Current transcriptomic (Tassic et al. 2018, Gala et al. 2019: Yao et al. 2021 2023) and optical methods (Cardin et al., 2010 Kravitz al. 2013 Lee et al. 2020) are effective but require extensive sample preparation or specialized equipment, limiting their scalability and applicability for in-vivo stud- ies (Lee et al. 2024). Recently, calcium imaging has been utilized in conjunction with molecular approaches to identify cell-types (Bugeon et al. 2022; Mi et al. (023). This approach has low temporal resolution and requires substantial post hoc effort to align molecular imaging with cal- cium data, making it unsuitable for closed-loop in vivo experiments. A promising alternative is to use the electrophyssological properties of recorded neurons as they capture some of the variability of the transcriptomic profile (Bomkamp et al.) 2019). Simple clectrophysiological features from neuron's 's EAP and spiking activity are commonly used to identify putative cell types, such as exci- tatory and inhibitory cells (Frank et al. 2001 Gouwens al. 2019). Automated methods including
2

---

Published as a conference paper at ICLR 2025
a
Average wavetorms
Spiking activity
Autooorrelogram (ACG) images
|
10
VIP
Neuropirees Ultra
Frling dedie
10
litrg roto (ocleeee))
SST
6 pm
60 um
PV
aau.
40
100ms
500
500
Time from triggered spike (m14)
C
ercnder
Embedding space
Lineor projector
ms
Spike-sorted units
Chetgertoon
ACG encoder
Linear projecter
ACC rep:
Downstream analyses: type diassification brain-region decoding
Figure 1:
Schematic illustration of NEMO. (a) Neuropixels Ultra YEe al. 2023b recordings capture
activity from many different cell-types which have distinct extracellular action potentials (EAPs) and spiking activity. We present waveform and spiking activity snippets from five example neurons for each cell-type. (b) We transform the spiking activity of each neuron into compact autocorrelogram (ACG) image from (Beau al. 2025) that accounts for variations in the firing rate (see Section 4. (c) NEMO utilizes CLIP-based objective where an EAP encoder and ACG image encoder are trained to embed randomly augmented EAPs and ACG image from the same neuron close together while keeping di different neurons separate, The learned representations can he utilized for downstream tasks such as cell-type and brain-region classification.
EAP- Ase-sicccfiic methods (Lee al. 2021) and activity-based methods (Schneider al. 2023a) an improvement comparison to manual features. Most recently, multi-modal cell-type classifica- tion methods including PhysMAP (Lee et al.) (0224) and VAE-based algorithm (Beau al. 2025 Herzfeld et al.) 2025) have been introduced which make use of multiple physiological modalities such the EAP, activity, or peri-stimulus time histogram (PSTH).
2.3 BRAIN REGION CLASSIFICATION
Brain region classification consists of predicting the location neuron or electrode based on the recorded physiological features (Steinmetz et al. 2018 Davis al. 2023). Rather than predicting a 3D location, the task is to classify the brain region a neuron or electrode occupies, which can be estimated using post-insertion localization via histology (Sunkin al. 2012). Brain region classifi- cation is an important task for understanding fundamental differences in physiology between brain areas and for targeting regions that are hard hit via insertion. Most importantly, brain region clas- sification can provide a real-time estimate of the probe's 's location in the brain during experiments. Additionally, insertions in primates and human subjects often lack histological information, instead relying on the experimental heuristics that lack standardization between laboratories. As this task relatively new, only simple features of the EAPs have been utilized for classification (Tia et al. 2019 Tolossa et al. 2024
DATASETS
For cell-type classification. we use two mouse datasets: an patt-ttgged dataset from the visual cortex recorded with Neuropixels Ultra (NP Ultra; Ye et al. 2023h) and dataset from the cerebellar cortex

---

Published as a conference paper at ICLR 2025
recorded with Neuropixels (Beau et al. 2025). For brain region classification, we utilize the IBL Brain-wide Map of neural activity from mice performing a decision -aaion task (IBL et al. 2023).
NP Ultra opto-tagged mouse data. This dataset consists of NP Ultra recordings of spontaneous and opto-stimulated activity from the visual cortex of mice. We included spontaneous periods only and excluded units that have less than 100 spikes after removal of stimulation periods (see Supplement We obtained 462 ground-truth neurons with three distinct cell-types. The ground-truth neu- rons are composed of 237 parvalbumin (PV). 107 somatostatin (SST), and 1 118 vasoactive intestinal peptide cells (VIP). There are also 8491 unlabelled neurons that can utilize for pre-tralee pre-training.
C4 cerebellum dataset. This dataset consists of Neuropixels recordings from the cerebellar cortex of mice. Opto-tagging is utilized to label 202 ground-truth neurons with five distinct cell-types. The ground-truth neurons are composed 27 molecular layer interneurons (MLI, 18 Golgi cells (GoC), 30 mossy fibers (MF). 69 Purkinje cell simple spikes (PCss), and 58 Purkinje cell complex spikes (PCcs). There are 3,090 unlabelled neurons for pretraining.
IBI. Brain-wide Map. This dataset consists of Neuropixels recordings from animals performing decision-making task. Each neuron is annotated with the brain region where it is located. We utilize 675 insertions from over 100 animals. yielding 37017 'good' quality neurons after spike sorting and quality control (Banga et al. 2022 IBL et al. 2022). Each brain is parcellated with 10 brain atlas annotations divided into 10 broad areas: isocortex, olfactory areas (OLF), cortical subplate (CTXsp), cerebral nuclei (CNU), thalamus (TH), hypothalamus (HY), midbrain (MB), hindbrain (HB), cerebellum (CB) and hippocampal formation (HPF). We divide this dataset into training, validation, and testing set by insertion such that we can evaluate each model on heldout insertions.
NEMO
We introduce Neuronal Embeddings via MultimOdal contrastive learning (NEMO), which learns multimodal embedding of neurophysiological data. To extract representations from multiple modal- ities, utilize Contrastive Language- Image Pretraining (CLIP; Radford et al. 2021). CLIP uses contrastive objective to learn joint representation of images and captions. For NEMO, we utilize the same objective but with modality-specific data augmentations and encoders (see Figure Ie).
4.1 PREPREPROCESSING
We construct paired dataset of spiking activity and EAPs for all recorded neurons. Using the open-source Python package NeuroPyxels (Beau et al. 2021), we computed an autocorrelogram (ACG) image for each neuron by smoothing the spiking activity with a 250-ms width boxcar filter, dividing the firing rate distribution into 10 deciles, and then building ACGs for each decile (see Figure [p]. This ACG image
useful representation because the activity autocorrelations of a neuron can change as function of its firing rate. By computing ACGs for each firing rate decile, the ACG image will account for firing rate dependent variations in the autocorrelations, allowing for comparisons between different areas of the brain, behavioral contexts, and animals (Beau al. 2025). For the EAPs, we construct a 'template' waveform which is the mean of ~500 waveforms for that neuron. For NP Ultra, we utilize multi-channel templates which take advantage of the detailed spatial structure enabled by the small channel spacing; we use nine channels with the highest peak- to-peak (ptp) amplitude, re-ordered from highest to lowest amplitude. For the C4 and IBL dataset, all main text results utilize templates consisting of one channel with maximal amplitude.
4.2 DATA AUGMENTATIONS
Previous work on contrastive learning for spiking activity utilizes data augmentations including sparse multiplicative noise (pepper noise), Gaussian noise, and temporal jitter (Azabou et al., 2021) )). As it is computationally expensive to construct ACG images for each batch during training, we in- stead design augmentations directly for the ACG images rather than the original spiking data. Our augmentations include temporal Gaussian smoothing, temporal jitter, amplitude scaling, additive Gaussian noise, and multiplicative pepper noise (see Supplemental B for more details and Supple- mentary Figure 14 for an ablation). For single channel templates, we use additive Gaussian noise as our only augmentation. For multi-channel templates, we also include electrode dropout and ampli- tude jitter as described in Supplementary Table] 1
4

---

Published as a conference paper at ICLR 2025
b
0.90
Waveform
ACG
Waveform ACG
0.85
Linear probe fine-tuned
Balenoed Accuracy
Pl 9ST
0.75
VIP
0.70
0.60
0.55
2au.
29uu.
23.1
0.45
NEMO viE (xo0))
Phystipp
NEMO aperrieed
viE (ocen) VAE (rad)
Models supervised
NEMO frozen
NEMO
VAE
PhysMAP
PV
0.90
0.04
0.05
PV
0.94
0.02
0.04
PV
0.93
0.02
0.05
PV
0.53
0.20
0.27
PV
0.93
0.02
0.05
label
SST
0.06
0.73
0.21
SST
0.09
0.71
0.21
SST
0.09
0.70
0.21
SST
0.27
0.52
0.22
SST
0.09
0.21
VIP
VIP
0.25
VIP
PV
SST
VIP
SST
VIP
VIP
SST
VIP
PV
SST
VIP
Predicted label
Figure 2: Comparing NEMO to baseline models the NP Ultra opto-tagged dataset. (a) UMAP visual- ization the pretrained NEMO representations of unseen opto-tagged visual cortex units, colored by different cell-types. Neurons the same class form clusters, particularly when combined modalitics are used. (b) Bal- anced accuracy and (c) Confusion matrices for the NP Ultra classification results, normalized by ground truth label and averaged across 5 random seeds. NEMO outperforms the other embedding methods by significant margin across all cell-types and evaluation methods.
4.3 ENCODERS
We employ separate encoders for each electrophysiological modality. For the ACG image encoder, we use a version of the convolutional architecture introduced in (Beau et al. 2025) with 2 layers and Gaussian Error Linear Units (GeLU) (Hendrycks & Gimpel| 2016). For the waveform encoder, use 2 layer multilayer perceptron (MLP) with GeLU units. The representation sizes are 200 di- mensional and 300 dimensional for the ACG image encoder and the waveform encoder, respectively. We set the projection size to be 512. For details about hyperparameters, see Supplement
4.4 CONTRASTIVE OBJECTIVE
We utilize the contrastive objective defined in CLIP. Let Pacy and zwf be the I.2 normalized projec- tions of each modality. For batch B, the objective is as follows,
B
expl exp(Znce, Zwt, + log
exp Zace, /T)
L = B
log
(1)
exp Zacg, /T),
Σ 2==1 exp( (2pg, Zwt, /
where 7 a temperature parameter which we fix during training. The objective function encourages the model to correctly match Racg, with its corresponding Zwf, and vice versa, over all other possible pairs in the batch. This loss can easily be extended to more than two modalities including PSTHs.
4.5
SINGLE-NEURON AND MULTI-NEURON BRAIN REGION CLASSIFICATION
Brain region classification using electrophysiologicll datasets is a new problem that requires novel classification schemes. We develop two classification schemes for our evaluation: a single-neuron and multi-neuron classifier. For our single-neuron classifier, neuron independently using its embedding. For our multi-neuron classifier, we predict the brain region for each 20 micron bin along the depth of the probe by ensembling the predictions of nearby neurons within a 60-micron radius (i.e., averaging the logits of the single-neuron model) as shown in Figure 3h (ii). When more than five neurons fall within this range, only the nearest five selected.
predict the brain area for each
5

---

Published as a conference paper at ICLR 2025
Table 1: Cell-type classification for the NP Ultra dataset. 5-Fold accuracy and F1 Fi-scores are reported for three conditions: (i) a linear layer and (ii) MIP on top the frozen pretrained representations, and (iii) after
MLP finetuming. Chance level is 0.33 for this dataset.
Model
Linear
MLP
MLP fine-tuned
Acc
F1
Acc
F1
Acc
F1
Supervised
N/A
N/A
N/A
N/A
0.79 + 0.00
0.78 + 0.00
PhysMAP (Lee at 2024
0.5:
0.52
N/A
N/A
N/A
N/A
VAE (10D latntt Beau et al. 2025
0.74 = 0.00
0.73 = 0.00
0.74 + 0.00
0.73 = 0.00
0,78 = 0.00
0.79 = 0.00
(500D) rep)
0.76 0.00
0.75
0.00
+ 0.00
0.77 0.00
0.00
0.79
NEMO (500D rep)
0.78 0.01
0.78
0.00
0.80 + 0.00
0.79 + 0.00
0.80 = 0,00
0.80 +: 0.00
Table 2: Cell-type classification for the C4 cerebellum dataset. 5-Fold accuracy and F1-scores are reported for three conditions: (i) a linear layer and (ii) ML. on top of the frozen pretrained representations, and (iii) after MLP finetuning. Chance level is 0,20 for this dataset.
Model
Linear
MLP (5-fold)
Finetuned MLP (5-fold)
Acc
F1
Acc
F1
Ace
F1
Supervised
N/A
N/A
N/A
N/A
0.82 0.00
0.82 0.00
PhysMAP
0
0.49
N/A
N/A
N/A
N/A
VAE (10D latent: Beaau et al. 2025
0.79 : 0.00
0.78 0.01
0.74 = 0.01
0.73 = 0.02
0.82 + 0.00
0.81 = 0.00
VAE (500D rep)
0.82 : 0,00
0.81 0.00
0.82
= 0.00
0.82 = 0.00
0.83 + 0.00
0.83 = 0.00
NEMO (500D rep)
0.85 -: 0.01
0.85 0.01
0.85+ 0,00
0,85 0.00
0.86 + 0.01
0.86 d: 0.01
5
EXPERIMENTAL SETUP
5.1 BASELINES
For our baselines, we compare to PhysMAP (Lee et al. 2024). VAE-based method (Beau et al., ). and a fully supervised MLP. For fair comparison, we utilize the same encoder architectures for NEMO and the VAE-based method. We include two versions of the VAE baseline: (1) the latent space (10D) is used to predict cell-type or brain region (from Beau et al. (2025), or (2) the output of the layer before the latent space (500D) is used to predict cell-type brain region. Although this approach was not proposed in Beau et al. (2025), we find that utilizing the 500D representations be- fore the latent space performed better than using the 10D latent space and also outperformed VAE trained with a 500D latent space. For the VAËS, we use default hyperparameters from Beau et al.) (2025) (see Supplement| N for a hyperparameter sensitivity analysis). For the supervised bascline, we again use the same encoder architectures as NEMO. For training NEMO, we use an early stop- ping strategy which utilizes validation data. For the VAE-based method, we use the training scheme introduced in Beau et al. (2025). We fix the hyperparameters for all methods across all datasets. For more details about baselines, training, and hyperparameters, see Supplements B and
5.2 EVALUATION
For both NEMO and the VAE-based method, the representations from the ACG image and EAPs are concatenated together before classification or fine-tuning. We apply three classification schemes for evaluation including (1) freezing the model and training linear classifier on the final-layer outputs, (2) freezing the model and training MLP-based classifier on the final-layer outputs, (3) fine-tuning both the original model and a MLP-based classifier on the final layer. To ensure balanced training data, we implement dataset resampling prior to fitting the linear classifier. For PhysMAP comparisons, we utilize the weighted graph alignment approach provided in Lee al. (2024) for all comparisons. For our classification metrics, we utilize the macro-averaged Fl score, calculated as the unweighted mean of F1 scores for each class, and balanced accuracy, which measures average accuracy per class. For additional details about baseline hyperparameters, see Supplement B 5.3 EXPERIMENTS
NP Ultra opto-tagged dataset. For the NP Ultra dataset, we pretrain NEMO and the VAE-based method on 8491 unlabelled neurons. This pretraining strategy is important for reducing overfitting to the small quantity of labeled cell-types. To evaluate each model after pretraining, we perform the three evaluation schemes introduced in Section 5.2 freezing + linear classifier, freezing + MLP
We utilize PhysMAP's anchor alignment technique (which is deterministic) to evaluate its performance.

---

Published as a conference paper at ICLR 2025
Table 3: Single-neuron brain region classification for the IBL dataset. The accuracy and Fi-scores
reported for three conditions: (i) a linear layer and (ii) MI. on top of the frozen pretrained representations, and (iii) after MIP finetuning. Chance level is O. 1.10 for this dataset.
Model
Linear
MLP
MLP fine-tuned
Acc
Acc
Acc
F1
Supervised
N/A
N/A
N/A
N/A
0.45 1 0.01
0.42 = 0.01
PhysMAP
0.31
0.27
N/A
N/A
N/A
N/A
VAE (500D rep)
0.40 = 0.01
0.37 0.01
0.41 = 0.00
0.37 0.00
0.45 + 0.01
0.43 0.00
NEMO (500D rep)
0.42 0.00
0.40 0.00
0.45
0.01
0.42 0.00
0.47 0.01
0.44 0.01
Table 4: Multi-neuron brain region classification for the IBI. dataset. The accuracy Fl Pl-scores are reported for three conditions: (i) a linear layer and (ii) MI. on top of the frozen pretrained representations, and (iii) after MLP finetuning. Chance level is 0.10 for this dataset.
Model
Linear
MLP
MLP fine-tuned
Acc
F1
Acc
F1
Acc
F1
Supervised
N/A
N/A
N/A
N/A
0,50 1 0.00
0.48 1 0.01
VAF (500D rep)
0.45 + 0.01
0.42 + 0.01
0.46 + 0.00
0.43
0.00
0.51
0.01
0.49 + 0.01
NEMO (500D rep)
0.48 +: 0.00
0.45
0.00
0.50 + 0.00
0.48
0.00
0.51
0.00
0.50 - 0.00
classifier, and full end-to-end finetuning, For PhysMAP, we utilize the anchor alignment technique introduced by Lee et al. (2024). For all methods and evaluation schemes, we perform 5-fold cross- cross-validation with 10 repeats to evaluate each model.
C4 cerebellum dataset. For the C4 dataset, we pretrain all methods on 3090 unlabelled neurons. To evaluate each model after pretraining, we perform the three evaluation schemes introduced in Section [.2. For all classifiers, we do not utilize input layer information, nor do we exclude neurons based on confidence threshold as done in Beau al. (2025), as we were interested in evaluating the predictiveness of the features directly without additional information. For all methods and evaluation schemes, we perform f-lold cross- vols-valiittionn with 10 repeats evaluate each model.
IBL Brden-wide Map. For the IBL dataset, we randomly divide all insertions (i.e., Neuropixels recordings) into a 70-10-20 split to create a training, validation, and test set for each method. We then pretrain NEMO and the VAE-based method on all neurons the training split. We then perform the three evaluation schemes introduced in Section [.2] For PhysMAP, we utilize the anchor align- ment technique. We train both a single-neuron and multi-neuron classifier using the representations learned by NEMO and the VAE-based method. For PhysMAP, we only evaluate the single-neure neuron classifier. We compute the average and standard deviation of the metrics using five random seeds.
RESULTS
6.1 CLASSIFICATION
NP Ultra cell-type classifier. The results for the NP Ultra opio-tagged dataset are shown in Table For all three evaluation schemes, NEMO achieves the highest macro-averagee Fl score and balanced accuracy by a significant margin. Notably, its largest improvement is in the linear and frozen MI.P evaluations, indicating that NEMO captures cell-type-discrimnee without fine-tuning. After fine-tuning, all methods are closer in performance, suggesting that there is some saturation for this dataset. These results demonstrate that NEMO is an accurate method for cell-type classification in visual cortical microcireuits even without fine-tuning.
T and Figure 2
features even
C4 cerebellum cell-type classifier. The results for the C4 cerebellum dataset are shown in Table and Supplementary Figure g For all evaluation schemes, NEMO outperforms the baseline models, achieving the highest macro-averaged F1 score and balanced accuracy, with the largest gains again in the linear and frozen MI.P evaluations. These findings demonstrate that NEMO can accurately classify cell types in the cerebellum without any hyperparameter adjustments.
IBL single-neuron region classifier. We then aim to investigate how much relevant information NEMO extracts from each neuron about its anatomical location, ie., brain region. We investigate this by training classifiers that use single neuron features identify anatomical regions for the

---

Published as a conference paper at ICLR 2025
a
Real
b
Single-neuron decodar with fine-tuned MLP
Multi-neuron decoder
with fine-funed MLP
Isocorbee
Multmodal rep.
NEMO
NEMO
AGB
WVF
CNU
CNU
ML Linner
Individual uni popits
True label
True label
CTXpp
HB
HPF
#YY
HY
Isoeartex
OLF
13
OLF
TH
Lvv
7M
leroon Ka
Final predict
Predicted label
0.75
d
Linear probe
MLP
Average balanced accuracy by modality
0.45
0.5
NOMO
WAE
Balanced aocuracy
0.40
Balanced accuracy
Balanced aalnnaadaccuaaoo
gocuracy
0.4
0.40
0.3
0.35
0.35
WIE Pezzo
0.2
Nodel
0.30
0.30
supprreedd NEVO freee NEVO
VAE
Wid Foozz
0.25
0.25
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
both
3D-ACG Modality
WVF
Label
Figure 3: Results for NEMO on the IBL brain region classification task. (a) Schematic for multi-neuron classifier. (i) At cach depth, the neurons within 60 um were used to classify the anatomical region. Only the nearest 5 neurons were selected if there
more than 5 neurons within that range. (ii) For logits averag- ing, single-neuron classifier logits are predicted using linear model/ML.P trained the representations of our two physiological modalities. The final prediction based on the average Confusion matrices for the single-neuron region classifier for fine-tuned NEMO. (c) Confusion matrices for the NEMO multi-neuron region classifier, averaged across 5 runs. (d) Single neuron balanced accuracy with linear classifier and the MLP head for cach model trained/fine- MLP-clasifficttion balanced accuracy for each modality separately and the combined representation.
individual logits. (b)
with different label ratios. (c) Single-neuron
IBL. dataset (see Table and Figure 3 for results). The confusion matrix for the VAE, supervised MLP, and PhysMAP shown in Supplementary Figure 9 We find that NEMO outperforms all other methods using both the linear and MLP-based classification schemes. Without end-to-end fine-tuning, NEMO with an MLP classification head is already on par with the supervised MLP. NEMO's s success with both the linear and MLP classifier with frozen encoder weights indicates that NEMO is able extract a region-discrime digiondiscrimmnative representation of neurons without additional fine- tuning. This representation can be further improved by fine-tuning NEMO. All methods have closer performance after fine-tuning potentially due the substantial amount of labeled data.
IBL multi-neuron region classifier. We investigate whether combining information from multiple neurons at each location can improve brain region classification. We use the nearest-neurons ensem- bling approach as described in 4nd shown in Fi. Averaging the logits of predictions from single neurons improves classification performance over the single-neuron model. NEMO still has the best region classification performance especially for the linear and frozen MI.P evaluations (see Table 4 and Figure 3 for results). Again, all methods have closer performance after fine-tuning.
6.2 CLUSTERING
We next examine the clusterability of NEMO representations for the IBL Brain-wide Map. We fol- lowed the clustering strategy used in Lee et al. (022I) by running Louvain clustering on UMAP graph constructed from the representations extracted by NEMO from the IBL training neurons. We adjusted two main settings: the neighborhood size in UMAP and the resolution in Louvain clus- tering. We selected these parameters by maximizing the modularity index, which had the effect of minimizing the number resulting clusters (Figure 4e). The clustering results relative to the region labels are presented in Figures and b. The UMAP visualization of the NEMO represen-
8

---

Published as a conference paper at ICLR 2025
region colored
b
cluster colored
# clusters
200
500
1000
Cluster
# neighbors
GB
Chaiter
CTXXp
CNU
Clastar 3
Chaster 2
Modularity Index
Cluster Number
Cheller
0.68-
8
Clustar
isccorlex
Challer
50
MB
Cluster g
0.84-
OLF
Clastar 9
Ckeiter
d
Horizontal
0.62
0.5
1.0
Resolution
Sagillal Slice
Coronal Slice
2mmm
2mm
2mm
2mm
2mm
2mm
Figure 4: IBL neuron clustering using NEMO pretraining, A UMAP visualization of the representations that NEMO extracts from training data colored by anatomical brain region. (b) The same UMAP as shown (a) but instead colored by cluster labels using a graph-based approach (Louvain clustering). (c) We tuned the neighborhood size in UMAP and the resolution for the clustering. These parameters were selected by maximizing the modularity index which minimized the number of clusters. (d) 2D brain slices across three brain views with the location of individual neurons colored using cluster shown (b). The black lines show the region boundaries the Allen mouse atlas (Wang et al. 2020). The cluster distribution found using NEMO is closely correlated with the anatomical regions and is consistent across insertions from different labs.
tations, colored by region label, demonstrates that the regions separable in the representation space. Notably, there is distinct separation of thalamic neurons from other regions, along with an isolated cluster of cerebellar neurons. Neurons from other regions also well organized by gion labels within the NEMO representation space, allowing for their clustering into several distinct clusters. Additionally, overlaying the neurons colored by their cluster IDs onto their anatomical lo- cations (Figure 4 reveals a cluster distribution closely correlated with anatomical regions which is consistent across insertions from different labs (Supplementary Figure TTI. We find that clustering NEMO's representations leads to a more region-selective clustering than when we use the raw fea- tures directly (Supplementary Figures 12 and 13p. These results demonstrate that NEMO extracts features that capture the electrophysiological diversity across regions even without labels.
6.3 ABLATIONS
Label ratio sweep. We assess whether NEMO requires less labeled data for fine-tuning by conduct- ing a label ratio sweep with single-neuron region classifiers. train both linear and MLP classifiers under two conditions: with frozen weights and full end-to-end fine-tuning, using 1%, 10%. 30%, 50%, 80%, and 100% the labeled data. Accuracy results are shown in Figure 31 (F1 scores in Supplementary Figure 6). The fine-tuned NEMO model outperforms all other methods across all label ratios. Notably, with only 50% of the training labels, both the linear and fine-tuned NEMO models outperform the corresponding VAE models and the supervised MLP, even when the latter models are trained on the full dataset.
Single modality classifier. We examine whether combining modalities improves region-relevant information and if NEMO enhances feature extraction by aligning their embeddings. We compared the classification performance of the MLP classifier with encoder weights frozen and end-to-end fine-tuned, across all models using: waveforms only 2) ACGs only 3) waveforms and ACGs.

---

Published as a conference paper at ICLR 2025
a
Linear
MLP
b
Average balanced accuracy by modality
0.45
0.5
NEMO SimCLR
Balanoed sccuracy
Balanced BOQuraKy
0.45-
Galane plannco accuracy
0.40
0.40
0.3
0.35
0.35-
0.2
SmC Froo
0.30
SmxCLR
0.1-
NEMD
NEMO
SinOLR trezze
0.25
0.25
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Waveform ACG ACG
Waveform
Label ratio
Label ratio
Modality
Figure 5: Ablating joint vs. independent learning for NEMO. To evaluate the importance of learning shared representation between modalities, train version of NEMO the IBL brain classification task where each modality independently embedded using SimCLR. (a) Across all label ratios and classifiers, we find that NEMO trained with CLIP outperforms SimCLR version. (b) NEMO trained with CLIP also extracts more informative representations for each modality than when training with SimCLR.
Brain region classification balanced accuracies are shown in Figure 3 (for F1, see Supplementary Figure We found that bimodal models generally outperform unimodal models, suggesting that combining both modalities provides extra information the anatomical location of neurons. Once again, NEMO achieves the best performance, demonstrating its ability to enhance single-modality information extraction by leveraging the other modality. After fine-tuning, NEMO and the VAE have closer performance potentially due to the substantial amount of ground-truth labels.
Joint ys. independent learning for NEMO. To ablate the importance of learning a shared represen- tation of each modality, we train version of NEMO where we independently learn embedding for each modality using a unimodal contrastive method, SimCLR (Chen et al., (020). The results for brain region classification are shown in Figure 5h where NEMO trained with CLIP outperforms NEMO trained with SimCLR for all label ratios and classification methods. NEMO trained with CLIP also able to extract more informative representations from each modality as shown in Fig- ure 5p. These results demonstrate that learning a shared representation of the two modalities is important for the downstream performance of NEMO. Supplementary Table and I0 show that joint training with CLIP also leads to an improvement over SimCLR for cell-type classification.
DISCUSSION
In this work, we proposed NEMO, pretraining framework for electrophysiologica data that utilizes multi-modal contrastive learning. We demonstrate that NEMO is able to extract informative repre- sentations for cell-type and brain region classification with minimal fine-tuning across three different datasets. This is especially valuable in neuroscience, where ground truth data, like opto-tagged cells, costly, labor-intensive, even impossible obtain (e.g., in human datasets).
Our work has several limitations. First, we focus on shared information between two modalities, assuming this is most informative for identifying cell identity or anatomical location. However, modeling both shared and modality-specific information could further improve performance, as each modality may contain unique features relevant cell identity or anatomical location (Liu et al.
2024 Liang et al., 2024). Additionally, NEMO utilizes the activity of each neuron independently. which ignores neuron correlations that can help distinguish cell-types (Mi al.) 2023). Extending NEMO to encode population-leval features is an exciting future direction.
NEMO opens up several promising avenues for future research. Our framework can be adapted for studies of peripheral nervous systems, such as the retina (Wu et al. 2023). NEMO can also be com- bined with RNA sequencing to find features that are shared between RNA and electrophysiologica data (Li et al. 2023). It will also be possible to correlate the cell-types discovered using NEMO with animal behavior to characterize their functional properties. Finally, we imagine that the repre- sentations extracted by NEMO can be integrated with current multi-animal pretraining approaches for neural activity to provide additional cell-type information which could improve generalizability unseen sessions animals (Azabou et al. 2023
Ye al. 2023a Zhang et al. 2025 2024
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENTS
We thank Jonathan Pillow and Tatiana Engel for providing feedback on this manuscript and Sahar Minavi and Shawn Olsen for their contributions to collecting and sharing the NP Ultra data. We also thank Maxime Beau and the other authors of Beau al. 2025 for sharing the C4 cerebel- lum dataset. This project was supported by the Wellcome Trust (PRF 209558, 216324, 201225, and 224688 to MH, SHWF 221674 to LFR, collaborative award 204915 MC. MH and TDH), National Institutes of Health (UU1NNS1277) under Cooperative Agreement PHY-2229929 (The NSF AT Institute for Artificial and Natural In- telligence), the Kavli Foundation, the Gatsby Charitable Foundation (GAT3708), the NIH BRAIN Initiative (UOINSI MANNS11223 13252 NAS, SRO, and TDH), the Pew Biomedical Scholars Program (NAS), the Max Planck Society (GL), the European Research Council under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No 834446 to GL and AdG 695709 to MH), the Giovanni Armenise Harvard Foundation (CDA to LFR), the Human Technopole (ECF to LFR), the NSF (IOS 21 2500 to NBS), the Klingenstein-Simen Simons Fellowship in Neuroscience (NAS), the NINDS
the Simons Foundation, the DoD OUSD (R&E)
REER awards 12999,99, the NINDS R2INS135361, the NINDS F3 F31NS131018, the NSF CA- as well as generous gifts from the McKnight Foundation, and the CI- FAR Azrieli Global Scholars Program. GM is supported by Bochringer Ingelheim Fonds PhD Fellowship. The primate research procedures were supported by the NIH P51 (OD010425) to the WaNPRC, and animal breeding was supported by NIH U42 (OD011123). Computational modeling work was supported by the European Union Horizon 2020 Research and Innovation Programme under Grant Agreement No. 945539 Human Brain Project SGA3 and No. 101147319 EBRAINS 2.0 (GTE and TVN), Computational resources for building machine learning models were provided by ACCESS, which is funded by the US National Science Foundation.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 8623-2633 2019.
Antonis Antoniades, Yiyi Yu, Joseph Canzano, William Wang, and Spencer Vere Smith. Neu- roformer: Multimodal and multitask generative pretraining for brain data. arXiv preprint arXiv:2311.00136. 2023.
Thomas Aynaud. python-louvain x.y: Louvain algorithm for community detection. httpps://githin http://www...///aaaaaaa//////aaa 2020.
Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran-Nain Max Dabagia, Bernardo Avies-Pires, Lindsey Kitchell, Keith B Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2/02. 10106, 2021.
Mchdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael J Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva L Dyer. A unified, scalable framework for neural population decoding. arXiv preprint arXiv:2310.1604 2023.
Kush Banga, Julien Boussard, Gaälle Chapuis, Mayo Faulkner, Kenneth Harris, JM Hunten- burg. Cole Hurwitz, Hyun Dong Lee, Liam Paninski, Cyrille Rossant, et al. Spike sorting pipeline for the international brain laboratory. 2022.
Peter Barthó, Hajime Hirase, Lenaïc Monconduit, Michael Zugaro, Kenneth D Harris, and Gyorgy Buzsaki. Characterization neocortical principal cells and interneurons by network interactions and extracellular features. Journal of neurophysiology. 92(1):600-608, 2004.
Maxime Beau, Federico D) Agostino, Ago Lajko, Gabriela Martínez, and Dimitar Kostadinov. Neuropyxels: loading, processing and plotting neuropixels data python. Zenodo https://dii. org/10.5281//EENODD 55097333 5509733, 2021.
Maxime Beau, David Herzfeld, Francisco Naveros, Marie E. Hemelt, Federico D' D'Agsstino, Mar- lies Oostland, Alvaro Sánchez-López, Young Yoon Chung, Michael Maibach, Hannah N. Stabb, et al. A deep-learning strategy to identify cell types across species from high-density extracellular recordings. Cell, 2025. doi: 10. cell..l.
Claire Bomkamp, Shreejoy Tripathy, Carolina Bengtsson Gonzales, Jens Hjerling-Leffler, Ann Marie Craig, and Paul Pavlidis. Transcriptomic correlates electrophysiological and mor- phological diversity within and across excitatory and inhibitory neuron classes. PLoS computa- tional biology, 15001113, 2019.
Julien Boussard, Charlie Windolf, Cole Hurwitz, Hyun Dong Lee, Han Yu, Olivier Winter, and Liam Paninski. Dartsort: A modular drift tracking spike sorter for high-density multi-electrode probes. bioRxiv, pp. 2023-08, 2023.
Alessio P Buccino, Cole L Hurwitz, Samuel Garcia, Jeremy Magland, Joshua H Siegle, Roger Hurwitz, and Matthias Hennig. Spikeinterface, a unified framework for spike sorting. Elife, 9: e61834, 2020.
Stephane Bugeon, Joshua Duffield, Mario Dipoppa, Anne Ritoux, Isabelle Prankerd, Dimitris Ni- coloutsopoulos, David Orme, Maxwell Shinn, Han Peng. Hamish Forrest, al. A transeriptomic axis predicts state modulation of cortical interneurons. Nature, 607(791))3330-1 -38,0 2022.
Jessica A Cardin, Marie Carlén, Konstantinos Meletis, Ulf Knoblich, Feng Zhang, Karl Deisseroth, Li-Huei Tsai, and Christopher Moore. Targeted optogenetic stimulation and recording of neu- rons vivo using cell-type-speciff expression of channelrhodopsin Nature protocols, 5(2): 247-254, 2010.
12

---

Published as a conference paper at ICLR 2025
Hanbo Chen, Jiawei Yang, Daniel lascone, Lijuan Liu, Lei He, Hanchuan Peng, and Jianhua Yao. Treemoco: Contrastive neuron morphology representation learning. Advances in Neural Infor- mation Processing Systems, 01-25000-250777 2022.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffre Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Zachary W Davis, Nicholas M Dotson, Tom P Franken, Lyle Muller. and John H Reynolds. Spike- phase coupling patterns reveal laminar identity in primate cortex. Elife, 12:c84512, 2023.
Sven Dorkenwald, Peter H Li, Michal Januszewski, Daniel R Berger, Jeremy Maitin-Shepard Shepard, Agnes L Bodor, Forrest Collman, Casey M Schneider-Me Mizell, Nuno Maçarico da Costa, Jeff W Lichtman, et al. Multi-layered maps of neuropil with segmentation-guiee guided contrastive learning. Nature Methods, 20(12):201 (000)2011-20200 1-2020, 2023.
Loren M Frank, Emery N Brown, and Matthew A Wilson. A comparison of the firing properties of putative excitatory and inhibitory neurons from cal and the entorhinal cortex. Journal of neurophysiology. 86(4):2029-2D0 2040, 2001.
Rohan Gala, Nathan Gouwens, Zizhen Yao, Agata Budzillo, Osnat Penn, Bosiljka Tasic, Gabe Mur- phy, Hongkui Zeng, and Uygar Sümbül. A coupled autoencoder approach for multi-modal anal- ysis of cell types. Advances in Neural Information Processing Systems, 32, 2019.
Nathan W Gouwens, Staci A Sorensen, Jim Berg, Changkyu Lee, Tim Jarsky, Jonathan Ting, Su- san M Sunkin, David Feng, Costas A Anastassiou, Eliza Barkan, et al. Classification of electro- physiological and morphological neuron types in the mouse visual cortex. Nature neuroscience, 22(7):1182-1199 2019.
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In Ijcai, volume 17, pp. 1753-1759, 2017.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
16Xvv:1606008455 2016.
Matthias H Hennig, Cole Hurwitz, and Martino Sorbaro. Scaling spike detection and sorting for next-generation electrophysiology. In Vitro Neuronal Networks: From Culturing Methods to Neuro-Technological Applications, pp. 171-184, 2019.
David Herzfeld, Nathan Hall, and Stephen Lisberger. Strategies to decipher neuron identity from extracellular recordings in the cerebellum of behaving non-human primates. bioRxiv, pp- 0025-01. 2025.
Wei Huang, Andi Han, Yongqiang Chen, Yuan Cao, Zhiqiang Xu, and Taiji Suzuki. On the comparison between multi-modal and single-modal contrastive learning. arXiv preprint arXiv:2411.02837, 2024.
IBL. Kush Banga, Julius Benson, Niccolò Bonacchi, Sebastian Bruijns, Rob Campbell, Gaëlle A Chapuis, Anne K Churchland, M Felicia Davatolhagh, Hyun Dong Lee, al. Reproducibility of in-vivo electrophysiological measurements in mice. bioRxiv, pp. 2022-05, 2022.
IBL, Brandon Benson, Julius Benson, Daniel Birman, Niccolo Bonacchi, Matteo Carandini, Joana A Catarino, Gaelle A Chapuis, Anne Churchland, Yang Dan, et al. A brain-wide map of neural activity during complex behaviour. bioRxiv, pp. 2023-07, 2023.
Xiaoxuan Jia, Joshua H Siegle, Corbett Bennett, Samuel Gale, Daniel J Denman, Christof Koch, and Shawn R Olsen. High-density extracellular probes reveal dendritic backpropagation and facilitate neuron classification. Journal of neurophysiology, 121(5): 1831-187, 2019.
Jonathan Jouty, Gerrit Hilgen, Evelyne Sernagor, and Matthias H Hemnig. Non-parametric physi- ological classification of retinal ganglion cells in the mouse retina. Frontiers in Cellular Neuro- science, 12:481, 2018.
13

---

Published as a conference paper at ICLR 2025
James Jun, Nicholas A Steinmetz, Joshua H. Siegle, Daniel Denman, Marius Bauza, Brian Barbar- its, Albert Lee, Costas A Anastassiou, Alexandru Andrei, Çağatay Aydın, et al. Fully integrated silicon probes for high-density recording of neural activity. Nature, 551(7679):222 236, 2017.
Alexxai V Kravitz, Scott F Owen, and Anatol C Kreitzer. Optogenetic identification of striatal projection neuron subtypes during in vivo recordings. Brain research, 1511:21-32, 2013.
Patrick Latuske, Oana Toader, and Kevin Allen. Interspike intervals reveal functionally distinct cell populations in the medial entorhinal cortex. Journal of Neuroscience, 15911160131 2015.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. leee Access, 8: 1919997-1999338 2020.
Candice Lee, Andreanne Lavoie, Jiashu Liu, Simon Chen, and Bao-hua Liu. Light up the brain: the application of optogenetics in cell-type specific dissection of mouse brain circuits. Frontiers in neural circuits, 14:18, 2020.
Eric Kenji Lee, Hymavathy Balasubramanian, Alexandra Tsolias, Stephanie Udochukwu Anakwe, Maria Medalla, Krishna V Shenoy, and Chandramouli Chandrasekaran. Non-linear dimensional- ity reduction on extracellular waveforms reveals cell type diversity in premotor cortex. Elife, 10: e67490, 2021.
Eric Kenji Lee, Asim Gul, Greggory Heller, Anna Lakunina, Santiago Jaramillo, Pawel Przytycki, and Chandramouli Chandrasekaran. thysmmp-interppetaalee vivo neuronal cell type identifica- tion using multi-modal analysis of clectrophyyiological data. bioRxiv, pp. 2024-02, 2024.
Qiang Li, Zuwan Lin, Ren Liu, Xin Tang, Jiahao Huang, Yichun He, Xin Sui, Weiwen Tian, Hao Shen, Haowen Zhou, al. Multimodal charting of molecular and functional cell states via in situ electro-sequencing. Cell, 186(9)2002-22117, 2023.
Paul Pu Liang, Zihao Deng, Martin Q Ma, James Y Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36, 2024.
Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang. Jinyang Li, Suhas Diggavi, Mani Srivastava, and Tarek Abdelzaher. Focal: Contrastive learning for multimodal time-series sensing signals factorized orthogonal latent space. Advances in Neural Information Processing Systems, 36, 2024.
Jeremy Magland, James J Jun, Elizabeth Lovero, Alexander Morley, Cole Lincoln Hurwitz, Alessio Paolo Buccino, Samuel Garcia, and Alex H Barnett. Spikeforest, reproducible web-facing ground-truth validation of automated neural spike sorters. Elife, 9:e55167, 2020.
Richard H Masland. Neuronal cell types. Current Biology. 14(13)) 2004.
RT Matthews and WL Lee. A comparison of extracellular and intracellular recordings from medial septum/diagonal band neurons in vitro. Neuroscience, 42(2):451-462, 1991.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint /802.0346, 2018 8a.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018b.
Lu Mi, Trung Le, Tianxing He, Eli Shlizerman, and Uygar Sümbül. Learning time-invariant rep- resentations for individual neurons from population dynamics. Advances in Neural Information Processing Systems, 36:46007-46026, 2023.
Veroo Mountcastle, William H Talbot, Hideo Sakata, and J Hyvärinen. Cortical neuronal mecha- nisms in flutter-vibration studied in unanesthetized monkeys. neuronal periodicity and frequency discrimination. Journal of neurophysiology, 32(3):452 484, 1969.
Lionel G Nowak, Rony Azouz, Maria V Sanchez- Vives, Charles M Gray, and David A McCormick. Electrophysiological classes of cat primary visual cortical neurons vivo as revealed by quanti- tative analyses. Journal of neurohyssology, 89(3)1541-11666 2003.
14

---

Published as a conference paper at ICLR 2025
Marius Pachitariu, Shashwat Sridhar, Jacob Pennington, and Carsen Stringer. Spike sorting with kilosort4. Nature Methods, pp. 1-8, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, Pp- 8748-8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: /908. 10084, 2019.
Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3875-3879, IEEE, 2021.
Aidan Schneider, Mehdi Azabou, Louis McDougall Vigier, David Parks, Sahara Ensley, Kiran Bhaikaran-Nair, Tomasz Nowakowski, Eva L Dyer, and Keith B Hengen. Transcriptomic cell type structures vivo neuronal activity across multiple timescales. Cell reports, 42((), 2023a.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):360-333 2023b.
Nicholas A Steinmetz, Christof Koch, Kenneth Harris, and Matteo Carandini. Challenges and opportunities for large-scale electrophysiology with neuropixels probes. Current opinion in robiology, 50:92-100 2018.
Nicholas A Steinmetz, Cagatay Aydin, Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, et al. Neuropixels 2.0: A minia- turized high-density probe for stable, long-term brain recordings. Science, 372(6539))aabf4588, 2021.
Susan M Sunkin, Lydia Ng, Chris Lau, Tim Dolbeare, Terri I. Gilbert, Carol I. Thompson, Michael Hawrylycz, and Chinh Dang. Allen brain atlas: integrated spatio-temporal portal for exploring the central nervous system. Nucleic acids research, (1(DI):D96--DIO) 2012.
Bosiljka Tasic, Zizhen Yao, Lucas T Graybuck, Kimberly A Smith, Thuc Nghi Nguyen, Darren Bertagnolli, Jeff Goldy, Emma Garren, Michael N Economo, Sarada Viswanathan, et Shared and distinct transcriptomic cell types across neocortical areas. Nature, 563(7729):72-78, 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16. pp. 776-794. Springer, 2020.
Gemechu B Tolossa, Aidan Schneider, Eva I. Dyer, and Keith Hengen. conserved code for anatomy: Neurons throughout the brain embed robust signatures of their anatomical location into spike trains. bioRxiv, pp. 07, 2024.
Eric M Trautmann, Janis K Hesse, Gabriel M Stine, Ruobing Xia, Shude Zhu, Daniel J O'Shea, Bill Karsh, Jennifer Colonell, Frank F Lanfranchi, Saurabh Vyas, et Large-scale high-density brain-wide neural recording in nonhuman primates. bioRxiv, pp. 2023-02, 2023.
Carolina Urzay, Nauman Ahad, Mehdi Azabou, Aidan Schneider, Geethika Atamkuri, Keith B Hen- gen, and Eva L Dyer. Detecting change points in neural population activity with contrastive metric learning. In 2023 11th International IE IEEE/EMBS Conference Neural Engineering (NER), pp- 1-4. IEEE, 2023.
Ganesh Vigneswaran, Alexander Kraskov, and Roger N Lemon. Large identified pyramidal cells in macaque motor and premotor cortex exhibit "thin spikes": implications for cell type classification. Journal of Neuroscience, 31 31(40): 14235-14442 2011.
Ankit Vishnubhotla, Charlotte Loh. Akash Srivastava, Liam Paninski, and Cole Hurwitz. Towards robust and generalizable representations of extracellular data using contrastive learning. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
Quanxin Wang. Song-Lin Ding, Yang Li, Josh Royall, David Feng, Phil Lesnar, Nile Graddis, Maitham Naeemi, Benjamin Facer, Anh Ho, et al. The allen mouse brain common coordinate framework: 3d reference atlas. Cell, 181(4):936-9)) 2020.
Keiko Weir, Oriane Blanquie, Werner Kilb, Heiko Luhmann, and Anne Sinning. Comparison of spike parameters from optically identified gabaergic and glutamatergic neurons in sparse cortical cultures. Frontiers in cellular neuroscience, 8:460, 2015.
Eric G Wu, Andra M Rudzite, Martin Bohlen, Peter H Li, Alexandra Kling, Sam Cooler, Colleen Rhoades, Nora Brackbill, Alex R Gogliettino, Nishal P Shah, et al. Decomposition of retinal ganglion cell electrical images for cell type and functional inference. bioRxiv, 2023.
Zizhen Yao, Cindy TJ Van Velthoven, Thue Nghi Nguyen, Jeff Goldy, Adriana Sedeno-Cortes, Fahimeh Baftizadeh, Darren Bertagnolli, Tamara Casper, Megan Chiang, Kirsten Crichton, et al. A laxonomy of transcriptomic cell types across the isocortex and hippocampal formation. Cell, 184(12):222-3241, 2021.
Zizhen Yao, Cindy TJ van Velthoven, Michael Kunst, Meng Zhang, Delissa McMillen, Changkyu Lee. Won Jung, Jeff Goldy, Aliya Abdelhak, Matthew Aitken, et al. A high-resolution transcrip- tomic and spatial atlas of cell types the whole mouse brain. Nature, 624(7991):117-332, 2023.
Joel Ye, Jennifer Collinger, Leila Wehbe, and Robert Gaunt. Neural data transformer multi- context pretraining for neural spiking activity. bioRxiv, pp. 2023-09, 2023a.
Zhiwen Ye, Andrew M Shelton, Jordan R Shaker, Julien Boussard, Jennifer Colonell, Daniel Bir- man, Sahar Manavi, Susu Chen, Charlic Windolf, Cole Hurwitz, et al. Ultra-high density elec- trodes improve detection, yield, and cell type identification neuronal recordings. bioRxiv, 2023b.
Yizi Zhang, Hanrui Lyu, Cole Hurwitz, Shuqi Wang. Charles Findling, Felix Hubert, Alexandre Pouget, International Brain Laboratory, Erdem Varol, and Liam Paninski. Exploiting correlations across trials and behavioral sessions to improve neural decoding. bioRxiv, 2024.
Yizi Zhang, Yanchen Wang, Donato Jiménez-Benetó, Zixuan Wang, Mehdi Azabou, Blake Richards, Renee Tung, Olivier Winter, Eva Dyer, Liam Paninski, et al. Towards a" universal translator" for neural dynamics at single-cell, single-spike resolution. Advances in Neural Infor- mation Processing Systems, 37:80495-80522 2025.
16

---

Published as a conference paper at ICLR 2025
A IBL BRAIN-WIDE MAP DATASET
The full IBL Brain-wide Map dataset contains 699 insertions. Among those, only 675 successfully went through the pre-processing. with 37017 units that pass the IBL spike-sorting quality control. The region distribution of those units are shown in Supplementary Figure
8000
6000
Counts
4000
2000
CNU
CTXsp
HPF
HY lsocortex
MB
TH
Regions
Supplementary Figure 1: Region distribution of the IBI. Brain-wide Map dataset. The dataset very imbalanced and has small number of units in hypothalamus and cortical subplates.
B BASELINES AND HYPERPARAMETERS
B.1 MODEL ARCHITECTURE
a
1
b
to2
2d covolutional
average pooling
10
18
172
sctivation
hully connected
S
cropout
comr
Supplementary Figure 2: Encoder architecture illustration: a) A two-layer multilayer perceptron for waveform encoding, and b) A convolutional neural network for ACG image encoding.
17

---

Published as a conference paper at ICLR 2025
B.2 MODEL PARAMETERS AND AUGMENTATIONS
We apply the model augmentations in table on the training data during NEMO. p is the proba- bility am augmentation gets applied an instance. We use the Adam optimizer with learning rate 0.0005 and Cosine Annealing scheduler with To 20. Other model hyperparameters are in table Supplementary Figure shows each of the ACG augmentations on an example ACG image.
Supplementary Table 1: NEMO Augmentations
Augmentation
Description
Type
Gaussian noise
Gaussian noise with mcan 0 and std 0.1 xtd of WVF
WVF
0.3
Electrode dropout
channel with with probablity Rescales chunnel's umplitude by uniform value between 0.9 and 1.1 Smooths ACG using Gaussian filter along the temporal axis
Randomly
MC-WVF
0.1
Amplitude Jitter noise
MC-WVF
0.3
Temporal Gaussian smoothing
ACG
0.5
with
Jitters the temporal axis of an ACG by random inleger between
Temporal jitlering
ACG
-3 and 3 inclusive
Scales the amplitude of an ACG by random number between
Amplitude sculing
ACG
0.5
Additive Gaussian noise
Adds Gaussian noise with mean and std 0. maximum of ACG
Multiplicative pepper noise
Each value ACG has a 5% being set
ACG
0.5
Supplementary Table 2: NEMO Hyperparameters
Parameter
Description
Value
Epochs Batch Size Learning Rate
Numbers of epochs to run
6000
Number of items processed in single operation
1024
Learing rate of the model
0.0005
Log every steps
Save model and cross validate results every epochs
100
dim .embedded
Latent dimension size
512
Temperature
in the formula for contrastive loss
0.5
ACG dropout
Dropout probability ACG encoder
0.2
Waveform dropout
Dropout probability in waveform encoder
0.1
3D-ACG augmentation
original
temporal gaussian smooihing
temporal jitter jitter
80
Firing (spikesis)
Firing rate decile
10
100
1003
10
100
1000
10
100
1000
multiplicative
emplitude jitter
additive gaussian noise
pepper noise
0
10
100
1000
0
10
100
1000
0
10
100
1000
Time from triggered spike (ms)
Supplementary Figure 3: Illustration of the augmentations on spiking activity ACGs used by NEMO.
B.3 MLP FUNIIN
For cell-type classification, we fix the hyperparameters the MLP classifier for all methods to those used in Beau et al. (2025). This is because have very few labels for evaluation and it challeng-
18

---

Published as a conference paper at ICLR 2025
ing to hold out any for hyperparameter tuning. For the brain region classification experiments, tune the optimizer, learning rate, model size and dropout using the validation set of insertions.
B.4 SIMCLR-BASED NEMO
For the SimCLR-based NEMO, we randomly apply the same set of augmentation methods to each data point, generating two correlated augmented views. We then train each encoder and linear pro- jection layer to maximize agreement using contrastive learning. Compared to other methods, we use stronger augmentations for SimCLR. This because SimCLR focuses on the differences between augmentations from the same data point, whereas CLIPP-ased learning strategies compare augmen- tations across different modalities. Stronger augmentations improved SimCLR's performance.
Both methods use contrastive learning to maximize similarity between augmented views of the same neuron, while keeping views of different neurons distinct. The main difference between the SimCLR bassed method and the CLIP-based method is that CLIP is multimodal and SimCLR is uni- modal. For the SimCLR-basee Let 2viewl and vieew2 be the normalized projections of two augmented views from the same modality. For a batch B, the objective is as follows:
method, we use the contrastive objective from (Chen et al. 2020).
exp (vieww, -view2, /T)
log
B
B Σ
=1 (ittt) exp( (zriewll 2view2,/ /T) + E 1 (ixt) exp(2riewl, Zviewl, (ileww,/T] /T)
exp( /Zviow,/TT)
+ log
E/11 1 (ittt exp(2view2, Zview1, 2view1,/T) +
Σ 1 (ive) exp(2view2 Zyiew2, (T)
(2)
For SimCLR, we use the same types of augmentation as CLIP with increased strength: for the EAPs, we applied Gaussian noise with standard deviation 1 with probability 0.5, rather 0.1 standard with a probability of than 0.3. For the ACG images, we keep the augmentation methods consistent across all models.
In SimCLR-based method, each modality, which are then combined to train downstream tasks such as cell-type and brain region classification.
the pretrained encoder to obtain separate embeddings for
C REMOVING STIMULATION ARTIFACTS FROM THE NP ULTRA DATA SET
The optotagging experiments for the Neuropixels Ultra dataset included two distinct photostimu- lation patterns: a 10 ms square wave pulse and s raised cosine ramp, each delivered at of three light intensities (0.2, 4.1, and 10.0 m mW/mm??, randomly interleaved. The average inter-trial interval was 2 s. To eliminate stimulus- stimulusinduuced induced effects, we excluded spikes occurring within 1100 of each stimulus onset and computed ACG images using only the remaining data. The ACG im- ages followed the approach in (Beau et al. 2021). with one modification: we normalized each bin's value solely by the number of spikes whose time lag around that bin's 's spiking time did not overlap with any stimulus period (i.c., the 1 100 ms post-stimulus window). This adjustment mitigates edge effects introduced by blank periods during stimulation.
D DATASET SPLIT AND VALIDATION STRATEGY
Given the limited quantity of ground-truth labels for the NP Ultra dataset and the C4 cerebellum dataset, we evaluate the performance of each model using 5-fold cross-validation of the labeled cells with 10 repeats. Since we do not have separate validation set while training NEMO, we utilize a nested cross-validation approach to choose an evaluation checkpoint for NEMO. In other words, for each vross-validation fold, perform nested 5-fold cross-validation a linear classifier to choose the best checkpoint for NEMO. We then use this NEMO checkpoint to perform the evaluation the heldout fold of the original 5-fold cross-validation This checkpoint choosing procedure is done only on the training folds and does not use any information from the testing fold.
the training folds with
For the IBL dataset, we use a standard 70-10-20 split for the training, validation, and test sets, respectively. During training, we monitor the performance of the model using linear classifier
19

---

Published as a conference paper at ICLR 2025
trained the training set and validated the validation set. We then compute evaluation metrics on the test set using the checkpoint with the highest validation Fl.
E SINGLE-CHANNEL NP ULTRA CELL-TYPE CLASSIFICATION
Utilizing single-channel waveforms, NEMO outperforms baseline methods across all evaluation methods with default parameters. See Supplementary Table for summary of these results, Com- pared to multi-channel results in the main text, however, the performance of all methods are lower, indicating that there is important cell-type information in the multi-channel templates.
Supplementary Table 3: Cell-type classification for the NP Ultra dataset. The accuracy and Fi-scores are reported for three conditions: (i) a linear layer and (ii) MLP of the frozen pretrained representations (for VAE and NEMO), and (iii) after MLP finetuning.
Model
Linear
MLP
MLP fine-tuned
Acc
F1
Acc
Acc
Supervised
N/A
N/A
N/A
N/A
0.73 = 0.01
0.73 = 0.01
PhysMAP
0.53 + 0.00
0.49 + 0.00
N/A
N/A
N/A
N/A
VAE (10d latent)
0.74 + 0.01
0.74 + 0.01
0.74 + 0.00
0.74 + 0.00
0.75 0.00)
0.75 0.00
VAE (500d rep)
0.74 - 0.01
0.74 = 0.01
0.74 + 0.01
0.75 0.01
0.76 0.01
0.76 0.00
NEMO (500d rep)
0.75 + 0.01
0.75 0.01
0.77 0.01
0.77 1 0.01
0.78 0.00
0.78 + 0.00
F CELL-TYPE CLASSIFICATION ON THE C4 CEREBELLUM DATA
We evaluate NEMO in comparison all baseline models on the C4 cerebellum opto-tagged dataset using 5-fold cross validation. We utilize the same training pipeline for the VAE as proposed in Beau al. (2025) including the data augmentations. Supplementary Figure show the results of this analysis. Across all evaluation schemes, NEMO outperforms the baseline models.
Waveform
ACG
Waveform ACG
0.85
Belanced Accuraoy
0.80
0.75
0.70
0.85
Lace'ss
GoC
0.55
Lineer probe
MLP fine-tuned
PkC
0.45
a.U.
a.u.
S.U.
NEMO
(rib))
NEMO hotw
(ube)
unt man
Modele supervised
NEMO linear
NEMO
VAE
PhysMAP
Goc
OcC BBr 0200220000 D.B8..0
GoC-O.CD 04000400030000 1111
Goc
Mu 0.17 0.175
0010000.000
0.000.00
MLl
0.24 24
000000000000
MU
0.00 190:0002200000 0:30
MU
9.220 2200c00000 100000300000 0.04
000000000
0000000
1000000000000000000000000 1000000
4.07 4700000
-020.0 3
PAC_
PKC,
PHO.
PKC,
M//
MPM
PaC
MP3
MI
MPS
mel
MB
PH
Predicted latel
Supplementary Figure 4: Comparing NEMO to bascline models the C4 cerebellum opto-tagged dataset. (a) UMAP visualization of the pretrained NEMO representations of unseen opto-tagged cerebellum units, colored by different cell-types. Neurons of the same class form clusters, particularly when combined modalities are used. (b) Balanced accuracy and (c) Confusion matrices for the C4 cerebellum classification results, normalized by ground truth label and averaged across 5 random seeds. NEMO outperforms the other embedding methods by significant margin across all cell-types and evaluation methods.
20

---

Published as a conference paper at ICLR 2025
G TRANSFER LEARNING BY PRETRAINING ON THE IBL BRAIN-WIDE MAP
After pretraining NEMO on the IBL Brain-wide Map dataset, we evaluated how well the model performs at single-channel cell-type classification of the NP Ultra dataset before and after fine- tuning. For this experiment, we have three models: (1) NEMO (Ultra) pretrained and evaluated on NP Ultra data, (2) NEMO (IBL) pretrained on IBL data and evaluated on NP Ultra data, (3) NEMO (IBL) pretrained on IBL. data and NP Ultra data and evaluated on NP Ultra data. Interestingly. we found that the pretrained IBL model performed well on cell-type classification even without training examples from NP Ultra (see Supplementary Table 4).
Supplementary Table 4: Transfer learning using IBL data for pretraining and the NP Ultra dataset for evaluation. The accuracy and Fl Fl-sccsss are reported for three conditions: (i) linear layer and (ii) MLP on top of the frozen pretrained representations (for VAE and NEMO), and (iii) after MLP finetuning.
Model
Linear
MLP
MLP fine-tuned
Acc
F1
Acc
F1
Acc
F1
NEMO (Ultra)
0.75 0.01
0.75 0,01
0.77
0.01
0.77 + 0.01
0.78
0.00
0.78 : 0.00
NEMO (IBL)
0.75 0.01
0.75
0.01
0.77
0.01
0.77
0.01
0.78
0.00
0.78 0,00
NEMO (IBL+Ultra)
0.76 0.01
0.76 0.01
0.77
0.01
0.77 - 0.01
0.77 0.01
0.78 0.01
H MULTI-CHANNEL IBL BRAIN REGION CLASSIFICATION
For the IBL. multichannel experiments, we use 25 channel templates centered on the peak channel if possible. We add two additional template augmentations: amplitude jitter (Vishnubhotla et al., 2024) and electrode dropout. Amplitude jitter rescales a channel's amplitude by a uniform value between 0.9 and 1.1 with p 0.3. Electrode dropout zeros out a channel with p 0.1. If all channels are zeroed out, we leave the peak channel. Our model architecture is the same as NEMO with the only difference being the input size of the waveform encoder.
We tune hyperparameters for both the linear and MLP classification models. For the linear model, we tune the inverse of the regularization strength between le and 1e4 using the python mod- ule optuna (Akiba et al.) (209)). For the MLP model, we do grid search over the dropout probability 0011 0.2, 0.3, 0.4), learning rate(1e- 4, le - 5, le 6), number layers 2, and layer size 128, 256, 512).
Supplementary Table 5: Multi-channel, single-neuron brain region decoding for the IBI. dataset
Model
Linear
MLP
MLP fine-tuned
Ace
F1
Acc
F1
Acc
F1
NEMO
0.42 + 0.00
0.40 0.00
0.45 0.01
0.42 = 0.00
0.47 1 0.01
0.44 + 0.01
NEMO (25-channel)
0.45 + 0.01
0.42 0.00
0.46 + 0.00
0.43
0.00
0.48 + 0.00
0.45 + 0.00
Supplementary Table 6: multi-neuron, multi-channel brain region classification for the IBL dataset
Model
Linear
MLP
MLP fine-tuned
Acc
Acc
F1
Acc
F1
NEMO
0.48 + 0.00
0.45
0.00
0.50 + 0.00
0.48 = 0.00
0.51 + 0,00
0.50 + 0.00
NEMO (25-channa)
0.50 + 0.01
0.47 = 0.00
0.51 +: 0.01
0.48 = 0.01
0.52 1 0.00
0.51 : 0.01
I PICKING PARAMETERS FOR IBL UNIT REPRESENTATION CLUSTERING
We used the eethon-implemented UMAP package (McInnes al. 201 8b) and the Python-Louvain package (Aynaud, 2020) for our clustering analyses. For our clustering analysis, we aim to find the most informative clustering with the smallest number of clusters. There
parameters to tune: 1) the size of local neighborhood used for the UMAP graph manifold approximation resolution that controls the community size in Louvain clustering (Mneighhor). and 2) the resolution y that
21

---

Published as a conference paper at ICLR 2025
determines the size of the communities. We tuned Aneighhor while keeping the resolution to be 1.0 and tracked the final number of clusters. We picked the 'elbow' that has the smallest Rneighbor (200). The resolution was chosen by maximizing the modularity index Q of Louvain clustering with Rneiighor = 200. The modularity index of graph is defined as:
Le
he 2m
Q
where each e represents a community in the graph, m is the number of edges, Le is the number of intra-community links for community c, ke is the sum of degrees of the nodes in community c, and is the resolution parameter.
J PARAMETERS FOR CLASSIFIERS AND FINE-TUNING METHODS
Supplementary Table 7: Linear probe best hyperparameters
Hyperparameter
Value
maTster
1000
tol
le-5
NP Ultra Celltype
0.02
IBL NEMO joint
0.02
IBL VAE e
0.4
IBL supervise
0.001
IBL NEMO independent c
2.5
Supplementary Table 8: IBI. MLP hyperparameters
Hyperparameter
Value
Maaer
layerO size
256
Dropout rate
0.2
Supervised scheduler Supervised To
CosineAmnealing WarmRestarts
20
Supervised Trmut
1
Other models' schedular
StepLR
StepLR slepsize
200
StepLR 7
0.8
Supervised Ir
1 10-1
Other models'
x 10-5
22

---

Published as a conference paper at ICLR 2025
J.1 UMAP EMBEDDINGS WITH RAW FEATURES, RANDOM PROJECTION AND VAE
We visualize the UMAP embeddings the raw features, randomly initialized encoder projection and the VAE projection for the C4 cerehellum dataset in Supplementary figure [ We find that NEMO representations (shown in Supplementary Figure are visually more structured.
Labes
Waveform
ACG VAE
Waveform ACG
GsC
MLI
PKC_44
a.u.
aa.u.
2au
2au..
Random projection
MFB
PKC_es
a a.u.
2 a.u..
2auu.
2a.u.
Raw
Lasels
MU
PKC_
a.u.
B.U.
2au..
2.u.
a.u.
Supplementary Figure UMAP visualizations using VAE, random projections, and raw features. The random projections are representations which are passed through randomly initialized encoders with the same architecture NEMO.
23

---

Published as a conference paper at ICLR 2025
a
Linear probe
MLP
b
Macro averaged F1 by modality
0.45
NEMO VAZ
0.4
Macro averaged F1
Macro averaged F1
0.40
averaged
0.40
0.3
0.35
0.35
0.2
0.30
VSE Beadn VIE
0.30
0.1
NESOO Teois
saperite
0.25
0.25
NEMM
0.0
0.0
0.2
0.6 Label ratio
0.4
0.8
1.0
0.0
0.2
0,4
0.6 Label ratio
0.8
1.0
both
3D-ACG Modality
WVF
Supplementary Figure 6: Left and middle, macro-averaged F1 scores for linear and MLP-based single-neuron classification of brain region by label ratio, as in Fig. (jd) replacing accuracy with F1 score. Right, single-neuron ILP-classification balanced F1 scores for uni- and bimodal models, as in Fig. e)), replacing accuracy with F1 score.
J.2 IBL REGION CLASSIFICATION WITH LABEL RATIO SWEEP
we show the macro-averaged F1 scores for single neuron classification of brain region by label ratio as complementary to Figure
In Supplementary Figure 6
In Supplementary Figure 7 we study the effect of varying the ratio of labeled data used to train brain region classifier. Fully supervised methods used the same labeled examples, but were not pretrained. Means and standard-devinee
bands computed over five random initializations for each label ratio. Supplementary Figure g shows the macro- macro-averaged FI scores for single neuron classification of brain region by label ratio with the IBL dataset, complementary to Figure
24

---

Published as a conference paper at ICLR 2025
NEMO NEMO freeze
SimCLR
Supervise
WAE freeze
SimCLR freeze
VAE
Precision
Recall
F1-Score
0.6
0.5
F1-Score
Recall
8
0.4
0.3
0.2
0.2
1.0
0.0
0.0
Label
Label
Precision
Recall
F1-Score
F1-Score
0.4
ONU
Recall
0.3
0.2
0.2
0.3
0.1
1.0
Label Ratio
Label Ratio
Label
Precision
Recall
F1-Score
0.12
0.15
CTXsp
0.10
Recall
0.2
F1-Score
0.10
0.08
0.08
0.05
0.0
Label Ratio
Label Ratio
Label Ratio
Precision
Recall
F1-Score
0.30
0.6
0.35
0.25
0.5
F1-Soore
Recall
0.30
0.20
0.25
0.15
Label Ratio
Label Ratio
Label Ratio
Precision
Recall
F1-Score
0.5
0.5
0.4
F1-Score
HPF
Recail
0.3
0.3
0.2
0.1
0.5
1.0
0.5
0.0
0.5
Label Ratio
Label Ratio
Label Ratio
25

---

Published as a conference paper at ICLR 2025
NEMO
SimCLR
Supervise
VAE freeze
NEMO freeze
SImCLR freeze
VAE
Precision
Recall
F1-Score
0.10
0.4
0.150
0.125
0.08
0.3
F1 F1-Soore
Recall
0.06
0.2
0.075
0.04
0.1
0.050
0.5
1.0
0.0
1.0
0.0
0.5
1.0
Label
Label Ratio
Precision
Recall
F1-Score
0.6
0.7
Isoconex
F1-Score
0.6
Recail
0.6
0.5
0.5
0.5
0.4
0.4
1.0
0.0
0.0
Label Ratio
Label Ratio
Label Ratio
Precision
Recall
F1-Score
0.45
0.50
0.4
0.40
0.45
Recail
F1-Score
MB
0.40
0.3
0.35
0.30
0.2
0.25
1.0
0.0
0.0
Label Ratio
Label Ratio
Label Ratio
Precision
Recall
F1-Score
0.25
0.3
0.20
Recail
0.3
F1-Score
0.2
0.15
0.2
0.10
0.1
0.1-
0.05
0.0
0.0
0.5
1.0
0.0
0.0
0.5
Label Ratio
Label Ratio
Label Ratio
Precision
Recall
F1-Score
0.70
0.65
0.7
E
Recail
0.60
F1-Scare
0.6
0.60
0.55
0.55
0.50
0.50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Label Ratio
Label Ratio
Label Ratio
Supplementary Figure 7: Detailed comparison of NEMO and SimCLR for region classification performance IBL data with varying label ratio. We show the precision, recall and F1 score for each class. NEMO shows superior performance in most of the classes, except hypothalamus (HY), which has small sample size compared other classes.
26

---

Published as a conference paper at ICLR 2025
K INDEPENDENT LEARNING ABLATION FOR NEMO
We ablate the cell-type classification performance between NEMO with independent (SimCLR) and joint learning (CLIP). The results are shown Supplementary table and Supplementary table I0] Joint training improves cell-type classification for both datasets.
Supplementary Table 9: Single-channel cell-type classification for the NP Ultra dataset for indepen- dent joint learning NEMO.
Model
Linear
MLP
Finetuned MLP
Acc
Acc
Acc
F1
independent NEMO
0.74 1 0.05
0.74 + 0.01
0.75 + 0.01
0.75 - 0.00
0.76 + 0.02
0.77 =: 0.00
joint NEMO
0.75 0.01
0.75 + 0.01
0.77 + 0.01
0.77 0.01
0.78 + 0.00
0.78 0.00
Supplementary Table 10: Single-channel cell-type classification for the C4 dataset for independent vs. joint learning NEMO
Model
Linear
MLP (5-fold)
Finetuned MLP (5-fold)
Ace
Aec
F1
Acc
independent NEMO (500d rep) joint NEMO (500d rep)
0.82 + 0.00
0.81
1 0.00
0.82 1 0.01
0.82 1 0.00
0.84 1 0.01
0.84
0.01
0.01
0.85
0.85:
0.85 + 0.00
0.86 + 0.01
0.86
a
Linear probe
MLP
averaged by modality
0.45
0.40
Medro averaged F1
0.4
Mecro avereged
0.40
Macro averaged
0.35
0.30
0.30
NEMO NEMO besun SinCUR SinOLR froeco
SinCLR
0.1
NEMO
0.25-
NEMO fieuce
SiMCUR
NEMO
0.20
0.0
0.0
0.2
0.4
0.6
0.8
1.0 0.0
0.2
0.4
0.6
0.8
1.0
Waveform ACG ACG
Waveform
Label
ratio
Label ratio
Modality
Supplementary Figure 8: Left and middle, macro-averaged F1 scores for linear and MLP-based single-neuron classification of brain region by label ratio for independent learning ablation, in Fig. 5(a), replacing accuracy with Fl score. Right, single-neuron MLL-classification balanced scores for uni- and bimodal models, as in Fig. 5jb), replacing accuracy with Fl score.
27

---

Published as a conference paper at ICLR 2025
a
Single-neuron decoder with fine-funed models
Single-neuron decoder with PhysMAP
Supervised
VAE
CB
CNU
CNU
CTKsp
True label
puu label
+PF
0:5
HY
lsccortex
MB
MR
OLF
OLF
Hy lsocuren
CRU
Predicted label
0.00
0.75
Predidted label 0.00
0.75
Multi-ncuron decodor with fine-tuned models
Suparvised
VAE
CNU
CTXap
6.H
Trun label
HPF
HY
loocortev
OLF
Hy laneurtea
MG
Hy loncuree
Predicted label
0.00
0.75
Supplementary Figure 9: Classification results for the VAE, supervised MLP, and PhysMAP on the IBL. brain region classification task. Due label imbalance, PhysMAP unable predict CTXsp and HY, which leads to a low balanced accuracy and Fl score.
28

---

Published as a conference paper at ICLR 2025
L IBL UNIT CLUSTERING RESULTS WITH NEMO
Supplementary Figure 10 shows averages and standard deviations for each cluster's template wave- forms and ACG images. Supplementary Figure shows the distribution of cluster labels over brain regions, separated by individual insertions and by the recording lab.
a
Cluster templates
Class 0 Waveforms (429 units)
Class Waveforms (3920 units)
Class Waveforms (3183 units)
Class 3 Wavetorms (2034 units)
Class Waveforms (966 units)
Class 5 Waveforms (4682 units)
Class (2627 units)
Class (2236 units)
Waveforms (805 units)
Waveforms (853 unils)
(3378 units)
ms
ms
ms
ms
b
Average images
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
Firing rale decle
10
10
10
Firing rate (spikesls)
10
100
10
100
100
100
1003
-11
Cluster
Cluster
Cluster
Cluster
Cluster
Firing rate decie
100
100
1000
10
100
1000
100 1000
100 1000
100 1000
Time from triggerad spike (ms)
C
Standard devistion of ACG images
Cluster
Cluster
Cluster
Cluster
Cluster
Cluster
Fining rale docile
Firing rate (seliie (
10
100 10
1030
10
100
1000
10
100
1003
10
100 1000
10
100 1000
10
100
1000
10
Firing
100
10
100
1000
10 100
1000
10
100
1000
10 100
1000
10 100
1000
Time triggered spike (ms)
Supplementary Figure 10: Average and standard deviation of the template waveforms and ACGs for the units in each cluster, as shown in Figure 4 (a) The waveforms are consistent within clusters and distinct across clusters. (b) The ACG images are are also distinct across clusters. These results suggest that NEMO is able to find distinct clusterings of neurons across the whole-brain.
M IBL UNIT CLUSTERING RESULTS WITH RAW FEATURE INPUT
shows clustering results for the IBL dataset. The hyper-parameten were selected with similar criteria as in Section Since the number of clusters does not show a similar 'elbow,' but keeps decreasing Rneighhor increases, lution that maximizes the the modularity and minimizes the number of clusters. These clusters are less spatially organized compared the clusters clustered using NEMO. As shown in Supplemen-
Supplementary Figure 12
picked Nneighhor 1000 and used reso-
29

---

Published as a conference paper at ICLR 2025
Depth relalive UG-Thalamus (mm)
4
5
3 4
5
6
7
2
2
3
4
1
2
3
5
2.0
RE
PPC
1.5
1.0
0.5
0.0
-0.5
-1.0
MB
-1.5
-2.0
Supplementary Figure 11: Visualization of clustering results across repeated site in the IBL dataset (IBL et al., 2022). The Neuropixels probes target the same brain regions (including posterior pari- etal cortex, hippocampus, and thalamus) in these insertions. The color of the labels on top of each column indicates the lab of each insertion. Our results reveal that the clusters are highly dis- tinguishable by region, with each region containing distinct group of neurons. Moreover, the dominant cluster IDs for the same region remain consistent across different insertions.
tary Figure 13 the clustering result based on NEMO overall shows lower entropy, which indicates higher region-selectivityy
a
b
25
region colored
cluster colored
Cluster number
10
Regione
Cb
Clssters
Cluster
0
200
500
1000
Cluster
neighton
HPF
HB
Cluster 4
Clueter
Cluster 3
Modularity Index
0.70-
0.71
40
Number
uni
OU*
M6
unt
Cluster
Cluster Cluster
0.69-
30
units
TH
units
0.88-
20
0.67
Horizontal Slice
10
0.5
Resolution
d
Coronal Slice
Sagittal Slice
2mm
2mm
2mm
2mm
2mm
Supplementary Figure 12: IBL neuron clustering using raw features. (a) A UMAP visualization of raw features on the training data colored by anatomical brain region. (b) The same UMAP as shown in (a) but instead colored by cluster labels using a graph-based approach (Louvain clustering). (c) We tuned the neighborhood size in UMAP and the resolution for the clustering. These parameters were selected by maximizing the modularity index which minimized the number of clusters. (d) 2D brain slices across three brain views with the location of individual neurons colored using the cluster IDs shown in (b). The black lines show the region boundaries of the Allen mouse atlas.
30

---

Published as a conference paper at ICLR 2025
a
NEMO
b
Raw feeture
CNU
GB-
GB
0.8
1.25
CTXsp
0.5
HB
HB
0.4
HPF
HPF
1.10
Isccortex
MB
HY
Isocortex
HY
0.2
1.00
1.05
OLF
OLF
0.1
0.95
raw feature
TH
0.90
NEMO
Clusters
10
Clusters
CB
CNU
CTXsp
HB
HPP
HY
bocarten
MB
OLF
TH
Region
Supplementary Figure 13: Region distribution of clustering results using different NEMO features and raw features (normalized by region). For each region, we get cluster distribution vector. We then computed the normalized entropy of that distribution. Cluster result based on NEMO overall shows lower entropy, which indicates higher region-selectiiityy
N SENSITIVITY TO PARAMETERS
N.1 MULTI-CHANNEL NP ULTRA HYPERPARAMETER SWEEP
To assess the sensitivity of hyperparameters for NEMO and the VAE baseline, we specify range of values for each hyperparameter and randomly sample across 50 models on one seed. For the learning rate, we draw samples from a log-uniform distribution between le-4 and 5e-3. Dropout values are sampled from uniform distribution between and 0.4. The latent dimension size is randomly selected from the set (10, 20, 256, 512, 1024). For the VAE, the learning rate is restricted to the range le-4 to 5e-4 to prevent gradient explosions. We report both median and best performance for each model across hyperparameter configurations. Notably, since this dataset lacks a validation set, the best best-performing model is reported for illustrative purposes only. Supplementary Table [T] shows the median performance and Supplementary Table 12] shows the best performance. We find that NEMO outperforms the VAE-based model for both the median and best performing models.
Median performance the VAE and NEMO model for different hyperpa- rameters. The accuracy Fi-scores are reported for three conditions: (i) a linear layer and (ii) MI. on top of the frozen pretrained representations (for VAE and NEMO), and (iii) after MLP finetuning.
Supplementary Table 11:
Model
Linear
MLP
MLP fine-tuned
Acc
F1
Acc
F1
Acc
F1
VAE (latent; Beau et al. (2025))
0.74
0.73
0.73
0.72
0.78
0.78
VAE (representation) NEMO
0.76
0.76
0.77
0.77
0.78
0.79
0.78
0.78
0.80
0.79
0.80
0.80
Supplementary Table 12: Best evaluation performance of the VAE and NEMO model for different
hyperparameters. accuracy and Fi-scores are reported three conditions: (i) linear layer and (ii) MLP on top of the frozen pretrained representations (for VAE and NEMO), and (iii) after MLP finetuning.
Model
Linear
MLP
MLP fine-tuned
Aсc
F1
Ace
F1
Ace
F1
VAE (latent; Beau et al. (2025)
0.77
0.76
0.77
0.77
0.79
0.79
VAE (representation) NEMO
0.77
0.77
0.79
0.79
0.80
0.80
0.80
0.80
0.81
0.81
0.81
0.81
31

---

Published as a conference paper at ICLR 2025
N.2 AUGMENTATION ABLATIONS
To assess the importance of augmentations, we conducted an ablation experiment two parts: (1) starting with the full set of augmentations, removed one augmentation at time, and (2) starting with no augmentations, we added one augmentation at a time. The resulting balanced accuracy and macro-averaged Fl scores are presented in Supplementary Figure I4. The results indicate that additive Gaussian noise in the ACG images is the most impactful augmentation, while the absence of augmentations significantly degrades performance. For other augmentations, the combined effect is large while the individual contribution is smaller.
Augmentation Ablation Macro Averaged F1 Score
0.80
Macro Averaged F1
0.78
0.76
0.74
0.72
0.70
Augmentation Ablation Balanced Accuracy
0.80
Balanced Accuracy
0.78
0.76
0.74
0.72
Remove one Add one
0.70
Ene IR
00
D
D poroorn
0 due
the auoa
your prnnnnee phuppe
pooeeee unnear Doneoo
Desppe
aaaag biccide
poseal dus
spenni do
soddag angappe
ueismee
Terroteedeee reodun
Bompord
Supplementary Figure 14: To assess the impact of our augmentations, we performed two analyses on the multi-channel UHD cell-type classification dataset: (1) removing one augmentation at time, starting from the full set of augmentations, and (2) applying one augmentation at time, starting from no augmentations. For each condition, we calculated the mean and standard deviation of (a) balanced accuracy and (b) macro-averaged FI score with linear classifier.
32

---

Published as a conference paper at ICLR 2025
N.3 MULTI-NEURON NEIGHBOR SWEEP
To evaluate the significance of the number of neurons we choose in our multi-neuron brain region classification model, we conduct a sweep over the maximum number neighboring neurons (3, 5, 9, 15, 25) that can be selected within a 60-micron radius on the IBL dataset. As illustrated in Supplementary Figure [5| the overall classification performance improves as the maximum number of neighboring neurons increases, reaching saturation point when this number reaches ~10.
Average F1 score by rumber reuatons
Juarage Selanced accuracy by numberr of neurom
0.42
Mab chansel
0.4
044
5ng sthurmrl
Mab cramei ME3"
Sky echansel Ha Reseuured
Nah charne V& fraetared
0.44
20
25
Mamber ofne.mm
Number of TaNEISE
Supplementary Figure 15: Macro-averaged Fl and Balanced accuracy over 5 seeds of NEMO with multineuron classifier over different number of max neighboring neurons for single and multi- channel IBL with linear, frozen MLP, and MLP fine-tuning.
N.4 IBL CLUSTERING WITH DIFFERENT RANDOM SEEDS
We trained five instances of NEMO with different random seeds and clustered the neurons using the same hyperparameters. The clustering results are visualized in Supplementary Figure T6 where UMAP was applied for dimensionality reduction, with colors distinguishing clusters. To ensure di- rect comparison, we maintain consistent UMAP visualizations across clustering results. The average adjusted Rand Index (ARI) for these outcomes is 0.48 + 0.04, while the adjusted mutual informa- tion (AMI) score is 0.58 + 0.02. These metrics indicate moderate level of agreement among the clustering results.
The boundaries of the clusters subtle, introducing stochasticity into the clustering results. In addition, clustering the full brain dataset results in a coarse segmentation, while significant vari- ability is expected within individual regions. Therefore, these clustering results should mainly be interpreted as exploratory.
O STATISTICAL TESTS
To evaluate performance differences between models, we conducted two-sample one-tailed t-tests (significance threshold < .05) on the macro-averaged F1 scores and balanced accuracies across five random seeds. The results are summarized in significance matrix comparing models: red indicates the model in the column is significantly better than the model in the row, blue indicates it is significantly worse, and white indicates no significant difference.
0.1 CELL-TYPE CLASSIFICATION
Multi-channel NP Ultra opto-tagged mouse data For both linear and MLP classifiers, NEMO significantly outperforms all baselines (Supplementary Figure 17).
C4 opto-tagged mouse data. For both linear and MLP classifiers, NEMO significantly outperforms all baselines (Supplementary Figure 18
O.2 BRAIN REGION CLASSIFICATION
NEMO consistently outperforms all other baselines for both linear and MLP classifiers (Supple- mentary Figure 19 For single-modality brain region classification with the MLP classifier (Sup-
33

---

Published as a conference paper at ICLR 2025
Supplementary Figure 16: Clustering results from five NEMO instances trained with different ran- dom seeds, visualized using UMAP with consistent dimensionality reduction across runs. Colors indicate distinct clusters.
plementary Figure 20
NEMO with the fine-tuned MLP either significantly outperforms or shows no significant difference compared to all other baselines.
O.3 COMPARISON ACROSS LABEL RATIOS
We further compared classifier performance under varying label ratios. For both the linear classifier (Supplementary Figures 21 and 22) and MIP classifier (Supplementary Figures and 24), NEMO significantly outperforms other baselines each label ratio. Notably, with only 50% of the labels, the linear classifier with NEMO either significantly outperforms or shows no significant difference compared all baselines using 100% of the labels. Similarly, the MLP classifier with NEMO achieves comparable or significantly better performance than all baselines using 100% of the labels with only 80% of the labels.
Overall, these results demonstrate that NEMO consistently outperforms or matches the performance of other baselines across multiple evaluation criteria, including cell-type classification and brain re- gion decoding tasks. Its robustness is evident across classifiers, label ratios, and metrics, establishing NEMO as a highly effective model for these challenging neurobiological classification tasks.
34

---

Published as a conference paper at ICLR 2025
NP-Ultra cell-type classification (multi-channel)
Linear classifier balanced accuracy
Linear classifier macro averaged F1
NEMO
NEMO
VAE
VAE
NEMO
HENO
MLP balanced accuracy
MLP macro averaged
NEMO
NEMO
supervised
supervised
MEMO
NEMO frozz
VAE
UAE feettt
sxperisedd
MEMO
NEMO tearr
MAEE
VIE foare
suserviend dddd
significantly better
significantly worse
n.s.
Supplementary Figure 17: One-tailed t-test results for NP Ultra cell-type classification. For both
linear and MLP classifiers, NEMO significantly outperforms matches all baselines.
35

---

Published as a conference paper at ICLR 2025
cell-type classification
accuracy
macro averaged
NEMO
NEMO
#mCUR
6mcLR
VAE
WAE
NEMO
use
NSSO
GMCLR
MLP balanced accuracy
MLP averaged Ft
NEMO
NEMO feece
NEMO freze
SMOUR
SIMCLR
sIMCUR hoczo
SMOLR froze
VAE
VAE hoozo
VAE frecza
HEto
HEMO treer
BinCLR
WmmC braczte
ViE
ME trasop
aupavited
HEMO
NEMO feezr
sinCUUR truize
ViEE
Resom
asseritedd
significantly better
significantly worse
n.s.
Supplementary Figure 18: One-tailed t-test results for NP Ultra cell-type classification. For both
linear and MI.P classifiers, NEMO significantly outperforms matches all baselines.
36

---

Published as a conference paper at ICLR 2025
IBL brain region classification
Linear classiffer balaaneed ascurany
Linear classifeer macro averaged F1
NEMO
NEMO
SmCLR
SimCR
NEMO
GimCLR
ViEE
NEMO
GimCI
MLP balanced accuracy
MLP macro averaged F1
SmCLR 9962
SimCLR freee
WAE
VAE
VAE fro02
WAE free20
supervise
supervise
NEMO
NEMO Remaap
SmXCLR Enear
VAE
UNE treuse
experetto
MEMO
NEMO Erasos
SimCL.R
SaNOUR fmoud
WAE
USE freute
supervito
significantly better
significantly worse
n.s.
Supplementary Figure 19: One-tailed t-test results for IBI. brain region classification. For both
linear and MLP classifiers, NEMO significantly outperforms all baselines.
37

---

Published as a conference paper at ICLR 2025
Single modality balanced accuracy
ACG
Waveform
NEMO
NCMG
NEMO ftee
NEMO Feeze
SimCUR freec
Cin Feeze
VAE frezzo
WAE frezze
supervise
supervise
NESO
HEMO toeso
SenCLA
SinCL feeezz
VRE freeze
acperrio
NEMO
NESO tmeza
SinCUR
Binn freeae
VAE
VKE tower
aqaerie
Single modality macro averaged F1
ACG
Waveform
NEMO frezzo
NEMO frezze
SimCLR
SimCLR
SiMCLR frezz
WAE
VAE
supervise
suparvise
NEMO
NEMO fowas
SmnDL
SimCL feeez
VRE
UsE frante
NEMO
NEMO frenza
SinGLR
Sten freus
VRE
VEE fewzz
significantly better
significantly
Supplementary Figure 20: One-tailed t-test results for IBL brain region classification with single modality and MI.P classifier with end-to-end fine-tuning. For both modality, NEMO significantly outperforms or matches all baselines.
38

---

Published as a conference paper at ICLR 2025
One-tailed t-test linear classifier balanced accuracy
0.01
0.1
0.3
Label
0.8
1.0
SimCUR
NEMO
SimCLR
VAE
NEMO
SIMCLR
VAE
NEMO
SIMCLR
VAE
NEMO
SimCLR
VAE
NEMO
SIMCLR
VAE
NEMO
SImCLR
VAE
0.01
0.1
0.3
0.5
0.8
1.0
Label ratio
significantly better
significantly worse
n.s.
Supplementary Figure 21: One-tailed t-test results on balanced accuracy for IBL brain region clas- sification with linear classifier across different label ratios. For all label ratios, NEMO significantly outperforms all other baselines. With only 50% of the labels, the linear classifier with NEMO either significantly outperforms or shows no significant difference compared to all baselines using 100% of the labels.
39

---

Published as a conference paper at ICLR 2025
One-tailed t-test linear classifier macro averaged F1
NEMO
SimCLR
0.01
VAE
NEMO
0.1
SimCLR
VAE
NEMO
SimCLR
Label ratio
VAE
NEMO
0.5
SimCLR
VAE
NEMO
0.8
SimCLR
VAE
NEMO
1.0
SimCLR
VAE
NEMO
SimCLR
VAE
NEMO
SimCLR
VAE
NEMO
SimCLR
VAE
NEMO
SimCLR
VAE
NEMO
SimCLR
VAE
NEMO
SimCLR
VAE
0.01
0.7
0.3
0.5
0.8
1.0
Label ratio
significantly better
significantly worse
n.s.
Supplementary Figure 22: One-tailed I-test results on macro averaged score for IBL brain region classification with linear classifier across different label ratios. For all label ratios, NEMO signifi- cantly outperforms all other baselines. With only 50% of the labels, the linear classifier with NEMO significantly outperforms all baselines using 100% of the labels.
40

---

Published as a conference paper at ICLR 2025
One-tailed t-test MLP balanced accuracy
NEMO
NEMO freeze
SMOLLR
0.01
Simol frem
WNE VAE freeze
NEMO
CLR
SimCLR freeze
WAE
WNE freeze
supervise
NEMO
NEMO
Label ratio
0.3
BiNCLR
WAE
VAE freeze supervice
NEMO
NFMO freeze
SimCLR
0.5
SimCUR
VAE freeze superiiee
NEMO
NEMO freeze
0.8
SirCI R
SimCLR freeze
WAE
suparvise
NEMO
NEMO frazee
1.0
SIECLR
SImCLR freeze
WAE
w OWENN
scent OwNN
Orise
OREaY
0.z0g 300
some OWEN
3bn 3vm
socueeti
OWER OWEN
ednay OWENN
OWEEN
Tokees
100
0.3
8'0
1.0
Label ratio
significantly better
significantly worse
Supplementary Figure 23: One-tailed t-test results on balanced accuracy for IBL brain region clas- sification with MLP classifier across different label ratios. For almost all label ratios (except 0.01), NEMO significantly outperforms all other baselines. With only 80% the labels, the MLP classi- fier with NEMO either significantly outperforms or shows no significant difference compared to all baselines using 100% the labels.
41

---

Published as a conference paper at ICLR 2025
One-tailes t-test MLP macro averaged F1
SinCLR
0.01
SimCLR frezz
VAE
VAE frezze
EIMCLR
0.1
SimCLR freeze
VAE freeze
supervise
NEMO
NEMO freze
Eim
0.3
Label ratio
VAE freeze suparvise
NEMO
NEMO frezz
Sim
0.5
SIMCLA
supervise
NEMO
NEMC freeze
0.8
SinCLR freeze
EImCLR
VAE
NEMU frece
1.0
EIrCLR
SimCLR freeze
VAE VAE
- Jvn
stpe ovEN
aoee JM
sowel DMSN
3t
OVEN
OPEN
Cousp
100
01
8'
1.0
Label ratio
significantly better
significantly worse
n.s.
Supplementary Figure 24: One-tailed t-test results on macro Fl score for IBL. brain region classi- fication with MLP classifier across different label ratios. For almost all label ratios (except 0.01), NEMO significantly outperforms all other baselines. With only 50% the labels, the MLP classi- fier with NEMO either significantly outperforms or shows no significant difference compared to all baselines using 100% the labels.
42


Reviews:
[{"review_id": "4N8YGieNHO", "reviewer": "Reviewer_iQCh", "summary": "This paper presents a novel application of contrastive learning to solve two important problems in systems neuroscience by incorporating just two electrophysiological recording modalities: spiking activity and extracellular action potentials (EAPs). The authors developed a new framework called Neuronal Embeddings via Multimodal Contrastive Learning (NEMO) by combining the well-established Contrastive Language-Image Pretraining (CLIP) framework with task-specific data augmentations and encoders. The authors demonstrate the multimodality as well as the power and utility of NEMO by evaluating its performance on two very different downstream tasks:\n\n1.\tcell-type classification among parvalbumin (PV), somatostatin (SST), and vasoactive intestinal peptide (VIP) inhibitory interneurons using opto-tagged Neuropixels Ultra (NP Ultra) recordings from the mouse visual cortex, and\n\n2.\tbrain-region classification among 10 broad areas using the public International Brain Laboratory (IBL) brain-wide map data set.\n\nThis paper’s novelty mainly stems from the utilization of a paired data set that combines an autocorrelogram (ACG) image of every neuron’s spiking activity and a waveform template of the neurons’ EAPs, and from the application of two separate encoders for the aforementioned two modalities. In both cell-type and brain-region classification tasks, the authors show that NEMO outperforms the state-of-the-art multimodal cell-type embedding methods including PhysMAP and VAE-based methods as well as a fully supervised method in terms of balanced accuracies and macro-averaged F1 scores with minimal fine-tuning.", "questions": "The waveform template is restricted to one channel with maximal amplitude and multi-channel template results are shown in Supplement E. What exactly is the definition of a channel in this context?\n\nWhy was additive Gaussian noise chosen as the sole data augmentation? Can a brief rationale for this specific choice be included in the paper? The authors demonstrate that adding two template augmentations: amplitude jitter and electrode dropout does not significantly improve the performance of the two downstream classification tasks. How do other data augmentation types impact the performance and results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a very well-written paper with clear organization of the figures and sound presentation of the data sets used, methods applied, and results obtained. The authors apply a successful contrastive learning method to develop a new framework that outperforms comparable state-of-the-art methods in both the cell-type classification task and the less widely-explored brain-region classification task. The task relies on two electrophysiological recording modalities: spiking activity and EAPs, which are more accessible, and the decent classification performances come with minimal fine-tuning. NEMO is able to differentiate between VIP and SST cells, which is highly valued in systems neuroscience, and its ability to yield data embeddings that lead to separable classification regions is impressive. The method described in this paper will be particularly helpful and useful to systems neuroscientists interested in applying this technique with the goal of decoding the neural circuitry underlying multiple biological functions.", "weaknesses": "Detailed graphical representations of the architectures used in the authors’ method may help the readers understand the details of NEMO better. A more comprehensive description, such as including explanations of 10D and 500D in the VAE baseline versions’ latent spaces of the encoder architectures, would have further aided clarity to the method’s explanations.\n\nThe authors restrict the number of example neurons, whose recorded activities are inputted to the overall architecture, to five neurons without an explanation of why the input neuron number was kept low. Providing a rationale for limiting the number of input neurons to five as well as an explanation of how the results of the authors' method change with varying input neuron number would be greatly appreciated.\n\nThe authors state that they fixed the hyperparameters for all methods across the two data sets used in the experiments due to the limited number of labels for evaluation for the cell-type classification task. It is unclear whether this choice led to fair performance comparisons among the state-of-the-art methods. It would help to know that separate hyperparameter optimization among the different methods and data sets would not yield different results. Providing a brief analysis on the sensitivity of the results and the overall performance to variations in specific hyperparameters will notably strengthen the claims made in this paper.\n\nMinor comments:\n\nThere seems to be a citation error or a missing preposition in Section 1 when the authors cite IBL et al.\n\nRadford et al. 2021 to Radford et al., 2021 in Section 4\n\nTable 6 is non-existent in Section 6.1. The authors may be referring to Table 1.\n\nThere seems to be a citation error or a missing preposition in Section 6.3 when the authors cite Chen et al. (2020).\n\nFigure 5 (b) caption's word \"then\" should be changed to \"than.\"\n\nIn Section 7, the phrase \"should also be useful these down-stream tasks\" should be changed to \"should also be useful in these down-stream tasks.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fuw1prNwZy", "reviewer": "Reviewer_vN2L", "summary": "The authors proposed a multimodal contrastive learning approach for joint modeling of extracellular action potential data and spiking activity data. The proposed approach can be fine-tuned for different downstream tasks, including in vivo cell-type and brain region classification. More specifically, the author applied the classic contrastive learning framework established in CLIP on extracellular action potential data and spiking activity data. Although the theoretical innovation is relatively limited, the authors made the earliest efforts (as far as I know) to apply contrastive learning for joint modeling of extracellular action potential data and spiking activity data.", "questions": "1.The authors emphasized “in vivo”, however, are there any results about the computational efficiency of the NEMO model that can support it? Does it support real-time processing?\n\n2.I would expect a validation of the data augmentation strategy. It is understandable that the construction of ACG images is computationally expensive. But the authors are encouraged to validate the data augmentation strategy adopted in the study, augmentations directly for the ACG images, is reasonable by showing a couple examples.\n\n3.The authors compared NEMO with the VAE-based method. However, in my opinion, it seems that a critical comparison is missing: the comparison between naive models, specifically, NEMO without fine-tuning and the VAE-based method without fine-tuning. This comparison would highlight the representational power of the two methods .", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "As mentioned above, this study is among the earliest efforts to apply contrastive learning for joint modeling of extracellular action potential data and spiking activity data. Most key components in the proposed analytical framework are fetched from previous work (e.g., CLIP contrastive learning and the ACG encoder), increasing the reproducibility of the study. In addition, the study was well presented and easy to follow.", "weaknesses": "As mentioned above, the theoretically contribution of the study is relatively limited. The readers may expect to see some components that are specifically designed with consideration for the unique characteristics of the data and the problem being  addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xYLIn8hAIm", "reviewer": "Reviewer_YH4G", "summary": "This paper introduces NEMO (Neuronal Embeddings via MultimOdal contrastive learning), a method for classifying neurons by their cell type and brain region location using electrophysiological data. NEMO uses CLIP-like contrastive learning to jointly analyze two types of neural data: the shape of neural electrical signals (waveforms) and patterns of neural activity over time (autocorrelograms). \n\nThe authors evaluated NEMO on two datasets: an opto-tagged mouse visual cortex dataset for cell-type classification and the International Brain Laboratory dataset for brain region classification. In comparative experiments, NEMO achieved higher classification accuracy than existing methods including PhysMAP and VAE-based approaches. The authors also demonstrated that using multiple units improved performance, and that the method maintained effectiveness even with limited labeled training data. The paper includes ablation studies examining the importance of joint versus independent learning of the two data modalities.", "questions": "- The VAE training is unclear to me. Do you jointly embed waveforms and autocorrelograms, or do you use two separate encoders/decoders with a shared latent space (which would require cross-decoding, i.e. encode waveform and decoder autocorrelogram)?\n- How important are the data augmentations? Can you provide an ablation experiment for that?\n- Waveforms and autocorrelograms are two reasonable choices for input modalities. However they are not the only conceivable choices. Have you thought/tried other choices or thought about learning an encoding of spiking activity directly?\n- What are the results when you cluster on the latent embeddings directly instead of running UMAP first? How stable is the clustering? I.e. if you train two models of NEMO from different seeds and then cluster neurons, how consistently are two neurons assigned to the same cluster (as measured by adjusted rand index or similar)?\n- Can you provide a definition for the modularity index?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written, the figures are well made, descriptions are generally clear.\n- The paper contains extensive additional material for more details.\n- The training and experimental setup seems to be carefully chosen and sound.\n- Limitations are discussed at the end.", "weaknesses": "I only found minor weaknesses. \n\n- Although, the authors did a great job providing necessary details, sometimes the training specifics for the control models and the clustering analyses were a bit too short in my opinion. It would be great if you could provide a bit more detail on this.\n- It would be great to see an ablation for the data augmentation to see how important that is (see questions).\n\n**Minor (do not expect you to respond to this)**\n\n- Figure 3 typo in legend “Supervise” instead of “Supervised”\n- Supplementary Figure 7 is blurred likely due to plotting settings\n- As multi-unit has a particular meaning in neuronscience, I find the wording “multi-unit brain region classification”. I think you mean “multi-neuron” here. Unless I misunderstood and you are really using multi-unit activity, I would change the wording.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IS0CyiSwny", "reviewer": "Reviewer_a6D4", "summary": "The authors used multimodal (spike waveform + firing pattern) contrastive learning to classify three types of inhibitory neurons (PV/SST/VIP) and ten different brain regions. The proposed NEMO model is based on the widely used CLIP (Contrastive Language-Image Pre-Training) model. NEMO outperforms two previous multimodal models: PhysMAP and VAE.", "questions": "1. In Figure 1a, the text \"Neuropixels 1.0\" should be replaced with \"Neuropixels Ultra\" since your VIP/SST/PV data comes from NP Ultra, not NP 1.0, which is used in the IBL dataset for classifying brain regions. Also, please update the inset to show the schematic of NP Ultra instead of NP 1.0.\n\n2. Figure 3b is not mentioned in the text. It could be referenced between L350 and L361.\n\n3. Figure 3c is also not mentioned in the text. It might fit well between L362 and L366.\n\n4. Consider changing the title of Figure 3b and 3c from \"unit\" to \"neuron\" to be consistent with the terminology used in the main text.\n\n5. In the PhysMAP paper that you benchmarked, they applied three public datasets from S1, A1, and V1/Hippocampus. Have you tested these datasets? While I am not requesting additional experiments, if you have tested them, it would be helpful to include the results in the supplementary materials.\n\n6. Please specify how you computed the two primary metrics: macro-averaged F1 score and balanced accuracy. Does the first metric equal the unweighted mean of all the per-cell-type F1 scores? Does the second metric equal the unweighted mean of sensitivity (True Positive) and specificity (True Negative)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The writing is clear, easy to understand, and follows a smooth logical flow, with almost no typos and well-presented figures.\n\n2. The paper provides a thorough review of relevant literature in this field. I have closely followed the cell-type classification area, and all the papers I am aware of (and some I was not) have been accurately cited, except for one (see Weakness). Notably, the authors benchmarked two very recent models that are still in preprint format.\n\n3. This is the second work to use contrastive learning for cell-type classification and the first to combine two modalities.\n\n4. The multimodal approach outperforms single-modal models, which is consistent with previous studies.", "weaknesses": "1. In the best-case scenario (cell type classification, L344, Figure 2c), NEMO outperforms VAE by 11%. However, in brain region classification, the improvement is minimal. For example, in Figure 3e, comparing the deep orange (NEMO) to the deep blue (VAE) bars, the difference in balanced accuracy is less than 0.05. Are these differences statistically significant?\n\n2. Joint training shows little to no benefit over independent training. For example, in Figure 5b, comparing the deep orange (NEMO) to the deep violet (SimCLR) bars, the difference in balanced accuracy is around 0.01. Additionally, in Supplementary Table 9, independent NEMO performs either better (0.84 vs. 0.83) or similarly (0.83 vs. 0.84, 0.87 vs. 0.88) to joint NEMO. Are these differences statistically significant?\n\n3. To my knowledge, there is no neuroscience evidence suggesting a strong pairwise correlation between spike waveform and firing patterns. For example, layer 5 pyramidal neurons and cortical PV neurons both fire a large number of spikes (both spontaneously and evoked), but their spike waveforms are broad and narrow, respectively (Cortical connectivity and sensory coding, KD Harris, TD Mrsic-Flogel - Nature, 2013). Additionally, burst firing can be evoked in both excitatory (Chattering cells: superficial pyramidal neurons contributing to the generation of synchronous oscillations in the visual cortex. CM Gray, DA McCormick - Science, 1996) and SST neurons (Somatostatin-expressing neurons in cortical networks, Urban-Ciecko, AL Barth - Nature Reviews Neuroscience, 2016). This is fundamentally different from the relationship between an image and its description in CLIP. In other words, the word \"puppy\" and an image of a \"puppy\" represent the same concept, but a broad spike could be associated with either burst or dense firing, depending on whether the neuron is located in layer 2/3 or layer 5.\n\n4. This is the second paper to use contrastive learning for cell-type classification. The first is CEED (Vishnubhotla et al., 2023), which used an unsupervised model (SimCLR) to classify cell types and benchmarked against WaveMap (the single-modality version of PhysMAP). While the CEED work does not significantly compromise the novelty of this study, it should be more clearly acknowledged. The current citation, \"Contrastive learning has been applied to raw electrophysiological recordings (Vishnubhotla et al., 2024),\" is inappropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)