Neuroformer: Multimodal and Multitask Generative Pretraining for Brain Data
Authors: Antonis Antoniades
Yiyi Yu
Joe S Canzano
William Yang Wang
Spencer Smith
Venue: ICLR 2024 poster (2024)
Keywords: GPT
Pretraining
Transformers
Multimodal
Multitask
Neuron
Neuroscience
Neuronal
Spikes
Brain
Cortex
Human
Mice
Biology
Primary area: applications to neuroscience & cognitive science
TL;DR: Introduce a generative framework to train large transformer models on multimodal and multitask system neuroscience datasets.

Abstract: State-of-the-art systems neuroscience experiments yield large-scale multimodal data, and these data sets require new tools for analysis. Inspired by the success of large pretrained models in vision and language domains, we reframe the analysis of large-scale, cellular-resolution neuronal spiking data into an auto-regressive spatiotemporal generation problem. Neuroformer is a multimodal, multitask generative pre-trained transformer (GPT) model that is specifically designed to handle the intricacies of data in systems neuroscience. It scales linearly with feature size, can process an arbitrary number of modalities, and is adaptable to downstream tasks, such as predicting behavior. We first trained Neuroformer on simulated datasets, and found that it both accurately predicted simulated neuronal circuit activity, and also intrinsically inferred the underlying neural circuit connectivity, including direction. When pretrained to decode neural responses, the model predicted the behavior of a mouse with only few-shot fine-tuning, suggesting that the model begins learning how to do so directly from the neural representations themselves, without any explicit supervision. We used an ablation study to show that joint training on neuronal responses and behavior boosted performance, highlighting the model's ability to associate behavioral and neural representations in an unsupervised manner. These findings show that Neuroformer can analyze neural datasets and their emergent properties, informing the development of models and hypotheses associated with the brain.

Published as a conference paper at ICLR 2024
NEUROFORMER: MULTIMODAL AND MULTITASK GENERATIVE PRETRAINING FOR BRAIN DATA
Antonis Antoniades; Yiyi Yu, Joseph Canzano, William Wang, Spencer LaVere Smith University of California, Santa Barbara
arxivssubmiit/5444555 [q biN 15 Mar 2024
ABSTRACT
State-of-the-art systems neuroscience experiments yield large-scale multimodal data, and these data sets require new tools for analysis. Inspired by the success of large pretrained models in vision and language domains, reframe the analy- sis of large-scale, celoularresoluutin neuronal spiking data into an autoregressive spatiotemporal generation problem. Neuroformer is multimodal, multitask gen- erative pretrained transformer (GPT) model that is specifically designed to handle the intricacies of data in systems neuroscience. It scales linearly with feature size, can process an arbitrary number of modalities, and is adaptable to downstream tasks, such as predicting behavior. We first trained Neuroformer on simulated datasets, and found that it both accurately predicted simulated neuronal circuit activity, and also intrinsically inferred the underlying neural circuit connectivity, including direction. When pretrained to decode neural responses, the model pre- dicted the behavior of a mouse with only few-shot fine-tuning, suggesting that the model begins learning how to do directly from the neural representations themselves, without any explicit supervision. We used an ablation study to show that joint training neuronal responses and behavior boosted performance, high- lighting the model's ability to associate behavioral and neural representations unsupervised manner. These findings show that Neuroformer can analyze neu- ral datasets and their emergent properties, informing the development of models and hypotheses associated with the brain.
INTRODUCTION
Systems neuroscience experiments are growing in complexity with a cascade of technical advances. In addition to recording neuronal activity from hundreds to thousands of neurons in parallel from multiple brain areas (Yu et al. 2021 Stirman et al. 2016 de Vries et al. 2020a b), experimenters are also sinultaneously acquiring hehavioral data including eye positions and body movements (Stringer et al. 2019; Steinmetz al. 2019). Neuroscientists seek models for these rich datasets to obtain insights into how neural circuitry is involved with behavior. However, complex datasets and experimental designs present challenges for model development, particularly in incorporating the various modalities, and subsequently creating models for a host of different applications.
In recent years, deep neural networks (DNNs) have demonstrated their potential in modeling neural activity and circuitry (Yamins et al. 2014 Kubilius et al. 2018, Richards al. 2019). Brain-inspired DNNs have exhibited responses similar to visually evoked neural responses in visual cortical areas (Yamins al., 2014; Kindel et al. 2019, Mineault al. 202T] Yamins & DiCarlo 2016). These models were trained in goal-driven approach, such as the classification of images. Then intermediate representations in the model were compared observed neuronal activities in biological brains. In other work, autoencoder-based latent variable models have been applied to the analysis of neuronal activities, with the assumption that the single-trial spiking activity of behavior task depends on underlying dynamics (Sussillo et al. 2016 Ye & Pandarinath, 202T Zhao & Park 2017 Schneider et al.| 2022). These approaches can provide insight into principles of neural circuitry, but often entail inductive biases that are not always idcal. General purpose,
multimodal models can provide a complementary approach.
"Contact: antonis ucsh.edu Code: https:://gttubb.ccm/// wwt.ittoonnde///eeee
ades http:///aaaatttaaaaae////eoorrrr

---

Published as a conference paper at ICLR 2024
To develop models for correlating neuron activities, sensory input, and behavior, we sought powerful multimodal architecture and training objective. The transformer architecture has demonstrated its flexibility in modeling data from various domains, finding widespread adoption in applications including vision, language, and sound (Liu al., [2021a] [Brown et al.. 2020a] Huang it al. 2018 Radford et al., [021 Gan et al. 2020 Li ct al. (021). In particular, transformer-basee
language-vision models exhibit high performance for learning multimodal representations for text
and images, enabling text-to-image generation (Ramesh et al. 2021 Radford et al., (021). Key to this is that a) the transformer is scalable; i. it does not suffer from the vanishing gradient problem (Pascanu et al. 2012) b) it makes few assumptions about the input data (minimal inductive bias) @Jaelee et al. 202 lb a and c) can contextualize information within long-duration multimodal con- texts. Here we present a transformer-basee model for multimodal datasets in systems neuroscience, including neuronal activity, stimuli, and behavior. We developed a generative model which firstly aligns and subsequently fuses neuronal activities, input signals (c.g., visual stimuli) and behavior.
Our key contributions are as follows: a) Introduce an approach that reframes the population re- suusally-masked spatiotemporal autoregressive generation problem.
sponse inference problem b) Show how to employ a generative pretraining paradigm for neural data. e) Investigate the po- tential of multimodal, multitask, generative transformer models to analyze and parse neuroscience data, and obtain insights.
RELATED WORK
Representational and Functional Similarities between DNNs and Mammalian Brains There
similarities between the hierarchical representations of convolutional DNNs and the visual and
inferior-temporal cortices (Yamins et al. 2014; Kubilius et al. 2018 Kindel et al. 2019 Mineault et al.] 2 Yamins & DiCarlo 2016). More recently, parallel-patte networks trained using contrastive-reedictive predictive objective were shown to separate into pathways that mirrored the functional specializations of the dorsal and ventral streams in mouse visual cortex (Bakhtiari al. 2021). Sim- ilarly, there are parallels between human speech and language models (Caucheteux & King 2022] Millet al. 2022). Lastly, reports have shown the plausibility of transtormers replicating specific neural functions and circuits (Whittington et al., [021] Bricken & Pehlevan (022). In our work, establishing a similarity between DNN and the brain is not our focus. Instead, we are focused on developing an analytical and functional tool. This tool, Neuroformer, can exhibit similarities in representational and functional characteristics with biological neural circuitry, and we explore that here.
contrastive-
Learning Latent Dynamics of Population Activity Variational auto-encoders have been used to infer latent dynamics embedded in complex, high-dimensional neural data (Sussillo et al., 2016 Zhou & Wei, 2020). Recently, transformer architecture has been used in place of RNNs for the same application (Ye & Pandarinath, 2021), and another project has trained a latent model using a contrastive objective (Schneider et al., 2022). These empirical findings show that this approach avoids over-fitting and outperforms previous methods. In our work, we integrate these paradigms, by employing a contrastive objective at the front end of a generative model.
Large Multimodal and Multitask Models The transformer has proven effective at processing a
variety of modalities including vision, languange, and sound (Liu et al. [021 Brown et al.) 2020aa
Huang et al. 2018)). Subsequently, the field introduced multimodal models. For example, models for interactions between images and text (Tan & Bansal [2019] Chen et al. 2019 Lu et al. 2019). Some models attempt to align the modalities with separate feature encoders using contrastive learning approaches (Radford al., 202T Gan et al. 2020), and alignment and fusion (Li et al., (021). We experimentally validate these methods on neural data, by using our own generative and contrastive learning objective. Furthermore, architecture innovations have resulted in models that are agnostic to the input modalities and can be applied to a large variety of tasks (Jaegle et al.) bb a). Lastly, it has been recently observed, that large models pretrained on massive datasets of text exhibit emergent properties. This has been observed in Large Language Models (LLMs) as in (Brown al. 2020bb (Wi et al. 2023 Jung et al. 2022 Creswell et al. 2022) Ganguli et al. 2023) but also in scientific Bakhtiari al. 2021 We take inspiration from these lines of work to
domains (Rives al. 2021
2

---

Published as a conference paper at ICLR 2024
build framework for modelling neural data that efficiently scale large imputs and an arbitrary number modalities.
3 MODEL
Token
Current State
Neural History
Visual
Arbitrary Modality
Masked Token
Enc.
Multi-modal feature fusion
AR Decoding
ID Proj.
Feature Encoding
Seif Attertton
Self Actembion
Dross Atherttre
Selft Actinttonn
Cross Atterition
Self Artertion
Contractive Matching inggrret
di Proj.
Sy
Figure I: Neuroformer architecture. Inputs undergo contrastive matching to ensure efficient repre- sentations for downstream processing. Neural activity, stimuli, and other modalities are added to the Current State using cascading self- and cross-attention modules. Finally, the output of spike predictions (neuron and time interval tokens) are read out. During inference time, neuron IDs autoregressively populate the block, and the corresponding predicted time tokens act as additive temporal encodings.
Workflow As input, our model takes action potential data, i.e., spike trains, from multiple neurons recorded or imaged simultaneously, and breaks them up into the Current State I and past state 7 according specified temporal context windows (We: wp) (Fig. [ blue and purple, respectively). We do this to discretize them and precisely align the neural representations with features from other modalities. Model inputs, including neural, visual, and other features, are optionally processed by feature encoders. The modalities are first aligned via a multimodal contrastive module (Radford jet al., 2021 Chen et al., 2020 Oord al., 2018) which aims to learn matching representations the neural features and the features from other modalities. Subsequently, cross-modal transformer fuses the current neural response Ie with all other features. We then decode our fused representation using causally-maske transformer decoder. All training is sulf-supervised and requires data labelling.
Feature Backbones We assign an ID (number) each neuron in our dataset and model token. At each step, learnable look-up table projects each neuron spike contained in our Current and Past States onto an embedding space E, resulting in vectors (T., E) and (Tp: E), where T,, Tp are the corresponding state's sequence length (number of spikes + padding). For video frames, we optionally use a single layer 3D Convolutions (Hara et al.) (017)) before splitting them into patches of size T', H, W. E' ), is standard for visual transformers (Fig. [ Feature Encoding, green) (Dosovitskiy et al. 2020 Liu al. 2021b et 2019). We then feed these patches into
the cross cross- layers.
Multimodal Contrastive Representation Learning
Drawing from recent developments in vision
and language representation learning (Radford al. 2021 Zellers al. 2021 et al. 2021). our approach leverages contrastive learning objective before fusing our feature sets (Fig. 1 Con- trastive Matching). Our architecture allows alignment of an arbitrary number modalities; how- ever, here we focus the three-modal case. We use Visual features Fp. (B, T, H, W, E), Current State neural embeddings I. (B, Te. E), and Behavior features A ~ (B. E) as our modality representatives, where B Batch Size and T sequence length. These projected to a latent dimension d using three linear layers g 9. g.. We calculate pairwise similarities be- tween the modalities, yielding three matrices of dimensions (B,, Bi), (B,, Ba), (Ba, By) where
3

---

Published as a conference paper at ICLR 2024
B B, B, = Ba B. The diagonal entries correspond to ground-truth pair similarities. These scores are then softmax- normax-ed and used to compute the contrastive loss via cross-entropy H. where the probability should be 0 and I for the negative and positive pairs y, respectively. We generalize the computations with pair notation (m,n) where m and n are any of the modalities (either or A):
exp(s(m, na)/T)
smm, n) = 9m (mp.c) 9m (nc)
Pk mn
(I)
Σ exp(s(m, nx)/T)
We then apply these equations to each pair (F.I), (L.A), (A,F), and compute the contrastive loss:
Lynaa (F) pl) (1'" (1), + (A),
(A) (A))]
(2)
By explicitly imposing alignment, we encourage the model to uncover the representational common- alities between biologically relevant features (Schneider al. 2022). This is particularly helpful in low-data settings, mitigating the over-fitting that can occur with variable neural signals. We observe consistent performance improvement upon adding the contrastive objective (see Fig. 6 10
Feature Fusion Following alignment, the masked Current State, Ie is fused with all other features, in our case the visual F and neural Pust State features, Ip by cascading cross-attention modules. This stage of the model is similar to the (Jaegle et al., 202 lb a). where cross-attention between smaller latent array (Te, E), in our case the Current State and larger byte arrays (Ty.$E), in our case the Past State neural features, is iteratively carried out. This architecture decision has two main advantages: a) It introduces a bottleneck imposed by the smaller Le array, of length size T. that upon operating with larger feature array of length T,, reduces the complexity of attention from ((T}) to O(T,Ty): ). When To much less than T'y (expressed mathematically limm.-0077 -, Te/T, 0), the complexity scales linearly with Ty. This enables the use of finer-grained features, like large- resolution visual features or in our case raw video frames, which are essential for modeling lower- level visual areas. b) The iterative nature of the algorithm enables us to fuse an arbitrary number of modalities with our Current State.
Causal Spike Modelling After it has been fused with all other modalities, the resulting Current State is decoded using a causally masked transformer (Brown et al. 2020a). At each prediction step, the model predicts both which neuron will fire, and within which sub-interval (dt(n) using two linear projections from the last self-attention layer (Fig. [ AR Decoding) mapping to the corresponding (Pi, Pat) label distributions. Notice that two Spikes occur the same time, and therefore by predicting bins within which neurons fire, we can sequentially predict spikes that occur simultaneously. We optimize for this objective, by minimizing the cross entropy between the predicted and ground truth distributions:
wa),
Lee(1) = Egaya H (y),PI)
Lee(dt) = Esdty H (yet Pat)
Loss Function
The model is jointly optimized end-to-end using the contrastive objective Lumes and cross-entropy loss for both L. and dt (Lee(1) Lee(ar)). The resulting loss is weighted average of the three losses:
= Lunc + (H) Loe()) + (1 7 - 11) Loo(d))
(3)
Inference Generating Simulations At inference time, the model autoregressively predicts the distributions of the nath step. (I,(n),dt,nn))) outputting an end-of-state [EOS] token to indicate it has completed the Current Target State W., in similar fashion to work in image-text generation (Ramesh et al. 2021), (Fig. AR Decoding). To sample from the respective distributions (pr, Pas) we use nucleus sampling (Holtzman et al.) 2019). The predicted Current Target State neuron IDs are then incorporated into the Past State according the Past State Context Window Wo together with the predicted sub-intervals dt,,, which act as additive temporal embeddings (Fig, [ dotted arrow from AR Decoding).
4

---

Published as a conference paper at ICLR 2024
Finetuning- Transferring to New Tasks A primary motivation for redefining the spike inference problem in a manner compatible with contemporary pretraining paradigms is the ability to utilize a broad spectrum of techniques developed within this framework. To transfer to task, whether it be hehavioral prediction or any other downstream task, we adopt a straightforward heuristic. First, if necessary, the data distribution is discretized into n classes. Then, mean-pooles representation from our final latent layer is projected onto the appropriate feature space P. E n. The loss is calculated using the standard multi-class cross-entropy for classification (Howard & Ruder 2018 Yosinski et al., 2014 Hinton et al. 2015) of mean- -auared error for regression.
Training We trained all Neuroformer models with 8 layers and heads for the decoder and each of the cross-attention modules, with embedding dimension of 256 and 80/20 train/test split. VI+AL and Visnav models equated to a total of 40, 000, 000 and 100, 000, 000 parameters respec- tively. We emprically observed an increase in performance up until the specified parameter sizes. We used AdamW optimizer (Loshchiloy & Hutter 2019) in Py PyTorch. Learning rate was 2e-4, with linear warmup and cosyne decay schedules. Batch size was kept at 224 for pretraining and 32 for fine-tuning.
4 RESULTS
To evaluate performance of Neuroformer, used both artificial data sets, with known ground truth, and real neural data sets, with authentic variability and real-world relevance. We analyzed the per- formance in terms of inference and features.
4.1 DATA
Simulated Dataset We simulated spiking neural network characterized by directional connec- tivity with three hub neurons using Brian2 simulator (Stimberg et al. 2019) (Hub-net simulation). The network has N = 300 leaky integrate-and-fir neurons with stochastic current. The membrane potential (V₂) of each neuron was simulated as a stochastic differential equation:
dV. dt
V/ 7
2
(4)
The membrane time constant was set to r = 10ms as a Gaussian random variable with zero mean and standard deviation. We scaled the random variable by a 0.5. The total dataset equated million tokens.
Two-photon calcium imaging datasets We assessed the performance of Neuroformer using real neural datasets. These consisted of neuronal activity recorded from awake mice, which were either passively viewing visual stimuli (Passive-Stim data), or actively engaged in risually-guide navi- gation task (Visnav data). The activity was recorded using two-photon calcium imaging, (Dombeck) et al. 2007; Yu et al. 2021 2022 and neuronal spikes were deconvolved from calcium traces via the suite2p calcium imaging processing pipeline, (Stimberg et al., 2019) complemented by a Bayesian spike inference method (Pnevmatikakis et al., 2013). The Passive-Stim data comprised recordings from 386 neurons in the primary visual cortex and a higher visual area (VI AL.). These neurons responded reliably to drifting grating stimuli and naturalistic movie. Two Visnav datasets were derived from recordings in the lateral visual cortex (2022 neurons) or L2/3 medial (1905 neurons), spanning VI and multiple higher visual areas. During these recordings, the mice were engaged in a visually- guideed navigation task within a virtual environment, with their movements tracked by a ball system (Harvey et al. (2009) that controlled the first-person view within the virtual environment. The Visnav neurons conveyed information about both the visual input and the speed of the animal's movements. The resulting dataset sizes equated to 80 x 10 800 x 10 and 10 tokens.
4.2 ATTENTION REVEALS THE FUNCTIONAL CONNECTIVITY OF SIMULATED NEURONAL CIRCUITS
To test our hypothesis that attention can provide insights into neural circuitry, we first trained the Neuroformer simulated dataset with known ground-truth connectivity. This dataset is unimodal,
5

---

Published as a conference paper at ICLR 2024
b
Ground truth
Ground fruth network
Syneptic weight
Red neurons are hub neurons
0.8
Torgel neurrnn ID
100
0.8
Spike data from ground truth network Predictions from Neuroformer
300
100
200
300
-0.2
Source neuron ID
12
d
Pearson correlation of neuronal activity
e
Inferred from Neuroformer
Neuron
de 15 toen mo
15 & 1
using average attention
0.05
Jum 11. a
2.8 V/ 8A
0.04
2100 m 11.5..5
Target nouron
100
0.5
Target nouron
188.
0.03
AM
1:P4
115,
0.02
Count
40g
15
0.01
20
-0.4
0.4
0.8
300
Source neuron ID
100
200
300
300
Source neuron ID
100
200
300
0
Test prediction correlation
Figure 2: Validating Neuroformer with ground truth data. (a) A model spiking neural network was generated to provide ground truth for validation. To provide salient network feature, three "hub" neurons are strongly connected to many other neurons. (b) The ground truth connectivity matrix provides reference for the network. (c) Raster plots (top) for neurons show that Neuroformer provides spike predictions (red) that closely match the spiking of the ground truth network (black). Correlations between the predicted and ground truth spike trains were generally positive, and 20.1 (bottom). (d) A simple Pearson correlation analysis reveals three subnetworks, but not the hub neurons, because of the lack directionality or causality. (e) Attention mechanisms Neuroformer infer causality and thus reveal the hub neurons.
so we simply omitted the visual cross-attention module, essentially reducing a vanilla GPT archi- tecture. After training, we ran masked inference on the data, aggregating the attention values that are assigned each neuron, and counting how many times neurons attend each other during this pro- cess. This results to two square matrices Na. N2 of size (N, N), where N is equal to our neuronal population's size (300). By dividing (N,) by (Nz) we compute neurons assign to one another. Our results recovered the hub connectivity structure of the ground truth neuronal circuit ( I E 50, 150, 200 ) using only the spiking activity (Fig. 2}. This attention analysis identified the neurons that were driving circuit activity. Compared to simple correlation matrix, the attention analysis matrix provides directional, potentially causal, information.
average attention ano. (i. i) that
4.3 REALISTIC AUTOREGRESSIN SIMULATION OF MULTIMODAL NEURONAL DATASET
To validate Neuroformer a viable analysis tool for neuronal population datasets we tested its ability to model real neuronal data. We trained Neuroformer to predict visually evoked neuronal activity in population of 386 neurons, recorded using two-photon calcium imaging in mouse. The neurons were recorded simultaneously, and spanned two visual cortical areas: primary visual cortex (VI) and the anterolateral (AL) higher visual area. After training, Neuroformer is able toregressively generate whole-trit trial simulations (96 seconds) of the neuronal population that elosely resemble held-out trials (Fig. 3)
4.3.1 NEUROFORMER OUTPERFORMS GLM
The Generalized Linear Model (GLM) (Truccolo et al. 2005 Pillow et al. 2008b) is a versatile and widely used statistical modeling framework that generalizes linear regression accommodate non-normal response variables and non-linee relationships. It has been widely used by neurosci- entists performant tool for stimulus-response population decoding. Although useful, it has

---

Published as a conference paper at ICLR 2024
Multi-region calcium imaging
Nauron 1D379
Neuron tD 104
0.52
PM
TO
Nouron ID 43
Neuron ID
mm
conr. 0.76
con: 0.59
Simmau
Troiprad (eazrooree))
0.0
Gratings
Gratings
Naturalisic
36
Neuron Layer
Layer 3,
Layer
8.
Du2 234 4.2
0/8
Figure 3: Demonstrating utility of Neuroformer with real data. (a) We measured neural activity in two visual cortical areas, V1 and AI,, mice using large field-of-view two-photon calcium imaging- After screening for reliable neuronal responses stimuli, 386 neurons were included for analysis. (b) Neuroformer modeled the spiking data accurately, for responses to two classes of visual stimuli: gratings and naturalistic videos (black ground-truth, red generated). (c) Neuroformer vs. GLM population response prediction comparison. Our model's predictions where more correlated with the ground-truth (t-test, p 0.0196) (d) Attention provided data-driven maps of the parts of stimuli that had statistical dependencies with neuronal responses.
some significant disadvantages, in that it requires heavy inductive bias (linear kernel, non-linearity, Poisson spiking). This limits both its generality and flexibility in simultaneously integrating many modalities. We compared the predicted, trial-averaged responses from a GLM to Neuroformer on the VI+AL dataset and found that our model's predictions where more closely correlated with the ground- truth (Fig, (3)) Our results revealed that our model could better capture the variability across the diverse set of stimuli in the dataset. We empirically observed that a single GLM could perform well on either the diffraction gratings naturalistic movies, but not both at the same time.
4.3.2 BIOLOGICALLY RELEVANT FEATURES EMERGE FROM STIMULUS- STTMULLS-RESPONSE CROSS-ATTENTION
Again, we turned to the attention parameters in the Neuroformer, as we had for the hub neuron anal- ysis. We plotted the spatial locations on visual stimulus frames where cross-attention was allocated, ie., sort of attention map. These maps exhibited localized features and structure. These maps are related to the concept of receptive fields, but are specifie to the full context of ongoing visual stimuli and neuronal activity. Thus, they can reveal novel relationships and insights into potentially causal relationships among neuronal activity and stimuli (Fig. 3).
4.3.3 HIGHLY ACCURATE. AND FEW-SHOT PREDICTION OF MOUSE BEHAVIOR
In this section, we introduce the notion of finetuning (Howard & Ruder 2018 for neuronal data. By reframing the neuronal modelling problem be compatible with modern NLP approaches, we can leverage similar techniques to solve tasks that can benefit from neuronal pretraining. One such task predicting the behavior of mouse from neuronal responses. This is a common thing to do in neuroscience analysis, where researchers want to investigate the relationship between neuronal activity and behavior. Furthermore, such paradigms could be useful brain-computer interface ap- plications. For this analysis, we used neural and behavior data from mouse running in virtual reality setup. Neuroformer can support both classification with a mutli-class cross-entropy objec- tive regression using mean-squared error, which be chosen according to the nature of the task. Here we report results using regression (Fig. 4 Table [] and have included classification and regression inference results for all models in the appendix (Figs 1T 12). Similarly to how multi-

---

Published as a conference paper at ICLR 2024
task models in the field of vision and language work, finetuning model for task involves projecting the features of the last layer onto the new feature space. (Howard & Ruder 2018
Lasso Reg
GIM
Uidirrectionll GRU
Neuroformer
Wilhy
Figure 4: Raw Speed vs. Predicted for methods for medial (top) and lateral (bottom).
Despite the inherent stochasticity in the speed of the mouse, the predictions of our model when trained on the full dataset are highly cor- related with the held-out ground-truth. Ad- ditionally, Neuroformer significantly outper- formed conventional approaches, achieving pearson r 0.97, with the held-out ground truth for the respective lateral and medial datasets (Table 1). To determine the effec- tiveness of the masked neuronal prediction pre- MLP
Table 1: Behavior Prediction
Pearson Corr (r)
Model
Lateral
Medial
Lasso Regression
0.62
0.73
GLM
0.69
0.81
0.83
0.85
Bidirectional GRU
0.83
0.88
training we compared the performance of pre- trained vs. randomly initialized models on 10% and 1% of the finetuming data. The pretrained few-shot models quickly approached the performance of the model trained on the full dataset (Fig. 5b,c). Crucially, we observed that the pretrained model that was finetuned on 1 % of the data out- performed the randomly-initialized model that was finetuned on 10% of the data. This suggests that pretraining enables the model to begin learning representations that go beyond the optimization objective as has been observed in generative models within natural language generation. (Radford
Neuroformer
0.97
0.97
eet al. 2019 Sanh et al. 2022
1.5
r- 0.97
No Preinivng 1% Ko Pretraiving 10% Prsture
1.0
Pasturn,
1.0
p
Ground-Truth Core atton Conclation
Finstans, 10%
Finetune 10% Training
Behantor Loos tholdoutt
0.9
Preizalring..
0.5
Predicted
0.6
0.8
0.0
0.4
-05
0.7
-100
0.2
0.8
-1.
-1.5
-1.0
-0.5
1.0
1.5
60
Epocc
Figure 5: (a) Velocity prediction Ground Truth. Neuroformer's predictions highly corre- lated (pearson r full = 0.97) with the actual speed of the mouse. (b) Pretrained neuronal models exhibit few-shot learning of mouse speed. The pretrained model finetuned using 1% of speed data was able to significantly outperform non-pretrained equivalent trained on 10% of the data. (pearson ft%.ft 0.51 vs. 110%. 0.33 respectively) (c) Behavior loss. The pretrained models (green) were able to converge faster and to a lower testing loss compared to the non aneerrrradded equivalents.
5 ABLATIONS
To explore the impact of each component of Neuroformer, we implemented a range of model ver- sions. Starting from just decoding from the Current State, progressively incorporated Past State,
8

---

Published as a conference paper at ICLR 2024
Video, Behavioral modalities, and Alignment using contrastive learning. With each addition, noted an enhancement in the model's ability to generate realistic neuronal responses (Fig, 6 10
Naturalistic, V1 AL
Visnav, Medial
Visnav, Lateral
0.45
0.32
0.34
0.30
0.32
0.28
t
0.26
t
0.24
0.22
0.25
0.20
State
Paur
Video
State
State
Video
Bahawior
Contrastive
State
Stat
Video
Banavior
Contrastive
Figure 6: Model component effect on performance. Each of the components (Current State, Past State, Video, Behavior, and Contrastive) improve the predictive capability of the model across datasets and brain regions. Points color-coded according to model architecture (see Fig. 1)
Of particular interest, was the Lateral dataset where the behavioral modality incorporated for the ablation study was eye position (see Fig. [3 for predictions). We hypothesized that by learning extrapolate from the eye position the visual field of the mouse, the model could improve its performance, which is what we observed. While more investigation needs to go into understanding exactly how Neuroformer incorporated eye position within this context, this result highlights the potential of our approach.
By observing the fluctuation in performance as components within the model are selectively in- corporated, scientists can examine hypotheses about the relevance of various input modalities or phenomenological behaviors to different brain regions. Ultimately, the development of modular and performant foundation models of the brain will create new avenues for the scientific exploration of the principles that underpin neuronal function and intelligence.
CONCLUSION
We have developed have demonstrated its utility accross four novel neural datasets with diverse structure. Our initial ob- servations showed that pretraining not only adopted representations that mirrored the causal patterns found in neuronal circuitry, but was also able to effectively enhance performance in downstream applications, enabling rapid few-shot adaptation behavior-related tasks.
method to enable generative pretraining of multimodal neural data, and
Our model outperformed conventional approaches, while in comparison incorporating minimal in- ductive biases and retaining an identical architecture across datasets and experiments. Overall, the model was applied to 4 modalities (neural data, video, speed, eye position), and 3 decoding tasks (neuron ID, neuron dt, speed). The architecture enables easy expansion of the model to more modal- ities and tasks, making it compatible with almost any type of systems neuroscience dataset and decoding objective.
Concluding, the Neuroformer enables systems neuroscience to leverage the full suite of tools in generative, multitask and multi-modal transformers. It paves the way for scaling training of neuro- science models and potentially connecting them with Large Language Models. This advancement could foster models with new emergent capabilities, benefiting not just neural data analysis but also various other applications. Our work sets the stage for promising research within this domain. !
For thorough discussion of limitations, please see Appendix A.

---

Published as a conference paper at ICLR 2024
REFERENCES
Shahab Bakhtiari, Patrick Mineault, Tim Lillicrap, Christopher C. Pack, and Blake A. Richards. The functional specialization of visual cortex emerges from training parallel pathways with self- supervised predictive learning. bioRxiv, 2021. doi: 10.1 202.222000061111. 448989. httptsss:
/ww..biisss www.wbior.iv.oommmmwwwwwwwwwww.. wwt. /ww.bbboor.....rrr//ottttt///e/e /ttpa//www......ooooo///////////
/2021 06 448989
Trenton Bricken and Cengiz Pehlevan. Attention approximates sparse distributed memory, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlísh, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, (bs/2005.14166 2020a. URL https: /ttes///rxxaa /tw.iiiiivv.////a///aaa///aaaava org////////o.ooo abs/ 22055.005.00 14165
Tom B. Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b.
Charlotte Caucheteux and Tean-Rémi King. Brains and algorithms partially converge in natural lan- guage processing. Communications Biology, 5(1): Feb 2022. ISSN 3399-3642. doi: 10. s42003-022- -2-00363---111
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL pettps://ariiv.oo abs//
2002. 05709)
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: learning universal image-text representations. CoRR, abs//199.1111 2019. URL
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large lan- guage models for interpretable logical reasoning, 2022.
Saskia E J de Vries, Jerome A Lecoq. Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, Kate Roll, Marina Garrett, Tom Keenan, Leonard Kuan, Stefan Mihalas, Shawn Olsen, Carol Thompson, Wayne Wakeman, Jack Waters, Derric Williams, Chris Barber, Nathan Berbesque, Brandon Blan- chard, Nicholas Bowles. Shiella D Caldejon, Linzy Casal, Andrew Cho, Sissy Cross, Chinh Dang. Tim Dolbeare, Melise Edwards, John Galbraith, Nathalic Gaudreault, Terri L Gilbert, Fiona Grif- fin, Perry Hargrave, Robert Howard, Lawrence Huang. Sean Jewell, Nika Keller, Ulf Knoblich, Josh D Larkin, Rachael Larsen, Chris Lau, Eric Lee, Felix Lee, Arielle Leon, Lu Li, Fuhui Long, Jennifer Luviano, Kyla Mace, Thuyanh Nguyen, Jed Perkins, Miranda Robertson, Sam Seid, Eric Shea-Brown, Jianghong Shi, Nathan Sjoquist, Cliff Slaughterbeck, David Sullivan, Ryan Valenza, Casey White. Ali Williford, Daniela M Witten, Jun Zhuang, Hongkui Zeng, Colin Farrell, Lydia Ng. Amy Bernard, John W Phillips. R Clay Reid, and Christof Koch. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neur- science, 23(1) 151, January 2020a. ISSN 1097-6256. doi: 10.10388:41533-019--555 648932
URL wtt.http:://o.... http:::s///euuopppmm....
Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature neuroscience, 23(1):138-151, 2020b.
Daniel A Dombeck, Anton N Khabbaz, Forrest Collman, Thomas L Adelman, and David W Tank. Imaging large-scale neural activity with cellular resolution in awake, mobile mice Newron, 56 (1):43-57, 2007.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. URL.ttts:: http:://///xwv.oo/r///////
201 0. 1 1929
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-languuag representation learning. CoRR, abs/2006.06 061955, 2020. URL
bttpp::/aaxivv org/ http:///b///0000 htTp:://axxxv.orrr//////2205 06195
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer EI Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Cauffeld-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. The capacity for moral self-correction in large language models, 2023.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Learning spatio-temporal features with 3d residual networks for action recognition, 2017. URL https::/aaxiv. wtw.///////////..//.//////.
07632
Christopher D. Harvey, Forrest Collman, Daniel A. Dombeck, and David W. Tank. Intracellular dy- namics of hippocampal place cells during virtual navigation. Nature, 461((266):9419944 2009. doi: 10.1038/aauure084499 URL http:///oo..oooo.....o/////tttta
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2019. URL https::::
09751
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-339, Melbourne, Australia, July 2018, Association for Com- putational Linguistics. doi: 10.18653/v1/P18-. 1031. URL bttps://claatthhoogg.. P18-1031
Cheng-Zhi Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, An- drew M. Dai, Matthew D. Hoffman, and Douglas Eck. improved relative self-attention mech- anism for transformer with application to music generation. CoRR, 08881142 2018. URL http:://rre
org/ abs///b///// 1et//9989900011111
Andrew Jacgle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira, Perceiver IO: A general architecture for structured inputs & outputs. CoRR, abs/2107.14795, 2021a. URL
httg/bb//// abss 2tabs//110.. 2107. 14795
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General perception with iterative attention. CoRR, abs/2103.03206 2021b. URL
hattps://axee
org. obg///s////3...... 03206
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations, 2022.
W. F. Kindel, E. D. Christensen, and J. Zylberberg. Using deep learning reveal the neural code for images in primary visual cortex. J. Vis, 19., 2019.
J. Kubilius, M. Schrimpf, A. Nayebi, D. Bear, D. L. K. Yamins, and J. DiCarlo. CORnet: Modeling the Neural Mechanisms of Core Object Recognition. bioRxiy, 1-9, 2018.

---

Published as a conference paper at ICLR 2024
Junnan Li, Ramprasaath R. Selvaraju. Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation, 2021. URL/ttps:: /ttp//iiivvv...//../... wttpv//.rrrabbbb//1/1/ 2107.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Kei Chang. Visualbert: A simple and performant baseline for vision and language, 2019. URL https://arxiv. www. abs/ 1908. 03557
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CoRR, abs/2103. la2 URL/ttts:://are wwwpaaawwww.gga//////a/////// 14030
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows, 2021b. URL https:
14030
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi- task vision and language representation learning. CoRR, abs/1912.02315,, 2019. URL http:
7/ 7/arxiv.com wwwxiv.org/aaa//9000 02315
Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King. Toward a realistic model of speech processing in the brain with self-super learning, 2022. URL httpps///aaxxi..or//bbo..
01685
Patrick Mineault, Shahab Bakhtiari, Blake Richards, and Christopher Pack. Your head is there to move you around: Goal-driven models of the primate dorsal pathway. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 28757-28771. Curran Asso- ciates, Inc., 2021. URL http:tttss/s/ooceeeeeeeeo mew.eeeeddddnnne.ee............. cc pctpppppp/222//11ee http://w.......o...oopoooo/////.
f1676935f930 FIL:6335633033763377772222222223
Paper. pdf
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representationn learning with contrastive predic- tive coding, 2018. URL
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks, 2012. URL
http///xxiiv.oo/////1/1111 abs/ 5063
Jonathan W. Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M. Litke, E. J. Chichilnisky, and Eero P. Simoncelli. Spatio-temporal correlations and visual signalling in a com- plete neuronal population. Nature, 454(7200):995-999) July 2008a. doi: 10. 10.1008/aauree07 140. URL
Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan Litke, Chichilnisky, and Fero P Simoncelli. Spatio-temporal correlations and visual signalling in complete neuronal population. Nature, 454(7207):995-999 Aug 2008b. ISSN 1476-4687 (Elec- tronic); 0028-0836 (Print); 0028-0836 (Linking). doi: 10. 10.1008/nature77140
Eftychios A. Pnevmatikakis, Josh Merel, Ari Pakman, and Liam Paninski. Bayesian spike inference from calcium imaging data. In 2013 Asilomar Conference on Signals, Systems and Computers, pp. 349-353, 2013. doi: 10. 1..11//ASCC..113.661100000 10293.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. htttt::ss://aat www.vorg/aaaae habgg//////// /2003. 30000
12

---

Published as a conference paper at ICLR 2024
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. URL httpp:://axxi......//
abs/2102. 12092
Blake A. Richards, Timothy Lillicrap, Philippee Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lind- say, Kenneth D. Miller, Richard Naud, Christopher C. Pack. Panayiota Poirazi, Pieter Roelf- sema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Kon- rad P. Kording. deep learning framework for neuroscience. Nature Neuroscience, 22(11): doi: 10. 10.10388/41193--19-00222
1761-1770, Nov 2019. ISSN 1546- 1726.
https:
//doi.org// 10. 1038/ 00338//41193 -00-0-11----00 0520-2
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e001623311 2021. doi: 10.1073/pnas:201623991
URL https: www.wwww........................ org/
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioral and neural analysis, 2022. URL httttttt//xxxiii/ org/ abs /tw.b//2222 2 00673
Nicholas A Steinmetz, Peter Zatka-Haas, Matteo Carandini, and Kenneth D Harris. Distributed coding of choice, action and engagement across the mouse brain. Nature, 576(7786):2662773, 2019.
Marcel Stimberg, Romain Brette, and Dan FM Goodman. Brian 2, an intuitive and efficient neu- ral simulator. el.ife, 8,e4331, aug 2019. ISSN 2050-084X. doi: 10.7 URL
pettps://doi.orgg 10. 7554/ fe. 47314
Jeffrey N. Stirman, Ikuko T. Smith, Michael W. Kudenov, and S. S. Wide field-of-view. multi- region two-photon imaging of neuronal activity, Nat. Biotechnol, 34, 2016.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth D Harris. Spontancous behaviors drive multidimensional, brainwide activity. Science, 364(6433))ccay789) 2019.
David Sussillo, Rafal Józefowicz, L. F. Abbott, and Chethan Pandarinath. LFADS latent factor analysis via dynamical systems. CoRR, abs/1608.06311 2016. URL attp:://aaxa /rtp:///rxxv./////////
abs abs/1608. 06315
Hao Tan and Mohit Bansal. LXMERT: learning cross- repres eowtremooodiiiyyoooddrrrreennnno from trans- formers. CoRR, abs/1908.07499 2019. URL
07490
Wilson Truccolo, Uri Eden, Matthew Fellows, John P Donoghue, and Emery N Brown. A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. J. Neurophysiol. 93(2):1074-1089 Feb 2005. ISSN 0022-3077 (Print); 0022-3077 (Linking). doi: 10. 10.11522jn.00667722000
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of of-thought prompting elicits reasoning in large language models, 2023.
13

---

Published as a conference paper at ICLR 2024
James C. R. Whittington, Joseph Warren, and Timothy E. J. Behrens. Relating transformers
to models and neural representations the hippocampal formation, 2021. URL https:
wwwplv/org//ab///// 04035
Daniel L K Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3:356--35,, March 2016.
Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optmmized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, (11(23))611-8624, 2014. doi:
10. 10.1073/pnas. 1403112111.
URL https:: http//wwwppp. pnas wwtp/oigggoooooooooooooott..oo.o /ttp://ooo/abs/111113.11
1403112111
Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers, 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks?, 2014.
Che-Hang Yu, Jeffrey N Stirman, Yiyi Yu, Riichiro Hira, and Spencer I. Smith. Diesel2p mesoscope with dual independent scan engines for flexible capture of dynamics in distributed neural circuitry. Nature communications, 12(1):6639, 2021.
Yiyi Yu, Jeffrey N Stirman, Christopher R Dorsett, and Spencer L. Smith. Selective representations of texture and motion in mouse higher visual areas. Current Biology, 32((1):2810- (01-))222202200 2022.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models, 2021. URL
https:
7/arxiv orl. /abs /2106. 62636
Yuan Zhao and Memming Park. Variational latent gaussian process for recovering single-trial dynamics from population spike trains. Neural Computation, 2965):193--1316 may 2017. doi: 10.1162/neco a 00953. URL/LLLp:///0...o///1.1.. 11t..og///...1122hhecc...c. a 00953
Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. 2020. doi: 10.46550/ARXIV.0011
URL
https://axxiv org/ abs 2011.111100 04798
14

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIALS
A LIMITATIONS
In preliminary experiments, the Neuroformer was trained on simulated data, revealing potential in identifying hub network connectivity. However, its reliability varied across datasets. For instance, multi-channe simulation, neuron attention sometimes accurately identified the correct channel, but at other times was noisy. In another test with neurons responding to naturalistic video via 3D gabor filters, the method's reliability in uncovering receptive fields was inconsistent. We think this technique could benefit from more advanced methods for calculating the attention like Grad-CAM.
Our model, the first of its kind using an auto-regressive decoder- based generative pre-training ap- proach, contrasts with encoder-based methods like LFADS or CEBRA. While these methods explic- itly exploit the intrinsic dimensionality of neuronal populations, the Neuroformer, inspired by the GPT objective, takes different approach. Out attempts to approaches yielded poor results (see Fig. 10). These methods are more typically applied to motor control tasks for BCI applications. Direct comparisons with methods like LFADS needed, as they might currently offer better representation learning due to explicit low-dimensional constraints. In contrast, our method is more akin an in-silico model of the brain.
data with representation learning
While transformers, like the Neuroformer, offer advantages like being modality- and con- textualizing information over extended sequences, traditional methods like the GLM and CNNs have their merits. They are grounded in neuroscientific assumptions, offering interpretability. Although the Neuroformer showed promise in decoding tasks, the reasons for its performance advantages remain unclear. Factors like overparameterization, ability to model -on-lincarisies, and effective representation learning might contribute.
Our aim with the Neuroformer was two-fold: 1) to introduce a tool for neuroscience to flexibly incorporate data from increasingly diverse and complex experimental settings and 2) to explore generative pre-training for such data. The true potential of this method might be realized with increased scale, given the vast data available in systems neuroscience. The Neuroformer marks step towards this, but challenges persist. We're keen scaling these methods and finding synergies between large language and neuroscience models.
B ETHICAL STATEMENT
This project involves data from mouse experiments. Our AAALAC (Association for Assessment and Accreditation of Laboratory Animal accrediidd institution is committed the humane and use of animals research and education. Our Institutional Animal Care and Use Committee (IACUC) oversees animal care and use, facilities, and procedures. The attending veterinarian and IACUC ensure that research activities involving mice meet the ethical and legal requirements as set by relevant policies and guidelines.
15

---

Published as a conference paper at ICLR 2024
C
EXPERIMENTAL SETUP
Naturalistic Complex
Visual Navigation
Generate Spikes
Multi-Region Brain imaging
Locomotion
Decode Variables
Teuroformer
Heanc
Align Modalities
Eye Tracking
Reward
mis
Figure 7: Neuroformer can incorporate a diverse range of modalities that are recorded in large-scale systems neuroscience experiments. Here, we show some of the modalities collected in the visual navigation experiment used to train our model.
16

---

Published as a conference paper at ICLR 2024
D HYPERPARAMETERS
D.1 NEUROFORMER
Hyperparameter
Symbol
VI+AL
Visnav
Video resolution (s)
dirrames
0.05
0.05
dt token resolution
dt
0.01
0.005
Behavior feature resolution (s)
dt delluvior
0,05
Dropout (p)
p
0.35
0.35
Behavior Dropout (p)
Pochavior
0,45
Contrastive Learning Temperature
Telip
0.25
0.25
Contrastive Learning Latent Dimension (n)
Eciip
1024
1024
Embedding Dimension (n)
E
512
512
Window current state (s)
0.05
0.05
Window past state (s)
up
0.25
0.15
Window frames (s)
Wirames
0.3
0.2
Window behavior (8)
Wbchavior
0.05
3D Convolution Kernel Size
Okernel
(6, 8, 8)
(4, 5, 5)
3D Convolution Stride Size
Cstride
(6, 8, 8)
(2, 5, 5)
Number of state layers (n)
Layete_
Number of state stimulus layers (n)
Rstate stimulus layers
8
Number of self-attention layers (n)
Thself attention layers
Number of behavior layers (n)
Thehaviorr
Number of heads (n)
Thhcad
Table Hyperparameters used for training models on V1+AL and Visnav datasets. The units are added in the Hyperparameter column where applicable: for seconds, n for number (count), and probability for dropout rates.
E COMPARISON MODEL DETAILS
E.1 POPULATION RESPONSE
E.1.1 GLM
A generalized linear model (GLM) for neuron data was adapted from (Pillow et al. 2008a). The instant firing rate (r)( individual neurons was modeled by a linear-nonlinear cascade, including linear stimulus components (k æ) and spike history components (h y) followed by an exponential nonlincarity. represents the baseline firing rate of the neuron. The stimulus kernel (k) and history kernel (h) are modeled by cosine bases. The parameters are optimized by minimizing the log- likelihood for spike prediction.
ri(t) exp(k h
(5)
E.2 BEHAVIOR DECODING
E.2.1
MLP
The MLP used for the hehavior comparisons was comprised of 5 stacked MLP layers, with a hidden state of (h 1024), GeLU non-linearity, dropout 0.01, layer-normalization between each of the MLPs. No improvement was observed upon adding more layers.
E.2.2 GRU
The GRU model used for the behavior comparisons had the following architecture: 5 Bi-directional GRU layers with dropout 0.01, norraaoormliizaiion, and tapering r=0.5 for each consecutive hidden state. Tapering allowed us to stack more GRU layers without undergoing collapse. More layers resulted in decrease in performance and eventual collapse. The first hidden state of the
17

---

Published as a conference paper at ICLR 2024
model was initialized as the neural past state defined in the paper, allowing the model to integrate past information. We employed Neuroformer-style tokenization which shortened the sequence of spikes and resulted in better performance.
18

---

Published as a conference paper at ICLR 2024
Populate Next
Interval(n)
Block Index...
Interval(n +1)
SOS
PAD
PAD
PAD
PAD
PAD
PAD
T1
T2
13
T4
T4
15
T6
17
TN
EOS Reached, Send Buffer Start Next Interval
Masked/Pad Token
ID Token
Figure 8: Current State causal block masking and padding. During training, the block is populated with all Current State spikes, and causally masked. To retain the same length of block, (and enable batched training) we right- pad the remaining sequence, and zero-out the loss corresponding from the PAD positions. During Inference, (e.g.. video frames) are passed into the model. The block is autoregressively populated starting from the SOS position with ID tokens, until EOS is reached. The predicted spikes are then fed into the buffer, so they can be incorporated into the Past State.
Current State is initiated, and all corresponding modalities
F BLOCK-SPARSE CAUSAL MASKING
To preciscly align the corresponding neural features with all other modalities, broke state features into a Current State w. and a Past State Wp according to specified temporal windows (see Table 2).
Because of this, we cannot simply model the neural features as a continuous string of spikes (or the equivalent language tokens) in the case of standard GPT model (Radford et al., 2019). The unique challenge of arranging our data in this way is that each of these intervals contains a different number of spikes. Our situation is similar (Ramesh et al., (02TT) where during inference time, the block is autoregressively populated by the predicted tokens. The difference in our case is that each resulting "block" is of different size. We therefore employ right- masting and padding enable efficient training and causal inference (see Fig. 8
Note that by just masking our Current State and projecting from the T,, row of the block during in- ference time, we are able to fuse with any other non- masked features, while preventing any backflow of information, and retaining causal aspect of our model.
19

---

Published as a conference paper at ICLR 2024
G CALCULATING CONNECTIVITY WITH OTHER METHODS
Partial
Ground Truth
1003
Teree
40
A00
200
Source Nearnn ID
Scurce Futuron
Soaree Neuren
Figure 9: Comparing derived connectivity with other methods. In addition to Pearson correlation, compared the connectivity from attention to that derived from partial correlation and Granger causality. The square root the sum of the absolute squares of elements of the attention matrix was 140.4, for Granger causality it was 138.1, 1, and for partial correlation, it 156.9. These values indicate how each method compares in terms similarity to the ground-truth connectivity matrix, with lower values suggesting closer resemblance.
20

---

Published as a conference paper at ICLR 2024
H CAPTURING TRIAL-TO-TRIAL VARIABILITY
When investigating the performance of the model as we progressively incorporated each module, examined Fl scores, and the ability of the model capture the trial-to-trial variation across the data sets.
To do this, we computed the correlation across trial-averaged spikes for each neuron of four trials. We compared a holdout group to the corresponding set generated by the Neuroformer, and another corresponding group of trials from the data set. We observed steady improvement in the distribution of correlations when compared to the ground-truth (Fig. T0 with the full model's predictions closely matching those of the ground-truth sets.
groups
PSTH Corre ations (V1 AL)
State
-Simuluss
60
70
80
70
60
30
75
50
20
20
25
10
10
0.2
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.4
0.0
0.2
0.4
0.6
Pearson
Figure 10: Trial-averaged correlations between groups of trials. The trials generated by the model progressively converged to the correlation distribution of the ground-truth trials as we added com- ponents, The model configurations were: Current State, Current State Pust State, Current State Past State + Stimulus, Current State + Past State Contrastive.
21

---

Published as a conference paper at ICLR 2024
I PRECISE BEHAVIOR PREDICTIONS
An advantage of the Neuroformer architecture, is that modalities can be flexibly incorporated as an input label. In the case of labels, variables can be modeled as both classes in multi-class logistic regression using a cross-entropy objective (Fig, [T]. or raw values as a linear regression using mnen--suared error (Fig. 12). Our model vastly outperformed other baselines, and performed reliably across datasets and objectives, as opposed to other methods, whose performance fluctuated.
p-r EACL
410
24:2:
<a COT
40 40
Figure 11: Speed Predictions, Classification. The speed variable was broken into 25 discrete classes and optimized with multi-class crossentropy loss.
Haurffomer
1-232
5:4
60
15
43 00
10.
5-118l
C001
$.0000
-18
Figure 12: Speed Predictions, Regression. We used MSE loss to optimize the models.
22

---

Published as a conference paper at ICLR 2024
J MULTITASK DECODING
In addition to speed, Neuroformer is able to jointly predict many other variables. Here we show that the model could effectively predict the eye position the mouse even in the absence of input stimulus. We used a single model checkpoint to decode all three variables. Speed and longitudinal latitudinal angle for eye position.
Visnav lateral Multitask Decoding Speed Eye Gaze (phi, th)
25-
0.93
0.84
0.92
20
< 0.001
0.001
p 0.001
Predicted speed
Predicted phi
Predicted 2
-..5
-14--
-2.
-1a
True speed
True phi
True
Figure 13: Decoding Speed (m/s) and Eye position phi, th (?).
K
MULTIMODAL LATENT VARIABLES
The contrastive part of Neuroformer, enables us to align any the corresponding auxilliary modali- ties with neural data, including visual features, behavioral features reward. Here show the raw features extracted from the contrastive stage that was trained to align speed with neural features.
No dimensionality reduction method (like T-SNE) was used for this visualization. These the raw features, extracted from our contrastive alignment module, by setting Ealip = 3.
ms
Figure 14: Contrastive Latents: Neural Features Separate According to their Speed.
23

---

Published as a conference paper at ICLR 2024
L ANALYSIS OF INTERMEDIATE FEATURES AND PROBING SPEED
L.1 INTERMEDIATE FEATURES
The features represented in Figure 14 were derived from the contrastive alignment module by setting the Eelip to 3, effectively projecting the features to 3 dimensions for alignment during training, as detailed Hyperparameters Our initial hypothesis posited the need for two dimensions for direc- tion-considering movement occurs on 2D plane-and one dimension for magnitude. Although we did not expect intermediate representations to fully segregate according to speed without explicit supervision, the process proved beneficial for analytical purposes. As indicated by the toy model experiment in Figure 2, a convergence towards speed-based separation was observed. Additional in- termediate representations were extracted from transformer blocks of model not trained speed, and linear regression was performed to map these to speed. These were compared against random arrays of identical shape control. supplementary material. As more modalities fused, the resulting representation becomes more linearly separable in terms of speed.
L.2 LINEARLY PROBING SPEED
Feature
R²
Random
0.0521
Past State
0.2299
Past State Stimulus
0.5341
Past State Stimulus Current State
0.6357
Table 3: Linear regression R² values for different features when probing for speed. Colors coded according to model architecture (Fig.
24

---

Published as a conference paper at ICLR 2024
M CEBRA BEHAVIOR RESULTS
We also attempted to compare Neuroformer behavior predictions with the SOTA in representation learning methods, CEBRA (Schneider al. 2022). Despite attempting an extensive hyperparameter search accross batch size, temperature, learning rate, offset, and decoding method (KNN and Lasso Reg.) we were unable to get the model generalize, and typically observed heavy overfitting as seen in Fig. 15
Training
Evailation (-KNN decoding)
100 training iterations
decoding r= -0.02
1.0
35
05
10
00
-10
5000 training ilerations
decoding - 0.51
Chandin
-10
20
Doenown 2
42
10000 training Iterations
decoding
-10
-35
21.5 12-00 Mae Sa
-25
-30
Figure 15: CEBRA decoding. The visualization depicts the latent features produced, colored by speed.
25


Reviews:
[{"review_id": "VJTFmLxEnS", "reviewer": "Reviewer_HBVf", "summary": "This manuscript introduces a multi-modal, multi-task generative pretrained transformer as a tool for analyzing the increasing volume of data generated by large-scale experiments in system neuroscience. The goal of this tool is to create a better neural spiking model while taking external variables into account.  In particular, they applied the Perceive IO architecture (Jaegle et al. 2021) to the neural domain and modified it accordingly. During the process of decoding, they also developed feature backbones, which enabled the specialized architecture to track the activity of individual neurons. Their loss function has a component that deals with alignment as well as one that deals with spike creation. The alignment component explicitly enforces representational commonalities among biologically significant features. The causal spike modeling is used by the spike generation component to do an autoregressive decoding of brain spikes. They used a simulated dataset in addition to two different two-photon calcium imaging datasets in order to verify the accuracy of this neuroformer. They demonstrated, with the simulated dataset, that the neuroformer is capable of recovering the hub-neuron structure that is comparable to the ground truth. Using the calcium imaging dataset, they were able to demonstrate that the suggested neuroformer performed better than GLM when it came to creating neuronal spikes. They also showed that a pretrained neuroformer has more accurate predictive features of mouse behaviors than baselines like Lasso regression, GLM, MLP, and GRU. In addition to this, they presented the results of an ablation investigation, which demonstrated that each module contributes progressively to predicting eye position.", "questions": "Does this architecture outperform any of those previous approaches in terms of generating realistic neural spikes? Is it possible to show the performance of the neuroformer on the Neural Latent Benchmark? \n\n \n\nWhat is the range of T_p in those calcium imaging datasets? Is it possible to demonstrate long-term inference with a neuroformer? \n\n \n\nIn Fig. 3d, the attention maps seem to suggest that the intermediate blocks of the neuroformer contain interesting features. It is common for the community that pretrains transformers for time series (like HuBert or Whisper for speech) to use features from intermediate blocks for decoding. Is it possible to show how well these intermediate blocks can be used to predict behavior? \n\n \n\nMinor question: I could guess that the red dot is the model being used to generate b) or d). It would be helpful if the authors could clarify this in the figure caption.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The loss function combines multi-task representation loss with two losses relevant to generating neural spikes. To the best of my knowledge, this particular application of multi-task learning to modeling neural activity is new. \n\nThe feature backbones in the Neurofomer are able to dissect single neurons. A common limitation of previous machine learning approaches to modeling population activity is that they lose single neurons. This feature seems to circumvent such a limitation.", "weaknesses": "Lack of comparison with strong baselines is my main concern for this submission. Prior to this paper, there were a couple notable publications that leveraged the transformer architecture to generate neural spikes. Albeit those most well-known ones are single modality only, it is still worth a comparison in terms of neural modeling. This work only showed its comparison with simple baselines (MLP, GRU) when it compared the quality of neural spike generation. If this neuroformer does not perform as well as other transformer-based architectures, I would hope the authors may include more elaborated discussion on whether the appeal of cross-modality representation outweighs its limited performance. \n\n1) Liu 2022 Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers\n\n2) J. Ye and C. Pandarinath, “Representation learning for neural population activity with Neural Data Transformers,” Neurons, Behavior, Data analysis, and Theory, Aug. 2021\n\nThe F1 scores in Figure 5 are rather low at their absolute values. It would be helpful if the authors put the F1 score in perspective (why is this F1 score indicating good performance?). Is it possible for the authors to comment on the dip of the F1 score after adding the video modality in the Visnav, Lateral dataset? \n\nThe correlation difference in Fig 3C is also low in comparison with GLM. Such a comparison will be a lot stronger if it is versus another transformer architecture or more elaborate architecture that is capable of expressing neural activity fully. \n\nSpeed is misspelled as “spped”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8Pfl2ZEow", "reviewer": "Reviewer_pMwz", "summary": "The paper presents a multi-modal Transformer-based pretraining paradigm for learning joint representations of neural activity, behavior and sensory inputs. It mainly follows recent work in vision-language modeling and adapts these approaches to the neuroscience setting. The paper shows that (a) the attention maps can reveal the circuit structure in a simulated toy dataset with a few hub neurons, (b) it can predict neural activity based on past activity and stimulus slightly better than a generalized linear model, and (c) it can decode running speed of a mouse from neural activity in a few-shot manner.", "questions": "Overall I think this is a super interesting and potentially very useful paper, albeit with some weaknesses, which I will detail below. If the authors can address these points in their response, I am happy to reconsider my score.\n\nWhile I appreciate the experiments, I believe the authors could do a better job at demonstrating that their approach actually works well. \n\n### Fig. 2\n\nIt is well known that inferring connectivity from correlations is not a good idea. The most straightforward way of getting closer to connectivity (albeit with a number of limitations and caveats as well) is using partial correlations instead of Pearson correlations. I would predict that the partial correlation matrix would correspond much more closely to the ground truth than the Pearson correlation matrix shown in Fig. 2d. Does your approach perform on par with it or even better?\n\n### Fig. 3\n\nI am somewhat underwhelmed by the result in Fig. 3: All this effort only to improve the correlation between prediction and ground truth by 2%? It may be significant, but the effect size is tiny. Papers on predicting activity from visual stimuli typically show quite more substantial improvements of neural nets over GLMs (e.g. McIntosh, NeurIPS 2016, Klindt et al., NeurIPS 2017, ...). I would like to see some stronger baselines here. The baseline model form the recent Sensorium competition (https://www.sensorium-competition.net) would be a fairly straightforward starting point.\n\nA second point on this figure: I am unsure how to interpret the attention maps in panel d). What exactly do they show? The word \"neuron\" seems overloaded here. Since the Transformer does not have a token for each neuron (or did I misunderstand something here? According to p.3 bottom the current state does not have a neuron dimension, only batch, time and embedding), I don't quite understand, what, e.g., \"Neuron 1, Layer 0, Head 0\" means. If it refers to neurons in the brain, please explain how and explain what we don't see localized receptive fields as we would expect from V1. If it refers to something else, please explain what and what we see. \n\n\n### Architecture not clear\n\nI had a hard time following the description of the method on pages 3+4. Some examples:\n\n 1. This sentence on p.3 could be unpacked: \"At each step, a learnable look-up table projects each neuron spike contained in our Current and Past States onto an embedding space E, resulting in vectors (T_c,E) and (T_p,E), where Tc,Tp are the corresponding state’s sequence length (number of spikes + padding).\" <-- What does \"vectors (T_c,E)\" mean? In particular, what is the meaning of the parenthesis around T_c,E? T_c appears to be the sequence length, i.e. scalar. Does it mean you literally concatenate a scalar that indicates the length with a vector E? If so, why? If not, what is happening here?\n\n 1. It is not clear to me what the decoder outputs. From the text and figures I'm guessing it's some form of sparse representation of the spiking in the future, where ID is the row (column) and dt the column (row) of a non-zero entry in a binary matrix (size: #neurons x #timesteps) that contains the spikes. However, I am unsure about this interpretation and cannot map it onto the losses in Eqs. 3+4. Part of the problem might be that p_i and p_dt are not defined and I am not sure how to interpret the cross-entropies. Also, I don't understand why there are two losses. Why not simply output a vector of zeros and ones that is the same size as there are neurons? What is the meaning of dt? I thought you're predicting the next frame?\n\n 1. The last paragraph on p.4 is equally opaque to me and I couldn't make sense of it. The sentence with nucleus sampling is unclear and the meaning of sub-intervals is also not clear.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ Promising self-supervised learning paradigm for large-scale, multi-modal neuroscience data\n + Nice set of experiments from simple toy model with known ground truth to real, large-scale data\n + Overall well-written and mostly easy to follow (with some exceptions)", "weaknesses": "1. Weak baselines in Figs. 2+3\n 1. Small effect in Fig. 3\n 1. Architecture (especially decoder) not entirely clear from paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iu907nYjDQ", "reviewer": "Reviewer_dTQU", "summary": "This paper proposes a multimodal, multitask generative pre trained transformer model called Neuroformer. This model uses an arbitrary number of modalities, such as neural responses,  external stimuli, and behavior, to perform downstream tasks. The authors apply this model to predict simulated neural circuit activities and behavior of a mouse from its neural recordings, where four modalities are used: neural responses, video, speed, and eye position. They also perform ablation studies to explore the impact of each model component.", "questions": "* What kind of modalities are used in section 4.3? Although we may infer these modalities in section 5, it seems there are no clear describing sentences about them in the whole section 4.3.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* Propose a multimodal, multitask transformer-based model for neural data modeling.", "weaknesses": "* In the experiments, the Neuroformer is only evaluated in terms of behavior prediction, which is not enough in evaluating neuroscience tasks. The choices of models (GLM, GRU, Lasso Regression) for comparison are also not convincing to me. One suggestion is to follow the evaluation criteria in [Neural Latents Benchmark](https://eval.ai/web/challenges/challenge-page/1256/overview) and compare the Neuroformer with the top leading models there, such as S5 and LFADS on the multimodal calcium imaging datasets. It is critical to see whether the proposed model is a solid technical innovation with practical influences in neuroscience research.\n\n* As the authors mentioned in Appendix A, the Neuroformer has poor results (Figure 10) in low-dimensional latent space learning. One possible reason is that the transformer-related neural networks are too expressive so that good dynamics in latent space are no longer necessary. But in terms of interpretability, neuroscientists prefer to observe meaningful latent space in many experimental scenarios, which may be more important than a good behavior prediction performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jc5tZvohik", "reviewer": "Reviewer_M1PZ", "summary": "In the paper \"Neuroformer: a multimodal, multitask GPT framework for brain data at scale\", the authors suggest a transformer-based architecture (Neuroformer) for fitting high-dimensional neural spike train recordings, that can incorporate a CLIP-like contrastive learning objective to use visual stimuli and/or behavioural recordings. The authors argue that Neuroformer can slightly outperform classic GLM models for spike prediction and strongly outperform simpler models for behavioural prediction.", "questions": "MAJOR COMMENTS\n\n* Section 3: I could not understand the details of the architecture from this description. I could not even understand the basic setup... The text says that each neuron is one token, is that right? So the model is limited by O(1000) neurons, as attention layers scale quadratically with the number of tokens, right? Next, what is one training example: one time bin? For each neuron and each time bin, we have some integer number of spikes. How is this number converted into an embedding vector? What exactly are past states (how many past states) and how are they passed into the model? How does prediction (as in section 4.3.1) work: what is passed as the input instead of neural states?\n\n   Perhaps this is confusing because one would naively think about time-series modeling along the lines of GPT, where time bins (and not neurons) would be tokens. So I think the architecture setup requiers a more detailed explanation.\n   \n* Section 4.3.1: the sentence \"our model's predictions w[h]ere more closerly corelated with the ground-truth\" should contain some quantification, e.g. the fraction of neurons (or what are the dots in figure 3c: are these neurons?) for which Neuroformer outperforms GLM, and also the p-value (0.02) which is currently only mentioned in the figure caption. The evidence here is not very strong, so the authors should not oversell.\n\n* Table 1 is the strongest result in the paper, as Neuroformer strongly outerforms all other models. However, neither the comparison models (Lasso, GLM, etc) nor the prediction task are described in sufficient detail. The task here to predict behaviour from neural responses, but what exactly is \"behaviour\", what is the input (how many time steps?) and the output (how many time steps) of this prediction problem, etc.? The authors should present their experiment such that it is clear the comparison models in Table 1 are not \"strawman\" in some sense.\n\n\nMINOR COMMENTS\n\n* \\citep and \\citet should be used instead of \\cite. Current citation formatting is not following the ICLR template.\n\n* Schneider et al 2022 has been published in Nature.\n\n* page 6: what are N_a and N_z matrices? Unclear.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is interesting because it applies modern transformer architectures to modeling neural data, and shows competitive results.", "weaknesses": "Overall I thought that the paper would perhaps be more suited for a computaional neuroscience journal or for NeurIPS that traditionally has some amount of comp neuro papers. At ICLR, this topic is an outlier, as there is very little (next to none) computaitonal neuroscience there.\n\nWhereas the paper is generally well-written, I found many model details not sufficiently clear (examples below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)