# Deep Learning of Mesoscale Cortical Dynamics for Real-Time Classification of Forelimb Movement in Mice

Clément Picard, Anton Dogadov, Daniel E Shulz, Isabelle Férézou, Valerie

Ego-Stengel, Luc Estebanez

► To cite this version:

Clément Picard, Anton Dogadov, Daniel E Shulz, Isabelle Férézou, Valerie Ego-Stengel, et al. Deep Learning of Mesoscale Cortical Dynamics for Real-Time Classification of Forelimb Movement in Mice. IEEE 35th International Workshop on Machine Learning for Signal Processing (MLSP 2025), Aug 2025, Istanbul, Turkey. 10.1109/MLSP62443.2025.11204287. [hal-05352153](https://hal.science/hal-05352153v1)

HAL Id: hal-05352153

https://hal.science/hal-05352153v1

Submitted on 14 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# DEEP LEARNING OF MESOSCALE CORTICAL DYNAMICS FOR REAL-TIME CLASSIFICATION OF FORELIMB MOVEMENT IN MICE

Clément Picard, Anton Dogadov, Daniel E. Shulz, Isabelle Férézou, Valérie Ego-Stengel, Luc Estebanez

Institut des Neurosciences Paris-Saclay, CNRS, 91400 Saclay, France

## ABSTRACT

Recent studies suggest that mesoscale cortical activity patterns encode movement-related information, such as direction and speed. We developed an approach to decode forelimb movement direction from mesoscale cortical dynamics in mice using wide-field GCaMP6f imaging. By simultaneously recording cortical activity and 3D kinematics of specific body points, we built a dataset linking 450 ms windows of cortical activity to movement categories (forward, backward, still). A convolutional neural network (CNN) trained on this dataset achieved real-time decoding at 100 Hz and sub-10 ms latency, with a correlation of $r = 0.72$ between predicted and ground truth values. Although separate models were trained per animal, output distributions were highly consistent (inter-animal $r \ge 0.91$), suggesting that mesoscale cortical dynamics related to movement are consistent across individuals. These results demonstrate the feasibility of the online decoding of movement types from wide-field cortical signals, towards minimally invasive upper limb neuroprosthetic.

**Index Terms—** Brain-machine interface, Neuroprosthesis, Cortical dynamics, Deep learning, Motor control

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** From mesoscale cortical activity to movement prediction.</figcaption></figure>

Wide-field calcium imaging (A) captures mesoscale cortical activity, which is preprocessed to enhance signal quality (B). These images are then fed into a 3D CNN (C). After being thresholded (D), its output allows the identification of the current movement category at 100Hz.

## 1. INTRODUCTION

Brain-machine interfaces (BMIs) often rely on invasive electrophysiological recordings in the primary motor cortex (M1) to decode motor plans. But they degrade over time, and provide limited coverage of the area of interest [1, 2, 3]. In contrast, mesoscale surface measurements — such as ECoG and wide-field calcium imaging with calcium indicators like GCaMP6 — offer spatial coverage and do penetrate the brain [4]. Recent studies in mice have correlated mesoscale somatosensory-motor cortex dynamics and body movements [5], suggesting that motor plans can be decoded from neural activity at this scale. In primates, researchers have shown that functional ultrasound imaging of the posterior parietal cortex can predict movement direction in real-time [6], while the propagation direction and speed of neural activity in M1 encode hand velocity [7]. In mice, local cortical activity recorded via wide-field calcium imaging has been harnessed for BMI control of a cursor [8, 9]. Despite these efforts to extract movement feature from mesoscale cortical signals, the real-time decoding of mesoscale cortical dynamics for BMI applications remains underexplored.

Here, we present a novel approach for real-time decoding of forelimb movement direction from mesoscale cortical dynamics (Fig. 1). In mice, we simultaneously recorded cortical activity from wide-field GCaMP6f signals and 3D limb kinematics, then run CNN-based classification of stillness, forward and backward movements. We achieve real-time movement classification at 100 Hz, with a mean absolute error of 0.34 between model output and actual movements (compared to a chance level of 0.67; $p < 0.05$, Mann–Whitney U test), even when excluding forelimb primary somatosensory regions. This paves the way for online, low-latency BMI control of upper limb prostheses.

## 2. MATERIALS AND METHODS

### 2.1. Mice and surgical procedures

Experiments were performed on 2 male and 1 female adult (> 8 weeks old) Emx1-Cre (Jax # 5628) x Ai95 (Jax # 24105) transgenic mice expressing GCaMP6f in a majority of
cortical excitatory neurons. Protocols were in accordance
with the French and European (2010/63/UE) legislation
and our licence (Ethical Committee #59, APAFIS#25932-
2020060813556163 v7). The mice were isoflurane anes-
thetized (3 to 4%, maintenance 1 to 1.5%), and a head fixation
bar was attached, as well as a transparent window above a
6-mm craniotomy centered on the M1 and the somatosensory
S1 cortices.

**2.2. Cortical and behavioral data capture**

2.2.1. *Experimental setup*

We performed simultaneous recordings of cortical activity
and body movements (Fig. 2A). The cortex was illuminated
by 200 Hz alternation of blue and green light. Cortex im-
ages were acquired through a green band-pass filter (Edmund
Optics 525/45, OD6) by a JAI GO-2400M-PMCL camera
mounted on a macroscope made of two photographic lenses
in a face-to-face configuration: a 50 mm f/1.2 Nikon lense
(sample side), and a 12.5 – 75 mm f/1.2 Fujinon zoom lens
entric illumination). This reduced hemodynamic contamination of
the GCaMP6 signal [9]. A 100 Hz camera (DMK 33UX273,
The Imaging Source) provided side and bottom body views
via a 45° mirror (Fig. 2A,B).

2.2.2. *Images preprocessing*

Cortical image were stabilized with an optical flow-based
method that canceled any whole-frame image shifts ver-
sus the first frame. Next, high-frequency spatial noise was
removed by a temporal sliding window (10 frames) and a
Gaussian spatial filter (29 pixels square kernel, SD: 4.7 pix-
els), followed by x8 downsampling to reduce computational
load.

From the body images, we tracked anatomical landmarks
with DeepLabCut [10] across the two body views by triangu-
lation (Fig. 2C). We finally classified forelimb movements as
forward, backward or stationary.

2.2.3. *Data acquisition sessions*

We located the limb S1 areas as peak cortical responses to
forelimb/hindlimb stimulations with a piezoelectric actuator
in the anesthetized mice (Fig. 4A left). To prevent sensory
contaminations of our decoder, the manually outlined S1 fore-
limb area was excluded from the input data (Fig. 4A). In
awake mice, we triggered reactive forelimb movements by
manually sliding a card underneath the mouse right paw.

**2.3. Movement decoding from cortical dynamics**

We built a dataset of 450 ms segments of cortical activity
and matching forelimb movements (0: main body axis). For

<figure><img src="image_3.png" /><figcaption>**Fig. 2. Simultaneous cortical imaging and movement recording.**</figcaption></figure>

A. A sliding plastic surface beneath the mouse's paw is moved back and forth to elicit a compensatory movement of the forelimb, imaged at 100 Hz by a body camera. Simultaneous 100 Hz imaging of the brain surface uses alternating green and blue light to reduce the hemodynamic signal and isolate the GCaMP signal. B. Side and bottom view of the mouse are imaged by the same camera thanks to a 45° mirror below the mouse. Color dots: body parts tracking. C. 3D position of the same body parts. D. The 3D speed of the forelimb allows us to label forward and backward movements, as well as periods of stillness. By linking these movement periods with cortical activity, we construct a dataset of 450 ms segments of cortical data paired with the corresponding movement category.
every frame during each forward or backward movement, we extracted one 450 ms segment, aligning the movement frame 100 ms after the segment's start. For example, a 100 ms movement at 100 Hz (10 frames) yields 10 overlapping 450 ms windows. "Still" segments were found at timepoints halfway between consecutive movements.

We implemented a CNN made of three 3-D convolutional layers followed by two dense layers (Fig. 3) to capture spatial and temporal features simultaneously. Its interpretability enabled us to localize the cortical regions that contribute most to movement prediction. The first convolutional layer applied 8 filters (kernel = 3 × 6 × 6, stride = 3 × 3 × 3); the second used 16 filters (kernel = 3 × 4 × 4, stride = 3 × 2 × 2); the third increased to 32 filters (kernel = 2 × 2 × 2). All convolutional layers used ReLU activations. The resulting feature map was flattened, passed through a fully connected layer with 128 ReLU units, and finally fed to a single linear output neuron.

We framed anteroposterior movement decoding as a scalar regression task: each sample was labelled with $t \in \{-1, 0, +1\}$ representing “backward”, “still”, and “forward”, respectively. The model was trained to minimize mean absolute error (MAE). MAE’s linear penalty limits the influence of occasional noisy outliers — unlike the quadratic growth of mean-squared error — and match our objective of reducing the absolute directional error along a single axis, yielding more stable gradients. At inference, we thresholded the predicted value to recover discrete classes (> 0.6 : 'forward'; < -0.6 : 'backward', in between: 'still') while retaining a continuous output.

We benchmarked our 3D CNN against two classical regressors trained on the exact same segments and cross-validation splits. First, multivariate linear regression, map-

ping the flattened cortical images directly to the continuous movement label (-1 to +1). Then, support vector regression (SVR) with the default radial basis function (RBF) kernel (as in scikit-learn), using the same flattened input. Both models were fit on each training fold, and their continuous outputs were thresholded at -0.6 and +0.6 to yield the same movement classes as the CNN.

For each mouse, data were collected across over 4–6 sessions, 1 per day. We evaluated generalizability with a leave-one-day-out cross-validation: in each fold the network was trained on all but one day and tested on the held-out day. The network was optimized for 50 epochs with mini-batch gradient descent (batch size = 32) using the Adam optimizer (default parameters). A custom data generator presented mini-batches in a round-robin fashion: it drew one segment from each class in turn (forward → still → backward → ...). When the generator exhausted the segments of a given class it reshuffled that class and continued sampling so that, within every batch and across an epoch, each class contributed exactly one-third of the training examples, preventing the model from collapsing onto the dominant “still” class. After training ten random-seed initializations per fold, we selected the run with the lowest average absolute error across all movement categories as the best-performing model. Final performance metrics were computed as a weighted average across the three best-performing models (one per mouse), with weights proportional to the number of samples in each validation set.

We quantified performance with two standard metrics in machine learning: *Precision* is proportion of predicted events that are correct (TP / (TP + FP)), while *recall* is the proportion of real events that are correctly predicted (TP / (TP + FN)), with TP: true positives; FP: false positives, and FN: false negatives.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** A 3D CNN to detect forelimb movements and stillness from cortical activity.</figcaption></figure>

The network processes 450 ms segments of cortical activity using successive 3D convolutional layers, capturing spatiotemporal patterns at multiple scales. The extracted features are flattened and passed through dense layers to predict the movement category: forward (+1), still (0), or backward (-1).
## 3. RESULTS

### 3.1. Decoding performance metrics

In cross-validation folds, the architecture achieved a MAE of 0.34 ± 0.03, compared to a chance level of 0.67 (Mann-Whitney U test, *p* < 0.05). Additionally, the Pearson correlation coefficient between predicted and actual movement labels was *r* = 0.72, demonstrating strong alignment of predicted outputs with ground truth.

Despite the variability of the data and class imbalance, the model achieved accurate decoding with an average of 143 forward movements, 294 still periods, and 145 backward movements: this data efficiency limits the need for challenging data collection in awake behaving subjects.

### 3.2. Movement-specific classification analysis

Using the best model for each fold, we examined how the model output values were distributed across ground-truth movement categories (Fig. 4B). For each true category, the output distribution was centered around the matching value, indicating that the model outputs reflect underlying movement distinctions. We then computed confusion matrices based on thresholded predictions (Fig. 4C). The 3D CNN average matrix (Fig. 4C) revealing diverging performance across metrics. While precision was high for forward (0.84) and backward (0.95) movements, recall values were considerably lower (0.56 and 0.59 respectively), indicating that these movements were often missed. The “still” category showed the inverse pattern with excellent recall (0.96) but moderate precision (0.56), suggesting frequent false positives.

<figure><img src="image_5.png" /><figcaption>**Fig. 4. Decoding performance.**</figcaption></figure>

A. Left: Map of cortical pixels activated by passive stimulation of the forelimb (red), hindlimb (blue), or both (purple). Right: forelimb S1 mask. B. Distribution of the continuous output values for the best-performing model of each fold, grouped by ground-truth movement category. C. Confusion matrices comparing our 3D CNN (left), linear regression (middle), and SVR rbf (right). Each matrix shows the average predicted label proportions (± standard deviation) for forward, still, and backward classes across cross-validation folds. D. Best performance confusion matrix: Average confusion matrix of the single most accurate model for each mouse. E. Example processing of continuous data. Top: model output over time. Blue: output > 0.6 (forward predicted). Red: < -0.6 (backward predicted). Bottom: ground truth of anteroposterior speed of the front paw. Predictions were shifted 350 ms earlier to account for the temporal lag between cortical signals and movement onset, and the duration of forward and backward predictions was extended by 150 ms on each side of the period to enhance visual clarity. F. Histogram of decoding latency.
To contextualize these findings, we compared our CNN against two classical regressors: multivariate linear regression and SVR-RBF. The CNN correctly labeled forward, still, and backward movements at 56 %, 96 %, and 59 %, respectively, versus 47 %, 55 %, and 46 % for the linear model, and 32 %, 92 %, and 34 % for SVR-RBF. This better performance of the CNN highlights its ability to extract spatiotemporal features from cortical dynamics.

When focused on the best model for each mouse (Fig. 4D), performance improvements varied notably by category. Most dramatically, backward movement recall increased substantially from 0.59 to 0.90 while precision improved from 0.95 to 0.98. In contrast, forward movement detection showed only modest improvement in recall (from 0.56 to 0.67) despite increased precision (from 0.84 to 0.96). The “still” category showed intermediate improvement, with precision increasing from 0.56 to 0.70 while recall remained high at 0.96.

To evaluate model predictions on continuous data (Fig. 4E), we implemented a post-processing module within the decoding pipeline to ensure temporal consistency. A temporal filtering step removed unstable predictions: predicted labels were retained only when no conflicting prediction occurred within a ±100 ms window. In cases of ambiguity, the label associated with the highest absolute model output was selected.

## 3.3. Cross-subject consistency of decoder outputs

We trained separate models for each mouse using the same CNN architecture, and evaluated the consistency of output distributions across subjects. Despite being trained independently, models produced highly similar output distributions for each movement category (Fig. 4B), with Pearson correlation coefficients of *r* = 0.79 ± 0.15 (forward), *r* = 0.94 ± 0.04 (backward), and *r* = 0.99 ± 0.01 (still). This suggests a consistent structure of the cortical signals across animals — possibly reflecting common mesoscale dynamics well suited to our CNN architecture.

## 3.4. Real-time decoding performance

We run the decoder at 100 Hz and benchmarked its latency over 3 × 10⁴ consecutive frames (5 minutes) on our experimental setup, using a Windows 10 workstation equipped with an Intel Core i7-9700 CPU (8 cores, 3.0 GHz base, no GPU acceleration), with frame acquisition and buffering handled by an onboard FPGA. Image frames were timestamped by the camera hardware. The mean end-to-end delay up to the generation of the corresponding class label was 9.00 ± 0.97 ms (Fig. 4F), comfortably within the 10 ms budget required for real-time processing at 100 Hz.

# 4. DISCUSSION

## 4.1. Achievements and implications

We showed that movement direction can be effectively decoded from wide-field calcium imaging at 100 Hz with latencies suitable for real-time applications such as BMIs. The correlation (*r* = 0.72) between predicted and actual movement labels indicates that mesoscale cortical dynamics contain rich information about movement parameters, again relevant motor BMIs. Compared to the online readout of mesoscale dynamics via ultrasound [6], our system relies on a simpler and more accessible signal, and supports the notion that mesoscale signals, even at the surface, can provide robust movement control signals. Compared to surface ECoG in humans [11], our approach offers higher frequency (100 Hz vs. 10 Hz) with shorter decoding windows (450 ms vs. 1 s). Overall, our findings highlight that surface optical signals enable fast, minimally invasive decoding of motor states, and may offer a scalable and generalizable pathway for future minimally invasive BMIs. To our knowledge, this is one of the first demonstrations of real-time decoding from wide-field calcium imaging at 100 Hz, supporting the use of surface optical signals for fast, online motor intention decoding.

## 4.2. Movement-specific performance analysis

The forward and backward movement predictions is encouraging, especially when minimizing false negatives is critical (Fig. 4). However, the model shows high specificity but lower sensitivity for movement detection, and the reverse for stillness. This suggests that the neural signatures underlying movement initiation may be more subtle or variable, especially for certain directions. In particular, while backward movement recall improved substantially with model selection — likely due to more distinct and consistent cortical activation patterns — forward movement recall remained stable, hinting at weaker or more distributed signals. Together, these results point to fundamental differences in how movement directions are encoded in M1, which we will have to better understand.

## 4.3. Data perspectives

We identified several directions for refinement of our data acquisition strategy. First, in our experiments, movement was assessed through video, capturing only dynamic contractions where muscles change length. We frequently observe mice performing isometric contractions by pushing against the floor, which calls for additional sensing [12]. Second, our experimental design relied on externally movement triggers rather than spontaneous or goal directed behaviors, potentially biasing the cortical activity patterns we observe. In future work, measurements during voluntary actions could provide more naturalistic neural data.
## 4.4. Model perspectives

So far, our model outputs a movement category. Our next step will be to generate a continuous motor parameter, such as forelimb speed. This could allow for the fine-grained BMI control required for a neuroprosthesis.

While each model was trained individually per animal, the strong consistency of decoder outputs across subjects suggests that movement-related cortical signals share common structure across animals. However, it remains to be determined whether the models rely on similar underlying features or converge on functionally equivalent solutions. This opens promising perspectives for the development of generalized decoders trained on pooled datasets, which could reduce the need for subject-specific calibration in BMI applications.

Finally, our current approach does not address the stability of decoding performance over extended time periods (weeks to months), which would be crucial for practical neuroprosthetic applications [13]. Future work must evaluate longer-term stability and explore adaptive decoding strategies to maintain performance as neural representations may evolve over time.

## 5. REFERENCES

[1] Manuele Gori, Gianluca Vadalà, Sara Maria Giannitelli, Vincenzo Denaro, and Giovanni Di Pino, “Biomedical and Tissue Engineering Strategies to Control Foreign Body Reaction to Invasive Neural Electrodes,” vol. 9, pp. 659033.

[2] Ashley N. Dalrymple, Sonny T. Jones, James B. Fallon, Robert K. Shepherd, and Douglas J. Weber, “Overcoming failure: Improving acceptance and success of implanted neural interfaces,” vol. 11, no. 1, pp. 6.

[3] Mikhail A. Lebedev and Miguel A. L. Nicolelis, “Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation,” vol. 97, no. 2, pp. 767–837.

[4] Angela K. Nietz, Laurentiu S. Popa, Martha L. Streng, Russell E. Carter, Suhasa B. Kodandaramaiah, and Timothy J. Ebner, “Wide-Field Calcium Imaging of Neural Network Dynamics In Vivo,” vol. 11, no. 11, pp. 1601.

[5] Francesco Resta, Elena Montagni, Giuseppe De Vito, Alessandro Scaglione, Anna Letizia Allegra Mascaro, and Francesco Saverio Pavone, “Large-scale all-optical dissection of motor cortex connectivity shows a segregated organization of mouse forelimb representations,” vol. 41, no. 6, pp. 111627.

[6] Whitney S. Griggs, Sumner L. Norman, Thomas Deffieux, Florian Segura, Bruno-Félix Osmanski, Geeling Chau, Vasileios Christopoulos, Charles Liu, Mickael Tanter, Mikhail G. Shapiro, and Richard A. Andersen, “Decoding motor plans using a closed-loop ultrasonic brain-machine interface,” vol. 27, no. 1, pp. 196–207.

[7] Wei Liang, Karthikeyan Balasubramanian, Vasileios Papadourakis, and Nicholas G. Hatsopoulos, “Propagating spatiotemporal activity patterns across macaque motor cortex carry kinematic information,” vol. 120, no. 4, pp. e2212227120.

[8] Kelly B. Clancy and Thomas D. Mrsic-Flogel, “The sensory representation of causally controlled objects,” vol. 109, no. 4, pp. 677–689.e4.

[9] Anton A. Dogadov, Daniel E. Shulz, Isabelle Ferezou, Valérie Ego-Stengel, and Luc Estebanez, “Operant conditioning of cortical waves through a brain-machine interface.”

[10] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge, “DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning,” vol. 21, no. 9, pp. 1281–1289.

[11] Alexandre Moly, Thomas Costecalde, Félix Martel, Matthieu Martin, Christelle Larzabal, Serpil Karakas, Alexandre Verney, Guillaume Charvet, Stephan Chabardes, Alim Louis Benabid, and Tetiana Aksenova, “An adaptive closed-loop ECoG decoder for long-term and stable bimanual control of an exoskeleton by a tetraplegic,” vol. 19, no. 2, pp. 026021.

[12] Amy Claire Kristl, Turgay Akay, and Andrew Miri, “Recording Forelimb Muscle Activity in Head-Fixed Mice with Chronically Implanted EMG Electrodes,” no. 205.

[13] Ethan Sorrell, Michael E. Rule, and Timothy O'Leary, “Brain-Machine Interfaces: Closed-Loop Control in an Adaptive System,” vol. 4, pp. 167–189.