DOI: 10.5061/dryad.2bvq83c34
title: Intracortical brain-computer interface for navigation in virtual reality in macaque monkeys

description: We present an innovative intracortical Brain-Computer Interface (BCI) to
 bridge the gap between laboratory settings and real-world applications.
 This BCI approach introduces three key advancements. First, we utilized
 neural signals from three macaque brain regions – primary motor, dorsal
 and ventral premotor cortex – enabling precise and flexible decoding of
 real-time three-dimensional (3D) sphere/avatar velocities. Second, we
 developed a realistic, immersive 3D virtual reality setup with dynamic
 camera tracking, allowing continuous navigation and obstacle avoidance
 that closely mimic real-world scenarios. Finally, our BCI approach is very
 well suited for use by paralyzed patients, featuring a brief passive
 fixation without overt movements and closed-loop operation without
 retraining of the decoder during online decoding, relying on the user’s
 neural plasticity and the decoder’s robust generalization across tasks.
 Our BCI adapted to different environments, targets, and obstacles,
 illustrating its potential to substantially enhance the quality of life
 for paralyzed patients by enabling natural, reliable and flexible control
 in complex settings.
description: # Intracortical brain-computer interface for navigation in virtual reality
 in macaque monkeys Dataset DOI:
 [10.5061/dryad.2bvq83c34](https://doi.org/10.5061/dryad.2bvq83c34) ##
 Description of the data and file structure This dataset contains
 trial-level data from three rhesus macaques performing brain–computer
 interface (BCI) navigation tasks in Unity-based virtual environments. Data
 are provided as Python pickle files (.pkl) containing trial structures
 with timestamps, behavioral trajectories/velocities, and multiunit
 activity (spike threshold crossings) from Utah arrays implanted in motor
 cortical areas. For many sessions, synchronized eye-tracking and surface
 EMG recordings are also included as separate .pkl files. Spike detection
 thresholds used during each session are provided as .ccf files. ### Files
 and variables #### File: Method_paper_data.zip A compressed archive
 containing all session data organized by subject (“Monkey 1/2/3”) and
 task. #### Folder organization inside `Method_paper_data.zip` ```
 Method_paper_data/ Monkey 1/ (Loki) Monkey 2/ (Vino) Monkey 3/ (Maui) ```
 Within each Monkey folder, task folders are present only for tasks
 performed by that animal: * Monkey 1 (Loki): performed 2 tasks (fewer task
 folders). * Monkey 2 (Vino): performed 5 tasks. * Monkey 3 (Maui):
 performed 6 tasks (includes first-person perspective continuous
 navigation). ``` Method_paper_data/ Monkey 1/ (Loki) Center-out/
 Continuous Navigation/ Monkey 2/ (Vino) Center-out/ Continuous Navigation/
 3D Center-out/ Respawn/ Obstacle/ Monkey 3/ (Maui) Center-out/ Continuous
 Navigation/ 3D Center-out/ Respawn/ Obstacle/ Continuous Navigation (first
 person perspective)/ ``` ##### Structure inside each task folder Each task
 folder has the following structure: ``` Task_Name/ trainingFiles/
 navtraining_____trials.pkl ___.ccf navdecoding_____trials.pkl ``` ##### Always present ###### trainingFiles/ Contains files required for decoder training: ``` trainingFiles/ navtraining*_trials.pkl — training trial data (passive observation phase) *.ccf channel configuration file (spike thresholds) used during that session ``` ###### Decoding files Located directly inside the task folder: * navdecoding*_trials.pkl — closed-loop BCI trial data * *.ccf channel configuration file (spike thresholds) used during that session There are multiple navdecoding*.pkl, navtraining*.pkl, and .ccf files within each task folder — one set per recording session. ##### Optional additional folders (session-dependent) Some task folders may also contain: ``` eyeMovementData/ offlineAnalysisPerArea/ emgData/ ``` * eyeMovementData/ — continuous eye position recordings. * offlineAnalysisPerArea/ — area-specific offline decoding results (e.g., M1, PMd, PMv). * emgData/ — continuous sEMG recordings. These folders are present only when that modality or analysis was collected/performed. Task folders contain multiple sessions (different dates/times/session letters), each with its own training/decoding files. #### Tasks and corresponding filename prefixes Each session typically includes: * a training file (navtraining…) used to train the decoder from passive observation * a decoding file (navdecoding…) containing closed-loop online BCI trials Task names used in the paper correspond to these filename prefixes: | Task (paper) | Decoding file prefix | Training file prefix | | :----------------------------------------------- | :--------------------------- | :------------------- | | Center-out | `navdecodingsphere_` | `navtrainingsphere_` | | Continuous Navigation | `navdecoding_` | `navtraining_` | | 3D Center-out | `navdecodingsphere_` | `navtrainingsphere_` | | Respawn | `navdecodingsphererespawn_` | `navtrainingsphere_` | | Obstacle | `navdecodingsphereobstacle_` | `navtrainingsphere_` | | Continuous Navigation (first-person perspective) | `navdecodingfpp_` | `navtrainingfpp_` | #### File naming convention (recommended for reuse) Most data files follow: `_____.pkl` Examples: * navdecoding_Maui_20240304_1131_A_trials.pkl * navtrainingsphere_Vino_20240206_0857_A_trials.pkl Where: * Subject is one of: Loki, Vino, Maui * YYYYMMDD is the recording date (YearMonthDay) * HHMM is local start time in 24h format (HourMinute) * SessionLetter is a session label (A, B, C, …) used internally during acquisition * suffix indicates content (see below) #### File types included **(1) Trial data (.pkl)** * *_trials.pkl : training or decoding trial structures (core dataset) * *_emg.pkl : surface EMG data (available for a subset of sessions) * *_eyeMovementX.pkl and *_eyeMovementY.pkl : eye position traces (available for a subset of sessions) **(2) Spike threshold configuration (.ccf)** * *.ccf: spike detection thresholds and electrode metadata used during that session **(3) Offline analysis results** * offlineAnalysisPerArea/decoding_results.pkl (folder structure may repeat per monkey/task/date)\ Contains offline decoding predictions using different electrode selections (per area or combinations). #### Number of sessions per task and monkey The number of recording sessions (i.e., number of navdecoding*_trials.pkl files) per task and per animal is: * Center-out task * Monkey 1 (Loki): 5 sessions * Monkey 2 (Vino): 10 sessions * Monkey 3 (Maui): 19 sessions * Continuous Navigation task * Monkey 1 (Loki): 6 sessions * Monkey 2 (Vino): 7 sessions * Monkey 3 (Maui): 15 sessions * 3D Center-out task * Monkey 2 (Vino): 12 sessions * Monkey 3 (Maui): 10 sessions * Respawn task * Monkey 2 (Vino): 11 sessions * Monkey 3 (Maui): 9 sessions * Obstacle task * Monkey 2 (Vino): 10 sessions * Monkey 3 (Maui): 11 sessions * Continuous Navigation task (first person perspective) * Monkey 3 (Maui): 3 sessions Each decoding session corresponds to one navdecoding*_trials.pkl file and one associated .ccf file.\ Training sessions are stored separately as navtraining*_trials.pkl files. A single training session may be used to train a decoder that is applied to multiple decoding sessions. This archive contains 128 decoding files, 65 training files, and 128 .ccf files. The same .ccf file is stored inside trainingFiles/ and duplicated in the task root directory for convenience. ### Usage notes: how to open and view the files #### Opening .pkl files (Python) All .pkl files were generated in Python and can be loaded using: ``` import pickle with open("filename.pkl", "rb") as f: obj = pickle.load(f) ``` The loaded object is typically: * a **list of trials**, or * a **dictionary** containing per-trial arrays and metadata (offline analysis). If you prefer a safer/modern loader, **joblib** may also work depending on how the file was saved, but **pickle** is the default. #### Viewing .ccf files .ccf files are XML-based channel configuration files exported by the Blackrock/Cerebus acquisition system. They contain per-channel metadata used during that recording session, including electrode labels, acquisition settings, and spike detection threshold levels. They can be: * opened in any text editor (e.g., VS Code, Notepad++, etc.), or * parsed programmatically using an XML parser in Python (e.g., xml.etree.ElementTree). Each .ccf file includes: * Electrode/channel labels (e.g., elec1) * Channel and bank identifiers * Analog-to-digital scaling parameters (units typically µV) * Hardware filter settings * Spike detection threshold values (stored as integer level codes in the XML) Key fields relevant for this dataset are described below under “Variables”. ### Variables and data dictionaries (with units) #### A) Training and decoding *_trials.pkl files Each *_trials.pkl file is a 2-tuple: * session_config (dict): task/session parameters (e.g., task, threeDimensions, targetWindowRadius, timeToStayInTargetWindow, dataDirectory, isTraining, …) * trials *(list of dicts)*: one dictionary per trial Below describes the per-trial dictionary keys (exact presence depends on task). ##### Session configuration (session_config) Per-session task parameters exported from Unity. Keys may vary by task/version. * task (str): task variant (e.g., movingCamera, fixedCamera). "movingCamera" is for the Continuous Navigation and Continuous Navigation (first person perspective) task, the rest is labeled "fixedCamera". * threeDimensions (bool): if True, movement/control is 3D; if False, movement is constrained (typically planar) even if Unity stores 3D coordinates. * targetWindowRadius (float, Unity units): radius of the target acceptance region used for success detection. * timeToStayInTargetWindow (int, ms): dwell time required inside the target window to count as target acquisition. * curvedPath (bool): whether the task enforces a curved/trajectory constraint during the training (task-specific). * isTraining (bool): whether the session/trials were run in training mode (as set in Unity). Logged for reproducibility: * dataDirectory (str): acquisition path on the recording computer. * useDebugLogLevel (bool): Unity logging verbosity. ##### Per-trial dictionary ###### 1) Trial metadata * trial (int): trial index. * start (float, ms): trial start time relative to the start of the recording file. * stop (float, ms): trial end time relative to the start of the recording file. * answer (int): trial outcome code. * 1 = correct * 3 = incorrect: target went off-screen * 5 = incorrect: target not reached within max trial time * 6= incorrect: sphere/avatar went off-screen * other values = aborted trials ###### 2) Target / task state * targetPosition (np.array, shape (3,), Unity units): target position (x, y, z) in the local task coordinate frame, expressed relative to the avatar’s start position at (0, 0, 0) at the beginning of the trial. * targetOnset (float, ms): time the target appeared. * photoEvents (np.array, ms): timestamps when the target was visible on screen (photodiode / screen-sync events). Task-dependent fields: * targetJumpPosition (np.array, shape (3,), Unity units): respawn/jump target position (Respawn task only; otherwise typically [nan, nan, nan]). * unityTargetJumpTime (float, ms; often nan): target jump/respawn time recorded from Unity (Respawn task only). * unityTargetPosition (np.array of dict, dtype=object; Unity units): target position as logged by Unity in absolute world coordinates (often a 1-element array containing {'x','y','z'}). * obstaclePosition (np.array, shape (3,), Unity units): obstacle position (Obstacle task only; otherwise typically [nan, nan, nan]). ###### 3) Behavior * avatarTrajectory (dict): sphere/avatar position time series * time (np.array, ms) * x, y, z (np.array, Unity units) in the same local coordinate frame as targetPosition * may include avatarRotation (np.array, shape (T,), degrees): avatar yaw angle around the vertical axis (Unity Euler Y), sampled at the same timestamps as x,y,z. * avatarVelocity (dict): sphere/avatar velocity time series * time (np.array, ms) * vx, vy, vz (np.array, Unity units/s) ###### 4) Neural data * muaA *(dict)*: multiunit threshold crossings per electrode/channel * keys like elec1, elec2, … * each value is an array of spike timestamps *(ms, relative to recording start)* * contA*(dict)*: local field potential (LFP) signal per electrode. * each channel stores: * samplingRate *(float, Hz; typically 1000.0)* * data*(np.array,* µV*): *continuous LFP signal Optional / task- or session-dependent: * muaB, contB: same structure as above for a second bank/array. ###### Units note (Unity) Unity uses arbitrary Unity units for spatial quantities. In this dataset: * positions are stored in Unity units * velocities are stored in Unity units/s * timestamps in trial dicts and behavior time series are in ms, all relative to the start of the recording file. All timestamps in *_trials.pkl (trial events, behavior samples, and neural event times) are expressed in milliseconds relative to the start of the recording file, enabling direct alignment. Continuous signals (LFP, EMG, eye) are aligned to trials using their samplingRate and the trial start/stop times. Spike times and event timestamps are stored in milliseconds relative to recording start. While some logged Unity fields (e.g., unityTargetPosition) are spatial metadata rather than time-synchronized traces. #### B) .ccf files (spike threshold configuration) Files are XML-based Blackrock Cerebus Channel Configuration Files (CCF, Version 12) containing per-channel acquisition parameters used during that recording session. Each file includes: * Electrode/channel labels (e.g., elec1) * Channel ID and bank information * Analog-to-digital scaling parameters (including units, typically µV) * Hardware filter settings * Spike detection threshold levels * Sampling and display parameters Spike thresholds are stored in the acquisition system’s digital units, together with scaling parameters that define their relationship to physical units (µV). These files can be opened with any text editor. #### C) sEMG data *_emg.pkl Each file contains a list of trials, each trial holding a dictionary: * data (np.array, shape (T,), µV): EMG voltage over time (microvolts) * sample rate(float, Hz): sampling rate  EMG channels correspond to biceps and thumb placements as described in the related manuscript. EMG recordings are available for a subset of decoding sessions (60 files). #### D) Eye data *eyeMovementX.pkl and *eyeMovementY.pkl Each file contains a list of trials, each trial holding a dictionary: * data (np.array, shape (T,), volts): raw eye position voltage from the acquisition system. * X coordinate in \_eyeMovementX  * Y coordinate in \_eyeMovementY  * sampling_rate(float, Hz): typically 10000. 122 pairs for each movement files (X and Y) are available. #### E) Offline analysis (offlineAnalysisPerArea/) ##### Decoding results **Where it lives:**\ offlineAnalysisPerArea///decoding_results.pkl with one of: All, M1, PMd, PMv, M1+PMd, M1+PMv, PMv+PMd, and  Online_simulation. **Availability:**\ Offline analysis folders are present only for: * Monkey 1 and Monkey 3 * Center-out and Continuous Navigation tasks For these sessions, one decoding_results.pkl file is provided **per electrode set** for each corresponding decoding session. In total, 360 offline decoding files are included. **What it is:**\ decoding_results.pkl contains the output of the offline decoding pipeline: offline-predicted velocity commands for each online trial, using a model trained with the electrode subset corresponding to that folder (e.g., M1-only). Each folder (e.g., M1/, PMd/, All/) contains predictions generated using only that specific electrode subset. ##### File format (Python dict) Top-level keys: * trial_answers *(list of int; length = #trials)*\ Outcome codes for the corresponding trials (same convention as in *_trials.pkl, e.g. 1=correct, 5=timeout, etc.). * allPredictions *(list; length = #trials)*\ Offline velocity predictions aligned to each trial.\ allPredictions[i] is a list over time bins for trial i. Each time bin is a dict: * x : np.array shape (1,) — predicted velocity along Unity X (Unity units/s) * y : np.array shape (1,) — predicted velocity along Unity Y (Unity units/s) * z : np.array shape (1,) — predicted velocity along Unity Z (Unity units/s) * Notes: In 2D tasks, y is often always zero. * The number of time bins per trial depends on trial duration and the binning used in the offline pipeline (typically the same bin size as decoding, e.g. 50 ms). * allRewards *(list; often empty)*\ Legacy field; usually unused in this dataset. ##### Electrode selection metadata **Where it lives:** offlineAnalysisPerArea/electrode_selection.pkl **What it is** Dictionary mapping each decoding session to the electrode subsets used for area-specific offline decoding. It specifies which electrode labels belong to: M1, PMd, PMv for each session. This file defines the electrode grouping used to generate the All, M1, PMd, PMv, and combined electrode-set folders. **File format (Python dict**) Top-level keys are decoding session identifiers matching decoding filenames without "_trials.pkl". Example structure: {\ "navdecodingsphere_Loki_20230825_1023_A": {\ "M1": ["A_elec67", "A_elec139", ...],\ "PMd": ["A_elec215", "A_elec218", ...],\ "PMv": ["A_elec15", "A_elec17", ...]\ },\ ...\ } For each session: * Keys: "M1", "PMd", "PMv" * Values: list of electrode labels used for that area (typically 27 electrodes per area) The All condition corresponds to the union of M1, PMd, and PMv for that session. Electrode assignments are session-specific and may vary slightly across dates. A total of four electrode_selection.pkl files are included, one per Monkey/Task combination for which offline analysis is provided ## Missing data, abbreviations, and notes * Missing data codes: Missing values are typically stored as NaN (e.g., fields not applicable to a task). * Some modalities are session-dependent: * EMG and eye tracking are not available for all animals/sessions. * Abbreviations: * mua= multiunit activity (threshold crossings) * fpp = first-person perspective ## Code/Software * Data files were generated and analyzed using Python. * To load .pkl files, use Python’s pickle module (example above). * Unity tasks were implemented in Unity 2021.3.x (used during data collection). * Related code to reproduce offline analyses and figures: Zenodo [https://zenodo.org/records/15791187](https://zenodo.org/records/15791187) ## Access information Other publicly accessible locations of the data: * None Data was derived from the following sources: * N/A