DOI: 10.5061/dryad.cvdncjt7n
title: Data from: Using adversarial networks to extend brain computer interface decoding accuracy over time

description: Existing intracortical brain computer interfaces (iBCIs) transform neural
 activity into control signals capable of restoring movement to persons
 with paralysis. However, the accuracy of the “decoder” at the heart of the
 iBCI typically degrades over time due to turnover of recorded neurons. To
 compensate, decoders can be recalibrated, but this requires the user to
 spend extra time and effort to provide the necessary data, then learn the
 new dynamics. As the recorded neurons change, one can think of the
 underlying movement intent signal being expressed in changing coordinates.
 If a mapping can be computed between the different coordinate systems, it
 may be possible to stabilize the original decoder’s mapping from brain to
 behavior without recalibration. We previously proposed a method based on
 Generalized Adversarial Networks (GANs), called “Adversarial Domain
 Adaptation Network” (ADAN), which aligns the distributions of latent
 signals within underlying low-dimensional neural manifolds. However, we
 tested ADAN on only a very limited dataset. Here we propose a method based
 on Cycle-Consistent Adversarial Networks (Cycle-GAN), which aligns the
 distributions of the full-dimensional neural recordings. We tested both
 Cycle-GAN and ADAN on data from multiple monkeys and behaviors and
 compared them to a third, quite different method based on Procrustes
 alignment of axes provided by factor analysis. All three methods are
 unsupervised and require little data, making them practical in real life.
 Overall, Cycle-GAN had the best performance and was easier to train and
 more robust than ADAN, making it ideal for stabilizing iBCI systems over
 time.
description: <strong>Electrophysiology</strong>:  
 Depending on the task, we implanted a 96-channel Utah electrode
 array (Blackrock Neurotech, Inc.) in either the hand or arm representation
 area of the primary motor cortex (M1), contralateral to the arm being used
 for the task. The implant site was pre-planned and finally determined
 during the surgery with reference to the sulcal patterns and the muscle
 contractions evoked by intraoperative surface cortical stimulation. For
 each of monkeys J, S, G, and P, we also implanted intramuscular leads in
 forearm and hand muscles of the arm used for the task in a separate
 procedure. Electrode locations were verified during surgery by stimulating
 each lead. <strong>Behavioral
 task</strong>: Monkeys J and S were trained to
 perform an isometric wrist task, which required them to control the cursor
 on the screen by exerting forces on a small box placed around one of the
 hands. The box was padded to comfortably constrain the monkey’s hand and
 minimize its movement within the box, and the forces were measured by a 6
 DOF load cell (JR3 Inc., CA) aligned to the wrist joint. During the task,
 flexion/extension force moved the cursor right and left respectively,
 while force along the radial/ulnar deviation axis moved the cursor up and
 down. Each trial started with the appearance of a center target requiring
 the monkeys to hold for a random time (0.2 – 1.0 s), after which one of
 eight possible outer targets selected in a block-randomized fashion
 appeared, accompanied with an auditory go cue. The monkey was allowed to
 move the cursor to the target within 2.0 s and hold for 0.8 s to receive a
 liquid reward. For both decoding and alignment analyses, we only used the
 data within each single trial (from ‘trial start’ to ‘trial end’). We did
 not do any temporal alignment with the trials, so the lengths of the
 trials were different from each other. <br> <br>Monkeys P and
 G were trained to perform a grasping task, which required them to reach
 and grasp a gadget placed under the screen with one hand. The gadget was a
 cylinder for monkey P facilitating a power grasp with the palm and the
 fingers, while a small rectangular cuboid for monkey G facilitating a key
 grasp with the thumb and the index finger. A pair of force sensitive
 resistors (FSRs) were attached on the sides of the gadgets to measure the
 grasping forces the monkeys applied. The sum and the difference of the FSR
 outputs were used to determine the position of the cursor on the vertical
 axis and the horizontal axis respectively. At the beginning of each trial
 the monkey was required to keep the hand resting on a touch pad for a
 random time (0.5 – 1.0 s). A successful holding triggered the onset of one
 of three possible rectangular targets on the screen and an auditory go
 cue. The monkey was required to place the cursor into the target and hold
 for 0.6 s by increasing and maintaining the grasping force applied on the
 gadget. For this task we extracted trials from ‘gocue time’ to ‘trial
 end’, as the monkeys’ movements were quite random before the gocue.
   Monkeys C and M were trained to perform a center-out
 (CO) reaching task while grasping the upright handle of a planar
 manipulandum, operated with the upper arm in a parasagittal plane. Monkey
 C performed the task with the right hand, monkey M with the left. At the
 beginning of each trial the monkey needed to move the hand to the center
 of the workspace. One of eight possible outer targets equally spaced in a
 circle was presented to the monkey after a random waiting period. The
 monkey needed to keep holding for a variable delay period until receiving
 an auditory go cue. To receive a liquid reward, the monkey was required to
 reach the outer target within 1.0 s and hold within the target for 0.5 s.
 For this task we extracted trials from ‘gocue time’ to ‘trial end’, since
 the monkeys kept static before the gocue.  Monkey M was
 also trained to perform a random-target (RT) task, reaching a sequence of
 three targets presented in random locations on the screen to complete a
 single trial. The RT task used the same apparatus as the CO reach task. At
 the beginning of each trial the monkey also needed to move the hand to the
 center of the workspace. Three targets were then presented to the monkey
 sequentially, and the monkey was required to move the cursor into each of
 them within 2.0 s after viewing each target. The positions of these
 targets were randomly selected, thus the cursor trajectory for each trial
 presented a ‘random-target’ manner. For this task we extracted trials from
 ‘trial start’ to ‘trial end’. All surgical and
 experimental procedures were approved by the Institutional Animal Care and
 Use Committee (IACUC) of Northwestern University under protocol
 #IS00000367, and are consistent with the Guide for the Care and Use of
 Laboratory Animals. <strong>Data collection and
 preprocessing</strong>:   M1 activity was
 recorded during task performance using a Cerebus system (Blackrock
 Neurotech, Inc.). The signals on each channel were digitalized, bandpass
 filtered (250 ~ 5000 Hz) and converted to spike times based on threshold
 crossings. The threshold was set with respect to the root-mean square
 (RMS) activity on each channel and kept consistent across different
 recording sessions (monkeys J, C and M: -5.5 x RMS; monkey S: -6.25 x RMS;
 monkey P: -4.75 x RMS; monkey G: -5.25 x RMS). The time stamp and a 1.6 ms
 snippet of each spike surrounding the time of threshold crossing were
 recorded. For all analyses in this study, we used multiunit threshold
 crossings on each channel instead of discriminating well isolated single
 units. We applied a Gaussian kernel (S.D. = 100 ms) to the spike counts in
 50 ms, non-overlapping bins to obtain a smoothed estimate of firing rate
 as function of time for each channel.  The EMG signals
 were differentially amplified, band-pass filtered (4-pole, 50 ~ 500 Hz)
 and sampled at 2000 Hz. The EMGs were subsequently digitally rectified and
 low-pass filtered (4-pole, 10 Hz, Butterworth) and subsampled to 20 Hz.
 EMG channels with substantial noise were not included in the analyses. For
 monkeys C and M, we recorded the positions of the endpoint of the reach
 manipulandum at a sampling frequency of 1000 Hz using encoders in the two
 joints of the manipulandum.
description: This data set includes behavioral recordings and extracellular
 neural recordings from the primary motor cortex of six rhesus macaque.
 Christian Ethier, Stephanie N. Thacker, Matthew G. Perich and Xuan Ma
 collected these data in the laboratory of Lee E. Miller in Northwestern
 University. Xuan Ma and Fabio Rizzoglio processed the data for use in [1],
 which utilized adversarial neural networks to "align" the neural
 signals over a long period of time to extend the accuracy of an iBCI
 decoder calibrated only on the very first day (day-0).
 <strong>Please cite this article if you are using this
 dataset:</strong> <strong>[1] Ma X,
 Rizzoglio F, Perreault EJ, Miller LE, Kennedy A. Using adversarial
 networks to extend brain computer interface decoding accuracy over time.
 eLife 2023;12:e84296. DOI: https://doi.org/10.7554/eLife.84296</strong> This dataset contains data files from 6 monkeys and 4 types of behavioral tasks (see Methods). The data files from one monkey/task were compressed into a `.7z` file, which could be opened and extracted by '7-Zip'. The `.7z` files are named like 'Name_Task_XXXX.7z', where 'Name' is the full name of the monkey ('Jango': monkey J; 'Spike': monkey S; 'Greyson': monkey G; 'Pop': monkey P; 'Chewie': monkey C; 'Mihili': monkey M), 'Task' indicates the behavioral task ('ISO': isometric wrist task; 'PG': power grasp task; 'Key': key grasp task; 'CO': center-out reach task; 'RT': random-target reach task), and 'XXXX' indicates the year when the data files were recorded.  Data files could be extraxted from the `.7z` files once they are downloaded. Please check the readme.md file along with this data set and also this document to see more details about them.
description: # Title of Dataset:"Using adversarial networks to extend brain
 computer interface decoding accuracy over time" Xuan Ma, Fabio
 Rizzoglio, Stephanie N. Thacker, Christian Ethier, Matthew G. Perich, Lee
 E. Miller Northwestern University Correspondence Email:
 xuan.ma1@northwestern.edu This data set includes behavioral recordings and
 extracellular neural recordings from the primary motor cortex of six
 rhesus macaque. Christian Ethier, Stephanie N. Thacker, Matthew G. Perich
 and Xuan Ma collected these data in the laboratory of Lee E. Miller in
 Northwestern University. Xuan Ma and Fabio Rizzoglio processed the data
 for use in [1], which utilized adversarial neural networks to
 "align" the neural signals over a long period of time to extend
 the accuracy of an iBCI decoder calibrated only on the very first day
 (day-0). Please cite this article if you are using this dataset: [1] Ma X,
 Rizzoglio F, Perreault EJ, Miller LE, Kennedy A. Using adversarial
 networks to extend brain computer interface decoding accuracy over time.
 eLife 2023;12:e84296. DOI: https://doi.org/10.7554/eLife.84296 ##
 Description of methods **Electrophysiology**: Depending on the task, we
 implanted a 96-channel Utah electrode array (Blackrock Neurotech, Inc.) in
 either the hand or arm representation area of the primary motor cortex
 (M1), contralateral to the arm being used for the task. The implant site
 was pre-planned and finally determined during the surgery with reference
 to the sulcal patterns and the muscle contractions evoked by
 intraoperative surface cortical stimulation. For each of monkeys J, S, G,
 and P, we also implanted intramuscular leads in forearm and hand muscles
 of the arm used for the task in a separate procedure. Electrode locations
 were verified during surgery by stimulating each lead. **Behavioral
 task**: Monkeys J and S were trained to perform an isometric wrist task,
 which required them to control the cursor on the screen by exerting forces
 on a small box placed around one of the hands. The box was padded to
 comfortably constrain the monkey’s hand and minimize its movement within
 the box, and the forces were measured by a 6 DOF load cell (JR3 Inc., CA)
 aligned to the wrist joint. During the task, flexion/extension force moved
 the cursor right and left respectively, while force along the radial/ulnar
 deviation axis moved the cursor up and down. Each trial started with the
 appearance of a center target requiring the monkeys to hold for a random
 time (0.2 – 1.0 s), after which one of eight possible outer targets
 selected in a block-randomized fashion appeared, accompanied with an
 auditory go cue. The monkey was allowed to move the cursor to the target
 within 2.0 s and hold for 0.8 s to receive a liquid reward. For both
 decoding and alignment analyses, we only used the data within each single
 trial (from ‘trial start’ to ‘trial end’). We did not do any temporal
 alignment with the trials, so the lengths of the trials were different
 from each other. Monkeys P and G were trained to perform a grasping task,
 which required them to reach and grasp a gadget placed under the screen
 with one hand. The gadget was a cylinder for monkey P facilitating a power
 grasp with the palm and the fingers, while a small rectangular cuboid for
 monkey G facilitating a key grasp with the thumb and the index finger. A
 pair of force sensitive resistors (FSRs) were attached on the sides of the
 gadgets to measure the grasping forces the monkeys applied. The sum and
 the difference of the FSR outputs were used to determine the position of
 the cursor on the vertical axis and the horizontal axis respectively. At
 the beginning of each trial the monkey was required to keep the hand
 resting on a touch pad for a random time (0.5 – 1.0 s). A successful
 holding triggered the onset of one of three possible rectangular targets
 on the screen and an auditory go cue. The monkey was required to place the
 cursor into the target and hold for 0.6 s by increasing and maintaining
 the grasping force applied on the gadget. For this task we extracted
 trials from ‘gocue time’ to ‘trial end’, as the monkeys’ movements were
 quite random before the gocue. Monkeys C and M were trained to perform a
 center-out (CO) reaching task while grasping the upright handle of a
 planar manipulandum, operated with the upper arm in a parasagittal plane.
 Monkey C performed the task with the right hand, monkey M with the left.
 At the beginning of each trial the monkey needed to move the hand to the
 center of the workspace. One of eight possible outer targets equally
 spaced in a circle was presented to the monkey after a random waiting
 period. The monkey needed to keep holding for a variable delay period
 until receiving an auditory go cue. To receive a liquid reward, the monkey
 was required to reach the outer target within 1.0 s and hold within the
 target for 0.5 s. For this task we extracted trials from ‘gocue time’ to
 ‘trial end’, since the monkeys kept static before the gocue. Monkey M was
 also trained to perform a random-target (RT) task, reaching a sequence of
 three targets presented in random locations on the screen to complete a
 single trial. The RT task used the same apparatus as the CO reach task. At
 the beginning of each trial the monkey also needed to move the hand to the
 center of the workspace. Three targets were then presented to the monkey
 sequentially, and the monkey was required to move the cursor into each of
 them within 2.0 s after viewing each target. The positions of these
 targets were randomly selected, thus the cursor trajectory for each trial
 presented a ‘random-target’ manner. For this task we extracted trials from
 ‘trial start’ to ‘trial end’. All surgical and experimental procedures
 were approved by the Institutional Animal Care and Use Committee (IACUC)
 of Northwestern University under protocol #IS00000367, and are consistent
 with the Guide for the Care and Use of Laboratory Animals. **Data
 collection and preprocessing**: M1 activity was recorded during task
 performance using a Cerebus system (Blackrock Neurotech, Inc.). The
 signals on each channel were digitalized, bandpass filtered (250 \~ 5000
 Hz) and converted to spike times based on threshold crossings. The
 threshold was set with respect to the root-mean square (RMS) activity on
 each channel and kept consistent across different recording sessions
 (monkeys J, C and M: -5.5 x RMS; monkey S: -6.25 x RMS; monkey P: -4.75 x
 RMS; monkey G: -5.25 x RMS). The time stamp and a 1.6 ms snippet of each
 spike surrounding the time of threshold crossing were recorded. For all
 analyses in this study, we used multiunit threshold crossings on each
 channel instead of discriminating well isolated single units. We applied a
 Gaussian kernel (S.D. = 100 ms) to the spike counts in 50 ms,
 non-overlapping bins to obtain a smoothed estimate of firing rate as
 function of time for each channel. The EMG signals were differentially
 amplified, band-pass filtered (4-pole, 50 \~ 500 Hz) and sampled at 2000
 Hz. The EMGs were subsequently digitally rectified and low-pass filtered
 (4-pole, 10 Hz, Butterworth) and subsampled to 20 Hz. EMG channels with
 substantial noise were not included in the analyses. For monkeys C and M,
 we recorded the positions of the endpoint of the reach manipulandum at a
 sampling frequency of 1000 Hz using encoders in the two joints of the
 manipulandum. ## Description of data and file structure **Summary of files
 and naming conventions**: This dataset contains data files from 6 monkeys
 and 4 types of behavioral tasks (see above). The data files from one
 monkey/task were compressed into a `.7z` file, which could be opened and
 extracted by ['7-Zip'](https://www.7-zip.org/). The `.7z` files
 are named like 'Name\_Task\_XXXX.7z', where 'Name' is
 the full name of the monkey ('Jango': monkey J;
 'Spike': monkey S; 'Greyson': monkey G;
 'Pop': monkey P; 'Chewie': monkey C;
 'Mihili': monkey M), 'Task' indicates the behavioral
 task ('ISO': isometric wrist task; 'PG': power grasp
 task; 'Key': key grasp task; 'CO': center-out reach
 task; 'RT': random-target reach task), and 'XXXX'
 indicates the year when the data files were recorded. Data files could be
 extraxted from the `.7z` files once they are downloaded. Please check
 [this
 document](https://cdn.elifesciences.org/articles/84296/elife-84296-fig2-data1-v1.docx) to see more details about them. For monkeys J, S, G, C and M, neural and behavioral data are organized in a data structure called `xds` (cross-platform data structure) and saved in MATLAB '.mat' format. The naming convention is like 'Name\_YYYYMMDD\_XXX.mat', where 'Name' is the name of the monkey, 'YYYYMMDD' shows the date when the recording session was made, and 'XXX' is the serial number of the recording session on a day (like '001'). These files could be directly opened in MATLAB, so you can develop your own MATLAB codes to analyze them. At the same time, we provide \*\*Python codes ('xds', https://github.com/limblab/xds) \*\*to load these `.mat` files in Python (h5py and scipy are needed). **A tutorial for `xds`** can be found [here](https://github.com/limblab/adversarial_BCI/blob/main/xds_tutorial.ipynb). In summary, when analyzing these data, there is no need to write codes from scratch by yourself using `h5py` or `scipy` to load the data files , since `xds` has already done it for you. Specifically, the `xds` where all data are encapsulated is a MATLAB struct with 20+ fields. These fields are consistent in MATLAB and Python. Some fields may not be present in some datasets because the corresponding types of data were not recorded or unavailable. Details are explained below. **'meta'** is a struct containing the essential information of the recording session. **'spikes'** is a 1 by N cell array containing the spike timings on each of the `N` cortical electrodes. Normally, `N` should be 96 because there are 96 electrodes on an Utah array. However, an electrode may be skipped in file writing if zero spike is detected on it in a recording session. Therefore, `N` is less than 96 in some data files. We've demonstrated how to do zero padding for the convinience of data analysis in [this notebook](https://github.com/limblab/adversarial_BCI/blob/main/Cycle_GAN_aligner.ipynb). **'spike\_counts'** is a T by N array for the binned spike counts for the N electrodes, where `T` is the total number of bins determined by the length of the recording session. Apparently, 'spike\_counts' is computed from 'spikes'. The default bin width is 1 ms. **'bin\_width'** gives the default bin width for the data files, which is 1 ms. **'spike\_waveforms'** is a 1 by N cell array containing the waveforms of detected spikes. Each element in the cell array corresponds to a cortical electrode. Each spike waveform has 48 data samples. In some datasets (Chewie\_CO\_2016 and Mihili\_CO\_2014) the spike waveforms are not provided so this field is not present in the data files. **'unit\_names'** are actually the names of the electrodes, since no spike sorting has been done to identify single units for this data set. The ordering of 'unit\_names' corresponds to the columns of 'spike\_counts', as well as the elements of 'spikes' and 'spike\_waveforms' **'EMG'** contains the EMG signals. They have been resampled to 1000 Hz following the 1 ms default bin width and synchronized with 'spike\_counts'. Please check the codes for the processings steps in our data analysis [here](https://github.com/limblab/adversarial_BCI/blob/187857d4963dcffbdbf633502b1e41dafa4cd09a/utils.py#L14). Specifically, the data points from each EMG channel were clipped to be no larger than the mean plus 6 times the S.D. of that channel. Within each session, we removed the baseline of each EMG channel by subtracting the 2nd percentile of the amplitudes and normalized each channel to the 90th percentile. But **please note in the data files you download the EMGs were not processed, no clipping, no normalizing and no baseline removal**. **'EMG\_names'** give the names of the muscles where EMGs were recorded from, corresponding to the columns of 'EMG'. **'force'** contains the force signals measured by the JR3 load cell when monkey performing the tasks. It is a 2-D array for x-axis and y-axis respectively, and has been resampled to 1000 Hz and also synchronized with 'spike\_counts'. **'curs\_p', 'curs\_v' and 'curs\_a'** are the **p**ositions, **v**elocities, and **a**ccelerations of the cursors on the screen controled by the monkey through the device during the task, also resampled to 1000 Hz and synchronized with 'spike\_counts'. **'time\_frame'** gives the time frame of data samples. It has the same length as 'spike\_counts', 'EMG', 'force' and cursors, following the 'bin\_width'. **'trial\_info\_table' and 'trial\_info\_table\_header'** contain a collection of all trial related information. Since we extracted important items and made them as stand-alone fields in the data structure, it is not necessary to visit these two fields. **'trial\_start\_time', 'trial\_gocue\_time', 'trial\_end\_time', 'trial\_result'** provide trial informations and the names are quite self-explanatory. The unit for time is second. **'trial\_target\_dir'** gives the direction of the target on the screen for each trial in degrees. **'trial\_target\_corners'** provides the position of the target on the screen for each trial. **'has\_EMG', 'has\_cursor', 'has\_force'** use values **0** or **1** to indicate if the corresponding type of data is present in a data file. In 'Greyson\_Key\_2019' cursor data is not available during recordings so `has_cursor` is **0** and fields `curs_p`, `curs_v`, `curs_a` are absent. For 'Mihili\_CO\_2014', 'Mihili\_RT\_2013\_2014', and 'Chewie\_CO\_2016', both `has_force` and `has_EMG` are 0 and fields `EMG`, `EMG_names`, `force` are absent since forces and EMGs were not recorded at all. The timings for force onset were not provided directly in the data structure, but can be computed using the functions we provided in `xds`, please check [the tutorial for `xds`](https://github.com/limblab/adversarial_BCI/blob/main/xds_tutorial.ipynb) for more details. For monkey P, only trial-splitted data were provided. The data files are saved in `.pkl` format, and can be loaded in Python using [`pickle`](https://docs.python.org/3/library/pickle.html). The data files are named as 'Pop\_YYYYMMDD\_pg.pkl', where 'Pop' is the name of the monkey, 'YYYYMMDD' gives the date when the data file was recorded, and 'pg' indicates the behavioral task is power grasp. Please use the codes in 'load\_monkey\_pop\_data.py' compressed with monkey P data files to load them. Specifically, when using those codes to load one data file, three lists could be obtained, which are `spike_counts`, `EMG`, and `force`. Each element in the lists is a `numpy` array corresponding to a trial. Please note the EMG data in Pop\_PG\_2021 were processed. The data points from each EMG channel were clipped to be no larger than the mean plus 6 times the S.D. of that channel. Within each session, we removed the baseline of each EMG channel by subtracting the 2nd percentile of the amplitudes and normalized each channel to the 90th percentile. Besides the paper published with eLife, we also provide the codes of our method to align the neural signals for BCI stabilization, which can be found [here](https://github.com/limblab/adversarial_BCI).