Dual-primate avatar model for motor paralysis

To test volitional motor control in awake-behaving animals without confound of intact efferent/afferent spinal pathways, we first devised a novel dual-primate motor paralysis model.  Here, two adult Rhesus monkeys (Macaca mulatta) were separately designated as either a master or an avatar on different sessions.  The monkey functioning as the master was responsible for controlling movement based on cortically recorded neural activities, and the other sedated monkey functioned as the avatar and was responsible for generating movement based on distal spinal cord and/or muscle stimulations.  Since the sedated avatar was a separate animal from the master and therefore had no physiological connection with the master, the master had no direct afferent or efferent influence on the avatar’s movement and was, therefore, fully paralysed from the functional standpoint (Fig.  1).  The two monkeys were interchangeably used as the master or the avatar on alternate sessions, meaning that we effectively tested two masters and two avatars in this study. 

Figure 1: Schematic illustration of the dual-primate set-up.  The master is displayed on top and the avatar is displayed on the bottom.  Note that on decoding-based sessions, the master had a joystick during training that was then disconnected during the real-time neural prosthetic trials.  On adaptive-based sessions, no joystick was used at any time.  Under ‘empiric mapping’, the arrows indicate the estimated intended target of movement based on neural activity and the colour codes illustrate the corresponding stimulator channel used to elicit limb movement in the animal.  The left versus right pairings indicate the two possible mappings to movement to each of the targets for the first and second avatars, respectively.  Full size image

During each session, the master was seated in a primate chair placed within a radiofrequency shielded recording enclosure.  Simultaneous multiple-unit recordings were made from the master’s premotor cortex using chronically implanted planar multielectrode arrays (NeuroNexus Technologies Inc., MI).  Signals were digitized and processed to extract action potentials in real-time by a Plexon workstation (Plexon Inc., TX).  The avatar was fully sedated (using a combination of ketamine, xylazine and atropine) and was seated in a separate enclosure.  The avatar’s limb was attached to a planar, free range-of-motion (360 degree), spring-loaded joystick that controlled a cursor displayed on the master’s screen. 

All trials during the task began with presentation of a small circular target that was positioned at two random locations on the screen.  The radius of the targets displayed to the masters was 3.75 cm, with each target occupying ~8% of the screen surface at 24 cm × 24 cm.  After presentation of a go cue, the master had to reach the displayed target by directing a cursor, from the centre of the screen to the displayed target, using stimulation-elicited limb movement in the avatar (Fig.  2).  The cursor had to occupy the target circumference for 100 ms or more in order for the master to receive reward (see Methods). 

Figure 2: Behavioural task.  Schematic illustration of the trial presentation and timeline proceeding from left to right.  On decoding-based sessions, a central green circle was used as a go cue.  On adaptive, single-neuron based sessions, no go cue was given.  Only the displayed target is shown on this particular trial (that is, the other possible target/movement locations are not shown).  During model training and adaptive sessions, the monkeys had up to 3,000 ms to make a movement.  Full size image

Decoding intended movements using neural population activity

We first used a population decoding approach that estimated the intended target of movement based on changes in the firing activity of neurons recorded within the master’s premotor cortex.  Before performing the real-time neural prosthetic experiments, the spiking activity of all premotor neurons in the masters were modelled as an inhomogeneous Poisson process in a training session in which they used the joystick to perform the task7,8,10,28.  This target-decoding approach is based on prior work by our group and others showing that multiple targets or target sequences can be accurately decoded from premotor neurons before movement (see Methods). 

We recorded a total of 125 well-isolated premotor neurons (11–20 units per session) over 10 sessions.  To predict the planned movement before its execution, we analysed the neuronal activity during the target presentation period before the go cue.  We find that, of the 125 cells recorded, 64 (51%; one-tailed Z-test, P<0.01; Fig.  3a) significantly predicted which target the animals were intending to move to.  When further examining model predictions based on the population activity at different time points, accuracy was 82±12% (mean±s.d.)  by 500 ms after target presentation and 95±10% by 1,000 ms after target presentation across sessions consisting of 854 trials.  By the time the go cue was first displayed, mean cross-validated prediction accuracy was 96±10% (one-tailed t-test, n=10; P<0.01; Fig.  3b).  Training across sessions was performed for an average of 85±19 trials (mean±s.d.)  or ~6 min before performing the real-time experiments. 

Figure 3: Neural population decoding. ( a) Averaged (grey) and model estimated (black) peristimulus histogram (top) and raster (bottom) of a premotor neuron during movement planning and aligned to presentation of two different targets (left and right, respectively). ( b) Mean population decoding performance across all trials and 95% confidence bounds aligned to target presentation (and up to the go cue) during a sample session.  Full size image

Testing stimulation-elicited limb movements

To elicit movement in the avatars, stimulating electrodes were chronically implanted in the cervical spinal cord of both monkeys.  Two 16-contact iridium oxide stimulating electrodes (100–500 kΩ) were each inserted at the C5 and C6 levels of both avatars (NeuroNexus).  In addition to spinal cord electrode implantation, percutaneous electrodes (100–200 kΩ) were placed in the long and short head of the triceps muscle of avatar #1.  This was done to provide added range of movements not available by spinal cord stimulation in that particular avatar (see Methods for further discussion and Results). 

Before running the real-time combined recording-elicitation experiments, the stimulating electrodes were tested under different amplitudes and contact locations to determine the range of target locations that can be reached.  Specifically, we identified which limb movement direction and amplitude will be produced per stimulation setting and contact location in the avatars and, therefore, to which precise target in space the cursor could reach.  Similar to prior reported stimulation experiments in anesthetized animals22,23,24,29,30,31, we tested each contact location at serially incremented amplitudes. 

In all cases, stimulation frequency was 200 Hz and pulse width was 0.2 ms, with cathodal pulse leading.  Stimulation duration was 500 ms and was selected to mimic the time it naturally took the masters to move to and hold a target during the normal joystick movement task (that is, non-prosthetic controlled).  The tested stimulation amplitudes ranged from 10–80 μA and were incremented by 10–20 μA intervals per electrode contact.  For both avatar 1 and 2, stimulations were tested across all 32 electrode contacts located within the ventral spinal cord at the C5–6 level (4-mm deep from the dorsal surface and 2-mm lateral from the median sulcus).  For avatar 1, stimulations were also tested across four acute contacts located within the short and long head of the triceps muscle.  In this case, stimulation frequency was 200 Hz and pulse width was 0.2 ms with cathodal pulse leading.  Stimulation duration was 500 ms.  The tested stimulation amplitudes ranged from 100–200 μA incremented by 10–20 μA intervals per electrode contact. 

We defined the targets of elicited movement based on the angle and amplitude of cursor displacement during the last 100 ms of stimulation.  Here, the last 100 ms of stimulation was defined as the target ‘hold-time’ and was required in order for the master to receive reward.  On average, each electrode array produced movements over a range of end target locations.  These individual clusters ranged in width from ~6 to 7 cm.  Triceps muscle stimulations were somewhat more confined, producing a cluster of end target locations 3.8-cm wide (Fig.  4a,b). 

Figure 4: Microstimulation-induced limb movement during testing. ( a) Scatter plot indicating the mean cursor position during the last 100 ms of stimulation (during which the target was held) for all tested electrode sites.  These together define the possible range of movements elicited in the two avatars during testing.  For avatar #1, blue dots indicate the end targets of movement produced by C5 electrode stimulation and red dots indicate the end targets of movement produced by triceps stimulation.  For avatar #2, green dots indicate the end targets of movement produced by C5 electrode stimulation and black dots indicate the end targets of movement produced by C6 stimulation. ( b) Distribution of elicited movement directions in relation to centre.  Full size image

Of the above tested contact locations and stimulation parameters, two were used for each session and avatar to produce stimulation-induced limb movements.  These parameters were chosen to produce movements to targets that were distinct and as radially distant from each other as possible using the available implanted electrodes (in principal, however, and as discussed further below, any other stimulation parameters and corresponding elicited limb movements could be used for the task).  The mean angle of separation between the two tested movements was 176 degrees for avatar #1 and 34 degrees for avatar #2 (Fig.  5).  For avatar #1, mean path length of movement was 10.3±0.2 cm and mean velocity was 86.4±9.0 cm s−1 (mean±s.d.).  For avatar #2, mean path length of movement was 9.4±0.1 cm and the mean velocity was 59.6±3.0 cm s−1.  Mean deviation of movement (that is, how much the movement trajectory deviated from a straight line) was 1.0 cm for avatar #1 and 3.1 cm for avatar #2. 

Figure 5: Selected targets and elicited limb movement trajectories used for the real-time sessions.  Target locations and the stimulation-induced limb movement trajectories are shown for avatar #1 (left) and avatar #2 (right).  For avatar #1, upward movements (blue) were produced by C5 electrode stimulation and downward movements (red) by triceps stimulation.  For avatar #2, upward movements (green) were produced by C5 electrode stimulation and righward movements (black) by C6 stimulation.  Full size image

Real-time neural prosthetic control of limb movement

On the basis of the above testing, we could now predict the intended target of movement based on recorded premotor activity in the master and determine which spinal cord and/or muscle electrode locations and stimulation parameters elicit limb movement to the different targets positioned on the screen.  Next, we approximated changes in motor intent with movement production using the master–avatar primates in real-time on a trial-by-trial basis. 

As the two targets were displayed in random order on the screen, neuronal activity was continuously recorded from the master and was used to predict trial-by-trial changes in the masters’ intended target of movement.  Therefore, if neural activity recorded during the trial predicted that the master was intending to move to target #1, the system would activate the stimulating electrode in the avatar that was previously observed to produce limb movement to that exact target location.  Alternatively, if neuronal activity predicted that the master was intending to move towards target #2, the system would activate another electrode that produced a movement towards that target.  This way, the neural prosthesis continuously matched the master’s planned target of movement with stimulation parameters/electrode locations that elicited movement in the avatar’s limb to the same target.  Importantly, selected stimulation parameters and electrode contacts were set to produce limb movement that precisely reached and held the intended target in their 2D space (Fig.  1). 

As noted above, mean cross-validated performance during model training was 96±10% when tested across training sessions with 854 training trials (Fig.  2b).  We find that, when the masters performed the same task as before but target selection was controlled by joystick movements made by the avatar using the neural prosthetic, performance was slightly lower but still significantly higher than chance.  Overall, the primates reached the displayed targets during these real-time recording-elicitation sessions in 84±14% (mean±standard error of the mean (s.e.m.))  of the 561 trials tested (binomial test, P<0.01).  Most incorrect trials (11%) occurred because of decoding error (that is, selecting the wrong target).  Only a few errors (5%) occurred because the avatar-controlled cursor failed to reach the spatial confines of the displayed target (that is, to maintain the cursor location within the 0.75 radians or 8% screen surface of the target for 100 ms). 

No movements were made by the master during the decoding period, before the go cue presentation (that is, the trial would abort if any movement was made).  To further confirm that no movements were made, we recorded electromyography (EMG) activity from the master as the task was being performed over one session (see Methods).  We found no difference in activity during the decoding period, before the go cue, between planned movements (t-test, P=0.53; Fig.  6). 

Figure 6: EMG tracings display robust peri-movement activity during movement to both targets. ( a) Averaged EMG tracings in millivolts over the course of a real-time recording-elicitation session.  The grey area indicates the time during which neural decoding was performed and the arrow (time zero) indicates the time of the go cue.  The thick line indicates the average activity and the thin lines the s.e.m. ( b) EMG tracings are broken down into target of movement (red for up and blue for right). ( c) Raw EMG tracings over two individual trials with the same colour convention as in b. Full size image

Motor control based on adaptive sensorimotor responses

In many circumstances, such as in the setting of full motor paralysis, it may not be possible to train models based on the subject’s natural movement.  Also decoders trained on physical movement by the subjects may not accurately model the subjects’ planned movement during direct neural prosthetic control32,33, and consequently adaptive changes in neural activity may allow for improved performance over time.  We tested this possibility by assigning individual neurons within the same premotor population (that is, the same recording electrodes) to control the avatar’s limb movement by volitionally modulating their activity.  Similar to a sensorimotor conditioning approach described previously21,34,35, we randomly selected individual neurons and assigned them to control the target of movement by naturally varying their firing activity from trial to trial as the monkeys performed the same real-time neural prosthetic task above.  In particular, depending on whether the firing activity of the assigned individual neuron went above or below a fixed firing rate threshold, movement would be elicited to one of the two targets (see Methods).  Importantly, the mapping between the neuron’s firing activity and target location was chosen randomly, and no joystick was used by the master at any time (Fig.  7a). 

Figure 7: Adaptive activity based approach for controlling movement. ( a) The firing rate and corresponding spike time indicators (immediately blow the x axis) of a single selected premotor neuron recorded over two individual trials aligned to target presentation for two different targets.  The vertical bars indicate the time during which the firing rate of the neuron reached the top threshold (left, blue) and bottom threshold (right, red; see Methods for further detail on threshold definition).  The arrows indicate the time of stimulation and mapped elicited limb movement for each trial. ( b) Behavioural performance and 99% confidence bounds over successive trials witihin a session (100% performance indicates that the primates correctly selected and spatially reached the displayed target on all trials). ( c) Example of the normalized firing activities (averaged firing rate within successive 1,000 ms windows divided by the mean firing rate across the entire session) of a selected (green) and a non-selected (purple) premotor neuron recorded over a single representative session.  Full size image

Using this approach, we recorded from 40 population cells.  During these recordings, the primates performed 580 trials over five sessions.  Starting performance by the selected cells was 37±10% (mean±s.e.m.)  over the first 10 trials.  Correct target selection, however, then rapidly improved, reaching an average performance of 77±12% (two-tailed t-test, n=5; P=0.02; Fig.  7b).  Overall, it took the monkeys 28±13 trials, or ~3 min, to reach statistical learning criterion (see Methods)36,37.  The highest noted asymptotic performance was 90%.  The mean firing rate of selected cells remained the same over the time course of these sessions when comparing the first and second half of the session (two-tailed t-test; P=0.88) as did variance in their firing rates (two-tailed t-test; P=0.92). 

During these adaptive sessions, no joystick was given to the monkeys and no visible movement was noted during the task.  To further confirm that no subthreshold movements were being made, we recorded EMG activity from the master over one session.  We found no change in EMG activity during selection of one target versus the other (area under the curve; two-tailed t-test, P=0.62) and no difference in EMG activity before versus after target presentation (windows=3,000 ms; two-tailed t-test, P=0.33; Fig.  8). 

Figure 8: EMG tracings in the master during the real-time adaptive recording-elicitation sessions. ( a) An example of raw EMG tracings recorded in the master over individual trials during selection of the top (black) versus bottom (grey) target.  In both trials, the window over which neuronal threshold was reached lied between 1,500–2,500 ms. ( b) Average EMG activity for all top versus bottom movements over the course of the session, with the same colour convention as in a. The thick lines indicate the mean EMG activity in millivolts and the thin lines indicate their s.e.m.  Full size image

Response of non-selected neurons during adaptive control

Since each selected neuron can only encode a binary response using this potential approach (that is, high versus low threshold), we wanted to determine whether other neurons not involved in controlling movement respond similarly to the intended targets.  This may, therefore, provide insight into the potential capacity of larger neural populations and distinct selected cells to adaptively direct movement to more than two targets at a time. 

By simultaneously recording from multiple neurons during these sessions, we found that neighbouring neurons in the premotor population displayed surprisingly little correspondence with the activities of the selected neurons.  Thirty five premotor cells were recorded over the same five sessions in addition to the selected cells.  However, none of the recorded cells demonstrated a significant correlation (either positive or negative) when comparing the time-varying firing rates of selected cells to the other non-selected cells (Pearson’s correlation, n=35; P>0.05).  In other words, cells that were not selected to control movement did not consistently increase their firing activity when the activity of the selected cell increased during targeted movement or vice versa (note that these single cells were selected randomly from the population and the mapping between their firing rates and target selection was chosen arbitrarily).  An example of two such cells is shown in Fig.  7c (also, note that both cells still markedly fluctuated their activities from trial to trial).  Similarly, there was little correlation in activity when considering variations in the firing activities across all pairings (mean correlation r=−0.012±0.022; mean±s.e.m.).  These findings, therefore, suggested that cells in the population not directly assigned to controlling the movement do not necessarily covary their activities with the intended target during artificial motor control.