DOI: 10.5061/dryad.gtht76hhk
title: Dynamic stimulation of visual cortex produces form vision in sighted and blind humans

description: A visual cortical prosthesis(VCP) has long been proposed as astrategy for
 restoringuseful vision to the blind, under the assumptionthat visual
 percepts of small spots of lightproduced withelectrical stimulation of
 visual cortex (phosphenes) will combine into coherent percepts of visual
 forms, like pixels on a video screen.We tested analternative strategy in
 which shapes were traced on the surface of visual cortex by stimulating
 electrodes in dynamic sequence. In both sighted and blind participants,
 dynamic stimulation enabled accurate recognition of letter shapespredicted
 by the brain’s spatial map ofthe visual world.Forms could be presented and
 recognized rapidlyby blind participants, up to 86 forms per minute.These
 findings demonstrate that a brainprosthetic can produce coherent
 perceptsof visualforms.
description: We investigated the use of new electrical stimulation paradigms
 to produce form perception in sighted and blind subjects. Traditional
 approaches to electrical stimulation of visual cortex have been restricted
 both due to the small number of electrodes implanted, and the limited way
 in which groups of electrodes were activated simultaneously in an attempt
 to produce form vision. In an attempt to overcome these restrictions and
 produce more effective form vision, we used two new techniques: 1) dynamic
 stimulation, in which a series of electrodes defining a form are rapidly
 activated in sequence rather than simultaneously, and 2) current steering,
 in which two electrodes located nearby are activated simultaneously at
 various current ratios to produce ‘virtual electrodes’ in between the
 physical electrodes. Sighted subjects were epilepsy
 patients undergoing invasive monitoring in the epilepsy monitoring unit
 (EMU). In those subjects, we recorded receptive fields for each electrode,
 and screened each electrode to see which electrodes produced phosphenes
 when electrical stimulation was delivered. For each electrode that
 produced a phosphene, we determined the approximate threshold current
 required to produce a phosphene, and current values slightly above
 threshold were used when investigating dynamic pattern stimulation. Once
 electrodes producing phosphenes and threshold currents were established we
 picked various sets of electrodes to stimulate in dynamic stimulation
 patterns in an attempt to convey particular forms to the subject. We had
 the subjects give a verbal report of what they saw and make drawings on a
 touchscreen placed in front of them. In some subjects, we additionally had
 them perform alternative forced choice discrimination tasks in which we
 presented different dynamic stimulation sequences, corresponding to
 different perceived characters or letters, in random order, and the
 subject reported which character they perceived on each trial.
 The blind subjects used in this study were one subject (BAA) who
 was implanted with a Neuropace stimulation device with 8 electrodes over
 their occipital lobe as part of an early feasibility assessment prior to
 the Orion Visual Cortical Prosthesis Clinical Trial sponsored by Second
 Sight Medical Inc. The other subject (03281) was one of six subjects
 enrolled in the actual Orion clinical trial, and this subject had an array
 of 60 electrodes implanted over their occipital lobe. In blind subjects,
 we tested each electrode to see which produced phosphenes when stimulated
 in isolation and at what current magnitudes. We then selected dynamic
 stimulation sequences to produce the percept of lines in visual space or
 of particular forms or characters. As with sighted subjects, subjects
 verbally described what they saw and we recorded drawings they made on a
 touchscreen in response to the various stimulation sequences. In addition,
 we conducted forced choice tasks in which the subjects discriminated
 between forms. In subject BAA, we also conducted current steering
 experiments in which nearby electrodes on the cortex were stimulated
 simultaneously at various current ratios to create ‘virtual electrodes’ in
 between the two physical electrodes. Finally, in subject BAA we also
 examined how quickly we could deliver dynamic electrical stimulation
 sequences to the subject to allow rapid discrimination between various
 forms. A more detailed description of all methods is
 included in a separate file titled ‘FullMethods.pdf’.
description: Notes for usage of datasets associated with “Dynamic Stimulation
 of Visual Cortex Produces Form Vision in Sighted and Blind
 Humans” (This information is also is included
 as 'UsageNotes.pdf' in the main folder of our
 submission) <b>First: Download the zip archive
 ‘DynamicStimulationDataAndCode.zip’.</b> Expand the file, and keep
 the directory structure that results without moving files to new
 locations. Then the below notes apply to the data and code found in each
 directory. CurrentSteering03281 folder:
 Current steering experiments presented in this report were
 conducted with subject 03-281, a blind subject enrolled in the Orion
 clinical trial at the Baylor College of Medicine test site. Two types of
 basic current steering experiments were performed. In the first
 experiment, we used current steering between two physical electrodes to
 generate three virtual electrodes in between the two physical electrodes.
 In the second type of experiment, we used what we refer to as continuously
 variable dynamic current steering (CVDCS) in which the ratio of currents
 used across two electrodes was varied in small steps approximately every
 8ms which was the fastest allowed by the Orion system. There were five
 physical electrodes arranged in a row on the electrode array, and CVDCS
 was used first between the first pair of electrodes, then the second pair
 of electrodes, and so on, so as to rapidly move the locus of activation
 across the entire row of electrodes including many intervening points
 between each pair of physical electrodes. In this
 repository we include:
 <b>‘CurrentSteeringInfo.xslx’</b> This file contains
 information about the dynamic electrical stimulation sequences used for
 testing current steering between two electrodes, including the current
 ratios for each condition, and the location of the resulting
 phosphenes. <b>The Matlab file
 ‘03281_CSPairData.mat’ </b>contains the data for the experiment
 testing current steering between two electrodes. In this file, the
 ‘pair.trials’ data structure contains the electrical stimulation
 parameters, monitor settings, and drawing data for each trial. The x and y
 fields contain the original subject drawing, and xDeg and yDeg are the
 drawing data converted into degrees of visual space. The ‘segment’ field
 is used to divide the drawings into the discrete drawings in different
 locations on the screen made during single trials, since the subject was
 drawing more than one phosphene per trial on the touchscreen.
 <b>The Matlab file ‘03-281_CVDCSData.mat’</b>
 contains data from the experiment testing continuously variable current
 steering across a row of five electrodes. The ‘cvdcs.trials’ data
 structure contains the electrical stimulation parameters, monitor
 settings, and drawing data for each trial. The fields x and y contain the
 original drawings made by the subject and the xDeg and yDeg fields contain
 the drawings converted into units of degrees of visual space.
 <b>The Matlab file ‘03281_PlotCSPair.m’</b> is used
 to plot the results from the two-electrode current steering experiment.
 The code imports data from ‘03281_CSPairData.mat’ and generates a plot
 showing the average location for each of the five phosphenes the subject
 perceived in this experiment with error ellipses. Two of the phosphene
 locations correspond to the physical electrodes used, and three resulting
 from the current steering conditions. The units are azimuth and elevation
 in degrees of visual space. <b>Matlab files
 ‘errorEllipse.m’ and ‘getSegmentCenter’</b> are helper functions
 required for ‘03281_PlotCSPair’. They are used to align drawings made on
 separate trials, and to obtain error ellipse estimates for each
 point. <b>The file ‘03281_PlotCVDCS.m’</b>
 is used to plot results from the experiment with continuously variable
 current steering. This code imports data from ‘03-281_CVDCSData.mat’ and
 generates a plot of the drawing made by the subject on one trial of the
 experiment. In this experiment, the subject was asked to trace the overall
 line they perceived rather than plot the location of individual
 phosphenes, so there is just one line segment. The units are azimuth and
 elevation in degrees of visual space.
 <b>‘Figure2.pdf’</b> This is Figure 2 from our Cell
 manuscript. It shows the results from the two electrode current steering
 experiment, and the location of the electrodes on the subjects occipital
 lobe. <b>‘Figure3.pdf’ </b>This is Figure 3
 from our Cell manuscript. It shows the results from the continuously
 variable dynamic current steering experiment, and the location of the
 electrodes on the subjects occipital lobe.
 ReceptiveFieldsYBN folder: For most
 experiments shown in this report, individual phosphene locations were used
 to plan dynamic stimulation sequences. However, in sighted subject YBN,
 receptive field (RF) locations were used instead. We used receptive field
 mapping and analysis methods that we have used in before in other reports
 (Yoshor et al 2007; Bosking et al 2017). During RF mapping the subject
 performed a letter detection task at the central fixation point while
 small checkerboard stimuli were flashed briefly in various locations on
 the screen. In this repository we include:
 <b>‘SuppFigure1.pdf’</b> This is supplementary figure
 1 from our Cell manuscript. It shows sample receptive field data
 information for two of the electrodes on the electrode array used in
 subject YBN. <b>Data
 subfolder:</b>
 <b>‘YBNDatafile051_ch105.mat’ …
 ‘YBNDatafile051_ch128.mat’</b>  The data subfolder contains a series
 of files containing the raw voltage recording obtained during the RF
 mapping in subject YBN for each of the 24 mini-electrodes located over the
 medial wall of the occipital lobe. Channel 105-128 corresponds to
 mini-electrodes 1-24.
 ‘<b>YBNDatafile051_timeStamp.mat’</b> This file
 containing the timestamps for the presentation of visual stimuli and other
 events during the experiment ‘YBNDatafile051_timeStamp.mat’.
 <b>Code subfolder:</b>
 <b>Matlab analysis function “generateRF.m”</b>. This
 function is called with one of the channels 105-128 specified as an input
 argument to obtain the RF for that channel. For example:
 ‘generateRF(105)’. The remaining files are helper functions necessary for
 generateRF to execute. The code generates two figures. The first shows the
 average voltage traces across trials for each of the positions in visual
 space tested. The second shows the result of fitting a two-dimensional
 gaussian to the average root mean square signal for each position in
 visual space. The remaining files in this folder are the helper functions
 required to run ‘generateRF.m’. Dynamic electrical
 stimulation used to evoke letter percepts in sighted subjects
 We used dynamic sequences of electrical stimulation of multiple
 electrodes to attempt to produce perception of specific characters or
 letters in sighted epilepsy monitoring unit (EMU) subjects.
 In this repository we include:
 <b>CharacterDrawingYAY folder:</b>
 <b>‘SubjectYAYcurrentSteeringExample.mp4’ </b>This is
 a slightly longer version of Supplementary Movie 1 included with the Cell
 manuscript. The subject draws the pattern they perceived following a
 dynamic current steering stimulation sequence, and provides additional
 description of the percept.
 <b>‘SupplementaryMovie1.mp4’</b>  This is a slightly
 edited version of the above movie, included with Cell
 manuscript. <b>‘Figure4.pdf’</b> This is
 Figure 4 from our Cell manuscript. Panels E-H shows the location of the
 electrodes on the brain in this subject, the location of the phosphenes
 when each electrode was stimulated in isolation, the stimulation sequence
 used, and the subject drawing.
 <b>CharacterDiscriminationYBN folder:</b>
 <b>‘BehavioralDataYBN’ </b>This Excel file gives
 information from a four alternative forced choice (4AFC) discrimination
 experiment in which the subject was asked to discriminate between four
 different character patterns (V,C,N,S) based on four different dynamic
 electrical stimulation patterns. Before this experiment was run, we had
 asked to the subject to draw the pattern perceived resulting from each of
 the four stimulation patterns (this is in Figure 5 in the Cell
 manuscript). Note that during the initial drawing of characters associated
 with each sequence the subject reported one of the sequences as a “U” but
 later reported this same dynamic stimulation sequence as a “V” during the
 4AFC task. The spreadsheet shows the electrode sequence used on each
 trial, the target or expected character for that trial, the character the
 subject actually reported perceiving, and whether the trial was correct or
 wrong. <b>‘Figure4.pdf’</b> This is Figure
 4 from our Cell manuscript. Panels A-D show the location of the electrodes
 on this subject, the receptive fields for each electrode on the array, the
 stimulation sequence, and the drawings made by the subject for four
 different dynamic sequences. Dynamic electrical
 stimulation used to evoke letter percepts in blind subjects
 We used dynamic electrical stimulation to convey letters in two
 blind subjects. Subject BAA had a Neuropace device implanted, with eight
 electrodes located over her occipital cortex, as part of an early
 feasibility assessment for the Orion clinical trial sponsored by Second
 Sight Medical Inc. As previously mentioned, Subject 03-281 is a
 participant in the Orion clinical trial enrolled at the Baylor College of
 Medicine test site. In this repository we
 include: <b>CharacterDiscriminationBAA
 folder:</b> Subject BAA made drawings of seven
 different characters perceived as a result of seven different dynamic
 electrical stimulation sequences, and later performed a five alternative
 forced choice task using five of these patterns.
 <b>‘BAAindividualPhospheneDrawings.pdf’ </b>This file
 shows the location of each of the phosphenes that the subject perceived
 when electrodes were stimulated one at a time. These locations were used
 to plan the dynamic sequences used to convey various characters.
 ‘<b>BAAcharacterDrawingsOriginal.mat’ </b>This file
 contains the drawings the subject made for each of the 7 different
 electrical stimulation sequences with four trials per sequence. The
 subject identified each sequence with a particular character (backwards G,
 N, backwards R, upside down U, upside down V, W, Z). There are 28 data
 structures corresponding to the data from each trial, for each character.
 For example, ‘xy_G_T1_Original’ contains the x and y coordinates from
 trial 1 for the electrical stimulation sequence that produced the percept
 of the backwards G character.
 ‘<b>BAAcharacterDrawingsResampled.mat’ </b>This file
 contains the same drawing data, but resampled to have the same number of
 x, y coordinates (100) for each drawing. This transformation is used to
 prepare the data for the correlation and MDS analysis.
 <b>‘CorrDataBAA.mat’ </b>This file contains a matrix
 with the correlation between each possible pair of drawings. This is the
 input to the MDS analysis.
 <b>‘corr2_allPairsBAA.m’ </b>This Matlab file is the
 code used to generate the correlation matrix from the resampled
 drawings. <b>‘corr2_allPairsOneRefBAA.m’
 </b>This Matlab file is a helper function required by
 ‘corr2_allPairsBAA.m’. <b>‘MDS_analysisBAA.m’
 </b>This Matlab file is used to perform the actual MDS analysis
 using the correlation matrix. It outputs a figure showing the separation
 of the different character groups in multi-dimensional space.
 <b>‘BehavioralDataBAA5AFC.xlsx’ </b>This Excel file
 contains information from the 5AFC discrimination experiment in which the
 subject discriminated between five different characters corresponding to
 five different dynamic stimulation sequences. The characters used were
 backwards R, Z, upside down V, W, and upside down U. The spreadsheet shows
 the stimulation sequence, expected or target character, and actual
 reported character for each trial.
 <b>‘Figure5.pdf’</b> This is Figure 5 from our Cell
 manuscript. It shows the subject plotting on the touchscreen, location of
 the electrodes on her brain, location of individual phosphenes for each
 electrode stimulated in isolation, the electrical stimulation sequence
 timing, drawings made by the subject in response to seven different
 dynamic stimulation sequences, corresponding to seven different
 characters, and the results of MDS analysis on those drawings.
 <b>‘SupplementaryMovie2.mp4’</b> This is
 supplementary movie 2 included with our Cell manuscript. It shows the
 subject plotting the location of individual phosphenes, and then drawing
 the characters she perceived with certain dynamic stimulation
 sequences. CharacterDiscrimination03281
 folder: Subject 03281 performed a 4AFC task in which
 he both drew the characters he perceived and gave a verbal report of what
 character he perceived on each trial. There were four different dynamic
 stimulation sequences which he perceived as the characters M, N, U, and
 W. <b>‘M1_resampled.mat’, ‘N1_resampled.mat’,
 ‘U1_resampled.mat’, ‘W1_resampled.mat’</b> etc.  We include the
 Matlab data files for each of ten trials for each of the four dynamic
 sequences corresponding to the four different charactes the subject
 perceived (M, N, U, W). Each file contains the original x and y
 coordinates for the drawing from one trial, and the resampled coordinates
 used as input for the MDS analysis.
 <b>‘CorrData03281.mat’ </b>This file contains a
 matrix with the correlation between each possible pair of drawings. This
 is the input to the MDS analysis.
 <b>‘corr2_allPairs03281.m’ </b>This Matlab file is
 the code used to generate the correlation matrix from the resampled
 drawings. <b>‘corr2_allPairsOneRef03281.m’
 </b>This Matlab file is a helper function required by
 ‘corr2_allPairs03281.m’. <b>‘MDS_analysis03281.m’
 </b>This Matlab file is used to perform the actual MDS analysis
 using the correlation matrix. It outputs a figure showing the separation
 of the different character groups in multi-dimensional space.
 <b>‘ElectrodeInfo4AFCsubject03281.xslx’</b> This
 Excel files gives the electrodes and currents used in the 4AFC character
 discrimination experiment.
 <b>‘CharacterDiscrimBehData03281.xlsx’</b> This Excel
 file gives the behavioral data including expected or target character, and
 actual verbal report from the subject for each trial of the 4AFC
 discrimination experiment.
 <b>‘03281_4AFC_AlignLetters.m’ </b>This file contains
 the Matlab code used to align character drawings obtained on each
 trial. <b>‘removeCellOffset.m’ </b>This is
 a helper function required by ‘03281_4AFC_AlignLetters’.
 <b>‘03281_4AFC_LetterDiscrimData.mat’ </b>This Matlab
 file contains the data from each trial of the 4AFC discrimination
 experiment. The ‘data.trials’ structure contains the electrical
 stimulation parameters including the sequence of electrodes used, monitor
 settings, target character (letter and stimCharacterID), actual subject
 response (response), and drawing data (xDeg, yDeg). The stimCharacterID
 for each character used are M = 1, N =2, U = 3, W = 4.
 <b>‘Figure6.pdf’</b> This is Figure 6 from our Cell
 manuscript. It shows the subject drawing one of the characters perceived
 on the touchscreen, location of electrodes on his brain, location of the
 individual phosphenes, the timing of the stimulation sequences used, the
 subject drawings for each of the four dynamic sequences corresponding to
 four different characters, the results of the MDS analysis, and the
 percent correct for rapid direction discrimination (2AFC) and character
 discrimination (3AFC).
 <b>‘SupplementaryMovie3.mp4’</b> This is
 supplementary movie 3 from our Cell manuscript. It shows the subject
 performing the 4AFC character discrimination task.
 RapidFormDiscrimination03281 folder: In
 blind subject 03-281 we conducted two types of experiments to assess the
 ability of subjects to use dynamic sequences to rapidly discriminate
 between different forms. In one experiment, we used continuously variable
 dynamic current steering sequences to present a rapidly drawn line in
 visual space and varied whether the progression moved in the upward or
 downward direction. In the other experiment, dynamic sequences tracing
 four different characters were presented rapidly to the subject and the
 subject gave a verbal report indicating which character was perceived on
 each trial. In this repository we include:
 <b>‘03281_2AFC_DirectionStreamData.mat’ </b>This
 Matlab file contains the data from the rapid direction discrimination
 task. The ‘data.trials’ structure contains the electrical stimulation
 information, monitor settings, expected or target direction (up or down),
 and the actual verbal report from the subject. stimDirectionID 1
 corresponds to the up direction, stimDirectionID 2 correspond to the down
 direction. <b>‘03281_3AFC_LetterStreamData.mat’
 </b>This Matlab file contains the data from the rapid letter
 discrimination task. The ‘data.trials’ structure contains the electrical
 stimulation information, monitor settings, expected or target character
 (U, C, or backwards C), and the actual verbal report from the subject.
 stimCharacterID 1 corresponds to the the letter C, stimCharacterID 2
 correspond to the backwards C, and stimCharacterID 3 corresponds to the
 letter U. <b>‘RapidCharacterDiscrimBehData.xlsx’
 </b>This Excel file contains a summary of the behavioral data from
 the rapid character discrimination task including the target or expected
 character for each trial and the actual subject report.
 <b>‘RapidDirectionDiscrimBehData.xlsx’ </b>This Excel
 file contains a summary of the behavioral data from the rapid direction
 discrimination task including the target or expected character for each
 trial and the actual subject report.
 <b>‘RapidCharacterDiscrim03281.mov’ </b>This movie
 shows the subject performance during the rapid character discrimination
 task.
 <b>‘RapidCharacterDiscrim03281withOverlays.mp4’
 </b>This is the same movie but with overlays we added to show the
 target character based on the dynamic electrical stimulation sequence used
 on each trial.
 <b>‘RapidDirectionDiscrimBlock1.mov’ </b>This movie
 shows the subject performance during the first block of ten trials of the
 rapid direction discrimination task. Additional movies are included for
 the second and third block of ten trials.
 <b>‘RapidDirectionDiscrimBlock1withOVerlays.mp4’
 </b>These movies are the same as above but with overlays we added to
 show the expected or target direction for each trial.
 <b>‘SupplementaryMovie4.mp4’</b> This is
 supplementary movie 4 from our Cell manuscript. It shows the subject
 performing one block of the rapid direction discrimination 2AFC task, and
 the entire rapid character discrimination 3AFC task.