Cognitive-based Rehabilitation Platform of Hand Grasp After Spinal Cord Injury Using Virtual Reality and Instrumented Wearables
Phase: NA
Conditions: Spinal Cord Injuries, Hand Weakness
Sponsor: VA Office of Research and Development

Cognitive-based Rehabilitation 
Platform of Hand Grasp after Spinal 
Cord Injury using Virtual Reality and 
Instrumented Wearables 
NCT04577573 
07/29/2021 
Study Protocol and Statistical Analysis Plan 

 
 
PROTOCOL 
(a) Background and Significance  
The VA provides care to 27,000 veterans with SCI-related issues each year. The most frequent neurological 
category of SCI is incomplete tetraplegia (40.8%), which involves sensory loss and motor paralysis of the 
arms [1]. For persons with cervical level SCI, restoration of hand function is the top priority [2]. Physical therapy 
at rehabilitation centers is still the most effective approach to recover significant function for this population [3]. 
Advanced modes of rehabilitation are increasingly used and include computerized interfaces (e.g., virtual 
reality (VR) [4]) and powered devices (e.g., prosthetics, exoskeletons) [5-8]. In any case, physical therapy 
requires intense participant commitment during and in between sessions. Methods that cognitively motivate 
and engage the person are critical to maximize rehabilitation outcomes. Cognitive agency refers to the 
perception of control during voluntary action [9]. Cognition of movement implies better physical function, but 
agency has not been prioritized as a primary rehabilitation goal compared to increased strength or practiced 
skill.  Incorporating cognitive-based features, such as agency, into rehabilitation platforms may be the key to 
unlock the promise of advanced rehabilitation using VR and powered devices. In this project, we propose 
rehabilitation platforms for grasping and reaching that leverage cognitive factors of (1) agency and (2) sensory-
driven learning using virtual reality and instrumented wearables.  
By the concept of intentional binding [10], agency is reinforced when one performs an action and perceives, or 
expects, sensory consequences sooner in time. A growing body of evidence indicates that greater cognitive 
engagement [4, 11] and sensory guides (visual, auditory, and haptic) [12-14] can accelerate neuromotor 
learning. Powered devices such as exoskeletons and prostheses inject mechanical energy to physically assist 
the user. But the user must cognitively integrate their own intention against observed movements for better 
performance. More ‘natural’ interfaces involve the recording of user biosignals such as muscle 
electromyography (EMG) [15] or brain electroencephalography (EEG) to command devices. We seek to create 
a rehabilitation platform for both grasp and reach that leverages cognitive agency and sensory feedback to 
improve user abilities for both independent function and potential EMG-control of powered devices. The 
expected outcome of this research is to provide accessible, cognitive-based platforms that are entertaining 
and more effective in rehabilitating hand grasp and reach for persons with cervical SCI. Within the scope of this 
SPiRE, we will establish cognitive approaches for immediate (post-training) improvement in performance of 
grasp and reach after SCI. These methods will then be adapted to train better user control of powered assistive 
devices (e.g., exoskeletons [16-18] with functional neuromuscular stimulation [6, 19-21]) for reach and grasp in 
a subsequent MERIT award.        
(b) Preliminary Studies 
Prior to 2017, our team developed and deployed feedback control systems for neuroprostheses and 
exoskeletons to assist movement function for persons with SCI [6, 16, 17, 22]. The laboratory at Stevens (est. 
2017) has been investigating the roles of control perception and sensory feedback in further improving 
performance of reaching and grasping tasks [5, 23, 24]. Virtual reality, motion capture, force and flex 
instrumentation, and physiological signals (EMG, EEG) have been utilized to determine the links across visual 
feedback, agency, and performance of hand grasp tasks. The three major results from these studies include: 
Result #1: Agency and performance of reach and grasp are co-modulated through sensory feedback  
Experiments were conducted to investigate modulation of agency and computerized performance by varying 
control modes observed visually during reach-to-grasp in VR (Fig 1A), and grasp-only on a force-sensitive 
pinch apparatus (Fig 1B). The reach-to-grasp task involved controlling a virtual hand (Modular Prosthetic Limb, 
[25]) in a physics-based computer environment (MuJoCo, [26]). The subject controlled the virtual hand, viewed 
in an Oculus headset, from marker-based motion tracking (Optitrack) of their own hand. Evaluation of 
performance was based on minimizing the reaching pathlength to a target sphere. The grasp-only task 
involved applying precision pinch (thumb and index finger loads) onto the apparatus. Pinch force magnitude 
was projected as a force trace intended to track a displayed ramp that rose constantly with time. Performance 

 
 
was based on minimizing the error between the force trace and target ramp. For both tasks, control of either 
the virtual hand or the force trace was modified across modes typical for tuning assistive devices. These 
modes included variations in speed [27, 28], noise mitigation [29, 30], and automation [31, 32] from a ‘baseline’ 
mode best representing 
the subject’s intentions for 
control. For both tasks, 
agency was measured 
using time-interval 
estimation methods for 
intentional binding [33]. 
Following task completion 
(e.g., contacting sphere, 
end of ramp), a sound 
beep would be delay-
activated at a variable 
time-interval the subject 
would estimate. Relative underestimation of intervals 
indicated greater agency due to intentional binding [9]. We 
identified significant positive relationships (linear regression, 
slope>0, p<0.05) between performance and agency for both 
reaching (Fig 2A) [23, 34, 35] and grasping either rigid or 
compliant surfaces (Fig 2B) [23, 24]. We also observed 
patterns of increased neuromuscular coupling (Fig 2C) during 
states of high agency. These results suggest for reach and 
grasp, agency (perception of control) and performance are 
positively related and modulated across varied control modes 
perceived by visual feedback. This finding motivates the 
design of devices that utilize sensory feedback to enhance 
perception of control and improve functional performance.  
Result #2: Instrumented “cognition” glove to compute 
secure grasp and provide sensory feedback improves 
functional grasp 
Our lab has developed and tested a functional prototype of an 
instrumented glove to alert the user about secure grasp [36] 
of objects. Onboard force and flex sensors provide inputs to a 
machine learning algorithm (artificial neural network, ANN) to 
estimate secure grasp based on previously collected training 
data. The glove enhances agency by alerting the user to 
secure grasp through sensory feedback modules (visual – 
LED, audio – beeper, tactile – vibrator) (Fig 3A). Each user 
undergoes a simple calibration procedure to use the glove 
(Fig 3B). We have demonstrated reliable ANN prediction of 
secure grasp from signal pattern recognition compared to 
analytical methods assuming force vectors [37] (Fig 3C). For 
an able-bodied study (n = 15), the glove provided visual and audio feedback during training blocks for a 
precision grasp task to pick-up and place a small cubic object (Fig 4). Vibration feedback was omitted in this 
study but expected to assist sensation for persons with neuromuscular deficits [38]. Feedback from the glove 
was provided either ‘instantly’ after secure grasp or at time-intervals ‘decaying’ from 1 to 0 seconds after 
secure grasp. We hypothesized that decaying feedback would progressively induce agency-based 
performance by priming the person to perceive shorter intervals. Compared to no feedback, decay feedback 
 
Fig 1: Experiments to demonstrate links between agency and performance of reach-
to-grasp (A) and grasp-only (B).    
Fig 2: Experimental results for positively 
relating performance and agency during reach 
(A) and grasp (B). For grasp, high states of 
agency demonstrate spectral spiking (C) in the 
beta band for neuromuscular coupling 
(coherence between EEG and EMG).    

 
 
improved immediate post-training performance 
with reduced time for task completion and 
minimized pathlength of object motion. This 
preliminary study suggests that training with the 
cognition glove can generate immediate 
increases in performance of functional precision 
pinch. Our next step is to investigate if sensory 
feedback from the glove, and from VR, will 
improve grasp performance for persons with 
SCI.    
Result #3: Electromyography patterns can be 
regulated with sensory feedback training 
In creating a multi-sensory platform to train 
better muscle control for potentially improving 
independent function and command of 
myoelectric devices, we have accomplished the 
following: 1) established a method using systematic 
forms of visual feedback to mold EMG patterns during 
movement, and 2) progress in developing a virtual arm 
training platform with sensory feedback modalities. For 
accomplishment 1, we investigated the consistency to 
peak EMG magnitude across all measured muscle 
groups during squat with visual feedback. The squat 
exercise produces large bursts in muscle activations 
that are sensitive to small variations in technique [39]. 
Electromyography was measured (Delsys Trigno 
wireless system) for major flexor-extensor muscle 
groups of the ankles, knees, hips, and lower torso. The 
various forms of visual feedback included abstract 
(sinusoid) versus representative (stick-figure 
represents ‘body’). Increased consistency in EMG, 
indicated by lower standard deviations to the 
peak, were observed with representative 
feedback (Fig 5A). Other variations of 
feedback were examined based on 
complexity and intermittency (results not 
shown here) [23, 40]. From these squat 
studies, we have directives to present virtual 
reality feedback to train greater consistency 
in EMG movement. Specifically, our results 
indicate that complex body-representative 
visual cues presented more intermittently 
with progressively improved performance 
has the greatest potential for acutely shaping 
EMG patterns during movement. We plan to 
employ this approach for VR movement 
training in Aim 2.  For accomplishment 2, we 
have completed utilization of EMG pattern 
classifiers to control a cursor in a 3D virtual reality environment (Fig 5B) for completing a maze task from 
isometric force loading of the upper-arm and torso. Several classifiers were trained, but only results for the 
 
Fig 3: Instrumented glove with onboard sensors and modules 
providing sensory feedback (A). Glove accommodates each 
user hand after 10 repetitions of simple calibration procedure 
(B). Machine learning for sensor signal pattern recognition 
improved prediction rate of secure grasp (C).   
 
Fig 4: Using cognition glove to train better 
performance of functional grasp-and-place task (A). 
Following provision of sensory feedback from the 
cognition glove, there was significant reduction in time 
to complete the task and in pathlength of object 
movement (B).  
 
Fig 5: Visual feedback paradigms for modulating EMG patterns for 
better performance (A) or monitoring EMG patterns to drive 
movement performance in a virtual environment (B).  

 
 
support vector machine (SVM) and linear discriminant analysis (LDA) are shown. The SVM represents a 
traditional computational technique for controlling myoelectric devices [41] while LDA has markedly higher 
potential for reducing training times [42]. In our pilot study [43], we limited computational time to <10 minutes 
and observed better performance with LDA three muscle sets (ANOVA, p<0.01), but peak performance was 
achieved by SVM on an intermediate muscle set. This result suggests a unique and effective set of muscle 
inputs for myoelectric control can be identified for our platform within practical time constraints. This is a 
promising result for real-world SCI rehabilitation settings that must consider various degrees of muscle 
impairment and limited time to train users. 
 
 

 
 
 
(c) Research Design and Methods  
The Research Design for this two-year SPiRE proposal to RR&D will comprise of testing for two cognitive-
based training platforms for functional grasp and reach, respectively, on persons with cervical SCI (Fig 6A). 
The experimental set-ups developed at Stevens are modular and portable for conducting tests with Veterans 
having SCI at the James J. Peters VAMC. Testing of each platform represents a unique proposal aim. 
Execution of each aim will be done independently in each of the two years and similarly include: 1) adaptation 
of platforms for use with persons having SCI, 2) recruitment of five participants per year, 3) execution of a 
single 8-hour experimental session for each participant. Each session includes 3 hours for data collection, and 
the remaining time for preparation, rest, feeding, bathroom trips, and 90-minute washout of each experimental 
condition. Inclusion criteria for subjects include: 1) aged 18 to 65 years, post-SCI for more than 1 year, SCI 
between C1-T1 with hand weakness (2-4 out of 5 manual muscle test in intrinsic hand muscles of thumb or 5th 
digit).  
The schedule for each 12-month aim: recruitment (months 1-6), refining protocols for SCI (months 1-6), SCI 
participant testing (months 6-9), data and statistical analysis (months 7-10) and dissemination via conference 
and paper writings and submissions (months 9-12). Recruitment will be done via IRB-approved flyers at local 
SCI clinics; study posting at clinicaltrials.gov; and communication with previous participants at our VAMC. 
Among the ten total subjects (five per aim), our enrollment plan includes at least three females, three Latinos, 
and three African Americans.  
For each aim, the data analysis will involve 
within-subject comparisons across three 
conditions: (1) control with minimal-to-no 
cognitive feedback, (2) intermediate level of 
cognitive feedback, and (3) enhanced 
cognitive feedback. Enhanced feedback 
refers to addition of VR to glove feedback 
in Aim 1, and addition of vibration to visual 
feedback in Aim 2. For each condition, 
there will be three consecutive trial blocks 
(Fig 6B): 1) baseline – performing real-
world task 25 times with no feedback 
before training, 2) training – performing 
training task 50 times with feedback 
specific to condition, 3) retention – performing real-world task 25 times with no feedback after training. 
Repeated-measures MANOVA will be utilized given the three condition groups and two or more performance 
metrics specific to each aim. Our previous analyses for Result #1 with able-bodied persons suggest that n = 5 
is sufficient to show significant difference (p<0.05) for MANOVA across three condition groups and two 
performance metrics (completion time, motion pathlength) at 90% power. The primary hypothesis for each 
aim is that performance after training is significantly greater with enhanced feedback. Both aims of this 
study will also capture 32-channel EEG recordings for exploratory analysis of coherence computations with 
select recordings of muscle EMG as in Fig 2C. 
 
AIM 1: Investigate the effect of a “cognition” glove and virtual reality on grasp performance in SCI 
persons with partial hand paralysis.  
The cognition glove protocol from Result #1 will be extended to include VR feedback to persons with SCI 
performing the grasp-and-place of a small cubic object.  
Fig 6: Project overview - objective of each Aim (A) and flow chart for 
testing a condition, three conditions per session, for each Aim (B). 
Note: the third (final) condition does not require washout.   

 
 
PROTOCOL – Each trial involves: firstly, secure 
grasp of the cubic object placed in front of the 
dominant arm; secondly, pick-up the object and 
move towards target in front of non-dominant arm; 
thirdly, accurate placement onto target. The three 
condition groups for training task performance in 
Aim 1: 1) no feedback from glove as ‘control’, 2) 
‘decay’ feedback from glove that includes audio, 
visual, and vibration feedback from onboard 
modules, and 3) enhanced feedback with the 
addition of mixed-mode virtual reality. ‘Instant’ 
feedback is omitted given the observed 
improvement from baseline with ‘decay’ in persons 
without SCI (Fig 4). Enhanced feedback is ‘mixed 
mode’ since there still exists grasp onto a physical 
cubic object with the glove but interactions are viewed in VR. Subjects observe their own hand registered in VR 
using the LEAP® hand-tracker compatible with Oculus. The virtual cubic object is position-calibrated with the 
real object at the start of each trial. The glove communicates with the VR environment to cue real-time sensory 
feedback following detection of secure grasp. Enhanced VR feedback includes the small cubic object changing 
color to green and a view panel whose border also turns green. The panel also allows for posting of 
customized messages or images meaningful to the subject. The subject will pre-select a message, image, and 
sound to positively affirm achievement of secure grasp from a pool of options provided by the experimenters. 
Sample message and image include ‘great job’ and cartoon sunshine, respectively. At this stage, the novel VR 
feedback display is intended to be simple and evident ‘reward’ [44] to accelerate motor rehabilitation over the 
glove-only condition. Functional performance metrics for Aim 1 include: 1) time to achieve secure grasp 
upon initial contact, 2) time to complete pick-up and placement of object, 3) motion pathlength in moving 
object, 4) error in placing object onto target. Higher performance is indicated by reduction in these metrics.  
AIM 2: Investigate the effect of a “sensory” brace and virtual reality on reach performance in SCI 
persons with partial hand paralysis.  
The broad objective of this platform is to train greater 
strength and muscle control while interacting in an 
engaging environment. The virtual reality environment 
from Result #2 has been extended to include 
successful communication between all electronic 
modules. Subjects have EMG control of a virtual arm 
prosthesis in Unity (Fig 8). We are currently 
developing a size- and position-adjustable arm brace 
with weight-support capability and housing for 
vibration motors and EMG sensors. Position 
adjustment allows for physical therapists to find and 
recommend arm postures that are clinically relevant to 
each person. The participant can then isometrically 
push/resist against the brace to strengthen target 
muscles while performing VR reach-to-touch. The 
person will receive visual feedback from the virtual 
environment to train movement performance and 
vibrotactile feedback at tendons to subconsciously 
adjust their muscle activation patterns. Tendon 
vibration is proven to elicit movement sensations 
affecting muscle activations [45-47].  
 
Fig 8: Pilot experimental set-up providing visual, audio, 
vibrational feedback to participant held in isometric 
(static) position in EMG-control of virtual arm in VR. 
Note: an arm brace is currently being constructed to 
replace ad hoc set-up shown and used for pilot testing.  
 
Fig 7: Experimental protocol for enhanced training feedback 
condition in Aim 1. Condition includes mixed-mode 
feedback with manipulation of real object while observing 
VR environment with enhanced visual and audio feedback.     

 
 
PROTOCOL - The fundamental task for this aim is to reach-to-touch designated targets with a real arm 
(baseline, retention) or virtual arm (training). Each trial involves: firstly, moving the hand towards a specific 
target; secondly, minimizing pathlength of hand motion; thirdly, accurately touching the center of the target. For 
this Aim, prior to training across three conditions, data will be collected to train an LDA pattern classifier based 
on each subject’s EMG patterns. This data collection involves 5 maximum isometric voluntary exertions within 
the brace in each of six command directions (front, back, right, left, up, and down). The three condition groups 
for training task performance in Aim 2: 1) no feedback and simply practicing the virtual arm task with the 
classifier, 2) visual feedback to control the virtual arm with complex-representative-intermittent visual feedback 
of a transparent guide arm to be followed, 3) visual and tactile feedback whereby vibro-tactile feedback will be 
added near tendon locations for muscles on the chest (pectoral), back (latissimus dorsi) and upper arm 
(biceps, triceps). Able-bodied subject experiments, conducted outside the scope of this proposal, will 
demonstrate how to provide vibration feedback that optimizes (maximizes) EMG inter-cluster distances [48] 
and augments isometric myoelectric control [49] with the trained classifier. Isometric training and distinct EMG 
activation patterns have demonstrated improved rehabilitation performance [50-52]. Functional performance 
metrics for Aim 2 include: 1) time to touch each target, 2) motion pathlength of reaching arm, 3) contact 
position error from center of each target. Higher performance is indicated by reduction in these metrics.   
Challenges/Alternatives – There are two primary challenges to this project: 1) For Aim 1, the ability of a person 
with SCI and compromised sensation to find and skillfully grasp the real object while viewing the virtual 
environment. If the subject is overly challenged to grasp the real object, then virtual feedback will be provided 
in desktop mode. VR goggles will be removed, and the participant can see and grasp the real object directly 
while a virtual environment is projected onto a background monitor. We expect viewing the virtual environment 
peripherally will still have additive reinforcement effects [53]. 2) For Aim 2, subjects with various levels of SCI 
and conditioning may experience more fatigue than others given a total of 300 trials per session. For each Aim, 
condition 1 allows us to re-calibrate our procedures based on participant behavior and apparent onset of 
fatigue. Re-calibration includes longer rest periods and fewer trials. A pilot analysis in persons without SCI for a 
reach-to-touch task done in our lab [54] suggests MANOVA significant differences in performance (pathlength 
and contact accuracy) across two feedback conditions (reward, punishment) could be observed with only 40 
trials per condition. For both Aims, fatigue will also be monitored from EMG recordings [55].  
Implications for MERIT award: Following the 
SPiRE, we will  investigate these cognitive 
training approaches on three separate device 
testbeds (Fig 9). The first testbed would 
involve reach-to-touch with a physical robotic 
arm. We are developing a low-cost (<$1000) 
VR training toolkit which will include the 
cognition glove, sensory brace, and the real 
robot arm to be utilized at physical 
rehabilitation centers. The robotic arm provides 
a fun, real-world alternative to test muscle 
control after virtual arm training. The second 
testbed would be for training better user control 
of a powered hand exoskeleton (HEXOES, 
[18]) for grasp assistance. We would employ a 
hybrid approach to combine motor actuation from the glove with one-channel FES (g.tec FES stim) to elicit 
grasp from tenodesis [56, 57]. Our cognitive platform would be used to compute secure grasp and provide 
sensory feedback as with the cognition glove to enhance agency with this device. The third testbed would be 
incorporation of sensory brace training with use of an arm exoskeleton (HERCULES, in progress) driven by 
EMG signals. The same training approach from Aim 2 would be utilized to provide feedback and achieve better 
reach-to-touch performance with the arm exoskeleton from more distinct EMG commands.  
 
Fig 9: Device testbeds with future MERIT proposal. Real-world 
robot arm serves as a ‘fun’ alternative for performance testing 
after training with virtual arm from Aim 2 (A). Powered hand 
exoskeleton would include FES and be adapted to provide 
sensory training feedback as the cognition glove with VR as in 
Aim 1 (B). Arm exoskeleton (in progress) training would 
incorporate a brace for sensory feedback as in Aim 2 (C).