Usability of Diabetes Dashboard Embedded Within a Patient Web Portal: A Prospective Longitudinal Study
Phase: NA
Conditions: Diabetes Mellitus, Type 2
Sponsor: Vanderbilt University Medical Center

 
 
 
 
 
 
 
 
 
 
Usability of Diabetes Dashboard Embedded Within a Patient Web Portal: A Prospective 
Longitudinal Study 
 
NCT03786718 
 
Protocol and Statistical Analysis Plan 
 
May 3, 2020 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
Table of Contents: 
 
Study Schema 
1.0 
Background 
2.0 
Rationale and Specific Aims 
3.0 
Inclusion/Exclusion Criteria 
4.0 
Enrollment/Randomization 
5.0 
Study Procedures 
6.0 
Statistical Analysis Plan 
7.0 
Qualitative Analysis Plan 
 

 
 
1.0 
Background 
Diabetes is a leading cause of kidney failure, heart disease, stroke, visual impairment, 
and non-traumatic lower limb amputations. Diabetes self-care and preventative health 
services can prevent or delay many diabetes-related complications, yet few patients 
consistently receive all recommended services or engage in recommended self-care 
behaviors that are challenging to implement and sustain. Patient activation (i.e., 
knowledge, skills and confidence to manage their own health and care) is essential to 
achieving optimal diabetes self-management and is associated with lower costs. 
 
By providing an engaging and convenient means to track and visualize health data, 
obtain education and guidance, and connect patients and doctors, patient portals offer a 
promising platform to increase patient activation, enhance care and promote self-
management while overcoming the limitations of costly and difficult to scale face-to-face 
interventions. We recently applied user-centered design sprint methodology and key 
strategies for patient engagement to develop a patient portal intervention for patients 
with diabetes called the Diabetes Dashboard (also known as My Diabetes Care).   
 
The Diabetes Dashboard is embedded within an established patient portal, My Health at 
Vanderbilt (MHAV).  The Diabetes Dashboard is a multi-faceted intervention designed 
to help patients better understand their diabetes health data as well as support self-
management. The Diabetes Dashboard uses graphics to visualize and summarize 
patients' diabetes health data, incorporates motivational strategies (e.g., social 
comparisons), and provides literacy-level sensitive educational resources. The Diabetes 
Dashboard is grounded in a well-established Chronic Care Model adapted for eHealth 
(i.e., healthcare practices supported by electronic processes and communication). By 
leveraging elements within the Model’s five domains (self-management support, 
delivery system design, decision support, clinical information systems, and eHealth 
education), the Diabetes Dashboard has the potential to create more informed and 
activated patients leading to improved outcomes.   
 
2.0 
Rationale and Specific Aims 
In order to assess dashboard usability, usage patterns, and user experience over time, 
and to uncover errors in dashboard functionality prior to a larger interventional trial, we 
will conduct a longitudinal usability study. The study will utilize questionnaires to 
quantify participants’ perceptions and responses to the dashboard and semi-structured 
interviews to comprehensively assess users’ experience and barriers to use. Combining 
qualitative and quantitative assessments of usability identifies more usability concerns 
than quantitative (e.g., questionnaires) assessments alone. Each participant will have 
access to the dashboard for 1 month from the time of enrollment. This duration allows 
sufficient time for participants to become familiar with the dashboard and determine if 
and how to use the dashboard in the management of their diabetes. 
 
In the current study, we aim to assess: (1) the usability and acceptability of the Diabetes 
Dashboard, (2) users’ attitudes about dashboard features and potential improvements, 
and (3) the potential impact of the dashboard on secondary cognitive and behavioral 
outcomes including patient activation.  

 
 
3.0 
Inclusion/Exclusion Criteria 
Participants will be eligible for the study if they have type 2 diabetes mellitus, are 
currently being treated with at least one diabetes medication, are able to speak and 
read in English, are age 21 or over, have an existing MHAV (i.e., Vanderbilt patient 
portal) account, and self-report reliable access to a computer with internet access. 
 
We will exclude patients living in long term care facilities, patients with known cognitive 
deficits, patients with severe visual or hearing impairment, patients with unintelligible 
speech (e.g., dysarthria), and patients currently participating in another diabetes related 
research study. 
 
4.0 
Enrollment/Randomization 
Participants will be able to complete an electronic consent form and enroll online via 
Research Electronic Data Capture (REDCap) version 5.0.8. 
 
5.0 
Study Procedures 
Setting. The Vanderbilt Adult Primary Care (VAPC) clinics and Eskind Diabetes Clinics 
are located within a large ambulatory care facility in Nashville, TN. An electronic health 
record (Epic Systems Corp.) stores all clinical data and patients receive access to their 
clinical data via an integrated and highly-adopted patient web portal, MHAV, that is 
accessible on desktops and via a native mobile app for iOS and Android mobile 
operating systems. 
 
Participants and Recruitment. Potential participants will be identified automatically using 
VUMC’s Subject Locator to query the EHR for patients with upcoming clinic 
appointments who meet the discrete inclusion and exclusion criteria. We will send 
identified patients a letter describing the study. Study flyers will be placed in 
participating clinic sites. Interested patients will contact a research assistant to learn 
more about the study and confirm eligibility.  
 
Participants will complete an electronic consent form and enroll online via REDCap. Our 
target enrollment is 70 participants. In accordance with best practices and to reflect a 
range of patient experience with diabetes and groups with distinct usability needs, we 
will recruit a purposive sample to achieve at least 20% representation of each of the 
following characteristics: (a) limited health literacy and (b) age 65 or over. Some 
participants may have one or more of these characteristics. Health literacy will be 
assessed using a validated on one-item screener that asks respondents to rate their 
confidence independently filling out medical forms. Consistent with prior studies, we will 
categorize participants noting any lack of confidence filling out medical forms as having 
limited health literacy. 
 
Data Collection and Outcome Measures. Study participants will complete 
questionnaires electronically via email using REDCap at two time points:  enrollment 
(T0) and end of study (T1). The enrollment questionnaire (T0) will include basic 
demographic questions, items about computer/smartphone usage, and validated 
measures of health literacy and eHealth literacy.  Both questionnaires will contain 

 
 
validated scales to assess study outcomes. Participants will be compensated $40 for 
completing the enrollment questionnaire, $35 for completing the study end 
questionnaire, and $5 for first 10 minutes of dashboard use.  
 
The primary outcome measures will be: (a) ease of use and satisfaction as assessed by 
the System Usability Scale at T1, (b) user experience as assessed by end of study 
questionnaire and semi-structured interviews at T1. Secondary outcomes will include 
system usage (total number of dashboard visits, total duration of dashboard use, and 
use of embedded educational links, secure messaging, and a link to the American 
Diabetes Association Online Community) and the potential impact of the dashboard on 
the following secondary cognitive and behavioral outcomes assessed using validated 
scales at T0 and T1: patient activation, diabetes self-efficacy, diabetes knowledge, 
diabetes care understanding, diabetes self-care, and diabetes distress. 
 
At the conclusion of their one month of dashboard access, we will conduct one-on-one, 
semi-structured interviews with a subsample of at least 10 participants. This 
methodology is most appropriate for in-depth assessment of patients’ perceptions and 
reactions regarding a proposed intervention. Interviews are preferred over focus groups 
for understanding usability because focus groups can amplify bias and individual 
opinions. Interviews will take place by phone or in person and will last approximately 30 
minutes. A trained interviewer will use a semi-structured interview guide to facilitate the 
interview and elicit in-depth understanding of participants’ perceptions and experiences 
with specific dashboard functionality as well as barriers to use. Additional participants 
will be enrolled until saturation is reached. Saturation will be defined as no new usability 
concerns raised in the preceding two interviews and typically occurs between 10 and 30 
interviews. Participants will be compensated an additional $40 for the interview.  
 
6.0 
Statistical Analysis Plan  
Statistical Analysis. We will use descriptive statistics to characterize the study 
participants and survey responses. We will use a one sample t-test to compare the 
mean SUS score at T1 to the threshold score of 68 indicative of “above average” 
usability and a two-sample t-test to compare SUS scores of independent groups (e.g., 
participants with limited vs. adequate health literacy). If SUS score distributions 
suggests asymmetry or nonnormality, we will use the one sample median test to 
compare the median SUS scores at T1 to the threshold score of 68 and Wilcoxon-
Mann-Whitney test to compare the SUS score distributions of two independent groups.   
 
To assess whether there was a significant improvement in the continuous secondary 
cognitive/behavioral outcomes from baseline to end of study (T0 to T1), we will perform 
two-sided paired t-tests on the pairwise differences. If any of the distributions of pairwise 
differences suggest asymmetry or nonnormality, the non-parametric Wilcoxon Signed 
Rank Sum test will be performed in lieu of paired t-tests. We will use the McNemar’s 
test for to compare paired proportions and Fischer’s Exact test for independent 
proportions. All analyses will be completed using statistical software under the 
supervision of a biostatistician in the Vanderbilt Center for Diabetes Translational 
Research (CDTR).  

 
 
 
Sample Size. In the context of the primary outcome, SUS score, and assuming a 
standard deviation of 12 based on prior studies, with a sample size of 50, a one sample 
t-test would detect an absolute difference in mean SUS scores of at least 5 points 
above the threshold score of 68 with 82% power. Conservatively assuming study 
dropout of 20%, we aim to enroll 70 participants. 
 
7.0 
Qualitative Analysis Plan 
We will transcribe participants’ semi-structured interviews for coding and analysis of 
user experience. We will use selective coding to identify participants’ statements 
addressing six established elements of user experience (i.e., the core category): (1) 
useful - fulfilling a need, (2) useable - ease of use, (3) accessible - easy to access and 
comprehend, (4) desirable - design elements that evoke emotion or appreciation, (5) 
findable - easy to navigate, and (6) credible - trustworthy and believable. Two research 
assistants will independently code all interviews then resolved any differences by 
consensus. We will review participant statements in each category for inform potential 
revisions that could improve user experience with the dashboard.