Video Information Provider for HIV-Associated Non-AIDS (VIP-HANA) Symptoms
Phase: NA
Conditions: HIV (Human Immunodeficiency Virus), AIDS (Acquired Immunodeficiency Syndrome)
Sponsor: Columbia University

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Protocol Title: Video Information Provider for HIV-Associated Non-AIDS (VIP-HANA) 
NCT03182738 
Date: 04/16/18 
 

 
Protocol: IRB-AAAP5958 
Principal Investigator: Rebecca Schnall 
Protocol Title: Video Information Provider for HIV-Associated Non-AIDS (VIP-HANA) 
 
 
STUDY PURPOSE AND RATIONALE 
 
The purpose of this study is to use technology to improve symptom status and ultimately improve 
patient centered outcomes in PLWHA. The primary purpose of the intervention (VIP- HANA) is to 
improve symptom status. We hypothesize that VIP-HANA will improve symptom frequency and 
intensity, our primary outcomes. 
 
SPECIFIC AIMS 
 
HIV has evolved from an acute to a chronic illness largely due to antiretroviral therapy (ART) 
(Chiasson, Shaw, Humberstone, Hirshfield, & Hartel, 2009). As a result, people living with 
HIV/AIDS (PLWHA) are living longer and by 2015 half of PLWHA in the US will be over 50 years 
old. As PLWHA age, they are developing chronic illnesses and co-morbid conditions that are 
often seen in older HIV negative patients (Parsons et al., 2005). Fifty to sixty percent of deaths   
in HIV-infected persons occur from HIV-associated non-AIDS (HANA) causes and people 
suffering from these conditions are more likely to be affected by adverse symptoms (Parsons et 
al., 2006). HANA conditions (e.g., cardiovascular disease, liver disease, diabetes, and asthma) 
are becoming more common as PLWHA age. 
An individual's ability to identify and self-manage symptoms of HIV illness has been shown to 
improve patient outcomes and quality of life ("Endpoints and other measures in a multisite HIV 
prevention trial: rationale and psychometric properties. NIMH Multisite HIV Prevention Trial," 
1997; Parsons, Missildine, et al., 2004; Parsons, VanOra, Missildine, Purcell, & Gomez, 2004). 
Holzemer (study consultant) developed a paper-based symptom management manual with self-
care strategies for 21 common (non-HANA) HIV/AIDS symptoms. The efficacy of this manual for 
improving symptoms was established in a 775-person RCT over three months at 12 sites 
(Chiasson et al., 2005). To enhance uptake of these strategies by PLWHA, we developed and 
pilot tested the Video Information Provider (VIP), a web application (app) that delivered HIV-
related symptom self-care strategies for PLWHA (P30NR010677- 03S1)(Parsons, Halkitis, 
Wolitski, & Gomez, 2003). Results from our 3-month pilot study (N=42) study overwhelmingly 
demonstrated the feasibility of the system. Participants reported a decrease in HIV-related 
symptom frequency and intensity after using the VIP system for 12 weeks. 
However, Holzemer’s symptom management manual and VIP were not developed for managing 
symptoms related to HANA conditions. Little is known about the symptom experience or self-
care strategies used by PLWHA with HANA conditions. Based on the existing evidence and our 
previous work, we hypothesize that providing PLWHA with HANA conditions with tailored self-
care strategies in their everyday settings has the potential to improve symptom status, reduce 
healthcare visits, decrease frailty, and improve ART adherence and quality of life. Therefore, 
there is a need to identify the symptom experience of PLWLHA with HANA conditions and the 

self-care strategies they use to manage the symptoms. For this study, we propose to: 
Specific Aim 1a: Identify the most prevalent symptoms and self-care strategies of PLWHA with 
HANA conditions in order to modify the VIP system into the VIP-HANA system. 
To achieve this aim, we will conduct an online survey with 1,000 PLWHA with HANA 
conditions, recruited from national social networking websites, to learn about their symptom 
experience and self-care strategies that they use to mitigate their symptoms. Our study team 
will incorporate the most frequently reported symptoms and appropriate self-care strategies 
into our VIP system. The VIP-HANA system, which will provide tailored self-care strategies 
based on HANA conditions and gender. 
Specific Aim 1b: Evaluate the VIP-HANA system for violations of usability principles. 
Specific Aim 2: Compare the efficacy of VIP-HANA to a control arm for ameliorating symptom 
frequency and intensity and secondary health outcomes in 100 PLWHA with HANA conditions 
over 6 months. 
Our hypotheses are that use of VIP-HANA as compared to a control arm will: 
a) Decrease symptom frequency and intensity (primary outcome measures) in the 
treatment arm as compared to the control arm; 
b) Decrease frailty as measured by Fried’s frailty phenotype; 
c) Improve functional and emotional status, quality of life, and morbidity; and 
d) Improve adherence to antiretroviral therapy (ART). 
 
Specific Aim 3: Understand PLWHA’s perceptions of the predisposing, enabling, and 
reinforcing factors for VIP-HANA use with theoretically-guided focus group sessions. 
We will conduct focus groups, using the PRECEDE-PROCEED model with Aim 2 intervention 
arm participants, to understand their perceptions related to VIP-HANA use. 
 
 
DESIGN AND METHODS TABLE 
 
Table 1 presents and overview of the design, participants and data analysis plan  
 
 
Table 1. Overview of Design, Methods, Participants, and Data 
Analysis 
Aim Design/Methods Participants 
Data Analysis 
1a 
Online Survey 
PLWHA with HANA 
conditions(N=1000) 
and PLWHA without 
HANA conditions 
(N=1000) 
Descriptive and thematic analysis of most 
frequently reported symptoms, symptom 
attribution, self-care strategies and 
information source. 

1b 
Heuristic 
evaluation, 
Usability testing 
(including: cognitive 
walkthrough, user 
testing using 
think-aloud 
protocol) 
Heuristic evaluation: 
human-computer 
interaction experts 
(N=5); User testing: 
PLWHA (N=20) 
Quantitative and qualitative summary of 
heuristic violations; thematic analysis of 
think-aloud protocol, quantitative summary 
of mouse clicks, time, etc. 
2 
Randomized 
controlled design 
PLWHA with HANA 
conditions 
(pilot: N=5; trial: 
N=100; 
intervention=50, 
control=50) for 26 
weeks 
Descriptive statistics; Linear regression 
model 
3 
Descriptive: 
focus groups 
Focus groups: 
Participants in the 
intervention and 
control arms of the 
RCT (N=60) 
Descriptive, thematic analysis to examine 
post-intervention perceptions related to VIP- 
HANA use, usefulness and impact on overall 
health 
 
Population, Target Enrollment and Subject Compensation 
Aim 
Description 
Participants 
Target 
Enrollment 
Compensation 
1a 
Online Survey 
PLWHA with HANA 
conditions(N=1000) and 
PLWHA without HANA 
conditions (N=1000) 
2,500 
$10 incentive offered 
female participants 
through paid 
campaign and on 
site female survey 
takers to reduce 
gender disparity in 
data collection; we 
have also offered 
$10 for 
participants who 
have taken the 
follow-up survey 
online. 
1b 
Heuristic evaluation, 
Cognitive 
walkthrough and 
user testing using 
think-aloud protocol 
Evaluators: Human- 
computer interaction 
experts (N=5); Cognitive 
Walkthrough Expert 
(member of study team) 
(N=1); End User testing 
( (N=20) 
26 
Heuristic Evaluators: 
$150 
PLWHA: $35 

2 
Randomized 
controlled design 
PLWHA with HANA 
conditions 
(pilot: N=5; trial: N=100; 
intervention=50, 
control=50) for 26 weeks 
105 
$250  
3 
Descriptive: 
focus groups 
Participants in the 
intervention and 
control arms of the 
RCT (N=60) 
60 
$30 

 
Study Design, Subject Population, Setting and Sample, Procedures, are all detailed by 
Aim below: 
 
Specific aim 1a: Identify the most prevalent symptoms and self-care strategies of PLWHA 
with HANA conditions in order to modify the VIP system into the VIP-HANA system. We 
will also include PLWHA without HANA conditions for a comparison analysis. In response 
to the reviewers’ feedback, we have completely revised Aim 1 so that we can collect the best 
patient-reported evidence that is available. The goal of our work in Aim 1a is to identify the most 
frequently reported symptoms and self-care strategies of PLWHA with HANA conditions and 
integrate them into our existing VIP system. We will expand and refine the existing symptom list 
(21 symptoms) to include symptoms and self-care strategies by adding new symptoms that 
are the most frequently reported by survey participants with HANA conditions in Aim 1. The 
additional information that is collected will be synthesized and incorporated into the VIP 
system which will result in the VIP-HANA system. The VIP-HANA system will deliver self-
care strategies to PLWHA with HANA conditions tailored to symptom reporting, HANA 
condition(s) and gender. VIP-HANA will be a web-application (app) and can be used on a 
computer or a mobile device. 
Online Survey with 2,000 PLWHA: 
This web-based survey will enroll approximately 1,000 PLWHA with HANA conditions and 
approximately 1,000 PLWHA without HANA conditions into an anonymous online survey that 
inquires about symptoms and self-care strategies of PLWHA with HANA conditions. Study 
participants will be recruited online from various social media sites such as Facebook.com (the 
largest online social networking site) and POZ (poz.com). Dr. Hirshfield (Co-I) has had great 
success using this recruitment strategy (Chiasson et al., 2005; Hirshfield, Remien, 
Humberstone, Walavalkar, & Chiasson, 2004; Hirshfield, Remien, Walavalkar, & Chiasson, 
2004). 
Eligibility criteria: 1. HIV+; 2. Age 18 or over; 3. Able to read and respond in English; 4. Reside 
within the US; 5. .Willing to participate in an online survey. We have chosen the following HANA 
conditions because they are the most frequently occurring conditions that result in mortality 
and/or morbidity (Crothers et al., 2006; A. Justice & Falutz, 2014; A. C. Justice, 2010; A. C.  
Justice, 2014; A. C. Justice et al., 2004; Kendall et al., 2014; Messeri, Lee, & Berk, 2009; So-
Armah & Freiberg, 2014).  
HANA Conditions: Cardiovascular Disease, Liver Disease, Asthma, Bronchitis, COPD, 
Osteoporosis, Diabetes, arthritis and/or Renal Failure  
 
Exclusion: 1. HIV-negative; 2. under age 18; and 3. Unwilling to provide key data (i.e., age, 
information about symptoms) on the online survey. 
Recruitment: We are using two online venues to recruit the survey sample. Online recruitment 
will enable us to reach more PLWHA and will ensure a more diverse sample by age, geography, 
and engagement in their healthcare services. We have had success in recruiting 
racially/ethnically diverse PLWHA from social networking sites, like Facebook and POZ, in 
previous studies (Chiasson et al., 2005; Hirshfield, Remien, Humberstone, et al., 2004; Hirshfield, 
Remien, Walavalkar, et al., 2004). POZ is a U.S.-based website serving the community of people 
living with, and those affected by, HIV/AIDS since 1994. POZ.com provides health information 

about HIV, sexually transmitted infections, and other chronic health conditions. POZ.com also 
has a national health services directory, covers HIV-related local and international news, as well 
as blogs, forums, and a personals section for HIV-positive individuals, which currently has over 
150,000 U.S. male and female members. Dr. Hirshfield has an   ongoing collaboration with 
POZ.com, which includes an online pilot study of 463 HIV+ men, a publication based on this 
collaboration, and an NIH-funded online national intervention for HIV+ virally unsuppressed 
MSM recruited through POZ Personals. The Research and Evaluation   Unit at Public Health 
Solutions has developed a registry of research participants, including a large number of HIV+ 
men who do not meet criteria for other ongoing studies. For the proposed study, Dr. Hirshfield 
will work with POZ to recruit the sample. We will use psychographic targeting (in-depth publicly 
available consumer data such as interests, occupation, and city) to recruit PLWHA with HANA 
conditions from around the US. 
See Attachment: Qualtrics Survey VIP. HANA.pdf (stamped 04/25/2016) 
Study banners will be created by Public Health Solutions for recruitment of PLWHA with HANA 
conditions for the aforementioned survey. The study banners will route people to our study 
landing page. We conservatively estimate that at least 50% of all PLWHA in the US suffer from 
at least 1 HANA condition based on our review of the literature (Crothers et al., 2006; A. Justice 
& Falutz, 2014; A. C. Justice, 2010; A. C.  Justice, 2014; A. C. Justice et al., 2004; Kendall et al., 
2014; Messeri et al., 2009; So-Armah & Freiberg, 2014). Given this estimate, of the 150,000 
PLWHA who use POZ.com as well as additional PLWHA with HANA conditions who use 
Facebook, we should easily be able to collect our study data within our study timeframe. Dr. 
Hirshfield collaborated with POZ on a brief online pilot in NY, CA and GA, sending emails to a 
defined subset of HIV+ male members identifying as gay or bisexual (n=5,262). POZ’s email 
system can target members by location, sexual orientation, race/ethnicity, age, and most recent 
log-on. Of the 5,262 emailed, 1,696 (32%) clicked through; 479 of those who clicked through 
consented (28%); and 464/479 completed the survey (97%). In 7 days, through direct email with 
POZ Personals members, we were able to collect 463 completed cases. For the proposed study, 
we will be using banner ads on POZ and may not have the same response rate; however, 4 
months is a sufficient amount of time to recruit 1,000 participants, given our inclusion criteria. 
During the initial stages of survey collection for Aim 1a we detected a significant gender disparity 
in our population being mostly males completing the survey. Due to this difficulty recruiting 
women, in an effort to increase the number of female participants we will close the enrollment for 
males at this time. We have modified our survey to include female reproductive system 
questions and created a recruitment flyer offering $10 as an incentive for female survey takers. A 
new survey has been created for this purpose. 
 
See Attachment: VIP HANA Women Qualtrics Survey.pdf 
For this purpose, we will launch a paid online campaign through POZ.com as well as flyers 
posted around the Columbia hospital and surrounding areas as well as other social media sites 
to include more women and reduce gender disparity. Online female survey takers will receive 
a $10 amazon gift code while on site female survey takers, recruited through posted flyers, will 
receive $10 in cash. 
Procedures: PLWHA who click our banner advertisements posted online will be automatically 

directed to a study landing page that will screen to see if they qualify for participation. If they 
do qualify then they will be directed to complete the consent form. Those who consent, by 
clicking a consent button, will then complete a one-time only anonymous online survey. 
Employing the methods used by Holzemer (consultant) and Bakken (Co-I) when they 
originally developed the HIV Symptom Self-Management Manual (Bakken et al., 2000; 
Corless et al., 2002), participants will be asked to identify the 3 most disruptive symptoms 
they experienced in the past month. 
Survey questions related to Symptoms and Self-care strategies are in Table 4.The web-based 
survey will be hosted by Qualtrics software. Columbia University Medical Center has a license 
with Qualtrics and all data collected through this software is HIPAA compliant. Qualtrics is also 
CUMC-certified and PHI secured which is a requirement of the Columbia University IRB for data 
collection via the Internet. Qualtrics will automatically assign random subject ID#s to each 
participant. We will collect information on symptom reporting and self-care strategies. Data will 
also be collected regarding additional secondary measures such demographics, psychosocial 
measures, ART adherence and scheduled/missed healthcare visits. As we mentioned before we 
will include PLWHA without HANA conditions for a comparison analysis. At this time we have 
decided to close the enrollment for males and direct more efforts into including female 
participants. 
Data Analysis: Frequency of symptom reporting will be calculated. Our analysis will look for 
differences in symptom reporting and self-care strategies based on HANA condition, age 
and/or gender. We anticipate that there will be differences in gender but not in age (See 
Section C.2.1 below). Symptoms must be attributed to a HANA condition to be included as 
part of the final list of symptoms in our intervention. We anticipate that some self-care 
strategies will be the same for different conditions. In other cases, strategies will be different 
and the VIP-HANA system will deliver a self-care strategy tailored to symptom reporting, 
HANA condition(s) and gender, if applicable. Content analysis will be used to analyze the 
self-care strategies and attribution data for inductive development of the category schemes. 
Initial codes or categories of self-care strategies will be identified inductively from the data. To 
reduce the number of categories, initial codes will be consolidated on the basis of theoretical 
similarities and clinical significance after consultation with our study team. Drs. Schnall, 
Siegel, Bakken and the research assistant will collaborate on this analysis. We will also 
categorize the self-care strategies by information source reported by the survey respondent. 
Strategies which are potentially harmful or illegal (e.g., substance use) will not be included in 
VIP-HANA. We will have precautions in place to advise patients who appear to be at-risk for 
negative outcomes (e.g., report of severe depression) to contact their healthcare provider, call 
911 or go directly to the emergency room. 
 
Study Team and Experts will finalize the symptom list and self-care strategies: In year 2, we will 
conduct a meeting with the study team and the EAB to finalize the symptom list and the self- 
care strategies to be included in the VIP-HANA system. Once this has been finalized, technical 
refinement of our existing web app will take place at CBITs (see facilities and resources). We 
have partnered with CBITs since they use Purple, an extensible, modular, and re-purposable 
system created for the development of Web-based and mobile-based applications for health 
behavior change (Schueller, Begale, Penedo, & Mohr, 2014). Dr. David Mohr (PI: CBITs) is the 
Site PI for this project. Figure 4 illustrates a sample screen created by CBITS of self-care 
strategies for depressive symptoms. In response to the reviewer’s request, a letter of support 

from the technical team is included. 
 
Specific Aim 1b: Evaluate the VIP-HANA system for violations of usability principles. 
The goal of usability testing is to improve system design, identify potential problems with 
usability, and increase the likelihood of technology acceptance. To achieve this goal we will 
evaluate the user interface and system functions of the VIP-HANA system developed at CBITs. 
We will conduct three types of usability evaluations: A) Heuristic Evaluation B) Cognitive 
Walkthrough, and C) PLWHA Usability Testing. Using this combination of methods, which 
includes both experts and end-users, can be used to more comprehensively identify potential 
usability concerns (Jaspers, 2009; Tan, Lui, & Bishu, 2009) One set of user scenarios will be 
used across all three evaluation methods to facilitate comparisons between findings (Tan et al., 
2009). 
Heuristic Evaluation: 
 
Sample: Five informaticians will be recruited as usability experts (J. Nielsen & Molich, 1990). 
Nielsen recommends using three to five evaluators since one gains little additional information by 
adding additional evaluators (J. Nielsen & Molich, 1990). Experts must have training in human- 
computer interaction and have published in the field of informatics. We will recruit faculty and 
doctoral students from the Columbia University Department of Biomedical Informatics via 
personal correspondence (see attachment Letter for Heuristic Evaluators dated 5/5/2017). 
Procedures: Usability experts will be provided with an Alpha version of the VIP-HANA system. 
Similar to procedures used in our prior work (Gordon et al., 2012; Lee et al., 2011; Schnall, 
Kaufman, Hyun, & Bakken, 2009; Sheehan, Lee, Rodriguez, Tiase, & Schnall, 2012) each 
usability expert will be asked to evaluate the system using the Heuristic Evaluation Checklist 
and to think-aloud while doing so (Bertini, Gabrielli, & Kimani, 2006). Morae software™ 
(Techsmith Corporation, Okemos, MI) (TechSmith, 1995) will be used to record sound and 
screens during the heuristic evaluation for subsequent analysis. The participants will be asked 
to describe what they are thinking, seeing and trying to do as they perform the tasks required in 
the scenarios developed for usability testing in this study (see attachment “VIP HANA user 
scenarios with embedded symptom reports 05.05.17”). If a participant is silent for a long time, 
the researcher will remind the participants to think out loud. When a user finds errors or the 
researchers finds critical incidents that are characterized by comments, silence or looks of 
puzzlement, the researcher will record the users’ activities. Recording the users’ interactions 
and vocalizations provides additional feedback that can highlight problems not identified with 
static screen shots (Gibson, Gaffey, & Spendlove, 2003). In addition to the user scenarios, a 
symptom report will be provided to both end-users and their health care providers. Examples 
of how these symptom reports will appear are included in the user scenarios and will be 
presented to the heuristic evaluators, expert study member performing cognitive walkthrough, 
and the end users. 
Instrument: Nielsen (2005) proposed a list of ten recommended heuristics for a usable interface 
design. Bright, Bakken, and Johnson (2006) developed a Heuristic Evaluation Checklist based 
on Nielsen’s ten heuristics (Bertini et al., 2006) (Appendix). Each heuristic will be evaluated with 
one or more items and the overall severity of identified heuristic violations will be rated. Heuristic 
evaluators will complete an online questionnaire while working though their assessment of the 
system (see attachment Heuristic evaluation survey 05.01.17). Heuristic evaluators will be asked 

the following four questions regarding the symptom report: 1) What are your initial thoughts when 
you look at the symptom report? 2) How can we modify the report to more appropriately meet the 
needs of end-users? 3) How can we modify these reports to effectively share information with 
health-providers? 4) Do you have any additional feedback or comments that may help us to 
improve these reports? Please feel free to create any additional sketches that you think would 
help communicate your ideas.  
Data Analysis: The frequencies of usability issues will be calculated according to the heuristic 
principles adapted from Nielsen’s checklist. Mean severity scores will be calculated for each 
heuristic principle. Evaluators’ comments about usability problems on the evaluation form and 
the Morae recording will be grouped and content analyzed according to the usability factors of 
Nielsen’s heuristics (1994). Responses to questions regarding the symptoms report will be 
compared for similarities and possibility for improvements will be determined. 
 
Cognitive Walkthrough: 
 
Sample: An expert study team member will identify usability concerns within the system. 
 
Procedures: The study team member who is a usability expert will be provided with the same 
set of user scenarios symptom reports and will be asked to simulate a novice user and the 
process that user would go through to achieve the goals set forth in each of the scenarios 
(Jaspers, 2009). By doing this, the evaluator creates a cognitive model that simulates the 
cognitive processes a user might go through to effectively accomplish tasks within the system. 
There are four tasks that the expert will need to complete for each scenario: 1) The user must 
set an end goal to accomplish, 2) The user inspects available actions on the user screen (menu 
items, buttons, etc.), 3) The user must select one of those actions as the next step that leads to 
the end goal, and 4) The user would perform that action and evaluate system feedback for 
evidence that progress is being made.  
 
For each action needed to accomplish a task, the evaluator will need to answer 4 questions: 1) 
Will the user try and achieve the desired effect, 2) Will the user notice that the correct action is 
available, 3) Will the user associate the correct action with the desired effect, and 4) Will the 
user notice that progress is being made toward the final goal. If there are positive answers to all 
of the questions, then it can be determined that there are not any usability concerns at this 
stage. 
 
Instrument: The evaluator will be asked to use the scenarios included in the attached document 
(see attachment “VIP HANA user scenarios with embedded symptom reports 05.05.17”) as a 
guide to the goals and sub-goals that users of the system will have to successfully complete. 
The expert will be asked to document each step that they deem necessary to successfully 
complete the tasks in the user scenarios and then answer the four questions above for each of 
the tasks identified in the user scenarios.  
 
Data Analysis: Answers to questions from the cognitive walkthrough will be evaluated to 
determine if any potential usability concerns have been identified. Specific comments and 
recommendations will be compared to those identified in the heuristic testing and any concerns 
will then be aggregated and summarized for presentation to system developers for 
improvements. For any negative responses to the questions, specific concerns and 

recommendations will be recorded.  
 
Usability Testing with PLWHA: After the initial two usability tests, we will perform usability testing 
with PLWHA to identify violations of usability principles and any potential obstacle to their 
effective use of the VIP-HANA system. This is an iterative process that involves testing the 
system and then using the findings to change it to better meet users' needs. 
Sample: We will recruit 20 PLWHA with HANA conditions to participate in the evaluation of the 
VIP-HANA system.  
Procedures: Initially, participants will be provided a brief explanation of the app and what it is capable of 
doing and then will be provided with the same set of scenarios detailing ways in which they might 
use the VIP-HANA app as well as the example versions of the symptom reports that are 
associated with the user scenarios (see attachment “VIP HANA user scenarios with embedded 
symptom reports 05.05.17”). Each participant will be asked to perform tasks which closely mirror 
the intended end use of the application. The process will be recorded using Morae software™ 
(Techsmith Corporation, Okemos, MI) (TechSmith, 1995) which allows the researcher to record 
and analyze the audio recording and screen shots that are captured. After the usability 
evaluation, participants will be asked to rate the prototype’s perceived ease of use and 
perceived potential usefulness using standardized questionnaires. The instruments are 
described below and are included in the Appendix. 
Instruments: We will measure self-reported ease of use and usability with the Health Information 
Technology (IT) Usability Evaluation Scale (Health-ITUES) (Davis, 1989; Yen, Wantland, & 
Bakken, 2010). This tool varies from most traditional measurement scales in that it is designed to 
support customization at the item level to match the specific task/expectation and health IT 
system while retaining standardization at the construct level. The Health-ITUES supports 
evaluation of three levels of task/expectation: user-system, user-system-task, and user-system- 
task-environment. Schnall has published on the usefulness of the Health-ITUES for evaluating 
the usability of mobile health technology (Brown, Yen, Rojas, & Schnall, 2013). End users will be 
asked the following four questions regarding the symptom report: 1) What are your initial 
thoughts when you look at the symptom report? 2) How can we modify the report to more 
appropriately meet your needs? 3) How can we modify these reports to effectively share 
information? 4) Do you have any additional feedback or comments that may help us to improve 
these reports? Please feel free to create any additional sketches that you think would help 
communicate your ideas.  
Data Analysis: The analysis will be based on the Morae recordings of user sessions, 
transcriptions, notes, and the user surveys. The mean task performance time will be calculated. 
The research assistant under the guidance of Dr. Schnall will search for critical incidents 
(Andersson & Nilsson, 1964), an event that indicates something (positive or negative) about 
usability and review these in detail using Morae. These incidents and users’ written comments 
will be summarized. Content analysis, a technique for making replicative and valid inferences 
from data, will be performed. The comments will be categorized according to the positive 
characteristics, negative characteristics, and recommendations made by the usability testers. 
Survey data will be analyzed using SPSS version 20.0 (IBM, Armonk, NY). Descriptive statistics 
will be calculated to complement the findings from the usability assessment. The result of these 
analyses will inform refinements to the VIP-HANA system for subsequent use in the RCT (Aim 
2). The research team is experienced with these methods (Lee et al., 2011; Schnall, Kaufman, et 

al., 2009; Sheehan et al., 2012). 
Specific aim 2: Compare the efficacy of VIP-HANA to a control arm for ameliorating 
symptom frequency and intensity and secondary health outcomes in 100 PLWHA with 
HANA conditions over 6 months. 
Setting and Sample 
Eligibility Criteria: 
In order to participate in the RCT participants must 
1) 
 Be diagnosed with HIV/AIDS 
2) 
Self-report of at least one of the following co-morbid/HANA conditions: cardiovascular 
disease, liver disease, asthma, bronchitis, COPD, osteoporosis, diabetes, arthritis 
and/or renal failure 
3) 
Age of 18 years or older 
4) 
Able to provide written informed consent 
5) 
Able to communicate in English 
6) 
Report at least 8 symptoms. Possible symptoms include: fatigue or loss of energy, 
sadness/depression, muscle aches/pains, difficulty falling asleep, difficulty staying asleep, 
nervousness/anxiety, difficulty remembering, pain/numbness/tingling in hands or feet, 
trouble concentrating, decreased sex drive, diarrhea/loose bowel movement, difficulty 
achieving or maintaining an erection (males only), shortness of breath, 
constipation/bloating/gas, thirst/dry mouth, difficulty with balance/clumsiness/dropping 
things frequently, unplanned weight changes, dizziness/lightheadedness, heartburn/acid 
reflux, dry eyes, changes in appetite, ringing in ears or intolerance to noise, cough, 
nausea/vomiting, fevers/chills, sweats, difficulty with urination, speech difficulties, pain or 
discomfort during sex 
7) 
Taking antiretroviral therapy (ART). 
 
While we anticipate that HANA conditions will be more prevalent in persons over age 50, we 
have not limited our inclusion criteria to this age range. We are collecting participants’ age and 
will include that as a covariate in our final analysis. If our results in Aim 1 demonstrate a gross 
difference in symptom reporting and self-care strategies related to age of participants then the 
study team and external advisory board will meet and consider revising the inclusion criterion of 
age for the RCT. 
Exclusion criteria include: pregnancy, unable to understand the consent procedure, and self- 
reporting no symptoms within the previous week or concurrent participation in a mobile app 
study for PLWHA including text messaging studies and any clinical problems that would not 
allow someone to use a cell phone or fulfill study procedures including meeting the criteria for 
dementia. Prior to participating in any study procedure, all potential participants must voluntarily 
provide informed consent. 
Sample: We will recruit 5 PLWHA with HANA conditions to participate in a pilot and 100 PLWHA 
with HANA conditions to participate in a 6-month trial (see recruitment plan). They will be told the 
purpose of the study is to improve the quality of life of PLWHA. Sample size and statistical power 
calculations are based on the hypothesis test of improvement of symptom status after the use of 
VIP-HANA and on the comparison of the effect of VIP-HANA to the control group. 
The intervention group will have access to VIP-HANA, a web app that provides self-care 
strategies tailored to symptom reporting, HANA condition(s) and gender. Both the control and 

intervention groups will receive reminders to complete their symptom assessments every week. 
Details of the difference between the study arms can be found in Table 2. 
 
 
Table 2. Detailed Comparison of Intervention vs. Control Group 
Intervention 
Control 
VIP-HANA: Self-care strategies tailored to symptom 
reporting, HANA condition(s) and gender 
 
Symptom assessment every  week 
 
 
Secure messaging reminders to complete symptom 
assessment 
 
 
 
The criterion for significance (alpha) has been set at 0.05 for the two-sided test. We will recruit 
100 PLWHA with at least 1 HANA condition for the trial with a proposed sample size of 50 in each 
of the two groups. To account for a potential attrition rate of 20% by the end of study (week 26 or 
month 6 after the intervention), we propose to achieve a final sample of 80 patients (40 per 
group). Additionally, each patient will have a 20% chance to miss at least one of the symptom 
reporting points between week 2 and week 24 after the start of the intervention. We believe the 
20% attrition assumption is conservative since our feasibility study had a 20% attrition rate we 
did not provide study participants with smartphones and data plans (Schnall, Wantland, Velez, 
Cato, & Jia, 2014). 
The power calculation for this study is not based on the feasibility study effect size as this was a 
pre-post study design and does not present equivalency to this study design. The purpose of the 
feasibility study was to show that participants are willing to use the system and that there was 
some meaningful change in symptom reporting after use of the VIP system (Bowen et al., 2009). 
The power analysis for this study is calculated to detect a clinically meaningful change in 
symptom frequency. For symptom frequency outcomes, 100 subjects gives more than 80.7% 
power to detect 13% decrease in symptom frequency (for example, from an average 8 
symptoms to an average 7 symptoms) from the baseline to the end of study (i.e., 26th week). 
For the comparison between the control group and the VIP-HANA group, 100 subjects gives 
82.4% power to yield a statistically significant result (assuming an intention-to- treat principle for 
the analysis) to test 10% difference in VIP-HANA over the control group (i.e., a decrease in 
symptom frequency). For the symptom intensity outcomes, the proposed sample yields at least 
92.3% power to detect decrease in symptom intensity score of a small effect size of 0.1 from the 
baseline to the end of study. For the comparison of the two groups, this study will have about 
87.4% power to yield a statistically significant result to test a small effect size of 0.24 difference 
in symptom intensity score for the VIP-HANA group over the control group.  
 
Procedures 
 
Recruitment Plan: We will post flyers at local clinics and community based organizations 
throughout New York City. Potential participants will be pre-screened for eligibility over the 
phone by study staff.  In order to ensure that an eligible participant arrives at their visit, they will 
receive a confirmation email for their baseline visit which will be generated by REDCap.   
 
Pilot: We will conduct a recruitment pilot that enrolls 5 PLWHA with at least 1 HANA condition 

before the 6-month trial. This pilot will allow us to determine ease and feasibility of recruiting 
PLHWA that have HANA conditions, as well as examining any issues there may be in retention 
and adherence with regard to usage of the VIP-HANA app. Those enrolled in the pilot will follow 
the same procedure (randomization, data collection) as those enrolled in the 6-month trial. 
 
Trial: We will enroll 100 PLWHA with HANA conditions in a 6 month trial with a baseline, 3 
month and 6 month measurement. 
Randomization: This will be a single-blinded study, with subjects blinded to study group 
assignment. Patients will be randomized (1:1) to VIP-HANA or an attention control group. Both 
groups will receive secure messaging reminders via their smartphone and will also be asked to 
report their symptoms on their smartphone via a web app every week. Participants in the VIP-
HANA (intervention) group will also receive symptom strategies tailored to symptom reporting, 
HANA condition(s) and   gender. Details of the differences between the intervention and control 
group can be found above in Table 5 (see Section D.1.b). 
Data collection: We will have two levels of data collection. 
1) Participants will report on their symptom frequency and intensity every week on the VIP-
HANA app. Participants in both groups will receive reminder messages to report symptoms 
every week via the revised Sign and Symptom Check-List for HIV delivered via email. If a 
participant has not completed the checklist in more than 3 weeks, he/she will receive a reminder 
email or phone call. Intervention arm participants will complete the revised Sign and Symptom 
Check-List for HIV every week and receive tailored self-care strategies based on their symptom 
reporting, HANA condition, and gender. Control arm participants will complete the revised Sign 
and Symptom Check-List for HIV, but will NOT receive strategies. 
2) Participants will complete all secondary study-related questionnaires (see outcome measures 
below) via an electronic survey at baseline, at a 3-month follow-up visit (after 12 weeks), and at the 
end of the study at a 6-month follow-up visit (week 26). Participants will complete the study 
questionnaires at our study site. The survey will be housed on Qualtrics software which is a web-
based survey program freely available to Columbia University Medical Center employees. 
Qualtrics provides a secure HIPAA protected environment. Participants will also complete physical 
measures at all 3 time points.  Participants’ height, weight, grip strength, and time taken to walk 15 feet 
will be measured.  We will also abstract information from their medical records, such as CD4 counts and 
viral load, as indicated in the grant submission. In addition all participants will complete demographic 
information and other related measures including PHQ-2 (Hirshfield et al., 2008; Kroenke, 
Spitzer, & Williams, 2003), GAD-2 (Kroenke, Spitzer, Williams, Monahan, & Lowe, 2007), HIV 
stigma (Berger, Ferrans, & Lashley, 2001; Bunn, Solomon, Miller, & Forehand, 2007), health 
literacy (Weiss et al., 2005), eHealth literacy (Norman & Skinner, 2006), and perceived social 
support (Brock, Sarason, Sarason, & Pierce, 1996; Kalichman et al., 2006). In response to the 
reviewers’ concern, all participants will complete the social desirability scale (Reynolds, 1982) 
which will be included in our final analytic model. 
 
Participants will receive $25 for completion of study questionnaires at baseline, $35 at the 3-
month follow-up visit, and $40 at the 6-month follow-up visit for completing the assessment 
tools.  Each week that a participant completes a session of the VIP HANA app, they will be 
given $5 (26 weeks x $5/week = $130).  If a participant does not miss more than 5 sessions, 
they will earn a bonus of $20.  Participants are eligible for up to $250.   
 
Outcome Measures. Our primary outcome measures, symptom frequency and symptom 
intensity, will be measured at baseline and every week for 26 weeks. We used the same study-

related questionnaires in our feasibility trial. Based on our feasibility trial, symptom reporting 
takes about 10 minutes to complete. The remainder of the assessment tools will either be 
collected only at the 3-month follow-up and the 6-month follow-up or baseline, 3-month follow-up, 
and 6-month follow-up. Completion of all surveys takes about 30 minutes (see Table 6 and the 
Appendix for details on the surveys). The UCSF School of Nursing Symptom Management 
Model (The University of California, 1994) is based on the premise that effective management of 
symptoms demands that three dimensions are considered: Symptom Experience, Self-Care 
Symptom Strategies, and the Outcomes (Dodd et al., 2001). The interrelatedness of these three 
dimensions of symptom management is not consistently taken into account in research (Figure 
5) (Lenz, Pugh, Milligan, Gift, & Suppe, 1997). 
Outcome Measures for Aim 2 RCT is listed in Table 3 below: 
Primary outcome: The primary purpose of the intervention is to improve symptom status. We 
hypothesize that VIP-HANA will improve symptom frequency and intensity, our primary 
outcomes. 
Secondary outcomes: The potential benefits of the intervention are quite broad, so multiple 
secondary outcomes are of interest (see Table 3). As a result of improving symptom status and 
frequency among PLWH, several other factors that contribute to a subject's overall well-being 
may be affected. Changes in symptom status should have an effect on secondary health 
outcomes including quality of life, frailty, adherence to ART, and system use. Importantly, 
symptom burden has been identified as a predictor of ART adherence (Gay et al., 2011). Unlike 
treatments for other illnesses, ART medications are more likely to contribute to greater 
discomfort and treatment discontinuation (Chesney, Morin, & Sherr, 2000). As a result it is not 
surprising that only 25% of PLWHA in the US are virally suppressed. Therefore symptom 
management is of critical importance for improving clinical outcomes in PLWHA and of public 
health importance (Services, 2014). 
 
 
 
Table 3. Outcome Measures 
Outcomes 
Tool 
Collection Time 
Primary: 
Symptom Status 
The revised Sign and Symptom Check- 
List for HIV (SSC-HIVrev)(Holzemer, 
Hudson, Kirksey, Hamilton, & Bakken, 
2001) 
Every 1 week 
Secondary Outcomes 
Quality of Life 
Medical Outcomes Study Standard 
Form (SF 12) (Ware, Kosinski, & 
Keller, 1996) 
Week 0, 3-month follow-
up, 6-month follow-up 
Engagement with 
Health care Provider 
Engagement with Health care Provider 
Scale (Bakken et al., 2000) 
Number of healthcare visits 
Medical Record 

CD4, Viral Load 
Clinical outcomes 
Abstraction 
 
ART Adherence 
Visual Analogue Scale Self-report 
(VAS) (Arnsten et al., 2001; Chesney, 
Ickovics, et al., 2000; De Costa et al., 
2010; Walsh, Mandalia, & Gazzard, 
2002)  
Week 0, 3-month follow-
up, 6-month follow--up 
Frailty 
Fried’s Frailty Phenotype (Brothers 
et al., 2014; Fried et al., 2001); Used 
in numerous HIV-related studies 
(Desquilbet et al., 2007; Desquilbet 
et al., 2009; Piggott et al., 2013) 
System Use 
Log Files 
Automated 
 
We are paying particular attention to the symptom experience of the study participants over time, 
the components of the symptom management strategies (e.g. content and frequency of use) 
through log files, and symptom status as our primary outcome but multiple secondary outcomes 
are also of interest. Of particular relevance, one of the assumptions of the model is that once 
symptoms have been successfully addressed then the secondary outcomes should show 
improvement. Findings from our RCT will inform this hypothesis. 
Data Management and Analysis: All survey data will be stored on a secure server. All 
multivariate analyses will be preceded by standard descriptive statistics to profile participants in 
each study group and to examine distributions of all outcome variables and to identify outliers. 
Descriptive statistics (i.e., means, standard deviations, frequencies, and percentages) will be 
used to examine demographic characteristics of the sample and the frequency and intensity of 
the symptoms. We will use intention-to-treat principles for the primary outcome analysis. 
 
Intention to treat implies all subjects are considered in analyses. Linear mixed model and 
generalized linear mixed model (negative binomial model) will be used for the analysis of 
symptom intensity outcomes and symptom frequency outcomes, respectively. Linear and 
generalized linear mixed models are statistical models that are particularly well suited to analyze 
repeated measured symptom data. This method does not require equal waves of data and 
allows missing outcome measures during the course of the study. This is an attractive method for 
longitudinal analyses, where participant attrition can be a problem, because participants often 
drop out and return for later follow-up or drop out entirely. Therefore, it is useful in retaining 
power in statistical analyses. The mixed models incorporate correlation between the reports of 
symptoms as these reports are measured from the same subject repeatedly over time and can 
account for this correlation for the model inference (Ugrinowitsch, Fellingham, & Ricard, 2004). 
An assumption for using linear and generalized linear mixed model methods is that the cases 
missing to subsequent follow-up assessments are missing at random. If missing data is related 
to intervention assignment or symptom outcome, we will estimate the probability of not missing a 
data point (i.e., the probability of reporting symptom) and include the inverse of this probability in 
the model as a selection weighting factor to correct for bias due to missing data.  
 
For analysis of secondary outcomes, if the outcomes are measured more than once, we will use 

linear and generalized linear (logistic or negative binomial model) mixed model. If the outcomes 
are measured only once, we will use linear or generalized linear model. We will also asses any 
net benefits in the patients’ report of their healthcare provider’s engagement with their treatment 
and care, quality of life, frailty, ART adherence and biological outcomes (CD4 and viral load). 
Analysis of additional hypotheses includes assessment of differences in the self- care strategy 
used to help alleviate a symptom and the recorded level of intensity following the strategy use 
will also be assessed. Based on the Symptom Management Model, we anticipate that VIP-HANA 
will be effective at improving symptom status and then we will see a positive change in our 
secondary outcome measures: quality of life, frailty, ART adherence, and CD4/ viral load. 
Specific Aim 3: Understand PLWHA’s perceptions of the predisposing, enabling, and 
reinforcing factors for VIP-HANA use with theoretically-guided focus group sessions. 
The goal of these focus group sessions is to understand the experience of PLWHA using the 
VIP-HANA system. The adaption of this model for HIT was originally developed by Kukafka et al. 
(2003). Dr. Schnall has used this model in previous studies to understand the barriers and 
facilitators to implementation of health information technology systems (Schnall, Clark, Olender, 
& Sperling, 2013; Schnall, Gordon, Camhi, & Bakken, 2011). If VIP-HANA is found to be 
efficacious in our trial, then our follow-up focus groups will inform dissemination and 
implementation of the VIP-HANA intervention. If the intervention is not efficacious then we will 
learn why participants did not find the intervention to be useful or improve their outcomes. 
Sample: Post-intervention, we will conduct five focus group sessions with approximately 6-12 
participants per group. Focus group participants will be drawn from the VIP-HANA group 
(intervention arm) and the attention control group. We anticipate that 60 participants will agree to 
participate in the follow-up focus groups. 
Procedures: The focus groups will be 60-90 minutes in length. We will include $30 
reimbursement for participants’ time. Following completion of the informed consent process, all 
focus group sessions will be audio-recorded. The PI, who has conducted focus groups for a 
number of studies in the past, will convene the groups with the study participants and will act as 
a facilitator (Lee et al., 2011; Okoniewski, Lee, Rodriguez, Schnall, & Low, 2013; Schnall et al., 
2013; Schnall, Kaufman, et al., 2009; Schnall, Odlum, Gordon, & Bakken, 2009). The focus 
group guide will be informed by the Predisposing, Reinforcing, and Enabling Constructs in 
Evaluation (PRECEDE) portion of the PRECEDE- PROCEED Model of health program planning 
and evaluation (Green, Kreuter, Deeds, & Partridge, 1980). The integration of these frameworks 
for application in HIT implementation evaluation has been proposed by a number of authors as a 
strategy for assessing predisposing, enabling, and reinforcing factors for use and acceptance of 
HIT (Kukafka et al., 2003; Schnall et al., 2011). Sample focus group questions are: 1) Describe 
your use of the VIP-HANA system 2) What are some of the ways that your overall health 
benefited through the use of the VIP-HANA system (reinforcing factors)? 3) What were the 
barriers to your use of VIP-HANA in your everyday life? (predisposing factors) 4) What are your 
ideas about strategies for overcoming these barriers? (enabling factors) 5) What did you 
perceive as useful about the VIP-HANA system? (predisposing factors) 6) How would you 
envision VIP-HANA being used in your everyday life? (implementation) 7) How do you imagine 
that VIP-HANA would be shared with others PLWHA? (dissemination) 8) Any there any other 
thoughts about your experience using the VIP-HANA system that you would like to share? 
The team will adhere to qualitative research processes to ensure the credibility, confirmability, 
dependability and transferability of the qualitative data from these analyses. To support            

the credibility of the data, we will conduct peer debriefing and triangulate findings across multiple 
data sources (surveys, focus group data). In addition, we will use “member checks,” i.e., sharing 
of initial data interpretations with participants, to ensure accurate interpretations. 
Triangulation of findings, along with reflexivity, will enhance the confirmability of the 
interpretations. The investigators will carefully record an audit trail and keep extensive field notes 
to facilitate transferability of study findings into other contexts (Guba, 1981). Each focus group 
recording will be transcribed; transcripts will be analyzed independently for content by research 
team members. 
Data Analysis: Field notes and transcripts will be analyzed by the researchers using NVivo™ 
(QSR International, Victoria, Australia) software. Participants’ statements will be captured using 
memoing and then sorted into the following categories of interest: predisposing, enabling, and 
reinforcing. These activities will result in a greater understanding of the use, usefulness, 
dissemination and implementation of VIP-HANA. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

References 
 
Andersson, B.-E., & Nilsson, S.-G. (1964). Studies in the reliability and validity of the critical 
incident technique. Journal of Applied Psychology, 48(6), 398-403. 
doi:10.1037/h0042025 
Arnsten, J. H., Demas, P. A., Farzadegan, H., Grant, R. W., Gourevitch, M. N., Chang, C. J., . . . 
Schoenbaum, E. E. (2001). Antiretroviral therapy adherence and viral suppression in 
HIV-infected drug users: comparison of self-report and electronic monitoring. Clin Infect 
Dis, 33(8), 1417-1423. doi:10.1086/323201 
Bakken, S., Holzemer, W. L., Brown, M. A., Powell-Cope, G. M., Turner, J. G., Inouye, J., . . . 
Corless, I. B. (2000). Relationships between perception of engagement with health care 
provider and demographic characteristics, health status, and adherence to therapeutic 
regimen in persons with HIV/AIDS. AIDS Patient Care STDS, 14(4), 189-197. 
doi:10.1089/108729100317795 
Berger, B. E., Ferrans, C. E., & Lashley, F. R. (2001). Measuring stigma in people with HIV: 
psychometric assessment of the HIV stigma scale. Res Nurs Health, 24(6), 518-529.  
Bertini, E., Gabrielli, S., & Kimani, S. (2006). Appropriating and assessing heuristics for mobile 
computing. Proceedings of the working conference on Advanced visual interfaces, 119-
126. doi:10.1145/1133265.1133291 
Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D., . . . Fernandez, 
M. (2009). How we design feasibility studies. Am J Prev Med, 36(5), 452-457. 
doi:10.1016/j.amepre.2009.02.002 
Bright, T. J., Bakken, S., & Johnson, S. B. (2006). Heuristic evaluation of eNote: an electronic 
notes system. AMIA Annu Symp Proc, 864.  
Brock, D., Sarason, I., Sarason, B., & Pierce, G. (1996). Simultaneous assessment of perceived 
global and relationship-specific support. Journal of Social and Personal Relationships, 
13, 143-152.  
Brothers, T. D., Kirkland, S., Guaraldi, G., Falutz, J., Theou, O., Johnston, B. L., & Rockwood, 
K. (2014). Frailty in people aging with human immunodeficiency virus (HIV) infection. 
Journal of Infectious Diseases. doi:10.1093/infdis/jiu258 
Brown, W., 3rd, Yen, P. Y., Rojas, M., & Schnall, R. (2013). Assessment of the Health IT 
Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) 
technology. J Biomed Inform, 46(6), 1080-1087. doi:10.1016/j.jbi.2013.08.001 
Bunn, J. Y., Solomon, S. E., Miller, C., & Forehand, R. (2007). Measurement of stigma in people 
with HIV: a reexamination of the HIV Stigma Scale. AIDS Educ Prev, 19(3), 198-208. 
doi:10.1521/aeap.2007.19.3.198 
Chesney, M. A., Ickovics, J. R., Chambers, D. B., Gifford, A. L., Neidig, J., Zwickl, B., & Wu, A. 
W. (2000). Self-reported adherence to antiretroviral medications among participants in 
HIV clinical trials: the AACTG adherence instruments. Patient Care Committee & 
Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials 
Group (AACTG). AIDS Care, 12(3), 255-266. doi:10.1080/09540120050042891 
Chesney, M. A., Morin, M., & Sherr, L. (2000). Adherence to HIV combination therapy. Soc Sci 
Med, 50(11), 1599-1605.  
Chiasson, M. A., Hirshfield, S., Humberstone, M., Difilippi, J., Koblin, B. A., & Remien, R. H. 
(2005). Increased high risk sexual behavior after September 11 in men who have sex 
with men: an Internet survey. Arch Sex Behav, 34(5), 527-535. doi:10.1007/s10508-005-
6278-5 
Chiasson, M. A., Shaw, F. S., Humberstone, M., Hirshfield, S., & Hartel, D. (2009). Increased 
HIV disclosure three months after an online video intervention for men who have sex 
with men (MSM). AIDS Care, 21(9), 1081-1089. doi:10.1080/09540120902730013 
Corless, I. B., Bunch, E. H., Kemppainen, J. K., Holzemer, W. L., Nokes, K. M., Eller, L. S., . . . 

Chou, F. Y. (2002). Self-care for fatigue in patients With HIV. Oncol Nurs Forum, 29(5), 
E60-69. doi:10.1188/02.ONF.E60-E69 
Crothers, K., Butt, A. A., Gibert, C. L., Rodriguez-Barradas, M. C., Crystal, S., Justice, A. C., & 
Veterans Aging Cohort 5 Project, T. (2006). Increased COPD among HIV-positive 
compared to HIV-negative veterans. Chest, 130(5), 1326-1333. 
doi:10.1378/chest.130.5.1326 
Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of 
information technology. MIS Quarterly, 13(3), 319-340.  
De Costa, A., Shet, A., Kumarasamy, N., Ashorn, P., Eriksson, B., Bogg, L., & Diwan, V. K. 
(2010). Design of a randomized trial to evaluate the influence of mobile phone reminders 
on adherence to first line antiretroviral treatment in South India--the HIVIND study 
protocol. BMC Med Res Methodol, 10(25). doi:1471-2288-10-25 
Desquilbet, L., Jacobson, L. P., Fried, L. P., Phair, J. P., Jamieson, B. D., Holloway, M., . . . 
Multicenter, A. C. S. (2007). HIV-1 infection is associated with an earlier occurrence of a 
phenotype related to frailty. J Gerontol A Biol Sci Med Sci, 62(11), 1279-1286.  
Desquilbet, L., Margolick, J. B., Fried, L. P., Phair, J. P., Jamieson, B. D., Holloway, M., . . . 
Multicenter, A. C. S. (2009). Relationship between a frailty-related phenotype and 
progressive deterioration of the immune system in HIV-infected men. J Acquir Immune 
Defic Syndr, 50(3), 299-306. doi:10.1097/QAI.0b013e3181945eb0 
Dodd, M., Janson, S., Facione, N., Faucett, J., Froelicher, E. S., Humphreys, J., . . . Taylor, D. 
(2001). Advancing the science of symptom management. J Adv Nurs, 33(5), 668-676.  
Endpoints and other measures in a multisite HIV prevention trial: rationale and psychometric 
properties. NIMH Multisite HIV Prevention Trial. (1997). AIDS, 11 Suppl 2, S37-47.  
Fried, L. P., Tangen, C. M., Walston, J., Newman, A. B., Hirsch, C., Gottdiener, J., . . . 
Cardiovascular Health Study Collaborative Research, G. (2001). Frailty in older adults: 
evidence for a phenotype. J Gerontol A Biol Sci Med Sci, 56(3), M146-156.  
Gay, C., Portillo, C. J., Kelly, R., Coggins, T., Davis, H., Aouizerat, B. E., . . . Lee, K. A. (2011). 
Self-reported medication adherence and symptom experience in adults with HIV. J 
Assoc Nurses AIDS Care, 22(4), 257-268. doi:10.1016/j.jana.2010.11.004 
Gibson, L., Gaffey, E., & Spendlove, J. (2003). The Usability Factor: More than just a Pretty 
Interface.  
Gordon, P., Camhi, E., Hesse, R., Odlum, M., Schnall, R., Rodriguez, M., . . . Bakken, S. 
(2012). Processes and outcomes of developing a continuity of care document for use as 
a personal health record by people living with HIV/AIDS in New York City. Int J Med 
Inform, 81(10), e63-73. doi:10.1016/j.ijmedinf.2012.06.004 
Green, L., Kreuter, M., Deeds, S., & Partridge, K. (1980). Health education planning: A 
diagnostic approach.  
Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. 
Educational Communication and Technology Journal, 29(2), 75-91.  
Hirshfield, S., Remien, R. H., Humberstone, M., Walavalkar, I., & Chiasson, M. A. (2004). 
Substance use and high-risk sex among men who have sex with men: a national online 
study in the USA. AIDS Care, 16(8), 1036-1047. doi:10.1080/09540120412331292525 
Hirshfield, S., Remien, R. H., Walavalkar, I., & Chiasson, M. A. (2004). Crystal 
methamphetamine use predicts incident STD infection among men who have sex with 
men recruited online: a nested case-control study. J Med Internet Res, 6(4), e41. 
doi:10.2196/jmir.6.4.e41 
Hirshfield, S., Wolitski, R. J., Chiasson, M. A., Remien, R. H., Humberstone, M., & Wong, T. 
(2008). Screening for depressive symptoms in an online sample of men who have sex 
with men. AIDS Care, 20(8), 904-910. doi:10.1080/09540120701796892 
Holzemer, W. L., Hudson, A., Kirksey, K. M., Hamilton, M. J., & Bakken, S. (2001). The revised 
Sign and Symptom Check-List for HIV (SSC-HIVrev). J Assoc Nurses AIDS Care, 12(5), 

60-70.  
Jaspers, M. W. (2009). A comparison of usability methods for testing interactive health 
technologies: methodological aspects and empirical evidence. Int J Med Inform, 78(5), 
340-353. doi:10.1016/j.ijmedinf.2008.10.002 
Justice, A., & Falutz, J. (2014). Aging and HIV: an evolving understanding. Curr Opin HIV AIDS, 
9(4), 291-293. doi:10.1097/COH.0000000000000081 
Justice, A. C. (2010). HIV and aging: time for a new paradigm. Curr HIV/AIDS Rep, 7(2), 69-76. 
doi:10.1007/s11904-010-0041-9 
Justice, A. C. (2014). PLWHA with HANA conditions.  
Justice, A. C., McGinnis, K. A., Atkinson, J. H., Heaton, R. K., Young, C., Sadek, J., . . . 
Veterans Aging Cohort 5-Site Study Project, T. (2004). Psychiatric and neurocognitive 
disorders among HIV-positive and negative veterans in care: Veterans Aging Cohort 
Five-Site Study. AIDS, 18 Suppl 1, S49-59.  
Kalichman, S. C., Cherry, C., Cain, D., Pope, H., Kalichman, M., Eaton, L., . . . Benotsch, E. G. 
(2006). Internet-based health information consumer skills intervention for people living 
with HIV/AIDS. J Consult Clin Psychol, 74(3), 545-554. doi:10.1037/0022-006X.74.3.545 
Kendall, C. E., Wong, J., Taljaard, M., Glazier, R. H., Hogg, W., Younger, J., & Manuel, D. G. 
(2014). A cross-sectional, population-based study measuring comorbidity among people 
living with HIV in Ontario. BMC Public Health, 14, 161. doi:10.1186/1471-2458-14-161 
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2003). The Patient Health Questionnaire-2: validity 
of a two-item depression screener. Med Care, 41(11), 1284-1292. 
doi:10.1097/01.MLR.0000093487.78664.3C 
Kroenke, K., Spitzer, R. L., Williams, J. B., Monahan, P. O., & Lowe, B. (2007). Anxiety 
disorders in primary care: prevalence, impairment, comorbidity, and detection. Ann 
Intern Med, 146(5), 317-325.  
Kukafka, R., Johnson, S. B., Linfante, A., & Allegrante, J. P. (2003). Grounding a new 
information technology implementation framework in behavioral science: a systematic 
analysis of the literature on IT use. J Biomed Inform, 36(3), 218-227.  
Lee, Y., Johnson, S. B., Trembowelski, S., Schnall, R., Sheehan, B., & Bakken, S. (2011). 
Cognitive walkthrough evaluation of an investigator profiles directory. Proc AMIA Annu 
Fall Symp.  
Lenz, E. R., Pugh, L. C., Milligan, R. A., Gift, A., & Suppe, F. (1997). The middle-range theory of 
unpleasant symptoms: an update. ANS Adv Nurs Sci, 19(3), 14-27.  
Messeri, P., Lee, G., & Berk, S. (2009). Prevalence of Chronic Diseases & Comorbid Conditions 
in the CHAIN Cohort of PLWHA.  
Nielsen, J. (1994). Heuristic evaluation. In J. M. Nielsen, R. L. (Ed.), Usability Inspection 
Methods: John Wiley & Sons. 
Nielsen, J. (2005). 10 Usability Heuristics for User Interface Design. Retrieved from 
https://www.nngroup.com/articles/ten-usability-heuristics/ 
Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Paper presented at the 
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 
Seattle, Washington, USA.  
Norman, C. D., & Skinner, H. A. (2006). eHEALS: The eHealth Literacy Scale. J Med Internet 
Res, 8(4), e27. doi:10.2196/jmir.8.4.e27 
Okoniewski, A. E., Lee, Y. J., Rodriguez, M., Schnall, R., & Low, A. F. (2013). Health 
Information Seeking Behaviors of Ethnically Diverse Adolescents. J Immigr Minor 
Health. doi:10.1007/s10903-013-9803-y 
Parsons, J. T., Halkitis, P. N., Wolitski, R. J., & Gomez, C. A. (2003). Correlates of sexual risk 
behaviors among HIV-positive men who have sex with men. AIDS Educ Prev, 15(5), 
383-400.  
Parsons, J. T., Missildine, W., Van Ora, J., Purcell, D. W., Gomez, C. A., & Seropositive Urban 

Drug Injectors, S. (2004). HIV serostatus disclosure to sexual partners among HIV-
positive injection drug users. AIDS Patient Care STDS, 18(8), 457-469. 
doi:10.1089/1087291041703683 
Parsons, J. T., Schrimshaw, E. W., Bimbi, D. S., Wolitski, R. J., Gomez, C. A., & Halkitis, P. N. 
(2005). Consistent, inconsistent, and non-disclosure to casual sexual partners among 
HIV-seropositive gay and bisexual men. AIDS, 19 Suppl 1, S87-97.  
Parsons, J. T., Severino, J., Nanin, J., Punzalan, J. C., von Sternberg, K., Missildine, W., & 
Frost, D. (2006). Positive, negative, unknown: assumptions of HIV status among HIV-
positive men who have sex with men. AIDS Educ Prev, 18(2), 139-149. 
doi:10.1521/aeap.2006.18.2.139 
Parsons, J. T., VanOra, J., Missildine, W., Purcell, D. W., & Gomez, C. A. (2004). Positive and 
negative consequences of HIV disclosure among seropositive injection drug users. AIDS 
Educ Prev, 16(5), 459-475. doi:10.1521/aeap.16.5.459.48741 
Piggott, D. A., Muzaale, A. D., Mehta, S. H., Brown, T. T., Patel, K. V., Leng, S. X., & Kirk, G. D. 
(2013). Frailty, HIV infection, and mortality in an aging cohort of injection drug users. 
PLoS One, 8(1), e54910. doi:10.1371/journal.pone.0054910 
Reynolds, W. M. (1982). Development of reliable and valid short forms of the marlowe-crowne 
social desirability scale. Journal of Clinical Psychology, 38(1), 119-125.  
Schnall, R., Clark, S., Olender, S., & Sperling, J. D. (2013). Providers' perceptions of the factors 
influencing the implementation of the New York State mandatory HIV testing law in two 
Urban academic emergency departments. Acad Emerg Med, 20(3), 279-286. 
doi:10.1111/acem.12084 
Schnall, R., Gordon, P., Camhi, E., & Bakken, S. (2011). Perceptions of factors influencing use 
of an electronic record for case management of persons living with HIV. AIDS Care, 
23(3), 357-365. doi:933810483  
Schnall, R., Kaufman, D., Hyun, S., & Bakken, S. (2009). Cognitive Evaluation of Case 
Managers’ Use of a Continuity of Care Document for Persons Living with HIV. Paper 
presented at the AMIA Annual Symposium, San Francisco, CA.  
Schnall, R., Odlum, M., Gordon, P., & Bakken, S. (2009). Barriers to implementation of a 
Continuity of Care Record (CCR) in HIV/AIDS care. Stud Health Technol Inform, 146, 
248-252.  
Schnall, R., Wantland, D., Velez, O., Cato, K., & Jia, H. (2014). Feasibility testing of a web-
based symptom self-management system for persons living with HIV. J Assoc Nurses 
AIDS Care, 25(4), 364-371. doi:10.1016/j.jana.2013.09.002 
Schueller, S. M., Begale, M., Penedo, F. J., & Mohr, D. C. (2014). Purple: a modular system for 
developing and deploying behavioral intervention technologies. J Med Internet Res, 
16(7), e181. doi:10.2196/jmir.3376 
Services, U. S. D. o. H. a. H. (2014). HIV CARE CONTINUUM. Retrieved from 
https://www.aids.gov/federal-resources/policies/care-continuum/ 
Sheehan, B., Lee, Y., Rodriguez, M., Tiase, V., & Schnall, R. (2012). A comparison of usability 
factors of four mobile devices for accessing healthcare information by adolescents. Appl 
Clin Inform, 3(4), 356-366. doi:10.4338/ACI-2012-06-RA-0021 
So-Armah, K. A., & Freiberg, M. S. (2014). Cardiovascular disease risk in an aging HIV 
population: not just a question of biology. Current Opinion in HIV and AIDS, 9(4), 346-
354 doi:10.1097/COH.0000000000000065 
Tan, W.-s., Lui, D., & Bishu, R. (2009). Web evaluation: Heuristic evaluation vs. user testing. 
International Journal of Industrial Ergonomics, 39(4), 621-627. 
doi:10.1016/j.ergon.2008.02.012 
TechSmith. (1995). Morae Usability and Web Site Testing. Retrieved from 
https://www.techsmith.com/morae.html 
The University of California, S. F. S. o. N. S. M. F. G. (1994). A model for symptom 

management. Image J Nurs Sch, 26(4), 272- 
276.  
Ugrinowitsch, C., Fellingham, G. W., & Ricard, M. D. (2004). Limitations of ordinary least 
squares models in analyzing repeated measures data. Med Sci Sports Exerc, 36(12), 
2144-2148.  
Walsh, J. C., Mandalia, S., & Gazzard, B. G. (2002). Responses to a 1 month self-report on 
adherence to antiretroviral therapy are consistent with electronic data and virological 
treatment outcome. AIDS, 16(2), 269-277.  
Ware, J., Jr., Kosinski, M., & Keller, S. D. (1996). A 12-Item Short-Form Health Survey: 
construction of scales and preliminary tests of reliability and validity. Med Care, 34(3), 
220-233.  
Weiss, B. D., Mays, M. Z., Martz, W., Castro, K. M., DeWalt, D. A., Pignone, M. P., . . . Hale, F. 
A. (2005). Quick assessment of literacy in primary care: the newest vital sign. Ann Fam 
Med, 3(6), 514-522. doi:10.1370/afm.405 
Yen, P. Y., Wantland, D., & Bakken, S. (2010). Development of a Customizable Health IT 
Usability Evaluation Scale. AMIA Annu Symp Proc, 2010, 917-921.