The Wise App Trial for Improving Health Outcomes in People Living With HIV
Phase: NA
Conditions: HIV/AIDS
Sponsor: Columbia University

 
 
 
 
 
 
 
 
 
 
 
 
Title: The Wise App Trial for Improving Health Outcomes in PLWH 
NCT Number: NCT03205982    
Document Date: 6/17/2021 

 
 
 
The United States (US) HIV epidemic continues to be a significant public health challenge affecting 1.2 
million Americans 1. Persons living with HIV (PLWH) confront a range of psychological and behavioral 
challenges, including adherence to medication regimens, accessing healthcare services, changes in quality of 
life, stigma, uncertainty about physical and psychological decline, and death 2. At the same time, the US HIV 
epidemic is highly concentrated demographically in racial and ethnic minorities 3-5. 
Antiretroviral Therapy (ART) adherence is central to therapeutic success for PLWH. Strict 
adherence to ART is key to sustained HIV suppression, reduced risk of drug resistance, improved overall 
health, quality of life, and survival 6 as well as decreased risk of HIV transmission 7-10. Achieving adherence to 
ART is a critical determinant of long-term health outcomes in HIV-infected patients. For many chronic 
diseases, such as diabetes or hypertension, drug regimens remain effective even after treatment is resumed 
following a period of interruption. In the case of HIV, however, loss of virologic control as a consequence of 
non-adherence to ART may lead to emergence of drug resistance and loss of future treatment options. 
Developing effective interventions to enhance ART adherence is essential. Currently only about 25% of PLWH 
in the US are virally suppressed, supporting the urgent need for interventions to engage people in the HIV care 
continuum and improve ART adherence 11.The shift to simpler regimens alone is insufficient for optimal 
adherence 12. 
mHealth is the use of mobile devices -- such as cellular phones, tablets, smartphones and other 
wireless devices -- for the delivery of health services and information 13. The ubiquitous nature of mobile 
technologies in daily life creates opportunities for health behavior management tools that were not 
previously possible 14. mHealth can rapidly assess and modify health-related behavior and transform 
patients’ decision-making about their health 13. Smartphones can house mHealth applications (apps) designed 
to be used by patients and providers for diagnostics, behavioral prompts, reminders, illness monitoring and 
self-management programs that extend far beyond the limits of a physical clinic. The potential for information 
and communication technology, such as mobile apps, to enhance medication adherence through the provision 
of support (information, education, reminders, etc.) for behavior change has been well-documented over the 
last decade 15-22. A growing body of research confirms the benefits of empowering healthcare consumers with 
information and decision-making support 23-25. Patient participation in their health leads to increased patient 
satisfaction as well as positive changes in adherence patterns translating into improved clinical outcomes 26-28. 
Individuals who use mobile apps to manage their health may perceive these tools as more private, potentially 
increasing patients' willingness to disclose non-adherence and seek support tools 29,30. 
mHealth also has the potential to bridge a divide in healthcare delivery among underserved 
racial and ethnic minority groups. The use of mobile technology has made a huge impact on 
communication, access, and information/resource provision to minority and underserved populations 31. Nearly 
2/3 of Americans are now smartphone owners, particularly those from racial/ethnic minority and low 
socioeconomic groups 32. Furthermore, emerging evidence suggests that underserved populations use 
smartphones as their primary method for accessing the Internet 33. The use of mHealth can reduce geographic 
and economic disparities and personalize healthcare 34,35, which can be particularly relevant to PLWH since a 
majority of them are from underserved and minority groups 3-5. 
mHealth has the potential to be an efficacious tool for improving the health of PLWH. mHealth 
tools can promote the management and prevention of chronic illnesses, such as HIV 36 and have the potential 
to address many of the healthcare needs of PLWH including adherence to HIV medications and retention in 
HIV care and treatment. Medication reminders delivered via text messaging have been shown to improve ART 
adherence in a number of studies 37. 
Self-report of medication adherence is often criticized since it typically overestimated adherences 
especially in unmasked trials 38. Our study will use real-time, wireless monitoring strategies via the 
CleverCapTM Lite dispenser, for measuring ART adherence. This will overcome this often cited 
limitation and provide novel opportunities to proactively prevent virologic rebound and treatment failure. In 
contrast, current adherence assessments, such as patient recall, pill counts and pharmacy refill data, typically 
detect missed doses long after they occur. All of these methods are assessed on an intermittent basis, such 
that missed doses are detected several weeks to months after they occur, which can lead to treatment failure 
and drug resistance 39. Real-time medication adherence monitoring can allow for the detection of adherence 
lapses and initiation of interventions to resume treatment prior to the development of virologic rebound and 
drug resistance, which has not previously been possible 40. 
STUDY PURPOSE AND RATIONALE 

CleverCapTM Lite, a smart pill box container, (Compliance Meds Technologies, North Miami Beach, 
Florida) communicates dosing behavior in real-time by transmission of a patient identifier and date-time stamp 
over existing cellular networks when the container is opened to take medications. Because cellular network 
coverage is becoming ubiquitous, the technical infrastructure now exists for real-time adherence monitoring in 
resource-limited settings. A similar device, called The Wisepill container, was pilot tested in a small sample 
(N=10) of PLWH in Africa and shown to be useful for providing real-time adherence counseling in PLWH 41. 
The Wisepill has also been successfully used to monitor adherence to anti-depressants by Dr. Mohr (Co-I) on 
this proposed project 42. 
The Wise App will motivate and engage participants over time. Although mHealth technology may 
be promising, several studies have shown that adherence to mHealth interventions is low and decreases over 
time 43, even though these studies showed a significant effect on health outcomes 44. Therefore there is a need 
for strategies that motivate people to use, and keep using, the technologies offered. Gamification, the use of 
game-like rewards and incentives, paired with desired behaviors, is one such strategy that can be used to 
increase motivation to sustain habits of individuals over time 45. Companies have widely accepted and adopted 
gamification as a means to increase initiation and retention of desired behaviors 46. In fact, it has been 
estimated that 60% of health initiatives in workplaces now include gamification elements 47. PatternHealth 
Technology’s HealthStar mobile app, referred throughout this protocol as “The Wise App,” will use persuasive 
games to motivate end-users 48. End-users can set health and fitness goals and use persuasive gaming to help 
meet the goal. In the case of the Wise App, participants will set goals, have a tally bar, and receive reminders to 
motivate them to complete their goals. 
mHealth tools, including mobile apps, for PLWH have not been well-developed or evaluated. 
Currently there are a number of mHealth apps for PLWH 49 but few have been developed by using information 
gathered from potential end-users (PLWH) or healthcare providers regarding desired content and features of a 
mobile app. Of the limited number of studies specifically focused on mobile apps for PLWH, one study identified 
the preferences for a mobile app in HIV positive young mothers 50, but did not rigorously evaluate the app    
after its development. In another study, researchers developed a mobile app consisting of a music program      
to improve adherence to ART for adult PLWH, but did not engage the intended end-users in the development  
of the intervention 51. Although mobile phone apps are increasingly being used for the care of HIV and         
other sexually transmitted diseases, most available apps have failed to attract user attention and positive 
reviews. In a review of existing HIV apps, researchers found that apps were infrequently downloaded (median 
100 to 500 downloads) and not highly rated (an average customer rating 3.7 out of 5 stars). Based on this 2012 
review, less than 0.3% of the more than 29,000 health-related apps available for iPhone and Android 
consumers were dedicated to HIV/STD information and prevention49. 
Our proposed study is timely and relevant since there is currently a dearth of evidence on the use of 
mHealth interventions for improving health outcomes in PLWH. Of the relevant research including text 
messaging to support HIV care, studies have largely been developed and tested outside the US in low and 
middle income countries 52. Two systematic reviews on the use of text messaging for treatment adherence in 
PLWH have been conducted, but the studies were focused on populations outside the US 37,52. In another 
systematic review on the use of health tools for PLWH, all of the US studies in the review focused on the use 
of mobile technology as a data collection tool for targeting smoking cessation in PLWH 53. 
Given the dearth of useful and likeable apps 54, the need for improving medication adherence in PLWH 
and the great promise that mHealth holds, we propose to test a smartphone app linked to a smart pill box 
called CleverCapTM Lite (The Wise App) for PLWH targeting ART adherence and guided by a rigorous and 
relevant theoretical framework55. The feasibility of the Wise App is supported by data demonstrating that use 
of mobile devices is nearly ubiquitous in the U.S. with persons from underserved groups being more likely to 
own a smartphone and more likely to use a smartphone (as compared to a computer) to access the Internet. 
The current trial is significant in representing one of the first principled and systematic efforts to build a  
mHealth app intervention based on user-centered design work and linked to a smart pill box for improving 
health outcomes. Our proposed study, based on our empirical findings from our preliminary work 56-61, uses 
rigorous methods and a theory-based approach 55 to develop and test a mHealth intervention (Wise App) for 
improving ART adherence for PLWH. 
 
 
 
The goal of this study is to test the efficacy of The Wise App for improving health outcomes (as measured by 
improving ART adherence) in PLWH. Our secondary outcomes are: (a) healthcare access (b) technology 
STUDY DESIGN 

acceptance (c) technology use. Table 1 presents and overview of the design, participants and data analysis 
plan for each aim. 
 
 
Table 1. Overview of Design, Methods, Participants, and Data Analysis 
Aim Design/Methods 
Participants 
Data Analysis 
1 
Cognitive 
Walkthrough, and 
user testing using 
think-aloud protocol 
Cognitive Walkthrough: human- 
computer interaction experts 
(N=5); User testing: PLWH (N=30) 
Quantitative and qualitative summary of heuristic 
violations; thematic analysis of think-aloud 
protocol, quantitative summary of mouse clicks, 
time, etc. 
2 
Randomized 
controlled design 
PLWH 
(N=200; intervention=100, control 
n=100) for 6 months 
Descriptive statistics; Linear regression model 
2b 
In-depth Interviews 
PLWH 
(N=100; intervention group only) 
Qualitative coding of in-depth interviews related 
to CleverCap device; thematic analysis to 
examine CleverCap experience and acceptability 
3 
Descriptive: focus 
groups 
Focus groups: 
Participants in the intervention arm 
of the RCT (N~50) 
Descriptive, thematic analysis to examine post- 
intervention perceptions related to The Wise App 
use, usefulness and impact on overall health 
3b 
Descriptive: focus 
groups 
Focus groups: 
Participants in the control arm of 
the RCT (N~60) 
Descriptive, thematic analysis to examine post- 
intervention perceptions related to the Fitness 
Tracker use, usefulness and impact on overall 
health 
3c 
Descriptive: in-
depth interview 
In-depth interview: 
Participants in the intervention and 
control arms of the RCT (N~20) 
Descriptive, thematic analysis to examine 
perceptions related to the recruitment process 
and research participation to the RCT  
 
C. Aim 1: Build a functional app for HIV self-management linked to a smart pill (Wise App) for PLWH 
and assess its usability 
 
C.1. Intervention Components. 
Medication adherence can be monitored using the CleverCapTM Lite bottle, which emits a cellular signal when 
it is opened, allowing for near real-time acquisition of data regarding patient pill taking activity. The 
CleverCapTM Lite adherence monitor (see Compliance Meds Technologies, North Miami Beach, Florida) 
communicates dosing behavior in real-time by transmission of a patient identifier and date-time stamp over 
existing cellular networks when the container is opened to take medications. 
 
The primary intervention component will include: 
• 
Medication Reminders – Reminders will be tailored to participants’ 
medication schedule (Figure 1). To avoid notification fatigue, a cellular signal 
is sent to the phone when the servers detect a pill bottle opening, which 
prevents the launching of the reminder notifications on the phone. The app will 
launch an in-phone pop-up alert before a scheduled dose time is missed. If the 
Wise App does not receive notification that the pill bottle has been opened, 
additional reminders will be launched on the phone after the scheduled     
dose, asking the patient to taken their medication. 
Additional app components are listed below and organized according to 
Fogg’s Functional Triad: 
Figure 1. Medication 
Schedule 

 
Figure 2. Mobile App 
Screen of User 
History 
• Missions (Tool) – Missions are intended to help users keep track of activities they enjoy. 
This feature will display a checklist of tasks participants aim to complete (e.g. go to the 
gym, text/email a friend, etc.) 
• Fitness Tracker (Medium) – This feature will allow users to track their daily steps. Each 
user will receive a fitness tracker, which will be synced to the Wise App. A daily step goal 
will be set for all users and progress will be tracked on the app, allowing users to see the 
effects of their physical activity. 
• History (Medium) – Informed by the CleverCapTM Lite dispenser, users will be able to 
view their medication adherence over time through a graphical representation (Figure 2). 
The CleverCapTM Lite dispenser uses cellular technology to transmit information on time of 
pill bottle opening. Users will also be able to view trends of step goals and other app 
features. 
• Games (Medium) – Wise App supports two types of gamification: 1) A virtual pet that 
reacts to a user’s medication adherence, and 2) Challenges that can be designed to 
award/deduct points to users based on select activities completed (Figure 3). 
 
• 
Chat (Social Actor): This feature will allow study participants to anonymously chat 
with other users under specific discussion topics. To ensure privacy protection, 
only study participants will be a part of this chat and all posts will need to be 
approved by an app administrator prior to being posted. 
• 
Testimonials of Lived Experiences (Social Actor) – Videos of PLWH who share their 
experiences which has the potential to provide social support.Videos will include 
Figure 3. Mobile App 
Screen of Challenges 
topics such as a new diagnosis, living with HIV for 20 years or longer, and disclosing 
HIV status. 
After the Wise App has been built, we will conduct usability testing at the 
Columbia University School of Nursing. 
 
C.2. Usability Testing. 
The goal of usability testing is to improve the design and increase the likelihood of 
technology acceptance. To achieve this goal we will evaluate the user interface of the 
Wise App developed at Pattern Health Technologies. We will conduct two types of 
usability assessments: A) Cognitive Walkthrough and B) End-User Usability Testing. 
C.2.1. Cognitive Walkthrough: 
 
The purpose of a CW evaluation is to evaluate the ease with which users can perform a task with little or no 
formal instruction or informal coaching.62 The CW has historically been conducted by experts or researchers63 
and it is highly structured with a goal of understanding a technology’s learnability.64 A CW session’s input 
should include 1) a detailed design of the user interface 2) a task scenario 3) explicit assumptions about the 
user population 4) the context of use and 5) a sequence of actions that would allow the user to successfully 
complete the task.62 For the purposes of this study, we chose the tasks outlined in the procedures section to 
analyze. 
There are four steps in the context of a task that the user will need to complete using scenarios: 1) The user 
would set an end goal to be accomplished, 2) The user would inspect available actions on the user screen 
(e.g. menu items, buttons, etc.), 3) The user would select one of those actions as the next step that leads to 
the end goal, and 4) The user would perform the action and evaluate system feedback for evidence that 
progress is being made toward accomplishment of the user’s goal. 
 
Sample: Five informaticians will be recruited as usability experts with training in human-computer interaction 
and who have published in the field of informatics. We will recruit the evaluators through direct contact from the 
Informatics Departments at Columbia University and/or Weill Cornell Medical College, both of which have large 
cadres of informatics researchers. 
 
Procedures: Each of usability experts will be provided with a beta version of the Wise App, a Clever Cap, and 
a fitness tracker to conduct the study. Each expert will be provided with a set of user scenarios utilizing the 

Wise App. The expert will be asked to simulate a novice user and assess a user interface by analyzing the 
cognitive processes a user would go through to effectively accomplish each of the high-level tasks within the 
app. While they are performing the tasks required for the scenarios, the expert will be asked to say aloud what 
they are thinking, seeing and trying to do and will be asked to explicitly document each step that they deem 
necessary to successfully complete each task in the user scenarios, if needed. The expert will then asked to 
answer aloud the following four questions for each of the tasks identified in the user scenarios: 1) Will the user 
try and achieve the desired effect?, 2) Will the user notice that the correct action is available?, 3) Will the user 
associate the correct action with the desired effect?, and 4) Will the user notice that progress is being made 
toward the final goal?. The process will be recorded using Morae software™ (Techsmith Corporation, Okemos, 
MI),65 which allows the researcher to record and analyze the audio recording and screen shots that are 
captured during the heuristic evaluation. Participants will also complete an electronic survey. Participants will be 
compensated $150.00 for their time. 
 
Data Analysis: Verbal commentary, answers to the four questions from the CW, and screen-recordings from 
the Morae software will be reviewed and evaluated to determine if any potential usability concerns have been 
identified. If there are positive answers to all of the questions, then it can be determined that there are not any 
usability concerns at this stage; if there is a negative answer to any of the four questions, the specified action is 
not free of usability problems.64 Specific comments and recommendations will be discussed by the researchers 
and any concerns will then be aggregated and summarized for presentation to system developers for 
improvements. 
C.2.2. End-User Usability Testing: 
We will conduct usability testing to examine task performance by our end-users, persons living with HIV 
(PLWH). It is intended to identify usability problems and any potential obstacle to their effective use of the Wise 
App. This is an iterative process that involves testing the system and then using the test results to change it to 
better meet users’ needs. 
 
Sample: 30 PLWH will be recruited to identify usability concerns within the Wise App. Of the 30 PLWH, 15 will 
be iPhone users and 15 will be Android users. In usability testing the minimum percentage of problems 
identified rose from 82% up to 95% when the number of users was increased from 10 to 20.66 
 
Procedures: Participants will be provided with a beta version of the Wise App, a Clever Cap, and a fitness 
tracker. They will be also provided with the same set of user scenarios used for CW by experts. Each 
participant will be asked to perform tasks which should closely mirror the intended end users of the app. Each 
participant will be also encouraged to verbalize their thoughts about the tasks they are performing. The 
process will be audio- and screen-recorded using Morae software™ (Techsmith Corporation, Okemos, MI).65 
After the usability evaluation, participants will be asked to complete an online survey including a demographic 
questionnaire and user satisfaction assessment via Qualtrics®. Participants will be compensated $25.00 for 
their time. 
 
Data Analysis: The analysis will be based on the Morae recordings of user sessions, transcriptions, and notes. 
Dr. Schnall will search for critical incidents which will be characterized by comments, silence, and repetitive 
actions. Dr. Schnall will review these incidents in detail using Morae software. The incidents identified and      
the users’ written comments will be summarized. Content analysis, a technique for making replicative and    
valid inferences from data, will be performed by the research assistant under Dr. Schnall’s supervision. The 
comments will be categorized according to the positive characteristics, negative characteristics, and 
recommendations made by PLWH. Results from the user satisfaction assessment will be analyzed using SPSS 
(IBM, Armonk, NY) to calculate the descriptive statistics to complement the findings from the usability 
assessment in which the PLWH will be using the Wise App. Using the findings from these activities, we will 
refine the components of the app for use in the RCT (Aim 2). 
 
D. Aim 2: Evaluate the impact of the Wise App on medication adherence in underserved PLWH 

D.1. Design Overview. 
A randomized controlled trial will be conducted with 200 PLWH over 6 months. Participants will be randomly 
assigned to the Wise App (intervention) or a control arm. 
Details on the differences between the two groups can be 
found in Table 2. 
D.2. Eligibility Criteria for Participants. 
Participants must be at least 18 years old, have a diagnosis of 
HIV, speak and understand English or Spanish, and live in the 
US. Participants must have a smartphone and be on ART 
medications. Our participants will all be Medicaid eligible and 
most of our participants, as can be seen from our preliminary 
work, are from racial and ethnic minority groups. 
To comply with the AHRQ Policy on the Inclusion of 
Priority Populations (NOT-HS-03-010), our study participants 
are chronically ill, inner-city, low-income and minority. We anticipate that about 40% of our study participants 
will include women. 
In addition, participants must report past 30 days adherence of 80% or less as measured using the 
Visual Analogue Scale (VAS) 67 (Appendix) or have a viral load of over 400 copies/mL. Exclusion criteria are: 
participation in any other mobile app study for PLWH, not including text messaging studies, and any clinical 
problems that would not allow someone to use a cell phone. Prior to participating in any study procedure, 
participants must voluntarily provide informed consent. 
D.3. Recruitment Plan. 
We will post flyers at each of the recruitment sites listed below and will provide compensation for time that it 
takes to complete the study questionnaires. The flyer will have the name, contact number, and e-mail for the 
study coordinator. PLWH from these sites are almost exclusively from low socioeconomic backgrounds. If we 
are unable to recruit participants after posting flyers, we will then hand distribute flyers to potential participants 
in the waiting rooms of our sites. We will also be using advertisements through social media that will lead 
users to a short Qualtrics survey, The Wellness Study Interest Survey, asking for contact information and 
some eligibility criteria. The purpose of this short survey is to attract more individuals towards our study, 
provide them with necessary study-related information, and to determine initial eligibility before a staff 
member reaches back out to screen for enrollment eligibility. Our main sites for recruitment are those where 
we have previously conducted our studies (HRSA H97HA08483, NINR P30NR010677-03S1 and CDC 
U01PS003715-01).  
1) The Comprehensive Health Program at Columbia University/NewYork-Presbyterian Hospital provides 
primary care, HIV specialty care and care coordination services to over 2,000 PLWH. 95% of individuals are 
of either Black or Latino background and women represent half of the overall population. In addition, the 
program has   28 direct clinical providers, 8 social workers, and 10 additional care coordinators. Of note, Dr. 
Olender (Co-I) is an attending physician at the Clinic. 
2) AIDS Service Center (ASC) is a community based organization for PLWH and persons at risk for HIV. ASC 
has more than 90 staff, 85 peer interns, and more than 1,800 clients who come for services each year. In 
addition, another 18,000 people are reached through the peer education and community outreach initiatives 
of ASC. Given our past experience and success in recruiting participants and using the recruitment strategies 
described above, we are confident that we will be able to recruit the number of participants needed to conduct 
the study within the study timeframe. See details in Section A.3.3. Preliminary Work. 
 
 
 
D.4. Power and Sample Size. 
We will recruit 200 individuals and have greater than 80% power to detect a less than 10% difference in 
adherence to ART medications between the Wise App and the control group. We assume a 25% attrition 
rate by the end of trial for both arms. The power calculations are based on the analysis of main outcome, 
ART adherence. All calculations are based on 2-sided test with alpha at 0.05 level. Our calculations are 
based on the assumption that each person is on a once daily regimen and the adherence rate is ≤ 80% at 
baseline. 
Many of our participants will be on regimens that are two or three times per day which will provide greater 
power for our study since we have a greater number of measurements (e.g., 180 for once daily vs. 360 for 
twice daily regimen). To illustrate, if the adherence rate for the control group does not change, we will have 
90.7% power to detect a 6% increase in ART adherence for the intervention group (e.g., from 80% to 86% at 
STATISTICAL PROCEDURES 
Table 2. Comparison of Intervention vs. 
Control Group 
Intervention 
Control 
The Wise App 
• 
Medication 
reminder 
• 
Step Goal 
reminder 
✓ 
✓ 
✓ 
✓ 
CleverCapTM Lite 
dispenser 
✓ 
✓ 

the end of the trial). If the adherence rate for the control group increases 4%, from 80% to 84% at the end of 
the trial, we will have 88.6% power to detect a 5% difference in adherence between the intervention and 
control group. In this case, the intervention group’s adherence rate increased an additional 5% (or 
4%+5%=9%) from 80% to 89% at the end of the trial. 
D.5. Baseline Screening and Assessment. 
Potential study participants will complete a phone screening that assesses eligibility. If eligible, participants will 
attend a baseline session described below. 
D.6. Randomization. 
Study participants will be randomized (1:1) to Wise App intervention or control arm. We will use a variable 
permuted randomized block design 68 where the block size itself is randomly selected (i.e., blocks of four to 
eight). The advantage of the permuted block design is that treatment assignment is pre-determined before the 
trial begins and then assignment remains static throughout the course of the trial 69. Participants will be 
randomized based on the use of computer-generated random numbers at baseline. The randomization 
database will be stored on a password protected computer at Columbia University and will only be accessible 
to Dr. Schnall. 
D.7. Procedures. 
Upon enrollment in the study, participants will come to the study site, Columbia University School of Nursing to 
complete consent forms. Following informed consent, study participants will complete a number of baseline 
questionnaires including socio-demographic characteristics, physical health measurements, behavioral data, 
health status and social desirability (See Table 3) 70. Survey instruments will be collected through Qualtrics, a 
secure, Web-based application designed to support data capture for research studies, providing an intuitive 
interface, audit trails, and automated export. Qualtrics is a free service offered through Columbia University 
Medical Center. 
Following the completion of the baseline study instruments, study participants in both arms will be given a 
CleverCapTM Lite dispenser and trained on how to use it, as well as a fitness tracker that will be synced to 
Wise App. Study participants in both arms will be trained by study team members on how to use the different 
features of the Wise App. Those in the intervention group will receive medication reminders while those in the 
control group will receive reminders about completing their step goal. Participants in both arms will be given a 
document that includes contact information for the project team, information about compensation, study 
reminders, and will also receive a telephone call from a research assistant 1 week after enrollment to assist  
with any technical problems with the app or the CleverCapTM Lite dispenser and biweekly thereafter. 
Participants will be texted appointment reminders for study visits via Qualtrics. Hard to reach participants will 
also be sent scheduling reminders through the Qualtrics system and encrypted e-mails from the study’s 
secure CUMC e-mail address.  
Token of appreciation: Participants will be given $40 at the baseline visit, $50 + up to an extra $25 for app 
challenges completed at the 3 month visit, and $60 + up to an extra $25 for app challenges completed at the 6 
month visit (up to a total of $200 for participation) and for providing a blood sample (or recent lab results) for 
viral load/ CD4 testing and completion of study questionnaires. Participants will also have the option to have 
the cost of their transportation to and from the study site paid for, through a ridesharing service. 
D.8. Operationalization of Outcome Measures. 
The primary outcome will be ART adherence, measured by the CleverCapTM dispenser. The CleverCapTM 
dispenser will automatically record each time a participant opens the dispenser. We will collect adherence data 
each day from the start to the end of trial (day 1 to 6 months), and it is a count response (number of times 
taking medication each day). Frequency of app use will be counted each time a participant opens the app. 
 
Table 3. Study Measures 
Tool 
Measurement Time Points 
Demographic Variables and Potential Confounders 
Socio- 
demographic 
characteristics 
Gender, age, education, income, employment, health insurance, 
housing 
Baseline 
Behavioral Data 
Use of condoms, tobacco, alcohol, substance use 
Baseline, 3 and 6 months 
Health Literacy 
Newest Vital Sign71; Short Test of Functional Health Literacy in 
Adults (S-TOFHLA) 72 
Baseline 
Health Status 
RAND-36; Symptom Distress Module;73 PROMIS-29;74 Self- 
Management Scale75   Baecke Questionnaire for the Elderly76 
Baseline, 3 and 6 months 
Social 
Desirability 
Social desirability scale70 
Baseline 

Mental Health 
Patient Health Questionnaire (PHQ-9);77 Mood Disorder 
Questionnaire (MDQ);78 Prodromal Questionnaire (PQ-16);79 APA 
Severity Measures for Generalized Anxiety Disorder80 
3 and 6 months 

 
Physical Health 
Measures 
Height, weight, hip, and waist measurements 
Baseline, 3 and 6 months 
Housing Status 
Housing Status Assessment Tool 
Baseline, 3 and 6 months 
Primary Outcome 
ART Adherence 
Primary Measure: 
CleverCapTM Lite Dispenser 
Daily 
CASE Adherence Index81 
Baseline, 3 and 6 months 
CD4 and viral load 
Baseline, 3 and 6 months 
Secondary Outcomes 
Health Care 
Access 
Number of Primary Care Visits 
Baseline, 3 and 6 months 
Engagement with Health care Provider Scale 
Baseline, 3 and 6 months 
Caregiver Survey 
6 months 
System Use 
Automated Log Files 
Ongoing 
Technology 
Acceptance 
Health-ITUES;82 PSSUQ;83 Perceived Ease of Use and Potential 
Usefulness Questionnaire84; eHEALS: the Health Literacy Scale85 
Trust in Health Information Sources 
Baseline, 3 and 6 months 
Physical Activity 
Measures 
Neighborhood Environment Survey, Social Capital Scale, & Self- 
Efficacy Scale 
6-months 
 
We will also use additional measurements of medication adherence to validate the findings from the 
CleverCapTM Lite dispenser. We will do blood draws at baseline, 3 and 6 months to measure CD4 and viral 
load data on each study participant; alternatively, participants will be given the option of providing recent lab 
results of their CD4 and viral load data The blood draw at baseline will also be tested for genotyping, GSA chip 
and qPCR, Hormone blood levels (Estrogen and FSH), and cytokine blood levels (IL-1β, IL-2, IL-4, IL-6, IL-8, 
IL-10, IL-13, TNFα, and CRP). The genotyping with GSA chip and qPCR, cytokine blood level and CRP, and 
hormone blood tests will only be collected once during the course of the study, at Baseline or at the 6-month 
follow-up if not completed at Baseline. The genotyping, GSA chip and qPCR will be performed by the Feinstein 
Institute for Medical Research at Northwell Health. We will also ask participants to complete the Center for 
Adherence Support Evaluation (CASE) Adherence Index, a simple composite measure of self-reported 
antiretroviral therapy (ART) adherence 81. The CASE Adherence Index consists of three unique adherence 
questions. Past research has shown that the CASE Adherence Index correlated strongly with the three-day 
self-reported adherence data (p<0.001) and was more strongly associated with HIV outcomes, including a 1-
log decline in HIV RNA level (maximum OR = 2.34; p<0.05), HIV RNA < 400 copies/ml (maximum OR = 2.33; 
p<0.05) and performed as well as the three-day self-report when predicting CD4 count status. The CASE 
Adherence Index is an easy to administer instrument that provides an alternative method for assessing ART 
adherence in clinical settings. Items are scored such that higher values indicate better adherence, and the 
maximum total score      is 16. Scores of 11 or higher on this index indicate good adherence (Cronbach’s α= 
0.79) 86. 
Our primary outcome is ART adherence. The CleverCapTM Lite dispenser is our objective measure of 
ART adherence and the CASE Adherence index as a subjective measure. We are including the subjective 
measure since there is potential bias in that participants’ in the control group may not use the device or may 
not use the device for the entire study period. 
We will assess for contamination between the study groups at the end of the trial by asking control 
group participants whether they had used any HIV or medication related apps in the past 6 months. In addition, 
participants will return at 3 and 6 months to complete the remainder of the measures listed in Table 3 and 
included in the Appendix. We will also collect data on app use through automated log files so there will not be 
response burden associated with this measure. 
Secondary Outcome Measures: Our measures are listed in Table 3 and included in the Appendix. 
D.9. Data Analysis. 
We will use intention-to-treat principles for the primary outcome analysis. Intention to treat implies all subjects 
are considered in analyses. The outcome variable y_i will be the total number of times the CleverCapTM Lite 
dispenser is opened at day t (from day 1 to day 180) for person i, so y_i follows binomial distribution with 
parameter of p_i and n where p is probability of taking medication (adherence rate) and n is number of 
medications to take each day. We use a generalized linear mixed model (GLMM) with logit link function to 
analyze adherence rate. The basic form of the model is: log (p_i/(1-p_i ))=β_0+β_1×t+β_2×t×INT+μ_i. In the 
model, μ_i is person level random effect and follows normal distribution. This model is also called individual 
growth model. Independent variable INT is an indicator for intervention group (INT=1 for the intervention group 
and INT=0 for the control group). Regression parameter β_1 is change in adherence rate for the control group 
and β_2 is the difference in change in adherence rate between the intervention group and the control group. 

This model can include personal level factors as covariates so we can test for potential confounding by 
covariates (i.e., age, gender, and health literacy) in this model and inspecting for changes in the point estimate 
of the relation between study group and adherence. 
We will use GLMM for analyzing secondary outcomes. In our analysis of the surveys related to the SDT 
framework, the variable time (t) will be at 3 time points (baseline, 3 month, and 6 month) and the link function 
will be identified for continuous outcomes. In the analysis of app use, the unit of analysis will be at daily level for 
each participant, and system use will be analyzed using a GMMM with log link function (Poisson model). 
For the analysis of CD4 and viral load, we will analyze difference-in-difference using GLMM with identity or log 
link function when these variables are treated as continuous outcomes. Viral load will also be treated as a 
binary outcome (detectable vs. undetectable) and analyses of viral suppression will use the GLMM with logit 
link. In a secondary “as treated” analyses, missing adherence and viral load data will be ignored. The genotype 
analysis will be run by the Feinstein Institute for Medical Research at Northwell Health. Finally, we  will assess the 
relationship between adherence (independent variable) and virologic suppression (dependent variable) using 
GLMM logistic regression and including missing viral load as detectable. This model will include personal 
level factors as covariates so we can test for potential confounding by covariates (i.e., age, gender, and 
health literacy) in this model and inspecting for changes in the point estimate of the relation between study 
group and adherence. 
 
E. Aim 2b: Assess the acceptability of the CleverCapTM dispenser through In-depth interviews, which 
will focus on the individual experience of PLWH with the device. 
 
E.1. Sample. At the 3-month follow-up visit, semi-structured in-depth interviews will take place with participants 
in the intervention group. These participants will be the same participants from the intervention study arm, if 
they choose to participate (N=100) 
 
E.2. Procedures. The in-depth interviews will be approximately 20-30 minutes in length. This will be an 
optional activity for participants in the intervention group and no additional compensation will be provided. 
Once participants complete the informed consent process, all interviews will be audio-recorded. The in-depth 
interviews will be conducted by the study coordinator and research assistants, the members responsible for 
conducting follow-up visits. The brief interview guide will be informed by past studies focusing on the use and 
acceptability of electronic pill dispensers.87-89 This interview will aim to fill gaps in the literature by taking a 
qualitative approach to understanding how our population of interest engages with the CleverCap device, 
which captures our primary measure for change in medication adherence. Some of the questions the 
questionnaire will include are: 1) Please tell me a little bit about your experience using the electronic pill bottle, 
2) What have you used in the past for storing and receiving reminders about your medications? How does this 
compare? 3) What are some of the things that you like about using your pill bottle? 4) What do you dislike 
about the pill bottle? 5) How has the pill bottle changed how you take your medications 6) If a friend asked you 
about your experience using the clevercap, how would you describe your experience so far? and 7) At the 
completion of this study, would you want to keep using the device, why or why not? 
 
The team will adhere to qualitative research processes to ensure the credibility, confirmability, 
dependability, and transferability of the qualitative data from these analyses90. To support the credibility of the 
data, we will conduct peer debriefing and triangulate findings across multiple data sources (surveys, focus 
group data). In addition, we will use “member checks,” i.e., sharing of initial data interpretations with participants 
to ensure accurate interpretations. Triangulation of findings, along with reflexivity, will enhance the confirmability 
of the interpretations. The investigators will carefully record an audit trail and keep extensive field notes           
to facilitate transferability of study findings into other contexts. 
 
E.3. Data Analysis. All in-depth interviews will be transcribed verbatim and then coded. The development and 
application of a coding scheme is an integral component of the data analysis process. It enables the systematic 
examination and interpretation of the data related to the primary analytic foci. The coding scheme is 
conceptualized as a multilevel structure. At the highest level are the primary analytic foci coded as headings. 
Specific aspects or dimensions of the headings are assigned core codes. Specific aspects or dimensions of the 
core codes are assigned sub codes. We use ATLAS.ti, a software program for qualitative analysis, to facilitate 
the analysis. 
The following 7 steps will be used to develop the coding scheme: 
Step 1: Identify the principal issues discussed by interviewees. 

Step 2: Construct definitions of the primary analytic themes. 
Step 3: Develop and apply core codes and sub-codes to the initial set of interviews. 
Step 4: Develop a provisional coding scheme. 
Step 5: Test and refine the provisional coding scheme. 
Step 6: Reconcile coding differences and construct an updated and final coding scheme. 
Step 7: Apply the coding scheme to the full data set and assess inter-coder reliability. 
After all transcripts have been coded, we will extract and examine the content of text segments linked to core 
codes and sub-codes relevant to understanding the technology acceptance of the CleverCap and barriers and 
facilitators to its use. Based on the coded data, we will propose ways in which certain themes are analytically 
related. A careful examination of the coded text will reveal the associations among these themes, and may lead 
to more refined data searches. Once we establish patterns of relationships among themes and issues, we      
will identify participants’ accounts that support or refute these patterns. Identifying and accounting for cases that 
deviate from an interpretative pattern enables us to test and confirm the pattern’s validity and robustness. 
Schnall has done extensive work understanding technology acceptance with particular emphasis on end-users’ 
needs 91-94 
 
E.4. Triangulation of Findings. The research team will use the findings from our quantitative data analysis of 
medication adherence and technology acceptance in Aim 2 to understand use and perceived usefulness of the 
CleverCap. Field notes and transcripts will be analyzed by the researchers using NVivo™ (QSR International, 
Victoria, Australia) software. Participants’ statements will be captured using memoing and then sorted into the 
following categories of interest: predisposing, enabling and reinforcing. These activities will result in a greater 
understanding of the use of the CleverCap based upon the predisposing, enabling, and reinforcing factors 
identified in the in-depth interviews. Each interview recording will be transcribed; transcripts will be analyzed by 
Dr. Schnall, who is experienced in qualitative analysis and a research assistant. The research team is 
experienced with these methods 91,95-97. 
 
F. Aim 3: Assess PLWH perceptions of the predisposing, enabling, and reinforcing factors for Wise 
App use through theoretically-guided focus group sessions. 
F.1. Sample. Post-intervention, we will conduct five focus group sessions with approximately 8-10 participants 
per group. Focus group participants will be drawn from the intervention study arm (N~50). 
F.2. Procedures. The focus groups will be 60-90 minutes in length. We will include $30 reimbursement for 
participants’ time. Following completion of the informed consent process, all focus group sessions will be 
audio-recorded. The PI, who has conducted focus groups for a number of studies in the past, will convene the 
groups with the study participants and will act as a facilitator 93,96,98,99. The focus group guide will be informed 
by the predisposing, reinforcing, and Enabling Constructs in Evaluation (PRECEDE) portion of the PRECEDE- 
PROCEED Model of health program planning and evaluation 100. The integration of these frameworks for 
application in HIT implementation evaluation has been proposed by a number of authors as a strategy for 
assessing predisposing, enabling, and reinforcing factors for use and acceptance of HIT 97,99,101.The following 
structured questions related to Wise App use will be posed: 1) What are some of the ways that your overall 
health might benefit through the implementation and dissemination of the Wise App (reinforcing factors)? 2) 
What do you perceive as barriers to implementation and use of the Wise App? (predisposing factors) 3) What 
do you perceive as barriers to implementation and use of the Wise App? (predisposing factors) 4) What are 
your ideas about strategies for overcoming these barriers? (enabling factors). 
The team will adhere to qualitative research processes to ensure the credibility, confirmability, 
dependability, and transferability of the qualitative data from these analyses90. To support the credibility of the 
data, we will conduct peer debriefing and triangulate findings across multiple data sources (surveys, focus 
group data). In addition, we will use “member checks,” i.e., sharing of initial data interpretations with participants 
to ensure accurate interpretations. Triangulation of findings, along with reflexivity, will enhance the confirmability 
of the interpretations. The investigators will carefully record an audit trail and keep extensive field notes           
to facilitate transferability of study findings into other contexts 
F.3. Data Analysis. All focus groups will be transcribed verbatim and then coded. The development and 
application of a coding scheme is an integral component of the data analysis process. It enables the systematic 
examination and interpretation of the data related to the primary analytic foci. The coding scheme is 
conceptualized as a multilevel structure. At the highest level are the primary analytic foci coded as headings. 
Specific aspects or dimensions of the headings are assigned core codes. Specific aspects or dimensions of the 

core codes are assigned sub codes. We use ATLAS.ti, a software program for qualitative analysis, to facilitate 
the analysis. 
The following 7 steps will be used to develop the coding scheme: 
Step 1: Identify the principal issues discussed by participants. 
Step 2: Construct definitions of the primary analytic themes. 
Step 3: Develop and apply core codes and sub-codes to the initial set of interviews. 
Step 4: Develop a provisional coding scheme. 
Step 5: Test and refine the provisional coding scheme. 
Step 6: Reconcile coding differences and construct an updated and final coding scheme. 
Step 7: Apply the coding scheme to the full data set and assess inter-coder reliability. 
After all transcripts have been coded, we will extract and examine the content of text segments linked to core 
codes and sub-codes relevant to understanding the technology acceptance of the Wise App and barriers and 
facilitators to its use. Based on the coded data, we will propose ways in which certain themes are analytically 
related. A careful examination of the coded text will reveal the associations among these themes, and may lead 
to more refined data searches. Once we establish patterns of relationships among themes and issues, we      
will identify participants’ accounts that support or refute these patterns. Identifying and accounting for cases that 
deviate from an interpretative pattern enables us to test and confirm the pattern’s validity and robustness. 
Schnall has done extensive work understanding technology acceptance with particular emphasis on end-users’ 
needs 91-94 
F.4. Triangulation of Findings. The research team will use the findings from our quantitative data analysis of 
medication adherence and technology acceptance in Aim 2 to understand use and perceived usefulness of the 
Wise App. For example, if a participant uses the app component of appointment reminders but does not score 
well on the Health Care Climate Questionnaire, then further exploration through the focus group sessions will 
be conducted to determine whether the appointment reminder component is improving health behaviors. 
Field notes and transcripts will be analyzed by the researchers using NVivo™ (QSR International, Victoria, 
Australia) software. Participants’ statements will be captured using memoing and then sorted into the following 
categories of interest: predisposing, enabling and reinforcing. These activities will result in a greater 
understanding of the use of the Wise App based upon the predisposing, enabling, and reinforcing factors 
identified in the focus groups. Each focus group recording will be transcribed; transcripts will be analyzed by 
Dr. Schnall, who is experienced in qualitative analysis and a research assistant. The research team is 
experienced with these methods 91,95-97. 
 
G. Aim 3b: Assess PLWH perceptions, experience, and satisfaction on the use of the Fitness Tracker 
that was synced to the HealthStar app through focus group sessions. 
 
G.1. Sample. Post-intervention, we will conduct up to eight focus group sessions with approximately 8-12 
participants per group. Focus group participants will be drawn from the control study arm (N~60). 
 
G.2. Procedures. The focus groups will be 60-90 minutes in length. We will include $30 reimbursement for 
participants’ time. Following completion of the informed consent process, all focus group sessions will be 
audio-recorded. The PI who have conducted focus groups for a number of studies in the past, and her 
research team members who are pre/post docs will convene the groups with the study participants and will act 
as a facilitator 93,96,98,99. This will be an optional activity for participants in the control group. This focus group  
will be to understand participant’s experience, perception and satisfaction on the use of the fitness tracker 
which was synced to the HealthStar app during the 6 month duration of the study. Some of the prompts in the 
focus group include: 1) What are some of the things that you liked about using the fitness tracker syncing to the 
Healthstar app? 2) Please describe your experience using the fitness tracker. 3) Please describe your 
experience monitoring your daily activity. 4) How do you think that your physical activity (i.e., daily step goal 
completion) changed after using the fitness tracker for 6 months? 5) What are some of the ways that your 
physical activity may change through the use of the HealthStar app linked with the fitness tracker? (Focus 
groups guide included as attachments entitled “WiseApp Fit Bit Focus Group Guide“) 
The team will adhere to qualitative research processes to ensure the credibility, confirmability, 
dependability, and transferability of the qualitative data from these analyses90. We will use “member checks,” 
i.e., sharing of initial data interpretations with participants to ensure accurate interpretations. The investigators 
will carefully record an audit trail and keep extensive field notes to facilitate transferability of study findings into 
other contexts 

G.3. Data Analysis. 
Each focus group recording will be transcribed. Transcripts will be analyzed independently for content by 
research team members experienced in these methods. 
 
H. Aim 3c: Assess PLWH perceptions and experience on the recruitment process and research 
participation to WiseApp study through in-depth interview. 
 
H.1. Sample. We will conduct in-depth interview with 20 participants. Interview participants will be drawn from 
both control and intervention arms. 
 
H.2. Procedures. The in-depth interview will be 30 minutes in length. We will include $20 reimbursement for 
participants’ time. Following completion of the informed consent process, all interviews will be audio-recorded. 
The PI who have conducted interviews for a number of studies in the past, and her research team members who 
are pre/post docs will convene the groups with the study participants and will act as a facilitator 93,96,98,99. This will 
be an optional activity for participants. This in-depth interview  will be conducted to understand participant’s 
experience and perception on the recruitment process and research participation. Some of the questions in the 
interview include: 1) How comfortable were you with the recruitment and screening process for the study? 
2) How would you modify the recruitment process to improve it or make it easier? 3) Please describe your 
experience in the enrollment process. 4) Would you prefer the recruitment and enrollment process to be 
online or in-person? 5) What motivated you to participate in this research study? (Interview guide included as 
attachments entitled “WISEAPP_Interview Guide on Recruitment “) 
The team will adhere to qualitative research processes to ensure the credibility, confirmability, 
dependability, and transferability of the qualitative data from these analyses90. We will use “member checks,” 
i.e., sharing of initial data interpretations with participants to ensure accurate interpretations. The investigators 
will carefully record an audit trail and keep extensive field notes to facilitate transferability of study findings into 
other contexts. 
 
H.3. Data Analysis. 
Each interview recording will be transcribed. Transcripts will be analyzed independently for content by research 
team members experienced in these methods. 
 
 
 
 
The study protocol will be reviewed and ultimately receive approval of the Columbia University Medical Center 
(CUMC) Institutional Review Board (IRB). As a principal investigator (PI) of a number of funded studies for 
persons living with HIV, Dr. Schnall has experience securing IRB approval with this special study population. 
Risks to Human Subjects 
a. Human Subjects Involvement, Characteristics, and Design 
Characteristics of Subject Population: 
We anticipate two categories of research subjects in our study: 1) persons living with HIV/AIDS and 2) experts 
in human computer interaction. 
Persons living with HIV/AIDS in this study will be recruited from the HIV clinic at CUMC/NewYork-Presbyterian 
Hospital and AIDS Service Center. 
There is no more than a minimal risk associated with any of the proposed study activities. The study 
activities meet the general definition found in Subpart A (46.102) that the probability and magnitude of harm 
or discomfort anticipated in the research are not greater in and of themselves than those ordinarily 
encountered in daily life or during the performance of routine physical or psychological examinations or tests. 
Following IRB approval, all study participants will provide informed consent prior to their participation in this 
study. 
RISKS, BENEFITS & MONITORING 

b. Sources of Materials 
Subjective data (e.g., health status, quality of life) and system use will be obtained from subjects for the 
specific research purposes of this study. 
b.1. Access to Individually Identifiable Private Information about Human Subjects 
Access to individually identified private information about human subjects will be limited to research team 
members who collect and manage the data, the project coordinator, and the PI. De-identified data will be 
accessible by all members of the research team involved in the data analysis. 
c. Potential Risks 
The risks of participating in this study are few. Some of the questions on the questionnaires may make the 
subject uncomfortable or upset, but the subject is free to decline to answer any questions. Participation in 
research can involve loss of privacy. Information about the subject will be handled as confidentially as possible. 
All study data will be maintained in a completely secure and HIPAA compliant environment. All CUMC servers 
have HIPAA compliant security. All signed consent forms and payment receipts used in this study will be kept in 
locked files so only the investigators will have access to the files. 
Adequacy of Protection against Risks 
a. Recruitment and Informed Consent/Assent 
Recruitment for participation in the study will occur following approval by the CUMC IRB. The PI will determine 
eligibility for inclusion, explain the purpose of the study, answer any questions, and obtain written consent from 
the participants. 

Patients who agree to participate will sign a consent form. Recruitment strategies will be developed in 
collaboration with clinics and community centers. Potential risks and strategies for managing risks will be 
carefully explained as part of informed consent procedures. All HIPAA requirements will be applied to this 
study. The study plan, advertisements or recruitment letters, lay description of the study and all consent forms 
will be submitted to the IRB at CUMC following proposal acceptance and prior to study initiation. Dr. Schnall 
will be responsible for obtaining IRB approval for this study. 
b. Protections against Risks 
This study will be submitted to the IRB of CUMC before starting the study. We will be careful to ensure that no 
coercion occurs during the recruitment periods by ensuring that the healthcare providers and patients are 
voluntarily participating. All participants will be screened to assess for study eligibility. 
Should any physical or psychological manifestation be exhibited at any time during the study, Dr. Schnall will 
consult with her co-investigators regarding clinical situations as they arise. If an urgent clinical situation should 
arise, Dr. Schnall or designee will access an urgent appointment at the HIV clinic at NewYork-Presbyterian 
Hospital or the emergency room at CUMC. Dr. Schnall will complete an adverse event form and report to the 
CUMC IRB. If there are any serious adverse events, they will be reported within 48-72 hours to the 
National Institute of Health (NIH), AHRQ and the CUMC IRB. 
Patient volunteers will be told that they can withdraw from study at any point without any effect on their 
healthcare treatment or status as patients. During the usability testing, the participants might become frustrated 
with the system; however, all efforts will be made to provide a comfortable environment. Data will be stored in 
password-protected computers or double locked file cabinets. All reported data will be de-identified. 
Non-participation will not affect patients’ medical care in any way. Each group of participants will be made 
aware of the nature of the questions prior to consenting to participate, but nonetheless may become upset by 
some of the questions. Participants may opt to omit or not answer any questions at any time. 
There are no benefits to participants. However, if a previously unrecognized problem is observed, the 
participant may access appropriate services for evaluation and treatment. 
Subjects will be informed of the potential risks in the consent form. All subject names and medical data will be 
kept confidential, and all data will be identified only by a code number. The list matching names and code 
numbers will be kept locked in the PI's office and will be destroyed at the end of the study period. 
Loss of confidentiality. All study data will be stored in password protected computers or file cabinets in locked 
offices. All research team members will pass the protection of human subjects and research HIPAA exams and 
sign a protocol-specific conflict of interest. 
All procedures have been designed to protect each participant's privacy and allow for anonymous participation. 
All study data will be maintained in a completely secure and HIPAA compliant environment. All CUMC servers 
have HIPAA compliant security. 
Potential Benefits of the Proposed Research to Human Subjects and Others 
This study has not been designed for the direct benefit of its participants. However, there are a number of ways 
in which they may derive benefit. The proposed research will inform the delivery of care for persons living with 
HIV. The knowledge gained will contribute to the body of knowledge regarding the use of health information 
technology for improving the lives of persons living with HIV. 
Importance of the Knowledge to be Gained 
The research has the potential to improve patient-centered outcomes for persons living with HIV. 
 
Data Safety and Monitoring Plan 
The Data and Safety Monitoring Plan for the proposed study incorporates the Policies on Data and Safety 
Monitoring specified by Columbia University. Data safety and monitoring will be the responsibility of the PI. 

This trial does not pose more than minimal risk and therefore requires the establishment of a Data and Safety 
Monitoring Board (DSMB). 
 
The project team will meet weekly during the project period and minutes will be taken. All adverse events will 
be reported per the requirements of the CUMC IRB. 
The PI will report any of the following to AHRQ and NIH in a timely fashion: 
a. Unanticipated problems or unexpected serious adverse events that may be related to the study protocol 
(within 48-72 hours) 
b. IRB-approved revisions to the study protocol that indicate a change in risk for participants 
c. A summary of recommendations made by the DSMB or other monitoring entity and the action plan for 
response 
d. Notice of any actions taken by the IRB or regulatory bodies regarding the research and any responses 
to those actions 
 
The DSMB will be comprised of a small group of experts from the Columbia University School of Nursing and 
Case Western School of Nursing (at least 2 members who are independent of the protocol) who will review 
data from this study. The DSMB will be comprised of Drs. Larson (Associate Dean for Research), Arcia 
(Columbia University School of Nursing faculty) and Webel (Case Western University School of Nursing faculty) 
who will review the data every six months and meet once a year since the study procedures are of minimal 
risk to study participants. During their yearly monitoring, the DSMB will assess participant recruitment, 
intervention effects, and potential adverse events. The Board will create an action plan if concerns arise. All 
adverse events will be reported per the CUMC IRB, the NIH and the AHRQ. In addition, an external advisory 
board for the project will meet in person annually at CUMC and will review progress on enrollment, the study 
outcomes, and progress on the project. 

1. 
Centers for Disease Control and Prevention. HIV in the Unites States: At A Glance. 2015;  
http://www.cdc.gov/hiv/statistics/overview/ataglance.html. Accessed February 25, 2016. 
2. 
Remien RH, Mellins CA. Long-term psychosocial challenges for people living with HIV: let's not forget 
the individual in our global response to the pandemic. AIDS. Oct 2007;21 Suppl 5:S55-63. 
3. 
Center for Disease Control and Prevention (CDC). HIV Among Latinos. 2014;  
http://www.cdc.gov/hiv/risk/racialEthnic/hispanicLatinos/facts/index.html. Accessed June 11, 2014. 
4. 
Centers for Disease Control and Prevention (CDC). HIV Among African Americans. 2014;  
http://www.cdc.gov/hiv/risk/racialEthnic/aa/facts/. 
5. 
Hall HI, Song R, Rhodes P, et al. Estimation of HIV incidence in the United States. JAMA : the journal 
of the American Medical Association. Aug 6 2008;300(5):520-529. 
6. 
Chesney MA. The elusive gold standard. Future perspectives for HIV adherence assessment and 
intervention. J Acquir Immune Defic Syndr. Dec 1 2006;43 Suppl 1:S149-155. 
7. 
Cohen MS, Chen YQ, McCauley M, et al. Prevention of HIV-1 Infection with Early Antiretroviral 
Therapy. New England Journal of Medicine. 2011/08/11 2011;365(6):493-505. 
8. 
Attia S, Egger M, Muller M, Zwahlen M, Low N. Sexual transmission of HIV according to viral load and 
antiretroviral therapy: systematic review and meta-analysis. AIDS. Jul 17 2009;23(11):1397-1404. 
9. 
Montaner JS, Lima VD, Barrios R, et al. Association of highly active antiretroviral therapy coverage, 
population viral load, and yearly new HIV diagnoses in British Columbia, Canada: a population-based 
study. Lancet. Aug 14 2010;376(9740):532-539. 
10. 
Das M, Chu PL, Santos GM, et al. Decreases in community viral load are accompanied by reductions in 
new HIV infections in San Francisco. PLoS One. 2010;5(6):e11068. 
11. 
US Department of Health and Human Services. The National HIV/AIDS Strategy Washington DC: The 
White House;2010. 
12. 
Hanna DB, Hessol NA, Golub ET, et al. Increase in single-tablet regimen use and associated 
improvements in adherence-related outcomes in HIV-infected women. J Acquir Immune Defic Syndr. 
Apr 15 2014;65(5):587-596. 
13. 
Kumar S, Nilsen W, Pavel M, Srivastava M. Mobile Health: Revolutionizing Healthcare Through Trans- 
disciplinary Research. Computer. 2013;46(1):28-35. 
14. 
Estrin D, Sim I. Health care delivery. Open mHealth architecture: an engine for health care innovation. 
Science. Nov 5 2010;330(6005):759-760. 
15. 
Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of 
computerized information services. A review of 98 randomized clinical trials. Arch Fam Med. May 
1996;5(5):271-278. 
16. 
Lewis D. Computer-based approaches to patient education: a review of the literature. J Am Med Inform 
Assoc. Jul-Aug 1999;6(4):272-282. 
17. 
Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based  
vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res. 
Nov 10 2004;6(4):e40. 
18. 
Nguyen HQ, Carrieri-Kohlman V, Rankin SH, Slaughter R, Stulbarg MS. Supporting cardiac recovery 
through eHealth technology. J Cardiovasc Nurs. May-Jun 2004;19(3):200-208. 
19. 
Nguyen HQ, Carrieri-Kohlman V, Rankin SH, Slaughter R, Stulbarg MS. Internet-based patient 
education and support interventions: a review of evaluation studies and directions for future research. 
Comput Biol Med. Mar 2004;34(2):95-112. 
20. 
Eysenbach G, Powell J, Englesakis M, Rizo C, Stern A. Health related virtual communities and 
electronic support groups: systematic review of the effects of online peer to peer interactions. BMJ. May 
15 2004;328(7449):1166. 
21. 
Brennan PF, Aronson AR. Towards linking patients and clinical information: detecting UMLS concepts 
in e-mail. J Biomed Inform. Aug-Oct 2003;36(4-5):334-341. 
22. 
Brennan PF, Caldwell B, Moore SM, Sreenath N, Jones J. Designing HeartCare: custom computerized 
home care for patients recovering from CABG surgery. Proc AMIA Symp. 1998:381-385. 
23. 
Mumford E, Schlesinger HJ, Glass GV, Patrick C, Cuerdon T. A new look at evidence about reduced 
cost of medical utilization following mental health treatment. Am J Psychiatry. Oct 1984;141(10):1145- 
1158. 
24. 
Vickery DM. Demand management. Toward appropriate use of medical care. Healthc Forum J. Jan- 
Feb 1996;39(1):14-19. 

25. 
Vickery DM, Kalmer H, Lowry D, Constantine M, Wright E, Loren W. Effect of a self-care education 
program on medical visits. JAMA : the journal of the American Medical Association. Dec 2 
1983;250(21):2952-2956. 
26. 
Latham CE. Is there data to support the concept that educated, empowered patients have better 
outcomes? J Am Soc Nephrol. Dec 1998;9(12 Suppl):S141-144. 
27. 
Renders CM, Valk GD, Griffin SJ, Wagner EH, Eijk Van JT, Assendelft WJ. Interventions to improve the 
management of diabetes in primary care, outpatient, and community settings: a systematic review. 
Diabetes Care. Oct 2001;24(10):1821-1833. 
28. 
Wagner EH, Austin BT, Von Korff M. Improving outcomes in chronic illness. Manag Care Q. Spring 
1996;4(2):12-25. 
29. 
Bangsberg DR, Bronstone A, Chesney MA, Hecht FM. Computer-assisted self-interviewing (CASI) to 
improve provider assessment of adherence in routine clinical practice. J Acquir Immune Defic Syndr.  
Dec 15 2002;31 Suppl 3:S107-111. 
30. 
Johnson MO, Chesney MA, Goldstein RB, et al. Positive provider interactions, adherence self-efficacy, 
and adherence to antiretroviral medications among HIV-infected adults: A mediation model. AIDS 
Patient Care STDS. Apr 2006;20(4):258-268. 
31. 
Lenhart A, Hitlin P, Madden M. Teens and Technology. 2005;  
http://www.pewinternet.org/Reports/2005/Teens-and-Technology/06-Communications-Tools-and-   
Teens/17-Cell-phone-text-messaging-emerges-as-a-formidable-force.aspx?r=1. Accessed March 28, 
2010. 
32. 
Smith A. US Smartphone Use in 2015. Washington, DC: Pew Research Center;2015. 
33. 
Duggan M. Cell Phone Activities 2013. PewResearch Internet Project 2013;  
http://www.pewinternet.org/2013/09/19/cell-phone-activities-2013/. Accessed August 26, 2014. 
34. 
Akter S, Ray P. mHealth - an Ultimate Platform to Serve the Unserved. Yearbook of medical 
informatics. 2010:94-100. 
35. 
Klasnja P, Pratt W. Healthcare in the pocket: mapping the space of mobile-phone health interventions. 
J Biomed Inform. Feb 2012;45(1):184-198. 
36. 
Kirk GD, Himelhoch SS, Westergaard RP, Beckwith CG. Using Mobile Health Technology to Improve 
HIV Care for Persons Living with HIV and Substance Abuse. AIDS research and treatment.  
2013;2013:194613. 
37. 
Horvath T, Azman H, Kennedy GE, Rutherford GW. Mobile phone text messaging for promoting 
adherence to antiretroviral therapy in patients with HIV infection. Cochrane Database Syst Rev. 
2012;3:CD009756. 
38. 
Nieuwlaat R, Mistry N, Haynes R. Mobile text messaging and adherence of patients to medication 
prescriptions: A txt a da keeps da doctr awa? JAMA Internal Medicine. 2016. 
39. 
Parienti J-J, Das-Douglas M, Massari V, et al. Not All Missed Doses Are the Same: Sustained NNRTI 
Treatment Interruptions Predict HIV Rebound at Low-to-Moderate Adherence Levels. PLoS ONE. 
2008;3(7):e2783. 
40. 
Bangsberg DR, Deeks SG. Spending more to save more: interventions to promote adherence. Ann 
Intern Med. Jan 5 2010;152(1):54-56; W-13. 
41. 
Haberer JE, Kahane J, Kigozi I, et al. Real-time adherence monitoring for HIV antiretroviral therapy. 
AIDS Behav. Dec 2010;14(6):1340-1346. 
42. 
Mohr DC, Montague E, Stiles-Shields C, et al. MedLink: a mobile intervention to address failure points 
in the treatment of depression in general medicine. Paper presented at: Proceedings of the 9th 
International Conference on Pervasive Computing Technologies for Healthcare2015. 
43. 
de Vette F, Tabak M, Dekker-van Weering M, Vollenbroek-Hutten M. Engaging Elderly People in 
Telemedicine Through Gamification. JMIR Serious Games. 2015;3(2):e9. 
44. 
Huis in 't Veld RM, Kosterink SM, Barbe T, Lindegård A, Marecek T, Vollenbroek-Hutten MM. Relation 
between patient satisfaction, compliance and the clinical benefit of a teletreatment application for 
chronic pain. Journal of Telemedicine and Telecare. September 1, 2010 2010;16(6):322-328. 
45. 
King D, Greaves F, Exeter C, Darzi A. ‘Gamification’: Influencing health behaviours with games. Journal 
of the Royal Society of Medicine. 2013;106(3):76-78. 
46. 
Fankhauser D. Is Gamification Just a Fad? 2013; http://mashable.com/2013/05/17/gamification-  
buzzword/. Accessed February 20, 2015. 
47. 
Ferguson B. Games for Wellness—Impacting the Lives of Employees and the Profits of Employers. 
Games for Health Journal. 2012/06/01 2012;1(3):177-179. 

48. 
Bakkes S, Tan CT, Pisan Y. Personalised gaming: a motivation and overview of literature. Proceedings 
of The 8th Australasian Conference on Interactive Entertainment: Playing the System; 2012; Auckland, 
New Zealand. 
49. 
Muessig KE, Pike EC, Legrand S, Hightow-Weidman LB. Mobile phone applications for the care and 
prevention of HIV and other sexually transmitted diseases: a review. Journal of medical Internet 
research. 2013;15(1):e1. 
50. 
Ramanathan N, Swendeman D, Comulada WS, Estrin D, Rotheram-Borus MJ. Identifying preferences 
for mobile health applications for self-monitoring and self-management: focus group findings from HIV- 
positive persons and young mothers. Int J Med Inform. Apr 2013;82(4):e38-46. 
51. 
Holstad M, Ofotokun I, Farber E, et al. There's an App for That! Promoting Antiretroviral Medication 
Adherence in Rural Georgia: The Music for Health Project. Journal of Mobile Technology in Medicine. 
2013;2(4s):18-18. 
52. 
Catalani C, Philbrick W, Fraser H, Mechael P, Israelski DM. mHealth for HIV Treatment & Prevention: A 
Systematic Review of the Literature. Open AIDS J. 2013;7:17-41. 
53. 
Free C, Phillips G, Galli L, et al. The effectiveness of mobile-health technology-based health behaviour 
change or disease management interventions for health care consumers: a systematic review. PLoS 
Med. 2013;10(1):e1001362. 
54. 
IMS Institute for Healthcare Informatics. Patient Apps for Improved Healthcare: From Novelty to 
Mainstream. 2013. 
55. 
Fogg BJ. Persuasive Technology: Using Computers to Change What We Think and Do. Science \& 
Technology Books; 2002. 
56. 
Schnall R, Bakken S, Carballo-Dieguez A, Brown Iii W, Iribarren SJ. Usabilty Evaluation of a Prototype 
Mobile App for Health Management for Persons Living with HIV NI2016; 2016; Geneva, Switzerland. 
57. 
Schnall R, Bakken S, Rojas M, Travers J, Carballo-Dieguez A. mHealth Technology as a Persuasive 
Tool for Treatment, Care and Management of Persons Living with HIV. AIDS Behav. Jun 2015;19 
Suppl 2:81-89. 
58. 
Schnall R, Higgins T, Brown W, Carballo-Dieguez A, Bakken S. Trust, Perceived Risk, Perceived Ease 
of Use and Perceived Usefulness as Factors Related to mHealth Technology Use. Stud Health Technol 
Inform. 2015;216:467-471. 
59. 
Schnall R, Mosley JP, Iribarren SJ, Bakken S, Carballo-Dieguez A, Brown Iii W. Comparison of a User- 
Centered Design, Self-Management App to Existing mHealth Apps for Persons Living With HIV. JMIR 
Mhealth Uhealth. Sep 18 2015;3(3):e91. 
60. 
Schnall R, Rojas M, Bakken S, et al. A user-centered model for designing consumer mobile health 
application (apps). J Biomed Inform. Feb 20 2016. 
61. 
Schnall R, Rojas M, Travers J, Brown W, 3rd, Bakken S. Use of Design Science for Informing the 
Development of a Mobile App for Persons Living with HIV. AMIA Annu Symp Proc. 2014;2014:1037- 
1045. 
62. 
Polson PG, Lewis C, Rieman J, Wharton C. Cognitive walkthroughs: a method for theory-based 
evaluation of user interfaces. International Journal of Man-Machine Studies. 1992/05/01/ 
1992;36(5):741-773. 
63. 
Georgsson M, Kushniruk A. Mediating the Cognitive Walkthrough with Patient Groups to achieve 
Personalized Health in Chronic Disease Self-Management System Evaluation. Stud Health Technol 
Inform. 2016;224:146-151. 
64. 
Jaspers MW. A comparison of usability methods for testing interactive health technologies: 
methodological aspects and empirical evidence. Int J Med Inform. May 2009;78(5):340-353. 
65. 
TechSmith. Morae Usability and Web Site Testing: TechSmith Corporation. 1995;  
http://www.techsmith.com/products/morae/default.asp. 
66. 
Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. 
Behav Res Methods Instrum Comput. Aug 2003;35(3):379-383. 
67. 
Chesney MA, Ickovics JR, Chambers DB, et al. Self-reported adherence to antiretroviral medications 
among participants in HIV clinical trials: the AACTG adherence instruments. Patient Care Committee & 
Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group 
(AACTG). AIDS Care. Jun 2000;12(3):255-266. 
68. 
Altman DG, Bland JM. Treatment allocation in controlled trials: why randomise? BMJ. 
1999;318(7192):1209-1209. 

69. 
Hedden Sl, Woolsen RF, Malcolm RJ. Randomization in substance abuse clinical trials. Substance 
Abuse Treatment, Prevention, and Policy. 2006;1:6. 
70. 
Reynolds WM. Development of reliable and valid short forms of the marlowe-crowne social desirability 
scale. Journal of Clinical Psychology. 1982;38(1):119-125. 
71. 
Weiss BD, Mays MZ, Martz W, et al. Quick Assessment of Literacy in Primary Care: The Newest Vital 
Sign. The Annals of Family Medicine. November 1, 2005 2005;3(6):514-522. 
72. 
Parker RM, Baker DW, Williams MV, Nurss JR. The test of functional health literacy in adults: a new 
instrument for measuring patients' literacy skills. J Gen Intern Med. Oct 1995;10(10):537-541. 
73. 
Marc LG, Wang M-M, Testa MA. Psychometric evaluation of the HIV symptom distress scale. AIDS 
care. 03/12 2012;24(11):1432-1441. 
74. 
Viney R, Norman R, Brazier J, et al. An Australian discrete choice experiment to value eq-5d health 
states. Health economics. Jun 2014;23(6):729-742. 
75. 
Webel AR, Asher A, Cuca Y, et al. Measuring HIV self-management in women living with HIV/AIDS: a 
psychometric evaluation study of the HIV Self-management Scale. J Acquir Immune Defic Syndr. Jul 1 
2012;60(3):e72-81. 
76. 
Hertogh EM, Monninkhof EM, Schouten EG, Peeters PHM, Schuit AJ. Validity of the Modified Baecke 
Questionnaire: comparison with energy expenditure according to the doubly labeled water method. The 
International Journal of Behavioral Nutrition and Physical Activity. 2008;5:30-30. 
77. 
Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a Brief Depression Severity Measure. 
Journal of General Internal Medicine. 2001;16(9):606-613. 
78. 
Hirschfeld RMA. The Mood Disorder Questionnaire: A Simple, Patient-Rated Screening Instrument for 
Bipolar Disorder. Primary Care Companion to The Journal of Clinical Psychiatry. 2002;4(1):9-11. 
79. 
Ising HK, Veling W, Loewy RL, et al. The Validity of the 16-Item Version of the Prodromal 
Questionnaire (PQ-16) to Screen for Ultra High Risk of Developing Psychosis in the General Help- 
Seeking Population. Schizophrenia Bulletin. 2012;38(6):1288-1296. 
80. 
Diagnostic and statistical manual of mental disorders : DSM-5. Fifth edition. Arlington, VA : American 
Psychiatric Publishing, [2013] ©2013; 2013. 
81. 
Mannheimer SB, Mukherjee R, Hirschhorn LR, et al. The CASE adherence index: A novel method for 
measuring adherence to antiretroviral therapy. AIDS care. 2006;18(7):853-861. 
82. 
Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for 
Usability Assessment of Mobile Health Technology: Validation Study. JMIR Mhealth Uhealth. 
2018;6(1):e4. 
83. 
Lewis JR. Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies. 
International Journal of Human–Computer Interaction. 2002/09/01 2002;14(3-4):463-488. 
84. 
Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information 
technology. MIS quarterly. 1989:319-340. 
85. 
Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. Journal of medical Internet research. 
2006;8(4):e27. 
86. 
Gwadz M, Cleland CM, Applegate E, et al. Behavioral intervention improves treatment outcomes 
among HIV-infected individuals who have delayed, declined, or discontinued antiretroviral therapy: a 
randomized controlled trial of a novel intervention. AIDS Behav. Oct 2015;19(10):1801-1817. 
87. 
Haberer JE, Robbins GK, Ybarra M, et al. Real-Time Electronic Adherence Monitoring is Feasible, 
Comparable to Unannounced Pill Counts, and Acceptable. AIDS and Behavior. February 01 
2012;16(2):375-382. 
88. 
Hall J, Bond C, Kinnear M, McKinstry B. Views of patients and professionals about electronic 
multicompartment medication devices: a qualitative study. BMJ Open. 2016;6(10). 
89. 
Pellowski JA, Kalichman SC, White D, Amaral CM, Hoyt G, Kalichman MO. Real-time medication 
adherence monitoring intervention: Test of concept in people living with HIV infection. The Journal of 
the Association of Nurses in AIDS Care : JANAC. 2014;25(6):646-651. 
90. 
Guba EG. Criteria for assessing the trustworthiness of naturalistic inquiries. Educational 
Communication and Technology Journal. 1981;29(2):75-91. 
91. 
Lee Y, Johnson SB, Trembowelski S, Schnall R, Sheehan B, Bakken S. Cognitive walkthrough 
evaluation of an investigator profiles directory. . Proc AMIA Annu Fall Symp. 2011. 
92. 
Schnall R, Cimino JJ, Bakken S. Development of a prototype continuity of care record with context- 
specific links to meet the information needs of case managers for persons living with HIV. Int J Med 
Inform. Aug 2012;81(8):549-555. 

93. 
Schnall R, Odlum M, Gordon P, Bakken S. Barriers to implementation of a Continuity of Care Record 
(CCR) in HIV/AIDS care. Stud Health Technol Inform. 2009;146:248-252. 
94. 
Sheehan B, Lee Y, Rodriguez M, Tiase V, Schnall R. A comparison of usability factors of four mobile 
devices for accessing healthcare information by adolescents. Appl Clin Inform. 2012;3(4):356-366. 
95. 
Wantland D, Odlum M, Schnall R, et al. Barriers and Facilitators to Use of a Continuity of Care 
Document (CCD) in HIV/AIDS Care: Persons Living With HIV/AIDS (PLWH) and Health Care Provider 
Perspectives. AMIA Annual Symposium Proceedings. 2009. 
96. 
Schnall R, Clark S, Olender S, Sperling JD. Providers' Perceptions Of The Factors Influencing The 
Implementation Of The New York State Mandatory HIV Testing Law In Two Urban Academic 
Emergency Departments. Academic emergency medicine : official journal of the Society for Academic 
Emergency Medicine. 2013;20(3):279-286. 
97. 
Odlum M, Gordon P, Camhi E, Schnall R, Teixeira P, Bakken S. Perceptions of predisposing, enabling, 
and reinforcing factors influencing the use of a continuity of care document in special needs PLWH. J 
Health Care Poor Underserved. Nov 2012;23(4):1457-1476. 
98. 
Okoniewski AE, Lee YJ, Rodriguez M, Schnall R, Low AF. Health information seeking behaviors of 
ethnically diverse adolescents. J Immigr Minor Health. Aug 2014;16(4):652-660. 
99. 
Schnall R, Cimino JJ, Currie LM, Bakken S. Information needs of case managers caring for persons 
living with HIV. J Am Med Inform Assoc. May 01 2011;18(3):305-308. 
100. 
Green L, Kreuter M, Deeds S, Partridge K. Health education planning: A diagnostic approach. . 
Mountain View, CA: Mayfield; 1980. 
101. 
Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information technology 
implementation framework in behavioral science: a systematic analysis of the literature on IT use. J 
Biomed Inform. Jun 2003;36(3):218-227.