Evaluation of the Initial Deployment of VA's Annie Texting System (PEC 15-470)
Phase: NA
Conditions: Hepatitis C
Sponsor: VA Office of Research and Development

eHealth Partnered Evaluation Initiative - Evaluation of the Initial Deployment of VA’s Annie Texting 
System (Annie Eval) 
ClinicalTrials.gov ID: NCT03898349 
April 10, 2017 

Objective  
 
We describe the first evaluation of the VA’s automated texting system (aTS). Consistent with an 
implementation-effectiveness hybrid 2 study design, our aims were to: (1) qualitatively and quantitatively 
assess implementation outcomes for the aTS; and (2) assess impact of the aTS on HCV clinical outcomes 
in a real-world setting. 
 
Methods 
Our evaluation was guided by the Practical, Robust, Implementation and Sustainability Model 
(PRISM), which defines a set of factors for consideration when designing, implementing, sustaining and 
evaluating interventions.i PRISM posits that the extent to which an intervention achieves results can be 
linked to the four PRISM domains, namely the intervention’s characteristics (via patient and 
organizational perspectives), the intervention recipients (via patient and organizational perspectives), the 
external environment, and the implementation and sustainability infrastructure.  
 
Design 
This was a multi-site, mixed-methods, randomized, two-group hybrid type 2 study design 
comparing the effectiveness of usual implementation (UI) and augmented implementation (AI).ii Matched 
comparison (i.e., no intervention) sites helped to determine effectiveness of the aTS in aiding HCV 
treatment. The study was reviewed by the Institutional Review Board (IRB) at the Edith Nourse Rogers 
Memorial Veterans Hospital in Bedford, Massachusetts and determined to be a quality improvement 
study (VA Handbook 1058.05).iii The project was conducted from February 2017 through February 2018. 
 
Setting & Participants 
The group of clinics selected reflects a purposive sample based on several criteria, including 
clinic size and complexity, and geography. HCV clinics were recruited via a national HCV provider email 
listserv and monthly HCV provider phone call. The seven HCV clinics selected were randomly assigned 
to either UI or AI, using set randomization, which, with small sample sizes, helps to achieve balance on a 
set of relevant characteristics – in this case, urban/suburban setting and on HCV patient volume.iv Two 
additional comparison clinics were selected purposively because they had similar patient volume, clinic 
complexity, and geographic locale to the other participating clinics.v Thus in total, data were collected 
from nine VA clinics: four AI clinics, three UI clinics, and two comparison clinics. Care teams within the 
HCV clinics had different compositions and involved pharmacists, nurse practitioners, registered nurses, 
and social worker to varying degrees.  
 
Intervention: Text Messaging for HCV treatment  
At the time of implementation, usual care for HCV included starting a patient on daily oral 
medication for 8-16 weeks, and follow-up in-person visits, blood lab work, and medication refills at 2- or 
4-week intervals. The aTS for HCV was intended to improve processes and outcomes of care, and 
satisfaction with care. The HCV texting protocol included reminder text messages about medication 
taking, appointment keeping, and laboratory completion, and motivational text messages to encourage 
continued engagement in treatment. To be consistent with the standard treatment processes, each of the 
seven clinics worked collaboratively with study team members (VY, KM, and national aTS program 
office) to tailor the HCV texting protocol to align with their clinic’s workflow (e.g., adjusting messaging 
logic from 2 to 4 weeks for different treatment intervals). Motivational messages were supportive in 
nature and intended to promote self-management and increase feelings of connection to the treatment 
team.  
Message content could be tailored for content (e.g., adding clinic name and phone number) and 
timing (e.g., adjusting time of day that medication reminder is sent) through patient and provider 
discussion at the time of aTS enrollment, or later to reflect patient preferences. Veterans who did not use 
the aTS received otherwise standard HCV care. 
 

Usual Implementation Facilitation and Augmented Implementation Facilitation 
Facilitation was employed to support adoption of the aTS at participating clinics, with facilitation 
delivered by a primary (VY) and a secondary (KM) external facilitator. During the four-month pre-
implementation phase, the functions of the external facilitators included engaging local, regional and 
national stakeholders to garner support for the aTS. The implementation phase took place over six months 
and differed between usual implementation (UI) and augmented implementation (AI), as detailed below. 
The post-implementation (evaluation) phase took place over three months.  
UI clinics: UI clinics received the start-up experience that VA designed for all new clinics 
instituting the aTS. This involved a live virtual demonstration of the aTS, and access to an aTS resource 
website that included promotional materials and training guides. UI clinics could receive troubleshooting 
assistance from the external facilitators by phone or email, but only if and when they reached out to the 
them. 
AI clinics: In addition to the start-up experience for UI sites described above, AI clinics received 
an implementation toolkit, support for local champion development, and proactive outreach by the 
primary external facilitator. The toolkit was developed by our team based on a formative evaluation that 
involved visits to 5 VA medical centers around the country to conduct interviews with patients and 
clinical staff who were using a pilot version of the aTS for conditions other than HCV. The toolkit 
contained sections on evidence of texting in healthcare, gaining leadership and clinic support for 
technology like the aTS, use of champions to support aTS adoption, tips and tools on how to use the aTS, 
and aTS promotional materials to encourage clinic and patient participation. Each AI clinic received one 
in-person visit from the primary external facilitator early in their implementation efforts. Additionally, the 
primary external facilitator-initiated check-ins with AI clinic champions throughout implementation. 
Facilitation was delivered via email, phone, and in-person. In the pre-implementation period, to 
establish rapport and trust, there was more emphasis on phone calls and in-person meetings, whereas 
during implementation, those modes were used less while use of emails increased. Facilitation calls lasted 
from 5 minutes to 90 minutes (40-minute average) and site visits by the external facilitator lasted two to 
four hours. The most common facilitation strategies employed were local technical assistance, assessing 
for readiness, site visits, identifying and preparing champions, developing and distributing educational 
materials, building a coalition, and tailoring implementation to context. vi,vii The template for intervention 
description and replication (TIDieR) checklist was used to guide intervention description (Additional File 
1).viii  
Measures and Data Collection 
PRISM domains (implementation and sustainability infrastructure, intervention characteristics, 
recipients via both patient and organizational perspectives, external environment) guided the measures 
and data collection and are denoted in parentheticals.  
Implementation Processes: Facilitation (Implementation and Sustainability Infrastructure). The 
external facilitator (VY) logged facilitation events on a tracking sheet, including facilitation date, length 
of time, mode of delivery (i.e., email, phone call, or in-person visit), purpose, notes, and other 
observations. From the perspective of the primary facilitator, if multiple facilitation events occurred in 
one day, only one event per person, per day was counted.  
Implementation Outcomes: Texting Use (Intervention). Providers logged the number of patients who 
were offered the aTS and noted whether patients enrolled or declined, including the reason for declining. 
The content of patient text message replies was extracted from the aTS portal. To be eligible for the 
texting protocol, patients had to be starting HCV medication treatment. There were four steps to initiate a 
patient on the aTS: (1) providers verbally offered patients the aTS, (2) providers registered interested 
patients in the aTS portal and assigned them the HCV protocol, and (3) once a patient was registered, the 
aTS would send an automated text message requesting the patient authenticate themselves by replying to 
this initial text message thus prompting the assigned HCV protocol to begin, and (4) patients actively text 
with the aTS.  
Clinical Effectiveness Outcomes. Medication adherence was measured via patient text response rate, 
operationalized as the number of days of text-confirmed medication taking divided by the number of days 

receiving medication reminder texts. Consistent with other adherence standards, an affirmative text 
response rate of ≥80% was considered high adherence.ix Clinical data, including HCV treatment regimen 
and duration, and lab dates and results, were extracted from VA’s National HCV Dashboard with data 
from VA’s Corporate Data Warehouse.x The goal of treatment is to achieve cure, or sustained virologic 
response (SVR), where there is an undetectable HCV lab result 12 weeks after completion of treatment.  
Questionnaires (Recipients, intervention, implementation). Patients at each clinic completed baseline 
and follow-up (after 8-12 weeks of using the aTS) questionnaires. The comparison clinics followed the 
same schedule, though without any use of texting. Patient questionnaires covered the topics of self-rated 
health status, adherence, illness perception, health engagement and activation, technology use, experiences 
with the aTS (usability, usefulness, working alliance), and demographics.xi,xii,xiii,xiv,xv,xvi Provider 
questionnaires followed the same schedule and covered topics of technology experience, quality 
improvement culture, climate and readiness for implementation, satisfaction with current local HCV care 
processes, experiences with the aTS (usability and usefulness), and demographics.xvii,xviii,xix,xx 
Questionnaires were pre-tested for clarity, redundancy, and relevancy by two patients and four providers 
and implementation scientists independent of the study team.  
For effectiveness outcomes measures, we combined patients who were using the aTS regardless of 
whether they were in UI or AI clinics. These were referred to as “texters”. In contrast, “non-texters” could 
come from UI or AI clinics (patients who agreed to participate in the project but never completed the last 
step of authenticating themselves with the aTS, and thus never received any text messages), as well as 
from the two comparison clinics that did not implement the aTS. 
Semi-structured interviews (all PRISM domains). Qualitative semi-structured telephone interviews 
were conducted with patients and providers who used or were invited to use the aTS. Interviews were 
conducted in the follow-up period during September and October 2017. The interview guides were 
informed by PRISM domains (denoted in parentheticals) and explored issues regarding barriers to and 
facilitators to aTS uptake and use (intervention; implementation and sustainability), usability and 
usefulness of the aTS (intervention), and how the aTS was experienced by patients and providers 
(recipients) in the course of treatment and daily practice (external environment). Interviews were 
conducted by members of the study team not involved in facilitation (BP, CG) and lasted about 30 
minutes.  
For each questionnaire or qualitative interview completed, patients received a $10 store gift card to 
compensate them for their time.  
 
Data Analysis  
Descriptive and bivariate analyses of facilitation log data were conducted to compare facilitation 
dose, comparing UI and AI implementation groups. Descriptive and bivariate analyses were conducted on 
provider and patient questionnaires, text message frequencies, and clinical data to assess differences 
between implementation groups (UI and AI) in implementation outcomes. We then compared clinical 
effectiveness outcomes between texters and non-texters. We examined patient progression through the 
aTS initiation process, by calculating the percent retained from one step to the next, by UI and AI 
implementation group. Chi-square tests were used to assess differences between the two groups. All 
analyses were conducted in RStudio Version 1.0.153 and statistical significance was defined as p<0.05.xxi 
Qualitative interviews were audio recorded, transcribed verbatim, and then analyzed using NVivo 
11 software.xxii Thematic analysis of all qualitative data (interview transcripts, meeting notes, text 
messages) was conducted.xxiii PRISM domains provided the deductive a priori codes and other codes 
emerged with inductive coding. The triangulation of quantitative and qualitative data served as a final step 
of analysis.  
 
 

References 
i Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research 
findings into practice. The Joint Commission Journal on Quality and Patient Safety. 2008 Apr 1;34(4):228-43. 
ii Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of 
clinical effectiveness and implementation research to enhance public health impact. Medical care. 2012 Mar;50(3):217. 
iii Department of Veterans Affairs. VA Handbook 1058.05, VA Operations Activities That May Constitute Research. 2011. 
iv Efird J. Blocked randomization with randomly selected block sizes. International journal of environmental research and public 
health. 2010 Dec 23;8(1):15-20. 
v Veterans Health Administration. VHA Facility Complexity Model 2017. 
http://opes.vssc.med.va.gov/FacilityComplexityLevels/Pages/default.aspx. Accessed Nov 2018. 
vi Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation 
of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. 
Implementation Science. 2015 Dec;10(1):21. 
vii Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Kirchner JE, Proctor EK, Gonzalez R, Park A, Ross D, Morgan TR, Chartier 
M. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. 
Implementation Science. 2017 Dec;12(1):60. 
viii Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Altman DG, Barbour V, Macdonald H, Johnston M, 
Lamb SE. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. 
Bmj. 2014 Mar 7;348:g1687. 
ix Haynes RB. A critical review of the “determinants” of patient compliance with therapeutic regimens. In: Sackett DL, Haynes 
RB, editors. Compliance with therapeutic regimens. Baltimore, MD: Johns Hopkins University Press; 1976. pp. 26–39. 
x Department of Veterans Affairs. HIT Dashboard Workgroup Hepatitis C Virus Analytics.  
xi Fisher L, Glasgow RE, Mullan JT, Skaff MM, Polonsky WH. Development of a brief diabetes distress screening instrument. 
The Annals of Family Medicine. 2008 May 1;6(3):246-52. 
xii Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the 
patient-reported outcomes measurement information system (PROMIS) global items. Quality of Life Research. 2009 Sep 
1;18(7):873-80. 
xiii Centers for Disease Control and Prevention (CDC). Behavioral Risk Factor Surveillance System Survey Questionnaire. 
Atlanta, Georgia: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. 
xiv Mayberry LS, Gonzalez JS, Wallston KA, Kripalani S, Osborn CY. The ARMS-D out performs the SDSCA, but both are 
reliable, valid, and predict glycemic control. Diabetes research and clinical practice. 2013 Nov 1;102(2):96-104. 
xv Davis FD. User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. 
International journal of man-machine studies. 1993 Mar 1;38(3):475-87. 
xvi Horvath AO, Greenberg LS. Development and validation of the Working Alliance Inventory. Journal of counseling 
psychology. 1989 Apr;36(2):223. 
xvii Hannon PA, Helfrich CD, Chan KG, Allen CL, Hammerback K, Kohn MJ, Parrish AT, Weiner BJ, Harris JR. Development 
and pilot test of the workplace readiness questionnaire, a theory-based instrument to measure small workplaces’ readiness to 
implement wellness programs. American Journal of Health Promotion. 2017 Jan;31(1):67-75. 
xviii Jacobs SR, Weiner BJ, Bunger AC. Context matters: measuring implementation climate among individuals and groups. 
Implementation Science. 2014 Dec;9(1):46. 
xix Wen KY, Gustafson DH, Hawkins RP, Brennan PF, Dinauer S, Johnson PR, Siegler T. Developing and validating a model to 
predict the success of an IHCS implementation: the Readiness for Implementation Model. Journal of the American Medical 
Informatics Association. 2010 Nov 1;17(6):707-13. 
xx Brooke J. SUS-A quick and dirty usability scale. Usability evaluation in industry. 1996 Sep;189(194):4-7. 
xxi Team R. RStudio: integrated development for R. RStudio, Inc., Boston, MA URL http://www. rstudio. com. 2015 Jun;42:14. 
xxii Bazeley P, Jackson K, editors. Qualitative data analysis with NVivo. Sage Publications Limited; 2013 Apr 25. 
xxiii Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive 
coding and theme development. International journal of qualitative methods. 2006 Mar;5(1):80-92.