Using Behavioral Economics to Enhance Appointment Reminders and Reduce Missed Visits
Phase: NA
Conditions: Appointment Reminders
Sponsor: VA Office of Research and Development

1 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
Protocol and Statistical Analysis Plan 
 
Study Title 
“Using Behavioral Economics to Enhance Appointment Reminders and Reduce Missed 
Visits” 
 
ClinicalTrials.gov ID 
NCT03850431 
 
Date approved by VA Portland Institutional Review Board 
06/03/2020 
 

2 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
 
Research Protocol/Local Protocol Addendum  
 
TITLE 
“Using Behavioral Economics to Enhance Appointment Reminders and Reduce 
Missed Visits” 
 
INVESTIGATORS 
Alan Teo, MD, MS (Principal Investigator) 
 
SPECIFIC AIMS/PURPOSE 
“No-shows,” which are appointments neither attended nor canceled, are a persistent 
problem in all health care systems. They compromise patient access, lengthen wait 
times, increase health care inefficiencies, and worsen clinical care2,4–7.  Despite efforts 
by VHA, overall no-show rates nationally have remained at 13% for years, are 
especially high in mental health settings (~19%),and result in a staggering nine million 
missed visits in VHA outpatient clinics. Appointment reminders are vital to reducing no-
shows because forgetting is the top reason for missing appointments8. Research has 
established the effectiveness of reminder features such as timing and modality of 
delivery9–14. However, little is known about the effect of modifying what messages are 
contained in appointment reminders9,15. 
 
Behavioral economics (BE) suggests that behavioral “nudges” can be used to make it 
easier for people to do the “right” thing while retaining individual autonomy. We draw 
from BE, psychology and related fields to identify concepts that can be applied to 
innovating the field of appointment reminders16–19. One concept is social norms, which 
suggests that Veterans are likely to attend appointments if they sense that it is the norm 
for Veterans to do so2022. Another concept is based on the idea that providing clear 
instructions and an implementation plan increases the target behavior 1,23,24. A third 
approach is to highlight potential harms or losses to the Veteran from missing an 
appointment, while a fourth approach instead highlights potential negative 
consequences of no-showing to other Veterans25–29. Each of these approaches can be 
transformed into messages that can be incorporated into appointment reminders, which 
in turn may trigger improvements in patients’ appointment adherence. For instance, 
stating that “More than 80 out of 100 Veterans attend their VA appointments” uses 
social norms to reduce likelihood of a no-show. Several large quasi-experiments and 
randomized trials in the federal government, U.K.’s National Health Service, and 
elsewhere have demonstrated that seemingly small changes incorporating these types 
of approaches can produce large benefits when taken to scale19,30–36.  
 
 

3 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
 
We believe now is the idea l moment to apply these types of nudges to the development 
of enhanced appointment reminders in VHA. The overarching objective of this proposal 
is to conduct a cluster randomized controlled trial of simple and scalable enhancements 
to appointment reminders in mental health and primary care, and prepare for larger-
scale implementation. We will design and evaluate four intervention groups varying in 
type of nudge included in appointment reminder letters or postcards. Appointments will 
be randomly allocated to one of the four interventions or a usual care control group at 
the provider-level. Our preliminary work shows that Veterans are dissatisfied with 
current appointment reminders and find an intervention with enhanced reminder 
messages acceptable. We have established key partnerships with national VA leaders, 
constructed and analyzed data sets for our primary quantitative outcomes, and 
conducted a pilot intervention of over 8,000 appointment letters that shows promise for 
reducing no-shows and boosting appointment attendance. 
 
Aim 1: Develop and iteratively refine BE-informed messages based on Veterans’ 
perceptions, and incorporate them into enhanced appointment reminders.  
Hypothesis: Perceptions of messages will be consistent with our conceptual 
framework and not associated with emotional harms. 
 
Aim 2: Determine the effect of four versions of enhanced appointment reminders 
on measures relevant to treatment access, compared with usual reminders.  
Hypotheses: 1. Each intervention group will have a lower no-show rate, as 
compared with usual care. 2. Each intervention group will have a higher 
advanced cancellation rate, as compared with usual care. Exploratory 
Hypothesis: Wait time will be shorter in intervention groups than with usual care. 
 
Aim 3: Evaluate differences in treatment effect associated with four versions of 
enhanced appointment reminders.  
Hypothesis: Effect size of nudges will vary by intervention arm such that some 
types of nudges will have a larger proportional effect than others. 
 
Aim 4: Characterize potential barriers and facilitators to widespread 
implementation of enhanced appointment reminder messages. We will conduct 
semi-structured interviews with key informants (10 Veterans locally and 30 VHA staff 
nationally). Interview guides will be informed by selected domains and constructs from 
the Consolidated Framework for Implementation Research.  
 
SCIENTIFIC RATIONALE AND SIGNIFICANCE  

4 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
BACKGROUND 
Why are no-shows a critical health services issue now?  
Providing and maintaining patient access to care has been declared by leadership as 
the Veterans Health Administration’s (VHA) highest priority. Deficits in access have 
been shown to contribute to decreased patient satisfaction37, higher mortality rates38 
and diminished quality of care39. Amidst an environment of limited access and long wait 
times, VHA is simultaneously struggling with “no-shows.”  
 
No-shows are defined as scheduled appointments that are neither attended nor 
canceled, and represent a “missed opportunity” on several levels. First, for patients, no-
shows may contribute to reduced access and increased wait time. A VA Office of 
Inspector General audit noted that “reducing unused outpatient appointments could 
improve timeliness of Veterans’ patient care”7. Another report found that inefficiencies 
caused by no-shows in VHA make it “difficult to meet the performance measure of 
seeing new patients within 30 days”40. Second, no-shows contribute to inefficient use of 
healthcare resources. No-shows result in appointment slots that unexpectedly go 
unused; further, after a no-show, clinics frequently spend time re-scheduling. The same 
Office of Inspector General audit estimated that suggested that reducing no-shows in 
could allow VHA facilities to more efficiently use millions of dollars each year7. Third, no-
shows represent a missed clinical opportunity. The appointment was scheduled for a 
clinical reason, which does not get addressed when there is a no-show. Many 
vulnerable, at-risk groups, including psychiatric patients, ethnic minorities, and low-
income patients are disproportionately impacted by no-shows4,41,42, which can 
contribute to further disparities in clinical outcomes and quality of care4,41.  
 
Why are current approaches to address no-shows in VHA inadequate? 
VHA struggles with elevated no show-rates, which have not improved in recent years. 
VHA’s overall no-show rate continues to be approximately 13-14%, far above Secretary 
Shulkin’s stated goal of five percent. In FY2015 this rate led to a total of over nine 
million no-shows. Despite VHA efforts aimed at reducing no-shows, such as recall 
scheduling and new patient orientation clinics, no-show rates have been static for years, 
according to VHA Support Service Center (VSSC) data (see Figure 1)15. 

5 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
 
 
 
 
 
 
 
 
 
What Veterans are likely to no-show and why? 
Research led by our operational partner Dr. Davies has described some common 
characteristics of Veterans who miss visits: mental health and rehabilitation are the 
service lines with the highest no-show rates; no-show rate increases with age of the 
appointment (i.e., how long ago it was scheduled); new patients have higher no-show 
rates; male Veterans have somewhat higher no-show rates than females until age 65, 
after which it equalizes2. 
 
The National Initiative to Reduce Missed Opportunities was a large VHA project that 
established many of the reasons for and risk factors related to no-shows, and their 
results reinforce the importance of appointment reminders. One of their national surveys 
(n=4,749) determined that forgetting was the most frequent reason (19%) cited by 
Veterans for a no-show8. In contrast, issues that would not be addressed by reminders 
(e.g., poor weather, sickness, and transportation issues) were much less common 
reason for no-shows (all cited by <8% of Veterans). Furthermore, Veterans who 
received a reminder letter, postcard, or call were significantly more likely to remember 
their appointment (85%) than those who did not receive a reminder (75%).  
 
Why use appointment reminder systems and mailed letters or postcards to 
address access issues? 
Appointment reminders are a key ingredient to promoting appointment attendance and 
cancellation of appointments that are no longer needed. (In this proposal, we use the 
term “appointment reminder” to refer to either reminders sent before an appointment 
or messages sent after a no-show.) Decades of research has definitively established 
the efficacy of appointment reminders in reducing no-show rates11,14,43 and recent trials 
have suggested that appointment reminders can trigger more patients to cancel, rather 
than no-show, presumably because they no longer feel the appointment is 
necessary31,44.  
 

6 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
 
No-shows are not simply limited to a small group of outlier patients. Our preliminary 
data compiled from VSSC show that 40% of unique patients in mental health clinics, 
and 15% in primary care, had at least one no-show in VA Portland Health Care System 
(VAPORHCS) in 2016.  
 
Appointment reminders letters and cards will continue to play an essential role in VHA. 
Rachel Goffman, MHA, former National Project Manager of the National Initiative to 
Reduce Missed Opportunities (NIRMO) at the VA Pittsburgh Healthcare System’s 
Veterans Engineering Resource Center (VERC) noted that appointment reminders are 
“not going away” despite an evolving access landscape. In addition, VA Directive 1230, 
released recently on July 15, 2016, reinforces the use of multiple appointment 
reminders as standard practice in mental health.  
 
Although other modalities may be used, in this proposal, we focus on enhancing letter 
and/or postcard reminders for several reasons: 1) a VA systematic review concluded 
that “evidence does not demonstrate that any particular reminder type (i.e., phone, 
postal, or text) has a clear net benefit over any other”15; 2) other approaches such as 
phone and text message reminders frequently do not reach their targeted 
audience31,44,45; 3) letters and postcards are reliably received by Veterans (see 
“Intervention Feasibility” in section “C” below); and 4) letters and postcards are VHA’s 
most common reminder strategy, used in 87% of VHA service lines nationally46, 
providing major advantages for large-scale implementation of a reminder intervention.  
 
Why change current appointment reminders? 
Appointment reminders typically only provide a “bare bones” reminder of time, date, and 
place, which—while certainly helpful—is more or less as was done 30 years 
ago11,14,43,47. The graphic above (see Figure 2) shows the body of a current reminder 
letter at VAPORHCS, and our review of appointment reminders from several VHAs 
across the nation (provided to us by the Office of Mental Health Operations) revealed 
similarly basic reminders. While some reminder features have been frequently 
examined, the content of the reminder message has rarely been studied10,12–15,44,47–51.  
 
Figure 2. VAPORHCS Current 
Reminder Letter 

7 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
What theoretical concepts can be applied to improve appointment reminders?  
We use concepts from behavioral economics (BE)52 and the Theory of Planned 
Behavior53 to inform our intervention content. Figure 3 presents our conceptual 
framework for this study. One of the overarching principles within BE is that people, 
while often well-intentioned, are often imperfect. To help overcome these predictable 
limitations, “nudges” can be designed that makes it easier for people to do the “right” 
thing while still retaining choice. In addition, the Theory of Planned Behavior suggests 
other drivers of a patient’s likelihood of performing a target behavior (here, attendance 
of a healthcare appointment), including their level of intention, which is in turn influenced 
by attitudes, norms, and perceived behavioral control. We believe effective nudges to 
reduce no-shows can be designed using both BE and TPB concepts and inserted into 
appointment reminders.  

8 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
1. Social Norms: 
The field of social 
psychology has 
established how the 
social environment 
influences our 
perceptions and 
behaviors. In 
particular, the 
tendency for people 
to model their own 
behavior on what 
people around them 
are doing is called a 
descriptive social 
norm. Conforming to social norms is often an individual’s chosen behavior because it 
provides a convenient decision-making heuristic that obviates the need to think critically 
about the consequences of each decision before acting on it, a process called fixed-
action patterns54.The effect of social norms may be especially strong among Veterans 
who tend to view themselves as part of a tight knit peer group55.  
 
2. Behavioral Instructions and Intentions: Providing easy-to-understand instructions 
is a simple but effective technique to increase the likelihood of a person following 
through on the target behavior23. In addition, nudging people to develop a plan in the 
form of “If situation A occurs, I will implement B response” (called implementation 
intentions) increases attainment of goal. Both lab-based and real-world experiments 
demonstrate effectiveness of this nudge1,24. 
 
3. Consequences for Self: The Theory of Planned Behavior suggests the perceived 
consequences of a behavior can be a potent motivator. Further, BE research on taking 
risks suggests that we feel losses to be about twice as painful as the pleasure from an 
objectively equivalent gain, and has led to use of the term loss aversion25,26. More 
broadly, a large literature on framing effects suggests that stating a behavior in terms of 
a potential loss (e.g., to one’s health or to their scheduled appointment) can also be an 
important motivator for health behaviors27.  
 
4. Consequences for Others: Another major behavioral insight is that, in contrast to 
traditional economic theory, humans do not act purely out of self-interest; fairness and 
consideration to others can also be a significant influence on our behavior,28,29 
particularly with people belonging to the same group. The “band of brothers” belief in 

9 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
caring for and respecting other Veterans acts as an especially strong social and moral 
bond, and qualitative interviews suggest Veterans’ perceptions of “other Veterans” play 
a major role in decisions to attend appointments and engage in treatment56,57. 
Therefore, pointing out harm of missing an appointment to other Veterans may promote 
the pro-social sharing use of public goods such as VA appointments 58.  
 
What knowledge gaps need to be addressed to effectively apply these insights to 
reduce no-shows and enhance access? 
How to adapt BE and Theory of Planned Behavior principles into messages in 
enhanced appointment reminder letters or postcards: While the theoretical and 
empirical basis for nudges is fairly well-established, the operationalization and 
translation of these concepts into the setting of appointment reminders is not. Moreover, 
it is not known whether psychological harms might be associated with these messages.  
 
Relative impact of different nudge strategies: There is evidence to suggest the utility of 
each individual BE principle in enhancing appointment reminders 16,18,19,21,31,59. However, 
studies evaluating multiple nudges and appointment reminders are lacking. It is also 
unclear whether effect sizes vary depending on the particular concept or principle used 
in a nudge. A recent large systematic review concluded that studies including multiple 
appointment reminder strategies are lacking10.  
 
Effectiveness of enhanced appointment reminders in varying clinical and patient 
populations: Although not all studies have been consistent, empirical data often suggest 
that the largest impacts on reducing no-show rates with appointment reminders have 
been in mental health11,44,47,60. This may result in part from higher baseline no-show 
rates in mental health compared to other specialties. Having more room for 
improvement may be important for success of BE interventions trying to improve patient 
adherence61. Primary care is also a vital setting to conduct this research. First, 
treatment access in primary care is a VHA high priority. Second, the risk for missed 
appointments may be rising in primary care62. Because many VHA primary care clinics 
are shifting to patient-centered scheduling (from recall scheduling), more appointments 
are being scheduled months in advance (i.e., at the time of the last appointment), a 
well-known risk factor for no-shows2. Reminders may be especially vital for new 
patients: they are known to have higher no-show rates than established patients2, may 
be unfamiliar with clinic procedures, and especially responsive to appointment-related 
messages63. 
 
SIGNIFICANCE AND INNOVATIONS 
This proposal directly tackles the priority areas of access and mental and 
behavioral health. In personal discussion with Dr. Teo, Senior Medical Advisor to the 

10 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
Office of Veterans’ Access to Care, Dr. Mike Davies, has described reducing no-shows 
as “critical to efforts to improve access,” a belief supported by empirical data linking no-
shows to access and wait time7,40. We use mental health clinics as one of the treatment 
settings in this proposal because of their elevated no-show rates6,10,43 and because 
mental health appointments are typically allocated more time, with therefore fewer 
available slots.  
 
Applying insights from BE to enhance appointment attendance is innovative. With 
the exception of a few studies in the U.K.’s National Health Service21,31, BE has not 
been incorporated and evaluated in the design and content of appointment reminders. 
Nonetheless, BE has a robust scientific grounding, with a broad array of successful 
applications in contexts ranging from financial planning to consumer energy use to 
healthcare19,27,64,65.  
 
This proposal is closely aligned with VHA operational partners and their 
priorities. Our partnerships include the: 
• 
Office of Veterans Access to Care (OVAC): We have had discussions with Senior 
Medical Advisor to OVAC, Dr. Mike Davies, who is strongly supportive of strategies to 
reduce no-shows, views this proposal as addressing key needs in improving Veteran 
access, and has offered to advise on future implementation opportunities.  
• 
Office of Mental Health Operations (OMHO): We have discussed our research plan with 
leaders including Executive Director Dr. David Carroll and Senior Consultant Dr. Lisa 
Kearney who are enthusiastic about the proposal, its potential to enhance Veterans’ 
mental health care, and dissemination in VHA mental health.  
• 
National Initiative to Reduce Missed Opportunities (NIRMO): We have had discussions 
with leadership at the Pittsburgh Veterans Engineering Resource Center (VERC), where 
NIRMO is managed. They strongly support a research focus on appointment reminders, 
and will also be proactive in advising on dissemination of improvements to appointment 
reminders. 
• 
VAPORHCS Executive Leadership: Multiple members of our local leadership team, 
including the Acting Chief of Staff, vigorously support this proposal, deem it a high 
priority and high-impact area, and fully back its implementation.  
VHA is uniquely and strategically positioned to magnify small intervention effects 
at a population-level. VHA is an ideal target for applying BE insights to healthcare 
because of the opportunity to scale up across VHA settings. If large numbers of 
Veterans in VHA receive small but effective behavioral “nudges” through improved 
appointment reminder letters and/or postcards, the benefits quickly accrue. Even a one 
percent reduction in no-show rates nationally would translate, annually, into 90,000 
additional completed patient visits. 
 
This proposal has potential to directly and rapidly improve routine clinical 
practice, and perhaps the Triple Aim of healthcare66. This proposal was intentionally 

11 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
designed to focus on making modifications to existing appointment reminder systems in 
order to be integrated at minimal cost with routine practice in VHA. Intervention design 
includes multiple elements that enhance real-world effectiveness (e.g., lack of exclusion 
criteria, no demands on clinicians, and a simple format)67. 
 
Preliminary Studies 
In this proposal, the project team will leverage its considerable experience with 
randomized trials in VHA settings, health services research on mental health and 
access, and Veteran and other stakeholder engagement. The preliminary work in 
preparation for this proposal includes:  
1. VHA administrative data preparation: The team has extensive experience using 
and generating local, VISN-level, and national data reports from the Corporate 
Data Warehouse (CDW) and VSSC databases, which contain accurate and up-
to-date information on no-shows, cancellations, and other appointment 
information68–72. In addition, through an ongoing quality improvement project, 
Drs. Teo and Kaboli have working relationships with VSSC data scientists, 
constructed data definitions on appointment status and no-shows, and prepared 
and tested code to extract CDW data of sufficient granularity for our proposed 
analyses. 
2. Large project management experience: Our team has the skills and experience 
to manage a RCT examining BE and access-related outcomes. Dr. Dobscha has 
served as PI of two HSR&D-funded cluster RCTs in VHA75,76. Dr. Zikmund-Fisher 
has current and recent experience as PI of numerous federally-funded grants 
testing interventions using communication strategies related to BE77–79.  
3. Intervention acceptability: Two local VHA patient educators have provided input 
on draft interventions to ensure appropriateness for Veterans of lower literacy 
and numeracy backgrounds. In addition, Dr. Teo conducted, recorded, and 
transcribed four focus groups with a total of 14 Veteran stakeholders at 
VAPORHCS. Qualitative comments revealed the following key results. Veterans 
do receive, review, and rely on appointment letters and/or postcards from VHA 
(some even bring them to appointments, especially for visits in a new clinic). 
Veterans are frequently dissatisfied and confused by current appointment letters 
and postcards. The kinds of changes proposed in our interventions are 
acceptable to Veterans. They described sample intervention reminders as 
“interesting,” “caring,” and “engaging.”   
4. Intervention feasibility: We have conducted several tests to ensure smooth 
implementation of our proposed intervention. First, we have verified that patients 
have a very high probability of actually receiving pre-appointment letters or 
postcards. To evaluate this, Dr. Teo used administrative data to construct a 
sample of 1,110 VAPORHCS patients who had had an in-person primary care 
visit and positive depression screen in the last year. Of these patients, postal 

12 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
addresses were correct for 95.6% based on phone and postal verification efforts. 
Second, we have confirmed that the vast majority of appointments are scheduled 
at least two weeks in advance, which is a timeframe used to determine whether 
to mail a reminder letter or postcard. To determine this, we extracted data 
comparing scheduled appointment “create date” and appointment date. Results 
from the VSSC pending appointment report showed that in July 2016, 92.8% of 
all VHA scheduled appointments nationally in mental health were created at least 
two weeks in advance (94.7% in primary care), with similarly high rates at 
VAPORHCS.  
5. Pilot effectiveness study: We conducted a pilot effectiveness study of a version of 
the most basic proposed intervention. Reminder templates were changed on July 
1, 2016 in randomly selected mental health clinics across VAPORHCS, and no-
show rates were compared between intervention and control clinics in the pre- 
(April-June 2016) and post-intervention (July-Sept. 2016) periods. Difference-in-
difference results showed an absolute decrease of 0.6% in no-shows, and 3.8% 
increase in appointment attendance in association with the intervention. This 
suggests that in three months in just a handful of clinics, over 100 no-shows were 
averted by making a one-time change to pre-appointment reminder templates. 
Assuming this same effect size and application to all mental health and primary 
care clinics in VAPORHCS, we estimate this basic intervention alone could avoid 
over 1,500 no-shows [.006*(102,060 mental health + 157,587 primary care 
appointments/year)].    
 
Research Design and Methods  
A. Design Overview:  
The primary objective of this study is to systematically develop and test the 
effectiveness of a series of changes to appointment letters or postcards in a cluster 
RCT with the goal of reducing no-shows and increasing access. First, we will iteratively 
obtain feedback and refine messages in our draft interventions (Aim 1). Then, we will 
evaluate the effects of four enhanced appointment reminder interventions informed by 
BE and the Theory of Planned Behavior, and compare them to usual care and each 
other (Aims 2 and 3). Appointments will be randomly allocated to one of the four 
interventions or usual care at the provider-level, with providers blinded to allocation. The 
trial will be conducted at eight locations across VAPORHCS in both mental health and 
primary care. In addition, we will conduct a qualitative assessment with key informants 
(Aim 4) to provide data that will be useful for future implementation (if the intervention is 
effective) or suggest modifications to the intervention (if it is ineffective).  
Sites: The effectiveness trial will be conducted in mental health and primary care clinics 
at eight locations, both metropolitan and rural, across VAPORHCS: one Medical Center 
(Portland), one Division (Vancouver), and six Community-Based Outpatient Clinics 
(Hillsboro, Fairview, West Linn, Salem, Bend, and North Coast). Conducting this study 

13 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
in a single VA Health Care System containing numerous locations allows us to harness 
sufficient power for our analyses and a shared geography for logistical ease, while still 
maintaining representation of the broader VHA population. Primary care is included as a 
second treatment setting to evaluate generalizability of treatment effects, given 
distinctive characteristics of patients in mental health. Analyses will be conducted 
separately for each specialty.  
 
B. Data Source: Aims 1, 2, 3: All research material will be derived from existing records. 
VHA Corporate Data Warehouse (CDW) and VHA Support Service Center (VSSC) 
databases will be used to identify the main study sample and provide descriptive 
characteristics of the scheduled appointments and patients in our sample. VHA Survey 
of Healthcare Experiences of Patients (SHEP) data will be used to assess appointment 
wait time. Permissions will be obtained and data use agreements completed as 
appropriate to access data from CDW, VSSC, and SHEP. CDW data will be accessed 
via VA Informatics and Computing Infrastructure (VINCI) and will be saved locally. The 
research team has extensive experience obtaining permissions for, and working with, 
these data sources.  
 
We will obtain data from these databases for up to two years preceding the study start 
date and two years following the study completion date. Variables extracted from these 
databases will include patient demographics (age, sex, race/ethnicity, service 
connection status, and rurality), patient diagnoses (proportion with common chronic 
conditions such as depression, PTSD, diabetes, and hypertension), appointment 
features (provider, location, specialty, clinic type, new vs. follow-up, date, outcome [i.e., 
completed, canceled, or no-show].  
 
Aim 4: Research materials will include audio files, transcripts, and cards/notes from 
qualitative interviews with patients. These materials will be obtained specifically for this 
research project. 
 
C. Study Population: 
Sample and Recruitment: This project has two primary study populations involved in 
the 4 Specific Aims: 
1. Patients treated at VAPORHCS with at least 1 scheduled individual outpatient 
appointment in a primary care or mental health clinic during the 12-month 
intervention period (Aims 1, 2, and 3). 
2. VHA staff located throughout VHA nationally (Aim 4).  
 
We expect our study sites to yield patients with characteristics (except for race) very 
similar to elsewhere in VHA. Specifically, CDW data from the 12 months ending June 
2016 indicate that our no-show rates vary from national figures by just 0.1% in mental 

14 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
health and 1.5% in primary care (<0.5 standard deviation of variation). Similarly, there is 
<1% difference in proportion of Veterans with service disability, PTSD diagnosis, or 
male gender between local and national figures; mean age differs by just 1.6 years. 
Proportion of racial minorities (including unknown/declined) is 21% locally vs. 28% 
nationally. 
 
A sample for Aim 1 interviews in this project cannot be pre-specified as the study design 
involves iterative refinement to the intervention based on ongoing input from the 
participants. However, we will conduct at minimum three waves of six interviews (n=18). 
Interviews will be audio-recorded and last approximately one hour.  
 
The number of subjects for Aims 2 and 3 in this project is estimated based on the 
number of patients and providers associated with the approximately 102,060 and 
157,587 scheduled appointments in mental health and primary care clinics, respectively, 
over a 12-month period. This includes up to 64,000 unique patients, 275 unique mental 
health providers, and 150 unique primary care providers. The number of subjects for 
Aim 4 in this project is estimated as 40, consisting of 10 patients at VAPORHCS and 30 
VHA staff nationally. 
 
Based on our recruitment procedure and data available regarding the number of eligible 
appointments, unique patients, and providers in mental health and primary care at 
VAPORHCS, we do not anticipate any difficulties achieving our recruitment goals for 
Aim 2 and Aim 3. For Aim 1, our sample size (n=18+) is small and the pool of potentially 
eligible subjects large. Similarly, for Aim 4, our sample size (n=40) is small and the pool 
of potentially eligible subjects large. If subject recruitment and enrollment lags, we will 
try to identify and address causes in meetings with research team members and Co-
Investigators. 
 
For Aim 1 subjects who agree to join and do not withdraw from the study before all 
procedures are complete, participation in this study will last for up to one year. For Aim 
2 and Aim 3 subjects, participation will last for four years. For Aim 4 subjects, 
participation will last for up to one year. 
 
D. Subject Identification/Recruitment:  
Inclusion/Exclusion Criteria: All subjects will be a minimum of 18 years old. The 
health status of the patients will be variable, although all will be patients in ambulatory 
care settings. Race and ethnicity will not be used in determining inclusion or exclusion 
of subjects.  
 
Aim 1 

15 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
• Eligibility Criteria for Subjects: 1) patient with at least 1 ambulatory appointment 
scheduled at least two weeks before the appointment date during the study 
period; 2) appointment is in either primary care (limited to stop code 323) or 
mental health (limited to individual stop codes used by VSSC and tracked 
nationally: 502, 534, 513, and 540); and 3) 1 of the above eight VAPORHCS 
locations. 
o Exclusion Criteria for Veteran Subjects: 1) patients with hearing problems; 
2) lack of regular access to a telephone 
• Recruitment of Subjects: First, we will use CDW to identify all patients meeting 
the above eligibility criteria. Patients will be stratified along several criteria to 
promote purposive sampling based on: number of scheduled appointments in the 
prior 12 months (both primary care and mental health), number of scheduled 
appointments attended vs. no-showed in the prior 12 months, new vs. 
established patient, service era, and gender. Second, we will mail opt-out study 
invitations in batches of 50 letters. Third, we will conduct a phone call to 
determine Veteran interest in an in-person interview, as well as capacity to give 
informed consent. Fourth, during an in-person study visit, we will consent and 
enroll participants who indicated interest and have apparent capacity to give 
consent. We will enroll between 18-36 participants for the Aim 1 interviews and 
design sessions.  
 
Aim 2 and Aim 3 
• Eligibility Criteria for Subjects: 1) patient with at least 1 ambulatory appointment 
scheduled at least two weeks before the appointment date during the study 
period; 2) appointment is in either primary care (limited to stop codes 179, 
322,323, 324, 338, 690, 692, or 693) or mental health (limited to individual stop 
codes used by VSSC and tracked nationally: 502, 513, 527, 528, 542, 545, 534, 
and 540); and 3) 1 of the above eight VAPORHCS locations. 
o Exclusion Criteria for Veteran Subjects: 1) patients with hearing problems; 
2) lack of regular access to a telephone 
• Recruitment of Subjects: These subjects will not be individually enrolled. Instead, 
we will modify pre-appointment reminder letters or postcards sent routinely to 
Veterans with scheduled individual outpatient mental health and primary care 
appointments. We will obtain a waiver of authorization and informed consent for 
screening/recruitment, and waiver of informed consent documentation for 
subjects. Based on FY 2015 data, we estimate our locations to provide 102,060 
and 157,587 scheduled appointments in mental health and primary care clinics, 
respectively, over a 12-month period. These appointments are associated with a 
total of about 64,000 unique Veterans, 275 unique mental health providers, and 
150 unique primary care providers.  

16 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
Aim 4 
• Eligibility Criteria for Subjects: Aim 4 subjects will consist of two groups of key 
informants: Veterans and VHA staff. Eligibility for Veteran subjects will be the 
same as described above for Aim 1.  
o Exclusion Criteria for Veteran Subjects: 1) patients with hearing problems; 
2) lack of regular access to a telephone 
• Eligibility for VHA staff will require being a full-time VHA employee. Administrators 
recruited for interviews will have administrative oversight of access, scheduling, or 
appointment operations and procedures. This would include group practice 
managers. At a national level, this would include leadership at VHA offices such 
as OVAC and NIRMO, as well as the Office of Connected Care, which is leading 
development and implementation of text messaging-based appointment 
reminders via the “Annie” app. Other VHA staff that will be recruited for interviews 
will be those directly involved in patient scheduling, who have responsibilities 
involving the building, allocation, and/or printing of appointment reminder letters.  
• Recruitment of Veterans: Although Aim 4 Veteran subjects will be distinct from the 
Aim 1 Veteran subjects, recruitment processes will be similar: 1) use of CDW data 
to identify and sample potential participants; 2) opt-out study invitation letters 
mailed in batches of 50 letters; 3) phone calls to determine Veteran interest, 
capacity to give informed consent, and interview modality preference (in-person, 
telephone, or video conference – if this modality is preferred we will obtain the 
potential participant’s email address to send invite for consent discussion 
meeting); and 4)consent and enrollment of the participant (in-person, telephone, 
or video conference), followed by an interview (either in-person, over the phone, 
or over a video conferencing application). Veterans (n=10) will be purposively 
sampled to provide representation of new and established patients in both primary 
care and mental health settings at VAPORHCS. Among established patients, 
those with both lower (<10%) and higher (>20%) no-show rates will be included. 
Ten interviews are typically sufficient for obtaining key themes when using the 
rapid assessment process proposed in our data analysis for this aim81.  
• Recruitment of VHA staff (n=30;): will be recruited via email, with support from our 
operational partners (OVAC and NIRMO). After recruitment via email, before the 
interview is conducted, we will ask the employee if they would prefer to be 
interviewed on VA time or non-VA time. If they would prefer to be interviewed on 
VA time, we will obtain permission from their supervisor to conduct the interview 
on their VA time. VHA staff will only participate in one phone/virtual interview – no 
other study activities or procedures will be conducted; however, we can contact 
the VHA staff after the interview for follow-up questions and/or clarification. 
 

17 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
Required and Maximum Enrollment 
A maximum of 84 participants will be consented and enrolled. Patients will be 
enrolled at the VA Portland health Care System for a maximum of 51 patients. 
The maximum number of VHA staff to be consented and enrolled is 33. These 
maximums are based on a 10% screening failure/decline rate such that: 
 
Human Subjects 
Required 
Maximum 
       (Required + 10%) 
Aim 1 patients 
18-36 
40 
Aim 4 patients 
10 
11 
Aim 4 VHA staff 
30 
33 
Totals 
76 
84 
 
A maximum of 70,400 records will be accessed. This maximum is based on a 
10% failure rate such that: 
 
Records 
Required 
Maximum 
(Required + 10%) 
Aims 2 and 3 
64,000 unique 
veteran 
records 
70,400 
Totals 
64,000 
70,400 
 
E. Intervention Development: 
Intervention Development (Aim 1): We will conduct qualitative interviews informed by 
design thinking to iteratively develop and refine our draft interventions 82. The goals of 
this intervention development process are to compare whether message perceptions 
align with our predicted BE theoretical constructs, understand how patients perceive  
each intervention and its component messages, and minimize risks of emotional harm 
induced by the messages. 
 
During interviews, we will conduct two activities. The first will involve a pile sorting 
exercise. Pile sorts, also called card sorts, have methodological roots in anthropology 
and can be a simple but powerful way to explore and map relationships between 
concepts83. Participants will be introduced to our conceptual framework and the 
meanings of key concepts (e.g., social norms, behavioral intentions). Then, they will 
receive cards with various brief messages written on them, including our draft 
messages. They will sort messages into piles, with each pile representing a different 

18 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
concept. Blank label cards will be included to allow introduction of new piles (concepts) 
in which to categorize the draft messages. In the context of our interviews, pile sorting 
will facilitate a conversation in which participants organize and map out the relationship 
between messages and their underlying meanings. It would be difficult for a Veteran to 
directly answer the question “What concept is represented by X message?” The pile 
sort approach helps determine what a participant includes in their universe, or their 
thinking, and grounds generalizations in particulars. In addition, by doing pile sorting 
first, we will be able to sufficiently orient the participant to our conceptual framework to 
facilitate elicitation of feedback on appointment reminder messages in the second 
phase of the interview. Once the participant has sorted the cards, the interviewer will 
take a photograph of how the cards are laid out on the table, to be referred to later 
during data analysis. The participant, along with any PHI/PII, will not be included in the 
photograph. 
 
After pile sorting, the second activity during the interview will be a user experience 
design session84, an area that Co-I Zikmund-Fisher has expertise in from his work 
designing health communication tools and decision aids. This process, which draws on 
techniques used in design thinking, prioritizes consulting with prospective users early 
and often, and will help the study team to better incorporate patient perspectives in the 
design of the final appointment reminder messages. In the design session, we will 
present participants with: a) a status quo appointment reminder letter; and b) draft 
intervention appointment reminder letters. After identifying the changes with 
highlighting, we will query participants about how each message is perceived, 
interpreted, and comprehended. Questions will also specifically probe about potential 
harms (e.g. feelings of guilt) and the principles we are trying to evoke (e.g., social 
norms).  
 
Iterative design process: We will begin with a wave of six interviews (pile sort + design 
session). After each wave of design sessions, we will construct a spreadsheet compiling 
participants’ feedback for each draft message. This spreadsheet will function similarly to 
a codebook—facilitating identification of themes in participants’ interpretations of the 
messages. The research team will collectively review these responses to make 
consensus decisions about revisions to the interventions. Members of the Veteran 
Engagement Group (VEG) at our HSR&D Center to Improve Veteran Involvement in 
Care (CIVIC) will also participate in selected design meetings to offer additional 
Veteran-centered input. Depending on whether messages map onto our proposed BE 
and Theory of Planned Behavior concepts and how messages are perceived, the team 
may change wording in messages, delete messages, or develop multiple message 
versions to test in the next wave. We will continue these six-interview waves until we 
reach stable intervention content. Because the design process is iterative, a final 
sample size cannot be pre-specified, although based on experience we expect to 

19 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
conduct at minimum of three waves (18+ interviews). Six or more waves are possible. 
Interviews will last approximately one hour, and be audio-recorded to facilitate accurate 
capture of all relevant comments.  
 
We expect two deliverables from this work. First, we will possess refined intervention 
messages and an empirically-validated mapping of messages onto theoretical concepts. 
Second, we will prepare one or two manuscripts, tentatively titled “Operationalizing 
Behavioral Economic Principles in Appointment Reminders: A Qualitative Study” and 
“Pile Sorting and Design Sessions as Methods for Qualitative Analysis.” 
 
F. Randomization: 
Randomization (Aims 2 and 3): The unit of randomization will be provider, with two 
independent randomization procedures for primary care and mental health providers. 
Providers will be blinded to study group (treatment) assignment because appointment 
reminder templates are managed administratively without providers seeing them. 
Randomization will be blocked by location to account for variation in clinic procedures, 
no-show rates, and to ensure balance in treatment assignments within each location. 
Because some providers work in multiple locations and clinics, they may be randomized 
more than once.  
 
G. Study Groups and Intervention Description: 
Overview: There will be a total of five study 
groups, four intervention conditions and usual 
care (control group), with equal treatment 
allocation. There will be a total of five study 
groups, four intervention conditions and usual 
care (control group), with equal treatment 
allocation. As shown in Table 1, each 
intervention arm contains two or more different 
types of behavioral nudges: social norms, 
behavioral instructions, consequences for self, 
consequences for others, and caring. As 
described in the Background and conceptual 
framework, we used principles from BE and 
Theory of Planned Behavior to inform the 
messages in each intervention arm. For example, 
Intervention 1 will test the nudges based on the 
concept of social norms and behavioral 
Table 1: Summary of Type of  
Nudge in Each Study Group 
Study Group 
Nudge 
Intervention 1 
Social Norms  
Behavioral Instructions 
Intervention 2 
Caring  
Consequences for Others   
Behavioral Instructions 
Intervention 3 
Caring  
Consequences for Self 
Behavioral Instructions 
Intervention 4 
Caring  
Consequences for Others  
Consequences for Self  
Social Norms  
Behavioral Instructions 
Control 
None 

20 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
instructions messages. All intervention arms will include techniques to increase salience 
of nudges (e.g., selective bolding of text).  
 
By examining different types of nudges 
in different intervention arms, this 
project will help advance scientific 
understanding of the mechanisms 
behind behavior change related to 
appointment attendance. If successful, 
it would also lay the groundwork for a 
subsequent study where we would 
have the resources to conduct full 
factorial testing, identify optimal 
combinations of messages/nudges, 
and determine whether synergistic 
effects can be achieved by combining 
messages. Table 2 contains a more 
detailed description of what kinds of 
messages fall into each of the four 
categories of nudges. For most of the 
four concepts, multiple draft messages 
are presented.  
 
Intervention 1 (Social Norms + Behavioral Instructions): A letter with two types of 
nudges. One points out the common behavior of attending appointments. And one 
provides clear, specific instructions for making appointment changes. The appointment 
reminder also includes usual care (basic appointment information on date, location, and 
phone number(s) for scheduling changes). 
 
Research has shown that social norms messages have a positive impact on behavioral 
action, however it is not clear how effective it is in comparison to messages framed 
around consequences. Additionally, we did not use the caring message in this 
intervention arm to allow for comparison of whether the caring message on it’s own has 
a positive impact.  
 
Intervention 2 (Caring + Consequences for Others + Behavioral Instructions): A 
letter with three types of nudges. One suggests that the institution cares about the 
patient. One highlights a potential negative consequence for others if the patient no-
shows. And one provides clear, specific instructions for making appointment changes. 
Table 2: Messages included in the appointment  
reminder based on the related concept 
Concept 
Message 
Caring 
We’re here for you. 
Consequences 
for Others  
Call now so we can help another 
Veteran in need. 
Consequences 
for Self  
Attending appointments lowers your 
chances of being hospitalized. 
Consequences 
for Self  
If you miss your appointment, you 
may have to wait a while to be seen. 
Social Norms 
Most Veterans make a point to attend 
their VA appointments. If they can’t 
make their appointments, most 
Veterans also make an effort to let us 
know. 
Behavioral 
Instructions 
Need to change or cancel your 
appointment? 
Take these 2 simple steps today: 
1) Call the clinic at 503-555-5555. 
2) Give your name, last 4, and 
appointment information. 
It’s fine to leave a message. 

21 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
The appointment reminder also includes usual care (basic appointment information on 
date, location, and phone number(s) for scheduling changes). 
 
During the development of the conceptual messages the gain framed consequences for 
others message and the caring message both received favorable responses, and 
participants often put these messages together. These messages emphasize a 
personal connection between the recipient of the letter, the clinic, and other patients at 
the clinic. This positive message and personal connection calls upon a sense of social 
responsibility, which we anticipate will have a positive impact on no-show rates.  
 
Intervention 3 (Caring + Consequences for Self + Behavioral Instructions): A letter 
with three types of nudges. One suggests that the institution cares about the patients. 
One highlights potential negative consequences for the patient if s/he no-shows. And 
one provides clear, specific instructions for making appointment changes. The 
appointment reminder also includes usual care (basic appointment information on date, 
location, and phone number(s) for scheduling changes). 
 
On their own, each consequences-for-self message had mixed reactions. By including 
both messages we hope that if one concept does not appeal to a participant, that the 
second will. By including the caring message, we are conveying the idea that the 
reminder of the consequences if they no-show is coming from a place of caring for the 
recipient of the letter, and thus whether framed as a personal gain or loss the recipient 
may be more open to hearing the consequences-for-self messages and be gently 
nudged to action. 
 
Intervention 4 (Caring + Consequences for Others + Consequences for Self + 
Social Norms + Behavioral Instructions): A letter with all four types of nudges 
combined. One suggests that the institution cares about the patients. One highlights a 
potential negative consequence for others if the patient no-shows. One highlights 
potential negative consequences for the patient if s/he no-shows. One points out the 
common behavior of attending appointments. And one provides clear, specific 
instructions for making appointment changes. The appointment reminder also includes 
usual care (basic appointment information on date, location, and phone number(s) for 
scheduling changes). 
 
While we anticipate that each intervention arm will have an impact on No-Show rates, it 
is not clear which messaging combination will have the greatest impact. While we 
combine some messages, we also anticipate that combining more messages will not 
have an additive effect. Instead, too many messages may dilute the effectiveness of all 
the messages. To test this, we developed this intervention arm to include every 

22 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
message, to determine if behavioral nudges are best used in shorter and smaller 
combinations.   
 
H.  Procedure: Because VHA appointments are managed in the Veterans Health 
Information Systems and Technology Architecture (VistA) and related data systems, 
letters and postcards will be designed and formatted within the constraints of VistA and 
related system requirements. The research team will work with and support clinical 
application coordinators in mental health and primary care to modify, audit, and 
maintain letter template changes. Our research team is experienced with staffing 
resources and procedures necessary in preparing large and customized patient 
mailings85. Because letter templates are unique to each provider and his/her associated 
clinic(s), the research team will take advantage of this specificity when tailoring 
messages. Intervention content will be applied to pre-appointment reminder letters 
and/or postcards, which in VAPORHCS are mailed at the time of scheduling. (Our 
preliminary data shows that, as of July 2016, in VAPORHCS, appointments are 
scheduled, on average, 22-33 days before the appointment in primary care and mental 
health.) In addition, intervention content will also be adapted to no-show letters and/or 
postcards, which are sent after a no-show, to help reduce risk of repeated no-shows.  
 
 
I.  Measures:  
Outcomes for Aim 2 
(Determine the effect of 
four versions of 
enhanced appointment 
reminders on measures 
relevant to treatment 
access, compared with 
usual reminders) and 
Aim 3 (Evaluate 
differences in treatment 
effect associated with 
four versions of enhanced appointment reminders): Table 3 summarizes the outcomes 
used in these Aims.  
Our primary outcome will be no-show rate, or the proportion of total appointments that 
are classified as a no-show, relative to the total number of appointments scheduled. The 
numerator (“no-shows”) consists of appointments marked as a no-show and 
appointments canceled by the patient or clinic after the appointment time. The 
denominator (“total appointments”) consists of no-shows and completed appointments. 
No-show rates will be measured at the provider-level.  
 

23 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
Our secondary outcome will be cancellation rate, defined as the proportion of scheduled 
appointments that are marked as canceled. In addition to overall cancellation rate, we 
will also determine the rate amongst the subset of cancellations that are advance 
cancellations and patient-initiated cancellations. Advance cancellations, defined here as 
at least five days before appointment, provide a metric with a reasonable window of 
time (informed by preliminary data provided by our schedulers) that allows the 
opportunity to fill a cancelled slot with another waiting Veteran. Patient-initiated 
cancellations provide a more specific indication of intervention effects on cancellation 
behavior. Cancellation rates will be measured at the provider-level. 
Finally, as exploratory outcomes, we will calculate wait time and number of unique 
patients seen per day. The latter is simply the number of daily completed encounters, 
adjusting for patients who have multiple encounters in the same day. For wait time, we 
will use multiple validated metrics:  
• 
Raw wait time is defined as the length of time in days between the date an appointment 
was created and the date the appointment was completed74. If an appointment is 
cancelled or a no-show, the wait time will be calculated based on the next appointment 
that is scheduled and completed in the same clinic stop code.  
• 
Clinically indicated wait time accounts for the “return to clinic” date assigned by the 
patient’s provider. In this metric for established patients, wait time is defined as the 
length of time in days between the provider-entered clinically indicated date and the date 
of the appointment. This is to account for the fact that appointment create date as the 
start date for measuring wait times may not reflect patient or provider preferences of 
when the patient should be seen. Limitations in VHA scheduling software have 
prevented systematic capture of these data until summer of 2017 when a technological 
fix was implemented. Co-I Dr. Prentice will be examining this metric in the spring of 2018 
using a VHA national dataset, and we will include this measure if validated.  
• 
Waiting room time will help determine intervention impact on wait time on the day of the 
appointment. To assess this, we will examine clinic- and facility-level data from the VHA 
Survey of Healthcare Experiences of Patients (SHEP), which includes several variables 
related to patient experience and satisfaction with wait time (e.g., “Wait time includes 
time spent in the waiting room and exam room. In the last 6 months, how often did you 
see this provider within 15 minutes of your appointment time?”). Our research team has 
been in communication with SHEP leadership and confirmed the feasibility of obtaining 
and analyzing these measures for our study sites. We also have experience with 
obtaining data use agreements for SHEP and linking data to individual Veterans. A 
sample size of approximately 4,000 subjects is available annually from SHEP data 
obtained from our study sites, many of who are likely to be included in the trial. We will 
conduct descriptive comparisons of Veterans in intervention and control arms. 
• 
New mental health appointment wait time helps capture wait time for an important group: 
Veterans new to mental health. We will calculate this using another validated metric, 
defined as the length of time in days between the date the consult was created and the 
date the consult was completed89. 
 

24 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
These no-show, cancellation, and wait time metrics are consistent with ones used by 
our operational partners and VSSC. To construct these standardized measures, we will 
extract administrative data in CDW on appointment/encounter variables, including date, 
location, provider, stop code, and cancellation remarks. Wait time measures will be 
stratified based on new vs. established patients. “New patients” are those who have not 
completed another appointment in a given specialty and parent VA facility during the 
prior 24 months. 
 
Baseline characteristics (Aims 2 and 3): To demonstrate balance among study arms, we 
will extract information from CDW on subject demographics (age, sex, race/ethnicity, 
service connection status, and rurality), diagnoses (proportion with common chronic 
conditions: depression, PTSD, diabetes, and hypertension), and appointment features 
(clinic type, new vs. follow-up, provider, and time between appointment creation and 
actual appointment). We will use protocols and code for data extraction previously 
established by Drs. Prentice and Dobscha for these variables. 
 
Qualitative assessment (Aim 4): The primary goals of our semi-structured interviews are 
to identify features of our interventions that act as facilitators or barriers to widespread 
implementation, and to suggest potential modifications or adaptations to the 
interventions that would promote larger-scale implementation, long-term maintenance, 
and adaptability to other reminder modalities. We will develop and pre-test interview 
guides for VHA staff and Veterans.  
 
The VHA staff interview guide will be informed by selected constructs from domains (i.e. 
intervention characteristics, outer setting, and inner setting) in Consolidated Framework 
for Implementation Research (CFIR). CFIR is a widely used implementation framework, 
and bears relevance for our goals of conducting a formative evaluation prior to 
extensive implementation90. Constructs selected will be those most relevant to the stage 
of pre-implementation, consistent with the scope of this proposal91. Interviews with VHA 
staff (n = 30, conducted remotely) will address issues such as the interventions’ relative 
advantage, adaptability, complexity, and design quality and packaging, as well as 
features across and within varying VA facilities and operational offices that might impact 
implementation. Potential barriers to implementation, such as unintended 
consequences of the intervention, impacts on clinical workflow, and existing clinic 
practices (e.g., overbooking), will bear special attention in interviews. Likewise, we will 
solicit suggestions on how to overcome such barriers. In order to promote potential 
adaptation and implementation of the intervention to text messaging-based appointment 
reminders, interviews will explicitly explore issues related to use of this newer modality 
of reminder delivery. Interviews with the Office of Connected Care, which oversees the 
“Annie” app that employs text messages, will be especially targeted to these 

25 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
discussions. Interviews will last approximately 45-60 minutes, and be audio-recorded 
and transcribed.  
 
The Veteran interview guide will be informed in part by the intervention characteristics 
domain within CFIR. Interviews with Veterans (n = 10, conducted in-person at 
VAPORHCS or remotely) will identify relative advantage of letter and postcard 
reminders, adaptability to other Veteran populations, and Veterans’ support needs if the 
intervention were used implemented more broadly. We will also probe Veteran 
perspectives on barriers and facilitators to expanding the intervention to other non-
postal modes of appointment reminder delivery. Specifically, we will probe adaptations 
to the intervention (e.g., timing, frequency, and length of reminders) that would make it 
more appropriate to deliver by phone (AudioCare), secure messaging (MyHealtheVet), 
and/or text messaging (Annie app, VEText). 
 
J. Data Analysis: 
Aims 2 and 3: We will first conduct univariate analyses to describe the “Table 1” 
baseline demographic, diagnostic, and appointment utilization characteristics of 
subjects and clinics in the usual care and intervention conditions. Then we will tabulate 
the total number of no-shows and total number of unique subjects with at least one no-
show during the study period. For subjects with no-show(s), we will calculate the 
proportion with a single vs. multiple no-shows, and the mean and median number of no-
shows. To analyze our primary outcome, no-show rate, we will use standard mixed 
effects correlated logistic regression since the outcome is binary at the individual-level. 
(We will use a similar approach to analyze intervention effects on our secondary 
outcomes, cancellation rates.) These will model the odds of no-shows as log[ odds(pij) ] 
= bi + β0+β1*trt(1)i+ β2*trt(2)i+ β3*trt(3)i+ β4*trt(4)i+ locationj, where pij is the no-show 
probability for provider i in location j, trt(k) are indicators for the 4 interventions, and 
locationj corresponds to the location for provider i. The random effect, bi, is assumed to 
be normal with mean 0 and constant variance (2b) and will be used to account for 
repeated measures correlation for the individual subject outcomes for each provider; the 
model parameters, (β1, β2, β3, β4), represent log odds ratios of no-shows comparing the 
four intervention groups to the control group.  They will be estimated using maximum 
likelihood estimation and will be used to test the hypotheses of Aim 1. This mixed model 
is a longitudinal model and can accommodate some forms of missing data, provided the 
missingness mechanism is non-informative. We can also estimate absolute risk 
differences between the interventions and usual care. No-show rates will also be 
presented in several ways: overall rate (i.e., for all study participants in each of the five 
study arms), stratified by specialty (mental health vs. primary care), stratified by patient 
type (new vs. follow-up), and over multiple follow-up periods (six and 12 months to 
determine duration and decay over time of any effect). We will also conduct exploratory 
subgroup analyses, which may identify particular patient populations where 

26 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
effectiveness of interventions varies. For example, we will explore whether the effect of 
the interventions differ between men and women. To analyze our exploratory outcomes 
on appointment wait time, we will evaluate time-to-completed appointment using an 
extended multivariate Cox regression model to account for multiple appointments at the 
patient level and clustering at the provider level. 
Aim 2 Hypotheses: To address Hypothesis 1 (Each intervention group will have a 
lower no-show rate, as compared with usual care) and Hypothesis 2 (Each 
intervention group will have a higher cancellation rate, as compared with usual care), 
we will conduct pairwise comparisons to test for differences between the 
interventions and usual care as H0: βj=0 for Interventions j=1,2,3,4. 
 
Aim 3 Hypothesis (Effect size of nudges will vary by intervention arm such that some 
categories of nudges have a larger proportional effect than others): We will conduct 
pairwise comparisons of no-show and cancellation rates by testing the linear 
combinations between each of the four intervention arms (e.g., Intervention 1 and 
Intervention 2; H0: β1=β2). 
 
Aim 4: ATLAS.ti software will be used for data management. We will employ a “rapid 
assessment process,” which is useful for quickly developing understanding from an 
insider’s perspective and informing future implementation efforts81,92. This multi-step 
process will be as follows: 1) create domain names for each interview question; 2) 
create and test a summary template; 3) summarize interview transcripts using the 
summary template, and 4) transfer summaries into a matrix (respondent x domain) data 
display93. Our summary template will consist of domains and constructs from our 
interview guides (i.e., CFIR), and we will compose a templated summary for each 
interview. We will review these summaries on a regular basis (approximately every five 
interviews) to ensure interviews build iteratively on emergent findings. 
 
K.  Power Analyses: For our Aim 2 and Aim 3 power calculations, we used FY2016 
stop code-level no-show rates at each of the eight locations and the provider-level 
number of scheduled appointments at each location to conduct Monte Carlo 
simulations. We assumed two possible differences in no-show and cancellation rates for 
the interventions (1.9% and 2.7%), which represent changes in appointment attendance 
on par with our pilot data and two prior RCTs using similar variations in BE messages in 
appointment reminders31. Because baseline no-show rates in the RCTs were lower than 
ours, leading to possible floor effects, these estimated effects are relatively 
conservative94. We assumed a total of 256 providers randomized to the interventions. In 
addition, to account for intra-class correlation among providers and the effects of over-
dispersion, we allowed for correlated binary measurements to be correlated with 
parameter rho = (2b/2b + 2/3). We varied rho between two and five percent. All 
hypothesis tests were two-sided, with no adjustments for multiple comparisons. As the 

27 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
number of providers, patients and scheduled appointments within providers and no-
show rates differ between primary care and mental health, separate Monte Carlo 
simulations were performed to address power and sample size requirements for the two 
specialties. Each simulation was run a minimum of 500 times. We calculated power as 
the proportion of times the stated statistical hypothesis was correctly rejected.  
 
Power calculations are conducted separately for primary care and mental health. In 
primary care, results of the power analyses show that there is >99% power to detect a 
no-show rate difference of 2.7% between the intervention and usual care, and 82% 
power to detect a no-show rate difference of 1.9% (2.7%-0.6%) between two 
interventions. In mental health, results of the power analyses demonstrate that there is 
80% power to detect a no-show rate difference of 2.7% between the intervention and 
usual care, and 45% power to detect a no-show rate difference of 1.9% (2.7%-0.6%) 
between two intervention groups. In summary, the power analyses demonstrate it is 
possible to detect relatively modest effects in our Aim 2 outcomes (i.e., 2.7% differences 
in no-show or cancellation rates between interventions and usual care); it may also be 
possible to detect smaller effects in our Aim 3 outcomes (e.g., 1.9% differences in no-
show or cancellation rates between different intervention arms).  
 
L. Limitations and Alternatives 
Single-site effectiveness study: We considered whether to design this study as a multi-
site trial to address concerns about external validity. Ultimately, we favored testing our 
interventions across one VA Health Care System for three reasons. First, a single site 
allows us to examine effectiveness across two distinct specialties (mental health and 
primary care), which would be much more difficult in a multi-site study given resource 
limits. Second, given potential for differing appetites for intervention implementation 
across VHA sites, we agree on the need for a multi-site study, but see this as best 
pursued in a subsequent trial. Such a trial could be a multi-site hybrid type 2 trial95 
informed by the effectiveness (Aims 2 and 3) and pre-implementation data (Aim 4) 
from the current proposal to then evaluate multi-site effectiveness and implementation 
strategies. Third, our single site provides adequate power for our proposed analyses.  
 
Modality of appointment reminder: We considered testing our interventions using phone, 
text messaging, or other technological-savvy reminder modalities. We opted not to test 
these approaches, particularly text messaging, in the current proposal because our 
conversations with VHA informatics and other leaders suggested feasibility of 
widespread implementation is still years away, largely because the VistA system for 
appointment scheduling and appointments remains in place. Nonetheless, we believe a 
text messaging intervention would be an ideal next step, particularly once intervention 
content and message wording is refined through results from Aim 1. We have ongoing 
conversations with the VHA Office of Connected Care, which has developed the text 

28 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
messaging “Annie app,” and we anticipate the BE principles in this proposal could be 
evaluated in Annie’s text message-based appointment reminders, through VEText, or 
through MyHealtheVet upgrades that will allow appointment reminders through secure 
messaging. 
 
Intervention content: We recognize there are numerous possible interventions, varying 
by BE principles and what personalization is included. We considered a factorial design 
whereby each possible combination is evaluated, but power constraints make that 
difficult. Ultimately, it was both scientific and pragmatic reasons that guided our 
selection of messages included in each intervention.  
 
Contamination risk: We selected provider as the unit of randomization for pragmatic and 
statistical reasons. On a pragmatic level, individual patient randomization is not feasible 
for this intervention utilizing VHA’s appointment scheduling architecture in VistA. 
Statistically, provider randomization allows for enough clusters for statistical power. We 
understand there is contamination risk, as Veterans have appointments across 
providers. Our preliminary work suggests contamination risk is low. 
 
Alternative plans for Aim 4 (pre-implementation study) if the intervention proves 
ineffective: In this case, we will change the content for interviews with VHA staff to 
obtain two types of data. First, we will conduct a detailed inquiry into the acceptability, 
feasibility, and desired features of enhanced appointment reminders delivered through 
different modes (e.g., text message, secure messaging, or automated phone 
reminders). This is because a reason for trial failure could be related to the use of letters 
and/or postcards as a mode of reminder delivery. Together with the validated reminder 
wording created from Aim 1, these new data from Aim 4 could provide preliminary 
support for a reminder intervention using a different delivery mechanism. Second, we 
will elicit suggestions on modifications to the intervention for particular subpopulations of 
interest. These subpopulations will be identified through Aim 2 and Aim 3 post-hoc 
analyses meant to identify groups of patients with evidence of intervention effectiveness 
(e.g., established patients on the VHA list of scheduled patients with a high probability 
of no-show).  
 
M.  PROJECT MANAGEMENT PLAN 
This project will last for 3 years. Please see the Gantt Chart in Table 5 for information 
on timing of study activities. The majority of project personnel will be located at the 
Center to Improve Veteran Involvement in Care (CIVIC). CIVIC will provide office space, 
equipment, and administrative research support to project staff. As PI, Dr. Teo will lead 
team meetings with co-investigators and the project manager at least monthly (biweekly 
during certain study periods such as intervention development and data analysis of trial 

29 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
results) to discuss scientific issues. Dr. Teo will also have weekly meetings with other 
study personnel on operational and logistical issues. Drs. Zikmund-Fisher and Kaboli 
will fly to Portland to attend a study kick-off meeting. Subsequently, they will attend 
research meetings every two to four weeks via phone or videoconference. Dr. Zikmund-
Fisher will participate in all research team meetings, as well as ad hoc one-on-one 
meetings with Dr. Teo, during Year 1 to assist with intervention development and 
refinement, and Year 3 to assist with interpretation of Aim 1 and Aim 2 results. Dr. 
Kaboli will attend team meetings throughout the study. Drs. Teo and Kaboli will 
supervise the study’s project manager and analysts. Drs. Kaboli and Zikmund-Fisher 
will not have access to VA identifiable data. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
INFORMED CONSENT 
Patients: For Aims 1 and 4, we will seek Waivers of Authorization and Informed Consent 
Process for Screening/Recruitment Purposes. For Aim 2 and Aim 3, we will request a 
Waiver of Informed Consent Process. 
 
For Aim 1, the Waiver of Authorization and Informed Consent Process for 
Screening/Recruitment Purposes will allow us to review patients’ medical records, 
CDW, and VSSC data to identify potential patients for recruitment for interviews. After 
identifying a pool of potentially eligible subjects (Veterans), the potential subjects will be 
stratified along several criteria to promote purposive sampling based on: number of 
scheduled appointments in the prior 12 months (both primary care and mental health), 
number of scheduled appointments attended vs. no-showed in the prior 12 months, new 
vs. established patient, service era, and gender. After a potential subject is identified, 
we will send information about the study and an invitation to participate or opt-out. 
These letters will be sent out in batches to 50 patients at a time. Patient who do not opt-
Q
1
Q
2
Q
3
Q
4
Q
1
Q
2
Q
3
Q
4
Q
1
Q
2
Q
3
Q
4
Study start-up (hiring, training, SOPs)
Recruit participants (Aim 1)
Conduct interviews/design sessions & analyze (Aim 
1)
Finalize intervention content (Aim 1)
Finalize data extraction procedures/coding (Aims 2 & 
3)
Prepare for and conduct randomization (Aims 2 & 3)
Revise appointment letter templates in VistA (Aims 2 
& 3)
Collect longitudinal data (Aims 2 & 3)
Analyze quantitative data (Aims 2 & 3)
Finalize interview guides and pre-testing (Aim 4)
Recruit and conduct interviews (Aim 4)
Analyze interview data (Aim 4)
Disseminate study results (All Aims)
Table 5. Gantt Chart
Project Activities
Year 1
Year 2
Year 3

30 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
out will be contacted via their preferred method of communication (mail, phone, or 
email) to arrange an in-person study meeting at VAPORHCS. Consent and enrollment 
will take place at the same time during the in-person visit. Participants who enroll in the 
study will be involved for just one day for the completion of an interview, but data can be 
retained for the duration of the study. During the in-person study visit, we will review the 
Informed Consent Sheet and HIPAA Authorization. We will provide a copy of the Study 
Contact Sheet. After signing the Informed Consent and HIPAA Authorization, a member 
of the study team will conduct the interview with the Veteran. We will audio record the 
interviews. Audio recordings may also be sent to the VHA Salt Lake City (VHASLC) 
Centralized Transcription Services Program (CTSP) for transcription services. After the 
completed interview, the Veteran will receive an incentive of either a gift card or cash 
voucher of equivalent value for their participation. We can contact the Veteran after the 
interview to follow-up after the visit. We might call him/her on the phone to follow-up 
from the study visit. We might need to ask a clarifying question or to fill in any missing 
data.  
 
For Aim 2 and Aim 3, the Waiver of Authorization and Informed Consent Process will 
allow us to review administrative databases (CDW and VSSC) to identify potentially 
eligible patients for inclusion in the intervention. The waiver of informed consent process 
is the only feasible method to accomplish the objective of this part of the project, which 
is to analyze associations between appointment reminders and appointment outcomes 
for large groups of patients at VAPORHCS. Importantly, there will be no interaction 
between the research team and patients in Aim 2 and Aim 3; no patients will be 
contacted in any way. As part of the intervention, appointment reminder templates, 
which contain no personally identifying information about the patient, will be modified. 
Appointment reminders containing basic appointment information are routinely and 
automatically mailed to patients with scheduled individual outpatient mental health and 
primary care appointments. Wait time data from SHEP will be linked with CDW data at 
the patient level. 
 
For Aim 4, 
 
Patients: The Waiver of Authorization and Informed Consent Process for 
Screening/Recruitment Purposes, will allow us to review patients’ medical records, 
CDW, and VSSC data to identify potential patients for recruitment for interviews. 
Patients will be stratified along several criteria to promote purposive sampling based on: 
number of scheduled appointments in the prior 12 months (both primary care and 
mental health), number of scheduled appointments attended vs. no-showed in the prior 
12 months, new vs. established patient, service era, and gender. After a potential 
subject is identified, we will contact the patient via mail with information about the study 
and an invitation to participate or opt-out. Patients who do not opt-out will be contacted 

31 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
via their preferred method of communication (mail, phone, or email) to arrange a time 
for their informed consent and interview. Interviews will occur in one of three ways: 1)in-
person at VAPORHCS; 2) over the phone; or 3) over a video conferencing application.  
 
For in-person interviews, consent and enrollment will take place at the same time during 
the in-person visit. Participants who enroll in the study will be involved for just one day 
for the completion of an interview, but data can be retained for the duration of the study. 
During the in-person study visit, we will review the Informed Consent Sheet and HIPAA 
Authorization. We will provide a copy of the Study Contact Sheet. After signing the 
Informed Consent and HIPAA Authorization, a member of the study team will conduct 
the interview with the Veteran.  
 
For interviews conducted over the phone, once a Veteran has expressed interest in the 
study, the Veteran will be mailed copies of the informed consent document and HIPAA 
Authorization with a return envelope. Research staff delegated to obtain informed 
consent will set up a time to speak with the Veteran over the phone, explain the study, 
walk them through both documents and answer questions. If the Veteran chooses to 
participate, they will be asked to sign the informed consent document and HIPAA 
Authorization and mail them back to the research team in the provided envelope. The 
research team will then schedule a second phone call with the participant to complete 
the interview at least two weeks in advance to allow time for mailed documents to be 
received and processed. The research staff will document consent in CPRS. Once the 
research team receives the consent and HIPAA Authorization in the mail, the person 
who obtained consent will sign the forms and add an additional note in CPRS to explain 
the date discrepancy. Research staff will make copies of the forms and mail copies back 
to the participant. Interviews will only occur after signed and mailed consent and HIPAA 
documents have been received by the research team. Participants who enroll in the 
study will be involved for one day for completion of the interview, and data will be 
maintained for the duration of the study.  
 
For interviews that are conducted over a video conferencing application, the consent 
process will be similar to participants consented over the phone. Once a Veteran has 
expressed interest in the study, the Veteran will be mailed copies of the informed 
consent document and HIPAA Authorization with a return envelope. Research staff 
delegated to obtain informed consent will set up a time to speak with the Veteran over 
an agreed upon and VA-approved video conferencing application (i.e. VA Video 
Connect, although in specified circumstances other video conferencing applications 
such as Apple Facetime, Facebook Messenger, Skype, Zoom, Cisco WebEx, or Google 
Hangouts may be used. Public facing applications will not be used). Patients will be 
informed that certain third-party video conferencing applications may introduce 
additional privacy risks. Research staff will use all available encryption and privacy 
modes. During the first video conference meeting, the delegated research staff will 

32 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
explain the study, go through the documents, and answer questions. If the Veteran 
chooses to participate, they will be asked to sign the informed consent document and 
HIPAA Authorization and mail them back to the research team in the provided envelope. 
The research team will then schedule a second video visit with the participant to 
complete the interview at least two weeks in advance to allow time for mailed 
documents to be received and processed. The research staff will document consent in 
CPRS. Once the research team receives the consent and HIPAA Authorization in the 
mail, the person who obtained consent will sign the forms and add an additional note in 
CPRS to explain the date discrepancy. Research staff will make copies of the forms and 
mail copies back to the participant. Interviews will only occur after signed and mailed 
consent and HIPAA documents have been received by the research team. Participants 
who enroll in the study will be involved for one day for completion of the interview, and 
data will be maintained for the duration of the study.  
 
 
We will audio record the interviews using a handheld VA audio recorder. Audio 
recordings may also be sent to the VHA Salt Lake City (VHASLC) Centralized 
Transcription Services Program (CTSP) for transcription services. After the completed 
interview, the Veteran will receive an incentive of either a gift card or cash voucher of 
equivalent value for their participation. We can contact the Veteran after the interview to 
follow-up after the visit. We might call him/her on the phone to follow-up from the study 
visit. We might need to ask a clarifying question or to fill in any missing data.  
 
Staff: We will seek a Waiver of Authorization and Informed Consent Process for 
Screening/Recruitment Purposes and a Waiver of Consent Documentation for this study 
population. VHA staff will be recruited with the assistance of our operational partners. 
These partners will identify specific staff at VHA sites and services, and national 
leadership at VHA offices working on access, no-shows, and appointment reminders 
(e.g., OVAC, NIRMO, and the Office of Connected Care). The research team will 
contact potential subjects via email, phone, or mailed letter, and provide a description of 
the study. Staff who express interest in participating will be scheduled for a phone or 
video conference (on applications such as Zoom, Cisco WebEx, or Skype) meeting to 
discuss and obtain informed consent (including consent to be audio-recorded) and to 
conduct an interview. After expressing interest, we will ask if they would prefer to be 
interviewed on VA time or non-VA time. If they would prefer to use VA time, we will ask 
for their supervisor’s information and explain that we will be emailing him/her to obtain 
permission to conduct the interview during his/her VA Tour of Duty. We will obtain 
permission from their supervisor prior to conducting the interview on VA time. We will 
send them the research information sheet via email for them to review. After we allow 
them time to ask questions, we will ask if they would like to schedule the interview. 
These interviews will be audio recorded. Before the interview, we will obtain and record 
a verbal consent for recording. Audio recordings may also be sent to the VHA Salt Lake 

33 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
City (VHASLC) Centralized Transcription Services Program (CTSP) for transcription 
services. 
 
No compensation will be provided.  
 
We can contact Staff after the interview to follow-up after the visit. We might call them 
on the phone to follow-up from the study visit. We might need to ask a clarifying 
question or to fill in any missing data.  
 
Provider Clinic Information: Although providers are not subjects in this study, provider 
clinic information will be used to assign patients to study arms, and appointment letters 
or postcards will be changed, which may impact appointment attendance rates. 
Therefore, the request in our waiver of authorization and informed consent process for 
screening/recruitment will include provider-level data. There will be no interaction 
between the research team and providers; no providers will be interviewed or otherwise 
contacted in any way, and individual provider data will not be reported. We will also 
inform providers of this study via emailed announcements describing the study. 
Providers will have the opportunity to opt-out from having their patients included as 
subjects via an email reply to the study team.  
 
RISK AND SIDE EFFECTS  
Because the intervention in this study does not involve any direct contact with subjects, 
this study is highly unlikely to involve risks to patient. Nonetheless, it will be the 
responsibility of the project manager to monitor the progress of individual participants 
and report any concerns to the PI during regular study meetings. Interviewers and 
anyone else who has contact with study participants during study activities have the 
responsibility to monitor for any potential patient safety issues and report any concerns 
to the PI. The balance of risk to benefit will be continuously monitored by the PI, and the 
study may be modified or terminated if risks begin to outweigh benefits. 
 
Potential Risk 
For all study activities involving administrative data extraction (i.e., screening and 
recruitment for all Aims, and data analysis for Aim 2 and Aim 3), the risk to subjects is 
minimal. The primary risk is breach of confidentiality via inadvertent disclosure of 
personal health information (PHI). For study activities involving direct contact with 
subjects (i.e., Aim 1 and Aim 4 activities), the risk to subjects is also minimal due to the 
nature of the interaction. The primary risks are breach of confidentiality via inadvertent 
disclosure of personal health information (PHI) and discomfort during interviews or other 
contacts. 
 
Protection Against Risk 

34 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
Numerous efforts will be made to minimize the likelihood of the risk of a breach of 
confidentiality. No personally identifying information will be released outside of VHA. All 
subjects will be assigned a coded study ID number with password secure cross-walk 
located behind the VA firewall on a secure server or in locked file cabinets, separate 
from other study data. Identifying information will be removed from datasets prior to 
statistical analyses. CDW, VSSC, and SHEP databases are accessed only within the 
secure VHA electronic environment. We will analyze and report subject data in 
aggregate form only, and no PHI will be reported for any subject. All investigators and 
team members who will have access to personal identifiers or cross-walks linking data 
to identifiers will have received appropriate VHA background checks as part of VHA 
hiring and/or credentialing and will have completed Data Security Training within the 
prior 12 months.  
 
All information linking study data to PHI will be kept within VHA electronically in secure 
computer files stored behind firewalls requiring password access, or in hardcopy form in 
locked file cabinets in locked offices. All patient identifiers will be removed prior to 
analysis. All investigators and team members who will have access to the data will have 
received appropriate background checks as part of hiring and/or credentialing, and will 
have completed Data Security Training within the prior 12 months. 
 
For Aim 1 and Aim 4, individual interviews with Veterans will be digitally audio-recorded. 
After interviews, digital recordings will be uploaded directly to a network folder on a 
password secure server behind the VA firewall. Recording will then be deleted from the 
audio recorder. All audio recordings will be destroyed when analyses are complete. 
Filenames of digital recordings will not include participant names; only a unique study ID 
number will be listed in filenames. Transcriptions will be de-identified during the 
transcription process by personnel who have training and experience in transcription. 
Only study personnel will have access to de-identified transcripts. Any hardcopies made 
of transcripts will be kept in locked files in investigators’ offices. We will ensure that any 
quotations used in manuscripts or reports do not contain identifying information.  
 
For Aim 4, interviews with Staff will be digitally audio-recorded. After interviews, digital 
recordings will be uploaded directly to a network folder on a password secure server 
behind the VA firewall. Recording will then be deleted from the audio recorder. Audio 
recordings will not be destroyed since they will contain the verbal consent for the 
recorded interviews. Filenames of digital recordings will not include participant names; 
only a unique study ID number will be listed in filenames. Transcriptions will be de-
identified during the transcription process by personnel who have training and 
experience in transcription. Only study personnel will have access to de-identified 
transcripts. Any hardcopies made of transcripts will be kept in locked files in 
investigators’ offices. We will ensure that any quotations used in manuscripts or reports 
do not contain identifying information.  

35 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
The project manager will conduct a review of study files every six months to assure 
compliance with approved procedures. The research team will work closely with the 
VAPORHCS Information Security Officer and Contracting Officer to ensure that any 
data transfer, storage, and handling by non-VA entities adhere to VA security policies. 
 
PARTICIPANT SAFEGUARDS: 
The risk to participants is minimal and participation is entirely voluntary. Participants may 
withdraw from the study at any point without repercussion. Participants also have the 
choice to not answer interview questions or other study activities if they choose. This study 
will not include vulnerable populations and will only enroll participants with apparent 
capacity to give consent. 
 
SUICIDALITY: 
If concerns arise about a participant being at significant potential risk for suicide, the PI 
or his designee will be notified. Study team members conducting video, telephone, 
and in- person visits will have emergency contact information available at all times for 
the PI or designee and emergency services (e.g. VA National Veterans Crisis Line). All 
study team members will receive training and supervision on suicide risk assessment 
and suicide crisis procedures. These procedures include reporting these incidents to 
the PI and initiating appropriate emergency response interventions as indicated. 
 
If a participant appears to be at potentially elevated risk of suicide, a study team 
member will contact the PI or his designee to further review risk potential.  This may 
involve medical record review, speaking with the patient directly, or contacting the 
participant’s mental health or primary care provider to facilitate ongoing care as 
indicated. Additional care options in the case of more urgent circumstances may 
include clinical evaluation by the study PI or his designee, contacting the local suicide 
prevention coordinator, escort to the Portland VA Medical Center Emergency Room, or 
”warm-transfer” to the Veteran’s Crisis Line 
(http://www.portland.va.gov/research/documents/hrpp/warm-transfer.doc). 
 
BENEFITS:  
For Aim 1, patient participants will receive a $20 stipend for time and transportation 
costs associated with participating in interviews. Members of the Veteran Engagement 
Group will also participate and they will be compensated with a $25 stipend for their 
participation ($25 is the standard per hour compensation for VEG members). For Aim 4, 
patient participants will be compensated with a $20 stipend for time and transportation 
costs associated with participating in interviews. Otherwise, there will be no direct 
benefit to other subjects (patients, providers, staff). However, patients, providers, and 
staff in the future may benefit as per below. This information is also included in the 
Informed Consent Forms.  

36 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
This study has the potential to considerably improve patient care experiences by 
reducing no-show rates, reducing wait times, and increasing VHA healthcare access. In 
addition, results and products of this study may also help VHA providers and staff by 
reducing healthcare inefficiencies associated with patient no-shows, cancellations, and 
rescheduling. The risk to Veterans, as described above, is low relative to the potential 
benefit to VHA and Veterans.   
 
PROTECTED HEALTH INFORMATION: 
All information linking study data to PHI will be kept within VHA electronically in secure 
computer files stored behind firewalls requiring password access, or in hardcopy form in 
locked file cabinets in locked offices. Patient identifiers (i.e., names, address, SSN) will 
be removed prior to analysis.  
 
The identifiers and health information collected and used during analysis are the 
minimum necessary needed to conduct the research and cannot be further reduced. For 
patients in Aim 1 and Aim 4, as well as VHA staff in Aim 4, name and contact 
information (address, telephone, and email in the case of VA employees and Veterans 
who opt for a video conferencing modality for their interview) are essential to contacting 
subjects about their enrollment in the study. For patients in Aim 1 and Aim 4, 
participants social security number will be used to link locally obtained data with those 
accessible in the CDW as necessary for completion of these aims, including checking 
medical records to assess study eligibility. For patients in Aim 2 and Aim 3, dates 
related to appointments are essential to analyzing the intervention outcomes.  
 
All investigators and team members who will have access to the data will have received 
appropriate background checks as part of hiring and/or credentialing, and will have 
completed Data Security Training within the prior 12 months. 
 
MULTI-SITE STUDY CONCERNS  
N/A - All study recruitment and data analysis for this study is being conducted entirely at 
the VA Portland Health Care System. 
 
 
RESOURCES AVAILABLE 
 
The study will take place at the VA Portland Health Care System (VAPORHCS) in 
Portland, Oregon. VAPORHCS is the fastest growing unique population amongst 1A 
facilities in 11 primary care clinics. Ambulatory primary care clinics are located at the 
main hospital in Portland, on the Vancouver, Washington campus, and at nine 
additional community based outpatient clinic sites; VAPORHCS serves both urban and 
rural areas and represents a diverse population of Veterans. The VAPORHCS HSR&D 
Center of Innovation is the Center to Improve Veteran Involvement in Care (CIVIC) is 

37 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
where the science of this research study will take place, in collaboration with other 
CIVIC investigators, staff, and the CIVIC Veteran Engagement Group.  
  
The majority of project personnel will be located at the Center to Improve Veteran 
Involvement in Care (CIVIC). CIVIC will provide office space, equipment, and 
administrative research support to project staff. Dr. Teo and his research staff retain 
private office space in CIVIC. CIVIC has approximately 4,500 square feet of office space, 
which includes the second and third floors of VAPORHCS Building 6. This contiguous 
space allows extraordinary opportunities for collaboration and synergy. All necessary 
equipment including computers, printers, copiers, and scanners are available to the 
research team. Dr. Teo will devote 3/8 time to this study, and his study coordinator will be 
devoting 8/8 time to this study. 
 
COST TO SUBJECTS 
There are no costs to subjects in the proposed study. 
 
SUBJECT COMPENSATION 
For Aim 1, patient participants will receive a $20 stipend for time and transportation 
costs associated with participating in interviews. Members of the Veteran Engagement 
Group will also participate and they will be compensated with a $25 stipend for their 
participation ($25 is the standard per hour compensation for VEG members). For Aim 4, 
patient participants will be compensated with a $20 stipend for time and transportation 
costs associated with participating in interviews. Otherwise, there will be no direct 
benefit to other subjects (patients, providers, staff). However, patients, providers, and 
staff in the future may benefit as per below. These payments are reasonable and 
commensurate with the expected contributions of the subjects. Payments are fair and 
appropriate and do not constitute undue pressure or influence on, or coercion of, the 
prospective research subjects to volunteer for participation in this study. 
 
PRIVACY AND CONFIDENTIALITY 
Numerous efforts will be made to minimize the likelihood of the risk of a breach of 
confidentiality. No personally identifying information will be released outside of VHA. All 
subjects will be assigned a coded study ID number with password secure cross-walk 
located behind the VA firewall on a secure server or in locked file cabinets, separate 
from other study data. Identifying information will be removed from datasets prior to 
statistical analyses. CDW, VSSC, and SHEP databases are accessed only within the 
secure VHA electronic environment. We will analyze and report subject data in 
aggregate form only, and no PHI will be reported for any subject. All investigators and 
team members who will have access to personal identifiers or cross-walks linking data 
to identifiers will have received appropriate VHA background checks as part of VHA 
hiring and/or credentialing and will have completed Data Security Training within the 
prior 12 months. 

38 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
All information linking study data to PHI will be kept within VHA electronically in secure 
computer files stored behind firewalls requiring password access, or in hardcopy form in 
locked file cabinets in locked offices. All patient identifiers will be removed prior to 
analysis. All investigators and team members who will have access to the data will have 
received appropriate background checks as part of hiring and/or credentialing, and will 
have completed Data Security Training within the prior 12 months. 
 
For Aim 1 and Aim 4, individual interviews will be digitally audio recorded. After 
interviews, digital recordings will be uploaded directly to a network folder on a password 
secure server behind the VA firewall. Recording will then be deleted from the audio 
recorder. All audio recordings will be destroyed when analyses are complete, except for 
audio recordings of interviews with VA staff, which will include the staff member’s 
consent for audio recording. Filenames of digital recordings will not include participant 
names; only a unique study ID number will be listed in filenames. Transcriptions will be 
de-identified during the transcription process by personnel who have training and 
experience in transcription. Only study personnel will have access to de-identified 
transcripts. Any hardcopies made of transcripts will be kept in locked files in 
investigators’ offices. We will ensure that any quotations used in manuscripts or reports 
do not contain identifying information.  
 
Audio recordings may also be sent to the VHA Salt Lake City (VHASLC) Centralized 
Transcription Services Program (CTSP) for transcription services. 
 
For Aim 1, a personal smartphone with a camera will be used to capture photographs of 
the cards laid out on the table from the pile sorting task, but the participant will not be 
included in the photograph. Photographs will be emailed from the interviewer’s personal 
email on her smartphone to the research coordinator’s VA email, uploaded to a network 
folder on a password secure server behind the VA firewall, and labeled using a unique 
study code. No personally identifying information will be included in the photographs or 
their labels.  
 
Study data may also be stored in a designated study folder within VA Informatics and 
Computing Infrastructure (VINCI), a centralized data storage server. VINCI is a 
partnership between the VA Office of Information Technology (OI&T) and the Veterans’ 
Health Administration Office of Research and Development (VHA ORD). VINCI 
provides the storage and server technologies to securely host suites of databases 
integrated from select national data. These servers reside at the Austin Information 
Technology Center (AITC), located in Austin, Texas. To ensure the protection of 
Veterans data, VINCI maintains compliance with the guidelines set forth by Veterans 
Health Administration (VHA) Handbook 1200.12, Use of Data and Data Repositories in 

39 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
VHA Research and all other applicable VA and VHA policies and regulations. In 
addition, VINCI has undergone all security certification activities in support of obtaining 
an Authorization to Operate (ATO). 
 
The project manager will conduct a review of study files every six months to assure 
compliance with approved procedures. The research team will work closely with the 
VAPORHCS Information Security Officer and Contracting Officer to ensure that any 
data transfer, storage, and handling by non-VA entities adhere to VA security policies. 
 
CERTIFICATE OF CONFIDENTIALITY 
N/A 
 
DATA MANAGEMENT 
This project will be housed at the VA Portland Health Care System (VAPORHCS).  
 
All information linking study data to PHI will be kept within VHA electronically in secure 
computer files stored behind firewalls requiring password access, or in hardcopy form in 
locked file cabinets in locked offices. All patient identifiers will be removed prior to 
analysis. All investigators and team members who will have access to the data will have 
received appropriate background checks as part of hiring and/or credentialing, and will 
have completed Data Security Training within the prior 12 months. 
 
For Aim 1 and Aim 4, individual interviews will be digitally audio-recorded. After 
interviews, digital recordings will be uploaded directly to a network folder on a password 
secure server behind the VA firewall. Recording will then be deleted from the audio 
recorder. All audio recordings will be destroyed when analyses are complete, except for 
audio recordings of interviews with VA staff, which will include the staff member’s 
consent for audio recording. Filenames of digital recordings will not include participant 
names; only a unique study ID number will be listed in filenames. Transcriptions will be 
de-identified during the transcription process by personnel who have training and 
experience in transcription. Only study personnel will have access to de-identified 
transcripts. Any hardcopies made of transcripts will be kept in locked files in 
investigators’ offices. We will ensure that any quotations used in manuscripts or reports 
do not contain identifying information.  
 
The project manager will conduct a review of study files every six months to assure 
compliance with approved procedures. The research team will work closely with the 
VAPORHCS Information Security Officer and Contracting Officer to ensure that any 
data transfer, storage, and handling by non-VA entities adhere to VA security policies. 
 
Transfer of Data Ownership 
N/A 

40 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
Data and Safety Monitoring Plan (DSMP) 
Safety Monitoring: Because the intervention in this study does not involve any direct 
contact with subjects, this study is highly unlikely to involve risks to patient. 
Nonetheless, the PI or anyone else who has contact with study participants during study 
activities have the responsibility to monitor for any potential adverse events and protocol 
deviations. Any adverse participant event will be reported immediately to PI Dr. Teo, 
who will contact the participant and determine if additional intervention is needed to 
ensure participant safety. Protocol deviations will also be immediately reported to Dr. 
Teo who will ensure that adverse events deemed to be unanticipated problems and 
protocol deviations are properly reported to the IRB in a timely manner. Detailed written 
documentation will be kept for all adverse events that occur over the course of the 
study. PI Dr. Teo holds weekly meetings with his study personnel where they will 
discuss adverse events and protocol deviations associated with this project and ways to 
reduce repeat occurrences. Research staff will examine all cumulative adverse events 
quarterly to determine if there are any systematic problems and to implement protocol 
corrections as needed after receiving IRB approval. 
 
Data Monitoring: We will use password protected excel documents to store study data, 
track data screening, enrollment and participation, create surveys, generate reports, 
and enhance project management. All information linking study data to PHI will be kept 
within VHA electronically in secure computer files stored behind firewalls requiring 
password access, or in hardcopy form in locked file cabinets in locked offices. All patient 
identifiers will be removed prior to analysis. All investigators and team members who 
will have access to the data will have received appropriate background checks as part 
of hiring and/or credentialing, and will have completed Data Security Training within the 
prior 12 months. The project manager will conduct a review of study files every six 
months to assure compliance with approved procedures. The research team will work 
closely with the VAPORHCS Information Security Officer and Contracting Officer to 
ensure that any data transfer, storage, and handling by non-VA entities adhere to VA 
security policies. 
 
Per VHA guidelines, data resulting from this study will be stored locally on VHA password-
secure folders until enterprise-level resources become available for long-term storage and 
access. Requests for data access will be considered and responded to within one month 
of the request and datasets will be made available electronically. Requests must be made 
in writing to the study PI and provide information on the purpose for accessing the data.  
 
All data used in final, published results will be made available for sharing. Published data 
will be available upon request to any investigator in order to enable independent validation 
and interpretation of published data. 
 

41 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
 
References & Literature Cited  . 
1. Milkman, K., Beshears, J., Choi, J., Laibson, D. & Madrian, B. Using implementation 
intentions prompts to enhance influenza vaccination rates. Proc. Natl. Acad. Sci. U. S. A. 
108, 10415–20 (2011). 
2. Davies, M. L. et al. Large-Scale No-Show Patterns and Distributions for Clinic Operational 
Research. Healthcare 4, 15 (2016). 
3. Goffman, R. M. National Initiative to Reduce Missed Opportunities Mental Health Operations 
Call. (VA Pittsburgh Veterans Engineering Resource Center (VERC), 2016). 
4. Hwang, A. et al. Appointment ‘no-shows’ are an independent predictor of subsequent quality 
of care and resource utilization outcomes. 30, 1426–33 (2015). 
5. Nelson, E. A., Maruish, M. E. & Axler, J. L. Effects of discharge planning and compliance 
with outpatient appointments on readmission rates. Psychiatr. Serv. Wash. DC 51, 885–889 
(2000). 
6. Sparr, L. F., Moffitt, M. C. & Ward, M. F. Missed psychiatric appointments: who returns and 
who stays away. Am. J. Psychiatry 150, 801–805 (1993). 
7. Audit of Veterans Health Administration’s Efforts to Reduce Unused Outpatient 
Appointments. (VA Office of Inspector General, 2008). 
8. Goffman, R. M. National Initiative to Reduce Missed Opportunities Mental Health Operations 
Call. (2015). 
9. McLean, S. et al. Targeting the Use of Reminders and Notifications for Uptake by 
Populations (TURNUP): a systematic review and evidence synthesis. (NIHR Journals 
Library, 2014). 
10. McLean, S. M. et al. Appointment reminder systems are effective but not optimal: results of 
a systematic review and evidence synthesis employing realist principles. Patient Prefer. 
Adherence 10, 479–499 (2016). 
11. Bigby, J., Giblin, J., Pappius, E. M. & Goldman, L. Appointment reminders to reduce no-
show rates. A stratified analysis of their cost-effectiveness. JAMA 250, 1742–1745 (1983). 
12. Guy, R. et al. How effective are short message service reminders at increasing clinic 
attendance? A meta-analysis and systematic review. Health Serv. Res. 47, 614–632 (2012). 
13. Lefforge, N. L., Donohue, B. & Strada, M. J. Improving session attendance in mental health 
and substance abuse settings: a review of controlled studies. Behav. Ther. 38, 1–22 (2007). 
14. Macharia, W. M., Leon, G., Rowe, B. H., Stephenson, B. J. & Haynes, R. B. An overview of 
interventions to improve compliance with appointment keeping for medical services. JAMA 
267, 1813–1817 (1992). 
15. Peterson, K., McCleery, E., Anderson, J., Waldrip, K. & Helfand, M. Evidence Brief: 
Comparative Effectiveness of Appointment Recall Reminder Procedures for Follow-up 
Appointments. (Evidence-based Synthesis Program (ESP) Coordinating Center, 2015). 
16. Ariely, D. The (honest) truth about dishonesty: how we lie to everyone--especially ourselves. 
(2012). 
17. Emanuel, E. J. et al. Using Behavioral Economics to Design Physician Incentives That 
Deliver High-Value Care. Ann. Intern. Med. 164, 114–119 (2016). 
18. Nirenberg, T. D., Sobell, L. C. & Sobell, M. B. Effective and inexpensive procedures for 
decreasing client attrition in an outpatient alcohol treatment program. Am. J. Drug Alcohol 
Abuse 7, 73–82 (1980). 

42 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
19. Thaler, R. H. & Sunstein, C. R. Nudge: improving decisions about health, wealth, and 
happiness. (Yale University Press, 2008). 
20. Hallsworth, M., List, J., Metcalfe, R. & Vlaev, I. The Behavioralist As Tax Collector: Using 
Natural Field Experiments to Enhance Tax Compliance. NBER Programs PE (2014). 
doi:10.3386/w20007 
21. Martin, S. J., Bassi, S. & Dunbar-Rees, R. Commitments, norms and custard creams - a 
social influence approach to reducing did not attends (DNAs). JRSM 105, 101–104 (2012). 
22. Goldstein, N. J., Cialdini, R. B. & Griskevicius, V. A Room with a Viewpoint: Using Social 
Norms to Motivate Environmental Conservation in Hotels. J. Consum. Res. 35, 472–482 
(2008). 
23. Dolan, P. et al. Influencing behavior: The mindspace way. J. Econ. Psychol. 33, 264–277 
(2012). 
24. Gollwitzer, P. Implementation intentions: Strong effects of simple plans. Am. Psychol. 54, 
493–503 (1999). 
25. Hastie, R. & Dawes, R. Rational Choice in an Uncertain World: The Psychology of 
Judgment and Decision Making. (Sage Publications, Inc., 2001). 
26. Kahneman, D. & Tversky, A. Prospect Theory: An Analysis of Decision under Risk. 
Econometrica 47, 263–292 (1979). 
27. Emanuel, E. J. et al. Using Behavioral Economics to Design Physician Incentives That 
Deliver High-Value Care. Ann. Intern. Med. 164, 114–119 (2016). 
28. Fehr, E. & Schmidt, K. A Theory of Fairness, Competition, and Cooperation. Q. J. Econ. 
114, (1999). 
29. Andreoni, J. Impure altruism and donations to public goods: A theory of warm-glow giving. 
Econ. J. 100, 464–477 (1990). 
30. Fogarty, A. W., Sturrock, N., Premji, K. & Prinsloo, P. Hospital Clinicians’ Responsiveness 
to Assay Cost Feedback: A Prospective Blinded Controlled Intervention Study. JAMA Intern 
Med 1654–1655 (2013). doi:doi:10.1001/jamainternmed.2013.8211 
31. Hallsworth, M. et al. Stating Appointment Costs in SMS Reminders Reduces Missed 
Hospital Appointments: Findings from Two Randomised Controlled Trials. PLoS ONE 10(9): 
e0137306., (2015). 
32. Hallsworth, M. et al. Provision of social norm feedback to high prescribers of antibiotics in 
general practice: a pragmatic national randomised controlled trial. The Lancet 387, 1743–
1752 (2016). 
33. Motto, J. A. & Bostrom, A. G. A randomized controlled trial of postcrisis suicide prevention. 
Psychiatr. Serv. Wash. DC 52, 828–833 (2001). 
34. Carter, G. L., Clover, K., Whyte, I. M., Dawson, A. H. & D’Este, C. Postcards from the EDge: 
24-month outcomes of a randomised controlled trial for hospital-treated self-poisoning. Br. J. 
Psychiatry J. Ment. Sci. 191, 548–553 (2007). 
35. Social and Behavioral Sciences Team. Social and Behavioral Sciences Team (2016). 
Available at: Web-based Inquiry of the General Services Administration for the National 
Science and Technology Council’s Subcommittee on the Social and Behavioral Sciences 
Team [online]. (Accessed: 30th September 2016) 
36. Allcott, H. Social norms and energy conservation. J. Public Econ. 95, 1082–1095 (2011). 
37. Pizer, S. D. & Prentice, J. C. What Are the Consequences of Waiting for Health Care in the 
Veteran Population? J. Gen. Intern. Med. 26, 676–682 (2011). 

43 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
38. Prentice, J. C. & Pizer, S. D. Delayed Access to Health Care and Mortality. Health Serv. 
Res. 42, 644–662 (2007). 
39. Prentice, J. C., Fincke, B. G., Miller, D. R. & Pizer, S. D. Outpatient wait time and diabetes 
care quality improvement. Am. J. Manag. Care 17, e43-54 (2011). 
40. Kinsel, P. A. Case Study: South Texas Veterans Health Care System’s Communication 
Center. 
41. Siminoff, L. A., Hausmann, L. R. M. & Ibrahim, S. Barriers to obtaining diagnostic testing for 
coronary artery disease among veterans. Am. J. Public Health 98, 2207–2213 (2008). 
42. Menendez, M. E. & Ring, D. Factors associated with non-attendance at a hand surgery 
appointment. Hand N. Y. N 10, 221–226 (2015). 
43. Parikh, A. et al. The effectiveness of outpatient appointment reminder systems in reducing 
no-show rates. Am. J. Med. 123, 542–548 (2010). 
44. Shah, S. J. et al. Targeted Reminder Phone Calls to Patients at High Risk of No-Show for 
Primary Care Appointment: A Randomized Trial. J. Gen. Intern. Med. 31, 1460–1466 
(2016). 
45. Teo, A. R., Forsberg, C., Marsh, H. E., Saha, S. & Dobscha, S. K. No-Show Rates Higher 
When Phone Appointment Reminders are Not Directly Delivered. Psychiatr. Serv. (In 
Press). 
46. Monte, R. et al. Veterans Health Administration National Initiative to Reduce Missed 
Opportunities. (Veterans Engineering Resource Center, VA Pittsburgh Healthcare System, 
2011). 
47. Henry, S. R., Goetz, M. B. & Asch, S. M. The Effect of Automated Telephone Appointment 
Reminders on HIV Primary Care No-Shows by Veterans. J. Assoc. Nurses AIDS Care 23, 
409–418 (2012). 
48. McInnes, D. K. et al. Retaining homeless veterans in outpatient care: a pilot study of mobile 
phone text message appointment reminders. Am. J. Public Health 104 Suppl 4, S588-594 
(2014). 
49. Hashim, M. J., Franks, P. & Fiscella, K. Effectiveness of telephone reminders in improving 
rate of appointments kept at an outpatient clinic: a randomized controlled trial. J. Am. Board 
Fam. Pract. Am. Board Fam. Pract. 14, 193–196 (2001). 
50. Junod Perron, N. et al. Text-messaging versus telephone reminders to reduce missed 
appointments in an academic primary care clinic: a randomized controlled trial. BMC Health 
Serv. Res. 13, 125 (2013). 
51. Morse, D. L., Coulter, M. P., Nazarian, L. F. & Napodano, R. J. Waning effectiveness of 
mailed reminders on reducing broken appointments. Pediatrics 68, 846–849 (1981). 
52. Samson, A. (Ed). The Behavioral Economics Guide 2015 (with an introduction by Dan 
Ariely). (2015). Available at: www.behavioraleconomics.com.  
53. Ajzen, I. The Theory of Planned Behavior. Organ. Behav. Hum. Decis. Process. 50, 179–
211 (1991). 
54. Cialdini, R. B. Influence: Science and Practice. (Allyn and Bacon, 2001). 
55. Long, J. A., Jahnle, E. C., Richardson, D. M., Loewenstein, G. & Volpp, K. G. Peer 
mentoring and financial incentives to improve glucose control in African American veterans: 
a randomized trial. Ann. Intern. Med. 156, 416–424 (2012). 
56. Mouthaan, J., Euwema, M. C. & P, M. Band of Brothers in U.N. Peacekeeping: Social 
Bonding among Dutch Peacekeeping Veterans. Mil. Psychol. 17, 101–114 (2005). 

44 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
57. Fischer, E. P. et al. Overcoming Barriers to Sustained Engagement in Mental Health Care: 
Perspectives of Rural Veterans and Providers. J. Rural Health Off. J. Am. Rural Health 
Assoc. Natl. Rural Health Care Assoc. 32, 429–438 (2016). 
58. Robinson, L. A. & Hammitt, J. K. Behavioral Economics and the Conduct of Benefit-Cost 
Analysis: Towards Principles and Standards. J. Benefit-Cost Anal. 2, 1–51 (2011). 
59. Moses, J. & Hewitt, A. Get Action Instead of Indifference: Using Behavioral Economics 
Insights to Deliver Benefits Messages. Benefits Q. 3, (2013). 
60. Molfenter, T. Reducing appointment no-shows: going from theory to practice. Subst. Use 
Misuse 48, 743–749 (2013). 
61. Reese, P. P. et al. Two Randomized Controlled Pilot Trials of Social Forces to Improve 
Statin Adherence among Patients with Diabetes. J. Gen. Intern. Med. 31, 402–410 (2016). 
62. Yee, C., Legler, A., Pizer, S. & Prentice, J. Scheduling Policy and New Patient Access. 
(Partnered Evidence-based Policy Resource Center (PEPReC)). 
63. Gardner, L. I. et al. A low-effort, clinic-wide intervention improves attendance for HIV 
primary care. Clin. Infect. Dis. Off. Publ. Infect. Dis. Soc. Am. 55, 1124–1134 (2012). 
64. Patel, M. S. et al. Using default options within the electronic health record to increase the 
prescribing of generic-equivalent medications: a quasi-experimental study. Ann. Intern. Med. 
161, S44-52 (2014). 
65. Jeffery, R. W. Financial incentives and weight control. Prev. Med. 55 Suppl, S61-67 (2012). 
66. Shapiro, M. F. No-Show Patients and the Triple Aim. J. Gen. Intern. Med. 30, 1392–1393 
(2015). 
67. Thorpe, K. E. et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool 
to help trial designers. J. Clin. Epidemiol. 62, 464–475 (2009). 
68. Basham, C. et al. Characteristics and VA health care utilization of U.S. Veterans who 
completed suicide in Oregon between 2000 and 2005. Suicide Life. Threat. Behav. 41, 287–
296 (2011). 
69. Dobscha, S. K., Morasco, B. J., Duckart, J. P., Macey, T. & Deyo, R. A. Correlates of 
prescription opioid initiation and long-term opioid use in veterans with persistent pain. Clin. 
J. Pain 29, 102–108 (2013). 
70. Denneson, L. M., Corson, K., Helmer, D. A., Bair, M. J. & Dobscha, S. K. Mental health 
utilization of new-to-care Iraq and Afghanistan Veterans following suicidal ideation 
assessment. Psychiatry Res. 217, 147–153 (2014). 
71. Dobscha, S. K. et al. Correlates of suicide among veterans treated in primary care: case-
control study of a nationally representative sample. J. Gen. Intern. Med. 29 Suppl 4, 853–
860 (2014). 
72. Dobscha, S. K. et al. Predictors of Improvements in Pain Intensity in a National Cohort of 
Older Veterans With Chronic Pain. J. Pain Off. J. Am. Pain Soc. 17, 824–835 (2016). 
73. Prentice, J. C., Fincke, B. G., Miller, D. R. & Pizer, S. D. Primary Care and Health Outcomes 
among Older Patients with Diabetes. Health Serv. Res. 47, 46–67 (2012). 
74. Prentice, J. C., Davies, M. L. & Pizer, S. D. Which Outpatient Wait-Time Measures Are 
Related to Patient Satisfaction? Am. J. Med. Qual. 29, 227–235 (2014). 
75. Dobscha, S. K. et al. Depression decision support in primary care: a cluster randomized 
trial. Ann. Intern. Med. 145, 477–487 (2006). 
76. Dobscha, S. K. et al. Collaborative care for chronic pain in primary care: a cluster 
randomized trial. JAMA 301, 1242–1252 (2009). 

45 
<Protocol version date: 05/15/2020>  
Research Service Template: 7/11/17 
 
 
 
77. Rothberg, M. B. et al. The effect of information presentation on beliefs about the benefits of 
elective percutaneous coronary intervention. JAMA Intern. Med. 174, 1623–1629 (2014). 
78. Zikmund-Fisher, B. J., Sarr, B., Fagerlin, A. & Ubel, P. A. A matter of perspective: choosing 
for others differs from choosing for yourself in making treatment decisions. J. Gen. Intern. 
Med. 21, 618–622 (2006). 
79. Zikmund-Fisher, B. J., Fagerlin, A. & Ubel, P. A. ‘Is 28% good or bad?’ Evaluability and 
preference reversals in health care decisions. Med. Decis. Mak. Int. J. Soc. Med. Decis. 
Mak. 24, 142–148 (2004). 
80. Prentice, J. C., Davies, M. L. & Pizer, S. Using Health Outcomes to Validate Access Quality 
Measures. Am. J. Manag. Care 19, e367-77 (2013). 
81. Beebe, J. Rapid Assessment Process: An Introduction. (AltaMira Press, 2001). 
82. Roberts, J., Fisher, T., Trowbridge, M. & Bent, C. A design thinking framework for 
healthcare management and innovation. Healthcare 4, 11–14 (2016). 
83. Yeh, H. et al. Sorting it Out: Pile Sorting as a Mixed Methodology for Exploring Barriers to 
Cancer Screening. Qual. Quant. 48, 2569–2587 (2014). 
84. Research-based web design & usability guidelines. (U.S. Department of Health and Human 
Services: U.S. General Services Administration, 2006). 
85. Bush, N. E. et al. A Virtual Hope Box smartphone app as an accessory to therapy: proof-of-
concept in a clinical sample of veterans. Suicide Life. Threat. Behav. 45, 1–9 (2015). 
86. Social and Behavioral Sciences Team 2015 Annual Report. (Executive Office of the 
President, National Science and Technology Council, 2015). 
87. Hwang, A. S. et al. Appointment ‘no-shows’ are an independent predictor of subsequent 
quality of care and resource utilization outcomes. J. Gen. Intern. Med. 30, 1426–1433 
(2015). 
88. Mazar, N., Amir, O. & Ariely, D. The Dishonesty of Honest People: A Theory of Self-Concept 
Maintenance. J. Mark. Res. 45, 633–644 (2008). 
89. Pizer, S., Davies, M. & Prentice, J. Consult Timeliness Strongly Predicts Patient 
Satisfaction” American Journal of Accountable Care. (In Press). 
90. Damschroder, L. J. et al. Fostering implementation of health services research findings into 
practice: a consolidated framework for advancing implementation science. Implement. Sci. 
IS 4, 50 (2009). 
91. Kirk, M. A. et al. A systematic review of the use of the Consolidated Framework for 
Implementation Research. Implement. Sci. 11, 72 (2016). 
92. Hamilton, A. Qualitative Methods in Rapid Turn-Around Health Services Research. (2013). 
Available at: 
http://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cf
m?SessionID=780. (Accessed: 3rd November 2016) 
93. Averill, J. B. Matrix analysis as a complementary analytic strategy in qualitative inquiry. 
Qual. Health Res. 12, 855–866 (2002). 
94. Grissom, R. J. & Kim, J. J. Effect Sizes for Research: Univariate and Multivariate 
Applications, Second Edition. (Routledge, 2011). 
95. Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M. & Stetler, C. Effectiveness-
implementation hybrid designs: combining elements of clinical effectiveness and 
implementation research to enhance public health impact. Med. Care 50, 217–226 (2012).