Personalized Patient Data and Behavioral Nudges to Improve Adherence to Chronic Cardiovascular Medications (The Nudge Study)
Phase: NA
Conditions: Cardiovascular Diseases, Adherence, Medication, Medication Adherence, Diabetes Mellitus, Hypertension, Hyperlipidemias, Coronary Artery Disease, Atrial Fibrillation
Sponsor: University of Colorado, Denver

Personalized Patient Data & Behavioral Nudges to Improve Adherence to 
Chronic CV
Medications (Nudge)
Study Start Year: 2019
Principal Investigator Michael Ho, MD, PhD
ClinicalTrials.gov NCT ID# NCT03973931
NIH Grant Number 1UG3HL144163-01
Date of first UH3 notice of Grant Award 2019-07-01

1 
1 
2 
COMIRB Protocol 
3 
4 
Personalized patient data and behavioral nudges to improve adherence to chronic 
5 
cardiovascular medications (Nudge) 
6 
Co-Principal Investigators: Michael Ho, MD, PhD; Sheana Bull, PhD, MPH 
7 
Protocol: 18-2779 
8 
Version date:  5/23/2023 
9 
10 
Up to 50% of patients do not take their cardiovascular medications as prescribed, resulting in 
11 
increased morbidity, mortality, and healthcare costs. Interventions to improve adherence, such 
12 
as patient education, reminders, pharmacist support, and financial incentives, have produced 
13 
mixed results—some demonstrating benefits, but many producing small to negative results. 
14 
Adherence interventions have been limited by 1) including adherent patients who may not need 
15 
an intervention; 2) resource-intensive approaches involving pharmacists; and 3) lack of attention 
16 
to evidence-based strategies to motivate human behavior.   
17 
Brief behavioral interventions can influence decision-making and are impactful. Principles of 
18 
behavioral economics have been incorporated into health interventions to “nudge” people to 
19 
achieve improved health outcomes. A behavioral nudge is a small change in framing choice that 
20 
alters people’s behavior in a predictable way. A prior study testing financial incentives through 
21 
elimination of copayments for cardiovascular medications in the year after acute myocardial 
22 
infarction improved adherence from 4% to 6%; however, financial incentives are not 
23 
generalizable and are unlikely to be sustainable. Behavioral nudges such as commitments (e.g. 
24 
asking patients for demonstrated commitment to change through a pledge), norms (using 
25 
examples of others who take action), and salience (making information or recommendations 
26 
resonant through use of stories) build on a well-evidenced body of behavioral science theory 
27 
and have been shown to improve health behaviors such as smoking cessation and weight loss. 
28 
These have yet to be tested to improve medication adherence. 
29 
30 
Mobile and digital technologies for health promotion and disease self-management 1-3 offer an 
31 
intriguing and as of yet untested opportunities to adapt behavioral ‘nudges’ using ubiquitous cell 
32 
phone technology to facilitate medication adherence.  
33 
34 
The objectives of our two-part, multi-center study are as follows over the course of 4 years: 
35 
Aim 1: Conduct a pragmatic patient-level randomized intervention across 3 HCS to improve 
36 
adherence to chronic CV medications. The primary outcome will be medication adherence 
37 
defined by the proportion of days covered (PDC) using pharmacy refill data. Secondary 
38 
outcomes will include intermediate clinical measures (e.g., BP control), CV clinical events (e.g., 
39 
hospitalizations), healthcare utilization, and costs. 
40 
Aim 2: Evaluate the intervention using a mixed methods approach and applying the RE-AIM 
41 
(reach, effectiveness, adoption, implementation, and maintenance) framework. In addition, 
42 
assess the context and implementation processes to inform local tailoring, adaptations and 
43 
modifications, and eventual expansion of the intervention within the 3 HCS more broadly and 
44 
nationally. 
45 

2 
46 
BACKGROUND AND SIGNIFICANCE 
47 
Patients commonly fail to adhere to cardiovascular medications, resulting in an increased risk of 
48 
adverse outcomes. Pharmacy refill data is routinely used to describe the prevalence and 
49 
outcomes of medication non-adherence using one of two measures, the proportion of days 
50 
covered (PDC) or the medication possession ratio (MPR). Both measures are calculated by the 
51 
number of days supplied for a medication divided by the observation period with a range of 0 to 
52 
1.0, with 1.0 implying perfect adherence. Non-adherence is commonly defined as a PDC or 
53 
MPR <0.80. Prior work has found 20% to 50% of patients with cardiovascular diseases (e.g. 
54 
hypertension, hyperlipidemia, diabetes, atrial fibrillation, or coronary artery disease) have poor 
55 
medication adherence.  For example, in our prior work, over a quarter (28%) of patients were 
56 
non-adherent to dabigatran, a direct oral anti-coagulant that is intended to reduce the risk of 
57 
thromboembolic events among patients with atrial fibrillation. In the same study, we 
58 
demonstrated poor adherence was associated with increased risk of mortality or stroke (HR 
59 
1.13, 95% CI 1.07-1.19 per 10% decrease in adherence as measured by the PDC). A similar 
60 
association between medication non-adherence and adverse outcomes has been demonstrated 
61 
for other classes of medications including anti-platelet medications, B-blockers, ACE inhibitors 
62 
or ARBs, oral diabetes medications, and statins, all of which are used to treat cardiovascular 
63 
diseases. The accumulated literature has shown that medication non-adherence to CV 
64 
medications is common and results in suboptimal outcomes; thus effective interventions are 
65 
needed to improve medication adherence. 
66 
Mobile telephone text messaging interventions, a form of mHealth technology, may be 
67 
promising in improving medication adherence. Mobile telephones and text messaging are 
68 
ubiquitous - this technology is increasingly used regardless of age, socioeconomic class and 
69 
primary language. A recent meta-analysis of mobile telephone text messaging medication 
70 
adherence interventions for chronic diseases demonstrated that text-messaging interventions 
71 
approximately double the odds of medication adherence. This increase translates into 
72 
adherence rates improving from 50.0% (assuming this baseline rate in patients with chronic 
73 
disease) to 67.8%, or an absolute increase of 17.8%. Another meta-analysis of text messaging 
74 
to improve health behavior outcomes, including but not limited to medication adherence, found 
75 
that personalized messages have greater effects than those that do not. While these meta-
76 
analyses demonstrate the potential for text messaging interventions, the underlying studies 
77 
were markedly heterogeneous, leaving questions about the best strategies for message design, 
78 
the impact of generic vs. tailored approaches, optimal message timing and intensity and 
79 
whether bi-directional messaging is useful. Authors of both reviews concluded the results should 
80 
be interpreted with caution given the short duration of studies and reliance on self-reported 
81 
measures. These preliminary data on the influence of text messaging interventions are cause 
82 
for optimism. However, it is not yet known how this type of intervention might be optimized at 
83 
scale and whether doing so will improve outcomes.  
84 
Behavioral “nudges” from the fields of behavioral economics and cognitive psychology have the 
85 
potential to augment the impact of text messaging interventions to further enhance medication 
86 
adherence. Normative theories of decision making, such as expected utility theory, are based 
87 
on the ideal that all people approach decisions rationally and are able to weigh the risks and 
88 
benefits of various interventions. In medication adherence for example, a normative approach 
89 
would mean someone would take the medication provided the benefits outweigh the risks. 
90 
Descriptive theories of decision making, such as the Dual-Process theory and Prospect theory 
91 
(two of the foundational theories supporting Dan Kahneman’s 2002 Nobel prize in economics), 
92 
demonstrate that humans are subject to cognitive biases that cause decision-making to deviate 
93 
from the normative or rational. The Dual-Process Theory of decision making states that people 
94 
make decisions either ‘intuitively,’ quickly drawing on emotion and past experiences or 
95 

3 
‘reasonably’ using a thoughtful, analytic approach. Nudges take advantage of the intuitive 
96 
aspects of decision-making. A nudge is defined as a small change in choice framing or choice 
97 
architecture that “alters people’s behavior in a predictable way without forbidding any options or 
98 
significantly changing their economic incentives.” A technology-delivered nudge should 
99 
positively influence individuals’ behaviors through the use of non-intrusive education, social 
100 
norm setting, and reciprocity expectation.  
101 
There are three types of nudge interventions strongly supported by prior literature that could be 
102 
feasibly implemented as text messages within the context of medication adherence:  
103 
1) Communicating social norms: Social norms can activate and guide behavior in positive ways
104 
when a message normalizes positive behaviors, such as medication adherence, placing
105 
non-adherence outside the definition of typical behavior. In other contexts, social norms
106 
have been shown to improve healthy food choices, physical activity, everyday health
107 
behaviors (e.g. using the stairs vs. elevators) and even reduce home energy use. However,
108 
little or no research has tested the influence of social norm communication via text
109 
messaging to improve medication adherence.
110 
2) Behavioral commitments: A behavioral commitment is something like committing to filling
111 
one’s prescription. Prior research has demonstrated a strong desire among individuals to act
112 
consistently with their prior commitments, and eliciting commitments to engage in a specific
113 
behavior has been shown to be effective at improving a range of behaviors, including
114 
judicious use of antibiotics among clinicians. Commitments to fill one’s prescription could be
115 
elicited via text messaging and may lead to greater concordance between individuals’
116 
commitment and their behaviors.
117 
3) Narrative Stories: Narrative stories are increasingly recognized as an important way to
118 
increase vividness and comprehension of medical outcomes. One issue underlying
119 
medication non-adherence is likely a failure to recognize or understand the potential
120 
negative consequences the behavior, e.g., stroke, heart attack, or even death. Narrative
121 
interventions—particularly ones that describe stories of negative outcomes—may be
122 
particularly effective at helping patients concretely understand potential risks of non-
123 
adherence, spurring them to take action (improving medication adherence) to prevent
124 
negative outcomes.
125 
Although text messaging has been used with positive effect to influence medication 
126 
adherence,1-4 such messages are not theoretically informed to influence social norms, 
127 
behavioral commitment and/or use narratives. 
128 
PREVIOUS WORK 
129 
Throughout 2018, we established the procedures and start-up necessary to initiate this four-
130 
year study successfully with an informed approach (COMIRB #18-0630). We have developed a 
131 
message library, and vetted and developed through a series of N of 1 trials and stakeholder 
132 
panel input. IT infrastructure has been developed across three healthcare systems (see below 
133 
for additional details).  
134 
We have also piloted and refined our patient identification procedures, opt-out methods, and the 
135 
text messaging intervention through a pilot study (in progress at this time). 
136 
SETTING 
137 
The proposed study setting includes Denver Health, UCHealth, and the Denver VA healthcare 
138 
systems (HCS). 
139 
APPROACH 
140 

4 
  Over the next four years, we propose conducting a patient-level, randomized pragmatic trial 
141 
testing a variety of strategies “nudging” patients through text messages to encourage 
142 
medication adherence to already-prescribed cardiovascular medicines.  
143 
Inclusion & Exclusion Criteria 
144 
Programmers will identify eligible patients based on the presence at least 1 of the  
145 
cardiovascular conditions listed in Table 1 and with a prescription for at least 1 of the classes 
146 
of medications to treat the cardiovascular conditions listed in Table 1. Participants must be 
147 
patients at UCHealth, Denver Health, and the VA. Patients must have had a fill for a 
148 
medication of interest within the 100 days prior to the cohort build date.  
149 
There will be minimal exclusions criteria: 1) patients who have neither a landline or cellphone; 
150 
or 2) enrolled in hospice or palliative care; or 3) Non-English or Spanish speaking; 4) residing 
151 
out of the state of Colorado.  
152 
Patients with a refill gap of at least 7 days within the past year utilizing pharmacy refill data will 
153 
be eligible to receive nudges. We will specifically target patients from the following patient 
154 
groups: 1) Age > and <= 65 years of age; 2) Male and female patients; 3) one versus multiple 
155 
cardiovascular condition of interest; and 4) English and Spanish speaking patients.  
156 
Table 1. Inclusion criteria for the study based on cardiovascular conditions and medications 
157 
classes of interest  
158 
Condition 
Classes of medications 
Hypertension 
Beta-blockers (B-blockers), Calcium Channel Blocker (CCB), Angiotensin 
converting enzyme inihibitors (ACEi), Angiotensin Receptor Blockers 
(ARB), Thiazide diuretic 
Hyperlipidemia 
HMG CoA reductase inhibitor (Statins) 
Diabetes 
Alpha-glucosidase inhibitors, Biguanides, DPP-4 inhibitors, Sodium 
glucose transport inhibitor, Meglitinides, Sulfonylureas, 
Thiazolidinediones, and statins 
Coronary artery 
disease 
PGY-2 inhibitor (Clopidogrel, Ticagrelor, Prasugrel, Ticlopidine), B-
blockers, ACEi or ARB and statins 
Atrial fibrillation 
Direct oral anticoagulants, B-blockers, CCB 
Opt-out Consent Process 
159 
Programmers will develop identical study databases within each HCS to store potential 
160 
participants. All data will be stored on secure, separate servers or on RedCap.  
161 
We will use this pool to send opt-out consent letters to potential participants. An opt-out 
162 
consent packet will be sent to all eligible patients prior to randomization. The packet will 
163 
contain an introductory letter with information about the study, an opt-out form, and a self-
164 
addressed, stamped envelope. An opt-out survey will also be included. This opt-out survey 
165 
aims to gauge the reason for patients opting-out. The survey is optional.  
166 
All materials will be available and sent to patients in both English and Spanish for UCHealth 
167 
and Denver Health patients. If they have previously specified in their contact preferences that 
168 
they prefer English or Spanish, we will send materials in their preferred language. As is 
169 
standard, materials will be sent in English only at the VA. All materials will be sent on 
170 
letterhead and branding appropriate and specific to each HCS. The letter will be signed by 
171 
either the primary care provider for the patient or the site principal investigator. 
172 

5 
Should an opt-out packet be returned by the United States Postal Services (USPS) due to an 
173 
incorrect address, etc, a member of the study team will call the potential participant no more 
174 
than two times to verify their address; should they not hear from the participant the patient will 
175 
not be included in the study. Upon the deadline for response to the opt-out consent has 
176 
expired, patients that have not opted-out will be randomized accordingly. Signed and returned 
177 
opt-out forms will be securely stored in a locked filing cabinet (returned opt-out forms for VA 
178 
patients will be securely stored at the VA). 
179 
180 
Patient Identification Process 
181 
From this pool of candidates, patients will be followed forward and those identified to have a gap 
182 
of at least 7 days in one of the CV medications at any time during the two-year monitoring 
183 
period will be randomized to one of the four study arms. For patients who are prescribed 
184 
multiple CV medications, eligibility for randomization will be triggered by the first 7-day gap for 
185 
any medication.   
186 
We have already developed algorithms that queries pharmacy data for a defined medication 
187 
refill gap based on the date that the medication was supplied and the number of medication 
188 
days supplied. This algorithm is in use for a medication adherence study currently. We will apply 
189 
this algorithm across the 3 HCS to identify non-adherent patients.  
190 
Pharmacy data from UCHealth and Denver Health will be accessed through both hospital-
191 
specific pharmacy data and Surescripts, a network with existing relationships with both HCS 
192 
that provides pharmacy refill data from over 95% of US pharmacies. Daily pharmacy data via 
193 
Surescripts is currently available at Denver Health, and is projected to become available at 
194 
UCHealth in the fall of 2019.  
195 
Every 3 months for up to two years, we will assess whether there are new patients who have 
196 
met entry criteria in the interim, based on existing patients who now meet the eligibility criteria 
197 
or new patients to the clinic. These patients will be sent an opt-out package and provided a 
198 
reasonable timeline to return the opt-out form. We will continue to add new patients until 
199 
months 24 following the start of the pragmatic study at each site to allow for at least 12 months 
200 
of patient follow-up to assess for non-adherence. 
201 
202 
Mobile Messenger 
203 
Mobile Messenger (Upland Communications, Austin, TX) is an online platform specializing in 
204 
text message transmission. Three separate accounts have been created to ensure each HCS 
205 
data are stored separately; protected health information (PHI) between sites will never be 
206 
combined. Each account will have a separate login and password. To access the Mobile 
207 
Messenger account for the VA, Nudge study team members must be signed in behind the VA 
208 
firewall. 
209 
A trained member of the Nudge study team will upload phone numbers, first name, medication 
210 
class, and first three digits of the zip code. First, the researcher will identify the patient has a 
211 
listed cell phone. Upon confirming the phone number, the patient will be entered into the 
212 
appropriate study arm. Message schedules and example messages (attached) are available in 
213 
this application packet. 
214 
The program will allow patients to STOP if they would like to opt-out of the study, or indicate 
215 
DONE if they have filled their medication or if their physician has cancelled the medication of 
216 
interest. There will also be an option to receive messages in Spanish. 
217 
218 

6 
Message arm descriptions 
219 
The study will randomize at the patient level. Once randomized, patients will remain in the 
220 
same study arm for the entire study whether or not they have subsequent refill gaps. 
221 
1) Usual Care: This group will not receive an intervention. We have included a usual care
222 
group to demonstrate the impact of the text messaging interventions above and beyond usual
223 
care given that many prior medication adherence interventions have demonstrated small to
224 
negligible effects.
225 
2) Generic nudge: A generic reminder text will be delivered to patients to refill their
226 
medication at days 1, 3, 5, and 9 after they been labeled as non-adherent. During aim 1, we
227 
will further assess the optimal timing for delivery of these messages. We have put in the
228 
specific days to provide an example. In the day 1 text message, patients will have another
229 
opportunity to opt out of the study with text such as “text STOP if you wish to withdraw from
230 
this study.” The texts will stop once a patient has filled their medication.
231 
3)
Optimized nudge: A behavioral nudge text will be delivered to patients to remind them to
232 
refill their medications at days 1, 3, 5 and 9 after they have been labeled as non-adherent.
233 
During aim 1, we will further assess the optimal timing for delivery of these messages. We
234 
have put in the specific days to provide an example. In the day 1 text message, patients will
235 
have another opportunity to opt out of the study with text such as “text STOP if you wish to
236 
withdraw from this study.” The texts will stop once a patient has filled their medication. The
237 
content of the behavioral nudge text messages will vary with each text and will be derived from
238 
the text message library built during the pilot year (COMIRB #18-0630)
239 
4)
Optimized nudge plus AI Chat Bot: A behavioral nudge text will be delivered to patients
240 
to remind them to refill their medications at days 1 and 3 after they have been labeled as non-
241 
adherent. In the day 1 text message, patients will have another opportunity to opt out of the
242 
study with text such as “text STOP if you wish to withdraw from this study.” The texts will stop
243 
once a patient has filled their medication. If the patient has not filled their medication on days 5
244 
and 9, an AI will conduct interactive chat via a chat bot to assess barriers filling the medication
245 
as described in Aim 1 above.
246 
The AI Chat bot will assess for common barriers to medication adherence 1) socioeconomic 
247 
factors, 2) provider-patient/health care system factors; 3) condition-related factors; 4) therapy 
248 
related factors and 5) patient-related factors using a developed script that we are currently 
249 
employing in a medication adherence study. Communication about all of these barriers will be 
250 
pre-programmed to use as algorithms in the chat bot automated program. For each barrier, the 
251 
AI Chat bot will problem-solve with the patient and identify commonly used successful 
252 
approaches to overcome barriers, and will ask patients to choose and enact one solution to 
253 
improve medication adherence. The AI chat bot library will include algorithms to support 
254 
specific strategies to circumvent the adherence barriers responsible for each instance of a 
255 
medication refill gap. For example, patients would be queried to determine if they have 
256 
difficulty remembering what medications to take and when to take them; those that do would 
257 
be asked if using a medication diary, involving a caretaker, or setting an alarm on their phone 
258 
would help. Patients would be asked if they would like to try one of these strategies; for those 
259 
that agree and identify a strategy, the AI chat program will include an algorithm to check in one 
260 
week later to see how this strategy is going. Those who do not agree and/or identify a strategy 
261 
will be offered other options and the process repeated until they do. If there are issues that 
262 
arise that are not pre-programmed into the AI chat bot library, the AI Chat bot will refer the 
263 
patient to the study pharmacist at each site for consultation and assistance with the issue. For 
264 
example, a patient may have stopped taking his medication due to a side effect. The AI Chat 
265 
bot will document this information through interactive chat then refer the patient to a study 
266 

7 
pharmacist to see if there are alternative medications. Dr. Bull has programmed libraries very 
267 
similar to this AI Chat bot approach and utilized them for behavior change in prior interventions. 
268 
Patients will be followed for up to 24 months. They will remain in the same study arm for the 
269 
duration of the study and receive the same intervention for subsequent episodes of non-
270 
adherence. 
271 
272 
Plan for responding to text messages from patients 
273 
a) If a patient texts “stop” to unsubscribe, the patient is automatically withdrawn from
274 
receiving the intervention (messages). Patients may also text misspellings of the word
275 
"stop” (such as “tsop,” etc.), or send a message indicating the patient would like to be
276 
removed from the study. We will continue to view already-collected, standard of care
277 
data from these patients.
278 
In the initial version of the consent form, we described a “stop” response as a way to
279 
“fully withdraw” from the study. These patients were opted out from the study entirely,
280 
and data are no longer being collected on these patients. In v.12.3.19 of the info sheet,
281 
the language was amended. For patients that have hit "stop" in the original cohort,
282 
researchers will call the patients (please find script) to obtain verbal consent to collect
283 
their data moving forward. If patients do not consent or cannot be reached after two
284 
attempts, we will continue to treat the patients as fully opted out and we will not view
285 
their data moving forward. Patients from this original cohort that hit "stop" in the future
286 
will be flagged and the calling process will be implemented.
287 
b) Should a patient respond “done” to indicate they have already filled their prescription or
288 
a physician has cancelled their medication, we will stop sending patients any further
289 
text messages about refilling the medication in which they had a delay until their
290 
medication is again due for a refill (i.e. 30 – 90 days in the future). We will immediately
291 
remove the patient from further text messages for this instance of medication non-
292 
adherence.
293 
Misspellings of “done” or messages indicating they have already filled their prescription
294 
or that a physician cancelled their medications will be treated as if they responded
295 
“done.” If a physician has stopped their medication, we will no longer monitor for
296 
medication refills for that specific medication. For other medications, we will resume
297 
following the other medication refills. If the patient has a refill gap for the same or for
298 
other medications again, we will start delivering text messages within the same arm
299 
that they had been previously randomized to.
300 
c) Patients may request Spanish messages at any time via text. We will start to deliver
301 
Spanish language texts following the request. If the patient meets additional criteria for
302 
texts during the duration of the study, they will receive Spanish language texts.
303 
d) There will be text messages that do not fall into any of the categories above. A
304 
Research Assistant will monitor these responses and will triage the messages
305 
depending on the content of the messages. In our pilot study, some patients sent
306 
responses to the text messages that (a) requested additional information about the
307 
study and/or (b) requested more detail on the specific medication that required a refill,
308 
even though the text message they were responding to did not solicit this information.
309 
For these types of unsolicited messages, the Research Assistant will respond with a
310 
link to our study webpage where we will post information about the study, sponsors,
311 
participating institutions, and a contact number they can call for more information. This
312 
webpage will include a “Frequently Asked Questions” (FAQ) about the Nudge Study,
313 

8 
and we will post responses to anticipated questions (e.g. Does my provider know about 
314 
this study?; How did you get my cell number?; What if I don’t want to participate?; etc.).  
315 
We anticipate that patients may text unsolicited information about a side effect or 
316 
adverse event related to their medications. In these cases, we will have the site study 
317 
pharmacists call the patient to find out more about the issue. We will also have the 
318 
pharmacist contact the patient’s PCP to make them aware of the issue.  
319 
320 
Other responses, such as questions about the intervention or requests for information about 
321 
their medication, will be triaged and responded to by a research assistant, pharmacist, or 
322 
physician, as deemed appropriate. A call script for the research assistant is included. We will 
323 
catalogue the messages that we receive from patients and if there is a theme, we will develop 
324 
a FAQ and place information on the study website. In cases where the patient still has not 
325 
refilled their medication after 5 days of their reply “done”, a research assistant on the study 
326 
team will first confirm that the medication refill has not been completed. If the medication has 
327 
not been refilled after chart review, they will then contact the patient to see if he or she is 
328 
having issues with refilling the medication and try to resolve any issues with the patient.  
329 
330 
In addition, we will update the patient’s care provider on a regular basis about medication gaps 
331 
and other medication issues of their patients. The frequency and mode of contact will be based 
332 
on their indicated preference, which we will solicit at the start of the study during our clinic 
333 
presentation. If the provider did not indicate a preference, we will send this information via email 
334 
on a weekly basis. 
335 
336 
337 
Quantitative Analysis 
338 
339 
The study will be a randomized controlled study with four treatment arms.  When patients are 
340 
identified through pharmacy refill data to have a 7-day gap in any prescribed CV medication 
341 
refills, they will be randomized to one of four arms, described in Intervention below. 
342 
Randomization will be stratified within each of the eight clinics, and within strata defined by 
343 
number of other CV medication classes that are prescribed at randomization (1-4), using blocks 
344 
of 4 patients to ensure balance within clinics over time. Thus, within each clinic and number of 
345 
other medication stratum, each set of 4 consecutively enrolled subjects will be randomized to 
346 
the four study arms. Treatments will be initiated immediately upon randomization, in response to 
347 
the 7-day gap.  
348 
349 
The primary outcome is adherence to CV medications as measured by 12-month proportion of 
350 
days covered (PDC).  There are several complications in using this outcome, including a) a 
351 
need to account for periods when subjects are not at risk of depleting their medication supply 
352 
(see below for details), b) defining PDC when considering five medication classes 
353 
(antihypertensives, statins, oral hypoglycemics, antiplatelets, and anticoagulants), c) accounting 
354 
for number of other prescribed CV medications (more opportunities to miss days, and more 
355 
opportunities to receive reminders), d) modeling the non-normal bounded distribution of PDC, 
356 
and e) a desire to express results on a PDC difference (not risk ratio or odds ratio) scale. 
357 
Analyses accounting for these complications are described below.  Secondary outcomes 
358 
include clinical events (e.g., event times for stroke, MI, mortality), utilization of care (e.g., 
359 
hospitalizations or clinic visits for CV-related reasons), and costs of the interventions and of 
360 
medical care. Given survival, subjects will be followed for at least 12 months following 
361 
randomization to assess these secondary outcomes. Subjects who have more than one year of 
362 
follow-up (up to 3 years depending on when they are enrolled during years 2-3) will continue to 
363 
be followed for secondary outcomes.   
364 

9 
365 
Ascertainment of Outcomes 
366 
Data for the primary outcome PDC will be obtained using pharmacy records from each of the 
367 
healthcare systems during the 365-day follow-up.  To address complications a and b above: a) 
368 
The assessment will be based on the number of outpatient days a patient has a medication 
369 
available, among all days they were at risk of depleting that medication.  At-risk days will be 
370 
days during which a patient was prescribed the medication and should have depleted their 
371 
supply, and will exclude days following death.  Inpatient days will not be considered in the 
372 
calculation.  Note that at-risk days is medication specific, as medications may change during the 
373 
one-year assessment period.  b) We will consider three definitions of PDC: i) Medication-
374 
specific PDC: calculate PDC for each class of medication a patient is prescribed, ii) All-
375 
medication PDC: calculate PDC requiring all prescribed and at-risk medications be available on 
376 
a day, and iii) Average PDC: calculate the average PDC across medications the patient is at risk 
377 
of depleting on a day, and average over all days when at risk of depleting at least one 
378 
medication. 
379 
380 
Secondary outcomes for clinical events and utilization of care will be captured from the 
381 
electronic health records (EHRs) at each of the three health care systems.  Several sources of 
382 
cost data will be used. To capture costs of development, implementation and maintenance of 
383 
the intervention, we will develop instruments (e.g. time logs) and procedures to prospectively 
384 
capture resource use associated with the intervention including what was done, who did it, how 
385 
long it took, and what nonhuman resources were required. Intervention costs will be the long-
386 
term average cost of implementing the intervention excluding research and development costs. 
387 
Medical care costs will be estimated using a resource-based method previously developed to 
388 
assign costs to encounter data. Inpatient utilization will be measured using diagnostic-related 
389 
groups (DRGs), outpatient utilization using relative value units (RVUs), and cost of pharmacy 
390 
utilization using the midpoint between the Federal Supply Schedule (FSS) and the National 
391 
Average Drug Acquisition Cost (NADAC). Inpatient costs will be estimated by applying national 
392 
payment weights to DRGs, outpatient costs by applying a national conversion factor to RVUs, 
393 
and pharmacy costs as the median between the FSS and NADAC. 
394 
395 
Planned Analysis 
396 
Analyses will be based on the intent to treat principle, using all patients who were randomized. 
397 
398 
Descriptive analyses 
399 
Descriptive analyses will be used to describe the cohort and to check for balance across study 
400 
arms within strata (clinics and number of other medications prescribed).  Primary outcome PDC 
401 
will be calculated during the one-year period following treatment initiation.  Simple descriptive 
402 
estimates of each patient’s PDC on each medication will be used for descriptive analyses.  
403 
404 
Modeling PDC 
405 
Formal analyses will be based on daily data, using a binomial-type model with logistic link for 
406 
the number of days covered by medication, which will be 365 but excluding days not at risk of 
407 
depleting as described above.  We will model the three definitions of PDC described above: 
408 
409 
i) Medication-specific PDC: For a given medication class, the model will include fixed effect
410 
terms for treatment arm, clinic, patient covariates, and number of other CV medications the
411 
patient is prescribed, and a random subject effect for a subject’s tendency to have higher or
412 
lower PDC compared with other subjects.  Covariates to be adjusted for will be selected a priori
413 
based on clinical considerations and covariate data quality, the latter assessed after we have
414 
enough pilot and baseline data to make the assessment but before examining outcome data.
415 
All covariates included in outcome models will also be included in imputation models (see
416 

10 
below). The analysis will be carried out using daily data to account for the differing numbers of 
417 
at-risk days across patients, and the possibility that the number of other prescribed CV 
418 
medications may change during the one year PDC modeling period.  The parameter being 
419 
modeled is the probability that a given day is covered, conditional on fixed and random effects.  
420 
This approach will account for complications a) varying numbers of at-risk days for different 
421 
patients and different medication classes, b) defining PDC when considering multiple 
422 
medication classes, c) adjusting for number of other CV medications prescribed, and d) the non-
423 
normal bounded distribution of PDC.  Due to non-independence of days for a given subject, the 
424 
binomial assumption will not hold and inference will need to be carried out using bootstrap or 
425 
Markov chain Monte Carlo (MCMC) methods as described below. 
426 
427 
ii) All-medication PDC: Each day for each subject will be coded as whether they have all
428 
prescribed and at-risk medications available or not.  The same modeling process described
429 
above for medication-specific PDC will be used.
430 
431 
iii) Average PDC: Medication-specific PDC models will be estimated as described above, and
432 
will be used to calculate estimates of average PDC using the standardization methods
433 
described below.
434 
435 
Treatment comparisons 
436 
Discussions with clinicians have indicated expressing results on a linear scale (PDC 
437 
differences), as opposed to odds ratios or risk ratios, will be most appropriate for interpretation.  
438 
Several such modeling approaches exist including Modified Binomial using robust standard 
439 
errors, Additive Binomial using maximum likelihood, and standardization using counterfactual 
440 
methods.  We plan to base primary analyses on the standardization approach, which allows 
441 
flexibility in estimating treatment comparisons and provides population average estimates that 
442 
are parallel to simple proportion estimates of PDC.  In this approach a mixed logistic regression 
443 
model will be estimated using maximum likelihood, and the estimated model is used to calculate 
444 
probabilities that each day is covered for each subject and each medication.  Quantities of 
445 
interest are estimated from the relevant averages of probabilities.  For example, the estimated 
446 
probability of medication availability for a given medication on a given treatment is the average 
447 
of all estimated probabilities for all exposed days across all subjects assuming they received the 
448 
specified treatment and were prescribed the specified medication, regardless of which treatment 
449 
they actually received and which medication(s) they were actually prescribed.  Contrasts of 
450 
interest are formed from these estimates.  Primary hypotheses involve pairwise comparisons 
451 
between each of the four study arms, and will be conducted using the pairwise contrasts.  
452 
Standard errors and confidence intervals will be calculated using bootstrap methods, since the 
453 
binomial assumption of the model will not hold.  Bonferroni adjustments will be used to correct 
454 
for the 6 pairwise treatment comparisons.   
455 
456 
The secondary outcomes of clinical events, care utilization and cost will be analyzed using 
457 
similar approaches but based on appropriate models, e.g. Cox survival models for time to 
458 
clinical event or rehospitalization, generalized gamma regression for cost, etc.  Similar 
459 
standardization methods allow results to be expressed on interpretable scales such as risk 
460 
difference.  Data will be analyzed using SAS (SAS Institute Inc., Cary, NC) and R software. 
461 
462 
Missing data:  Patients with missing data in covariates, treatments (unlikely) or outcomes will be 
463 
retained in the intent to treat analyses and their missing values imputed using multiple chained 
464 
equation methods.  The imputation model will include all covariate, exposure and outcome 
465 
variables used in the outcome model.  If patients are randomized but later opt out or drop out, 
466 
their outcome data will be collected up to the point that they leave, and will be used as a 
467 
variable in the imputation.  When outcome data cannot be obtained, every effort will be made to 
468 

11 
document reasons for these missing observations. We will carry out the recommended 
469 
sensitivity analyses based on pattern mixture models, by assuming various values for difference 
470 
in means between observed and unobserved data and assessing differences in model 
471 
conclusions. 
472 
473 
Secondary analyses of primary outcome:  We will consider several sensitivity analyses.  First, 
474 
an alternate estimation approach using MCMC instead of maximum likelihood will be considered. 
475 
This approach would eliminate the need for bootstrapping, and would also allow imputation 
476 
during the model estimation by treating missing covariates and outcomes as parameters in the 
477 
Bayesian model and estimation.  Second, we will fit mixed linear regression models to the 
478 
binary 0/1 daily medication coverage outcome.  Treatment differences will be on the PDC 
479 
difference scale.  Robust sandwich estimators will be used in the estimation, and bootstrap will 
480 
be used for inference.  We expect results of these sensitivity analyses to be similar to the 
481 
primary analysis results.  We will also use the methods described above to conduct several 
482 
secondary analyses evaluating a) predictors of patients having an initial gap of at least 7 days, 
483 
using the initial pool of candidate patients, to identify types of patients or prescription 
484 
characteristics for patients at highest risk for non-adherence; b) predictors of subjects having a 
485 
gap of at least 7 days while enrolled in the study; c) PDC for the individual medication that 
486 
triggered enrollment in the study, to allow analyses of heterogeneity of treatment effect (HTE) by 
487 
drug class; and d) analysis of alternate definitions of primary outcome, including time to first gap, 
488 
PDC below 0.8, and number of 7-day gaps, during the one year follow-up.  We will also carry 
489 
out an analysis to estimate possible HTE between the three health care systems, using an 
490 
interaction of treatment arm with health care system. 
491 
492 
Health system costs: This analysis will examine intervention costs and incremental medical 
493 
costs associated with the intervention. Cost data will be analyzed using the same methods 
494 
described above with factors for study arm and health care system/clinic strata, using 
495 
generalized Gamma regression, which include as special cases lognormal, Gamma and Weibull.  
496 
The primary dependent variable will be healthcare cost, in total and separated into inpatient, 
497 
outpatient, and pharmacy cost buckets. The primary independent variable will be the 
498 
intervention arm the patient was randomized to. Results will be stratified by healthcare system.  
499 
500 
Power Assessment 
501 
We first estimated sample size required to achieve 80% power for the desired change in the 
502 
primary outcome, then used current data to show we will be able to achieve this sample size. 
503 
504 
Power and sample size: Required sample size was estimated for the primary outcome PDC 
505 
during the 12 months following randomization. Preliminary data from the VA were used for these 
506 
estimates.  We made the following assumptions:  a) Significance using two-sided level 0.05 
507 
tests, b) Power at least 80%; c) Difference between treatments in PDC of 10 percentage points; 
508 
d) Bonferroni adjustment for the 6 pairwise comparisons among the 4 study arms, resulting in
509 
adjusted level 0.05/6; e) Analysis stratified by health care system; and f) Within-system and
510 
within-treatment residual standard deviation of 12 month PDC equal to 0.247 (mean 0.771),
511 
obtained by analysis of 594,466 veterans during the period 01/01/2010 – 09/30/2012 who were
512 
prescribed CVD medications. The large SD is due to a highly left-skewed PDC distribution.  (We
513 
base sample size estimation on t-tests rather than the proposed binomial models since we do
514 
not have estimates of quantities needed to carry out power simulations.  We may be able to
515 
obtain data for the PDC outcome for a set of baseline patients, in which case we will consider a
516 
refined power analysis.)  Using these assumptions and estimates, and comparing any two
517 
treatments using a linear model with the above residual standard deviation of PDC, we estimate
518 
using sample size functions in R that we will need N=154 subjects per treatment arm, total
519 

12 
across the three health care systems, for a total of 616 subjects to be randomized across the 
520 
three health care systems. 
521 
522 
Available sample sizes:  We obtained data from each of the three health care systems on the 
523 
number of patients at the eight specific clinics (VA, 2 clinics; UCHealth, 4 clinics; DH, 2 clinics) 
524 
to be included in this study. Figure 1 below shows estimated numbers of patients seen, 
525 
numbers with CVD conditions and prescribed CVD medications, and numbers with 7 day gap.  
526 
These patients will be invited to participate, and assuming 80% agree to participate, and 90% of 
527 
those yield usable outcome data, we expect to have usable outcome data for about 4,364 
528 
patients, or about 1,091 per treatment arm. Thus, we expect to have ample subjects (about 
529 
seven times as many as needed) to achieve the necessary power for the primary analysis of 
530 
PDC.  Additional subjects will provide power for secondary analyses, and for analyses of 
531 
secondary outcomes. 
532 
533 
Exploratory 
534 
535 
Covid Messaging 
536 
537 
On April 6, 2020, the Nudge Study started to send Covid messages to patients enrolled in the 
538 
study (Amendment PAM006-1, approved March 18, 2020). 
539 
540 
This study plans to scale the concept of the Nudge Study to other applications in the future. We 
541 
propose pulling data regarding Covid testing of patients sent messages after April 6, 2020  to 
542 
compare with Arm 1 (non interventional arm).   
543 
544 
Comparing opt out vs opt in demographics 
545 
546 
There is a growing interest in learning more about who is opting out of our study, as well as who 
547 
is opting in. 
548 
549 
Our study plans to compare the demographics of patients (race, age, ethnicity, gender) that 
550 
have opted in vs opted out of our study. This requires pulling data from Electronic Health 
551 
records to compare both cohorts.  
552 
553 
We propose (12/1/20) to retrospectively pull demographic data from the EHR for patients that 
554 
have opted out of the study (race, age, ethnicity, gender) to compare these data with patients 
555 
that have opted in to the Nudge study. Moving forward, we will include a statement in the opt out 
556 
packet (Information Sheet, v 12/1/20) to inform potentially-eligible patients that receive an opt 
557 
out packet that demographic information will be drawn. 
558 
559 
Qualitative Analysis 
560 
Assessment of patient perspectives 
561 
In year 5, after the intervention and follow-up period has ended, we will survey patients via text 
562 
message and Nof1 telephone interviews. We have conducted similar interviews with patients 
563 
following adherence interventions. These interviews will help inform further refinement of the 
564 
interventions65 as we plan for broader dissemination of the intervention (if demonstrated to be 
565 
effective) to more clinics and patients with other chronic conditions. 
566 
Text message survey 
567 
We will randomly sample 150 patients across the three sites to participate in a voluntary survey 
568 
about their experience with the intervention over the University-hosted Qualtrics platform 
569 

13 
(https://ucdenver.co1.qualtrics.com/jfe/form/SV_5jBlah5rV7tcSoK). The survey will not collect 
570 
or solicit any PHI (see Attachment – Satisfaction Survey v 9.23). The sample will be stratified 
571 
evenly across patients who received one of the three intervention arms. 
572 
Mailed survey 
573 
If the response rate of the text messaged surveys is below 20%, the surveys will be mailed out 
574 
to patients who previously received the survey via text but did not respond. A $2 incentive will 
575 
be included in the mailed packet to encourage responses. An information sheet will be 
576 
included to explain the survey and incentive to the patient (Attachment – Information Sheet 
577 
v2.23). 
578 
579 
Nof1 Interviews 
580 
In a random sample of up to 50 patients who respond to the survey and indicate they would be 
581 
interested in participating in an additional interview, we will contact them via telephone to get 
582 
more in-depth feedback through qualitative interviews on the intervention (Attachment - Nof1 
583 
interview guide, v. 9.23). No deeply personal information or PHI will be solicited. Interviews will 
584 
take place over Zoom, Cisco Jabber (DH/UCH), and secure hospital phone lines (VA). 
585 
Interviewers will utilize detailed notetaking with an interview guide, and interviews will be 
586 
recorded to be transcribed (at DH and UCH only). Interviews will cease once information 
587 
saturation is reached. Nof1 participants will receive an electronic gift card of $30. A postcard 
588 
consent will be read to the patient prior to the commencement of the interview (Attachments – 
589 
Satisfaction Postcard Consent v9.30 (for DH/UCH), Satisfaction Postcard Consent v5.23 (for 
590 
the VA)). 
591 
592 
593 
Assessment of provider and health systems leaders perceptions 
594 
We will conduct key-informant interviews with up to 2-3 providers (6-9 across the 3 HCS) 
595 
from each setting whose patients have received the intervention to get their feedback about 
596 
the intervention and the intervention effects on their patient’s medication taking behavior. 
597 
For some providers, they may have received a note from the study team informing them that 
598 
their patient did not refill their medications and we will also interview the providers on their 
599 
perceptions of that process. We will also conduct key-informant interviews with health 
600 
systems leaders (3-6 interviewees) in each setting who are responsible for institutional 
601 
policies related to patient data-management, informatics and pharmacy. In these 
602 
interactions, we will share findings from the research and gauge their reaction to the findings. 
603 
With any indication of positive outcomes, we will ask participants to describe their likelihood 
604 
to maintain the system within their setting, and to discuss any barriers to maintenance and 
605 
specific actions needed to overcome these barriers.  
606 
Study Evaluation 
607 
The study will be evaluated using the Practical, Robust Implementation and Sustainability 
608 
Model (PRISM) and RE-AIM framework components of Reach, Effectiveness, Adoption, 
609 
Implementation and Maintenance. We will also develop tools and a sustainability plan to 
610 
broadly disseminate the intervention, if effective. PRISM considers important implementation 
611 
concepts from Diffusion of Innovations, the Chronic Care Model, the Model for Improvement, 
612 
and the RE-AIM framework and highlights four components that influence implementation 
613 
success: 1) organizational and participants characteristics; 2) intervention characteristics from 
614 
the organizational (health care system and providers) and participants’ perspectives (i.e., 
615 
patients); 3) implementation and sustainability infrastructure (training and support); and 4) 
616 

14 
external environment. These four elements will be assessed in a formative manner and will be 
617 
critical to understanding how to further disseminate the intervention if demonstrated to be 
618 
effective. PRISM also identifies a set of important outcomes from the RE-AIM model (i.e., 
619 
Reach and Effectiveness, Adoption, Implementation, and Maintenance) for evaluation. We will 
620 
incorporate the assessment of the 4 components that influence implementation success into 
621 
our evaluation and this will be further discussed in the implementation evaluation section below 
622 
(Table 2).  
623 
Table 2. RE-AIM measures 
624 
RE-AIM 
Domain 
 Domain Description Measure 
Data Source 
Reach 
Degree to which 
target 
population 
is impacted. 
1. Number of eligible patients.
2. % of patients who did not opt out.
3. % patients with 7-day gap
4. Representativeness of participants
Study database derived 
from EHR clinical and 
pharmacy data 
Effectiveness 
Success of the 
intervention to 
change outcomes. 
1. Improvement in medication adherence
(PDC) and reduction in utilization/clinical
outcomes/costs
Study database 
Adoption 
Degree to which 
interventions are 
taken up by 
organizations. 
1. Clinics approached and willingness to
participate in the intervention
Study database 
Implementatio
n 
Degree to which 
interventions are 
implemented as 
intended. 
1. Among patients with gap, how many
interventions were delivered per patient
2. Proportion reached and by method
3. Among patients in arm #4, Proportion where
AI chat bot was used and the barrier identified
4. Barriers and facilitators to implementation
5. Budget impact/ Cost of the program and
replication costs
6. Qualitative interviews focused on: 1)
organizational and participants characteristics;
2) intervention characteristics from the
organizational (health care system and
providers) and participants’ perspectives (i.e.,
patients); 3) implementation and sustainability
infrastructure (training and support); and 4)
external environment.
1. Qualitative interviews
2. Study database
 Maintenance 
Can the program 
be sustained over 
time? 
1. Identifying barriers to maintenance at the
end of the study.
2. Intent to continue intervention following
grant support
3. Can intervention be extended to other
patient populations with different conditions
1. Post-implementation
qualitative interviews
2. Study database
625 
Data Management 
626 
 We will set up a distributed network across the 3 HCS and each HCS will manage their own 
627 
data in the delivery of the study intervention. We will set up parallel processes so that each of 
628 
the sites can monitor patients for gaps in medication refills and can deliver the intervention 
629 
when needed. This will keep patient PHI within each of the HCS without the data leaving the 
630 
HCS. We envision that this will be the process for other HCS that want to adopt the 
631 
intervention if the intervention is demonstrated to be effective across the NIH Collaboratory 
632 
sites.  
633 

15 
At UCHealth and DH, pharmacy data is available for prescriptions filled within the HCS 
634 
pharmacy as well as non-HCS pharmacy. Data for prescriptions filled at non-HCS pharmacy 
635 
are automatically obtained at both UCHealth and DH as part of their routine clinical operations 
636 
in the care of their patients through Surescripts. For example, we have all pharmacy data for 
637 
UCHealth patients, which is derived from the following sources: ~15% from UCHealth 
638 
pharmacy and ~85% from non-UCHealth pharmacy. Surescripts pharmacy data are available 
639 
at the current time for Denver Health and will be estimated to become available in the Fall of 
640 
2019 for UCHealth. 
641 
642 
Data Movement 
643 
Programmers will upload data to the appropriate secure server in an appropriate format. All data 
644 
will be kept separate from the data associated with the other affiliated healthcare systems. Data 
645 
will be accessed by a centralized programmer from the Nudge Team via secure server to put 
646 
into Mobile Messenger (Figure 2). 
647 
Figure 2. Data Flow and Transfer for Proposed Trial 
648 
649 
650 
Stakeholder Panel 
651 
Stakeholder engagement in research is an important and challenging task. On one hand, we 
652 
want to avoid tokenism and want stakeholders to be as involved as they would like to be. On the 
653 
other hand, meaningful engagement can require a substantial time commitment. In a previous 
654 
study (COMIRB #18-0630), we developed a standing stakeholder panel. We will continue to 
655 
convene the Stakeholder panel quarterly, as needed. 
656 

Participants: The stakeholder panel consists of up to 5 people from each HCS: 1-2 of
657 
the following: patients, pharmacists, providers and persons involved in the leadership or
658 
operations of the health system from each HCS. Members previously recruited through
659 
relationships of the investigators will be welcome to join. New members will be identified
660 
through existing relationships / snowball recruiting.
661 

Location: The panel will meet in a central, secure location with ample free parking.
662 
Each member of the panel will be reimbursed $50 per meeting.
663 

Meeting content: The Nudge study team will present the ongoing study progress to
664 
obtain feedback on the project, such as the implementation challenges and brainstorm
665 
with the investigators strategies to mitigate these. This partnership between the study
666 
Data storage: 
PHI pulled from 
EHR of eligible 
patients will be 
saved on secure, 
restricted, and 
appropriate 
servers.  
All VA data will 
remain behind the 
VA firewall. 
Identification of 
eligible patients 
Data Storage: 
Returned (paper) 
opt-out forms  to be 
saved in secure, 
locked cabinet. 
Patients opting out 
of the study will be 
saved on the 
appropriate server. 
VA mailings will be 
sent via the VA mail 
room; VA opt-out 
consents will remain 
in secure, locked 
cabinet at VA. 
Opt-out letters 
sent to patients 
Data storage: 
Programmers will 
identify refill gaps 
by querying daily 
pharmacy data 
pulls. Data will be 
saved on secure, 
restricted, and 
appropriate 
servers.  
All VA data will 
remain behind the 
VA firewall. 
Monitor for 
pharmacy gaps 
Data Storage: 
Patient phone 
number, first 
name, medication 
class will be 
uploaded to 
Mobile 
Messenger. 
Analysts / PRAs 
will log in to the 
VA firewall before 
entering into 
Mobile 
Messenger. 
Nudges sent to 
participating 
patients 
Data Storage: 
Patients' 
participation from 
Mobile 
Messenger will be 
recorded and 
saved on secure 
servers. 
All VA data will 
remain behind the 
VA firewall. 
Message output 

16 
team and our stakeholders (patients, providers, and health system leaders) will help to 
667 
make the intervention components and products more sustainable. 
668 

Payment: Participants in the panel will receive $50/meeting they attend, resulting in up
669 
to $200 per year.
670 
671 
Protection of Human Subjects 
672 
This research project involves human subjects recruited from three health care systems (HCS): 
673 
1) University of Colorado Health; 2) Denver VA Medical Center; 3) Denver Health and Hospital
674 
Authority. We will identify eligible patients from the 3 HCS using the electronic medical record
675 
system of each HCS. Patients must meet all of the following inclusion criteria to participate in
676 
the study: the presence of 1 or more of the following cardiovascular conditions (hypertension,
677 
hyperlipiemia, diabetes, coronary artery disease, and/or atrial fibrillation) and are prescribed 1 or
678 
more of the classes of medications to treat the cardiovascular conditions (b-blockers, calcium
679 
channel blockers (CCB), angiotensin converting enzyme inhibitors (ACE-I), angiotension
680 
receptor blockers (ARB), HMG-CoA reductase inhibitor (statins), thiazide diuretics, PGY-2
681 
inhibitors (anti-platelets), direct oral anticoagulants, oral diabetes medications). These inclusion
682 
criteria are designed to identify a group of adults with suboptimal medication adherence who
683 
would benefit from the proposed intervention. Participants are not eligible if they meet any
684 
exclusion criterion: 1) patients who do not have either a landline nor cellphone; 2) enrolled in
685 
hospice or palliative care; 3) Non-English or Spanish speaking. We plan to enroll up to 100
686 
subjects from each of the healthcare systems, resulting in enrollment of up to 300 patients total.
687 
The study team believes that this project poses minimal risk to all subjects involved. There is no 
688 
clinical intervention being proposed, and no deeply personal matters will be discussed. The goal 
689 
of this study is to ensure that patients take their medications as prescribed. 
690 
Among patients who fulfil these eligibility criteria, we will send them a letter outlining the study 
691 
and include an opt-out postcard. Within the opt-out packet will include an introductory letter, 
692 
information about the study, the opt out form, and a self addressed, stamped envelope. The 
693 
envelope will be directed to a PRA within the Data Science to Patient Value (D2V) initiative, who 
694 
will remove the participant from the potential list, and save the consent in secure, locked cabinet, 
695 
as noted in the IRB application. Should an opt-out package be returned, a D2V PRA will call the 
696 
potential participant no more than 2 times to verify their address; should we not hear from the 
697 
participant, we will not include them in the study. 
698 
For patients who do not return the letter after the proposed deadline, we will include them in the 
699 
study. Additionally, we will engage our stakeholder panel in designing, refining, and 
700 
implementing the intervention. Stakeholder participants have been previously consented via 
701 
postcard consent. 
702 
703 
Study subject identification 
704 
Study subjects will be identified using electronic data. For the Stakeholder Panel, we will 
705 
continue to involve our existing Stakeholder Panel in the study, and recruit through existing 
706 
relationships and/or snowball recruitment techniques as needed. 
707 
We will request a waiver of documented consent from study subjects as this study is very low 
708 
risk and could not be reasonably completed without such. As described previously, we will 
709 
provide all potential participants with a packet of information about the study, an opt-out letter, 
710 
and a self-addressed stamped envelope. The text messages will also allow for opting out by 
711 
texing STOP (as described above).  
712 

17 
The study will be conducted according to Good Clinical Practice guidelines, the U.S. Code of 
713 
Federal Regulations (CFR) Title 21 CFR (Part 50 – Protection of Human Subjects and Part 56 – 
714 
Institutional Review Boards) and the Declaration of Helsinki.  
715 
716 
Sources of Materials 
717 
Trained and certified professional staff will manage all data according to detailed study protocols. 
718 
Data will be used specifically for research purposes. All materials will be created for low-literacy 
719 
populations, and also be translated to Spanish for our Latino population. Materials will be 
720 
reviewed by our Stakeholder Panel as applicable. 
721 
722 
Potential Risks to Participants 
723 
We do not anticipate any substantial risks to be associated with participation in this study. As 
724 
with any study involving participants with chronic disease, however, there is some risk of 
725 
psychological discomfort related to discussing disease management.  Participants will be 
726 
informed that if they choose to discontinue the study at any time, this will not interfere with their 
727 
usual medical care. 
728 
729 
As with all research, there is always a slight risk of loss of confidentiality. We have standard 
730 
operating procedures for data acquisition and data management designed to protect against 
731 
data loss and maintain patient confidentiality. Computer files will be password protected. Files 
732 
containing names, addresses, or other personal identifiers will have a separate password and 
733 
will be accessible only to personnel who need to contact subjects. 
734 
735 
Behavioral incentives (nudges) will be administered to the participants as a method to reinforce 
736 
the treatment plan prescribed by physicians that patients have already received. The nudges 
737 
themselves will not prescribe medicines nor introduce new instructions/treatments. However, 
738 
there is some risk that patients will interpret the text message information that is ‘tailored’ to 
739 
them has been approved by their physician or that a computerized system is foolproof. 
740 
741 
742 
Adequacy of Protection of Risk 
743 
744 
To mitigate the above risks, the text message nudges will contain a secondary opportunity for 
745 
opt-out, and will contain pharmacy contact information, should the patient decide to discontinue 
746 
their participation or want to contact the pharmacist with questions or need more information 
747 
about the intervention or their prescribed medications. Contact information for those opting out 
748 
of the study will be maintained in a separate file and deleted as soon as recruitment is complete. 
749 
The opt out consent will also state that the text messages are not from their doctor. 
750 
751 
As is the standard, all staff participating in the project will complete compliance and human 
752 
subject research training and all recruitment materials and consent forms will be approved by 
753 
the Colorado Multiple Institutional Review Board (COMIRB).  
754 
755 
Potential Benefits to Subjects 
756 
No claim is made that subjects will benefit from participation in this project. However, the results 
757 
of the study may improve care to the extent that our study improves medication adherence.  
758 
759 

18 
Data Quality, Transfer, and Security 
760 
Designated study team members will oversee all Nudge Study data-related activities, including 
761 
data quality monitoring and data security. All data will be stored on a secure server, separate for 
762 
each site. Designated research team members will oversee local and central data quality 
763 
checks for proper formatting, completeness and consistency. A data privacy and security 
764 
protections plan, consistent with the Health Insurance Portability and Accountability Act and 
765 
Sarbanes-Oxley Act, will be in place prior to project commencement. Research team members 
766 
will establish data use or business associate agreements for sharing data.  
767 
768 
Nof1 Addendum proposal (1/24/21) 
769 
770 
Interview overview 
771 
Trained qualitative researchers will conduct Nof1 (one-on-one) qualitative interviews to assess 
772 
the usability and preference of a proposed research study that is being built off of the existing 
773 
Nudge Study. Interviews will be held over Zoom (Sample questions are available in Appendix I; 
774 
sample messages are demonstrated in Appendix II). Meetings should not exceed 60 minutes 
775 
and patients will be reminded they can stop at any time. Patients will be notified that the Zoom 
776 
meeting will be recorded. 
777 
778 
Patient population  
779 
We will purposively sample diverse patients from each setting to gain a balance of older and 
780 
younger; male, female and non-gender conforming; Spanish and English speakers; and persons 
781 
who identify as Latinx and/or Black proportional to their representation as patients at Denver 
782 
Health. Patients must reside in the State of Colorado, speak English or Spanish, own a cell 
783 
phone, and have the ability to participate in a virtual Zoom interview. 
784 
785 
Recruitment methods  
786 
Nudge Study PRAs will recruit patients from a Denver Health cardiovascular clinic under the 
787 
direction of Site PI Pamela Peterson, MD, MSPH. PRAs will recruit an estimated 20 (no more 
788 
than 30) patients with at least one cardiovascular condition and prescribed at least one 
789 
medication to treat the condition of interest. Due to the Covid-19 January surge, PRAs will 
790 
recruit patients that would otherwise be recruited in person via the phone to reduce Covid 
791 
transmission.  
792 
793 
Consent 
794 
Patients will be read a verbal postcard consent for this study over the phone as minimal risk is 
795 
introduced in these interviews; no deeply personal matters will be discussed.  
796 
797 
Compensation 
798 
Patients participating in interviews will be offered a gift card for an online store valued at $50 for 
799 
their participation. 
800 
801 
802 
803 
1.
Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R, Car J. Mobile phone
804 
messaging reminders for attendance at healthcare appointments (Review). Cochrane
805 
Database of Systematic Reviews. 2013(12).
806 

19 
2.
de Jongh T, Gurol-Urganci I, Vodopivec-Jamsek V, Car J, Atun R. Mobile phone
807 
messaging for facilitating self-management of long-term illnesses (Review). Cochrane
808 
Database of Systematic Reviews. 2012(12).
809 
3.
Vodopivec-Jamsek V, de Jongh T, Gurol-Urganci I, Atun R, Car J. Mobile phone
810 
messaging for preventive health care (Review). Cochrane Database of Systematic
811 
Reviews. 2012(12).
812 
4.
Head KJ, Noar SM, Iannarino NT, Harrington NG. Efficacy of text messaging-based
813 
interventions for health promotion: A meta-analysis. Social Science & Medicine.
814 
2013;97:41-48.
815 
816 

EFFECTIVENESS STATISTICAL ANALYSIS PLAN (SAP) – REVISED 04/20/2023 
1 
 
2 
Summary of revision 
3 
 
4 
This revision of the analytic plan was motivated by several external disruptions during the 
5 
conduct of the study, and by a brief examination of treatment-blinded outcome data.  The main 
6 
changes to the original Statistical Analysis Plan (SAP) involve the method used to model the 
7 
primary PDC outcome, detailed specification of adjustment covariates, and adjustment for 
8 
differing lengths of follow-up due to death, changes in prescriptions, or external disruptions.  No 
9 
changes to primary or secondary outcomes are proposed.  The changes of substance occur 
10 
only in Sections 3.1.a-e. 
11 
 
12 
1. Study summary 
13 
 
14 
The primary outcome will be medication adherence defined as the proportion of days covered 
15 
(PDC) in the 12 months post randomization.  This will be a composite measure across a 
16 
patient’s multiple medications and will be assessed using pharmacy refill data. Secondary 
17 
outcomes will include intermediate clinical measures (e.g., BP control), CV clinical events (e.g., 
18 
hospitalization for myocardial infarction) and procedures (e.g. PCI), medication-associated 
19 
clinical events (e.g., syncope in patient on anti-hypertensive therapy), healthcare utilization, and 
20 
costs.  These outcomes will also be assessed at 12 months, with cost and utilization also 
21 
assessed for specific components as described below. 
22 
 
23 
Study description: This individually randomized three-center trial will compare three behavioral 
24 
nudges based on text and chatbot messaging with usual care to improve adherence to 
25 
cardiovascular medications. 
26 
 
27 
Setting: The study will be conducted in 17 primary care clinics within 3 health care systems 
28 
(University of Colorado Health, VA Eastern Colorado Health Care System, and Denver Health 
29 
Medical Center). 
30 
 
31 
Design: The study will be an individually randomized controlled study with four treatment arms 
32 
(details of interventions below).  Patients become eligible for the study when identified through 
33 
pharmacy refill data to have a 7-day gap in any prescribed CV medication refills. Eligible 
34 
patients will be randomized to one of four arms. Randomization will be stratified within each of 
35 
the clinics, and further stratified within clinics by patients with 1-2 vs 3 or more active CV 
36 
medication classes at baseline, and using blocks of 4 patients to ensure balance within clinics 
37 
over time. Thus, within each clinic and number of other medication stratum, each set of 4 
38 
consecutively enrolled subjects will be randomized to the four study arms. Treatments will be 
39 
initiated immediately upon randomization, in response to the 7-day gap.  
40 
 
41 
Intervention: The four treatment arms will be 1) usual care (control); 2) generic text message 
42 
reminder; 3) text message behavioral nudges only; or 4) text message behavioral nudge plus a 
43 
pre-programmed AI interactive chat bot designed to identify and resolve barriers to medication 
44 
refill and adherence. 
45 
 
46 
Delivery of treatments will be as follows: 
47 
For patients randomized to one of the three active intervention arms: 
48 
- 
If a patient gaps on a medication at baseline, they receive a text to refill that medication. 
49 
- 
If a patient gaps on multiple medications at baseline, they receive only one text about all 
50 
of the medications they gapped on. 
51 

- 
If a patient later gaps on any study medication, they will receive a reminder, regardless 
52 
of whether they had gapped on or been prescribed the medication at baseline. 
53 
- 
If a patient is removed from a medication, no further texts are sent for that medication. 
54 
Patients randomized to the control arm receive no texts or other reminders. 
55 
 
56 
2. Study outcomes 
57 
 
58 
Primary and secondary outcomes: The study outcomes have been selected based on input 
59 
directly elicited from patients and other stakeholders. The primary outcome is adherence to CV 
60 
medications as measured by 12-month composite proportion of days covered (PDC).  
61 
Secondary outcomes include alternate measures of CV medication adherence, clinical events 
62 
(e.g., event times for stroke, MI, mortality), utilization of care (e.g., hospitalizations or clinic visits 
63 
for CV-related reasons), and costs of healthcare utilization. Subjects will be followed for at least 
64 
12 months following randomization to assess these secondary outcomes. Subjects who have 
65 
more than one year of follow-up (up to 3 years depending on when they are enrolled during 
66 
years 2-3) will continue to be followed for secondary outcomes.   
67 
 
68 
Ascertainment of Outcomes: Data for the primary outcome PDC will be obtained using 
69 
pharmacy records from each of the healthcare systems during the 365-day follow-up. The 
70 
medication refill data needed to assess PDC is routinely collected in the pharmacy databases of 
71 
each of the participating sites.  
72 
 
73 
During the UG3 phase, the data management workgroup developed definitions and 
74 
specifications for the secondary outcomes that will be captured from the EHR at each of the 
75 
three healthcare systems.  International Classification of Diseases, Ninth and Tenth 
76 
Revision (ICD 9 and 10), CPT and DRG codes identifying CV clinical events, CV procedures, 
77 
and adverse medication associated clinical events have been compiled to ensure accurate 
78 
identification these outcomes. Outcome measures for BP and LDL have also been standardized 
79 
using NIH Collaboratory definitions for these standardized EHR data elements (Appendix V). 
80 
The data core successfully used EHR and administrative data to assess for each of these 
81 
outcomes during the UG3 pilot year.  
82 
  
83 
Several sources of cost data will be used. To capture costs of development, implementation and 
84 
maintenance of the intervention, we will develop instruments (e.g. time logs) and procedures to 
85 
prospectively capture resource use associated with the intervention including what was done, 
86 
who did it, how long it took, and what nonhuman resources were required. This is further 
87 
discussed in greater detail in the health economics plan. In brief, intervention costs will be the 
88 
cost of implementing the intervention excluding research and development costs. Costs 
89 
associated with healthcare utilization will also be estimated using a resource-based method 
90 
previously developed to assign costs to encounter data. Inpatient utilization will be measured 
91 
using diagnostic-related groups (DRGs), outpatient utilization using relative value units (RVUs), 
92 
and cost of pharmacy utilization using the midpoint between the Federal Supply Schedule (FSS) 
93 
and the National Average Drug Acquisition Cost (NADAC). Inpatient costs will be estimated by 
94 
applying national payment weights to DRGs, outpatient costs by applying a national conversion 
95 
factor to RVUs, and pharmacy costs as the median between the FSS and NADAC. Please refer 
96 
to health economic plan analysis for further details.  
97 
 
98 
3. Statistical analysis plan 
99 
 
100 
Analyses will be based on the intent to treat principle, using all patients who were randomized. 
101 
 
102 

3.1 Analysis of primary composite PDC outcome 
103 
 
104 
3.1.a Summary of modeling approach 
105 
We propose to calculate a composite PDC ratio of days covered for each subject and analyze it 
106 
using a standard longitudinal model for monthly PDC values, with a linear mean model, and with 
107 
robust standard error estimation and associated inference based on weighted Generalized 
108 
Estimating Equations (GEE), e.g. Robins et al., 1995; Preisser et al., 2002.  In more detail and 
109 
with rationale: 
110 
Outcome distribution:  Since data are in the form of a discrete proportion, our original 
111 
proposal was based on Bernoulli/binomial models.  However, treatment-blinded examination of 
112 
a sample of outcome data showed that the binomial assumption will not apply due to spikes 
113 
caused by common prescription lengths (e.g. 30 and 90 days), and strong overdispersion or 
114 
equivalently lack of independence of daily outcome data over time.  We considered alternative 
115 
approaches such as Poisson with offset for number of observation days but similar challenges 
116 
of spikes, overdispersion and lack of independence remain.  Ultimately, with the large number of 
117 
subjects available for analysis (~10,000) and the bounded distribution of PDC (between 0 and 
118 
1), central limit theorem arguments make standard linear modeling valid in this situation (e.g. 
119 
Lumley et al., 2002), and robust inference methods described below provide additional 
120 
assurance of valid inference even with heteroscedastic variance in the outcomes. 
121 
Missing and truncated follow-up:  Missing data and truncated follow-up for PDC have 
122 
occurred for several reasons.  Patients with inpatient stays have medications supplied by the 
123 
hospital and thus don’t deplete their own supplies, so such days are omitted from both 
124 
numerator and denominator of PDC.  Other situations resulted in early termination of PDC 
125 
calculation.  Prescribed medications are sometimes terminated by providers, or patients may die 
126 
during follow-up.  Additionally, initial study procedures did not allow for collection of outcome 
127 
data on patients who opted out, and while this was fixed early in the study a small percentage of 
128 
early patients have shorter PDC observation.  Finally, treatment delivery was disrupted by two 
129 
external situations specific to health care systems, one involving a new VA messaging system 
130 
that would confound delivery of Nudge treatment messages, and one involving gap identification 
131 
for patients at UCHealth.  In both cases, when the issue occurred our enrollment targets had 
132 
been surpassed at each system.  Delivery of Nudge treatments was stopped and collection of 
133 
daily PDC outcome data was truncated.  The disruptions due to these two exogenous events 
134 
resulted in data that can be assumed missing completely at random (MCAR), however this is 
135 
likely not true for the other cases of early termination of PDC observation.  Further examination 
136 
of the treatment-blinded outcome data showed an increasing trend over time from time zero in 
137 
PDC during periods shortly after randomization, due to the initial gaps.  This together with early 
138 
termination could result in biased estimation. 
139 
Longitudinal models and GEE:  To account for the situations of shortened PDC observation 
140 
and possible resulting biases we will use a longitudinal model with PDC calculated for monthly 
141 
intervals.  We will use GEE with identity link and independence with unequal variances for the 
142 
covariance structure.  This approach provides robust and likely conservative (Hernán et al., 
143 
2002) estimation using empirical (sandwich) variance estimates, but is valid only for MCAR 
144 
data.  However, use of weighting can extend GEE to data missing at random (MAR).  In this 
145 
approach, observation-specific weights are equal to the inverse probability that the longitudinal 
146 
value was observed.  Probability of observation will be estimated by a logistic regression model 
147 
for whether the value was observed (y/n), with covariates listed in section 3.1.c below.  Weights 
148 
greater than the 95th percentile of weights will be set to the 95th percentile weight.  These 
149 
methods can be implemented for example in SAS PROC GEE (Lin & Rodriguez, 2015) or R 
150 
package wgeesel (Xu et al., 2018).  We will use multiple imputation to impute missing 
151 
covariate data.  We will carry out the recommended sensitivity analyses to the MAR assumption 
152 
using methods based on pattern mixture models and imputation, by assuming a range of 
153 

perturbations of imputed values and assessing differences in model conclusions (e.g. White et 
154 
al., 2011; Fiero et al., 2017). 
155 
Treatment comparisons:  Discussions with clinicians have indicated expressing results on a 
156 
linear scale (PDC differences), as opposed to odds ratios or risk ratios, will be most appropriate 
157 
for interpretation.  This occurs naturally with the linear specification in 3.1.a and identity link in 
158 
GEE.  Estimation and inference will be carried out using the parameter estimates from the 
159 
robust GEE estimation to construct expected PDC treatment differences.  The estimand 
160 
comparisons will be 12-month PDC, calculated by summing the 12 monthly longitudinal 
161 
parameters, which incorporate the adjustments for early termination of some subjects using the 
162 
weighted GEE approach.  Primary hypotheses involve pairwise comparisons between each of 
163 
the four study arms, and will be conducted using a multistage gatekeeper approach to account 
164 
for the multiple treatment comparisons. In stage 1 of this approach, each of the three active 
165 
intervention arms is compared with the control arm using significance level 0.05/3. In stage 2, if 
166 
any of the three stage 1 tests is significant, the three pairwise comparisons among active 
167 
intervention arms are tested with the Holm method using significance level (R/3)*(0.05/3), where 
168 
R is the number of stage 1 tests that were significant (Dmetrienko et al., 2008).  
169 
 
170 
3.1.b Defining composite PDC for multiple medications 
171 
There are 13 CV medication classes considered in this study, and the primary outcome is 
172 
composite PDC across medications a patient is prescribed.  There are several considerations in 
173 
defining composite PDC, including which of a patient’s medications to include in the composite 
174 
PDC calculation, and calculation of composite PDC with differing lengths of follow-up for 
175 
different medications. 
176 
Medications to include in composite PDC:  We have considered three ways of selecting 
177 
medications to include in the primary PDC outcome: PDC1) all medications on which a patient 
178 
gapped at baseline, PDC2) all medications a patient ever gaps on, calculating PDC from the 
179 
time of gap, and PDC3) all medications a patient was prescribed at baseline.  Medications 
180 
prescribed after the baseline gap and enrollment will not be included in any of these definitions 
181 
of PDC, though they will be considered in secondary analyses.  Each definition (1-3) has 
182 
benefits and shortcomings.  PDC1 is closest to a pure effect of the intervention, but exploratory 
183 
analyses of pilot data have shown that many patients who gap in a medication and are 
184 
randomized also gap in another medication shortly thereafter.  Specifically, preliminary data 
185 
indicate that ~16% of patients gap on an additional medication within 30 days of baseline and 
186 
~37% within 90 days.  PDC1 will omit these other medications prescribed but not gapped on at 
187 
baseline. PDC2 provides a similar estimate incorporating these later-gapped medications but 
188 
excludes medications the patient may never gap on due to reminders for other medications.  
189 
PDC3 provides an estimate of the overall effect of the interventions on medication patterns, 
190 
incorporating indirect effects of reminders, but risks inflating PDC and diluting intervention 
191 
effects by including medications the patient never gaps on.  Our primary analyses will use 
192 
PDC1, while PDC2 and PDC3 will be considered in secondary analyses. 
193 
Composite PDC with differing lengths of follow-up:  In general, PDC is the sum of 
194 
observation days covered divided by the sum of observation days.  Due to variations from the 
195 
planned 365 days of assessment, there are several ways of calculating composite PDC for 
196 
multiple medications.  Information on hospitalizations, and on medication changes and 
197 
cancellations, will be available from the patient’s electronic health record. Note that at-risk days 
198 
is medication specific, as medications may change during the one-year assessment period.  Our 
199 
primary PDC calculation will be PDC-C1 = (sum of numerators of medication-specific PDCs) / 
200 
(sum of denominators of medication-specific PDCs).  This can also be viewed as a weighted 
201 
sum of medication-specific PDCs with weights equal to the proportion of the 365 target days for 
202 
which PDC was observed, and equally weights each day on each medication.  PDC-C1 will be 
203 
the outcome definition in the longitudinal models described in section 3.1.a.  As a sensitivity 
204 

analysis we will also consider an alternate definition, PDC-C2 = average of medication-specific 
205 
PDCs, which equally weights medications regardless of their length of observation.  When all 
206 
medications are observed for 365 days these definitions are equal.  For the longitudinal 
207 
analyses planned, these definitions will be applied within each longitudinal interval.  We will 
208 
carry out the recommended sensitivity analyses based on pattern mixture models, by assuming 
209 
various values for difference in means between observed and unobserved data and assessing 
210 
differences in model conclusions. 
211 
 
212 
3.1.c Adjustment/propensity covariates 
213 
The linear weighted GEE model for PDC and the logistic model for propensity of observation will 
214 
each contain terms for randomization stratification variables health care system (VA, Denver 
215 
Health or CUHealth), number of CV medications prescribed at baseline (1-2 or 3+), calendar 
216 
month of randomization; treatment arm, follow-up month from randomization in longitudinal 
217 
analysis, and treatment by follow-up month interaction; patient demographics age, race, 
218 
ethnicity, insurance status, and marital status; and comorbidity variables hypertension, 
219 
hyperlipidemia, coronary artery disease, diabetes, atrial fibrillation, chronic heart failure, chronic 
220 
kidney disease, cerebrovascular disease, prior myocardial infarction, prior revascularization, 
221 
depression, PTSD, and substance abuse. 
222 
 
223 
3.1.d Alternate measures of medication adherence 
224 
We will consider several alternate analyses for medication adherence.   
225 
Alternate definitions of PDC:  Alternate definitions PDC2 and PDC3 and alternate composite 
226 
PDC-C2 defined above will be analyzed using the same methods as for the primary definition 
227 
PDC1-C1.  These analyses will provide a better understanding of indirect effects of treatments, 
228 
and of effects of treatments on all medications a patient is prescribed.   
229 
Sensitivity to inactive medications:  Among medications gapping at baseline, we will repeat 
230 
the primary analysis limiting our analyses to those medications that had at least one fill during 
231 
the follow-up period.  This will allow us to assess treatment effects among medications that are 
232 
known to be active, removing medications that were cancelled but not indicated as such by the 
233 
medical record. 
234 
Medication-specific adherence:  We will calculate and analyze PDC for each medication 
235 
individually, to allow analyses of heterogeneity of treatment effect (HTE) by drug class.  Multiple 
236 
comparisons will be adjusted for separately for each analysis using the gatekeeper approach 
237 
described above. 
238 
Medication gaps:  We will consider alternate ways of describing medication adherence 
239 
behavior, including length of initial gap, time to subsequent gap, average length of gaps, and 
240 
number of 7-day gaps. 
241 
 
242 
3.1.e Secondary examinations of medication adherence 
243 
In secondary analyses we will examine several other questions related to medication 
244 
adherence, including: 
245 
Predictors of medication gaps:  We will examine predictors of patients having an initial gap 
246 
of at least 7 days, using the initial pool of candidate patients, to identify types of patients or 
247 
prescription characteristics for patients at highest risk for non-adherence.  We will also examine 
248 
predictors of subjects having a gap of at least 7 days while enrolled in the study. 
249 
Mechanism of treatment effects:  We will carry out analyses examining mechanisms or 
250 
mediators of treatment effect by considering direct responses to reminders, including time from 
251 
first reminder to refill, and measures of patient engagement, e.g. number of patient text 
252 
responses to reminders (intensive text and chatbot arms only). 
253 
Heterogeneity of treatment effect:  We will carry out analyses examining HTE, specifically i) 
254 
HTE between the three health care systems, and ii) HTE for patient characteristics or subgroups 
255 

of particular interest. HTE analyses will be carried out using interactions as recommended in 
256 
Kent et al., 2010 23 and will be considered exploratory. 
257 
 
258 
3.1.f Original power analysis 
259 
We first estimate sample size required to achieve 80% power for the desired change in the 
260 
primary outcome, then used current data to show we will be able to achieve this sample size. 
261 
Power and sample size: Required sample size was estimated for the primary outcome PDC 
262 
during the 12 months following randomization. Preliminary data from the VA were used for these 
263 
estimates. We made the following assumptions:  a) Significance using two-sided level 0.05 
264 
tests, b) Power at least 80%; c) Difference between treatments in PDC of 10 percentage points; 
265 
d) Bonferroni adjustment for the 6 pairwise comparisons among the 4 study arms, resulting in 
266 
adjusted level 0.05/6 (a conservative alternative of the sequential gatekeeper approach to be 
267 
used in the final analysis as described above, to simplify the calculation of power); e) Analysis 
268 
stratified by health care system; and f) Within-system and within-treatment residual standard 
269 
deviation of 12 month PDC equal to 0.22 (mean 0.732), obtained by analysis of 2,859 veterans 
270 
during the period 01/01/2017 – 12/31/2017 who were prescribed the medications of interest in 
271 
this study. The large SD is due to a highly left-skewed PDC distribution.  (We base sample size 
272 
estimation on t-tests rather than the proposed binomial models since we do not have estimates 
273 
of quantities needed to carry out power simulations. We may be able to obtain data for the PDC 
274 
outcome for a set of baseline patients, in which case we will consider a refined power analysis.)  
275 
Using these assumptions and estimates, and comparing any two treatments using a linear 
276 
model with the above residual standard deviation of PDC, we estimate using sample size 
277 
functions in R that we will need N=119 subjects per treatment arm, total across the three health 
278 
care systems, for a total of 476 subjects to be randomized across the three health care systems. 
279 
Available sample sizes:  We obtained data from each of the three health care systems on 
280 
the number of patients at the seventeen specific clinics (VA, 4 clinics; UCHealth, 5 clinics; DH 8 
281 
clinics) to be included in this study. Figure 1, which we believe to present conservative 
282 
estimates of enrollment, shows estimated numbers of patients with CVD conditions and 
283 
prescribed CVD medications across the 3 HCS. Patients will be sent a letter with the opportunity 
284 
to opt-out. In addition, care providers will be provided lists of their patients who are potentially 
285 
eligible for the study to see if there are patients that they feel should not be included in the 
286 
study. Assuming that 75% of patients have a gap, another 15% of patients opt-out of the study 
287 
following randomization, and 10% of patients do not have usable outcome data, we expect to 
288 
have usable outcome data for about 7,740 patients across the four study arms.  
289 
Planned enrollment:  In the patient accrual proposal sent to NHLBI (March 20, 2019), we 
290 
proposed to enroll 5,000 patients which is a conservative estimate in case the number of 
291 
patients opting out is higher than in the pilot or the number of patients with 7-day gaps is lower 
292 
than estimated. Even with this conservative estimate, we expect to have ample subjects (nearly 
293 
ten times as many as needed) to achieve the necessary power for the primary analysis of PDC. 
294 
Additional subjects will provide power for secondary analyses, and for analyses of secondary 
295 
outcomes. 
296 
Achieved sample sizes 3/2023 
297 
We surpassed our overall enrollment goal of 5,000 patients, enrolling 1,235 patients the VA, 
298 
7,266 patients at Denver Health, and 1,000 patients at UCHealth. 
299 
 
300 
3.2 Analysis of secondary outcomes 
301 
 
302 
The secondary outcomes of clinical events, care utilization and cost will be analyzed using 
303 
similar approaches as for the primary outcome but based on appropriate models, e.g. Cox 
304 
survival models for time to clinical event or rehospitalization, generalized gamma regression for 
305 
cost, etc. Standardization methods allow results to be expressed on interpretable scales such 
306 

as risk difference (e.g. Sjolander, 2016).22  Data will be analyzed using SAS (SAS Institute Inc., 
307 
Cary, NC) and R software. In the table below, we provide an estimate of the secondary 
308 
outcomes of interest based on a cohort of patients who would have been potentially eligible for 
309 
the study from 2 of our health systems. This illustrates that we are able to capture these 
310 
outcomes. 
311 
 
312 
Outcomes of interest for the pragmatic trial. This is based on the cohort of patients identified from 2017-2018 
who would have been eligible for the study and we followed them over time to assess outcomes 
Outcome 
Patient population based on the 
presence of specific comorbidity for 
which the outcome is relevant  
Outcomes of 
interest (DH) 
Outcomes of 
interest (VA) 
Systolic BP - Mean (SD) mm Hg 
All 
131.2 (17.9) 
133.8 (18.6) 
Diastolic BP - Mean (SD) mm Hg 
All 
78.7 (10.8) 
79.4 (10.3) 
LDL - Mean (SD) 
All 
85.5 (38.6) 
90.1 (33.1) 
Hemoglobin A1c - Mean (SD) 
All 
7.8 (1.9) 
7.0 (1.6) 
All Cause Hospitalization (1 Yr.) 
All 
8.7% (792/9149) 
13.6% (332/2447) 
     Cause Specific Hospitalization 
 
% (number of 
patients with 
events/number of 
eligible patients for 
the outcome) 
% (number of 
patients with 
events/number of 
eligible patients for 
that outcomes) 
     Hypertension Emergency  
Hypertension 
0.2% (14/7364) 
0.4% (8/1877) 
     Myocardial infarction (MI) 
HTN/Hyperlipidemia/Diab/CAD 
0.2% (22/9119) 
0.3% (7/2402) 
     Stroke 
HTN/Hyperlipidemia/Diab/CAD/AF 
0.1% (12/9149) 
0.2% (4/2447) 
     Heart Failure  
HTN/CAD/AF 
0. 8% (58/7546) 
1.1% (23/2052) 
     Hyperglycemia  
Diabetes 
0.1% (5/5122) 
0.1% (1/1048) 
     Atrial fibrillation (AF) 
AF 
2.3% (8/352) 
2.9% (9/311) 
All Cause ED Visit (1 Yr) 
All 
1.9% (1700/9149) 
4.5% (1093/2447) 
Cause Specific ED Visit 
 
 
 
     Hypertension Emergency 
Hypertension 
0 (1/7364) 
0.2% (3/1877) 
     MI 
HTN/Hyperlipidemia/Diab/CAD 
0 (0/9119) 
0.1% (3/2402) 
     Stroke 
HTN/Hyperlipidemia/Diab/CAD/AF 
0 (0/9149) 
0.1% (2/2447) 
     Heart Failure 
HTN/CAD/AF 
0 (3/7546) 
1.2% (25/2052) 
     Hyperglycemia 
Diabetes 
0.9% (48/5122) 
2.1% (22/1048) 
     AF 
AF 
1.7% (6/352) 
5.8% (18/311) 
Procedures 
 
 
 
     PCI 
HTN/Hyperlipidemia/Diab/CAD 
0.5% (45/9119) 
0.8% (20/2402) 
     CABG 
HTN/Hyperlipidemia/Diab/CAD 
0 (0/9119) 
0.2% (4/2402) 
     Cardioversion 
AF 
2.6% (9/352) 
1.6% (5/311) 
 
313 
 
314 
References 
315 
Lumley T, Diehr P, Emerson S, Chen L. (2002) The importance of the normality assumption in large 
316 
public health data sets. Annual Review of Public Health 23:151-169. 
317 
 
318 
Bell ML, Horton NJ, Dhillon HM, Bray VJ, Vardy J. (2018) Using generalized estimating equations and 
319 
extensions in randomized trials with missing longitudinal patient reported outcome data. Psycho-
320 
Oncology, 27:2125-2131. 
321 
 
322 
Dmitrienko A, Tamhane AC, Wiens BL. (2008) General multistage gatekeeping procedures.  Biometrical 
323 
Journal 50: 667-677. 
324 
 
325 
Fiero MH, Hsu C-H, Bell ML. (2017) A pattern-mixture model approach for handling missing continuous 
326 
outcome data in longitudinal cluster randomized trials.  Statistics in Medicine 36:4094-4105. 
327 

 
328 
White IR, Horton NJ, Carpenter J, Pocock SJ. (2011) Strategy for intention to treat analysis in randomized 
329 
trials with missing outcome data.  British Medical Journal 342:d40. 
330 
 
331 
Hernán, M. A., Brumback, B. A., & Robins, J. M. (2002). Estimating the causal effect of zidovudine on 
332 
CD4 count with a marginal structural model for repeated measures. Statistics in medicine, 21(12), 1689-
333 
1709. 
334 
 
335 
Preisser, J.S., Lohman, K.K. and Rathouz, P.J., 2002. Performance of weighted estimating equations for 
336 
longitudinal binary data with drop-outs missing at random. Statistics in medicine, 21(20), pp.3035-3054. 
337 
 
338 
Robins, J.M., Rotnitzky, A. and Zhao, L.P., 1995. Analysis of semiparametric regression models for 
339 
repeated outcomes in the presence of missing data. Journal of the American Statistical Association, 
340 
90(429), pp.106-121. 
341 
 
342 
Lin G & Rodriguez RN. (2015) Weighted methods for analyzing missing data with the GEE procedure.  
343 
SAS paper 166-2015, https://support.sas.com/resources/papers/proceedings14/SAS166-2014.pdf. 
344 
 
345 
Xu C, Li Z, Wang M (2018). wgeesel: Weighted Generalized Estimating Equations and Model Selection. 
346 
R package version 1.5, https://CRAN.R-project.org/package=wgeesel. 
347 
 
348 
 
349