Flu2Text: A Multi-Site Study Assessing Text Message Reminders for 2nd Dose of Influenza Vaccine
Phase: NA
Conditions: Influenza, Vaccination
Sponsor: Columbia University

 
 
 
 
 
 
 
 
 
 
 
Official title:  Flu2Text: Text Message Reminders for 2nd Dose of Influenza Vaccine 
 
NCT number: NCT03287830 
 
SAP Document date: September 12, 2019 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
Flu2Text: A Multi-Site Study assessing an Intervention for 2nd Dose of Influenza Vaccine 
 
Overview of the study 
This  study will assess the impact of text message reminders on the receipt of the second dose of influenza 
vaccine and will take place in 40-50 practices over two influenza seasons. 
 
This study includes a sample of up to 4,000 parents over the course of two flu seasons (up to approximately 40-
250 parents any of 40-50 sites per influenza season). Eligible parents will have a child who received their first 
vaccine dose and are in need of 2 doses that season. The study will include parents who have a 6-month 
through 8-year-old child (or a child who is of the age that necessitates 2 doses, should age recommendations 
change as determined by the CDC) who received their first vaccine dose and is in need of 2 doses that season. 
Current recommendations require 2 doses for certain children 6 months through 8 years old. 
 
Within each practice site, parents will be randomized to receive either: 
1) a series of text message reminders embedded with influenza vaccine health-literacy promoting information, or 
2) usual care 
• 
Season 1 (2017-2018): usual care alone 
• 
Season 2 (2018-2019): usual care plus a text message containing a link to health and parenting information 
from the American Academy of Pediatrics website  
 
The primary outcome will be receipt of two doses of influenza vaccine by the end of the season (April 30). A 
secondary outcome will be timeliness of receipt of 2nd dose. 
 
The study will be conducted over the 1st season (September 2017–April 2018) as an effectiveness trial and the 
2nd season (July 2018–April 2019) as a replication study to validate the approach and allow refinement based on 
parent and practice feedback. The intervention targets the parent-child dyad. 
 
 
Background 
Influenza, as evidenced by seasonal epidemics, remains a significant and largely preventable source of morbidity 
and mortality resulting in an estimated 31 million outpatient visits, 226,000 hospitalizations, and 3,000 to 49,000 
deaths annually (depending on the virulence of the circulating strain), along with a high cost burden from direct 
medical expenses and days lost from work.1-3 Young children, the focus of this proposal, are at especially high 
risk for hospitalization and serious influenza complications including pneumonia and neurological outcomes like 
seizures and encephalopathy. They are also a source of transmission to other high-risk household members.5-12 
However, influenza vaccine coverage is low.4 Only 58.9% of U.S. children 6 months to 17 years-old receive at 
least one needed influenza dose annually.4 Young children <9 years are at particular risk for under vaccination 
because they require 2 doses at least 4 weeks apart in the first influenza season they are vaccinated, since one 
dose does not confer adequate influenza protection12-19; some previously vaccinated children also require 2 
doses in subsequent seasons depending on their previous influenza vaccinations.12-15 Even among those who 
need two doses in a given season and who receive the first dose, only 40-60% receive the second dose.20-22 
 
Timeliness is also key, as children are not fully protected until two weeks after receiving the second dose;12, 24, 25 
many children do not receive this second dose close to 4 weeks after the first, as recommended.20 This delay 
leaves many children unprotected when influenza virus begins circulating.26 Increasing timely receipt of second 
doses is also a worthwhile focus as parents (meaning the adult in a parenting role, including guardians) have 
already “accepted” vaccination as evidenced by their child receiving the first dose. 
 
 

 
 
 
References 
1. 
Centers for Disease Control and Prevention. Estimates of deaths associated with seasonal influenza --- United 
States, 1976-2007. MMWR Morb Mortal Wkly Rep. Aug 27 2010;59(33):1057-1062. 
2. 
Thompson WW, Shay DK, Weintraub E, Brammer L, Bridges CB, Cox NJ, Fukuda K. Influenza-associated 
hospitalizations in the United States. JAMA. Sep 15 2004;292(11):1333-1340. 
3. 
Molinari NA, Ortega-Sanchez IR, Messonnier ML, Thompson WW, Wortley PM, Weintraub E, Bridges CB. The 
annual impact of seasonal influenza in the US: measuring disease burden and costs. Vaccine. Jun 28 
2007;25(27):5086-5096. 
4. 
Centers for Disease Control and Prevention. Seasonal Influenza (Flu) FluVaxView Influenza Vaccination 
Coverage. Available at http://www.cdc.gov/flu/fluvaxview/. Accessed on January 31, 2015. 
5. 
 Louie JK, Schechter R, Honarmand S, Guevara HF, Shoemaker TR, Madrigal NY, Woodfill CJ, Backer HD, Glaser 
CA. Severe pediatric influenza in California, 2003-2005: implications for immunization recommendations. 
Pediatrics. Apr 2006;117(4):e610-618. 
6. 
Neuzil KM, Zhu Y, Griffin MR, Edwards KM, Thompson JM, Tollefson SJ, Wright PF. Burden of interpandemic 
influenza in children younger than 5 years: a 25-year prospective study. J Infect Dis. Jan 15 2002;185(2):147-
152. 
7. 
Neuzil KM, Mellen BG, Wright PF, Mitchel EF, Jr., Griffin MR. The effect of influenza on hospitalizations, 
outpatient visits, and courses of antibiotics in children. N Engl J Med. Jan 27 2000;342(4):225-231. 
8. 
Jules A, Grijalva CG, Zhu Y, Talbot HK, Williams JV, Poehling KA, Chaves SS, Edwards KM, Schaffner W, Shay DK, 
Griffin MR. Influenza-Related Hospitalization and ED Visits in Children Less Than 5 Years: 2000-2011. Pediatrics. 
Jan 2015;135(1):e66-74. 
9. 
Zhou H, Thompson WW, Viboud CG, Ringholz CM, Cheng PY, Steiner C, Abedi GR, Anderson LJ, Brammer L, 
Shay DK. Hospitalizations associated with influenza and respiratory syncytial virus in the United States, 1993-
2008. Clin Infect Dis. May 2012;54(10):1427-1436. 
10. Newland JG, Laurich VM, Rosenquist AW, Heydon K, Licht DJ, Keren R, Zaoutis TE, Watson B, Hodinka RL, Coffin 
SE. Neurologic complications in children hospitalized with influenza: characteristics, incidence, and risk factors. 
J Pediatr. Mar 2007;150(3):306-310. 
11. Mistry RD, Fischer JB, Prasad PA, Coffin SE, Alpern ER. Severe complications in influenza-like illnesses. 
Pediatrics. Sep 2014;134(3):e684-690. 
12. Fiore AE, Uyeki TM, Broder K, Finelli L, Euler GL, Singleton JA, Iskander JK, Wortley PM, Shay DK, Bresee JS, Cox 
NJ. Prevention and control of influenza with vaccines: recommendations of the Advisory Committee on 
Immunization Practices (ACIP), 2010. MMWR Recomm Rep. Aug 6 2010;59(RR-8):1-62. 
13. Neuzil KM, Jackson LA, Nelson J, Klimov A, Cox N, Bridges CB, Dunn J, DeStefano F, Shay D. Immunogenicity and 
reactogenicity of 1 versus 2 doses of trivalent inactivated influenza vaccine in vaccine-naive 5-8-year-old 
children. J Infect Dis. Oct 15 2006;194(8):1032-1039. 
14. Grohskopf LA, Olsen SJ, Sokolow LZ, Bresee JS, Cox NJ, Broder KR, Karron RA, Walter EB, Centers for Disease 
Control Prevention. Prevention and control of seasonal influenza with vaccines: recommendations of the 
Advisory Committee on Immunization Practices (ACIP) -- United States, 2014-15 influenza season. MMWR 
Morb Mortal Wkly Rep. Aug 15 2014;63(32):691-697. 
15. Grohskopf LA, Sokolow LZ, Olsen SJ, Bresee JS, Broder KR, Karron RA. Prevention and Control of Influenza with 
Vaccines: Recommendations of the Advisory Committee on Immunization Practices, United States, 2015-16 
Influenza Season. MMWR Morb Mortal Wkly Rep. Aug 7 2015;64(30):818-825. 
16. Hofstetter AM, Natarajan K, Martinez RA, Rabinowitz D, Vawdrey DK, Stockwell MS. Influenza vaccination 
coverage and timeliness among children requiring two doses, 2004-2009. Prev Med. Mar 2013;56(3-4):165-
170. 
17. Pabst LJ, Chaves SS, Weinbaum C. Trends in compliance with two-dose influenza vaccine recommendations 
among children aged 6 months through 8 years. Vaccine. Jun 28 2013;31(31):3116-3120. 
18. Bhatt P, Block SL, Toback SL, Ambrose CS. A prospective observational study of US in-office pediatric influenza 
vaccination during the 2007 to 2009 influenza seasons: use and factors associated with increased vaccination 
rates. Clin Pediatr (Phila). Oct 2010;49(10):954-963. 

 
 
 
19. Ritzwoller DP, Bridges CB, Shetterly S, Yamasaki K, Kolczak M, France EK. Effectiveness of the 2003-2004 
influenza vaccine among children 6 months to 8 years of age, with 1 vs 2 doses. Pediatrics. Jul 
2005;116(1):153-159. 
20. Allison MA, Daley MF, Crane LA, Barrow J, Beaty BL, Allred N, Berman S, Kempe A. Influenza vaccine 
effectiveness in healthy 6- to 21-month-old children during the 2003-2004 season. J Pediatr. Dec 
2006;149(6):755-762. 
21. Eisenberg KW, Szilagyi PG, Fairbrother G, Griffin MR, Staat M, Shone LP, Weinberg GA, Hall CB, Poehling KA, 
Edwards KM, Lofthus G, Fisher SG, Bridges CB, Iwane MK, New Vaccine Surveillance Network. Vaccine 
effectiveness against laboratory-confirmed influenza in children 6 to 59 months of age during the 2003-2004 
and 2004-2005 influenza seasons. Pediatrics. Nov 2008;122(5):911-919. 
22. Shuler CM, Iwamoto M, Bridges CB, Marin M, Neeman R, Gargiullo P, Yoder TA, Keyserling HL, Terebuh PD. 
Vaccine effectiveness against medically attended, laboratory-confirmed influenza among children aged 6 to 59 
months, 2003- 2004. Pediatrics. Mar 2007;119(3):e587-595. 
23. Briss PA, Rodewald LE, Hinman AR, Shefer AM, Strikas RA, Bernier RR, Carande-Kulis VG, Yusuf HR, Ndiaye SM, 
Williams SM. Reviews of evidence regarding interventions to improve vaccination coverage in children, 
adolescents, and adults. The Task Force on Community Preventive Services. Am J Prev Med. Jan 2000;18(1 
Suppl):97-140. 
24. Recommendations of the Advisory Committee on Immunization Practices, the American Academy of 
Pediatrics, and the American Academy of Family Physicians: use of reminder and recall by vaccination 
providers to increase vaccination rates. MMWR Morb Mortal Wkly Rep. Sep 4 1998;47(34):715-717. 
25. Szilagyi PG, Bordley C, Vann JC, Chelminski A, Kraus RM, Margolis PA, Rodewald LE. Effect of patient 
reminder/recall interventions on immunization rates: A review. JAMA. Oct 11 2000;284(14):1820-1827. 
26. Stockwell MS, Kharbanda EO, Martinez RA, Lara M, Vawdrey D, Natarajan K, Rickert VI. Text4Health: impact of 
text message reminder-recalls for pediatric and adolescent immunizations. Am J Public Health. Feb 
2012;102(2):e15-21. 
 
Protocol 
This proposed study will assess the impact of personalized text message reminders embedded with influenza 
vaccine health-literacy-promoting information versus usual care on the outcome of receipt of the second dose 
of influenza vaccine in a given influenza season. 
 
Study Aims: 
AIM 1: To compare the effectiveness in increasing receipt of the second dose of influenza vaccine of 
personalized text message reminders embedded with influenza vaccine health-literacy-promoting 
information versus usual care in a diverse pediatric population in need of two doses in a given influenza 
season. 
 
AIM 2: To compare the effectiveness in improving the timeliness of the second dose of influenza vaccine of 
personalized text message reminders embedded with influenza vaccine health-literacy-promoting information 
versus usual care in a diverse pediatric population in need of two doses in a given influenza season. 
 
AIM 3: To assess parental and child characteristics that modify text message effectiveness.  
 
 
 
 
 
 
 

 
 
 
Study procedures 
Practice inclusion criteria: 
• 
Ability to recruit a minimum of 40 English or Spanish speaking parents per season 
• 
Ability to check the office medical records and/or the city/state immunization information system of 
the child of the participating parent to confirm dates of first and second influenza vaccination and related 
visit dates in each season (and ability to provide these dates to the study team) 
 
Practice exclusion criteria: 
• 
Practices that already use or plan to use text message reminders for the second dose of influenza 
vaccine during the study period 
 
Parent inclusion criteria: 
• 
Parenting adult (or legal guardian) of a child that: 
o is between 6 months through 8 years at the time of enrollment (or a child who is of the age 
that necessitates 2 doses, should age recommendations change as determined by the CDC) 
o received 1st dose of influenza vaccine within last 7 days (6 days plus date of vaccination) 
o receives care at study site 
o needs 2 doses of influenza vaccine that season as determined by their provider at the study site. 
• 
Ability to communicate in English or Spanish 
• 
Has a cell phone that has text message capabilities 
 
Parent exclusion criteria: 
• 
Parenting adult of a child that: 
o already enrolled in the study in this or the previous season (see Attachments E.2 & I.2) 
• 
Unable to communicate in English or Spanish 
• 
Unable to receive or read text messages on their cell phone 
 
Below we outline the different procedures planned for the study: 
 
Season One: When parents/children come to the practice to receive an influenza vaccine, a member of 
the care team at the practice site will use the Eligibility Flow Sheet to determine their eligibility for the 
study. If eligible and parent is interested, the care team member will introduce the study to the parent by 
providing a parent study information card and a copy of the parent information sheet, available in either 
English or Spanish. Parents who are potentially interested in the study will be asked to send a text 
message to the study text message platform. Families will be called to be enrolled. Parental verbal 
consent and HIPAA authorization will be obtained via phone by study staff. 
 
Season Two: Parents presenting for visits with age-eligible children up to/including the initial flu 
vaccination visit will receive a study information sheet in either English or Spanish. Providers and practice 
staff will be asked to approach all eligible parents and provide them with the information sheet.  At the 
visit when the child will receive the first flu shot, a provider or member of practice staff will use the 
Eligibility Flow Sheet to determine whether the child needs two flu shots this season and whether the 
child’s parent can communicate in either English or Spanish. This recruiting practice staff member will 
present the Parent Information Sheet to the parent. If the parent is interested and the practice enrolled 
patients in the 2017-18 flu season, the recruiting practice staff member will assess whether the parent 
already participated in the study. If there is more than one eligible child in the family, the recruiting 
practice staff member will enroll the child whose birthday comes next in the calendar year. For example, 

 
 
 
if today is October 1 and one child’s birthday is January 21 and the next child’s birthday is March 4 
(regardless of year) then the child with the January 21 birthday should be selected. If the parent is eligible 
and wishes to participate, verbal consent and HIPAA authorization will be obtained by an HST-certified 
provider or practice staff member (recruiting practice staff member) at the study practice sites. 
 
 
Parent randomization 
 
      Randomization will occur with a 1:1 allocation ratio stratified by: 
• 
clinical site (such that each site will have a balance of intervention and usual care plus participants) 
• 
child’s age group (6-23 months vs. 2-8 years) 
• 
parental language (English versus Spanish preference) 
 
To ensure allocation concealment (a technique used to prevent selection bias), the CHOP/Penn statistician 
will prepare for each site a separate randomization list stratified by age and language, using randomly 
permuted blocks with varying block sizes. 
 
If the parent is assigned to usual care, they will receive the existing influenza vaccination protocol for any 
given practice. This may include other forms of influenza vaccine reminders. In season 2, caregivers 
randomized to usual care will be sent a message upon enrollment containing a link to health and 
parenting information from the American Academy of Pediatrics website. If a parent is assigned to the 
intervention arm, we will follow the protocol under Study Intervention: Text Messages.  
 
Study Intervention: Text Messages 
Messages will be sent in English or Spanish based on parent preference. 
Using date of initial vaccine as day 0, text messages will be sent to all parents randomized to the 
intervention on the following days: 
o Day 14: notifying parents that their child is in need of a second flu shot. 
o Day 21: educational text messages, with interactivity, through which parents can select to 
receive more information via text message. 
o Day 25: letting parents know when and where they will need to go to receive the second shot 
o Day 28 (day vaccine is due): it is now time to take [child] to be vaccinated (2nd flu shot). 
o Day 42 (2 weeks after it is due): another interactive message will assess potential vaccine barriers 
and offer selected responses. 
Texts will stop after day 42. We will not be able to know from the practices in time to stop 
messages after a child was vaccinated. However, as the shot is not due until day 28, it would likely 
only be the day 42 message that could come after the child was vaccinated. That message has 
been worded to first assess whether the child received the vaccine. 
 
Dates of influenza vaccinations and related visit dates: Dates of influenza vaccinations and related visit dates will be 
collected from the practices. The part of the Patient Enrollment/Outcomes Forms that stays with the site will have 
child’s first and last name and date of birth. The site will use this to identify the medical record, extract relevant 
influenza vaccination dates and related visit dates and transfer this information to the research team. Practices will 
do this by using their medical records and/or by checking their city or state immunization registry should the 
practice have no record of receipt of subsequent influenza doses for the current season. 
 

SAP v 1.6 | Sep 12, 2019 
1 
 
 
Statistical Analysis Plan (SAP) – Flu2Text - v1.6 
Date of final SAP: September 12, 2019 
This plan, based on the original proposal and the IRB protocol, will guide analyses. It 
follows the aims outlined in the proposal. In keeping with recent guidance of the 
American Statistical Association, we will avoid statements about “statistical 
significance”. When reported, p-values will be continuous, unless they are below 0.001, 
when we shall use the notation “p<0.001”, in keeping with general guidance on very 
small p-values. We shall report p-values with 2 decimal places if they are larger than 0.10 
and to 3 decimal places otherwise. Point estimates and confidence bounds will be the 
primary means of reporting results and their precision. Primary analyses are reported 
below where indicated. Software tools are also listed, but are subject to modification if 
they fail to converge or produce results. 
 
For overall guidance, we will follow those from New England Journal of Medicine (Last 
accessed, Sep 11, 2019). 
 
We shall date this SAP upon agreement and completion and then date all amendments. 

SAP v 1.6 | Sep 12, 2019 
2 
 
 
1.0 
Data management 
 
We shall examine each covariate to identify ambiguities and to categorize when 
appropriate to avoid zero and small categories. All refinement of covariates will occur 
prior to any examination of outcomes. 
 
Cross classifying X’s will determine which X’s are correlated, aliased, how many levels 
can be collapsed. The goal will be to find a set of covariates for use in describing the 
sample and comparing the two treatment groups for balance. In these analyses, we should 
not be looking at outcome. 
 
2.0 
Analysis (A) Primary analysis: As randomized (Aims 1 and 2). 
 
AIM 1: To compare the effectiveness in increasing receipt of the second dose of 
influenza vaccine of personalized text message reminders with embedded influenza 
vaccine health-literacy-promoting information versus usual care in a diverse pediatric 
population in need of two doses in a given influenza season. 
 
AIM 2: To compare the effectiveness in improving the timeliness of the second dose 
of influenza vaccine of personalized text message reminders with embedded influenza 
vaccine health-literacy-promoting information versus usual care in a diverse pediatric 
population in need of two doses in a given influenza season. 
 
2.0.1 Analyses will proceed separately by season 
 
Seasons 1 and 2 will be analyzed separately. 
 
Secondary analysis will combine seasons, adjusting for season as a stratifying factor. 
Site will be considered to be a separate stratum by season. 
 
2.1 
Analysis – binary outcomes – primary analysis and outcome 
 
Outcome = Receipt of the second dose of influenza vaccine, a binary outcome. 
By April 30, the end of the season - primary 
By Day 42 from the day of first dose - secondary 
 
Formal question to be answered: "If a group of children were alternatively assigned to the 
two treatment arms, what would be differences in completion of influenza vaccines by 
the end of the season, defined as April 30th of each season?" 
 
The alternative and secondary endpoint will evaluate receipt of the second dose as of day 
42 from the date of the child’s first dose. 
 
Randomization, if successful, should lead to exchangeability of two groups and permits 
estimation of the treatment differences (as the control group should be the same as the 
intervention group). 

SAP v 1.6 | Sep 12, 2019 
3 
 
 
Preliminary analysis of time from the first dose to time of randomization showed no 
difference between intervention and control groups, and reflects successful 
randomization. This analysis was conducted as part of assessing balance and without 
regard to outcomes. 
 
Primary analyses will proceed where possible with assumption-free methods. Dropout 
was negligible and so adjustment for dropout was not needed nor specified in the final 
SAP. 
 
2.1.1 Testing for balance from randomization based on patient-level covariates 
 
The initial check will be whether randomization produced balance of the sites and 
covariates. For this check, we will fit a model: 
 
Outcome is treatment assignment. Logistic regression – with all covariates included. 
 
1. Check that overall likelihood ratio statistic is not significant. 
2. Suggests overall balance consistent with successful randomization. 
 
Balance check covariates: 
 
• time from 1st (initial) dose to randomization 
• child age 
• child gender 
• text language 
• child health insurance 
 
2.1.2 Unadjusted association of treatment assignment and outcomes 
 
After checking for balance across patient characteristics, we shall estimate treatment- 
outcome associations assuming binary outcomes 
 
2.1.2.1 Relative risks 
 
Stratified (by site and language) Mantel Haensel 
Estimates = RR and confidence interval 
Make a new variable that is the combination of site and language = l_site 
Stata: cs outcome treatment, by(l_site) 
This is a "fixed effects" analysis and does not consider the effect on variance of variation 
across sites in the effect of treatment (See section 2.4). 

SAP v 1.6 | Sep 12, 2019 
4 
 
 
2.1.2.2 Binary outcomes – risk differences 
 
Risk differences are important measures and should be reported along with 
relative risks. But standard packages do not report risk differences using Mantel Haenszel 
approaches. There are two alternatives that might be indicated. 
 
Alternative #1. Newcombe 
 
SAS: Alternative – Mantel Haenzsel methods for risk difference with Newcombe 
confidence bounds. 
Let "strata" be the combination of site and language. 
PROC FREQ; 
Tables strata*treatment*outcome/riskdiff method=newcombe; 
Run; 
 
Alternative #2. Bohning (Biometrics 2000) - confirmatory 
 
Bohning proposed, and we have programmed in Stata, a Mantel Haenszel 
approach for estimating risk differences. We propose to use this as our primary approach 
and then to use Newcombe as a check on sensitivity. Both methods might be helpful 
because risk differences (stratified) are not as well-accepted as methods for stratified 
relative risk. This will be a secondary analysis only for purposes of confirmation. 
 
Binary outcomes are (as noted above): Second dose status as of April 30 
(primary), and as of day 42 from child’s date of first dose (secondary). 
 
2.1.3 Analysis with covariates (if check on balance fails) 
 
2.1.3.1 Conditional logit models (likely not applicable owing to randomization) 
 
Conditional logistic regression model, adjusted for covariates, stratified by site. 
 
Stata: clogit outcome treatment covariates, group(site) 
 
Compare with the model (covariate unadjusted) such as Mantel Haenszel approaches. 
 
Stata: clogit outcome treatment, group(site) 
 
Do the estimates of the effect of treatment differ between these two models? If 
yes, then some chance imbalance in covariates across treatment groups is imparting some 
bias in the unadjusted estimates. 
 
2.2. 
Continuous outcomes (if any) – not applicable 

SAP v 1.6 | Sep 12, 2019 
5 
 
 
We have none. This section is not developed. 
 
2.3 
Secondary analysis: Time to event (Aim 2) 
 
As a secondary outcome, and to confirm that statistical significance from our 
primary analysis applies equally to time to event data, we will use stratified (by practice 
site) time-to-event analysis for receipt of vaccine. 
Our goal will be to use, where possible, assumption free methods and avoid 
parametric approaches, and at the same time to present results in a format that can be 
interpreted easily to a clinical audience. For these reasons, we are deliberately avoiding 
traditional approaches such as Cox regression and common metrics such as hazard ratios. 
 
This section covers two separate topics: 
 
First, in keeping with our goal of using assumption-free, randomization-test based 
methods, we shall apply this approach to time-to-event analyses. 
 
Second, we shall estimate restricted mean survival times to arrive at more 
meaningful estimate of time to event and to avoid "hazard ratios" 
 
2.3.1 Time zero 
 
Time zero for all time to event analyses is the date of child’s initial vaccine dose. 
 
2.3.2 
Descriptive Kaplan-Meier (KM) curves of time from 1st to 2nd dose 
 
Non parametric Kaplan-Meier curves: 
Stata: "sts graph" 
These will be done with censoring date April 30 (primary) and day 42 (secondary). Both 
analyses will be reported and plotted 
 
We will use inverted KM curves. This graphical approach will be our primary analysis. 
Log rank tests will generate p-values if proportional hazards assumption met. 
Stratification will be by clinic site. 
 
2.3.3 Stratified randomization test based estimates - confirmatory 
 
Randomization-based estimates are now available for stratified data in Stata with the 
program strbee (White 2002). Later versions support stratification. These are iterative, 
grid search based estimates for inverting the test statistic. 
 
In Stata: Must download latest version using “net search strbee” 
The key is the use of "strata()" command below. Then the syntax is: 

SAP v 1.6 | Sep 12, 2019 
6 
 
 
Stset timevar, failure(failure_indicator) 
 
Strbee treat_var, test(logrank) endstudy(time_to_end_study_var) strata(site) 
adjvars(covariates) 
 
 
2.3.4 Risk difference (of completion of vaccine series) as of a given time 
 
The purpose of this analysis is to re-express the results of Kaplan Meier survival 
estimates to generate more interpretable results. It is therefore an alternative method to 
express the results and will not be considered as a multiple test in need of adjusting 
confidence intervals. This analysis is not separate from the Kaplan Meier analysis. Rather 
it only quantifies the values of the two curves (y) at a given value of time (e.g. 42 days). 
This is therefore, just a re-expression of the Kaplan Meier plot. 
 
Use of pseudo-observation approach with syntax: 
Stpsurv, at(time1) failure after(time0), 
Where time(1) can be as of day 42, 60, and 90. 
 
As April 30 does not account for different days of follow up per child, it will not 
be possible to use that date for analysis. Programming details appear in Stata program 
“stpm2” (Lambert, Andersson, Royston). 
 
Syntax is (approximately): 
 
standsurv, at1(treatment 0) at2(treatment 1) timevar(timevar) ci 
contrast(difference) 
 
See stpm2 syntax for “standsurv” 
“ci” refers to cumulative incidence 
“timevar” needs to take on the value 42, 60 and 90 
 
 
2.3.5 
Restricted mean survival curves (RMST) 
 
RMST will not be a primary analysis, and we will not consider it to be a separate 
look at the data. Rather, it is another method of reporting the data, and we will not adjust 
confidence bounds for multiple comparisons. 
 
We will implement restricted mean survival time analysis to compare intervention 
versus usual care groups (if needed, after adjusting for covariates that are not balanced 

SAP v 1.6 | Sep 12, 2019 
7 
 
 
between groups) and to arrive at meaningful estimates that do not require proportional 
hazards assumptions. 
We have moved beyond hazard ratios because of the assumptions underlying Cox 
models and the shortcoming of interpretations (Hernan 2010). If we set a truncation time 
(Day 42), then RMST has the interpretation "if we follow children for 42 days, they will 
receive the second vaccine on average by day X" (Royston 2011, 2013). Estimation of 
between group differences are also possible. The more theoretical interpretation is the 
area under the survivor function until the time of truncation. 
Preiminary analysis of available time per child (analysis blinded to outcome and 
to the treatment assignment) showed that between 84% and 90% (depending on season) 
of children has observation windows that extended to day 90 from date of first dose. 
 
2.3.5.1 RMST 
 
Stata: strmst2: 
 
RMST and curves are estimable using strmst2 (Cronin 2016) in Stata 
Will perform conventional RMST analysis: 
Requires “stset” as preliminary command. 
 
Strmst2 treat, tau(50) (will not allow factors), where "tau" is the largest time of 
observation and should be the minimum (by group) of the maximum follow up time 
(within group). 
 
Observation time starts at the time of the first dose of vaccine and ends on April 
30, of the season year or when the child received the second does, whichever was earlier. 
 
For this study, we pre-specify several values of tau: 
 
(i) 42 – the number of days within which children should receive the second dose 
(ii) 60 days 
(iii) 90 days 
 
Adjustment for covariates using strmst2pw 
Robust confidence intervals. 
This approach has no support for robust confidence intervals (or for strata). For 
that reason, we will check all confidence bounds by using bootstrap percentile methods, 
with 999 resamples at the site level (code will be developed for Stata). 
 
Note: Alternative RMST code available from Royston (2015) using Stata’s code 
strmst. This will be a backup in the event that Cronin’s code fails. 

SAP v 1.6 | Sep 12, 2019 
8 
 
 
2.4 
Tertiary analysis: Variation across practice sites - descriptive 
 
We anticipate that the effect of the intervention could vary across practice sites. 
 
2.4.1 Binary outcomes 
 
2.4.1.1 Mixed effects models 
 
To assess this potential for variation in effect across the intervention sites, we will 
consider generalized linear mixed effects models (logistic regression with random 
intercepts and slopes) to mimic the analysis that would occur in a meta-analysis in which 
variation across sites is expected. The advantage of this approach is that once the random 
slopes (treatment effects estimating the departure of each site from average) are 
estimated, we can identify the sites that have either high or low treatment effects. 
Individual site estimates are shrunk toward the overall average in these methods to avoid 
problems of unstable site-specific estimates and to reduce the problem of multiple 
comparisons. This method will, however, allow us to identify the sites with the greatest 
and least improvements. Once identified, we will attempt to characterize these sites and 
contact them to discover the possible barriers to improvement. 
 
If mixed effects models reveal no varation of intervention effect across sites, when that 
result will reported and this analysis will stop. 
 
Generalized linear mixed effects models. Stata melogit for binary outcomes: 
 
Random intercept (and slopes) for each site (assuming correlated intercept/slope) 
Overall test for variation across sites in treatment effect 
Likelihood ratio test of random slopes 
 
2.4.2 Time to event outcomes – variation across sites 
 
We will use the same approach for time to vaccination, with mixed effects models and 
letting site be a random effect. In other words, we allow estimated time to vaccination 
vary by site, and test whether this variation is significant. 
 
Statistical significance of the random site effect can be testing using likelihood ratio 
statistics. 
 
Stata: mestreg – 
 
mestreg treat covariates || site:, distribution(weibull) int(12) 
predict resite, reffects 

SAP v 1.6 | Sep 12, 2019 
9 
 
 
In this example, the variable "resite" will contain the value for the "prediction" at the site 
level. The prediction will then represent the site-level time to event, and by ordering the 
sites by this value, we can determine which sites fared best and which fared worst. 
 
R:  coxme 
 
Alternative but similar approaches are now in R (Therneau 2018) for the Cox model 
 
3.0 
AIM 3: To assess parental and child characteristics that modify text message 
effectiveness 
 
The goal for this Aim is to identify subgroups for which the intervention was either 
particularly effective or not effective. All subgroups should be pre-specified (see 
candidates below). They should also be few, owing to issues of multiple comparisons. 
For that reason, all confidence bounds of subgroup estimates will be adjusted so that 
bounds correspond to Bonferroni corrections for multiple contrasts. This aim is 
exploratory. 
 
3.0.1 Candidate effect modifiers 
 
The following factors represent the original candidate effect modifiers. 
 
3.0.1.1 Available for all children 
 
• child age-group (6-23 months; 2-8 years) 
• child gender (male; female) 
• child health insurance (public/unisured; commercial (including Tricare)). This 
variable is used as a partial indicator of socioeconomic status. 
• seasonality of the initial dose (early influenza season, mid-season, late season) 
 
3.0.1.2 Available for all caregivers 
 
• text language (English; Spanish) 
 
3.0.1.2 Available for all practice sites 
 
• appoitment scheduling for the 2nd dose (walk-in; appointment) 
• primary practice setting/urbanicity (urban, inner city; urban, not inner city; 
suburban; rural). Practice setting is self-reported and the categories align with the 
Periodic Survey from the American Academy of Pediatrics. 
 
3.0.1.3 Available for only questionnaire respondents 
 
Caregiver: 
 
• age 

SAP v 1.6 | Sep 12, 2019 
10 
 
 
• highest education level 
• relationship to a child 
• receipt of previous text message reminders from their practice site/doctor 
• type of text message plan 
• text messaging frequency 
• overall childhood vaccine hesitancy 
• flu shot safety 
• perceived risk of getting sick from the flu 
 
child: 
• reported health status 
• race/ethnicity (including sub-group) 
 
The first challenge is to try to identify potential subgroups based on groups of these 
individual factors without making the error of estimating each candidate factor one by 
one and then looking for “interaction” of treatment by factor. 
 
3.0.2 Effect modification on the additive scale 
 
We define effect modification on the additive scale for the effects of treatment on 
outcome. For binary outcomes, effect modification therefore means that the effect of 
treatment is in terms of “percentage points” of improvement, and effect modification is 
present if improvement (in terms of percentage points) from the intervention differs by 
the effect modifier grouping. 
 
3.0.3 Latent class analysis of patient-level factors - exploratory analysis 
 
Latent class analysis can reduce many patient level factors into a single dimension factor. 
(Lanza 2013). With our candidate factors distilled into a categorical latent class, we 
propose to simplify the process of identifying and reporting potential effect modification. 
These analyses will be exploratory 
 
3.1 Subgroup analyses – binary outcomes – primary analysis 
 
We will pre-specify subgroups of interest based on combinations of patient 
characteristics. Risk differences, rather than relative risks, will be the scale on which to 
estimate treatment*subgroup interactions, so that final results we can report in terms of 
percentage points of improvement. 
 
3.1.1 Estimating within site effects of subgroups 
 
The within-site effect modification estimate represents the difference in treatment effects 
across levels of the effect modifier contrasted within (rather than across) sites. For this 
estimate we shall first use a linear fixed effects model, as implemented in Stata command 
“xtreg,fe”, assuming that the outcome will lie in the probability range of 0.2 and 0.8. 

SAP v 1.6 | Sep 12, 2019 
11 
 
 
This approach will be a first estimate to be followed by the decomposition approach in 
section 3.1.2. 
 
3.1.2 Parametric subgroup analysis – decomposing the within and across site 
effects 
 
We assume that absolute rates (risks) of second vaccine will not lie at the ends of the 
interval [0,1]. If that is the case, then we can estimate risk differences with subgroups and 
then across subgroups in the following 3-step process. 
 
Step 1: Using a generalized linear effects, create a site-level variable that equals 
the mean rate of the group at the site. If at one site, for example, 20% of the children have 
a chronic medical condition, then create a factor =0.2 for that site (called group_bar). 
 
Step 2: Specify a generalized linear model with the following covariates where 
treatment and group are 0/1 factors: 
 
Treatment + group + treatment*group + group_bar + treatment*group_bar + covariates. 
 
This model will decompose the within-site effect of treatment by group and the 
across-site effect of treatment by group. 
 
3.2 
Subgroup analysis – time-to-event – secondary 
 
The goal is to identify subgroups of patients who responded well (or not at all) to the 
intervention in terms accelerating the time to completing of the vaccination series. 
Contrasts between groups will be estimated and reported as 
 
Restricted mean survival time analyses can be done with interactions 
(treatment*subgroup). 
 
Effect modification will be estimated on the difference scale. 
Using pseudo-observation approach: 
Stata: glm pseudo treat##subgroup, vce(cluster site) 
Results can be standardized for covariates (see above). 
 
3.2.2 Permutation-test based confirmatory subgroup analysis - confirmatory 
 
The STEPP package in R (in theory) can implement permutation-based analyses 
for binary outcomes (as well as count, continuous and survival) (Yip 2016). This 
approach will be confirmatory as well as exploratory, and as time permits, and will be 
applied to the case of multiple subgroups defined in terms of latent classes, and will be 

SAP v 1.6 | Sep 12, 2019 
12 
 
 
used for the primary outcome (binary) and for the time to event estimate. If the approach 
runs, we shall report all results, as confirmatory. 
 
[Tested using stepp data. Remains for testing using actual study data.] 
 
4.0 
Season specific estimation 
 
There is interest in estimating differences by season, and for that reason we will estimate 
treatment effects separately by season as well as combined overall. 
 
Proposal: Consider each season*site combination to be a stratum and do the analysis 
heavily stratified. If there is a need to consider differences by season, then treat season as 
a separate, site-level effect modifier. 
 
5.0 
Amendments 
 
We have made the following amendments on the dates notes and for the reasons noted. 

SAP v 1.6 | Sep 12, 2019 
13 
 
 
References: 
 
 
Allison PD. Fixed Effects Regression Models. Newbury Park, CA: Sage; 2009. 
 
Bohning D, Sarol J,Jr. Estimating risk difference in multicenter studies under baseline- 
risk heterogeneity. Biometrics. 2000; 56(1):304-8. [PMID: 10783812] 
 
Cronin A, Tian L, Uno H. strmst2 and strmst2pw: New commands to compare 
survival curves using the restricted mean survival time. The Stata J. 2016; 16(3):702- 
16. 
 
Devan V. Mehrotra, Xiaomin Lu & Xiaoming Li (2010) Rank-Based Analyses of 
Stratified Experiments: Alternatives to the van Elteren Test, The American 
Statistician, 64:2, 121-130, DOI: 10.1198/tast.2010.08121 
 
Guo S, 2009. "HODGESL: Stata module to perform Hodges-Lehmann aligned rank 
test," Statistical Software Components S457042, Boston College Department of 
Economics. 
 
Hernan, M. A. (2010). The hazards of hazard ratios. Epidemiology (Cambridge, 
Mass) 21, 13-15. 
 
Horiguchi M, Cronin AM, Takeuchi M, Uno H. A flexible and coherent test/ 
estimation procedure based on restricted mean survival times for censored time-to-event 
data in randomized clinical trials. Statistics in Medicine. 2018;37:2307–2320. 
https://doi.org/10.1002/sim.7661 
 
Hothorn T, Hornik K, van de Wiel M, Zeileis A. Implementing a Class of Permutation 
Tests: The coin Package. Journal of Statistical Software. 2008;28(8). 
 
Kaiser J, Lacy MG. A general purpose method for two-group randomization tests. The 
Stata Journal 2009;9:70-85. 
 
 
Lanza ST, Rhoades BL. Latent class analysis: an alternative perspective on subgroup 
analysis in prevention and treatment. Prev Sci. 2013; 14(2):157-68. [PMID: 21318625] 
 
 
Overgaard M, Andersen PK, Parner ET. Regression analysis of censored data using 
pseudo-observations: An update. Stata J. 2015;15(3):809-821. 
 
 
Royston P, Parmar MK. Restricted mean survival time: an alternative to the hazard 
ratio for the design and analysis of randomized trials with a time-to-event outcome. BMC 
Med Res Methodol. 2013; 13:152,2288-13-152. [PMID: 24314264] 

SAP v 1.6 | Sep 12, 2019 
14 
 
 
Royston P, Parmar MK. The use of restricted mean survival time to estimate the 
treatment effect in randomized clinical trials when the proportional hazards assumption is 
in doubt. Stat Med. 2011; 30(19):2409-21. [PMID: 21611958] 
 
Therneau T, Grambsch P, Pankratz VS. Penalized survival models and frailty, J 
Computational and Graphical Statistics. 2003; 12:156-175. 
 
Therneau T. (2018) https://cran.rproject.org/web/packages/coxme/vignettes/coxme.pdf 
 
Tian L, Zhao L, Wei LJ. Predicting the restricted mean event time with the subject's 
baseline covariates in survival analysis. Biostatistics. Apr 2014;15(2):222-233. 
 
 
Uno H, Claggett B, Tian L, Inoue E, Gallo P, Miyata T, Schrag D, Takeuchi M, 
Uyama Y, Zhao L, Skali H, Solomon S, Jacobus S, Hughes M, Packer M, Wei LJ. 
Moving beyond the hazard ratio in quantifying the between-group difference in survival 
analysis. J Clin Oncol. Aug 1 2014;32(22):2380-2385. 
 
 
White IR, Walker S, Babiker A (2002). strbee: Randomisation-based efficacy 
estimator. Stata Journal 2: 140-150. 
 
 
Yip WK, Bonetti M, Cole BF, Barcella W, Wang XV, Lazar A, et al. Subpopulation 
Treatment Effect Pattern Plot (STEPP) analysis for continuous, binary, and count 
outcomes. Clin Trials. 2016; 13(4):382-90. [PMID: 27094489]