Increasing Human Papillomavirus (HPV) Immunization Rates: The Rural Adolescent Vaccine Enterprise (RAVE)
Phase: NA
Conditions: Human Papilloma Virus
Sponsor: Oregon Health and Science University

 
V1 06/27/2018 
OHSU Knight Cancer Institute 
Minimal-Risk Protocol 
 
Increasing HPV Immunization Rates: The Rural Adolescent Vaccine 
Enterprise (RAVE) 
 
NCT03604393 
 
Principal Investigator:             
Lyle J. Fagnan, MD  
  
Co-Investigators:                 
Patty Carney, PhD; Brigit Hatch, MD, MPH; Melinda Davis, PhD; Michelle Berlin, MD, MPH; & Paul 
Darden, MD 
  
Biostatistician:                              
Miguel Marino, PhD, MS 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Final Protocol Date:  
09/10/2018 
 
 
Protocol Revision Dates: 
08/23/2018  
 
 
 
 
08/30/2018 
 
 
 
 
09/10/2018 
 
 
 

 
V1 06/27/2018 
TABLE OF CONTENTS 
1.0 
Background/ Rationale (required) ..................................................................................................... 3 
2.0 
Objectives.......................................................................................................................................... 4 
3.0 
Study design/Methodology (required) .............................................................................................. 5 
4.0 
Study Population (required) .............................................................................................................. 5 
5.0 
Inclusion/Exclusion Criteria (required) .............................................................................................. 6 
6.0 
Vulnerable Populations (required) .................................................................................................... 6 
7.0 
Setting (required) .............................................................................................................................. 7 
8.0 
Study Procedures and Schedule of Events (required) ....................................................................... 7 
9.0 
Data and Specimens (required) ......................................................................................................... 8 
10.0 
Risks to Subjects (required) ............................................................................................................. 10 
11.0 
Potential Benefits to Subjects (required) ........................................................................................ 10 
12.0 
Timeline & Milestones (required) ................................................................................................... 11 
13.0 
Bio-statistical Considerations (if applicable) ................................................................................... 11 
14.0 
Recruitment Methods (required) .................................................................................................... 13 
15.0 
Informed Consent (required, unless a waiver or alteration is approved by the IRB) ..................... 13 
16.0 
Changes to Protocol (required, ok to use the suggested language below) .................................... 14 
17.0 
Privacy, Confidentiality, and Data Security (required) .................................................................... 14 
18.0 
OHSU IRB Reporting of Unanticipated Problems and Adverse Events (only if applicable) ............ 14 
19.0 
OHSU Knight Cancer Institute Data and Safety Monitoring Plan (only if applicable) ..................... 14 
20.0 
Inclusion of Women, Minorities and Children (only if applicable) ................................................. 14 
21.0 
Inclusion of Children (only if applicable) ......................................................................................... 14 
22.0 
References (if cited above) ............................................................................................................. 15 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
V1 06/27/2018 
1.0 BACKGROUND/ RATIONALE  
Dangers Associated with the Human Papilloma Virus & Vaccine Recommendations:  HPV-16 and -18 
account for about 70% of cancers of the cervix, vagina, and anus and for approximately 30-40% of 
cancers of the vulva, penis, and oropharynx (4). Other cancers linked to HPV include non-melanoma skin 
cancer and cancer of the conjunctiva (3). About $8 billion is spent annually to manage the sequelae of 
HPV infections, primarily to manage abnormal cervical cytology and cervical neoplasia (1). This exceeds 
the economic burden of any other sexually transmitted infection except for the human 
immunodeficiency virus (1). 
In 2006, the Food and Drug Administration approved a vaccine to protect against strains of the virus 
(5), including HPV-16 and -18. Currently, three vaccines (Cervarix, Gardasil, and Gardasil 9) are available 
(13), and vaccination is recommended starting at age 11 or 12 years for both males and females. 
Unfortunately, the U.S. National Immunization Survey (NIS) has shown that completion of the vaccine 
series (three shots) often does not occur despite recommendations by several groups (6, 13, 14). 
Regarding females, national vaccine initiation among 13-17 year olds increased from 53% in 2011 to 60% 
in 2014. However, vaccine series completion was only 39.7% nationally, with regional variation by state 
(20.1%-56.9%) (6). Regarding males, modest increases occurred between 2014 to 2015 for initiating the 
HPV vaccine series (from 41.7% to 49.8%), with completion of the series increasing only from 21.6% to 
28.1% (7).  In late 2016, the Advisory Committee on Immunization Practices (ACIP) has approved a 
schedule change for HPV vaccination completion.  The ACIP recommends the use of a 2-dose schedule 
for girls and boys who initiate the vaccination series at ages 9 through 14 years.  Three doses remain 
recommended for persons who initiate the vaccination series at age 15 through 26 years and for 
immunocompromised persons (15).  This policy change will be taken into account in our analyses.  HPV 
vaccination coverage remains lower than meningococcal ACWY (MCV) and tetanus, diphtheria, and 
acellular pertussis (Tdap) vaccine rates (16). This gap points to the opportunity to reduce missed 
opportunities for HPV vaccinations.  
One of the first studies to use state immunization registry data to assess missed opportunities for 
HPV vaccines showed a missed opportunity rate of 43.9% between 2008-2012 among females aged 11-
26 years. Compared to urban areas, rural areas were more likely to have missed opportunities (OR 1.8) 
(17). Another study reported that 84% of unvaccinated adolescent females had at least one healthcare 
visit where non-HPV vaccines were administered and the HPV vaccine was not (18). Other adolescent 
vaccination rates are relatively high with MCV at 79% and Tdap at 87% (18).  
Barriers to HPV Vaccination: Studies on vaccine acceptability indicate most parents react positively to 
vaccinating their children against HPV, especially when parents believe the vaccine is effective, a 
physician recommends it, and HPV infection is likely (19-21). Between 6% and 12% of parents fear that 
vaccination would promote adolescent sexual behavior, a barrier to vaccination (20, 21). Race does not 
appear to affect acceptability of the HPV vaccine (18). Although cost has been reported as a barrier (22), 
the cost of the HPV vaccine for uninsured children under 18 years is covered by the federal Vaccines for 
Children Program (VFC). 
Rural areas have unique challenges regarding HPV and are understudied.  Access to pediatricians 
and family physicians is often challenging in rural areas, with preventive care underutilized (23-26). 
Lower socioeconomic status, health literacy, and religiosity/conservatism are prevalent in rural areas 
(27). A study of communication disparities (28) found that rural parents are least likely to engage in 
communication with their children’s healthcare clinicians, with lower rates of mutual information 
exchange, deliberation, and shared decision-making. This study, which used NIS-Teen data from 4,124 
parents of daughters aged 13-17 years, showed lower uptake of HPV in rural areas (28). 
Studies on medical practice barriers indicate that inadequate reimbursement (68%) and burden of 
determining insurance coverage (66%) all inhibit primary care practices from stocking the vaccine and 
providing it to adolescents (29, 30). Practices not providing the vaccine reported more concerns about 
HPV vaccine provision than practices providing it (n=71 practices: 6.0 vs. 4.5 concerns, p < 0.05) (29). 
Other barriers include parents deciding to delay rather than refuse vaccination. When this occurs, 
clinicians are hesitant to discuss the topic, while teens consider themselves passive in the decision-
making process (31a).  

 
V1 06/27/2018 
To overcome these barriers, clinicians either present the HPV vaccine as routine or as optional, while 
highlighting risks and benefits (31). Additional studies have examined why clinicians do not follow 
clinical practice guidelines (31b). One systematic review of 76 published studies (31) identified seven 
common barriers, including lack of awareness, familiarity with details, lack of agreement with guidelines, 
issues related to self-efficacy (ability to reach goals) (31), outcome expectancy (belief that given 
behaviors will lead to certain outcomes) (32), ability to overcome practice inertia, and external barriers – 
such as health insurance coverage (33). Importantly, barriers in one setting are not generalizable to 
other settings (34). Improving primary care practices’ rates of HPV vaccinations will require a tailored 
approach.   
Several studies have 
tested interventions to 
improve preventive services 
delivery (35-42). One meta-
analysis (37) found that 
organizational change, such 
as separate clinics or pods 
within a clinic devoted to 
prevention, use of a planned 
care visit for prevention, or 
designation of non-clinician 
staff to do specific 
prevention activities, were 
most effective intervention 
strategies, with adjusted 
odds ratios ranging from 2.5 
to 17.6 (37). Financial 
incentives, patient reminders 
and education, and clinic 
feedback were the next most 
effective intervention 
strategies, with adjusted 
odds ratios ranging from 1.1 
to 3.4 (37). Additional studies 
have found that clinician education and office-based systematic patient tracking can increase preventive 
service delivery by 12-90% (38-40). However, practice members must be involved in choosing what will 
work in their setting (39, 40). A 2015 systematic review of practice- and community-based interventions 
identified promising programs to close the gap in HPV vaccination, which had been implemented in 
diverse populations.  Interventions targeted providers with audit and feedback, involved school-based 
programs, and included social marketing campaigns (11, 36). Figure 1 summarizes the characteristics of 
recent studies that have addressed HPV vaccination interventions and highlights those features we will 
include in our intervention study. 
  
2.0 OBJECTIVES 
The overarching goal of this study is to engage rural primary care clinics and community organizations to 
test interventions designed to increase HPV vaccinations in both male and female patients aged 11-17 
years. Using a step-wedge randomized controlled trial, we will design and test the effectiveness of 
multicomponent primary care practice-based interventions on the completion of the HPV vaccine series 
and will explore implementation timing in high and low functioning practices to determine how specific 
characteristics affect the delivery of the full series. Additionally, we will design and test the impact of a 
community organization-based intervention intended to educate the public about HPV and cancer risk 
and risk reduction via vaccination. 

 
V1 06/27/2018 
3.0 STUDY DESIGN/METHODOLOGY  
Aim I: Primary Care and Community-based Organizations: Conduct a baseline assessment of how 
primary care practices, public health programs, and community-based organizations are initially 
addressing initiation and completion of the HPV vaccine series. 
We have already determined HPV vaccination rates in 53 rural primary care practices (available from 
the Oregon Immunization Program – OIP), which has helped us identify “hot spots” for intervention 
research. We will conduct a baseline assessment in 20 of these practices using direct observation of 
workflows and validated measures of team-based care effectiveness and practice-level knowledge of 
HPV and current HPV vaccine recommendations. This will assist us in developing tailored intervention 
approaches for busy rural primary care practices and enable us to assess which practice characteristics 
affect delivery of the full HPV series. We will also conduct an in-depth assessment of activities 
undertaken in rural Oregon communities to inform the public about cancer risks and HPV vaccines.   
 
Aim II: Primary Care: Implement and test, using a stepped-wedge cluster randomized trial design, the 
effectiveness of a multi-component primary care practice-based intervention on initiation of and 
completion of the HPV vaccine series as well as reduction in rates of missed opportunities to vaccinate. 
      The intervention will be conducted in 40 primary care practices we plan to enroll and will involve: 1) 
practice facilitation to engage clinics in a practice-wide planning process to redesign patient care and 
communication strategies to optimize HPV vaccine completion; 2) workflow mapping adapted to 
practice context to support HPV vaccine delivery; 3) a practice improvement model designed to firmly 
establish reminder and recall systems and standing orders; and 4) education for patients and parents 
that underscores HPV vaccination is safe, effective, and an important approach for reducing cancer risk.  
We will additionally explore uptake of the intervention in terms of implementation timing and 
component features in high- and low-functioning practices to determine which practice characteristics 
affect delivery of the full HPV series. 
 
Aim III: Community Organizations: To explore the extent to which an evidence-based social marketing 
campaign, implemented in partnership with participating practices, will increase HPV vaccine demand.  
      In this Aim, primary care practices will select a community-level organization to partner with, such as 
a regional Accountable Care Organization (ACO) Community Advisory Council (CAC), patient and family 
advisory group, or local public health program based on existing relationships. Together, the clinic and 
community group will attempt to implement an evidence-based community-level intervention (11,12) 
designed to improve knowledge among adolescents and their parents on HPV, cancer risk, and risk 
reduction via vaccination, as well as the safety and effectiveness of the vaccine series. The plan 
specifically explores the extent to which this partnership brings “vaccine ready” adolescent and parents 
to primary care clinics and increases the demand for HPV vaccination. 
 
Aim IV: National Dissemination: To explore the impact of sharing promising clinical and community 
intervention features in a toolkit with practices, state public health programs, and Accountable Care 
Organizations (ACOs). 
     We will create a practice and community HPV improvement toolkit for use by practices, ACOs, and 
state public health agencies. The toolkit will be electronically disseminated, with the assistance of our 
partners in the ACOs and at OIP, and we will assess its spread.   
 
4.0 STUDY POPULATION  
Rural adolescents aged 11-17 years, along with their parents, will be included in this study. In addition, 
rural family physicians and pediatricians and members of their clinical practices will be active 
participants and will uniquely partner with community-based organizations to undertake comprehensive 
communication, education, and social messaging approaches to address inequities in a much 
understudied population, rural America. Members of community-based organizations who partner with 
practice members to implement a social media campaign will also be participating. 

 
V1 06/27/2018 
Table 1. Total Planned Enrollment: 
The numbers below reflect the anticipated recruitment of 11-17  year old subjects in rural 
Oregon, according to 2015 U.S. Census Bureau Data from Oregon on gender, race, and 
ethnicity status. 
 
TARGETED/PLANNED ENROLLMENT: Number of Subjects 
Ethnic Category 
Sex/Gender 
Females 
Males 
Total 
Hispanic or Latino 
1,384 
1,355 
2,739 
Not Hispanic or Latino 
9,511 
9,324 
18,835 
Ethnic Category: Total of All Subjects * 
10,895 
10,679 
21,574 
Racial Categories 
 
American Indian/Alaska Native 
196 
192 
388 
Asian 
479 
470 
949 
Native Hawaiian or Other Pacific Islander  
44 
43 
87 
Black or African American  
228 
224 
452 
White 
9,948 
9,750 
19,698 
Racial Categories: Total of All Subjects * 
10,895 
10,679 
21,574 
* The “Ethnic Category: Total of All Subjects” must be equal to the “Racial Categories: Total of All Subjects.” 
5.0 INCLUSION/EXCLUSION CRITERIA  
We will enroll ORPRN primary care clinics located in rural communities that see adequate numbers of 
patients in the age range we are studying (age 11-17 years). To date, 62 clinics have committed to 
participating in the study. We decided to exclude urban practices in order to study the intervention 
specifically in under-resourced rural settings. Based on our sample size estimates, we need 20 practices 
for Aim I and 40 practices for Aim II of the study. By over-enrolling to 45 practices for Aim II, we 
accounted for an attrition rate of 11%. We are confident, based our extensive experience recruiting and 
retaining practices in research studies, we will easily retain the 40 practices needed to test study 
hypotheses for Aim II. 
 
6.0 VULNERABLE POPULATIONS  
No subject will be excluded from the study on the basis of racial or ethnic origin.  Both male and female 
clinicians (including pregnant clinicians) and practice members will be included in the study. 
Additionally, male and female patients (including pregnant patients) and parents of patients (including 
pregnant parents) at enrolled practices will be included in analyses. This protocol does involve male and 
female children ages 11 to 17 years. We will not include neonates, decisionally impaired adults, and 
prisoners in this study. We will not collect information that would identify a woman as pregnant. The 
study poses minimal risk to pregnant women. 
 

 
V1 06/27/2018 
7.0 SETTING  
Aim I: We will engage 20 rural primary care clinics throughout Oregon – and, their affiliated public 
health programs and community organizations – to understand workflows and clinical structures 
present in high/low/medium HPV vaccine-performing clinics. 
 
Eligibility. Primary care clinics in rural Oregon that report data to the state ALERT IIS program 
o Inclusion criteria: 
 
Located in a rural area of Oregon (defined using RUCA codes) 
 
Family Medicine or Pediatric Practice 
 
Provide primary care services (school based clinics, public health clinics that do 
not deliver primary care, pharmacies, etc. are not be eligible). 
 
Non-tribal 
 
Participating in VFC 
 
Aims II and III: We will engage at least 40 rural primary care clinics throughout Oregon – and, their 
affiliated public health programs and community organizations – to test interventions designed to 
increase HPV vaccinations in male and females aged 11-17 years, with an emphasis on 11-12 year olds.   
 
8.0 STUDY PROCEDURES AND SCHEDULE OF EVENTS  
The proposed study subjects are family physicians and pediatricians, other healthcare providers (nurse 
practitioners and physician assistants), office staff located and patients in 40-60 practices from ORPRN 
(practices that participate in Aim I can participate in Aim II). Additional participants will include members 
of community-based organizations who will partner with participating practices to implement a social 
media campaign to educate the public about the HPV vaccine. More specifically, patient participants will 
include: 1) male and female patients between the ages of 11 and 17 years (and their parents for 11-12 
year olds) who are either receiving routine healthcare at enrolled ORPRN clinics or reside within the 
catchment areas of these clinics.  
 
Aim I:  
 
Pre-Site Visit Phone Interview (45 min – 1 hour). The purpose of these calls is to coordinate the 
Formal Site Visit and to gain buy-in for the visit. This will include confirming locations for 
observation (front desk, patient encounters, back office, population outreach, lab) and gathering 
names of individuals for interviews. As time allows, we will also gather initial information on 
organizational structure, key stakeholders involved in the immunization process, and the 
practice’s basic approach for immunizations. 
 
Practice Intake Survey (5 min). This brief intake survey will be completed by the clinic point of 
contact and provide descriptive data regarding practice characteristics (ownership, QI team) and 
patient panel information (total visits, demographic profile).  
 
Formal Site Visit (1-2 days) with supplemental informal key-informant interviews. The 
observation visits will last 1.5 days and be conducted by two members of the study team.  
o The observation will be used to document clinical workflows, organizational culture, and 
other factors that may influence HPV vaccination rates. We are interested in vaccine 
delivery and missed opportunities. 
o Informal interviews will be conducted with clinical staff members to supplement and 
clarify onsite observations. These interviews will occur “on the fly” when there are 
breaks in clinical care.  
 
Community Partner Survey (10 min). We will administer surveys (either through SurveyMonkey 
or on paper) to approximately six people – from various sectors (healthcare services, public 

 
V1 06/27/2018 
health, education, policy, etc.) in the communities where Aim I practices are located – to gather 
knowledge and perceptions of the HPV vaccine and to understand how we could/should market 
the HPV vaccine in the community.  
 
Qualitative data from Aim I will be transferred into Atlas.ti for data management and analysis: 
 
Field notes from pre-site visit phone interview reviewed/finalized. 
 
Jottings from observation visits typed into field notes within 24 hours. 
 
Interviews audio transcribed. 
 
Quantitative data (brief practice intake survey, pre-observation visit call) will be transferred from Survey 
Monkey (or paper form) into Excel for descriptive analysis. Quantitative data from OIP (ALERT IIS) sent to 
OHSU via secure file transfer (SFT) and added into Excel for descriptive analysis.  
 
Aims II and III: Members in each practice (n=40) will be asked to complete a Practice Survey (PS) at 
baseline and again annually (for variables that might change). The PS ascertains information about 
practice and practice’s patient demographics, practice change, payer mix, revenue and payments, HPV 
vaccine priority, internal structure and strategies for improving HPV vaccine rates, external reporting, 
registries, and clinical guideline availability. A Staff Member Survey (SMS) will also be administered at 
baseline and annually (for variables that might change). It ascertains information on staff member 
demographics, adaptive reserve, patient needs, clinician and staff satisfaction and burnout, staff 
motivation, staff turnover, practice leadership, practice cohesion and rewards, and practice readiness to 
change. A Quality Improvement Change Assessment (QICA) will be administered every six months. It 
ascertains information on the foundational change concepts for building quality improvement (QI) 
capacity in primary care: Organized, Evidence-Based Care; Quality Improvement Strategy; Continuous & 
Team-Based Healing Relationships; and Care Coordination. Intervention Site Visits will be conducted 
monthly, which will involve workflow mapping; patient data collection: eligible patient population, 
patient demographics, and other characteristics of the patient population; quarterly immunization rates; 
quarterly community education data collection: existing education programs, efforts, and partnerships; 
PDSA cycle worksheets; and field notes. We will also conduct Patient/Parent Surveys every six months, 
which will ascertain children’s and parents’ knowledge about cancer risk reduction and the HPV vaccine 
series, where and how information about HPV and cancer risk reduction via vaccination was sought, and 
patient and parental demographic information. Lastly, we will conduct Pre- and Post-Community Partner 
surveys and Group Interviews that will assess perspectives on community acceptability of HPV 
vaccination. 
Table 2. Measures and Methods According to Solberg Model Constructs 
Aim I:  Data Collection 
Methods & Timing 
Measures 
Solberg Model 
Constructs 
Pre-Site Visit Phone Interviews 
– once at Baseline 
1. 
Staffing structure 
2. 
Presence of QI team 
3. 
EHR 
Priority 
Formal Site Visit – once at 
Baseline 
Baseline workflows; baseline patient data collection: eligible patient population, patient 
demographics, and other characteristics of the patient population; baseline 
immunization rates; baseline community education data: existing education programs 
and efforts, existing partnerships 
Content 
Practice Intake Survey (PS) – 
once at Baseline 
1. 
Ownership 
2. 
Presence of a designated QI team 
3. 
Relationship with community groups with regard to public health (and description, 
quantity) 
4. 
Percentage of minor patients 
5. 
Total patient visits in past week 
6. 
Patient demographics (Race & Ethnicity) 
Priority 

 
V1 06/27/2018 
ALERT IIS – once at Baseline 
1. 
Vaccine initiation rates  for children ages 11-12 
2. 
Vaccine completion rates for children ages 13-17  
3. 
Insurance status 
Priority 
Community Partner Survey – 
once at baseline 
Knowledge and perceptions of HPV vaccinations; relationship between HPV and cancer; 
perceived utility of a social marketing campaign on HPV vaccination, community health, 
and cancer rates 
Priority 
Content 
Aim II: Data Collection 
Methods & Timing 
Measures 
Solberg Model 
Constructs 
Practice Survey (PS) –  annually 
Practice & practice’s patient demographics, practice change, payer mix, revenue & 
payments, HPV vaccine priority 
Priority 
Internal structure & strategies for improving HPV vaccine rates, external reporting, 
registries & clinical guideline availability 
Capability 
Staff Member Survey (SMS) – 
annually 
Staff member demographics, adaptive reserve, patient needs, clinician and staff 
satisfaction and burnout, staff motivation, staff turnover, practice leadership, practice 
cohesion and rewards 
Priority 
Practice readiness to change 
Capability 
ALERT IIS – every six months 
(or more, TBD) 
1. 
Vaccine initiation rates  for children ages 11-12 
2. 
Vaccine completion rates for children ages 13-17  
3. 
Insurance status 
Priority 
Quality Improvement Change 
Questionnaire (QICA) – every 
six months 
Engaged Leadership 
Priority 
Organized, Evidence-Based Care; Quality Improvement Strategy; Continuous & Team-
Based Healing Relationships; Care Coordination 
Capability 
Site Visits – monthly 
Workflows; patient data collection: eligible patient population, patient demographics, 
and other characteristics of the patient population; quarterly immunization rates; 
quarterly community education data: existing education programs and efforts; existing 
partnerships 
Content 
Intervention Data Entries –  
monthly 
PDSA cycle worksheets, workflow mapping, field notes 
Content 
Aim III:  Data Collection 
Methods & Timing 
Measures 
Solberg Model 
Constructs 
Pre- and Post-Community 
Partner Group Interviews 
Acceptability of HPV vaccination 
Priority 
Patient/Parent Surveys – every 
6 months 
Child and parents’ knowledge about cancer risk reduction and the HPV vaccine series; 
where and how information about HPV and cancer risk reduction via vaccination was 
sought; demographic information  
Content 
 
As Aim II and III project materials are developed (e.g., surveys, QICA document, site visit materials), they 
will be submitted to the IRB. 
 
9.0 DATA AND SPECIMENS  
a) Handling of Data and Specimens 
ALERT Immunization Information System (IIS) Abstraction 
 
We will extract clinic-level data through ALERT IIS – via our collaboration with OIP – to investigate 
how the intervention we are testing affects patient outcomes (increases in HPV vaccine series 
completion) and practice outcomes (reductions in missed opportunities to vaccinate). These data will be 
used to determine how clinic rates of delivering HPV vaccination series changed throughout the study. 
We will identify established male and female patients in each practice who are between the ages of 11-
17 years. Information on number of visits and dates of services for each patient in the practice during 
the study years outlined in our application will be captured. 
 
b) Sharing of Results with Subjects 
Practices will receive their HPV vaccination rate data via ALERT IIS. 
 
 

 
V1 06/27/2018 
c) Data and Specimen Banking 
Not Applicable. 
 
10.0 
RISKS TO SUBJECTS  
Risks associated with this project are expected to be minimal; loss of confidentiality is the greatest 
potential risk. The surveys and assessments, mentioned above, pose minimal risk to practice members 
because the instruments contain minimal personal information. It is possible that someone in the 
practice could use information about a respondent’s role (e.g., Office Manager) to identify that person 
and his or her responses. For this reason, our study team collects these data personally, and the data we 
collect is kept completely confidential and only used by the study team. If data is reported to practices, 
we report these data in aggregate so that it is not possible to link an individual to his or her responses. 
The risk of loss of confidentiality for patients/parents is higher because the survey for these individuals 
contains more sensitive information about HPV knowledge and attitudes regarding vaccination. 
All data entered or sent to the study offices will be handled in a highly confidential manner 
consistent with the high standards established at OHSU and ORPRN. All employees at OHSU and ORPRN 
are required to sign a confidentiality statement and abide by standards of confidentiality and data 
security outlined in a manual developed for this purpose. A unique encrypted identifier will replace 
names on all study instruments and documents, including field notes and interview transcripts, and in a 
master database that will hold all study data.  Data presented in all presentations and publications will 
not be associated with the name of any participating person or practice. All computer systems at OHSU 
and ORPRN are protected from possible external access using network security systems. Only study 
researchers and staff will have access to the data.  
We will use industry standard Secure Sockets Layer (SSL) technology with server and client 
certificates to insure the confidentiality of data use and any data transfer. 
 
11.0 
POTENTIAL BENEFITS TO SUBJECTS  
Potential Benefits of the Proposed Research to the Subjects and Others 
 
The study may contribute long-term benefits of new scientific knowledge about providing HPV 
vaccinations to male and female patients aged 11-17 years. The participants will not directly benefit 
from their participation. The benefits of improving cancer risk reduction and vaccine delivery processes 
far outweigh the remote possibility of a breach in confidentiality.  
 
Importance of Knowledge to be Gained  
 
The information gained from this study may provide critical information on how to reduce the 
burden of cancer in morbidity and mortality. The findings from this research may be useful to other 
areas of vaccine research. Additionally, this study will help us understand how to improve QI within rural 
primary care practices, and it will help determine the relationship between community-based 
organizations and primary care practices. 
 

 
V1 06/27/2018 
12.0 
TIMELINE & MILESTONES  
Figure 2. Overall Project Timeline 
Figure 3. Aim 1 Timeline (below) 
 
13.0 
BIO-STATISTICAL CONSIDERATIONS  
Our proposed stepped-wedge cluster randomized trial (78) will evaluate the impact of the 
proposed intervention on HPV outcomes in 40 primary care practices (Aim II). We will 
specifically test two hypotheses: 2a) More patients will have initiated and completed the HPV 
vaccine series in intervention practices compared to control practices, and 2b) Practices 
receiving the intervention will have lower rates of missed opportunities to vaccinate compared 
to control practices. The primary outcome variables will include receipt rate of the initial (age 
11-12 years) and full series of HPV vaccines and a reduction in the rates of missed opportunities 
to vaccinate among males and females aged 11-17 years. We will conduct block randomization 
of practices in five waves every six months. Outcomes will be measured every three months in 
all clusters at every period, so that each cluster provides data points in both the control and 
intervention conditions. Practices in each wave will be stratified based on designation of 
pediatric or family medicine clinic to balance pediatric/family medicine clinics at each wave. 
Outcome data collection will be collected from ALERT IIS beginning in quarter four of the study 
and up to 17, for 14 longitudinal data collection points per practice. Dr. Marino, who has no 
contact with the study practices, will perform the randomization for all practices prior to the 
start of the first intervention wave. Randomized assignment will not be revealed to practices 
until their cluster begins the study.  

 
V1 06/27/2018 
 
The primary goal of Aim II is to establish whether the intervention is effective and the 
degree to which it is accomplishing its goals. Our outcome evaluation includes a rigorous 
research design so we may attribute positive findings to the intervention and not to existing 
threats to internal validity. To compare the effect of the intervention with usual care on HPV 
outcome measures in the context of a stepped-wedge design, we will utilize Generalized Linear 
Mixed Models (GLMMs) (79). In particular, to assess the intervention effect, we will use a 
Poisson model with random effects for practice. The general model is as follows: 
(1)  𝑙𝑜𝑔⁡(𝑌𝑖𝑡|𝑏𝑖) = 𝛽0 + 𝛽1𝑡+ 𝛽2𝐼𝑖𝑡+ 𝛽3𝐼𝑖𝑡(𝑡−𝑠𝑖) + 𝛽4𝑍𝑖+ 𝑏𝑖+ 𝑙𝑜𝑔(𝐸𝑖𝑡) 
where 𝑌𝑖𝑡 is the number of patients in practice i who had HPV completion during period 𝑡 for 𝑡∈
(0,1, … ,13). Data is collected every three months (i.e., quarterly), and 𝑡= 0 is the baseline 
quarter. Additionally, 𝐼𝑖𝑡 is an indicator variable where 𝐼𝑖𝑡= 1 if practice i has been assigned to 
the intervention at period 𝑡, and 𝐼𝑖𝑡= 0 otherwise. Term 𝑠𝑖 is the period when the intervention 
begins for practice i. The practice-level term to denote a family medicine or pediatric practice is 
denoted by 𝑍𝑖 where 𝑍𝑖= 1 if practice i is a family medicine practice and 𝑍𝑖= 0 if pediatric 
practice. 𝐸𝑖𝑡 is the number of patients in the practice i who were eligible to be counted during 
period 𝑡 and the 𝑙𝑜𝑔⁡(𝐸𝑖𝑡) term is considered the ‘offset’ in the Poisson regression model. The 
term 𝑏𝑖 denotes a random effect term for practice i with mean 0 and variance 𝜎𝑏
2. The 
estimation via model (1) takes into account the general time trend, and allows for the 
intervention effect to grow over time following intervention implementation. Model (1) 
estimates the intervention effect with the within site difference between HPV completion rates 
pre- and post-intervention, averaging across practices and accounting for possible secular trends 
which might confound with the timing of the intervention implementation. In model (1), 𝛽0 is 
the (log of) average pre-intervention HPV completion rate for pediatric practices (i.e., 𝑍𝑖= 0) 
with the random effect set to 0. 𝛽1 is the effect of the general time trend of the HPV completion 
rate without the intervention. 𝛽2 represents the intervention effect that appears in the first time 
period of the intervention. 𝛽3 represents the added intervention effect that accrues over time 
following intervention implementation. Model (1) will also apply for outcome of HPV initiation, 
completion, and rates of missed opportunities to vaccinate by replacing  𝑌𝑖𝑡 and 𝐸𝑖𝑡 with the 
appropriate numerator and denominator. Hypothesis tests will be two-sided, with type I error 
set to 0.05. Because statistical tests are specified a priori and our proposed outcome measures 
are highly related, we will follow recommendations to report p-values rather than adjust for 
multiple comparisons (80, 81). Goodness of fit statistics and model fitting diagnostics will be 
used to assess for influential points and over-dispersion and to evaluate alternative model 
specifications. 
Power Calculations: We assess the power to detect effects that are clinically meaningful and 
realistic. For this calculation, we assess the power to detect at least 80% power of a 2-sided 
significance test with 𝛼= 0.05. We are interested in comparing the rate of HPV completion in 
the sixth quarter after intervention with HPV completion rate at baseline. Given the following 
conditions: 1) Pre-specified total sample size of 40 practices; 2) Design (stepped-wedge with five 
cohorts of eight practices starting six months apart); 3) Two-sided significance level of 0.05; 4) 
Starting practice-level HPV completion of 24% as described above; 5) Intra-cluster correlation 
(ICC) of patients within practices of 0.08 (82, 83), we will have at least 80% power to detect a 
minimum of 19.4% difference in the absolute change (increasing from 24.0% to 43.4%). If we are 
successful at retaining all 45 practices that will be recruited, we will have at least 80% power to 
detect a 18.1% difference in absolute change. These estimates assume a conservative ICC of 
0.08. As most studies show an ICC<0.001 (84), the proposed study will have sufficient power to 
detect differences of health policy importance. Estimates of parameters for these power 

 
V1 06/27/2018 
calculations were obtained from model (1) and the most recent, up-to-date OIP dataset. These 
power calculations estimate the minimum numbers needed to detect minimum changes.  
Analysis Considerations: If we find significant practice differences between cohorts, we will 
utilize propensity score weighting methods to reduce the observed bias, help minimize external 
threats to the validity of results, and adjust for imbalances (85, 86). Additionally, model (1) can 
accommodate missing data resulting from practice attrition. We will explore model missingness 
by including relevant variables in the analysis as covariates (87, 88). If non-trivial levels of 
missing data are observed, we will use methods such as multiple imputation to include these 
practices in analyses (87). We will conduct a sensitivity analysis comparing parameter values for 
models using completers only, all practices without multiple imputation, and all practices with 
multiple imputation, to understand any biases introduced by differential attrition or missing not 
at random (88). Our project has control, intervention, and follow-up periods in all seasons, 
which will allow us to detect and account for the potential effect of seasonality on the receipt of 
indicated adolescent vaccines. 
 
14.0 
RECRUITMENT METHODS  
Participating clinicians and practice staff members will be verbally asked if they agree to participate in 
the study after receiving a study information sheet at the start of baseline data collection (Aim I) [Note: 
The clinic team will receive a fact sheet – via in-person meetings with ORPRN staff and/or e-mail 
communication – that describes the study and all study activities prior to being asked to participate.]. 
Participation will be voluntary and will consist of agreeing to allow researchers to observe practice 
operations and interview practice clinicians and staff. For Aim II, clinicians and practice staff members 
will also be asked to complete study surveys at designated intervals. Patients selected to receive surveys 
will also receive information about the study, explaining how the data they provide on their surveys will 
be used. An information sheet about the study will be included with the survey mailings.  Community 
partners will be identified by the participating practice members and project team members; these 
individuals will also receive a fact sheet about the study and an information sheet to help them decide 
whether or not to participate in the study. 
 
Women and Minorities 
 
No subject will be excluded from the study on the basis of racial or ethnic origin. Both male and 
female clinicians and practice members will be included in the study.  Additionally, male and female 
patients and parents of patients at enrolled practices will be included in analyses.  Clinicians and practice 
members will be asked to provide confirm interest at the start of the study. Parents will receive 
information about the study, explaining how the data they provide on their surveys will be used. We will 
not collect information that would identify a woman as pregnant. The study poses minimal risk to 
pregnant women. 
 
 
15.0 
INFORMED CONSENT  
Children 
 
This protocol does involve male and female children ages 11 to 17 years. In Aim III, we will be 
sending children (and their parents) that get seen in the clinics we will work with in Aim II surveys to 
complete that describe the children’s (and their parents’) knowledge, attitudes, beliefs, and behaviors 
relating to the HPV vaccine. In addition to the surveys, children and their parents will receive 
information about the study – via information sheets – explaining how the data they provide on their 
surveys will be used. These information sheets will be used in lieu of signed consent forms in order to 

 
V1 06/27/2018 
not collect patient names or other PHI. However, the sheets will contain check boxes for individuals to 
indicate their consent to participate in the study. If the study team receives an information sheet with a 
dissenting (“No, I would not like to participate in the study.”) remark, we will not review that patient’s 
survey data (if completed). A letter from the child’s physician will also accompany the survey and 
information sheet. This letter will describe why the clinic is participating in the study, what the survey 
results will be used for, and contact information for the study’s project manager so the child/parent can 
call and ask this individual questions relating to the study and their participation and/or the survey. If 
need be, we will create information sheets, surveys, and letters in Spanish, Russian, French, etc. to 
accommodate non-English speaking subjects.  
We are waiving documentation of consent due to the 1) desire for the study team to not obtain PHI-
related data from the children and/or their parents; we want to collect anonymous data, and 2) the 
large number of children and parents we hope to engage in this part of the study, making it not feasible 
for us to obtain consent in-person or over the phone with each child and their parent. 
Once Aim III materials are finalized, we will submit them to the IRB. 
 
16.0 
CHANGES TO PROTOCOL  
Any modification of this protocol must be documented in the form of a protocol revision or amendment 
signed by the principal investigator and approved by the Knight Cancer Institute and the IRB before the 
revision or amendment may be implemented. The only circumstance in which the amendment may be 
initiated without regulatory approval is for a change necessary to eliminate an apparent and immediate 
hazard to the patient. In that event, the investigator must notify the IRB in writing within 5 working days 
after the implementation.  
17.0 
PRIVACY, CONFIDENTIALITY, AND DATA SECURITY  
The data we collect will not contain any personal identifiers.  All data will be stored on secure servers as 
well as secure, password-protected computers in locked offices. We will use information sheets in lieu 
of signed consent forms for the interviews and surveys; participants and data/responses cannot be 
connected. The data will be maintained indefinitely..  
 
18.0 
OHSU IRB REPORTING OF UNANTICIPATED PROBLEMS AND ADVERSE EVENTS  
Not Applicable. 
19.0 
OHSU KNIGHT CANCER INSTITUTE DATA AND SAFETY MONITORING PLAN  
Not Applicable. 
20.0 
INCLUSION OF WOMEN, MINORITIES AND CHILDREN  
No subject will be excluded from the study on the basis of racial or ethnic origin. Both male and female 
clinicians and practice members will be included in the study.  Additionally, male and female patients 
and parents of patients at enrolled practices will be included in analyses.  Clinicians and practice 
members will be asked to confirm participation at the start of the study (via an information sheet). 
Parents will receive information about the study, explaining how the data they provide on surveys will 
be used.   
 
21.0 
INCLUSION OF CHILDREN  
This protocol does involve male and female children ages 11 to 17 years. We will attain children’s assent 
regarding completion of surveys by having them check a box on page 3 of the information sheet 
indicating whether they want to participate in the study or not. Children will receive information about 
the study via the information sheet, explaining how the data they provide on their surveys will be used.   

 
V1 06/27/2018 
 
22.0 
REFERENCES  
1. 
Centers for Disease Control and Prevention. Epidemiology and Prevention of Vaccine-
Preventable Diseases. Hamborsky, J., Kroger, A., Wolfe, S. (2015). Chapter 11, Human Papilloma 
Virus 13th ed. Washington D.C. Public Health Foundation. 
2. 
Genital HPV Infection - CDC Fact Sheet. Centers for Disease Control and Prevention (CDC). 2009-
11-24. Retrieved 2010-02-13. 
3. 
Muñoz, N., Castellsagué, X., Berrington de González, A., & Gissmann, L. (2006). Chapter 1: HPV in 
the etiology of human cancer. Vaccine, 2006, 24(Supp 3), S1-S10. 
4. 
  Lowy, D.R., Schiller, J.T. (2012). Reducing HPV-associated cancer globally. Cancer Prevention 
Research (Philadelphia), 5(1),18-23.  
5. 
  Koutsky, L. (2009). The epidemiology behind the HPV vaccine discovery. Ann Epidemiol, 19(4), 
239-44. 
6. 
Centers for Disease Control and Prevention, 2012b. National and state vaccination coverage 
among adolescents aged 13–17 years — United States, 2011. MMWR Morb. Mortal. Wkly Rep. 
61, 671–677. 
7. 
  Reagan-Steiner, S., Yankey, D., Jeyarajah, J., et al. (2016). National, Regional, State, and Selected 
Local Area Vaccination Coverage Among Adolescents Aged 13-17 Years - United States, 2015. 
MMWR Morb Mortal Wkly Rep, 65(33), 850-858. 
8. 
  Oster, N.V. (2005). Barriers to adolescent immunization. JABFM, 18(1). 
9. 
  Briss, P.A., Rodewald, L.E., Hinman, A.R., et al. (2000). Reviews of evidence regarding 
interventions to improve vaccination coverage in children, adolescents, and adults. The Task 
Force on Community Preventive Services. Am J Prev Med, 18(1 Suppl), 97–140.  
10. 
  Recommendations regarding interventions to improve vaccination coverage in children, 
adolescents, and adults. Task Force on Community Preventive Services. Am J Prev Med. (2000) 
18(1 Suppl), 92-96.  
11. 
  Niccolai, L.M. & Hansen, C.E. (2015). Practice- and Community-Based Interventions to Increase 
Human Papilloma Vaccine Coverage. A Systematic Review. JAMA Pediatrics, 169(7). 
12. 
  Smulian, E.A., Mitchell, K.R., Stokley, S. (2016). Interventions to increase HPV vaccination 
coverage: A systematic review. Human vaccines & immunotherapeutics, 12(6), 1566-1588. 
13. 
  Petrosky, E., Bocchini, J.A., Jr., Hariri, S., et al. (2015). Use of 9-Valent Human Papillomavirus 
(HPV) Vaccine: Updated HPV Vaccination Recommendations of the Advisory Committee on 
Immunization Practices. Morbidity and Mortality Weekly Report, 64(11), 300-304. 
14. 
  American Cancer Society:  Recommendations for Human Papilloma Virus (HPV) Vaccine Use to 
Prevent Cervical Cancer and Pre-cancers: 
http://www.cancer.org/cancer/cancercauses/othercarcinogens/infectiousagents/hpv/acs-
recommendations-for-hpv-vaccine-use (Accessed 7/2/16). 
15.  
  Meites, E., Kempe, A., Markowitz, L.E. (2016). Use of a 2-Dose Schedule for Human 
Papillomavirus Vaccination - Updated Recommendations of the Advisory Committee on 
Immunization Practices. MMWR. Morbidity and mortality weekly report, 65(49):1405-1408. 
16.  
  Jacobson, R.M., Rogacki, B., Thompson, D.M., Roberts, J.R., Margolis, B., Darden, P.M. (2015). 
Vaccination Rates among Adolescents in Minnesota as Compared with the United States: Not 
"Above Average". Minn Med, 98(11-12), 38-43. 
17.  
  Kepka, D. (2016). Statewide analysis of missed opportunities using registry data. Papillomavirus 
research, 2, 128-132. 
18.  
  Roberts, J.R., Naifeh, M., Jacobson, R.M., et al. (2015). Adolescent Vaccination Performance in 
South Carolina Compared to the United States. J S C Med Assoc, 111(4), 117-121. 

 
V1 06/27/2018 
19.  
  Brewer, N.T. & Fazekas, K.I. (2007). Predictors of HPV vaccine acceptability: A theory-informed, 
systematic review.  Preventive Medicine, 45(2-3),107-114. 
20.  
  Marlow, L.A.V., Waller, J., & Wardle, J. (2007). Parental attitudes to pre-pubertal HPV 
vaccination.  Vaccine, 25(11), 1945–1952. 
21.  
 Roberts, J.R., Thompson, D., Rogacki, B., et al. (2015). Vaccine hesitancy among parents of 
adolescents and its association with vaccine uptake. Vaccine, 33(14), 1748-1755. 
22.  
Katz, M. L., Reiter, P. L., Heaner, S., Ruffin, M. T., Post, D. M., & Paskett, E. D. (2009). Acceptance 
of the HPV vaccine among women, parents, community leaders, and healthcare providers in 
Ohio Appalachia. Vaccine, 27, 3945–3952. 
23.  
 Coyne, C. A., Demian-Popescu, C., & Friend, D. (2006). Social and cultural factors influencing 
health in southern West Virginia: A qualitative study. Preventing Chronic Disease, 3(4), A124. 
24.  
Ferrer, H.B., Trotter, C., Hickman, M., Audrey, S. (2014). Barriers and facilitators to HPV 
vaccination of young women in high-income countries: a qualitative systematic review and 
evidence synthesis.  BMC Public Health, 14, 700. 
25.  
Lyttle, N. L., & Stadelman, K. (2006). Assessing awareness and knowledge of breast and cervical 
cancer among Appalachian women. Preventing Chronic Disease, 3(4), A125. 
26.  
Elnicki, D. M., Morris, D. K., & Shockcor, W. T. (1995). Patient-perceived barriers to preventive 
health care among indigent, rural Appalachian patients. Archives of Internal Medicine, 155(4), 
421–424. 
27.  
Thomas, T., Blumling, A., & Delaney, A. (2015). The influence of religiosity and spirituality on 
rural parents’ health decision-making and Human papillomavirus vaccine choices. ANS Adv Nurs 
Sci, 38(4), E1-E12. 
28.  
Moss, J.L. (2016). Disparities in collaborative patient-provider communication about HPV 
vaccination. Hum Vaccin Immunother, 12(6),1476-83.  
29.  
Keating, K.M., Brewer, N.T., Gottlieb, S.L., Liddon, N., Ludema, C., & Smith, J.S. (2008). Potential 
Barriers to HPV Vaccine Provision Among Medical Practices in an Area with High Rates of 
Cervical Cancer.  J Adoles Health, 43(4), S61-167. 
30.  
Hughes, C.C., Jones, A.L., Feemster, K.A., Fiks, A.G. (2011). HPV vaccine decision making in 
pediatric primary care: a semi-structured interview study. BMC Pediatrics, 11, 74.    
31a. 
Pope, C.A., Escobar-Gomez, M., Davis, B.H., et al. (2016). The challenge of tetradic relationships 
in medically interpreted pediatric primary care visits: A descriptive study of communication 
practices. Patient education and counseling, 99(4), 542-548. 
31b.  Cabana, M.D., Rand, C.S., Powe, N.R., Wu, A.W., Wilson, M.H., Abboud, P.C., Rubin, H.R. (1999). 
Why Don't Physicians Follow Clinical Practice Guidelines? JAMA, 282(15), 1458-1465. 
32.  
Williams, D.M. (2010). Outcome Expectancy and Self-Efficacy: Theoretical Implications of an 
Unresolved Contradiction. Pers Soc Psychol Rev, 14, 417. 
33.  
Sommers, J.P. (2004). Co-pays, Deductibles, and Coinsurance Percentages for Employer-
Sponsored Health Insurance in the Non-Federal Workforce, by Industry Classification, 2004, 
Medical Expenditure Panel Survey, Agency for Healthcare Research and Quality: 
https://meps.ahrq.gov/data_files/publications/st145/stat145.pdf (Accessed 9/27/16). 
34.  
Oxman, A.D., Thomson, M.A., Davis, D.A., & Haynes, R.B. (1995). No magic bullets: a systematic 
review of 102 trials of interventions to improve professional practice. CMAJ, 153(10), 1423–
1431. 
35.   
Hannon, P.A., Harris, J.R. (2008). Interventions to Improve Cancer Screening Opportunities in the 
Workplace. Am J Prev Med, 35(1S), 10S-13S. 
36.  
Paskette, E.D., Tatum, C.M., D’Agostino, R., Rushing, J., Velez, R., Michielutte, R., Dignan, M. 
(1999). Community-based Interventions to Improve Breast and Cervical Cancer Screening: 

 
V1 06/27/2018 
Results of the Forsyth County Cancer Screening (FoCaS) Project1, Cancer Epidemiol Biomarkers 
Prev, 453. 
37.  
Stone, E.G., Morton, S.C., Hulscher, M.E., Maglione, M.A., Roth, E.A., Grimshaw, J.M., Mittman, 
B.S., Rubenstein, L.V., Rubenstein, L.Z., Shekelle, P.G. (2002). Interventions that increase use of 
adult immunization and cancer screening services: a meta-analysis. Ann Int Med, 136(9), 641-
651.  
38.  
Dietrich, A.J., Tobin, T.N., Cassells, A., Robinson, C.M., Greene, M.A., Hill Sox, C., Beach, M.L., 
DuHamel, K.N., Younge, R.G. (2006). Telephone Care Management To Improve Cancer Screening 
among Low-Income Women A Randomized, Controlled Trial. Ann Intern Med, 144, 563-571. 
39.  
Paskett, E.D., McMahon, K., Tatum, C., Velez, R., Shelton, B., Case, L.D., Wofford, J., Moran, W., 
Wymer, A. (1998). Clinic-based interventions to promote breast and cervical cancer screening. 
Preventive Medicine, 27(1), 120-128. 
40.  
McPhee, S.J., Detmer, W.M. (1993). Office-based interventions to improve delivery of cancer 
prevention services by primary care physicians. Cancer, 72(S3), 1100-1112. 
41.  
Zapka, J.G., Taplin, S.H., Solberg, L.I., & Manos, M.M. (2003). A Framework for Improving the 
Quality of Cancer Care: The Case of Breast and Cervical Cancer Screening. Cancer Epidemiol 
Biomarkers Prev, 12, 4. 
42.  
Centers for Disease Control and Prevention. How State and Local Immunization Programs Can 
Help Improve HPV Vaccination Rates to Reduce Future Cancers: 2-3 Actions to Increase HPV 
Vaccinations (2014). 
43.  
Carney, P.A., O'Malley, J., Buckley, D.I., Mori, M., Lieberman, D.A., Fagnan, L.J., Wallace, J., Liu, 
B., Morris, C. (2012). The Influence of Health Insurance Coverage on Breast, Cervical, and 
Colorectal Cancer Screening in Rural Primary Care Settings. Cancer. 
44.  
Fagnan, L.J., Shipman, S.A., Gaudino, J.A., Mahler, J., Sussman, A.L., Holub, J. (2011). To Give or 
Not to Give:  Approaches to Immunization Delivery in Oregon Rural Primary Care Practices. The 
Journal of Rural Health, 27, 385-393. 
45. 
Solberg, L.I. (2007). Improving Medical Practice: A Conceptual Framework.  Ann Fam Med, 5(3), 
251-256. 
46.  
Helfrich, C.D., Yu-Fang, L., Sharp, N.D., Sales, A.E. (2009). Organizational readiness to change 
assessment (ORCA): Development of an instrument based on the Promoting Action of Research 
n Health Services (PARIHS) framework. Implementation Science, 4(38), 4-38. 
47.  
Jaen, C.R., Crabtree, B.F., Palmer, R.F., et al. (2010). Methods for evaluating practice change 
toward a patient-centered medical home. Ann Fam Med, 8, S9-S20. 
48.  
Simon, J.S., Rundall, T.G., Shortell, S.M. (2007). Adoption of Order Entry with Decision Support 
for Chronic Care by Physician Organizations. J Am Med Inform Assoc, 14(4). 
49.  
Solberg, L.I., Kottke, T.E., Brekke, M.L., Calomeni, C.A., Conn, S.A., Davidson, G. (2001). Using 
continuous quality improvement to increase preventive services in clinical practice-- LI, Zander 
KM. The failure of a con- trolled trial to improve depression care: a qualitative study. Jt Comm J 
Qual Improv, 27(12), 639-650.  
50.  
Solberg, L.I., Kottke, T.E., Brekke, M.L., et al. (2000). Failure of a continuous quality improvement 
intervention to increase the delivery of preventive services. A randomized trial. Eff Clin Pract, 
3(3), m105-115.  
51.  
Fischer, L.R., Solberg, L.I., Kottke, T.E. (1998). Quality improvement in primary care clinics. Jt 
Comm J Qual Improv, 24(7), 361-370.  
52.  
Solberg, L.I., Kottke, T.E., Brekke, M.L., Magnan, S. (2000). Improving prevention is difficult. Eff 
Clin Pract, 3(3), 153-155.  

 
V1 06/27/2018 
53.  
Solberg, L.I., Reger, L.A., Pearson, T.L., et al. (1997). Using continuous quality improvement to 
improve diabetes care in populations: the IDEAL model. Improving care for Diabetics through 
Empowerment Active collaboration and Leadership. Jt Comm J Qual Improv, 23(11), 581-592.  
54.  
O’Connor, P.J., Desai, J., Solberg, L.I., et al. (2005). Randomized trial of quality improvement 
intervention to improve diabetes care in primary care settings. Diabetes Care, 28(8), 1890-1897.  
55.  
Fischer, L.R., Solberg, L.I., Zander, K.M. (2001). The failure of a controlled trial to improve 
depression care: a qualitative study. Jt Comm J Qual Improv, 27(12), 639-650.  
56.  
Solberg, L.I., Fischer, L.R., Wei, F., et al. (2001). A CQI intervention to change the care of 
depression: a controlled study. Eff Clin Pract, 4(6), 239-249.  
57.  
McDaniel, R.R., Jordan, M.E., Fleeman, B.F. (2003). Surprise, surprise, surprise! a complexity 
science view of the   unexpected. Health Care Management Review, 28(3), 266-278. 
58.  
Miller, W.L., McDaniel, R.R., Jr., Crabtree, B.F., Stange, K.C. (2001). Practice jazz: understanding 
variation in family practices using complexity science. J Fam Pract, 50(10), 872-878. 
59.  
Miller, W.L., Crabtree, B.F., McDaniel, R., Stange, K.C. (1998). Understanding change in primary 
care practice using complexity theory. J Fam Pract, 46(5), 369-376. 
60.   
McDaniel, R., Driebe, D.J. (2001). Complexity science and health care management. Advances in 
Health Care Management, 2, 11-36. 
61.  
Parchman, M.L., Fagnan, L.J., Dorr, D., et al. (2016). Study protocol for “Healthy Hearts 
Northwest”: a 2x2 randomized factorial trial to build quality improvement capacity in primary 
care. Implementation Science, 11(128), 1-9. 
62.  
Baskerville, N.B., Liddy, C., Hogg, W. (2012). Systematic Review and Meta-Analysis of Practice 
Facilitation Within Primary Care Settings. Ann Fam Med, 10(1), 63-74. 
63.  
Meropol, S.B., Schiltz, N.K., Sattar, A., Stange, K.C., Nevar, A.H., Davey, C., Ferretti, G.A., Howell, 
D.E., Strosaker, R., Vavrek, P., Bader, S., Ruhe, M.C., Cuttler, L. (2014). Practice-Tailored 
Facilitation to Improve Pediatric Preventive Care Delivery: A Randomized Trial, Pediatrics, 
133(6), e1664-e1675. 
64.  
Berta, W., Cranley, L., Dearing, J.W., Dogherty, E.J., Squires, J.E., Estabrooks, C.A. (2015). Why 
(we think) facilitation works: insights from organizational learning theory.  Implementation 
Science, 10, 141. 
65.  
Hussey, M.A., Hughes, J.P. (2007). Design and analysis of stepped wedge cluster randomized 
trials. Contemp Clin Trials, 28(2), 182-91. 
66.   
King, V.J., Davis, M.M., Gorman, P.N., Rugge, J.B., Fagnan, L.J. (2012). Perceptions of shared 
decision making and decision aids among rural primary care clinicians. Med Decis Making, 
32(4):636-644.   
67.   
Davis, M., et al., A Review of ORPRN Survey Research.  Oral presentation at the 38th NAPCRG 
Annual Meeting. Seattle, Washington. November 2010. 2010. 
68.  
Beebe, J. (2001). Rapid Assessment Process: An Introduction. Walnut Creek: CA: AltaMira Press. 
69.  
Miller, W.L., Crabtree, B.F. (1999). The dance of interpretation. In: Crabtree BF, Miller WL, eds. 
Doing qualitative research. 2nd ed. Thousand Oaks, CA: Sage Publications, Inc, 127-143. 
70.  
Cohen, D.J., Crabtree, B.F. (2008). Evaluative criteria for qualitative research in health care: 
controversies and recommendations. Ann Fam Med, 6(4), 331-339. 
71.  
Malterud, K. (2001). Qualitative research: standards, challenges, and guidelines. Lancet, 
358(9280), 483-488. 
72.  
Patton, M.Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services 
Research, 34(5), 1189-1208. 
73.  
Lincoln, Y.S., Guba, E.G. (1985). Naturalistic Inquiry. Newbury Park: CA: Sage. 
74.  
Wagner, E.H., Gupta, R., Coleman, K. (2014). Practice Transformation in the Safety Net Medical 
Home Initiative: A Qualitative Look. Medical Care, 52, S18-S20. 

 
V1 06/27/2018 
75.  
Agency for Healthcare Research and Quality.  EvidenceNOW:  Advancing Heart Health in Primary 
Care: http://www.ahrq.gov/professionals/systems/primary-care/evidencenow.html (Accessed 
09/09/16). 
76.  
Healthy Hearts Northwest: http://healthyheartsnw.org/ (Accessed 09/29/16). 
77.  
Simpson, D.D., Dansereau, D.F. (2007). Assessing Organizational Functioning as a Step Toward 
Innovation. Addiction Science and Clinical Preactice, 3(2), 20-28. 
77a. 
Fuller, J.E., Walter, E.B., Dole, N., O’Hara, R., Herring, A.H., Durkin, M.S., Specter, B., Wey, B. 
(2017). State-Level Immunization Information Systems: Potential for Childhood Immunization 
Data Linkages  Maternal and Child Health Journal, 21(1):29-35.   
77b.. 
Koepke, R., Petit, A.B., Ayele, R.A., Eickhoff, J.C., Schauer, S.L., Verdon, M.J., Hopfensperger, D.J., 
Conway, J.H., Davis, J.P. (2015). Completeness and Accuracy of the Wisconsin Immunization 
Registry: An Evaluation Coinciding With the Beginning of Meaningful Use.  J Public Health 
Management Practice, 21(3), 273–281. 
77c. 
Groom, H., Hopkins, D.P., Pabst, L.J., Murphy, M.J., Patel, M., Cologne, N., Coyle, R., 
Dombkowski, K., Groom, A.V., Kurilo, M.B., Rasulnia, B., Shefer, A., Town, C., Wortley, P., Zucker 
J. (2015). Immunizational information systems to increase vaccination rates: A community guide 
systematic review. J Pub Hlth Manag, 21(3):227-248. 
78.  
Hemming, K., Haines, T.P., Chilton, P.J., Girling, A.J., Lilford, R.J. (2015). The stepped wedge 
cluster randomised trial: rationale, design, analysis, and reporting. BMJ, 350, h391. 
79.  
Dean, C.B., Nielsen, J.D. (2007). Generalized linear mixed models: a review and some extensions. 
Lifetime Data Anal, 13(4), 497-512. 
80.  
Rothman, K.J. (1990). No adjustments are needed for multiple comparisons. Epidemiology. 1(1), 
43-46. 
81.  
Perneger, T.V. (1998). What's wrong with Bonferroni adjustments. BMJ, 316(7139), 1236-1238.   
82.  
Glynn, R.J., Brookhart, M.A., Stedman, M., Avorn, J., Solomon, D.H. (2007). Design of cluster-
randomized trials of quality improvement interventions aimed at medical care providers. Med 
Care, 45(10 Supl 2), S38-43. 
83.  
Cosby, R.H., Howard, M., Kaczorowski, J., Willan, A.R., Sellors, J.W. (2003). Randomizing patients 
by family practice: sample size estimation, intracluster correlation and data analysis. Family 
practice, 20(1), 77-78. 
84.  
Hayes, R.J., Alexander, N.D., Bennett, S., Cousens, S.N. (2000). Design and analysis issues in 
cluster-randomized trials of interventions against infectious diseases. Stat. Methods Med. Res, 
9(2), 95-116. 
85.  
Austin, P.C. & Stuart, E. A. (2015). Moving towards best practice when using inverse probability 
of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects 
in observational studies. Statistics in medicine, 34(28), 3661-3679. 
86.  
Little, T.D., Schnabel, K.U., Baumert, J. (2000). Modeling longitudinal and multilevel data 2000. 
New Jersey: Lawrence Earlbaum Associates. 
87.  
Little, R.J.A. & Rubin, D.B. (2002). Statistical Analysis with Missing Data. Hoboken, NJ: John Wiley 
& Sons. 
88.  
Schafer, J.L. (1997). Analysis of Incomplete Multivariate Data. London, UK: Chapman and 
Hall/CRC Press. 
89.  
Cates, J.R., Shafer, A., Diehl, S.J. & Deal, A.M. (2011). Evaluating a county-sponsored social 
marketing campaign to increase mothers' initiation of HPV vaccine for their preteen daughters 
in a primarily rural area. Social marketing quarterly, 17(1), 4-26. 
90.   
Cates, J.R., Diehl, S.J., Crandell, J.L. & Coyne-Beasley, T. (2014). Intervention effects from a social 
 
marketing campaign to promote HPV vaccination in preteen boys. Vaccine, 32(33), 4171-4178. 

 
V1 06/27/2018 
91.   
Mathematica Policy Research.  Evaluation of the Comprehensive Primary Care Initiative: Second 
Annual Report, April 2016:  https://innovation.cms.gov/files/reports/cpci-evalrpt2.pdf (Accessed 
09/27/16). 
92.   
Carney, P.A., Dietrich, A.J., Keller, A., Landgraf, J.M., O'Connor, G.T. (1992). Tools, Teamwork, 
and 
 
Tenacity: An Office System for Cancer Prevention. Journal of Family Practice, 35(4), 388-394. 
93.  
Dietrich, A.J., O'Connor, G.T., Keller, A., Carney, P.A., Levy, D., Whaley, F.S. (1992). Cancer: 
Improving early detection and prevention. A community practice randomized trial. British 
Medical Journal, 304, 687-691. 
94.   
Dietrich, A.J., Woodruff, C.B., Carney, P.A. (1994). Changing Office Routines to Enhance 
Preventive 
 
Care: The Preventive GAPS Approach. Archives of Family Medicine, 3, 176-183. 
95.   
Dietrich, A.J., Carney, P.A., Sox, C., Reed, S. (1997). An office systems approach to cancer 
prevention in primary care. Cancer Practice, 5(6), 375-381. 
96.   
Nease, D.E., Daly, J.M., Fernald, D., Hahn, D.L., Levy, B.T., Michaels, L, Simpson, M.J., Westfall, 
J.M., 
 
Fagnan, L.J. Impact of a Boot Camp Translation Intervention on Self-Management Support in 
Primary Care:  A report from the INSTTEPP* trial and the MetaLARC Consortium (JABFM, In 
Review). 
97.   
Fagnan, L.J., Morris, C., Shipman, S., Holub, J., King, A., Angier, H. (2007). “Characterizing a 
Practice 
 
Based Research Network, the Oregon Rural Practice-based Research Network (ORPRN) Survey 
Tools.” JABFM, 20, 2. 
98.   
Dorr, D.A., McConnell, K.J., Williams, P., Gray, K.A., Wagner, J., Fagnan, L.J., Malcom, E. (2015). 
Study protocol:  transforming outcomes for patients through medical home evaluation and 
redesign:  a cluster randomized controlled trial to test high value elements for patient-centered 
medical homes versus quality improvement.  Implementation Science, 10, 13. 
99.   
Dorr, D.A., Anastas, T., Ramsey, K., Wagner, J., Sachdeva, B., Michaels, L., Fagnan, L.J. (2016). 
Effect of a Pragmatic, Cluster-Randomized Controlled Trial on Patient Experience With Care:  The 
Transforming Outcomes for Patients Trough Medical Home Evaluation and redesign (TOPMED) 
Study. Med Care [Epub ahead of print]. 
100.  
Deyo, R.A., Bryan, M., Comstock, B.A., Turner, J.A., Heagerty, P., Friedly, J., Avins, A.L., 
Nedeljkovic, 
 
S.S., Nerenz, D.R., Jarvik, J.G. (2015). Trajectories of symptoms and function in older adults with 
low 
 
back disorders. Spine, 40(17), 1352-62. 
101.  
Agency for Healthcare Research and Quality: Meta-LALC: Mata-network Learning and Research 
Center: https://pbrn.ahrq.gov/sites/default/files/docs/page/Meta-Larc.pdf (Accessed 
09/27/16). 
102.  
Certificate Program in Practice-Based Research Methods. 
 
http://www.collaborativeohioinquirynetwork.com/pbrn-certificate-program.html (Accessed 
9/27/16). 
103.  
Westfall, J.M., Mold, J., Fagnan, L.J. (2007). Practice-Based Research-“Blue Highways on the NIH 
 
Roadmap.” JAMA, 297(4), 403-406. 
104. 
Dickey, L.L., Gemson, D.H., Carney, P.A. (1999). Office System Interventions Supporting Primary 
Care 
 
Based Health Behavior Change Counseling. American Journal of Preventive Medicine, 17(4), 299-
308. 

 
V1 06/27/2018 
105.  
Carney, P.A., Goodrich, M.E., O’Mahony, D.M., Tosteson, A.N., Eliassen, M.S., Poplack, S.P., 
Birnbaum, S., Harwood, B.G., Burgess, K.A., Berube, B.T., Wells. W.A., Ball, J., Stevens, M.M. 
(2000). 
 
Mammography in New Hampshire: Characteristics of the women and the exams they receive. 
Journal of Community Health, 25(3), 183-198. 
106.  
Poplack, S.P., Tosteson, A.N., Grove, M., Wells, W.A., Carney, P.A. (2000). The Practice of 
 
 
Mammography in 53,803 Women from the New Hampshire Mammography Network. Radiology, 
217, 832-840. 
107.  
Kerlikowske, K., Carney, P.A., Geller, B., Mandelson, M.T., Taplin, S.H., Malvin, K., Ernster, V., 
 
 
Rosenberg, R., Cutter, G., Ballard-Barbash, R. (2000). Performance of Screening Mammography 
Among Women with and without a Family History of Breast Cancer. Annals of Internal Medicine, 
133 (11), 855-863. 
108.  
Carney, P.A., Harwood, B.G., Weiss, J.E., Eliassen, M.S., Goodrich, M.E. (2002). Factors 
Associated with Interval Adherence to Mammography Screening in a Population-Based Sample 
of New Hampshire Women, Cancer, 95, 219-227. 
109.  
Barlow, W.E., Lehman, C.D., Zheng, Y., Ballard-Barbash, R., Yankaskas, B.C., Cutter, G.R., Carney, 
 
P.A., Geller, B.M., Rosenberg, R., Kerlikowske, K., Weaver, D.L., Taplin, S.H. (2002). Performance 
of diagnostic mammography for women with breast signs or symptoms of breast cancer. JNCI, 
94, 1151-1159. 
110.   
Ernster, V.L., Ballard-Barbash, R., Barlow, W.E., Zheng, Y., Weaver, D.L., Cutter, G., Yankaskas, 
B.C., Rosenberg, R.D., Carney, P.A., Kerlikowske, K., Taplin, S.H., Urban, N., Geller, B.M. (2002). 
Detection of Ductal Carcinoma in Situ in Women Undergoing Screening Mammography. JNCI, 
94(20), 1546-1554. 
111.   
Carney, P.A., Miglioretti, D.L., Yankaskas, B.C., Kerlikowske, K., Rosenberg, R., Rutter, C., Geller, 
B.M., Abraham, L.A., Dignan, M., Cutter, G., Ballard-Barbash, R. (2003). Individual and Combined 
Effects of Breast Density, Age, and Hormone Replacement Therapy Use on the Accuracy of 
Screening Mammography. Annals of Internal Medicine, 138(3), 168-175. 
112.   
Miglioretti, D.L., Rutter, C.M., Geller, B.M., Cutter, G., Barlow, W.E., Rosenberg, R., Weaver, D.L., 
Taplin, S.H., Ballard-Barbash, R., Carney, P.A., Yankaskas, B.C., Kerlikowske, K. (2004). The effect 
of breast augmentation on the accuracy of mammography screening and breast cancer 
outcomes. JAMA, 291(4), 442-450. 
113.   
Vacek, P.M., Carney, P.A., Buist, D.S.M., Oestreicher, N., Barlow, W.E., Ballard-Barbash, R., 
Taplin, S.H. (2004). Biennial vs. annual mammography and risk of late stage breast cancer by age 
and breast density. JNCI, 96(24), 1832-1839. 
114.   
Poplack, S.P., Carney, P.A., Weiss, J.E., Titus-Ernstoff, L., Goodrich, M.E., Tosteson, A.N.A. (2005). 
Screening Mammography: Costs and Screening-Related Services. Radiology, 234, 79-85. 
115.   
Yankaskas, B.C., Taplin, S.H., Ichikawa, L., Geller, B.M., Rosenberg, R.D., Carney, P.A., 
Kerlikowske, K., Ballard-Barbash, R., Cutter, G., Barlow, W.E. (2005). Association Between 
Mammography Timing and Measures of Screening Performance in the US. Radiology, 234(2), 
363-373. 
116.   
Carney, P.A., Harwood, B.G., Greene, M.A., Goodrich, M.E. (2005). Impact of a Telephone 
Counseling Intervention on Transitions in Stage of Change and Adherence to Interval 
Mammography Screening. Cancer Causes and Control,16(7), 799-807. 
117.   
Carney, P.A., Goodrich, M.E., Weiss, J.E., MacKenzie, T., Poplack, S.P., Wells, W.A., Titus-Ernstoff, 
L. (2005). Utilization of Mammography in New Hampshire. Cancer, 104(8), 1726–1732. 
118.   
Rosenberg, R.D., Yankaskas, B.C., Abraham, L., Sickles, E.A., Lehman, C., Geller, B.M., Carney, 
P.A., Kerlikowske, K., Buist, D.S.M., Weaver, D.L., Barlow, W.E., Ballard-Barbash, R. (2006). 
Performance Benchmarks for Screening Mammography. Radiology, 241(1), 55-66.  

 
V1 06/27/2018 
119.   
Carney, P.A., Steiner, E., Goodrich, M.E., Dietrich, A.J., Kasales, C.J., Weiss, J.E., MacKenzie, T. 
(2006). Discovery of Breast Cancers Within One Year of a Normal Screening Mammogram: How 
Are They Found? Annals of Family Medicine, 4(6), 512-518. 
120.   
Barlow, W.E., White, E., Ballard-Barbash, R., Vacek, P.M., Titus-Ernstoff, L., Carney, P.A., Tice, 
J.A., Buist, D., Geller, B.M., Rosenberg, R., Yankaskas, B.C., Kerlikowske, K. (2006). Prospective 
breast cancer risk prediction model for women undergoing screening mammography. JNCI, 
98(17), 1204-1214. 
121.   
Carney, P.A., Goodrich, M.E., Butterly, L., Dietrich, A.J. (2006). The design and development of a 
population-based colonoscopy registry. Jnl of Registry Management, 33(4), 91-99.  
122.   
Fenton, J.J., Taplin, S.H., Carney, P.A., Abraham, L., Sickles, E.A., D’Orsi, C., Berns, E.A., Cutter, G., 
Hendrick, R.E., Barlow, W.E., Elmore, J.G. (2007). Computer-aided detection and interpretive 
performance of screening mammography. NEJM, 356(14), 1399-1409. 
123.   
Carney, P.A., Abraham, L.A., Miglioretti, D.L., Yarbroff, R., Sickles, E.A., Kasales, C.J., Geller, B.M., 
Yankaskas, B.C., Rosenberg, R.D., Buist, D.S.M., Dignan, M.B., Kerlikowske, K. (2007). Factors 
Associated with Imaging and Procedural Events Used to Detect Breast Cancer Following 
Screening Mammography, AJR, 188, 385-392. 
124.   
Butterly, L., Olenec, C., Goodrich, M., Carney, P., Dietrich, A. (2007). Colonoscopy demand and 
capacity in New Hampshire. Am J Prev Med, 32(1), 25‑31. 
125.   
Taplin, S.H., Abraham, L.A., Barlow, W.E., Fenton, J.J., Berns, E.A., Carney, P.A., Cutter, G.R., 
Sickles, E.A., D'Orsi, C., Elmore, J.G. (2008). Mammography facility characteristics associated 
with screening mammography interpretive accuracy. JNCI, 100(12), 876-887. 
126.   
Carney, P.A., Hoffman, R.M., Lieberman, D., Hornbrook, M.C., Dietrich, A.J., Klabunde, C.N. 
(2008). Data Systems to Evaluate Colorectal Cancer Screening Practices and Outcomes at the 
Population Level. Medical Care, 46(9 Suppl 1), S132-137. 
127.   
Taplin, S., Abraham, L.A., Geller, B.M., Yankaskas, B.C., Buist, B.S.M., Smith-Bindman, R., 
Lehman, C., Weaver, D., Carney, P.A., Barlow, W.E. (2010). Effect of previous benign breast 
biopsy on the interpretive performance of subsequent screening mammography. JNCI, 102(14), 
1040-1051. 
128.   
Carney, P.A., Sickles, E., Monsees, B., Bassett, L., Brenner, J., Rosenberg, R., Feig, S., Browning, 
S., Tran, K., Berry, J., Kelly, M., Miglioretti, D.L. (2010). Identifying Minimally Acceptable 
Interpretive Performance Criteria for Screening Mammography. Radiology, 255(2), 354-61. 
129.   
Fenton, J., Abraham, L., Taplin, S., Geller, B., Carney, P.A., D'Orsi, C., Elmore, J.G., Barlow, W. 
(2011). Clinical Impact of Computer-Aided Detection in Community Mammography Practice. 
JNCI J Natl Cancer Inst, 103(15), 1152-1161. 
130.   
Carney, P.A., Cook, A.J., Miglioretti, D.L., Feig, S.A., Bowles, E.A., Geller, B.M., Kerlikowske, K., 
Kettler, M., Onega, T., Elmore, J.G. (2012). Radiologists’ Use of Clinical History and it’s Impact on 
Interpretive Performance. Journal of Clinical Epidemiology, 65(2), 219-30. 
131.   
Carney, P.A., Dietrich, A.J., Freeman, D. (1992). Improving Future Preventive Care through 
Educational Efforts at a Women’s Community Screening Program. Journal of Community Health, 
17(3), 167-174. 
132.  
Carney, P.A., Bogart, T.A., Geller, B.M., Haneuse, S., Kerlikowske, K., Buist, D.S.M., Smith, R., 
Rosenberg, R.D., Yankaskas, B.C., Onega, T., Miglioretti, D.L. (2012). Association Between Time 
Spent Interpreting, Level of Confidence, and Accuracy of Screening Mammography, AJR, 198, 
970-978. 
133.   
Buckley, D., Davis, M., Andresen, E. Does a standard measure of physical disability correlate with 
clinican perception of impairment related to cancer screening? Cancer. In press. 
134.   
Davis, M., Bond, L., Howard, A., Sarkisian, C. (2011). Primary care expectations regarding aging. 
The Gerontologist.  

 
V1 06/27/2018 
135.   
Davis, M., Goubaud, M., Fagnan, L.J. (2011). Tips for small practices: Lessons learned from care 
management implementation. In: Chun A, Schwartzberg J, Irmiter C, eds. Geriatric Care by 
Design: A Clinician's Handbood to Meet the Needs of Older Adults through Environmental and 
Practice Redesign. Chicago: American Medical Association. 
136.   
Davis, M., Hilton, T., Schott, J., et al. (2010). Unmet Dental Needs in Rural Primary Care: A Clinic, 
Community, and Practice-based Research Network (ORPRN/PROH) Collaborative. Journal of the 
American Board of Family Medicine, 23(4), 514-522. 
137.   
Fagnan, L.J., Davis, M., Deyo, R.A., Werner, J.J., Stange, K.C. (2010). Linking practice-based 
research networks and Clinical and Translational Science Awards: new opportunities for 
community engagement by academic health centers. Acad Med, 85(3), 476-483. 
138.   
Fagnan, L.J., Dorr, D., Davis, M., Mahler, J., Michaels, L., McGinnis, P. Turning on the Care 
Coordination Switch in Rural Primary Care: Voices from the Practices-Clinician Champions, 
Clinician Partners, Administrators, and Nurse Care Managers. Journal of Ambulatory Care 
management. In Press, 34(3). 
139.   
Findholt, N., Michael, Y., Davis, M. (2011). Photovoice engages rural youth in childhood obesity 
prevention. Publiic Health Nursing, 28(2), 186-192. 
140.   
Findholt, N., Michael, Y., Davis, M., Brogoitti, V. (2010). Environmental influences on children's 
physical activity and diets in rural Oregon: Results of a youth photovoice project. Online Journal 
of Rural Nursing and Health Care, 85(3), 476-483. 
141.   
Heintzman, J., Marino, M., Hoopes, M., Bailey, S.R., Gold, R., O'Malley, J., Angier, H., Nelson, C., 
Cottrell, E., DeVoe, J. (2015). Supporting Health Insurance Expansion: Do Electronic Health 
Records Have Valid Insurance Verification and Enrollment Data? Journal of the American 
Medical Informatics Association, 22(4), 909-13. 
142.   
Heintzman, J., Marino, M., Hoopes, M., Bailey, S., Gold, R., Crawford, C., Cowburn, S., O'Malley, 
J., Nelson, C., Devoe, J. (2014). Using Electronic Health Record Data to Evaluate Preventive 
Service Utilization Among Uninsured Safety Net Patients. Preventive Medicine, 67, 306-310. 
143.   
Marino, M. (2012). Statistical Issues in large-scale community-based studies. Cambridge, MA, 
Harvard University. 
144.   
Marino, M. & Li, Y. (2011). Variable selection for multilevel models in the presence of missing 
data. ENAR International Biometric Society; Miami, FL. 
145.   
Marino, M., Bailey, S., Gold, R., Hoopes, M., O'Malley, J.P., Huguet, N., Heintzman, J., Gallia, C., 
McConnell, K.J., DeVoe, J.E. (2016). Receipt of Preventive Services after Oregon's Randomized 
 
Medicaid Experiment. American Journal of Preventive Medicine, 50(2), 161-170. 
146.   
Marino, M., Li, Y., Pencina, M.J., D'Agostino, R.B., Berkman, L., Buxton, O. (2014). Quantifying 
 
Cardiometabolic Risk Burden Using Modifiable Non-self-reported Risk Factors. American Journal 
of Preventive Medicine, 47(2), 131-140. 
147.   
Marino, M., Li, Y., Rueschman, M., Winkelman, J., Ellenbogen, J., Solet, J., Dulin, H., Berkman, L., 
Buxton, O. (2013). Measuring sleep: accuracy, sensitivity, and specificity of wrist actigraphy 
compared to polysomnography. SLEEP, 36(11), 1747-1755. 
148.   
Olson, R., Thompson, S., Elliot, D.L., Hess, J.A., Rhoten, K.L., Parker, K.N., Wright, R., Wipli, B., 
Bettencourt, K., Buckmaster, A., Marino, M. (2016). Results of the COMPASS cluster randomized 
controlled trial: a total worker health intervention for home care workers. In press: American 
Journal of Public Health. 
149.   
Melnick, A., Rdesinski, R., Marino, M., Jacob-Files, E., Gipson, T., Kuyl, M., Hatt, W., Crawford, C., 
Dexter, E., Olds, D. Randomized Controlled Trial of Home-based Hormonal Contraceptive 
Dispensing for Women at Risk for Unintended Pregnancy. Perspectives on Sexual and 
Reproductive Health, 48(2), 93-99. 

 
V1 06/27/2018 
150.    Darden, P.M., Gustafson, K.K., Nietert, P.J., Jacobson, R.M. (2011). Extra-immunization as a 
clinical  
Indicator for fragmentation of care. Public Health Rep, 126 (Suppl 2), 48-59. 
151.   
Mennito, S.H. & Darden, P.M. (2010). Impact of practice policies on pediatric immunization 
rates. The Journal of pediatrics, 156(4), 618-622. 
152.   
McElligott, J.T., Darden, P.M. (2010). Are patient-held vaccination records associated with 
improved vaccination coverage rates? Pediatrics, 125(3), e467-472. 
153.   
Slora, E.J., Steffes, J.M., Harris, D., et al. (2008). Improving pediatric practice immunization rates 
through distance-based quality improvement: a feasibility trial from PROS. Clinical pediatrics, 
47(1), 25-36. 
154.   
Darden, P.M., Taylor, J.A., Brooks, D.A., et al. (2008). How should immunization rates be 
measured in the office setting? A study from PROS and NMA PedsNet. Clinical pediatrics, 47(3), 
252-260. 
155.   
Darden, P.M., Thompson, D.M., Roberts, J.R., et al. (2013). Reasons for not vaccinating 
adolescents: National Immunization Survey of Teens, 2008-2010. Pediatrics, 131(4), 645-651. 
156.   
Centers for Disease Control: AFIX (Assessment, Feedback, Incentive and eXchange): 
http://www.cdc.gov/vaccines/programs/afix/components.html#ui-id-5 (Accessed 09/19/16).