Implementing Cancer Prevention Using Patient-Provider Clinical Decision Support
Phase: NA
Conditions: Cancer Prevention
Sponsor: HealthPartners Institute

 
 
 
 
Implementing Cancer Prevention Using Patient – Provider 
Clinical Decision Support 
 
 
 
IRB Protocol Version:  v1.21 
IRB Study Acronym: Cancer Wizard 
National Clinical Trial (NCT) Identified Number: NCT02986230 
Principal Investigator(s): Thomas E. Elliott, MD HealthPartners Institute 
Melissa L. Harry, PhD Essentia Health 
Sponsor: HealthPartners Institute 
Grant Number: 5R01CA193396-02 
Project Period: 03/24/2016 – 02/28/2022 
Funded by: National Cancer Institute 
 
 
 2021-02-03 
 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 1 of 29 
 
·
Implementing Cancer Prevention Using Patient – Provider Clinical Decision Support 
 
Cancer Wizard Prevention (CPW) STUDY PROTOCOL 
Revised 02.03.2021 
 
Introduction 
We thank the reviewers for their thoughtful critiques and are pleased that they judged our application to be 
well designed and very significant with likely high impact and have expert, very experienced investigators; 
great innovation; outstanding environment and resources; appropriate aims and hypotheses; exceptional cost 
assessment; and, overall, an outstanding approach with only a few weaknesses. We have revised the 
application to address the reviewers' comments. Responses are highlighted and summarized below: 
Specify a Theoretical Framework: We strongly concur and now use the Reach Effectiveness Adoption- 
Implementation Maintenance (RE-AIM) and Consolidated Framework for Implementation Research (CFIR) 
conceptual frameworks to plan, organize, and evaluate the implementation and dissemination of our 
intervention. We have added Aim 3, which uses these frameworks to guide the intervention implementation 
process and outcomes, as well as inform future dissemination strategies to other healthcare systems, clinical 
domains, and patient populations. Based on these frameworks, the dependent variables for our evaluation 
plan have been expanded, and the mixed-methods approach has been strengthened. 
Modify the Scope of Work: Despite the current USPSTF recommendation statement on lung cancer 
screening and recent decisions by CMS to pay for this in some circumstances, reviewers strongly suggested 
that lung cancer screening is not ready for widespread dissemination and implementation, and the revised 
proposal did not include lung cancer screening as an intervention target. However, lung cancer prevention 
screening has since been added back into the protocol based on USPSTF recommendations. Reviewers 
were divided on the importance of primary cancer prevention interventions (smoking cessation and weight 
management); however, we retain these within the Cancer Prevention-Clinical Decision Support (CP-CDS) 
intervention tool and will be used as secondary analytical outcomes. 
Add Behavior Change Expertise and Provide More Detailed Strategy: We carefully considered the reviewer's 
request for more patient behavior change in the research strategy. Current evidence suggests that little 
behavioral counseling for weight loss or smoking cessation is done in primary care.1,2 However, innovative 
strategies are needed for delivering these services in primary care that may include leveraging technologies 
such as this proposed CP-CDS system. We posit that EHR-linked CDS systems best serve as a means to 
identify and refer patients needing behavior change services, which follows the screen, brief intervention, and 
refer to treatment (SBIRT) paradigm.3 The proposed CP-CDS physician interface will help them use the 5A's 
(Ask, Advise, Assess, Assist, Arrange)4 to motivate patients to change and connect with evidence-based 
resources, both internally and externally. We have added Dr. Nancy Sherwood to our team, who is PhD-
trained psychologist and an expert on obesity prevention and management in children, adolescents, and 
adults. She will oversee and guide the development and evaluation of components of the CP-CDS interfaces 
that invite patients to engage in smoking cessation and/or weight management decisions and referrals, select 
their preferences, and address readiness-to-change questions and also help providers use the 5 As 
embedded in their CP-CDS interface. She will also participate in the evaluation, interpretation, publishing, 
and dissemination of study results. 
Patient-centered Approach: Note that all 36 study clinics are National Committee for Quality Assurance 
(NCQA)-certified patient-centered homes with a patient-centered, team approach to care delivery. 
Specify Approach to Adolescents: We agree that adolescents aged 9-17 need different CP-CDS approaches 
than adults. Adolescents are subject only to the HPV vaccination component of the CP-CDS, and all CP-CDS 
material and HPV vaccination decisions are channeled exclusively through a parent or legal guardian. 
Patients needing smoking cessation and/or weight management interventions must be 18 or older. Although 
adolescent obesity and smoking in adolescents aged 9-17 are very important concerns, they are beyond the 
scope of this project. 
 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 2 of 29 
 
•
SPECIFIC AIMS 
The objective of this project is to close the gap in implementation of evidence-based, personalized cancer 
prevention in rural populations, thus accelerating progress towards Healthy People 2020 goals, with 
potential to prevent or delay as many as 100,000 cases of cancer annually.7,8 To achieve this objective, we 
aim to develop and implement a sophisticated electronic health record (EHR)-linked clinical decision 
support (CDS) system in rural primary care clinics to improve the quality of primary and secondary cancer 
prevention services in those at risk for common types of cancer. Rural primary care providers (PCPs) 
especially need effective and efficient tools due to suboptimal application of cancer prevention in their 
practices. Limited patient access, constrained time, and patients who are older, poorer, and less healthy 
than patients in urban areas characterize rural healthcare.9 To achieve this objective, the Cancer 
Prevention CDS (CP-CDS) links EHR data to evidence-based cancer prevention algorithms to deliver 
point-of-care, personalized, patient-centered cancer prevention recommendations to the patient through 
the patient's PCP. The Consolidated Framework for Implementation Research (CFIR)10 will be used to 
guide how implementation is planned, organized, and conducted. The RE-AIM framework11 will be used to 
evaluate impact and the optimum processes for implementing and disseminating a CP-CDS in rural 
primary care practices, which has not previously been done. This intervention builds on a decade of work 
by our research team, is specifically designed for widespread use in primary care settings that use EHR 
systems, and could improve quality of cancer preventive care and quality of life, decrease morbidity and 
mortality, advance implementation science as it pertains to rural primary care, and help achieve Healthy 
People 2020 goals.7,8 To accomplish these ambitious objectives, the project addresses these specific aims 
and hypotheses: 
Specific Aim 1. Conduct a cluster-randomized trial that includes 36 primary care clinics (34 randomized 
clinic units, with 3 clinics randomized together due to shared providers) with approximately 300 PCPs and 
more than 153,000 patients to one of three study arms: (a) PCP-Focused Group: PCP intervention using 
an EHR-linked, Web-based CP-CDS system, (b) shared decision-making tools (SDMT) Focused Group: 
PCP intervention using an EHR-linked, Web-based CP-CDS system including additional, or (c) usual care 
(UC) to assess whether this intervention can improve the delivery of evidence-based, personalized 
primary and secondary cancer preventive care in rural primary care settings. Specifically, the PCP-focused 
group will receive a point of care CP-CDS tool that is integrated into the EHR. The CP-CDS provides 
evidenced-based recommendations to the provider and patient related to overdue cancer prevention screening 
services and eligible for smoking cessation and obesity management referrals. The SDMT-focused group will 
also receive the CP-CDS. Furthermore, the SDMT-focused arm provides SDMT to the patient and provider at 
the time of the visit. The UC arm will not receive the CP-CDS or SDMT but will receive usual best practice 
advisory suggestions for cancer prevention screening and other usual care. 
Hypothesis 1 (H1). Eligible study subjects will have significantly different rates of appropriate 
screening tests for breast, colorectal, and cervical cancer as defined by the U.S. Preventive 
Services Task Force (USPSTF) during the 12 months after the index visit by study arm, with PCP-
focused group higher than UC and SDMT-focused group higher than UC. 
Hypothesis 2 (H2). Eligible study subjects will have significantly different rates of human 
papilloma virus (HPV) vaccination in appropriate cases, as defined by the Advisory Committee on 
Immunization Practices/Centers for Disease Control and Prevention during the 12 months after the 
index visit by study arm, with PCP-focused group higher than UC and SDMT-focused group higher 
than UC. 
Specific Aim 2. Assess the cost of the CP-CDS intervention from the health system perspective through 
measurement of healthcare utilization and use of diagnostic codes and billing levels related to cancer 
prevention services. 
Hypothesis 3 (H3). After controlling for demographics and baseline clinical risk factors, eligible 
study subjects will have significantly different overall healthcare costs during the 12 months after 
the index visit by study arm, with PCP-focused and SDMT-focused arms higher than the UC arm. 
Hypothesis 4 (H4). After controlling for demographics and baseline clinical risk factors, eligible study 
subjects will have significantly different frequencies of diagnostic codes related to cancer prevention 
services and billing levels in the 12 months after the index visit by study arm, with PCP-focused and 
SDMT-focused arms higher than the UC arm. 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 3 of 29 
 
•
Specific Aim 3. Describe critical facilitators and barriers for the CP-CDS implementation process, 
outcomes, and future dissemination strategies using a mixed-methods approach supported by the CFIR 
and RE-AIM conceptual frameworks. 
Results of this pioneering project will: (a) lead to improved delivery of personalized, evidence-based 
primary and secondary cancer preventive services, helping achieve Healthy People 2020 goals;7,8 (b) 
advance our understanding of how best to integrate effective personalized cancer preventive CDS into 
primary care workflows; and (c) guide effective dissemination and implementation of similar EHR-linked, 
Web-based, personalized CDS systems to other rural primary care clinics and delivery systems. 
RESEARCH STRATEGY 
1. SIGNIFICANCE 
1.1 Gaps in Cancer Prevention Care: Cancer remains a major cause of death and disability, despite 
significant advances in cancer treatment and improved understanding of cancer mechanisms of disease 
over the last 50 years.12,13 Although the U.S. age-adjusted cancer death rates have decreased very 
slightly over the last 20 years, the absolute number of people who will be diagnosed and die of cancer will 
continue to increase for the foreseeable future.13 Cancer is the second leading cause of death in the 
United States, but in several states, such as Minnesota, it is No. 1.14 Furthermore, the counties in 
Minnesota, Wisconsin, and North Dakota served by the healthcare system that this project will engage 
have the highest cancer death rates in their respective states.15 Since the sentinel article by Doll and Peto 
in 1981, massive evidence has accumulated confirming that nearly two-thirds of cancer deaths can be 
linked to tobacco use (30%), unhealthy diet/obesity (30%), and physical inactivity (5%).16-21 Unfortunately, 
the U.S. smoking rate remains a persistent problem (42 million adults still smoke),22 adult obesity and 
overweight rates have doubled over the last 40 years (to 35%  and 34%, respectively),23-27 and fewer than 
48% of adults meet the CDC's 2008 Physical Activity Guidelines.28 Moreover, less than 10% of adults 
regularly engage in all four lifestyle practices known to reduce cancer risk: physical activity, healthy 
eating, no tobacco use, and effective weight management.17-28 Innovative approaches to address primary 
as well as secondary prevention of cancer are urgently needed. 
Most U.S. healthcare is delivered in clinics by primary care providers (PCPs), but cancer prevention often 
receives insufficient attention due to time constraints, competing priorities, lack of clinical decision support  
(CDS), PCP skepticism about patient adherence to recommendations, patient and PCP knowledge 
deficits related to cancer prevention, and complicated clinical guidelines based on an ever-evolving set of  
demographic, genetic, and behavioral risk factors.27 In this scenario, electronic health record (EHR)-linked 
CDS systems would seem to have great potential to improve care, especially if it is addressed to both 
patients  and PCPs. As early as 1992, the Institute of Medicine anticipated that EHR systems and linked 
CDS would rapidly improve chronic disease care and preventive care services.29 However, this potential 
has been only slowly fulfilled.30 With respect to cancer, the full impact of EHR-linked CDS for primary and 
secondary cancer prevention has yet to be actualized. In a recent review article evaluating CDS for cancer 
screening, only 10 articles met criteria for adequate study design and analysis.30 None of the 10 studies 
combined two or more cancer screening tests, and most (8/10) implemented only simple reminder 
systems that were not very sophisticated. Furthermore, evidence demonstrating positive effects of CDS 
on cancer prevention clinical and economic outcomes is sparse, and studies that evaluate the relationship 
of Cancer Prevention (CP-CDS) systems to clinic and PCP workflow and efficiency, configuration of office 
teams, and interactions with patients and shared decision-making are nonexistent. To the best of our 
knowledge, no CP-CDS system that combines multiple current evidence-based cancer screening tests 
(secondary prevention) with primary cancer prevention interventions (smoking, obesity, human papilloma 
virus [HPV] vaccination) has been formally evaluated in primary care settings. Furthermore, we are aware 
of no dissemination and implementation research that has studied CP-CDS in primary care and 
determined effective processes and their outcomes. 
1.2 Healthcare Disparities in Rural Populations: About 20% of Americans (60 million) live in rural areas 
characterized by older age, lower per capita income, transportation barriers, less education, poorer access 
to healthcare, lower use of cancer prevention care, and fewer physicians per capita than most urban 
areas, resulting in substantial health disparities.9 Rural PCPs need effective and efficient EHR-linked CDS 
systems to automate personalized data to reduce these disparities. 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 4 of 29 
 
1.3 Implementing EHR-linked CDS Systems to Improve Cancer Prevention: Our project implements 
and evaluates a comprehensive CP-CDS system that embraces multiple components of primary and 
secondary prevention for common cancers. It is based on previous NIH-funded projects conducted by our 
team (DK068314, HL102144) demonstrating that a specific approach to EHR-linked CDS, providing 
evidence-based CDS to both the PCP and the patient at the point of care: (a) significantly improved 
important chronic disease outcomes (blood pressure [BP] and glucose control), (b) was consistently used 
by clinic teams at 75%-80% of targeted visits, and (c) received a 95% satisfaction rating from PCPs who 
used it. This EHR-linked CDS system is designed to incorporate evidence-based factors shown in meta-
analyses to be linked to success, including (a) providing advice automatically within the clinic workflow, (b) 
providing CDS advice to both patients and providers, (c) providing monitoring and feedback on use rates, 
and (d) including healthcare team members other than physicians.31-33 
Several recent developments make EHR-linked Web-based point-of-care CDS for cancer preventive 
services feasible on a large scale for the first time: (a) over 80% of PCPs now use EHR systems, (b) we 
have shown in a randomized controlled trial that our EHR-linked, Web-based approach to CDS 
significantly improves chronic disease care in high-risk adults,34,35 (c) this CDS system requires only minor 
adjustments in clinic workflow and does not slow PCPs down, (d) Web-based algorithms can be easily 
updated and are highly scalable to both large and small healthcare delivery systems, and (e) previous 
cost-effectiveness analysis indicates that this approach to CDS is highly cost-effective and may be cost-
saving at scale.34 After adapting previous work to the needs of our new clinical target, we will randomly 
assign 36 primary care clinics (34 clinic randomization units) to 3 groups (11 intervention clinic units in the 
PCP-focused group, 11 intervention clinic units in the SDMT-focused group, and 12 usual care control 
clinic units) with approximately 300 PCPs and more than 153,000 primary care patients. The number of 
clinic units to be randomized (34) is smaller than the number of clinics in the study (36) due to three small 
clinics being grouped together. A particular focus in our project is to compare the impact of a PCP-
focused CDS workflow with a SDMT-focused CDS workflow.  
1 INNOVATION  
Innovative features of this project include: (a) extension of existing CDS technology to identify adults and 
adolescents at risk for common cancers and provide evidence-based personalized cancer  preventive 
care recommendations at the point of care (multiple times, if necessary), (b) integration of both primary 
and secondary cancer prevention interventions in a single EHR-linked, Web-based algorithmic CDS tool 
has never been done before, (c) presentation of CDS recommendations to the patient and also to 
providers, (d) inclusion of cost and provider satisfaction assessments to guide future use of the 
technology, and (e) acquisition of valuable qualitative and quantitative information needed to optimize 
dissemination and implementation of CDS systems in rural healthcare settings. 
The degree of patient involvement in the development of the tool is also innovative. We work with a panel 
of patients and consultant experts to develop, pilot test, and optimize the patient and provider interfaces. 
Once finalized, interfaces are given to the patient and PCP just before the clinical encounter and enable 
visit planning by the PCP and expression of evidence-informed and personalized treatment preferences 
by the patient. This promotes evidence-informed patient decision-making and is an exciting advance in 
patient-centered care. The 3-arm research design allows testing of the comparative effectiveness of the 
CP-CDS with (SDMT-focused) and without (PCP-focused) SDMT at the point-of-care. Other innovative 
aspects of the project include: (i) using the patient interface to efficiently elicit personalized cancer 
prevention preferences of each patient before the PCP visit, (ii) establishing clinic workflows that support 
timely provision of personalized, evidence-based CDS at the point of care, (iii) incorporating all treatment 
algorithms in a Web site rather than  the EHR to enable efficient maintenance, quality assurance , and 
updates as guidelines evolve over time, and (iv) finally, this project creates a Web-based EHR-linked CDS 
template that is very scalable and can be rapidly disseminated to the 80% of primary care practices that 
now use EHRs, enabling consistent and reliable application of evidence-based CP-CDS to large 
populations of patients. 
 
2 PRELIMIARY STUDIES AND INVESTIGATORS 
2.2 Preliminary Studies and Previous Experience: The core investigators (Drs. Elliott, O'Connor, 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 5 of 29 
 
-
-
Sperl-Hillen, and Dehmer) form an experienced multidisciplinary team with expertise in oncology, primary 
care, CDS design and interfaces, health economics, biostatistics, health behavior change, and 
implementation science (HL102144, DK079861). The Pl, Dr. Elliott, is an experienced oncologist and 
health services researcher with extensive experience in cancer prevention in rural settings.36 His 
experience includes the Lake Superior Rural Cancer Care Project (CA056334), a cluster-randomized trial 
including 18 rural communities in Minnesota, Wisconsin, and Michigan that tested interventions to improve 
cancer treatment and patient outcomes, and the Minnesota Cancer Pain Project (CA057803), a cluster-
randomized trial in 6  Minnesota cities of interventions to improve cancer pain management and patient 
outcomes. Drs. O'Connor and Sperl-Hillen are currently conducting a randomized trial evaluating CDS 
tools for adults with high 10-year reversible cardiovascular risk (HL102144), and our team previously 
completed a study showing that EHR-linked CDS improves glucose and BP control in adults with diabetes 
(DK068314). The team has published detailed, model-specific articles with respect to colorectal cancer 
and other conditons.42-46 Consultants Drs. Stange and Crabtree have extensive experience in primary care 
research, preventive services, patient outcomes, and qualitative methods, with special focus on 
dissemination and implementation research. In our many previous projects, we have developed effective 
project-management strategies, refined methods to access EHR data, developed effective provider and 
patient interfaces, addressed data privacy concerns, and developed an EHR-Web link that can generate 
sophisticated point-of-care CDS to more than 600 patients a minute. Our previous and ongoing work in 
cancer care and pain research (CA56334, CA57803, NS045361), comparative effectiveness research 
(HS019912), lifestyle interventions (AG023410, CA128211), cardiovascular care (HL102144, HL093345, 
HL090965, HL089451), and cardiometabolic care (HC95183, HS019859, DK06650, HS10639, 
DK068314), along with more than 400 publications on these and related topics, makes our team uniquely 
qualified to successfully conduct this ambitious project.47-58 
2.3 Preliminary Assessment of the Study Population: Data collected 01/01/12 to 12/31/14 
demonstrate that there will be a sufficient number of Essentia Health (EH) patients to power this cluster-
randomized trial (Table 1). 
In addition, EH PCP 
primary cancer 
prevention activity at the 
study site, based on 
data collected 01/01/14 
to 12/31/14 show: 
Referrals for smoking 
cessation services= 
4.4% (1159 
referrals/26,574 active 
smokers);  
Prescriptions for quit 
smoking drugs= 7.1% 
(1901 
prescriptions/26,574 
active smokers);  
Referrals to dieticians or 
weight management 
programs = 1.9%  
(1521 referrals /80,730 
patients with BMI >25.0 
kg/m2). 
 
 
 
Table 1. Clinical Status of Essentia Health Patients aged 11-80 with 2+ primary 
care visits within 36 months, by age group, smoking status, obesity rates, rural-
urban residency, and cancer prevention procedure*+ 
Cancer screening/ 
prevention, ages 
N patients 
eligible for 
screening* 
% Up to date on 
screening or HPV 
vaccination1 
% 
Current 
smoker 
% 
Obese2 
Rural N 
(%) 
Urban N 
(%) 
Colorectal, 50-75 
87,019 
67.1 
16.5 
45.6 
52,368 
(60.2) 
34,651 
(39.8) 
Breast, 38-74 
44,542 
65.9 
16.0 
43.5 
26,940 
(60.4) 
17,602 
(39.6) 
Cervix, 21-65 
75,420 
53.9 
21.8 
41.1 
40,028 
(53.1) 
35,392 
(46.9) 
Subjects eligible 
for HPV vaccine, 
11-261 
40,297
3 doses: 5.1 
males, 19.5 
females# 
12.8 
18.4 
20,484 
(50.8) 
19,813 
(49.2) 
*Patients with missing data are not included in denominator. Total patient population 
is significantly larger for all categories. 
+Data collection timeframe: Jan. 1, 2012 to Dec. 31, 2014 
1Patients who received cancer screening test or human papilloma virus vaccine  
#1 or 2 doses of HPV vaccinations: 9.6% males, 30.1% females 
2BMl>30.0 
NB: Lung cancer screening was not assessed in this preliminary data 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 6 of 29 
 
3 Approach 
3.2 Theoretical Framework: This project will improve our understanding of how to implement and 
disseminate CDS systems in diverse primary care settings. The Reach, Effectiveness, Adoption, 
Implementation, and Maintenance (RE-AIM)11 and the Consolidated Framework for Implementation 
Research (CFIR)10 provide conceptual frameworks to study the implementation and dissemination of the 
CP-CDS in primary care. We use RE-AIM framework to quantify the reach and effectiveness of the CP-
CDS and its potential for dissemination and scalability by evaluating CDS use rates, provider workflow, 
and patient/provider experience. The CFIR framework consists of 5 domains (the intervention, internal 
context [practice], external context [system], participants, and implementation process) that will be used to 
guide how implementation is planned, organized, and conducted. We will use key CFIR components to 
identify influences and adaptation of the implementation, direct observation of its fidelity, and evaluate its 
sustainability and prospects for wider dissemination (Specific Aim 3). This mixed-methods approach will 
greatly improve our understanding of how to successfully implement and disseminate CP-CDS and similar 
systems to other practice settings, clinical domains, and patient populations. 
3.3 Study Design: Supported by CFIR and RE-AIM frameworks, the implementation process for CP-CDS 
has four phases: a) initial organizational engagement; b) adopting and piloting the CP-CDS with Essentia 
Health (EH) PCPs and patients; c) a cluster-randomized controlled trial with 3 arms in 36 EH primary care 
clinics (34 randomized clinic units), and d) evaluation of the implementation process and outcomes. 
Because three small clinics were randomized together, there are 34 randomized units in this study. Five 
randomization schemes were developed using random numbers and 1:1:1 allocation to the three study 
arms. The randomization scheme selected was pre-determined to be the first of five schemes (blind to 
clinic allocation) with adequate balance on factors of clinic urbanicity, % of patients who were current 
smokers, % of patients under 30, total number of patient visits per clinic, % of patients with Medicaid, 
breast cancer screening rate, geographic region, count of patient with serious mental illness, count of 
patient with pre-diabetes, colorectal cancer screening rate, access to low-dose lung screening, and 
participation in a community wellness initiative. Quantification of study arm balance was not based on a-
priori cut-offs but on subjective assessment of balance by the study team.  
The two arms were randomly assigned to either the PCP-focused intervention (CP-CDS) or SDMT-
focused intervention (CP-CDS + SDMT). The 12-usual care (UC) clinics for the Cardiovascular Wizard 
studies will continue as control clinics with the CP-CDS (Figure 1) in a 1:1:1 allocation. Because three 
small clinics were randomized together, there are 34 randomized units in this study. 
Eligible patients will be allocated to the study arm 
assigned to their clinic at their index visit. 
3.4 Study Sites: This study will be conducted at 36 
Essentia Health (EH) primary care clinics in North 
Dakota, Minnesota, and Wisconsin. Leaders at EH 
are fully committed to the project, as indicated in 
letters of support from Drs. Nikcevich and Bianco. 
These 36 clinics have used the EpiCare® EHR 
since 2004 and have, on average, 12 PCPs and 
3,996 eligible patients per clinic (range, 318-
11,823). Ten participating EH clinics are classified 
as rural, 7 as micro, 8 as small towns, and 9 as 
urban by the RUCA2-UR coding system. 
Study Participants: PCPs and Patients: To 
participate, PCPs must practice at one of the 
randomly assigned EH clinics and be a general 
internist, family physician, pediatrician, or adult or 
pediatric care nurse practitioner or physician 
assistant. There are currently 398 eligible PCPs at 
the 36 clinics.  
To be included in Aim 1 analysis, patients must meet all the following eligibility criteria: (a) be 9-75 years 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 7 of 29 
 
·
·
-
old, inclusive, at an index clinical encounter with their PCP during the accrual period, (b) are not pregnant, 
not in hospice care , and have no cognitive impairment (e.g., Alzheimer’s) diagnosis codes in the last 3 
years. For each eligible patient, the study period begins on the date of the first qualifying visit (index visit) 
at which a cancer screening (or HPV vaccination) is due over the 7.5 months of study accrual (Aug 1, 
2018 – March 16, 2019). To extract Wizard CDS-related clinical data at the end of eligible patient’s study 
period, the Wizard may be re-run manually outside a visit at 12 months following the index date. To 
complete the re-run, HealthPartners will send Essentia Wizard web service IDs for Essentia staff to 
determine patient IDs and manually fire the Wizard via Epic. Data is then extracted and stored per usual 
secure Wizard data flow. Based on preliminary EH data in Table 1, we estimate that 152,800 patients will 
meet eligibility criteria. Less than 5% of patients per year switch from their EH primary care clinic to 
another EH or non-EH clinic, and losses due to death will be <1% per year. Validated algorithms 
accurately match over 98% of patients to a regular PCP. Ultimately, after accounting for all exclusions, we 
anticipate that about 525 eligible adults per PCP (149,625 total patients) will be available for analysis. 
Plans for dealing with missing data are detailed in Section 2.2.8.5. We previously requested and received 
a waiver of informed consent for patients from the Institutional Review Board (IRB) because the care 
recommendations in the CP-CDS intervention are limited to evidence-based care already recommended 
in current national and regional clinical guidelines. The IRB has waived patient consent in similar 
circumstances. 
3.5   Description of Usual Care and Intervention Conditions 
3.5.1 Usual Care Condition (Arm C): In the usual care group of the study, clinics and their PCPs will 
have no access to the CP-CDS intervention. Some existing Epic reminder systems in EH clinics do 
address cancer screening tests with reminders that are pop-up alerts, which are easily ignored. These 
alerts are marginally effective, resulting in current suboptimal rates of cancer screening tests and very low 
rates of referrals for weight management and smoking cessation and HPV vaccinations (Section 3.2). 
There is no systematic CP-CDS system in EH clinics other than what will be implemented as part of this 
project in the 2 intervention arms. Simple alerts have generally been ineffective due to many factors, such 
as alert fatigue, provider burden, and no patient engagement.30,32,59 
3.5.2 Intervention Condition: CP-CDS is rooted in a series of antecedent studies that have developed 
successful forms of outpatient CDS.45-47 Our project's intervention will have 2 experimental arms:  
PCP-focused group and SDMT-focused group. Specific steps in design and implementation of the CP-
CDS intervention are described next. 
Step 1. Identify Eligible Patients, Extract Clinical Data from EHR, and Send to Web site: In 
HL102144, we   created and validated programming to (i) extract pharmacy, laboratory, vital signs, 
demographic, comorbidity and other data from the EHR, and (ii) export these data to a secure Web site 
inside the HealthPartners Medical Group firewall. Web site algorithms will use these EHR-extracted data 
to identify eligible patients needing primary and/or secondary cancer prevention during a routine, non-
urgent primary care visit. Patients younger than 18 years will not be identified for smoking cessation or 
weight management treatment or referrals due to the need for different approaches required by 
adolescents, which is beyond the study scope. Patients 9 to 17 years old will be identified for HPV 
vaccination only if a parent is present during the clinical encounter. Patients aged 18 and older will be 
identified for cancer screening tests, HPV vaccinations, smoking cessation, and weight management, if 
eligible, according to national guidelines previously presented. To avoid firing the CDS too often and 
burdening the PCP care team, the CDS algorithms may be suppressed. 
Step 2. Create Primary & Secondary Cancer Prevention CDS Algorithms: The Web-based treatment 
algorithms used in this project are based on national guidelines, approved in advance by primary care 
clinical leaders at EH, and reflect current evidence and community standards of care.60-71 Similar 
algorithms have been tested in 2 previous research projects (DK068314, HL102144). The algorithms for 
this project will include ICD-1072 codes and will be updated (and revalidated) as required by changes in 
national guidelines. The Breast Cancer Risk Assessment Tool5 (BCRAT, NCI), Colorectal Cancer Risk 
Assessment tool6 (CRCRAT, NCI), and the Risk-based National Lung Cancer Screening Trial Outcomes 
Tool (RNOT) lung cancer risk assessment tool will compute personalized risks for these cancers, which 
can be presented during the CP-CDS encounter.  
Step 3. Collect Patient-Centered Data Using Personalized Tools: During their office visit, patients in 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 8 of 29 
 
need of specific screenings or HPV vaccination will receive patient-centered, personalized tools, 
developed by the team based on national standards, including: Breast cancer shared decision-making tool, 
colorectal cancer decision-making tool, and HPV vaccination shared decision-making tools. Patients eligible 
for lung cancer screening may also receive the lung cancer shared decision-making tool from the CDC 
Agency for Healthcare Research and Quality (AHRQ) or a tool adapted from the AHRQ tool. Only patients 
needing a cancer prevention intervention will be presented these tools and questions.  
Step 4: Identify Available Treatment Options for Cancer Prevention Needs: Web-based algorithms 
identify evidence-based prevention options that address each unmet cancer prevention need. Specific 
CDS recommendations given to a patient are based on both (a) statements from USPSTF,60,62-63,66-70 
NCl,65 CDC,64 AHRQ, and ACIP,64 and (b) the specific patient's current clinical state, including age, sex, 
smoking status, BMI, HPV vaccine status, comorbid conditions, allergies, and past screening tests for the 
3 target cancers. Cancer screening and treatment recommendations for smoking cessation, HPV 
vaccination, and obesity management are given as needed. The PCP uses the CP-CDS Active Guidelines 
or “Wizard Guidelines” to refer the patient directly to appropriate internal or community resources for 
smoking cessation or obesity management and to order HPV vaccinations and appropriate cancer 
screenings. Active Guidelines are a manually accessed EHR form designed to fit into providers' workflow 
containing actionable functions such as order sets and referrals. 
Step 5: Present Cancer Prevention Options to PCPs and Patients: Clinic Workflow:  In this project, CP-
CDS recommendations are presented to the patient directly and the PCPs using a sequence of clinic staff steps 
successfully implemented in previous studies and pretested interface formats. Participating PCPs in 
intervention clinics are trained to use the provider and patient interfaces of CP-CDS as follows: When a 
patient needing cancer prevention interventions has a clinical encounter with a PCP, the following protocol 
is automatically implemented: (i) after vital signs are entered into the EHR, CP-CDS assesses cancer 
prevention needs, identifies target patients, and provides a best practice alert (BPA). In response to a 
single click, CP-CDS displays the interface screen to the CMA within 1 second (with no additional prompts 
or triggers needed). The CMA prints the patient and PCP versions of the CP-CDS sheet, (ii) if a patient's 
mental and physical status appears stable, the CMA hands the patient sheet (example in Appendix B) to 
the patient, saying “the caution marks show how you can prevent health problems such as cancer, 
diabetes, stroke, and heart attacks. If you are ready to work on any of these things, please talk with your 
doctor during your visit today.” (iii) for both the PCP-focused and SDMT-focused groups, a printed version 
of the provider CP-CDS (example in Appendix B) is either placed in the basket outside the exam room for 
rapid review by the PCP before entering the exam room or displayed on the EHR screen with one click on 
the EHR navigator bar, depending on the PCP preference, with the SDMT-focused group also receiving 
any relevant SDMT, and (iv) in both the PCP- and SDMT-focused groups, the PCP uses the CP-CDS 
PCP interface to guide changes in cancer prevention care and uses the CP-CDS patient interface to 
reinforce patient actions. The CP-CDS will embrace the 5 A's (Ask, Advise, Assess, Assist, Arrange) to 
promote patient activation by having these elements embedded in the provider interface.4 The patient 
version may be printed as part of the After-Visit Summary. After discussion with the patient, the PCP can 
order screening tests, medications, or make referrals to internal programs or community resources to 
address smoking and/or obesity by using the CP-CDS Active Guidelines. 
Provider Interface: CDS included on the provider interface is specific and based on whether the patient 
has significant cancer risk factors or is due for a cancer screening. Both PCP- and the SDMT-focused 
groups use the provider interface. Prototype CDS algorithms are based on the recommendations 
published by USPSTF, NCI, CDC, and ACIP.60-70 They will be updated over time to ensure ongoing 
congruence with national evidence-based guidelines. Primary and secondary cancer screening 
information displayed on the CP-CDS provider interface are: smoking and obesity management options, 
medical interventions (HPV vaccine status, and screening status for breast, colorectal, lung, cervical 
cancer). All treatment recommendations are labeled as suggestions, and the interface sheets emphasize 
that this CDS does not take the place of clinical judgment or a PCP's detailed knowledge of a particular 
patient. The provider interface is a powerful visit-planning tool that most PCPs prefer to view in print just 
before entering the exam room. 
Patient Interface: A simple visual approach is preferred, because patients may have low levels of 
numeracy and misinterpret probabilistic information. A visual display of recommended lifestyle 
modifications (smoking cessation, weight reduction/counseling), medical interventions (HPV vaccination, 
medications for tobacco cessation), and screening tests for the 4 target cancers are included, depending 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 9 of 29 
 
on the needs of the patient identified by the CDS algorithms. Similar visual patient interfaces we used in 
earlier studies have been well received by most adult and teen patients,73-74 accommodate low numeracy, 
and have been shown in studies by others to be a strong motivational strategy.75-79 However, the 
prototype interfaces shown in the Appendix B will be substantively modified to address key issues related 
to cancer prevention care, with extensive input from our adult and teen representatives from EH Patient 
Councils and extensive pilot testing before randomization.  
Four SDMTs: For patients in need of breast, lung, or colorectal cancer screen or who are not up to date 
on HPV immunizations, four separate SDMT tools will be available. Each tool will include five components: 
1) overview of screening; 2) screening options; 3) benefits of tests; 4) risks; and 5) how do I make a 
decision? 
Breast cancer screening: We created our own tool: a static, printable SDMT for women in need of 
screening ages 35 to 74. 
Colorectal cancer screening: We created our own tool: a static, printable SDMT for men and women 
in need of screening ages 50 to 75. 
HPV vaccination: We created two separate tools: a static, printable SDMT for parents of children in 
need of HPVV ages 9 to 17, and a static, printable SDMT for adults in need of HPVV ages 18 to 26. 
Lung cancer screening: We will use one of two tools: a tool developed by HealthPartners based on 
decision aids from the AHRQ: a static, printable SDMT for adults with a 30-year pack habit ages 55 to 
80.  
Step 6. PCP Activates CP-Active Guidelines: The Active Guidelines, or “Wizard Guidelines”, is built into 
the EHR and provides options for orders and referrals based on the wizard recommendations. The CP-
Active Guidelines will contain information on clinic- and community-based resources for smoking 
cessation and weight management services, counselors, and programs with locations, contact 
information, program type, and delivery methods. 
Step 7: Iterative Use of Cancer Prevention CDS over Series of Visits: A key design feature of this 
intervention is its repeated use at all routine office visits of eligible patients in intervention study arms A & 
B. Pilot data indicate that nonpregnant eligible adults will average 3-6 primary care visits during the 
anticipated 33- month accrual and follow-up periods.87-88 After the CP-CDS is activated during the accrual 
period, subsequent activations may be limited to once every 120 days to minimally disrupt clinic 
workflows. In a previous study (HL102144), rates of use of a cardiovascular CDS system by consented 
and non-consented PCPs were 70%-80% of targeted care visits. CP-CDS gives updated treatment 
suggestions for patients and PCPs to consider at each visit, because CDS evolves as the patient's clinical 
status, lifestyle, and cancer prevention needs change over time. 
4.6  Implementation of the Cancer Prevention CDS Intervention 
Throughout the implementation of the CP-CDS, we will conduct key informant and patient interviews, 
surveys, usability testing, and continuous quantitative and qualitative feedback between researchers 
and participants to measure the implementation processes and outcomes as recommended by the 
CFIR and RE-AIM frameworks. Organizational Engagement: In Phase 1, we will engage EH clinic 
leadership and managers, informatics personnel, PCPs, CMAs, and patients through meetings to 
identify potential influences of implementation of this project. Pilot Testing: All CP- CDS algorithms and 
interfaces will be extensively pilot-tested among stakeholders in Phase 2 to adapt the 
implementation. Representatives from EH Patient Councils will evaluate the patient interfaces to 
maximize patient-centeredness. We will then recruit two EH clinics not in the study and pilot test CP- 
CDS in eligible patients for 4 weeks. We will conduct in-depth interviews with up to 10 PCPs at the two 
pilot clinics pre- and post-pilot, asking probing questions about shared decision-making and shared 
decision-making tools. After further modification of the CP-CDS, the project will enter Phase 3. EH 
Patient Advisory Councils: EH has 18 volunteer patient advisory councils that engage patients and 
families as advisors, mentors, and educators. The main goal of this program is to improve the patient 
and family experience and quality of healthcare. Research team members will recruit representative 
members to regularly review and critique project pilot phase activities, CDS interfaces, survey tools, 
interview objectives and results, and project deliverables. This existing infrastructure is ideal for 
introducing CDS technology to primary care, gaining the reaction and input of patients and families, and 
refining adaptation of the intervention. 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 10 of 29 
 
Implementation of the Cluster-Randomized Trial (Phase 3): We will train intervention clinics to use CP-
CDS using strategies similar to those EH routinely uses to inform clinic teams of changes to the EHR. 
They include face-to-face group or individual meetings with all intervention clinic PCPs, CMAs, and clinic 
staff, plus email reminders with links to a short instructional video demonstrating CMA and PCP roles in 
CP-CDS use. Training will be completed and CP-CDS fully implemented at the 24 intervention clinics 
within 30 working days of the project "go-live" date.  
Strategies to Ensure PCP Care Team Use of the Intervention: Following implementation, all intervention 
clinic staff will receive monthly email reports showing CP-CDS use rates. Study team personnel will have 
ongoing communication with the intervention clinic nurse managers throughout the duration of the 
intervention period to assess continued use of CP-CDS and to gather feedback ensuring real-time 
observation of implementation fidelity. Support from EH leaders and clinic PCP leaders, plus monitoring 
and feedback, will help maximize provider adherence to CP-CDS study protocols, as demonstrated by 
CDS use at 75%-80% of targeted visits in previous projects.  
To further encourage use of the tool, clinics will be offered a best practice lunch and learn for printing at 
goal. If a clinic achieves four non-consecutive months at or above 75% print rate, the Wizard team will 
provide a catered project-related meeting. The Wizard team will provide a meeting agenda for the clinic 
manager and/or nurse supervisor with the expectation that notes from the meeting will be returned to the 
Wizard team. This lunch and learn will also be offered at 10 non-consecutive months at or above 75% 
including the previous four.  
Additional quality assurance (QA) will be conducted to ensure appropriate functioning of the Wizard 
algorithms, which determine clinical recommendations made by the Wizard to the provider and patient. 
The QA will require manual review of select patients’ electronic medical records. Key study personnel at 
Essentia Health will be responsible for the review. Manual chart review may also be required during 
implementation if necessary for QA of Wizard algorithms. Information reviewed during QA will not be 
retained for study analyses. 
Evaluation of Implementation Process and Outcomes (Phase 4): During the cluster-randomized trial, 
components of the RE-AIM and CFIR frameworks will be assessed to determine mediators of CP-CDS 
implementation, use, clinical outcomes, and future strategies for dissemination and implementation. 
4.7 Definition and Measurement of Dependent Variables 
We will use a mixed-methods approach to determine the facilitators and barriers for achieving the specific 
aims. Appendix A contains the definitions, sources, and metrics for study variables. 
Cancer Screening Tests (Aim 1, H1): The dependent variable for H1 is a binary variable indicating that 
the patient is up to date on the specific constellation of screening tests for breast, cervical, lung and 
colorectal cancer that were due at the index visit for the patient by 12 months after the index visit. 
Secondary endpoints for H1 will look at the same individual screenings, within 12 months of the index visit. 
Each patient will be classified as up to date or not (composite endpoint) on all appropriate screening tests, 
depending on sex, age, risk factors, and date of last needed screening tests according to active USPSTF 
recommendation statements. This variable will be based on EHR-captured procedure codes that are 
routinely gathered by the Wizard interface at actual and re-run visits, which manually force the capture of 
this data into the Wizard repository, and data will be obtained from the index visit date through 12 months 
past the last patient index date. 
HPV Vaccination Rates (Aim 1, H2): New CDC recommendations put into effect at Essentia Health 
January, 2017 are reflected in changes to the HPV vaccination sample and recommendations in this 
study. Children ages 9-14 are now recommended to receive 2 HPV vaccine doses, while 3 HPV vaccine 
doses are still recommended for adolescents and young adults ages 15-26. The dependent variable for 
H2 is a binary variable indicating that the patient has had all 2 or 3 HPV vaccinations recommended by 
ACIP/CDC, dependent on age group, within 12 months of the index visit. This variable will be coded as "1" 
for patients having all 2 or 3 vaccinations and "0" for patients with 0-2 vaccinations, dependent on age 
group. HPV vaccination data will be obtained from the EHR from the index visit date through 12 months 
past the last patient index date. 
12-month Health Care Costs (Aim 2, H3): Costs for this analysis are defined as intervention costs as 
well as the incremental medical care costs associated with the intervention from the health system 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 11 of 29 
 
•
perspective. 
Intervention costs include CDS implementation and maintenance, training, and incentives but exclude 
intervention research and development costs. Medical care costs include costs of all medical services- 
including laboratory, physician services, and screening tests-incurred in the 12-month post-index date 
period by participants in each study group, as indicated by EH billing and clinical encounter data. 
Described in detail in the Section 4.9.3 below, we will use relative value units (RVUs) and diagnosis-
related groups (DRG) to calculate inpatient and outpatient costs for patients randomly assigned to each 
study group and standard accounting methods to measure the cost of the CP-CDS intervention. 
Emergency visits and hospitalizations may be too infrequent in the study sample to accurately predict a 
population-wide impact of the study interventions. If there were no intervention impact on these utilization 
components during the study period, including their costs would substantially increase variance in the cost 
data without adding to the accuracy of the cost assessment. Therefore, we will first assess whether there 
is a differential impact between CP-CDS and usual care on use of these services and will include their 
costs only if a difference is observed. Reliance on EH billing records for measuring medical care use may 
miss costs incurred in other health systems; however, this opportunity is expected to be equal across 
randomized study arms, and cross-system medical utilization is expected to be relatively limited in the 
primarily rural study population. 
While market prices generally are a good estimate of the costs for medical services, the paid amount in this 
claims system is specific to HealthPartners at a particular time and may provide a biased view of costs between 
pre- and post-intervention periods. To address this, we will use Total Care Relative Resource ValuesTM 
(TCRRVs), which are a nationally standardized set of measures that have been endorsed by the National 
Quality Forum and are derived from Centers for Medicare and Medicaid Services (CMS) relative value units 
(RVUs).  TCRRVs extend CMS RVU measures to include utilization categories, such as laboratory services and 
medications, which do not have CMS RVU weights.  Specifically, TCCRVs defined during the midpoint of 
patient accrual and follow-up (e.g., 2019) will be used to convert claims to represent U.S. dollars. 
Health System Revenue (Aim 2, H4): Use of ICD-10 diagnostic codes and CPT-4 billing codes during 
primary care visits in study clinics will be used to predict health system revenue collection. ICD-10 diagnostic 
codes include those for breast, colorectal, cervical and lung cancers; HPV vaccinations (3 codes: up to 3 
vaccinations), obesity, and smoking. ICD-10 and/or CPT-4 codes for mammography, colonoscopy, stool-based 
tests for colorectal cancer, low dose CT lung scans, Pap test/cytology with and without HPV co-testing, referral 
to smoking cessation counselor, prescription of smoking cessation drugs (multiple), referral to weight 
management program, referral to dietician, referral to bariatric surgery program. Coding complexity of clinical 
encounters will be measured by billing levels for evaluation and management services during clinic visits.  
Specifically, complexity levels will be measured using CPT-4 codes for new patients, from 99201 (level 1, 
low complexity) to 99205 (level 5, high complexity), and established patients, from 99211 (level 1, low 
complexity) to 99215 (level 5, high complexity). 
Facilitators and Barriers of Implementation (Aim 3): For this analysis, we use the CFIR and RE-AIM 
frameworks to assess implementation process and outcomes.10-11 RE-AIM metrics are: a) percent and type of 
patients reached, b) for whom the intervention was effective, c) percent of clinics and providers that adopted 
the intervention, d) consistency and cost of intervention implementation, and e) proportion of intervention 
components and effects maintained. CFIR metrics include PCP perceptions of CDS source, strength and 
quality of the evidence base, iterative adaptability and testing of the CDS and usability testing; external 
policies and incentives affecting implementation; implementation climate and readiness; PCP 
engagement; fidelity of implementation and modifications made; and PCP knowledge, attitudes, beliefs, 
and self-efficacy about the CDS. Detailed descriptions of these metrics are in Appendix A. 
Our mixed-methods approach includes: a) semi-structured interviews conducted with 8-20 EH leaders and 8-20 
PCPs and CMAs during Year 1 to understand organizational support and culture to guide CDS implementation  
and Year 4 to plan future dissemination; b) meeting minutes from organizational engagement and study 
implementation, c) patient interviews during Years 2-4 (patients recruited from the CDS arms) to learn 
about their experiences with the CDS and perceived barriers and facilitators to act on the CDS 
personalized recommendations; d) interviews with PCPs and CMAs at intervention clinics as part of key 
stakeholders interviews in Year 2 and interviews with PCPs and CMAs at intervention clinics in Years 3-4 
to learn how to improve CDS use, effectiveness, and dissemination to other clinics; e) pre- and post-pilot 
in-depth interviews with up to 10 PCPs in pilot clinics; f) cross-sectional PCP surveys conducted in Year 1 
before randomization (pretest) and Year 4 (post-test) among PCPs in each study arm (approximately 400 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 12 of 29 
 
total) to assess perceptions of primary practice systems related to cancer screening and prevention, 
including PCP demographics, experience, and use of CDS;95 g) in Year 4, PCPs in Arms A, B, & C will 
complete the 10 item System User Scale (SUS, Appendix C) to assess their experience with the CDS, as 
well as questions presented in the baseline surveys;96 and h) patient surveys in Years 2-4 on a cohort of 
up to 100 from each arm (approximately 300 total) to assess their experience with cancer prevention and 
screening care in their primary care clinics. In addition to questions on primary (tobacco cessation, weight 
management, and HPV vaccination) and secondary (breast, cervical, colorectal, and lung) cancer 
prevention and screening care at Essentia Health, the survey will contain demographic questions, 
questions adapted from the Clinic and Group Consumer Assessment of Healthcare Providers and 
Systems (CG-CAHPS, Appendix C)97 and from the 10-item Decisional Conflict Scale (O’Conner, 1993, 
revised 2005). Patient consent will be implied by survey completion. Patient survey eligibility criteria 
include being eligible for the survey during their CP-CDS index visit (or comparable visit at the study 
control clinics) and completing the survey. Those patients that complete the survey will be offered their 
choice of a mailed $10 gift card to Target or an emailed e-gift card to Walmart. Patients will be mailed a 
letter on the study, a paper survey, and a $2 incentive approximately15 days from their index visit. 
Patients who do not respond to the mailed paper survey may be called up to 7 times by the Survey 
Research Center at HealthPartners Institute, where they will receive an option to take the survey over the 
phone. All surveys will be administered and tracked from the Survey Research Center. See the CPW 
Patient Survey Protocol_2018-06-29_FINAL.pdf document for complete details. 
Dependent Variables for Secondary Analyses: In Appendix A, primary and secondary dependent 
variables are defined, some of which assess potential moderators of intervention effects. Secondary 
analyses include (a) rural-urban comparisons, (b) separate binary outcomes for the 3 screenings tested as 
a composite in H1: colorectal, breast, and cervical cancer, (c) lung cancer screening, (d) a binary outcome 
of any HPV vaccinations vs. none within 12 months of the index date, and (e) binary indicators for referrals 
to internal or community programs for , smoking cessation, or medications prescribed for smoking 
cessation, (f) smoking cessation by 12 months following the index date. 
4.8 Measurement of Independent Variables Primary Predictor: The primary predictor is the treatment 
arm to which a clinic is randomized. This variable will be coded as a two degree of freedom contrast with 
the usual care (UC) arm as the reference category. Planned contrasts will examine pairwise differences in 
treatment arms. Patient and Provider Characteristics: Patient and provider characteristics listed in 
Appendix A will be documented so we can assess the extent to which results apply to subgroups of 
patients or whether patient or PCP characteristics modify intervention efficacy. Patient sex will be an 
important covariate and stratifying factor in the analysis because screening rates for H1 are likely to be 
different by sex. Clinic randomization may introduce random or selection-induced patient or PCP covariate 
imbalance, necessitating adjustment. Patient characteristics obtained from the EHR, include 
demographics, pre-intervention comorbidities (derived from dated ICD-10 diagnosis codes),  insurance 
status,  smoking status,  among others. Furthermore, primary care visit dates will link patients and PCPs. 
We will have complete data for PCP characteristics, including age,  sex, race, ethnicity,  physician or allied 
provider (i.e., nurse practitioner), department.  
4.9 Analysis Plan 
4.9.1 Aim 1 Analytic Approach 
Hypotheses 1 & 2 in Aim 1 posit that patients seen at clinics with the SDMT-focused or PCP-focused 
cancer screening CDS will be more likely than patients seen at UC clinics without the CDS to have 
appropriate screening for breast, cervical, and colon cancer (H1), and a full course of three HPV 
vaccinations (H2) within 12 months of an index visit. Because clinics are the unit of randomization and the 
outcome varies at the level of the patient, generalized linear mixed-model regression with a logit link and 
binomial error distribution will be used to test the effect of the interventions. The general form of the 
analytic model for H1 & H2 is: 
Dependent Variablekji = Yooo + Y100SDMT_CDS k + Y200PCP_CDSk + Yoo1SEXi + [VkQO + Ukj0 + ekji] 
The dependent variable in each model is a binary indicator of appropriate screening (H1) or complete course of 
HPV vaccination (H2). The dependent variable is predicted by the two fixed-effect terms representing the study 
arm contrast (SDMT_CDK and PCP_CDS). Patient sex is included as a fixed effect because screening rates for 
H1 and vaccination rates for H2 are likely to be different. We will screen patient characteristics to determine 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 13 of 29 
 
whether they differ across study arm and include patient covariates in the model when they are unbalanced by 
treatment arm. Random terms are included for clinic (vkoo), provider (ukjo), and patient (ekii). The provider random 
term will be removed if the provider intra-cluster correlation (ICC) is less than 0.005. A significant treatment 
group effect (P<0.025) and positive and significant parameters for the treatment group terms Y100 and Y200 for 
each hypothesis will support the H1 & H2 predictions that clinics with CDS interventions are more effective 
than clinics without them for increasing cancer screening and HPV vaccination rates. This model will 
include patient sex and patient variables unbalanced by study arm as well as the indicated random effects. 
Treatment group heterogeneity by sex will be assessed by including interaction terms between patient sex 
and the two study arm indicators.  
4.9.2 Aim 1 Sample Size Justification 
Hypothesis 1: Using a preliminary data pull of patient visits from 2011-2013, we estimate there are 9390 
male patients (3133 per study arm) aged 50-75 with clinic visits over 18 months who are not up to date on 
colon cancer screening and are associated with the estimated 75% of providers who will take part in the 
study. We estimate there are 33,264 female patients (11,088 per study arm) with clinic visits over 18 
months who are not up to date on all three screens for colorectal cancer (ages 50-75), breast cancer (ages 
50-74, not ages 38-74), and cervical cancer (ages 21-65) and are associated with the estimated 75% of 
providers who will take part in the study. The proportion of men up to date with colorectal cancer screening 
is 60%, and the proportion of women up to date on all  
three cancer screens is 34%, yielding a weighted pooled  
average of 40%. Among those not up to date on screening  
and thus study eligible, the anticipated pattern of effects for  
up-to-date screening rates is shown in Table 2  
(11% UC, 18% PCP CDS, 24% SDMT CDS).  
The effective sample size for the analysis is reduced to 
484-1754 per arm due to anticipated ICCs of 0.005 to 0.02, reflecting the clustering of patients within clinics. 
With this range of effective sample sizes and 10 clinics per arm, we will have 80% power (α2=0.05,  
R square of screening with other covariates from 0 to 0.1) to detect a minimum detectable difference of being 
up to date on screening 18 months after patient index visits of 11% in patients seen in UC clinics and 14%-18% 
for patients seen in clinics assigned to the PCP CDS study arm. This minimum detectable difference is for the 1 
df contrast comparing UC and PCP-focused CDS arm clinics. A larger difference in proportion of patients 
screened is expected between UC and SDMT-focused CDS arm clinics, so these detectable differences are 
adequate for this comparison. 
Hypothesis 2: From the preliminary data pull, we estimate that 
21,277 patients (7092 per study arm) aged 11-26 with clinic  
visits over 18 months have not received all 3 HPV vaccinations 
and associated with an estimated 75% of providers. The  
proportion of patients with all 3 HPV vaccinations is currently  
19.5% for females and 5.1% for males (12% pooled). Among  
those without all 3 HPV vaccinations and thus study eligible, the anticipated pattern of effects for receipt of 
all 3 HPV vaccinations is shown in Table 3 (20% UC, 33% PCP CDA, 40% SDMT CDA). The effective 
sample size for the analysis is reduced to 468-1562 per arm due to anticipated ICCs of 0.005 to 0.02, 
reflecting the clustering of patients within clinics. With this range of effective sample sizes and 10 clinics 
per arm, we will have 80% power (α2=0.05, R square of HPV vaccination with other covariates from O to 
0.1) to detect a minimum detectable difference of completion of all 3 HPV vaccinations at 18 months 
following patient index visits of 20% (pooled female and male) in patients seen in UC clinics and 26.0%-
31.3% for patients seen in clinics assigned to the PCP CDS study arm. Therefore, we anticipate 
adequately powered minimum detectable effect sizes in the anticipated pattern of effects for a range of 
ICCs and contributions of other variables such as patient sex to the prediction of the study endpoint. 
4.9.3 Aim 2 Analytic Approach 
12-Month Utilization (H3): Incremental medical costs will be estimated using standard health 
econometric methods. A generalized estimating equation (typically assuming a gamma distribution and log 
link function) will be used to estimate costs by study arm while allowing clustering by clinic and controlling 
for demographics and baseline clinical risk factors.90-94 The marginal effect of being assigned to an 
intervention clinic will provide an estimate of the incremental medical cost associated with the CP-CDS 
intervention. Urban and rural clinic- based eligible subjects receiving care from eligible PCPs will also be 
Table 2. Anticipated pattern of effects for 
H1 (composite of appropriate screening) 
UC 
11% (12% F, 7% M) 
PCP CDS 
18% (19% F, 14% M) 
SDMT CDS 
24% (25% F, 20% M) 
Table 3. Anticipated pattern of effects 
for H2 (receipt of 3 HPV vaccinations) 
UC 
20% (30% F, 10%M) 
PCP CDS 
33% (45% F, 20% M) 
SDMT CDS 
40% (55% F, 25% M) 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 14 of 29 
 
compared. The estimates assume that the CP-CDS is implemented in a large health plan with an EHR capable of 
exporting data to Web-based clinical algorithms. Implementation in settings without this capability would likely 
incur additional costs. 
12-Month Diagnostic and Billing Codes (H4): Similar to costs, a generalized estimating equation with a time 
× study group interaction term will be used to estimate use rates of diagnostic and billing codes for cancer 
preventive services or the encounter billing level by study arm while allowing for clustering by clinic and 
controlling for demographics and baseline clinical risk factors. The marginal effect of being assigned to an 
intervention clinic will provide an estimate of the incremental effect associated with the intervention. All 
statistical comparisons will be 2‐sided. P‐values of less than .05 will be considered statistically significant. 
4.9.4 Aim 3 Analytic Approach: Measures of centrality will be computed for each of the RE-AIM metrics. 
Mixed models will access changes in patient- and provider-reported outcomes with distributional and 
model modifications made as needed for each outcome. For provider and patient experience survey 
metrics, we will describe the proportion of respondents indicating responses for specific items at each time 
point and change over time for these items and composite scales. Based on the distribution of underlying 
scale data and item responses, parametric or nonparametric analyses will be used to examine 
relationships between items, scales, and other study outcomes. Qualitative analysis of interviews will be 
conducted using a content analysis process of iterative data reduction. Information will be extracted and 
organized into categories, themes, and patterns that emerge using grounded theory methods.205 We will 
use an open coding scheme followed by axial coding, then selective coding to develop themes and 
concepts. As appropriate, the CFIR framework will also be used in developing a coding frame, part of 
qualitative content analysis. After coding is complete, we will use descriptive data analysis to examine the 
weight and intensity of categories by their repetition within and across interviews, using counts and 
frequencies, using qualitative analysis software to facilitate these analyses.206 
We may descriptively compare provider baseline and follow-up survey results on a study arm level. This will 
involve utilizing provider’s assigned clinic from the baseline and follow-up survey administration data (stored at 
HealthPartners) and linking surveys responses to an assigned study arm. HealthPartners will share baseline 
and follow up survey results, not including study clinic, with Essentia for data analysis. Since comparisons are 
study arm level, no individual identifiers will be associated with the results. 
4.9.5 Missing Data: All analytic variables will be derived from EHR or administrative databases in which it 
is rare for care delivery information to be incompletely recorded, even if done outside EH, at each visit. 
Thus, most missing data will be considered missing completely at random. Several additional factors will 
effectively minimize missing data. The data source for the primary endpoints is a re-run of the automated 
Wizard process, which gathers EHR data at the time of an encounter. The re-run conducted 12 months 
following the index date will provide an EHR snapshot in time of diagnoses, procedures, test results, 
prescriptions, and referrals.  The rural setting of the study provides few care alternatives for patients for 
obtaining cancer screenings and HPV vaccinations beyond their primary care clinic. Finally, few patients 
switch from one EH primary care clinic to another. These factors all help to ensure that patients who will 
be coded in the analysis as lacking a cancer screening or HPV vaccinations are in fact lacking these 
events and have not obtained this care elsewhere. The planned likelihood-based analysis using all 
available data will ensure accurate parameter estimation, assuming the data are at least missing at 
random (MAR). 
4.9.6 Secondary Analyses:  In secondary analysis, the analytic model described for Aim 1 will also be 
used to examine each of the screening rates making up the H1 endpoint separately (colorectal, breast, 
cervix) as well as lung cancer screening. The analysis for H2 will test a secondary binary outcome of any 
HPV vaccinations vs. none received within 12 months of the patient index date. Additional secondary 
outcomes will include binary indicators for referrals to smoking cessation programs, medications 
prescribed for smoking cessation, and smoking status 12 months following the index date. To examine 
treatment heterogeneity, the analytic model for Aim 1 will be augmented with a clinic-level indicator of rural 
vs. urban clinic and study arm by rural/urban interaction terms to test the effect of clinic setting on 
endpoints used in H1 & H2 and to test for differences in treatment arm effects in different clinic settings. 
Patient-level covariates by study arm interaction terms will be included to test differences in treatment arm 
effects by number of patient encounters, insurance status, and patient gender.  
4.10 Data Sources, Data Quality and Data Management for Hypothesis Tests and Analysis: We will 
follow established data management procedures developed and refined sequentially in previous projects 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 15 of 29 
 
-
·
-
·
•
(DK068314, HL102144, HL115082). A relational database to store patient-level data extracted from 
administrative and EHR (Clarity) databases has been developed, linking information across service 
settings to a specific individual on a specific date using a fixed patient identifier. More detail on data 
standardization, data validation, and data security is in Appendix C.  
4.11 Organization of Project: Dr. Elliott will lead weekly meetings of the research team to ensure that 
tasks are completed according to study protocol, communicate with the IRB and DSMB, and oversee 
budgetary and administrative aspects of the project. Team members: Drs. Elliott, Saman, Harry, O'Connor, 
Sperl-Hillen, Asche, and project managers at EH and the Institute.  
4.12 Strengths and Limitations of the Study: Valid EHR data are key to this project, and some data 
may be missing or inaccurate. However, dates of all mammograms, colonoscopies, and PAP/HPV tests 
and smoking history are updated by CMAs at each office visit and are easily retrieved to inform both the 
CDS and for use in the analyses. In previous studies, others and we have validated EHR-derived data 
elements needed in the analysis,47-58 and our study endpoints use easily validated data. Budgetary 
constraints limit the primary cancer prevention intervention strategy to HPV vaccination, with secondary 
analysis of smoking cessation. These potential limitations should be weighed against the strengths of this 
ambitious, timely, and innovative project. National and regional data show dramatic deficits in preventive 
cancer care, especially in rural areas; the counties in this study are ranked in the bottom half of the 
counties in Minnesota, North Dakota and Wisconsin in health status and outcomes.15 Unfortunately, EHR 
systems have failed to deliver consistent clinical benefits in outpatient settings.30,59 However, over a series 
of NIH-funded projects, we developed a scalable EHR-linked, Web-based CDS system that significantly 
improved important chronic disease outcomes.47-58 We now adapt this CDS system to cancer preventive 
care and posit that directing provider and patient attention to cancer prevention at the point of care will 
improve delivery of cancer prevention services in rural primary care clinics. Our 3-arm design also 
compares the effectiveness of workflows with and without SDMTs to engage patients in cancer preventive 
services; previous studies suggest better outcomes when clinic staff initiate discussion of preventive care 
services before the PCP encounter.31,56,59 Economic analyses are an additional study strength. Other 
strengths include use of RE-AIM and CFIR conceptual frameworks, our expertise in CDS programming 
and evaluation, and the size of EH's rural population in three states. 
4.13 Dissemination and Future Plans: Using the CFIR and RE-AIM frameworks for reporting, the results 
of this project will be presented at national scientific meetings and in peer-reviewed journals. However, our 
main dissemination goal is to spread CP-CDS use widely to other primary care practices and delivery 
systems based on our findings and further informed by the implementation conceptual frameworks. Drs. 
Elliott, O'Connor, Sperl-Hillen, Sherwood, Stange, and Crabtree are ideally positioned to do this, as they 
are national leaders in primary care and quality improvement. 
5 PROTECTION OF HUMAN SUBJECTS 
5.1.1 Risks to Subjects 
Human Subjects Involvement and Characteristics: Potential study subjects include about 300 adult 
and pediatric primary care providers (PCPs) at Essentia Health (EH) in Minnesota, Wisconsin, and North 
Dakota. PCPs must practice at an EH randomized study clinic. 
Limited clinical data from automated diagnostic, utilization, pharmacy, vital signs, laboratory, and other 
EHR and administrative databases will be collected for patient study subjects for specified periods of time 
and will be used to (a) identify eligibility for the study and inclusion in the analysis, and (b) assess the 
impact of study interventions on implementation of evidence-based care recommendations. 
Sources of Materials: All necessary data to determine study eligibility, implement and operate the 
intervention, and test the study hypotheses are derived from (a) electronic medical records (EHRs), (b) 
health plan or medical group administrative databases, and (c) interviews and surveys of all participating 
PCPs and CMAs. 
Potential Risks: Risks to PCP study subjects who take part in interviews or surveys are considered 
minimal and principally involve consideration of the risk of violation of confidentiality of study data. If 
confidentiality were breached and quality of care were seriously out of range for one or more PCPs, EH 
leadership could conceivably use this information to release one or more PCPs from employment with EH. 
Therefore, no identifying information on individual PCP performance with respect to the clinical domains 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 16 of 29 
 
addressed in this study or any other aspect of care gathered as part of this research project will be made 
available to EH leaders who make employment, compensation, or disciplinary decisions. Furthermore, this 
research project will not alter the existing EH policy of using quality-of-care assessments to evaluate the 
performance of PCPs or other providers. 
Potential risks to study subjects who are patients include the possibility that the intervention may provide 
clinical decision support (CDS) advice to PCPs on the basis of the national evidence-based guidelines, 
which may be inappropriate for a given individual patient and, if applied without further checking the 
clinical status of a given patient, could lead to erroneous therapy, adverse events, disability, or death. 
However, the clinical recommendations are evidence-based and operationalize current national and 
regional standards of care and, therefore, the risk of untoward consequences of such clinical actions is 
considered minimal. Moreover, this potential risk is routinely present in every clinical encounter in the 
healthcare system. Additional risks to patients are also minimal and include principally the risk of violation 
of confidentiality. Measures to minimize these risks are also discussed below. 
5.1.2. Adequacy of Protection against Risks 
Recruitment and Informed Consent: We previously requested and received a waiver for PCPs 
(physicians, advanced practitioners, nurses conducting wellness checks) and CMAs to take part in 
interviews for this study. 
We have requested, and received, a waiver of informed consent for PCPs (physicians, advanced 
practitioners, nurses conducting wellness checks) and CMAs to participate in the study because the care 
recommendations in the CP-CDS intervention are limited to evidence-based care already recommended 
in current national and regional clinical guidelines. We have also requested, and received, a waiver of 
written informed consent for patients for evaluation of the CDS intervention for the following reasons: (a) 
All treatment options included in the Cancer Prevention CDS algorithms are based on current national 
guidelines and, therefore, the care recommendations provided conform to current standards of care and 
ought not to represent any risk to patients beyond the routine risk that all patients assume whenever they 
have contact with the medical care system. (b) At intervention clinic training sessions and on Cancer 
Prevention CDS displays in the EMR, we emphasize that medical and/or lifestyle treatment suggestions 
are intended only as suggestions, not as a mandate, and that it is inappropriate for a PCP to follow any 
such treatment suggestion without further checking the clinical status of a given patient. (c) It would be 
impractical to consent patients (due to large numbers) and impossible to answer the primary research 
questions (due to selection effects related to consent) if written informed consent of patients were 
required. 
Protection against Risk: The following measures will be taken to protect patients who take part in 
surveys or interviews from the risk of breach of confidentiality: A unique study ID code unrelated to the 
medical record number will be assigned to each patient study subject and used to link data from various 
sources and needed for analysis. A crosswalk table linking this code number to a patient medical record 
number will be destroyed within 12 months after completion of analyses needed to test study hypotheses.  
The following measures will be taken to minimize the risk that a PCP will act wrongly on the basis of 
information provided through Cancer Prevention CDS developed for this study: Each project-related 
communication to providers will include a written explanation indicating that the CDS is a suggestion, not a 
mandate, and that the action should only be taken if judged to be clinically appropriate by the treating 
provider on the basis of the patient's health, previous healthcare, current treatment, and other factors. 
5.1.3 Potential Benefits of the Proposed Research to Study Subjects and Others 
No claim is made in communications with PCP study subjects that any personal benefit will accrue from 
participating in this project. 
PCPs will have no defined benefits from participating in this project. However, the Cancer Prevention CDS 
intervention is designed to optimize the application of primary and secondary cancer prevention 
interventions. Some PCPs who are repetitively exposed to this new and potentially useful clinical 
information may use it to improve their clinical care during the study or afterward.  
Patient study subjects will have no defined personal benefit from participating in this project, and most 
patients will receive no compensation. Other than interviews, no communication between research team 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 17 of 29 
 
members and study subject patients is planned as part of the study protocol. Although some patients may 
receive better cancer prevention services as a result of this intervention, no claim of clinical benefit to an 
individual patient can or will be made. 
5.1.4 Importance of the Knowledge to be Gained 
If the intervention significantly improves primary care with respect to application of primary and secondary 
cancer care in people 9-80 years old, the risks of occurrence of the 4 target cancers and other cancers 
may be reduced later in life for large numbers of patients. If the interventions fail to significantly improve 
primary and secondary cancer prevention in eligible subjects, that knowledge will also be important 
because it will direct the attention of investigators to other potentially more fruitful lines of investigation. 
Thus, regardless of specific findings, the results of this trial will provide important new knowledge that may 
ultimately contribute to improved care for adolescents and adults with regard to cancer prevention. 
Collaborating Sites: HealthPartners Institute for Education and Research (the Institute) is the applicant 
organization with the required expertise to conduct this study. The Institute has assembled a research 
team with a 10-year history of successful research using clinical decision support tools and cluster-
randomized trial methods. This study will be conducted at Essentia Health, the performance site with a 
research team based at Essentia Institute of Rural Health. Institutional subcontracts are anticipated 
between the Institute and EIRH. Also, consultants from other organizations have key roles in the study. 
Special Populations: The study will not preferentially recruit any special populations. All patient study 
subjects will be 9-75 years of age at study entry, and none will be older than 76 years at study end.  
Patients aged 9-17 years will be exposed only to the HPV shared decision-making tools but not the 
smoking cessation and weight management recommendations, because these recommendations require 
different approaches than adults and are out of scope for this project. This group may include some 
subjects with mental health conditions of various types. However, it is important to systematically address 
cancer prevention care in this population, because such patients may have elevated cancer risk and 
because such patients often have been excluded or underrepresented in previous research studies. 
5.1.5. Data Safety and Monitoring Plan 
The data safety and monitoring board is being replaced by an Independent Project Safety Officer (IPSO), 
Dr. Cynthia Gross, a biostatistician and professor with the University of Minnesota, who we will show data to 
and who will identify any problems.  
The IPSO will provide input and guidance on the study evaluation and intervention protocols, including 
quality assurance and safety issues related to the protocols and intervention strategy, as well as data-
handling activities. The IPSO will provide periodic input via email, conference calls, and annual meetings.  
Consistent with NIH and Institute policy, the IPSO member will not be affiliated with HealthPartners or 
Essentia Health. 
A special focus of interest will be the safety of patients exposed to the study intervention. The intervention 
provides point-of-care CDS to PCPs and patients at study entry with evidence-based cancer prevention 
care options based on national guideline recommendations and further vetted by clinical leaders at EH 
and outside consultants who have agreed to participate in this project. Clinical decision support 
recommendations provided as part of the Cancer Prevention CDS intervention are designed to support 
clinicians' decision-making, not to override clinical judgment. 
Adverse-events information will be collected by the study coordinator and recorded on standard forms 
based on those used in other trials (HL102144). Consistent with NIH and Essentia Health IRB policies, all 
adverse events will be promptly reported in writing to the NIH, the IPSO, and Essentia's IRB. 
HealthPartners Institute will register this study on ClinicalTrials.gov (NCT02986230) and post results of the 
study when available, consistent with HP and EH research policies. For ClinicalTrials.gov results reporting, the 
Essentia study team will provide to the HealthPartners study team participant all-cause mortality data 
(occurrence of death due to any cause), extracted from EHR (Clarity) databases. 
5.2.1. Inclusion of Women 
The EH PCP population is roughly 52% female, while eligible adults seen in primary care clinics at EH are 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 18 of 29 
 
about 72% female due to uneven distribution of the 3 target cancers in the 9- to 80-year-old population, 
where females account for about 98% breast cancer and for 100% of cervical cancer. All eligible adult 
study subjects will be included, without regard to gender, race, or ethnicity. The demographics of both 
overall study populations (providers and patients) are further described in the enrollment tables. 
5.2.2 Inclusion of Minorities 
The EH PCP population is about 4% minority race and about 1% Latino ethnicity. The providers included 
in this project are about 8% minority race and about 1% Latino ethnicity. The study-eligible adolescent and 
adult patient population includes about 9% minority race, with roughly 0.3% Latino ethnicity; these figures 
also mirror the regional population demographics. Native American and Alaska Native patients are about 
3% of the EH- served population. 
All eligible study subjects will be included without regard to gender, race, or ethnicity. The demographics 
of both overall study populations (providers and patients) are further described in the human subjects 
enrollment tables. To the degree that race or ethnicity is associated with need for cancer prevention care, 
the proportion of minorities among study subjects may be somewhat higher than the proportion of 
minorities in the overall adult patient population.

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 19 of 29 
 
5.2.3. Inclusion of Children 
All study subjects will be aged 9 to 75 years, inclusive, on their index date. Inclusion of adolescents (aged 9-17 
years) is justified because the intent is to study methods to improve Human Papilloma Virus (HPV) vaccinations in 
this age range. The ACIP/CDC recommends HPV vaccination to adolescents and young adults aged 9-26 years 
to prevent HPV infections and, consequently, cervical cancer and other cancers directly related to HPV. 
Therefore, we are enrolling adolescents in this study, but they will be exposed only to the HPV vaccination 
recommendation. Also, their parent or guardian, as a requirement of participation, will accompany all patients 
younger than 18 years. Patients younger than 18 will not be exposed to the overweight/obesity or smoking 
cessation CDS interfaces. Cancer screening tests for breast, colorectal, or cervical cancers do not apply to 
children. 
6 RESOURCES AND DATA-SHARING 
Sharing of study procedures and outcomes is an essential element of this research proposal. We are determined 
to ensure that data sharing occurs on a local, regional and national level. Our plan includes the following: 
6.9 Local: We will work closely with clinical and administrative leaders at EH to ensure that our CP-CDS tool will 
be well accepted and locally relevant. During the intervention and analysis phases of the project, we will continue 
to meet with these leaders to update them on our findings. If desired, once the intervention period is complete, we 
will activate the CP-CDS at all EH primary care clinics. Findings of our research will be presented locally at the 
HealthPartners Celebration of Education and Research and to local EH clinical and administrative leaders. 
6.10 Regional: To ensure statewide availability and dissemination of results and interventions, Dr. Elliott is an 
influential regional and national opinion leader in cancer care. In addition, findings of this research will be 
presented at regional conferences such as the Minnesota Health Services Research Conference and others. 
Results will also be communicated to the leaders within the Minnesota Department of Health and to other regional 
and statewide medical groups through the Institute for Clinical Systems Improvement, a regional shared-learning 
quality-improvement organization of which HealthPartners and Essentia Health are members. 
6.11 National: Our main study findings will be presented at national meetings and published in peer-reviewed 
journals. Our outside consultants, Drs. Kurt Stange and Benjamin Crabtree, will help with national dissemination 
of our findings. In addition, our CP-CDS system is developed as a Web-based application to facilitate its adoption 
outside of EH. If the funding agency desires, and if permitted under then-current law, at the conclusion of the 
funding period, we may provide a deidentified data set to NCI (federal funding agency) at its written request for 
the use of other qualified researchers in the future. 
 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 20 of 29 
 
 
Appendix: Table of Contents 
Implementing Clinical Decision Support for Cancer Prevention in Primary Care 
 
 
Appendix A:  List of Dependent and Independent Variables .......................................................... 23 
Appendix B:  Prototype Clinical Decision Support Interfaces for PCP & Patient ............................. 28 
Appendix C:  Cancer Prevention CDS Function Operations .......................................................... 29 
Appendix D:  Qualitative Research Plan: Surveys and Interviews .................................................. 30 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 21 of 29 
 
Appendix A.  List of Dependent and Independent Variables. 
Figure A.1. Variables for Analysis including Description, Data Source, and Classification. ICD-9 
diagnosis codes will be replaced with ICD-10 diagnosis codes in 2015. Blue shaded variables are 
CFIR related metrics 
 
Variable 
Description 
Data Source 
Variable 
Classification 
Dependent Variables 
Up-To-Date Cancer 
Screening 
A binary variable indicating the patient is up to 
date on screening tests for breast, cervix, and 
colorectal cancer by 12 months following the 
index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
Up-To-Date Colorectal 
Cancer Screening 
A binary variable indicating the patient is up to 
date on colorectal cancer screening by 12 months 
following the index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
Up-To-Date Breast 
Cancer Screening 
A binary variable indicating the patient is up to 
date on breast cancer screening by 12 months 
following the index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
Up-To-Date Cervical 
Cancer Screening 
A binary variable indicating the patient is up to 
date on cervical cancer screening by 12 months 
following the index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
All HPV Vaccinations 
Complete, age 18-26 
A binary variable indicating the patient has had all 
HPV vaccinations as recommended by ACIP/CDC 
within 12 months of the index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
Some HPV Vaccinations 
Complete, age 18-26 
A binary variable indicating the patient has had at 
least 1 HPV vaccination as recommended by 
ACIP/CDC within 12 months of the index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
All HPV Vaccinations 
Complete, age 9-17 
A binary variable indicating the patient has had all 
HPV vaccinations as recommended by ACIP/CDC 
within 12 months of the index visit. 
EHR 
Procedure 
Results 
Binary 
Lung cancer screening 
A binary variable indicating the patient is up to 
date on lung cancer screening by 12 month 
following the index visit. 
EHR 
Procedure 
Results from 
Wizard re-run 
Binary 
Any quit attempts 
A binary variable indicating that the patient made 
any smoking quit attempts in the 12 months 
following the index date 
Wizard EHR 
Binary 
Smoking status at 12 
months 
A binary variable indicating that the patient has a 
non-smoker smoking status at 12 months 
following the index date 
Wizard EHR 
Binary 
Referral to Smoking 
Cessation Programs 
A binary variable indicating that a referral was 
provided to a patient for internal or community- 
based smoking cessation programs within 12 
months of the index visit. 
EHR 
Procedure 
Results 
Binary 
Prescription of Smoking 
Cessation Medications 
 
A binary variable indicating that a prescription was 
made for smoking cessation medications within 12 
months of the index visit. 
EHR 
Procedure 
Results 
Binary 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 22 of 29 
 
Generic and brand 
prescription names 
Generic and brand names of prescriptions 
ordered for smoking cessation 
EHR 
Procedure 
Results 
Categorical 
Dates of prescription 
orders 
Dates of orders of prescriptions for smoking 
cessation 
EHR 
Procedure 
Results 
Date 
CDS Usability Testing 
Provider post-intervention period: 
Systems User Scale 
Proportion indicating highest satisfaction 
User Survey 
Provider 
Survey 
Interval 
Patient Visit Experience: 
Access to Care, Provider 
communication, Office 
staff, Provider rating 
Pre and multiple post intervention time points. 
CAHPS Adult Visit Questionnaire 2.0 
Proportion indicating highest satisfaction 
Description of change in proportion over time 
Patient Survey 
Interval 
Provider Perceptions of 
Clinic Care Systems: 
Decision support, 
Clinical Information 
Systems, Self-
management support, 
Delivery System 
Redesign, Health Care 
Pre- & Post-Intervention time points: Provider 
Practice Connections Readiness Survey 
PPC-RS. Proportion indicating highest satisfaction 
Description of change in proportion over time. 
Provider 
Survey 
Interval 
Provider knowledge, 
attitudes, & beliefs of 
cancer prevention care 
Pre- & Post-Intervention time points Change over 
time. 
Provider 
Survey 
Interval 
Independent Variables 
 
 
 
Study Arm 
Control, PCP-focused CDS, CMA-focused CDS 
Administrative 
Nominal 
Patient Sex 
Male or Female 
Wizard EHR 
and 
Administrative 
Nominal 
Patient Age 
Years 
Wizard EHR 
and 
Administrative 
Interval 
Patient Race 
Standard categories 
1 
Wizard EHR 
and Clarity 
Nominal 
Patient Ethnicity 
Standard categories 
EHR Clarity 
Nominal 
Smoking Status 
Indicator variables for current, former, or never 
smoker. Used as a descriptive variable and to 
select patients for analysis. 
Wizard EHR 
Nominal 
Medicaid Status 
Insurance status. Medicare, Medicaid, 
Commercial, Other, None 
Administrative 
Nominal 
Coronary Heart Disease 
Status 
Indicator variable for CHD comorbidity based on 
one or more inpatient or outpatient ICD-9 codes 
410.xx-414.xx or 429.4 (To be updated to ICD-10 
codes.) 
Wizard EHR 
Diagnosis 
Codes,  
Nominal 
Diabetes Status 
Indicator variable for diabetes mellitus based on 
inpatient and outpatient diagnosis codes, filled 
prescription for a glucose-lowering medication 
other than metformin or TZD, and/or laboratory 
tests that indicate diabetes; or a combination of 
these.  Validated in DK06650 and DK068314. 
Wizard EHR 
Diagnosis 
Codes 
Nominal 
Hospice Care 
In a defined period of time, one or more HPMG 
special codes for Hospice Care on EHR Problem 
List.  Exclusion criteria for study eligibility. 
Wizard EHR  
Nominal 
Alzheimer’s Disease 
Status 
Indicator variable for Alzheimer’s disease based 
on one or more inpatient or outpatient ICD-9 
codes 
Wizard EHR 
 
Nominal 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 23 of 29 
 
Number of Office Visits 
Outpatient primary care visits, in a defined period 
of time. 
Wizard EHR 
 
Interval 
 
 
CDS Activations 
Count of CDS activations within 12 months of the 
patient index date within the intervention arms 
CDS 
Interval 
Index date for patients age 
9-17 
First date on which a patient age 9-17 has a 
primary care visit and is due for HPV 
vaccination 
EHR 
Date 
BPA fire for patients age 9-
17 for HPV vaccination 
Indication that the “needs HPV vaccination” 
Essentia Health BPA fired at the index 
encounter 
Administrative 
Binary 
Clinic ID 
Clinic identification number needed for 
randomization and for use as a random effect in 
the analysis. 
Wizard EHR 
and 
Administrative 
Nominal 
Dates of HPV vaccination 
prior to index date 
Dates of all HPV vaccination prior to the index 
date for patients age 9-17 
EHR 
Procedures 
Date 
Dates of HPV vaccination 
after index date 
Dates of all HPV vaccination in the 12 months 
following the index date for patients age 9-17 
EHR 
Procedures 
Date 
Clinic Location: Rural vs. 
Urban 
Based on RUCA2-UR codes 
Administrative 
Nominal 
Provider Type 
Primary care physician, CMA, other. Used to 
determine provider study eligibility. 
Administrative 
Nominal 
Provider Department 
FP, IM, Med/Peds, Other 
Administrative 
Nominal 
Provider Age 
Years 
Administrative 
Interval 
Provider Gender 
Male or female 
Administrative 
Binary 
 
 
Figure A.2 Protocol for Cancer Prevention-Clinical Decision Support Implementation Metrics 
Using the RE-AIM Framework (Tan Shaded) 
Domain 
Measure 
Data Source 
Reach 
Identification rate among patients –Number and proportion of 
patients identified as needing cancer prevention interventions. 
EHR prescription, patient 
surveys, & Smart Form data 
[Intervention only]) 
Representativeness of patients –age, gender and clinical 
characteristics of CDS clinic patients exposed and not 
exposed to the CP-CDS intervention 
EMR data 
Effectiveness 
Proportion who received a cancer prevention intervention 
(includes SA #1, H1 & H2) 
EHR data 
24-month adherence to a cancer prevention intervention (SA 
#1, H1 & H2) 
EHR data 
Proportion of physician practice patterns meeting evidence- 
based recommendations of the CP-CDS 
EHR data (CDS & UC clinics), 
CP-CDS Smart Form use 
Proportion of patients reporting participation in weight or 
smoking counseling or programs 
Patient surveys 
Adoption 
(Intervention 
Clinics) 
Proportion of primary care clinics agreeing to participate and, 
within clinics, proportion of primary care providers consenting 
to participate 
Meeting minutes, consent 
forms 
Representativeness of settings – Proportion of potential clinics 
and providers who participate; differences in their urban/rural 
location, size, specialty, etc. 
Essentia clinic & provider 
data 
Implementation 
(Intervention 
Proportion of patients needing cancer prevention identified 
with the ‘Best Practice Alert’ (BPA’s) among all identified 
EHR data 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 24 of 29 
 
Clinics) 
CP-CDS rates of use, defined as number of times used in 
eligible patient visits, divided by number of eligible patient 
Web-service data 
Print use of CP-CDS interface 
Web-service print data, 
provider & patient surveys 
Use rates of CP-CDS Smart Form 
EHR and Web-service data 
Maintenance 
Percent Intervention clinics continue using CP-CDS at 24 
months 
Meeting minutes, web-service 
data 
Percent primary care providers using the CP-CDS at 24 
months 
Web-service data 
Organizational commitment to sustaining the CP-CDS 
Provider focus groups, key 
informant interviews of 
leaders 
Implementation of CP-CDS in non-Intervention primary care 
clinics, including spread beyond usual care clinics 
Key informant interviews with 
organizational leaders, 
training of non-intervention 
 
Figure A.3 Request for Supplemental Clarity Data, Provider Attribute Data, Utilization Data 
(see “CPW Data Dictionary” excel table attached in iRIS) 
Purpose  
Obtain clarity data for patients in the denominators for H1, H2, Lung, Smoking 
 
-Descriptive data elements not available in wizard (e.g., insurance status) 
-Endpoints not available in wizard (e.g., referral to smoking cessation) 
 
 
Obtain provider variables for description and secondary analysis 
-Examples: provider type, dept, age, gender 
 
 
Obtain RUCA codes for the 36 clinics in study 
 
-Develop a lookup table or tie location from each encounter to a RUCA code 
 
Dehmer cost aims 
 
-See “Cost” tabs in data dictionary for this aim 
 
Secondary (if resources): HPV vaccination analysis 
 
-Identify sample 
 
-Identify index visit date, clinic 
 
-Gather patient description (e.g., gender) 
 
-HPV vaccination dates prior to index date 
 
-HPV vaccination dates in year after index date 
 
Samples 
1) H1 / H2 / Lung / Smoking sample: patients eligible for any of these denominators because they are due 
for cancer screening, HPV vaccination, or smoking interventions over the 7.5 mo sample accrual period: 
Aug 1 2018 – March 16 2019. Will later do 15 mo accrual period of index dates of 8/1/18 – 10/31/19 
-38053 wizard service calls (WSC) “Index 7_5 mo stacked” 
-Contains study_id, patient_id, Index_VisitDt, visit_provider, visit_location, flags for subsample 
eligibility (H1 / H2 / Lung) 
-The smoking sample will be a subset of the H1 / H2 / lung denominator. No separate pulls are 
needed for such patients. 
-Some patients show up in the file more than once if they are eligible for more than one subsample 
denominator. Such patients are included in the file multiple times because they sometimes have 
different index visit dates and visit_provider, and they are analyzed separately. 
 
1) Dehmer cost analysis denominator 
-38053 WSC “Index 7_5 mo stacked” covering the 7.5 month index date period 
-For the analysis this file will be limited on our end to just the first WSC per patient 
 
2) HPV sample: patients age 9-17 due for HPV vaccination 
 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 25 of 29 
 
Time frame 
-Some variables are the status as of the patient index date 
-Some variables are static (e.g., provider type, patient ethnicity) 
-The two smoking-related endpoints examine occurrence of an event over 12-month time period following 
the index date 
-Cost variables include information from the index visit as well as utilization over a 12-month time period 
 
Fields to extract and return: 
 
For Sample 1 (H1, H2, Lung, Smoking):  
-Variables and definitions listed in data dictionary in the “Austin variables” tab 
 
-For each record in Index 7_5 mo stacked, please attach the following variables: study_id, patient_id, 
ethnicity, insurance status, RUCA code of clinic at index visit, provider type, provider dept, provider age, 
provider gender (if readily available), referral to smoking cessation counseling 
 
-Attaching prescription orders might go as follows: In a separate file, for all records in “index 7_5 mo 
stacked”, provide all prescription orders from 8/1/18 – 3/16/20 with separate fields for NDC code (if 
available), prescription order date, generic name, brand name 
 
 
For Sample 1 (Dehmer utilization analysis): 
-See “Cost” tabs in data dictionary for this aim 
-There are four tabs (yellow) requesting data elements 
 
For Sample 2 (HPV sample: patients age 9-17 due for HPV vaccination) 
 
- Variables and definitions listed in “CPW data dictionary” in the “HPV” tab 
 
 
 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 26 of 29 
 
Appendix B.  Clinical Decision Support Interfaces for  
Primary Care Provider (Figure B.1) and Patient (Figure B.2). 
These are example screen shots of the interface design in R01CA193396 for adults overdue for cancer screenings. 
 
B.1  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
B.2 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 27 of 29 
 
  
Appendix C. Cancer Prevention CDS Function Operations 
 
Step 1: Web-based cancer prevention clinical decision support (CDS) algorithms identify eligible patients needing 
primary and/or secondary cancer prevention interventions. This automated web-based process will identify adult 
patients at the point of care at in-person office visits. The display is limited to once every four months to avoid 
overburden. 
Cancer Prevention Interventions 
Primary: 
Smoking 
any current use, both sexes, ages 18 – 80 
Obesity 
BMI >25 kg/m2, both sexes, ages 18 – 80 
HPV vaccination 
3-part series not complete, both sexes, ages 9 – 26 
Secondary: 
Breast cancer (mammography)  
females, ages 50 - 74  
Colorectal cancer (endoscopy, FIT)  
both sexes, ages 50 - 75  
Cervix cancer (pap test, HPV test) 
females, ages 21 - 65 
Step 2: Web-based CDS extracts clinical data from EHR and sends to secure website (inside HP firewalls) to populate 
appropriate risk prediction tools for breast cancer and/or colorectal cancer and/or cancer prevention guideline-based 
templates. 
Patient-centered, personalized tools and shared decision making: 
a. Breast cancer risk assessment tool (BCRAT): uses NCI’s 8 item tool. Data elements that exist and are 
complete in the EHR are auto populated into the CDS decision tools interface and presented to women 
age 50 to 75. A link to the Weill-Cornell breast cancer tool is also available in the risk assessment section 
of the CPW CDS. 
b. Colorectal cancer risk assessment tool (CRCRAT): uses NCI’s 15 item tool. Data elements that exist and 
are complete in the EHR are auto populated into the CDS decision tools interface and presented to men 
and women age 50 to 74 
c. Prioritizing cancer prevention needs: when a patient has two or more needs an CDS algorithm will be used 
that will rank these needs based on no risk, some risk, and high risk: 
> Smoking cessation will always be ranked #1 for current smokers, 
> Breast cancer and colorectal cancer screening will be ranked according to the risk score 
computed from the above tools, 
> HPV vaccination will be often ranked #1, or #2 if smoking, because it does not compete with breast and 
colorectal cancer screening, and will be ranked ahead of Pap test 
> Obesity will be categorized as no risk for BMI = 20-25, some risk for BMI = >25-30, and high risk for BMI 
>30. 
d. Patient education materials for breast, colorectal, cervical, lung cancer, and HPV for adults are available 
in the decision aids section of the CPW CDS. The materials are designed to provide information to 
patients and elicit shared decision making. 
Step 3: During rooming process web-based algorithms identify all evidence-based prevention options needed by 
each patient based on their personalized data and recommendations from USPSTF, NCI, CDC, and ACIP. 
a. After CMA completes rooming procedure and enters data in the EHR, a BPA alerts CMA that the 
patient has cancer prevention needs. 
b. With a single click on the best practice advisory the cancer prevention CDS displays the interface screen to 
the CMA that contains both provider and patient versions (each one-page displays). 
c. Both interface screen displays can be printed for use during the visit by the provider/CMA and the patient 
version given to patient to take home along with the after-visit summary (AVS). Abbreviated patient 
education materials are also printed for the associated cancers. Both versions will be shown on the screen 
in the exam room for provider and patient to discuss and make decisions. Full versions of education 
materials are available from the online version. 
d. Patient and CMA/provider discuss either displayed or printed recommendations, patient’s readiness to act or 
discuss and preferences, then proceed to decision-making. 

 
 
CPW Main Study Protocol v1.21 02.03.2021  
 
Page 28 of 29 
 
Appendix D.  Qualitative Research Plan: Surveys and Interviews 
(individual study protocols will be developed for each aspect) 
 
 
Survey tool, subjects 
Method 
Time 
Purpose 
Semi-structured interviews 
EH PCPs* & leadership  
Qualitative 
EH Team 
Year 1 
Year 4 
Learn organizational support & 
culture  
Develop future plans 
Patient interviews 
Arms A & B 
Qualitative 
EH Team 
Years 2-4 
Learn about experiences & 
barriers & facilitators to act 
PCP interviews 
Arms A & B 
Qualitative 
EH Team 
Years 2-4 
Learn how to improve CDS use, 
implementation & 
effectiveness/efficiency 
Cross-sectional PCP surveys  
3 Arms 
Quantitative 
e-survey 
Year 2 (pre-test) 
Year 4 (post-test) 
PCP Survey 
Arms A, B, and C,  
Quantitative 
e-survey 
Years 2-4 
Assess CDS experience 
Patient survey  
3 Arms 
Quantitative 
e-survey4 
Years 2-4 
Assess experience with cancer 
prevention care, access to care, 
provider communication & PCP 
rating. 
 
 
 
*PCP = Primary Care Provider 
1PPC-RS = Physician Practice Connections Readiness Survey 
2SUS = System User Scale 
3CAHPS = Consumer Assessment of Healthcare Providers and Systems 
4CAHPS survey triggered automatically 15 days after index visit and after an 18-month visit 
Patients age <18 years are excluded from surveys and interviews