Enhanced, EHR-Facilitated Cancer Symptom Control (E2C2) Trial
Phase: NA
Conditions: Hematopoietic and Lymphoid Cell Neoplasm, Malignant Solid Neoplasm
Sponsor: Mayo Clinic

 
 
 
 
 
E2C2 / 18-007779 
 
 
Enhanced, EHR-facilitated Cancer Symptom Control (E2C2) 
Pragmatic Clinical Trial 
 
 
NCT03892967 
 
 
Document Date: 03/04/2025 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 1 of 32 
 
IRB Minimal Risk Protocol Template  
 
General Study Information 
 
Principal Investigator:   Andrea L. Cheville, MD, MSCE 
 
 
 
 
 
 
 
 
Study Title:    Enhanced, EHR-facilitated Cancer Symptom Control (E2C2) trial 
 
Protocol version number: March 4, 2025 Version 24 
 
Research Question and Aims 
 
Hypothesis: Collaborative care management for control of moderate or worse sleep disturbance, pain, physical function 
impairment, anxiety, depression, fatigue (SPPADE symptoms), and physical dysfunction among cancer survivors and 
patients with cancer will improve quality of life, symptom severity, and adherence to cancer treatment, and will also 
reduce need for acute care. 
 
Aims: 
Specific Aim 1.  Conduct a cluster randomized pragmatic trial with a stepped wedge design to test the hypothesis that a 
symptom control-focused E2C2 intervention will significantly reduce SPPADE symptom and physical dysfunction scores ; 
reduce unplanned hospitalizations, emergency department (ED) visits, and intensive care unit (ICU) admissions; improve 
adherence to cancer therapies; and improve self-reported quality of life. 
H1a. SPPADE symptom scores will improve more, relative to prior scores, for patients who have been exposed to the 
intervention when compared with the improvement of patients that are not exposed. 
H1b. Among patients who report moderate or worse symptoms, SPPADE symptom scores will improve more for patients 
who have been exposed to the intervention when compared with the improvement of patients that are not exposed. 
Specific Aim 2. Evaluate the hypothesis that use of a multifaceted, evidence-based implementation strategy to support 
adoption and use of the E2C2 system will result in improvements in implementation and clinical outcomes. 
Specific Aim 3. Conduct a mixed methods evaluation to detect, understand and reduce disparities in the adoption and 
implementation of the E2C2 intervention among elderly and rural-dwelling patients with cancer.  
Background:   
Cancer and its treatment are often associated with severe, disabling symptoms that have been causally linked to 
diminished survival, increased healthcare utilization, degraded quality of life (QoL), unemployment, and non-adherence 
to recommended cancer treatments. The prevalence of inadequate symptom control has been reported to be as high as 
90%1 despite the availability of definitive care guidelines and accurate and efficient assessment tools, and is especially 
stark among the elderly and rurally-situated patients. This disjuncture is needless as most symptoms can be 
substantiallymitigated.2,3  
Reports suggest that two key factors have impeded efforts to achieve better symptom control through heightened 
clinical awareness.  First, the expectation that symptom screening and score reporting alone would lead to better 
management has been repeatedly refuted. Meta-analyses and high quality trials have failed to show that solely 
providing clinicians with patient reported outcome (PRO) scores improves outcomes.4-7  Second, symptoms are 
challenging to manage, even for experts with dedicated time. Success often requires diligent monitoring; multi-modal 
care plans spanning behavioral, pharmacological, and rehabilitative domains; as well as ongoing adjustment.  These 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 2 of 32 
 
demands are particularly salient for the most prevalent and destructive symptoms of the “SPPADE” hexad; Sleep 
disturbance, Pain, Physical function impairment, Anxiety, Depression, and Energy deficit.  Although guidelines exist for 
each symptom class, their enactment by disease-focused clinicians, who may lack both the training and time necessary 
to coordinate treatments, has proven unfeasible. 
Collaborative care (CC) approaches that include a multidisciplinary team-based patient care strategy with symptom-
specific goals, support for self-management training sustained patient follow-up, and decision support for medication 
changes, have considerable potential to improve the management of symptoms in patients with cancer8,9 and the 
potential to reduce healthcare disparities among vulnerable patient subgroups. A robust evidence base supports the 
effectiveness of CC approaches in mitigating anxiety and depression; meta-analyses and systematic reviews report 
positive effect sizes ranging from 0.34 to 0.61 in primary care settings,10-15 and among patients with co-morbid medical 
conditions.16,17  Additionally, more recent CC iterations have been shown to significantly improve pain in diverse 
populations, including patients with cancer.18,19  It should be noted that virtually all positive trials of PRO-based symptom 
monitoring and treatment in cancer populations have incorporated key CC components.20-21   
CC approaches have proven difficult to implement at scale, yet recent information technology advances permit their 
core components to be simulated through integrated systems embedded in current generation electronic health records 
(EHRs). Specifically, EHR internal logic and patient interface capabilities can provide automated, “stepped care” decision 
support that integrates patient and clinical data to match the aggressiveness of management with a symptom’s intensity 
and impact.  Patients with mild to moderate symptoms may achieve relief through self-management education or 
simple, first-line interventions.  Embedded EHR decision support rules can systematically identify these patients and 
automatically push the appropriate educational materials to them, or assist in the initiation of prescriptive therapies. 
Some patients, by virtue of the number, intensity, and functional impact of their symptoms, will require dedicated 
specialist attention.  This narrowing of the focus of CC providers to intensely symptomatic patients who fail first-line 
treatments can permit the optimal deployment of limited resources. 
We propose to comprehensively evaluate the implementation and effectiveness of a guideline-informed Enhanced, EHR-
facilitated Cancer Symptom Control (E2C2) system that uses two empirically supported levels of stepped care for 
improving control of SPPADE symptoms, low-touch automated CC for mild to moderate symptoms and conventional CC 
for more intense symptoms. We will conduct a Hybrid II stepped wedge cluster randomized pragmatic clinical trial at the 
population-level among patients with solid tumors across all phases of cancer care in community settings and clinics 
within an academic medical center. Malignant hematology (liquid tumor) patients will also be included in community 
practices. Successful completion of this trial will provide evidence regarding the impact of the E2C2 intervention on the 
management of SPPADE symptoms. 
The E2C2 intervention utilizes critical insights from the current evidence base to overcome barriers that have limited 
previous attempts at the development of scalable, pragmatic approaches to symptom management. Three recent 
findings are particularly relevant.  First, we now know that simply providing symptom patient reported outcome (PRO) 
data to overloaded, disease-focused clinicians limitedly improves clinical outcomes, as robustly attested by systematic 
reviews.4,22,23 24-26 Second, applying the Collaborative Care Model (CCM) which entrusts mid-level and allied health 
providers with measurement-based symptom management not only mitigates symptoms, as robustly reported in 
systematic reviews,13,27-29 but also results in improved function30 and quality of life (QoL), and reduced health care 
utilization.31 Third, up to 40% of patients experiencing mild or moderate cancer symptoms achieve relief with self-
management approaches.32-38 The proposed E2C2 trial leverages these findings, particularly those from the investigators’ 
successful and completed INCPAD and COPE trials which effectively generalized the CCM to patients with cancer to 
improve pain, depression, and physical function.39,40 The proposed E2C2 intervention’s emphasis on the CCM is further 
supported by the fact that other investigators’ successful PRO monitoring efforts in cancer populations were grounded 
in the model’s basic tenets – measurement-based care delivered by dedicated non-MD providers. For example, patients 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 3 of 32 
 
in Basch et al’s trial,41-43 received a mean of 13 nurse phone calls during a median 4 months on study, while patients in 
studies reported by Mooney et al20,44 and Cleeland at al21 received similar symptom-directed attention from dedicated 
nurse practitioners. 
The E2C2 intervention recognizes historic challenges the CCM’s broad implementation posed by its human resource 
requirements and proposes to overcome them through three validated strategies that capitalize on increased EHR 
penetrance and capabilities: 
1. EHR-embedded algorithms will deliver validated self-management education for specific symptoms. This will enable 
nurse symptom care managers (SCMs) to focus their collaborative care efforts on high-needs patients.  
2. EHR clinical decision support tools, proven effective in other contexts in improving the frequency of evidence-based 
care,45,46 will be utilized to strengthen the benefits of providing oncological clinicians with PRO score-linked care 
plans. The Symptom Monitoring II (SyMon II) trial presented oncological clinicians with scores for five PRO domains 
with guidance in score interpretation and symptom management. While anxiety significantly improved, 
demonstrating that the strategy can be effective, the other domains did not.47,48  The E2C2 intervention will enhance 
the SyMon II approach by including validated decision support tools to enable clinicians to efficiently manage 
patients’ symptoms.  
3. E2C2 will allocate nursing, social work, and physical therapy FTE to designate and train SCMs in efficient EHR-based 
CC. Cultivation of dedicated niche practitioners to achieve economies of scale has proven effective in both primary 
and specialty care.49 We believe that this strategy will prove broadly scalable and require limited, if any, incremental 
FTE because: 1) Experienced SCMs are highly efficient; in INCPAD, a single SCM managed 18 community practices;39 
2) Proactive symptom management can reduce unscheduled office visits;50 3) US oncology practices, on average, 
employ 6.9 nurses making the dedication of >0.5 FTE for CC practical (this average has steadily increased through 
practice consolidation into larger hospital- and health system-owned group and multi-specialty practices);51-53 and 4) 
Prior CCM iterations have used inefficient modes of SCM-care team communication, e.g. repeated phone calls, fax, 
email, rather than time-saving, seamless intra-EHR order entry by the SCM. 
 
The E2C2 trial takes advantage of our growing recognition of the interrelatedness of symptoms, as its targeting of 
commonly co-occurring symptoms will enhance our understanding of symptom clusters.  Emerging research 
demonstrates the substantial co-occurrence of cancer-related symptoms and, consequently, the impracticality and, 
indeed, artificiality of evaluating and treating symptoms in isolation54-61 rather than in clusters such as the SPPADE 
grouping (Sleep disturbance, Pain, Anxiety, Depression, and Energy deficit/fatigue), which represents a prevalent and 
potentially treatable group of overlapping symptoms.54,56,57,59,62-67 Cross-sectional analyses will provide insights into the 
co-occurrence rates of these 5 symptoms as well as their individual and additive effects on functional status and quality 
of life. Longitudinal analyses will determine the responsiveness of individual symptoms and overall symptom burden to 
treatment as well as patient- and disease-specific mediators and moderators of improvement.  
The E2C2 trial is clinically and methodologically significant because it will comprehensively evaluate the use of 
ubiquitous and inexpensive EHR functionalities – automated assessment, decision support, and implementation – to 
enhance symptom control.  While these capabilities are increasingly EHR agonistic, our utilization of Epic may be 
particularly beneficial as it is used across 250 U.S. healthcare organizations and as 65% of the US population has an Epic 
record, and 69 million Americans have Epic My Chart accounts.68  The Epic internal logic, assessment, audit and 
feedback, and alert functionalities used in the E2C2 intervention are straightforward to configure, requiring only 1 to 2 
weeks of builder time, thereby limiting the human resource requirements for E2C2 intervention implementation.  As 
noted above, these capabilities are increasingly EHR agonistic and while build specifics will differ in Cerner, Meditech, 
and other EHRs, the fundamental principles will remain constant.69  Our detailed Resource Sharing Plan emphasizes the 
development of manuals that outline build specifics that will be pertinent to both Epic and non-Epic using organizations.  

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 4 of 32 
 
This strategy has the potential to radically accelerate the uptake and broad implementation of E2C2 capabilities with 
limited resource investment.  
Study Design and Methods 
 
Methods:   
 
Overall Study Design: The Enhanced, EHR-facilitated Cancer Symptom Control (E2C2) is a pragmatic, cluster-randomized 
clinical trial that will rigorously assess the impact of the intervention in controlling SPPADE symptoms among patients 
with cancer spanning the cure-directed, survivorship, and palliative phases of disease management, while also exploring 
factors relevant to its implementation. The E2C2 trial is population based; all patients receiving care for solid tumors at 
the MC Cancer Center Rochester (MCR) will be included in the study sample irrespective of cancer type or stage. A 
similar sample will be included from community clinics within the Mayo Clinic Health System in Minnesota and 
Wisconsin, with the addition of malignant hematology patients at those community sites. Patients will be assigned the 
ePRO “Basic Symptom and Function Screen” (BSFS) questionnaire via their EHR approximately 4 days prior to their 
oncology appointment (in-clinic or virtual visit) as the standard of care process. The EHR algorithm will be set to assign 
questionnaires no more than every 2 weeks. The patient will have the ability to complete the BSFS via their portal but 
will be prompted to complete it via welcome tablet at check in if they do not. Additionally, if patients do not respond to 
questionnaires via portal or welcome tablet, they will be given an opportunity to complete it via Interactive Voice 
Response, otherwise known as IVR. When a patient fails to complete their assigned BSFS questionnaire, they will receive 
an automated phone call and will be prompted to answer the questions via touchpad. While Mayo Clinic is using an 
external vendor (IPT), all data is maintained on Mayo Clinic servers. We are not releasing data to IPT at any point in time. 
The ePRO (questionnaire) Health platform resides on our server and is not going outside of the firewall. Data going to 
the platform is transient in nature and is purged once the results are sent back into Epic. SPAD approval has been 
granted and there are Business Associate Agreements (BAA) and Information Security Agreements (ISA) in place, which 
covers how PHI can be used and stored. The trial’s stepped wedge design (SWD) will randomize the order of E2C2 
implementation among 15 clusters.  Clusters will be defined at the level of the cancer care team and will be randomized 
to one of five different tranches to receive the implementation at staggered 8 month intervals.  Between 2 to 20 
oncologists/hematologists and mid-level providers will comprise each of the 15 clusters. Clusters will be randomized to 
initiate the E2C2 intervention at each of the steps. This design will allow data to be sequentially collected from all 
clusters for a minimum of 7 months during pre-E2C2 usual care. This stepped wedge design will allow us to compare 
outcomes for each cluster both across intervention groups and with historical controls at the same clinic, providing a 
rigorous assessment of the impact of E2C2 on controlling SPPADE symptoms among patients with cancer while also 
exploring factors relevant to its implementation across all sites.     
The primary outcomes of the trial are a bundle of 6 SPPADE symptom and function scores assessed with numerical 
rating scales (NRSs) collected in association with medical oncology clinic appointments. The primary analysis will test 
differences in the change in mean joint SPPADE symptom NRS scores in all participants, and in those who experience 
elevated (4-10/10) symptoms, between participants who are exposed to the intervention and those who are not.   
Secondary outcomes will include PROMIS-CAT measures for anxiety, depression, pain, and physical function, assessed up 
to every four months depending on visit frequency. Additional secondary outcomes will include health care utilization, 
adherence to cancer treatment, and survival.  Variation in ePRO response rates and outcomes will be assessed for all 
participants, but special emphasis will also be given to disparities in outcomes among elderly and rurally based patients.  
Process measures will capture stakeholders’ use of the EHR graphic and decision support tools and will be abstracted 
from the EHR. All analyses will be performed using methods appropriate for cluster randomized stepped wedge trials. In 
parallel with conduct of the E2C2 trial, mixed methods will be used to comprehensively assess system usability, multi-
stakeholder experience, and barriers and facilitators for stakeholders’ use of the system.  Qualitative interviews will also 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 5 of 32 
 
be conducted to understand possible reasons for disparities in the adoption and implementation of the E2C2 
intervention among elderly and rural-dwelling patients with cancer. 
Cancer Patient Enrollment:  This is a population-based study, where all eligible patients seen by clinicians at the Medical 
Oncology Clinic in Rochester and hematology/oncology in the Midwest Mayo Clinic Health Systems (MCHS) will be 
enrolled. This inclusive enrollment approach substantially increases the generalizability and external validity of our real-
world pragmatic trial. Rochester campus will act as primary research site and there will be no study team responsibilities 
at any of the MCHS sites.  
Among the roughly 2.5K patients with new cancer diagnoses annually treated by the MC practices, 50% are rurally 
situated, 50% are elderly, and 18% are covered by Medicaid, comparable to the upper Midwest.70 The population’s 
ethnic and racial characteristics are also representative of the upper Midwest, excepting higher inclusion of Native 
Americans, ~3.8%. The anticipated distribution of cancer types and stages in the study population parallels those 
described in the American Cancer Society 2018 Facts and Figures,71 excepting higher proportions of patients with 
melanoma, neuroendocrine tumors, and sarcomas, which reflect MCR emphases.  
Transparency and Institutional Review. E2C2 will be sequentially implemented across the Mayo Clinic Midwest medical 
oncology practices, as per the stepped wedge random allocation.  Neither current regulations, nor the peer-reviewed 
literature explicitly defines when a formal patient consent process is required for pragmatic clinical trials, particularly 
when testing the implementation of proven effective interventions at scale. This type of research, sometimes referred to 
as “standard of care” research in which the primary research objective is to implement and evaluate a strategy that 
better satisfies the existing clinical standard of care, has been debated in recent high profile cases.72-74 Opinions from 
ethicists vary but all emphasize the vital importance of IRB engagement and assiduous efforts to foster disclosure and 
transparency across all relevant stakeholders.75-77 We believe, and in consultation with our local Research Ethics Consult 
Service have confirmed, that this study meets the criteria for being standard of care research in which the marginal 
incremental risks of the proposed interventions do not foreseeably introduce even incremental net risk to individuals in 
the participating practices being studied. The MC Cancer Center Patient and Family Advisory Council contributed to the 
current E2C2 “Transparency Promotion and Patient Preference Protection Plan.” All patients who have previously opted 
out of having their records used in medical research (~3% of the Mayo Clinic patients) will have their data redacted from 
any final analysis files.    
Transparency is both a vital human subject protection concern and an implementation tool with the potential to 
enhance patient engagement.  The E2C2 intervention will include a simply produced 2-minute video that will describe 
key aspects of the intervention and will visually communicate patients’ anticipated experience with it. It will emphasize 
the intervention as an improvement in care delivery and not as a research project. This video will be available on the 
patient facing website (cancersymptoms.mayoclinic.org). A welcome packet will be sent to all patients with upcoming 
appointments at the time of E2C2 rollout. This packet will include a welcome letter that gives information about how to 
view the video and other IRB and Office of Patient Experienced approved materials on the external website 
(cancersymptoms.mayoclinic.org). Patients will also have the opportunity to receive educational materials in the form of 
“toolkits,” which is discussed in the “Dispensing of self-management education modules” section. 
Provider and staff data collection 
In parallel to the E2C2 study, we will enroll clinical stakeholders at multiple levels to participate in surveys, interviews, 
and focus groups designed to explore factors relevant to the experience of implementing the intervention. Measures 
and methods for this process are as follows: 
System Stakeholder Context Survey and Interviews: Approximately 20 system-level stakeholders within medical oncology 
or related practice (e.g. administrators, practice leadership, information technology) will be recruited via an email 
invitation during the Pre-E2C2 stage to participate in a brief survey and semi-structured interview. We will use a 12-item 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 6 of 32 
 
Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of 
Intervention Measure (FIM) to assess system-level context prior to implementation (see survey drafts). Interviews will 
comprise 15-30 minute elaborations of the same constructs, with prompts informed by the Consolidated Framework of 
Implementation Research. The aim of this data collection is to understand the factors that may impact implementation 
of the intervention, including stakeholder perceptions of how it will impact their work. Interviews will be completed in 
person or by phone, and they will be recorded with permission. Participants will complete oral consent before the 
interview. After the interview, the study team member who completed the interview will write field notes summarizing 
the participant’s impressions of E2C2, readiness to implement E2C2, etc. He or she will also forward items requiring 
immediate action to the appropriate study team member or resource.  
Care Team Implementation Survey: The perspectives of care team members will be elicited during implementation using 
the Care Team Implementation Survey. At baseline, the Care Team Implementation Survey consists of questions from 
two widely used implementation questionnaires: 1) the 12-item IAM-AIM-FIM survey, and 2) a 23-item instrument 
based on Normalization Process Theory (i.e., The NoMAD questionnaire). The NOMAD questions will assess, among 
other things, care team members’ perceptions of intervention leadership, integrated workflow, training adequacy, and 
organizational understanding, as well as perceptions of the value of the intervention and its impact on working 
relationships. Just prior to initiation of the intervention (baseline), all care team members (e.g. physicians, nurses) within 
a given cluster will be invited to complete the surveys. Participants will be recruited at cluster engagement “kick-off” 
meetings and provided with a paper copy of the survey for completion. Care team members who are not recruited in-
person will be sent an email link to complete the surveys online. Version 2 of the Care Team Implementation Survey will 
be repeated at 2-4 months after initiation of the intervention and 12 months later throughout the funding period, via 
email link or paper copy. Version 2 includes the same IAM-AIM-FIM and NoMAD questions, along with a set of questions 
that assess the care team members’ current use of various components of E2C2. All surveys will be preceded with 
consenting information describing the study, risks, benefits, and alternatives, including the option to not participate. 
Care Team Implementation Focus Groups: To supplement implementation survey data, members of care teams will also 
be invited by email to participate in focus groups. Up to two focus groups will be held with each cluster. A trained focus 
group facilitator will moderate the groups using a semi-structured interview guide informed by survey constructs (e.g., 
acceptability, integration of E2C2 into workflow), as well as the preliminary results of the cluster’s survey data. Focus 
groups will function like reflective team meetings and will be recorded with permission and transcribed for analysis. 
Participants will complete oral consent prior to the focus group. 
System Stakeholder Implementation Interviews: To further contextualize and explain interesting findings from the survey 
data and to explore the insights of key stakeholders that do not correspond to individual care team clusters (e.g. system-
level implementation work), we will invite up to 35 stakeholders (MD, administration, IT, NP) (including all RN symptom 
care managers to participate in approximately 30-minute qualitative interviews. Stakeholders will be identified by 
members of the study team and the engagement workgroup, and potential participants will be recruited by email.  
Interviews will take place in person or by phone, based on participant preference, and they will be timed to follow 
analysis of surveys conducted at 2-4 months after initiation of the intervention and 12 months later. They will be 
conducted by a member of the study team using a semi-structured interview guide developed using NPT constructs and 
existing literature on implementation of EHR and symptom management innovations. Interviews will be audio recorded 
with permission and transcribed for analysis. Field notes will also be completed after each interview. Participants will 
complete oral consent prior to the interview. 
System Stakeholder Sustainability Assessment: Approximately 40 system-level stakeholders within medical oncology or 
related practice (e.g. administrators, practice leadership, information technology), including those who participated in 
the context assessment or stakeholder interviews, will be asked to complete the short form version of the Clinical 
Sustainability Assessment Tool (CSAT) . The CSAT short form is a 21-item assessment of clinical and organizational 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 7 of 32 
 
stakeholder perspectives on how sustainable a practice is over time (e.g., whether the practice has champions, whether 
it is easy to use, and whether it has sufficient resources).  Surveys will be distributed by paper or using an email link after 
12 months of intervention go-live. 
Implementation Documents: Implementation documents, including executive team meeting minutes, project 
correspondence and materials, periodic study reflections, and study logs maintained by study team members will also 
be collected for textual analysis of how the intervention unfolded in practice.  
Symptom Sage Surveys: As part of the E2C2 intervention, implementation facilitators (referred to as “Symptom Sages”) 
were recruited to facilitate cluster-level implementation and champion the intervention to their colleagues. This 
evidence-based implementation strategy was deployed at the cluster level, with facilitators typically representing 
clinicians or nurses on the care teams. To help us understand the experiences of Symptom Sages with the E2C2 
intervention, including whether they felt they had adequate resources (e.g., time, training, knowledge) to be effective in 
their role, we will administer a brief survey to Symptom Sages. Questions for the survey were informed by constructs 
from the Consolidated Framework for Implementation Research (CFIR) 2.0, as well as existing literature on the 
implementation facilitation strategy. 
Symptom Sage Survey Recruitment: All Symptom Sages who previously consented to participate in the E2C2 study and 
completed an E2C2 Care Team survey as part of the study protocol (up to n = 18) will receive an email inviting them to 
complete a web-based survey specific to their experience as a Symptom Sage. This email will introduce the survey and 
contain an embedded survey link (to a RedCap or Qualtrics survey). Those who do not complete the survey after the first 
email request will receive a second email request. Our third and final recruitment attempt to recruit non-responders will 
be done either by email, by phone, or in person with paper surveys depending on staff availability. The option of paper 
copies will come with a pre-addressed envelope to return the survey via intra-clinic mail if they wish to participate. This 
survey should take less than 10 minutes to complete. Each Symptom Sage will be remunerated $40 for their completion 
of the survey. Survey responses will help us understand how to improve implementation of similar projects in the future. 
Symptom Sage Interviews: To understand the experiences of Symptom Sages and the challenges they faced in their role 
in more detail, we will also conduct semi-structured interviews with Symptom Sages. Interview questions were designed 
in accordance constructs from the Consolidated Framework for Implementation Research (CFIR) 2.0, which assesses 
facilitators and barriers to implementation, and will help us understand how to improve the implementation strategies 
used to implement future initiatives. All interviews will be conducted telephonically or via a secure web-based platform 
(e.g., Zoom, Teams) and recorded. Recordings may be submitted for transcription to a Mayo approved vendor. 
Symptom Sage Interview Recruitment: All Symptom Sages who previously consented to participate in the E2C2 study and 
completed a Care Team survey (up to n = 18), will receive an email inviting them to participate in an ~30 -minute 
interview. These interviews will help us understand how to improve implementation of similar projects in the future. 
Each Symptom Sage will be remunerated $40 for their completion of the interview. If after up to 2 email contacts we do 
not hear back from a Symptom Sage about their interest (or lack of interest) in participating, a member of our study 
team may call, email, or visit in person to inquire about their interest participating in an interview. 
 
Provider and staff data analysis 
System Stakeholder Context Survey and Interviews: AIM-IAM-FIM surveys will be collected and entered into a secure 
REDCap database, and scores will be compiled for the overall group of participants. Interview data will be analyzed using 
methods of content analysis to identify impressions related to AIM-IAM-FIM and CFIR constructs. Analysis will be 
completed during the pre-E2C2 study period, and findings will be summarized in an analytic memo to facilitate further 
reflection on context-related findings at the completion of the study.  

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 8 of 32 
 
 
Care Team Implementation Surveys: The implementation survey will be collected and entered into a secure REDCap 
database. Analysis will be completed after each wave of data collection. Scores will be compiled for the 4 NPT construct 
levels within NOMAD and overall and within the 3 domains of Acceptability, Appropriateness, and Feasibility for IAM-
AIM-FIM and as a composite readiness for implementation score and team/cluster and tranche level scores will be 
determined for all outcomes at each scheduled time point. Differences will be explored between groups and across 
time.  
 
Care Team Implementation Focus Groups and System Stakeholder Implementation Interviews:  Preliminary qualitative 
analysis of focus group and interview transcripts, as well as field notes, will follow completion of each wave of 
qualitative data collection (e.g., the months following completion of focus groups/interviews in each tranche). Additional 
analysis (including between tranches and overall across the study) will be completed at the end of the study. The 
analyses will employ a theoretically driven thematic approach. The study team will begin by reading the textual data, 
discussing manifest and latent content, and developing a list of concepts represented in the data. These concepts will be 
reviewed in light of NPT and other relevant implementation constructs but not constrained by them. A coding 
framework will be developed using these concepts and applied to the text data. Coded data will be entered into NVivo 
11.4 (QSR International Pty Ltd.)—a qualitative analysis software package—to facilitate data organization and queries 
and maintain an audit trail of the analysis. Members of the study team will meet to discuss the data, and narrative 
memos will be used to describe themes or major findings in the data, along with variation in experience. Individual 
survey results, implementation metrics, and individual- or site-level characteristics of interview participants will be used 
as appropriate to aid in interpretation. Furthermore, a side-by-side comparison of qualitative and quantitative findings, 
along with study team discussion, will further explore how the different data findings converge or diverge in order to 
enrich understanding of E2C2 implementation. This type of data and methods triangulation, along with the inclusion of 
study team members from different disciplines, will reduce interpretive bias.  
 
System Stakeholder Sustainability Assessment: The CSAT will be scored through calculation of an overall average and 
averages per each of the subscales included in the instrument. Responses will be reported at the study level rather than 
by tranche or cluster. 
 
Implementation Document Analysis: Analysis of documents—collected during the course of study implementation—will 
take place at the end of the study and will use methods of content analysis. Documents will be reviewed for occurrences 
of information that describe theoretical constructs including those from CFIR.  Those will be coded in NVivo using a 
predetermined, theory-informed coding framework. New codes will be added if they arise during analysis.  
Symptom Sage Survey Analysis: We will report descriptive statistics on survey responses to help us understand Symptom 
Sages perceptions of the E2C2 intervention and their role as Symptom Sages. This will help us understand any challenges 
Symptom Sages may have faced, potentially limiting the effectiveness of the Symptom Sage role as an implementation 
strategy. 
Symptom Sage Interview Analysis: These interviews may be rapidly analyzed from audio recordings and/or transcripts. 
For Rapid Analysis, we will create Summary Templates that summarize the key points within each domain assessed by 
the interview questions and then transfer key points for each domain into a Data Matrix in excel. At least two 
researchers will review domains within the data matrix and identify key themes independently and then via discussion 
by preparing and comparing Memos. Transcripts may also be uploaded to qualitative software (e.g., Nvivo) for more in-
depth coding. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 9 of 32 
 
Patient enrollment, data collection and analysis 
Quantitative data collection and analysis.  In order to understand variation in the use of the E2C2 symptom 
management strategies, we will first use data from the EHR to identify if the intervention is similarly accessed/adopted 
and if symptom management and utilization patterns are similar across age groups and by patients in rural and urban 
areas. We will calculate ePRO response rates (dividing the number of ePROs returned/the number sent) during the pre-
E2C2 usual care period and then again during the intervention period, comparing rates by age, urban/rural residence, 
and age/rural residence.  To analyze response rate data, we will use both simple unadjusted descriptive statistics as well 
as generalized regression models.  We will calculate simple proportions of response rates for the total population and by 
age and urban/rural status.  We will then create regression models to determine predictors of being a high responder. 
Additional explanatory variables will include patient demographics (e.g., employment status, insurance status, gender), 
cancer type and stage and cluster.  Response rates at each time point will be calculated for analyses, and models will 
include a random intercept to account for repeated measures over patients. We will then aggregate data and examine 
trends over time to determine if, over time, response rates have increased significantly.  Similarly, we will calculate 
responses to PROMIS CAT scores (individually and composite scores) during the pre-E2C2 usual care period and then 
again during the intervention period, comparing rates by age, urban/rural residence, and age/rural residence. For the 
PROMIS CAT scores, we will use a similar analytic approach as with the response rate analysis.       
Qualitative data collection and analysis.  Quantitative data will inform subsequent qualitative interviews aimed at 
examining patient barriers and facilitators to optimal symptom management, understanding experiences with symptom 
management strategies included in E2C2 (e.g., patient education; collaborative care), and identifying feasible and 
patient-centric solutions for refining and improving E2C2 in order to reduce disparities in outcomes for the elderly and 
rural residents. We will recruit patients for interviews that will help us understand barriers and facilitators for ePRO 
reporting and to solicit potential refinements that could improve ePRO reporting success.  We will use a random sample 
of patients for interviews.  Starting with the clusters randomized to the first tranche, we will randomly select up to 100 
patients stratified by age (<65, 65-75, >75), sex, and by urban/rural residence. We will aim to complete up to 20 patient 
interviews per tranche, with balance on key characteristics such as PRO responders/non-responders, cancer type, cancer 
stage, and experience with E2C2. Additional random samples will be drawn as needed to meet interview goals. As the 
interviews continue, we may move to more purposive sampling to capture the broadest set of qualitative experiences. 
These procedures will be repeated for tranches 2-5, or until data redundancy or data saturation (i.e., the point at which 
little new information is produced with each additional interview).  
We will use quantitative metrics—including on symptom reports, response mode, and invention use—to create a 
dataset for interview recruitment. The criteria for sampling will be reviewed before each wave of interviews (e.g., new 
tranche), and criteria will be adjusted as needed to meet study goals. All patients who participate in the interviews will 
be consented. We will collect written HIPAA authorization for patients as part of this consenting process. If the subject is 
agreeable, documentation of HIPAA authorization will involve the use of Electronic Informed Consent (otherwise 
referred to as Docu-Sign) for HIPAA authorization forms. This is an institutionally approved process for documenting 
consent/HIPAA authorization using an on-line process. The subject may print or electronically save the consent form or 
may contact the study team to provide a copy of the consent.  If the subject prefers not to use Electronic Informed 
Consent, the study team will utilize a paper form for enrollment of these subjects, either via mail or in-person. 
 
A random sample will be drawn from the dataset of eligible patients. After participants are randomly selected, the study 
coordinator will send a letter describing the study, along with a HIPAA form and reply envelope and a copy of the HIPAA 
form for the patient’s records. Patients will be directed to return the HIPAA form if they are interested in participating or 
call the study staff with any questions. The study coordinator will follow up with a phone call.  Those who are interested 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 10 of 32 
 
in participating will be read the oral consent script and scheduled for an interview date and time.  Prior to a scheduled 
interview, a variety of techniques will be utilized to remind participants of their upcoming interview. Depending on 
participant preference, they will be able to choose the method in which they are reminded: by telephone, by text 
message to a mobile device, by mail (letter), or by e-mail. Participants will also be given the option of having a calendar 
invitation sent to them to remind them of their upcoming interview.  E-mail address will be collected for participants 
who opt to have an e-mail reminder and/or calendar invitation sent to them. For participants who elect to have their 
reminder sent by text message, we will collect phone carrier information (this data is needed for determining the 
appropriate SMS gateway domain). 
 
Interviews will be conducted over the telephone or Zoom and are expected to last 30-45 minutes.  If the patient prefers 
to use video conferencing, such as Zoom, the interview will be scheduled using a MC Health account, which is approved 
for PHI.  They will be audio recorded with permission and transcribed for analysis and the video conference interview 
recorded.  Upon completion of the interview, participants will be sent a hand-written thank you note. Participants will 
receive $40 in remuneration for their time. The interview schedule will follow an iterative process of data collection and 
analysis, a standard technique in qualitative methodology designed to ensure the identification and testing of analytic 
categories until thematic saturation has been achieved.  Methods of data analysis will be similar to those used for 
clinical stakeholders, although the study team will also include methods of framework analysis. Initial analysis steps are 
comparable to those for clinical stakeholder data, but the coding framework will be applied in order to generate a matrix 
of results,78,79 allowing for within- and across-case analysis. This approach will facilitate systematic but flexible analysis of 
variation in experiences by characteristics such as rural/urban and elderly/non-elderly.  
 
See the below timeline for a summary of when clinical stakeholder and patient implementation data will be collected: 
 
Pre 
E2C2, 
funding 
outset 
Baseline** 2-4 Months 
after 
Intervention 
Initiation 
6 Months after 
Intervention 
Initiation 
12 Months 
after last 
assessment 
System-level practice stakeholder 
context assessment IAM-AIM-FIM 
surveys and interviews 
X 
 
 
 
 
 
 
 
 
 
 
 Care Team Implementation Survey  
 
X 
X 
 
X 
Cluster implementation work focus 
groups * 
 
 
X 
 
X 
System implementation work key 
stakeholder interviews * 
 
 
X 
 
X 
System Stakeholder Sustainability 
Assessment 
 
 
 
 
X 
Interviews to assess patient disparities 
in implementation and outcomes* 
 
 
 
X 
 
 
*Interviews and focus groups will commence following survey assessment.  
**For survey administration, baseline refers to the period prior to implementation and just after the tool has been 
introduced to the care team.  
 
Clusters 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 11 of 32 
 
Disease-focused clinical teams, e.g., breast, colorectal, etc. will comprise the clusters.  All clusters will include both 
medical oncologists and mid-level providers. 
Allocation. There will be 5 sequences in our design, with 3 clusters randomized to initiate the intervention for each 
sequence. Pre-E2C2 baseline assessment will occur for all clusters starting 7 months prior to the first sequence.  Each 
sequence will occur 8 months apart; data collected prior to the intervention will provide a within-site control group for 
each site. The allocation is illustrated in Figure 1. To ensure balanced cluster characteristics we will stratify 
randomization based on cluster size and site.   
Figure 1. Final allocation of 15 cancer care sites to 5 sequences over 6 time periods.  
 
 
 
 
 
 
 
 
 
Interventions  
The E2C2 intervention will include patient- and clinician-directed elements that are designed to increase the frequency 
with which patients receive individualized, preference-concordant, and guideline-based care for their symptoms and to 
increase rates of symptom control.   

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 12 of 32 
 
Automated Symptom Monitoring:  Following the transition from pre-E2C2 usual care to the E2C2 intervention, the 
ePROs used to assess SPPADE symptoms, physical function impairment, social participation, distress, and QoL will not 
change, however, assessment frequency will increase, particularly for symptomatic patients. After initiating E2C2, the 
Symptom Control Bundle will be activated and patients and clinicians will be able to access graphic displays for SPPADE 
score tracking, and patients will be able to enter ePROM data via a telephonic symptom assessment in addition to the 
web, smart phone, and tablet modes.  
Presentation of symptom data to patients and clinicians.  For PROM data to be useful in clinical practice, patients and 
clinicians need to understand what the scores mean. In order to facilitate intuitive ePROM interpretation, the E2C2 
intervention will include a dedicated view in the clinician-facing Synopsis section of the Epic EHR and in the patient-
facing My Chart interface.  The views will be identical for both patients and clinicians to support communication and will 
be specified per the findings of Snyder et al. to optimize accurate interpretation.80 Data collected regarding symptom 
intensity and context will also be presented to clinicians in brief auto-generated text that will populate their clinical 
notes with a patient’s longitudinal SPPADE scores and a hyperlink to the graphic Synopsis view. Presenting data to 
clinicians in graphic form for decision support has been shown to have a small to moderate effect size in improving 
patient outcomes.81,82 
EHR Triage and Decision Support Algorithms. EHR algorithms are foundational to the E2C2 intervention.  Algorithms 
populated by automatically abstracted PROM and clinical data from the EHR will determine which symptom-directed 
services are offered to patients at multiple points in the intervention.  Algorithms will be built into Epic using 
specification capabilities integral to the foundation system and available in all Epic iterations after 2014.   
Triage to Level 1 or 2.  Patients’ responses to the BSFS questionnaire will initially assign symptomatic patients to one of 2 
incremental Levels of stepped care.  Patients will be triaged to Level 1 (moderate symptoms) unless they meet Level 2 
(severe symptoms) criteria based on SPPADE symptom or physical dysfunction intensity; non-response to treatment; 
patient preference for SCM contact; or clinician referral. Patients may move between the “no symptom” state and 
Levels 1 and 2, as depicted in, depending on treatment response, and symptom onset and/or worsening. 
Dispensing of self-management education modules. A second algorithm, irrespective of Level assignment, will determine 
which self-management education modules (also referred to as “Symptom Toolkits”) will be automatically dispensed to 
symptomatic patients either via their web portal, where they will be directed to the E2C2 patient facing website 
(cancersymptoms.mayoclinic.org) or in print. Each printed symptom toolkit will contain an assortment of patient 
education and IRB approved materials in paper and digital form. Patients’ preferences for mode of delivery will be 
electronically captured in their EHRs.  Criteria that will determine whether and which materials are dispensed include 
symptom number and type, and request for reissue.  
Order pre-configuration for first-line symptom management.  A third algorithm for patients triaged to Level 2 will 
identify first-line, guideline-supported prescriptive interventions matched to those listed in the self-management 
modules and will automatically present these to the clinicians as pre-configured orders. 
Automated provision with guideline concordant, self-management education and resource information (all levels). All 
patients, irrespective of Level status, will receive validated educational modules on SPPADE symptom and physical 
dysfunction self-management in the form of a paper booklet.  Subsequently, patients will receive Portal-based links to 
videos and additional written content modules matched to the symptoms that they endorse at a moderate or greater 
level, with the content of these educational materials.  The educational materials are based upon cognitive-behavioral 
principles, and were developed and refined through 5 previous studies conducted by Given et al. Each symptom module 
was developed using evidence-based guidelines written at the 8th grade level.83-86Each module is presented in an 
identical format (frequently asked questions): what the symptom is, how people describe the symptom, the causes of 
the symptom including medications, and a set of strategies presented in bullet points for managing the symptom. For 
each symptom, there are indications as to when and for what reasons to contact the oncology team, and other 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 13 of 32 
 
resources for management are listed. All module information is based on NCCN guidelines, Mayo Clinic expert opinion, 
and the Oncology Nursing Society PEP guides.87-89 The content of the symptom modules will be cross-referenced 
annually against current guidelines.  
Nurse Symptom Care Manager (SCM)-facilitated Collaborative Care (Level 2).  Symptom-specific medication 
management, specialist referrals, and reinforced self-management will be provided by the SCMs, Social Worker and 
Physical Therapist. Level 2 is designed to optimally align treatment with each patient’s specific symptoms, preferences, 
and response to therapy; thus, the frequency and content of nurse calls will vary with our “treat to target” approach.  
Similar to our INCPAD and COPE trials,39,40 we will systematically document in detail the frequency, duration, and 
content of all SCM calls.  All self-management strategies discussed will be documented on a structured checklist for each 
call.  Additionally, the time spent on the call reinforcing self-management and discussing medication management will 
be documented. This will allow us to secondarily examine the independent effect of the intensity and content of nurse 
contacts as mediators of outcomes. Phone calls between the patients and SCM will be recorded for fidelity monitoring 
and quality assurance purposes. Patients will give verbal consent to, and will have the ability to opt out of, having the 
recording used. Due to the setup of the call system and processes set in place by Legal, when a patient opts out of 
having their recording used for fidelity purposes it (the recording) because the responsibility of the Medical Oncology 
department. The study team will maintain records of the patients who do not wish to have their recording used so that 
they are not included in the fidelity monitoring and quality assurance. 
The SCM will contact patients in response to ePRO symptom-triggered alerts and trend reports. Patients will be able to 
view a short video (SCM Intro Video) explaining the role of the SCM when they are completing their BSFS questionnaire 
and indicating whether they’d like the assistance of a nurse Symptom Care Manager. In efforts to create awareness of 
the SCM role, patients will have the opportunity to receive a printed “face sheet” with photos of the SCMs and details 
on their role and how to request intervention materials. The E2C2 Face Sheet will be handed out by clinical Oncology 
staff. The SCM will monitor trend reports weekly, respond to automated monitoring clinical alerts and patient calls daily, 
configure orders for the oncology care team via the Epic EHR, and serve as the “triangulating” linchpin between patient 
and the oncology care team (including the supervising oncologist).  The Symptom Care Managers will assess patients’ 
needs for resistance band and pedometer (up to 2 pedometers per patient) interventions/therapies and will request 
these for the patient via Clinical Research Coordinator who will mail the items with the IRB approved cover letter and 
patient education materials.  For patients who have a local Oncologist in addition to being seen by Mayo Oncology, the 
Symptom Care Manager will send a letter to the local Oncologist (if there is a current authorization to do so). This letter 
will contain information outlining the SCM’s involvement with the patient and the discussed action plan for the 
identified symptom(s). For E2C2, the SCM, Social Worker and Physical Therapist will have weekly case management 
sessions with the Palliative Care Service physician to review new Level 2 patients and patients not responding to 
therapy.  In addition, the Clinical Research Coordinator will send a portal message to 2 groups of patients: a group who 
requested contacted from a Symptom Care Manager (SCM) and a group without SCM contact.  This message will include 
a link to patient education material and to Mayo Clinic Connect.  
Implementation Bundle 
Implementation Strategies. The Implementation Bundle will be initiated at the same time as the Symptom Control 
Bundle as part of the stepped-wedge design. 
Practice Facilitation:  Practice facilitation, supportive services provided to clinicians by a trained individual to assist with 
engagement in clinical practice changes, has been shown to be moderately effective in changing clinician behavior.90,91 
Practice facilitators use a variety of organizational, project management, and quality improvement approaches and 
methods to support practice change or redesign.  We will implement a practice facilitation model wherein “Symptom 
Sages” will be trained to support adoption and use of the E2C2 EHR clinical decision support tools.92 The Symptom Sage 
practice facilitator role will ensure readiness and sustained support for use of the symptom monitoring and 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 14 of 32 
 
management system. The facilitator will provide hands-on training and support to clinicians to foster understanding of 
and engagement with the symptom monitoring and management system.  
Point-of-Care Computer Reminders: Point-of-care computer reminders/alerts, delivered through a computer system 
routinely used by clinicians, have been demonstrated to be moderately effective in changing clinician behavior.93,94 We 
will use point-of-care computer reminders/alerts embedded within the EMR as Epic Best Practice Advisories (BPA) which 
will appear as alerts on the clinician EMR interface. The EMR will record the actions that clinicians take, and these data 
can then be aggregated and presented to the user in graphic form (see audit-and-feedback) and used as a measure of 
fidelity to the intervention. Clinicians may respond to the best practice advisory by linking to the pre-configured orders 
or Synopsis SPPADE summary view.  
Audit and Feedback:  Audit and feedback, the collection and summary of clinical performance data over a specified time 
period provided to clinicians in written, verbal, or electronic form, has been shown to be moderately effective in 
changing clinician behavior.94 Clinical performance data including the frequency with which a clinician; 1) responds to 
the Best Practice Advisories, 2) accesses the SPPADE summary view, 3) issues orders pre-configured in SmartSets, 4) 
issues orders queued by the SCM, and 4) retains SPPADE symptom-related text in clinical notes will be collected through 
automated Epic EHR tracking.  Individual- and group-level quarterly performance data will be provided to clinicians as 
feedback via Epic’s dashboard functionality.  Upon initiation of the Symptom Control Bundle, the dashboard will be 
presented weekly on the landing page of the clinician when they first log into the Epic EHR. Clinicians can also freely 
access the dashboard via the Epic Navigator. The EMR will record the frequency and duration of dashboard access and 
whether a clinician drills down more granularly into their performance data.  These data will be used to assess the 
degree to which use of the dashboard may be associated with Implementation Bundle effects. 
Resources:   
Department of Physical Medicine and Rehabilitation 
The Mayo Clinic Department of Physical Medicine and Rehabilitation (PM&R) provides comprehensive and long-term 
rehabilitation to over 10,000 patients each year. The Department of PM&R occupies approximately 60,000 square feet 
in Mayo-Clinic Rochester facilities that include group and individual treatment areas, patient kitchens, a patient 
apartment, fully equipped gyms and treatment areas, and patient computers. The Chronic Disease Rehabilitation 
Program spans an outpatient clinic, inpatient consultation service, and designated beds on the acute inpatient 
rehabilitation ward. Drs. Cheville serves regularly as physician consultant in the Chronic Disease Rehabilitation Clinic. The 
outpatient Chronic Disease Rehabilitation Clinic is housed within the PM&R Department’s 45,688 square foot outpatient 
treatment facility. This facility was renovated in 2009 and includes 20 physician exam rooms, 33 therapy treatment 
rooms, two therapy gyms (PT and OT), 35 physician offices (including Dr. Cheville’s), 41 therapist workstations, eight 
physician workstations, 24 secretarial stations, and three patient education stations.   
Department of Health Sciences Research 
The main activities related to this project take place in the Department of Health Sciences Research (HSR). 
Division of Biomedical Statistics and Informatics  
The Division of Biomedical Statistics and Informatics (BSI) is comprised of over 300 members, including faculty, 
statisticians, bioinformatics specialists, statistical programmer analysts, and support personnel. BSI provides integrated 
collaborative research support in biostatistics, bioinformatics, and medical informatics. The Division has provided 
consultation on design and analysis for the clinical and laboratory research staff at Mayo Clinic since it was introduced in 
1932. The Division of BSI includes 31 PhD statisticians and 17 PhD bioinformaticians who are engaged in collaborative 
research activities with Mayo’s investigators. In addition to working with faculty, investigators can request access to the 
technical expertise provided by 197 MS and BS statisticians and 43 MS bioinformaticians that are respectively part of the 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 15 of 32 
 
technical statistic and bioinformatics core. The BSI team provides their expertise to more than 2,000 ongoing 
investigation projects, many funded by NIH research grants. The BSI is organized in four sections, described below. 
Section of Medical Informatics 
The Section of Medical Informatics supports clinical data normalization, natural language processing, information 
retrieval, machine learning, knowledge management, clinical terminology and formal ontology development and 
deployment. These services are enriched by a broad program of basic informatics research and embedded informatics 
cores in partnership with clinical and basic biology researchers; faculty are PI or co-PI on three core resources, and eight 
R01s. Members of the Section contribute prominently to national and international health informatics and standards 
communities. The Section comprises two senior faculty, five junior faculty, and five research assistants, and support 
personnel. This team is enriched by dedicated research IT support including programmer and analyst resources. The 
team meets weekly for alternating update and methodology seminars. 
Division of Medical Oncology The Division of Medical Oncology includes over 45 physicians who see hundreds of 
patients with cancer weekly. Inpatient and outpatient facilities include state-of-the-art treatment units on Gonda 
building floors 9 and 10, and on three different units at Rochester Methodist Hospital.  
The outpatient clinical area includes 51 examining rooms, an infusion facility with 18 chairs and 13 beds, and a small 
adjacent laboratory for processing blood, urine, and other samples for pharmacologic studies; multiple large rooms for 
multidisciplinary group consultation; an appointment office; offices for oncologists, mid-level providers, and nurses; a 
large waiting room; secretarial space; a patient library; a medical oncology staff library; a seminar room; and a 
conference room. 
Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery 
The Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery (CSHCD) is a Mayo Clinic 
Board designated transformative center that focuses resources and expertise to analyze, evaluate, and implement care 
delivery models that improve value for patients and leads to better health outcomes. Led by Andrew H. Limper, MD, 
Associate Dean of Practice Transformation, and a pulmonologist, the center brings together and expands Mayo Clinic's 
already extensive efforts in health care delivery research; systems engineering; and value, quality and policy analysis. 
The CSHCD was created in 2011 to accelerate the clinic’s efforts to improve care delivery systems. Through the 
integration and application of scientific disciplines (engineering, health services research, epidemiology, and economics), 
this group of researchers will create evidence based and sustainable care delivery systems to provide higher value, 
decrease variability, and increase reliability and quality of care for all patients. The CSHCD serves as both an actual 
physical (housed in the Harwick Building) and virtual connection for health care practitioners and multidisciplinary 
researchers to integrate knowledge and investigate solutions needed for the future of health care.  
The center works with physicians, medical researchers and experts from other academic disciplines (including systems 
engineers, health economists, social scientists, and epidemiologists) to design, study, implement and share new models 
of clinical care. Success requires data analysis, engineering principles and scientific rigor as well as a history of proven 
results translated into patient-centered care. As a result of the Center’s work, patients receive improved health care 
value and better outcomes. Evidence and approaches for higher value care are shared broadly by Center staff. This 
center, which is internally and externally funded (over $20 million), is organized around the following key focus areas 
which make up the center’s five programs: 
Offices  
Drs. Cheville, Rutten, Griffin, Ridgeway and Storlie all have offices on the second floor of the Harwick Building. Dr. 
Cheville has an additionally office on the 14th floor of the Mayo Building.  Drs. Ruddy, Okuno, and Hubbard have offices 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 16 of 32 
 
on the 10th floor of the Gonda building which adjoins the Mayo building.  Dr. Chlan has an office in the Rosa Parks 
Pavilion.  Dr. Leppin has an office on the 3rd floor of the Plummer Building.   
 
  (1a)  This is a multisite study involving Mayo Clinic and non Mayo Clinic sites. When checked, describe in detail the 
research procedures or activities that will be conducted by Mayo Clinic study staff. 
  (1b)  Mayo Clinic study staff will be engaged in research activity at a non Mayo Clinic site.  When checked, provide a 
detailed description of the activity that will be conducted by Mayo Clinic study staff. 
 
Subject Information 
 
Target accrual is the proposed total number of subjects to be included in this study at Mayo Clinic. A “Subject” may 
include medical records, images, or specimens generated at Mayo Clinic and/or received from external sources.    
 
Target accrual:  All patients being followed by Medical Oncology at Mayo Clinic Rochester and Hem Onc in Midwest 
MCHS. 
 
Subject population (children, adults, groups):  Adults 18 years of age or older 
 
Inclusion Criteria:  Being seen for a solid or liquid cancer at a Midwest Mayo Clinic MCHS site or for a solid tumor at 
Mayo Clinic Rochester. 
 
Exclusion Criteria:   No Minors (under age 18) will receive the questionnaires, but no further exclusions can be made 
based on the standard of care for questionnaire assignment.  
 
 
Research Activity 
 
Check all that apply and complete the appropriate sections as instructed.  
  
1. 
  Drug & Device:  Drugs for which an investigational new drug application is not required. Device for which (i) an 
investigational device exemption application is not required; or the medical device is cleared/approved for 
marketing and being used in accordance with its cleared/approved labeling. (Specify in the Methods section) 
 
2. 
  Blood:  Collection of blood samples by finger stick, heel stick, ear stick, or venipuncture.  
 
3. 
  Biological specimens other than blood:  Prospective collection of human biological specimens by noninvasive 
means that may include urine, sweat, saliva, buccal scraping, oral/anal/vaginal swab, sputum, hair and nail clippings, 
etc. 
 
4. 
  Tests & Procedures:  Collection of data through noninvasive tests and procedures routinely employed in clinical 
practice that may include MRI, surface EEG, echo, ultrasound, moderate exercise, muscular strength & flexibility 
testing, biometrics, cognition testing, eye exam, etc.  (Specify in the Methods section) 
 
5. 
   Data (medical record, images, or specimens):  Research involving use of existing and/or prospectively collected 
data. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 17 of 32 
 
 
6. 
  Digital Record:  Collection of electronic data from voice, video, digital, or image recording. (Specify in the 
Methods section) 
 
7. 
  Survey, Interview, Focus Group:  Research on individual or group characteristics or behavior, survey, interview, 
oral history, focus group, program evaluation, etc.  (Specify in the Methods section) 
 NIH has issued a Certificate of Confidentiality (COC).  When checked, provide the institution and investigator named 
on the COC and explain why one was requested. ________________________ 
 
Review of medical records, images, specimens – Category 5 
 
For review of existing data:  
Date Range: 
 
Check all that apply (data includes medical records, images, specimens).  
 
  (5a)  Only data that exists before the IRB submission date will be collected.   
 
 
    (5b)  The study involves data that exist at the time of IRB submission and data that will be generated after IRB 
submission. Include this activity in the Methods section.  
Examples 
• 
The study plans to conduct a retrospective chart review and ask subjects to complete a questionnaire.  
• 
The study plans to include subjects previously diagnosed with a specific disease and add newly diagnosed 
subjects in the future.  
 
  (5c)  The study will use data that have been collected under another IRB protocol. Include in the Methods section 
and enter the IRB number from which the research material will be obtained. When appropriate, note when subjects 
have provided consent for future use of their data and/or specimens as described in this protocol.  
 
Enter one IRB number per line, add more lines as needed 
 
 Data    
 Specimens  
 Data & Specimens  ______________________________________ 
 
 Data    
 Specimens  
 Data & Specimens  ______________________________________ 
 
 Data    
 Specimens  
 Data & Specimens  ______________________________________ 
 
 
  (5d)  This study will obtain data generated from other sources. Examples may include receiving data from 
participating sites or an external collaborator, accessing an external database or registry, etc.  Explain the source and 
how the data will be used in the Methods section.  
 
  (6)  Video audio recording: Describe the plan to maintain subject privacy and data confidentiality, transcription, store 
or destroy, etc.  
 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 18 of 32 
 
All paper-based and digital audio data will be transferred within a locked case or on an encrypted device to secure Mayo 
servers. Names will be removed from any audio transcripts and survey records will use study IDs to identify participants 
after linking their data from a master file. Completed paper surveys will be stored in locked cabinets. 
 HIPAA Identifiers and Protected Health Information (PHI) 
 
Protected health information is medical data that can be linked to the subject directly or through a combination of 
indirect identifiers.  
 
Recording identifiers (including a code) during the conduct of the study allows you to return to the medical record or 
data source to delete duplicate subjects, check a missing or questionable entry, add new data points, etc. De-identified 
data is medical information that has been stripped of all HIPAA identifiers so that it cannot be linked back to the subject. 
De-identified data is rarely used in the conduct of a research study involving a chart review.   
 
Review the list of subject identifiers below and, if applicable, check the box next to each HIPAA identifier being 
recorded at the time of data collection or abstraction.  Identifiers apply to any subject enrolled in the study including 
Mayo Clinic staff, patients and their relatives and household members.  
 
Internal refers to the subject’s identifier that will be recorded at Mayo Clinic by the study staff. 
External refers to the subject’s identifier that will be shared outside of Mayo Clinic. 
 
Check all that apply: 
INTERNAL 
EXTERNAL 
Name 
X 
 
Mayo Clinic medical record or patient registration number, lab accession, specimen 
or radiologic image number  
X 
 
Subject ID, subject code or any other person-specific unique identifying number, 
characteristic or code that can link the subject to their medical data   
X 
 
Dates: All elements of dates [month, day, and year] directly related to an individual, 
their birth date, date of death, date of diagnosis, etc.   
Note: Recording a year only is not a unique identifier.  
X 
 
Social Security number 
 
 
Medical device identifiers and serial numbers 
 
 
Biometric identifiers, including finger and voice prints, full face photographic images 
and any comparable images 
X 
 
Web Universal Resource Locators (URLs), Internet Protocol (IP) address numbers, 
email address 
X 
 
Street address, city, county, precinct, zip code, and their equivalent geocodes 
X 
 
Phone or fax numbers 
X 
 
Account, member, certificate or professional license numbers, health beneficiary 
numbers 
 
 
Vehicle identifiers and serial numbers, including license plate numbers 
 
 
Check ‘None’ when none of the identifiers listed above will be recorded, 
maintained, or shared during the conduct of this study.  (exempt category 4) 
 None 
 None 
 
 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 19 of 32 
 
Data Analysis 
 
Power analyses may not be appropriate if this is a feasibility or pilot study, but end-point analysis plans are always 
appropriate even if only exploratory. Provide all information requested below, or provide justification if not including all 
of the information.  
 
Power Statement:   
 
Stepped wedge cluster randomization trials typically have more 
statistical power than other cluster randomized designs.95 This is 
because each cluster is able to server as its own control, 
accounting directly for the within cluster correlation of 
outcomes. Because of the complex nature of the design and 
statistical model, we estimate statistical power using 
simulation.96 
A version of the statistical model above was fit to 79 weeks of 
preliminary baseline data (i.e., with no intervention effect Xkt ) 
in order to estimate the parameters (e.g., 𝝂஺, 𝜮஺) needed to 
simulate longitudinal data for the 15 clusters.  Clusters were 
randomly allocated to the steps in Figure 1, and treatment effects due to intervention were generated on a grid of eight 
values 𝜈ଵ஻ൌ0.00, 0.01, 0.02, 0.03, 0.05, 0.10, 0.15, 0.20, with the remaining 𝜈ଶ஻ൌ𝜈ଷ஻ൌ𝜈ସ஻ൌ𝜈ହ஻ൌ0 to assume the 
most conservative case (i.e., only a single symptom is affected by the intervention).  The cluster effects were then 
generated according to 𝑩௞∼𝑁ሺ𝝂஻, √𝜈ଵ஻𝜮஺ሻ.  This data generation process was used to simulate 100 data sets under 
each of the eight grid values for 𝜈ଵ஻.  The full model above was then used to estimate 𝝂஻ and test the hypothesis that all 
𝜈௝஻ൌ0, via simultaneous credible bounds .97  .  The results are provided in Figure 3  As can be seen in the figure, the 
test keeps very close to the nominal level (i.e., 0.01, 0.05, or 0.10, respectively) when there is no effect. However, the 
power rises quickly to detect even a small average score difference of even 0.10 with 70% power (at 0.05 LOS) and a 
difference of 0.20 with nearly 100% power.  
Data Analysis Plan:  
Specific Aim 1 Analyses.  
We will summarize patient-, cluster-, and survey-level characteristics (e.g., sex, age, cancer type and stage, insurance 
status) by intervention status (pre-E2C2, E2C2) of the encounter.  All analyses will be conducted on an intention to treat 
basis. The primary analysis will test differences in change in joint SPPADE symptom NRS scores in all participants, and in 
those who report elevated (4-10/10) symptoms, between participants who are exposed to the intervention and those 
who are not.  We have two primary hypotheses.  The first will assess the effect of the intervention across the full cohort 
of patients enrolled, Hypothesis H1a.  The second will assess the effect of the intervention among patients who report 
moderate or worse symptoms, Hypothesis H1b. The rationale for Hypothesis H1b is that some intervention components 
are only offered to participants who report elevated symptom scores.  Additionally, a large proportion of the trial cohort 
will only report mild or no symptoms.  Among these participants there is little or no opportunity to improve symptoms 
and no practical exposure to the intervention. The original hypotheses regarding population-level effects will retain 
primary importance, yet it is critical to test intervention effects in the subset of patients that ever have elevated 
symptom scores.   
 
We will include a common set of variables in all models testing the effectiveness of the intervention. Cancer type and 
stage will be included, as well as 7 variables selected as likely to impact the use of the portal and completion of the 
Figure 3.  Power simulations for E2C2 Interventions 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 20 of 32 
 
surveys: from the EHR we will capture age (categorized into 10-year increments), sex (male or female), and severity of 
illness (Elixhauser score as a categorical variable). We will also link the patient 9-digit zip code to Census data to get 
estimates of educational level, income, and broadband access, which will be categorized into quartiles.  
 
Hypotheses H1a 
To test the primary hypothesis H1a for our primary outcomes, we will use generalized linear mixed models to assess the 
effects of the intervention on each of the 6 SPPADE scores90.  All models will account for correlation of outcomes within 
clusters and across patients, and for changes in this correlation over time. We will adjust for time, measured by study 
period, to account for temporal effects.  Each score will be adjusted for the prior score, with a coefficient that depends 
on the time t since the prior score; to account for potential changes in correlation of outcomes by over time we will 
include autoregressive terms at the patient level.91 To account for changing correlation within clusters over time we will 
incorporate a discrete-time correlation structure.98   
 
Our primary analysis will treat each SPPADE score as a continuous variable and employ a mixed effects generalized 
multivariate model; we will model all 6 scores jointly, which will allow us to account for the correlation between them 
and avoid correcting for multiple hypothesis tests.  
 
Specifically, let Yijkm be the ith score (i=1,…,6) of the jth encounter for the kth patient in the mth cluster (m =1,…,15), for j = 
2,… ; the first score for each patient (j=1) is a baseline score collected during the first visit, not an outcome. Let w=1,…,6 
indicate the trial step during which the survey is collected.  
 
Then to test the hypothesis that the intervention impacts the 6 symptom scores we will estimate the joint models: 
 
𝑌௜௝௞௠= 𝐴௜௞௠+ 𝑩𝒊𝑋௝௞௠+ 𝑳𝒊(𝑡)𝑌௜ሾ௝ିଵሿ௞௠+ 
  𝑭𝒁𝒌+ 𝑫𝑺𝒎+ 𝑸𝑞௝௠௞+ 𝜇௜௠௪+ 𝜈௜௧          ;   i=1,…,6            (1) 
where   
 
 X௝௞௠ is an intervention indicator which equals 1 if encounter j for patient k in cluster m was post intervention and 0 
otherwise; 
𝑩𝒊 is a vector of fixed effects for the intervention on symptom score i, i = 1,…,6; 
𝑳𝒊 is a vector of time dependent lags, representing the effect of the most recent measurement on the current one, 
as a function of time t in days between measures;  
Zk is a vector of patient factors for patient k, F is a corresponding vector of fixed effects;   
 
Sm is a cluster level set of indicators for randomization strata, D is a vector of fixed effects;  
Q is a vector representing the secular time effect, where qjmk is the calendar quarter; and,  
 the time elapsed since encounter j-1; for j=2,….  
 
In addition 
 
Aikm is a vector of random effects, [A1km,…,A5km] specified as varying across patients nested within clusters 
 
 
Aikm ~ [Ai + αikm + γim ];  i=1,..,6 
[αikm] ~ N(0,Σα) 
[γim ] ~ N(0,Σγ) 
 
where Σs are exchangeable correlation matrices. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 21 of 32 
 
 
 
μimw , i=1,…,6 is a discrete-time vector of random effects for each cluster m and step w; and, 
 
νit , i=1,…,6 is a multivariate autoregressive process of order 1, AR(1) with Gaussian noise, to account for time 
dependent correlation of responses within each patient over time; and, 
 
Prior to estimating this model, the distributions of the SPPADE scores will be examined and, if appropriate, transformed 
to have approximately normal distributions. Covariates will be similarly reviewed and either transformed or categorized 
before inclusion in the model.  
 
In general, fitting the above models (1) which has several random effects (in both time and across clusters) can be 
challenging. However, Markov Chain Monte Carlo estimation is very effective at fitting such models and are appropriate 
for assessing stepped-wedge cluster randomized designs.99  Moreover, by incorporating historical time trends into our 
priors for the effect of calendar time, a Bayesian model can increase statistical power relative to a frequentist 
approach.100  All other priors will be flat. In the event that model (1) can’t be estimated, we will simplify the error 
structure and other features using review of variable distributions and correlations, including intra-cluster correlations 
and time dependent correlation, to inform the simplifications. Then, using our final model we will calculate posterior 
probabilities and credible intervals to assess whether coefficients differ from zero; specifically, we will achieve the 
primary aim of the study by calculating the posterior probabilities that each of  𝐵ଵ, … , 𝐵଺ is greater than 0; we will 
accept that the intervention is effective if PP[abs(Bi ) > 0, i=1,..6] ≥ 0.80.   
 
In the event that the model with patient surveys as the unit of analysis cannot be computed, we will use an alternative 
cluster-period model for the primary analysis.  We will aggregate outcomes and covariates for each cluster for each time 
period and use these cluster-level estimates to assess the effect of the intervention. We will calculate the mean of each 
of the 6 different SPPADE NRS scores collected during trial cluster-periods. We will also calculate the mean baseline 
score for each participant that reports one or more ePROMs. We will then test the first primary hypothesis using a 
multivariate regression model using these 15 clusters x 6 period = 90 cluster-period values, including an indicator for 
intervention status for each cluster-period, and adjusting for mean baseline scores and including indicators for study 
step to account for secular trends and for indicators for cluster to account for cancer type. We will adjust for cluster-
period averaged values of factors of patients reporting during each period for which the SMD between intervention and 
control cluster-periods are >0.2. 
 
To assess the robustness of using the CP model, we will replicate the main analyses for each symptom NRS score.  We 
will use the originally specified generalized mixed effects model, as well as the CP model adapted to the univariate 
dependent variable. This former model will be estimated at the survey level, with random effects for surveys within 
patients and patients within clusters; these 6 models (one for each symptom score) also included all pre-specified 
patient covariates, baseline symptom score, the lag in weeks since the baseline measurement and the interaction of the 
lag time with the baseline effect.  
 
Because there is potential for the intervention to impact the baseline scores as well, we will replicate the cluster-period 
and survey level models each symptom score, including patients with only 1 survey score. 
 
We will estimate models using the STAN software package.  
 
Hypotheses H1b 
To test whether the intervention has a differential effect on those patients with at least one elevated symptom score, 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 22 of 32 
 
we will construct an indicator, Eijkm, varying by measurement time, which indicates whether the ith prior symptom score 
was elevated (>3).  By interacting these indicators with the main intervention effects Bi  we can assess the impact of the 
intervention on the corresponding scores of patients with at least one elevated score:  
 
𝑌௜௝௞௠= 𝐴௜௞௠+ 𝑩𝒊𝑋௝௞௠+ 𝑪𝐸௜௝௞௠+ 𝑫𝒊𝐸௜௝௞௠𝑋௝௞௠+ 𝑳𝒊(𝑡)𝑌௜ሾ௝ିଵሿ௞௠+ 
  𝑭𝒁𝒌+ 𝑫𝑺𝒎+ 𝑸𝑞௝௠௞+ 𝜇௜௠௪+ 𝜈௜௧          ;   i=1,…,6            (2) 
 
 
We will then test the hypothesis as above by calculating the PP[Abs(Di)>0, i=1,…,6] 
 
 
In the event that the model with patient surveys as the unit of analysis cannot be computed to test Hypotheses H1b, we 
will use the cluster-period approach outlined for testing H1a, but will summarize SPPADE NRS scores for each cluster-
period according to whether the patient had any prior elevated scores; we will then estimate a model analogous to the 
cluster-period model used to test Hypothesis H1a but including an indicator for post-elevation and the interaction with 
the intervention indicator. 
Finally, as a secondary objective of this aim, to better understand the effectiveness of different components of the E2C2 
intervention in reducing symptoms, we will perform a separate set of analyses to assess the impact of specific process 
measures (D.2.d) on SPPADE scores and other clinical outcomes. For each process measure we will first examine the 
distribution of values (frequency or duration) and, if highly skewed, categorize into 2 or more categories. We will then 
estimate for each outcome a mixed effects linear model with a random cluster effect, where the outcome is the 
dependent variable and the main independent variable is the continuous or categorized process measure, adjusting for 
patient covariates as described above. By testing for an overall process measure effect, we can assess which processes 
contribute most to reducing outcomes; we will also report measures of variance explained to enable relative comparison 
of process measure effects. 
Specific Aim 2 Analyses. We will summarize patient characteristics (sex, age, insurance status) by intervention status 
(baseline, intervention). We will also summarize all survey responses by cluster implementation status, including 
clinician characteristics (age, sex, years of practice). All patients will be analyzed on an intention to treat status; this 
principle will be extended to the cluster status, so that delays in implementation of an intervention will not affect the 
intervention status of patients.101 All methods will account for correlation of outcomes within clusters and across 
patients. As a secondary objective of this aim, we will assess the change in clinician survey constructs over time. This 
analysis will include each clinician with at least two completed surveys at least 6 months apart.    
As in Aim 1, we will also assess the effect of process measures on SPPADE scores and other clinical outcomes. We will 
use models appropriate for count data, selecting standard or zero-inflated Poisson or negative binomial models 
according to Aikake’s Information Criteria, or logit models for binary measures. For each outcome we will estimate a) 
bivariate models including only one survey score; b) a model with all scores; and c) a final model which also includes 
clinician characteristics.  
Aim 3 
Endpoints 
Primary:  
SPPADE and physical function NRSs will serve as both the E2C2 primary outcome and a critical component of the E2C2 
intervention since NRS scores populate the EHR algorithms that determine whether patients are triaged to Level 1 or 2, 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 23 of 32 
 
and whether patients are offered specific intervention components (education, medications, etc.).  Patients will 
complete NRSs for SPPADE symptoms and physical function prior to medical oncology clinic appointments at all stages of 
the trial, i.e., prior to and after E2C2 initiation. ePROs will be more frequently administered to symptomatic patients 
outside of their appointment-linked assessments as part of the intervention.  To avoid potential bias from oversampling 
of symptomatic patients, only SPPADE symptom NRSs collected in association with clinic appointments will be included in 
the primary analysis.  
NRSs.  Simple 11-point NRSs have been used for decades in the assessment of diverse latent traits among patients with 
cancer of all types and stages.102-104  These single-item assessments have been the most often-used QOL and symptom 
measures in National Cancer Institute (NCI) cancer control clinical trials,105  and they feature prominently in the most 
reliable and valid symptom assessment tools, both generic and cancer-specific.106,107 The NRSs utilized in E2C2 to assess 
the SPPADE symptoms and physical function will be patterned on the extensively validated MD Anderson Symptom 
Assessment Scale and  Edmonton Symptom Assessment Scale with respect to the  24 hour recall period, and verbal 
anchors identical to the, e.g., “No pain” and “Worst possible pain,”respectively.108,109  NRSs have been shown to be more 
responsive than either Likert or verbal rating scales,110 and preferred by patients over visual analogue scales.111  
 
Secondary:  
PROMIS CATs. Depression, anxiety, pain, and physical function will be assessed with the PROMIS CATs 
(www.promis.org).  In addition to strong endorsement by the NIH,112,113  use of CATs is justified because IRT-based 
instruments have generally better discrimination across the entire trait range than legacy PROs, being less prone to floor 
and ceiling effects.114,115  Moreover, administering IRT-modeled banks with CATs enhances the efficiency and precision of 
measurement relative to short forms.116,117 The PROMIS item banks utilized in E2C2 have been validated, with robust IRT 
calibration across different clinical populations.118-120 
Health care utilization. Healthcare utilization for E2C2 will consider health care encounters including hospitalizations, ED 
visits, ICU admissions, clinic outpatient visits, and calls to the oncology care team.  EHR entries and administrative billing 
data will be aggregated to construct a comprehensive data set of all clinical encounters.  Data collected for 
hospitalizations will include admission and discharge diagnoses, length of stay, ICU admission/transfer, and whether an 
admission was planned for anti-cancer treatment or unplanned, and if the latter, initiated from the ED or an office visit.  
For ED encounters, we will capture diagnoses and whether an encounter resulted in hospital or ICU admission.  Clinic 
visits will be captured using billing data which will include CPT codes, ICD-10 codes, location, and clinician NPI numbers.  
We will be able to ascertain clinician discipline and specialty from the NPI numbers. Calls to the oncology care team are 
not billed but are reliably captured in the EHR.  A recent audit established that 87% of calls to medical oncology 
practitioners were recorded in the EHR, and that rates of capture did not vary systematically across disciplines.  
However, call characteristics, e.g., duration, were not reliably recorded.  Therefore we will use count data in analyses to 
model call frequency.   
We will estimate mixed effects Poisson regression models with utilization events during the trial period as dependent 
variables. Because utilization events will be available for all trial participants, we will perform analyses with cohorts 
comprised of all trial participants and those who completed >1 ePROM, and > 2 ePROMs.   
The majority, but not all, of patients’ health care utilization will occur at MC facilities.  Established patterns suggest that 
75% to 80% of patients care will occur at an MC facility and that this frequency will not vary across E2C2 clusters, or 
phases of the E2C2 intervention, e.g., pre-E2C2, Stage 1.  However, we will perform analyses that include only patients 
residing within driving distance of a trial site to assess the impact of distance.  Additionally, we will use imputation 
methods and sensitivity analyses to assess the impact of mis-ascertainment of utilization data.    
Cancer treatment adherence.  The literature does not offer a definitive definition of cancer treatment adherence, even 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 24 of 32 
 
for medications, or clarify which of the many candidate measures should be considered.121 We will therefore assess 
adherence broadly and examine the frequency of missed imaging, medical oncology clinic visits, and chemotherapy 
appointments.  Additionally, we will capture any unplanned hiatuses in cancer treatment or dose reductions.  This 
information is routinely recorded on electronic chemotherapy administration forms from which discrete structured data 
elements can be abstracted.  For oral hormonal regimens we will use the accepted definition of at least 80% of 
recommended doses to create a binary variable.122  Dr. Ruddy (E2C2 Co-I) has contributed seminally to the literature on 
adherence to breast cancer adjuvant therapies.123,124   We will apply this criterion to oral chemotherapy and 
immunotherapy as well.  
Vital status. Vital status will be verified through death certificates, the Mayo Clinic EMR, next-of-kin reports, the Mayo 
Clinic Tumor Registry and the Social Security Death Index website.   
Reporting mode and rates.  Patient reported outcomes will be collected via several modes for the E2C2 trial.  These 
include remote modes; the patient portal and interactive voice response, and point of care modes;  tablets and provider 
entry via Epic Care.  Data regarding patients’ mode of PRO reporting are automatically stored by the Epic EHR.  We will 
perform descriptive statistics to characterize how reporting modes differ across subgroups defined by demographic, 
clinical, cancer-related, and symptom characteristics.  Additionally, we will create uni- and multi-variate mixed models to 
estimate associations of reporting frequency and mode with patient characteristics. 
Receptivity to receipt of symptom support.  The E2C2 intervention provides low and high touch support for moderate 
and severe symptoms, respectively.  Patients who endorse moderate symptoms (4-6/10) are offered the option to 
receive symptom self-management education materials for specific symptoms, Sleep disturbance, Pain, Anxiety, 
Depression, Fatigue, and Deficits in physical function.  Similarly, patients who endorse severe symptoms (7-10/10) are 
offered the opportunity to discuss the same symptoms with a nurse symptom care manager.   We will calculate 
descriptive statistics to characterize patients’ preferences with regard to receipt of symptom support in subgroups 
defined by symptom type and intensity, as well as the stability of their preferences for symptom support. Additionally, 
we will estimate associations between patients’ demographic and clinical characteristics and their interest in receiving 
support by constructing multivariate mixed models. 
 
 
BIBLIOGRAPHY AND REFERENCES CITED 
 
1. 
Cheville AL, Shen T, Chang M, Basford JR. Appropriateness of the treatment of fatigued patients with 
stage IV cancer. Support Care Cancer 2013;21(1):229-33. DOI: 10.1007/s00520-012-1515-7. 
2. 
Barsevick AM, Sweeney C, Haney E, Chung E. A systematic qualitative analysis of psychoeducational 
interventions for depression in patients with cancer. Oncology nursing forum 2002;29(1):73-84; quiz 85-
7. DOI: 10.1188/02.ONF.73-87. 
3. 
Laoutidis ZG, Mathiak K. Antidepressants in the treatment of depression/depressive symptoms in cancer 
patients: a systematic review and meta-analysis. BMC Psychiatry 2013;13:140. DOI: 10.1186/1471-
244X-13-140. 
4. 
Boyce MB, Browne JP. Does providing feedback on patient-reported outcomes to healthcare 
professionals result in better outcomes for patients? A systematic review. Qual Life Res 
2013;22(9):2265-78. (In eng). DOI: 10.1007/s11136-013-0390-0. 
5. 
Mooney KH, Beck SL, Friedman RH, Farzanfar R, Wong B. Automated monitoring of symptoms 
during ambulatory chemotherapy and oncology providers' use of the information: a randomized 
controlled clinical trial. Support Care Cancer 2014;22(9):2343-50. DOI: 10.1007/s00520-014-2216-1. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 25 of 32 
 
6. 
Carpenter JS, Rawl S, Porter J, et al. Oncology outpatient and provider responses to a computerized 
symptom assessment system. Oncol Nurs Forum 2008;35(4):661-9. DOI: 10.1188/08/ONF.661-669. 
7. 
Yount SE, Rothrock N, Bass M, et al. A randomized trial of weekly symptom telemonitoring in 
advanced lung cancer. J Pain Symptom Manage 2014;47(6):973-89. DOI: 
10.1016/j.jpainsymman.2013.07.013. 
8. 
Katon W, Guico-Pabia CJ. Improving quality of depression care using organized systems of care: a 
review of the literature. Prim Care Companion CNS Disord 2011;13(1). DOI: 
10.4088/PCC.10r01019blu. 
9. 
Katon W, Unutzer J, Wells K, Jones L. Collaborative depression care: history, evolution and ways to 
enhance dissemination and sustainability. Gen Hosp Psychiatry 2010;32(5):456-64. (In eng). DOI: 
10.1016/j.genhosppsych.2010.04.001 
S0163-8343(10)00062-9 [pii]. 
10. 
Coventry PA, Hudson JL, Kontopantelis E, et al. Characteristics of effective collaborative care for 
treatment of depression: a systematic review and meta-regression of 74 randomised controlled trials. 
PLoS One 2014;9(9):e108114. DOI: 10.1371/journal.pone.0108114. 
11. 
Sighinolfi C, Nespeca C, Menchetti M, Levantesi P, Belvederi Murri M, Berardi D. Collaborative care 
for depression in European countries: a systematic review and meta-analysis. J Psychosom Res 
2014;77(4):247-63. DOI: 10.1016/j.jpsychores.2014.08.006. 
12. 
Craven MA, Bland R. Better practices in collaborative mental health care: an analysis of the evidence 
base. Can J Psychiatry 2006;51(6 Suppl 1):7S-72S. (https://www.ncbi.nlm.nih.gov/pubmed/16786824). 
13. 
Archer J, Bower P, Gilbody S, et al. Collaborative care for depression and anxiety problems. Cochrane 
Database Syst Rev 2012;10:CD006525. DOI: 10.1002/14651858.CD006525.pub2. 
14. 
van Straten A, Hill J, Richards DA, Cuijpers P. Stepped care treatment delivery for depression: a 
systematic review and meta-analysis. Psychol Med 2015;45(2):231-46. DOI: 
10.1017/S0033291714000701. 
15. 
Hudson JL, Bower P, Archer J, Coventry PA. Does collaborative care improve social functioning in 
adults with depression? The application of the WHO ICF framework and meta-analysis of outcomes. J 
Affect Disord 2016;189:379-91. DOI: 10.1016/j.jad.2015.09.034. 
16. 
Tully PJ, Baumeister H. Collaborative care for comorbid depression and coronary heart disease: a 
systematic review and meta-analysis of randomised controlled trials. BMJ open 2015;5(12):e009128. 
DOI: 10.1136/bmjopen-2015-009128. 
17. 
Panagioti M, Bower P, Kontopantelis E, et al. Association Between Chronic Physical Conditions and the 
Effectiveness of Collaborative Care for Depression: An Individual Participant Data Meta-analysis. 
JAMA Psychiatry 2016;73(9):978-89. DOI: 10.1001/jamapsychiatry.2016.1794. 
18. 
Kroenke K, Krebs EE, Wu J, Yu Z, Chumbler NR, Bair MJ. Telecare collaborative management of 
chronic pain in primary care: a randomized clinical trial. JAMA 2014;312(3):240-8. DOI: 
10.1001/jama.2014.7689. 
19. 
Cheville AL, Moynihan T, Basford JR, et al. The rationale, design, and methods of a randomized, 
controlled trial to evaluate the effectiveness of collaborative telecare in preserving function among 
patients with late stage cancer and hematologic conditions. Contemp Clin Trials 2017. DOI: 
10.1016/j.cct.2017.08.021. 
20. 
Mooney KH, Beck SL, Wong B, et al. Automated home monitoring and management of patient-reported 
symptoms during chemotherapy: results of the symptom care at home RCT. Cancer Med 2017;6(3):537-
546. DOI: 10.1002/cam4.1002. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 26 of 32 
 
21. 
Cleeland CS, Wang XS, Shi Q, et al. Automated symptom alerts reduce postoperative symptom severity 
after cancer surgery: a randomized controlled clinical trial. J Clin Oncol 2011;29(8):994-1000. (In eng). 
DOI: 10.1200/JCO.2010.29.8315 
JCO.2010.29.8315 [pii]. 
22. 
Gilbody S, Sheldon T, House A. Screening and case-finding instruments for depression: a meta-analysis. 
CMAJ 2008;178(8):997-1003. (In eng). DOI: 10.1503/cmaj.070281 
178/8/997 [pii]. 
23. 
Kroenke K, Unutzer J. Closing the False Divide: Sustainable Approaches to Integrating Mental Health 
Services into Primary Care. J Gen Intern Med 2017;32(4):404-410. DOI: 10.1007/s11606-016-3967-9. 
24. 
Mularski RA, White-Chu F, Overbay D, Miller L, Asch SM, Ganzini L. Measuring pain as the 5th vital 
sign does not improve quality of pain management. J Gen Intern Med 2006;21(6):607-12. DOI: 
10.1111/j.1525-1497.2006.00415.x. 
25. 
Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in 
clinical practice: a systematic review of the literature. Qual Life Res 2008;17(2):179-93. DOI: 
10.1007/s11136-007-9295-0. 
26. 
Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported 
outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health 
Serv Res 2013;13:211. DOI: 10.1186/1472-6963-13-211. 
27. 
Hsiang H, Karen MT, Joseph MC, Nahida A, Amritha B, Rachel K. Collaborative Care for Women 
With Depression: A Systematic Review. Psychosomatics 2017;58(1):11-18. DOI: 
10.1016/j.psym.2016.09.002. 
28. 
Nierenberg AA, McIntyre RS, Sachs GS. Improving outcomes in patients with bipolar depression: a 
comprehensive review. J Clin Psychiatry 2015;76(3):e10. DOI: 10.4088/JCP.13091ip1. 
29. 
Atlantis E, Fahey P, Foster J. Collaborative care for comorbid depression and diabetes: a systematic 
review and meta-analysis. BMJ Open 2014;4(4):e004706. DOI: 10.1136/bmjopen-2013-004706. 
30. 
van Eck van der Sluijs JF, Castelijns H, Eijsbroek V, Rijnders CAT, van Marwijk HWJ, van der Feltz-
Cornelis CM. Illness burden and physical outcomes associated with collaborative care in patients with 
comorbid depressive disorder in chronic medical conditions: A systematic review and meta-analysis. 
Gen Hosp Psychiatry 2017;50:1-14. DOI: 10.1016/j.genhosppsych.2017.08.003. 
31. 
Jacob V, Chattopadhyay SK, Sipe TA, et al. Economics of collaborative care for management of 
depressive disorders: a community guide systematic review. Am J Prev Med 2012;42(5):539-49. DOI: 
10.1016/j.amepre.2012.01.011. 
32. 
Boland L, Bennett K, Connolly D. Self-management interventions for cancer survivors: a systematic 
review. Support Care Cancer 2017. DOI: 10.1007/s00520-017-3999-7. 
33. 
Bennett S, Pigott A, Beller EM, Haines T, Meredith P, Delaney C. Educational interventions for the 
management of cancer-related fatigue in adults. Cochrane Database Syst Rev 2016;11:CD008144. DOI: 
10.1002/14651858.CD008144.pub2. 
34. 
Kobleder A, Raphaelis S, Glaus A, et al. Recommendations for symptom management in women with 
vulvar neoplasms after surgical treatment: An evidence-based guideline. Eur J Oncol Nurs 2016;25:68-
76. DOI: 10.1016/j.ejon.2016.10.003. 
35. 
Hammer MJ, Ercolano EA, Wright F, Dickson VV, Chyun D, Melkus GD. Self-management for adult 
patients with cancer: an integrative review. Cancer Nurs 2015;38(2):E10-26. DOI: 
10.1097/NCC.0000000000000122. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 27 of 32 
 
36. 
Te Boveldt N, Vernooij-Dassen M, Leppink I, Samwel H, Vissers K, Engels Y. Patient empowerment in 
cancer pain management: an integrative literature review. Psychooncology 2014;23(11):1203-11. DOI: 
10.1002/pon.3573. 
37. 
Howell D, Harth T, Brown J, Bennett C, Boyko S. Self-management education interventions for patients 
with cancer: a systematic review. Support Care Cancer 2017;25(4):1323-1355. DOI: 10.1007/s00520-
016-3500-z. 
38. 
Lovell MR, Luckett T, Boyle FM, Phillips J, Agar M, Davidson PM. Patient education, coaching, and 
self-management for cancer pain. J Clin Oncol 2014;32(16):1712-20. DOI: 10.1200/JCO.2013.52.4850. 
39. 
Kroenke K, Theobald D, Wu J, et al. Effect of telecare management on pain and depression in patients 
with cancer: a randomized trial. JAMA 2010;304(2):163-71. DOI: 10.1001/jama.2010.944. 
40. 
Cheville AL, Moynihan T, Basford JR, et al. The rationale, design, and methods of a randomized, 
controlled trial to evaluate the effectiveness of collaborative telecare in preserving function among 
patients with late stage cancer and hematologic conditions. Contemp Clin Trials 2018;64:254-264. DOI: 
10.1016/j.cct.2017.08.021. 
41. 
Basch E, Deal AM, Dueck AC, et al. Overall Survival Results of a Trial Assessing Patient-Reported 
Outcomes for Symptom Monitoring During Routine Cancer Treatment. JAMA 2017;318(2):197-198. 
DOI: 10.1001/jama.2017.7156. 
42. 
Basch E, Deal AM, Kris MG, et al. Symptom Monitoring With Patient-Reported Outcomes During 
Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol 2016;34(6):557-65. DOI: 
10.1200/JCO.2015.63.0830. 
43. 
Kroenke K, Cheville AL. Symptom Improvement Requires More Than Screening and Feedback. J Clin 
Oncol 2016;34(27):3351-2. DOI: 10.1200/JCO.2016.67.7708. 
44. 
Kolb NA, Smith AG, Singleton JR, et al. Chemotherapy-related neuropathic symptom management: a 
randomized trial of an automated symptom-monitoring system paired with nurse practitioner follow-up. 
Support Care Cancer 2017. DOI: 10.1007/s00520-017-3970-7. 
45. 
Goldzweig CL, Orshansky G, Paige NM, et al. Electronic health record-based interventions for 
improving appropriate diagnostic imaging: a systematic review and meta-analysis. Ann Intern Med 
2015;162(8):557-65. DOI: 10.7326/M14-2600. 
46. 
Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic 
review with a focus on meaningful use. Ann Intern Med 2014;160(1):48-54. DOI: 10.7326/M13-1531. 
47. 
Cella D, Choi S, Garcia S, et al. Setting standards for severity of common symptoms in oncology using 
the PROMIS item banks and expert judgment. Qual Life Res 2014;23(10):2651-61. DOI: 
10.1007/s11136-014-0732-6. 
48. 
Cella D, Choi S, Rosenbloom S, et al. A novel IRT-based case-ranking approach to derive expert 
standards for symptom severity. Quality of Life Research 2008;17(1):A-32. 
49. 
SR E. Health Economics. Efficiency, quality, and equity. Westport, CT: Auburn House, 1992. 
50. 
Lorig KR, Sobel DS, Stewart AL, et al. Evidence suggesting that a chronic disease self-management 
program can improve health status while reducing hospitalization: a randomized trial. Med Care 
1999;37(1):5-14. (https://www.ncbi.nlm.nih.gov/pubmed/10413387). 
51. 
Forte GJ, Hanley A, Hagerty K, Kurup A, Neuss MN, Mulvey TM. American Society of Clinical 
Oncology National Census of Oncology Practices: preliminary report. J Oncol Pract 2013;9(1):9-19. 
DOI: 10.1200/JOP.2012.000826. 
52. 
American Society of Clinical O. The State of Cancer Care in America, 2017: A Report by the American 
Society of Clinical Oncology. J Oncol Pract 2017;13(4):e353-e394. DOI: 10.1200/JOP.2016.020743. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 28 of 32 
 
53. 
Balch C, Ogle JD, Senese JL. The National Practice Benchmark for Oncology: 2015 Report for 2014 
Data. J Oncol Pract 2016;12(4):e437-75. DOI: 10.1200/JOP.2015.008458. 
54. 
Barsevick AM. The elusive concept of the symptom cluster. Oncol Nurs Forum 2007;34(5):971-980. 
(http://www.ncbi.nlm.nih.gov/pubmed/17878126). 
55. 
Barsevick AM, Whitmer K, Nail LM, Beck SL, Dudley WN. Symptom cluster research: conceptual, 
design, measurement, and analysis issues. J Pain Symptom Manage 2006;31(1):85-95. 
(http://www.ncbi.nlm.nih.gov/pubmed/16442485). 
56. 
Barsevick AM. The concept of symptom cluster. Semin Oncol Nurs 2007;23(2):89-98. 
(http://www.ncbi.nlm.nih.gov/pubmed/17512435). 
57. 
Dodd MJ, Miaskowski C, Paul SM. Symptom clusters and their effect on the functional status of 
patients with cancer. Oncol Nurs Forum 2001;28(3):465-470. 
(http://www.ncbi.nlm.nih.gov/pubmed/11338755). 
58. 
Fox SW, Lyon DE. Symptom clusters and quality of life in survivors of lung cancer. Oncol Nurs Forum 
2006;33(5):931-936. (http://www.ncbi.nlm.nih.gov/pubmed/16955121). 
59. 
Given BA, Given CW, Sikorskii A, Hadar N. Symptom clusters and physical function for patients 
receiving chemotherapy. Semin Oncol Nurs 2007;23(2):121-126. 
(http://www.ncbi.nlm.nih.gov/pubmed/17512439). 
60. 
Ferreira KA, Kimura M, Teixeira MJ, et al. Impact of cancer-related symptom synergisms on health-
related quality of life and performance status. J Pain Symptom Manage 2008;35(6):604-616. 
(http://www.ncbi.nlm.nih.gov/pubmed/18362059). 
61. 
Hadi S, Fan G, Hird AE, Kirou-Mauro A, Filipczak LA, Chow E. Symptom clusters in patients with 
cancer with metastatic bone pain. J Palliat Med 2008;11(4):591-600. 
(http://www.ncbi.nlm.nih.gov/pubmed/18454612). 
62. 
Carr D, Goudas L, Lawrence D, et al. Management of cancer symptoms: Pain, depression, and fatigue.  
Evidence Report/Technology Assessment No. 61 (Prepared by the New England Medical Center 
Evidence-based Practice Center under contract No 290-97-0019).  AHRQ Publication No. 02-E032. 
Rockville, MD: Agency for Healthcare Research and Quality; 2002. 
63. 
Donovan KA, Jacobsen PB. Fatigue, depression, and insomnia: evidence for a symptom cluster in 
cancer. Semin Oncol Nurs 2007;23(2):127-135. (http://www.ncbi.nlm.nih.gov/pubmed/17512440). 
64. 
Fleishman SB. Treatment of symptom clusters: pain, depression, and fatigue. J Natl Cancer Inst Monogr 
2004(32):119-123. (http://www.ncbi.nlm.nih.gov/pubmed/15263052). 
65. 
So WK, Marsh G, Ling WM, et al. The symptom cluster of fatigue, pain, anxiety, and depression and 
the effect on the quality of life of women receiving treatment for breast cancer: a multicenter study. 
Oncol Nurs Forum 2009;26:205-214. 
66. 
Yamagishi A, Morita T, Miyashita M, Kimura F. Symptom prevalence and longitudinal follow-up in 
cancer outpatients receiving chemotherapy. J Pain Symptom Manage 2009;37(5):823-830. 
(http://www.ncbi.nlm.nih.gov/pubmed/18804946). 
67. 
Brown LF, Kroenke K. Cancer-related fatigue and its association with depression and anxiety: a 
systematic literature review. Psychosomatics 2009;50:440-447. 
68. 
J F. Epic: Past Accomplishments and Future Challenges.  2017 Epic Users' Group Meeting. Verona, 
WI2017. 
69. 
Wright A, Sittig DF, Ash JS, Sharma S, Pang JE, Middleton B. Clinical decision support capabilities of 
commercially-available clinical information systems. J Am Med Inform Assoc 2009;16(5):637-44. DOI: 
10.1197/jamia.M3111. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 29 of 32 
 
70. 
KFF. Percent of People Covered By Medicaid/CHIP, 2015. The Henry J. Kaiser Family Foundation. 
(https://www.kff.org/interactive/medicaid-state-fact-sheets/). 
71. 
American, Cancer, Society. Cancer Facts & Figures 2018. Atlanta: American Cancer Society; 2018. 
72. 
Miller FG. Research on medical records without informed consent. J Law Med Ethics 2008;36(3):560-6. 
DOI: 10.1111/j.1748-720X.2008.304.x. 
73. 
Wilfond BS, Magnus D, Antommaria AH, et al. The OHRP and SUPPORT. N Engl J Med 
2013;368(25):e36. DOI: 10.1056/NEJMc1307008. 
74. 
Macklin R, Shepherd L, Dreger A, et al. The OHRP and SUPPORT--another view. N Engl J Med 
2013;369(2):e3. DOI: 10.1056/NEJMc1308015. 
75. 
Largent EA, Joffe S, Miller FG. Can research and care be ethically integrated? Hastings Cent Rep 
2011;41(4):37-46. (https://www.ncbi.nlm.nih.gov/pubmed/21845922). 
76. 
Faden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a 
learning health care system: a departure from traditional research ethics and clinical ethics. Hastings 
Cent Rep 2013;Spec No:S16-27. DOI: 10.1002/hast.134. 
77. 
Kass NE, Faden RR, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. The research-treatment 
distinction: a problematic approach for determining which activities should have ethical oversight. 
Hastings Cent Rep 2013;Spec No:S4-S15. DOI: 10.1002/hast.133. 
78. 
Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of 
qualitative data in multi-disciplinary health research. BMC medical research methodology 
2013;13(1):117. 
79. 
Srivastava A, Thomson SB. Framework analysis: a qualitative methodology for applied policy research.  
2009. 
80. 
Snyder CF, Smith KC, Bantug ET, et al. What do these scores mean? Presenting patient-reported 
outcomes data to patients and clinicians to improve interpretability. Cancer 2017;123(10):1848-1859. 
DOI: 10.1002/cncr.30530. 
81. 
Miller K, Mosby D, Capan M, et al. Interface, information, interaction: a narrative review of design and 
functional requirements for clinical decision support. J Am Med Inform Assoc 2017. DOI: 
10.1093/jamia/ocx118. 
82. 
Main C, Moxham T, Wyatt JC, Kay J, Anderson R, Stein K. Computerised decision support systems in 
order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the 
effects and cost-effectiveness of systems. Health Technol Assess 2010;14(48):1-227. DOI: 
10.3310/hta14480. 
83. 
B G, C. G. Family Home Care for Cancer. A Community-Based Model. 1987-1996. 
84. 
B G, C. G. Family Home Care for Cancer. A Community Based Model. NCI; 1997-2008. 
85. 
CW. G. Automated Telephone Monitoring for Symptom Management. NCI; 2003-2009. 
86. 
C. G, B. G. Rural Partnership Linkage for Cancer Care. NCI; 1992-1997. 
87. 
ONS. PEP Rating System Overview. (https://www.ons.org/practice-resources/pep). 
88. 
NCCN. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®). 
(https://www.nccn.org/professionals/default.aspx). 
89. 
NCCN, Adult, Cancer, Pain, Panel, Members. NCCN Clinical Practice Guidelines in Oncology: Adult 
Cancer Pain. (http://www.nccn.org/professionals/physician_gls/pdf/pain.pdf). 
90. 
Baskerville NB, Liddy C, Hogg W. Systematic review and meta-analysis of practice facilitation within 
primary care settings. Ann Fam Med 2012;10(1):63-74. DOI: 10.1370/afm.1312. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 30 of 32 
 
91. 
Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results 
from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 
2015;10:21. DOI: 10.1186/s13012-015-0209-1. 
92. 
Taylor EF, Machta RM, Meyers DS, Genevro J, Peikes DN. Enhancing the primary care team to provide 
redesigned care: the roles of practice facilitators and care managers. Ann Fam Med 2013;11(1):80-3. 
DOI: 10.1370/afm.1462. 
93. 
Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, 
point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev 
2009(3):CD001096. DOI: 10.1002/14651858.CD001096.pub2. 
94. 
Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care 
computer reminders on physician behaviour: a systematic review. CMAJ 2010;182(5):E216-25. DOI: 
10.1503/cmaj.090578. 
95. 
Woertman W, de Hoop E, Moerbeek M, Zuidema SU, Gerritsen DL, Teerenstra S. Stepped wedge 
designs could reduce the required sample size in cluster randomized trials. J Clin Epidemiol 
2013;66(7):752-8. DOI: 10.1016/j.jclinepi.2013.01.009. 
96. 
Baio G, Copas A, Ambler G, Hargreaves J, Beard E, Omar RZ. Sample size calculation for a stepped 
wedge trial. Trials 2015;16:354. DOI: 10.1186/s13063-015-0840-9. 
97. 
Storlie CB, Fugate ML, Higdon DM, Huzurbazar AV, Francois EG, McHugh DC. Methods for 
Characterizing and Comparing Populations of Shock Wave Curves. Technometrics 2013;55(4):436-449. 
(In English). DOI: 10.1080/00401706.2013.805662. 
98. 
Kasza J, Hemming K, Hooper R, Matthews J, Forbes AB. Impact of non-uniform correlation structure 
on sample size and power in multiple-period cluster randomised trials. Stat Methods Med Res 
2019;28(3):703-716. DOI: 10.1177/0962280217734981. 
99. 
Cunanan KM, Carlin BP, Peterson KA. A practical Bayesian stepped wedge design for community-
based cluster-randomized clinical trials: The British Columbia Telehealth Trial. Clin Trials 
2016;13(6):641-650. DOI: 10.1177/1740774516656583. 
100. 
Zhan D, Ouyang Y, Xu L, Wong H. Improving efficiency in the stepped-wedge trial design via Bayesian 
modeling with an informative prior for the time effects. Clin Trials 2021;18(3):295-302. DOI: 
10.1177/1740774520980052. 
101. 
Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin 
Trials 2007;28(2):182-91. DOI: 10.1016/j.cct.2006.05.007. 
102. 
Aktas A, Walsh D, Kirkova J. The psychometric properties of cancer multisymptom assessment 
instruments: a clinical review. Support Care Cancer 2015;23(7):2189-202. DOI: 10.1007/s00520-015-
2732-7. 
103. 
Locke DE, Decker PA, Sloan JA, et al. Validation of single-item linear analog scale assessment of 
quality of life in neuro-oncology patients. J Pain Symptom Manage 2007;34(6):628-38. 
(http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=
17703910 ). 
104. 
Paice JA, Cohen FL. Validity of a verbally administered numeric rating scale to measure cancer pain 
intensity. Cancer Nurs 1997;20(2):88-93. (https://www.ncbi.nlm.nih.gov/pubmed/9145556). 
105. 
Singh JA, Satele D, Pattabasavaiah S, Buckner JC, Sloan JA. Normative data and clinically significant 
effect sizes for single-item numerical linear analogue self-assessment (LASA) scales. Health Qual Life 
Outcomes 2014;12:187. DOI: 10.1186/s12955-014-0187-z. 
106. 
Kumar SP. Utilization of brief pain inventory as an assessment tool for pain in patients with cancer: a 
focused review. Indian J Palliat Care 2011;17(2):108-15. DOI: 10.4103/0973-1075.84531. 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 31 of 32 
 
107. 
Kroenke K, Theobald D, Wu J, Tu W, Krebs EE. Comparative responsiveness of pain measures in 
cancer patients. J Pain 2012;13(8):764-72. DOI: 10.1016/j.jpain.2012.05.004. 
108. 
Cleeland CS, Mendoza TR, Wang XS, et al. Assessing symptom distress in cancer patients: the M.D. 
Anderson Symptom Inventory. Cancer 2000;89(7):1634-46. 
(http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=
11013380 ). 
109. 
Richardson LA, Jones GW. A review of the reliability and validity of the Edmonton Symptom 
Assessment System. Curr Oncol 2009;16(1):55. (https://www.ncbi.nlm.nih.gov/pubmed/19229371). 
110. 
Hjermstad MJ, Fayers PM, Haugen DF, et al. Studies comparing Numerical Rating Scales, Verbal 
Rating Scales, and Visual Analogue Scales for assessment of pain intensity in adults: a systematic 
literature review. J Pain Symptom Manage 2011;41(6):1073-93. DOI: 
10.1016/j.jpainsymman.2010.08.016. 
111. 
Jeter K, Blackwell S, Burke L, et al. Cancer symptom scale preferences: does one size fit all? BMJ 
Support Palliat Care 2016. DOI: 10.1136/bmjspcare-2015-001018. 
112. 
Cella D, Riley W, Stone A, et al. The Patient-Reported Outcomes Measurement Information System 
(PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-
2008. J Clin Epidemiol 2010;63(11):1179-94. DOI: 10.1016/j.jclinepi.2010.04.011. 
113. 
Garcia SF, Cella D, Clauser SB, et al. Standardizing patient-reported outcomes assessment in cancer 
clinical trials: a patient-reported outcomes measurement information system initiative. J Clin Oncol 
2007;25(32):5106-12. DOI: 10.1200/JCO.2007.12.2341. 
114. 
Fries JF, Krishnan E, Rose M, Lingala B, Bruce B. Improved responsiveness and reduced sample size 
requirements of PROMIS physical function scales with item response theory. Arthritis Res Ther 
2011;13(5):R147. DOI: 10.1186/ar3461. 
115. 
Fries J, Rose M, Krishnan E. The PROMIS of better outcome assessment: responsiveness, floor and 
ceiling effects, and Internet administration. J Rheumatol 2011;38(8):1759-64. DOI: 
10.3899/jrheum.110402. 
116. 
Rose M, Bjorner JB, Gandek B, Bruce B, Fries JF, Ware JE, Jr. The PROMIS Physical Function item 
bank was calibrated to a standardized metric and shown to improve measurement efficiency. J Clin 
Epidemiol 2014;67(5):516-26. DOI: 10.1016/j.jclinepi.2013.10.024. 
117. 
Choi SW, Reise SP, Pilkonis PA, Hays RD, Cella D. Efficiency of static and computer adaptive short 
forms compared to full-length measures of depressive symptoms. Qual Life Res 2010;19(1):125-36. 
DOI: 10.1007/s11136-009-9560-5. 
118. 
Pilkonis PA, Yu L, Dodds NE, Johnston KL, Maihoefer CC, Lawrence SM. Validation of the depression 
item bank from the Patient-Reported Outcomes Measurement Information System (PROMIS) in a three-
month observational study. J Psychiatr Res 2014;56:112-9. DOI: 10.1016/j.jpsychires.2014.05.010. 
119. 
Amtmann D, Kim J, Chung H, et al. Comparing CESD-10, PHQ-9, and PROMIS depression 
instruments in individuals with multiple sclerosis. Rehabil Psychol 2014;59(2):220-9. DOI: 
10.1037/a0035919. 
120. 
Junghaenel DU, Schneider S, Stone AA, Christodoulou C, Broderick JE. Ecological validity and clinical 
utility of Patient-Reported Outcomes Measurement Information System (PROMIS(R)) instruments for 
detecting premenstrual symptoms of depression, anger, and fatigue. J Psychosom Res 2014;76(4):300-6. 
DOI: 10.1016/j.jpsychores.2014.01.010. 
121. 
Geynisman DM, Wickersham KE. Adherence to targeted oral anticancer medications. Discov Med 
2013;15(83):231-41. (https://www.ncbi.nlm.nih.gov/pubmed/23636140). 

Minimal Risk Protocol Template [IRB 10429.010] 
 
Effective: 9/20/2017 
 
 
 
 
 
 
 
 
Page 32 of 32 
 
122. 
Greer JA, Amoyal N, Nisotel L, et al. A Systematic Review of Adherence to Oral Antineoplastic 
Therapies. Oncologist 2016;21(3):354-76. DOI: 10.1634/theoncologist.2015-0405. 
123. 
Ruddy KJ, Partridge AH. Adherence with adjuvant hormonal therapy for breast cancer. Ann Oncol 
2009;20(3):401-2. DOI: 10.1093/annonc/mdp039. 
124. 
Ruddy K, Mayer E, Partridge A. Patient adherence and persistence with oral anticancer treatment. CA 
Cancer J Clin 2009;59(1):56-66. DOI: 10.3322/caac.20004.