A Feasibility Study to Improve Colorectal Cancer Screening Among Racially Diverse Zip Codes in a Persistent Poverty County Using Navigation and Machine Learning Predictive Algorithms
Conditions: Colorectal Cancer
Sponsor: Abramson Cancer Center at Penn Medicine

Improving Colorectal Cancer Screening 
in Racially Diverse Zip Codes Using 
Navigation and Machine Learning 
(PCSNaP) 
 
NCT05383976 
October 26, 2021 

Program Director/Principal Investigator (Last, First, Middle):  Vonderheide, Robert H.   
PA-20-272 “Administrative Supplements to Existing Grants and Cooperative Agreements (Parent Admin 
Supplement)”: A Feasibility Study to Improve Colorectal Cancer Screening among Racially Diverse Zip 
Codes in a Persistent Poverty County using Navigation and Machine Learning Predictive Algorithms 
 
Abstract 
The overarching goals of this Administrative Supplement to Existing NIH Grants and Cooperative Agreements 
(PA-20-272, Parent Admin Supplement) is to support the Abramson Cancer Center (ACC) of the University of 
Pennsylvania in carrying out its mission to increase colorectal cancer (CRC) screening completion among 
high-risk individuals living in a persistent poverty county by designing, conducting, disseminating and 
evaluating an electronic health record-based automated identification program to target effective, culturally-
sensitive CRC screening navigation to individuals who have not completed an ordered colonoscopy or fecal 
immunochemical test (FIT).  
 
Specifically, the goals of this supplement are to: 1) Adapt a previously validated electronic health record 
(EHR)-based machine learning algorithm to predict CRC detection by retraining the model using data from 
patients seen in primary care clinics serving zip codes with a high proportion of racial and ethnic minorities 
living in Philadelphia County, a persistent poverty county; and 2) Implement and evaluate the feasibility and 
effectiveness of an algorithm-based CRC navigation program to increase colorectal cancer screening among  
patients in Philadelphia county who are at high risk of CRC and have uncompleted colonoscopies.  
 
Together, these novel projects aim to be the first to combine use machine learning algorithms and patient 
navigation to increase guideline-based cancer screening in order to reduce the burden of CRC among high-risk 
individuals living in a persistent poverty county through targeted, culturally-sensitive navigation that addresses 
social factors that prevent CRC screening. This study will generate preliminary data in support of an R01 
application to scale algorithm-informed cancer screening navigation in persistent poverty areas. 
Specific Aims 
Aim 1: Adapt a previously validated machine learning algorithm to predict risk of colorectal cancer or 
adenoma among a retrospective cohort of Penn Medicine patients living in vulnerable zip codes within 
Philadelphia County. We will improve upon a previously validated machine learning algorithm1 to predict risk 
of colorectal cancer by (1) re-training the model to predict both cancer and adenoma detection; (2) integrating 
additional clinical and socioeconomic risk factors for colorectal cancer risk; and (3) retraining the model in a 
cohort of individuals living in Philadelphia County zip codes that are minority-predominant and have lowest 
median incomes in the county. We will compare algorithm performance to existing criteria, based on age and 
prior colon cancer screening history, which are currently used to stratify colorectal cancer risk among Penn 
Medicine patients. We hypothesize that machine learning models will maintain good performance in predicting 
CRC risk among a community living in a persistent poverty area.  
 
Aim 2. Test the feasibility, usability, and effectiveness of algorithm-informed navigation on colorectal 
cancer screening rates and adenoma detection rates among Penn Internal Medicine and Family 
Medicine patients residing in racially and ethnically diverse zip codes. We will use the machine learning 
algorithm developed in Aim 1 to inform the CRC screening navigation program process and evaluate its impact 
on the identification of Penn Internal Medicine and Family Medicine patients with adenomas and CRC. We will 
generate automated monthly lists of Penn Medicine Internal and Family Medicine patients from diverse 
communities ranked by predicted cancer risk. High-risk patients will be prioritized for navigation of 
socioeconomic and financial barriers to colonoscopy through an existing navigation program created and 
managed by our team at the University of Pennsylvania.2 In addition to assessing feasibility and usability, we 
will evaluate the effectiveness of the program on rates of completed colonoscopies and rates of CRC/adenoma 
detection, compared to historical rates prior to the intervention. We hypothesize that algorithm-informed 
navigation will improve CRC screening rates and cancer/adenoma detection rates.  
 
Need for Cancer Control Program in Philadelphia County 
Nearly 150,000 individuals will be diagnosed with colorectal cancer in 2020 in the United States, with incidence 
and cancer-associated mortality higher for African-Americans and other minorities.3,4 The ACC’s 12-county 
catchment area is comprised of urban and suburban counties in southeastern Pennsylvania (Bucks, Chester, 
Delaware, Montgomery, Philadelphia), southern New Jersey (Atlantic, Burlington, Camden, Gloucester, 
Mercer, Ocean), and Delaware (New Castle) (Fig. 1). These 12 counties have a population of 7,015,781 
residents and capture the residences of 81.4% of the patients seen at ACC. 

Program Director/Principal Investigator (Last, First, Middle):  Vonderheide, Robert H.   
 
There is greater incidence and mortality from CRC in the ACC catchment 
area compared to the general US population. (Table 1). Philadelphia 
County, which contributes 21.5% of the patients seen at ACC, has 
an even higher CRC incidence and mortality compared to the rest of 
the catchment area and the US. Furthermore, Philadelphia County is 
a persistent poverty county, with 23% of patients living in poverty (90% 
confidence interval 21.8-24.2%).5 Philadelphia County has the second 
highest poverty rate in the state of Philadelphia, with over 352,748 
residents living in poverty. In 25 of the city’s 46 residential zip codes, 
more than 1 in 5 individuals are living below the poverty line.6 In 
Philadelphia, the poverty rate is roughly 19% among whites and 29% 
among African-Americans. Poverty is strongly associated with CRC incidence and with minority communities.7–
9 Furthermore, there are racial inequities in CRC incidence and 
mortality in all of the geographic areas analyzed:  African-
American residents consistently have a higher incidence 
and mortality from CRC compared to White residents.  
While colorectal cancer screening may reduce cancer-
associated mortality, African-Americans in Philadelphia County 
are over 10 percentage-points less likely to be adherent to 
colorectal cancer screening guidelines compared to Whites.  
Despite the proven benefit of regular colonoscopy, CRC 
screening adherence rates remain only around 50% for Penn 
Medicine primary care patients. Furthermore, screening rates 
are consistently worse for African-Americans than White 
individuals – and this disparity is widening at Penn Medicine 
(Fig. 2). For this reason, Penn Medicine, a safety net 
provider for low income, underserved patients in 
Philadelphia, has instituted a system-wide goal of 
improving CRC screening for African-American patients.  
 
Existing Services 
Colorectal cancer screening navigator program: Adverse social determinants of health such as lack of 
transportation access contribute to CRC screening nonadherence disproportionately for African-American and 
other minority communities. Recognizing this, in 2011, Penn Medicine created a navigation program to address 
poor CRC screening rates and increase access to screening colonoscopies for patients in underserved areas 
of West, South, and Southwest Philadelphia.2 The navigation program provides services to address barriers to 
cancer screening; these include transportation assistance, detailed instructions on bowel prep, or mailed FIT 
test for average-risk patients who refuse colonoscopy (see Fig. 3). Eligible patients have had a colonoscopy 
order placed in the past six months and have not scheduled, cancelled, or no-showed to their colonoscopy. 
Among eligible patients contacted by the navigator program, 50% agree to a colonoscopy over the phone. Of 
those patients, 80% complete a colonoscopy. The CRC screening navigator program is currently staffed by 1.5 
FTE staff, including the nurse lead of the navigator program Diann Boyd, BSN, RN, OCN (see attached Letter 
of Support) and a part-time clinical assistant. Our prior research has shown that navigated patients have a 
high rate of colorectal cancer screening completion (79%) and a significantly lower rate of cancelled and/or 
skipped appointments.2 In addition, those who were screened as a result of the program had a much higher 
Table 1. Age-adjusted rates per 100,000 for colorectal cancer incidence and mortality by race and geography 
  
Entire ACC catchment 
area 
Philadelphia County 
non-Philadelphia 
catchment area  
United States 
 Colorectal 
Incidence 
Mortality 
Incidence 
Mortality 
Incidence 
Mortality 
Incidence 
Mortality 
  
W† 
B 
W 
B 
W 
B 
W 
B 
W 
B 
W 
B 
W 
B 
W 
B 
41.8 
45.0 
14.4 
17.5 
46.8 
47.1 
15.1 
20.4 
41.1 
43.2 
14.3 
14.7 
38.4 
45.7 
14.1 
19.4 
Sources: PA, NJ and DE state cancer registries, SEER; 2011-2015.*ages 0-14; †W - White; B - Black; red shade > US, green shade < 
US (non-overlapping 95% CIs and > 10% difference) 
 
Figure 1 
Figure 2. CRC screening in Philadelphia county 

Program Director/Principal Investigator (Last, First, Middle):  Vonderheide, Robert H.   
adenoma detection rate (40%). An adenoma detection rate of 40% is an increase of at least 15 percentage 
points, or a 60% increase, over the national benchmark of 15% to 25%.  
Patients and clinics served: Currently the On July 1, 2015, the program was enlarged and institutionalized by 
Penn Medicine with expansion from 5 to 11 zip codes to 
include Southwest and South Philadelphia. On January 1, 
2021, the program expanded from 11 to 18 zip codes 
throughout Philadelphia County. Currently patients receiving 
care at one of 4 Internal Medicine practices located in 
Philadelphia County participate in the program. Another key 
goal of Penn Medicine is to expand navigation services 
beyond Internal Medicine to other types of primary care 
practices, including 3 Family Medicine practices in 
Philadelphia County. As part of the resulting expansion 
of patients requiring navigation, additional FTE 
navigation staff will be needed and are requested as 
part of this Administrative Supplement. Additionally, a 
key challenge in scaling this navigation program is 
identifying patient populations at increased risk of CRC, who 
may benefit most from timely navigation. While risk 
stratification guidelines exist to determine who should be 
prioritized for colonoscopy10, these require time-consuming 
chart review and may not be adequately sensitive. 
Automated machine learning algorithms based on routine 
EHR data accurately estimate a patient’s relative risk of 
CRC in multi-institutional studies.1,11,12 
 
CRC risk algorithm: The Penn Predictive Healthcare Group 
is a Penn Medicine-employed team of data and human 
factors scientists. Their objective is to use EHR data to 
develop real-time, automated predictive models for use in 
inpatient and outpatient care settings to inform clinical care delivery. In collaboration with the Predictive 
Healthcare Group, we have adapted a previously published EHR-based ML algorithm1 based on complete 
blood count records from EHR data, adding additional variables of significance including age, gender, BMI, 
alcohol and smoking status, and relevant 
comorbidities to generate a model with 
good discrimination (AUC 0.73), sensitivity 
(44%), and specificity (90%) in identifying 
future risk of CRC or adenoma (see Fig. 4). 
During the period of this award, the Data 
Science Team will retrain the algorithm in a 
racially and ethnically diverse subgroup 
residing and receiving care within Penn 
Medicine Internal and Family Medicine 
clinics located in Philadelphia County.  
 
Strategies to Elicit Needs 
Given that we are utilizing a central 
navigator program that exists outside of the 
clinic and does not require additional clinic 
staff, in addition to the fact that outcomes will be identified from routine EHR records and not from dedicated 
research staff within clinics, we do not anticipate that there will be issues implementing this program into our 
target clinics given our preexisting infrastructure. However, we will take several steps to ensure clinics’ 
feedback prior to Aim 2.  
 
Focus groups: First, to ensure buy-in and feedback of the additional clinics participating in this program, we will 
conduct focus groups with relevant leadership from each of the 3 new clinics and each of the 4 older clinics. 
Figure 3. CRC navigator workflow 
Figure 4. ML algorithm performance in unselected Penn Medicine cohort 

Program Director/Principal Investigator (Last, First, Middle):  Vonderheide, Robert H.   
The focus group will garner feedback about the intervention and strategies to integrate automated navigation 
services into routine workflow. This intervention will occur without physician referral; however, given that some 
automated predictions may be incorrect, we will display results from an algorithm that predicts both CRC and 
adenoma vs. a model that predicts CRC alone. We will elicit perspectives from these clinic groups about which 
algorithm they agree with the most, in order to determine which algorithm to proceed with for Aim 2. 
 
PDSA cycles: After training the algorithm, we will conduct a 3-month pilot using patients from one primary care 
practice, using the Plan-Do-Study-Act (PDSA) Framework13 to identify patient and clinic barriers to this 
research program. In the “Plan” stage, the critical elements are: (1) integrating flags for algorithm-predicted 
high-risk patients into existing automated lists given to the navigator program; (2) selection, training and 
defining the functions of the new CRC screening navigator; and (3) establishment of a network of clinic 
partners serving patients in the high poverty navigation program. The “Do” stage is focused on the 
implementation of an algorithm-facilitated navigation process to address the clinical and social barriers to CRC 
screening. The “Study” stage is defined by tracking enrolled patients and evaluating program measures that 
include completion of screening and adenoma detection rates and, then, in the “Act” stage, optimizing the 
program to expand the reach of the program to serve a broader population of patients, potentially even outside 
of Penn Medicine. 
 
Approach 
Aim 1. Validation of ML algorithm to predict colon cancer and adenoma risk 
Study Design. Retrospective cohort study  
Cohort. 60,270 unique patients without a prior CRC diagnosis who had a colonoscopy order placed by a Penn 
Internal Medicine or Family Medicine provider living in one of 18 minority-predominant zip codes in Western 
and Southwestern Philadelphia County; completed a colonoscopy at Penn between January 1st, 2010 and 
December 31st, 2019; and were seen by a Penn primary care physician in the preceding year. We will split the 
cohort into training (70%) and validation (30%) sets at the patient-level so that patients are represented in 
either the training or validation set only.  
Outcome. The primary outcome will be biopsy-proven CRC or adenoma, based on (1) diagnostic or procedural 
codes for CRC or adenoma using previously published administrative code mappings14,15; and (2) internal 
datasets on Adenoma Detection Rate (ADR), generated by the Data Analytics Center using Provation and 
Clarity pathology data. Preliminary data suggest that 4,210 cases of CRC or adenomas exist in our cohort. 
Predictors. We will use PennChart Clarity data form the 6 months prior to the colonoscopy as predictors in the 
model. Predictors will be adapted from prior literature and will include demographic (e.g. age, gender, EtOH 
history), comorbidity (e.g. anemia, inflammatory bowel disease), laboratory (e.g. hemoglobin, BUN), utilization 
(e.g. prior colonoscopy orders), and socioeconomic (e.g. area-level income and education) data.1,11,12 
Additionally, we will use the EPIC social determinants of health tool to extract features such as income, 
education level, and homelessness that may be association with CRC risk. .  
Modelling strategy. We will compare the cross-validation predicted performance of several ML strategies, 
including logistic regression, random forest, gradient boosting, and recurrent neural networks, to identify the 
model with optimal predictive performance. We will report algorithm performance on the hold-out validation set, 
specifically testing for performance differences between the algorithms. Finally, we will compare performance 
of the best ML algorithm against an existing PennChart risk stratification model that uses only age and prior 
colorectal cancer screening utilization as predictors. 
Performance metrics. Primary performance metrics will include area under the receiver operating characteristic 
curve (AUC), area under the precision-recall curve (AUPRC), and sensitivity.  
Limitations. Administrative codes and internal ADR data may underestimate the true outcome rate, although 
the number of cases and event rate (7.0%) should still allow adequate model development.  
 
Aim 2. Pilot testing of ML-facilitated colon cancer screening navigation 
Study Design. Prospective cohort study with 3-month enrollment period and 3-month follow-up.  
Cohort. The cohort will consist of patients residing in 18 zip codes in Western and Southwestern Philadelphia 
who have primary care providers in 4 Penn Medicine Internal Medicine practices and 3 Penn Medicine Family 
Medicine Practices. Patients will have had a colonoscopy order placed in the past 6 months and have not 
scheduled, cancelled, or no-showed to their colonoscopy.  
Intervention. This intervention will utilize the existing Penn Medicine CRC patient navigation program; 
administrative supplement funding will be used to expand the staff from 1.5 FTE to 2.5 FTE. In the revised 
intervention, we will provide a monthly list of ~300 patients with unfilled colonoscopies, risk-stratified according 

Program Director/Principal Investigator (Last, First, Middle):  Vonderheide, Robert H.   
the ML algorithm and select high-risk criteria.10 The navigation team will prioritize timely outreach and 
navigation to high-risk patients according to a script that communicates risk.  
Outcome. The co-primary outcomes, measured within 3 months of enrollment, will be (1) enrollment in 
navigator program (feasibility); (2) completion of colonoscopy or FIT testing; and (3) adenoma detection rate.  
Analysis Plan. We will use an intention-to-treat analysis. We will use propensity score matching using clinical 
and sociodemographic characteristics to create matched cohorts of patients between the intervention 
participants and historical controls from the same clinics in the year prior. We will then use logistic regression 
to assess the association between the intervention and the two primary outcomes, adjusted for age, gender, 
comorbidity, and area-level socioeconomic status. 
Statistical Power. A sample of 344 participants (172 per group) will provide >90% power, assuming Type I 
error rate with a 2-sided α 0.05 and 50% dropout/refusal to participate to navigation, to detect a 20 percentage-
point (pp) increase in colon cancer screening completion (40  60%, SD 20%) in the intervention group.   
 
Work plan and timeline 
As seen in our Gantt chart, we plan to complete all Aims 
within one year. This feasibility study will generate 
preliminary data within 1 year that will lead to an R-level 
grant application to support a well-powered cluster-
randomized controlled trial that tests the impact of ML-
guided patient navigation services on colorectal cancer 
screening rates and detection among Penn Medicine 
primary care practices within and outside of Philadelphia 
County serving underserved, high poverty areas. This R-
level funding application will also be used support 
additional patient navigators to support this trial. 
 
Sustainability  
Our team maintains strong working relationship with the 
Colorectal Cancer Alliance (CCA) and Get Your Rear in 
Gear (GYRIG), both which are 501(c)3 nonprofit 
organizations whose mission is to reduce the burden of 
CRC in communities. The CCA and GYRIG have 
generous supported the Penn Medicine CRC screening 
navigation program with through their community health partnerships grant program for over eight years. We 
believe that our continued engagement with the CCA and GYRIG, in addition to planned R01 level funding, 
could lead to the organization expanding the navigator program to additional underserved communities and 
practices in zip codes beyond Philadelphia County. Additionally, future funding will help to support a dedicated 
data science specialist to improve the algorithm by mining unstructured clinical notes and pathology reports in 
order to better target the navigator program towards high-risk individuals.   
Qualifications of program leads 
The multidisciplinary team members proposed are:  Dr. Robert Vonderheide, Director of the ACC at the 
University of Pennsylvania, Vice Dean for Cancer Programs for the Perelman School of Medicine at the 
University of Pennsylvania, and Vice President for Cancer Programs for the University of Pennsylvania Health 
System will serve as PI. Dr. Ravi B. Parikh, Assistant Professor of Medicine and Medical Ethics and Health 
Policy, is proposed as PD and will work with Dr. Guerra to oversee all aspect of the proposed work including 
design, implement and evaluation of the proposed work.  Dr. Carmen Guerra, MD, MSCE, Associate 
Professor of Medicine, practicing general internist and Associate Director of Diversity and Outreach and co-
leader of COE at the ACC is proposed as co-investigator and will work with Dr. Parikh to oversee all aspects of 
the proposed work including design, implement and evaluation of the proposed work. We will hire a research 
coordinator senior at the ACC, who will work with Drs. Vonderheide, Parikh, and Guerra to expand the reach 
of the navigator program; integrate algorithm output into the workflow of the navigator program; solicit feedback 
from patients, clinics, and navigators via regular questionnaires; assist in navigation of patients under 
supervision of Ms. Boyd; create the database and collect data; conduct analyses for the program evaluation; 
and present the results.  In this application, the team will collaborate with Michael Draugelis, MS (see Letter 
of Support), Chair of the Penn Medicine Predictive Healthcare group, in order to integrate algorithm output 
into the navigator workflow.  
Tasks 
Q1 
Q2 
Q3 
Q4 
Aim 1: Algorithm development 
Data cleaning and feature extraction 
 
 
 
 
Algorithm validation 
 
 
 
 
Algorithm IT integration 
 
 
 
 
Preparation of Manuscripts 
 
 
 
 
Aim 2: Feasibility study 
Focus groups 
 
 
 
 
PDSA cycles 
 
 
 
 
Recruitment for feasibility study 
 
 
 
 
Analyze data and post-study questionnaires 
 
 
 
 
Preparation of manuscripts 
 
 
 
 
R01 preparation