Development and Internal Validation of Models to Predict Time-to-event for Major Medical Complications Within 30-days After Planned Colectomy: a Retrospective Population Cohort Study
Conditions: Colorectal Cancer, Inflammatory Bowel Diseases, Diverticulitis, Colectomy, Predictive Model, Complications, Postoperative
Sponsor: University of British Columbia

1 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
1
Development and internal validation of models to predict time-to-event for major 
medical complications within 30-days after planned colectomy: a retrospective 
population cohort study 
Study Protocol 
Principal Investigators 
Cox proportional hazards analysis paper:  
Dr. Alana Flexman, MD, FRCPC 
Clinical Associate Professor, Department of Anesthesiology, Pharmacology & 
Therapeutics, University of British Columbia (UBC) 
Department of Anesthesia, Providence Health Care  
 
Dr. Xue Chen (Janny) Ke, MD, FRCPC 
Anesthesiologist, Department of Anesthesia, St. Paul's Hospital 
Clinical Instructor, Department of Anesthesiology, Pharmacology & Therapeutics, 
UBC; Adjunct, Department of Anesthesia, Pain Management & Perioperative 
Medicine, Dalhousie University 
 
Machine learning analysis paper:  
Dr. Matthias Görges, PhD  
Assistant professor, Department of Anesthesiology, Pharmacology & Therapeutics,  
            University of British Columbia; Research Institute, BC Children’s Hospital 
 
Co-Investigators 
Dr. Rama Sreepada, PhD 
Postdoctoral research fellow, Department of Anesthesiology, Pharmacology & 
Therapeutics, UBC; Research Institute, BC Children’s Hospital 
 
Dr. Long Ngo, PhD 
Associate Professor, Department of Biostatistics, Harvard School of Public Health 
Division of General Medicine, Beth Israel Deaconess Medical Center 
Harvard Medical School 
 
Sihaoyu (Sherry) Gao, PhD Candidate 
Department of Statistics, Faculty of Science, UBC 
 
Dr. Lang Wu, PhD 
Professor, Department of Statistics, Faculty of Science, UBC 
 
Dr. Stephan Schwarz, MD, FRCPC 
Professor, Department of Anesthesiology, Pharmacology & Therapeutics, UBC;  
Department of Anesthesia, Providence Health Care 
 
Dr. Carl Brown, MD, MSc, FRCSC  
Head, Division of General Surgery, St. Paul’s Hospital; Clinical Assistant Professor,  
            Department of Surgery, Faculty of Medicine, UBC; Chair, Section of Colorectal  
            Surgery, UBC; Provincial Lead for Surgical Oncology, BC Cancer Agency 
 
Jeff Rueger, Patient Partner 
 
Primary Contact:  
Dr. Xue Chen (Janny) Ke  jannyke@alumni.ubc.ca 

2 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
2
Development and internal validation of models to predict time-to-event for major 
medical complications within 30-days after planned colectomy: a retrospective 
population cohort study 
 
Introduction  
Planned (elective or time sensitive) colectomy are performed for indications including cancer, 
inflammatory bowel disease (IBD), and diverticulitis (1,2). After colectomy, patients are at 
risk of a variety of major medical complications, including pneumonia, myocardial infarction 
(MI), cerebral vascular event (CVA), venous thromboembolism (VTE), acute renal failure 
(ARF), and sepsis (2,3). However, different complications tend to happen at different times 
after surgery. Accurate risk prediction, not only whether a complication may occur in a 
patient, but also when, is crucial for patient education, monitoring, and disposition planning. 
While several studies have explored the incidence and binary risk prediction for major 
complications after surgeries (1–3), there has been scarce literature on time-to-complication 
prediction modeling in recent population cohort data.  
 
 
Objectives 
1. Develop and internal validate Cox proportional hazards models to predict time-to-
complication for each individual major medical complication captured in the 
American College of Surgeons National Surgical Quality Improvement Program 
(NSQIP) dataset (pneumonia, myocardial infarction (MI), cerebral vascular event 
(CVA), venous thromboembolism (VTE), acute renal failure (ARF), and sepsis) or 
adverse outcomes (mortality, readmission), that started within 30-days after elective 
colectomy.  
2. Develop and internal validate machine learning models to predict time-to-
complication for major medical complications and adverse outcomes (same as in 
objective 1) within 30-days after elective colectomy in NSQIP. The best machine 
learning model for each complication will be compared to the Cox proportional 
hazards model in terms of discrimination, and calibration. 
 
 
Methods 
Following approval from the University of British Columbia Clinical Research Ethics Board 
with waived informed consent, we will conduct a time-to-event survival analysis in a 
retrospective cohort using NSQIP®, a prospectively-collected multicentre dataset with more 
than 150 clinical variables within 30 days after surgery (4). This dataset includes information 
on whether the patient was diagnosed with major complications (in- or out-of-hospital) as 
well as the number of postoperative days to the diagnoses of complications, as defined by a 
standardized criteria within the NSQIP operations manual (4). The general dataset will be 
linked with the NSQIP® Procedure Targeted Colectomy dataset, which contains additional 
colectomy-specific variables. The study will be pre-registered prior to data analysis 
(Clinicaltrials.gov) and reported according to the Transparent reporting of a multivariable 
prediction model for individual prognosis or diagnosis (TRIPOD) guidelines and Guidelines 
for Developing and Reporting Machine Learning Predictive Models in Biomedical Research 
(5). 
 
 
 
 

3 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
3
Study Population 
The study will include all patients who are 18 years or older undergoing elective colectomy, 
whose data has been collected in the NSQIP® Procedure Targeted Colectomy dataset from 
2014-2019, inclusively. The estimated sample size is approximately 130,000 patients. 
 
We will exclude patients with American Society of Anesthesiologists (ASA) Physical Status 
(PS) V (defined as “5-Moribund”) (ASA PS 6 - organ donation is not included within 
NSQIP). We will also exclude patients who are undergoing urgent or emergency surgery, or 
have indication for colectomy consisting of “Acute diverticulitis”, “Enterocolitis (e.g. C. 
Difficile)”, and “Volvulus” due to the non-elective nature of these pathologies. Patients with 
disseminated cancer, wound infection (i.e. potentially recent surgery), systemic sepsis, or 
ventilator-dependence preoperatively will also be excluded.  
 
Model Prediction Endpoints 
We will create a distinct time-to-event prediction model for each of the following new 
postoperative complications or adverse outcomes within 30 days postoperatively:  
• Major medical complications: pneumonia, myocardial infarction (MI), cerebral 
vascular event (CVA), venous thromboembolism (VTE), acute renal failure (ARF), 
and sepsis or septic shock 
• Adverse outcomes: mortality, readmission 
 
The definitions of the complications, outcomes, and time-to-complication date criteria 
collected from NSQIP variables are included in Appendix 1 (Data Extraction Form).  
 
Candidate predictors  
To predict time-to-complication, we selected candidate predictors from the NSQIP variables 
based on literature and clinical experience, and obtained consensus within our 
multidisciplinary team. For generalizability and future external validation, we selected 
predictors that are routinely collected in hospital electronic health records (EHRs) such as at 
our institution whenever possible. Candidate predictors are as follows (please see Appendix 1 
for detailed definitions): 
• 
Age, sex, race, body mass index (BMI) 
• 
ASA, diabetes, smoker within one year preoperatively, severe obstructive chronic 
pulmonary disease (COPD), ascites, congestive heart failure (CHF), hypertension on 
medications, preoperative renal failure (acute renal failure, dialysis, and/or glomerular 
filtration rate (GFR) <60 based on preoperative creatinine), preoperative functional 
status, dyspnea, coagulopathy 
• 
Steroid and/or immunosuppressant use preoperatively 
• 
Chemotherapy within 90 days preoperatively 
• 
Primary indication for colectomy (recategorized as): 
– Cancer 
– Non-malignant polyp  
– Inflammatory bower disease (IBD)  
– Diverticulitis  
– Bleeding 
– Other 
• 
Surgical approach: minimally invasive vs. open   
• 
Total operation time 
• 
Use of regional anesthesia as an additional anesthetic technique (epidural, peripheral, 
spinal)  
• 
Wound classification (end of surgery) 

4 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
4
 
For the two models to predict time-to-event for adverse outcomes (mortality and 
readmission), we will perform sensitivity analysis to determine whether including the 
occurrence of any major complications (pneumonia, myocardial infarction (MI), cerebral 
vascular event (CVA), venous thromboembolism (VTE), acute renal failure (ARF), and 
sepsis) as individual predictors improve discrimination and calibration.  
 
Statistical Analysis  
1. Preprocessing  
• Convert age from the NSQIP character variable to continuous variable, with 
Age “90+” recoded as 90  
• Standardizing the coding of missing values: missing coded in NSQIP as “-99”, 
“unknown”, “None Assigned”, or “NULL”. Each variable will be individually 
examined to ensure that the missing value is correctly standardized. 
• Pre-processing variables for survival analysis: creation of a censored 1 vs. 0 
variable for each outcome to be modelled based on time-to-event and the 
occurrence of complications.  
• Missing values  
• Missing values for candidate predictors will be quantified. If the % 
missing is: 
1. >10% then exclude the predictor from the model 
2. <1% then delete (complete case analysis) 
3. >=1% and <10% then multiple imputation by chained 
equations (time-to-event variables are never imputed) 
• Note that patients with missing values for complications will be coded 
as zero for the complication (i.e. assumed to not have the 
complication), as the presence of complications would have been in the 
health records  
• The % missing data for height and weight will be examined in cohort 
characteristics. If <1% is missing, then BMI will be calculated using 
these variables.   
• Patients with a complication or adverse event but missing time-to-
event data will be excluded from modeling. The extent of bias (i.e. if 
these patients have different cohort characteristics compared to patients 
without missing time-to-event data) will be assessed in sensitivity 
analysis. Note that it is expected for patients without a complication or 
adverse event to have missing time-to-event data, thus these patients 
are not excluded.  
• Check collinearity with variance inflation factor and Pearson’s correlation 
coefficient: if present, combine information from collinear variables if feasible 
(e.g. new variables is “yes” if any of the variables it aggregates are “yes”), 
otherwise eliminate the variable that has more missing values or would be less 
accurately ascertained  
• Techniques for class imbalance: e.g. resampling within the derivation set to 
improve model performance in the case of rare outcomes  
2. Descriptive statistics  
 Cohort characteristics will be stratified by mortality, by each complication and 
by derivation vs. validation sets (please see list below). Continuous variables 
will be presented as mean (standard deviation) and median (IQR) for 
parametric and nonparametric data, respectively. Categorical variables will be 

5 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
5
presented as frequency (%). For comparison amongst groups, ANOVA will be 
used for parametric data, Kruskal-Wallis for nonparametric data, and Chi-
square tests for categorical data. Standardized mean difference will be 
presented. Exploratory visualizations will be performed.  
• Distribution of time to complication data (time to any complication, and time 
to each complication): failure curves, i.e. plotting the 1-S(t) (survival function) 
to determine the risk of complication daily within 30 days  
• Exploratory analysis for complications that tend to occur concurrently or are 
associated with each other  
3. Model building 
• Models will be created using the entire dataset and using bootstrap validation  
• Individual models will be built for each outcome.  
• Assumptions for the Cox model will be checked prior to model 
building, including but not limited to:  
1. Proportionality: visual inspection of the Kaplan-Meier curves, 
Log Log S(t) test, Schoenfeld residuals. If these assumptions 
hold, we will use the Cox proportional hazards (CPH) 
modeling; if not, we will use the accelerated failure time (AFT) 
modeling.  
2. Competing risk: complications are assumed to be independent 
from each other. If mortality is >10%, it will be modeled as a 
competing risk (however, we do not anticipate mortality to 
be >10% for elective colectomy).  
• Sensitivity analyses 
1. After examining the temporal distributions of events, we may 
include complications that tend to happen earlier as predictors 
for complications that happen later, and may be related to 
earlier complications  
2. For the two models to predict time-to-event for adverse 
outcomes (mortality and readmission), we will perform 
sensitivity analysis to determine whether including the 
occurrence of major complications (pneumonia, myocardial 
infarction (MI), cerebral vascular event (CVA), venous 
thromboembolism (VTE), acute renal failure (ARF), and 
sepsis) as predictors improve discrimination and calibration 
3. Restricted cubic splines for continuous predictors (age, BMI, 
duration of surgery) to explore non-linear relationships  
4. Compare cohort characteristics for patients with a complication 
or adverse event but missing time-to-event data, vs. patients 
without missing time-to-event data, to evaluate for the impact 
of bias from the exclusion of patients with missing time-to-
event data 
• Additional analysis will be performed to improve time-to-event 
predictions, including but limited to: 
1. Accelerated failure time (AFT) modeling to directly model 
failure time. Compared to CPH, APT is more interpretable and 
robust with omitted predictor variables. 
2. Machine learning models that can facilitate time-to-event 
predictions: including but not limited to boosting trees, neural 
networks.  
4. Model evaluation, with bootstrapped 95% confidence intervals and optimism 

6 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
6
• AUROC, pseudo R-squared, scaled Brier, calibration plots   
5. Sample size calculation  
• We used the PMSampsize package in R and followed guidance specifically for 
time-to-event multivariable modeling (6). Based on an event rate of 0.01 over 
30 days (i.e. 0.00033333 per patient per day in the study period totalling 30 
days), Cox-Snell R-squared 0.11 (the maximum for the event rate of 0.01 over 
30 days) (6), 0.05 acceptable difference in apparent & adjusted R-squared, 
shrinkage 0.9, a minimum of 51 events (5091 patients, 15,2730 person-days of 
follow-up) are required for modeling with 30 predictor variables.  
Data will be analyzed using R and/or Python.  
 
List of variables in cohort characteristics (Please see Appendix 1 for details) 
 
Preoperative and intraoperative variables:  
• 
Age, sex, race, height, weight, body mass index (BMI) 
• 
ASA, diabetes, smoker within one year preoperatively, severe obstructive chronic 
pulmonary disease (COPD), ascites, congestive heart failure (CHF), hypertension on 
medications, preoperative renal failure (acute renal failure, dialysis, and/or GFR <60 
based on preoperative creatinine), preoperative functional status, dyspnea, coagulopathy 
• 
Steroid/immunosuppressant use preoperatively 
• 
Chemotherapy within 90 days preoperatively 
• 
Primary indication for colectomy (categorized and full list) 
• 
Surgical approach: minimally invasive vs. open (categorized and full list)  
• 
Total operation time 
• 
Use of regional anesthesia as an additional anesthetic technique (epidural, peripheral, 
spinal)  
• 
Wound classification (end of surgery)  
 
Postoperative variables:  
• Myocardial infarction  
• Stroke/CVA 
• VTE (composite: pulmonary embolism, DVT) 
• Pneumonia 
• Acute renal failure  
• Sepsis or septic shock (composite of systemic sepsis and septic shock) 
• Readmission: also present % unplanned  
• Mortality  
• Complication or death (composite of the major complications modeled)  
• Time to first complication or death (composite of the major complications modeled)  
• Wound issues (composite: superficial and deep infection, wound disruption)  
• Non-home discharge 
• Bleeding requiring transfusion   
• Anastomotic leak  
• Prolonged postoperative ileus (nothing by mouth or nasogastric tube)  
• Urinary tract infection  
• Transfusion (packed red blood cell (PRBC) within the first 72 hours of surgery start time) 
• Cardiac Arrest Requiring cardiopulmonary resuscitation (CPR) 

7 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
7
• On ventilator >48h postoperatively 
• Still in Hospital > 30 Days 
• Unplanned reoperation  
 
 
 
 
 

8 
 
Nov 9, 2021             Protocol Version #2  
                Ke et al.  
      REB# H21-02670 
 
8
References 
1.  
Morris MS, Deierhoi RJ, Richman JS, Altom LK, Hawn MT. The Relationship Between 
Timing of Surgical Complications and Hospital Readmission. JAMA Surg. 2014 Apr 
1;149(4):348–54.  
2.  
Scarborough JE, Schumacher J, Kent KC, Heise CP, Greenberg CC. Associations of 
Specific Postoperative Complications With Outcomes After Elective Colon Resection: 
A Procedure-Targeted Approach Toward Surgical Quality Improvement. JAMA Surg. 
2017 Feb 15;152(2):e164681.  
3.  
Thompson JS, Baxter BT, Allison JG, Johnson FE, Lee KK, Park WY. Temporal 
Patterns of Postoperative Complications. Arch Surg. 2003 Jun 1;138(6):596–603.  
4.  
American College of Surgeons. ACS NSQIP Participant Use Data File [Internet]. 
American College of Surgeons. [cited 2021 Sep 18]. Available from: 
http://www.facs.org/quality-programs/acs-nsqip/participant-use 
5.  
Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for 
Developing and Reporting Machine Learning Predictive Models in Biomedical 
Research: A Multidisciplinary View. J Med Internet Res. 2016 Dec 16;18(12):e323.  
6.  
Riley RD, Snell KI, Ensor J, Burke DL, Harrell Jr FE, Moons KG, et al. Minimum 
sample size for developing a multivariable prediction model: PART II - binary and time-
to-event outcomes. Stat Med. 2019;38(7):1276–96.