Evaluating the efficacy of pre-operative fistula risk assessment.

We read with great interest the recent manuscript by Roberts and colleagues which introduced a pre-operative fistula risk metric for patients undergoing a pancreatoduodenectomy (PD).1 As a post-operative pancreatic fistula (POPF) is the most common complication after PD, the authors argue that such a tool could be useful for identifying at-risk patients, and adjusting the management of their care accordingly. Their study presents a thoughtful and balanced report regarding the merits and deficiencies of their approach; however, we believe this manuscript introduces issues worthy of further discourse, particularly, POPF risk assessment. As risk scores become more prevalent, it is imperative to ensure their fidelity through a rigorous external validation process prior to ubiquitous acceptance. This requires multi-centre assessment with a sizable number of patients. With this in mind, we attempted to externally validate the model proposed by Roberts. Pre-operative risk scores were derived for 1345 patients (four institutions and six surgeons) undergoing PD and authenticated using the web-based calculator.1 Patient characteristics evaluated for the model were body mass index (BMI) and pancreatic duct diameter. POPFs were graded in accordance with ISGPF standards and designated as either biochemical or clinically relevant fistulae (CR-POPF). Risk scores were entered into logistic regression models with POPF and CR-POPF as the outcomes of interest. The overall POPF rate was 27.4%, of which, 46.1% were clinically relevant (12.6% overall). The median BMI and pancreatic duct diameter measurements were 25.7 kg/m2 and 4 mm, respectively. The mean and median risk score was 16.0 and 13.0. Regression analysis identified the risk score as a significant predictor for overall POPF and CR-POPF (P < 0.001). Sensitivity and specificity for POPF prediction was 12.20% and 95.80%. The positive and negative predictive values (PPV and NPV) were 52.3% and 74.3%. Sensitivity and specificity for CR-POPF prediction was 0% and 99.7% moreover, the PPV and NPV was 0% and 87.3%. Calibration of both models was deemed unacceptable by Hosmer–Lemeshow χ2 testing (POPF: P < 0.001; CR-POPF: P = 0.029). The c-statistics for predicting POPF and CR-POPF were 0.689 and 0.666, respectively (P < 0.001). This analysis demonstrates that the risk score proposed by Roberts and colleagues is not sufficient to replace other established metrics for fistula risk assessment. Fistula incidence was predicted in only 12.2% of cases, whereas the absence of fistula was correctly predicted 74.3% of the time. Although the absence of CR-POPF was predicted with high accuracy (NPV = 99.7%), the positive prediction of these fistula – arguably more important in regards to preoperative counselling – was poor (PPV = 0%). The area under the curve (AUC) values were <0.7 for the prediction of POPF and CR-POPF – lower than in the original report by Roberts – as well as the externally validated Fistula Risk Score (FRS); the latter was validated using a nearly identical cohort.2,3 Furthermore, Hosmer–Lemeshow tests produced P-values <0.05, which signified a poor model fit. In addition to concerns about the model's predictive capacity, we have other reservations. First, disease pathology categorization paired pancreatic cancer with ampullary cancer and cholangiocarcinoma, which contrasts with findings in the literature that indicate greater fistula risk with the latter pathologies. Pathologies other than pancreatitis and pancreatic cancer usually correlate with soft glands and non-dilated ducts, unfavourable scenarios for anastomotic connection.4 Perhaps, confounding factors caused patients with pancreatic, ampullary and bile duct cancers to have similar fistula rates during univariate analysis. Clustering pathologies based upon pre-existing literature and clinical intuition would be preferable, and may have led to the inclusion of pathologic diagnosis in the model's construct, as with the FRS.2,5 Of greater concern is Roberts' collective prediction of biochemical and CR-POPF. In our current assessment, the c-statistic improved when predicting the overall occurrence of fistula rather than only CR-POPF. As biochemical fistulae have no clinical impact, the benefit derived from their pre-operative prediction is questionable. An advantage of pre-operative risk assessment is the ability to provide informed consent to the patient regarding their operative risk for morbidity and mortality. If a surgeon informs a patient that they have a high risk for fistula development, yet is unable to say whether the fistula would be innocuous or clinically impactful, what is the benefit of such a prediction or consent process? In this external assessment, predicting any type of fistula, rather than just CR-POPF, predicted 52.3% more patients to have a fistula when one actually occurred; however, 13% more patients developed a fistula when one was not predicted. Striking the right balance between PPV and NPV is not a simple process, but a risk model must have a high NPV when ruling out patients for significant complications. Next, the prediction of POPF is inadequate when only pre-operative variables are considered. While pre-operative prediction may have a role in counselling, we do not believe it realistically influences surgical decision making. Do we really decide not to provide a resection to a patient because they have a defined risk of this single, usually manageable, complication? We believe information obtained at the point of anastomotic reconstruction is relevant for predicting anastomotic failure. This is reflected by the importance of intra-operative blood loss as a contributor to CR-POPF risk.5 Furthermore, BMI and other measurements of pancreatic adiposity only estimate pancreatic texture. We contend that this risk factor is most accurately assessed intra-operatively by an experienced pancreatic surgeon. To conclude, although pre-operative risk assessment may offer some value, it should not be implemented at the cost of predictive capacity, particularly if the score is also used for comparative research or performance assessment. Previously validated fistula risk metrics, which include intra-operative criteria, should not be replaced until the advent of a more accurate pre-operative risk score.