Additional file 2 of Prediction of American Society of Anesthesiologists Physical Status Classification from preoperative clinical text narratives using natural language processing
Creators: name: Chung, Philip
name: Fong, Christine T.
name: Walters, Andrew M.
name: Yetisgen, Meliha
name: O’Reilly-Shah, Vikas N.
Publisher: name: figshare
Year: 2023
Subjects: Medicine
Biological Sciences not elsewhere classified
Information Systems not elsewhere classified

description: Additional file 2: Supplemental Figure 1. BioClinicalBERT Model Architecture with additional prediction heads for fine-tuning and prediction of modified ASA-PS. Supplemental Figure 2. Attribution of input text features to predicting modified ASA-PS for the BioClinicalBERT model on Note512 task. Model prediction is ASA I, Anesthesiologist assigned case ASA IV-V. Notable findings include the model focusing on pertinent negatives on trauma exam and imaging findings and a normal hematocrit of 33 all of which support predicting a ASA-PS I. The same pertinent negatives as well as a Glasgow Coma Scale (GCS) of 15 are negatively Shapley values for ASA-PS IV-V, which reduce the probability of predicting ASA IV-V. Despite the anesthesiologist’s assignment of ASA IV-V, the text description does not suggest the patient has severe systemic disease with constant threat to life (ASA IV) or is moribund and requires the operation to survive (ASA V). Text examples are de-identified by replacing ages, dates, names, locations, and entities with pseudonyms to achieve data obfuscation while preserving structural similarity to the original passage. Supplemental Figure 3. Attribution of input text features to predicting modified ASA-PS for the BioClinicalBERT model on Note512 task. Model prediction is ASA I, Anesthesiologist assigned case ASA IV-V. Notable findings include the model associating chest tube with ASA IV-V. The model has trouble with consistently attributing the multiple mentions of eyelid laceration with a specific ASA-PS. The model may be inappropriately assigning mention of left pneumothorax to ASAI. This example depicts a challenge for the model in which a relatively minor injury (eyelid laceration) is simultaneously present with a potentially severe injury (pneumothorax), though the severity of the pneumothorax is not mentioned and thus the text predominantly supports ASA I (healthy) or ASA II (mild systemic disease). This kind of mixed illness/injury example coupled with a narrative that does not clearly describe disease severity may be a struggle for the model. Text examples are de-identified by replacing ages, dates, names, locations, and entities with pseudonyms to achieve data obfuscation while preserving structural similarity to the original passage. Supplemental Figure 4. Attribution of input text features to predicting modified ASA-PS for the BioClinicalBERT model on Note512 task. Model prediction is ASA IV-V, Anesthesiologist assigned case ASA I. Notable findings include: young age associated with ASA I and ASA IV-V, but negatively associated with ASA II and III; diagnosis of perforated appendix and procedure of laparoscopic appendectomy negatively associated with ASA I and positively associated with higher ASA-PS; model identifying broad-spectrum antibiotics such as ertapenem to be associated with ASA IV-V, but narrower-spectrum antibiotics such as metronidazole, cefazolin to be heavily associated with ASA I; inpatient medications such as subcutaneous heparin and ondansetron negatively associated with lower ASA-PS and positively associated with higher ASA-PS. Text examples are de identified by replacing ages, dates, names, locations, and entities with pseudonyms to achieve data obfuscation while preserving structural similarity to the original passage. Supplemental Figure 5. Attribution of input text features to predicting modified ASA-PS for the BioClinicalBERT model on Note512 task. Model prediction is ASA IV-V, Anesthesiologist assigned case ASA I. Notable findings include medical conditions and interventions associated with higher ASA-PS such as cardiomyopathy, internal cardiac defibrillator (ICD) generator change, paroxysmal ventricular tachycardia, left ventricular assist device (LVAD), heart failure, possible transplantation, tricuspid valve repair, and patent foramen ovale (PFO) closure; history of chronic cigarette smoking and snoring associated with ASA IV-V. The text description is at least ASA III (severe systemic illness), and can be argued to be ASA IV (severe systemic disease with constant threat to life) if heart failure is progressively worsening. In this example the model appears to make a more appropriate ASA-PS prediction than the anesthesiologist. Text examples are de-identified by replacing ages, dates, names, locations, and entities with pseudonyms to achieve data obfuscation while preserving structural similarity to the original passage. Supplemental Table 1. (A) Matthew's correlation coefficient (MCC) and (B) AUCµ for each model architecture and task on the held-out test set compared to baseline models. MCC is a categorical analog of Pearson’s correlation coefficient. AUCµ is a multiclass generalization of AUROC and U statistic and is more theoretically grounded than macro-average AUROC, but less commonly reported. Standard errors are reported in parenthesis. Supplemental Table 2. (A) Macro-average F1 and (B) class specific F1 for each model architecture and task on the held-out test set compared to baseline models. Standard errors are reported in parenthesis. Supplemental Table 3. (A) Macro-average precision and (B) class-specific precision for each model architecture and task on the held-out test set compared to baseline models. Standard errors are reported in parenthesis. Supplemental Table 4. (A) Macro-average recall and (B) class-specific recall for each model architecture and task on the held-out test set compared to baseline models. Standard errors are reported in parenthesis. Supplemental Table 5. (A) Macro-average AUROC and (B) class-specific AUROC for each model architecture and task on the held-out test set compared to baseline models. Standard errors are reported in parenthesis. Supplemental Table 6. (A) Macro-average AUPRC and (B) class-specific AUPRC for each model architecture and task on the held-out test set compared to baseline models. Standard errors are reported in parenthesis. Supplemental Table 7. P-values for all pairwise comparisons which were not statistically significant. Reported p-values are  corrected for multiple hypothesis testing using the Benjamini-Hochberg procedure with α =0.01.