title: Towards trustworthy machine learning for clinical application using structured data and clinical notes
Creators: name: Hegselmann, Stefan
Publisher: name: University of Muenster, Institutional Repository MIAMI
Year: 2024
Subjects: subject: 000 Computer science, information, general works
subject: 610 Medicine and health

description: Die Digitalisierung der Medizin macht stetige Fortschritte und viele Krankenhäuser nutzen elektronische Patientenakten. Gleichzeitig hat sich die Leistungsfähigkeit von maschinellem Lernen drastisch verbessert. Trotz dieser Entwicklungen bleibt die Integration von maschinellem Lernen in den klinischen Alltag begrenzt. Diese Arbeit untersucht diese Diskrepanz zwischen Forschung und der klinischen Anwendung, wobei der Fokus auf den Schritten der Entwicklungspipeline liegt, die Vertrauenswürdigkeit fördern.
Wir befassen uns mit vier Bereichen: Reproduzierbarkeit, Interpretierbarkeit, kleine Datensätze und Leistungskennzahlen. Zu unseren wichtigsten Beiträgen gehört die Entwicklung einer vollständig reproduzierbaren Pipeline zur Überlebensvorhersage sowie die Evaluierung von erklärbaren Modellen in einem klinischen Umfeld. Darüber hinaus untersuchen wir die Effektivität vortrainierter großer Sprachmodelle in Szenarien mit begrenzten Trainingsdaten und erforschen die Anwendung dieser Modelle zur Erstellung von Patientenzusammenfassungen.
description: The digitization of medicine has progressed steadily, with a significant number of hospitals adopting electronic healthcare records. At the same time, the performance of machine learning systems has drastically improved. Despite these advances, the integration of machine learning methods into daily clinical practice still remains limited. This thesis explores the gap between machine learning research and its clinical application, focusing on methods within the machine learning development pipeline that foster trustworthiness.
We address four key areas: reproducibility, interpretability, small datasets, and performance metrics. Our main contributions include the establishment of a fully reproducible survival prediction pipeline and the evaluation of explainable boosting machines within a real clinical setting. In addition to that, we examine the effectiveness of pretrained large language models in scenarios with limited training data and investigate the application of these models for generating patient summaries.
description: 1 Introduction ..... 1
i Foundational Concepts
2 Interpretable Machine Learning ..... 7
2.1 Potential Problems of Learning from Associations ..... 7
2.1.1 Biased Predictions ..... 8
2.1.2 Relying on Unaware Correlations ..... 8
2.1.3 Missing Robustness ..... 9
2.2 Evaluation of Interpretable Machine Learning ..... 9
2.3 Inherently Interpretable Models ..... 11
2.3.1 Explainable Boosting Machines ..... 12
2.4 Post-Hoc Explainability ..... 12
2.4.1 Local Interpretable Model-Agnostic Explanations ..... 13
2.4.2 Shapley Additive Explanations ..... 14
3 Large Language Models ..... 15
3.1 Large Language Model Components ..... 15
3.1.1 Tokenization ..... 15
3.1.2 Positional Encoding ..... 17
3.1.3 Attention Mechanism ..... 17
3.1.4 Transformer Architecture ..... 18
3.2 Large Language Model Architectures ..... 19
3.2.1 Encoder-Decoder Models ..... 20
3.2.2 Encoder-Only Models ..... 20
3.2.3 Decoder-Only Models ..... 21
3.2.4 Model Pre-Training Objectives ..... 21
3.3 Adaptation of Large Language Models ..... 22
3.3.1 In-Context Learning (Prompting) ..... 22
3.3.2 Full Fine-Tuning ..... 23
3.3.3 Parameter-Efficient Fine-Tuning ..... 23
3.4 Pretrained Large Language Model Families ..... 24
3.4.1 T5 Models ..... 24
3.4.2 GPT Models ..... 25
3.4.3 LLaMA Models ..... 25
ii Reproducible Machine Learning for Healthcare
4 Reproducible Cancer Survival Prediction ..... 29
4.1 Introduction ..... 29
4.2 Related Work ..... 31
4.3 Literature Review ..... 31
4.3.1 Study Selection ..... 31
4.3.2 Information Extraction ..... 33
4.3.3 Experiments ..... 35
4.3.4 Reproducibility ..... 36
4.4 Cohort ..... 37
4.4.1 Cohort Selection ..... 37
4.4.2 Data Extraction ..... 39
4.4.3 Feature Choices and Target Label Generation ..... 39
4.5 Methods ..... 41
4.6 Results ..... 47
4.7 Discussion ..... 51
iii Interpretable Machine Learning for Healthcare
5 Evaluating the Doctor-Interpretability of Explainable Boosting Machines ..... 57
5.1 Introduction ..... 58
5.2 Related Work ..... 59
5.3 Dataset ..... 60
5.4 Methods ..... 60
5.4.1 Training Explainable Boosting Machines ..... 61
5.4.2 Study Participants and Instruction Procedure ..... 61
5.4.3 Questionnaire Development ..... 62
5.4.4 Statistical Analysis ..... 65
5.5 Results ..... 66
5.5.1 Training Explainable Boosting Machine ..... 66
5.5.2 Questionnaire Development ..... 68
5.5.3 Statistical Analysis ..... 69
5.6 Discussion ..... 72
6 Interpretable 3-Day Intensive Care Unit Readmission Prediction ..... 77
6.1 Introduction ..... 78
6.2 Related Work ..... 79
6.3 Dataset ..... 81
6.3.1 Variables ..... 84
6.3.2 Features ..... 88
6.3.3 Cohort from MIMIC-IV Database ..... 88
6.4 Methods ..... 91
6.4.1 Study Setting and Preregistration ..... 91
6.4.2 Baseline Models ..... 91
6.4.3 Development of Model with Limited Risk Functions ..... 93
6.4.4 Inspection of Model by a Multidisciplinary Team ..... 94
6.4.5 External Validation on MIMIC-IV Database ..... 94
6.5 Results ..... 97
6.5.1 Development of Model with Limited Risk Functions ..... 97
6.5.2 Inspection of Model by a Multidisciplinary Team ..... 99
6.5.3 Performance Compared to Baseline Models ..... 103
6.5.4 External Validation on MIMIC-IV Database ..... 103
6.6 Discussion ..... 103
iv Learning from Small Healthcare Datasets
7 Large Language Models are Few-Shot Clinical Information Extractors ..... 109
7.1 Introduction ..... 110
7.2 Related Work ..... 111
7.3 Methods ..... 113
7.3.1 Predicting Structured Outputs with Language Models ..... 113
7.3.2 Dataset Annotation ..... 114
7.4 Clinical Sense Disambiguation ..... 115
7.4.1 Dataset ..... 116
7.4.2 Prompting and Resolver ..... 116
7.4.3 Model Distillation via Weak Supervision ..... 117
7.4.4 Baselines ..... 117
7.4.5 Evaluation ..... 118
7.4.6 Example Prompts ..... 118
7.5 Biomedical Evidence Extraction ..... 120
7.5.1 Dataset ..... 120
7.5.2 Prompting and Resolver ..... 120
7.5.3 Baselines ..... 121
7.5.4 Token-level Results (Proxy Task) ..... 121
7.5.5 Oracle Definition for Evaluation ..... 122
7.5.6 Arm Identification Results ..... 123
7.5.7 Example Prompts ..... 123
7.6 Coreference Resolution ..... 125
7.6.1 Dataset ..... 125
7.6.2 Prompting and Resolver ..... 125
7.6.3 Baselines ..... 126
7.6.4 Results ..... 126
7.6.5 Example Prompts ..... 127
7.7 Medication Extraction ..... 128
7.7.1 Recognition + Status Classification ..... 128
7.7.2 Recognition + Relation Extraction ..... 132
7.7.3 Example Prompts ..... 134
7.8 Discussion ..... 138
8 Few-Shot Classification of Longitudinal Healthcare Data ..... 141
8.1 Introduction ..... 142
8.2 Related Work ..... 143
8.3 Dataset ..... 145
8.3.1 Public Tabular Datasets ..... 146
8.3.2 Large Healthcare Claims Dataset ..... 147
8.4 Methods ..... 148
8.4.1 Large Language Models for Tabular Data Classification ..... 148
8.4.2 Our Instantiation of a Large Language Model for Tabular
Data Classification ..... 149
8.5 Experimental Setup ..... 151
8.5.1 Large Language Model and Fine-Tuning ..... 151
8.5.2 Baseline Models ..... 152
8.5.3 Serializations ..... 153
8.5.4 Feature Importance Analysis ..... 157
8.6 Results ..... 158
8.6.1 Effects of Serialization ..... 158
8.6.2 Public Tabular Datasets ..... 163
8.6.3 Large Healthcare Claims Dataset ..... 169
8.7 Discussion ..... 171
8.8 Task Templates ..... 173
8.9 Example Serializations ..... 175
v Performance Metrics for Machine Learning for Healthcare
9 Data-Centric Hallucination Reduction for Generating Patient Summaries ..... 185
9.1 Introduction ..... 186
9.2 Related Work ..... 187
9.3 Dataset ..... 189
9.3.1 MIMIC Discharge Instructions Datasets ..... 189
9.3.2 Hallucination Datasets ..... 195
9.3.3 Derived Datasets ..... 195
9.4 Methods ..... 195
9.4.1 Generation of Patient Summaries ..... 195
9.4.2 Protocol for Labeling Hallucinations ..... 199
9.4.3 Hallucinations in MIMIC and Generated Patient Summaries ..... 200
9.4.4 Data-Centric Hallucination Reduction ..... 202
9.4.5 Quantitative Evaluation ..... 203
9.4.6 Qualitative Evaluation ..... 203
9.4.7 Automatic Hallucination Detection ..... 206
9.5 Results ..... 213
9.5.1 Hallucinations in MIMIC and Generated Patient Summaries ..... 213
9.5.2 Data-Centric Hallucination Reduction ..... 214
9.5.3 Quantitative Evaluation ..... 215
9.5.4 Qualitative Evaluation ..... 215
9.5.5 Automatic Hallucination Detection ..... 216
9.6 Discussion ..... 218
9.7 Further Labeling Examples ..... 220
10 Conclusion ..... 223
Bibliography ..... 227