A Multi-Modal and Multitask Benchmark in the Clinical Domain

Healthcare represents one of the most promising application areas for machine learning algorithms , including modern methods based on deep learning . Modern deep learning algorithms perform best on large datasets and on unstructured modalities such as text or image data ; advances in deep learning have often been driven by the availability of such large datasets . Here , we introduce Multi-Modal Multi-Task MIMIC-III ( M3 ) — a dataset and benchmark for evaluating machine learning algorithms in the healthcare domain . This dataset contains multi-modal patient data collected from intensive care units — including physiological time series , clinical notes , ECG waveforms , and tabular inputs — and defines six clinical tasks — including predicting mortality , decompensation , readmission , and other outcomes — which serve as benchmarks for comparing algorithms . We introduce new multi-modal and multi-task models for this dataset , and show that they outperform previous state-of-the-art results that only rely on a subset of all tasks and modalities . This highlights the potential of multi-task and multi-modal learning to improve the performance of algorithms in the healthcare domain . More generally , we envision M3 as a general resource that will help accelerate research in applying machine learning to healthcare . 1 INTRODUCTION . Healthcare and medicine are the some of the most promising areas in which machine learning algorithms can have an impact ( Yu et al. , 2018 ) . Techniques relying on machine learning have found successful applications in dermatology , ophthalmology , and many other fields of medicine ( Esteva et al. , 2017 ; Gulshan et al. , 2016 ; Hannun et al. , 2019 ) . Modern machine learning techniques — including algorithms based on deep learning — perform best on large datasets and on unstructured inputs , such as text , images , and other forms of raw signal data ( You et al. , 2016 ; Agrawal et al. , 2016 ) . Progress in modern machine learning has in large part been driven by the availability of these types of large datasets as well as by competitive benchmarks on which algorithms are evaluated ( Deng et al. , 2009 ; Lin et al. , 2014 ) . Recently , machine learning algorithms that combine data from multiple domains and that are trained to simultaneously solve a large number of tasks have achieved performance gains in domains such as machine translation and drug discovery ( Johnson et al. , 2017 ; Ramsundar et al. , 2015 ) . Current research in this area is driven by widely adopted computational benchmarks , particularly in the field of natural language processing ( Wang et al. , 2018a ; 2019 ) . In this paper , we argue that multi-modal and multitask benchmarks can similarly drive progress in applications of machine learning to healthcare . In many healthcare settings , we have access to data coming from diverse modalities — including radiology images , clinical notes , wearable sensor data , and others — and we are solving many tasks — for example , estimating disease risk , predicting readmission , and forecasting decompensation events . These kinds of settings are naturally suited to modern deep learning algorithms ; developing models that effectively leverage diverse tasks and modalities has the potential to greatly improve the performance of machine learning algorithms in the clinical domain . As a first step in this research direction , we introduce in this paper Multi-Modal Multi-Task MIMICIII ( M3 ) 1 , a dataset and benchmark for evaluating machine learning algorithms in healthcare that is inspired by popular multitask benchmarks in other application domains , such as natural language processing ( Wang et al. , 2018b ; McCann et al. , 2018 ) . Previous clinical datasets and benchmarks have either focused on specific tasks in isolation as in Khadanga et al . ( 2020 ) or on multiple tasks over a single input modality ( Harutyunyan et al. , 2019 ) . Our work is the first to combine multiple tasks and modalities into one benchmark . More specifically , we propose a dataset that is derived from the MIMIC-III database and is comprised of data collected from over forty thousand patients who stayed in intensive care units ( ICUs ) of the Beth Israel Deaconess Medical Center between 2001 and 2012 ( Johnson et al. , 2016 ) . As part of this dataset , we have collected data from four modalities — including physiological time series , clinical notes , ECG waveforms , and tabular data — and have defined six clinical tasks — mortality prediction , decompensation , readmission , and others . We also propose an evaluation framework to benchmark models on this dataset . As a demonstration of how the M3 benchmark can drive progress in clinical applications of machine learning , we propose a first set of multi-modal and multitask models and evaluate them on our new benchmark . We find that these models achieve high performance levels and may serve as strong baselines for future work . In particular , our models outperform previous state-of-the-art results that only rely on a subset of all tasks and modalities . These results highlight the potential of multitask and multi-modal learning to improve the performance of algorithms in the healthcare domain . We envision M3 as a general resource that will help accelerate research in applying machine learning to healthcare . To facilitate such uses , we release M3 and our models as an easy-to-use open-source package for the research community . Contributions . In summary , our paper makes the following contributions . • We define a new benchmark for machine learning algorithms in the clinical domain . It defines six clinical tasks , and is the first to collect data across multiple modalities . • We introduce new multi-modal and multitask machine learning models which outperform previous state-of-the-art methods that only rely on a subset of tasks or modalities . This highlights the importance of multi-modal and multitask learning in clinical settings . • We package our benchmark into an easy to use format such that the clinical machine learning community can further build upon our work . 2 BACKGROUND . Machine Learning in the Clinical Domain . Machine learning has been successfully applied throughout healthcare , including in areas such as medical imaging , drug discovery , and many others ( Rajpurkar et al. , 2017 ; Vamathevan et al. , 2019 ) . In this paper , we restrict our attention to a specific healthcare setting — intensive care . The Medical Information Mart for Intensive Care ( MIMIC-III ) database is one of the most important resources for applying machine learning to intensive care ( Johnson et al. , 2016 ) . Data collected in the ICUs includes vital signs , lab events , medical interventions , and socio-demographic information . Multi-Modal and Multi-Task Learning . Multitask learning trains models to simultaneously solve multiple tasks ( Ruder , 2017 ) . Successful applications of multitask learning include machine translation and drug discovery ( Johnson et al. , 2017 ; Ramsundar et al. , 2015 ) . Current research in this area is driven by popular benchmarks , particularly in the field of natural language processing ( Wang et al. , 2018b ; 2019 ; Rajpurkar et al. , 2016 ) . Multi-modal machine learning combines and models data of different modalities such as vision , language , speech . A key challenge in multi-modal learning is to combine representations over diverse input types . Applications of multi-modal learning include image captioning and visual question answering ( Anderson et al. , 2018 ; Agrawal et al. , 2016 ; Moradi et al. , 2018 ; Nguyen et al. , 2019 ) . 1Our code is available here : https : //github.com/DoubleBlindGithub/M3 The clinical domain lends itself naturally to multi-modal learning , given the prevalence of data in different modalities , including physiological signals , clinical notes , medical images , tabular inputs and genome sequences . In this work , we propose the first multitask and multi-modal benchmark over this diverse clinical data . 3 THE MULTI-MODAL MULTI-TASK MIMIC BENCHMARK . Next , we introduce Multi-Modal Multi-Task MIMIC-III ( M3 ) , a dataset and benchmark for evaluating machine learning algorithms in healthcare . This dataset is derived from the MIMIC-III database and is comprised of data collected from over forty thousand patients . It contains data from four diverse modalities and defines six clinical tasks . It also proposes an evaluation framework to benchmark models and comes in an easy-to-use open source package . Each patient in the dataset completes a number of stays in an ICU , with 13 % of patients completing more than one stay . Over the course of each stay , the ICU collects measurements of vital signs and other clinical variables at irregular intervals , as well as clinical notes describing the state of the patient . 3.1 DESIGN PRINCIPLES . The goal of the M3 benchmark is to accelerate progress in applications of machine learning to healthcare . The design of this benchmark is guided by the principles of multi-modality — we collect data from diverse and unstructured modalities found in healthcare and that can be leveraged by modern deep learning algorithms — as well as task relevance , diversity , and evaluability — we choose diverse real-world clinical tasks that enhance model performance when solved jointly and that have well-defined success metrics . Finally , we are concerned with accessibility : in order to ensure that our benchmark can be widely adopted , we base it on datasets that can be obtained with minimum overhead and that satisfy privacy and legal requirements . 3.2 TASKS . The M3 benchmark includes 6 different clinical prediction tasks . Each task is performed at specific time points within the ICU stay of a patient.More details can be found in Table 1 . In Hospital Mortality . We observe the first 48 hours of a patient ’ s data , and then predict whether the patient will die by the end of their stay ( Harutyunyan et al. , 2019 ; Khadanga et al. , 2020 ; Purushotham et al. , 2017 ; Wang et al. , 2020 ; Tang et al. , 2020 ) . Mortality is one of the major concerns for any ICU unit , with limiting mortality being an ultimate goal for most ICUs . Decompensation . Starting from the fifth hour of the stay , a prediction is made at every hour about whether the patient will die within the next 24 hours given all the data collected to that point . Unlike in the IHM task , predictions are made on an hourly basis rather than after a set amount of time and concern the next 24 hours rather than the entire stay . As such , this task may better reflect the changing landscape of available patient information . Length of Stay . We predict the total duration time from admission to discharge . This task could provide useful information for medical resources allocation and scheduling . We formulate this task as a multiclass classification problem with three classes/bins ( 0-3 days , 3-7 days , and longer than 7 days ) using only data from the 24 hours of the stay . Phenotyping . A prediction of the patient ’ s phenotype is made at discharge time . This is a multilabel classification task . The target label is derived from the billing code at a patients discharge , which we then convert to our 25 labels following the procedure from Harutyunyan et al . ( 2019 ) . Readmission . We predict if another ICU stay will occur to the same patient after the discharge time of an ICU stay . Predicting readmission is useful to identify higher risk patients and minimize the waste of financial resources . We define this as a multiclass classification problem with 5 classes — readmission within 7 , 7-30 , 30-90 , 90-365 , and 365+ days or no readmission . Long-Term Mortality . For each ICU stay , we predict if the patient survives for more than 1 year after discharge . We frame this as a binary classification problem . Predicting long-term mortality is useful for assessing patients ’ well-being after discharge .

Summary: This paper defines a new task for clinical data by combing multi-modal and multi-task settings into one task. It collects a dataset called M3 as the benchmark for the multi-modal and multi-task benchmark in the clinical domain. The dataset has 6 prediction tasks, i.e., in-hospital mortality, decompensation, length of stay, phenotyping, readmission, and long-term mortality, and it has 4 modalities, i.e., physiological time series, clinical notes, tabular data, and waveforms. Specifically, this paper also provides a multi-modal multi-task model where the time series data are encoded by LSTM, clinical notes are encoded by text CNN and tabular data are also encoded by existing methods.  In experiments, the authors conduct an ablation study and compare the proposed method with the method of Harutyunyan et al. and Khadanga et al.