A Multi-Modal and Multitask Benchmark in the Clinical Domain
Authors: Yong Huang
Edgar Mariano Marroquin
Volodymyr Kuleshov
Venue:  (2021)
Keywords: multi-modal
multitask
machine learning in healthcare
benchmark

Abstract: Healthcare represents one of the most promising application areas for machine learning algorithms,  including modern methods based on deep learning.   Modern deep learning algorithms perform best on large datasets and on unstructured modalities such as text or image data; advances in deep learning have often been driven by the availability of such large datasets. Here, we introduce Multi-Modal Multitask MIMIC-III (M3) — a dataset and benchmark for evaluating machine learning algorithms in the healthcare domain.  This dataset contains multi-modal patient data collected from intensive care units — including physiological time series, clinical notes, ECG waveforms, and tabular inputs — and defines six clinical tasks — including predicting mortality, decompensation, readmission, and other outcomes — which serve as benchmarks for comparing algorithms. We introduce new multi-modal and multitask models for this dataset, and show that they outperform previous state-of-the-art results that only rely on a subset of all tasks and modalities. This highlights the potential of multitask and multi-modal learning to improve the performance of algorithms in the healthcare domain. More generally, we envision M3 as a general resource that will help accelerate research in applying machine learning to healthcare.

Under review as a conference paper at ICLR 2021
A MULTI-MODAL AND MULTITASK BENCHMARK IN THE CLINICAL DOMAIN
Anonymous authors Paper under double-blind review
ABSTRACT
Healthcare represents one of the most promising application areas for machine learning algorithms, including modern methods based on deep learning. Mod- ern deep learning algorithms perform best on large datasets and on unstructured modalities such as text or image data; advances in deep learning have often been driven by the availability of such large datasets. Here, introduce Multi-Modal Multi-Task MIMIC-III (M3) - a dataset and benchmark for evaluating machine learning algorithms in the healthcare domain. This dataset contains multi-modal patient data collected from intensive care units - including physiological time se- ries, clinical notes, ECG waveforms, and tabular inputs and defines six clinical tasks - including predicting mortality, decompensation, readmission, and other outcomes which serve benchmarks for comparing algorithms. We introduce new multi-modal and multi-task models for this dataset, and show that they out- perform previous state-of-the- results that only rely on a subset of all tasks and modalities. This highlights the potential of multi-task and multi-modal learning to improve the performance of algorithms in the healthcare domain. More generally, we envision M3 as a general resource that will help accelerate research in applying machine learning to healthcare.
INTRODUCTION
Healtheare and medicine are the some of the most promising areas in which machine learning al- gorithms can have an impact (Yu et al. 2018). Techniques relying on machine learning have found successful applications in dermatology, ophthalmology. and many other fields of medicine (Esteval
jet al. 2017 Gulshan et al.) 2016 Hannun et al.) 2019).
Modern machine learning techniques including algorithms based deep learning perform best on large datasets and on unstructured inputs, such as text, images, and other forms of raw signal data (You et al. 20166] Agrawal et al.) (0066). Progress in modern machine learning has in large part been driven by the availability of these types of large datasets as well as by competitive benchmarks on which algorithms are evaluated (Deng al. 2009 Lin al. 2014).
Recently, machine learning algorithms that combine data from multiple domains and that are trained simultaneously solve large number of tasks have achieved performance gains in domains such as machine translation and drug discovery (Johnson et al., 2017]] |amssundar et al., 2015). Current research in this area is driven by widely adopted computational benchmarks, particularly in the field of natural language processing (Wang et al.) 2018a 2019).
In this paper, we argue that multi-modal and multitask benchmarks can similarly drive progress in applications of machine learning to healthcare. In many healthcare settings, we have access to data coming from diverse modalities - including radiology images, clinical notes, wearable sensor data, and others and we are solving many tasks - for example, estimating disease risk, predicting readmission, and forecasting decompensation events. These kinds of settings are naturally suited to modern deep learning algorithms; developing models that effectively leverage diverse tasks and modalities has the potential to greatly improve the performance machine learning algorithms in the clinical domain.

---

Under review as a conference paper at ICLR 2021
As first step in this research direction, we introduce in this paper Multi- Multi-Task MIMIC- III (M31) a dataset and benchmark for evaluating machine learning algorithms in healthcare that is inspired by popular multitask benchmarks in other application domains, such natural language processing (Wang et al. 2018b; McCann et al.; 2018). Previous clinical datasets and benchmarks have either focused specific tasks in isolation in Khadanga al. (2020)
multiple tasks
over a single input modality (Harutyunyan et al. 2019). Our work is the first to combine multiple tasks and modalities into one benchmark.
More specifically, we propose a dataset that is derived from the MIMIC-III database and is com- prised of data collected from over forty thousand patients who stayed in intensive care units (ICUs) of the Beth Israel Deaconess Medical Center between 2001 and 2012 (Johnson et al. 2016). As part of this dataset, we have collected data from four modalities - including physiological time series, clinical notes, ECG waveforms, and tabular data and have defined six clinical tasks mortality prediction, decompensation, readmission, and others. We also propose an evaluation framework to benchmark models on this dataset.
As demonstration of how the M3 benchmark can drive progress in clinical applications of machine learning, we propose a first set of multi-modal and multitask models and evaluate them our new benchmark. We find that these models achieve high performance levels and may serve as strong baselines for future work. In particular, our models outperform previous state-of-the art results that only rely on a subset of all tasks and modalities.
These results highlight the potential of multitask and multi-modal learning to improve the perfor- mance of algorithms in the healthcare domain. We envision M3 as a general resource that will help accelerate research in applying machine learning healthcare. To facilitate such uses, we release M3 and our models as an easy-to-use open-source package for the research community.
Contributions. In summary, our paper makes the following contributions.
We define new benchmark for machine learning algorithms in the clinical domain. It defines six clinical tasks, and the first to collect data across multiple modalities.
We introduce new multi-modal and multitask machine learning models which outperform previous state-of-the-art methods that only rely on subset of tasks or modalities. This highlights the importance of multi-modal and multitask learning in clinical settings.
We package our benchmark into an easy to use format such that the clinical machine learn- ing community can further build upon our work.
2 BACKGROUND
Machine Learning in the Clinical Domain. Machine learning has been successfully applied throughout healthcare, including in areas such as medical imaging, drug discovery, and many others (Rajpurkar et al. 2017; Vamathevan et al. 2019). In this paper, restrict our attention to specific healthcare setting intensive care.
The Medical Information Mart for Intensive Care (MIMIC-III) database is one of the most important resources for applying machine learning to intensive care (Johnson et al., 2016). Data collected in the ICUs includes vital signs, lab events, medical interventions, and socio demograppmaa information.
Multi-Modal and Multi-Task Learning. Multitask learning trains models to simultancously solve multiple tasks (Ruder 2017). Successful applications of multitask learning include machine translation and drug discovery (Johnson et al. [2017, Ramsundar et al., (015).. Current research in this area driven by popular benchmarks, particularly the field of natural language processing (Wang et al. 201 8b 2019: Rajpurkar et al. 2016).
Multi-modal machine learning combines and models data of different modalities such as vision, lan- guage, speech. A key challenge in multi-modal learning is to combine representations over diverse input types. Applications of multi-modal lcarning include image captioning and visual question answering (Anderson et al. 2018 Agrawal et al. 2016 Moradi et al. 2018 Nguyen et al. 2019).
Our code is available here: http://giiuakccmm///bahhhhiiiiin
2

---

Under review as a conference paper at ICLR 2021
Table 1: Overview of the M3 Benchmark. The M3 benemark is comprised of six tasks de- compensation, length of stay, in-hospital (IH) mortality, long-term (LT) mortality, phenotyping, and readmission. For each task, we specify the number of available labeled training instances, the time(s) within the ICU stays at which predictions are made, the type of classification task, and the main metric used for evaluation. The M3 dataset also contains data from four modalities; we specify the dimensionality of each modality and the number of ICU stays containing data from it.
Training Instances
Task
Prediction time(s)
Type
Main metric
7 Train
Val
Test
Decomp.
2495.8k .8k
360.3k
523.1k
every hour
binary
aucpr
Length of Stay
25.7k
3673
5280
24 hour
multiclass
aucroc ovr
Mortality (IH)
15.4k
2209
3234
48 hour
binary
aucpr
Phenotyping
30.7k
4383
6278
discharge time
multilabel
macro aucroc
Readmission
27.4k
3894
5659
discharge time
multiclass
aucroc ovr
Mortality (LT)
27.4k
3894
5659
discharge time
binary
aucpr
ICU stays
Modality
Dimensionality
Train
Val
Test
time series
30701
4383
6278
59
clinical notes
30507
4368
6247
tabular inputs
30701
4383
6278
6
waveforms
5567
686
1060
1
The clinical domain lends itself naturally multi-modal learning, given the prevalence of data in different modalities, including physiological signals, clinical notes, medical images, tabular inputs and genome sequences. this work, we propose the first multitask and multi-modal benchmark over this diverse clinical data.
THE MULTI-MODAL MULTI-TASK MIMIC BENCHMARK
Next, we introduce Multi-Modal Multi-Task MIMIC-II (M3), a dataset and benchmark for evaluat- ing machine learning algorithms in healthcare. This dataset is derived from the MIMIC-III database and is comprised of data collected from over forty thousand patients. It contains data from four diverse modalities and defines six clinical tasks. It also proposes an evaluation framework bench- mark models and comes in an casy-to-use open source package.
Each patient the dataset completes number of stays an ICU, with 13% of patients completing more than one stay. Over the course of each stay, the ICU collects measurements of vital signs and other clinical variables irregular intervals, as well as clinical notes describing the state of the patient.
3.1 DESIGN PRINCIPLES
The goal of the M3 benchmark is to accelerate progress in applications of machine learning to healthcare. The design of this benchmark is guided by the principles of multi-modality we collect data from diverse and unstructured modalities found healthcare and that can leveraged by modern deep learning algorithms choose diverse real-world clinical tasks that enhance model performance when solved jointly and that have well-defined success metrics. Finally, ensure that our benchmark can be widely adopted, we base it on datasets that can be obtained with minimum overhead and that satisfy privacy and legal requirements.
well as task relevance, diversity, and evaluability
concerned with accessibility: in order to
3.2 TASKS
The M3 benchmark includes 6 different clinical prediction tasks. Each task is performed at specific time points within the ICU stay of a patient. More details can be found in Table
3

---

Under review as a conference paper at ICLR 2021
In Hospital Mortality. We observe the first 48 hours of patient's data, and then predict whether the patient will die by the end their stay (Harutyunyan et al. 2019 Khadanga et al.) 2020 Pu- rushotham et al., 2017; Wang et al. 2020 Tang et al. 2020). Mortality is one of the major concerns for any ICU unit, with limiting mortality being an ultimate goal for most ICUs.
Decompensation. Starting from the fifth hour of the stay, a prediction is made at every hour about whether the patient will die within the next hours given all the data collected to that point. Unlike in the IHM task, predictions are made on an hourly basis rather than after set amount of time and concern the next 24 hours rather than the entire stay. As such, this task may better reflect the changing landscape of available patient information.
Length of Stay. We predict the total duration time from admission to discharge. This task could provide useful information for medical resources allocation and scheduling. We formulate this task multiclass classification problem with three classes/bins (0-3 days, 3-7 days, and longer than days) using only data from the 24 hours of the stay.
Phenotyping. A prediction of the patient's phenotype is made at discharge time. This is a multi- label classification task. The target label derived from the billing code patients discharge, which we then convert to our 25 labels following the procedure from Harutyunyan et al. (011).
Readmission. We predict if another ICU stay will occur to the same patient after the discharge time of an ICU stay. Predicting readmission is useful to identify higher risk patients and minimize the waste of financial resources. We define this as a multiclass classification problem with 5 classes readmission within 7, 7-30, 30-90, 90-365, and 365+ days or no readmission.
Long-Tew Term Mortality. For each ICU stay, we predict the patient survives for more than year after discharge. We frame this as binary classification problem. Predicting long-tem mortality is useful for assessing patients' well-being after discharge.
3.3 MODALITIES
Our dataset contains data from four diverse and unstructured modalities that are typically found in healthcare settings.
Physiological Time Series. We select 59 temporal physiological variables from MIMIC-II, such diastolic blood pressure, systolic blood pressure, oxygen saturation. The full list is in Table These physiological variables are recorded irregularly, and they are important indicators of the patient's condition during the ICU stay.
g
Clinical Notes. Clinical notes are written by clinicians and nurses during the ICU stay and usually summarize topics such as reasons for admission, details of treatment, nutrition, and the patients' respiratory conditions. These clinical notes are also temporal, and they are charted sporadically.
Tabular Data. For every patient, we have access to their recorded sex, age, height, and weight upon entry to the ICU, the type of the ICU, and other tabular inputs. We consider only the initial values upon entry to the ICU. Some of these fields, such as weight, may fluctuate throughout the ICU stay, and are also part of the time Series data.
Waveforms.
Our dataset includes electrocardiogram (ECG) data from MIMIC Waveform (Moody 2020). About 16% of patient stays have recorded waveform data.
3.4
DATASET AND BENCHMARK
We employed the preprocessing steps described in Harutyunyan al. (2019). excluding ICU stays with missing events or missing length-of-stay and excluding patients younger than 18 years old because of the significant difference between adult and pediatric physiology.
4

---

Under review as a conference paper at ICLR 2021
Canaa
inggat
mopel
Iuo model
mrodel
mope
Lim model
Mull-modal
Muti-moddi enzoder
Multi-nodal encoder
Mult-modal
Ciinical
Tabular
Time
Climical
Time Tabular
notes
series
data
noles
series data
notce
series datta
Hour
Hour 24
Hour 48
Hour
Figure 1: Multi-modal multi-task models for the M3 benchmark. We propose architecture con- sisting of an encoder that outputs multi-modal embeddings based on text, time series, and tabular inputs as well as recurrent connections from earlier time steps (green horizontal lines). At each prediction time, these embeddings are used by task-specific components (e.g.. "Decomp model" and others) to output predictions. Predictions evaluated by task- losses and an overall multitask loss.
croozr
PUB coneeeeee moneeecee..oo
Cinoal Bdla
lost oneccar
Noclo sacodor
Task www.pccccooooccoooooooooooooooo comoomeet
Figure 2: Multi-modal encoder and task-specific components. At each time step f! are the encoded time series features, f!²) are the encoded tabular inputs, and f(() is the encoded clinical notes. The O, is concatenated global embedding.
For binary classification problems, define AUC-PR and AUC-ROC the evaluation metrics. For multiclass classification problems, we define AUC-ROC-OVR (one versus rest), which compares AUC-ROC for the multilabel classification
the AUC of each class against the rest. We problem; it is the unweighted mean of AUC-ROC for each label.
MULTI-MODAL MULTI-TASK CLINICAL MODELS
As demonstration of how our work drive progress in clinical applications of machine learning. we propose first set of multi-modal and multitask models for our new benchmark.
Our proposed architecture consists of encoder that outputs multi-modal embeddings based on text, time series, and tabular inputs as well as recurrent connections from earlier time steps. The embeddings are fed task speciff models that perform predictions.
4.1 ENCODERS
The multi-modal encoder is comprised of one child encoder per input modality (Figure 2 The multi-modal embedding is a concatenation of the outputs from each child encoder. We describe these below. Note that we do not use waveforms, as we found that we not have enough inputs of this modality improve model performance.
5

---

Under review as a conference paper at ICLR 2021
Time Series. Given patient's s ICU stay of length of T hours. resample time series data with hour interval and get [TS,] from t 1 to T. We impute missing values as described in modalities section. Our time series encoder is an LSTM (Hochreiter & Schmidhuber [997). The input TS; at time step / directly fed to the LSTM model along with the previous hidden state and cell state, and the next hidden state is our extracted feature, denoted by fe Formally, the feature extraction step is described as:
S LSTM(TSS, 121)
For simplicity, cell states omitted here.
Clinical Notes For each ICU stay, have N clinical notes [Note, charted sporadically, the
charted time of these notes are ( where N is usually smaller than T. We use the text CNN proposed in Kim (2014) extract text features. To create embeddings from N, notes collected 2) preceding the current time step t (i.e. Time(i) S for every i S N2), we first extract
times Time
features [g] N;
] of each note by the text CNN separately, and then average the extracted features
KKadanga al. 2020), assigning more weight recent notes:
N weight(t,,
2, TextCNN(Notc.)
weight(t, i) = exp(-A(t Time(i))
(00
=1
Here, is a scaling factor and y normalization term.
Tabular Data We have a total tabular inputs that non-temporal. To process the tabular inputs, we learn an embedding table for every categorical input dimensioníde Brebisson et al. 2015). The embeddings for the tabular features are concatenated into one tabular embedding.
4.2 TASK-SPECIFIC COMPONENTS
Multi-modal embeddings used the input task specific components, of which there is one per task. Each task specific component is composed of a fully connected layer with h2 hidden units, dropout layer, ReLU activation, and output layer matching the shape of that component's respective task. Each task specific component, regardless of architecture, has no explicit connection any other task specific layer, but does share the same multi-modal embedding: thus the task specific components share the modal encoder.
each task specific component has task specific loss. To learn across all tasks simultaneously, we take the weighted sum of all the losses resulting to form the multi-task loss.
As shown in Figure [
5
EXPERIMENTS
We now report performance on the M3 dataset. Our models achieve high performance levels and represent strong baselines for future work. In particular, we outperform previous state-of-the-an results that only rely on subset of all tasks and modalities.
5.1 SETUP
Table 2: Implementation details for tabular encoder and task specific components.
(a) Tabular input features
(b) Task specific components
Tabular Feature
num values
Embed dim
Task
hidden units(h;)
dropout prob(o:))
ICU unit
8
32
Decomp.
128
0.8
Dbsource
4
32
Mortality (IH)
108
0.85
sex
4
32
Pheno.
512
0.1
age
100
32
Len. of Stay
0.8
height
100
32
Mortality (LT)
64
0.8
weight
40
32
Readm.
128
0.1

---

Under review as a conference paper at ICLR 2021
Encoders. We used an I-layer LSTM with 256 hidden units the time series encoder. Clinical notes were mapped to Word2Vee embeddings (Mikolov et al. 2013). The Text CNN has three 1D kernels of size 2,3 and 4 with 128 filters each. We replaced all missing inputs with zeros.
Task specific components. We have task specific components. Each task specific component has linear layer with h, hidden units, followed by dropout layer with dropout probability o;, ReLU and an output layer. See Table 2b for details.
Preprocessing. The 17 clinical variables used by Harutyunyan et al. 2019) were normalized as in their paper. We imputed missing values during inference, taking the most recent value for the feature before the current time point. If there is most recent value, then we impute based on predefined imputed values. For the additional time series features, we take the raw inputs and impute them in the same manner. Due to computational limits, we truncated each clinical note most 500 tokens and we limit the number of notes we can process per stay to 150. Inputs were pero-pad& as necessary.
Training Every task used the cross entropy loss. To find multitask weights, we used uncertainty weighting (Cipolla al. 2018)) followed by manual search. We train with the Adam optimizer with a learning rate of le-4. and employ early stopping.
Baselines We compare to Harutyunyan et al. (2019). who report performance physiological time series in a multi-task setting using LSTMs and channel-wis. LSTMS, We also compared with Sheikhalishahi al. (2020) who used BiLSTMs for time series data. For the text modality. We compared with Grnarova et al. (2016); Khadanga et al. (2020); Boag et al.] (2018); Sheikhalishahi al. (2020). Additionally, KKadanga al. (2020) report a multi-modal result using time series and text. More details are in Table
5.2
MAIN RESULTS
Table details the performance across every task for various input modalities and in both the single and multitask setting. Having additional modalities improves performance on every task except for length of stay, where it does not significantly change performance. Multitasking further increases performance on many tasks, such in-hospital mortality, readmission, and long term mortality.
Table shows the performance models from different papers on the same tasks. Our models outperform both of our state-of-the-ar baselines that only rely on subset of all tasks and modalities.
Table 3: Results on all tasks by input modalities of time series (TS), clinical notes (Text) to tabular inputs (Tab). Tasks, in order, are Decompensation, In-Hospital Mortality, Phenotyping, Length of Stay, Long Term Mortality, and Readmission. Error bounds were computed by sampling with replacement 6000 samples 1000 times from the test set and computing metrics. Reported value is the median, bounds are based off of the 2.5 and 97.5 percentiles.
Metrics
Decomp. aucpr
Mortality (IH)
Pheno.
Len. of Stay Mortality (LT)
Readm.
Modalities
Num Tasks
aucpr
macro aucroc
aucroc ovr
aucpr
aucroc ovr
TS
ST
.299 1.01
.427 1.053
.788 1.004
.722 1.012
.291 1.027
.551 ..02
TS
MT
.280 1.01
.427 1.053
.77 4.005
.735 1.012
.366 1.036
.500 ..02
TS-Text
ST
.359 1.01
.580
+.052
.794 1.005
.751 4.011
.357 1.030
.700 1.017
TS- Text
MT
.409 +.01
.604 +.05
.773 4.005
.754
414+:037
.696 #.017
TS Text-Tab
ST
.404 :.01
.700
+.051
4.004
.737 2112
.406 4.031
.695 #.017
TS-Text-Tab
MT
.408 0.01
.600 +.05
812
.741 #:011
.4501.033
.708+.016
RELATED WORK
The paper by HHauutyuyan et al. 2019) is the closest work to ours in the literature. They report the performance multi-task models on a dataset derived from MIMIC-III that consists of 4 of our clinical prediction tasks and subset of the physiological time series data that use. Purushotham jet al.] (2017) introduced another benchmark covering tasks such as mortality prediction task, ICD code grouping task, and length-of-stay prediction with different sets of time series features.

---

Under review as a conference paper at ICLR 2021
Models
Decomp.
Mortality (lH)
Pheno.
Len. of Stay Mortality (LT) Readm.
aucpr
aucpr
macro aucrne
aucroc ovr
aucpr
aucroc ovr
Single Modal Time Series
Is-LSTM
0.299
0.427
0.788
0.722
0.291
0.551
CW LSTM
0.326
0.486
0.788
0.774
0.341
0.594
BiLSTM
0.345
0.458
0.782
0.782
0.336
0.656
Single Modal Text
LSTM
NA
NA
0.546
NA
NA 0.563
CNN
0.312
0.517
0.541
0.711
0.269
Multi Modal
TS Text
Is-LSTM + CNN
0.359
0.580
0.794
0.751
0.357
0,700
ew-LSTM CNN
0.387
0.600
0.790
0.760
0.400
0.692
Multi Modal TS + Text + Tab
Is-LSTM CNN Tab
0.404
0.570
0.813
0.737
0.406
0.695
Table 4: Comparing Single modal baselines and increasing modalities. Harutyunyan et al. (2019 use time series LSSMsM()-LSTM) and channelwise LSTMs(cw-LSTMss their work. (Sheikhalisahhi et al. (2020) achieve their best performance with BiLSTMs. For text modality. LSTM text classifier are reported achieve state-of-the- performance on phenotyping task (Grnarova et al. 2016). For all other tasks, the best model we found in our literature search is Text CNN (Khadanga al.) 2020
Boag al.) 2018 Sheikhalishahi et al. 2020). Khadanga al. (2020) reported a TS+Text Multi Modal model IS-LSTM+CNN.
Khadanga al. (2020) are the first to consider the multi-mode setting, specifically clinical notes and time series data. They focus three of our clinical prediction tasks (decompensation. length of stay, and in-hospital mortality), but not perform multi-task learning. Our paper improves over both Harutyunyan et al. (2019) and KKadanga et al. (2020). We create benchmark that combines both multiple modalities (including two modalities not found in Khadanga et al. (2020) and multiple tasks (including two new tasks not present in earlier works).
7 DISCUSSION
We release our benchmark in an easy-to-use open source package. This resource may be used to improve architectures for biomedical multi-modal and multi-task learning, as well as to explore research directions in causality, generative models, fairness, and other areas.
Future Research Directions. Within the field of causal inference, our dataset can be used to study ways of inferring latent causal factors from unstructured modalities such as images. It may also drive experiments in new generative model architectures and ways in which generative models can address practical problems such as missing data inputs or the estimation uncertainties a multi-task regime. Our diverse dataset may offer insight into questions of fairness and privacy, such as on the extent to which medical images and clinical notes reveal sensitive patient information such (e.g., race gender), and on how to address this leakage. We hope that our benchmark will serve as a resource to a broad subset of the machine learning community.
CONCLUSION
In this paper, we argued that developing new machine learning models in healthcare that effectively leverage diverse modalities - such radiology images, clinical notes, and sensor data - and that can simultaneously solve multiple tasks e.g., estimating disease risk, predicting readmission the potential to significantly impact applications in the clinical domain. We proposed the first bench- mark for the healthcare domain that focuses multi-modal and multi-task learning and new models that outperform previous state-of-tha fhf--hhart art methods that only rely on subset of tasks or modalities.
has
Our results highlight the importance of multi-moda and multitask learning in clinical settings. More generally, we envision our work being a general resource that will help accelerate research in apply- ing machine learning to healthcare.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lci Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018.
Willie Boag, Dustin Doss, Tristan Naumann, and Peter Szolovits. What's in a note? unpacking predictive value in clinical note representations. AMIA Summits on Translational Science Pro- ceedings, 2018:26, 2018.
Roberto Cipolla, Yarin Gal, and Alex Kendall. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1 10.1109900p:22218800777 URL http://dx doi. org/10.
11C9/CVPR. 2 00781
Alexandre de Brébisson, Étienne Simon, Alex Auvolat, Pascal Vincent, and Yoshua Bengio. Artifi- cial neural networks applied to taxi destination prediction, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-244220 255. leee, 2009.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sehastian Thrun. Dermatologísttlevel classification of skin cancer with deep neural networks. nature, (4277639):115-111 2017.
Paulina Grnarvv,, Florian Schmidt, Stephanie L Hyland, and Carsten Eickhoff. Neural document embeddings for intensive care patient mortality prediction. arXiv preprint 1xXi::6612.00477, 2016.
Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama. 3 316(22):2402-444 2016.
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpamahi, Geoffrey H Tison, Codie Bour, Mintu Turakhia, and Andrew Y Ng. Cardiologist-level arrhythmii detection and classifica- tion in ambulatory electrocardiograms using deep neural network. Nature medicine, 25(1 (66,, 2019.
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Mul- titask learning and benchmarking with clinical time series data. Scientific Data 6(1):96, 2019. ISSN 2052-4463. doi: 10. 00.1038/411577-09-9011 941597-019-0103-
URL htttps://dm org 10. 1038/
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Alistair EW Johnson, Tom Pollard, Lu Shen, Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Melvin Johnson, Mike Schuster, Quoe V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, al. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351. 2017.
Swaraj Khadanga, Karan Aggarwal, Shafiq Joty, and Jaideep Srivastava. Using clinical notes with time series data for icu management, 2020.

---

Under review as a conference paper at ICLR 2021
Yoon Kim. Convolutional neural networks for sentence classification. 1rXiv:100.58800 2014.
arXiv preprint
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen. Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp- 31 5111-3111 19, 2013.
Moody G. Villarroel M. Clifford G. Silva I. Moody, B. Mimic-iii waveform database, 2020.
Mehdi Moradi, Ali Madani, Yaniv Gur, Yufan Guo, and Tanveer Syeda-Mahmood. Bimodal network architectures for automatic generation of image annotation from text, 2018.
Binh D. Nguyen, Thanh- Toan Do, Binh X. Nguyen, Tuong Do, Erman Tjiputra, and Quang D. Tran. Overcoming data limitation in medical visual question answering, 2019.
Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmark of deep learning models large healthcare mimic datasets, 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprini arXiv: (6666655500 2016.
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn L Ball, et al. Mura: Large dataset for abnormality detec- tion in musculoskeletal radiographs. arXiv preprint arXiv: 1712.06957, 2017.
Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprint arXiv: 0002.22072222 2015.
Sebastian Ruder. An overview of multi-task learning in deep neural networks, 2017.
Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. Benchmarking machine learn- ing models multi-centre eicu critical care dataset. Plos one, 15()))e0235424, 2020.
Shengpu Tang, Parmida Davarmanesh, Yanmeng Song. Danai Koutra, Michael W Sjoding, and Jenna Wiens. Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data. Journal of the American Medical Informatics Association, 10 2020. ISSN 1527-974X. doi: 10. 1093//amia/oocaal... URL /tttps:///ci..om jamia/ocaal:3 ocaal39.
1093/
Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learn- ing in drug discovery and development. Nature Reviews Drug Discovery, 18(6):463-44 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pp. 353-355, Brussels, Belgium, November 2018a. Association for Computational Lin- guisties. doi: 10. 10.18653/11/118-54442 URL https:/ www. acl web. org
wwtp/ottt/aaoooooooooooooo ant ogy/
W18 5446)
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint
2018b.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems, pp. 3266-3266 3666-220, 2019.
10

---

Under review as a conference paper at ICLR 2021
Shirly Wang, Matthew B. A. McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C. Hughes, and Tristan Naumann. Mimic-extract. Proceedings of the ACM Conference on Health, Inference, and Learning, Apr 2020. doi: 10.1145/368555388446 URL http: / //tp doi.
org/10.
3384469)
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 4651-4659, 2016.
Kun-Hsing Yu, Andrew L Beam, and Isaac S Kohane. Artificial intelligence in healthcare. Nature biomedical engineering, 2(10):719-73, 2018.
A APPENDIX
A.1
FULL PERFORMANCE PER TASK
Table 5: Length of stay
AUCROC
AUCROC
TS-ST (ours)
0.722 (0.711, 0.734)
0.692 (0.682, 0.703)
TS-MT (ours)
0.735 (0.723, 0.746)
0.700 (0.690, 0.710)
Ts- Ts-Text-ST (ours)
0.751 (0.740, 0.762)
0.730 (0.719, 0.741)
TS-TexIMMT (ours)
0.754 (0.743, 0.765)
0.734 (0.723, 0.745)
TS- S5--Tet--ab-- Tab-ST (ours)
0.737 (0.725, 0.748)
0.713 (0,702, 0.724)
TS- Text- -ee---TT------- Tab-MT' (ours)
0.741 (0.730, 0.752)
0.719 (0.709, 0.730)
Table Long term mortality
AUCROC
AUCPR
TS-ST (ours)
0.659 (0.640, 0.677)
0.291 (0.266, 0.318)
TS-MT (ours)
0.736 (0. (0.720, 0. 0.753)
0.366 (0.337, 0.396)
TS-TexI-ST (ours)
0.701 (0.683, 0.718)
0.357 (0.328, 0.388)
TS-Text-Mi (ours)
0.760 (0.744, 0.775)
0.414 (0.382, 0.445)
TS- T5--eet--Ta--S (ours)
0.744 (0.728, 0.760)
0.406 (0.375, 0,438)
TS-TexI-Tab-MT (ours)
0,784 (0.768, 0,798)
0,450 (0.417, 0,482)
Table 7: Readmission
AUCROC ovr
AUCROC ovo
TS-ST (ours)
0.551 (0.532, 0.571)
0.532 (0.517, 0.546)
TS-MT (ours)
0.530 (0.510, 0.550)
0.524 (0.508, 0.540)
TS- IS-Text-ST (ours)
0.700 (0.683, 0.717)
0.600 (0.588, 0.612)
TS (ours)
0.696 (0.679, 0.713)
0.603 (0.590, 0.616)
TS- Tab-ST (ours)
0.695 (0.678, 0.712)
0.603 (0.589, 0.617)
TS-Text-Tab-Mi (ours)
0.708 (0.692, 0.725)
0.605 (0.594, 0.617)
A.2 COMPLETE PHOSIGGGI TIME SERIES VARIABLES
11

---

Under review as a conference paper at ICLR 2021
Time series var
Alanine aminotransferase Albumin
Alkaline phosphate
Anion gap
Asparate aminotransserase
Basophils
Bicarbonate
Bilirubin
BBlood urea nitr)gen(BUN)
CO2 (ETCO2; PCO2; etc.)
Calcium
Calcium
lonized
Calcium ionized
Capillary refill rate
Creatinine
Diastolic blood pressure
Eosinophils
Fraction inspired oxygen
Glascow coma scale eye opening
Glascow coma scale motor response Glascow coma scale total
Glascow coma scale verbal response
Glucose
Heart Rate
Hematocrit
Hemoglobin
Lactate
Lactate dehydrogenase
Lactic Lyphh.yye
Magnesium
Mean blood pressure
Mean corpuscular hemoglobin
Mean corpuscular hemoglobin concentration
Mean corpuscular volume Monocytes
Neutrophils
Oxygen saturation
Partial pressure of carbon dioxide
Partial pressure of oxygen
Partial thromboplastin time
Peak inspiratory pressure
Phosphate
Platelets
Positive end-expiratory pressure
Potassium
Prothrombin time
Red blood cell count
Respiratory rate
Sodium
Systolic blood pressure
Temperature
Urine output
Weight
White blood cell count
pH
Table List of all used Time series vars
12

---

Under feview as a cnnterence paper ICLR 2021
1.019 291 #.027 .5$1 1 11.02//55 03.5332222 +.015
1.0016 366 1.036 .53 1 22.022554444 524 1.016
.700 0..7/6000 + 0.1
2.011 .760 1.016/ 4144 + -.002 .696 +.017/ 7/ + 01.013
.744 .744 1.0 406 406 + .695 4.017/6603 7/.603 # 014
aucroc oVr/ovo
1081.016..00::10
Readm.
.701 11.018..33 357 + 2..00
Mortality (LT)
.784 0166
qucroc/aucpp
.659
.736
.850 1. 0230 427 1.053 .788 1. .000//// 837 L.004 .722 1 1L..11//6992 1.011
.735 1.012/ .700 1.010
F.011
.737 +.012/773 # #.011
1.011
Lon. of Stay aucroc ovr/ovo
.51 +. 011.1331003-000
.711 1,011//719
.754 +44
.77 +. 00.005//.5 82 1.005
.794 # 005/ 840 +. 004
.773 +. 0.005/ $25 006
.813 4.00 855 855 #. 000
003
aucroc macro/micn
4.000/854
812
+. 02000 427 1.053
.800 01..1888588 580 +.052
01..18...600 18.66444444 + 05
.87 +. 1..01.77570 0.051
+.05
Mortality (IH)
aucroc/accy aeeepr
600
.858
.800
.887
ST .996 +. 0.000/ 299 1.01
MT 892 1. 003/ .280 +.01
ST .925 + 002/ 359 F.01
MT .926 + 0030000//400 409 2.01
ST .977 + 10..000440 404 +.01
#.01
aucroc/ auept
408::
.40
Decomp.
MT .34 1.0
Table 9: Full Full results with all metrics
Num Tasks
Time Scries- Tex- Tew--TTTTTlaaaaa Tabular
Time Series- Text- Tex--Teeelllele Tabular
Metrics
Time Series- Text
Time Series- Text
Time Series
Time Series
Modalities
13


Reviews:
[{"review_id": "eb3T_pyAqF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There are two objectives for this work:\n1. A new MIMICIII benchmark data that supports multi tasks and multi modalities \n2. A proposed multi-modal multi-task AI model\n\nFor the first objective, the data supports 6 clinical classification tasks (which were proposed before in a earlier MIMICIII benchmark) but also supports 4 different modalities: physiological time series (the standard modality that used in the prior benchmark), clinical notes, baseline data, and waveform (although not being used in the method/experiments). For the second objective, a modality-specific embedding model was applied to each modality independently and then an aggregation of all embeddings was used as an input to a task-specific predictive model. The weighted linear combination of losses over all tasks was used for the multi-task settings. \n\n\n\nI have the following concerns/comments: \n- MIMIC-IV was released long time back and MIMIC-V was just released a few months back. Why the authors benchmarking MIMIC-III and not the recent versions? \n- The authors have mentioned \"We also propose an evaluation framework to benchmark models on this dataset.\". I could not see that in the paper \n- \"We release M3 and our models as an easy-to-use open-source package for the research community\". Where is the link? I can not evaluate \"easy-to-use\" open source package.\n- The number of stays in Table 1, is that when all modalities are available or the number when any modality is available? \n- I hope in the open-source package (which is not provided) if the clinical notes are processed so that it can be used by other researchers\n- There is no clear description about how waveforms are processed.\n- \"we resample time series data with 1 hour\", what would be the case if multiple data points are available within 1 hour? Do the authors average them or take the recent value? \n- For tabular data \"To process the tabular inputs, we learn an embedding table for every categorical input dimension\". why do not learn embedding use all of them instead of learning embedding for each feature independently. The correlation among these features might not be captured. \n- I give the authors a credit of using weighted combination of losses using the uncertainty of each task \n- In the baselines section, the authors mentioned \" We replicate their test set to compare to their results\", what does it mean? What would be the case if the part of the test data in the \"Khadanga et al. (2020)\" is part of the training in this work \n- \"Having additional modalities improves performance on every task except for length of stay,\" The same also applies for in-hospital mortality\n- The presentation of Table 3 could be significantly improve if you start with the entire model (all modalities) and compare it to counter-part single task. Then in a separate table, you perform ablation analysis to see which modality is important. \n- Table 4 is confusing. Maybe another representation could be a figure where in x-axis the metric for the baseline and y-axis is the metric for the proposed model and each do represents one comparison. For example, the first part of the Table could be a dot (model) where y-axis is 0.408 and x-axis is 0.344, which compares the proposed model versus the baseline model. The legend of the figure spells out the task name (and any other info to be added). \n- The numbers in Table 4 do not match, for example the second part of the table shows that the proposed models has 0.417 AUCPR, where does that number come from? It is not in Table 3. \n- The discussion and future work is not quite related to the paper, for example, it was mentioned about isng images, but images is not one of the modalities in the proposed benchmark. Do you plan to add images to he benchmark? Does MIMIC has images? \n- There is no comparison versus other multi-modal multi-task baselines \n- Change \"We proposed he first benchmark\" to \"We proposed the first benchmark\"\n- Change \"Given a patient’s ICU stay of length of stay T hours\" to \"Given a patient’s ICU stay of length T hours\"\n- There is no section about how the hyperparameters were optimized", "extra_scores": {}, "extra_text": {}}, {"review_id": "eFDNKCEZq3f", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper defines a new task for clinical data by combing multi-modal and multi-task settings into one task. It collects a dataset called M3 as the benchmark for the multi-modal and multi-task benchmark in the clinical domain. The dataset has 6 prediction tasks, i.e., in-hospital mortality, decompensation, length of stay, phenotyping, readmission, and long-term mortality, and it has 4 modalities, i.e., physiological time series, clinical notes, tabular data, and waveforms. Specifically, this paper also provides a multi-modal multi-task model where the time series data are encoded by LSTM, clinical notes are encoded by text CNN and tabular data are also encoded by existing methods.  In experiments, the authors conduct an ablation study and compare the proposed method with the method of Harutyunyan et al. and Khadanga et al.\n\nQuality: The overall quality of the paper is marginally below the acceptance threshold. The problem definition, data collection, and model design are reasonable. However, I am concerned about the paper presentation and the experiment design. The authors need to compare the proposed multi-modal multi-task model with state-of-the-art single-modality models.\n\nClarity: The presentation of the paper is easy to follow, but the structure of the presentation may need to be improved.\n\nOriginality: The collected dataset is also new because of the new setting. But in terms of model design, the feature embeddings are learned by existing methods.\n\nSignificance of This Work: The direction of this work is significant and worth being paid attention to. Considering multi-modal multi-task settings in the clinical domain is useful for the development in this area. \n\nPros: \n1. The multi-modal multi-task setting is interesting and important for future related research. This paper provides a new direction for moving machine learning forward in the clinical domain.\n2. Collecting data from MIMIC-III is a reasonable choice for creating a multi-modal multi-task dataset. It provides a solution on how to create a new dataset for clinical prediction task with new settings. \n3. The model design is reasonable. I think the authors choose the right frameworks for dealing with different modalities.\n\nDespite this, I am concerned about the presentation and experimental design of the paper, which are summarized as the cons as follows.\n\nCons: \n1. The paper structure could be improved by incorporating Section 2 “background” and Section 6 “related work” together. The mentioned work in Section 6 is related to “machine learning in the clinical domain” in Section 2. \n2. There are other forms of multi-modal learning in the clinical domain, and the authors should take them into consideration and discuss them in the background. For example, Moradi et al. and Nguyen et al. have some work about the text and image multi-modal learning in the clinical domain.\n3. As for the experiment, Table 3 is useful in showing the model design, but I think the result presentation and experiment design in Table 4 can be improved. The authors could provide how the choices of encoders can influence task performance if there are only two baselines. This can help the readers see what’s the potential of doing research on this dataset and directions.  For example, the method of Harutyunyan et al. only uses the time series modality in Table 4, but it can be used as the encoder to process time series in the proposed model design, and maybe the authors should consider this setting as a baseline. To sum up, I think the comparison experiment in Table 4 is not compelling enough to illustrate the model design. \n4. There are many existing works using single modality data towards the six tasks. The authors should compare those models on a single modality to demonstrate improvement when incorporating a multi-modal multi-task dataset.\n\nSome typos: In the conclusion section, “he first benchmark” should be “the first benchmark”\n\nReferences\nMoradi, M., Madani, A., Gur, Y., Guo, Y., & Syeda-Mahmood, T. (2018, September). Bimodal network architectures for automatic generation of image annotation from text. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 449-456). Springer, Cham.\nNguyen, B. D., Do, T. T., Nguyen, B. X., Do, T., Tjiputra, E., & Tran, Q. D. (2019, October). Overcoming data limitation in medical visual question answering. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 522-530). Springer, Cham.\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "VgPPN9KGOj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "################################ \n\nSummary:\n\nThis paper discusses the Multi-Modal Multi-Task MIMIC-III (M3) dataset and benchmark, which extends previous efforts in this space to provide a benchmark on the MIMIC-III dataset. In particular, this work considers the inclusion of multiple modalities, including time series, clinical notes, ECG waveforms, and tabular input. It also defines six clinical tasks, some of which overlap with existing efforts and others which appear to be new.\n\n################################ \n\nReasons for score:\n\nOverall, I lean toward reject. This appears to be a simple extension of existing work in the area. While in many ways it's a meaningful contribution to this area it also rests at the boundary of the scope for the CFP. Either alone might not be grounds for rejection, but together they likely put this work below the acceptance threshold.\n\n################################ \n\nStrengths:\n\n- Unifying existing work. In looking to the repository (https://github.com/DoubleBlindGithub/M3), it appears that the incorporates existing benchmarks by Harutyutyan et al. This approach of building on existing benchmarks rather than reinventing the wheel seems like a great start toward unifying work in this space.\n\n- Multi-Modal. While prior works note extensibility to include additional modalities, they tend to focus on a single modality and merging multiple modalities in a composable manner may require some additional work. This paves the way to provide additional multi-modal work in this area.\n\n- Timely domain. There has been increasing interest in healthcare due to the current COVID-19 pandemic. The availability of good benchmarks stands to channel that interest and energy into improvements that matter.\n\n################################ \n\nWeaknesses:\n\n- Clinical actionability / best practices. In cited works, particularly Wang et al., there is a presence of a \"gap\" introduced in the tasks to discourage overly easy predictions from the data. This work appears to remove those gaps and thus benchmark results may be unusually high and omit what appears to be a best practice for work in this space. \n\n- Unclear takeaway. The models that are reported support a claim that additional information (in the form of additional modalities) are able to improve performance in general, but do not clarify the importance of multi-task in this setting. Expanding the discussion may help elucidate the insights that should be taken away from this work.\n\n- Missing reference. Tang et al.'s \"Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data\" (https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocaa139/5920826) should be added as related work.\n\n################################ \n\nQuestions:\n\n- Can additional context be provided for this work as it relates to the scope of ICLR?\n\n- Can you clarify the takeaways from the results of this work, or clarify if the paper serves primarily to introduce the benchmark?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has two contributions. A novel benchmark for clinical multi-modal multi-task learning based on the already released MIMIC III and a multi-modal multi-task machine learning model. While the paper does show value in providing a curated benchmark and combining/unifying existing approaches to a timely problem, the reviewers agree that the paper provides insufficient novelty to warrant publication.
Decision: Reject