Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ################################ 

Summary:

This paper discusses the Multi-Modal Multi-Task MIMIC-III (M3) dataset and benchmark, which extends previous efforts in this space to provide a benchmark on the MIMIC-III dataset. In particular, this work considers the inclusion of multiple modalities, including time series, clinical notes, ECG waveforms, and tabular input. It also defines six clinical tasks, some of which overlap with existing efforts and others which appear to be new.

################################ 

Reasons for score:

Overall, I lean toward reject. This appears to be a simple extension of existing work in the area. While in many ways it's a meaningful contribution to this area it also rests at the boundary of the scope for the CFP. Either alone might not be grounds for rejection, but together they likely put this work below the acceptance threshold.

################################ 

Strengths:

- Unifying existing work. In looking to the repository (https://github.com/DoubleBlindGithub/M3), it appears that the incorporates existing benchmarks by Harutyutyan et al. This approach of building on existing benchmarks rather than reinventing the wheel seems like a great start toward unifying work in this space.

- Multi-Modal. While prior works note extensibility to include additional modalities, they tend to focus on a single modality and merging multiple modalities in a composable manner may require some additional work. This paves the way to provide additional multi-modal work in this area.

- Timely domain. There has been increasing interest in healthcare due to the current COVID-19 pandemic. The availability of good benchmarks stands to channel that interest and energy into improvements that matter.

################################ 

Weaknesses:

- Clinical actionability / best practices. In cited works, particularly Wang et al., there is a presence of a "gap" introduced in the tasks to discourage overly easy predictions from the data. This work appears to remove those gaps and thus benchmark results may be unusually high and omit what appears to be a best practice for work in this space. 

- Unclear takeaway. The models that are reported support a claim that additional information (in the form of additional modalities) are able to improve performance in general, but do not clarify the importance of multi-task in this setting. Expanding the discussion may help elucidate the insights that should be taken away from this work.

- Missing reference. Tang et al.'s "Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data" (https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocaa139/5920826) should be added as related work.

################################ 

Questions:

- Can additional context be provided for this work as it relates to the scope of ICLR?

- Can you clarify the takeaways from the results of this work, or clarify if the paper serves primarily to introduce the benchmark?

review_title: Expands Prior Work

final_score: rating: 5: Marginally below acceptance threshold
review_content: There are two objectives for this work:
1. A new MIMICIII benchmark data that supports multi tasks and multi modalities 
2. A proposed multi-modal multi-task AI model

For the first objective, the data supports 6 clinical classification tasks (which were proposed before in a earlier MIMICIII benchmark) but also supports 4 different modalities: physiological time series (the standard modality that used in the prior benchmark), clinical notes, baseline data, and waveform (although not being used in the method/experiments). For the second objective, a modality-specific embedding model was applied to each modality independently and then an aggregation of all embeddings was used as an input to a task-specific predictive model. The weighted linear combination of losses over all tasks was used for the multi-task settings. 



I have the following concerns/comments: 
- MIMIC-IV was released long time back and MIMIC-V was just released a few months back. Why the authors benchmarking MIMIC-III and not the recent versions? 
- The authors have mentioned "We also propose an evaluation framework to benchmark models on this dataset.". I could not see that in the paper 
- "We release M3 and our models as an easy-to-use open-source package for the research community". Where is the link? I can not evaluate "easy-to-use" open source package.
- The number of stays in Table 1, is that when all modalities are available or the number when any modality is available? 
- I hope in the open-source package (which is not provided) if the clinical notes are processed so that it can be used by other researchers
- There is no clear description about how waveforms are processed.
- "we resample time series data with 1 hour", what would be the case if multiple data points are available within 1 hour? Do the authors average them or take the recent value? 
- For tabular data "To process the tabular inputs, we learn an embedding table for every categorical input dimension". why do not learn embedding use all of them instead of learning embedding for each feature independently. The correlation among these features might not be captured. 
- I give the authors a credit of using weighted combination of losses using the uncertainty of each task 
- In the baselines section, the authors mentioned " We replicate their test set to compare to their results", what does it mean? What would be the case if the part of the test data in the "Khadanga et al. (2020)" is part of the training in this work 
- "Having additional modalities improves performance on every task except for length of stay," The same also applies for in-hospital mortality
- The presentation of Table 3 could be significantly improve if you start with the entire model (all modalities) and compare it to counter-part single task. Then in a separate table, you perform ablation analysis to see which modality is important. 
- Table 4 is confusing. Maybe another representation could be a figure where in x-axis the metric for the baseline and y-axis is the metric for the proposed model and each do represents one comparison. For example, the first part of the Table could be a dot (model) where y-axis is 0.408 and x-axis is 0.344, which compares the proposed model versus the baseline model. The legend of the figure spells out the task name (and any other info to be added). 
- The numbers in Table 4 do not match, for example the second part of the table shows that the proposed models has 0.417 AUCPR, where does that number come from? It is not in Table 3. 
- The discussion and future work is not quite related to the paper, for example, it was mentioned about isng images, but images is not one of the modalities in the proposed benchmark. Do you plan to add images to he benchmark? Does MIMIC has images? 
- There is no comparison versus other multi-modal multi-task baselines 
- Change "We proposed he first benchmark" to "We proposed the first benchmark"
- Change "Given a patient’s ICU stay of length of stay T hours" to "Given a patient’s ICU stay of length T hours"
- There is no section about how the hyperparameters were optimized
review_title: bencahmark MIMIC-III data but not robust evaluation for the proposed model 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper defines a new task for clinical data by combing multi-modal and multi-task settings into one task. It collects a dataset called M3 as the benchmark for the multi-modal and multi-task benchmark in the clinical domain. The dataset has 6 prediction tasks, i.e., in-hospital mortality, decompensation, length of stay, phenotyping, readmission, and long-term mortality, and it has 4 modalities, i.e., physiological time series, clinical notes, tabular data, and waveforms. Specifically, this paper also provides a multi-modal multi-task model where the time series data are encoded by LSTM, clinical notes are encoded by text CNN and tabular data are also encoded by existing methods.  In experiments, the authors conduct an ablation study and compare the proposed method with the method of Harutyunyan et al. and Khadanga et al.

Quality: The overall quality of the paper is marginally below the acceptance threshold. The problem definition, data collection, and model design are reasonable. However, I am concerned about the paper presentation and the experiment design. The authors need to compare the proposed multi-modal multi-task model with state-of-the-art single-modality models.

Clarity: The presentation of the paper is easy to follow, but the structure of the presentation may need to be improved.

Originality: The collected dataset is also new because of the new setting. But in terms of model design, the feature embeddings are learned by existing methods.

Significance of This Work: The direction of this work is significant and worth being paid attention to. Considering multi-modal multi-task settings in the clinical domain is useful for the development in this area. 

Pros: 
1. The multi-modal multi-task setting is interesting and important for future related research. This paper provides a new direction for moving machine learning forward in the clinical domain.
2. Collecting data from MIMIC-III is a reasonable choice for creating a multi-modal multi-task dataset. It provides a solution on how to create a new dataset for clinical prediction task with new settings. 
3. The model design is reasonable. I think the authors choose the right frameworks for dealing with different modalities.

Despite this, I am concerned about the presentation and experimental design of the paper, which are summarized as the cons as follows.

Cons: 
1. The paper structure could be improved by incorporating Section 2 “background” and Section 6 “related work” together. The mentioned work in Section 6 is related to “machine learning in the clinical domain” in Section 2. 
2. There are other forms of multi-modal learning in the clinical domain, and the authors should take them into consideration and discuss them in the background. For example, Moradi et al. and Nguyen et al. have some work about the text and image multi-modal learning in the clinical domain.
3. As for the experiment, Table 3 is useful in showing the model design, but I think the result presentation and experiment design in Table 4 can be improved. The authors could provide how the choices of encoders can influence task performance if there are only two baselines. This can help the readers see what’s the potential of doing research on this dataset and directions.  For example, the method of Harutyunyan et al. only uses the time series modality in Table 4, but it can be used as the encoder to process time series in the proposed model design, and maybe the authors should consider this setting as a baseline. To sum up, I think the comparison experiment in Table 4 is not compelling enough to illustrate the model design. 
4. There are many existing works using single modality data towards the six tasks. The authors should compare those models on a single modality to demonstrate improvement when incorporating a multi-modal multi-task dataset.

Some typos: In the conclusion section, “he first benchmark” should be “the first benchmark”

References
Moradi, M., Madani, A., Gur, Y., Guo, Y., & Syeda-Mahmood, T. (2018, September). Bimodal network architectures for automatic generation of image annotation from text. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 449-456). Springer, Cham.
Nguyen, B. D., Do, T. T., Nguyen, B. X., Do, T., Tjiputra, E., & Tran, Q. D. (2019, October). Overcoming data limitation in medical visual question answering. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 522-530). Springer, Cham.

 
review_title: A Work on Clinical Prediction with A New Setting

Meta-review: The paper has two contributions. A novel benchmark for clinical multi-modal multi-task learning based on the already released MIMIC III and a multi-modal multi-task machine learning model. While the paper does show value in providing a curated benchmark and combining/unifying existing approaches to a timely problem, the reviewers agree that the paper provides insufficient novelty to warrant publication.
Decision: Reject