Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors showcase a pipeline intended to standardize and industrialize AI model development and testing for medical time series.

It is very clear that the authors have put in a tremendous amount of work in building their pipeline. As someone who works on ML for healthcare, I appreciate it and look forward to using such pipelines.

As far as the *paper* is concerned, however, I’m not sure if the paper (as it is written) and the venue (ICLR) are a good fit. For papers describing such frameworks, I’d prefer to read about:
1. pipeline design and tradeoffs: why did the team make the decisions they did . For e.g. why did they decide to design the API interface the way they did, why did they decide to offer some ML techniques over others, what are the tradeoffs between standardization and flexibility for using such a pipeline, when would a user use Clairvoyance over starting fresh.
2. benefits of using this pipeline over other pipelines or no pipeline: the authors benchmarked their pipeline’s performance over off-the-shelf ML models for some tasks, which is great! It would also be good to see the benefits of using Clairvoyance in terms of time to setup, time to train, etc.
3. user interviews, feedback, and adoption to demonstrate how quickly new users can learn this framework and the benefits they observe while using it

I look forward to hearing back from the authors and I’m open to changing my score.
review_title: Exciting and useful pipeline; concerned that the paper doesn't do justice to the authors' work

final_score: rating: 6: Marginally above acceptance threshold
review_content: The manuscript introduces and illustrates an end-to-end software pipeline, called Clairvoyance, for medical machine learning on time-series data. The authors must be congratulated for having designed and developed this wonderful resource to accelerate the adoption of these computational techniques in clinical practice as a way to support people’s judgement and decision-making. The manuscript excels in describing and relating its contributions with related work. It has also included a convincing set of experimentation on datasets from three medical environments that are supplementary to each other. My only concerns with this paper are (i) describing and justifying these experiments, their materials (also research ethics), and their processing, evaluation, and statistical significance testing methods to an extent that allows the reader to comprehend these original studies that are now a part of the pipeline release paper (perhaps an appendix or references to separate original studies would do), (ii) release details of the pipeline seem to be missing (e.g., where to get the code, what is the licence of the release, and how are the authors facilitating people adopting the toolkit), and (iii) I am uncertain if ICLR is the right venue for the manuscript to obtain envisioned impacts of the conclusion section of the paper — I would have seen this contribution published in a medical journal instead, and the lacking details related to the item (ii) before make it even harder to assess this paper. However, the submission is excellent, and the program committee should discuss this case further.
review_title: Review of the submission called Clairvoyance: A Pipeline Toolkit for Medical Time Series

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

The authors present a new package aimed at improving the design and validation of pipelines using medical time series data. The pipeline covers many aspects of time series pipelines including pre-processing, prediction, treatment effect estimation, calibration, etc. The package, as depicted in the paper, appears to be very comprehensive and well motivated.

##########################################################################

Rationale for score: 

The primary reason for my recommendation of reject is this paper is ill-suited for a venue such as ICLR. The paper is a descriptive paper for a new software package and it is aimed at the healthcare/informatics sub-community. While I appreciate the considerable effort the authors have clearly put into this software package, I think the paper would have a better chance of reaching it's intended audience in a more focused venue such as the Journal of the American Medical Informatics Association (JAMIA), the Machine Learning for Healthcare Conference (MLHC), or the Journal of Statistical Software (JSS). I also believe that the ICLR format does a disservice to the authors, as 8 pages is not enough room to fully elaborate on their work and the current version is very compressed which makes for difficult reading. Given the space constraints, the paper feels more like an advertisement for the package vs. an elaboration and explanation document. I think there is an impressive amount of work on display here, but I think that ultimately a paper such as this would be better served in a different venue. 
 
##########################################################################

Pros: 

1. The pipeline, as it is proposed in the paper is quite impressive. Moreover, the authors do an excellent job at motivating the need for such a pipeline not only as a tool but as a means of standardization and benchmarking, something that is sorely needed in many healthcare applications of machine learning. 

2. The figures and table 1 do a good job at summarizing the proposed approach and existing alternatives.

##########################################################################

Cons: 

1. There are no empirical evaluations against alternative methods in this paper. At minimum there should be some kind of head to head evaluations against the existing packages.
 
2. I found the vignettes too condensed to be helpful and it's unclear how the proposed pipeline was used to produce the results in the tables. The authors do a good job at setting up the clinical problem, but (likely due to space constraints) it is unclear how the problem reduces to a series of pipeline steps. Having one fully worked example with code (even in the appendix) would greatly help to understand how the proposed pipeline works.

3. Several basic details are missing from the paper. For example, is this a python package? From the code snippets, I assume it is but this is never stated in the paper. If it is a python package, what versions of python is it compatible with, what are the dependencies, is GPU acceleration supported, etc? I was able to find some of this by digging through the included code but these details should be included in the paper.

4. Many acronyms are used but never defined in the text, e.g. CRN and R-MSN from the examples. If, as the authors claim, they would like their package to be used by clinicians and ML practitioners alike, the should define these acronyms in the text to aid the reader. 

5. Are all of these modules complete or are some still in the alpha phase of development?
review_title: This paper provides a new software package to standardize many common tasks associated with clinical time series data, along with some illustrative examples

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Very interesting and timely project. 

Major concern:
For a well-planned model development with large enough dataset, it is recommended to separate the case and controls from the very beginning and apply the pre-processing and imputation on the training dataset only. Feature selection should also be based on the training data alone, which is not the case in this pipeline from my understanding. The way, the pipeline is described, the processing, including imputation and feature selection are performed on the full dataset, which is then passed to the modeling phase. During the modeling phase, where model training and validation and testing will be performed.  

This can be cause of over-fitting since the testing dataset was to some level "seen" before the model testing. This has to be stated in the limitation of the study design. 

Having a team science approach with clinician scientists as part of the team is integral part of the study, which seems that this paper is all about.  



Minor:
Some grammatical /stylistic error. ex: “While issues such as data cleaning, algorithmic fairness, and privacy and heterogeneity have import, they are beyond the scope of our software.”  revise the sentence “have import”.


Finally, I am not a software engineer and will leave that level of evaluation to my colleagues.

review_title: Timely project for healthcare application - some concern about the study design

Meta-review: The paper addresses a pressing problem for applications involving clinical time series and introduce a pipeline that handle many of the issues pertaining to data preprocessing.

An important contribution is the software that makes the processing more seamless, which will, without a doubt, be useful to the community given the need for reproducibility.

The authors have responded suitably to reviewer comments with the main 'leftover criticism' being that such a paper may not be the best fit for ICLR. This isn't a typical paper. However, something that introduces this level of automation and flexibility in handling time series has not been presented at this conference (or other ML conferences) to the best of my knowledge. It seems it could work in conjunction (as opposed to competing) with any new time series models/techniques that may be introduced.
Decision: Accept (Poster)