Clairvoyance: A Pipeline Toolkit for Medical Time Series
Authors: Daniel Jarrett
Jinsung Yoon
Ioana Bica
Zhaozhi Qian
Ari Ercole
Mihaela van der Schaar
Venue: ICLR 2021 Poster (2021)
Keywords: reproducibility
healthcare
medical time series
pipeline toolkit
software

Abstract: Time-series learning is the bread and butter of data-driven *clinical decision support*, and the recent explosion in ML research has demonstrated great potential in various healthcare settings. At the same time, medical time-series problems in the wild are challenging due to their highly *composite* nature: They entail design choices and interactions among components that preprocess data, impute missing values, select features, issue predictions, estimate uncertainty, and interpret models. Despite exponential growth in electronic patient data, there is a remarkable gap between the potential and realized utilization of ML for clinical research and decision support. In particular, orchestrating a real-world project lifecycle poses challenges in engineering (i.e. hard to build), evaluation (i.e. hard to assess), and efficiency (i.e. hard to optimize). Designed to address these issues simultaneously, Clairvoyance proposes a unified, end-to-end, autoML-friendly pipeline that serves as a (i) software toolkit, (ii) empirical standard, and (iii) interface for optimization. Our ultimate goal lies in facilitating transparent and reproducible experimentation with complex inference workflows, providing integrated pathways for (1) personalized prediction, (2) treatment-effect estimation, and (3) information acquisition. Through illustrative examples on real-world data in outpatient, general wards, and intensive-care settings, we illustrate the applicability of the pipeline paradigm on core tasks in the healthcare journey. To the best of our knowledge, Clairvoyance is the first to demonstrate viability of a comprehensive and automatable pipeline for clinical time-series ML.

Published as a conference paper at ICLR 2021
CLAIRVOYANCE:
A PIPELINE TOOLKIT FOR MEDICAL TIME SERIES
Daniel Jarrett"
Jinsung Yoon*
University of Cambridge, UK daniel.jee jaa..jarraee eea.rrretteeee. eeterrtttmaathtt @maths.............. cam ac. uk
Google Cloud AI, Sunnyvale, USA University of California, Los Angeles, USA
j jinsungyoon@googll. com
Ioana Bica
University of Oxford, UK
Zhaozhi Qian
The Alan Turing Institute, UK
University of Cambridge, UK
ioana. bica@eng. wc..acc.a..o
zhaozhi qian@maths. cam ac uk
Ari Ercole
Mihaela van der Schaar
University of Cambridge, UK Cambridge University Hospitals NHS Foun- dation Trust, UK ael 1 cam. ac.uk
University of Cambridge, UK University of California, Los Angeles, USA
The Alan Turing Institute, UK mv 7 2 cam. ac uk
ABSTRACT
Time-series learning is the bread and butter of data-driven clinical decision support, and the recent explosion in ML research has demonstrated great potential in various healtheare settings. At the same time, medical time-series problems in the wild are challenging due to their highly composite nature: They entail design choices and interactions among components that preprocess data, impute missing values, select features, issue predictions, estimate uncertainty, and interpret models. Despite exponential growth in electronic patient data, there a remarkable gap between the potential and realized utilization of ML for clinical research and decision support. In particular, orchestrating a world project lifecycle poses challenges in engi- neering (i.c. hard to build), evaluation (i.e. hard to assess), and efficiency (i.c. hard to optimize). Designed to address these issues simultancously, poses unified, end-to-end, ML-rMM-frrendlly pipeline that Clairvoyance pro- software toolkit, (ii) empirical standard, and (iii) interface for optimization. Our ultimate goal lies in facilitating transparent and reproducible experimentation with complex infer- ence workflows, providing integrated pathways for (1) personalized prediction, (2) treatment-effee estimation, and (3) information acquisition. Through illustrative ex- amples on real-world data in outpatient, general wards, and intensive-care settings, we illustrate the applicability of the pipeline paradigm on core tasks the health- care journey. To the best of our knowledge, Clairvoyance is the first to demonstrate viability of comprehensive and automatable pipeline for clinical time-series MI.
Python Software http:///////.//////////o/o/ooaaa
INTRODUCTION
With the increasing availability and accessibility of electronic patient records, machine learning for clinical decision support has made great strides offering actionable predictive models for real-world questions S 9]. In particular, plethora of methods-baat based research has focused on addressing specific problems along different stages of the clinical data science pipeline, including preprocessing patient data [10, imputing missing measurements [6) issuing diagnoses and prognoses of discases and biomarkers [17]
Inference over time series is ubiquitous in medical problems 177
estimating the effects of different treatments [
26-3 31, optimizing measurements
332 36 capturing
Authors contributed equally

---

Published as a conference paper at ICLR 2021
uncertainty [37 411 and interpreting learned models 42 46 On the other hand, these component tasks are often formulated, solved, and implemented as mathematical problems (on their own), result- ing in stylized range of methods that may not acknowledge the complexities and interdependencies within the real-world clinical MI. project lifecycle (as a composite). This leads to an often punishing translational barrier between state-of-the-ar MI techniques and any actual patient benefit that could be realized from their intended application towards clinical research and decision support [47)51).
Three Challenges To bridge this gap, we argue for more comprehensive, systematic approach to de- velopment, validation, and clinical utilization. Specifically, due to the number of moving pieces, man- aging real-world clinical time-series inference workflows is challenging due the following concerns:
First and foremost, the engineering problem is that building complex inference procedures involves significant investment: Over 95% of work in typical mature project is consumed by software tech- nicals, and <5% addressing real scientific questions [52]. As a clinician or healtheare practitioner, however, few resources are available for easily developing and validating complete workflows. What is desired is a simple, consistent development and validation workflow that encapsulates all major aspects of clinical time-series ML-froo initial data preprocessing all the way the end.
Second, the evaluation problem is that the performance of any component depends on its context; for instance, the accuracy of prediction model is intimately tied to the data imputation method that pre- cedes it [3 [14]. As an ML researcher, however, current empirical practices typically examine the merits of each component individually, with surrounding steps configured as convenient for ensuring "all else equal" conditions for assessing performance. What is desired is a structured, realistic, and reproducible method of comparing techniques that honestly reflects interdependencies in the gestalt.
Lastly, the efficiency problem is that sophisticated designs tend to be resource-intensive to optimize, and state-of-the-arr deep learning approaches require many knobs to be tuned. As clinical ML practitioner alike, this computational difficulty may be compounded by pipeline combinations and the potential presence of temporal distribution shifts in time-series datasets (53). What desired is a platform on which the process of pipeline configuration and hyperparameter optimization can be automated and through which new optimization algorithms to that effect may be built and tested.
Contributions We tackle all three issues simultaneously. The Clairvoyance package a unified, end-to-end, -frindly pipeline for medical time series. (i) As a software toolkir, it enables development through single unified interface: Modular and composable structures facilitate rapid experimentation and deployment by clinical practitioners, as well as simplifying collaboration and code-sharing. (ii) As an empirical standard, it serves as a complete experimental benchmarking envi- ronment: Standardized, end-to-end pipelines provide realistic and systematic context for evaluating novelties within individual component designs, ensuring that comparisons are fair, transparent, and reproducible. (iii) Finally, as an interface for optimization over the pipeline abstraction, Clairvoyance enables leveraging and developing algorithms for automatic pipeline configuration and stepwise selection, accounting for interdependencies among components, hyperparameters, and time steps. Through illustrative examples on real-world medical datasets, we highlight the applicability of the proposed paradigm within personalized prediction, personalized treatment planning, and personalized monitoring. To the best of our knowledge, Clairvoyance is the first coherent effort to demonstrate viability of comprehensive, structured, and automatable pipeline for clinical time-series learning.
Static Data pgs. gender ctc)
Temporal mormenss
clairvoyance
Palint Journay
Proprocesing
What potlictt snicemec the moot likaly?
ICU Ad Miliiooo, Kovullly Destaarg
Imputation
Personaieed Prodiction
Festure Selection
Ialagntedl
Penstaalzed Treatment Planning
trastevasts ingrove extcessc?
Mechal soclp-
cuza Evartlaao, Medoutien
Information socumulated over
Integeetstion
Uncertaitty
Specifs metiocll koll, bich mecaorro rako,
Adarission
Carrest
1ime
Caliboriio
Penoutlizaal Monitoring
Figure 1: Clairvoyance and the Patient Journey. The healthcare lifecycle revolves around asking (1) what out- comes are most likely, (2) which treatments may best improve them, and (3) when taking additional measurements is most informative. Utilizing both static and temporal data, Clairvoyance provides corresponding pathways for personalized prediction of outcomes, personalized estimation of treatment-effictss, and personalized monitoring.
2

---

Published as a conference paper at ICLR 2021
2 THE CLAIRVOYANCE PIPELINE
The Patient Journey Consider the typical patient's 's interactions with the healthcare system. Their healthcare lifecycle revolves tightly around (1) forecasting outcomes of interest (i.e. the prediction problem), (2) selecting appropriate interventions (i.e. the treatment effects problem), and (3) arranging followup monitoring (i.e. the active sensing problem). Each of these undertakings involve the full complexity of preparing, modeling, optimizing, and drawing conclusions from clinical time series. Clairvoyance provides model pathways for these core tasks in the patient journey (see Figure ! integrated into single pipeline from start to finish (see Figure 2 Formally, these pathways include:
Predictions Path. Let {(s, XI.T.) )) denote any medical time-series dataset, where s,, is the vector of static features for the n-th patient, and XIT,, - {{x,t)]== Tr, is the vector sequence of temporal features. One-shot problems seek to predict vector of labels yn from (Sn; I., ): e.g. prediction of mortality or discharge, where yn E {0, 1}. Online problems predict some target vector Yn,e from (s,, Xn,1:t) at every time step: e.g. -hseep-hhead prediction of biomarkers yn, S Xn,0+T-
Treatment Effects Path. For individualized rreament-effeer estimation 26:31, we additionally identify interventional actions and S Xn,t at each time step (e.g. the choices and dosages of pre- scribed medication), well corresponding measurable outcomes Yn.; S Xn,t+r. The learning problem now consists in quantifying the (factual or counterfactual) potential outcomes Yn,t+r that would result from any specific sequence of interventions and patient covariates ( Xn,1:t; 1,,1:))-
Active Sensing Path. In addition to mapping (already-mentt measured) covariates to targets, the very decision of what (and when) measure is also important under resource constraints. In medical settings, active sensing deals with balancing this trade-off between information gain and acquisition costs (3). With reference some downstream task (e.g. predicting yn yn,t+1), the aim is select a subset of covariates Kn,; at each & to maximize the (net) benefit of observing (In,t.kkkkK..
As Software Toolkit Engineering complete medical time-series workflows is hard. The primary barrier to collaborative research between ML and medicine seldom lies in any particular algorithm. Instead, the difficulty is operational 66 48 54 ie. in coordinating the entire data science process, from handling missing/irregularly sampled patient data all the way to validation on different popu-
Predictions
Statie Imputation
LSTM/ GRU
Interpretation
TCN / WaveNer
Fest. Importance
Pipeline Input
Mean Median
Attention Mech.
Feat. Additivity
Static Tatte
MissForest
INVASE
Temporal
MICE
Pipeline Output
GAIN
Treatment Eff.
Predictions /Messurenents
CRN
Uncertainty
Preprocessing
Uncertainnies and Intervals
Ensemble
Normalization
Temporal
T-GANITE
MC Dropout
Fearure / Temp- oral Inportances
L-hot Encoding
Imputation
Fext. Embedding
Moan/ /Medim
...
...
Performance Metrics
Outiier Detection
Interpolation
Active Sensing
Calibration
M-RNN
ASAC
Platt's
T-GAIN
Randomice
Isotunic
DeepSensing
None
PSC
<g. FLASH. SKL, etc.
Pipelines x Models x Timestops
4g RPS, DKL, etc.
SMS
Optimization Interface
Figure 2: Clairvoyance Pipeline Overview. (Dashed) purple cells denote pipeline inputs/outputs, and (solid) gray cells denote pipeline components. Orange options give main pathway models, and blue options give surrounding components. (Solid) gray arrows indicates pipeline workflow, and (dashed) orange the optimization interface.
3

---

Published as a conference paper at ICLR 2021
"Confiigree Data Prepracessing" preprocessing PipelIneComposer(
"CConfigure Pathway Model
prediction nodell = Prediction(model_nnnee par paraneter_ dicttsr.d w.)
FilterNegatiiee(... Onello:Encader(.... Nornalizer(.),, -))
"*Load Datasets""
"Conffigure Problen Specification"
datantrain, data_test Dataloader,,oad
speci fication ( "roblen.clas==-onlinne mex_seq_len=24,
static_dira'... temporal_dirs'''s -.),
Dataloader,, wawaee.ooassaaticciirr..a.... temporal.dir=".")
venetiltvvt treatment-None, window'4, ..)
"Execute Pipeline" for component [preprocessing,
'"Configure Data Imputation"
imputation Pipellineoopposer
specification,
Inputecion(tyye""staat model_nane= Imputation(tyye==terppo model_name="..
inputation,
festura_selectionn:
data_train component, transtrrrnnooomttata traaaa)
"Couriguree Feature Selection""
data_test compenent. twwnssooonnaaaaaaa.ooooo test)
feature_stlection PigelineConppser(( FeatureSelection (typesttttttty""ssattii model_nane="".. (tarrreeeeeecootteeeeeeeeeee
prediction_nodal
wtt.rnnneee....taata.tttt..... fitt(dataaattrain -.)
model_name="...
test_outout prrplittoonnoeel..eeetttttee
...)
Figure 3: Illustrative Usage. A prototypical structure of API calls for constructing a prediction pathway model. Clairvoyance modularized to abide by established firtreicsorm/predicc design patterns. (Green) ellipses de- note additional configuration; further modules (treatments, sensing, uncertainty, etc.) expose similar interfaces.
lations 4 (5-60).
Clairvoyance gives a single unified roof under which clinicians and researchers alike can readily address such common issues-wite wwith the only requirement that the data conform to the standard EAV open schema for clinical records (i.e. patient key, timestamp, parameter, and value).
Under a simple, consistent API, Clairvoyance encapsulates all major steps of time-series modeling. including (a.) loading and (b.) preprocessing patient records, (c.) defining the learning problem, handling missing or irregular samples in both (d.) static and (e.) temporal contexts, (f.) conducting feature selection, (g.) fitting prediction models, performing (h.) calibration and (i.) uncertainty estimation of model outputs, (j.) applying global or sestance-wise methods for interpreting learned models, (k.) computing evaluation metrics, and (1.) visualizing results. Figure shows a high-level overview of major components in the pipeline, and Figure 3 shows an illustrative example of usage.
All component modules are designed around the established fi-ttransform-pree the modeling workflow is based around single chain of API calls. In this manner, each stage in the pipeline is extensible with little effort: Novel techniques developed for specific purposes (e.g. new state-of-thee tfte-off-hhe-rt imputation method) can be seamlessly integrated via simple wrappers (see Appendix forrr example of how this can be done for any existing method, e.g. from sklearn). This stepwise composability aims to facilitate rapid experimentation and deployment for research, as well as simpli- fying collaboration and code-sharing. Package docummnntation/rutoriilss give further software details.
paradigms, and
As an Empirical Standard Evaluating any algorithm depends on its context. For instance, how well proposed classifier ultimately performs invariably coupled with the upstream feature-selcction method it is paired with 44. Likewise, the accuracy of a state-of- th-t-e-rrt imputation method cannot be assessed on its own: With respect to different downstream prediction models, more sophisticated imputation may actually yield inferior performance relative to simpler techniques [3] 14 especially
components are not jointly optimized [15. While current research practices typically seek to isolate individual gains through "all-else-equa" equal" configurations in benchmarking experiments, the degree of actual overlap in pipeline configurations across studies is lacking: There is often little commonality in the datasets used, preprocessing done, problem types, model classes, and prediction endpoints. This dearth of empirical standardization may not optimally promote practical dascssmeen/rqproouubbbltyy and may obscure/entangle
progress. (Tables 7 in Appendix give a more detailed illustration).
Clairvoyance aims to serve as a structured evaluation framework to provide such an empirical standard. After all, in order to be relevant from real-world medical standpoint, assessment of any single proposed component (e.g. a novel ICU mortality predictor) can-and should-be contextualized in the entire end-to-end workflow as whole. Together, the 'problem-maker', "pipeline-composer" and all the pipeline component modules aim to simplify the process of specifying, benchmarking. and (self- Joocumenting full-fleed fledged experimental setups for each use case. At the end of the day, while results from external validation of is often heterogeneous 2 59 61] improving transparency and
reproducibility greatly facilitates code re-use and independent verification [54] 56 57]. Just as the "environment" abstraction in OpenAl Gym does for reinforcement learning, the "pipeline" abstraction Clairvoyance seeks to promote accessibility and fair comparison as pertains medical time-series.
4

---

Published as a conference paper at ICLR 2021
(a) Example: SASH 'decomposed' as SMS (fulfilled he- (b) Example: SPSC "decomposed" as PSC (fulfilled he- re by DKL. followed by combiner (stacking ensemble): re by SKI.) followed by SMS (fulfilled here by DKI.):
s stepwise, se_ wtwphhaappptwayy.oomm pathway_nodels = 0
pipeline_ = [list_of_stte _ttttiicc______i__________ stt.iosttttccccitttttiiiooo May 1.. ... list_ htt.oo___attaa_csssssee
"Optinize Each Class"
for klass
*PSC Optimization"
sms_agent (neepiiseeethhoosdddd
psc_agent Componentoiiseerrtthoie
klass, data_train, netric)
pipel appeline_llasee data_train, test, netric)
models, scores ams.. pon..oottiiiiinunnnnerrsstt
components, score psc_agennt www.appt.ottmiieennntteeeesss
sus nodel Step:iseeisennileesssll. scores)
pathway data_train data_test test htt.ppathayycclssadddddooo
"Ensenble Over Classes"
SSS Optimization"
for model in hatpwiss_pachayyrooee for step in stepwise. se, nodel:
sms_agent = Stepwisn (method==""
pathway_classs data_train, netric) nodels, scores # wtw.gnnntootiiiiennnnttteeee
step.Load_jndee((ee Saapppge__ppaaaaeeee pattpppthsttpppeedeee nd)e
pathway_ model Stacki
pathway_ _oddll = Stepwisnnnsembl (modees (models, scores) pathway boddel.lodd_nodee (..) test_output pathway_nade hawpaay_ndrl.ppeeeeee (datttcttcaatt.esss
pathway_nodel. trainn, ..) test_putput mothway_nodel.ppe tftts
Figure 4: Optimization Interface. Example code using the optimization interface to conduct stepwise (i.c. across time steps) and componentwise (i.c. across the pipeline) configuration. Each interface is implementable by any choice of newlexisting algorithms. The DKL implementation of SMS is provided for use the Section examples.
As an Optimization Interface Especially in cross-disciplinary clinical research-ang and during initial stages of experimentation automated optimization may alleviate potential scarcity of exper- tise in the specifies of design and tuning. The Clairvoyance pipe- line abstraction serves as a software interface for optimization algorithms-throwe
algeritiaa
CASH
pipelines
PSC
over
which new/existing techniques can be ap- plied, developed, and tested in a more systematic, realistic setting. In particular, by focusing on the temporal aspect of medical time series, this adds a new dimension to classes of autoML problems,
algenitma
SASH
pipdiros
DVST
Figure 5: Degrees of Optimizations. Clairvoyance allows optimizing over algorithms, pipelines, and time steps.
Briefly (see Figure 5). consider the standard task of hyperparam- eter optimization (for a given model) (62] By optimizing over classes of algorithmss, the combined algorithm selection and hyperparameter optimization ("CASH") problem 663)65) has been approached in healthcare settings by methods such as progressive sampling. filtering, and fine-tuning (50 66). By further optimizing over combinations of pipeline components, the pipeline selection and configuration ("PSC") problem 67 has also been tackled in clinical model-
ing via such techniques as fast linear search ("FLASH")
68 and structured kernel learning ("SKL") [67 [69] Now, what bears further emphasis is that for clinical time series, the temporal dimension critical due to the potential for temporal distribution shifts within time-series data-a common phenomenon in the medical setting (we refer to [53] 70, 71 for additional background). Precisely to account for such temporal settings, the stepwise model selection ("SMS") problem [7]] has recently been approached by such methods as relaxed parameter sharing ("RPS") (53) well deep ker- nel learning ("DKL.") 71 (22. Further, what the pipeline interface also does is to naturally allow extending this define the stepwise algorithm selection and hyperparameter optimization ("SASH") problem, or even-in the most general case-the stepwise pipeline selection and configuration ("SPSC") problem. Although these latter two are new-and clearly hard-problems (with no existing solutions), Figure 4 shows simple examples of how the interface allows minimally adapting the SMS and PSC sub-problems (which do have existing solutions) to form feasible (approximate) solutions.
Two distinctions are due: First, Clairvoyance is pipeline toolkit, not an autoML toolkit. It is not our goal to (re- jimplement newlexisting optimization algorithms-whic abound literature. Rather, the standardized interface is precisely what enables existing implementations to be plugged in, as well as allowing new autoML techniques to be developed and validated within a realistic medical pipeline. All that is required, is for the optimizing agent to expose an appropriate 'optimize' method given candidate components, and for such candidates to expose a gett-hpprrprameter-sppe no less importantly-wwe must emphasize that we not advocating removing human oversight from the healthcare loop. Rather, the pipeline simply encourages systematizing the initial development stages in clinical MI, which stands to benefit from existing literature on efficient al techniques.
method. Second but
5

---

Published as a conference paper at ICLR 2021
Design Principles
Our philosophy is based on the authors' experience in prototyping and developing real-world collaborative projects in clinical time-series. * Pipeline First, Models Second: Our first emphasis is on reproducibility: The process of engineering and evaluating complete medical time-series workflows needs to be clear and transparent. Concretely, this manifests in the striet "separation of concerns" enforced by the high- level API of each component module along the pipeline (see e-g. Figure 3 With the "problem-maket and "problem-composer as first-class objects, the central abstraction here is the pipeline itself, while the intricacies and configurations of individual model choices (c.g. a specific deep learning temporal imputation method) are limited to within each component module. Be Minimal and Unintrusive: Our second emphasis is on standardization: While workflow development needs to be unified and systematic, learning to use the framework should be intuitive as well. Concretely, this manifests in the API's adherence to the existing and popular 'fi-transform-prreide paradigm (see e.g. sklearn) in all component modulcs-bont "along' the pipeline steps, well as 'across' the pathways that define the patient's 's healthcare lifecycle (see Figure 2). This enables easy adoption and rapid prototyping- quuaiiiinnguulittsssss that are paramount given the degree of collaborative research and cross disciplinary code-sharing required in healthcare- Encourage Extension: Our third emphasis is on extensibility: Given that novel methods are proposed in the ML community every day, the pipeline components should be casily extensible to incorporate new algorithms. Concretely, this manifests in the encapsulated design for models within each component module: Specifically, in order integrate a new component method (c.g. from another rescarcher's code, or from an external package) into the framework, all that is required is a simple wrapper class that implements the 'fit', 'predict', and gee-byypeparameeer-spacc methods; likewise, for an optimization agent (see subsection on optimization interface below), all that is required is to expose an 'optimize" method.
research.
Worked Examples
For discussion of the choice of built-in techniques to include with the initial release, see Appendix C Appendix
[ gives worked example of using the Clairvoyance pipeline train and use model in standard setting (for this, we use the predictions pathway). Appendix[[]iive a worked example of using the optimization interface to perform stepwise model selection (for this, we use the treatment effects pathway for variety). Appendix [ gives model/algorithm that is not already implemented in the current version of Clairvoyance. Finally, the software repository contains Jupyter notebooks and top-level API code with examples of pathways and optimization.
example of how generic wrapper can be written for integrating an external
3 RELATED WORK
Clairvoyance is a pipeline toolkit for medical time-series machine learning research and clinical decision support. As such, this broad undertaking lies the intersection of three concurrent domains of work: Time-series software development, healthcare joulmmy modeling, and automated learning.
Time-series Software First and foremost, Clairvoyance a software toolkit, Focusing on challenges common to clinical time-series modeling, it is primarily differentiated by the breadth and fiexibility of the pipeline. While there exists a variety of sophisticated time-series packages for different purposes, they typically concentrate implementing collections of algorithms and estimators for specific types of problems, such as classification [79]. forecasting [77), feature extraction (76), reductions between tasks (78). or integrating segmentation and transforms with estimators 75). By contrast, our focus is orthogonal: Clairvoyance aims at end-to-end development along the entire inference workflow, including pathways and pipeline components important medical problems (see Table I.. Again indeed-if so desired, and as mentioned above- algorithms from [73779] can be integrated into Clairvoyance workflows through the usual fit-transfform-rree preeiictt interface, with little hassle.
Healthcare L Lifecycle For specific use cases, clearly a plethora of research exists in support of issuing diagnoses 24]. prognostic modeling [17-21], treatment-effect estimation [2) optimizing measurements 36), among much more. The key proposition that Clairvoyance ad advances is the underlying commonality across these seemingly disparate problems: It abstracts and integrates along the time-series inference workflow, across outpatient, general wards, and intensive-care environments, and-above all-stongst a patient's journey of interactions through the healthcare system that call for decision support in predictions, treatments, and monitoring (Figure Ip. Now, it also important to state what Clairvoyance is not: It is not an exhaustive list of algorithms; the pipeline includes a collection of popular components, and provides standardized interface for extension. It is also not solution considerations: While issues such as data cleaning, algorithmic fairness, and privacy and heterogeneity are important, they are beyond the scope of our software.
to

---

Published as a conference paper at ICLR 2021
Table Clairvoyance and Comparable Software." "Note that vernacularly, "pipelining" simply refers to the proce- dural workflow (i.e. from inputs training. cross-validation, outputs, and evaluation); existing packages focus on implementing algorithms for prediction models alone, with minimal preprocessing In contrast, Clairvoyance pro- vides support along the data science pipeline, and across different healthcare pathways requiring decision support.
cesium
tslearn
seglearn
ts csfresh
pys t0
sktimm 78
pyts 79
Clairvoyance
73
74
75
76
Preprocessing
Temporal Imputation
Feature Selection
x
Predictions
Static Features
x
X
x
Online Targets
/
Predictions
Treatment Effects
2
X
X
Active Sensing
X
X
X
3441
Interp.
Feat. Importance
x
X
Feat. Additivity
x
x
Instance wise
x
Uncertainty
X
x
x
x
x
x
x
Calihration
X
x
x
x
x
x
x
End-to-End Pipelining"
x
X
X
x
/
Optimization Interface
x
x
Automated Learning Finally, tangentially related is the rich body of work on autoML for hyper-
parameter optimization [62]. algoritmm/pipeline configuration [63 65 67 and stepwise selection [71], as well as specific work for healthcare data [50] 53 666 68 77 71). In complement these threads of research, the Clairvoyance pipeline interface enables if so lessred-leverraingg existing implementations, validating novel ones in efficiently accounting for the temporal dimension.
ILLUSTRATIVE EXAMPLES
Recall the patient's journey of interactions within the healthcare system (Fi this section, our goal is to illustrate key usage scenarios for Clairvoyance in this journey-for personalized (1) predic- tion, (2) treatment, and (3) monitoring-in outpatient, general wards, and intensive-care environments.
Specifically, implicit in all examples is our proposition that: (i) as software toolkit, constructing an end-to-end solution to each problem is easy, systematic, and self-documenting: (ii) as an empirical standard, evaluating collections of models by varying single component ensures that comparisons are standardized, explicit, and reproducible; and (iii) as an optimization interface, the flexibility of selecting over the temporal dimension-ing and of
out interesting research avenue.
Medical Environments Our choices of time-series environments are made to reflect the heteroge- neity of realistic cases envisioned for Clairvoyance. For the outpatient setting, consider a cohort of patients enrolled in the UK Cystic Fibrosis Registry (CYSTIC) [80), which records longitudinal follow-up data for 5,800 individuals with the disease. On the registry, individuals
Medical Environment
Outpatient
General Wards
Intensive Care
Dataset
UKCF [80
WARDS [T
MIMIC [22
Duration of Trajectories
Avg. ~-5.3 years (4-6-7 years)
Avg. r9911 days
Avg. 855.4 hours (27-47-91 hours) Per hour
Variance (25%-50%%-75%
(6-9-15 days)
Frequency of Measurements
Per 6 months
Per 4 hours
Different Types of Static and Temporal Features Dimensionality of Features
Demo., Comorbidities, Infections, Treatments
Admiss. Stats, Vital Signs, Lab Tests static, temporal
Demo. Vital Signs, Lab Tests, Medications static, temporal
11 static, 79 temporal
Number of Samples Endpoints (cf. Predictions)
55,800 patients
~6,300 patients Admission to ICU
230 patients Mechanical Ventilation
FEV1 Result
Class label Imbalance
(Continuous-valued)
~5.0%%to--55.0.
~~3.8% 63..2%
Table Medical Environments. consider the range of settings, incl. outpatient, general wards, and ICU data.
7

---

Published as a conference paper at ICLR 2021
chronic patients monitored over infrequent visits, and for which long-term decline is generally expected. For the general wards setting, we consider a cohort of ~,,00 patients hospitalized in the general medicine floor in the Ronald Reagan Medical Center (WARDS) 88] In contrast, here the population of patients presents with a wide variety of conditions and diagnoses (1,600+ ICD-9 codes), and patients are monitored more frequently. The data is highly non-stationary: on the hospital floor, deterioration is an unexpected event. For the intensive-care setting, we consider ~23, 100 individuals from the Medical Information Mart for Intensive Care (MIMIC) that more or less "anything Setthingg-a----e -aapha"p""", and physiological data streams for each patient are recorded extremely frequently. Varying across the set of environments are such characteristies as the average durations of patient trajectories, the types of static and longitudinal features recorded, their frequencies of measurement, and their patterns and rates of missingness (Table ?? presents some brief statistics).
Here, the setting is virtually
Example 1 (Lung Function in Cystic Fibrosis Patients) The most common genetic disease in Cau- casian populations is cystic fibrosis [83] which entails various forms of dysfunction in respiratory and gastrointestinal systems, chiefly resulting in progressive lung damage and recurrent respiratory infections requiring antibiotics-an in severe cases may require hospitalization and even mechanical ventilation in an ICU (see Example 2) [84]]]]]] [8). While classical risk scores and survival models utilize only a fraction of up-to-datt measurements, recent work has leveraged deep learning to incorporate greater extents of longitudinal biomarkers, comorbidities, and other risk factors [86]. An essential barometer for anticipating the occurrence of respiratory failures is the gauge of lung function by forced expiratory volume (FEVI): Accurate prediction yields an important tool for assessing severity of patient's 's disease, describing its onset/progression, and as an input to treatment decisions [85] 87
This is an archetypical rolling-window time-series problem for Clairvoyance's predictions pathway, Consider the models in Table 3 (i) As a clinical professional, it goes without saying that building the pipeline for each or extending additional models through wrappers has low barrier to entry (see Figure wtwte/ooiiao//dduueetttttoooooii
(ii) As an ML researcher, one can rest assured that such compar- isons are expressly standardized: Here, all results are explicitly from same pipeline using min-max normalized features, GAIN for static missing values, M-RNN for temporal imputation, no feature selection, and each model class shown. (iii) Lastly, to highlight the utility of the interface for selection over time, the final row presents results of approaching SASH using the example method of Figure 4 [)), and-for fair comparison- with the pipeline kept constant. This simple approach already yields some gains in performance, laying precedent and the pipeline infrastructure for further research.
Dataset (Label)
UKCF (FEV1 Result)
WARDS (Admission to ICU)
MIMIC (Mech. Ventilation)
Evaluation
RMSE
MAE
AUC
APR
AUC
APR
Attention
(N/A)
(N/A)
0.888 + 0.016
0.551 = 0.024
(N/A)
(N/A)
RNN-GRU
0.064 1 0.001
0.035 1 0.001
0.865 1 0.010
0,487 : 0.048
0.898 1 0.001
0.774 1 0.002
RNN-LSTM
0.062 + 0.001
0.033 + 0.001
0.841 + 0.014
0.412 = 0.032
0.901 + 0.001
0.776 = 0.002
Temporal CNN
0.120 1 0.004
0.096 + 0.003
0.826 1 0.020
0.319 = 0.048
0.884 1 0.004
0.749 1 0.007
Transformer
0.081 + 0.002
0.050 + 0.002
0.846 1 0.006
0.472 = 0.045
0.889 1 0.002
0.761 +:200 0.004
Vanilla RNN
0.070 0.001
0.043 0.001
0.794 + 0.018
0.277 = 0.063
0.898 + 0.001
0.771 + 0.002
SASH
0.059 + 0.001
0.030 + 0.001
0.891 + 0.011
0.557 = 0.031
0.917 + 0.006
0.809 + 0.013
Table 3: Predictions Pathway Example. In addition (online) 6-month ahead predictions of FEVI in UKCF, we also test (onc-shot) predictions of admission to ICU after 48 hours on the floor in WARDS, and (online) 4-hours ahead predictions of the need for mechanical ventilation in MIMIC (these are extended to treatment and sensing problems below). As the WARDS prediction is one-shot, what is denoted 'SASH' for that excludes the SMS en- sembling step. Note that the canonical atlention mechanism does not permit (variable-length) online predictions.
Example 2 (Mechanical Ventilation on Intensive Care) Mechanical ventilation is an invasive, pa- inful, and extremely unpleasant therapy that requires induction of artificial coma, and carries a high risk of mortality [8]. It is also expensive, with typical ICU ventilator admission >$30,000 [89] To the patient, the need for mechanical ventilation-dae due to evidence of respiratory/ventilttoyy failure is by itself an adverse outcome, and is unacceptable to some, even if it means they will not survive. is possible that alternative strategies employed earlier may alleviate the need for ventilation, such as high flow oxygen, non-invasive ventilation, or or-in this example- example-appropriatt appropriate use of antibiotics [88] Now, little is known about optimal timing of courses of antibioties; in most cases a routine number of days is simply chosen when blood is typically sterile after first dose. On the one hand, there is a clear biologically plausible mechanism for incompletely treated infection to lead to longer periods of
8

---

Published as a conference paper at ICLR 2021
critical care, esp. requiring ventilation. On the other hand, antibiotic stewardship is crucial: Over- use of broad spectrum antibiotics leads to resistance, and is by itself a global health emergency [90].
This is an archetypical problem for the treatment effects pathway. Table 4 shows the performance of the two sfate-of-the-ar models for estimating effects of treatment decisions over time while adjusting for time-dependent confounding- that is, since actions taken in the data may depend on time-varying variables related to the outcome of interest 30 31I We refrain from belaboring points (i), (ii), (iii)
above but their merits should be clear. From the patient's 's perspective, accurate estimation of the effect of treatment decisions on the risk of ventilation may assist them and their carers in achieving optimal shared decision- making about the care that they would like to receive. From the hospital's 's perspective. many ICUs around the world operate at ~ 100% bed occupancy, and delayed admission is typically an independent predictor of mortality (91| therefore accurate estimation of the need for escalation or continued ICU ventilation is logistically important for resource planning and minimization of delays.
Time Horizon
Estimating
Day Ahead
Estimating 2 Duys Ahead
Estimating 3 Days Ahead
Evaluation
AUC
APR
AUC
APR
AUC
APR
RMSN
0.860
0.005
0.889 = 0.007
0.790 + 0.004
0.883 + 0.003
0.726
0.015
0.852 0.009 0.869 + 0.007
CRN
0.865
0.003
0.892 = 0.004
0.783 = 0.009
0.872 + 0.013
0.767 0.010
SASH
0.871
0.007
0.902 0.005
0.792 + 0.003
0.885 + 0.009 0.771
0.005 0.873 0.003
Table 4: Treatment Effects Pathway Example. Results for estimation over different horizon lengths. Note that this uses -6,000-paticn subset (from those in Table (??) who received antibiotics at any point, based on daily decisions antibiotic treatment, over spans of up to 20 days, with labels distributed 58.9%- 1 1% overall.
Example 3 (Clinical Deterioration of Ward Patients) Given the delay-critical nature of ICU ad- mission w.r.t. morbidity/mortulityy what is often desired is an automated prognostic decision support system to monitor ward patients and raise (early) alarms for impending admission to ICU (as a result of clinical deterioration) [25] 94 95. However, observations are costly, and the question of what (and when) to measure is by itself an active choice under resource constraints 36): For instance, there is less reason to measure a feature whose value can already be confidently estimated on the basis of known quantities, or if its value is not expected to contribute greatly to the prognostic task at hand.
This is an archetypical problem for Clairvoyance's s active sensing pathway. Table 5 indicates the performance of different models for balancing this trade-off between information gain and acquisition rate with respect to admissions to ICU of ward patients. At various budget constraints (i.e. amounts of measurements permitted), each active sensing model learns from the training data to identify the most informative features to measure test-time, so to maximize the performance of admission predictions. (To allow some measurements to be costlier than others, they can simply be up-weighted when computing the budget constraint). As before, our propositions (i), (ii), and (iii) are implicit here.
Measure Rate
With 50% Measurements
With 70% Measurements
With 90% Measurements
Evaluation
AUC
APR
AUC
APR
AUC
APR
ASAC
0.714 + 0.018
0.235 + 0.034
0.781 + 0.015
0.262 = 0.037
0.841 + 0.016
0.414 t: 0.033
DeepSensing
0.707 + 0.020
0,230 + 0.036
0.772 + 0.016
0.255 = 0.033
0.829 + 0.017
0.409 + 0.038
Randomize
0.677 + 0.021
0.217 + 0.033
0.729 # 0.019
0.249 = 0.032
0.788 + 0.017
0.269 + 0.039
SASH
0.725 + 0.015
0.248 + 0.032 0.793 + 0.013 0.278 = 0.043
0.849 + 0.014
0.420 + 0.037
Table 5: Active Sensing Pathway Example. Results at different acquisition rates (using GRUs as base predictors).
5 CONCLUSION
Machines will never replace a doctor's 's medical judgment, nor ML researcher's technical innovation. But as a matter of data-driven clinical decision support, Clairvoyance enables rapid prototyping. benchmarking, and validation of complex time-series pipelines-so doctors can spend more time on the real scientific problems, and MI. researchers can focus on the real technical questions. Moreover, collaborative research between medical practitioners and ML researchers increasingly common (48). To help grease the wheels, we developed and presented Clairvoyance, and illustrated its flexibility and capability in answering important and interesting medical questions in real-world environments.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We would like thank the reviewers for their generous and invaluable comments and suggestions. This work was supported by Alzheimer's Research UK (ARUK), The Alan Turing Institute (ATT) under the EPSRC grant 1012510 The US Office of Naval Research (ONR), and the National Science Foundation (NSF) under grant numbers 1407712, 1462245, 1524417, 1533983, and 1722516.
REFERENCES
[1] Romain Pirracchio. Mortality prediction in the icu based on mimic-ii results from the super icu learner algorithm (sicula) project. Springer: Secondary Analysis of Electronic Health Records, 2016.
[2] Alistair EW Johnson, Tom J Pollard, and Roger G Mark. Reproducibility in critical care: mortality prediction case study. Machine Learning for Healthcare Conference (MLHC), 2017.
[3] Sanjay Purushotham, Chuizheng Meng, Zhengping Che, and Yan Liu. Benchmarking deep learning models large healthcare datasets. Journal of Biomedical Informatics, 2018.
[4] Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M Dai, Nissan Hajaj, Michaela Hardt, Peter Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, al. Scalable and accurate deep learning with electronic health records. Nature Digital Medicine, 2018.
[5] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Nature Scientific Data, 2019.
[6] Beau Norgeot, Benjamin S Glicksberg, Laura Trupin, Dmytro Lituiev, Milena Gianfrancesco, Boris Oskotsky, Gabriela Schmajuk, Jinoos Yazdany, and Atul Butte. Assessment of a deep learning model based electronic health record data forecast clinical outcomes in patients with rheumatoid arthritis. Journal of the American Medical Association (JAMA), 2019.
[7] Carl Waldmann, Neil Soni, and Andrew Rhodes. Critical Care: Oxford Desk Reference. Oxford University Press, 2008
[8] Eren Gultepe, Jeffrey P Green, Hien Nguyen, Jason Adams, Timothy Albertson, and Ilias Tagkopoulos. From vital signs to clinical outcomes for patients with sepsis: a machine learning basis for a clinical decision support system. Journal of the American Medical Informatics Association (AMIA), 2014.
[9] Steven Horng, David A Sontag, Yoni Halpern, Yacine Jernite, Nathan Shapiro, and Larry A Nathanson. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PloS one, 2017.
[10] Andreas Philipp Hassler, Ernestina Menasalvas, Francisco José Gareía-Gareía. Leocadio Rodríguez-Mañizl and Andreas Holzinger. Importance of medical data preprocessing in predic- tive modeling and risk factor discovery for the frailty syndrome. BMC Medical Informatics and Decision Making, 2019.
[11] Daniel Trujillo Viedma, Antonio Jesús Rivera Rivas, Francisco Charte Ojeda, and María José del Jesus Diaz. A first approximation to the effects of classical time series preprocessing methods on Istm accuracy. International Work-Conference on Artificial Neural Networks, 2019.
[12] Dimitris Bertsimas, Agni Orfanoudaki, and Colin Pawlowski. Imputation clinical covariates in time series. NeurIPS 2018 Workshop Machine Learning for Health (ML4H), 2018.
[13] Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[14] Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, al. Multivariate time series imputation with generative adversarial networks. Advances in Neural Information Processing Systems (NeurIPS), pages 1,59-11007, 2018.
[15] Jinsung Yoon, William R Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering (TBME), 2018.
10

---

Published as a conference paper at ICLR 2021
[16] Paul Nickerson, Rahelch Baharloo, Anis Davoudi, Azra Bihorac, and Parisa Rashidi. Compar- ison of gaussian processes methods to linear methods for imputation of sparse physiological time series. International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018.
[17] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. Machine Learning for Healthcare Conference (MI.HC), 2016.
(18) Joseph Futoma, Mark Sendak, Blake Cameron, and Katherine A Heller. Scalable joint mod- cling of longitudinal and point process data for disease trajectory prediction and improving management of chronic kidney disease. Conference on Uncertainty in Artificial Intelligence (UAI), 2016.
[19] Bryan Lim and Mihacla van der Schaar. Discase-atlas: Navigating disease trajectories with deep learning. Machine Learning for Healthcare Conference (MLHC), 2018.
[20] Ahmed M Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease progres- sion. Advances in Neural Information Processing Systems (NewrIPS), 2019.
[21] Daniel Jarrett and Mihaela van der Schaar. Target-embedding autoencoders for supervised representation learning. International Conference on Learning Representations (ICLR), 2020.
[22] Zachary C Lipton, David C Kale, Charles Elkan, and Randall Wetzel. Learning to diagnose with Istm recurrent neural networks. International Conference on Learning Representations (ICLR), 2016.
[23] Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. Patient subtyp- ing via time-aware Istm networks. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2017.
[24] Huan Song, Deepta Rajan, Jayaraman Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. AAAI Conference on Artificial Intelligence
(AAAI), 2018
[25] Ahmed M Alaa and Mihaela Van Der Schaar. A hidden absorbing semi-markov model for informatively censored temporal data: Learning and inference. Journal of Machine Learning Research (JMLR), 2018.
[26] Jason Roy, Kirsten J Lum, and Michael J Daniels. A bayesian nonparametric approach to marginal structural models for point treatments and a continuous or survival outcome. Biostatis- lics, 2016.
[27] Yanbo Xu, Yanxun Xu, and Suchi Saria. A bayesian nonparametric approach for estimating in- dividualized treatment-response curves. Machine Learning for Healthcure Conference (MLHC), 2016.
[28] Peter Schulam and Suchi Saria. Reliable decision support using counterfactual models. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[29] Hossein Soleimani, Adarsh Subbaswamy, and Suchi Saria. Treatment-response models for counterfactual reasoning with continuous time, continuous-valued interventions. arXiv preprint, 2017.
[30] Bryan Lim. Forecasting treatment responses over time using recurrent marginal structural networks. Advances in Neural Information Processing Systems (NeurIPS), 2018
[31] Ioana Bica, Ahmed M Alaa, James Jordon, and Mihaela van der Schaar. Estimating counterfac- tual treatment outcomes over time through adversarially balanced representations. International Conference on Learning Representations (ICLR), 2020.
[32] Shipeng Yu, Balaji Krishnapuram, Romer Rosales, and R. Bharat Rao. Active sensing. Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), 2009.
[33] Kartik Ahuja, William Zame, and Mihaela van der Schaar. Dpscreen: Dynamic personalized screening. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[34] Jinsung Yoon, William Zame, and Mihaela van der Schaar. Deep sensing: Active sensing using multi-directional recurrent neural networks. International Conference on Learning Representations (ICLR), 2018
11

---

Published as a conference paper at ICLR 2021
[35] Jaromír Janisch, Tomáš Pevný, and Viliam Lisy. Classification with costly features using deep reinforcement learning. AAAI Conference on Artificial Intelligence (AAAI), 2019.
[36] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Asac: Active sensing using actor-critic models. Machine Learning for Healthcare Conference (MLHC), 2019.
[37] Joseph Futoma, Sanjay Hariharan, and Katherine Heller. Learning to detect sepsis with a multitask gaussian process rnn classifier. International Conference on Machine Learning (ICML), 2017.
[38] Li-Fang Cheng, Gregory Damell, Bianca Dumitrascu, Corey Chivers, Michael E Draugelis, Kai Li, and Barbara Engelhardt. Sparse multi- multi-output gaussian processes for medical time series prediction. arXiv preprint, 2017.
[39] Edmon Begoli, Tanmoy Bhattacharya, and Dimitri Kusnezov. The need for uncertainty quantifi- cation in mashine-assisted medical decision making. Nature Machine Intelligence, 2019.
2019.
[40] Marco Lorenzi, Maurizio Filippone, Giovanni B Frisoni, Daniel C Alexander, Sébastien Ourselin, Alzheimer's s Disease Neuroimaging Initiative, et al. Probabilistic disease progres- sion modeling to characterize diagnostic uncertainty: application to staging and prediction in alzheimer's disease. Neurolmage, 2019.
[41] Aven Samareh and Shuai Huang. Uq-chi: An uncertainty quantification-basee contemporaneous health index for degenerative disease monitoring, IEEE Journal of Biomedical and Health Informatics (JBHI). 2019.
[42] Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in Neural Information Processing Systems (NeurIPS), 2016.
[43] Tian Bai, Shanshan Zhang, Brian L Egleston, and Slobodan Vucetic. Interpretable represen- tation learning for healthcare via capturing disease progression through time. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2018.
[44] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. International Conference on Learning Representations (ICLR), 2018.
[45] Camburu Oana-Mariz Giunchiglia Eleonora, Foerster Jakob, Lukasiewicz Thomas, and Blun- som Phil. Can trust the explainer? verifying post-hoc explanatory methods. NeurIPS 2019 Workshop on Safety and Robustness in Decision Making, 2019.
[46] Sana Tonekaboni, Shalmali Joshi, David Duvenaud, and Anna Goldenberg. What went wrong and when? instance-win wise feature importance for semess models. arXiv preprint, 2020.
[47] Beau Norgeot, Benjamin S Glicksberg, and Atul J Butte. A call for deep-learning healthcare. Nature Medicine, 2019.
[48] Brett Beaulieu-Jones, Samuel G Finlayson, Corey Chivers, Irene Chen, Matthew McDermott, Jaz Kandola, Adrian V Dalca, Andrew Beam, Madalina Fiterau, and Tristan Naumann. Trends and focus of machine learning applications for health research. Journal of the American Medical Association (JAMA), 2019.
[49] Raag Agrawal and Sudhakaran Prabakaran. Big data digital healthcare: lessons learnt and recommendations for general practice. Nature Heredity, 2020.
[50] Gang Luo, Bryan L Stone, Michael D Johnson, Peter Harczy-Hornoch Adam B Wilcox, Sean D Mooney, Xiaoming Sheng, Peter J Haug, and Flory L Nkoy. Automating construction of machine learning models with clinical big data: proposal rationale and methods. JMIR research protocols, 2017.
[51] Duncan Shillan, Jonathan AC Sterne, Alan Champneys, and Ben Gibbison. Use of machine learning to analyse routinely collected intensive care unit data: a systematic review. Critical Care, 23(1):284, 2019.
[52] D Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, and Michael Young. Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems (NeurIPS), 2014.
[53] Jeeheh Oh, Jiaxuan Wang, Shengpu Tang, Michael Sjoding, and Jenna Wiens. Relaxed weight sharing: Effectively modeling time-varying relationships in clinical time-series. Machine
12

---

Published as a conference paper at ICLR 2021
Learning for Healthcare Conference (MLHC), 2019.
[54] Fei Wang, Lawrence Peter Casalino, and Dhruv Khullar. Deep learning in medicine medicine-ppommis,, progress, and challenges. Journal of the American Medical Association (JAMA), 2019.
[55] Maarten van Smeden, Ben Van Calster, and Rolf HH Groenwold. Machine learning compared with pathologist assessment. Journal of the American Medical Association (JAMA), 2018.
[56] Nilay D Shah, Ewout W Steyerberg, and David M Kent. Big data and predictive analytics: recalibrating expectations. Journal of the American Medical Association (JAMA), 2018.
[57] Eric J Topol. High-performance medicine: the convergence of human and artificial intelligence.
Nature Medicine, 2019.
[58] Cao Xiao, Edward Choi, and Jimeng Sun. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. Journal of the American Medical Informatics Association (AMIA), 2018.
[59] Ben Van Calster, Ewout W Steyerberg, and Gary S Collins. Artificial intelligence algorithms for medical prediction should be nonproprietary and readily available. Journal of the American Medical Association (JAMA), 2019.
[60] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Nature Scientific Reports, 2018.
[61] Richard D Riley, Joie Ensor, Kym IE Snell, Thomas PA Debray, Doug G Altman, Karel GM Moons, and Gary S Collins. External validation of clinical prediction models using big datasets from e-health records or ipd meta-analysis: opportunities and challenges. The British Medical Journal (BMJ), 2016.
[62] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Hyperparameter optimization. Chapter 1, Automated machine learning. 2019.
[63] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Advances in neural information processing systems (NeurIPS), 2015.
[64] Lars Kotthoff, Chris Thornton, Holger Hoos, Frank Hutter, and Kevin Leyton-Browe Auto- weka 2.0: Automatic model selection and hyperparameter optimization in weka. Journal of Machine Learning Research (JMLR), 2017.
[65] Randal S Olson and Jason Moore. Tpot: A tree-based pipeline optimization tool for automat-
ing machine learning. ICMI. Workshop on Automatic Machine Learning. 2016.
[66] Xueqiang Zeng and Gang Luo. Progressive campling-bssed bayesian optimization for efficient and automatic machine learning model selection. Health information science and systems, 2017.
[67] Ahmed M Alaa and Mihaela van der Schaar. Autoprognosis: Automated clinical prognostic modeling via bayesian optimization with structured kernel learning. International Conference on Machine Learning (ICML), 2018.
[68] Yuyu Zhang, Mohammad Taha Bahadori, Hang Su, and Jimeng Sun. Flash: fast bayesian optimization for data analytic pipelines. ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD), 2016.
[69] Zi Wang, Chengtao Li, Stefanie Jegelka, and Pushmeet Kohli. Batched high-dimensional bayesian optimization via structural kernel learning. International Conference on Machine Learning (ICML), 2017.
[70] Jenna Wiens, John Guttag, and Eric Horvitz. Patient risk stratification with time-varying parameters: multitask learning approach. Journal of Machine Learning Research (JMIR), 2016.
[71] Yao Zhang, Daniel Jarrett, and Mihacla van der Schaar. Stepwise model selection for sequence prediction via deep kernel learning. International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
[72] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. International Conference on Artificial Intelligence and Statistics (AISTATS), 2016.
[73] Brett Naul, Stéfan van der Walt, Arien Crellin-Quick, Joshua Bloom, and Fernando Pérez. Cesium: ooen-source platform for time-series inference. Annual Scientific Computing with
13

---

Published as a conference paper at ICLR 2021
Python Conference (SciPy), 2016.
[74]
Romain Tavenard, Johann Faouzi, Gilles Vandewiele, Felix Divo, Guillaume Androz, Chester Holtz, Marie Payne, Roman Yurchak, Mare Rußwurm, Kushal Kolar, and Eli Woods. tslearn: A machine learning toolkit dedicated to time-series data. Gitflu,, 2017.
[75] David M Burns and Cari M Whyne. Segleam: A python package for learning sequences and time series. Journal of Machine Learning Research (JMLR), 2018.
[76]
Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W Kempa-Lichr. tsfresh: time series feature extraction basis of scalable hypothesis tests. Neurocomputingg. 2018.
[77] Ahmed Guecioueur, pysf: supervised forecasting of sequential data in python. GitHlu, 2018.
[78] Markus Löning, Anthony Bagnall, Sajaysurya Ganesh, Viktor Kazakov, Jason Lines, and Franz. J Király. sktime: A unified interface for machine learning with time series. NeurlPS 2019 Workshop on Systems for MI., 2019.
[79] Johann Faouzi and Hicham Janati. pyts: A python package for time series classification. Journal of Machine Learning Research (JMLR), 2020.
[80] David Taylor-Robinsee
Olia Archangelidi, Siobhán B Carr, Rebecca Cosgriff, Elaine Gunn, Ruth H Keogh, Amy MacDougall, Simon Newsome, Daniela K Schlüter, Sanja Stanojevic, et al. Data resource profile: the uk cystic fibrosis registry. International Journal of Epidemiology. 2018.
[81] Ahmed M Alaa, Jinsung Yoon, Scott Hu, and Mihaela Van der Schaar. Personalized risk scoring for critical care prognosis using mixtures of gaussian processes. IEEE Transactions Biomedical Engineering, 2017.
[82] Alistair EW Johnson, Tom Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Nature Scientific Data, 2016.
[83] Shawn D Aaron, Anne I. Stephenson, Donald W Cameron, and George A Whitmore. A statistical model to predict one-year risk of death in patients with cystic fibrosis. Journal of Clinical Epidemiology, 2015.
[84] Theodore G Liou, Frederick R Adler, and David Huang. Use of lung transplantation survival models refine patient selection in cystic fibrosis. American Journal of Respiratory and Critical Care Medicine, 2005.
[85] Lionelle Nkam, Jérôme Lambert, Aurélien Latouche, Gil Bellis, Régise-Régis Burgel, and MN Hocine. A 3-year prognostic score for adults with cystic fibrosis. Journal of Cystic Fibrosis, 2017.
[86] Changhee Lee, Jinsung Yoon, and Mihaela Van Der Schaar. Dynamic-deephit: A deep learning approach for dynamic survival analysis with competing risks based on longitudinal data. IEEE Transactions Biomedical Engineering, 2019.
[87] Dan Li, Ruth Keogh, John P Clancy, and Rhonda D Szezesniak. Flexible semiparametric joint modeling: an application to estimate individual lung function decline and risk of pulmonary exacerbations in cystic fibrosis. Emerging Themes in Epidemiology, 2017.
[88] Joëlle Texereau, Dany Jamal, Gérald Choukroun, Fierre-Régis Burgel, Jean- Dichl, Antoine Rabbat, Philippe Loirat, Antoine Parrot, Alexandre Duguet, Joël Coste, et al. Determinants of mortality for adults with cystic fibrosis admitted in intensive care unit: a multicenter study. Respiratory Research, 2006.
[89] Joseph F Dasta, Trent P McLaughlin, Samir H Mody, and Catherine Tak Piech. Daily cost of an intensive care unit day: the contribution of mechanical ventilation. Critical Care Medicine, 2005.
[90] Geneva: World Health Organization. Antibiotic resistance. World Health Organization: News- room Antibiotic Resistance Fact Sheet, 2020.
[91] Jason Phua, Wang Jee Ngerng, and Tow Keang Lim. The impact of a delay in intensive care unit admission for community-aqquired pneumonia. European Respiratory Journal, 2010.
[92] Vincent Liu, Patricia Kipnis, Norman W Rizk, and Gabriel J Escobar. Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system. Journal of Hospital Medicine, 2012.
14

---

Published as a conference paper at ICLR 2021
[93] Michael P Young. Valerie J Gooder, Karen McBride, Brent James, and Elliott S Fisher. Inpatient transfers to the intensive care unit. Journal of General Internal Medicine, 2003.
[94] Jinsung Yoon, Ahmed Alaa, Scott Hu, and Mihaela Schaar. Forecasticu: a prognostic decision support system for timely prediction of intensive care unit admission. International Conference on Machine Learning (ICML). 2016.
[95] Ahmed M Alaa, Scott Hu, and Mihaela van der Schaar. Learning from clinical judgments: Semi-markov--oodultee marked hawker processes for risk prognosis. International Conference on Machine Learning (ICML), 2017.
[96] loana Bica, Ahmed M Alaa, and Mihaela van der Schaar. Time series deconfounder: Estimating treatment effects over time in the presence of hidden confounders. International Conference on Machine Learning (ICML), 2020.
[97] Jinsung Yoon, James Jordon, and Mihaela Van Der Schaar. Gain: Missing data imputation using generative adversarial nets. arXiv preprint arXiv: 02920, 2018.
[98] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. In International Conference on Learning Representations, 2018.
A NEED FOR EMPIRICAL STANDARDIZATION
'All else' is seldom 'equal'. As examples, a review of recent, state-of-the-art research on medical time-series imputation and prediction models demonstrates the following: While the benchmarking performed within each individual study strives to isolate sources of gain through "all-else-equal" experiments, the degree of overlap in pipeline settings across studies is lacking. Such a dearth of empirical standardization may not optimally promote effective assessment of true research progress:
Pruposed Impulation Method
Downstream Prediction Component
Dataset(s)
Technique
Evaluation
Problem Type
Eadpoint
Model(s
Evaluation
Used
Ono-shot
10-year Risk of
Framingham
med. impute
Imputation MSE
Classilcation
Stroke
LogR
Prediction ROC
Heart St. (FHS)
BRITS LB
Impulation MAF, MRF
shot
In-Hosiiaal
NN
Prediction ROC
PhysioNei ICU (MIMIC)
Classification
Death
GRUI GAN P
None
One-shot
In-Hospital Death
LogR, SVM,
Prediction ROC
PhysioNet ICU (MIMIC)
Classification
RNN
UF Shands
G based 1C
Imputation MSE
None
N/A
N/A
N/A
Hospital Data
Ouline
Various
NN, RF. LogR,
Various Medical
X- RNN 15
Imputation MSE
Classification
Endpoints
XGB
Prediction ROC
Datasets
Table 6: Research on Medical Time-series Data Imputation. Typically, proposed imputation methods rely on some downstream dummy prediction task for evaluating utility. However, the datasets, problem types, prediction endpoints, and time-series models themselves do not often coincide. Depending on specific cases, this dearth of standardized benchmarking does not promote accessible comparison across different proposed techniques.
Proposed Prediction Method
Upstream Imputation Component
Dataset(s) Used
Technique
Endpoint
Evaluation
Imputation
Model(s)
Evaluation
Multi-label
ROC, Fl, preci-
Forw-, Back-, Mean-fill
Ch. Hospital LA PICU
LSTX-DO-TR AR
Diagnosis
10
Yes
None
Regression;
MSC: tests for
Data Pre-
Parkinson's (PPMT)
T-1.5TM 2
Subtyping
group effects
imputed
N/A
Nane
Sepsis Onset
ROC, PRC.
Multitask
Duke UHS
MG2-RMD -
Prediction
precision
GPs
None
Imputient
Survival;
Median
Cystic Fibrosis
D-At las
Forecasting
PRC,
Mean-fill
None
(UKCF)
Regression;
PRC,
Masking
PhysioNet ICU
SAND
Classification
MAPE
as Input
N/A
None
(MIMIC)
Table 7: Research on Medical Time-series Prediction Models. Typically, proposals of prediction models pay short attention the choices regarding upstream imputation of missing and/or irregularly sampled data. In comparative experiments, single imputation method is usually fixed w.r.t. all prediction models for evaluation. The datasets, imputation methods, and even prediction endpoints themselves have little overlap across studies.
15

---

Published as a conference paper at ICLR 2021
B BACKGROUND REFERENCES FOR EXPERIMENTS
Example
Example 2
Example 3
Lung Function Cystic Fibrosis Patients
Mechanical Ventilation on Intensive Care
Clinical Deterioration
Modeling Question
Ward Patients
Data Resource Profile: The UK Cystic Fibrosis Registry [0
The Medical Information Mart for Intensive Care [2]
The Ronald Reagan Medical Center General Wards Dataset [8]
Original description of dataset
As specified in the original study [86]. specifically as in their Appendix A
As specified in the software corresponding the implementation of the study in
Às specified in the original study [4],
Initial data cleaning/ selection used
specifically in their Section 5
This is vanilla
Decision support with counterfactuals [28].
Active sensing as an original problem 12) models for personalized screening [3]] and most recent deep learning methods for the active sensing problem 34
time-series prediction, so any technical study should have adequate background, such as our references [7 through [25] in the introduction this paper
marginal structural models [26] and their
Theoretical
background on model pathway involved
deep equivalents [30],
and the most recent: counterfactual recurrent networks
Background on Cystic
Mechanical ventilation as an adverse outcome of interest [88] ventilation as pertains mortality (9T [M] and
Prior methods and importance of forecasting deterioration of patients in general wards (25 (95 and
Fibrosis [3]884 84], studies of classical risk scores [8]. studies of joint modeling approaches for survival analysis [87].
Original works describing and justifying the modeling task
the importance of antibiotic stewardship vs. reduction the need for critical care
specifically developing forecasting systems for clinical decision
well as more recent deep learning approaches to
survival analysis [86]
support
Data collection, data coverage, ethics, etc.
See Sections 1-3
80
See Sections 2-3 in 82
See Section in 81
Table 8: Background References. Additional table of background references for the experiments in Scction
C
NOTE ON CHOICE OF BUILT-IN TECHNIQUES
Given our "Pipeline First" focus (see "Key Design Principles" in Section 2}. especially in the context of medical applications, rather than (re- (te-jimplementing every time-series model in existence, our primary contribution is in unifying all three key pathways in a patient's healthcare lifecycle (i.e. predictions, treatments, and monitoring tasks; Section [ "The Patient Journey") through single end-to-end pipeline abstraction for which Clairvoyance is the first (see Table "Clairvoyance and Comparable Software"). For the predictions pathway, while there is virtually infinite variety of time-series models in the wild, we choose to include standard and popular classes of deep learning models, given their ability to handle large amounts and dimensions of data, well as the explosion of their usage in medical time-series studies (see e.g. virtually any of the paper references in Section [ For both the treatment effects and active sensing pathways, there much less existing work available; for these, we provide state-of-the-an models (e.g. CRN, R-MSN, ASAC, DeepSensing) implemented exactly as given in their original research papers. With that said, as noted throughout, recall that all component modules (including the various other pipeline components) are easily extensible: For instance, if more traditional time-series baselines from classical literature were desired for comparison purposes, existing algorithms from packages such as 773-79] can be integrated into Clairvoyance by using simple wrapper classes with little hassle (for an explicit demonstration of this, see Appendix G).
Note Time to Train: Our computations for the examples included in Section 4 were performed using a single NVIDIA GeForce GTX 1080 GPU, and each experiment took approximately ~24 72 hours. Of course, this duration may be shortened through the use of multiple GPUs in parallel.
16

---

Published as a conference paper at ICLR 2021
D ADDITIONAL DETAIL ON EXPERIMENTS
In our experiments for UKCF (used in Example 1), out of the total of 10,995 entries in the registry data, we focused on the 5,883 adult patients with followup data available from January 2009 through December 2015, which excludes pediatric patients and patients with no follow-up data from January 2009. This includes a total of 90 features, with 11 static covariates and 79 tame-varying covariates, which includes basic demographic features, genetic mutations, lung function scores, hospitalizations, bacterial lung infections, comorbidities, and therapeutic management. Within the 5,883 parients, 605 were followed until death (the most common causes of which were complications due to transplantation and CF- associacd liver disease); the remaining 5,278 patients were right-censored.
In our experiments for WARDS (used in Examples and 3), the data comes from 6,321 patients who were hospitalized in the general medicine floor during the period March 2013 through February 2016, and excludes patients who were reverse transfers from the ICU (i.e. initially admitted from the ICU, and then returned to the ward subsequent to stabilization in condition). The heterogeneity in patient conditions mentioned in the main text include such conditions as shortness of breath, hypertension, septicemia, sepsis, fever, pneumonia, and renal failure. Many patients had diagnoses of leukemia or lymphoma, and had received chemotherapy, allogeneic or autologous stem cell transplantation, and treatments that cause severe immunosuppression places them at risk at developing further complications that may require ICU admission. Here, the recorded features include 8 static variables (admission-tins time statistics) and 37 temporal physiological data streams (vital signs and laboratory tests); vital signs were taken approximately every 4 hours, and lab tests approximately every 24 hours.
In our experiments for MIMIC (used in Example for predictions, and Example for estimating treatment effects), for the predictions example we focus on 22,803 patients who were admitted to ICU after 2008, and consider 11 static variables (demographics information) and 40 physiological data streams in total, which includes 20 vital signs which were most frequently measured and for which missing rates were lowest (e.g. heart rate, respiratory rate), as well as 20 laboratory tests (e-g. creatinine, chloride); vital signs were taken approximately every hour, and laboratory tests approximately every 24 hours. For the treatment effects pathway (used in Example 2), we focus on the 6,033 patients who had received antibiotics at any point in time, based on daily decisions on antibiotic treatment, with maximum sequence length of 20 days. Note that the class-label imbalance between the pure prediction task (Example 1) and treatment effects task (Example 2) slightly different per the different populations included, and the numerical results should not be compared directly. The code for extracting this data is included under 'mimic_dan _att__exttaction in the repository.
In all experiments, the entire dataset is first randomly partitioned into training sets (64%), validation sets (1 (16%), and testing sets (20%). The training set is used for model training, the validation set is used for hyperparameter tuning, and the testing set is used for the final vaaluttion-whte the performance metrics. This process itself is then repeated randomly for a total of 10 times, with the means and spreads of each result used in generating results Tables 3 As usual, the entire pipeline (with the exception of the pathway model corresponding to each row) Is fixed across all rows, which in this case uses min- max normalized features, GAIN for static missing values, M-RNN for temporal imputation, and no prior feature selection; where hyperparameters for such pipeline components are involved (i.e. GAIN and M-RNN here), these are also-as as they should nensonttant across all rows.
generates
In order to highlight our emphasis on the temporal dimension of autoML in Clairvoyance, the results for SASH isolate precisely this effect alone: Each result for SASH is generated using the simple approach of Figure aa that is, by 'naively' decomposing SASH into collection of SMS problems (for each model class considered), subsequent to which the stepwise models for each class are further ensembled through stacking. Note that the point here is not argue for this specific technique, but merely to show that even this (simplistic) approach already yields some gains, thereby illustrating the potential for further autoML research (which can be conveniently performed over Clairvoyance's pipeline abstraction) to investigate perhaps more efficient solutions with respect to this temporal dimension. Briefly, in DKL the validation performance for each time step is treated as a noisy version of black box function, which leads to multiple black-box function optimization problem (which DKL solves jointly and efficiently); we refer to [7]] for their original exposition. In our experiments we complete 100 iterations of Bayesian optimization in DKL for each model class. For reproduc- ibility, the code for our implementation of DKL used for experiments is included in the repository.
17

---

Published as a conference paper at ICLR 2021
E WORKED EXAMPLE: USING THE FULL PIPELINE
This section gives fully worked example of using the Clairvoyance pipeline (via the predictions pathway). To follow along, the user should have their static and temporal datasets for training and testing, named as follows-when 'data_name' is replaced by the appropriate name of the dataset:
dat @ati__amm__.eeppoaaaaraaaaaa.a. csv.
dat das___eaee_sttatccccccoootttdtt
data__rme__staii__tsss___e
and placed within the / aatiatttddttaaamme//aaaaa As described in Section ("As a Software Toolkit"), the requirement is that the data conform to the standard EAV open schema for clinical records (i.e. patient key, timestamp, parameter, and value). See Figure summary of the pipeline workflow that we shall be walking through and executing in the following subsections:
1. Load Dataset: Extract csv files from the original raw datasets located in the data directory.
2. Preprocess Dataser: Preprocess the raw data using various filters, such as replacing negative values NaN, doing one-hot encoding for certain features, and normalizing feature values.
3. Define Problem: Set the prediction problem (one-shot or online). the label (the target of predictions), the maximum sequence length, and (optionally) the treatment features (not used here). Also define the metric for evaluation and the task itself (classification regression).
4. Impute Dataser: Impute missing values in the preprocessed static and temporal datasets- for each selecting among data imputation methods of choice, and return the complete datasets.
5. Feature Selection: Select the relevant static and temporal features for the labels (e.g. recursive or greedy addition/deletion, or simply skip this step by setting the method be None.
6. Model Training and Prediction: After finishing the data preparation steps, we define the model used for time-series prediction, and train the model using the training dataset. After training is finished, we use the trained model to predict the labels using the testing dataset. Estimate Uncertainty: Estimate uncertainty of the predictions made by the predictor model.
8. Interpret Predictions: Compute the mostance-wwss)) feature and temporal importance weights.
9. Visualize Results: Output predictions, performance metrics, uncertainties, and importances.
Stanaard forms 20 GOC
Impudation
Tomperal
Preprocess Nomaa butees
Define problem
Loaed data
Defined
Ckovbel pood miner
Terperal vall iigoo
Oiservatins
Outlier
cPt 39%
Bestauts dalke
Toe tamels
Static
gendec apn
Mamaily noer
Lubil sifee Rudieiil vullaad
Marking
TrainTestSpiin
K-BACCY
Interpeetation
Interpeet
Time-satic
peslict
Feskas
Iomeenal any wttnen
Titee lite--seris prediction models
Fecatuee selection
IKN LSTM, Jtlll
TrewsiiioN
Toining data
Remunk Suace
Main outputs
Common opticts
Seouby leaas
specific model
Packe
Monz
Output visualiaaioon
Uncertainty
Unsectaiill estimate
Terting cita
Confilens transl
Figure 6: Pipeline Workflow, Step-by-step schematic corresponding to the procedure in this worked example.
18

---

Published as a conference paper at ICLR 2021
Import necessary packages for this example:
# Necessary packages
from futuree__ import absolute_import from future__ import division
from future_.. import print_function
import numpy np
import warnings; warnings. filtterwrnings((ignoon import sys; sys.path. aathhappendd".../
from utils import PipelineComposer
E.1 LOAD DATASET
Extract esv files from the original raw datasets located in the directory. The CSVLoader is responsi- ble for loading esv files from the original raw datasets in the /settssets directory. In this example we use data from MIMC, so here the 'data_name is 'mimic' throughout:
Load Dataset
from datasets import CSVLoader
Define data name data_name 'mimic'
Define data directory date_directory = ht.dddaasttsddaaaattttaaaaa 1/" + data_name '2'
Load train test datasets
@ata__oader_trainingg
'static_train_data cww.p/attaccc.g'....o.
Lemporai_file-atttddirrrtto "tempora__train_data. csv.gz')
@ata_loader_testing =
"static_test_data.cs csv.gz,,
demmooral_esst_ddtteeeev csv.gz')
dataset_training data gosta_loaeer_traiinng....
dataset_testing = data_loader_ttesiigg..u
print('Finish data loading.")
E.2 PREPROCESS DATASET
Preprocess the raw data using multiple filters. In this example, we replace all the negative values to NaN (using F MilteeNegativee do one-hot encoding on 'admission_ type" feature (using OneHotEncoder), and do MinMax Normalization (using Normalizer). Preprocessing done for both training and testing datasets; note as should be the case-the "fit_traa transformm method called on the training dataset, and only the "cransform' method executed on the testing dataset:
Preprocess Dataset
from preprocessing import FilterNegative, OneHotEncoder, Normalizer
(1) filter out negative values regative_filter = FilterNegative()
(2) one-hot encode categorical features one ome_hot_encee hoo__enco "admission_type" onehot_encodes =
Normalize features: options (minmax, standard, none)
19

---

Published as a conference paper at ICLR 2021
normalization 'minmax'
normalizer Normalizer (normalization)
i Data preprocessing
filter_pipeline TipelInnccompeeeeegaaaiieeee onehot_encoder, normalizer)
dataset_training fil.err.ppeii....i.ttaasmommmmm trainnnng
dataset_testing = filter_pipel transfomm(dattsst
print('Finish preprocessing.'
E.3 DEFINE PROBLEM
The prediction problem can be defined as: 'one-shot" (one time prediction) or 'online 'onninee'rolling (rolling window prediction). The "max_seq_len" The 'label name" is the column name for the label(s) selected as the prediction target. The "treatment' the column name for the actions selected treatments (not used here in this example. in the predictions pathway). The "window specifies the prediction window (i.e. how many hours ahead to predict). The 'metric_nante specifies the performance metric of interest, e.g. 'auc', "apr', 'msc', 'mac', and the "task" is classification regression. In this example, are interested in issuing online predictions for whether the patient will require mechanical ventilation after 4 hours:
the maximum sequence length of time-series sequence.
Define Problem
from preprocessing import ProblemMaker
Define parameters
problem 'online'
max_seq_len 24
label_name = 'vntilatoo''
treatment None window
# Define problem problem_maker eemmmkkerrreeee croblemMaker(rrobleeee max_seq max_sse_lennmax_sqq_uu treatment-trretment,, window window)
label=[label_name),
dataset_training = problem_maker www.aaaaee..tttssfrroooooooooe fil_ttraaffommttttsstrrrann
dataset_testing problem_makeer @ealemm_mkke... fit_rraassformmttttstteessssss
Set other parameters metric_name 'auc'
task "classification"
metric_sets [metric_name] metric_parameterss ('problem': problem, 'label_name': [label_name]]
print('Finish defining problem.")
E.4
IMPUTE DATASET
For static imputation there are options such as mean, median, mice, missforest. knn, gain. For temporal imputation there are options such as mean, median, linear, quadratic, cubic, spline, mrnn, tgain. In this example we simply select median imputation for both static and temporal data:
Impute Dataset
from imputation import Imputation
Set imputation models iratic_imputation.mm modelllloommlllleeeoe = 'median' intprratlmmuutaiiinnn = 'median'
20

---

Published as a conference paper at ICLR 2021
Impute the missing data static_imputation Imnutationnimmuuaaiion.. _anettiiommmmm___meeeee static_.iputation.noiii
data_type 'static")
temporal_imputttion = (iputationimmuuatiionn sasioon_mdee__meeee = temporal_imputation.nooo
data_type "temporal')
Imputation_pipeline
temporal_imputttion
bataset_trainiie = imputation_pipell heappeline.fitttraaaeeee tttt.trrnnfommmattttt.tttt training)
dataset_testing imputation_pipell transform(datasst__eestt
print("Finish imputation.')
E.5 FEATURE SELECTION
In this step, we can perform feature selection for the most relevant static and temporal features the labels. In the simplest case, skip the feature selection step entirely (as do here). The user can select from among sreedy-adation greedy-deletion letion, seeursive-adde ion, recursive-dell ion, and None. The feature_number specifiss the number of selected features:
Feature Selection
from feature_selection import FeatureSelection
Set feature selection parameters
sraticcfeaturesselettion...oo = None
staticc.faturresselecttooonnn moodel = None Cemporal_feature__eeecttooon
None # Select relevant features seaiic_featur__seeeeee = (eetee
stattc_.eature_selettoooooo
feature_type 'static',
feature <etbbere_ummbrr>> = static_festure_selettiii task = task, metric_nand metric_name.
metric_parameters eetric_parameters)
laaorra_feature_seeleee festureeeelctioooffffttteeeeess aaato__ooel__ammm____ temporal_eesturre.elcctttoooooo
feature_typs 'temporal',
feature_number feaorral_ffatur_sseeettte
task = task, netric_name = metric_name,
metric_parameters metric_parrameters.
feature_selecte let___eeeeccooippeellleeee harprccccccooeetttttttceettterre
temporal_feature_selltttt
dataset_training
fiL_tranffommstascc_rraiiiin
dataset_testing feature_sellcttiooppppp.... transform(sttsstt_testtnn)
print('Finish feature selection.")
E.6 TRAINING AND PREDICTION
After finishing the data preparation, we define the predictive models. Existing options include RNN, GRU, LSTM, Attention, Temporal CNN, and Transformer, and as the case for the other pipeline modules, and as discussed in Section i easily extensible through the standard fil-transform- predict paradigm. We now train the model using the training dataset. We set the validation set the 20% of the training set for early stopping and for saving the best model. After training, we use the trained model to predict the labels of the testing dataset. Here the parameters include model _ame: rnn, gru, Istm, attention, ten, transformer; mode wpp_ppaaaaeeee network parameters,
21

---

Published as a conference paper at ICLR 2021
such hdim: hidden dimensions, n layer: number of layers, p_head: number of heads (for transformer model), batch size: number of samples in mini-batch, epochs: number of epochs, Tea rni ng_rate: learning rate, stat c_mred: method of incorporating static features (e.g. by concatenation), t time_mode: method of incorporating temporal information (e.g. concatenate), etc.:
Training and Prediction
from prediction import prediction
# Set predictive model
model_name 'gru'
model parameters model_parameterss
('h_dim': 100,
'n_layer': 'n_head': 2, bbatch_size:: 128,
'epoch': 20, @model__ypee:: model_name, "learning_rate': 0.001,
"static_mode": 'Concatenate', 'Concatenate',
ttiee_moee
'verbose': : True}
up validation for early stopping and best model saving lataset_training wtt.aatttestssplttpooooaaaaaaa
prob_ test 0.0)
Train the predictive model
pred_class (mmoecclcioommee_____.. laame,, model paramettrs,,,, task) pred_class. (it(daatset_ttaannnnng
# Return the predictions on the testing set
test_y_hat httpcccasssppeeiccccttaaaaa
print('Finish predictor model training and testing.") ')
E.7 ESTIMATE UNCERTAINTY
Estimate uncertainty of the predictions (which we name 'test_c ci hat below) made by the predic- tor model. In this example, use the method of ensembling to model uncertainty in prediction output:
Estimate Uncertainty
from uncertainty import uncertainty
Set uncertainty model
uncertainty_model.na 'ensemble'
# Train uncertainty model uncertainty_model
unerrtaintyuuneertaitty..mm hatierttaittyymmee_aaeeeeee
parel_parameters, pred_class, task)
# Return uncertainty of the trained predictive model
test_ci_hat uncertainty_medte uncerfeiityymmbb..ppeettttttdtte
print("Finise uncertainty estimation')
E.8 INTERPRET PREDICTIONS
Compute feature importance weights (which we name 'test s hat below). In this example, we the method of (temporal) INVASE to model instance-wise feature/temporal importance weights:
22

---

Published as a conference paper at ICLR 2021
Interpret Predictions
from interpretation import interpretation
# Set interpretation model interpretationnmoellnnaa = 'tinvase'
Train interpretation model
interpretor ineerpeeatiioninneepprtaaooooood nmmee
madel__aaraeters. pred_class, task)
interpretor fit(dátaase__taaining
Return instance-wise temporal and static feature importance test_s_hat = http:ttor..preicccttttteeeeeeee
print('Finish model interpretation')
E.9
VISUALIZE RESULTS
Here we visualize the performance of the trained model (using the perforacc method):
Visualize Performance
from evaluation import Metrics from evaluation import print_performance
# Evaluate predictor model result (eetricc metric_parameters) evaluaaee dataset_testing. label, test_y_hat)
print( Finish predictor model evaluation.")
print('Overall formanccc"""
print_performanceeresl.. (result,, metric_sets, metric_parameters)
Performance metric: auc
ventilator
0325
Perforaance
0.75
0.80
Sequence Length
Similar methods can be used to visualize model predictions, uncertainties, and importances (by importing pwiit__preeictttt.prrprpppp prediction, print_uuceeeeee wnc.rtainty, and print interpretati methods). See Jupyter notebook tutorial for complete sample code, including inputs, outputs, and visualizations:
0ther Visualizations
from evaluation import print_prediction, print_uncertainty,, grint_interpretation
# Set the patient index for visualization index = [1]
print('Each prediction')
metric_parameters)
print("Uncertainty estimations')
print_uncertainty (test_y.hat(iidex),, test_ci_hat((inex), metric_parameters)
print("Model interpretation')
print_interpretation [test_s_hat(index]
dataset_training feature_name,
metric_parameters, amettess model_parameters)
23

---

Published as a conference paper at ICLR 2021
F WORKED EXAMPLE: USING THE AUTOML INTERFACE
This section gives fully worked example of using the Clairvoyance optimization interface (via the treatment effects pathway). Here the basic structure remains the same as in Section E but in the model training step (here
CRN for the treatment effects model) show example of performing stepwise model selection (SMS) as well. We assume the reader is familiar with the details in Section E and do repeat similar descriptions. Instead, we organize the code as in a standard
experiment using 'main' function wrapper with top-level arguments to enable ease of inspection.
Import necessary packages, begin main function, and basic parameters:
from future__ import insolute_impor
from future__ import division
from future__ import print_function
import argparse import numpy np
import warnings; warnings. Tiliterwariings(iignrr append('../') sys.path.appeen(
import sys; sys.path,
from datasets import CSVLoader
from preprocessing import FilterNegative, OneHotEncoder, Normalizer, ProblemMaker from imputation import Imputation
from feature_selection import FeatureSelection
from treatments. CRNNCRRNMMdde import CRN_Madel
from prediction import AutoËnsemble
from nuttoml.mode) import AutoTS
from evaluation import Metrics, BOMetric from evaluation import print_performance, print_prediccton from utils import PipelineComposer
Begin Main Function
def main (args):
Args:
data loading parameters:
data_names, mimic, ward, cf, mimic_antibiotics
pr ereroocess parameters ters:
normalization: minmax, standard, None
owe_hoo__encoding input features that need to be one-hot encoded
problem: "one-shot" 'online'
'one-shot': one time prediction the 'online': preditcion at every time stamps of the time-series max_seq_len: maximum sequence length after padding 1abel_name: the column name for the label(s)
the time-series
treatment: the column name for treatments imputation parameters:
static_iipuutaiio__eddde mean, median, mice, missforest, knn, gain, etc.
mean, median, linear, quadratic, cubic, spline, etc.
feature selection parameters:
feature_selectionn.n moddell greedy-addtion, recursive-addition, etc.
feature_number: selected feature number
predicctor_parmetteess
epochs: number of epochs
bo_itr: bayesian optimization iterations
static_mode: how to utilize static features (concatenate or None)
time_mode: how to utilize time information (concatenate None)
task: classification regression
metric_name: auc, apr, mae, mse
...
# Set basic parameters metric_sets wargs..etric..o met metric_parameters ('problem': args.probles 'label_name' [args. 1aabel_name]
24

---

Published as a conference paper at ICLR 2021
F.1 LOAD DATASET
Load Dataset
(continued within 'def main')
# File names
data_directory "../daaasets/datta"" trgs.data_name
args. dataddaataannaaa name
data_loader- wra.loodddrrrriiinggggooo
'static_train_date dav.ndaa.aa....z csv.gz',
Cemporal_filleddaadddrreeee Cemporal__train_ddtaeeee csv.gg))
@ata_loader_tees _estter_tettingg =
CSWLoader aal.aaferssaaacccfflllllaaaaaaad
'static_test_datae csv.gz',
Lemporal_Filleedattddiirrttt "temporal_test_dataaee gz')
dataset_training
data_loader wtw.oaddee_rraiinn...oooo
dataset_tessting = tett_loader_teetingg...
print('Finish loading.')
F.2 PREPROCESS DATASET
Preprocess Dataset
(continued within 'def main')
# (0) filter out negative values (Automatically) negative_filter = FilterNegative()
one-hot encode categorical features
onehot_encoder OneHotEncoder wwwt__eecodig_featureese ooeallarrg..sshhhooeooeeeeeeeeeo http:r/rrrroooeeoeeeeeeeeeeeoede mon_ohinnncoodngg)
# (2) Normalize features: 3 options (minmax, standard, none)
normalizer Normalizer )
filter_pipeline PippLinncooppperreaaattleeeeeeie onehot_encoder, normalizer)
dataset_training filter_pipelle fit_.rrasfoommdatttttttrrrii
dataset_testing filter_pipeline. transfomm(dataeetl.esttt)))
p'int('Finish preprocessing.')
F.3
DEFINE PROBLEM
Define Problem
(continued within 'def main')
problem_maker
(proomleaarrroooorerrrreee problem,
max_seq_len=args.. maaeggsssaa__sseeeee treatment=Cargs. treatmenttttttt
dataset_training problem_maker fitt.taanfoommcattttttttrrininn
dataset_testing prwblee_maker.errrree fit_ htt.:ttraasssmmttttttttttettttie
print("Finish defining problem.")
F.4 IMPUTE DATASET
Impute Dataset
(continued within main')
static_imputation = Imputation(
imputation,
mat_emooeee_eeeeeee satell_nmmaarggss www.ssttttttaitt tattttiiiiutaatiinnmee...ee model, dnta_type='statiee static')
25

---

Published as a conference paper at ICLR 2021
temporal al_immutation Imputation(
imputation
watriioommoeeeeaaaaarrrre _namm=args...... htw.rrr.....oma....mmmttoooooooo deta_.tyee='temporal
imputationnpipellnne = Pipe_//ecccopperrrttaaaiiiiititt temporal_imputttion
dataset_training http:p/pppe.ii...i..traaaforoooe training)
dataset_testing
print(FFinish imputation.
F.5
FEATURE SELECTION
Feature Selection
(continued within 'def main')
"tatic_feature_seleette FeatureSelection
feature_type= 'static', stttrre_ttp==sttiii...
traturre_umber-argee task=args. task,
wwt.rrrgggeetriicaaaaaaaa metric_name-argg..eer netricc.peramneessmmttrr.....ppp
temporaa_.eeatur__seletttto FeatureSelection((
festure_selection_nee aorirriioo__odee__am-aarrrggg ott.::/oo.ooooo...e.eeeeeeeeeeee
feature_type=' temporal',
"eeturr_typ==""tmmprralll
feature_numberaargs
task=args. task,
retric_name-args..etrrr
PipoLLnCCmmperrssttiicfeeeeerrrr resture_ellctioonpppellie
Lemporaa_faaturreeelletioo)
dataset_training feature_select reature_selectionnpppee... htt.eieiiiiitttssfoorrtsssseeerr
dataset_testing Beaur
transsoommmaaasstttessssn print("Finish feature selection.")
F.6 OPTIMIZATION AND PREDICTION
Since want do stepwise model selection, this step differs from that in Section particular, here we are not just relying on a single pathway model (CRN); we are calling the 'AutoTs' module to perform Bayesian optimization-watt which implements SMS by DKL., exactly as described 77]
Optimization and Prediction
(continued wit main')
# CRN model model @itee_pprrmmtern
p'proecttionhhoiiie mhonizon pecedder_mxx_apphaaaa :1,
decoder paaadeer_m___pphaa"___ 1,
sitatic_mode':: 'concatenate', 'time_mode": "concatenate')
crn_model CRN_Mode (task=' (lassss classificatiinn))
crn_model wooooel.seeee set_params wttptodde__pprramteeeeeeee
model_class= crn_model
# train_validate split
dataset_training htspsainnggtrraaaaaaattssssssste ttt.tsssplltprro__llllla.... prab_test=0.2)
26

---

Published as a conference paper at ICLR 2021
# Bayesian Optimization Start
metric B0Metric(metric='aue auc", fold=0, split='test')
# Run 80 for selected model class
30_model A trriiiing, model_class, metric)
models, bo_score mo__moel.. (anpning_ooppuuu_ittereee auto_ens_model AutoEnsemble (models, bo_score)
# Prediction of treatment effects
test_yyhat auto_ens_modee conns_mdel..g pondee..peeicttdaaaet (datsstttttttttttttttin rest_split="test]
print("Finish AutoML model training and testing.")
F.7 MODEL EVALUATION
Output performance evaluation for the final trained model, using metric parameters as defined above:
Model Evaluation
(continued within "def main')
result Metrics metric_ parrricppaammeerss.eem .evetttteee..lllllttttttt.. test_y_hat) print('Finisi ITE model evaluation.')
p'int('Overall performance') frrrperrffrnaacceeesst,,,, metric_sets, metric_parameters)
F.8 TOP-LEvEL PARSER
Define and Parse Arguments
n_me___ maiiannn
parser = argparse. ArgumentParser()
parser.add_nd add___runn_rreeett
'--data_name
choices=['mimic" 'ward', 'cf', 'mimic_antibiotiis"]]
/effult=inimic_anibbbiisss
type=str)
parser.adddaaggmmntt
'--normalization",
Dooices=='minmax'
'standard', None],
default='minmax'.
type=str)
parser.add_argumentt
--=one_hot_encoding
sefault="admiistiontttpee,
type=str)
aarser.add__rgumeett
'--problem',
Phoices=["onnine"
'one-shot"] ]
default="online",
type=str)
parser..add_argumentt
'--max_seq_len'
help='maximum sequence length',
default=20,
type=int)
parser..ddd_argumer.
'==1abee_namee,
default="ventilator,,
type=str)
parser..ad__aruumett
'--treatment',
default="antibioticss
type=str)
27

---

Published as a conference paper at ICLR 2021
parser a
ittatic_immputaiion.oeee
"oiices=['mean'' 'median', 'mice', 'missforest', 'knn', 'gain'], default='mediaan' type=str)
parser.add_argumentt
"==temporal_iimuutatiooomm
choices=['mean 'median', 'linear', 'quadratic', 'cubic', 'spline',
'mrn'', 'tgain'],
default='mediann,
type=str)
parser.add__agumenss
prolice=s(greedyyaaddiiiooo "greedy-deletion', 'recursive-additee tion',
'recursive-deletion Nonel,
default=None,
type=str)
parsse..adu.arrumentt
default=10,
type=int)
parserraddd_agumentt
(roicces("greedlyaadddtio 'greedy-deletion', 'recursive-adds tion", None],
Nene,
type=str)
parser..ad__argument
default=10,
type=int)
parser..add_argument
'--epochs',
default=20,
type=int)
parser.
addser.add__rgumment
'--bo_itr',
default=20,
type=int)
padsse..add_argumentt
'--static_mode',
choices=("connatennee ,None],
default="'cocatenate:
type=str)
parser..addaargument
'--time_mode',
defaullt=/comaatenttee,,
type=str)
parser..add_argument
'--task',
'regression',,
ceffuutt=(classiiccction type=str)
barsse..add_argumeett
'--metricc_nme',
choices=['aur au'''' 'apr', 'mse", 'mae") default='auc', type=str)
# Call main function
args parser.parse_are parse_args() main(args)
28

---

Published as a conference paper at ICLR 2021
G EXTENSIBILITY: EXAMPLE WRAPPER CLASS
Since novel methods are proposed in the ML community every day, the pipeline components should be easily extensible to incorporate new algorithms, To integrate new component method (e.g. from another researcher's code, or from an external package) into the framework, all that required is simple wrapper class that implements the 'fit', 'predict', and "gechhyperarameterrsppacc methods. Here we show an example of how a classical time-series prediction model (ARIMA) can be integrated.
ARIMA Wrapper Class
Necessary packages i import os
import pmdarima from datetime import datetime base import BaseEstimator, PredictorMixin import numpy as np
class ARLMA(BaseEstimaton PredictorMixin):
"""Attributes:
task: classification or regression
MA degree
d: Di Differncing degree
q: AR degree
time_mode mode: 'concatenate' or None model_id: the name of model
model_path: model path for saving
verbose: print intermediate process
def init (self,
task-None,
p=None,
d=None,
q=None,
Lime_mode=None:
model_id='auto_eee auto_arima',
model_path= tmp',
verbose=false):
super O init__
self. task task
self.p =
self.d =
self.q
self. time_mode time_mode mode
self soff.moeel_pth = model_path
model_id self. verbose verbase
# Predictor model & optimizer define self. aef.ppredictor_mode
None
self. task 'classification
raise ValueError (* A'Aima model cannot be used for classification
# Set path for model saving if not os, path (mode_(ho))))) os m save_fileeleeeeeeeeeeeeer. datetime nowe) wtw.eetttttt.e.memeee ( "HXMKS') + '.hdf5'
'O//O'.format' (model_path, model_id)
strftime
new(self, model_id):
"""Create new model with the same parameter as the existing one. Args:
model_ id: an unique identifier for the model
29

---

Published as a conference paper at ICLR 2021
Returns:
new ARIMA
return ARIMA task,
self.p,
self.d,
self.q,
self. time_mode, model_id,
self self.modellppth,
self verbose)
fit
fit(self, dataset, fold=0, brain_spli=='ratiie calid_split='val" Arima model fitting does not require an independent training set.
pass
predict
def predict(self, dataset, fold=0, pest_split='test'): """Return the predictions based on the trained model.
Args:
dataset: temporal, static, label, time, treatment information
fold: Cross validation fold
test_split: testing set splitting parameter
Returns:
test_y_hat: predictions on testing set
test_x, test_y seFF_daaa_.rerrossssaaaaat fold, test_split)
shape@ shap__y.shapeee]
shape1 = test_y.shappe(1
shape2 s2)
yointttestyyshagpg
assert shape2 ==
# y: N_sample, max_ max_seq_len, dim fited_list list = []
for in rshaees::
ye test_y[i, :,
model amm.rrima.Ae
(rrrmaaaAAitt(ordrrssefff,. self.d, self.q), suppress_wwrmings=True))
try:
model.fit
y_hat model. oat.pracct......aaaccccaaaccrra
except Exception:
y_hat np. likes_)
y_hat mp. wtw.stttcctfeddi.ieee axis=0)[;, i, None]
return y_hat
@staticmethod
get_ eet_hyperpaaameter.sso
def get_hyperparameter..oo
hyp. [C (C'nane": 'p', "type's "discrete", "donain': 1ist(range(1, 6)), 'dimensionality": 1).
('nane": 'd', "type's "discrete', "donain': list(rarge(1, 6)), 'dimensionality's 1);
('nane": 'q'. "type's "discrete", "donain': list(range(1, 6)), 'dimensionality': 12]
hyp.
30

---

Published as a conference paper at ICLR 2021
H SOME FREQUENTLY ASKED QUESTIONS
Q1. Does Clairvoyance include every time-series model under the sun?
A1. That is not our purpose in providing the pipeline abstraction (see Section 2
"As a Software
Toolkit"), not to mention generally impossible. We do include standard classes of models (e.g. pop- ular deep learning models for prediction), and an important contribution is in unifying all three key tasks involved in a patient's healthcare lifecycle under a single roof, including the treatment effects pathway and active sensing pathway (both for which we provide state-of-the-ee time-series models) in addition to the predictions pathway (see Section 2 "The Patient Journey", Figures T2 as well as Table T Moreover, as noted throughout, modules are easily extensible: For instance, if more traditional time-series baselines from classical literature are desired for comparison purposes, existing algorithms from [73)79 can be integrated into Clairvoyance by using wrappers, with little hassle.
Q2. Isn't preprocessing, imputation, selection, etc. already always performed?
A2. Yes, and we are not claiming that there is anything wrong with individual studies per se. However (per Section 2 "As an Empirical Standard", and Appendix [ Tables 7 while current research practices typically seek to isolate individual gains, the degree of clarity and/or overlap in pipeline configurations across studies is lacking. This dearth of empirical standardization may not optimally and may obscure/entangle true progress, By providing
promote practical assessmend/rprooucibbiig software toolkit and empirical standard, constructing an end-to-end solution to each problem is easy, systematic, and self-documenting (see Figures 2 3 and evaluating collections of models by varying a single component ensures that comparisons are standardized, explicit, and reproducible.
Q3. How about other issues like regulations, privacy, and federated learning?
A3. Per the discussion in Section Clairvoyance is not solution for pnfeerecee/appllccaionssss fic considerations such cohort construction, data cleaning and heterogeneity, patient privacy, algor- ithmic fairness, federated learning, or compliance with government regulations. While such issues are real/important concerns (with plenty of research), they are firmly beyond the scope of our software; it is designed to operate in service to clinical decision support- not at all to replace humans in the loop.
Q4. What are these interdependencies among components and time steps?
A4. Componentwise interdependencies occur for any number of reasons. We have discussed several examples (see Section 2 "As Empirical Standard"), but it is not our mission to convince the reader from scratch: For that, there exists plethora of existing MutoML/medicas literature (see e.g. Section 3}. However, the pipeline abstraction serves as a succinct and standardized interface to anyone's 's favorite autoML algorithm (see Section "As an Optimization Interface"). Moreover, here we do specifically highlight the temporal dimension of model selection opened up by the time-series nature of the pipeline (see Figure 55. In particular, each example in Section Q speecifically illustrates the gains in performance that already occur-cetted paribus-usiin a simple approach to SASH as in Figure a).
Q5. Where is all the background and theory each module?
A5. The scope of the software toolkit is purposefully broad, but is not our intention to provide a technical introduction to each of the topics involved (which would-in any case- impossible in the scope of a paper). While Clairvoyance lowers the barrier to entry in terms of engineering/evallatin it is not intended to be used black-box solution. For instance, expect that user desiring to conduct treatment effects estimation using the CRN component to be familiar with its basic theory and limitations. That said, in addition to the various references provided throughout the description of each aspect of Clairvoyance, the following may serve as more concise background information on original problem formulations and solutions: For treatment effects estimation over time we refer to [31]; for active sensing we refer to [34]; for time-series data imputation we refer to [5]: for interpretation by individualized variable selection we refer to [44]; for autoML in general we refer to refer to Section 3. 1 in [7]; and for the stepwise model selection (SMS) problem we refer to Sections 2-3 in [71]; moreover, Figure 5
633; for the pipeline configuration and selection (PSC) problem
shows how new problems (e.g. SASH) directly result from combining their optimization domains.
Q6. How do you know what clinicians want?
A6. With clinicians as developers/authors, it is our central goal to understand realistic usage scenarios.
31

---

Published as a conference paper at ICLR 2021
1 GLOSSARY OF ACRONYMS
ASAC: Active sensing by actor critic, first defined in [36].
APR: Area under the precision-recall curve.
AUC: Area under the receiver-operating characteristic curve.
CASH: Combined algorithm selection and hyperparameter optimization, first defined in 63
CRN: Counterfactual recurrent network, first defined in [31
DKL: Deep kernel learning, first defined in 722
FLASH: Fast linear search, first defined in 68
GAIN: Generative adversarial imputation network, first defined in 97
GANITE: Generative adversarial network for individualized treatment effects, first defined in 99
GRU: Gated Recurrent Units, type of recurrent neural network.
INVASE: Instance-wise variable selection, first defined in 44
LSTM: Long-short term memory, type of recurrent neural network.
MAE: Mean absolute error
MICE: Multiple imputation by chained equations.
MissForest: Missing value imputation using random forest.
MRNN: Multi-directional recurrent neural networks, first defined in [5
PSC: Pipeline selection and configuration, first defined 66].
RMSE: Root mean squared error.
RMSN: Recurrent marginal structural network, first defined 30
RPS: Relaxed parameter sharing, first defined in [53
SASH: Stepwise algorithm selection and hyperparameter optimization, first defined in Section 2
SKL: Structured kernel learning, first defined in [69]
SMS: Stepwise model selection, first defined 71
TCN: Temporal convolutional network.
Note: prefix of "T" to certain techniques simply indicates its temporal counterpart (e.g. "T-GAIN" refers to the method of GAIN using recurrent neural network for handling the temporal dimension).
32


Reviews:
[{"review_id": "mC5gsJsIW4o", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript introduces and illustrates an end-to-end software pipeline, called Clairvoyance, for medical machine learning on time-series data. The authors must be congratulated for having designed and developed this wonderful resource to accelerate the adoption of these computational techniques in clinical practice as a way to support people’s judgement and decision-making. The manuscript excels in describing and relating its contributions with related work. It has also included a convincing set of experimentation on datasets from three medical environments that are supplementary to each other. My only concerns with this paper are (i) describing and justifying these experiments, their materials (also research ethics), and their processing, evaluation, and statistical significance testing methods to an extent that allows the reader to comprehend these original studies that are now a part of the pipeline release paper (perhaps an appendix or references to separate original studies would do), (ii) release details of the pipeline seem to be missing (e.g., where to get the code, what is the licence of the release, and how are the authors facilitating people adopting the toolkit), and (iii) I am uncertain if ICLR is the right venue for the manuscript to obtain envisioned impacts of the conclusion section of the paper — I would have seen this contribution published in a medical journal instead, and the lacking details related to the item (ii) before make it even harder to assess this paper. However, the submission is excellent, and the program committee should discuss this case further.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RvLeelbIlfT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe authors present a new package aimed at improving the design and validation of pipelines using medical time series data. The pipeline covers many aspects of time series pipelines including pre-processing, prediction, treatment effect estimation, calibration, etc. The package, as depicted in the paper, appears to be very comprehensive and well motivated.\n\n##########################################################################\n\nRationale for score: \n\nThe primary reason for my recommendation of reject is this paper is ill-suited for a venue such as ICLR. The paper is a descriptive paper for a new software package and it is aimed at the healthcare/informatics sub-community. While I appreciate the considerable effort the authors have clearly put into this software package, I think the paper would have a better chance of reaching it's intended audience in a more focused venue such as the Journal of the American Medical Informatics Association (JAMIA), the Machine Learning for Healthcare Conference (MLHC), or the Journal of Statistical Software (JSS). I also believe that the ICLR format does a disservice to the authors, as 8 pages is not enough room to fully elaborate on their work and the current version is very compressed which makes for difficult reading. Given the space constraints, the paper feels more like an advertisement for the package vs. an elaboration and explanation document. I think there is an impressive amount of work on display here, but I think that ultimately a paper such as this would be better served in a different venue. \n \n##########################################################################\n\nPros: \n\n1. The pipeline, as it is proposed in the paper is quite impressive. Moreover, the authors do an excellent job at motivating the need for such a pipeline not only as a tool but as a means of standardization and benchmarking, something that is sorely needed in many healthcare applications of machine learning. \n\n2. The figures and table 1 do a good job at summarizing the proposed approach and existing alternatives.\n\n##########################################################################\n\nCons: \n\n1. There are no empirical evaluations against alternative methods in this paper. At minimum there should be some kind of head to head evaluations against the existing packages.\n \n2. I found the vignettes too condensed to be helpful and it's unclear how the proposed pipeline was used to produce the results in the tables. The authors do a good job at setting up the clinical problem, but (likely due to space constraints) it is unclear how the problem reduces to a series of pipeline steps. Having one fully worked example with code (even in the appendix) would greatly help to understand how the proposed pipeline works.\n\n3. Several basic details are missing from the paper. For example, is this a python package? From the code snippets, I assume it is but this is never stated in the paper. If it is a python package, what versions of python is it compatible with, what are the dependencies, is GPU acceleration supported, etc? I was able to find some of this by digging through the included code but these details should be included in the paper.\n\n4. Many acronyms are used but never defined in the text, e.g. CRN and R-MSN from the examples. If, as the authors claim, they would like their package to be used by clinicians and ML practitioners alike, the should define these acronyms in the text to aid the reader. \n\n5. Are all of these modules complete or are some still in the alpha phase of development?", "extra_scores": {}, "extra_text": {}}, {"review_id": "QAIL2GPu7tL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Very interesting and timely project. \n\nMajor concern:\nFor a well-planned model development with large enough dataset, it is recommended to separate the case and controls from the very beginning and apply the pre-processing and imputation on the training dataset only. Feature selection should also be based on the training data alone, which is not the case in this pipeline from my understanding. The way, the pipeline is described, the processing, including imputation and feature selection are performed on the full dataset, which is then passed to the modeling phase. During the modeling phase, where model training and validation and testing will be performed.  \n\nThis can be cause of over-fitting since the testing dataset was to some level \"seen\" before the model testing. This has to be stated in the limitation of the study design. \n\nHaving a team science approach with clinician scientists as part of the team is integral part of the study, which seems that this paper is all about.  \n\n\n\nMinor:\nSome grammatical /stylistic error. ex: “While issues such as data cleaning, algorithmic fairness, and privacy and heterogeneity have import, they are beyond the scope of our software.”  revise the sentence “have import”.\n\n\nFinally, I am not a software engineer and will leave that level of evaluation to my colleagues.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EmxX_YVHkCJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors showcase a pipeline intended to standardize and industrialize AI model development and testing for medical time series.\n\nIt is very clear that the authors have put in a tremendous amount of work in building their pipeline. As someone who works on ML for healthcare, I appreciate it and look forward to using such pipelines.\n\nAs far as the *paper* is concerned, however, I’m not sure if the paper (as it is written) and the venue (ICLR) are a good fit. For papers describing such frameworks, I’d prefer to read about:\n1. pipeline design and tradeoffs: why did the team make the decisions they did . For e.g. why did they decide to design the API interface the way they did, why did they decide to offer some ML techniques over others, what are the tradeoffs between standardization and flexibility for using such a pipeline, when would a user use Clairvoyance over starting fresh.\n2. benefits of using this pipeline over other pipelines or no pipeline: the authors benchmarked their pipeline’s performance over off-the-shelf ML models for some tasks, which is great! It would also be good to see the benefits of using Clairvoyance in terms of time to setup, time to train, etc.\n3. user interviews, feedback, and adoption to demonstrate how quickly new users can learn this framework and the benefits they observe while using it\n\nI look forward to hearing back from the authors and I’m open to changing my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses a pressing problem for applications involving clinical time series and introduce a pipeline that handle many of the issues pertaining to data preprocessing.

An important contribution is the software that makes the processing more seamless, which will, without a doubt, be useful to the community given the need for reproducibility.

The authors have responded suitably to reviewer comments with the main 'leftover criticism' being that such a paper may not be the best fit for ICLR. This isn't a typical paper. However, something that introduces this level of automation and flexibility in handling time series has not been presented at this conference (or other ML conferences) to the best of my knowledge. It seems it could work in conjunction (as opposed to competing) with any new time series models/techniques that may be introduced.
Decision: Accept (Poster)