Clairvoyance: A Pipeline Toolkit for Medical Time Series

Python Software Repository : https : //github.com/vanderschaarlab/clairvoyance 1 INTRODUCTION . Inference over time series is ubiquitous in medical problems [ 1–7 ] . With the increasing availability and accessibility of electronic patient records , machine learning for clinical decision support has made great strides in offering actionable predictive models for real-world questions [ 8 , 9 ] . In particular , a plethora of methods-based research has focused on addressing specific problems along different stages of the clinical data science pipeline , including preprocessing patient data [ 10 , 11 ] , imputing missing measurements [ 12–16 ] , issuing diagnoses and prognoses of diseases and biomarkers [ 17–25 ] , estimating the effects of different treatments [ 26–31 ] , optimizing measurements [ 32–36 ] , capturing ∗Authors contributed equally uncertainty [ 37–41 ] , and interpreting learned models [ 42–46 ] . On the other hand , these component tasks are often formulated , solved , and implemented as mathematical problems ( on their own ) , resulting in a stylized range of methods that may not acknowledge the complexities and interdependencies within the real-world clinical ML project lifecycle ( as a composite ) . This leads to an often punishing translational barrier between state-of-the-art ML techniques and any actual patient benefit that could be realized from their intended application towards clinical research and decision support [ 47–51 ] . Three Challenges To bridge this gap , we argue for a more comprehensive , systematic approach to development , validation , and clinical utilization . Specifically , due to the number of moving pieces , managing real-world clinical time-series inference workflows is challenging due the following concerns : • First and foremost , the engineering problem is that building complex inference procedures involves significant investment : Over 95 % of work in a typical mature project is consumed by software technicals , and < 5 % addressing real scientific questions [ 52 ] . As a clinician or healthcare practitioner , however , few resources are available for easily developing and validating complete workflows . What is desired is a simple , consistent development and validation workflow that encapsulates all major aspects of clinical time-series ML—from initial data preprocessing all the way to the end . • Second , the evaluation problem is that the performance of any component depends on its context ; for instance , the accuracy of a prediction model is intimately tied to the data imputation method that precedes it [ 13 , 14 ] . As an ML researcher , however , current empirical practices typically examine the merits of each component individually , with surrounding steps configured as convenient for ensuring “ all else equal ” conditions for assessing performance . What is desired is a structured , realistic , and reproducible method of comparing techniques that honestly reflects interdependencies in the gestalt . • Lastly , the efficiency problem is that sophisticated designs tend to be resource-intensive to optimize , and state-of-the-art deep learning approaches require many knobs to be tuned . As a clinical or ML practitioner alike , this computational difficulty may be compounded by pipeline combinations and the potential presence of temporal distribution shifts in time-series datasets [ 53 ] . What is desired is a platform on which the process of pipeline configuration and hyperparameter optimization can be automated—and through which new optimization algorithms to that effect may be built and tested . Contributions We tackle all three issues simultaneously . The Clairvoyance package is a unified , end-to-end , autoML-friendly pipeline for medical time series . ( i ) As a software toolkit , it enables development through a single unified interface : Modular and composable structures facilitate rapid experimentation and deployment by clinical practitioners , as well as simplifying collaboration and code-sharing . ( ii ) As an empirical standard , it serves as a complete experimental benchmarking environment : Standardized , end-to-end pipelines provide realistic and systematic context for evaluating novelties within individual component designs , ensuring that comparisons are fair , transparent , and reproducible . ( iii ) Finally , as an interface for optimization over the pipeline abstraction , Clairvoyance enables leveraging and developing algorithms for automatic pipeline configuration and stepwise selection , accounting for interdependencies among components , hyperparameters , and time steps . Through illustrative examples on real-world medical datasets , we highlight the applicability of the proposed paradigm within personalized prediction , personalized treatment planning , and personalized monitoring . To the best of our knowledge , Clairvoyance is the first coherent effort to demonstrate viability of a comprehensive , structured , and automatable pipeline for clinical time-series learning . 2 THE CLAIRVOYANCE PIPELINE . The Patient Journey Consider the typical patient ’ s interactions with the healthcare system . Their healthcare lifecycle revolves tightly around ( 1 ) forecasting outcomes of interest ( i.e . the prediction problem ) , ( 2 ) selecting appropriate interventions ( i.e . the treatment effects problem ) , and ( 3 ) arranging followup monitoring ( i.e . the active sensing problem ) . Each of these undertakings involve the full complexity of preparing , modeling , optimizing , and drawing conclusions from clinical time series . Clairvoyance provides model pathways for these core tasks in the patient journey ( see Figure 1 ) —integrated into a single pipeline from start to finish ( see Figure 2 ) . Formally , these pathways include : • Predictions Path . Let { ( sn , x1 : Tn ) } Nn=1 denote any medical time-series dataset , where sn is the vector of static features for the n-th patient , and x1 : Tn . = { xn , t } Tnt=1 is the vector sequence of temporal features . One-shot problems seek to predict a vector of labels yn from ( sn , xn,1 : Tn ) : e.g . prediction of mortality or discharge , where yn ∈ { 0 , 1 } . Online problems predict some target vector yn , t from ( sn , xn,1 : t ) at every time step : e.g . τ -step-ahead prediction of biomarkers yn , t ⊆ xn , t+τ . • Treatment Effects Path . For individualized treatment-effect estimation [ 26–31 ] , we additionally identify interventional actions an , t ⊆ xn , t at each time step ( e.g . the choices and dosages of prescribed medication ) , as well as corresponding measurable outcomes yn , t ⊆ xn , t+τ . The learning problem now consists in quantifying the ( factual or counterfactual ) potential outcomes yn , t+τ that would result from any specific sequence of interventions and patient covariates ( sn , xn,1 : t , an,1 : t ) . • Active Sensing Path . In addition to mapping ( already-measured ) covariates to targets , the very decision of what ( and when ) to measure is also important under resource constraints . In medical settings , active sensing deals with balancing this trade-off between information gain and acquisition costs [ 32–36 ] . With reference to some downstream task ( e.g . predicting yn , t+1 ) , the aim is to select a subset of covariates Kn , t at each t to maximize the ( net ) benefit of observing { xn , t , k } k∈Kn , t . As a Software Toolkit Engineering complete medical time-series workflows is hard . The primary barrier to collaborative research between ML and medicine seldom lies in any particular algorithm . Instead , the difficulty is operational [ 6 , 48 , 54 ] —i.e . in coordinating the entire data science process , from handling missing/irregularly sampled patient data all the way to validation on different popu- Active Sensing . Treatment Eff .. Predictions . `` `` Configure Data Preprocessing '' '' preprocessing = PipelineComposer ( FilterNegative ( ... ) , OneHotEncoder ( ... ) , Normalizer ( ... ) , ... ) '' '' Configure Problem Specification '' '' specification = ProblemMaker ( problem_class= ‘ online ’ , max_seq_len=24 , label= [ ‘ ventilator ’ ] , treatment=None , window=4 , ... ) '' '' Configure Data Imputation '' '' imputation = PipelineComposer ( Imputation ( type= ‘ static ’ , model_name= ‘ ... ’ , ... ) , Imputation ( type= ‘ temporal ’ , model_name= ‘ ... ’ , ... ) ) '' '' Configure Feature Selection '' '' feature_selection = PipelineComposer ( FeatureSelection ( type= ‘ static ’ , model_name= ‘ ... ’ , ... ) , FeatureSelection ( type= ‘ temporal ’ , model_name= ‘ ... ’ , ... ) ) '' '' Configure Pathway Model '' '' prediction_model = Prediction ( model_name= ‘ ... ’ , parameter_dict= { ... } , ... ) '' '' Load Datasets '' '' data_train , data_test = DataLoader.load ( static_dir= ‘ ... ’ , temporal_dir= ‘ ... ’ , ... ) , DataLoader.load ( static_dir= ‘ ... ’ , temporal_dir= ‘ ... ’ ) '' '' Execute Pipeline '' '' for component in [ preprocessing , specification , imputation , feature_selection ] : data_train = component.fit_transform ( data_train ) data_test = component.transform ( data_test ) prediction_model.fit ( data_train , ... ) test_output = prediction_model.predict ( data_test , ... ) Figure 3 : Illustrative Usage . A prototypical structure of API calls for constructing a prediction pathway model . Clairvoyance is modularized to abide by established fit/transform/predict design patterns . ( Green ) ellipses denote additional configuration ; further modules ( treatments , sensing , uncertainty , etc . ) expose similar interfaces . lations [ 4 , 55–60 ] . Clairvoyance gives a single unified roof under which clinicians and researchers alike can readily address such common issues—with the only requirement that the data conform to the standard EAV open schema for clinical records ( i.e . patient key , timestamp , parameter , and value ) . Under a simple , consistent API , Clairvoyance encapsulates all major steps of time-series modeling , including ( a . ) loading and ( b . ) preprocessing patient records , ( c. ) defining the learning problem , handling missing or irregular samples in both ( d. ) static and ( e. ) temporal contexts , ( f. ) conducting feature selection , ( g. ) fitting prediction models , performing ( h. ) calibration and ( i . ) uncertainty estimation of model outputs , ( j . ) applying global or instance-wise methods for interpreting learned models , ( k. ) computing evaluation metrics , and ( l. ) visualizing results . Figure 2 shows a high-level overview of major components in the pipeline , and Figure 3 shows an illustrative example of usage . All component modules are designed around the established fit-transform-predict paradigms , and the modeling workflow is based around a single chain of API calls . In this manner , each stage in the pipeline is extensible with little effort : Novel techniques developed for specific purposes ( e.g . a new state-of-the-art imputation method ) can be seamlessly integrated via simple wrappers ( see Appendix G for an example of how this can be done for any existing method , e.g . from sklearn ) . This stepwise composability aims to facilitate rapid experimentation and deployment for research , as well as simplifying collaboration and code-sharing . Package documentation/tutorials give further software details . As an Empirical Standard Evaluating any algorithm depends on its context . For instance , how well a proposed classifier ultimately performs is invariably coupled with the upstream feature-selection method it is paired with [ 44 ] . Likewise , the accuracy of a state-of-the-art imputation method can not be assessed on its own : With respect to different downstream prediction models , more sophisticated imputation may actually yield inferior performance relative to simpler techniques [ 13 , 14 ] —especially if components are not jointly optimized [ 15 ] . While current research practices typically seek to isolate individual gains through “ all-else-equal ” configurations in benchmarking experiments , the degree of actual overlap in pipeline configurations across studies is lacking : There is often little commonality in the datasets used , preprocessing done , problem types , model classes , and prediction endpoints . This dearth of empirical standardization may not optimally promote practical assessment/reproducibility , and may obscure/entangle true progress . ( Tables 6–7 in Appendix A give a more detailed illustration ) . Clairvoyance aims to serve as a structured evaluation framework to provide such an empirical standard . After all , in order to be relevant from a real-world medical standpoint , assessment of any single proposed component ( e.g . a novel ICU mortality predictor ) can—and should—be contextualized in the entire end-to-end workflow as a whole . Together , the ‘ problem-maker ’ , ‘ pipeline-composer ’ , and all the pipeline component modules aim to simplify the process of specifying , benchmarking , and ( self- ) documenting full-fledged experimental setups for each use case . At the end of the day , while results from external validation of is often heterogeneous [ 2 , 59 , 61 ] , improving transparency and reproducibility greatly facilitates code re-use and independent verification [ 54 , 56 , 57 ] . Just as the “ environment ” abstraction in OpenAI Gym does for reinforcement learning , the “ pipeline ” abstraction in Clairvoyance seeks to promote accessibility and fair comparison as pertains medical time-series . ( a ) Example : SASH ‘ decomposed ’ as SMS ( fulfilled here by DKL ) followed by combiner ( stacking ensemble ) : ( b ) Example : SPSC ‘ decomposed ’ as PSC ( fulfilled here by SKL ) followed by SMS ( fulfilled here by DKL ) : As an Optimization Interface Especially in cross-disciplinary clinical research—and during initial stages of experimentation— automated optimization may alleviate potential scarcity of expertise in the specifics of design and tuning . The Clairvoyance pipeline abstraction serves as a software interface for optimization algorithms—through which new/existing techniques can be applied , developed , and tested in a more systematic , realistic setting . In particular , by focusing on the temporal aspect of medical time series , this adds a new dimension to classes of autoML problems . Briefly ( see Figure 5 ) , consider the standard task of hyperparameter optimization ( for a given model ) [ 62 ] . By optimizing over classes of algorithms , the combined algorithm selection and hyperparameter optimization ( “ CASH ” ) problem [ 63–65 ] has been approached in healthcare settings by methods such as progressive sampling , filtering , and fine-tuning [ 50 , 66 ] . By further optimizing over combinations of pipeline components , the pipeline selection and configuration ( “ PSC ” ) problem [ 67 ] has also been tackled in clinical modeling via such techniques as fast linear search ( “ FLASH ” ) [ 68 ] and structured kernel learning ( “ SKL ” ) [ 67 , 69 ] . Now , what bears further emphasis is that for clinical time series , the temporal dimension is critical due to the potential for temporal distribution shifts within time-series data—a common phenomenon in the medical setting ( we refer to [ 53 , 70 , 71 ] for additional background ) . Precisely to account for such temporal settings , the stepwise model selection ( “ SMS ” ) problem [ 71 ] has recently been approached by such methods as relaxed parameter sharing ( “ RPS ” ) [ 53 ] as well as deep kernel learning ( “ DKL ” ) [ 71 , 72 ] . Further , what the pipeline interface also does is to naturally allow extending this to define the stepwise algorithm selection and hyperparameter optimization ( “ SASH ” ) problem , or even—in the most general case—the stepwise pipeline selection and configuration ( “ SPSC ” ) problem . Although these latter two are new—and clearly hard—problems ( with no existing solutions ) , Figure 4 shows simple examples of how the interface allows minimally adapting the SMS and PSC sub-problems ( which do have existing solutions ) to form feasible ( approximate ) solutions . Two distinctions are due : First , Clairvoyance is a pipeline toolkit , not an autoML toolkit . It is not our goal to ( re- ) implement new/existing optimization algorithms—which abound in literature . Rather , the standardized interface is precisely what enables existing implementations to be plugged in , as well as allowing new autoML techniques to be developed and validated within a realistic medical pipeline . All that is required , is for the optimizing agent to expose an appropriate ‘ optimize ’ method given candidate components , and for such candidates to expose a ‘ get-hyperparameter-space ’ method . Second—but no less importantly—we must emphasize that we are not advocating removing human oversight from the healthcare loop . Rather , the pipeline simply encourages systematizing the initial development stages in clinical ML , which stands to benefit from existing literature on efficient autoML techniques . Design Principles Our philosophy is based on the authors ’ experience in prototyping and developing real-world collaborative projects in clinical time-series . • Pipeline First , Models Second : Our first emphasis is on reproducibility : The process of engineering and evaluating complete medical time-series workflows needs to be clear and transparent . Concretely , this manifests in the strict “ separation of concerns ” enforced by the highlevel API of each component module along the pipeline ( see e.g . Figure 3 ) . With the ‘ problem-maker ’ and ‘ problem-composer ’ as first-class objects , the central abstraction here is the pipeline itself , while the intricacies and configurations of individual model choices ( e.g . a specific deep learning temporal imputation method ) are limited to within each component module . • Be Minimal and Unintrusive : Our second emphasis is on standardization : While workflow development needs to be unified and systematic , learning to use the framework should be intuitive as well . Concretely , this manifests in the API ’ s adherence to the existing and popular ‘ fit-transform-predict ’ paradigm ( see e.g . sklearn ) in all component modules—both ‘ along ’ the pipeline steps , as well as ‘ across ’ the pathways that define the patient ’ s healthcare lifecycle ( see Figure 2 ) . This enables easy adoption and rapid prototyping—qualities that are paramount given the degree of collaborative research and cross-disciplinary code-sharing required in healthcare-related research . • Encourage Extension : Our third emphasis is on extensibility : Given that novel methods are proposed in the ML community every day , the pipeline components should be easily extensible to incorporate new algorithms . Concretely , this manifests in the encapsulated design for models within each component module : Specifically , in order to integrate a new component method ( e.g . from another researcher ’ s code , or from an external package ) into the framework , all that is required is a simple wrapper class that implements the ‘ fit ’ , ‘ predict ’ , and ‘ get-hyperparameter-space ’ methods ; likewise , for an optimization agent ( see subsection on optimization interface below ) , all that is required is to expose an ‘ optimize ’ method . Worked Examples For a discussion of the choice of built-in techniques to include with the initial release , see Appendix C. Appendix E gives a worked example of using the Clairvoyance pipeline to train and use a model in a standard setting ( for this , we use the predictions pathway ) . Appendix F gives a worked example of using the optimization interface to perform stepwise model selection ( for this , we use the treatment effects pathway for variety ) . Appendix G gives an example of how a generic wrapper can be written for integrating an external model/algorithm that is not already implemented in the current version of Clairvoyance . Finally , the software repository contains Jupyter notebooks and top-level API code with examples of pathways and optimization .

Summary: The authors present a new package aimed at improving the design and validation of pipelines using medical time series data. The pipeline covers many aspects of time series pipelines including pre-processing, prediction, treatment effect estimation, calibration, etc. The package, as depicted in the paper, appears to be very comprehensive and well motivated.