ACES: Automatic Cohort Extraction System for Event-Stream Datasets

## Abstract

Abstract Reproducibility remains a significant challenge in machine learning (ML) for healthcare. Datasets, model pipelines, and even task/cohort definitions are often private in this field, leading to a significant barrier in sharing, iterating, and understanding ML results on electronic health record (EHR) datasets. This paper addresses a significant part of this problem by introducing the Automatic Cohort Extraction System (ACES) for event-stream data. This library is designed to simultaneously simplify the development of task/cohorts for ML in healthcare and also enable the reproduction of these cohorts, both at an exact level for single datasets and at a conceptual level across datasets. To accomplish this, ACES provides (1) a highly intuitive and expressive configuration language for defining both dataset-specific concepts and dataset-agnostic inclusion/exclusion criteria, and (2) a pipeline to automatically extract patient records that meet these defined criteria from real-world data. ACES can be automatically applied to any dataset in either the Medical Event Data Standard (MEDS) or EventStreamGPT (ESGPT) formats, or to any dataset in which the necessary task-specific predicates can be extracted in an event-stream form. ACES has the potential to significantly lower the barrier to entry for defining ML tasks that learn representations, redefine the way researchers interact with EHR datasets, and significantly improve the state of reproducibility for ML studies in this modality. ACES is available at: https://github.com/justin13601/aces . A short video demonstration of ACES is available at: https://youtu.be/i_hCaHDydqA .

## 1 Introduction

Machine learning (ML) for healthcare suffers from a severe and systemic reproducibility crisis (McDermott et al., ). This challenge is further exacerbated by the need to maintain private and secure datasets, but even with public datasets, ML pipelines are not reliably reproducible from published papers alone. For instance, in numerous attempts to reproduce ML for healthcare studies using the MIMIC-III dataset (Johnson et al., ), Johnson et al. found that more than half the time, the cohorts described in the studies could not be reliably reconstructed (Johnson et al., ).

This burden in reproducing even the basic task and problem definitions in ML for healthcare studies is profoundly detrimental. Beyond the obvious concerns it raises around the robustness of reported results and their readiness for deployment, our inability to reliably define shared, canonical, reproducible task definitions limits our capacity to perform meaningful model comparisons during methodological development. This is particularly notable in settings where not all researchers have mutual access to all datasets, as is common in healthcare. Given the critical role that open benchmarks play in the advancement of ML methods (Zhang & Hardt, ; Salaudeen & Hardt, ; Shirali et al., ), this deficit directly translates to a significant barrier in our ability, as a research community, to effectively experiment, iterate, and develop new ML methodologies in the healthcare space.

Given the clear import of this problem, the research community has naturally explored a number of prospective solutions. These can be largely categorized into two areas: (1) leveraging existing common data models (CDMs) to define reproducible task cohorts only for datasets within these schemas, and (2) defining static benchmarking tasks on individual public datasets. Both of these areas have generated numerous high-impact works. For example, in the area of CDM-driven tools, systems such as the ATLAS tool (OHDSI et al., ) for OHDSI’s OMOP CDM (OHDSI, ) and i2b2’s query tool for the i2b2 CDM (Murphy et al., ), as well as institution-specific tools such as the Stanford Research Repository (STARR) OMOP system and the NIHR Infections in Oxfordshire Database (IORD) cohort discovery platform have all been used to drive numerous new lines of inquiry. Unfortunately, these tools are also extremely limited in that they can only apply to the specific CDM or institutional data warehouse for which they have been defined. Further, because many of these CDMs have had limited penetrance into healthcare’s high-capacity, deep learning ecosystems, they are particularly ill-posed for task and cohort extraction within the healthcare deep learning communities. Conversely, public static benchmarks (Harutyunyan et al., ; McDermott et al., ; van de Water et al., ) over datasets such as MIMIC-IV (Johnson et al., ) or eICU (Pollard et al., ) have also been extremely impactful for the ML community. However, they are all tied to only a single or small set of datasets and tasks. Given the highly dynamic nature of clinical data and healthcare requirements, this is insufficient for the benchmarking and reproducibility needs faced by the ML for healthcare community.

When considering these existing solutions alongside the realities of healthcare data access and methodological development, it is clear that they are insufficient for three key reasons:

#### 1. The Need for Interoperability

The limited public datasets and only partially used CDMs cannot capture the diverse clinical populations, needs, and model capacities necessary for tangible ML progress in healthcare. To address this, systems for automated task extraction must be meaningfully interoperable across both public and private datasets with diverse input schemas.

#### 2. The Need for Flexibility

A single, static benchmark cannot encompass the variety of clinical tasks relevant to clinicians and informaticians. As existing tools (with only limited interfaces to define queries using per-set vocabularies) may struggle to generalize to new clinical tasks, ideal solutions must be sufficiently flexible to accommodate a myriad of new task definitions, criteria formats, and disease or deployment areas.

#### 3. The Need for Accessibility, Usability, & Applicability in Deep Learning Workflows

While many existing tools present with no-code interfaces (e.g., web platforms to build queries) that are essential for less technically-literate audiences, integrating such tools with deep learning workflows can prove challenging. Deep learning systems are often run in a semi-programmatic manner on siloed, private computational clusters where researchers have minimal control. Hence, existing tools can cause significant hindrance. Instead, successful software must be able to provide a Python and command-line interface (CLI) that offer significant ease of use to deep learning researchers, alongside shareable and readable configuration files that specify task definitions in a manner that can be readily ported across datasets and environments.

#### Our Solution: Automatic Cohort Extraction System for Event-Stream Datasets

In this work, we solve these problems with the Automatic Cohort Extraction System for Event-Stream Datasets (ACES). ACES offers a simple, expressive, and shareable configuration language for task definitions, as well as a reliable command-line interface and Python-accessible library for extracting labeled task dataframes (Figure 1).

Task definitions in ACES are naturally separated into simple dataset-specific event predicates and dataset-agnostic inclusion/exclusion criteria, thereby permitting the same task to be used in a conceptually identical manner across diverse datasets. By requiring users to specify predicates to realize their ML tasks on their specific datasets, ACES allows users to produce precise, locally specific, validatable cohorts that harmonize only the data elements needed for their task, regardless of how their input dataset is aligned or misaligned with existing ontologies or CDMs. Further, for datasets that are fully harmonized (e.g., through OHDSI vocabularies), ACES predicate definitions can be re-used across datasets without any loss of utility. In this way, ACES accommodates diverse datasets at various levels of data harmonization in a flexible, transparent manner. Overall, this approach not only enhances reproducibility but also facilitates community collaboration on task definitions, inclusion/exclusion criteria, and evaluation metrics for specific clinical use cases.

In contrast to prior task-definition systems such as ATLAS, ACES makes minimal assumptions about the input data structure or source vocabularies. In particular, ACES can be run on any dataset, provided the necessary task-specific predicates can be pre-extracted in an ”event-stream” format(^1^11e.g., in a dataframe with columns for a subject ID, timestamp, and numerical predicate values.), and can further be run from raw data directly for any dataset in the relatively low-level and flexible Medical Event Data Standard (MEDS) (Arnrich et al., ) or EventStreamGPT (ESGPT) (McDermott et al., ) formats. To demonstrate the utility and flexibility of ACES, we also release a collection of task definitions (Section 4) based on prior ML for healthcare works at both the dataset-agnostic criteria level and with dataset-specific predicates for the widely-used MIMIC-IV dataset (Johnson et al., ).

In sum, ACES represents 3 key contributions:

- 1.
ACES defines a shareable, simple, and flexible task configuration language that can define diverse sets of prediction tasks for ML in healthcare on any event-stream dataset.
- 2.
ACES provides an easy-to-use library to automatically extract these tasks from diverse sources of real-world, structured, and longitudinal electronic health record (EHR) data.
- 3.
Through these advancements, ACES is poised to significantly advance the state of reproducibility, interoperability, and effective development of ML methods for healthcare.

In the rest of this work, we will do the following. First, in Section 2, we will go through ACES in more depth, beginning by illustrating the key concepts of ACES using a running example with ACES CLI. Next, in Section 3, we will discuss the ACES design principles and briefly overview its core algorithm. Subsequently, in Section 4, we will demonstrate the use of ACES on diverse problem areas over real-world data, before ultimately discussing the limitations and future roadmap of ACES in Section 5, and offering concluding thoughts in Section 6.

Figure: Figure 1: ML task cohort extraction process (A) with and (B) without ACES. Predicates are dataset-specific concepts that are needed to conceptually capture a machine learning task. Windows are temporal segments on a patient’s health record and are dataset-agnostic, as they are defined relative to the predicates. This distinction allows researchers to share the more complex task logic that is independent of datasets, enabling conceptual reproducibility for ML tasks in healthcare.
Refer to caption: x1.png

## 2 Automatic Cohort Extraction System for Event-Stream Datasets (ACES)

In this section, we introduce ACES, a novel automated task and cohort definition and extraction system that fills the key gaps in *interoperability*, *flexibility*, and *accessibility* left by the existing tools outlined in Section 1.

To use ACES and extract a cohort for downstream ML tasks, a user only needs to do the following simple steps:

#### 1. Install ACES:

A fully functional version of ACES is pushed to PyPI, and any user can easily install it by simply running pip install es-aces. All dependencies are automatically set up with no further actions needed.

#### 2. Define Dataset:

A dataset in a permitted format, such as MEDS, ESGPT, or as direct predicates, is required. More information on the data formats is available in Section 2.1.

#### 3. Define Task:

A task configuration file is required to define the task that the user wishes to extract. This configuration language is simple, clear, yet flexible, permitting users to rapidly share and iterate over task definitions for their clinical settings. Configuration specification is given in Section 2.2 and the Appendix.

#### 4. Run the ACES CLI:

ACES can be directly run from the command-line using:

Additional details about the possible command-line arguments are detailed in Section 2.3. ACES can also be used as a direct Python import, as detailed in Section 2.5.

#### 5. Get Outputs:

The resulting output from ACES is a single unified dataframe with all valid patient instances extracted according to task specifications. Users can subsequently leverage the returned columns with original patient identifiers and health record timestamps for downstream ML tasks and benchmark creation. ACES also returns additional columns for the task label, as well as summaries of each of the predicates over windows of the patient record. Further information is available in Section 2.4 and in the example output displayed in Figure 3.

Critically, after *only these five simple steps*, a user can immediately, reproducibly extract a full cohort from their source dataset that matches their task definition, and begin using this task for downstream representation learning. Due to space constraints, full technical details on all aspects of ACES and on the precise details of the recursive algorithm used to extract task cohorts given a configuration file are limited to the Appendix and the ACES public documentation, available here: [https://eventstreamaces.readthedocs.io/en/latest/technical.html](https://eventstreamaces.readthedocs.io/en/latest/technical.html). Please also refer to the PDF compilation of the technical documentation in the Supplementary Material (Pages 41-51).

### 2.1 Dataset Configurations

ACES is extremely flexible and can handle different data formats, including data.standard=meds (Arnrich et al., ), data.standard=esgpt (McDermott et al., ), or data.standard=direct, where event-stream predicate values are pre-extracted by the user from any given dataset schema. Other CDMs are interchangeable with these formats, such as OHDSI OMOP through the MEDS OMOP ETL(^2^22[https://github.com/Medical-Event-Data-Standard/meds_etl/tree/main](https://github.com/Medical-Event-Data-Standard/meds_etl/tree/main)).

The use of direct predicates for extraction from datasets in a format that ACES does not natively support still significantly reduces the burden on users. Simply creating predicates is much less cumbersome than either fully converting the dataset to a CDM in order to use existing tools like ATLAS or i2b2, or performing the entire task extraction from scratch by writing in-house dataframe querying code. This demonstrates the significant improvement in utility that ACES brings across diverse data schemas compared to existing tools.

### 2.2 Task Configurations

In ACES, tasks are specified through configuration files that define a collection of dataset-specific event predicates, which are simple functions evaluated on individual events within a structured event-stream dataset. Additionally, task criteria are defined in a dataset-agnostic manner through a collection of interrelated windows, which specify regions of a patient’s record and are constrained by certain relationships. Please see Figure 2 for an example of a task configuration.

Figure: Figure 2: Example of a configuration file for the binary prediction of in-hospital mortality 48 hours after admission. References to predicates and windows are italicized and bolded, respectively. (A) Dataset-specific task predicates. These concepts are needed to conceptually capture this task and are used as constraints and boundaries for windows of the patient record. For instance, in this example, the value of “ADMISSION” denotes a hospital admission event in the external source dataset. (B) A window of the task specifying the task inputs for downstream models. Suppose we’d like to use all historic patient data up to and including 24 hours past the admission. An arbitrary criterion requiring more than 5 records can then be placed on this window to ensure that the extracted cohort contains sufficient input data. (C) The trigger events for the task, which are hospital admissions as we’d like to make a mortality prediction for each admission. (D) A window of the task specifying a gap in the patient timeline. Suppose we’d like to set a minimum length of admission for our cohort (e.g., 48 hours). A temporal constraint (minimum window duration) of 48 hours could then be set to represent this requirement. (E) A window of the task specifying the task target, which is set from the end of (D) to the immediately subsequent “discharge” or “death” predicate. This creates our binary classes for the task (i.e., $discharge=0$, $death=1$). All windows are interrelated on the patient timeline, as shown by how each window references another in the configuration file.
Refer to caption: x2.png

### 2.3 Command-Line Interface Arguments

#### Hydra Arguments

The Hydra framework (Yadan, ) enhances the CLI by enabling flexible run configurations and argument parsing for cohort extractions. For instance, specific arguments are required to define the external source dataset for data loading. Depending on the chosen format (the data.standard argument), either the path to the data file (for meds or direct) or the path to the dataset directory (for esgpt) must be specified to indicate the external source data from which ACES will extract the cohort. Additionally, cohort_dir and cohort_name are essential for locating and loading the task configuration file, as well as for saving results and logging operational data.

#### Scaling to Large Datasets

An overview of the computational profile of ACES is available in Section 4.1. Additionally, for users dealing with large datasets, ACES can also be run over a collection of sharded files, extracting and storing the matching cohort for each shard individually in matching file paths. This can greatly increase computational efficiency by facilitating the processing of different shards in parallel via Hydra’s multi-run launchers(^3^33[https://hydra.cc/docs/1.0/plugins/joblib_launcher/](https://hydra.cc/docs/1.0/plugins/joblib_launcher/)).

### 2.4 Extraction Output

Finally, with a dataset configured for predicates and a task configuration, ACES will execute the extraction for the cohort and return a table where each row is a valid instance as per the criteria defined in the configuration file. Hence, each instance can be included in our cohort used for the downstream ML task.
At the most basic level, the table contains the patient identifiers of our cohort, a user-defined timestamp that indexes prediction time, and a task label derived from a user-specified predicate.
In addition, for each of the interrelated windows, a start and end timestamp is provided to segment the patient record, along with a summary of the number of predicates evaluated in said window. Please see Figure 3 for a sample output table.

### 2.5 Python Usage

In addition to the command-line tool, we also provide a Python interface to allow researchers to easily leverage ACES for cohort extraction in their code pipelines. A full tutorial is provided at: [https://eventstreamaces.readthedocs.io/en/latest/notebooks/tutorial_meds.html](https://eventstreamaces.readthedocs.io/en/latest/notebooks/tutorial_meds.html). Please also refer to the PDF compilation of the tutorial notebook in the Supplementary Material (Pages 31-36).

## 3 ACES Technical Design

#### Key Design Principles

ACES is built with the following key design principles in mind:

- 1.
Structured Medical Data as Event-Stream Data: ACES views structured medical data as “event-stream” data1, enabling the extraction of meaningful tasks and cohorts from continuous patient records.
- 2.
Compatibility with Any Dataset: ACES is designed to be compatible with any dataset, provided the necessary dataset-specific predicates are extracted and formatted as an event-stream (this step is automated for any dataset that can be expressed in the MEDS (Arnrich et al., ) or ESGPT (McDermott et al., ) formats)(^4^44Note: This includes datasets that can be converted into these formats using existing ETLs, such as OMOP datasets through the MEDS OMOP ETL.).
- 3.
Separation of Configuration Components: ACES clearly separates dataset-specific configuration components from dataset-agnostic configuration components, ensuring flexibility and ease-of-use.
- 4.
Simple Configuration Language: ACES leverages a simple configuration language to maximize portability and facilitate community feedback and iteration on ML tasks and environments.
- 5.
Parallelization for Computational Efficiency: ACES relies on parallelization over patient shards to handle large datasets efficiently, ensuring scalability and performance.

#### Algorithm Overivew

The key novel principle behind the ACES algorithm is that we decompose the task of determining whether a section of a patient’s record meets certain criteria specified in a configuration file into a series of recursive analyses. These analyses evaluate whether specific windows, defined relative to the realizations of other windows in the patient record, meet their criteria. By imposing only minimal constraints on the structure of the task configuration file, we can ensure that tasks can always be extracted using this recursive approach, thus forming the foundation of the entire ACES system. For more details, please see the Appendix and the technical documentation in the Supplementary Materials.

Figure: Figure 3: Sample output table from ACES. Continuing the example from Figure 2, we obtain this table after executing ACES extraction. The subject_id, index_timestamp, and label form the foundation of the defined cohort and are what the user would need, at a minimum, to perform downstream tasks. In addition to these three columns, there exists a column for each of the defined windows in the task configuration, with its value being a structured object containing information about that particular window, such as the start and end times, as well as a summary of predicate counts. As seen in the values of this table, the timestamp of the end of the input window is always 24 hours after the trigger timestamp for each sample (row), and contains more than 5 of _ANY_EVENT—exactly matching our previously defined task configuration.
Refer to caption: x3.png

## 4 Using ACES: A Repository of Task Configuration Examples

To demonstrate the flexibility and utility of ACES, we define and publicly release the task configurations described in Table 1, both with dataset-agnostic criteria and with dataset-specific predicate realizations based on the MEDS version of the public MIMIC-IV dataset. Additional usage profiles, tasks, and areas, as well as details of the task cohorts extracted from MIMIC-IV, can be found in the Appendix and documentation. These various tasks have been previously studied, and ACES will facilitate their conceptual reproducibility to encourage benchmarking efforts and ensure robustness in ML for healthcare.

**Table 1: A collection of sample configuration files for various common predictive tasks on MIMIC-IV (Johnson et al., ). These tasks can be easily generalized to other datasets, such as e-ICU (Pollard et al., ) or other private intensive care unit (ICU) and inpatient datasets by simply swapping out appropriate predicate definitions. pastecode.io links have been provided.**
| Task Name | Description |
| --- | --- |
| [First 24h in-hospital mortality](https://pastecode.io/s/14qt3986) | Predict mortality within a hospital admission using the first 24 hours of data from that admission. |
| [First 48h in-hospital mortality](https://pastecode.io/s/2ta9vq7f) | Predict mortality within a hospital admission using the first 48 hours of data from that admission. |
| [First 24h in-ICU mortality](https://pastecode.io/s/ex5ruzga) | Predict mortality within an ICU admission using the first 24 hours of data from that admission. |
| [First 48h in-ICU mortality](https://pastecode.io/s/50q6soqs) | Predict mortality within a ICU admission using the first 48 hours of data from that admission. |
| [30d post-hospital-discharge mortality](https://pastecode.io/s/vhpjze41) | Predict mortality within 30 days of discharge. |
| [30d re-admission](https://pastecode.io/s/hc6m9qt7) | Predict hospital readmission within 30 days of discharge. |
| [Myocardial infarction 1-5Y phenotyping](https://pastecode.io/s/d8g8xqfd) | Predict myocardial infarction (MI) incidence 1-5 years after hospital admission. |
| [Reduced echo-derived LVEF 9m post-ECG](https://pastecode.io/s/snpjbp8j) | Predict reduced echo-derived left ventricular ejection fraction (LVEF) within 9 months of any ECG. |
| [CKD onset in diabetics 5Y from kidney panel](https://pastecode.io/s/wfhax5nf) | Predict chronic kidney disease (CKD) onset in diabetic patients within 5 years of any kidney panel laboratory test. |

### 4.1 Computational Profile

To establish an overview of the computational profile of ACES, a collection of various common tasks was queried on MIMIC-IV (Johnson et al., ) in the MEDS format. The MIMIC-IV MEDS schema has approximately 50,000 patients per shard with an average of approximately 80,500,000 total event rows per shard over seven shards.

**Table 2: Performance statistics for various common predictive tasks on a single MEDS shard of MIMIC-IV (Johnson et al., ), including the number of patients, the number of samples, total runtime (in seconds), and maximum memory usage (in MiBs).**
| Task | # Patients | # Samples | Total Time (s) | Max Memory (MiBs) |
| --- | --- | --- | --- | --- |
| First 24h in-hospital mortality | 20,971 | 58,823 | 363.09 | 106,367.14 |
| First 48h in-hospital mortality | 18,847 | 60,471 | 364.62 | 108,913.95 |
| First 24h in-ICU mortality | 4,768 | 7,156 | 216.81 | 39,594.37 |
| First 48h in-ICU mortality | 4,093 | 7,112 | 217.98 | 39,451.86 |
| 30d post-hospital-discharge mortality | 28,416 | 68,547 | 182.91 | 30,434.68 |
| 30d re-admission | 18,908 | 464,821 | 367.41 | 106,064.04 |
| Myocardial Infarction 1-5Y phenotyping | 3,329 | 8,319 | 198.04 | 33,427.70 |
| Reduced echo-derived LVEF 9m post-ECG | 14 | 17 | 210.02 | 35,385.79 |
| CKD onset in diabetics 5Y from kidney panel | 736 | 3,503 | 238.65 | 44,221.81 |

All experiments were executed on a Linux server with 36 cores and 340 GBs of RAM available. A single MEDS shard was used to provide a bounded computational overview of ACES, which is applicable even when scaled to larger datasets using Hydra3. For instance, if one shard costs $M$ memory and $T$ time, then $N$ shards may be executed in parallel with $N*M$ memory and $T$ time, or in series with $M$ memory and $T*N$ time.

## 5 Discussion

### 5.1 Additional Related Work

In addition to the existing tools discussed in Section 1, there are several other areas of related work relevant to ACES. Firstly, ACES could be directly connected with and support various health CDMs, such as OMOP, FHIR, PCORNET, and i2b2 (OHDSI, ; Bender & Sartipi, ; PCORnet, ; Murphy et al., ), rather than through ETLs. These models provide already-accepted standardized frameworks for organizing and analyzing healthcare data, and integrating ACES directly with them could greatly enhance its utility and interoperability. Similarly, frameworks such as DescEmb (Hur et al., ) and GenHPF (Hur et al., ) hold great synergistic potential with ACES, and we believe that they can be complementary in enabling new kinds of cross-dataset training, transfer learning, and evaluation. Static benchmarks that provide standardized datasets, evaluation metrics, and baseline methods for a range of clinical problems, such as YAIB (van de Water et al., ), Harutyunyan et al.’s multitask learning clinical prediction benchmarks (Harutyunyan et al., ), and EHR-PT (McDermott et al., ), can also be directly integrated with ACES to facilitate robust ML in healthcare. Lastly, ACES can be used in conjunction with various health data management tools, such as ESGPT (McDermott et al., ), TemporAI (Saveliev & van der Schaar, ), PyHealth (Yang et al., ), and OMOPLearn (Kodialam et al., ). These tools offer functionalities for pre-processing, managing, and analyzing health data for downstream tasks, and integrating ACES with them directly can streamline ML workflows.

Beyond healthcare, ACES is applicable to data from a variety of other domains, such as for financial, climate, or social media data—essentially, ACES could be used for any *structured*, *longitudinal* data that can be reformatted as an event-stream. This versatility makes ACES a powerful library for extracting and analyzing complex event-based datasets across different fields.

### 5.2 Limitations & Future Roadmap

ACES has several key limitations that can be addressed in future work. Firstly, while already very expressive, ACES’s task configuration language can still be further expanded. Expressing more complex kinds of predicates, window aggregations, labeling functions, and criteria would expand the scope of ACES significantly. ACES also seeks to provide direct support for cohort extraction based on unstructured data (notes and memos) in the future, though currently, such predicates need to be manually extracted by the user.

ACES is also very well poised to capture more complex patterns of task and cohort relationships, including prescribed systems of case/control matching, automated bias analyses, or propensity re-weighting over excluded populations. It is also possible to enable users to nest ACES configuration files to leverage extracted task labels as new predicates in more complex ACES extraction processes.

To enhance the scalability of ACES, we will ensure that the expanding MEDS standard will always be supported. Direct interoperability with existing resources in this space, in particular ATLAS and OHDSI vocabulary-derived cohort definitions, is a high priority area for future work.

Finally, with the standardization that ACES offers, new opportunities for human interaction with data are also made available, such as via a natural language interface to define ACES predicates or configuration files and, thus, to extract downstream tasks, patient cohorts, or derived datasets in a code-free manner on diverse input EHR formats. We aim to also explore the viability of leveraging large language models (LLMs) to directly format predicate and criteria definitions given a data dictionary; preliminary work on this front shows very promising results!

### 5.3 ACES for Enabling a New Kind of Benchmark

In addition to the clear impact of ACES on reproducibility, robustness, and accessibility of health datasets and ML for healthcare, we also feel that ACES is critical for a “new kind of benchmark” in the field—and, in so being, is a portent of what needs to come should ML for healthcare progress to a more productive, communal, and impactful stage.

In particular, we argue that for this field to progress in the manner desired by the community, and most likely to be maximally positively impactful for all patients, we need to develop methodologies to test, share, and develop ML solutions across diverse datasets in a meaningful and reproducible manner, even without said datasets being publicly available to general researchers. This capability is critical because, without it, we will never be able to offer new inductive insights about which methods are most likely to work best on novel, private data. In other words, if we cannot test our model training recipes across the diverse sets of clinical care settings, populations, and conceptual dataset schemas that exist in the real world, we similarly cannot expect those training recipes to generalize to said set of myriad downstream deployment areas.

Libraries like ACES, which make it as easy as possible for users to share the conceptual definitions of their tasks and prediction areas across datasets—in such a way that their colleagues can use them even over independent, private datasets—can help transform the kinds of benchmarking studies we can perform in ML for healthcare towards those that permit generalizable assessment of ML training recipes across datasets, clinical areas, and more.

## 6 Conclusion

In this work, we present the Automatic Cohort Extraction System for Event-Stream Datasets (ACES). ACES is a system designed to intuitively define cohorts and downstream tasks of interest for learning representations and reliably and automatically extract those cohorts from arbitrary datasets that are in an event-stream format. This system enables significantly greater shareability of task definitions, reproducibility of ML training and evaluation recipes, and is as easy to use as installing a package via pip and running a simple command-line tool. We feel that ACES will be integral in the development of new kinds of benchmarks in ML for healthcare, which can be explored across both public and private datasets alike, and characterize the populations and tasks of interest in a manner that cleanly separates dataset-specific components from shareable dataset-agnostic components. To learn more about ACES and use it today in your work, please visit our GitHub repository at: [https://github.com/justin13601/aces](https://github.com/justin13601/aces), and the online ACES Documentation at: [https://eventstreamaces.readthedocs.io/en/latest/index.html](https://eventstreamaces.readthedocs.io/en/latest/index.html).

## 7 Video Demonstration

To complement this paper and the documentation, we have provided a very short online video demonstration of ACES in action, available here: [https://youtu.be/i_hCaHDydqA](https://youtu.be/i_hCaHDydqA). It overviews the Python API and walks through a running example of extracting a cohort for an in-hospital mortality prediction task from a synthetic MEDS dataset. Additionally, it briefly covers several simple and complex real-world task configuration files.

## 8 Reproducibility Statement

We have provided links to the ACES library source code and pastecode.io snippets of configuration files used in the experiments. Reproducible tutorial notebooks and technical implementation details are included in the Supplementary Materials (Pages 31-36 and 41-51, respectively) and are accessible via the online public documentation at: [https://eventstreamaces.readthedocs.io/en/latest/](https://eventstreamaces.readthedocs.io/en/latest/).

## 9 Acknowledgments and Disclosure of Funding

MBAM gratefully acknowledges support from a Berkowitz Postdoctoral Fellowship at Harvard Medical School. JX greatly appreciates support from supervisors David W. Eyre (Big Data Institute, University of Oxford) and Curtis P. Langlotz (Center for Artificial Intelligence in Medicine & Imaging,
Stanford University). We also acknowledge contributions from Tom Pollard (Massachusetts Institute of Technology).

## Appendix A Documentation

The full ACES Online Documentation is available at: [https://eventstreamaces.readthedocs.io/en/latest/](https://eventstreamaces.readthedocs.io/en/latest/). We have also included a compiled PDF version of this documentation in the Supplementary Material.

To answer specific questions about ACES, please see the below index (links to the external online documentation, as well as references to chapters and page numbers of the ancillary PDF documentation are provided):

How do you use ACES?

- 1.
What is a task and how do you specify one?
Sample task descriptions and specifications are provided on the Task Examples page of the online documentation ([https://eventstreamaces.readthedocs.io/en/latest/notebooks/examples.html](https://eventstreamaces.readthedocs.io/en/latest/notebooks/examples.html)). Please refer to Chapter 3, Pages 21-26.
1.1.
What are predicates and how do you specify them?
For an overview of predicates and how they form the foundation of ACES, please refer to the Predicates DataFrame section ([https://eventstreamaces.readthedocs.io/en/latest/notebooks/predicates.html](https://eventstreamaces.readthedocs.io/en/latest/notebooks/predicates.html)) in Chapter 4, Pages 27-30.
1.2.
What are windows and how do you specify them?
A window in ACES is defined as a segment in the patient record ([https://eventstreamaces.readthedocs.io/en/latest/algorithm.html#window](https://eventstreamaces.readthedocs.io/en/latest/algorithm.html#window)). For details on how to define a window ([https://eventstreamaces.readthedocs.io/en/latest/readme.html#windows](https://eventstreamaces.readthedocs.io/en/latest/readme.html#windows)), please refer to Chapter 1.3.3, Page 9.
- 2.
How do you extract a task from a dataset?
For general ACES usage instructions ([https://eventstreamaces.readthedocs.io/en/latest/usage.html#quick-start](https://eventstreamaces.readthedocs.io/en/latest/usage.html#quick-start)), please refer to Chapter 2.1, Pages 13-16. Additionally, brief end-to-end instructions ([https://eventstreamaces.readthedocs.io/en/latest/readme.html#instructions-for-use](https://eventstreamaces.readthedocs.io/en/latest/readme.html#instructions-for-use)) are also available in Chapters 1.2 and 1.3, Pages 4-10.
2.1.
Detailed Usage Instructions for ACES CLI
For detailed instructions on using ACES CLI ([https://eventstreamaces.readthedocs.io/en/latest/usage.html#detailed-instructions](https://eventstreamaces.readthedocs.io/en/latest/usage.html#detailed-instructions)), please refer to the Usage Guide in Chapter 2.2, Pages 16-20.
2.2.
Tutorial for the ACES Python API
For a step-by-step tutorial on using the ACES Python API ([https://eventstreamaces.readthedocs.io/en/latest/notebooks/tutorial_meds.html](https://eventstreamaces.readthedocs.io/en/latest/notebooks/tutorial_meds.html)), please refer to the Code Example Notebook in Chapter 5, Pages 31-41.
- 3.
ACES with & vs. Other Tools
For an overview of how ACES could be used with other existing complementary tools for reproducible machine learning ([https://eventstreamaces.readthedocs.io/en/latest/readme.html#complementary-tools](https://eventstreamaces.readthedocs.io/en/latest/readme.html#complementary-tools)), please refer to Chapter 1.4.2, Page 10.
For an overview of ACES and other existing alternative tools for semi- or fully-automated cohort extraction ([https://eventstreamaces.readthedocs.io/en/latest/readme.html#alternative-tools](https://eventstreamaces.readthedocs.io/en/latest/readme.html#alternative-tools)), please refer to Chapter 1.4.3, Page 11.

How does ACES work?

- 1.
What is the formal configuration language specification for ACES?
For technical details on the ACES configuration language ([https://eventstreamaces.readthedocs.io/en/latest/configuration.html](https://eventstreamaces.readthedocs.io/en/latest/configuration.html)), please refer to the Configuration Language Specification section in Chapter 6.1, Pages 41-44.
- 2.
Glossary of ACES Terminology
For a glossary of terminology used throughout ACES ([https://eventstreamaces.readthedocs.io/en/latest/technical.html#algorithm-terminology](https://eventstreamaces.readthedocs.io/en/latest/technical.html#algorithm-terminology)), please refer to the Algorithm Terminology section in Chapter 6.3, Pages 46-47.
- 3.
What is the ACES extraction algorithm?
For technical details on the ACES algorithm ([https://eventstreamaces.readthedocs.io/en/latest/technical.html#algorithm-design](https://eventstreamaces.readthedocs.io/en/latest/technical.html#algorithm-design)), please refer to the Algorithm Design section in Chapter 6.4, Pages 48-51.
- 4.
Full ACES Module API Documentation
For the complete ACES module documentation ([https://eventstreamaces.readthedocs.io/en/latest/api/modules.html](https://eventstreamaces.readthedocs.io/en/latest/api/modules.html)), including doctests that ensure algorithm correctness, please refer to the Module API sections in Chapter 8, Pages 55-125.

How well does ACES work?

- 1.
Computational Profile
For an overview of the computational profile of ACES ([https://eventstreamaces.readthedocs.io/en/latest/profiling.html](https://eventstreamaces.readthedocs.io/en/latest/profiling.html)), please refer to the Computational Profile section in Chapter 7, Page 53.
- 2.
Further Examples
For additional examples of configuration files and criteria of different machine learning for health tasks, please refer to the MEDS-DEV benchmarking effort on GitHub ([https://github.com/mmcdermott/MEDS-DEV/tree/main](https://github.com/mmcdermott/MEDS-DEV/tree/main)).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: + How the two parts of "institution-specfic" and "generalizable" work along side each other? Besides standardizing the training part of ML models, how this work helps with what's discussed in 5.3 (i.e., helping with no access to private data).
soundness: 4
strengths: + The study is indeed very timely and sound. It addresses a critical area of concern in ML for health, by publishing offering a pipeline to standardize common cohort extraction tasks from the health datasets.
+ The paper does a nice job of providing public and anonymized resources. 
+ The study adopts several recent prior studies, and seems to nicely complement those.
summary: This study presents Automatic Cohort Extraction System (ACES) aiming to addresses the reproducibility in healthcare ML challenges due to the private nature of datasets and model definitions,  by providing a tool to define and reproduce ML cohorts across datasets. The tools are claimed to enable consistent cohort extraction by offering a configuration language and automatic pipeline, potentially improving reproducibility in ML studies on electronic health record data.
weaknesses: + One major unclear aspect is providing robust evidence that the proposed tool/pipeline indeed achieves a good performance extracting the right cohort. A natural question, how one can ensure this fairly complex procedure will not miss critical samples or include incorrect ones in the final results. Is there any way to compare the results against some sort of baseline or ground truth?
+ While the paper is relatively sound and straightforward, and while some examples are provided, the whole document still seems a bit unaccessible to even an average reader. The paper stays high-level, perhaps in favor generalizability, but risks missing the readers. Here are some example ways, the paper can be further accessible. Provide more concrete examples, especially by showing the outcome cohort. Present the concept of "data stream" in a more tangible and clear way in this application. Rely less on referring too much to appendix and external links. 
+ Relatedly, the main technical contribution can be further clarified. The recursive nature of the specified algorithm seems unclear. 
+ It is suggested that subjective and not substantive statements are removed. For instance, "preliminary work on this front shows very promising results!"

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 4
strengths: 1. The paper addresses a significant barrier in healthcare ML research: the complexity of EHR data, especially in cohort extraction. The motivation is clear and relevant.
2. The authors provide an open-source Python package with thorough documentation, which will support reproducibility and benefit future research.
3. ACES is built on the MEDS format, showing potential for broad applicability across various datasets and tasks.

**Note:** I think this library could offer value to ML in healthcare by simplifying EHR cohort extraction. However, the essential factor for such tools is flexibility and support across multiple datasets and tasks. Assessing these aspects thoroughly is challenging without extensive hands-on testing, so I have adjusted my confidence rating accordingly.
summary: This paper introduces ACES, an open-source Python library designed for event-stream data, aimed at automating the cohort extraction process through a unified API. Users define tasks via a configuration file specifying predicates, input, trigger, gap, and target, making the system both structured and adaptable.
weaknesses: A primary concern is the library’s utility for more complex tasks. For example, in the case of “CKD in diabetics within 5Y of kidney panel,” much of the effort lies in translating high-level criteria into specific medical features—a step that remains challenging and unresolved. Extracting cohorts from predefined features is relatively straightforward.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Have the authors compare the effort to ETL a new dataset to an existing standard (e.g. OMOP) and model multiple downstream tasks vs their proposed method? It would be useful to compare these timings. It is also known that in some situations all elements of a new dataset may not completely map to OMOP format exhaustively - is this a problem that the authors faced and they are trying to solve this challenge? Further comparative details on this would situate the paper better
soundness: 2
strengths: Overall this is a very interesting paper with some key strengths (but perhaps handicapped by some key weakness below). First considering the strengths we can identify the following

- The authors aim to improve standardization and interpretability in healthcare ML. While there have been many approaches to this including OMOP and i2b2, ACES aims to improve integration with potentially non-compliant sources by enabling reproducible task definitions across multiple datasets, which can aid in consistent benchmarking and model validation.
2. The authors propose a high level configuration language and support for Python and CLI interfaces facilitate broad adoption among researchers and compatibility with deep learning workflows. With this they aim to lower the barrier for integration
3. ACES's support for multiple data standards (like MEDS and ESGPT) and separation of dataset-specific and dataset-agnostic components allows flexible and adaptable cohort extraction across diverse clinical datasets.
summary: The paper introduces ACES, the Automatic Cohort Extraction System, designed to address reproducibility issues in machine learning for healthcare by streamlining task and cohort definitions in electronic health record (EHR) datasets. ACES provides a configuration language that supports dataset-specific and dataset-agnostic criteria, which enables sharing and reuse across datasets with minimal adjustments. It integrates with formats like MEDS and ESGPT, allowing users to define prediction tasks with flexibility, regardless of specific dataset schemas, and includes both Python and command-line interfaces. ACES aims to bridge gaps in interoperability, flexibility, and deep learning compatibility, potentially enhancing the reproducibility and accessibility of ML healthcare studies.
weaknesses: Given the promising aspects, they paper can be improved upon by addressing the following

1. First, the authors may have created the classic problem of `resolving n competing standards by ending up with n + 1 competing standards`. Specifically, despite claims of flexibility, ACES requires data to be formatted in supported structures, which could necessitate pre-processing for datasets outside MEDS or ESGPT. The authors doesn't describe in details the effort in taking a new dataset and making it compliant to MEDS/ESGPT. For OMOP, the authors mention about a connector - but the authors don't discuss whether the MEDS/ESGPT transformation are exhaustive and without data loss. 
2. Next moving on to the performance and scalability, while initial results are promising, more comprehensive evaluations on larger and more varied datasets would strengthen claims about ACES's efficiency and scalability. For example the authors claim these task specific transformations are cheaper than full scale ETL of datasets, the authors doesn't discuss whether it is cheaper in the long term, especially with multiple tasks. Specifically considering the eco-system of downstream processing like OHDSI ecosystem it might be cheaper to convert to OMOP. Even for deep learning systems several efforts like https://github.com/clinicalml/omop-learn and https://github.com/BiomedSciAI/DPM360 extends the ecosystem with low barriers for integrating with OMOP systems.  
3. Finally there are some limitations currently with ACES not able to directly handle unstructured data, which limits its application scope in tasks that require insights from clinical notes or similar records.


Overall while the authors have a novel solution it is not evident they are solving the right problem or who the solution is for. Identifying these two would enable the paper to be reviewed with the full context

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Please refer to points W1 through W3 for detailed concerns and recommendations.
soundness: 2
strengths: S1. The promotion of standardization and simplification in automatic cohort extraction is of considerable importance. This effort can enhance the reproducibility of machine learning applications in healthcare research. Furthermore, it has the potential to advance the digitalization of healthcare data, thereby providing substantial benefits to both clinicians and patients.

S2. The paper provides a relatively thorough overview of the current landscape of machine learning within healthcare research, including common data models, existing benchmarks, and representative applications/tasks.

S3. The ACES system is implemented as an open-source project, accompanied by comprehensive documentation.
summary: This paper concentrates on enhancing the reproducibility of machine learning for healthcare research and introduces the Automatic Cohort Extraction System (ACES), a library/tool specifically developed for handling healthcare data formatted as event streams. ACES functions as a configuration language that simplifies task definitions by abstracting them into two main components: dataset-specific event predicates and dataset-agnostic inclusion/exclusion criteria. This structured approach allows ACES to support a diverse array of representative predictive tasks essential for healthcare analytics.
weaknesses: W1. ACES primarily functions in healthcare analytics by performing preprocessing on broadly collected EHR data tailored to specific target applications, such as extracting particular features within a time window or performing aggregations. This process is ubiquitous in most EHR-related data mining tasks. ACES abstracts common dataframe transformation patterns across diverse healthcare applications, providing high-level encapsulation. Under the assumptions set by ACES, this approach simplifies the preprocessing to some extent. However, it is essential to note that this work can be seen as largely an engineering solution rooted in low-code programming concepts, with limited conceptual innovation. Additionally, it is challenging to discern any substantial transformative impact this encapsulation introduces to EHR data processing.

W2. Continuing from the previous point, ACES aims to implement a standard data extraction process for healthcare data in an event-stream format. However, the paper does not adequately articulate the specific challenges inherent to this process. This omission leaves readers uncertain of the unique problems ACES addresses, thereby diminishing the perceived contributions and impact of ACES.

W3. The paper lacks objective and quantitative evidence to substantiate the superiority of ACES over existing work aimed at solving the same research problem. This hence weakens the argument for ACES as a preferable solution in the competitive landscape of healthcare data processing and management technologies.

Meta-review: The paper introduces a new cohort extraction system for event-stream EHR datasets, designed to enable cohort replication across datasets, which in turn will facilitate the replication and comparison of analyses across datasets. The method encompasses a configuration language and an extraction pipeline. 

The problem addresses an important and insufficiently addressed problem in machine learning for healthcare. Most (3/4) of the reviewers have expressed the view that this paper brings a valuable contribution to the community. The reviewers appreciated how this work complements existing studies, that a well-documented python package is being released, and that it supports multiple data standards (like MEDS and ESGPT) seamlessly.

A single reviewer (z6xZ) expressed concerns that this is mainly an engineering effort, rather than conceptual innovation. However, the authors have rebutted this argument by pointing out that infrastructure development is within the scope of ICLR, and that ACES also defined a new language to facilitate access to EHR cohorts. I fully agree with the authors’ arguments in this.

Another point that was raised during the rebuttal was related to comparisons to existing cohort extraction tools. The authors have run additional experiments, showing that their method is generally more computationally efficient (in terms of runtime, and memory), as well as reducing the amount of code needed and time from an operator compared to existing techniques. I do find these results convincing, with the caveat that the human operators were familiar with the tool, and I do expect some training to be needed before it can be used effectively.

At any rate, I recommend acceptance of this paper, as it solves an important problem, provides an open-source framework, and has shown to be valuable in practical scenarios.

Decision: Accept (Poster)