ACES: Automatic Cohort Extraction System for Event-Stream Datasets
Authors: Justin Xu
Jack Gallifant
ALISTAIR JOHNSON
Matthew B.A. McDermott
Venue: ICLR 2025 Poster (2025)
Keywords: Automatic Task Specification
Cohort Extraction
Electronic Health Records
Open Source Software
Benchmarks
Datasets
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: A software library to standardize reproducible task and cohort extraction in machine learning for healthcare

Abstract: Reproducibility remains a significant challenge in machine learning (ML) for healthcare. Datasets, model pipelines, and even task or cohort definitions are often private in this field, leading to a significant barrier in sharing, iterating, and understanding ML results on electronic health record (EHR) datasets. We address a significant part of this problem by introducing the Automatic Cohort Extraction System (ACES) for event-stream data. This library is designed to simultaneously simplify the development of tasks and cohorts for ML in healthcare and also enable their reproduction, both at an exact level for single datasets and at a conceptual level across datasets. To accomplish this, ACES provides: (1) a highly intuitive and expressive domain-specific configuration language for defining both dataset-specific concepts and dataset-agnostic inclusion or exclusion criteria, and (2) a pipeline to automatically extract patient records that meet these defined criteria from real-world data. ACES can be automatically applied to any dataset in either the Medical Event Data Standard (MEDS) or Event Stream GPT (ESGPT) formats, or to *any* dataset in which the necessary task-specific predicates can be extracted in an event-stream form. ACES has the potential to significantly lower the barrier to entry for defining ML tasks in representation learning, redefine the way researchers interact with EHR datasets, and significantly improve the state of reproducibility for ML studies using this modality. ACES is available at: https://github.com/justin13601/aces.

Published as a conference paper at ICLR 2025
ACES: AUTOMATIC COHORT EXTRACTION SYSTEM FOR EVENT-STREAM DATASETS
Justin Xu
University of Oxford
justin.xu@nde ac.uk
Jack Gallifant
Massachusetts Institute of Technology jgallyemit.edu
Alistair E. W. Johnson
University of Toronto alistair. jocnson@utorontoo.co
Matthew B. A. McDermott
Harvard Medical School
wettthewwcdeeeee @hms harvard.
ABSTRACT
Reproducibility remains a significant challenge in machine learning (ML) for healthcare. Datasets, model pipelines, and even task or cohort definitions are of- ten private in this field, leading significant barrier in sharing, iterating, and understanding ML results on electronic health record (EHR) datasets. We address a significant part of this problem by introducing the Automatic Cohort Extrac- tion System (ACES) for event-stream data. This library is designed to simulta- neously simplify the development tasks and cohorts for ML healthcare and also enable their reproduction, both at exact level for single datasets and at conceptual level across datasets. To accomplish this, ACES provides: (1) a highly intuitive and expressive domain-specific configuration language for defining both dataset-specific concepts and dataset-agnostic inclusion or exclusion criteria, and (2) pipeline automatically extract patient records that meet these defined cri- teria from real-world data. ACES can be automatically applied to any dataset in either the Medical Event Data Standard (MEDS) Event Stream GPT (ES- GPT) formats, or to any dataset in which the necessary task-specifi predicates can be extracted in an event-stream form. ACES has the potential to significantly lower the barrier to entry for defining MI. tasks in representation learning, rede- fine the way researchers interact with EHR datasets, and significantly improve the state of reproducibility for MI. studies using this modality. ACES is available at: wtt. http://www....o.oonniooooo////1/
INTRODUCTION
Machine learning (ML) for healthcare suffers from a severe and systemic reproducibility crisis (Me- Dermott et al. 2021b). This challenge is further exacerbated by the need to maintain private and secure datasets, but even with public datasets, ML pipelines are not reliably reproducible from pub- lished papers alone. For instance, in numerous attempts to reproduce ML for healthcare studies using the MIMIC-III dataset (Johnson et al. 2016), Johnson et al. found that more than half the time, the cohorts described in the studies could not be reliably reconstructed. Specifically, experi- ments led to many discrepancies of up to 25% in cohort sizes, with one study reaching as high as 11,767 patients (Johnson al. 2017 This is primarily due sparse descriptions of cohorts in

---

Published as a conference paper at ICLR 2025
study methods with essential reproducibility details often omitted, along with the absence of openly available code.
This burden in reproducing even the basic task and problem definitions in ML for healthcare studies is profoundly detrimental (McDermott) 2025). Beyond the obvious concerns it raises around the robustness of reported results and their readiness for deployment, our inability to reliably define shared, canonical, and reproducible task definitions limits our capacity to perform meaningful model comparisons during methodological development. This is particularly notable in settings where not all researchers have mutual access to all datasets, as is common in healthcare, Given the critical role that open benchmarks play in the advancement of ML methods (Zhag & Hardi, 2024 Salaudeen & Hardt 2024: Shirali et al. (202),, this deficit directly translates to a significant barrier in our ability, as a research community, to effectively experiment, iterate, and develop new MI. methodologies in the healthcare space.
Given the clear import of this problem, the research community has naturally explored number of prospective solutions. These can largely be categorized into two areas: (1) leveraging existing com- mon data models (CDMs) to define reproducible task cohorts only for datasets within these schemas, and (2) defining static benchmarking tasks individual public datasets. Both of these areas have generated numerous high-impact works. For example, in the area of CDM-driven tools, systems such as the ATLAS tool (Gold et al., 2024) for OHDSI's OMOP CDM (Reich et al. 2024 and i2h2's PIC-SURE (Stedman et al.) 2024) for the i262 CDM (Murphy et al. 2010), as well as various institution-specine tools, have all been used to drive numerous new lines of inquiry. Unfortunately, these tools are also extremely limited in that they can only be applied to the specific CDM or insti- tutional data warchouse for which they have been defined. Further, because many of these CDMs have had limited penetrance into healthcare's high-capacity, deep learning ecosystems, they are par- ticularly ill- ill-posed for task and cohort extraction within the healthcare deep learning communities. Conversely, public static benchmarks (Harutyunyan et al. 2019 McDermott et al. [02 van de Water et al. 2024) over datasets such as MIMIC-IV (Johnson et al., (023) or eICU (Pollard et al., 2018) have also been extremely impactful. However, they are all tied to only a single or small set of datasets and tasks. Given the highly dynamic nature of clinical data and healthcare requirements, this is insufficient for the benchmarking and reproducibility needs faced by the ML for healthcare community.
When considering these existing solutions alongside the realities of healthcare data access and methodological development, it is clear that they are insufficient for three key reasons:
1. The Need for Interoperability The limited public datasets and only partially used CDMs can- not capture the diverse clinical populations, needs, and model capacities necessary for tangible ML progress in healthcare. To address this, systems for automated task extraction must be meaningfully interoperable across both public and private datasets with diverse input schemas.
2. The Need for Flexibility A single, static benchmark cannot encompass the variety of clinical tasks relevant to clinicians and informaticians. As existing tools (with limited interfaces for defining queries using per-set vocabularies) may struggle to generalize to new clinical tasks, ideal solutions must be sufficiently flexible to accommodate myriad of new task definitions, criteria formats, and disease or deployment areas.
3. The Need for Accessibility, Usability, and Applicability in Deep Learning Workflows While many existing tools feature no-code interfaces (e.g., web platforms to build queries) that are essential for less technically-literat audiences, integrating such tools with deep learning workflows can prove challenging. Deep learning systems are often run in a comi-programmatic manner on siloed, private computational clusters where researchers have minimal control. Hence, existing tools can cause significant hindrance. Instead, successful software must be able to provide a Python and command- line interface (CI.I) that offer significant ease of use to deep learning researchers, alongside shareable and readable configuration files that specify task definitions in a manner that can be readily ported across datasets and environments.
Our Solution: Automatic Cohort Extraction System for Event-Stream Datasets In this work, we solve these problems with the Automatic Cohort Extraction System for Event-Stream Datasets (ACES). ACES offers simple, expressive, and shareable configuration language for task and cohort
2

---

Published as a conference paper at ICLR 2025
definitions, as well as reliable CLI and straightforward Python library for extracting labeled task dataframes (Figure 2
Task definitions in ACES are naturally separated into simple dataset-specific event predicates and dataset- -agnost-e inclusion or exclusion criteria, thereby permitting the same task to be used in a conceptually identical manner across diverse datasets. By requiring users to specify predicates to realize their ML tasks on their specific datasets, ACES allows users to produce precise, locally- specific, verifiable cohorts that harmonize only the data elements needed for their task, regardless of how their input dataset is aligned or misaligned with existing ontologies or CDMs. Further, for datasets that are fully harmonized (e.g., through OHDSI vocabularies), ACES predicate definitions can be re-used across datasets without any loss of utility, In this way, ACES accommodates diverse a flexible, transparent manner. Overall, this
datasets at various levels of data harmonization approach not only enhances reproducibility but also facilitates community collaboration on task definitions, inclusion or exclusion criteria, and evaluation metrics for specific clinical use cases.
In contrast to prior task definition systems such as ATLAS, ACES makes minimal assumptions about the input data structure or source vocabularies. In particular, ACES can be run on any dataset, provided the necessary task-specific predicates can be pre-extracted in "event-stream" format (Figure T]. It can further be run from raw data directly for any dataset in the relatively low-level and flexible Medical Event Data Standard (MEDS) (Arnrich et al. 2024) or Event Stream GPT (ESGPT) (McDermott et al., 2023) formats in approximately five lines of template code, offering high efficiency.
Input EHR Dataset as an Event-Stream
ACES Cohort Extraction
Output Task Cohort
Stanc Messuremonts
Task Cohort
MEDS
Models
30b
SLee code
remorio, value
507224
CD DR/ AZTI
50724
mull
JEIG IT
58724 10U2611771
to00-000
DOn
230:96
58729 06/26-2011
subject
predection time
labol
68729 06/36-20 10,
231-95
TEMP
97.6
62729 DN 28/26/2311 235-5
64725 06/26-2010 10, 451152
DESC-ARRE
299684 06/112211 1751:s1
234684
HETO-T
176.211166
279104 15/38/1980,
000:00
DOB
Figure 1: Workflow for extracting cohorts using ACES. The pipeline shows the expected format for ACES-supported event-stream datasets and outcome cohorts. The transformation of raw data into the event-stream format is intentionally designed to be straightforward - primarily merging relational database tables minimizing data loss risks associated with other CDMs like OMOP.
Further, we align ACES with the concept of frictionless reproducibility for shared tasks proposed by David Donoho (Donoho 2024), especially for the clinical domain. This addresses the "Bring- Your-Own-Data Challenge", where much research relies on private patient outcomes data, often accessible only to a few credentialed researchers under strict usage agreements. Even when bench- marking platforms and shared code exist, the inability to share data directly remains significant barrier and often stifles progress. ACES seeks to overcome these challenges by offering a domain- specific language (DSL) and novel infrastructure ensure reproducibility without necessitating data sharing. Instead of relying on public datasets or reconfiguring code for diverse environments, ACES enables researchers to distribute task definitions through configuration files. These files provide standardized way to conceptually reproduce cohorts on private datasets or exactly reproduce them on public datasets.
In sum, ACES represents 3 key contributions:
1. ACES defines a shareable, simple, and flexible task configuration language that can define diverse sets of prediction tasks for ML in healthcare on any event-stream dataset.
2. ACES provides an casy-to-use library to automatically extract these tasks from diverse sources of real-world, structured, and longitudinal electronic health record (EHR) data.
3

---

Published as a conference paper at ICLR 2025
3. ACES introduces a novel DSL that leverages event-bounded query aggregations, enabling more expressive and efficient task definitions that better capture clinical timelines.
In the rest of this work, we first explore ACES in depth in Section 2 beginning with an illustration of its key concepts before briefly overviewing its core recursive algorithm. We then present a running example to illustrate how configuration files capture task logic. Next, in Section we demonstrate the use of ACES across diverse problem areas using real-world data, releasing a collection of task definitions based prior MI. for healthcare works both the dataset-agnostic criteria level and with dataset-specific predicates for the widely-used MIMIC-IV dataset (Johnson et al., 2023). Finally, we discuss the limitations and future roadmap of ACES in Section 4 and offer concluding thoughts in Section
Predicates:
A. With ACES
Diwer spaaio
code: BEATI
cefiee cohonts
Fistiuttan Specalic
diasfurge:
codo: DESOHNR2S
codo: OC/GIVES
COUIDIT adm solon
fualono SECENS
sife haw stara
Strrt Imggar
acd be criece
30 cotos
Gencrafazble
CE will
SABA
Hoc Hare dechhrr
Predict hosphha monslity for patients aigoosed with COMD- 9 orler to admiss on historic with B0e
Manual
Institution Specific Expensive Irreproducible
Hard to share & collaborate
B. Without ACES
Figure? 2: MI. task cohort extraction process (A) with and (B) without ACES, Predicates are dataset- specific concepts that needed to conceptually capture ML task. Windows temporal seg- ments on a patient's health record and are dataset-agnostic, as they are defined relative to the pred- icates. This distinction allows researchers to easily share the more complex task logic which is independent of datasets, facilitating conceptual reproducibility for ML tasks in healthcare.
AUTOMATIC COHORT EXTRACTION SYSTEM FOR EVENT-STREAM DATASETS (ACES)
In this section, we introduce ACES, a novel automatic task and cohort extraction system that fills the key gaps in interoperability, flexibility, and accessibility left by the existing tools outlined in Section [ To use ACES and extract cohort for downstream ML tasks, a user only needs to do the following simple steps:
Install ACES: A fully functional version of ACES is pushed PyPI, and any user can easily install it by simply running pip install es-aces. All dependencies are automatically set up with no further actions needed.
Define Dataset: A dataset in a permitted format, such as MEDS, ESGPT, or as direct predicates, required. More information on the data formats is available in Section 2.2
3. Define Task: A task configuration file is required to define the task that the user wishes to extract. This configuration language is simple, clear, yet flexible, permitting users to rapidly share and iterate over task definitions for their clinical settings. Configuration specification is given in Section 2.3
4

---

Published as a conference paper at ICLR 2025
4. Run the ACES CLI: ACES can be directly from the command line. Details about the
possible command- arguments are detailed Section 2.4 ACES can also be used as a direct Python import, as mentioned in Section 2.5
Sample command for basic cohort extraction using ACES CLI:
aces-cli cohort_name= -a..--a--mTTAAA""" data. path= $DATA_ PATH'
5. Get Outputs: ACES outputs single unified dataframe with all valid patient instances extracted according to task specifications. Users can subsequently leverage the returned columns with original patient identifiers and health record timestamps for downstream ML tasks. Further information is available in Section 2.6
Critically, after only these five simple steps, user can immediately, reproducibly extract full cohort from their source dataset that matches their task definition, and begin using this task for downstream representation learning.
2.1 ALGORITHM DESIGN
ACES addresses the challenge of extracting meaningful windows of data from patient records by using a recursive approach grounded in tree-structured configuration file. Each task is represented hierarchy of constraints, with nodes defining boundaries of windows of interest and edges spec- ifying temporal or event-based relationships between these windows. The algorithm begins by iden- tifying root anchor events in the dataset that correspond the triggering criteria of the task. It then recursively evaluates subtrees of constraints, aggregating predicate counts over defined windows either through temporal aggregations (e.g., over fixed time intervals) or event-bound aggregations (e.g., over windows bounded by specific clinical events, such as admissions, diagnoses, etc.). Each step ensures that the criteria of the subtree are met, filtering out invalid realizations before proceed- ing to child nodes. This recursive process guarantees that the specified configuration can always be resolved into valid windows that meet the task's s constraints. The final output a dataframe contain- ing all valid patients, task-specific labels, and prediction timestamps, and optionally, window start and end times as well as aggregated predicate counts. This ensures systematic and deterministic ex- traction of datasets for ML tasks. It also maintains flexibility and leverages the simple, transparent, and highly expressive DSL ACES (Figure 3).
2.2 DATASET CONFIGURATIONS
ACES is extremely flexible and can handle different input data formats, including MEDS (data standard-meds)) ESGPT (data. standard=esgpt), or direct predicates (data. standard= direct), where event-stream features are pre-extracted by the user from any given dataset schema. Other CDMs are interchangeable with these formats, such OHDSI OMOP through the MEDS OMOP ETI which transforms OMOP-compliant datasets into MEDS without data loss or seala- bility issues.
Using direct predicates to extract cohorts from formats that ACES does not natively support still significantly reduces the burden on users. Simply creating predicate features is much less cumber- some than either fully converting the dataset to a CDM in order to use existing tools like ATLAS or i2b2's platform, or performing the entire task extraction from scratch by writing in-house dataframe querying code. Additionally, as ACES configuration files are shareable and easily portable to other datasets (by simply swapping out predicate definitions), we believe ACES will offer long-term ef- ficiency benefits. This demonstrates the significant improvement in utility that ACES brings across diverse data schemas compared to existing tools.
2.3 TASK CONFIGURATIONS
In ACES, tasks are specified through configuration files that define collection of dataset-specific event predicates, which are simple functions evaluated on individual events within structured event-stream dataset. Predicate definitions can be stored in central "database" file specific to each
5

---

Published as a conference paper at ICLR 2025
Identify Subtree Root Anchors
predß prede
mark where predicate aggregations begin in recursion
Time/Event-based Aggregations
Window Reletionsnips
subject id
time predA pred8 predd
predD
Aggregations
...ihcc prunes Invalid realizations that violate subtree constraints
predA
predil
predc
predo
Filter by Subtree Criteria
ect predA predß pradC
summarizes predicate for each subtree
Figure 3: Overview of the ACES recursive algorithm. Given task generated from configura- tion file, ACES first identifies possible roots of the tree (task triggers) based on the associated pred- icate. then computes aggregations of predicate counts over time-based (i.e., windows with a time interval) or event-based (i.e., windows between specified events) periods to summarize predicates over the edges between the tree nodes. Finally, invalid branches are filtered if their predicate counts do not meet the specified criteria. This process is recursed for all child nodes of the task tree.
dataset, such that previously defined features could be easily reused for a variety of downstream tasks without further effort. Community predicate contributions for public datasets also streamline collaborative efforts for reproducibility. Additionally, task criteria are defined in a dataset-agnostic manner through collection of interrelated windows, which specify segments of patient's record and are constrained by certain relationships. Please see Figure 4 for an example of task configura- tion.
2.4 COMMAND-LINE INTERFACE
Hydra Arguments The Hydra framework (Yadan 2019) enhances the CLI by enabling flexible run configurations and argument parsing for cohort extractions. For instance, specific arguments are required to define the external source dataset for data loading. Depending on the chosen format (the data. standard argument), either the path to the data file (for meds direct) or the path to the dataset directory (for esgpt) must be specified to indicate the external source data from which ACES will extract the cohort. Additionally, cohort dir and cohort name are essential for locating and loading the task configuration file, as well as for results and operational logging.
Scaling Large Datasets An overview of the computational profile of ACES is available in Sec- tion 3.1 Additionally, for users dealing with large datasets, ACES can also be run over a collection of sharded files, extracting and storing the matching cohort for each shard individually in corre- sponding file paths. This can greatly increase computational efficiency by facilitating the processing of different shards in parallel via Hydra's multi-run launchers
http::///phhaaccccooo/////////pp 1auncher/

---

Published as a conference paper at ICLR 2025
Sample command for cohort extraction multiple MEDS shards using ACES CLI and Hydra:
aces-cli
-ultirun \
cohort_namest name=" config name>"
cohort_diae http:///aaaaaaaaooooo///////aaaa
data=sharded
data standard=meds
data. htta://////reettooo/////////////
data, shard= deepand_shaaaeeeee shards ifolder>/<nune <>um>>>
Sweeps over shards
In-hospital Mortality Prediction
C
E
48h
>= 5 Events
No Discharge OR Death
Admission
Discharge Death
Task Configuration
A
B
predicates:
admission:
input:
code: ADMISSION
start: NULL end: trigger 24 hours start_inclusive inc.usicclleeeeeeeeeeeeee True end ind_inclusive: True has:
discharge:
code: DISCHARGE
death:
code: DEATH
discharge death:
_ANY_ (5, None)
expr: or(discharge, death)
trigger: admission
gap:
D
start: trigger end: start 48 hours start start_inclus inclunicussiee False end ind_incuusive: True has:
target:
start: gap.end
end: start -> discharge death start start_inclusive: False
odmission: (None, 0)
discharge: (None, 0)
end inclusive: True
death: (None, 0)
label: death
Figure 4: Example configuration file for the binary prediction of in-hospital mortality 48 hours after admission. References to predicates and windows are italicized and bolded, respectively. (A) Dataset-specific task predicates. These concepts are needed conceptually capture this task and are used constraints and boundaries for windows of the patient record. For instance, in this example, the value of "ADM AAMMINS"ON" denotes a hospital admission event the source dataset. (B) A window of the task specifying the task inputs for downstream models. Suppose we'd like to use all historic patient data up to and including 24 hours past the admission. We could also place an arbitrary criterion requiring more than 5 records in this window ensure that the extracted cohort contains sufficient input data. (C) Trigger events for the task, which are hospital admissions as we'r 'd like make mortality prediction for each admission.
(D) A window of the task specifying gap in the patient timeline. Suppose we'd d like to set a minimum length of admission for our cohort (e.g., 48 hours). temporal constraint (minimum window duration) of 48 hours could then be set to represent this requirement. (E) A window of the task specifying the task target, which is set from the end of (D) the immediately subsequent discharge or death predicate. This creates our binary label classes for the task (i.e., discharge O; death = 1). All windows are interrelated on the patient timeline, as shown by how each window references another the configuration file.

---

Published as a conference paper at ICLR 2025
2.5 PYTHON API
In addition to the command-line tool, also provide Python interface to allow researchers to easily leverage ACES for cohort extraction in their deep learning code pipelines. full tutorial is available the ACES online documentation.
2.6 EXTRACTION OUTPUT
Finally, with a dataset configured for predicates and a task configuration file, ACES will execute the extraction for the cohort and return a table where each row is valid instance as per the criteria defined in the configuration file. Hence, each instance can be included in our cohort used for the downstream MI. task. At the most basic level, the table contains the patient identifiers of our cohort, a user-defined timestamp that indexes prediction time, and task label derived from a user-specified predicate. In addition, for each of the interrelated windows, start and end timestamp is provided to segment the patient record, along with summary the number of predicates evaluated that window.
3 USING ACES: A REPOSITORY OF EXAMPLE TASK CONFIGURATION FILES
To demonstrate the flexibility and utility of ACES, we define and publicly release the task con- figuration files described Table [ both with dataset-agnostic criteria and with dataset-specific predicate realizations based on the MEDS version of the public MIMIC-IV dataset. These vari- ous tasks have been previously studied, and ACES will facilitate their conceptual reproducibility to encourage benchmarking efforts and ensure robustness in ML for healthcare.
Table 1: A collection of sample configuration files for various common predictive tasks on MIMIC- IV. These tasks can be easily generalized other datasets, such e-ICU other private intensive care unit (ICU) and inpatient datasets by simply swapping out appropriate predicate definitions.
Task Name
Description
First 24h in-hospital mortality
Predict mortality within a hospital admission using the first 24 hours of data from that admission.
First 48h hospital mortality
Predict mortality within hospital admission using the first hours data from that admission.
First 24h In-ICU mortality
Predict mortality within an ICU admission using the first 24 hours of data from that admission.
First 48h inICU mortality
Predict mortality within ICU admission using the first 48 hours of data from that admission.
30d post-hospital hospital-dsschhrge discharge mortality
Predict mortality within 30 days of discharge. Predict hospital readmission within 30 days of discharge. Predict myocardial infarction (MI) incidence 1-5 years af- ter hospital admission.
30d re- re-admission
Myocardial infarction 1-5Y phenotyping
Reduced echo-derveed LVEF ECG
Predict reduced echo-derived left ventricular ejection fraction (LVEF) within 9 months of any ECG. Predict chronic kidney discase (CKD) onset in diabetic pa- tients within 5 years of any kidney panel laboratory test.
CKD onset in diabelics SY from kidney panel
3.1 COMPUTATIONAL PROFILE
To establish an overview of the computational profile of ACES, the collection of tasks from Table [] was extracted on MIMIC-IV, The MIMIC-IV MEDS schema has approximately 50,000 patients per shard with an average of approximately 00,500,000 total event rows per shard over seven shards. However, only a single shard was used to provide the bounded computational overview of ACES in applicable even when scaled to larger datasets using Hydra For instance,
Table 2 as the results one shard costs M memory and time, then shards may be executed in parallel with about N M memory and T time, in series with about M memory and T N time.
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance statistics for various common predictive tasks single MEDS shard of MIMIC-IV. This particular shard was 5,494.39 MiBs on disk as a parquet file and 9,661.41 MiBs when loaded in memory dataframe. consisted of 80,301,208 rows corresponding 47,954 unique patients. All experiments were executed on Linux server with 36 cores and 340 GBs of RAM available.
Task
f Patients
Samples
Total Time (s)
Max Memory (MiBs)
First 24h in-hospital mortality
20,971
58,823
363.09
106,367.
First 48h in-hospital mortality
18,847
60,471
364.62
108,913.95
First 24h in-ICU mortality
4,768
7,156
216.81
39,594.37
First 48h in-ICU mortality
4,093
7.112
217.98
39,451.86
30d post-hospitalddichaary mortality
28,416
68,547
182.91
30,434.68
30d re- addmiissioon
18,908
464.821
367.41
106,064.00
Myocardial Infarction 1-5Y phenotyping
3,329
8,319
198.04
33,427.70 70
Reduced echo-derived LVEF 9m post-ECO ECG
14
17
210.02
35,385,79
CKD onset in diabetics 5Y from kidney panel
736
3,503
238.65
44,221.81
4 DISCUSSION
4.1 ADDITIONAL RELATED WORK
In addition to the existing tools discussed in Section there are several other areas of related work relevant to ACES. Firstly, ACES serves as a middle ground between solutions that focus on specific CDMs, such as OHDSI's ATLAS Gold al. (2024) and i2b2's PIC-SURE (Stedman et al. 2024). Compared to these tools, ACES balances capability with greater ease of use and improved commu- nicative value. ACES is also not tied particular CDM. Built flexible event-streant format, ACES is a no-code solution with a descriptive input format, permitting easy and wide iteration over task definitions. It be applied to variety of schemas. making it versatile tool suitable for diverse research needs. ACES could also be directly connected with existing health CDMs through ETLs, such OMOP (Reich al. 2024), i2b2 (Murphy et al. 2010), FHIR (Bender & Sartipi 2013), and PCORnet (Fleurence et al. 2014). These models provide already-accepted standardized frameworks for organizing and analyzing healthcare data, and supporting them could greatly en- hance the utility and interoperability of ACES. Similarly, frameworks such as DescEmb (Hur et al., 2022) and GenHPF (Hur et al.) (202) hold great synergistic potential with ACES, and we believe that they can be complementary in enabling new kinds of cross-dataset training, transfer learning, and evaluation. Static benchmarks that provide standardized datasets, metries, and baseline meth- ods for range of clinical problems, such as YAIB (van de Water et al. 2024), multitask learning clinical prediction benchmarks (Harutyunyan et al. 2019). and EHR-PT PT McDermott et al. (021a), can also be directly integrated with ACES to facilitate robust ML in healthcare. Lastly, ACES can be used in conjunction with various health data management tools, such as TemporAl (Saveliev) & van der Schaar (023). PyHealth (Yang al.) 2023), OMOP-leam (Kodialam al.) 2021), and DPM360 (Suryanarayanan et al., 2021). These tools offer functionalities for pre-processing. manag- ing, and analyzing health data for downstream tasks, and integrating ACES with them directly can streamline ML workflows.
Beyond healthcare, ACES is applicable to data from a variety of other domains, such for finance, climate, or social media data essentially, ACES could be used for any structured, longitudinal data that can be reformatted an event-stream. This versatility makes ACES powerful library for extracting and analyzing complex event-based datasets across different fields.
4.2 LIMITATIONS FUTURE ROADMAP
ACES has limitations that can addressed in future work. Firstly, while already very expressive, the ACES task configuration language can still be further expanded. Expressing more complex kinds of predicates, window aggregations, labeling functions, and criteria would expand the scope of ACES significantly. ACES also seeks to provide direct support for cohort extraction based on unstructured data (notes and memos) in the future. Currently, such predicates need to be manually extracted by the user, but with the help of community contributions, we hope be able to incorporate automatic

---

Published as a conference paper at ICLR 2025
feature extraction from clinical notes, even images, and integrate them into configuration files for cohort extraction.
ACES is also very well poised to capture more complex patterns of task and cohort relationships, including prescribed systems case-control matching, automated bias analyses, or propensity re- weighting over excluded populations. It is also possible enable users to nest ACES configuration files to leverage extracted task labels as new predicates in more complex tasks and querying pro- cesses.
To enhance the scalability of ACES, we will seek to maintain support for the expanding MEDS standard. Direct interoperabilityy with other existing resources in this space, in partieular ATLAS and its OHDSI vocabulary-derrive cohort definitions, is a high priority area for future work.
Finally, with the standardization that ACES offers, new opportunities for human interaction with data are also made available, such as via natural language interface to define ACES predicates or configuration files and, thus, to extract downstream tasks, patient cohorts, or derived datasets code-free manner on diverse input EHR formats. We aim to explore the viability of leveraging large language models (LLLMs) to directly format predicate and criteria definitions given a data dictionary, and to automatically construct configuration files from natural language.
4.3 ACES AS A CATALYST FOR NEW ERA OF BENCHMARKING
In addition to the clear impact of ACES on reproducibility, robustness, and accessibility of ML for healthcare, we also feel that ACES is critical for a "new kind of benchmark" in the field-and and, in so being, is a portent of what needs to come should ML for healthcare progress to a more productive, communal, and impactful stage (McDermott 2025).
In particular, we argue that for this field to progress in the manner desired by the community and to maximize positive impact for all patients, we need to develop methodologies to test, share, and de- velop ML solutions across diverse datasets in meaningful and reproducible manner, even without said datasets being publicly available to general researchers. This capability is critical because, without it, we will never be able to offer new inductive insights about which methods are most likely to work best on novel, private data. other words, if we cannot test our model training recipes across the diverse sets of clinical care settings, populations, and conceptual dataset schemas that ex- ist the real world, we similarly cannot expect those training recipes to generalize well to a myriad of downstream deployment areas.
Libraries like ACES, which make as easy as possible for users to share the conceptual definitions of their tasks and prediction areas across datasets in such way that their colleagues them even over independent, private datasets - can help transform the kinds of benchmarking studies that Validation (MEDS-DEV) (McDermott et al. [024]] effort is one such step towards enabling the generalizable assessment of ML training recipes across datasets, clinical areas, and beyond.
perform in for healthcare. For instance, the MEDS Decentralized Extensible
CONCLUSION
In this work, we present the Automatic Cohort Extraction System for Event-Stream Datasets (ACES), ACES is a system designed to intuitively define cohorts and downstream tasks of inter- est for representation learning and reliably extract those cohorts from arbitrary datasets in event- stream formats. This system enables significantly greater shareability of task definitions, repro- ducibility of ML training and evaluation recipes, and is as easy to use as installing a package via pip and running a simple command-line tool. We feel that ACES will be integral in the develop- ment of new kinds of benchmarks MI. for healthcare, which can be explored across both pub- lic and private datasets alike, as well help characterize populations and tasks of interest in manner that cleanly separates dataset-specific components from shareable dataset-agnostic compo- nents. To learn more about ACES and use today your work, please visit our GitHub repository httpg//tthuuccoo/tsst/toooo///// 3601/acess and the ACES online documentation at: at: https::/eeeetssrremmccee readthedocs io/ en lattat
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
MBAM gratefully acknowledges support from a Berkowitz Postdoctoral Fellowship at Har- vard Medical School. JG is funded by the National Institutes of Health through NIH-USA R0ICA294033, JX greatly appreciates support from supervisors David Eyre (University of Ox- ford) and Curtis Langlotz (Stanford University). We also acknowledge valuable contributions by Tom Pollard (Massachusetts Institute of Technology) and by the broader MEDS ecosystem of con- tributors and users.
REFERENCES
Bert Arnrich, Edward Choi, Jason Alan Fries, Matthew B.A. McDermott, Jungwoo Oh, Tom Pol- lard, Nigam Shah, Ethan Steinberg, Michael Wornow, and Robin van de Water. Medical event data standard (MEDS): Facilitating machine learning for health. In ICLR 2024 Workshop on Learming from Time Series For Health, 2024. URL id [d-IsH2ebjiIG
net forum?
Duane Bender and Kamran Sartipi. HI7 fhir: An agile and restful approach to healthcare information exchange. In Proceedings of the 26th IEEE international symposium on computer-based medical systems, pp. 326-331. IEEE, 2013.
David Donoho. Data Science the Singularity. Harvard Data Science Review, 6(1), jan 29 2024. http:///ss.mmtreese.iim...uuu/us
Rachael L. Fleurence, Lesley H Curtis, Robert M Califf, Richard Platr, Joe V Selby, and Jef- frey Brown. Launching peomet, a national patient-centered clinical research network. Journal of the American Medical Informatics Association, 21(4))578-582, 05 2014. ISSN 1067-5027. doi: 10. 10011666mmiaal22214-007777
URL lattps://cc..og/// 1133/
am amia1n11-01140022
Sigfried Gold, Chris Knoll, Anthony Sena. Frank DeFalco, Anton Abushkevich, Pavel Grafkin, Vlad Belousov, Alex Saltykov, Vitaly Koulakov, Sergey Suvorov, Anastasiia Klochkova, Maria Pozhi- daeva, Jungmi Han, Anton Gackovka, Anton Stepanof, Mark Velez, Vlad Belousov, Chen Regen, Ekaterina Krivets, Semyon Titarenko, chgl, Joris Borgdorff, Valeri Antonov, Kwang Soo Jeong, Gowtham Rao, Alex Odysseus, Ajit Londhe, Alex Cumarav, Kai Kewley, Marc Suchard, Won- jun Hong. Tom White, Konstantin Yaroshovets, Shaun Turner, Taha Abdul- Richard D. Boyce Tiago Novo, and Rowan Parry. Atlas, June 2024. URL httph: / com/ OHDSI/Atlas. original-date: 2015-07-08T16:26:333
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Mul- titask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96. June doi: 10. 1038/03/4411 -97-019-0.---9. URL httthttt://// /ntttps://www. http:///wwwnnauue.ee.m nature. 03-
2019. ISSN 2052-4463
Publisher: Nature Publishing Group.
Kyunghoon Hur, Jiyoung Lee, Jungwoo Oh, Wesley Price, Younghak Kim, and Edward Choi. Uni- fying heterogeneous electronic health records systems via text-based code embedding, PMLR, pp. 183-203, Apr 2022. URL
m wtt.:m..r....s..emmmmmmmmmmm www.ppressslllll htt. html
Kyunghoon Hur, Jungwoo Oh, Junu Kim, Jiyoun Kim, Min Jae Lee, Eunbyeol Cho. Seong-Eun Moon, Young-Hak Kim, Louis Atallah, and Edward Choi. Genhpf: General healthcare predic- tive framework for multi-task multi-source learning. IEEE Journal of Biomedical and Health Informatics, 28(1):502- 513, Jan 2024. doi: http://oo.oo///11110////h..2.22. org/ alss 2208009927778
URL
Alistair E. W. Johnson, Tom J. Pollard, and Roger G. Mark. Reproducibility in critical care: mortality prediction case study. In Proceedings of the 2nd Machine Learning for Healthcare Conference, pp. 361-376. PMLR, November 2017. URL httpp:://ppett ISSN: 2640-3498.
ngs mmlr.
(eess//68//ohnsont 7a html
Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and
11

---

Published as a conference paper at ICLR 2025
Roger G. Mark. MIMIC-IV, freely accessible electronic health record dataset. Scientific Data, 10(1):1, Jamuary 2023. ISSN 2052-4463. doi: 10. (10-8///41157-002-1188999 URL https:::: Publisher: Nature Publishing
// //www. nature com/
Group.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Rohan Kodialam, Rebecca Boiarsky, Justin Lim, Aditya Sai, Neil Dixit, and David Sontag. Deep contextual clinical prediction with reverse distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 249-258. 2021.
Matthew McDermott. The (lack of?) science of machine learning for healthcare. In Proceedings of the 4th Machine Learning for Health Symposiam, volume 259 of Proceedings of Machine Learning Research, pp. 19-29. PMLR, 15-16 Dec 2025. URL https: / /trpss////ccceednggs
htm!
Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzych Ghassemi. A comprehensive ehr timeseries pre-training benchmark. In Proceedings of the Conference on Health, Inference, and Learning, pp. 257-278, 2021a.
Matthew B. A. McDermott, Shirly Wang. Nikki Marinsek, Rajesh Ranganath, Luca Foschini, and Marzyeh Ghassemi. Reproducibility in machine learning for health research: Still a ways to go. Science Translational Medicine, ((586))) March 2021b. ISSN 1946-6242. doi: 10.11126scirransseeeabbll6.
Matthew B. A. McDermott, Aleksia Kolo, Chao Pang, Edward Choi, Ethan Steinberg. Hyewon Jeong, Jack Gallifant, Jason Alan Fries, Jeffrey Chiang, Jungwoo Oh, Justin Xu, Kamilè Stankevičiūté, Kiril Vadimovic Klein, Mikkel Fruelund Odgaard, Nassim Oufattole, Patrick Rock- enschaub, Pawel Rene, Robin van de Water, Shalmali Joshi, Simon Austin Lee, Teya Berga- maschi, Tom Pollard, Vincent Jeanselme, Nigam Shah, Michael Wornow, Aparajita Kashyap, XinZhou Jiang, Yanwei Li, Young Sang Choi, Yuta Kobayashi, and Ryan King. Meds decentral- ized, extensible validation (meds-dev) benchmark: Establishing reproducibility and comparability in ml for health, 2024. BtL/https:/ /ttpss///ppeeeeeeeeeee www. mene//pfff http://wwww...eep//odoeeeeee/// Di=-DEx3ttRRe
Matthew B.A. McDermott, Bret Nestor, Peniel N Argaw, and Isaac S. Kohane, Event stream GPT: data pre-processing and modeling library for generative, pre-trained transformers over continuous-ting time sequences of complex events. In Thirty-sevntth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
//openreveew net.
Shawn N Murphy, Griffin Weber, Michael Mendis, Vivian Gainer, Henry C Chueh, Susanne Churchill, and Isaac Kohane. Serving the enterprise and beyond with informatics for integrat- ing biology and the bedside (i2b2). Journal of the American Medical Informatics Association, 17 (2):124-130) 2010.
Tom Pollard, Alistair E W Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. The eICU Collaborative Research Database, a freely available multi-center database for critical research. Scientific data, 501) 2018
Christian Reich, Anna Ostropolets, Patrick Ryan, Peter Rijnbeek, Martijn Schuemie, Alexander Davydov, Dmitry Dymshyts, and George Hripesak. Ohdsi standardized vocabularies large- scale centralized reference ontology for international data harmonization. Journal of the American Medical Informatics 31t 01 2024. ISSN 1527-974X. doi: 10. 10.1093/jamia/ wwg.:/o...r..ooo..ooo 1093/ jamia/ jataa//caaaaa" ocad247
ocad247. URL
Olawale Salaudeen and Moritz Hardt. Imagenot: A contrast with imagenet preserves model rank- ings. arXiv preprint arXiv:2404.021 12, 2024.
Evgeny S Saveliev and Mihaela van der Schaar. Temporai: Facilitating machine learning innovation in time domain tasks for medicine. arXiv preprint 1122002001.12660. 2023.
12

---

Published as a conference paper at ICLR 2025
Ali Shirali, Rediet Abebe, and Moritz Hardt. A theory of dynamic benchmarks. In The Eleventh International Conference on Learning Representations, 2023. URL /ttpttst/////eeeeeeaa
net peet//ooruumn1c-11119geee22
Jason Stedman, Jeremy Nix, James Peck, Nick Benik, Danielle Pillion, Anil Degala, Luke Sik- ina, Samantha Piatt, Simran Makwana, and Jaspreet Khanna. Pic-sure all-in-one. https: 2024. Parthasarathy Suryanarayanan, Prithwish Chakraborty, Piyush Madan, Kibichii Bore, William Ogallo, Rachita Chandra, Mohamed Ghalwash, Italo Buleje, Sekou Remy, Shilpa Mahatma, Pablo Meyer, and Jianying Hu. Disease progression modeling workbench 360, 2021. URL
https: //aaxiivvv... org/ abs/2105s 2106. 13265
Robin de Water, Hendrik Schmidt, Paul Elbers, Patrick Thoral, Bert Amnich, and Patrick Rock- enschaub. Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML, http:/bb/22000000 March 2024. URL [RRpiittp///axxiivoo
05109 arXiv:2306.05100 [cs].
Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, and Scott McLachlan. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association, 25(3):230-25) 08 2017. ISSN 1527-974X. doi: 10.1093/jamia/ocx077 URL /
jamia/ocx079
Omry Yadan. Hydra framework for elegantly configuring complex applications. Github, 2019.
URL http:://gggccb...cccccaceeeeeeee
Chaoqi Yang. Zhenbang Wu, Patrick Jiang, Zhen Lin, Junyi Gao, Benjamin P. Danek, and Jimeng Sun. Pyhealth: A deep learning toolkit for healthcare applications. In Proceedings of the 29th
ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '23, pp. 5788-5789, New York, NY, USA, 2023. Association for Computing Machinery, ISBN 9798400701030, doi: http:///oooo.orro111.111113/303. 359917
10.1145/3580305.35991
Guanhua Zhang and Moritz Hardt. Inherent trade-offi between diversity and stability in multi-task benchmark. arXiv preprint arXhv:2405.017/9 2024.
13

---

Published as a conference paper at ICLR 2025
A ACES ONLINE DOCUMENTATION INDEX
The full ACES online documentation is available at: htttp::/eveettee readthedocs io/ee//atest] We have also included compiled PDF version of this doc- umentation in the Supplementary Material.
To answer specific questions about ACES, please the below index for the PDF documentation (links to the associated online documentation are also provided).
How do you use ACES?
What is task and how you specify one?
Sample task descriptions and specificattions are provided in the Task Examples section in Chapter (fttps://eveenttrreeaasss htw.://aaaaaass...eadeeeeeeeeeee readhhedoos..o//e////aatttt notebooks/examples..oo html).
1.1. What are predicates and how do you specify them?
For an overview predicates and how they form the foundation of ACES, please refer to the Predicates DataFrame section in Chapter hhttps://eventstrrem.ccs readthedocs. /n//aees//a//////////// hotebboks/prrdiiatess.tt html).
1.2. What are windows and how do you specify them?
A window in ACES represents segment in the patient record. For details on how to define a window, please refer to Chapter 1.3.3 hnttps://eventstreee reat io/en/latest. readme, httheneehhtwliiiidwss..
readthedocs
2. How do you extract a task from a dataset?
For general ACES usage instructions, please refer to Chapter 2.1 (htpps: //eventstreeaacees www.eesssreatthhccc...o..o quick-start) Additionally, brief end-to-end instructions are also available in Chapters 1,2 and 1.3.
2.1. Detailed Usage Instructions for ACES CLI
For detailed instructions on using ACES CLI, please refer to the Usage Guide Chapter 2.2 Intpp:://eveetsstreeaaaee readthedocs reatt io/en/
latest/usager htm http:///eeaeeet.....eeeeeeeeeeee
2.2. Tutorial for the ACES Python API
For step-by-step tutorial on using the ACES Python API, please refer to the
Code Example Notebook Chapter 5 f
readthedocs.
htmll
3. ACES with /s. Other Tools
For an overview of how ACES could be used with other existing complementary tools for reproducible ML, please refer to Chapter 1.4.2 fatps:://evettttreemmeee.
readthedocs. r
dme. htm
For an overview of how ACES compares to other existing altemaatve tools for semi- fully-automated cohort extraction. please refer Chapter 1.4.3 rew.eeesseeadaaeeeeeeeeeeeee htt./////o...ee/ee/tttttt/////ee readme..
atn.laatteraative-toool.)
How does ACES work?
1. What is the formal configuration language specification for ACES?
14

---

Published as a conference paper at ICLR 2025
For technical details the ACES configuration language, please refer to the Configuration Language Specification section in Chapter 6.1 hetpss: //eventstrremaaces.o wtw.t/attt....oooeeooeeteeeeeeee http////aeeteeeetenaaaaaaaaaaea
html
contigguratioo--laggue----ee speci fication
2. Glossary of ACES Terminology
For glossary of terminology used throughout ACES, please refer to the Algorithm Ter- minology section in Chapter 6.3 Oftps://ereanstrremmcss..c. readt hedocs.
3. What is the ACES extraction algorithm?
For technical details on the ACES algorithm, please refer the Algorithm Design section in Chapter 6,4 (https: httt::::eeeveettsttrmmmeeee readthedocs. io/en/ Tltest/technicen
html hen
4. Full ACES Module API Documentation
For the complete ACES module documentation, including doctests that ensure algo- rithm correctness, please refer to the Module API sections in Chapter 8 (https:/7
eventstrranaces, rewtteedosssssccceecccccocoooee 1o http:////o...e/eeoooo/o/////ooo.
How well does ACES work?
1. Computational Profile
For an overview of the computational profile of ACES, please refer to the Computational Profile section in Chapter 7 (arreaee/eeeettttraaraeeee htw.aa/ettteteeeeeeeeaaaaceceaae 1o/
readthedocs
in/ html)
2. Further Examples
For additional examples of configuration files and criteria of different MI for health- care tasks, please refer to the MEDS- DEV benchmarking effort on GitHub: https:
7/github. com www./emottt.t/ceo/moo//mmocmcttt MEDS DDEDDDSD
15

---

Published as a conference paper at ICLR 2025
B COMPARATIVE EXPERIMENTS
We conducted preliminary experiments to quantitatively compare ACES with other alternative tools. Using the OMOP version of the MIMIC-IV Demo, well synthetic dataset of 1,000 patients generated using Synthea (Walonoski et al. 2017) and converted into OMOP, we queried four tasks using ACES and two of the most comparable tools, OMOP-leam and DPM360.
We collected metrics including script runtime, peak memory usage (in MiBs), lines of code required (including configuration files and any template code needed execute extraction), and human time spent. All experiments were conducted on default A100 GPU instance with 84 GB of RAM and 12 CPU cores from Google Cloud Platform's Compute Engine.
Table B.I: Quantitative comparison of ACES and other comparable cohort extraction tools across datasets and tasks.
Method
Task
Runtime
Menorry
Lines
Human Time
First 24h in-hospital morilly
0.386
385
35
120
30k posshsppini-fiscorrg momalit
0.236
351
32
ACES via MEDS
30d re-admission
Ead-af-life prodiction
Synthes:
First 24h in bospital mortaliy
3.932
340
205
2,126
30d pust-bospinnl dischurge murtaity
4.188
550
257
1,200
DPM360 via OMOP
admission
End-of-life prodiction
4.901
387
220
1,500
First 24h in hospital mortality
0.617
545
35
180
ACES via MEDS
30d post bospinal dischurgo mortality
0.301
509
32
303 re-adiission
22
MIMIC-IV Demo
End of life prediction
0.349
589
28
First 24h in-hospital mortality
12:220
688
172
3,621
OMOP-ieurn via OMOP
30d pust-hospiial- dischurge mortaliy
8608
199
2,441
50U re-admission
168
End of life prediction
24.540
932
251
12,000
We also qualitatively compared the adaptability of these approaches to any other given ML task. While ACES requires simple modifications to the task configuration file to capture new task logic or cohort criteria. new ATLAS executions, bespoke SQL queries, and changes to Python parameters may be needed for DPM360 and OMOP-leam.
While we acknowledge potential biases in these results due to our familiarity with ACES and only surface exposure to other extraction tools, have aimed for fair and objective evaluation. Based on our experience and preliminary user feedback, we find ACES to be intuitive and believe it offers significantly improved workflow with more comprehensive functionality.
16


Reviews:
[{"review_id": "EsWdp4HZuV", "reviewer": "Reviewer_G3hu", "summary": "This study presents Automatic Cohort Extraction System (ACES) aiming to addresses the reproducibility in healthcare ML challenges due to the private nature of datasets and model definitions,  by providing a tool to define and reproduce ML cohorts across datasets. The tools are claimed to enable consistent cohort extraction by offering a configuration language and automatic pipeline, potentially improving reproducibility in ML studies on electronic health record data.", "questions": "+ How the two parts of \"institution-specfic\" and \"generalizable\" work along side each other? Besides standardizing the training part of ML models, how this work helps with what's discussed in 5.3 (i.e., helping with no access to private data).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The study is indeed very timely and sound. It addresses a critical area of concern in ML for health, by publishing offering a pipeline to standardize common cohort extraction tasks from the health datasets.\n+ The paper does a nice job of providing public and anonymized resources. \n+ The study adopts several recent prior studies, and seems to nicely complement those.", "weaknesses": "+ One major unclear aspect is providing robust evidence that the proposed tool/pipeline indeed achieves a good performance extracting the right cohort. A natural question, how one can ensure this fairly complex procedure will not miss critical samples or include incorrect ones in the final results. Is there any way to compare the results against some sort of baseline or ground truth?\n+ While the paper is relatively sound and straightforward, and while some examples are provided, the whole document still seems a bit unaccessible to even an average reader. The paper stays high-level, perhaps in favor generalizability, but risks missing the readers. Here are some example ways, the paper can be further accessible. Provide more concrete examples, especially by showing the outcome cohort. Present the concept of \"data stream\" in a more tangible and clear way in this application. Rely less on referring too much to appendix and external links. \n+ Relatedly, the main technical contribution can be further clarified. The recursive nature of the specified algorithm seems unclear. \n+ It is suggested that subjective and not substantive statements are removed. For instance, \"preliminary work on this front shows very promising results!\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1KfULPDJRp", "reviewer": "Reviewer_ntJe", "summary": "This paper introduces ACES, an open-source Python library designed for event-stream data, aimed at automating the cohort extraction process through a unified API. Users define tasks via a configuration file specifying predicates, input, trigger, gap, and target, making the system both structured and adaptable.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a significant barrier in healthcare ML research: the complexity of EHR data, especially in cohort extraction. The motivation is clear and relevant.\n2. The authors provide an open-source Python package with thorough documentation, which will support reproducibility and benefit future research.\n3. ACES is built on the MEDS format, showing potential for broad applicability across various datasets and tasks.\n\n**Note:** I think this library could offer value to ML in healthcare by simplifying EHR cohort extraction. However, the essential factor for such tools is flexibility and support across multiple datasets and tasks. Assessing these aspects thoroughly is challenging without extensive hands-on testing, so I have adjusted my confidence rating accordingly.", "weaknesses": "A primary concern is the library’s utility for more complex tasks. For example, in the case of “CKD in diabetics within 5Y of kidney panel,” much of the effort lies in translating high-level criteria into specific medical features—a step that remains challenging and unresolved. Extracting cohorts from predefined features is relatively straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLgE5l9TDb", "reviewer": "Reviewer_Qm8X", "summary": "The paper introduces ACES, the Automatic Cohort Extraction System, designed to address reproducibility issues in machine learning for healthcare by streamlining task and cohort definitions in electronic health record (EHR) datasets. ACES provides a configuration language that supports dataset-specific and dataset-agnostic criteria, which enables sharing and reuse across datasets with minimal adjustments. It integrates with formats like MEDS and ESGPT, allowing users to define prediction tasks with flexibility, regardless of specific dataset schemas, and includes both Python and command-line interfaces. ACES aims to bridge gaps in interoperability, flexibility, and deep learning compatibility, potentially enhancing the reproducibility and accessibility of ML healthcare studies.", "questions": "Have the authors compare the effort to ETL a new dataset to an existing standard (e.g. OMOP) and model multiple downstream tasks vs their proposed method? It would be useful to compare these timings. It is also known that in some situations all elements of a new dataset may not completely map to OMOP format exhaustively - is this a problem that the authors faced and they are trying to solve this challenge? Further comparative details on this would situate the paper better", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Overall this is a very interesting paper with some key strengths (but perhaps handicapped by some key weakness below). First considering the strengths we can identify the following\n\n- The authors aim to improve standardization and interpretability in healthcare ML. While there have been many approaches to this including OMOP and i2b2, ACES aims to improve integration with potentially non-compliant sources by enabling reproducible task definitions across multiple datasets, which can aid in consistent benchmarking and model validation.\n2. The authors propose a high level configuration language and support for Python and CLI interfaces facilitate broad adoption among researchers and compatibility with deep learning workflows. With this they aim to lower the barrier for integration\n3. ACES's support for multiple data standards (like MEDS and ESGPT) and separation of dataset-specific and dataset-agnostic components allows flexible and adaptable cohort extraction across diverse clinical datasets.", "weaknesses": "Given the promising aspects, they paper can be improved upon by addressing the following\n\n1. First, the authors may have created the classic problem of `resolving n competing standards by ending up with n + 1 competing standards`. Specifically, despite claims of flexibility, ACES requires data to be formatted in supported structures, which could necessitate pre-processing for datasets outside MEDS or ESGPT. The authors doesn't describe in details the effort in taking a new dataset and making it compliant to MEDS/ESGPT. For OMOP, the authors mention about a connector - but the authors don't discuss whether the MEDS/ESGPT transformation are exhaustive and without data loss. \n2. Next moving on to the performance and scalability, while initial results are promising, more comprehensive evaluations on larger and more varied datasets would strengthen claims about ACES's efficiency and scalability. For example the authors claim these task specific transformations are cheaper than full scale ETL of datasets, the authors doesn't discuss whether it is cheaper in the long term, especially with multiple tasks. Specifically considering the eco-system of downstream processing like OHDSI ecosystem it might be cheaper to convert to OMOP. Even for deep learning systems several efforts like https://github.com/clinicalml/omop-learn and https://github.com/BiomedSciAI/DPM360 extends the ecosystem with low barriers for integrating with OMOP systems.  \n3. Finally there are some limitations currently with ACES not able to directly handle unstructured data, which limits its application scope in tasks that require insights from clinical notes or similar records.\n\n\nOverall while the authors have a novel solution it is not evident they are solving the right problem or who the solution is for. Identifying these two would enable the paper to be reviewed with the full context", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bvGPIph3Og", "reviewer": "Reviewer_z6xZ", "summary": "This paper concentrates on enhancing the reproducibility of machine learning for healthcare research and introduces the Automatic Cohort Extraction System (ACES), a library/tool specifically developed for handling healthcare data formatted as event streams. ACES functions as a configuration language that simplifies task definitions by abstracting them into two main components: dataset-specific event predicates and dataset-agnostic inclusion/exclusion criteria. This structured approach allows ACES to support a diverse array of representative predictive tasks essential for healthcare analytics.", "questions": "Please refer to points W1 through W3 for detailed concerns and recommendations.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "S1. The promotion of standardization and simplification in automatic cohort extraction is of considerable importance. This effort can enhance the reproducibility of machine learning applications in healthcare research. Furthermore, it has the potential to advance the digitalization of healthcare data, thereby providing substantial benefits to both clinicians and patients.\n\nS2. The paper provides a relatively thorough overview of the current landscape of machine learning within healthcare research, including common data models, existing benchmarks, and representative applications/tasks.\n\nS3. The ACES system is implemented as an open-source project, accompanied by comprehensive documentation.", "weaknesses": "W1. ACES primarily functions in healthcare analytics by performing preprocessing on broadly collected EHR data tailored to specific target applications, such as extracting particular features within a time window or performing aggregations. This process is ubiquitous in most EHR-related data mining tasks. ACES abstracts common dataframe transformation patterns across diverse healthcare applications, providing high-level encapsulation. Under the assumptions set by ACES, this approach simplifies the preprocessing to some extent. However, it is essential to note that this work can be seen as largely an engineering solution rooted in low-code programming concepts, with limited conceptual innovation. Additionally, it is challenging to discern any substantial transformative impact this encapsulation introduces to EHR data processing.\n\nW2. Continuing from the previous point, ACES aims to implement a standard data extraction process for healthcare data in an event-stream format. However, the paper does not adequately articulate the specific challenges inherent to this process. This omission leaves readers uncertain of the unique problems ACES addresses, thereby diminishing the perceived contributions and impact of ACES.\n\nW3. The paper lacks objective and quantitative evidence to substantiate the superiority of ACES over existing work aimed at solving the same research problem. This hence weakens the argument for ACES as a preferable solution in the competitive landscape of healthcare data processing and management technologies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)