ACES: Automatic Cohort Extraction System for Event-Stream Datasets

Authors: Justin Xu, Jack Gallifant, ALISTAIR JOHNSON, Matthew B.A. McDermott
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Automatic Task Specification, Cohort Extraction, Electronic Health Records, Open Source Software, Benchmarks, Datasets

Abstract:
Reproducibility remains a significant challenge in machine learning (ML) for healthcare. Datasets, model pipelines, and even task or cohort definitions are often private in this field, leading to a significant barrier in sharing, iterating, and understanding ML results on electronic health record (EHR) datasets. We address a significant part of this problem by introducing the Automatic Cohort Extraction System (ACES) for event-stream data. This library is designed to simultaneously simplify the development of tasks and cohorts for ML in healthcare and also enable their reproduction, both at an exact level for single datasets and at a conceptual level across datasets. To accomplish this, ACES provides: (1) a highly intuitive and expressive domain-specific configuration language for defining both dataset-specific concepts and dataset-agnostic inclusion or exclusion criteria, and (2) a pipeline to automatically extract patient records that meet these defined criteria from real-world data. ACES can be automatically applied to any dataset in either the Medical Event Data Standard (MEDS) or Event Stream GPT (ESGPT) formats, or to *any* dataset in which the necessary task-specific predicates can be extracted in an event-stream form. ACES has the potential to significantly lower the barrier to entry for defining ML tasks in representation learning, redefine the way researchers interact with EHR datasets, and significantly improve the state of reproducibility for ML studies using this modality. ACES is available at: https://github.com/justin13601/aces.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730125431457

thread_type:
official_review

<thread_object>
creator: reviewer_z6xz

<content>
summary:
This paper concentrates on enhancing the reproducibility of machine learning for healthcare research and introduces the Automatic Cohort Extraction System (ACES), a library/tool specifically developed for handling healthcare data formatted as event streams. ACES functions as a configuration language that simplifies task definitions by abstracting them into two main components: dataset-specific event predicates and dataset-agnostic inclusion/exclusion criteria. This structured approach allows ACES to support a diverse array of representative predictive tasks essential for healthcare analytics.

strengths:
S1. The promotion of standardization and simplification in automatic cohort extraction is of considerable importance. This effort can enhance the reproducibility of machine learning applications in healthcare research. Furthermore, it has the potential to advance the digitalization of healthcare data, thereby providing substantial benefits to both clinicians and patients.

S2. The paper provides a relatively thorough overview of the current landscape of machine learning within healthcare research, including common data models, existing benchmarks, and representative applications/tasks.

S3. The ACES system is implemented as an open-source project, accompanied by comprehensive documentation.

weaknesses:
W1. ACES primarily functions in healthcare analytics by performing preprocessing on broadly collected EHR data tailored to specific target applications, such as extracting particular features within a time window or performing aggregations. This process is ubiquitous in most EHR-related data mining tasks. ACES abstracts common dataframe transformation patterns across diverse healthcare applications, providing high-level encapsulation. Under the assumptions set by ACES, this approach simplifies the preprocessing to some extent. However, it is essential to note that this work can be seen as largely an engineering solution rooted in low-code programming concepts, with limited conceptual innovation. Additionally, it is challenging to discern any substantial transformative impact this encapsulation introduces to EHR data processing.

W2. Continuing from the previous point, ACES aims to implement a standard data extraction process for healthcare data in an event-stream format. However, the paper does not adequately articulate the specific challenges inherent to this process. This omission leaves readers uncertain of the unique problems ACES addresses, thereby diminishing the perceived contributions and impact of ACES.

W3. The paper lacks objective and quantitative evidence to substantiate the superiority of ACES over existing work aimed at solving the same research problem. This hence weakens the argument for ACES as a preferable solution in the competitive landscape of healthcare data processing and management technologies.

questions:
Please refer to points W1 through W3 for detailed concerns and recommendations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644436827

thread_type:
official_review

<thread_object>
creator: reviewer_qm8x

<content>
summary:
The paper introduces ACES, the Automatic Cohort Extraction System, designed to address reproducibility issues in machine learning for healthcare by streamlining task and cohort definitions in electronic health record (EHR) datasets. ACES provides a configuration language that supports dataset-specific and dataset-agnostic criteria, which enables sharing and reuse across datasets with minimal adjustments. It integrates with formats like MEDS and ESGPT, allowing users to define prediction tasks with flexibility, regardless of specific dataset schemas, and includes both Python and command-line interfaces. ACES aims to bridge gaps in interoperability, flexibility, and deep learning compatibility, potentially enhancing the reproducibility and accessibility of ML healthcare studies.

strengths:
Overall this is a very interesting paper with some key strengths (but perhaps handicapped by some key weakness below). First considering the strengths we can identify the following

- The authors aim to improve standardization and interpretability in healthcare ML. While there have been many approaches to this including OMOP and i2b2, ACES aims to improve integration with potentially non-compliant sources by enabling reproducible task definitions across multiple datasets, which can aid in consistent benchmarking and model validation.
2. The authors propose a high level configuration language and support for Python and CLI interfaces facilitate broad adoption among researchers and compatibility with deep learning workflows. With this they aim to lower the barrier for integration
3. ACES's support for multiple data standards (like MEDS and ESGPT) and separation of dataset-specific and dataset-agnostic components allows flexible and adaptable cohort extraction across diverse clinical datasets.

weaknesses:
Given the promising aspects, they paper can be improved upon by addressing the following

1. First, the authors may have created the classic problem of `resolving n competing standards by ending up with n + 1 competing standards`. Specifically, despite claims of flexibility, ACES requires data to be formatted in supported structures, which could necessitate pre-processing for datasets outside MEDS or ESGPT. The authors doesn't describe in details the effort in taking a new dataset and making it compliant to MEDS/ESGPT. For OMOP, the authors mention about a connector - but the authors don't discuss whether the MEDS/ESGPT transformation are exhaustive and without data loss. 
2. Next moving on to the performance and scalability, while initial results are promising, more comprehensive evaluations on larger and more varied datasets would strengthen claims about ACES's efficiency and scalability. For example the authors claim these task specific transformations are cheaper than full scale ETL of datasets, the authors doesn't discuss whether it is cheaper in the long term, especially with multiple tasks. Specifically considering the eco-system of downstream processing like OHDSI ecosystem it might be cheaper to convert to OMOP. Even for deep learning systems several efforts like https://github.com/clinicalml/omop-learn and https://github.com/BiomedSciAI/DPM360 extends the ecosystem with low barriers for integrating with OMOP systems.  
3. Finally there are some limitations currently with ACES not able to directly handle unstructured data, which limits its application scope in tasks that require insights from clinical notes or similar records.


Overall while the authors have a novel solution it is not evident they are solving the right problem or who the solution is for. Identifying these two would enable the paper to be reviewed with the full context

questions:
Have the authors compare the effort to ETL a new dataset to an existing standard (e.g. OMOP) and model multiple downstream tasks vs their proposed method? It would be useful to compare these timings. It is also known that in some situations all elements of a new dataset may not completely map to OMOP format exhaustively - is this a problem that the authors faced and they are trying to solve this challenge? Further comparative details on this would situate the paper better
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699780560

thread_type:
official_review

<thread_object>
creator: reviewer_ntje

<content>
summary:
This paper introduces ACES, an open-source Python library designed for event-stream data, aimed at automating the cohort extraction process through a unified API. Users define tasks via a configuration file specifying predicates, input, trigger, gap, and target, making the system both structured and adaptable.

strengths:
1. The paper addresses a significant barrier in healthcare ML research: the complexity of EHR data, especially in cohort extraction. The motivation is clear and relevant.
2. The authors provide an open-source Python package with thorough documentation, which will support reproducibility and benefit future research.
3. ACES is built on the MEDS format, showing potential for broad applicability across various datasets and tasks.

**Note:** I think this library could offer value to ML in healthcare by simplifying EHR cohort extraction. However, the essential factor for such tools is flexibility and support across multiple datasets and tasks. Assessing these aspects thoroughly is challenging without extensive hands-on testing, so I have adjusted my confidence rating accordingly.

weaknesses:
A primary concern is the library’s utility for more complex tasks. For example, in the case of “CKD in diabetics within 5Y of kidney panel,” much of the effort lies in translating high-level criteria into specific medical features—a step that remains challenging and unresolved. Extracting cohorts from predefined features is relatively straightforward.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730998947466

thread_type:
official_review

<thread_object>
creator: reviewer_g3hu

<content>
summary:
This study presents Automatic Cohort Extraction System (ACES) aiming to addresses the reproducibility in healthcare ML challenges due to the private nature of datasets and model definitions,  by providing a tool to define and reproduce ML cohorts across datasets. The tools are claimed to enable consistent cohort extraction by offering a configuration language and automatic pipeline, potentially improving reproducibility in ML studies on electronic health record data.

strengths:
+ The study is indeed very timely and sound. It addresses a critical area of concern in ML for health, by publishing offering a pipeline to standardize common cohort extraction tasks from the health datasets.
+ The paper does a nice job of providing public and anonymized resources. 
+ The study adopts several recent prior studies, and seems to nicely complement those.

weaknesses:
+ One major unclear aspect is providing robust evidence that the proposed tool/pipeline indeed achieves a good performance extracting the right cohort. A natural question, how one can ensure this fairly complex procedure will not miss critical samples or include incorrect ones in the final results. Is there any way to compare the results against some sort of baseline or ground truth?
+ While the paper is relatively sound and straightforward, and while some examples are provided, the whole document still seems a bit unaccessible to even an average reader. The paper stays high-level, perhaps in favor generalizability, but risks missing the readers. Here are some example ways, the paper can be further accessible. Provide more concrete examples, especially by showing the outcome cohort. Present the concept of "data stream" in a more tangible and clear way in this application. Rely less on referring too much to appendix and external links. 
+ Relatedly, the main technical contribution can be further clarified. The recursive nature of the specified algorithm seems unclear. 
+ It is suggested that subjective and not substantive statements are removed. For instance, "preliminary work on this front shows very promising results!"

questions:
+ How the two parts of "institution-specfic" and "generalizable" work along side each other? Besides standardizing the training part of ML models, how this work helps with what's discussed in 5.3 (i.e., helping with no access to private data).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732386043531

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Rebuttal Response & Thank You

comment:
Thank you to all reviewers and the AC for your service in making ICLR possible! We appreciate all the positive comments and feedback we've received, and believe they've significantly helped improve our paper.

We are particularly glad that the reviewers identified several key strengths of our paper, including:
- clear motivations for addressing a critical area of concern in ML for health (**Reviewer G3hu:** "...study is indeed very timely and sound..."; **Reviewer ntJe:** "...addresses a significant barrier in healthcare ML research...motivation is clear and relevant");
- the open-source nature of ACES accompanied with thorough documentation (**Reviewer ntJe:** "...an open-source Python package with thorough documentation...will support reproducibility and benefit future research"; **Reviewer z6xZ:** "...an open-source project, accompanied by comprehensive documentation"); and
- ACES' support for multiple data standards and separation of dataset-specific and dataset-agnostic components (**Reviewer ntJe:** "...showing potential for broad applicability across various datasets and tasks"; **Reviewer Qm8X:** "...separation of...components allows flexible and adaptable cohort extraction across diverse clinical datasets").

We have responded to each reviewer's concerns and questions individually as comments. We'd also like to highlight the following regarding the usability and flexibility of ACES:

## Existing public efforts that depend on ACES
Reviewers have noted challenges in discerning the transformative impact of ACES on EHR data processing and raised concerns about its utility for more complex tasks. In response, we highlight the MEDS-DEV benchmark, a public open-source effort described in Section 4.3 and the Appendix, which actively leverages ACES configuration files. This benchmark, involving researchers across multiple institutions worldwide, demonstrates ACES' capacity to address practical challenges such as standardizing cohort extraction pipelines and facilitating reproducibility even in scenarios where private data access is restricted (as discussed in Section 4.3). The adoption of ACES in MEDS-DEV provides concrete evidence of its emerging role in healthcare ML and that the growing MEDS community recognizes ACES' potential in ML4H.

## ACES complements existing tools

As mentioned in our updated paper, ACES is fully compliant with OMOP datasets via the MEDS-OMOP connector, which has been extensively tested by the MEDS team. As such, rather than becoming a separate tool for cohort processing, we believe ACES can complement many existing libraries and standards. Additionally, by interfacing with the relatively low-complexity MEDS schema, ACES can easily provide wide support for many CDMs as future ETLs and connectors continue to be developed.


We also added the following in Section 4.1 of the updated paper regarding existing alternatives for cohort extraction:
```ACES serves as a middle ground between solutions that focus on specific CDMs, such as i2b2 PIC-SURE and OHDSI ATLAS. Compared to these tools, ACES balances capability with greater ease of use and improved communicative value. ACES is also not tied to a particular CDM. Built on a flexible event-stream format, ACES is a no-code solution with a descriptive input format, permitting easy and wide iteration over task definitions. It can be applied to a variety of schemas, making it a versatile tool suitable for diverse research needs.```


## Thank you!
We sincerely thank all the reviewers and ACs involved in the review process. Your insights have been invaluable, and we always welcome any additional feedback and/or questions regarding ACES!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733250381601

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additional quantitative experimental results

comment:
We would like to once again thank all reviewers and the AC for your service, and appreciate all the feedback and comments during this reviewing period, which have been tremendously helpful in improving our work.

## **Experiments**
Some reviewers hoped to see additional quantitative experiments evaluating ACES in comparison to other cohort extraction tools. We sincerely apologize for the delay in posting these results as we wanted to ensure we conducted these experiments thoroughly and in good faith. Despite some challenges with OMOP ETLs, we would like to present the following:

- Using the OMOP version of MIMIC-IV-Demo, as well as a synthetic dataset of 1,000 patients generated using Synthea and converted into OMOP, we queried four tasks using ACES, OMOP-learn, and DPM360
- We collected metrics including script runtime, peak memory usage (in MiBs), lines of code required (including configuration files and any template code needed to execute extraction), and human time spent
- Experiments were conducted on a default GCP A100 GPU-instance with 84 GB of RAM and 12 CPU cores

Full results are shown in the table. ACES proved to be **generally computationally more efficient**, and does not rely on SQL connections and long queries with complex SQL scripts, which **can be brittle and time-intensive to implement**. As ACES does not require any additional code beside a single command-line invocation, it significantly **lowers the barrier to entry** for researchers without extensive experience with OMOP and SQL. We do acknowledge the potential for bias in estimating human time, so these estimates are offered as a rough measure. Nonetheless, the results suggest a clear advantage for ACES in terms of user experience and worked out-of-the-box with no further errors needed to be resolved.

| Method              | Dataset         | Task                                | Runtime (s)  | Peak Memory Cost (MiBs) | Lines of Code (#) | Human Time (s) | Adaptability to Other Tasks             |
|---------------------|-----------------|-------------------------------------|----------|------------------|---------------|------------|-----------------------------------------|
| ACES via MEDS       | Synthea-1000    | First 24h in-hospital mortality     | 0.386    | 389              | 35            | 120        | Edits to task configuration file        |
| ACES via MEDS       | Synthea-1000    | 30d post-hospital-discharge mortality | 0.236  | 351              | 32            | 90         |    -    |                                        |
| ACES via MEDS       | Synthea-1000    | 30d re-admission                    | 0.337    | 355              | 22            | 60         |     -    |                                        |
| ACES via MEDS       | Synthea-1000    | End-of-Life prediction              | 0.449    | 421              | 28            | 120        |    -    |                                        |
| DPM360 via OMOP     | Synthea-1000    | First 24h in-hospital mortality     | 5.932    | 390              | 205           | 2126       | Requires new ATLAS or custom SQL queries|
| DPM360 via OMOP     | Synthea-1000    | 30d post-hospital-discharge mortality | 4.188  | 550              | 257           | 1200       |        -                              |
| DPM360 via OMOP     | Synthea-1000    | 30d re-admission                    | 6.26     | 870              | 288           | 2020       |                 -           |
| DPM360 via OMOP| Synthea-1000    | End-of-Life prediction              | 4.901    | 387  | 222  | 1500       |        -      |
| ACES via MEDS | MIMIC-IV-Demo   | First 24h in-hospital mortality     | 0.617    | 545    | 35  | 180   | Edits to task configuration |
| ACES via MEDS       | MIMIC-IV-Demo   | 30d post-hospital-discharge mortality | 0.301  | 509       | 32            | 90         |   -  |
| ACES via MEDS       | MIMIC-IV-Demo   | 30d re-admission                    | 0.455    | 532        | 22    | 90         |              -       |
| ACES via MEDS       | MIMIC-IV-Demo   | End-of-Life prediction              | 0.349    | 589    | 28     | 300        |    -   |
| OMOP-learn via OMOP | MIMIC-IV-Demo   | First 24h in-hospital mortality     | 12.22    | 688  | 172           | 3623       | Requires new SQL scripts + changes to Python parameters |
| OMOP-learn via OMOP | MIMIC-IV-Demo   | 30d post-hospital-discharge mortality | 8.608  | 587  | 199           | 2441       |      -        |
| OMOP-learn via OMOP | MIMIC-IV-Demo   | 30d re-admission  | 19.71    | 640  | 168           | 2998       |        -   |
| OMOP-learn via OMOP | MIMIC-IV-Demo   | End-of-Life prediction     | 24.54    | 932              | 251           | 12000      |    -      |

We will include these additional experiments in our paper and hope that they help address any reviewer concerns. We look forward to continuing to improve ACES.

Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a new cohort extraction system for event-stream EHR datasets, designed to enable cohort replication across datasets, which in turn will facilitate the replication and comparison of analyses across datasets. The method encompasses a configuration language and an extraction pipeline. 

The problem addresses an important and insufficiently addressed problem in machine learning for healthcare. Most (3/4) of the reviewers have expressed the view that this paper brings a valuable contribution to the community. The reviewers appreciated how this work complements existing studies, that a well-documented python package is being released, and that it supports multiple data standards (like MEDS and ESGPT) seamlessly.

A single reviewer (z6xZ) expressed concerns that this is mainly an engineering effort, rather than conceptual innovation. However, the authors have rebutted this argument by pointing out that infrastructure development is within the scope of ICLR, and that ACES also defined a new language to facilitate access to EHR cohorts. I fully agree with the authors’ arguments in this.

Another point that was raised during the rebuttal was related to comparisons to existing cohort extraction tools. The authors have run additional experiments, showing that their method is generally more computationally efficient (in terms of runtime, and memory), as well as reducing the amount of code needed and time from an operator compared to existing techniques. I do find these results convincing, with the caveat that the human operators were familiar with the tool, and I do expect some training to be needed before it can be used effectively.

At any rate, I recommend acceptance of this paper, as it solves an important problem, provides an open-source framework, and has shown to be valuable in practical scenarios.