ComputAgeBench: Epigenetic Aging Clocks Benchmark

Authors: Dmitrii Kriukov, Evgeniy Efimov, Kuzmina Ekaterina, Ekaterina Khrameeva, Dmitry V. Dylov
Primary area: datasets and benchmarks
Keywords: biological age, epigenetic aging clocks, DNA methylation, aging biomarkers, longevity

Abstract:
The success of clinical trials of longevity drugs relies heavily on identifying integrative health and aging biomarkers, such as biological age. Epigenetic aging clocks predict the biological age of an individual using their DNA methylation profiles, commonly retrieved from blood samples. However, there is no standardized methodology to validate and compare epigenetic clock models as yet. We propose ComputAgeBench, a unifying framework that comprises such a methodology and a dataset for comprehensive benchmarking of different clinically relevant aging clocks. Our methodology exploits the core idea that reliable aging clocks must be able to distinguish between healthy individuals and those with aging-accelerating conditions. Specifically, we collected and harmonized 66 public datasets of blood DNA methylation, covering 19 such conditions across different ages and tested 13 published clock models. We believe our work will bring the fields of aging biology and machine learning closer together for the research on reliable biomarkers of health and aging.

Decision: reject
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730577881483

thread_type:
official_review

<thread_object>
creator: reviewer_jkrr

<content>
summary:
The paper proposes ComputAgeBench, a unified framework and benchmark dataset for evaluating epigenetic aging clocks, which are predictive models for estimating biological age based on DNA methylation data.  The framework aggregates 66 public datasets covering 19 aging-accelerating conditions, along with 13 published epigenetic clock models, to assess model performance consistently across a standardized dataset. The methodology incorporates rigorous evaluation criteria to test each model’s ability to distinguish between healthy individuals and those with accelerated aging conditions.

strengths:
### Strengths 

 The paper is clear and well-written, providing a solid foundation for its contributions. It presents a unified framework for evaluating epigenetic aging clocks, covering both first- and second-generation clocks. By introducing a benchmark dataset, the authors enable comprehensive testing of multiple epigenetic clock methods. 

This work has potential to significantly impact the field of biological aging, as it offers a standardized dataset that can facilitate consistent evaluation across various epigenetic clock methods. Such a resource will likely streamline method comparison and improve reliability in aging research.

weaknesses:
In reviewing the proposed benchmark in this paper, several key areas for improvement have emerged, particularly concerning data diversity, balance, and bias. 

  

### Weaknesses 

  

1. **Limited Report on Data Diversity**: The paper lacks adequate details on demographic and biological diversity, such as age, ethnicity, and health variations. Including these would improve the dataset's representativeness for broader applications. 

  

2. **Data Balance and Bias**: The authors do not address balance across categories (e.g., AACs vs. healthy controls) or potential sampling biases. This oversight may skew benchmarking results and limit generalizability. 

  

3. **Absence of Bias Mitigation**: No strategies are mentioned to detect or reduce dataset biases, which is crucial for fair benchmarking in aging prediction models, where demographic factors can affect DNA methylation patterns and model performance. Additional evaluation metrics for fairness would  increase the strength of this benchmark. 

  

4. **Put Together Publicly Available Dataset**: The proposed dataset, to my understanding,  is a collection of existing publicly available datasets. The authors do not present to the research community a new benchmarking dataset, they rather collect existing datasets that they put together with a published harmonization technique.  

The fact that the datasets already exist publicly, reduces the novelty of the benchmark. However, I cannot ignore that putting together 66 datasets  into a single dataset is a contribution that would facitilitate the comparison of epigenetic clock methods.

questions:
### Questions for the Authors

In evaluating the dataset and methodology presented, several questions arose that could help clarify the dataset’s potential applications and limitations.

1. **Applicability for Method Development**: Can this dataset be effectively used for developing new methods on epigenetic aging clocks, or is it primarily intended for benchmarking and evaluation? Are there features or structures in the dataset that support novel method exploration?

2. **Data Diversity and Representativeness**: How does the dataset account for demographic and biological diversity? Could the authors provide more details on the inclusion criteria to ensure the dataset is representative of a broad population?

3. **Addressing Balance and Bias**: Were any steps taken to balance the dataset across aging-accelerating conditions (AACs) and healthy controls, or to mitigate known biases in the sample selection process?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598530948

thread_type:
official_review

<thread_object>
creator: reviewer_3vvy

<content>
summary:
The authors present a benchmark study where they contrast different computational methods, namely aging clocks, for inferring biological age from epigenetics (methylation) data. A corpus of datasets relevant for the benchmark was built through a systematic search, and it is provided as a resource. Finally, the evaluation was performed on four different tasks, devised in such a way to capture different aspects of aging clocks' performances.

strengths:
The benchmark is well structured: (i) a variety of datasets and methods are included, and (ii) the tasks upon which the methods are evaluated are well defined and relevant for the domain. Furthermore, such type of benchmarks are quite timely, due to a continuously growing list of available aging clocks.

weaknesses:
My main criticism is that the paper is only marginally relevant with respect to the topics of the conference. Inferring the biological age of an individual can hardly be considered as learning representations. The machine learning methods used for deriving aging clocks are very well known and established, thus lacking novelty. The tasks presented in the paper to assess the clocks' performances are not totally novel, as the authors themselves point out in section 2.2.

From a technical point of view, an important aspect that the paper does not address is preprocessing. Several normalization methods exist for methylation data, and their impact to downstream analysis is well documented (see for example Teschendorff et al. 2013). A robust benchmark should try to evaluate the effect of different normalization methods on aging clock performances. 

A minor issue the authors may want to consider: the long list of reference at page 6 could be placed in the appendix, to ease reading

Andrew E. Teschendorff, Francesco Marabita, Matthias Lechner, Thomas Bartlett, Jesper Tegner, David Gomez-Cabrero, Stephan Beck, A beta-mixture quantile normalization method for correcting probe design bias in Illumina Infinium 450 k DNA methylation data, Bioinformatics, Volume 29, Issue 2, January 2013, Pages 189–196,

questions:
I would like to ask the authors to address the two main criticisms I listed in the "weaknesses" section:
- Overall, the opinion of this reviewer is that while the work has undoubtedly merit, it would be better suited for a forum more specific to biological age and aging clocks. 
- Regarding the normalization of methylation data, I would invite the authors to at least discuss whether the preprocessing of the included datasets match the recommended preprocessing of each aging clock (if any).
</content>
</thread_object>

</thread>
<thread>
creation date:
1731305334113

thread_type:
official_review

<thread_object>
creator: reviewer_xgqd

<content>
summary:
This paper benchmarks 13 different published biological clock models using a standardized test dataset that they compiled from more than 50 different publicly available studies. While no ground truth data is available for biological age (as it is a latent factor) or for age at death (as this data often isn’t published), the authors offer 4 compelling metrics by which to score the models accuracy and robustness. This paper presents a resource to the community in terms of a newly published benchmarking dataset, well-motivated metrics, and ratings for the current state of the art clock models. The paper also appropriately outlines limitations, such as the fact that some datasets had poor performance across all models, raising questions about dataset shift and for what kinds of data the clocks can be expected to make sound predictions. I believe this paper will help generate scientific discussion and progress in the aging clocks research community.

strengths:
- This paper is written very clearly, and did a great job walking the reader through the background to the problem, definitions of biological age, and different kinds of biological clock models. It’s graphics are informative, clear, and aesthetic. Truly a pleasure to read! 
- Provides colab notebook for reproducibility
- I believe this paper will be significant to those in the biological clocks community. It is a benchmarking paper, so while it doesn't offer a new methodology itself, it does offer original tasks/metrics for assessing the performance of these models (I think they are original, I asked for clarification in the questions section) and a standardized benchmarking dataset (I asked for clarity to confirm it will in fact be published along with this paper)

weaknesses:
- I was disappointed that the clock models weren't all re-trained on a standardized training dataset. Without standardizing the training data, it is impossible to know whether the methodology of the clock or the training data it used are contributing to better/worse performance. This insight would be critical to the community in improving clock methodologies going forward.
- The way that the authors chose to combine benchmarks in the cumulative score requires more justification. I am not sure why the different metrics should affect each other's weights so much. A simple sum, or weighted sum, of the four variables might be more appropriate if stronger justification is not supplied.
- Requires clarification: on the one hand, authors write "Clearly, the first task [AA1] provides a more rigorous way to test aging clocks [compared to AA2]" on the other hand, they write "The most rigorous of the four, AA2 task demonstrates..."
- Your description of the biomarker paradox could be improved. When I first read your description, I was left with questions. I had trouble finding more info on the "paradox of biomarkers" using the papers you cited (possibly due to paywall issues, I couldn't see the full articles), but you might consider adding this reference _Sluiskes, Marije H., et al. "Clarifying the biological and statistical assumptions of cross-sectional biological age predictors: an elaborate illustration using synthetic and real data." BMC Medical Research Methodology 24.1 (2024): 58._ as their explanation made me fully understand the problem, namely that "a (bio)marker that perfectly correlates with chronological age is useless in estimating biological age... in principle a nearly perfect chronological age predictor can be developed, as long as the sample size is large enough [35]. In such a case all signal related to biological aging would be lost."

More broadly, while I really enjoyed the paper, I am not sure it is a great fit for the ICLR community, as this model is a predictive regression model and not in the space of representation learning.

questions:
- Will you make your benchmarking dataset publicly available? Can you please add a link to it in your manuscript? I view this benchmarking dataset as a significant portion of your contribution in this work.
- Can you please confirm that your evaluation tasks/metrics are original, and add citations if not?
- Can you make a case for why the paper is a strong fit for ICLR, despite not truly being in the representation learning space?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731347874996

thread_type:
official_review

<thread_object>
creator: reviewer_px5y

<content>
summary:
The author introduces a benchmark designed to evaluate models of the epigenetic aging clock. The benchmark includes 66 datasets containing DNA methylation data that meet specific conditions and corresponding metadata, with a total sample size of 10,410. Four tasks are proposed to assess the models’ ability to distinguish between healthy individuals(HC) and age-accelerating conditions(ACC). Results of these four tests are summarized into Cumulative Benchmarking Score. The benchmark framework also includes 13 previously published models results.

strengths:
The author critiques previous benchmarks for being either small in scale, limited to predicting chronological age, lacking standardized datasets, comparing only a limited number of models, or relying on mortality and disease data that have restricted access. 

The proposed benchmark seems address all of these limitations. Derived from publicly accessible data, it includes processing of data from both age accelerating condition (ACC) and healthy control (HC) groups to test model’s ability to distinguish between these conditions. Diseases with ACC are well considered. The benchmark includes 4 well-defined tasks with a summary score and evaluates 13 previously published models.

weaknesses:
The paper is well-written and comprehensive overall, but several technical points need further clarification:

1. The selection of metrics for benchmark tasks requires more justification. Specifically, why do tasks 2, 3, and 4 report median instead of the mean? Additionally, task 4 mentions the "presence of covariate shift," but this shift is not clearly explained. Could the authors specify the covariate shift further ?

2. The rationale behind the summary benchmark score requires further explanation. Why was this scoring method chosen, and what are its advantages? Also, what does "positive bias" refer to in this context? In the Results section, it is stated that $S_{AA1}$ is adjusted by a ratio to penalize prediction bias, yet this concept of prediction bias remains unexplained. Further clarification on what prediction bias entails here would be beneficial.

3. It appears that plots C and D in Figure 3 may be incorrectly presented. Plot D should likely represent $Med(|\Delta|)$ rather than $Med(\Delta)$, as all points are above the diagonal. Please clarify if this is a mislabeling or if I have misunderstood the data shown.

questions:
Please see my questions in the above weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732119627349

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
Dear Reviewers,

We sincerely thank you for your thoughtful and encouraging reviews. Your recognition of our work as *“likely to streamline method comparison and improve reliability in aging research”* (JKRR), *“well-defined and relevant for the domain”* (3vvy), *“a pleasure to read”* with metrics that *“foster progress in aging clock research”* (XgQd), and addressing key limitations in the field (PX5Y) inspires us greatly.

Several reviewers have expressed concerns regarding the relevance of our work to the ICLR community. In response, we emphasize that **biological age represents an interpretable latent variable** derived from primary biomarkers, with wide-ranging applications in computational and practical domains, including predicting mortality, assessing morbidity risk, and evaluating the efficacy of potential longevity interventions. Additionally, biological age **inherently lacks ground truth values**, that casts similarities with out-of-distribution (OOD) detection, unsupervised, self-supervised learning and other actively discussed topics at ICLR that heavily relies on data representations. We view our work as an invitation to the broader data science community to explore this challenging and impactful application of representation learning, **bridging the gap between clinical practice and machine learning**.

We deeply value your constructive suggestions and will work to incorporate them to further strengthen the manuscript. Please kindly refer to individual responses below for our point-by-point responses.
Thank you again for your time and insights. We are available for the follow-up discussion and further clarifications.

Best regards,

Authors of Submission 7894
</content>
</thread_object>

</thread>
<thread>
creation date:
1732627289620

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Submission update by Authors

comment:
Dear Chairs and Reviewers,

We have updated the submission PDF to address the points raised in the Reviews. Below is the list of changes we have introduced:

1. In Benchmarking Methodology section 3.5, we have corrected the typo in the AA1 task description, replacing “**median** aging acceleration” with “**mean** aging acceleration”, as per our response to reviewer PX5Y.

2. In Benchmarking Methodology section 3.5, we have added the explicit mentions of the tasks: “Clearly, the first task (**AA2**) provides a more rigorous way to test aging clocks **compared to AA1**, because it helps to control potential covariate shifts, but the second task (**AA1**) deserves its place in the list, as it allows including more data into the panel to overcome data scarcity”, as per our response to reviewer XgQd.

3. In Benchmarking Methodology section 3.5, we replaced the last paragraph describing the 4th task (prediction bias task) with the following to include a deeper explanation of covariate shift, as per our response to reviewer PX5Y: “We introduced the fourth task, a prediction bias task, to evaluate the robustness of a given aging clock model to covariate shift between the original clock training dataset and the datasets from the proposed benchmark. Covariate shift, also referred to as batch effect in bioinformatics, denotes the shift between covariate distributions in two datasets. For instance, the distribution of methylation values for a given CpG site could be centered around 0.45 in one dataset and around 0.55 in the other one—a common scenario in DNAm and other omics data. Because each clock is trained on healthy controls, we expect age deviation of HC samples to be zero on average (*i.e.*, $E(\Delta_{HC})=0$). In practice, however, due to the presence of a covariate shift between the training and testing data, a clock might produce biased predictions, resulting in a systemic bias and adding or subtracting extra years for a healthy individual coming from an external dataset. The goal of the fourth task is to control for such systemic bias in clock predictions. Therefore, as a benchmarking metric for this task, we calculated median aging acceleration ($Med(\Delta)$) across HC samples from the entire dataset panel, which reflects the systematic shift in clock predictions caused by differences between datasets.”

4. In Benchmarking Methodology section 3.6, we added the following sentences before the last paragraph, addressing our discussion with reviewer PX5Y: “While designing our metric, we aimed for simplicity and interpretability. At the same time, we sought to include more data in the benchmark to address data scarcity caused by the underrepresentation of certain AACs.”

5. In the caption of Figure 3, we added the following, as per our response to reviewer PX5Y: “(C) illustrates that chronological age prediction accuracy is measured by median absolute error ($Med(|\Delta|)$) across all predictions. For a limiting case of prediction bias sketched in (D), all samples were predicted with positive age acceleration, leading to a strictly positive value of $Med(\Delta)$, graphically represented as a red arrow pointing to a cross.”

6. In Appendix section A.9, we clarified the rationale behind not performing any additional data procesing and inter-dataset normalization, as per our response to reviewer 3vvy, writing the following: “As there is no gold standard for DNAm pre-processing, each research group carries out their preferred pipeline that does not necessarily match the processing pipeline used for training the clock model, especially in case of applying earlier clocks (e.g., those by Hannum et al. (2013) or Horvath (2013)). Therefore, so as to retain this typical workflow and not to put any clock model into advantage by choosing the same processing that matches its own pipeline for every dataset, we did not perform any post-processing, inter-dataset normalization, or batch effect correction. In doing so, we also relied on two existing papers. First, compiling already pre-processed datasets without performing the same processing for all of them was done by Ying et al. (2023), another notable effort in the aging clock community. Second, we were also encouraged by a recent work by Varshavsky et al. (2023) who managed to create an accurate clock model by combining several blood datasets—without any additional normalization or correction procedure, using already pre-processed data from previous studies (some of which are included in our dataset as well), and thus demonstrating that the between-dataset normalization is not critical for this type of data.”

Best regards,

Authors of Submission 7894
</content>
</thread_object>

</thread>
<thread>
creation date:
1732782168436

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Post-discussion

comment:
Dear reviewers,

As the discussion period is about to end, we kindly want to thank all reviewers for their time and useful comments. Also, please let us know if there are any other questions or comments, which we are ready to address promptly.

Our submission in a nutshell:
- Clear **criteria and methodology** of assessment tasks for comparing clock models
- Systematic curation of **66 public datasets**
- Comparison of 13 published biological aging clock models
- 19 health conditions across different ages and populations

Sincerely, Authors of submission 7894
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a framework for evaluating 13 published pre-trained linear regression-based biological clock models. Because no ground truth exists for biological age (a univariate latent representation), the paper presents a scoring mechanism to evaluate them on four related tasks.

**Strengths:**
 - The paper is very well written and easy to follow. 
 - The compilation of the datasets is impressive and represents a thorough examination of how to compare epigenetic clock models.
 - The authors have participated heavily in the discussion period with lengthy responses to reviewer concerns.

**Weaknesses:**
 - The consistent and fundamental concern is whether this work is a good fit for the ICLR community. Put another way, it is not obvious whether the researchers who work on learning representations would benefit from this work. The models compared are all essentially the same linear regression model that have been pre-trained on different datasets. Several reviewers pointed out this flaw and remain unconvinced with the author rebuttal.
 - I want to acknowledge that the authors have made several arguments to defend the claim that the work is relevant to ICLR. They have stated that: a) biological age is inherently a learned representation lacking ground truth, b) they have cited several similar works published at comparable ML venues, and c) the key contribution of the work is not the compilation of datasets but instead "validating a clock's predictive ability to notice significant changes in patient's health, thus being a good indicator of biological age" (from author response to Reviewer 3vvy). 
 - After close examination of the paper, reviewer comments, and author responses, I remain unconvinced by a) and b). For a), if learning biological age is a relevant representation-learning task, it is unclear why other methods from the ICLR community are not used. The authors list out-of-distribution (OOD) detection, unsupervised, and self-supervised learning as relevant methods, but none of these are used in the linear-regression methods that are benchmarked. For b), the listed similar works (in response to Reviewer XgQd) are indeed ML-venue benchmark and methodological papers on aging models using -omics data. However, the listed works focus much more on novel architectures or benchmarking more sophisticated models such as language models or generative models. 
 - In response to c), the judgement about whether the work is clinically relevant is not the purview of ICLR. If this is indeed one of the main contributions of the work --- especially important in light of my thoughts on a) and b) --- then the work is better suited for a more relevant clinical venue.

The fit with ICLR's focus on methods, datasets, and benchmarks for learning representations weighed the most in my decision.