Benchmarking DNA Sequence Models for Causal Variant Prediction in Human Genetics

Authors: Gonzalo Benegas, Gökcen Eraslan, Yun S. Song
Primary area: datasets and benchmarks
Keywords: genomics, genetics, variant effect prediction, DNA, language models

Abstract:
Machine learning holds immense promise in biology, particularly for the challenging task of identifying causal variants for Mendelian and complex traits.  Two primary approaches have emerged for this task: supervised sequence-to-function models trained on functional genomics experimental data and self-supervised DNA language models that learn evolutionary constraints on sequences.  However, the field currently lacks consistently curated datasets with accurate labels, especially for non-coding variants, that are necessary to comprehensively benchmark these models and advance the field.  In this work, we present TraitGym, a curated dataset of genetic variants that are either known to be causal or are strong candidates across 113 Mendelian and 83 complex traits, along with carefully constructed control variants.  We frame the causal variant prediction task as a binary classification problem and benchmark various models, including functional-genomics-supervised models, self-supervised models, models that combine machine learning predictions with curated annotation features, and ensembles of these.  Our results provide insights into the capabilities and limitations of different approaches for predicting the functional consequences of genetic variants.  We find that alignment-based models CADD and GPN-MSA compare favorably for Mendelian traits and complex disease traits, while functional-genomics-supervised models Enformer and Borzoi perform better for complex non-disease traits.  All curated benchmark data, together with training and benchmarking scripts, will be made publicly available upon publication.

Decision: reject
Reviewer scores: 6, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730011100232

thread_type:
official_review

<thread_object>
creator: reviewer_3fjz

<content>
summary:
This paper conducted a benchmarking study of DNA sequence models for predicting causal genetic variant for 113 Mendelian and 83 complex traits. They tested the performance of three different types of methods, including functional-genomics-supervised models, self-supervised genomic language models (gLMs), and integrative models that combine machine learning predictions with annotation features (from the viewpoint of ensemble learning). The benchmark results demonstrated differences in model performance across different types of trait, different model classes, and different variant types

strengths:
1.	The consideration of both Mendelian and complex traits provides a quite comprehensive dataset of genetic variants, which is a valuable contribution to the field. Especially the performance difference across different types of traits could deepen the understanding of how a causal genetic variant may contribute to a trait or disease.
2.	The computational problem is well-defined by treating it as a binary classification of causal versus non-causal variants. The negative non-causal variants set should also be very useful in many computational models where negative controls are needed.
3.	The authors demonstrate that combining different types of computational models leads to performance improvements, especially for complex traits. This finding suggests that distinct models capture complementary information that can be leveraged for more accurate predictions.

weaknesses:
1.	The numbers of both variants and traits are small. The way for the benchmark study to define the causal variants is too simple and might be controversial. Both OMIM pathogenic annotation and PIP>0.9 do not imply the variant is causal.  
2.	Many baseline methods were missed. The authors may refer to a very recent variant effect prediction benchmark study (doi: 10.1186/s13059-024-03314-7). 24 methods for variant effect prediction were benchmarked. In another benchmark study in 2023 (doi: 10.15252/msb.202211474), 55 variant effect predictors were benchmarked. Many of those methods were ignored in this study.
3.	The evaluation metric used by the benchmark study is too simple based on the binary classification problem. In most of the cases, the number of non-causal variants are far more than causal variants. The auPRC will be affected by the imbalance ratio. Other metrics, such as auROC, precision, recall, f1-score are also necessary to provide more comprehensive benchmark results.
4.	Why only choose 9 non-causal variants as negative controls? The true causal variant proportion for different traits or diseases might be significantly different. Using a constant proportion here for different trait might not be consistent with the true biology.

questions:
1.	The causal variants of Mendelian traits were collected from a single source (Smedley et al., 2016). The causal variants of complex traits were collected from also a single source (UKBiobank). It would be more meaningful if the authors could incorporate more complex traits and disorders with well-defined causal variants, such as those from FinnGen or other large BioBanks, to improve the benchmark’s generalizability.
2.	The authors need to address how biases in the single dataset curation might affect benchmarking results and suggest methods for mitigating these biases, such as incorporating data from additional cohorts or stratified sampling.
3.	Since many deep learning models were included. Any way for enhancing the interpretability of the results by analyzing the biological relevance of model predictions? For instance, interpret how causal variants affect gene regulation networks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730402731994

thread_type:
official_review

<thread_object>
creator: reviewer_bb5y

<content>
summary:
TraitGym introduces a benchmarking framework designed to evaluate DNA sequence models in their ability to distinguish causal from control variants across a wide range of traits, covering 113 Mendelian and 83 complex traits. The benchmark relies heavily on established datasets, particularly those curated and described in Finucane et al. (2019, 2024) and databases like gnomAD and OMIM. TraitGym’s primary contributions focus on developing matched negative control sets for variants labeled as pathogenic in previous studies, as well as introducing a new genomic language model, gLM-Promoter. This model claims to specifically train on promoter regions to capture regulatory features that may play a role in causal variant prediction, addressing a gap in current modeling approaches.

The framework evaluates several model types, including functional-genomics-supervised, self-supervised, and integrative models, on a binary classification task aimed at identifying causal versus non-causal variants. To improve predictive performance, TraitGym further explores ensemble approaches, combining features from different models to leverage distinct predictive signals. This work include s an stratification of variant-type and traits to further understand model performance. Additionally, it incorporates a feature interpretation analysis to reveal insights into trait-specific effects across tissues, aiding in the understanding of trait relevance and tissue-specific regulatory mechanisms associated with the variants under study.

strengths:
TraitGym provides a diverse evaluation framework by assessing a range of models, including supervised, self-supervised, and integrative types, enabling a comprehensive comparison across different model architectures. The benchmark’s use of publicly available data sources and well-established resources like gnomAD and OMIM enhances accessibility and reproducibility, contributing to transparency and ease of use for the broader community. The inclusion of the gLM-Promoter model offers a unique approach focused on promoter regions, which may uncover regulatory insights specifically related to these key genomic areas. Additionally, TraitGym’s exploration of ensemble methods, trait specific stratification, and feature interpretation for both CADD and Borzoi is interesting.

weaknesses:
Data Curation and Benchmark Design

The threshold for minor allele frequency (MAF) in pathogenic OMIM variants, set at 0.1%, appears arbitrary, particularly since a 1% threshold is typically standard for rare variants. Conducting ablation studies would clarify the impact of this threshold on model performance and help justify this design choice. The control matching process would also benefit from closest gene-specific stratification. By matching positive and negative samples more closely at the closest gene level, gene-specific biases could be reduced, making the control set construction more rigorous. Finally, the methodology of matching controls based on Euclidean distance across TSS distance, MAF, and LD score assumes these features are equally relevant, which may not hold true. Given that MAF and LD score are more population-dependent, while TSS distance is more functionally related, this approach risks introducing biases in the control set.

Model Evaluation Framework:

The leave-one-chromosome-out (LOCO) validation approach in the linear probing evaluations, while aiming to test model generalization, has notable limitations. Chromosomes differ significantly in size and gene density, leading to inconsistencies in split sizes and class balance for variant groups. Smaller chromosomes, such as chromosome 21, have fewer variants, which can skew model performance and impact the validity of linear probing. Furthermore, features like minor allele frequency (MAF) and linkage disequilibrium (LD) score vary by chromosome and could add further bias if not carefully accounted for during these experiments.

Additionally, the evaluation of the CADD model introduces a potential risk of data leakage. CADD is a meta-predictor that integrates conservation scores and population data, among other annotations, and if any benchmark variants overlap with those in CADD’s training data, the evaluation may be inadvertently biased in CADD's favor. A thorough check of these overlaps would mitigate this risk and ensure an unbiased comparison.

gLM-Promoter Model Clarity

The gLM-Promoter model’s design and training process lack sufficient detail, which weakens its clarity and potential impact. Missing information includes the model's approach to defining promoter regions, the rationale behind choosing a 512bp window size, and how TSS coordinates are determined across different genomes. The current evaluation appears limited and does not specify if the model was exclusively tested on promoter variants, nor does it present a comparison across different variant types. Furthermore, it is unclear whether the evaluated variants are distinct from those seen in the model’s training, particularly when using the human reference genome. Without a clearer contribution to the benchmark, the inclusion of gLM-Promoter could be reconsidered unless region-specific analysis is expanded.

Organization and Clarity Issues

Several presentation issues detract from the paper's readability and clarity. Figure 1, for instance, is overly complex and could benefit from simplification to improve interpretability. Additionally, the writing style is often verbose, and adopting a more concise approach would enhance clarity. For example, the parenthetical explanation of Mendelian versus complex traits in the introduction disrupts the flow unnecessarily and could be integrated more smoothly into the text.

questions:
Could the authors clarify whether variants used in the evaluation overlap with those in CADD’s training data? This would help ensure unbiased results by addressing potential data leakage.

Why was a threshold of nine negative controls chosen per causal variant? An explanation of how this number impacts benchmark robustness would provide greater clarity on the control selection process.

Were ablation studies conducted to assess the impact of the 0.1% MAF threshold for OMIM variants on model performance? A comparison with a 1% threshold would align with conventional standards and strengthen the study’s design choices.

Were any ablation studies conducted to explore a range of posterior inclusion probability (PIP) thresholds for the complex trait benchmark? Examining how different PIP thresholds impact model performance could provide valuable insights into the robustness of the benchmark design for complex traits.

Could the authors provide additional information on their approach for trait- and tissue-based evaluations? Including positive and negative splits for these analyses, as well as integrating plausibly causal eQTLs (e.g., from GTEx fine-mapping), could enhance interpretability and contribute novel insights into trait- and tissue-specific variant effects.

Would the authors consider further expanding the trait analysis by integrating eQTL datasets (such as GTEx fine-mapping) that overlap with causal variants from Finucane et al. (2015) or OMIM? This approach could create a unique dataset combining eQTLs with fine-mapped UKBB GWAS variants, potentially providing additional validation and highlighting cases where expression changes may not fully capture phenotypic effects at the organism level.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694449282

thread_type:
official_review

<thread_object>
creator: reviewer_ub7o

<content>
summary:
This paper describes the curation of two benchmark datasets for evaluating approaches to predicting causal genetic variants. A study was performed to compare several such existing methods (including one trained by themselves) from three categories using the proposed benchmarks.

strengths:
-  After made available, the two curated benchmark datasets can help to advance the field of developing computational approaches to identifying genetic variants with major functional consequences. 

- It is interesting to see that considering alignment is important in self-supervised training for causal variants prediction. (By comparing gLM-Promoter and GPN-MSA if I understand correctly.) 

- Analyzed model predictions from varying prospectives and the obtained results are interesting

- It is interesting to see the ensembling of outputs of different models leads to the best predictions.

weaknesses:
- The technical contribution may be considered low with minimal innovation.

- More discussion about gLM promoter and its difference from GPN-MSA and other self-supervised trained models would be helpful. 

- A cross comparison between the curated genetic variants with those from ClinVar would strengthen this study.

questions:
- Why choosing to use common variants as controls in the study of Mendelian traits? The causal variants of Mendelian traits are typically rare (i.e., low MAF).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718123720

thread_type:
official_review

<thread_object>
creator: reviewer_zxnj

<content>
summary:
This manuscript presents a comprehensive benchmarking analysis for a wide range of DNA models in the task of predicting causal non-coding variants. The benchmark spans multiple model types, including supervised sequence-to-expression models, self-supervised DNA models, and baselines leveraging genome annotation-based summary statistics. The authors evaluate models on both Mendelian and complex traits, with a further distinction between disease and non-disease traits.

strengths:
- This paper offers rigorous benchmarking for DNA models on causal variant prediction, ensuring a balanced comparison by controlling for factors such as distance to the transcription start site (TSS) and minor allele frequency (MAF) in the positive-negative matching.
- It evaluates an extensive array of model types, from supervised and self-supervised DNA models to non-ML baselines using CADD features, which adds robustness to the benchmark.

weaknesses:
- The benchmark could be better positioned within the landscape of existing DNA benchmarks. While the authors have added a paragraph since my last review they haven't answered the question of what this particular benchmark brings in terms of insights on top of other existing ones. The discussion is just focused on the source of the variants. I appreciate that the CADD models provide a good baseline, and other benchmarks may not have this.

questions:
- Please highlight any distinct model insights provided by this benchmark on top of other similar benchmark which is not based on the source of variants
</content>
</thread_object>

</thread>
<thread>
creation date:
1733120068410

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their time and thoughtful feedback. We have uploaded a new version of our manuscript to address their questions and concerns.

We reiterate our main goals, which we believe will help advance future research at the intersection of machine learning and human genetics:
- A systematic benchmark for the challenging task of predicting causal variants for human Mendelian and complex traits, focusing on the often neglected non-coding part of the genome.
- An evaluation of the latest generation of broad classes of models: functional-genomics-supervised, self-supervised, and integrative.

In response to the reviewers’ questions and feedback, we have made the following modifications:
- **eQTLs** (Reviewers zXNj, Bb5Y): We analyze variants overlapping fine-mapped GTEx eQTLs (Fig. 8D).
- **Expanded negative sets** (Reviewers Bb5Y, 3FJZ): We provide and analyze additional expanded datasets with millions of negative variants (Fig. A.7).
- **Embedding metrics** (Reviewer zXNj): We evaluate additional gLM zero-shot scores such as L2 distance, cosine distance (Table A.8).
- **gLM-Promoter** (Reviewers Ub7o, Bb5Y): We have expanded our description of gLM-Promoter (Section A.2.2) and analyzed its performance across variant consequences (Figure A.8). gLM-Promoter was updated with the results of a longer training run and a slightly different architecture (Table A.6).
- **Related work** (Reviewers zXNj, Ub7o): We have clarified the differences with benchmarks in Tang and Koo (2024), BEND and ClinVar.
- **Additional experiments** (Reviewer Bb5Y): We evaluated the impact of different MAF thresholds (Fig. A.1), PIP thresholds (Fig. A.4), overlap with CADD variants (Fig. A.3) and matching by closest gene (Fig. A.2, A.5).
- **Additional metrics** (Reviewer 3FJZ): We report AUROC (Fig. A.6). 
- **Pleiotropy**: We analyzed performance for variants affecting multiple traits (Fig. 8C).
- **Sei**: We evaluated Sei, a functional-genomics-supervised model from a different research group than Enformer and Borzoi.

We thank the reviewers for their feedback and would be happy to answer any further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents TraitGym, a curated dataset designed to benchmark DNA sequence models for predicting causal genetic variants. The paper benchmarks various models and collects publicly available data sources for benchmarking.

Strengths included the comprehensiveness of the models and data and the colocation of benchmarking data within the framework for the community.

Several weaknesses were identified including alignment with existing threshold standards in the genetics community and organization and clarity of the paper.  

The paper seems to be specifically about non-coding variants and does not address variant effect prediction for coding variants (per author response).  The abstract states as motivation, "...the field currently lacks consistently curated datasets with accurate labels, especially for non-coding variants, that are necessary to comprehensively benchmark these models...", but then states "In this work, we present TraitGym, a curated dataset of genetic variants that are either known to be causal or are strong candidates..." The motivating statement alludes to the focus on non-coding variants, but the main claim of the paper drops the distinction. Further, the title does not clarify that the focus is on non-coding variants. While the reviewers generally appreciated the distinction, there was some confusion. It is critical to align the claims of the paper with the evidence provided so that readers can properly place the paper among the literature. While the authors have offered many edits to the paper in the rebuttal period, which certainly improve on some of the weaknesses, the paper may need further edits to align the claims and evidence before acceptance.