The Genomics Long-Range Benchmark: Advancing DNA Language Models

Authors: Evan Trop, Yair Schiff, Edgar Mariano Marroquin, Chia Hsiang Kao, Aaron Gokaslan, McKinley Polen, Mingyi Shao, Aymen Kallala, Bernardo P de Almeida, Thomas PIERROT, Yang I Li, Volodymyr Kuleshov
Primary area: datasets and benchmarks
Keywords: DNA, Language Models, Genomics, Benchmark

Abstract:
The advent of language models (LMs) in genomics necessitates benchmarks that can assess models’ capabilities and limitations. In contrast to protein models, DNA LMs can be used to study non-coding regions of the genome and must account for unique challenges, especially interactions across long sequence lengths. However, existing benchmarks for DNA LMs are defined over short sequence datasets and can involve tasks that are often not considered to be biologically meaningful. Here, we present the Human Genomics Long-Range Benchmark (LRB), which focuses on biologically meaningful tasks and supports long-range contexts. We complement our benchmark with fine-tuning recipes that meaningfully improve performance and affect model evaluation. We evaluate DNA LMs across nine compiled human genome tasks and observe that DNA LMs achieve competitive performance relative to supervised baselines on several tasks (e.g., genome annotation), but there remains a significant gap in domains, such as variant effect and gene expression prediction. Additionally, we introduce a visualization tool to examine model performance split by various genomic properties. Lastly, we present methods for context-length extrapolation of transformer-based models that enable studying the effect of context length on DNA LM performance.

Decision: reject
Reviewer scores: 6, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730569979187

thread_type:
official_review

<thread_object>
creator: reviewer_39ce

<content>
summary:
This paper introduces the Genomics Long-Range Benchmark (LRB), designed to evaluate DNA language models (LMs) on tasks that reflect biologically meaningful long-range interactions. The benchmark includes tasks across variant effect prediction, gene expression prediction, regulatory element detection, and chromatin feature identification.

strengths:
1. The LRB addresses a critical gap in DNA LM evaluation by focusing on biologically relevant tasks that require long genomic contexts. 
2. The experiments on DNA LMs, including zero-shot and fine-tuning performance across multiple tasks, reveal the strengths and limitations of the models.
3. Fine-tuning recipes and context-length extension methods provide a robust framework for DNA LM evaluation.

weaknesses:
- Lack of in-depth analysis of the experiments. Why do certain DNA LMs perform well on specific tasks?
- Potential unfairness in comparisons. DNABERT and HyenaDNA have significantly fewer parameters compared to NT500M, which may skew results. It would be beneficial to compare models with similar parameter counts where possible.
- Missing key long-range baseline LMs. The benchmark lacks important long-sequence models such as Caduceus [1] and Evo [2], which would provide a more comprehensive evaluation.
- Insufficient comparison in context extension experiments. The analysis of TSS distance effects lacks comparisons with other long-sequence models.
- References on benchmarks. In the first paragraph of the Introduction, the reference to ProteinGym [3] should have the publication year 2023 instead of 2024. Additionally, including relevant benchmarks like GenBench [4] and BEACON [5] would improve the coverage of related literature.

[1] Caduceus: Bi-directional equivariant long-range dna sequence modeling

[2] Sequence modeling and design from molecular to genome scale with Evo

[3] Proteingym: Large-scale benchmarks for protein fitness prediction and design

[4] GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models

[5] BEACON: Benchmark for Comprehensive RNA Tasks and Language Models

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665030299

thread_type:
official_review

<thread_object>
creator: reviewer_frd8

<content>
summary:
This paper proposes a new benchmark for evaluating DNA LMs on tasks with emphasis on long-range prediction, unlike other benchmarks which focus on short sequence tasks (<2k bp). The authors benchmark 4 DNA LMs on 4 tasks in their benchmark (variant effect prediction, gene expression prediction, and cis-regulatory element detection, chromatin feature identification). They compare performance on both zero-shot and full finetune settings. The authors also finetune NucleotideTransformer with extended context length.

strengths:
- Benchmarks for DNA language models are currently sparse, and this paper does a good job of curating a diverse set of tasks and benchmarking recent DNA LMs.

weaknesses:
- A main claim of this paper is that the tasks they curate are long-range tasks, and that they expect that model performance increases with longer context input. The claim that these tasks require long context would be strengthened by an ablation study over different input lengths.

- The authors should be clear that this is a human-only benchmark, ideally in the title and abstract. This is not mentioned until Section 3, and limits the usefulness of the benchmark as many DNA LMs like Evo are trained primarily on microbial sequences.

questions:
- Section 4 describes context length extension, specifically using NTK and an attention implementation with sqrt(L) chunks. The latter is not explained in the paper or in the supplement.

-The authors do not explain how the train/test splits are generated. How is train/test leakage avoided? Do they split by sequence similarity thresholds?

- The Evo model (https://www.biorxiv.org/content/10.1101/2024.02.27.582234v2) and Caduceus (https://arxiv.org/abs/2403.03234) should be benchmarked if possible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694566662

thread_type:
official_review

<thread_object>
creator: reviewer_gkl3

<content>
summary:
The authors present a new benchmark for evaluating DNA language models (LMs) with a focus on context size studies. They compiled a set of both long-range and short-range downstream tasks for DNA LMs, including variant effect prediction, gene expression, regulatory element, chromatin feature predictions, etc. The tasks and datasets are well-documented, and the benchmark comes with user-friendly features such as customizable context size downloads and visualization tools. Although this benchmark has potential utility for the field, it lacks some important results and discussions. Therefore, I currently recommend a weak rejection of this paper. I am open to raising my score if these issues are adequately addressed.

strengths:
1. The benchmark has a clear theme centered on the study of context sizes, which is currently an important topic in the development and application of DNA LMs. This benchmark can be expected to yield valuable insights for the field.
2. The authors selected a biologically relevant array of downstream tasks.
3. The paper includes a thorough review of existing benchmarks.
4. Dataset details are well-documented.
5. The visualization tool appears useful and well-designed.

weaknesses:
1. I see the most significant contribution/novelty of this benchmark to be facilitating studies on context size, a point emphasized in both the title and introduction. However, the results section provides only a general and superficial discussion on this topic, and the study of context length is quite restricted to NT. To provide more insight, there should be an in-depth analysis of the impact of context sizes on individual tasks and models. For instance, does each model empirically benefit from longer context lengths, and to what extent? Do certain tasks show a stronger dependence on longer context sizes as expected? There are more detailed evaluation results in Tables 11 and 12 in the appendix but lack an interpretation of the data. I would suggest creating some figures/tables to summarize these results and have more discussion on the impact of context size.
2. There is a missing discussion regarding alignment-based DNA LMs, which could have different context-size dependencies than single-sequence DNA LMs. Including this aspect is crucial for a complete and accurate narrative. This work builds on the GPN-MSA ClinVar and OMIM benchmarks but strangely does not include (nor even mention) the GPN-MSA model itself. GPN-MSA achieves SOTA on these tasks, performing better than CADD, and therefore better than all the DNA language models considered. It is important to include GPN-MSA in the benchmark (at least for zero-shot evaluations, if the authors deem fine-tuning to be cumbersome), since this could change the narrative in fundamental ways. First, it's not true that DNA language models do worse than CADD. This only applies to single-sequence models. Second, GPN-MSA achieves a good performance even with the smallest context. One interpretation is that with evolutionary context, one doesn’t need as much spatial context. Given that GPN-MSA is alignment-based, it could be reported by itself with a separator line in Table 3. Finally, even if the authors decide to restrict the discussion to single-sequence DNA LMs, it is conventional to compare with the actual SOTA on each task.

questions:
1. The sections on context length extrapolation read a bit disconnected from the rest of the manuscript. It appears to be an improvement on the NT model rather than directly related to the benchmark. If the authors claim it to be a generally applicable method to other DNA LMs, it should be made clear in the writing and preferably applied to at least another model. If it is for a focal investigation on the impact of context size on NT, the results should be more carefully analyzed and discussed.
2. In Table 3, Enformer is not a good baseline for the ClinVar task, since this set only contains missense variants and Enformer is a model focused on the non-coding genome.
3. In Section 3, the authors discussed why several tasks should be considered long-range, but did not discuss why the others are not. It would have been better to also include brief discussions on why those tasks are expected to be performed well with short-range models.
4. Section 3.1.3 and Line 1042: I’d like to point out that missense VEP is not necessarily a short-context task. Coding variants require a small protein context but a large genomic context, since the coding sequence is distributed across exons (which are very far away due to large introns).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714280377

thread_type:
official_review

<thread_object>
creator: reviewer_1nci

<content>
summary:
The paper presents the Genomics Long-Range Benchmark (LRB), a new suite of biologically meaningful tasks designed to evaluate DNA language models with a focus on long-range genomic contexts. The authors argue that existing benchmarks are limited by their emphasis on short sequences and sometimes biologically irrelevant tasks. They provide fine-tuning recipes to improve model performance, introduce a visualization tool for detailed analysis, and explore methods for extending the context length of transformer-based models.

strengths:
The paper identifies a significant gap in the evaluation of DNA LMs by focusing on long-range genomic interactions, which are key for understanding complex biological processes.

The LRB includes nine tasks covering variant effect prediction, gene expression prediction, regulatory element detection, and chromatin feature identification. This breadth ensures that models are tested on a variety of biologically relevant tasks.

Allowing users to select arbitrary sequence lengths for each task is very relevant for the field and facilitates the exploration of context length effects on model performance.

The authors demonstrate that full fine-tuning of DNA LMs significantly enhances performance compared to previous methods that froze the backbone model weights.

Exploring techniques to extend the context size of transformer-based models is a valuable contribution, especially given the computational challenges associated with long sequences.

weaknesses:
Although the paper compares DNA LMs to supervised baselines like Enformer and DeepSEA, it could include more recent or diverse models to strengthen the evaluation.

The results are presented with mean and standard deviation across folds, but there's no discussion of statistical significance. Including statistical tests would provide more confidence in the reported improvements.

questions:
I highly encourage the authors to:

Report statistical comparison between metrics.
Include more models in the benchmark. A good example is https://arxiv.org/abs/2406.10391
</content>
</thread_object>

</thread>
<thread>
creation date:
1733237451105

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of feedback and improvments

comment:
We thank the reviewers for their time and useful comments on our work.

We would like to summarize comments that were common to several reviewers and our high level responses here. For details please refer to the specific responses posted to each reviewer.

**Adding more models:** 3 new DNA LMs and 2 new supervised baselines
- We added results for a pre-trained Caduceus model (Schiff et al. 2024) on our benchmark
- We added Evo (Nguyen et al. 2024) on the zero-shot tasks 
- We added GPN-MSA (Benegas et al. 2024) for the zero shot-tasks
- We added two supervised baselines on to all fine-tuning tasks: A CNN model and a Caduceus model trained from scratch

**Probing the effect of context length:**

We added an experiment that examined the effect of context length for the long-range tasks in our benchmark. We found that for both of the new supervised training baselines, **there is a strong positive association between context length and performance,** validating our hypothesis about the importance of long-range modeling for these tasks.

Other improvements include:
- Adding statistical significance tests, per [Reviewer 1nci’s](https://openreview.net/forum?id=8O9HLDrmtq&noteId=bQVKep28TV) suggestion.
- Adding an extensive discussion about the drivers of varying DNA LM performance on downstream tasks, per [Reviewer 39ce’s](https://openreview.net/forum?id=8O9HLDrmtq&noteId=SVtqG7Iyaw) suggestion.

---

**References**

Benegas, Gonzalo, et al. "GPN-MSA: an alignment-based DNA language model for genome-wide variant effect prediction." bioRxiv (2023).

Nguyen, Eric, et al. "Sequence modeling and design from molecular to genome scale with Evo." Science 386.6723 (2024): eado9336.

Schiff, Yair, et al. "Caduceus: Bi-directional equivariant long-range dna sequence modeling." arXiv preprint arXiv:2403.03234 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a well-executed DNA benchmark paper. The reviewers were mostly well-aligned on their score in the eject to accept range. Compared to previous work like BEND - published at last year's ICLR - this paper adds five long range benchmarks.

This work definitely deserved publication. But given its incremental nature, arguably a specialized (bioinformatics) venue is a more obvious choice than ICLR.