Revisiting Convolution Architecture in the Realm of DNA Foundation Models

Authors: Yu Bo, Weian Mao, Yanjun Shao, Weiqiang Bai, Peng Ye, Xinzhu Ma, Junbo Zhao, Hao Chen, Chunhua Shen
Primary area: foundation or frontier models, including LLMs
Keywords: DNA modeling, foundation model, Genomic Language Model, Representation Learning

Abstract:
In recent years, A variety of methods based on Transformer and state space model (SSM) architectures have been proposed, advancing foundational DNA language models. 
However, there is a lack of comparison between these recent approaches and the classical architecture—convolutional networks (CNNs)—on foundation model benchmarks.
This raises the question: are CNNs truly being surpassed by these recent approaches based on transformer and SSM architectures? In this paper, we develop a simple but well-designed CNN-based method, termed ConvNova. ConvNova identifies and proposes three effective designs: 1) dilated convolutions, 2) gated convolutions, and 3) a dual-branch framework for gating mechanisms. 
Through extensive empirical experiments, we demonstrate that ConvNova significantly outperforms recent methods on more than half of the tasks across several foundation model benchmarks. For example, in histone-related tasks, ConvNova exceeds the second-best method by an average of 5.8\%, while generally utilizing fewer parameters and enabling faster computation.  In addition, the experiments observed findings that may be related to biological characteristics. This indicates that CNNs are still a strong competitor compared to Transformers and SSMs. We anticipate that this work will spark renewed interest in CNN-based methods for DNA foundation models.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729376647335

thread_type:
official_review

<thread_object>
creator: reviewer_2bpc

<content>
summary:
The paper argues that a well-designed convolutional neural network (CNN), when used as a DNA foundation model, can outperform Transformer- and SSM-based models not only in accuracy but also in inference speed, model size, and training cost. The authors introduce ConvNova, a model composed of dual-branched Gated Convolutional Blocks (GCBs). The GCBs incorporate dilated convolutions and avoid performance-degrading downsampling, as supported by empirical findings. Ablation studies demonstrate the benefits of the dilated convolutions, dual-branch architecture, and gated mechanisms within the GCBs. The proposed model achieves higher accuracy while maintaining a smaller parameter count compared to several state-of-the-art Transformer and SSM models.

strengths:
The authors conducted a series of carefully controlled experiments using different random seeds to demonstrate the accuracy advantages of their design. They compared the proposed ConvNova model against state-of-the-art Transformer- and SSM-based models across various tasks, including two benchmarks for short-range input sequences (the Nucleotide Transformer Benchmark and the Genomic Benchmark) and two benchmarks for long-range tasks (the Bend Gene Finding and the Chromatin Profile Prediction). Overall, the experiments and the ablation study are robust and effectively support the claims and the effectiveness of the proposed design.

weaknesses:
While the authors present solid empirical evidence, the paper lacks a clear discussion on the intuition behind the design, making it more like a technical report. For example, ConvNeXt [1] builds on a well-known variant, ResNeXt [2], and details the accuracy impact of each modification. In contrast, ConvNova does not seem to establish a clear rationale connecting it to previous CNN-based DNA foundation models, which makes the design choices appear somewhat ad hoc. Beyond demonstrating superior accuracy compared to Transformers and SSMs, it would be valuable to include an in-depth discussion and comparison with models like LegNet on block design, architecture, parameter count, and training schemes in the main text.

I found the organization of the paper somewhat confusing. For instance, the comparison of downsampling and dilation in Section 3.3 and Table 1 would be more appropriately placed in the experiments section. Additionally, given the variety of tasks with different state-of-the-art models and settings, I recommend adding a dedicated section that briefly outlines the experimental setup. This section should include the objectives of each task, descriptions of the baseline models, and the specific configurations used in each experiment.

[1] A ConvNet for the 2020s

[2] Aggregated Residual Transformations for Deep Neural Networks

questions:
- What is the intuition and rationale behind using dilation over self-attention in a DNA foundation model? And why is dilation outperforming self-attention?
- How is the receptive field for the dilated convolutions determined?
- What factors contribute to the dual-branch design outperforming a single-branch approach?
- In addition to the details in Table 11 and Section A.4, could you provide a comparison between ConvNova and LegNet in terms of structure and training scheme, and explain why your design is superior?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655846473

thread_type:
official_review

<thread_object>
creator: reviewer_sh7n

<content>
summary:
The paper proposes to revisit CNN based architectures for DNA sequence modeling, and propose a new CNN based architecture, ConvNova. The authors show, with an extensive empirical study,  that the model has state of the art performance on several DNA modeling task when controlling for model size.

strengths:
The authors have a high quality set of evaluation experiments for their method. This can be significant as it may lead to further research on CNN based architectures for sequence modeling. The writing is mostly clear and the author's proposal significantly improves the state of the art.

weaknesses:
The novelty of the approach is somewhat limited, as CNNs have already been applied to this setting, and the training procedure is broadly the same as in the HyenaDNA paper. 

The CNN architecture used is based on gated convolutions, which has already been proposed elsewhere, limiting the novelty. 

To better understand the novelty of the approach, a benchmark comparing with a more standard CNN architecture would have been useful. 

The paper seems to lack a clear motivation as to why this specific CNN architecture was proposed. 

I would have expected to find a description of the pretraining data in the main paper. 

I would say that calling the model a “foundation model” is somewhat misleading as the model used in the paper has 7M parameters and was pretrained for 14 hours on a single GPU.

questions:
What is the pretraining dataset?

Did you benchmark the model against a more traditional CNN architecture?

Is the architecture inspired by the DNA task in some way?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671086136

thread_type:
official_review

<thread_object>
creator: reviewer_pcer

<content>
summary:
The paper introduces a new convolutional-based DNA foundation model and shows that CNNs can be competitive with Transformers and SSMs both in terms of absolute performance as well as accuracy vs. speed tradeoffs

strengths:
The paper presents strong, rigorous, and detailed results, with several strong baselines
The significance of the model itself is high, both for the application domain but what it means in the context of SSMs.
To me, DNA seems like the prime use case that SSMs, based on their performance profile, should be the winning model. A pure convolutional model outperforming Cadesus is a very important result in the context, and adds to the growing body of work that show SSMs and their complexities may not be necessary.

weaknesses:
- The paper needs to do a better job, in the main text, of demonstrating what specifically about the model is novel, and what specifically contributes to the superior performance of this model compared to prior CNN-based DNA models. 

For example, is it just better pretraining data? Is it including the complement sequence? I saw dilations, and receptive field choice were shown to be a portion of this, and I saw the ablations on the different components. 

However, the specific novelty of this design is a bit ambiguous to me. The figures need to be clearer to emphasize the novel components compared to prior CNN-based models. Specifically, just answering the question in explicit detail: CNNs have been around for a while, why didn't we find a high-performing model like this well before this? What new has been added that wasn't found before?

There should be some more qualitative analysis in the main text, mainly I would like to see an understanding of how the different model classes compare in the types of errors they make. For example, does the smaller receptive field contribute to certain types of errors for tasks where longer range reasoning is necessary?

questions:
Also mentioned in the weaknesses section, CNNs have been around for a while, why didn't we find a high-performing model like this well before this? What new has been added that wasn't found before?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720476250

thread_type:
official_review

<thread_object>
creator: reviewer_nqxy

<content>
summary:
Since various Transformer and SSM-based models are proposed in DNA language modeling, the authors conduct empirical studies of whether CNNs are truly being surpassed by these recently proposed approaches. With analysis results, this paper develops a simple yet well-designed CNN-based method called ConvNova, which identifies and proposes three effective designs: 1) dilated convolutions, 2) gated convolutions, and 3) a dual-branch framework for gating mechanisms. Through extensive empirical experiments, we demonstrate that ConvNova significantly outperforms recent methods on more than half of the tasks across several foundation model benchmarks.

strengths:
* **(S1)** The paper addresses a critical problem in DNA embedding with a novel application of classical convolution, showing significant improvements in species-aware tasks.

* **(S2)** The overall presentation is well originated and easy to follow. It is clear that the authors provide step-by-step designs to improve the attention mechanism for better performance and efficiency.

weaknesses:
* **(W1)** Despite the efficient design, the proposed ConvNova is somewhat simple and lacks novelty and support. The (dilated) convolution with gating branch is not a new design (proposed by MogaNet [1] in 2022 and well studied by StarNet [2] and MambaOut [3] in 2024), especially after Mamba variants came out (i.e., the implicit long convolution with gating). The authors should discuss these background works (discussing in the related work section or comparing with them), and provide more supports of why the proposed design is specially useful in DNA applications (refer to Q1 for details).

* **(W2)** Although the authors have compared with several well-known DNA models, some recently published DNA models and pre-training works are overlooked (e.g., GPN [4], VQDNA [5], and DNABERT-S [6]). Meanwhile, there are various DNA benchmarks that should be referred to and compared (classical benchmarks like GUANinE v1.0 [7], BEND [8], and GUE [9]), especially some long-range benchmarks [10] where SSM-based models work well. From my perspective, I am still not sure or not convinced that the dilated convolution with gating aggregation could consistently outperform self-attention or SSM architectures.

### Reference

[1] MogaNet: Multi-order Gated Aggregation Network. ICLR, 2024.

[2] Rewrite the Stars. CVPR, 2024.

[3] MambaOut: Do We Really Need Mamba for Vision? arXiv, 2024.

[4] DNA language models are powerful predictors of genome-wide variant effects. PNAS, 2023.

[5] VQDNA: Unleashing the Power of Vector Quantization for Multi-Species Genomic Sequence Modeling. ICML, 2024.

[6] DNABERT-S: Learning Species-Aware DNA Embedding with Genome Foundation Models. arXiv, 2024.

[7] GUANinE v1.0: Benchmark Datasets for Genomic AI Sequence-to-Function Models. bioRxiv, 2023.

[8] BEND: Benchmarking DNA Language Models on biologically meaningful tasks. ICLR, 2024.

[9] DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome. ICLR, 2024.

[10] Advancing DNA Language Models: The Genomics Long-range Benchmark. ICLR Workshop on Machine Learning for Genomics Explorations, 2024.

questions:
* **(Q1)** Is there any empirical analysis or theoretical support to verify that the dilated convolution is capable and more efficient than self-attention or SSM modules on the DNA tasks? For example, the authors could visualize the reception field or analysis the learned patterns to show that the dilated convolutions with gating learn better.

* **(Q2)** Some questions about the hyper-parameters in the network. How to determine the kernel size and the dilated ratio in ConvNova? Are there any implementation details of the ablation studies and providing the corresponding configurations (in addition to Table 9)? Meanwhile, is there more ablation of different designs (like analysis in [1, 2, 3]) of the gating branches as mentioned in Eq. (1) and Appendix A.3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732900607466

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all the reviewers for their thoughtful feedback, constructive suggestions, and encouraging comments. Your insights have been invaluable in helping us refine our work and strengthen the manuscript. We appreciate your recognition of the contributions and potential impact of our approach and will continue to improve the paper by incorporating your suggestions. Thank you again for your time and effort in reviewing our submission.

Here, we again highlight the key contributions of our work:

1. In recent foundation model research, CNNs have largely ignored and not compared with existing methods. We are **the first to conduct a comprehensive comparison across diverse benchmarks, demonstrating that with proper design, CNNs can outperform other architectures**. This provides a clear answer to the question: **CNNs are still not surpassed by other architectures**.

2. Previous CNN models, such as Basenji, fail to perform well on all DNA sequence tasks. In contrast, we introduce ConvNova, **the first DNA foundation model based on classical CNN architecture to achieve strong performance across most tasks**. Through extensive experiments, we highlight three critical design principles: a well-designed dilation mechanism, a dual-branch structure, and gating convolution.

3. We are also **the first to explore the relationship between receptive fields and the local dependency characteristics of DNA sequences**. Our experiments reveal that **CNNs inherently possess inductive biases for neighborhood modeling, which may be essential for many DNA sequence applications**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper makes a strong case for reconsidering convolutional neural networks (CNNs) as competitive alternatives to Transformers and state-space models (SSMs) in DNA foundation modeling. The authors propose ConvNova, a CNN-based architecture incorporating innovative elements such as dilated convolutions, gated convolutions, and a dual-branch framework. Through extensive empirical evaluation, the authors demonstrate that ConvNova outperforms state-of-the-art models across various benchmarks, including both short- and long-range DNA tasks. ConvNova's performance improvements are coupled with lower parameter counts and faster computation, addressing practical concerns of scalability and efficiency.

The paper's contributions are particularly noteworthy in a field increasingly dominated by attention-based models. By leveraging the inductive biases inherent in CNNs, such as neighborhood modeling, the authors effectively challenge the assumption that Transformers and SSMs are universally superior. Moreover, the paper underscores the importance of architectural simplicity and principled design, showing that well-tuned CNNs can excel in specialized domains like genomics.