The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling
Authors: Andre Cornman
Jacob West-Roberts
Antonio Pedro Camargo
Simon Roux
Martin Beracochea
Milot Mirdita
Sergey Ovchinnikov
Yunha Hwang
Venue: ICLR 2025 Poster (2025)
Keywords: metagenomics
pretraining dataset
genomic language model
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We introduce the OMG dataset and gLM2. OMG is a massive Open MetaGenomic corpus totaling 3.1T base-pairs, and we train gLM2, the first mixed-modality genomic language model which learns zero-shot protein-protein interfaces.

Abstract: Biological language model performance depends heavily on pretraining data quality, diversity, and size. While metagenomic datasets feature enormous biological diversity, their utilization as pretraining data has been limited due to challenges in data accessibility, quality filtering and deduplication. Here, we present the Open MetaGenomic (OMG) corpus, a genomic pretraining dataset totalling 3.1T base pairs and 3.3B protein coding sequences, obtained by combining two largest metagenomic dataset repositories (JGI's IMG and EMBL's MGnify). We first document the composition of the dataset and describe the quality filtering steps taken to remove poor quality data. We make the OMG corpus available as a mixed-modality genomic sequence dataset that represents multi-gene encoding genomic sequences with translated amino acids for protein coding sequences, and nucleic acids for intergenic sequences. We train the first mixed-modality genomic language model (gLM2) that leverages genomic context information to learn robust functional representations, as well as coevolutionary signals in protein-protein interfaces and genomic regulatory syntax. Furthermore, we show that deduplication in embedding space can be used to balance the corpus, demonstrating improved performance on downstream tasks. The OMG dataset is publicly hosted on the Hugging Face Hub at https://huggingface.co/datasets/tattabio/OMG and gLM2 is available at https://huggingface.co/tattabio/gLM2_650M.

Published as a conference paper at ICLR 2025
THE OMG DATASET: AN OPEN META GENOMIC CORPUS FOR MIXED-MODALITY GENOMIC LANGUAGE
MODELING
Andre Cornman" "1
Jacob West-Roberts'
Antonio Pedro Camargo² Sergey Ovchinnikov
Simon Roux²
Martin Beracochea Yunha Hwang"
Milot Mirdita
Tatta Bio, USA
DOE Joint Genome Institute, Lawrence Berkeley National Laboratory, Berkeley, CA,
USA
3 European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, Cambridge, UK 4 School of Biological Sciences, Scoul National University, Scoul, Republic of Korea 6 Department of Biology, Massachusetts Institute of Technology, Cambridge, MA, USA Cooresponddnce:: (yunha, andre andre)]tatta.l @tatta.bio
ABSTRACT
Biological language model performance depends heavily on pretraining data quality, diversity, and size. While metagenomic datasets feature enor- mous biological diversity, their utilization as pretraining data has been limited due to challenges in data accessibility, quality filtering and dedupli- cation. Here, we present the Open MetaGenomic (OMG) corpus, genomic pretraining dataset totalling 3.1T base pairs and 3.3B protein coding se- quences, obtained by combining two largest metagenomic dataset reposito- ries (JGI's IMG and EMBL's s MGnify). We first document the composition of the dataset and describe the quality filtering steps taken to remove poor quality data. We make the OMG corpus available as model-mmdallity genomic sequence dataset that represents multi-gene encoding genomic sequences with translated amino acids for protein coding sequences, and nucleic acids for intergenic sequences. We train the first mixed-modality genomic language model (gLM2) that leverages genomic context informa- tion learn robust functional representations, as well as cocvolutionary signals in protein-protein interfaces and genomic regulatory syntax. Fur- thermore, we show that deduplication in embedding space can be used balance the corpus, demonstrating improved performance on downstream tasks. The OMG dataset publicly hosted on the Hugging Face Hub at attpps:/huuginggaaace wttp///aaaaatttttaaaaaa///////aa and gLM2 is avail- able at htttps://huginggteee http:////a....eeottaaaaao//////a
INTRODUCTION
Biological language models present an effective avenue for leveraging large amounts of un- structured sequence data and learn functionally meaningful representations. Similar to natural language processing (NLP) models Toouroon et al. 2023 Dodge et al. 2021 h, the quality and diversity of pretraining data dictate the behavior and performance of biolog- ical language models (Ding & Steinhardt 2024). To date, the most widely used datasets for biological language models (Hayes et al.] 2024 Lin et al. 2023 Madani et al. 2023
Nguyen et al. 2024) are derived from curated data repositories such as UniProt UniPrro
Consortium 2019 UniRef (Suzek al. 2007) and GTDB (Parks et al. 2022). However,
biological sequence diversity is immense and the above-mentioned data repositories cover
only small fraction of the full sequence diversity found in nature. In order for biological

---

Published as a conference paper at ICLR 2025
language models to improve, the size and diversity of pretraining data must also scale with the size the model.
Metagenomic sequences partial genomic sequences derived from direct sequencing of environmental (e.g. soil, ocean) or biological samples (e.g. human skin, gut). Because metagenomic sequencing circumvents the need for cultivation and isolation of biological organisms, metagenomes typically feature sequences derived from uncultivated and novel microorganisms (Tyson al. 2004). These microbial genomes encode high levels of molec- ular diversity and span previously unexplored branches of the tree of life (Hug et al. (2016). Metagenomic datasets are unstructured by nature and a large fraction of the data is not included in curated databases due to poor functional interpretability of these sequences. To date, metagenomic sequences have not been fully utilized in biological language models due to following limitations:
1. Metagenomic sequenees are not readily downloadable in single archive. To date, the download of raw contigs (assembled genomic segments) from the two main public repositories, Joint Genome Institute (JGI)'s IMG (Markowitz et al. 2012) and European Molecular Biological Laboratory (EMBL)'s MGnily (Richard- son et al., 2023). requires large number of database queries and/o /or rate-limited web API calls, as well as ad hoc approaches to robustly aggregate the results of these queries into single dataset.
2. Metagenomic sequences require extensive pre-processing. Raw metage- nomically assembled contigs first undergo gene calling in order to identify protein coding sequences and extract translated sequences. Additional quality filtering critical, as many metagenomes include poor mis-assemblee contigs.
3. Metagenomic sequences are difficult to deduplicate and balance. Like most biological sequence datasets, metagenomes feature sampling biases (e.g. over- representation of human gut microbiomes). Additionally, due to the lack of central- ized databases for metagenomes, submissions of identical metagenomes to different repositories result in duplicates. Unlike protein databases that can be deduplicated and balanced using computationally efficient clustering algorithms (e.g. MMseqs2 (Steinegger & Söding, 2017)), clustering of large dataset comprising genomic quences of arbitrary region and length is computationally costly. Furthermore, while curated genomic databases (e.g. GTDB (Parks et al. (202)) BV-BRC (O1-) son et al.) 2023 can be balanced with taxonomic labels, metagenomic sequences rarely have taxonomic assignment, and ad-hoc assignment (c.g. Kraken (Wood & Salzberg 2014)) ) is computationally expensive and not always reliable.
Here, document the collection and preprocessing steps of the OpenMetaGennoee (OMG) corpus. We then train the first mixed-modality genomic language model (gLM2) trained on OMG, which leverages genomic context information to learn contextualized functional representations of genomic elements. By training on mixed-modality data, gLM2 can per- form both protein and DNA downstrram tasks, and outperforms ESM2 (Lin al., 2023) on most protein tasks. Additionally, training on multi-protein contexts enables gLM2 to predict protein-proteis interfaces through co-evolutionary signal. Finally, we show that embedding based deduplication of the OMG dataset leads to improved functional represen- tations, especially for underrepresented sequences.
RELATED WORKS
Pretraining corpora preprocessing in NLP. A number of previous studies have de- veloped methods to improve the diversity and quality of pretraining corpora in NLP. For instance, raw snapshots Common Crawl (collection of webtext crawls) contain undesirable data (e.g. hate speech, placeholder text). Studies have demonstrated that careful deduplica- tion and rule-based filtering of Common Crawl (Dodge et al., 2021 improves overall model performance (Penedo et al. 2024). More recently, efforts have been made prune and bal- ance pre-training data in semantic embedding space to achieve increased training efficiency (Sorscher et al.) 2022 Tirumala et al. 2023
Abbas et. al. 2023 Dataset. preprocessing
2

---

Published as a conference paper at ICLR 2025
A
B
Gerome
Pacudemonaatea
Actioayyettoa
lota
Becteroidats
Bod 1ota,4
Hosbactoricta
Ofher
Figure 1: (A) UMAP visualization of the OG dataset examples, colored by taxonomic phylum, using embeddings from the 150M parameter gLM2 model. Distinct clusters form for different phyla in embedding space. (B) Semantic deduplication of the OG dataset, with pruned points highlighted in blue. Deduplication primarily removes samples from dense clusters corresponding to overrepresented phyla. We visualize the semantic deduplication on OG dataset to highlight taxonomic phyla most heavily pruned, and apply the same pruning process to the OMG dataset for model training. (C) Comparison of the OG and OMG datasets using a random 0.1% subset of cach. Notably, the metagenomic data (OMG) exhibits higher diversity.
presents important opportunity to minimize training resources, given the power-law na- ture of LLM scaling (i.e. exponentially increasing compute requirement for diminishing returns in performance improvement) (Hestness et al. 2017: Kaplan et al. 2020
Biological sequence language models and their training datasets. Biological se- quence language models are self-supervieed models trained on discrete protein sequences or genomic segments. Protein language models (pLMs) (Lin et al. 2023 Madani et al. 2023 Elnaggar et al.] 2022) are typically trained on high quality and curated publicly available datasets such as UniRef (Suzek et al.] 2007). UniRef is convenient for pLM training be- cause has been deduplicated using sequence similarity-
clustering (i.e. UniRef50 is deduplicated using 50% sequence identity). Previous efforts to increase the diversity of the pretraining data includes cluster-balanced sampling (e.g. UniRel50/D for FSM models [Rives et al. 2021) and sequence identity-based clustering of compiled protein databases beyond curated databases (e.g. BFD (Steinegger et al. 2019 Elnaggar et al.) 2022) )). Ge- nomic language models (gLMs) are trained on genomic sequences chunked at predefined length thresholds. Diversification efforts for genomic datasets include pretraining on MG- nify's metagenomic contigs (Hwang al. 2024) and balancing efforts in genomic pretraining datasets include taxonomy-aware sampling (Dalla- Torre et al. 2023 Nguyen et al. 2024) of
curated genomic databases such as RelSeq (Pruitt et al. 2014] IMG/VR (Carmaroo al.
2022 IMG/PR (Camargo el al. 2024) | and GTDB (Parks et al.
2022
Metagenomic datasets. In this study, we define metagenomic datasets as collections of gemomic contigs (contiguous genomic segments) computationally assembled from either short-read or long-read raw sequence libraries. Typically, melagenomic datasets are se- quenced from mixed community samples, which consist of multiple species, ranging from hundreds to thousands of distinct species (Bahram et al., 2021). Complete genomes are rarely obtained from metagenomic assemblies. Therefore, metagenomic assemblies require extensive taxonomic profiling (Parks et al.) 2021) and partial genome reconstruction through contig clustering (i.e. bimning). Because metagenomes are sequenced from diverse environ- ments without the need for cultivation, their sequences feature the highest level of molecular diversity amongst publicly available sequence datasets (Pavlopoulos et al. 2023 ). Metage- nomic datasets also vary in quality depending sequencing depth and sample type, where low quality metagenomes feature computational assembly errors, short contig lengths, and
truncated protein sequences (eende et al. 2012
Lai et al. 2022 Furthermore, while most.
3

---

Published as a conference paper at ICLR 2025
metagenomic datasets predominantly analyzed with focus microbial (archaca, bac- teria, viruses) communities, eukaryotic genomic material can comprise a substantial portion of the raw library (West et al. 2018). Many standard metagenomic (e.g. gene calling) fail on eukaryotic sequences, resulting in poor quality protein sequence predictions. Critically, quality filtering and dataset deduplication of metagenomes require domain-specific knowledge, yet there is little documentation of preprocessing steps needed to make these datasets suitable for biological language model pretraining. While pretraining on metagenomic datascts allows models to leverage rich molecular diversity and genomic context, these models are most suitable for microbial genomes and may result in out-of- dist.riuution elfects eukaryolic sequences.
cest-proeessing steps
THE OPEN META GENOME CORPUS
Here, we document the construction of the OMG corpus. The OMG is a 3.1T base pair (bp) pretraining dataset comprising EMBL's MGnify database"] and JGI's IMG database We utilize the gene predictions conducted by the databases; the gene calling protocols for IMG and MGnify are detailed in Huntemann tively. The combined dataset is pre-processed into mixed-modality dataset upon sequential element-by-elemeet quality-filtet filtering (described Section 3.1) The mixed-modalitt dataset of Open Metagenomes is made available as the OMG dataset (Fig. D containing 3.3 billion protein coding sequences (CDS) (Tab. 1 We also make available a 10x smaller subset of OMG that only consists of prokaryotic and viral genomes from INSD as the Open Genome mixed-modalitty dataset OG (Fig. [ Appendix B Finally, we make available protein-only dataset OMG 50o consisting of protein sequences derived from the OMG dataset, clustered at. 50% sequence identity (Appendix E) OMG prot contains 207M representative sequences from clusters with at least two members. representing >3-fold in- crease in sequence diversity compared to UniRef50 (Suzek et al. 2007). All three datasets are available for download from the Hugging Face Hub, and all dataset processing scripts are available at /https://gi //githb.. com cwm/comm/ttttc Tattaaio/ttoo As more metagenomic data bocomes available, we plan on regular updated releases the corpus in the future.
al. (2016) and Richardson et al. (2023) respec-
Table 1: Statistics for the datasets made available in this study. CDS: Coding sequences, IGS: Intergenic sequences. For reference, UniRef50 consists of 66M proteins.
# CDS
# IGS
Total (bps)
# Contig
Size (TB)
Description
Mixed-modality genomic se- quences with multiple protein coding genes (represented in AAs) interleaved with intergenic sequences (represented in NAs).
OMG
3.3B
2.8B
3.1T
271M
1.25
Fraction IMG data consisting of prokaryotic genomes with taxo- nomic metadata.
OG
0.4B
0.3B
0.4T
6.2M
0.16
Protein coding sequences clus- 0.05 tered at 50% sequence identity, ex- cluding singleton clusters. Clus- tering details Appendix
OMG _prot50
207M
'Snapshot date 3022-11-23 (excluding all embargged//vsttritted metagenomic samples, database statistics in Appendix
2 Snapshot date 2023-08-27 (excluding all mubargoed/resstriteed metagenomic samples and in- cluding IMG genomes dataset derived from NCBI.)
https:// /top:://wuu.e insde. org retrieved from TMGMM, metadata available in Appendix
4

---

Published as a conference paper at ICLR 2025
3.1 DATASET PREPROCESSING
Multi-modal data processing. Mctagenomic contigs often encode multiple genes on either strand of the sequence. A genomic language model can be trained on raw nucleic acid sequences (e.g. Evo (Nguyen et al., 2024), Nucleotide Transformers (Dalla-Toree et al. 2023 or by representing each genomic sequence as an order- and orientation-presevvee of translated coding sequences in amino acids (e.g. (Hwang et al., 2024)). For the former method, the context length needed to encode genomic sequences in nucleic acids can result in unfeasibly large compute requirements. Furthermore, recent study comparing muclcic acid (NA) models against amino acid (AA) models on protein functional representations demonstrated that NA may not be the most efficient input format for learning translated protein functions (West-Roberts et al. 2024). The latter method, while benefiting from the compressed sequence length and more expressive AA sequences for proteins, does not lever- age the information stored in intergenic regions. These intergenic regions contain important, yet, lesser characterized sequence patterns involved in transcription regulation and cellular function such as ncRNA, microRNA, promoters, and transcription factor binding sites. We developed mixed-modality dataset that represents a genomic contig a list of elements where an element is either a coding sequence (CDS) or an intergenic sequence (IGS) (see Fig. 2 CDS elements are represented in translated AA sequences and IGS elements are represented in NA sequences. We also store the strand information (+/-) of CDS elements and the order of all elements in the contig.
list
Edge-element removal. Metagenomic contigs are not complete genomic sequences, therefore, both edges of the sequences are more likely contain gene-calling errors. In our pre-processing, we remove edge CDS elements to address miscalled open reading frames (ORFs) and fragmented protcin sequences at the beginning and end of the metagenomic contigs (Steinegger & Salzberg, 2020). Specifically, if a scaffold starts/cnds with an inter- rupted CDS, we remove that CDS element.
a scaffold starts/ende with a non-coding region, we remove the IGS element and the CDS adjacent to the IGS element. This filtering step removes -1.4B genomic elements likely to be poor quality, partial sequences with high likelihood of assembly errors.
Contig mungt-based filtering and preprocessing. Assembll of shotgun metagenomic libraries results in many short contigs that are often low in quality. To limit the impact of the fragmented nature of metagenome assemblies, we first remove all metagenomic contigs that are shorter than 2kb from the raw databases. Secondly, we enrich the corpus with contigs that contain multiple genes by removing contigs that contain less than seven elements in total or less than three CDS elements. Only contigs that meet the length requirement the dataset. In preprocessing these contigs into Hugging Face datasets (Lhoest et al. 2021). we found that extremely large contigs resulted in process hanging errors and inefficient storage. To address this issue, we chunk large contigs into 1000 elements. Appendix C visualizes the distribution of contig length, as well as CDS and IGS clement longths.
added to
Assembly quality (N/ (M/X-frequency) filtering. Due to the computational nature of the metagenomic assembly, misassembled contigs comprise a nontrivial fraction of the data. The quality of the assembly differs significantly across samples, depending on the biological community composition, sample type, and sequencing depth (Vollmers et al., 2017 Lapidus & Korobeynikov 2021). Notably, the quality of assembly may vary across the contig, where a section the contig may contain assembly gaps due to shallow sequencing depth. One way to determine poorly assembled sequences is by identifying the fraction of Ns (gaps or ambiguous bases) in the raw DNA sequence (or Xs in the translated AA sequence). For OMG, we process each contig sequentially deement-by-eeenee 20% in invalid characters, we discard the element and start a new contig (Appendix. D). Importantly, only contigs that meet the length requirement above added to the dataset. This sequential processing allows high quality regions of the contigs to be preserved, while low quality stretches are discarded.
and if an element comprises
Element length-based filtering. A nontrivial portion of the metagenome can be eukary- otic, however, most metagenomic gene-calling software tools are not optimized for eukaryotic
5

---

Published as a conference paper at ICLR 2025
ORF prediction (Bruna et al. 2024 Additionally, metagenomes contain sequences from organisms that utilize alternative genetic codes (Borges et al. 2022 Cook et al. 2024), which may not all be correctly predicted by common tools. A saliett pattern observed for poor gene prediction is low coding density, (i.e. long stretches of IGS) or presence of very long CDS sequences. To identify these, we process each contig sequentially element-by-element and remove any CDS element >> 15,000 AAs or IGS element >4000 bps in length, and start new contig. These thresholds are designed to exclude regions of questionable gene calls, such as long intergenic regions where no genes are predicted, and giant protein sequences, which are prone assembly errors and require careful curation to verify
el al. 2023 This filtering slep removes 2.5e-5% of CDS, and 1e-4% of IGS elements Irom OMG.
Mataganomic contig
Gene callng
Mised-modal sequeenee
Masking
pass
gLM2
Reconstruction
Figure 2: Mixed-modality sequence processing and gLM2 masked language mod- cling. A gene-called metagenomic contig first preprocessod into a mixed-modality se- quence consisting of CDS elements (blue) and IGS elements (grey). The mixed-modalle sequence then undergoes masking at 30% and gLM2 is trained with a masked token recon- struction objective.
EXPERIMENTS
4.1 GLM2: A LIXED-MODALITY GENOMIC LANGUAGE MODEL
To showcase the efficacy of the OMG dataset for pretraining, we introduce gLM2: mixed-modality genomic language model pretrained OMG. gLM2 learns contextualized representations genomic contigs, which are represented as sequences of CDS and IGS elements. In order tokenize the mixed-modality sequence, CDS elements are tokenized using per-amino acid tokens, and IGS elements are tokenized using per-mucleotide tokens. To distinguish strand orientation for CDS elements, introduce and <->, which are prepended to each genomic element to indicate the positive and negative strands, respectively. gLM2 is trained using the masked language modeling objective, where 30% of both CDS and IGS vokens are masked. Cross-entropy loss is applied only on the masked tokens. gLM2 trained at two scales: 150M and 650M parameters. Both models are trained on the semantically deduplicated OMG dataset (Section 4.2 for 600k steps. We train gLM2 using a context window of 4096 tokens to allow for multiple (9.7 + 3.3) CDS and IGS elements to appear in each example. For model architecture and training hyperparameters refer to Appendix
special tokens: <+>
We benchmark gLM2 on the Diverse Genomic Embedding Benchmark (DGEB) (West- Roberts et al. 2024). DGEB is comprehensive benchmark that evaluates model represen-

---

Published as a conference paper at ICLR 2025
tations across diverse taxa and 18 tasks representing multiple axes of biological function, such as evolutionary distance similarity, remote homology prediction, enzyme classification, and retrieval sensitivity.
4.2 OMG CORPUS BALANCING WITH GENOMIC SEMANTIC DEDUPLICATION
Biological datasets exhibit significant biases that can inffuence the performance and gener- alizability of trained models (Ding & Steinhardt 2024 West-Roberts et al., 2024 Unlike protein databases, where short sequence lengths allow for clustering-base deduplication, (meta)genomic sequences have highly variable lengths (Appendix(C) making sequence-based clustering challenging. To address this challenge, we perform deduplication in embedding space by pruning examples with small cosine distance, following Semantic Deduplication (SemD (Abbas et al. 2023). SemDeDup previously showed efficacy in removing se- mantically similar examples over web-scale text and image datascts, demonstrating signifi- cant speed up in convergence for downstream tasks.
For genomic semantic deduplication, we first trained a 150M gLM2 on the tokenized OMG dataset for 600k steps. We then embed the entire OMG dataset, by extracting mean- pooled, per-example representation from the model's last hidden layer. The example-level embeddings correspond closely to the taxonomic classification available for the OG dataset (Fig. [A]. This motivates smbeddingg-assee duplicates while balancing taxonomic bias. We prune the OMG dataset 49% (i.c. 49% of the original data is removed) at the deduplication threshold 2e-3 (where examples with embeddings <2e-3 in cosine distance are deduplicated) (Appendix G The pruned exam- ples are saturated in highly dense clusters (Fig. [3) which results in taxonomic balancing (Appendix H. measured by increased entropies of distribution across taxonomic levels (Ap- pendix[] D.. We then trained 150M gLM2 on the pruned OMG dataset for an equal number of steps, and compared its performance against the un-pruned version on DGEB. While prun- ing results in modest increase in the aggregate DGEB score (0.48 vs 0.47), we observe improvements tasks that feature underrepresented taxa (c.g. ArchRctrioval, RpoB Arch phylogeny) (Appendix J]. This improved performance for underrepresented taxa appears to come at the cost of small regressions on lasks that are biased towards overrepresented taxa. Genomic SemDeDup presents a tunable method for effectively pruming unstructured genomic data without reliance on taxonomic labels.
deduplication as a method for removing near
0.52
0.50
0.43
0.44
DGEB
0.42
0.43
1026
10
Training FLOPs
ESM2
gLM2
gLM22 data pruning!
Embedding Dimension
Figure 3: Scaling performance on DGEB amino acid tasks for gLM2 and ESM2, relative to pretraining floating point operations (FLOPs). gLM2_ 150M trained with no data pruning is shown in black.

---

Published as a conference paper at ICLR 2025
4.3 GLM2 PERFORMANCE ON DGEB
We compare the performance of the 150M and 650M gLM2 models trained on the pruned OMG dalaset against the ESM2 series trained on the UniRel30/D dataset (Fig. 3 gLM2 outperforms ESM2 on the overall DGEB score al each parameter scale. In particular, gLM2's performance scales with pretraining floating point operations (FI.OPs) protein tasks where ESM2 plateaus in performance with scaling (i.e. Operon pair classification tasks, ModBC paralogy task) (Appendix K). Such improved functional representation learning is likely duc to gLM2's ability to leverage genomic context information, and thereby learn relationships between genomic elements. gLM2, being a mixad-modality model, also learns intergenic sequence representations. We compare gLM2's performance on DGEB nucleic acid (NA) tasks against the Nucleotide Transformer series (Appendix L). gLM2 performs similarly on NA tasks when compared to Nucleotide Transformers, despite only small fraction of the training tokens consisting of DNA sequences.
2ONK
Intan protein co co-ovalutoonry signal
1oo-1-8k-base
ESM2 650M
gLM2
Figure 4: gLM2 learns protein-protein interface co-evolutionary signal in the 2ONK (ModBC) complex. (A) ModB (Chain C) and ModC (Chain A) forms a structural complex with co-evolutionary signal between residues (in yellow). (B) Co-evolutionary signal extracted from multiple sequence alignment of 20NK Ovchinnikov et al. 2014). calculated and visualized using GREMLIN The region of inter-protein evolutionary signals are highlighted with a red box. (C) Zoomed-in region of inter-protein coevolutionary signal in B. (D) Categorical Jacobian calculated for Evo on the DNA quence encoding 2ONK_ A and 2ONK_ C (from 89,891 to 91,376 of genomic sequence NC 000917.1). The L2 norm was computed over the (3,4,3,4) tensor for every pair of codon positions to generate the contact map. (E) Categorical Jacobian calculated for ESM2 650M on the concalenated 2ONK A 2ONK C sequence. No inter-prolein co-evolutionary sig- nal is detected. (F) Categorical Jacobian calculated for gLM2 650M on the concatenaled 2ONK A__2ON__55 sequence. (G) Zoomed-in in region of inter-protein coevolutionary sig- nal in F.
(PDB_ benchmark alignments/ 20NK A2ONK A2ONK_C.fas) fass...
4.4 GLM2 LEARNS PROTEIN-PROTEIN INTERACTION INTERFACES
We test gLM2's ability to learn cocvolutionary signals between proteins in protein-protein interaction interfaces (Ovchinnikov et al.) 2014). Previous studies have shown that pLMs learn within-protein co-evolutionarr information that can be extracted with supervised contact prodiction head Lin et al. 2023) using an unsupervised "categorical Jacobian' cal- culation (Zhang et al. 2024). However, pLMs trained individual proteins protein families cannot learn co-evolutionary information across proteins. We calculate the categor- ical jacobian values from gLM2 650M the concatenated sequence of 2ONK A (ModC)
8

---

Published as a conference paper at ICLR 2025
and 2ONK C (ModB) (Appendix N We demonstrate that gLM2 leverages multi-protein context to learn protein-protein interfaces from a single concatenated sequence that closely matches the co-evolutionary signal that be learned from multiple sequence aligmment (MSA) based Potts model (GREMLIN (Kamisetty et al. 2013)) (Fig. [ Such protein- protein interface signals cannot. be extracted in existing language model methods such as ESM2 650M and Evo-1-8k-bass (Fig, A and F).. We validate the gLM2- ------eddceeed contacts directly with the ground truth contacts from 2ONK PDB structure (Fig. 5 as well 31 complexes previously described in (Ovchinnikov et al., 2014) (Appendix ??). The ability to extract interacting residues without supervision nor MSA presents an opportunity to predict novel protein-prolein interactions from sequence information alone.
20NK PDS Contacts 8.0
ZONE gLM2 Jacobian Contacts
20NK Contact Comparison
109
100
550
202
Gooston
100
100
180
2003
300
400
160
300
300
800
250
90
14D
150
350
8
GUGG POB Contacts x 8.0
eUGG g2M2 lacobiin Contacts
68GG Contact Comparison
10
20
20
30
40
posten
30
30
15
50
00
70
70
Figure 5: Ground truth comparisons Jacobian-detected contacts against PDB structures. (A) Left: Ground truth contacts derived from PDB structure (PDB: 2ONK; ModBC complex) shown Fig. 4 where contact is defined as residues that are within <8A. Middle: LLM2-preddcteer contacts using Categorical Jacobian. Right: Inter-protein region highlighting top highest scoring predicted contacts (red for true positive, blue for false positive) overlaying ground truth contacts (gray), where n is the number of inter- protein contacts identified in the ground truth. (B) Left: Ground truth contacts derived
from tRN A-Ap (PDB: 6UGG) shown in Fig. 6 Middle: gLM2-predicted contacts using Categorical Jacobian. Right: Top highest scoring contacts in gLM2 (red for true positive, blue for false positive) overlaying ground truth contacts (gray), where is the number of contacts within tRNA identified in the PDB ground truth excluding the diagonal.
4.5 GLM2 LEARNS REGULATORY SYNTAX INTERGENIC DNA
We demonstrate gLM2's ability to identify regulatory syntax and non protein-coding clc- ments in IGS regions. We first validate gLM2's ability predict contacts in tRNA-Asp against the ground truth 6UGG PDB structure (Fig. 5) We further demonstrate gLM2's ability to identify regulatory regions (sigma factor binding and terminator) in the genomic context of tRNA-Asp (Fig. G.. We additionally observe signal downstream of asp V and uptream of the terminator region. This region lacks annotation EcoCyc (Karp al. 2023) and presents the patential for g..M2-based unsupervised discovery of novel regulatory se- quence motifs, We examined 23 additional intergenic regions the E. colí K-12 genome that
https://colab research. GREMLIN _FF.K__FFFiipbb

---

Published as a conference paper at ICLR 2025
contain at least one terminator and one promoter regions according to EcoCye annotations.
We show conserved Categorical Jacobian patterns corresponding to previously validated annotations across diverse regions of the genome (Appendix Pl. We further conducted
(5'->3") containing a I,10 leader RNA gene and two ribosomal protein coding genes rplJ and rpll. (Appendix O We observe putative contacts between the L10 leader RNA and ribosomal protein RpIL, an experimentally evidenced interaction (Johnsen et al. 1982.. We also observe contacts between RpIJ and RplL, a known ribosomal protein complex. Furthermore, our analysis highlights co-cvolutionary signal between the Shine- Daagaro sequences (ribosomal binding sile) upstream of rplJ and rpll, suggesting gLM2 understanding of genome-specilie regula- tory motifs.
similar analysis on B. subtilis 168 genomic region -19.848-120,9788
lading
bidding
352
Figure 6: gLM2 learns intergenic regulatory syntax and tRNA structure. We vi- sualize co-cvolutionary signal in E. coli K-12 substr. MG1655 chromosomal region 236,866- 237, 237,087bp (5'->3") containing asp V (tRNA-Asp encoding gene) using the Categorical Ja- cobian. Structural signatures in ARNA-Asp sequence are visible. Other signals correspond to known regulatory syntax including sigma factor binding sites (-35 and -10), transcription initiation site (070 binding region), and rho-independent terminator sequence.
CONCLUSION
The OMG dataset is largo-scale mixed-modality biological pretraining corpus that lever- ages the immense volume and diversity of unstructured metagenomic (primarily prokaryotic and viral) sequences. We quality-filter and preprocess the raw melagenomic sequences into a mixed-modality format ready for language model training. We showcase the efficacy of modalityy input for genomic language modeling with gLM2. With genomic SemD- eDup, we present an efficient method for reducing the bias and duplication in genomic datasets. The gLM2 models trained on pruned OMG learn contextualized representations for both CDS and TGS, and demonstrate efficient scaling and improved performance across downstream tasks compared to uncontextualizer protein language models trained on curated databases. We further demonstrate the gLM2's ability to learn protein-protein interfaces at resid paving the path towards unsupervised protein-prote protein complex prediction. Finally, we show that gLM2 learns evolutionary couplings of regulatory motifs in the in- tergenic DNA, indicating model understanding of both modalities of the data. The OMG dataset. and gLM2 models well the supporting code publicly available for download.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This study aims to advance open science for genomics, by making the OMG corpus and gLM2 model publicly available on the HuggingFace Hub. The OMG corpus is constructed from publicly available data within JGI's IMG and EMBL's MGnify repositories. We exclude all embargoed and restricted data from the OMG corpus. As the data originates from environmental samples, no personally identifiable information is associated with the dataset.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos, SemDeDup: Data-efficient learning at web-scale through semantic deduplication. March 2023.
Mohammad Bahram, Tarquin Netherway, Clémence Frioux, Pamela Ferretti, Luis Pedro Coelho, Stefan Geisen, Peer Bork, and Falk Hildebrand. Metagenomic assessment of the global diversity and distribution of bacteria and fungi. Environ. Microbiol., (113:336,,, January 2021.
Adair L Borges, Yue Clare Lou, Rohan Sachdeva, Basem Al-Shayeb, Petar I Penev, Alexan- der L Jalfe, Shufei Lei, Joanne M Santini, and Jillian F Banfield. Widespread stop-codon recoding in bacteriophages may regulate translation of lytic genes. Nat Microbiol, 7(6): 918-927, June 2022.
Tomas Bruna, Alexandre Lomsadzee, and Mark Borodovsky. new gene finding tool GeneMark-ETT ETP significantly improves the accuracy of automatic annotation of large eu- karyotic genomes. bioRriv, April 2024.
Antonio Pedro Camargo, Stephen Nayfach, T-Min A Chen, Krishnaveni Palaniappan, Anna Ratner, Ken Chu, Stephan J Ritter, T B K Reddy, Supratim Mukherjee, Frederik Schulz, Lee Call, Russell Y Neches, Tanja Woyke, Natalia N Ivanova, Emiley Fadrosh,, Nikos C Kyrpides, and Simon Roux. IMG/W /IG/RR v4: an expanded database of uncultivated virus genomes within a framework of extensive functional, taxonomic, and ecological metadata. Nucleic Acids Research, (1(D1)):733-D443 11 2022. ISSN 0305-1048. doi: 10.1093/nai //r///a///// haa/aar/gggaea hgg///aal000 1037. URL btttps://doi... org/ 1093/ Sate///ar/ggae nar/kkk11037
Antonio Pedro Camargo, Lee Call, Simon Roux, Stephen Nayfach, Marcel Huntemann, Krishnaveni Palaniappan, Anna Ratner, Ken Chu, Supratim Mukherjeep, T B Reddy, I-Min A Chen, Natalia N Tvanova, Emiley A Floe-Fadrosh, Tanja Woyke, David A Baltrus, Salvador Castañeda- Barba, Fernando de la Cruz, Barbara E. Funnell, James P Hall, Aindrila Mukhopadhyay, Eduardo P C Rocha, Thibault Stalder, Eva Top, and Nikos C Kyrpides. IMG/PR- a database of plasmids from genomes and metagenomes with rich annotations and metadata. Nucleic Acids Res., 52(DI):DI64 D173, January 2024.
Ryan Cook, Andrea Telatin, George Bouras, Antonio Pedro Camargo, Martin Larralde, Robert A Edwards, and Evelien Adriaenssens. Driving through stop signs: predict- ing stop codon reassignment improves functional annotation of bacteriophages. ISME Commun, 4(1) Jamuary 2024.
Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, Guillaume Richard, Marcin Skwark, Karim Beguir, Marie Lopez, and Thomas Pierrot. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. September 2023.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL pottps://arxiv> http://////wxooo///////////aaa//
Frances Ding and Jacob Steinhardt. Protein language models are biased by unequal sequence sampling across the tree of life. March 2024.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. April 2021.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Toward understanding the language of life through Superiuee learning. IEEE Trons. Pattern Anal. Mach. Intell,, 44(10):7112 7127, October 2022.
12

---

Published as a conference paper at ICLR 2025
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkun- dri, Irhum Shafkat, Jun Gong, Alexander Derry, Raúl Santiago Molina, Neil Thomas, Yousuf A Khan, Chetan Mishra, Carolyn Kim, Liam J Bartie, Matthew Nemeth, Patrick D Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with a language model. July 2024.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, December 2017.
Laura A Hug, Brett J Baker, Karthik Anantharaman, Christopher T Brown, Alexander J Probst, Cindy J Castelle, Cristina N Butterfield, Alex W Hernsdorf, Yuki Amano, Kotaro Ise, Yohey Suzuki, Natasha Dudek, David A Relman, Kari M Finstad, Ronald Amundson, Brian C Thomas, and Jillian F Banfield. A new view of the tree of life. Nat Microbiol, 1: 16048, April 2016.
Marcel Huntemann, Natalia Ivanova, Konstantinos Mavromatis, H James Tripp, David Paez-Espino, Kristin Tennessen, Krishnaveni Palaniappan, Ernest Szeto. Manoj Pillay, I-Min A Chen, Amrita Pati, Torben Nielsen, Victor M Markowitz, and Nikos C Kyrpi- des. The standard operating procodure of the DOE-JGI metagenome annotation pipeline (MAP v.4). Stand. Genomic Sci., 11:17, February 2016.
Yunha Hwang, Andre I. Cormman, Elizabeth H Kellogg, Sergey Ovchinnikov, and Peter R Girguis. Genomic language model predicts protein co-regulation and function. Nat. Commun., 15(1):2880, April 2024.
M Johnsen, T Christensen, P Dennis, and N P Fiil. Autogenous control: ribosomal protein L10-L12 complex binds to the leader sequence of its mRNA. EMBO J., 188)9999-1004, 1982.
Hetunandan Kamisetty, Sergey Ovchinnikov, and David Baker. Assessing the utility of coevolution- residue residue contact predictions in sequence- and structure-rich era. Proceedings of the National Academy of Sciences, 110(39):15674 15679, 2013.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. January 2020.
Peter D. Karp, Suzanne Paley, Ron Caspi, Anamika Kothari, Markus Krummenacker, Pe- ter E. Midford, Lisa R. Moore, Pallavi Subhraveti, Socorro CGamaCCastin Victor H. Tier- rafria, Paloma. Lara, Luis Muñiz Rascado, César Bonavides- Martinez, Alberto Santos- Zavaleta, Amanda Mackic, Gwanggyu Sun, Travis A. Ahn-Horst, Hoejo Choi, Markus W. Covert, Julio Collado-Vide and Ian Paulsen. The ecocyc database (2023). EcoSal Plus, (00)) 2023. doi: 10.1128 /coosaalpluss.csp-000022222 //journals asm wwtg//////a///iiaaee///aa 10w.1111///coslplussee pw000000000000000200000033333333
URL https:
Senying Lai, Shaojun Pan, Chuqing Sun, Luis Pedro Coclho, Wci-Hua Chen, and Xing- Ming Zhao. metaMIC: reference-free misassembly identification and correction of de novo metagenomic assemblies. Genome Biol., (1):242 November 2022.
Alla Lapidus and Anton I Korobeynikov. Metagenomic data assembly - the way of de- coding unknown microorganisms.. Front. Microbiol., 12:613791, March 2021.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillann-Maorr, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Gochringer, Victor Mustar, François Lagunas, Alexan- der Rush, and Thomas Wolf. Datasets: A community library for natural language processing. Tn Proceedings of the 2021 Conference on Empirical Methods in Natural
13

---

Published as a conference paper at ICLR 2025
Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL
ittpp://aclaathologgy org/2021 momlp-ddmo- 221
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan Dos Santos Costa, Maryam Fazel-Zaraadii, Tom Sercu, Salvatore Candido, and Alexander Rives. Fvolutionary-scale prediction of atomic-level protein structure with language model. Science, (63(6) 1123 1130, March 2023.
Πly Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL
bttp::://axxi..orr/bb//////
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Jr, Caiming Xiong, Zachary Z Sun, Richard Socher, James Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Not. Biotechnol. 41(8):1099 1106, August 2023.
Victor M Markowitz, I-Min Chen, Krishna Palaniappan, Ken Chu, Ernest Szeto, Yuri Grechkin, Anna Ratner, Biju Jacob, Jinghua Huang, Peter Williams, Marcel Huntemann, lain Anderson, Konstantinos Mavromatis, Natalia N Ivanova, and Nikos Kyrpides. IMG: the integrated microbial genomes database and comparative analysis system. Nucleic Acids Res., 40(Database issue):D115 22, Jamuary 2012.
Daniel R Mende, Alison S Waller, Shinichi Sunagawa, Aino I Järvelin, Michelle M Chan, Manimozhiyan Arunugam, Jeroen Raes, and Peer Bork. Assessment of melagenomic as- sembly using simulated next generation sequencing data. PLoS One, 7(2):311886, Febru- ary 2012.
Eric Nguyen, Michael Poli, Malthew G Durrant, Armin W Thomas, Brian Kang, Jeremy Sullivan, Madelena Y Ng, Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A Baccus, Tina Mamnandez-Boussaae Christopher Ré, Patrick D Hsu, and Brian L Hie. Sequence modeling and design from molecular to genome scale with evo. March 2024.
Pascal Notin, Aaron W. Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Hansen Spinner, Nathan Rollins, Ada Shaw, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Rose Orenbuch, Yarin Gal, and Debora S. Marks. Proteingym: Large-scale benchmarks for protein design and fitness prediction. bioRriv, 2023. doi: 10.1101 2022. 2/0.//22222.2222 21..1.2.0700000 570727.. http:/ttt://///tt biorxiv. org/ wwwtentteearly//ee/e/e 08/2023. 00.007777777777
Robert D Olson, Rida Assaf, Thomas Brettin, Neal Conrad, Clark Cucinell, James J Davis, Donald M Dempsey, Allan Dickerman, Emily M Dietrich, Ronald W Kenyon, Mehmet Kuscuoglu, Elliot J Lefkowitz, Jian Lu, Dustin Machi, Catherine Macken, Chunhong Mao, Anna Niewiadomska, Marcus Nguyen, Gary Olsen, Jamie C Overbeek, Bruce Parrello, Victoria Parrello, Jacob S Porter, Gordon D) Pusch, Maulik Shukla, Indresh Singh, Lucy Stewart, Gene Tan, Chris Thomas, Margo VanOeffelen, Veronika Vonstein, Zachary Wallace, Andrew S Warren, Alice R Wattam, Fangfang Xia, Hyunsoung Yoo, Yun Zhang, Christian M Zmasek, Richard Scheuermann, and Rick L Stevens. Introducing the bac- terial and viral bioinformatics resource center (BVBBRC): resource combining PATRIC, IRD and ViPR. Nucleic Acids Res., (1())):D678-DD D689, Jamuary 2023.
Sergey Ovchinnikov, Hetunandan Kamisetty, and David Baker. Robust and accurate pre- diction of residue-residue interactions across protein interfaces using evolutionary infor- mation. Elife, 3:e02030, May 2014.
Donovan H Parks, Fabio Rigato, Patricia Vola-Wolf, Lutz Krause, Philip Hugenholtz, Gene W Tyson, and David L A Wood. Evaluation of the microba community profiler for taxonomic profiling of metagenomic datasets from the human gut microbiome. Front. Microbiol, 12:643682, April 2021.
14

---

Published as a conference paper at ICLR 2025
Donovan H. Parks, Maria Chuvochina, Christian Rinke, Aaron J Mussig, Picrre-All Alain Chaumeil, and Philip Hugenholtz, GTDB: an ongoing census bacterial and archaeal diversity through phylogenetically consistent, rank normalized and complete genome- based taxonomy. Nucleic Acids Res., 50(DD)::D77- -D794, January 2022.
Georgios Pavlopoulos, Fotis Baltoumas, Sirui Liu, Oguz Selvitopi, Antonio Pedro Camargo, Stephen Nayfach, Ariful Azad, Simon Roux, Lee Call, Natalia N Ivanova, I Min Chen, David Paez-Espino, Evangelos Karatzas, Toannis Diopoulls, Konstantinos Konstantinidis, James M Tiedje, Jennifer Pett-Ridge, David Baker, Axel Visel, Christos A Ouzounis, Sergey Ovchinnikov, Aydin Bulug, and Nikos C Kyrpides. Unraveling the functional dark matter through global metagenomics. Nature, 622(7983):594 602, October 2023.
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. June 2024.
Kim D Pruitt, Garth R Brown, Susan M Hiatt, Frangoise Thibaud-Niseen, Alexander As- lashyn, Olga Ermolaeva, Catherine M Farrell, Jemnifer Hart. Melissa J Landrum, Kelly M McGarvey, Michael R Murphy, Nuala A O'Leary, Shashikant Pujar, Bhanu Rajput, San- jida H Rangwala, Lillian D Riddick, Andrei Shkeda, Hanzhen Sun, Pamela Tamez, Ray- mond E Tully, Craig Wallin, David Webb, Janet Weber, Wendy Wu, Michael DiCuccio, Paul Kitts, Donna R Maglott, Terence D Murphy, and James M Ostell. RefSeq: an up- date mammalian reference sequences. Nucleic Acids Res., 42 4D(aabbas issue):D756-600 January 2014.
Lorna Richardson, Ben Allen, Germana Baldi, Martin Beracochea, Maxwell L, Bileschi, Tony Burdett, Josephine Burgin, Juan Caballero-Pérez, Guy Cochrane, Lucy J Colwell, Tom Curtis, Alejandra Escobar-Zepeda, Tatiana A Gurbich, Varsha Kale, Anton Ko- robeynikov, Shriya Raj. Alexander B Rogers, Ekaterina Sakharova, Santiago Sanchez, Darren J Wilkinson, and Robert D Fimn. MGnify: the microbiome sequence data analy- sis resource in 2023. Nucleic Acids Res., 51(D1))DD53-DPT DD59, January 2023.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protcin sequences. Proc. Noll. Acad. Sei. U. S. A., 118(15), April 2021.
Noam Shazeer. Glu variants improve transformer, 2020. URL https: org/aagg// abs/ 2002.05202
/rg/abs/
Ben Sorscher, Robert Geirhos, Shashank Shekhar, S Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Adv. Neural Inf. Process. Syst., abs/2206. 1 14866, June 2022.
Martin Steinegger and Steven Salzberg- Terminating contamination: large-scale search ifint more than 2,000,000 contaminated entries in GenBank. Genome Biol., 21 21))): 115, May 2020.
Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence search- ing for the analysis of massive data sets. Nat. Biotechmol, 35(1)::0026 1028, November 2017.
Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nat. Commun., 9(1):2542, June 2018.
Martin Steinegger, Milot Mirdita, and Johannes Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Not. Methods, 16(7): 603-606, July 2019.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Euhanced transformer with rotary position embedding, 2023. URL https: //trvvv.rgg/////////// abs/2104.09864
15

---

Published as a conference paper at ICLR 2025
Baris E Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. UniRef: comprehensive and non-redundant UniProt reference clusters. Bioinformatics, (3(10):1282-128) May 2007.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improving LLM pretraining via document De-Duplication and diversification. Adv. Nearal Inf. Process. August 2023.
Syst., 00s//200.112884
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinct, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. February 2023.
Gene W Tyson, Jarrod Chapman, Philip Hugenholtz, Eric E Allen, Rachna J Ram, Paul M Richardson, Victor V Solovyev, Edward M Rubin, Daniel S Rokhsar, and Jillian F Ban- field. Community structure and metabolism through reconstruction of microbial genomes from the enviromment. Nature, 428(6978):377-43 March 2004.
UniProt Consortium. UniProt: worldwide hub of protein knowledge. Nucleic Acids Res., 47(D1):D506 D515, January 2019.
John Vollmers, Sandra Wiegand, and Anne-Kristin Kaster. Comparing and evaluating metagenome assembly tools from microbiologist's perspective not only size matters! PLoS One, 12(1):c0169662, January 2017.
Patrick T West, Alexander J Probst, Igor V Grigoriev, Brian C Thomas, and Jillian F Ban- field.
for eukaryotes from complex natural microbial communities. Genome Res., 28(4):569-580, April 2018.
Jacob
Tuis Valentin-Alvarado, Susan Mullen, Rohan Sachdeva, Justín Smith, Laura A Hug, Daniel S Gregoire, Wentso Liu, Tzu- Yu Lin, Gabriel Husain, Yuki Amano, Lynn Ly, and Jillian F' Banfield. Giant genes rare but implicated in cell wall degra- dation by predatory bactoria. November 2023.
Jacob West-Roberts, Joshua Kravitz, Nishant Jha, Andre Cormman, and Yunha. Hwang- Diverse genomic embedding benchmark for functional evaluation across the tree of life. July 2024.
Derrick F. Wood and Steven I, Salzberg. Kraken: ultrafast metagenomic sequence classifi- cation using exact alignments. Genome Riol., 15(3):R46, March 2014.
Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https:
7/arxiv. org/ /tts/910....... 07467
Zhidian Zhang, Hannah K Wayment-Steele, Garyk Brixi, Haobo Wang, Matteo Dal Peraro, Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary statistics of interacting sequence motifs. January 2024.
16

---

Published as a conference paper at ICLR 2025
APPENDIX A DATA SOURCES
Type
Snapshot date
Samples
# contigs*
Total bps
# CDS
IMG
Metagenomes Genomes
2023-08-27 2023-08-27
36,273 131,744
182M
1.70T 0.4T
1.84B
6.2M
0.4B
MGnily
Metagenomes
2022-11-23
33,531
82M
1.03T
1.03B
Number of contigs after filtering and preprocessing.
APPENDIX B DATASET PREPROCESSING
Sequences (purple) undergo filtering steps (green), yielding three Hugging Face datasets (yellow) made available with this paper. 'NA' and 'AA' refer to nucleic acid and amino acid data modalities respectively.
Gusbenta
CD0 auga (oe)
Avartbly
DMS
SendiChap
IMO peromes anittammeetiii
APPENDIX C DATASET LENGTH DISTRIBUTIONS
Length distributions of the OMG corpus. (A) Distribution of contig lengths in the number of genomic elements (CDS and IGS). (B) Distribution of contig lengths in base pairs. (C) Distribution of CDS lengths in amino acids. (D) Distribution of IGS lengths in base pairs.
17

---

Published as a conference paper at ICLR 2025
A
B
3.5
3.5
3.0
Trequency
2.5
Frequency
2.0
2.0
3.5
0.5
0.0
0.0
50k
Genomic elements per scaffold
Base pairs per
143
3.0
144
1.4
2.5
Frequency
2.0
Frequency
2.0
0.8
1.5
0.6
0.2
200
400
600
800
1000
1200
1400
50
100
150
200
250
300
350
400
CDS length in amino acids
IGS length base pairs
APPENDIX D INVALID CHARACTER DISTRIBUTIONS
Distribution of the percent of characters per genomic clement considored as invalid ("X' for amino acids and 'N" for nucleotides) prior to applying the assembly quality filter from
Section 3.1 The assembly quality filter removes elements containing more than 20% invalid characters, result ing in 0.004% of CDS and 0.2% of IGS being filtered from OMG. We show the distribution for the subset of genomic elements containing at least 1 invalid character.
Distribution of amino acids CDS with at least 'X'
Distribution of nucleotides IGS with at least 'N'
0.30
0.25
0.25
0.20
0.20
Density
Density
0.15
0.15
0.10
0.10
0.05
0.05
0.00
10
20
30
40
50
10
40
50
60
Percent XX amino acids
Percent of 'N" nucleotides
APPENDIX E OMG PROT50 CLUSTERING METHOD
total of 4.2B protein sequences were first clustered to remove fragments using MMseqs2 linclust (Steinegger & Söding 2018) (commit f6c98, parameters: min-seq-id 0.9 -c 0.9 cov- mode 1). Subsequently, the resulting sequences were clustered at 50% sequence id and 90% sequence coverage using MMseqs2 linclust -min-seq-id 0.5 -c 0.9. Singleton clusters (only one sequence in the cluster across the full dataset) were removed and remaining 207M cluster representatives were uploaded the Hugging Face dataset.
18

---

Published as a conference paper at ICLR 2025
APPENDIX F GLM2 MODEL PARAMETERS
gLM2 is a transformer encoder optimized using AdamW Loshchilov Hutter 2019) and trained in mixed precision bfloat16. We set the AdamW betas to (0.9, 0.95) and weight decay of 0.1. We disable dropout throughout training. The lcarning rate is warmed up for 1k steps. followed by cosine decay to 10% of the maximum learning rate. gLM2
RoPE (Su et al. 2023) position encoding, SwiGLU (Shazeer 2020) feed-forward layers, and RMS normalization (Zhang Sennrich, 2019). We leverage Flash Attention 2 Dao 2023
to speed up attention computation over the sequence length of 4096.
Num heads
Num
Context
Learning Batch Pretraining rate
Dim
layers
length
size
tokens
gLM2-150M
640
10
30
4096
1e-3
128
315B
gLM2-650M
1280
20
33
4096
1e-3
128
315B
19

---

Published as a conference paper at ICLR 2025
APPENDIX G SEMANTIC DEDUPLICATION DISTANCE THRESHOLD
The percentage of remaining training examples as a function of the embedding distance threshold. Examples within the distance threshold in embedding space deduplicated.
80
70
50
40
30
20
64-04
13-03
20-03
30-03
46-07
58-03
1o-02
Deduplication Threshold
APPENDIX H TAXONOMIC DISTRIBUTION OF THE OG DATASET BEFORE AND AFTER PRUNING
Data pruning through semantic deduplication reduces dataset bias toward overrepresented phyla and orders.
A
B
P2
30
000 5011 2000
20

---

Published as a conference paper at ICLR 2025
APPENDIX I TAXONOMIC ENTROPY OF THE OG DATASET BEFORE AND AFTER PRUNING
Semantic deduplication of the OG dataset consistently increases the taxonomic entropy across all taxonomic ranks, indicating a more even distribution.
Domain
Phylum
Class
Drder
Family
Genus
7.00
3,0
En:ropy
0.08
Entropy
3.29
Entropy
Entropy
Entropy
1.00
2.0
0.8
14
2-
0.2
825
65
Original
Pruned
Original
Pruned
Original
Pruned
Original
Pruned
Original
Proned
Original
Pruned
21

---

Published as a conference paper at ICLR 2025
DGEB Score
0.474
0.482
We train two 150M parameter gLM2 models on the original and pruned OMG dataset, each for 600k steps. Both models are evaluated on the DGEB benchmark. Pruning improves performance, espocially for tasks with underrropreennted sequences.
RpoB Bacterial Phylo- geny
0.251
0.311
Vibrio Operonic Pair
0.547
0.566
MopB Clust- cring
0.786
0.820
Converg. ent Enzymes Class- ification
0.196
0.172
Euk Retri- eval
0.348
0.330
E.coli
Oper- onic Pair
0.670 0.628
0.639
MIBiG
Class- ifica. tion
0.660
EC Classi- fication
0.517
0.514
RpoB Archaeal Phylo- geny
0.312
0.353
ABLATION OF SEMANTIC DEDUPLICATION
FeFe Hydro- genase Phylo- geny
0.688
0.716
ModBC Para- logy BiGene
0.244
0.229
Cyano Oper. Retri oval
0.289 0.408
0.418
Arch Retri- eval
0.294
Bac
Arch BiGene
0.750
0.722
(with data proming)
(no data pruning)
APPENDIX J
gLM2 150M
gLM2 150M
22

---

Published as a conference paper at ICLR 2025
APPENDIX K PER TASK DGEB SCALING WITH FLOPS FOR ESM2 AND GLM2 MODELS IN AMINO ACID TASKS
Primary metric from the best scoring layer (between mid, and last) is reported for each task. To account for model-specific patterns in learning task-relevant functional information across different layers in the network (West-Roberts et al. 2024). DGEB calculates model performance for both mid and last layer and reports the best score between the two.
Tyct Seciered tylopon
1pc8 Haylogor
Nhtrr saget ase flydoaan
Last Ouseri Par
1.5
0.30
0.00
1.25
0.8
5:24
0.52
0.0
022
0.0
322
270
2220
200
2
3P9
1275
WP
Thanking
Thiring
0.30
1.4
0.32
5.37
0.21
0.20
5.5"
0.28
0.30
Thainin R.:0
R.0k
7100c
Trining
2C Dissttattion
Creepgen Eriperes Choornatie
NECI Clen Fostom
Higd Chovonnn
0.90
1.45
222-
0.71
0.30
g
0.75
1.5-
335
0.55
354
300
390
32*
30t1
309
204
10P
120
104
1902000 ILPP
T107in FLORS
0.22
Model Type Zest LayaT
0.35
2.73
3.2-
LEPIO2 CatA pnAeen
1.0
3:35
2.22
5.5
1.5
Trining
7UOPs
23

---

Published as a conference paper at ICLR 2025
APPENDIX L PER TASK DGEB SCALING WITH FLOPs FOR NUCLEOTIDE TRANSFORMERS AND GLM2 MODELS IN NUCLEIC ACID TASKS.
Primary metric from the best scoring layer (between mid, and last) is reported for each task. To account for model-specifie patterns in learning task- relevant functional information across different layers in the net work [West-Roberts et al. 2024), DGEB calculates model performance for both mid and last. layer and reports the best score between the two.
Praigeny
1:38
C28
0:25-
0.30-
0200
5.84
11.44-
0000
0.33
0.33
0.33
0100
0.31
Thining
APPENDIX M GLM2 PERFORMANCE ON PROTEIN GYM
Model name
Avg Spearman
Activity
Binding
Expression
Organismal Fitness
Stability
ESM2 650M
0.414
0.425
0.337
0.415
0.369
0.523
gLM2 650M prot
0.384
0.406
0.327
0.412
0.311
0.466
Deep Mutational Scanning (DMS) substitutions task. Because the DMS task is strictly a single-protein task (without context), we benchmark gLM2 650M after finetuning for one epoch of OMG prot50, the single-protein dataset introduced in Table [ While gLM2 650M prot performs slightly worse than ESM2_650M, note that the ProteinGym benchmark includes eukaryotic sequences, which are poorly represented in the OMG dataset.
We evaluate gLM2 on the ProteinGym Notin et al. 2023)
APPENDIX N MODB AND MODC SEQUENCE CONCATENATION
This concatenated sequence was derived from the 2ONK A 20NK C alignment used in (Ovchinikovv et. al. 2014).
TAA.GHAPPASSSSNTTNNTTTTTTTTTTTTS ALRITVGREDVSEGGG
24

---

Published as a conference paper at ICLR 2025
APPENDIX O PUTATIVE RNA-PROTEIN-PROTTE INTERACTIONS
We visualize a contiguous stretch (119,845 120,978bp, 5'->3") of the B, sutilis 168 reference genome. Putative residuc-level interactions between L10 leader RNA (IdLJ), proteins RpIJ and RpIL are highlighted in gray boxes. Shine-Dalgaraa sequences upstream of the two protein-coding genes highlighted and co-evolve.
L10 leader RNA
Pulstive RNA--Poleen
100
150
Shine Dalgamo sequeecc co-evolution
200
Rplj
RoLl-RpIL
250
300
350
400
RpIL
450
500
550
50
100
150
200
250
300
350
400
450
500
550
APPENDIX P ADDITIONAL FILES
Additional Files are found in https://zenodd.oor/rrer//////// hrt.oddoorrrrroord///1999988 14199868
Additional File 1. OG sample ID to original NCBI metadata. A JSON file mapping OG
sample ID (taxon_oif oid) to NCBI metadata (accessions, collection dates).
Additional File DOIs for MGnify samples. DOIs for MGnify samples that were included in this study, where available.
Additional File DOIs for IMG samples, DOIs for IMG samples that were included in this study, where available. Additional File 4. Comparison of gLM2 Jacobian Contacts on 20NK with (A) and without (B) the basepair IGS sequence flanking ModB and ModC. We show that the addition of IGS sequence does not change the results. Additional File 5. zip file containing all 32 evolutionary conserved complexes (ttp:/oort-roobodder----
PDB previously identified in (Ovehinnikov et al. 2014 https://openseq. org/ org/cplx. hts//cllx.pplll www.:.phpssorrttooo mode=pdb PDB contacts and gLM2 Jacobian Contacts are compared.
Additional File 6. A zip file containing Categorical Jacobian maps of 26 IGS regions in F.coli K-12 str. MG1655 (Genome ID: U00096) with at least one promoter (highlighted in red) and one terminator (highlighted in green) sites annotated in EcoCyc. File names and figure title correspond to the start and end positions in the U00096 genome.
25


Reviews:
[{"review_id": "YMlpBPUUMH", "reviewer": "Reviewer_jngi", "summary": "The authors present open metagenomic datasets combined from two separate sources and filtered to maintain high-quality data. They further present the data set's utility in a trained model.", "questions": "The importance of the results is not explained; could you present this with other protein-to-protein-related works? \nWhich conclusion can I reach with this data that I couldn't other wise?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors give an excellent introduction to the field of metagenomics. They adequately describe the difficulties of balancing and filtering biological data sets and how some issues are exacerbated for metagenomic specifically. They detail each of the steps required to filter the data. Finally they open the data sets to the public to use.", "weaknesses": "While the methods for data filtering are explained, their impact is not always clear.\nHow many elements are removed for the edge-element removal?  \nFor example, in the assembly filtering, a histogram of the number of invalid characters, why a threshold of 20% was chosen? What is the meaning of \"(>3 CDS >7 ...\"?\nFor the element length-based filtering section, again, what is the impact of why the specific threshold was chosen? Could you provide a supporting figure?\nEssentially, This presents a novel data set whose importance is still unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m3aFPvirRg", "reviewer": "Reviewer_HdWV", "summary": "This paper reports a large, well-curated, database of genomic sequences (protein coding and nucleic acids, genomic and metagenomic) with utility as a genomic pretraining dataset for biologic modeling. The dataset is unique and the authors demonstrate its utility in training a novel mixed-modality genomic language model (gLM2) that leverages genomic context information to learn functional representations and coevolutionary signals in protein-protein interfaces and genomic regulatory syntax.  These are very different kinds of tasks and that the model performs well in the vignettes shown is very promising for both the OMG corpus and the gLM. The gLM is not well described or validated; the corpus itself (OMG) is the focus of the paper. I agree that OMG is a very useful resource for the community, particularly in combination with gLMs but the authors do not sufficiently validate their gLM aside from a few interesting but poorly described vignettes.", "questions": "Is there a reason the gLM wasn't validated in some way? The examples seem very cherry picked, can you give us some sense of how you'd go about doing the validation for these very different biological questions (regulation, protein-protein interaction, etc). Do you think the gLM is really generalizable for all these different tasks? Where it struggles, why do you think it struggles?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "OMG is a novel resource that will be useful for pretraining models to address many different biological questions\nthe gLM is a novel way of representing genomic architecture and function simultaneously and could, as such be a very powerful resource", "weaknesses": "the gLM is poorly described and not validated, there are a few small vignettes describing it but these do not get at validation or generalizibility for the very interesting tasks described.  I have a hard time giving this paper the strongest recommendation because of these concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hCKejKDoH", "reviewer": "Reviewer_Sobx", "summary": "The Open Megagenomic Corpus (OMG) and gLM2 model are important and novel contributions to biologic sequence modeling. Authors make use of the largest repositories of metagenomic data to construct a new database for training that is larger than exiting, popular biologic LM training datasets. Using both coding sequences and intergenic sequences, authors show that their modeling strategy captures long range interactions of coding and non-coding genomic elements at multiple scales of biologic sequence interaction.\n\nThe paper should be accepted given the technical and strategic innovation database creation and the novelty in modeling strategy that performs on par with an existing state-of-the-art model and captures important biology across sequence modalities.", "questions": "Authors claim their modeling approach achieves \"more efficient scaling\" performance with flops on a set of benchmark tasks than a prior top performing model; however, they only show two training flops. When looking at Appendix J there are some tasks where scaling is observed and some where it does not. It would be a stronger claim if there were additional data points for gLM2. On a per-task basis, it appears that gLM2 has similar scaling to ESM2 even if at the aggregated score level the scaling looks slightly better (Figure 3).\n\nMinor comments:\n1. \"start a new contig\" should be \"start a new element\" (l.255, 266-267)\n2. DGEB is not defined (l.333)\n3. Figure 5 has two \"n\" that are mentioned but not present in the figure", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The work addresses the need for larger databases of sequence data that can be used to train language models for biologic applications by making use of rapidly growing repositories of metagenomic sequence data. Authors utilize knowledge and methods from recent insights in NLP to construct the large database used for training and develop their own genomic deduplication procedure that captures biologic reality. They also utilize deep domain knowledge about metagenomic sequencing to filter terabytes of raw data for database construction. The new curated database is made available to the community. The authors utilize a novel modeling approach, constituting of which includes multi-modal input in the form of amino acid and nucleotide data.", "weaknesses": "The language model trained does not explicitly include eukaryotic genomic sequences and it is unclear gLM2 is meant to be used to model eukaryotic sequences. Authors should be more explicit about the application for eukaryotic sequence modeling. Authors do mention contamination and likely inclusion of eukaryotic sequences in metagenomic samples but this in in the context of challenges and mitigation thereof with metagenomic sequence data.\n\nApplications of the gLM2 model are not explored. It would be interesting to see performance of gLM2 in areas where more and diverse training data is of importance, such as remote homology, or where the multi-modal training strategy is helpful, such as contig-level prediction of taxonomy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6DQnNx4OU", "reviewer": "Reviewer_GgLu", "summary": "The authors present the Open MetaGenomic (OMG) dataset, a substantial, mixed-modality genomic corpus designed to support biological language model training. OMG is sourced from two major repositories, JGI’s IMG and EMBL’s MGnify, encompassing 3.1 trillion base pairs and 3.3 billion protein-coding sequences. The authors address critical challenges in metagenomic data usage, including accessibility, quality filtering, and deduplication. They also introduce gLM2, the first mixed-modality genomic language model pretrained on OMG with 150M and 650M parameters, capable of leveraging both genomic and protein-coding information to capture complex functional and regulatory patterns. gLM2 demonstrates decent improvements in functional representation learning of both protein and NA and co-evolutionary information modeling, outperforming existing models on tasks involving genomic structure prediction, protein-protein interactions, and regulatory syntax learning.", "questions": "1. Regarding the training and modeling capabilities of gLM2:\n   - Is gLM2 trained on contig sequences containing both nucleotide sequences (IGS) and protein sequences? If so, does this limit the model to only modeling interactions between bio-sequences from the same contig? Can gLM2 predict contacts for Protein hetero-multimer interactions for example (also RNA-Protein interactions and DNA-Protein interactions)?\n\n\n2. For the 2ONK example:\n     * Are chains A and C from the same contig?\n     * If yes, is there IGS between them?\n     * What would be the effect on coevolution signal extraction if IGS were included?\n     * Why were only chains A and C analyzed when 2ONK has 4 chains?\n\n3. How ESM2 FLOPs were estimated? Or if this information comes from existing literature (e.g., the original ESM2 paper).\n\n4. Regarding model hyperparameter design:\n   - What is the rationale behind the chosen head number, layer number, and embedding dimensions?\n   - Were preliminary experiments conducted to support these choices?\n   - Contemporary LLMs like LLaMA typically use equal numbers of heads and layers, with embedding dimensions set to 128 times the number of heads", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **High-Quality, Diverse Metagenomic Dataset**: By aggregating two of the largest metagenomic repositories, the authors create a richly diverse dataset that addresses critical gaps in sequence variety and covers underrepresented taxa through rigorous filtering and preprocessing. This dataset is likely to become an essential training resource for future models, representing a major contribution to the field.\n\n2. **Mixed-Modality Model**: The authors propose an innovative modeling approach that integrates both protein and nucleic acid sequences, enabling the model to process protein-coding and regulatory sequences together. This combined approach opens the door to unique and complex downstream applications.\n\n3. **Co-Evolutionary Signals**: This work may be the first biomolecular language model to demonstrate coevolutionary signals across sequences using the Categorical Jacobian method, marking a novel and compelling contribution to biomolecular modeling.", "weaknesses": "Since I don't have much experience in metagenomic and its relevant data processing, I will leave these aspects to other reviewers. Here, I will focus more on gLM2 model and performance.\n\n1. The model is evaluated on the DGEB benchmark. However, insufficient information is provided about the evaluation methodology and associated metrics. I recommend adding a section discussing the evaluation process and metrics in detail.\n\n2. After checking the original DGEB paper, it appears that gLM2 can not match ProGen2 on amino acid (AA) tasks (Appendix J), which is not reported in the current manuscript. The authors should either include these results or justify their omission.\n\n3. From a protein language model perspective, the evaluation needs to be more comprehensive. The authors should include evaluations on standard benchmarks such as FLIP or ProteinGym. \n\n4. For the coevolution signal extraction experiment with 2ONK, the comparison would benefit from including NT2 and ProGen2 as additional baselines.\n\n5. While sections 4.4 and 4.5 present well-analyzed individual cases, for contact prediction in particular, more examples are needed to demonstrate that gLM2 consistently outperforms alternative methods. Might be interesting to consider datasets like [1][2] for RNA and  CAMEO for protein.\n\n\n[1] https://github.com/jaswindersingh2/SPOT-RNA-1D\n[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC7297115/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)