Large-Scale Multi-omic Biosequence Transformers for Modeling
  Peptide-Nucleotide Interactions
Authors: name: Sully F. Chen, name: Robert J. Steele, name: Beakal Lemeneh, name: Shivanand P. Lad, name: Eric Oermann

The transformer architecture has revolutionized bioinformatics and driven
progress in the understanding and prediction of the properties of biomolecules.
Almost all research on large-scale biosequence transformers has focused on one
domain at a time (single-omic), usually nucleotides or peptides. These models
have seen incredible success in downstream tasks in each domain and have
achieved particularly noteworthy breakthroughs in sequences of peptides and
structural modeling. However, these single-omic models are naturally incapable
of modeling multi-omic tasks, one of the most biologically critical being
nucleotide-peptide interactions.
  We present our work training the first multi-omic nucleotide-peptide
foundation models. We show that these multi-omic models (MOMs) can learn joint
representations between various single-omic distributions that are emergently
consistent with the Central Dogma of molecular biology, despite only being
trained on unlabeled biosequences. We further demonstrate that MOMs can be
fine-tuned to achieve state-of-the-art results on peptide-nucleotide
interaction tasks, namely predicting the change in Gibbs free energy
({\Delta}G) of the binding interaction between a given oligonucleotide and
peptide, as well as the effect on this binding interaction due to mutations in
the oligonucleotide sequence ({\Delta}{\Delta}G).
  Remarkably, we show that multi-omic biosequence transformers emergently learn
useful structural information without any prior structural training, allowing
us to predict which peptide residues are most involved in the
peptide-nucleotide binding interaction. Lastly, we provide evidence that
multi-omic biosequence models are non-inferior to foundation models trained on
single-omics distributions, suggesting a more generalized or foundational
approach to building these models.