Title: Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Geometric Deep Learning

Equivariance

Group Theory

mRNA Sequence Modeling

Language Models for Biology
Abstract: The growing importance of mRNA therapeutics and synthetic biology highlights the need for models that capture the latent structure of synonymous codon (different triplets encoding the same amino acid) usage, which subtly modulates translation efficiency and gene expression. While recent efforts incorporate codon-level inductive biases through auxiliary objectives, they often fall short of explicitly modeling the structured relationships that arise from the genetic code’s inherent symmetries. We introduce Equi‑mRNA, the first codon‑level equivariant mRNA language model that explicitly encodes synonymous codon symmetries as cyclic subgroups of 2D Special Orthogonal matrix ($\mathrm{SO}(2)$). By combining group‑theoretic priors with an auxiliary equivariance loss and symmetry‑aware pooling, Equi‑mRNA learns biologically grounded representations that outperform vanilla baselines across multiple axes. On downstream property‑prediction tasks including expression, stability, and riboswitch switching Equi‑mRNA delivers up to $\approx$ 10\% improvements in accuracy. In sequence generation, it produces mRNA constructs that are up to $\approx$ 4$\times$ more realistic under Fréchet BioDistance metrics and $\approx$ 28\% better preserve functional properties compared to vanilla baseline. Interpretability analyses further reveal that learned codon‑rotation distributions recapitulate known GC‑content biases and tRNA abundance patterns, offering novel insights into codon usage. Equi‑mRNA establishes a new biologically principled paradigm for mRNA modeling, with significant implications for the design of next‑generation therapeutics.


Peer reviews:
comments: SUMMARY:
In this paper, the authors propose a novel method to encode synonymous codons in mRNA language models. Their method, Equi-mRNA, represents synonymous codons as vector rotations in a base embedding space.

STRENGTHS AND WEAKNESSES:
The presented method is innovative and sound. It’s well motivated by the fact that current codon language models learn embeddings of synonymous codons independently, even though synonymous codons are subject to the same biological constraints at the protein level. 

The authors perform an extensive hyperparameter search and benchmark, evaluating different flavors of their proposed SO(2) embedding and demonstrating that their encoding outperforms competing models on a number of benchmark tasks. 

A potential weakness is that the authors do not seem to split the training corpus by sequence homology - this is common practice nowadays, especially for highly constrained sequences such as protein-coding genes. This may dampen the otherwise promising results on sequence generation.

QUESTIONS:
- It seems that even the Vanilla model is outperforming competitor models such as CodonBERT (Table 1 and 2). Do you have an explanation for that?
- Why did you restrict the training set to 512 or less codons? This seems a bit limiting, considering many proteins fall above this threshold. Modern hardware and architectures should easily handle contexts above 512 tokens. 
- Could you also report standard deviations for tasks other than MLOS?

LIMITATIONS:
The authors address the limitations of their work
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Equi-mRNA, a novel mRNA language model with symmetric codon-level embeddings. The model leverages the SO(2) group to encode synonymous codon relationships in the latent space. This new embedding method helps to improve the biological fidelity and performance of mRNA sequence modeling. The authors demonstrate that the Equi-mRNA model outperforms baseline models in multiple downstream tasks including property prediction tasks and sequence generation. They also show that the trained model angles variability correlates with GC and tRNA availability, which is a good sanity check given the model’s training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n- This paper is well-written with structured flow and solid theoretical inductions. \n- The use of group-theoretic principles to model codon symmetries is a novel and theoretically grounded approach. This method provides a structured way to incorporate biological knowledge into language models.\n- The paper introduced various design of embedding methods, which incorporate fixed or learnable rotation angles to adapt to different biological circumstances. \n- The model shows significant improvements over baseline models in various tasks, including expression prediction and sequence generation. It reported significant gains in accuracy and generative fidelity.\n- The model's ability to reflect known biological patterns, such as GC-content biases and tRNA is a good sanity check. This interpretability is crucial for applications in synthetic biology and mRNA therapeutics.\n- The paper includes a thorough evaluation across multiple datasets and tasks, providing a robust assessment of the model's capabilities.\n\n**Weaknesses:**\n- The introduction of group-theoretic embeddings and the associated computational increment may limit the model's scalability. \n- In the ablation study section, equivariance loss harms the performance of the model in multiple settings across different tasks. The results are suspicious since it contradicts the principle of the paper to design a symmetric embedding.\n- Evaluations and very limited as well the tests against other models. Many of the tests, as the authors admit, are problematic (e.g. no/fixed UTRs).\n- Moreover, some tests are not adaquetly described and their limitations not discussed. For example, the authors use iCodon and claim it's for over 40K human mRNA but the work seems to be based on 1-3K (depending on how you count) sequences in zebrafish used to select their setting for a genetic algorithm for their measure of optimility. We are guessing this was then used to predict for human mRNA? seems a problematic test and not correctly described. Another example is their mRFP. This one is based on ~1.5 codon variants *of the same gene with a fixed UTR*. The authors of this work get great performance (close to 0.8 correlation) just from fitting a random forest on the first 8AA. Yet, this is a test for a 25M parameter model.... It would have been good if the authors at least took the time to look into and acknowledge the how limited these evaluations actually are beyond a general comment at the end.\n- Related to the above it's unclear how train/test are done to prevent information leakage between similar samples. \n- The model is unable to incorporate UTRs, which is a major limitation for usability for the tasks they envision.\n\nIn summary, while we have significant concerns about the evaluations, the comparisons against other models and the usability of this model we believe there is potential (tbd) and the mathematical formulation itself should be of interest to NeurIPS audience.\n\nQUESTIONS:\nThe main question that came into our mind when reading this paper is simply \"why go through all this trouble?\" Yes, you can do this, and it's probably very satisfying to make such a sophisticated mathematical construction that appears to work, but it seems a lot of the sophistication comes to solve issues that can easily be addressed by much simpler embeddings such as softmax function over equivalent codons and adding priors on those (e.g. preference for GC or based on codon availability) which can also be learned. \n\nAnother question is how do they think this will combine with UTR elements?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Equi-mRNA, an mRNA language model that explicitly encodes synonymous codon relationships as cyclic SO(2) rotations in the embedding space. It implements three rotation schemes—fixed, learned, and fuzzy—together with an equivariance loss and SO(2)-aware pooling to enforce these biological symmetries. Across six downstream tasks, Equi-mRNA outperforms vanilla baselines. The interpretability analyses show that the learned rotation angles correlate strongly with GC content and tRNA abundance\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- Applying group theory, specifically SO(2) equivariance, to model the synonymous codon relationships is a novel and theoretically principled approach. \n- The proposed Equi-mRNA model shows consistent performance improvements across most downstream tasks.\n- The interpretability analysis is strong: linking learned geometric representations (codon angles) to known biological features (GC content, tRNA abundance) suggests the model captures biologically meaningful patterns. \n\nWeaknesses\n\nMajor\n- The model treats all synonymous codons for a given amino acid as one uniform cyclic group, which might be an oversimplification. For example, Serine’s six codons split into two distinct families, UCN (UCU, UCC, UCA, UCG) and AGY (AGU, AGC), that are not interconverted by single-nucleotide mutations. Forcing them into a single rotation cycle may impose a geometric constraint at odds with real genetic and evolutionary relationships.\n\nMinor\n- Line 315: an significant leap -> a significant leap\n- Line 367: Stiefel manifold rotations enhances -> Stiefel manifold rotations enhance\n\nQUESTIONS:\nWhat biological justification underlies modeling synonymous codons as planar rotations? Have alternative group structures been considered?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces Equi-mRNA, a novel language model for messenger RNA (mRNA) sequences that operates at the codon level.\n\nThe core contribution is a new embedding framework that explicitly models the symmetries inherent in the genetic code, where multiple codons (synonymous codons) encode the same amino acid. The authors represent these synonymous relationships using group theory, specifically by mapping codons within a synonym set to rotations in the 2D special orthogonal group, SO(2). This creates a structured, equivariant representation where synonymous codon substitutions correspond to predictable geometric transformations. The paper proposes several variations of this idea, including fixed, learned, and \"fuzzy\" (distributional) rotation angles, as well as extending rotations to learned 2D subspaces within a higher-dimensional embedding space using the Stiefel manifold. To enforce these symmetries throughout the model, an auxiliary equivariance loss and symmetry-aware pooling mechanisms are introduced.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Novel and Principled Approach\n\nThe core idea of using group theory, specifically SO(2) rotations, to model the synonymous codon relationships is highly original and biologically principled. It moves beyond simply treating codons as unrelated tokens or using simple clustering losses, embedding a fundamental biological symmetry directly into the model's architecture. This is a significant conceptual contribution to genomic language modeling.\n\n### Comprehensive Experimental Validation\n\nThe paper's empirical evaluation is thorough and convincing. The authors perform a detailed ablation study across 12 different model variants to dissect the contribution of each component (e.g., learned angles, Stiefel basis, equivariance loss). They test on a diverse set of six downstream tasks, covering expression, stability, and regulatory function. \n\n## Weakness\n\n### Lack of Comparison with Existing mRNA foundation models\n\nSeveral foundation models pre-trained on mRNA sequences have been established in the field of RNA research. This includes [CaLM](https://huggingface.co/multimolecule/calm), [mRNA-FM](https://huggingface.co/multimolecule/mrnafm), and [AIDO.RNA cds variants](https://huggingface.co/multimolecule/aido.rna-1.6b-cds). \nComparing with these methods would significantly improve the validation of the proposed method. \n\nSince a more detailed comparison with other methods is omission, I will not be able to grant a higher score.\n\nQUESTIONS:\nAlthough tasks such as vaccine degradation is about mRNA sequences, currently most methods are using ncRNA models on them. Given that the proposed method converts the nucleotides to SO(2), is it possible to apply the proposed method on the tasks requiring single-nucleotide resolution?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-04"}