Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
Authors: Mehdi Yazdani-Jahromi
Ali Khodabandeh Yalabadi
Ozlem Garibay
Venue: NeurIPS 2025 poster ()
Keywords: Geometric Deep Learning
Equivariance
Group Theory
mRNA Sequence Modeling
Language Models for Biology
Primary area: machine_learning_for_sciences

Abstract: The growing importance of mRNA therapeutics and synthetic biology highlights the need for models that capture the latent structure of synonymous codon (different triplets encoding the same amino acid) usage, which subtly modulates translation efficiency and gene expression. While recent efforts incorporate codon-level inductive biases through auxiliary objectives, they often fall short of explicitly modeling the structured relationships that arise from the genetic code’s inherent symmetries. We introduce Equi‑mRNA, the first codon‑level equivariant mRNA language model that explicitly encodes synonymous codon symmetries as cyclic subgroups of 2D Special Orthogonal matrix ($\mathrm{SO}(2)$). By combining group‑theoretic priors with an auxiliary equivariance loss and symmetry‑aware pooling, Equi‑mRNA learns biologically grounded representations that outperform vanilla baselines across multiple axes. On downstream property‑prediction tasks including expression, stability, and riboswitch switching Equi‑mRNA delivers up to $\approx$ 10\% improvements in accuracy. In sequence generation, it produces mRNA constructs that are up to $\approx$ 4$\times$ more realistic under Fréchet BioDistance metrics and $\approx$ 28\% better preserve functional properties compared to vanilla baseline. Interpretability analyses further reveal that learned codon‑rotation distributions recapitulate known GC‑content biases and tRNA abundance patterns, offering novel insights into codon usage. Equi‑mRNA establishes a new biologically principled paradigm for mRNA modeling, with significant implications for the design of next‑generation therapeutics.

Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
Mehdi Yazdani-Jahromi Department of Computer Science University Central Florida
Ali Khodabandeh Yalabadi Department of Industrial Engineering
University of Central Florida
Orlando, FL 32816
Orlando, FL 32816
yazdani edu
yalabadi@ucf edu
Ozlem Ozmen Garibay
Department of Computer Science and Industrial Engineering University of Central Florida Orlando, FL 32816 ozlemOucf. edu
Abstract
The growing importance of mRNA therapeutics and synthetic biology highlights the need for models that capture the latent structure of synonymous codon (different triplets encoding the same amino acid) usage, which subtly modulates translation efficiency and gene expression. While recent efforts incorporate codon-level induc- tive biases through auxiliary objectives, they often fall short of explicitly modeling the structured relationships that arise from the genetic code's 's inherent symmetries. We introduce Fqui-mRNA, the first codon-level equivariant mRNA language model that explicitly encodes synonymous codon symmetries as eyclic subgroups of 2D Special Orthogonal matrix (SO(2)). By combining crou--theoretic priors with an auxiliary equivariance loss and symmetry aaware pooling, mRRA learns bio- logically grounded representations that outperform vanilla baselines across multiple axes. On downstream property-prede prediction tasks including expression, stability, and riboswitch switching Equi-mRNA delivers up to R 10% improvements in accuracy. In sequence generation, it produces mRNA constructs that are up to x= 4x more realistic under Fréchet BioDistance metrics and x= 28% better preserve functional properties compared to vanilla baseline. Interpretability analyses further reveal that learned codon-rotation distributions recapitulate known GC-content biases and tRNA abundance patterns, offering novel insights into codon usage. Equi-mRNA establishes new biologically principled paradigm for mRNA modeling.
Introduction
RNA analysis has become central to modern molecular biology due to RNA's essential regulatory and functional roles within cellular systems 25 13]. Among RNA molecules, messenger RNA (mRNA) is especially critical, serving direct translator of genetic information into functional proteins, thereby underpinning both fundamental biological processes and therapeutic advancements [40]. At the nucleotide level, mRNA sequences structured triplets known codons, each specifying a single amino acid within the resultant protein. Due to redundancy in the genetic code, multiple synonymous codons encode the same amino acid, a surjective many-to-on relationship that introduces substantial complexity. Importantly, synonymous codons are not utilized equally; This phenomenon, termed codon bias, varies widely across species and even between genes within the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

same organism, influenced by both mutational biases and selective pressures aimed at optimizing translational efficiency [35] Thus, synonymous codons, despite coding for identical amino acids, can differentially impact mRNA stability, translational speed, and protein folding dynamics [26] 51 5
Such subtle yet critical differences have profound biological implications, with numerous synonymous mutations implicated in human diseases [31].
Understanding and leveraging codon-level nuances is therefore essential, particularly for applications
such precision-engineers mRNA vaccines, gene therapies, and synthetic biology tools [6 33)
Recent advances have demonstrated the significant potential of language models in analyzing biologi- cal sequences, particularly for proteins [2]
12 24 16] and DNA [28 56]. For messenger RNA, this success has not held, in part because biological sequences follow constraints and grammars that are inherently different from natural language.
Given the rich biological signal embedded in codon usage, there strong motivation to develop specialized language models tailored explicitly for coding regions of the genome. Such models could exploit inductive biases inherent in the genetic code such as the codon hierarchy [2 and arganis-speccin codon preferences to significantly enhance predictive modeling of biologically relevant properties.
To fully realize the potential of language models in mRNA modeling, we developed a theoretically principled framework for representing mRNA codon sequences, explicitly incorporating the biological symmetry and redundancy inherent in the genetic code (Fig. [lb). Using orthogonal group actions with cquivariance constraints, implemented as per-subspace rotations of shared amino acid embeddings. we produce interpretable embeddings that encode synonymous codon relationships. Integrating these diologically-inorree inductive biases significantly improve the data efficiency, robustness, generalizability, and generative performance of ANA-ffcused language models.
Veniita
Fixed Fiond
+2.0%
rleud a st.
13.2%
Flaed st. Erc.
Leaamed
Learned a tnc
+3.1%
Aaamod st.
Learond st. En.
+..2%
Funzy
every a sa.
+8.8%
Forry st. ea.
B
Relative
Mean Accuracy improvement over Vanilla (%)
Sane vecter
Rotelad
(b) Codon Embedding Visualization
(a) Component impact
Figure 1: Equi-mRNA Embedding Visualization: (a) Mean accuracy improvement over vanilla codon embeddings (Plain learned codon emheddings) for all twelve SO(2) variants: fixed, learned, and fuzzy rotation generators, each with and without equivariance. (b) Codon wheel for arginine: inner, middle, and outer rings denote the 1st 2nd and 3rd nucleotide positions, and codons are ordered by relative ID. A shared base vector (BV, blue) is rotated by multiples of 0 (4 x 0) to produce the rotated vector (RV, red), illustrating how cyclic SO(2) subgroup embeddings differentiate synonymous codons while preserving amino acid equivariance.
Our contributions are summarized follows:
We introduce Equi-mRNA, the first mRNA language model to explicitly encode codon-level symmetries via group- theorette priors, enforcing biologically grounded SO(2) equivariance with an auxiliary loss and symmetry-aware pooling mechanisms.
We curate and release a unified coding-region corpus of 25M protein-coding sequences plus stratified 1M sequence subset standardize benchmarking across studies.

---

Equi-mRNA achieves up to = 10% improvements in downstream property predic- tion, and up to 4x more realistic mRNA sequence generation with 28% better preservation of functional properties compared to vanilla baselines.
Through interpretability analyses, we link learned codon-rotee rotation distributions to GC-content biases and tRNA abundances, and demonstrate that Equi-mRN, outperforms oate-offthheart nucleotide and codon models with fewer parameters.
2 Methodology
In this section, we will introduce the group-theoretic embedding framework for codon-level language models. We will first describe the mathematical structure of the genetic code and how it can be represented as a group. Then, we will outline the embedding process and how can be integrated into existing language model architectures.
2.1 Problem Formulation
Let denote the set 64 codons (including start and stop), and let A be the set of 20 amino acids together with stop symbol. The genetic code defines a surjective map C + A. where codons mapping to the same amino acid are considered synonymous. For each a E A. define C. = π (a) as the corresponding synonym set, and let n. Caal- By construction, the sets C. disjoint partition of C. For preliminaries and notations please refer to Appendix A.1
A form
Cyclic action on synonymous codons. Fix an amino acid with its set of synonymous codons C, = {Ca.a,- ..... .a.,, -1) in chosen order. Let G. Zn, be the finite cyclic group with addition Ca.a.
modulo na- Define group action Pa Go x C. + C. by
Pa Ca.i) = Ca, (i++) mod n."
(1)
Here the group Ga. group element le E Ga, and a generator is any 9a G. such that repeated application of saa (ga. -) visits every codon in C.. This action is regular, it is free and transitive C.. When na 1, the action is trivial. This encoding treats each synonym set as a discrete symmetry class that reflects the degeneracy of the genetic code [22].
Definition 2.1. (Symonymous Codon Group). For each amino acid a E A with na |Ca > 1, let be an arbitrary bijection that assigns unique integer label to each codon Ca. We induce a cyclic group structure on C. by identifying with Zn, via Pa. where the group operation is defined as addition modulo Tha
Pa C. - 0, 1, 1.......1, -1
Under Definition 2.1 index the codons by labeling map la Ca Zn, with la(Ca,s i.
Let E G. = Zn. be a generator. We define the canonical substitution step the action of 1 any codon, that is,
Co.i H (1, Ca.s) = Ca, (++1) mod n.
which replaces a codon by the next synonymous codon in the fixed cyclic order. Repeated application of this step enumerates all codons in Ca, and the inverse step uses the action of to move to the previous codon.
This construction yields well-defined group action of G. = Zn. C.,, where substituting codon e E C, with another synonymous codon c' corresponds to applying an element j E Zn.: if (c((c) = k, then (c() + mod na. Thus, synonymous codon substitutions are formalized actions of a finite cyclic group. Our objective is to build codon embeddings that explicitly respect this group structure.
2.2 Mapping Codons to SO(2) via Cyclic Subgroups
While the codon group structure (Definition 2.1 captures the finite cyclic symmetry within each synonym set, it is non differentiable and thus incompatible with gradient based learning. To address this, we define a continuous group homomorphism Pa G. - SO( that maps & Zn,, to planar rotation with 6₂ 2T/ 2 or a learned angle constrained to preserve the homomorphism. This preserves the cyclic structure while embedding codons in a differentiable manifold suitable for neural models. Although the genetic code does not inherently suggest a rotational structure, the use of SO(2) serves as an inductive bias that reflects codon-level redundancy through structured, parameter-sharing representations. Its compact topology and continuous symmetry enable smooth optimization, enforce equivariant constraints, and promote generalization across contexts.
3

---

Construction 2.2. Let := 25 denote the angular increment corresponding to one step in the cyclic group Ga. We define homomorphism & from codons (O( by mapping each codon C. to rotation matrix:
(c) = ((pa(c)) 0a) = (Pos((pa (c))0
-(sn (sin (c)0.)
(c)(()(c)).)))
cos((a (c)0a)
The image of this mapping the set of nn evenly spaced rotations:
{R(0), R(Oa), R(20o)..... RR((n 1))a))
which forms a cyclic subgroup of (2) isomorphic Zm. The generator ga - - (1) is mapped to R(Oa), and we have ((y"") R(na- 0a) R((TT) consistent with the identity element in both groups.
(Appendix A.6). the map De. defines a group homomorphism from the codon substitution group G Zn, to the cyclic subgroup of SO(2) generated by R(Oa). This ensures that synonymous codon substitutions correspond to structured, differentiable rotations in the embedding space.
As formalized in Proposition A.
Example Serine is encoded by RReer 6 codons:
Cser = {UCU, UCC, UCA, UCG, AGU,
which we label via yoser 10,3, 4,5. Under 4 these codons are mapped to:
[((0"), R(120"), R(180)), R(240"), R(300)),
i.e. the vertices of a regular hexagon on the unit circle. Substituting UCU (label 0) by AGC
(label 5) corresponds to applying the group action five times: (AGC) = R((5sser) (UCCU) = R(-Oser) ((UCU) modulo 2T. Thus, Cser embedded in a cyclic subgroup of SO(2) of order 6. Since (2( is continuous Lie group, this embedding gives the discrete Zς symmetry a smooth, differentiable representation, making suitable for integration with gradient based learning.
2.3 Learning Codon Rotation Generators: Toward Adaptive and Robust Representations
The canonical construction maps each synonym set C, to cyclic subgroup of SO(2) by fixing 2m This produces uniform rotations that exactly realize the Zn,, action, but
generator angle 0g nu. also fixes the geometry priori and may fail capture organism specific or condition specific variation.
To retain differentiability while allowing data driven adaptation, use learnable parameterizations that preserve the group structure either exactly or approximately. A strict homomorphic parameteriza- tion sets and learns 0g under the constraint Pa (n,) T. This is enforced either by parameterizing 0_ 25m, with ma integer coprime na, by adding a penalty À [pa(nn) T|I? during train- ing. For higher dimensional embeddings, lift the action learned 2D subspaces by introducing an orthonormal basis Ua E pdxx and applying
Ua (R(kk.)) 12)
which rotates only the codon specific plane while leaving the orthogonal complement unchanged. These parameterizations keep the cyclic symmetry visible to the model, remain fully differentiable, and give the geometry enough freedom to reflect biological variability across datasets.
To overcome this limitation, we propose to treat the generator angle 0,, as a learnable parameter for each amino acid group. The codon embedding map is then defined as
P8. (c) R (c) 0m),
where 02 is optimized jointly with the model. Crucially, the generality of Proposition A. ensuress that the group homomorphism property continues to hold under this parameterization, provided that 02 na =0 mod 2n, which can be softly enforced or reparameterized during training.
Learning 0, introduces only one additional parameter per amino acid and provides a biologically grounded, symmetry-preservinn mechanism for fine-tuning codon embeddings. Unlike traditional fixed-vector approaches, maintains consistent relative geometry among synonymous codons while enabling global adaptation to species-specific context-dependent signals. This structured update mechanism improves flexibility and stability during downstream adaptation, offering a principled and efficient alternative to conventional embeddings.

---

linens--a extensions. To increase representational capacity, we also extend this con- struction nigher-dimensional embeddings via block-diagonal rotations in SO(d), enabling multiple independent subspaces per codon. This generalization preserves group-theoretic structure while allowing richer task-specific codon representations. Full details of this extension and its formal properties are presented in Appendix A.3
2.4 Fuzzy Codon Embeddings: Soft Group Actions from Learned Distributions
Strict cyclic embeddings assume that each codon corresponds to a fixed rotation derived from group generator. While this enforces clean algebraic structure, it may be too rigid in biological settings where codon preferences are not binary but graded, noisy. or context-dependent. To address this, we introduce fuzzy relaxation of the distribution over rotation angles, rather than fixed one.
mapping that allows each codon to induce
In fuzzy formulation, each amino acid group has learnable angle distribution number of discrete rotation prototypes. These prototypes can be uniform (e.g., evenly spaced over [0, 2T)) trainable. For each codon, we compute softmax over the angle logits angle_logts RIA xmxK, yielding smooth distribution over K angular components. The effective codon angle is then defined as a weighted average over angle bins:
fixed
K
2T
0(c)
Pej
J.
Kig
3=0
where Pa.j is the softmax probability assigned to codon e for angle bin j. and ka is the number of synonymous codons for amino acid a. This angle is then scaled by the codon's group label to form the final rotation: (c( O(c).
This fuzzy formulation retains the inductive bias of cyclic symmetry while allowing biologically meaningful deviations, such as soft substitutions asymmetric codon effects. It supports gradient- based optimization over both rotation parameters and angle distributions, making it amenable to fine-tuning and transfer. In the limit, it recovers the hard cyclic embedding, allowing the model to interpolate between strict and relaxed symmetry in a task-adaptive manner.
2.5 General Basis Rotation via Boeiill-Manfoldd--ooe
Subspaces
We extend codon embeddings by representing each synonymour codon as a rotation of a shared base embedding vector within learned 2D subspace of Rd. We work in an ambient space Rd For each amino acid a, let U. E Rdx2 have orthonormal columns that span a learned two dimensional subspace, and let ea Rd be shared base embedding. A synonymous codon with angle 0a,k is represented by
R(0) = cos 0 sin 0
La.k UU eas
sin 0
cos
Thus the embedding lives in Idd the rotation acts only in the learned two dimensional subspace, and the orthogonal complement is preserved.
For each amino acid A, let A E Rd denote a shared base embedding vector. We define rotation subspace for A spanned by an orthonormal pair of vectors (N^ v.) E RRd, where u := / | aligned with the direction the base vector. The second vector V E Rd constrained such that (uA, vA) = 0 and || = 1, forming an orthonormal frame in Rd We model this 2D subspace using the Stiefel manifold St(2, d), which is the space of all orthonormal 2-frames in Idd.
Each codon CA is associated with rotation angle de E [0, 2mr) and its embedding is computed via:
E(c) RiuA.va) (Q.)
(.)) ZA cos de n + sin de |
where R (0e) denotes the rotation in the plane spanned by {u v and acts as identity on the orthogonal complement. This embedding guarantees that all synonymous codons of A lie circle of radius | within a leskele semantic plane, enabling flexible and structured encoding of codon variation.
To ensure the basis (n,, v.) remains orthonormal during training, we parameterize the subspace using a point on the Stiefel manifold and optimize it via Riemannian gradient descent. In practice,
5

---

we implement this using the geoopt library, which constrains each learned matrix VA Rdx2 such that V V1 The rotated codon embeddings can then be expressed as:
E(c) VA R(o)) V ZA.
where R(oe)) E S0(2) is the canonical planar rotation matrix and VA E St(2, This formula- tion naturally supports group-equivariant representations while permitting the geometry of codon substitution be learned from data (see Proposition A.3)
2.6 Equivariance Enforcement via Auxiliary Loss
To ensure that codon-level symmetries are preserved beyond the embedding layer, we incorporate an auxiliary equivariance loss that encourages internal representations transform consistently under synonymous codon substitutions. This regularization promotes structured and predictable behavior aligned with the underlying group action, improving robustness, generalization to unseen codon contexts, and interpretability of the learned representations. It also facilitates stable fine-tuning of rotation generators 0g in transfer settings, ensuring that codon embedding geometry adapts smoothly to species- task-specific preferences. The full formulation, training objective, and mathematical properties this loss are provided in Appendix A.4
2.7 Equivariant Pooling Mechanisms for SO(2) Representations
To maintain architectural consistency with the (equi-qquvar codon embeddings, we implement specialized pooling mechanisms that preserve group structure during sequence-level aggregation. These include polar pooling, Fourier-based pooling, and direct angular averaging, each designed to respect the rotational symmetries inherent in the embedding space. For a full mathematical formulation and biological motivation of these (2))--aaae pooling strategies, please refer to Appendix A.5
3 Experiments
3.1 Experimental Setup
Pretraining Corpus We constructed a large-scale pretraining corpus by drawing 25 million anno- tated protein-coding sequences from 56 million RefSeq entries and retaining only those between 20 and 512 codons length. Untranslated flanking regions were discarded to focus on genuine coding signals. From this filtered set sampled stratified I million-sequence subset (preserving original taxonomic proportions) for controlled ablations, then used the full 25 million-sequence collection to train our final Equi-mRNA variant at scale (Appendix A.9.1
Downstream Tasks We evaluate on six biologically driven benchmarks spanning expression, stability, and regulatory switching (MLOS (23). mRFP (29), F. coli expression (10), Te-riboswitch switching factor [15), iCodon stability (9) SARS-CoV-2 degradation [8 see Appendix A.9.2 for
details).
Two-Stage Ablation Protocol We begin with an extensive ablation on the 1 M-sequence subset randomly sampled from the curated corpus to assess the impact of three embedding parameterizations: Fixed, Learned, and Fuzzy 0. Each is combined with or without the Stiefel rotation basis and with or without the equivariance loss Lequiv. yielding 12 total variants. Table summarizes these models, and Figure (Appendix A.2) provides a schematic overview. All variants share an identical GPT2 Transformer backbone [37]. ensuring that performance differences arise solely from embedding design. Note that the Stiefel and Fuzzy variants introduce additional parameters capture their
geometric transforms.
Full-Scale Training Following ablation, selected the top-performing configuration and retrained it on the full 25 M-sequence corpus to evaluate scalability and generalization under abundant data. Pretraining conducted thirty-two NVIDIA H100 GPUs (ablation used eight NVIDIA H200 GPUs); runtimes and resource utilization are detailed in Appendix A.
Hyperparameters All pretraining arguments and hyperparameters, as well as downstream genera- tion and property-pree prediction hyperparameters, are provided in Appendix A.10
3.2 Symmetry-Awan Property Prediction
We summarize the downstream performance of our twelve symmetry-aware variants across five bio- logical benchmarks. Figure La shows the average accuracy and Spearman correlation improvements

---

Table 1: Component-wise comparison of symmetry-aware variants: Equivariant SO(2) models improve downstream accuracy over vanilla across all tasks. Fuzzy and Learned variants with Stiefel basis and/or equivariant loss yield the best results, highlighting the benefit of structured group-theoretie priors.
Model
Stiefel
Equiv.
E. coli(A)
MLOS(S)
Tc-Ribo.(S)
mRFP(S)
COV Deg(S)
Vanilla
0.580
0.633 + 0.14
0.698
0.797
0.779
Fixed 0
/
0.602
0.667 + 0.19
0.688
0.840
0.803
Learned 0
/
x
0,633
0.657 + 0.10
0.701
0,871
0.790
Fuzzy
0.605
0.691 + 0.14
0.736
0.844
0.820
over the vanilla baseline for each rotation strategy and symmetry constraint. To highlight overall trends, [ the cop-performing configuration from each embedding group (fixed, learned, and fuzzy 0). The fuzzy 0 model with Stiefel basis and equivariant loss achieves the highest gains overall and was therefore selected for large-scale training on the full 25M sequence corpus. From this point forward, we refer to this fuzzy 0 Stiefel with equivariance configuration trained on the 25M sequence corpus as EQUI-MRNA Model. Full results for all variants and evaluation metrics are provided in Appendix A. Note that the MLOS dataset lacks standard splits and exhibits high variability due to its small size; we report standard deviations accordingly to reflect this sensitivity.
An in-depth component analysis in Table 14 (Appendix AAI12) isolates the contribution of each design element. Fixed-angle models with equivariance already outperform the vanilla baseline on E. coli and mRFP (0.600 and 0.841 vs. 0.580 and 0. 0.797), demonstrating that even static geometric priors can enhance biological alignment. Incorporating learned @ with a Stiefel projection further boosts performance, yielding the highest accuracy on E. coli (0.633) and peak Spearman correlation on mRFP (0.871), which suggests that data-driven codon embeddings capture species or tissue- speciffc regulatory signals. Equivariant training adds additional gains on structurally sensitive tasks such as Tc-Riboswitch (0.741) and MLOS long sequences (0.698). Finally, fuzzy models representing codons soft distributions over SO(2) deliver robust improvements noisy or low-resource assays like MLOS (0.691) and SARS-CoV-2 degradation (0.820), further motivating their adoption for full-scale training.
Table compares our EQUI-MRNA models (5M and 15M parameters) to established codon and mucleotide-based baselines trained on the full 25M-sequence corpus. The 15M EQUT-MRNA variant achieves the highest accuracy across all evaluated tasks while using only R 30% of the parameters of HELM. Even the smaller 5M model matches surpasses several larger architectures, high- lighting the efficiency of our symmetry-aware approach. While exact comparisons are limited by differences in pretraining protocols, objectives, and the public availability of models (e.g.. HELM'S pre- pre-training data is not publicly released), these results underscore the effectiveness of embedding codon symmetries directly. Specifically, our 5M parameter variant of EQUI-MRNA employs hybrid Mamba-Transee Transformer backbone, demonstrating that symmetry- SO(2) embeddings can achieve competitive performance with minimal overhead and be scamlessly extended to architectures such as state-space models.
3.3 Symmetry-Aware Sequence Generation
We conducted our experiments following the framework established by 52 For generative eval- uation, we used the iCodon thermostability corpus: up to I 000 test sequences were selected and truncated after two-thirds of their length, with the prefix serving as prompt for autoregressive complction of the remaining one-third via codon sampling (A: = 10, p = 0.95) at temperatures T E {0.2, 0.4, 0.6, 0.8, 1.0]. Generative quality was quantified by Fréchet BioDistance (FBD) be- tween synthetic and true suffixes, computed as in (46]. All FBD scores for equivariant variants appear in Figure 2 with complete results in Appendix A.1..I
To evaluate retention of functional properties, we assessed generated sequences on five downstream benchmarks (iCodon, MLOS, mRFP, Tc-Riboswith, and SARS-CoVe degradation). For each dataset, up to 000 sequences were truncated after two-third of their length, and the remaining one-thirds were generated under the same codon sampling protocol. We then applied pre-trained CodonBert- -hase property prediction model 23 to estimate task-specific labels and computed mean squared error (MSE) between predicted and true labels; lower MSE indicates better preservation of
7

---

Table 2: Performance comparison downstream benchmarks: Equi-mRNA models (trained 25M
sequences) outperform prior nucleotide codon-based baselines across biological tasks. 15M GPT-2 variant achieves the highest accuracy in out 6 datasets, demonstrating the effectiveness of symmetry embeddings at scale.
Model
E. coli(A)
MLOSS()
iCodon(S) Te-Ribo.(S) mRFP(S) COV Deg(S)
Nucleotide-Ban
RNA-FM
0.43
0,34
0,58
0.80
0.74
RNABERT (82 M)
0.39
0.16
0,47
0.40
0.64
Aido mRNA (L.6B)
0.576
0.504 0.23
0.23
0.472
0.492
0.683
0.743
CALM
0.430
0.170
0.376
0.625
0.546
0.773
mRNA-FM
0.509 + 0.154
0.458
0.690
0.564
0.714
Codon-Bacc
CodonBert M)
0.57
0.543
0.350
0.502
0.832
0.78
GPT2 (CLM)55M))
0.611
0.498
0.531
0.815
0.787
GPT2 (MLM)(50M))
0.653
0.503
0.569
0.753
0.801
HELM (CLM)
0.592
0.529
0.619
0.849
0.789
HELM (MLM)(50M)
0.701
0.525
0.626
0.822
0.833
Equi-mRNA (5M)
0.58/
0,705 + 0.12
0.519
0.764
0.853
0.756
Equi-mRNA (15M)*
0.613
0.710 + 0.13
0.537
0.737
0.855
0.791
Mamba-based hybrid Architecture;
GPT-2 Architecture both trained 25M datapoints
Trained Antibody mRNA sequences
biological function. All MSE reduction compared to Vanilla model for equivariant variants appear in Figure with complete results in Appendix A.I3.2
3.3.1 Enhanced Generative Fidelity with Equi-mRNA
Analyzing the generated sequences quality results, we observe a stark con- trast between the vanilla GPT-2 base- line and our symmetry-aware mod- els. The baseline's FBD increases with temperature indicating instabil- ity: whereas all equivariant variants exhibit smooth, pronounced decline in FBD T rises. The best per- former the equivariant Fuzzy 0 model achieves an FBD of 580 T 1.0, roughly 4.3 times improvement over vanilla. Equivariant Fixed 9 and Learned converge near 10a, about threefold gain.
Préchet Distance Temperature
2500
FBD (Fréchet Distance)
2250
2000
1750
1500
1250
1000
750
500
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Temporature
Model
vanilla
Leamed
Fuzzy Stietel
The sharp but smooth decline in FBD indicates that equivariant models in- creasingly capture latent sequence modes absent in the baseline but preva- lent in real data. Rather than degener- ating into noise, higher temperatures amplify those syntactic alternatives permitted by the group-theoretic prior, reducing distributional shift without sacrificing biologi- cal plausibility. In this way, symmetry-aware architectures and equivariance regularization convert temperature from a benign rescaling constant into a principled mechanism for structured sequence exploration.
Fised Stiefal
Figure 2: FBD vs. temperature for equivariant SO(2) variants and Vanilla model: FBD between generated and true suffixes iCodon improves (lower with temperature for all SO(2)-equivariant vari- ants, unlike the vanilla model which degrades. Fuzzy achieves the best score (~~580 at T=1.0), a 4.3x gain over baseline.

---

Compared Vanilla
High
Eq.
3.17
-3.31
Relative MSE Reduction (per dataset)
Eq.
Learned
Learned
3.07
6.23
1.45
1.19
4.38
Fuzzy + Eq.
7.42
28.14
4.68
6.49
19.69
Fuzzy St. Eq.
3.09
7.15
1.65
-0.73
9.14
ICodon
Tc-kibo.
COV
Low
Figure 3: MSE reduction for equivariant SO(2) variants vs. vanilla: SO(2)-equivariant variants consistently reduce MSE (lower 4) in property prediction across datasets, with up to 28% improvement over Vanilla model. Results reflect enhanced biological plausibility and downstream utility.
Lastly, scaling pretraining from 1M 25M sequences without any architectural modifications drives FBD down to 76.13 for the Equi-mRNA (5M) variant and 177.77 for Equi-mRNA (15M), representing significant leap forward generative fidelity at this scale. These results only demonstrate the remarkable scalability and robustness of symmetry-awe embeddings but also establish a new state of the for large-scale biological sequence generation.
3.3.2 Enhanced Property Retention under Equi-mRNA Embeddings
Beyond sequence quality, we assessed property prediction by training a CodonBert- based regression model on each dataset and measuring MSE [23]. Figure 3 demonstrates that equivariant SO(2) models consistently lower MSE relative to vanilla GPT-2 across all benchmarks achieving up to 28% reduction in error. Notably, the MLOS dataset, though small (<1000 sequences), comprises 1000 codon mRNAs, while Tc-Riboswitch sequences span only 26 codons; both contexts yield uniform MSE improvements under equivariance. Such substantial decrease in prediction error not only confirms the capacity of structured SO(2) embeddings to capture rotational symmetries and long-range dependencies, but also translates directly into more faithful preservation of functional properties. In practice, this level of accuracy can significantly reduce the cost and time of experimental validation by prioritizing higher-quality candidate sequences. For brevity, only equivariant-enforced variant results are shown here; full comparisons among fixed, learned, and fuzzy 0 models are provided in Appendix A32.
A.
3.4 Interpretability Analysis
To test whether the imposed SO(2) structure yields biologically meaningful representations within single species, we fine tuned a 5M parameter model on the human coding transcriptome (GRCh38) 27 and reserved 20% for evaluation. Focusing human controls for eross species confounders, lets us probe a consistent codon usage bias, and provides a clear path to codon optimization policies that are specific to organism rather than averaged across taxa. The goal is not only to interpret embeddings after the fact, but to validate that codon level rotational geometry captures constraints that matter for human design and optimization.
First, we quantified the uncertainty of learned codon angles using Shannon entropy and analyzed its dependence transcript GC content, which is known correlate of secondary structure and expression control. Binning sequences by GC proportion revealed a near linear trend mean angle entropy (r- 0.98, R² = 0.97, p < 10 10-11), suggesting that GC rich regions induce more uncertain codon rotations. This behavior is consistent with our parameterization, where the distribution over angles encode biological variability.
Second, we examined whether the codon embeddings capture translational supply constraints by comrlattnn learned angles with normalized human tRNA gene counts [18]. We selected the subspace block most predictive of tRNA levels and computed weighted composite angle per codon. Spear- man correlation of -0.69 indicates that codons with higher tRNA availability systematically

---

assigned smaller angles, consistent with efficient translation demands being reflected in the rotational geometry.
These results support the view that the group theoretic prior is both mathematically sound and aligned with human specific biology. Visualizations and further details are provided in Appendix A.8
Limitation and Future works
While Equi-mRNA introduces biologically grounded inductive biases by enforcing cyclic SO(2) rotations synonymous codons, it is currently constrained to protein coding regions and fixed triplet tokenization. This design overlooks non-coding elements such as untranslated regions (UTRs) and may obscure gene species-specific codon usage patterns. Furthermore, the incorporation of fuzzy angle distributions and Stiefel manifold rotations enhance representational flexibility but incurs additional parameterization and Riemannian optimization overhead, potentially hindering scalability in resource-constrained or large-scale settings. Current evaluations are limited to a narrow set of benchmarks, leaving the model's robustness on long transcripts, non-standard GC content, non-model organisms, and real-world sequence design tasks largely unexplored.
Future extensions may benefit from meta-leerning codon-specific priors using hyper-networks gen- erative modules that adapt rotation parameters dynamically across organisms, tissues, or experimental regimes with minimal fine-tume mun-tuning. Additionally, exploring richer group- theorett structures such as non-abelian or product groups could enable modeling of local codon interactions, frame-shift motifs, or positional dependencies. Expanding empirical validation applications will be essential for assessing generalization and informing deployment under realistic biological constraints.
diverse settings and practical
Conclusion
We introduced Equi-mRNA, a novel mRNA language modeling framework that embeds codon-level symmetries as differentiable group actions within the SO(2) Lie group. By aligning the genetic code's s redundancy with structured geometric priors, our model captures synonymous codon relationships through rotation-equivariant embeddings, auxiliary symmetry-enforcing losses, and biologically motivated pooling mechanisms. Across diverse benchmarks, Equi-mRNA consistently improves downstream property prediction accuracy and generative fidelity, achieving up to ~10% accuracy gains and 4.3x reductions in Fréchet BioDistance over strong baselines.
Beyond empirical performance, our interpretability analysis reveals that learned codon rotation patterns align with known GC-content and IRNA abundance biases, offering biologically grounded insights into translation regulation. Together, these results establish Equi-mRNA interpretable, and biologically faithful foundation for modeling protein-coding sequences, with broad implications for synthetic mRNA design and genomic modeling. Future work will extend this symmetry-awan paradigm to non-coding regions, richer group structures, and adaptive priors for cross-species generalization.
scalable,
Acknowledgments
This work used DeltaAl at NCSA through allocation CIS250398 from the Advanced Cyberinfrastruuc ture Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296 (4)
We thank Dr. Ivan Garibay for his thorough feedback on the manuscript and ongoing support throughout this research.
References
[1] Brandon Anderson. Truong Son Hy. and Risi Kondor: Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32. 2019.
[2] Ziga Avsec, Vikram Agarwal, Danicle Visentin, Joseph Ledsam, Agnieszka Grabska-Baree Barwinska, Kathryn R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene ex- pression prediction from sequence by integrating long-range interactions. Nature Methods, 18:1 18:1196-1203. 2021.
[3] Erik J Bekkers. B-spline cnns lie groups arXiv preprint 1205199999............
10

---

[4] Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. Access: Advancing innovation: Nsf's advanced cyberinfrasttuuctre coordination ecosystem: Services support. In Practice and Experience Advanced Research Computing (PEARC '23), pages 1-4, Portland, OR. USA, July 2023. ACM, doi: 10. 00.1145/3569951133955555 URL httts::///o/oo org/ 1145/3569911
3597559
[5] Florian Buhr, Sujata Jha, Michael Thommen, Joerg Mittelstaet, Felicitas Kutz, Harald Schwalbe, Marina Rodnina, and Anton A Komar. Synonymous codons direct cotranslational folding toward different protein conformations. Molecular cell, 61(3):341-351, 2016.
[6] NJ Caplen. Gene therapy progress and prospects, downregulating gene expression: the impact of rn interference. Gene therapy, 11(16;1241-1148, 2004.
[7] Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns homogencous spaces. Advances in neural information processing systems, 32, 2019.
[8] Flavio Dalla-Torre and al. Nucleotide transformer: Augmenting language models with nucleotide- level context genomics. bioRxiv, 2025. URL URLittpps//bbb.oog//0000////// 11.20 20. 00...667999 498 preprint.
[9] Michay Diez, Santiago Gerardo Medina-Muñoz Luciana Andrea Castellano, Gabriel da Silva Pescador, Qiushuang Wu, and Ariel Alejandro Bazzini. icodon customizes gene expression based on the codon composition. Scientific Reports, 12(1):12125) 2022.
[10] Zundan Ding, Feifei Guan, Guoshun Xu, Yuchen Wang, Yaru Yan, Wei Zhang, Ningfeng Wu, Bin Yao, Huoqing Huang, Tamir Tuller, al. Mpepe, a predictive approach to improve protein expression in coli based deep learning. Computational and Structural Biotechnology Journal, 20: 142-1153, 2022.
[11] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Prottrans: Toward understanding the language of life through self- self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7122-7127 2021.
[12] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgp12 is deep unsupervised language model for protein design. Nature communications, 13(1):4348, 2022.
[13] Xiang-Dong Fu. Non-coding ma: a new frontier in regulatory biology. National science review, 1(2): 190-204, 2014.
[14] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3) transffrrrrm::: 3d into-translation equivariant attention networks, Advances neural information processing systems, 33:1970- 1981, 2020.
[15] Ann- Ann-Christin Groher, Sven Jager, Christopher Schneider, Florian Groher, Kay Hamacher, and Beatrix Suess. Tuning the performance of synthetic riboswitches using machine learning, ACS synthetic biology. 8 (1):34 44, 2018
[16] Brian L Hie, Varun R Shanker, Duo Xu, Theodora UJ Bruun, Payton A Weidenbacher, Shaogeng Tang, Wesley Wu, John Pak, and Peter Kim. Efficient evolution of human antibodies from general protein language models. Nature Biotechnology, 42(2):275-283, 2024. 2024.
[17] Michael J. Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee whye Tch, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups In International conference on machine learning, pages 4533-4543 PMLR, 2021.
[18] James R Iben and Richard Maraia. trna gene copy number variation in humans. Gene, 536(():376-385 2014.
[19] Toshimichi Ikemura. Correlation between the abundance of *escherichia coli transfer mas and the occurrence of the respective codons in its protein genes: proposal for synonymous codon choice that optimal for the coli* translational system. Journal of Molecular Biology, 151:389-409, 1981,
[20] Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidircctional encoder representations from transformers model for dna- languang in genome. Bioinformatics, 37 2021.
[21] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International conference on machine learning, pages 2747-2755. PMLR, 2018.

---

[22] Reijer Lenstra. The graph, geometry and symmetries of the genetic code with hamming metric. Symmetry. 7(3):1211-1260. 2015.
[23] Sizhen Li, Saced Moayedpour, Ruijiang Li, Michael Bailey, Saleh Riahi, Lorenzo Kogler-Anele, Milad Miladi, Jacoh Miner, Dinghai Zheng, Jun Wang, et al. Codonbert: Large language models for mrna design and optimization. bioRuiv. pages 2023-09, 2023.
[24] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-seaa prediction of atomic-level protein structure with a language model. Science, 579(6637).1123-11
2023.
[25] Chuang Liu, Qiangqiang Shi, Xiangang Huang, Seyoung Koo, Na Kong, and Wei Tao. mrna-based cancer therapeutics. Nature Reviews Cancer, 23(8):526-5433 2023.
[26] Yi Liu, Qian Yang. and Fangzhou Zhao. Synonymous but not silent: the codon usage code for gene expression and protein folding. Annual review of biochemistry, 90(1):375 401, 2021.
[27] Jonathan M Mudge, Silvia Carbonell-Sala, Mark Diekhans, Jose Gonzalez Martinez, Toby Hunt, Irwin Jungreis, Jane E Loveland, Carme Arnan, If Barnes, Ruth Bennett, Andrew Berry. Alexandra Bignell, Daniel Cerdán-Vélee Kelly Cochran, Lucas T Cortés, Claire Davidson, Sarah Donaldson, Cagatay Dursun, Reham Fatima, Matthew Hardy, Prajna Hebbar, Zoc Hollis, Benjamin James, Yunzhe Jiang, Rory Johnson, Gazaldccp Kaur, Mike Kay, Riley Mangan, Miguel Maquedano, Laura Martínez Gómez, Nourhen Mathlouthi, Ryan Merritt, Pengyu Ni, Emilio Palumbo, Tamara Perteghella, Fernando Pozo, Shriya Raj. Cristina Sisu, Emily Steed, Dulika Sumathipala, Marie-Marthe Suner, Barbara Uszezynska- Ratajezak, Elizabeth Wass, Yucheng T Yang. Dingyao Zhang, Robert D Finn, Mark Gerstein, Roderic Guigó, Tim Hubbard, Manolis Kellis, Anshul Kundaje, Benedict Paten, Michael L Tress, Ewan Birney, Fergal J Martin, and Adam Frankish. Gencode 2025: reference gene annotation for human and mouse. Nucleic Acids Research. D3 D996, 11 2024. ISSN 1362-4962. doi: 10. 10.103//ar/gkaee100 CtRL/htts::// org/ /wtt////////aa//////
[28] Eric Nguyen, Michael Poli, Matthew G Durrant, Armin W Thomas, Brian Kang, Jeremy Sullivan, Made- lena Y Ng. Ashley Lewis, Aman Patel, Aaron Lou, al. Sequence modeling and design from molecular to genome scale with evo. BioRxiv, pages 2024 02, 2024.
[29] Thijs Nieuwkoop. Barbara R Terlouw, Katherine G Stevens, Richard A Scheltema, Dick De Ridder, John Van der Oost, and Nico Claassens. Revealing determinants of translation efficiency via whole-gene codon randomization and machine learning. Nucleic acids research, 51(5):2363-2376, 2023.
[30] Daniel A Nissley and Edward O'Brien. Timing is everything: unifying codon translation rates and nascent proteome hehavior. Journal of the American Chemical Society, 136662) 2014.
[31] Carlos Outciral and Charlotte M Deane. Codon language embeddings provide strong signals for use in protein engineering. Nature Machine Intelligence, 6(2): 17-1177, 2024.
[32] Carlos Outeiral, Daniel A Nissley, and Charlotte M Deane. Current structure predictors are not learning the physics of protein folding. Bioinformatics, 38(7): ( 188--1887 2022.
[33] Norbert Pardi, Michael J Hogan, Frederick W Porter, and Drew Weissman. mrna vaccines-a new era in vaccinology. Nature reviews Drug discovery, 17(4):261- 279, 2018.
[34] Joshua B. Plotkin and Grzegorz Kudla. Synonymous but not the same: the causes and consequences of codon bias. Nature Reviews Genetics. 12:32 42, 2011.
[35] Joshua B Plotkin and Grzegorz Kudla. Synonymous hut not the same: the causes and consequences of codon bias. Nature Reviews Genetics, 12(1)232-42, 2011.
[36] Pawel Pratyush, Callen Carrier, Suresh Pokharel, Hamid D Ismail, Meenal Chaudhari, and Dukka B Kc. Calmphoskan: prediction of general phosphorylation sites in proteins via fusion of codon aware embeddings with amino acid aware embeddings and wavelet-based kolmogorov arroll network. Bioinformatics, 41(4): btaf124. 24, 2025.
[37] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[38] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott. C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences. 118(15): e20162391 18, 2021.
12

---

[39] Aviv A Rosenberg, Ailie Marx, and Alex M Bronstein. Codon-specific ramachandran plots show amino acid backbone conformation depends identity of the translated codon. Nature communicationss 13(1): 2815, 2022.
[40] Ugur Sahin, Katalin Karikó, and Özlem Türeci. mrna- mrna-based therapeutics deweelopiessd a new class of drugs. Nature reviews Drug discovery. 13(10))759-78U) 2014.
[41] Raymond Sanabria, Soobin Kim, Jacob Linder, and Anshul Kundaje. Grover: A byle-pair encoded genomic language model learns regulatory elements. bioRxiv, 2024. URL htttt:://///oooo. org/ 11110111111014 02. 14. 580349 preprint.
[42] Rhodri Saunders and Charlotte Deane. Synonymous codon usage influences the local protein structure observed. Nucleic acids research, 38(11)667194 19-6728, 2010.
[43] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pages 9377-9388. PMLR, 2021.
[44] Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423, 1948.
[45] Paul M. Sharp and Wen-Hsiung Li. The codon adaptation index-a measure of directional synonymous codon usage bias, and its potential applications. Nucleic Acids Research, 15:1281-1295, 1987.
[46] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint 058441,402.05841, 2024.
[47] P Thölke and G De Fabritii.. Torchmd-net: Equivariant transformers for neural network based molecular potentials. arxiv 2022. arXiv preprint arXiv:2202.02544/ 2023.
[48] Hannah K Wayment- Steele, Wipapat Kladwang, Andrew M Watkins, Do Soon Kim, Bojan Tunguz, Walter Reade, Maggie Demkin, Jonathan Romano, Roger Wellington-OOuri. John Nicol, et al. Deep learning models for predicting rma degradation via dual crowdsoureing. Nature Machine Intelligence, (12):1174-1184, 2022.
[49] Matthew Wood, Mathieu Klop, and Maxime Allard, Helix-mmaa hybrid foundation model for full sequence mma therapeutics. arXiv preprint 2rXiv:2502.13785 2025.
[50] Ruidong Wu. Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnic Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pages 2022-0022000 07, 2022.
[51] Junjie Xu, Artem Moskalev, Tommaso Mansi, Mangal Prakash, and Rui Liao. Beyond sequence: Impact of geometric context for RNA property prediction... In The Thirteenth International Conference on Learning Representations, 2025. URL htt//ttcps///ppereeeeee. weet/forrmmid--hhteeee
[52] Mehdi Yazzani-Jahrona Mangal Prakash, Tommaso Mansi, Artem Moskalev, and Rui Liao. Helm: Hierarchical encoding for mrna language modeling. arXiv preprint arXiv:2410.12450 2024.
[53] He Zhang, Liang Zhang, Ang Lin, Congcong Xu, Ziyu Li, Kaibo Liu, Boxiang Liu, Xiaopin Ma. Fanfan Zhao, Huiling Jiang, al. Algorithm for optimized mrna design improves stability and immunogenicity. Nature, 621(7978):396- 403,, 2023.
[54] Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning-based sequence model. Nature Methods, 12(10):931-92) 2015.
[55] Zhemin Zhou, Yuning Ji, Han Liu, and Ramana V Davuluri. Dnabert-2: Efficient and interpretable pre- trained language models for genomie sequences. bioRxiv, 2024. URL htts::/t///o 1101/ preprint.
2024. 01. 10. 575570
[56] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-specics genome. arXiv preprint arXiv:2306. 15006, 2023.
13

---

A Technical Appendices and Supplementary Material
A.1 Preliminaries
This section introduces the algebraic and geometric foundations that motivate our model design, particularly the use cyclic groups, the special orthogonal group SO(2), and the Stiefel manifold. These constructs enable biologically grounded and differentiable representations of codon symmetries in mRNA sequences.
Group and Cyclic Groups. A group is G equipped with binary operation satisfying closure, associativity, identity, and invertibility. A group is called cyclic if there exists generator E G such that every element in G can be written as g for some integer The finite cyclic group of order is denoted En, with addition modulo n as its operation.
Group Action. A group action of G on a set X is function the identity transformation on X, and the action respects the group composition. In context, synonymous codons form equivalence classes C. C C, and the group Zn, acts on C. by cyclically permuting codons.
- X such that the identity acts
Lie Groups and (0(2). A Lie group is smooth manifold endowed with a group structure. The group S2( C R2x2 consists of 2D rotation matrices of the form
cos R(0)= = sin 0 cos
sin 0
forming compact and differentiable group under matrix multiplication. We construct a homomor- phism O Z,, (O()) by mapping codon labels to evenly spaced angles 0g 2rk/nna enabling gradient-based learning on codon symmetries.
Stiefel Manifolds.
The Stiefel manifold St(k, d) is the all orthonormal k-frames in Fd i.e.
St(k, d) {V E I d V = [1)-
This manifold defines a smooth subspace of kd with non-Euclidea. geometry, and is central to our parameterization of codon rotation subspaces. In Equi-mRNA, each amino acid defines local 2D subspace spanned by an orthonormal frame (u.. UA) E St(2, d), within which synonymous codons rotated.
A.2 Overview of Codon Embedding Variants
To complement the detailed description in the main text, Figure provides a unified visual summary of our codon embedding framework and its key variants. Each model encodes synonymous codons as rotated versions of a shared amino acid base embedding, differing along three key design axes: the rotation generator (fixed, learned/cyclic, learned Stiefel subspace). This schematic illustrates the modular structure of the embedding pipeline and how group-theoretic principles govern the construction of codon-level representations across all twelve variants evaluated in our experiments.
fuzzy), and the rotation basis (standard 2D plane or
A.3 Block-Diagonal Codon Embeddings in Higher Dimensions
While S0(2) suffices to model codon substitution as a planar rotation, is often beneficial in practice embed codons in higher-dimensional vector spaces. To preserve the group-theoretic structure while scaling up the representation space, we extend the S2( embedding to a block-diagonal 50(d) representation, where d is even.
We partition each codon embedding vector into d/2 disjoint 2D subspaces. Each subspace is then rotated independently by angle 0!0) E IR, where i 1, ... d/2 indexes the sub-blocks, and (( is a learnable generator parameter specific to amino acid a. Let e (0! ,...,0!//2 E dRd/2 denote the full set of Iearned rotation generators for amino acid
14

---

Compact View
Codon Relative
Input Seg: AAGGAAAAAAAAAAUUCACO...
Embedding Layer
Codenic))
Jnitall processing
onn En.Eeeiniiinngon Jay, idyy Leanable Base Vectors
OGI
Cuslons: AEG, AGA,ACA,, GAU, UUA, CGU,...
Bing big. buy, bug. bug, bug
'oaal mumthr of pndeaa for inal Indived from Dre
cadons
Gensmiee
Rossion Tlads
Aci (an) Seq Me: M.R.T.D........
Finul Embedking: by
odon Relative ids: (00,).55230 (0055/055..
Generator Types
Rotation
Fixed
Learned
Soft Funzy
Standard
Ma
Ug = (li....6""))
angle
Using 'R' from
Unilem angle
Embedding dimenson
2l Pey !J Piy probability of angle f Rg number symmymmnu codons for amno acid a
Rassdard
(PL())) Ua
Larmal (91a): oule) x nod Ze
Stiefel
(Lux Bra
Bus)
Bla Lakn
Indiating Stiffl veetrre
cosoi
G 5,
EEampls forAGA
R:
0
6. 660
Pe(AGA)
R-
Rebeive (1) (tr)
Fiazy
Rimmanies optimizer to upddist u. 7 weights
Aa - x. 240'
Rezy
[coe Spx Jai
LAN 9/4
8pa)
> Rggaarat - Vgy
Usng generator
SAAAYANA Codos Asssicnn Joandd Saliuare
Figure 4: Illustration of our codon embedding framework and the key variants evaluated in this study, Each codon is mapped to rotated copy of its amino acid base vector, using either fixed, learned (cyclic), or fuzzy angle generators. Rotation applied either a fixed 2D plane or learned subspace on the Stiefel manifold. This figure summarizes the geometric mechanisms underlying the twelve variants listed in Table 3
Table 3: Overview of codon embedding variants studied. Each can be trained with or without the equivariance loss. "Standard" uses a fixed 2D subspace, "General" uses a learned Stiefel manifold basis. "Cyclic" uses learned generator angles, "Fuzzy" uses soft angle distributions.
Model Name
Rotation Basis
Generator Type
S02-fixed-sttadare
Standard (fixed)
Fixed generator (uniform angles)
S02-fixed-generall
Learned basis (Stiefel)
Fixed generator (uniform angles) Learned generator (cyclic angles) Learned basis (Stiefel) Learned generator (cyclic angles)
502-cyclic-standarr
Standard (fixed)
$02-cyclic-generaal
$02-fuzzy-standard
Standard (fixed) Soft generator (fuzzy angles)
S02-fuzzy-generze
Learned basis (Stiefel) Soft generator (fuzzy angles)
Each codon c E C, is represented by the block Bock-diagooal matrix:
d/2
sin sin(
D =
COS
sin
cos cos
where R(0) E SO(2) is the standard planar rotation matrix. This generalization retains the equivariant group action structure: the embedding of codon c is obtained by applying the same group label (c)(c) E In, to all subspaces, but using separate generators for each.
Because each block acts independently and satisfies the same group multiplication rules as the base case, the full map Te. defines a homomorphism from Σ,,, into a block-diagone subgroup of SO(d). A formal proof is provided Appendix A.2 This construction enables each amino acid's codon group to learn richer, task- rotational structure across multiple subspaces, while still preserving group coherence. The learned parameters @,, thus define smooth, differentiable family of cyclic representations tailored to the downstream task.
15

---

A.4 Equivariance Enforcement via Auxiliary Loss
central goal of our codon embedding framework is to respect the symmetry structure induced by synonymous codons. Specifically, we aim to ensure that the learned representations are equivariant to the group action defined by codon substitutions: that is, rotating a codon within its synonymous group should induce a predictable and structured transformation in the model's internal representation.
While embedding construction encodes codon-level structure using SO(2) block-diagonal SO(d) transformations, these symmetries can be diluted or forgotten by subsequent layers in the model if not explicitly preserved. To address this, we introduce an equivariance regularization mechanism that reinforces the geometric structure throughout the network by aligning the transfor- mations induced by codon substitutions with transformations in representation space.
Equivariance objective. Let 2 E Rd denote the base embedding of amino acid A, and let R. E SO(d) be the rotation matrix corresponding to codon E Ca. The codon-specific embedding is then E(c) = RAZA To enforce equivariance in the downstream model f()), we pass both the base embedding A and the rotated codon embedding E(c) into the network. Specifically, we require that the output of the network behaves consistently under this transformation:
f(R,ZA) = Rzf(Za).
(2)
To promote this behavior, we define an equivariance loss term:
Lequi
[(A,)) [[((RZZA) - Ref(z))1
(3)
which penalizes deviations from equivariance the distribution of codons for each amino acid A. This term is added to the primary task loss, resulting in a total objective:
Loxcll
Lusk + Aequiv Lequiv
where Aequiv controls the strength of the symmetry regularization.
Enforcing equivariance yields several advantages. First, it ensures that synonymous codon substitu- tions, modeled as group actions, produce predictable and structured effects in the model's internal states, enhancing interpretability and robustness. Second, encourages the model generalize better across unseen synonymous codons or species with different codon usage, since synonymous substitu- tions correspond to smooth, equivariant transformations rather than arbitrary shifts in representation. Third, it introduces an inductive bias that acts as a regularizer, improving performance in low-data regimes and enabling transfer biologically diverse contexts.
Importantly, equivariance also facilitates fine-tuning of rotation angles 02 in downstream tasks. Since the network is trained be equivariant with respect codon group actions, small updates to the generator angles 0a (e.g.. to adapt to species-specific codon bias or task-specific codon effects) do not destabilize the representation space. Instead, the model can smoothly adjust the codon orbits without breaking its internal alignment, allowing the embedding geometry to shift in a controlled and interpretable manner. This makes the framework especially well-suited for transfer learning scenarios and downstream finetuning tasks where codon preference may shift but the overall group structure remains consistent.
A.5
Equivariant Pooling Mechanisms for SO(2) Representations
A.5.1 Cartesian Product of SO(2) Groups
The embedding space in our model consists of m independent 2D subspaces, each encoding a rotation in SO(2). These subspaces correspond to codon-specific blocks in the full emhedding vector of dimension D = 2m. Each block lies in TR2 and encodes a codon- or amino-acid-level transformation rotation vector (cos 0,, sin 0.)) for some angle 0 [0, 2Tr). To model this structure formally, consider the total embedding space as the Cartesian product of m copies of the special orthogonal group SO(2):
G SO(2) x SO(2) ... ... SO(2) S
m times
16

---

This group G = (0 is direct product of Lie groups and forms smooth manifold equipped with natural group operation defined component -iie.. Each element g E can be written tuple of rotation matrices:
(R, R((m))))
where R(O,) SO(2) is 2 matrix representing rotation in the i-th subspace:
cos 0,
sin 0g cos 0;
R(O,)
sin 0,
The group operation in G' is defined component-wise:
g h (R(0:) R(02) R((m.. R(om))
for g (R(6))....., .,. R(e....)) and h (R(61), R(om)).
This product structure induces natural notion of independent equivariance in each rotational subspace. That is, for a transformation to be equivariant under G, it must commute with the action of each R(0,) individually:
f(g x)) = g f(z), Ag E G, E 2
In our model, each codon embedding is represented in this G-equivariant space. Consequently, pooling sequence of such codon embeddings should ideally respect the product group structure. This justifies using blockwise angle-space aggregation, in which pooling operates independently within each (2( factor.
Biological Relevance. composition. Each codon position contributes independently the semantic representation of the sequence, analogous to how each gene segment contributes to protein folding or expression. Modeling the embedding space as SO(2)" allows for biologically meaningful invariances (e.g.. synonymous codon substitutions) to be explicitly encoded in the architecture, enhancing robustness and generalization.
The product structure SO(2) captures the modularity inherent in codon
Connection to Equivariant Pooling. Let (()) E Rm denote the angle vector at sequence position for each of the rotation blocks. The pooling operation
pposled Σ wi ((1) mod 2x
defines mapping from G - G that is group homomorphism when the weights wy form a convex combination. This mapping is the formal basis of our SO(2)-aware pooling mechanisms introduced in the next section.
To preserve the equivariant structure of codon embeddings under the action of the group SO(2), is crucial to aggregate sequence representations in a way that respects the geometric nature of the embeddings. Standard mean or max pooling Cartesian space may violate rotational equivariance, as these operations do not align with the circular topology of angle-based embeddings.
Given codon embeddings where each pair of dimensions represents 2D rotation vector (I,, y1) R² corresponding to rotation matrix R(O,) E SO(2), we aim to construct pooling function ø that maps the Cartesian product group G = G1 x ... x G, (where each G; C SO(2)) single SO(2) element while preserving group structure.
Homomorphic Averaging. Let each codon embedding correspond to a group element R(A,) e (2(,), and consider weights wi 2 0 such that Σ w, 1. Define the mapping
(((0))...... ((())...,,,()))) R
Σ wi Og mod 2x
17

---

We can verify that Φ is a homomorphism:
$(g h) = ووg)) @()) Vg, Jy. EG,
which ensures that equivariant properties are preserved under pooling. This motivates the of angle-space aggregation as a principled pooling strategy when working with rotation-encoded representations.
We now describe three such mechanisms:
A.5.2 Polar Pooling
Polar pooling converts each 2D embedding (ær, y) into its polar form (r; f), pools the representations in the complex plane using angitude-weightee averaging, and then converts the result back to Cartesian coordinates. Specifically, for a sequence of vectors {(rs, Bc) we compute the complex representation Tic and define:
2pooled
L
Σ
241 (Ipooled: Mpooled)
(pooleed) 3 (2pooled)
This method preserves the angular structure and is invariant to rotation magnitude noise. has the advantage of being simple implement and interpret while respecting the geometry of SO(2).
A.5.3 DFT-Based Rotation-Aware Pooling
We further introduce a frequency-domain pooling method that performs a Discrete Fourier Transform (DFT) along the sequence length for each SO(2) block. The core idea is to learn spectral filter W E C applied in the frequency domain:
X FxT(x), X X + W Xpooled (XLFFT(XXiteedd)
This allows the model to learn seruunccy-specifee weighting, acting as soft attention over periodic patterns the sequence. Additionally, angular modulation via learned sinusoidal positional weights enhances sensitivity to codon position and rhythm, making this approach well-suited for detecting periodic or repeating structures mRNA.
A.5.4 SO(2) Mean Pooling
In this approach, we directly pool in angle space by computing the weighted average of the rotation angles. For each codon embedding, we first convert angle space via:
Og atan2 atan2 (yr.),
and apply attention weights a; derived from the embedding magnitudes:
Opooted.j atan2 Σ oz sin Oz.j+ Σ Oz cos cos
The final embedding is then reconstructed as:
(Ipooled 3pooked) (cos pootod sin ppooood).
This pooling strategy explicitly equivariant under the action of SO(2) and allows interpretable analysis of angular dynamics within sequence.
Advantages. All three pooling mechanisms respect the group structure of S0(2) and support equivariant downstream processing. Compared to standard pooling, these approaches:
Preserve biologically meaningful angular relationships between synonymous codons.
Enable learnable or data-driven aggregation of periodic structures in sequence space.
Provide a theoretically grounded alternative to Cartesian pooling that aligns with group- theoretic design principles.
By incorporating these pooling methods, we enable robust, equivariant representations our mRNA models, which are crucial for generalization across different codon usage patterns and species.
18

---

A.6 Mathematical Details and Proofs
Proposition A.1 (Codon Mapping Homomorphism with Arbitrary Generator). Let Gg = Zn. be the group of synonymous codon substinutions for amino acid a, and let Pa Ca - 2m, be bijective labeling of its codons. Let Oa E Th. any fixed angle such that nl 0a (mod 2r). Define the map
28. C. - S0(2),
(c,(c) = R((pac))-000 0a).
Then De, is a group homomorphism from G. into SO(2), Moreover, if 02 is a generator of a cyclic subgroup of 50(2) of order nl (i.e., 0g 2mm with god(m, naa) = 1), then Po, is an isomorphism onto its image.
n
Proof. Let C1, C2 E C. with integer labels K = (a((c) and k2 = pa(()). The group operation in G. corresponds to codon composition
C3 = Vo ((ki + k2) mod ma).
Then,
De. (c1) Lo_ (c2) = R(() R(k20.) R((A) - -()))))))0.)) = 22 (ca),
by the additive property of rotations. Hence, Po., preserves the group operation and is homomor- phism. To establish injectivity (and thus isomorphism onto its image), note that R(j0.)) = R(j'0.) if and only [j - j') = (mod 2x).
This implies
j' (mod na) whenever n,,0, = 0 (mod 2T) and ged(m, na) = 1 where 0a 2mm In this case, $8. is injective on Zn, and hence bijective from C2 to its image in
SO(2).
The image &o. (C.) forms cyclic subgroup of SO(2) of order Ra. generated by R(O.). Therefore, Po. is an isomorphism from G. onto this subgroup.
Proposition A.2 (Cyclic Group Representation via Block-Diagondl Rotations). Let Go = Bn. be the substitution group for amino acid o, and let
θ = (0 " gld/2)) ) e RR/22
be a set of generator angles. Define the embedding map:
T ( = blockdiag ( R(jo(d/2))) E SO(d),
where R(-) E SO(2), and Zm...
Then &e. is a group homomorphism, from Zn. into (O)d), and its image forms cyclic subgroup of (O(d) of order dividing na. with equality if ged(na: m)) = for all i, where (
20mp
Th
Proof. Each subspace map j + R(jo.") is a homomorphism from Zn. (2(,), and by Propo- sition A.1 forms cyclic subgroup of SO(2) of order dividing na Since block lorekddaagnal matrix multiplication distributes over block components, the full map &e. (fr + j2) $ (h1 Po. (f2). proving that &e, is homomorphism. Injectivity (i.e., full cyclic order n..) holds if each o0¹¹ 2am.
ne
with ged mm, naa) ensuring each component cycles with full order. Thus the image of to. is cyclic subgroup of (O(d).
Proposition A.3 (Codon Embeddings via Stiefel-Maniaae Subspaces). Let VA E St(2, be an orthonormal basis for a subspace of Rdd, and lei ZA Rdd be the base vector for amino acid A. Define E(c) = VA VV. ZA. where R(oe) E SO(2). Then:
1. E(c) d lies a circle in the subspace spanned by VA.
2. The transformation VaR( V (d(d) acts as a rotation in the 2D subspace and as identity on the orthogonal complement.
Proof. (1) Since VIZA E R², the codon embedding E(c) = VAR(O.) zA is the image a 2D rotation applied to the projection of z into the subspace defined by Va. As R(o.) preserves Euclidean norm, all embeddings lie at radius ||V ZAll. tracing circle.
19

---

(2) The matrix Va VA is a similarity transformation acting only in the 2D plane spanned by VA. is orthogonal because:
(V (V = VARR - VAVA.
which is projection matrix onto the 2D subspace. Within that subspace, the operator behaves exactly R(Φ.) E (2).). On the orthogonal complement, it acts as identity since the projection of z zero. Hence the overall transformation is block rotation in SO(d).
Proposition A.4 (Sequence Homomorphism), Let's denote mapping
n w. 02 mod 2x)
(())), .... R(O,)) R
where each R(O:) is a rotation in a subgroup of SO(2) and wi are fixed (or learned) weights. want to show that & is homomorphism from the Cartesian product group
G Gi G2 Gn.
into H (2))). In other words, for any two elements
(R(o1), R(a2). .... R(an))
and
h (R(3)) ... R(B,))
in G, we need to prove that
6l p(g) o(h).
Proof. Group Operation in G: Since each G; is subgroup of SO(2), its group operation is given by addition of angles (modulo 2r). Thus, the group operation in G is defined coordinate-wise:
g h (R(01 B1), R(o2 + 92). .... R(a, + Bn)).
Using the definition of c. we have
ő(g h) = ( R(a1 + B1), .... R(an + Ba)) R Σ (a. + 3x) mod 2n
Because the weighted sum is linear, we can write
Σ wi (a, + B,)
Σ
wi 0q +
Σ
wi B,.
1=1
1=1
Therefore,
n
ö(g h) = R
Wi ai +
wi 8
mod 2T
R(a) R(b) R(a + mod 2T).
Thus, have
20

---

R
04
+
Bi R
Σ
Wi 0i
R
Σ wi Bi
i=1
By the definition of o,
n
( R
Σ
W/ 0i
@(h) R
Σ
wi B3
Therefore, we obtain
0g - ( o(h).
A.7 Background
Advances in genomic language modeling have introduced a range of strategies for representing biological sequences, each with different trade-offs in terms of biological fidelity, expressiveness, and computational efficiency. At the core of this challenge lies the question of whether models should operate on raw nucleotide sequences, protein- level amino acids, or intermediate representations such as codons. Each level captures distinct aspects of the biological signal: nucleotide-level models retain the full sequence information, amino acid models emphasize protein function, and codon-level models provide balance by preserving the coding frame and capturing synonymous variation. Understanding how these representations influence model performance critical for designing language models that are both biologically grounded and effective across wide range of downstream tasks.
A.7.1 Nucleotide-Ievel models (DNA/RNA as a sequence of bases)
Nucleotide-Ievel tokenization treats the coding sequence simply as a long sequence of nucleotides (characters A, G, T/U). Traditional sequence models like recurrent neural networks or trans- formers can be applied at the base level, sometimes using sliding windows or overlapping k-mers for tractability. For example, early genomic LMs such as DNABERT tokenized DNA sequences into overlapping k-mers (e.g. 6-mers) and then learned representations with the BERT architecture (20.. Nucleotide-level models have the advantage of retaining all information (including non-coding regions or regulatory motifs in the sequence context), but they do not inherently recognize the codon structure. Synonymous codons will be treated as completely unrelated tokens unless the model implicitly learns their interchangeabillty from data. The vocabulary (4 nucleotides or set of k-mers) relatively small, but sequences are three times longer at the nucleotide level than at the codon level, which can pose challenges for model input length and learning long-range dependencies. Models like DNABERT (20) and the Nucleotide Transformer (8] tokenize sequences into overlapping k-mers (e.g., 6-mers), striking a balance between biological context and computational tractability. While effective at capturing regulatory signals and enabling genome-scale analysis, these models lack an inherent inductive bias toward codon structure unless explicitly modeled. Recent approaches, such as DNABERT-2 [55] and GROVER [41), explore adaptive tokenization methods like byte-pair encoding to better reflect sequence regularities. Collectively, these models have demonstrated strong perfor- mance across a variety of genomics tasks by leveraging patterns learned directly from mucleotide-level input
A.7.2 Amino acid-level models (operate on the translated protein sequence)
Here, the DNA coding sequence is first translated to the corresponding amino acid sequence, and then protein language model is applied. This effectively discards the specific codon information. reducing the sequence to the 20-standard amino acid alphabet. Many state-of-the -hrt protein language models use this approach, treating proteins as sequences of amino acids (e.g., ESM and related transformer models (24 38 II). By focusing on amino acids, these models tap into signals of protein structure/function and avoid the complication extremely long nucleotide sequences. Indeed, large protein LMs have shown remarkable success in predicting structure and function directly from
21

---

sequence embeddings [8]. However, by design amino-acid models ignore synonymous codon variation. They cannot capture codon bias effects on translation or gene expression, since all synonymous DNA sequences collapse into the same amino acid sequence. For tasks where the protein phenotype paramount (e.g. structural modeling), this may be acceptable [88] but for tasks related to gene expression, regulatory control, or srecies- usage, amino-acid-onl, models miss a vital piece of information. As an example, codon usage biases differ by species, and aspects like translation elongation rates or mRNA stability are influenced by the specific codon sequence not just the resulting protein. Thus, while amino acid LMs leverage evolutionary signals in protein space, they lack sensitivity codon-level features by construction. Synonymous codon usage has been linked to specific structural characteristics of proteins [42 39]. and accumulating evidence highlights relationship between codon usage patterns and protein folding processes B0 26]. Such relationships suggest that codon-level signals may carry essential structural information currently missed by popular protein structure predictors, such ESMfold (2 and OmegaFold (50], which predominantly leverage language models designed to identify sequence-level correlations rather than explicit physical principles of folding. Given that existing deep learning approaches have been shown to inadequately capture the underlying biophysical mechanisms governing protein folding (32), incorporating synonymous codon usage signals could significantly improve structural predictions.
Codon-level or 3-mer models (using codons as tokens) A compromise approach is to tokenize the sequence in codons (triplets of nucleotides) or other 3-mer genomic words, so that each token represents a genomic substring of length three. In coding regions, aligning 3-mers precisely to the reading frame is particularly intuitive, each token directly corresponds to one amino acid (or stop signal) in the protein sequence. This representation preserves synonymous codon distinctions while simultaneously reducing sequence length by three-fold compared to nucleotide-level models. The resulting vocabulary of codons includes 64 tokens (61 sense codons plus 3 stop signals), a size
somewhat larger than the amino acid alphabet but still highly practical for language modeling. Recent approaches such as CodonBERT (23), HELM (52), CaLM (36), and DNABERT using 6-mers [56] have shown that codon-level representations can outperform standard amino acid models by capturing nuanced codon bias signals like species-specific preferences and rare codon placements [31 32
Overall, encoding sequences at the codon level represents a straightforward yet powerful strategy for embedding explicit knowledge of the coding frame into language models.
Codon-level tokenization captures synonymous differences but fails to reflect the biological relation- ships between codons that encode the same amino acid. Hierarchical approaches, such as HELM (Hierarchical Encoding for mRNA Language Modeling) [52), address this by explicitly modeling the codon-amino acid structure during pretraining. HELM introduces a codon hierarchy into the loss function, penalizing the model less for predicting synonymous codons, thus encouraging embeddings that cluster biologically equivalent codons together. This alignment with the genetic code significantly improves both predictive and generative performance, outperforming standard models by 8% across diverse tasks and producing sequences that better reflect natural codon usage. These results highlight the value of embedding biologically grounded inductive biases into language models for mRNA.
A.7.3 Limitations of Codon-Level Embedding Models and Motivation for Embeddings
Despite their advantages, codon-level embedding language models have important limitations that constrain their robustness and generalizability. A critical drawback is that the learned codon em- beddings typically lack interpretable or meaningful geometric relationships: although synonymous codons may cluster in the embedding space due to the structure of the loss function, these models do not explicitly encode biologically meaningful relationships among codons. For instance, while HELM (52) implicitly prefers synonymous codons due to optimization pressures, it does without establishing tangible structural or biological relationships between codons in the embedding space. As consequence, the embeddings learned by such models can be brittle; small perturbations in the embedding vectors or minor deviations from the training distribution may cause significant degradations in model performance even nonsensical outputs.
Another critical limitation stems from the difficulty of adapting these embeddings to new species or contexts via fine-tuning. Codon usage preferences vary significantly across organisms [35). and ideally, a model should easily adjust its embeddings to capture species-specifie codon biases. However, current codon-level embedding methods are typically not robust to fine-tuning, as the
22

---

entire downstream network heavily relies on the original embedding structure. Small adjustments in codon embedding vectors, intended to reflect new codon preferences, often propagate unpredictably throughout the network, effectively destabilizing the representations and yielding outputs that degrade biological fidelity or model accuracy. Thus, rather than enabling straightforward adaptation, attempts at fine-tuning may inadvertently disrupt the learned codon relationships, resulting in substantial loss of performance and biological interpretability.
A.7.4 Equivariant Architectures and Symmetry-Awar Modeling
Incorporating symmetry into neural architectures has emerged as powerful inductive bias for learn- ing over structured domains. Group equivariant neural networks (G-CNNs) extend the notion of translational equivariance in convolutional networks to more general symmetry groups, enabling the model to respect intrinsic invariances in the data [21 3 While early applications of equivariance were rooted in image domains, exploiting rotation and reflection symmetries. recent works have expanded these ideas into non- non-Euclidean settings and non- non-linear operators such as self-attention. For instance, the Tienoosoorrrrr 117 generalizes attention mechanisms to arbitrary Lie groups, demon- strating equivariant self-attention in diverse domains including molecular property prediction and physical trajectory modeling. Similarly, TorchMD--Ne (47) highlights the advantage of rotationally cquivariant transformers in modeling vectorial and tensorial outputs for molecular systems, showing improved generalization even for scalar-valued properties.
Rotationally equivariant neural networks have proven especially effective in domains where predicted outputs are vectors tensors. Trans))ranssrmerss [14], Cormorant []]. and equivariant message passing networks [3] leverage geometric symmetry to model 3D molecular systems with higher accuracy and efficiency. While these models were developed outside of biology, they highlight the broader value of enforcing structured equivariance. Inspired by these principles, our approach introduces form of rotational symmetry within the codon space of mRNA, aiming to reduce redundancy among synonymous codons through group-theoretic modeling-
To address these shortcomings, there is strong motivation to develop embedding frameworks that explicitly integrate the known symmetry and redundancy of the genetic code from the outset. Incor- porating codon synonym groups mathematically defined equivalence classes could substantially reduce model complexity and allow models to directly exploit biologically relevant variation, such as species-specific codon usage patterns, without the risk of destabilizing learned representations. This approach aligns closely with established biological knowledge and promises scientifically-gnee -rruuned Al models that inherently data-efficient, interpretable, and robust. Ultimately, creating such structured codon representations would not only address the current limitations of existing language modeling techniques for DNA and RNA but would also enable scalable Al systems capable of effectively navigating vast genomic sequence spaces for critical health-related applications, including predicting gene expression or designing optimized mRNAs for vaccines (53)
A.8
Interpretahility Analysis
A.8.1
Evaluation of Model Interpretability
To rigorously assess whether our SO(2)- struucuuue codon embeddings encode biologically meaningful signals, we performed two targeted analyses.
1. GC Content vs. Entropy of Codon Angles Genomic GC content a major driver of codon usage bias (34] We therefore measured how dispersed the learned codon angles are function of transcript GC%. For each coding sequence s of length N codons, let f1 oI,..., By E [0, 2Tr) denote the model's predicted angles. We estimate the empirical distribution
Σ 100 E bin,), i= K
Pi
by binning the circle into K equal-width intervals. The Shannon entropy of the angle distribution is then
H()) =
Σ
ps logg Pi.
23

---

Model Angle Distribution Entropy vs GC%
Blocks [44]; R7=0.48, p=-0.69
0.035
Spearman-1.00
Weighted mean rotation ainlle trad)
0.030
0.8m
Mean IRNA count
0.6R
Avg entropy
0.020
0.5m
0.015
0.010
0.3R
0.2m
0.000
0.0
0.2
0.2
0.3
0.5
0.6
0.7
0.8
10 Mean ERNA count (binned)
30
GC%
(a) Angle Distribution Entropy GC-Content
(b) Weighted Mean Angle vs. iRNA Abundance
Figure 5: Interpretability of SO(2) codon embeddings. (a) Mean Shannon entropy of learned codon angles, binned by transcript GC-content, exhibits a near-linear increase (Pearson r 0.98, R² 0.97, < 10 1), confirming that GC-driven biases are captured in angular dispersion. (b) Weighted mean rotation angle per codon versus normalized human tRNA gene copy number (top correlating block), showing that codons decoded by more abundant tRNAs are assigned systematically smaller angles (Spearman = 0.69, p < 10 10- indicating the embedding's alignment with translational efficiency.
We grouped all human transcripts into M GC% bins (e.g. 20 equal-width bins GC% [0, 1) and computed the mean entropy H in each bin. Fitting linear model
H + 3 GC% +
yielded B = 0.35 + 0.01 (SE), Pearson 0.98, R 0.97, p < 10-11, This near-perfect trend demonstrates that higher GC content known restrict codon choice to G/C-rich codons corresponds to lower angle uncertainty, validating that our embeddings capture mutational biases 44 34)
tRNA Abundance vs. Codon Angle Translational efficiency strongly influenced by IRNA gene copy number [19.. For each of the 61 sense codons let 0g be its learned angle (anchored so 0g E [0, 2T))), and let te be its normalized tRNA count. We assessed monotonic association via Spearman's rank correlation:
P = 1 (r.0)))) (612 1)
where r(.) denotes the rank. We observed p 0.66 (p < 10 10- indicating that codons with more abundant tRNAs are systematically placed at smaller angles. This alignment shows that the SO(2) embedding axis faithfully recapitulates the biological continuum of "optimal" to "non-optimal" codons 19 45)
Significance and Interpretation These results carry three key implications:
1. Biological alignment of latent space. The strong GC CC-ennropy trend (Pearson 0.98) confirms that the embedding's angular dispersion encodes mutational biases codon usage.
2. Translational efficiency signal. The negative Spearman p -0.69 substantiates that the continuous angular coordinate corresponds to IRNA-driven translation speed, without explicit supervision.
3. Enhanced interpretability. Constraining codon embeddings one-dimensional circle yields a latent space whose every point has direct biological meaning mutational constraint or translational optimality thereby increasing model transparency and trust.
Overall, these analyses justify our use of an SO(2) prior: it does not limit expressivity, but rather guides the model to learn the dominant forces shaping codon usage, producing a compact, interpretable representation amenable to downstream biological insights.

---

A.9 Experimental Setup
A.9.1 Pretraining Corpus Construction
All RefSeq files with the suffix ".rna.gbff.gz were downloaded via NCBI's FTP server. Using the Helical AI curation toolkit [49], we parsed each GenBank record, extracted 5'-UTR, CDS, and 3' 3'-UTR features, and retained only the CDS regions to ensure that our embeddings capture codon- level translational signals rather than untranslated flanking sequences. Sequences shorter than 20 codons or longer than 512 codons, or containing non non-canonical bases, were discarded.
Due to computational limits, randomly subsampled the cleaned set of approximately 56 million CDS entries down to 25 million, maintaining the following class proportions: 37.6% other vertebrates, 24.4% mammals, 22. invertebrates, 13.7% fungi, and 1.4% viruses (including major human pathogens such as SARS-CoV-2 influenza, RSV, HIV-1/2, HBV, HCV, HSV, EBV, VZV, Zika, and Dengue 1-4). For the low-data ablation, we sampled 1 million sequences from this pool with stratified sampling to preserve these ratios.
All retained coding sequences were then "codonized" by grouping every three nucleotides into single token, yielding a vocabulary of 64 codons. This pipeline, which includes data acquisition, filtering, codonization, and subsampling, ensures both biological relevance and computational tractability for our SO(2)- embedding experiments.
A.9.2 Downstream Evaluation
We evaluate our models on diverse biologically relevant datasets:
MLOS (Flu Vaccine mRNAs): Contains 543 sequences of influenza vaccine mRNAs
developed by Sanofi-Aventis The task focuses predicting the expression levels of these sequences, each approximately 1700 nucleotides long [23].
mRFP Expression: Includes 1459 synthetic mRNA sequences encoding red fluorescent protein (mRFP). Each sequence has a fixed length of 678 nucleotides, and the goal is to predict their expression levels quantitatively(2) [29].
E. coli Protein Expression: Consists of 6348 sequences from E. coli, varying from 171 to 3000 nucleotides. The classification task aims to determine whether a given mRNA sequence results in high or low protein expression. [10]
Comprises 355 riboswitch sequences ranging from 67 to 73 nucleotides. The regression task involves predicting the switching factor, a quantitative measure indicating regulatory activity [[3]
Te-Riboswitches:
iCodon Human mRNA Stability: A large-scale dataset of 41,123 human mRNA sequences with lengths ranging between 30 and 1497 nucleotides. The objective is predict mRNA stability as continuous value [9].
SARS-CoV-2 Vaccine Degradation: Contains 2400 sequences from SARS-CoV-2 vaccine candidates, each precisely 107 nucleotides. This regression task targets prediction of the degradation rate of vaccine mRNA molecules (48)
All datasets are split consistently into training, validation, and test subsets at ratios of 70%, 15%, and 15%, respectively, for model training and evaluation.
A.10
Hyperparameters
A.10.1
Pretraining Hyperparameters
All twelve SO(2) embedding variants were pretrained under an identical configuration to ensure fair comparison of their representational effects. We used a M-sequence subset for ablation, with a stratified 25 M corpus reserved for final scaling experiments. Training was run on a single node with eight NVIDIA GPUs, using a per-device batch size of 64 on GPUs and gradient accumulation over 2 steps (effective batch = 1024). Models were trained for 50 epochs, with a 5-epoch linear warmup and cosine Ir scheduler. Sequences were truncated or padded to a maximum length of 512 codons. We logged all runs to Weights&Biases and saved checkpoints to the specified output directory. Inputs
25

---

and defaults not overridden on the command line (e.g., learning rate, optimizer, scheduler) were left their parser-specified values. Table summarizss the key pretraining hyperparameters.
Table 4: Pretraining hyperparameters (identical for all 12 variants).
Hyperparameter
Value
Training file
1M Sample
Batch size
1024
Gradient accumulation steps
Epochs
50
Warmup epochs
Scheduler
Cosine
Max sequence length
512 codons
Precision
6f16-mixed
Embedding-laye variants
SO(2) configurations (fixedJcyclicffuzzy x standar//general)
Table 5: Pretraining hyperparameters for the 25 M-sequence corpus (identical for both GPT-2 and Mamba hybrid architecture).
Hyperparameter
Value
Training file
25 M Sample
Batch size
1024
Gradient accumulation steps Epochs
20
Warmup epochs
Scheduler
Cosine
Max sequence length
512 codons
Precision
bfl6-mixed
Embedding-hee layer variants
S02-fuzzy-generallttt
Downstream Hyperparameter Optimization To tune our downstream heads for both generative and property-rrediction tasks, we performed a structured grid search over key training parameters. We varied batch size {8, 16, 32, 64). learning rate E {1e-4, 1e-5), hidden-ratio E {2, 4, 8) (controls MLP width), and number of head layers E {2, 4, 7). We also swept over two finetuning regimes angle-only and last-two-layers- and five pooling strategies (so2 __aan, dft, polar, mean, lie_avg). Experiments used a fixed random seed (42), early stopping patience of 20 epochs, and scheduler patience of 10 on validation Spearman (regression) accuracy (classification). Invalid combinations (e.g., angle finetuning on fixed-8 models) were automatically pruned. The full search space and the selected best hyperparameters for each dataset reported in Appendix I..
A.10.2 Best Downstream Hyperparameters
To provide clarity on our tuning process, we present five tables ( through 12) that summarize the single best hyperparameter configuration across batch size, learning rate, MLP depth and width, finetuning regime, and pooling method for each downstream dataset. Each table lists the model variant (fixed, eyclic, or fuzzy), whether a Stiefel basis or equivariance loss was used, and the corresponding "Angle?" and "Last2?" flags. By consolidating these results, readers can directly see which configuration achieved the highest validation performance on each task without exhaustive per-dataset details.
26

---

Table 6: Downstream hyperparameter configurations for CDV Deg.
Model
Stiefel
Equiv.
#Layers
Batch
LR
Angle?
Last2?
Hidden
Pooling
Vanilla
4
8
0.0001
mean 8
Fixed 0
x
4
16
0.0001
4
so2_mean
Fixed A
x
4
0.0001
polar
Fixed 0
/
4
32
0.0001
4
lie avg
Fixed 0
/
4
le-05 16
X
4
so2 mean
Learned 0
X
4
32
0.0001
4
so2 mean
Learned 0
X
V
2
16
Ic-05
2
so2_ @cm_
Learned 0
V
4
16
Ic-05
2
so2_mean mcan
Learned 0
4
0.0001 32
2
mean
Fuzzy
x
2
32
0.0001
4
mean
Fuzzy
x
2
16
0.0001
4
mo2_mean
Fuzzy
4
0.0001
polar
Fuzzy
2
32
1e-05
2
dft
Table Downstream hyperparameter configurations for E. coli.
Model
Stiefel
Equiv.
#Layers
Batch
LR
Angle?
Last2?
Hidden
Pooling
Vanilla
2
8
Ic-05
4
mean
Fixed 0
x
x
2
8
0.0001
X
4
mcan
Fixed 0
x
/
4
16
le-05
X
2
dft
Fixed 0
/
2
8
le-05
4
so2,
Fixed 0
4
32
0.0001
polar
Learned 0
X
4
8
le-05
2
so2
Learned 0
X
2
0.0001
polar
Learned 0
4
1c-05
lie_avg
Learned 0
2
8
lc-05
mcan 4
Fuzzy
x
4
Ic-05
mcan dft
Fuzzy
x
2
8
le-05
2
Fuzzy
/
2
8
0.0001
4
polar
Fuzzy
2
lc-05 8
4
so2_ _ea_
A.11 Training Efficiency
To assess the computational overhead introduced by symmetry-awan embeddings, measured end-to-end wall-clock training times on both the 1M sequence subset and the full 25M sequence corpus. All models share the same 15M parameter Transformer backbone; variations arise only from the SO(2) embedding configuration (fixed vs. learned vs. fuzzy), choice of basis (standard vs. Stiefel), and inclusion of the equivariance loss.
Table 13 summarizes these results. On the 1M subset, the vanilla model completes in under whereas introducing fixed 0 embedding adds only modest 30 % overhead when equivariance loss used, and roughly doubles training time when the loss is activated. Learned 0 embeddings incur further cost up 4-5 slower than fixed, reflecting the extra parameterization needed for angle optimization. Stiefel-basis rotations amplify this effect, particularly once the equivariance penalty is applied. Fuzzy 0 variants exhibit the highest training times across all M experiments, as they combine a learned angle distribution with Riemannian updates on the Stiefel manifold.
Scaling to the full 25M corpus magnifies these trends: the hybrid fuzzy + Stiefel equivariance completes in just over one day, while the GPT-2 backbone requires nearly four days under the same configuration. These results highlight a clear trade-off between representational power and compute cost, guiding practical choices for large-scale pretraining under resource constraints.
backbone with
27

---

Table 8: Downstream hyperparameter configurations for MLOS Split Seed
Model
Stiefel
Equiv.
#Layers
Batch
LR
Angle?
Last2?
Hidden
Pooling
Vanilla
2
32
le-05
2
mean
Fixed 0
X
4
64
le-05
4
Fixed A
x
2
64
0.0001
so2_mase mo2_mean
Fixed 0
/
4
32
le-05
2
lie avg
Fixed 0
/
2
32
le-05
X
4
lie avg
Learned 0
X
8
64
0.0001
2
polar
Learned 0
X
V
2
64
0.0001
2
polar
Learned 0
V
2
8
0.0001
4
dft
Learned 0
4
32
0.0001
polar 4
Fuzzy
x
2
64
0.0001
2
so2 _e_m
Fuzzy
x
4
32
le-05
2
polar
Fuzzy
8
64
le-05
lie avg lie avg
Fuzzy
4
64
le-05
2
Table 9: Downstream hyperparameter configurations for MLOS Split Seed
Model
Stiefel
Equiv.
#Layers
Batch
Angle?
Last2?
Hidden
Pooling
Vanilla
2
32
0.0001
8
mean
Fixed 0
x
x
4
32
0.0001
X
4
dft
Fixed 0
x
/
4
32
0.0001
8
mean
Fixed 0
/
4
32
0.0001
4
lie_ lie_avg
Fixed 0
2
0.0001
so2_ _e2_
Learned 0
X
2
64
0.0001
V
4
Learned 0
x
2
le-05
mean
Learned 0
4
32
0.0001
4
so2_ so2_mcan
Learned 0
4
0.0001 32
2
lic_avg
Fuzzy
x
2
32
0.0001
dft
Fuzzy
x
2
32
0.0001
V
4
so2 __am
Fuzzy
2
32
0.0001
2
mean
Fuzzy
4
32
1e-05
2
mean
A.12
Ablation Study
To evaluate the effect of embedding design on predictive performance, we report results across three correlation-based metrics commonly used in biological sequence modeling: Spearman, Pearson, and coefficient of determination (R²). As shown Tables 14 15 and 16
A.13 Generation Experiments
This section presents the complete results of our generative evaluation, expanding upon the summary metrics reported in the main text. Table 17 rrpports Fréchet BioDistance (FBD) scores and generation diversity (IntDivge.) across SO(2)-based variants and sampling temperatures. Table 18 provides mean squared error (MSE) of predicted properties for generated suffixes temperature T.O. These results offer comprehensive view of fidelity and diversity trade-offs across embedding configurations and generation settings.
A.13.1 Fidelity
Table reports Fréchet BioDistance (FBD) scores and generation diversity (IntDivGGm) across SO(2) based variants and sampling temperatures.
28

---

Table 10: Downstream hyperparameter configurations for MLOS Split Seed 42.
Model
Stiefel
Equiv.
#Layers
Batch
LR
Angle?
Last2?
Hidden
Pooling
Vanilla
2
16
0.0001
4
mean
Fixed 0
x
4
32
0.0001
4
dft
Fixed A
x
2
lc-05
so2 mean
Fixed 0
/
4
16
0.0001
2
dft
Fixed 0
/
4
0.0001 8
X
2
dft
Learned 0
X
4
16
0.0001
2
dft
Learned 0
X
V
2
8
0.0001
4
dft
Learned 0
V
4
1c-05 8
2
dft so2_ so2_mean
Learned 0
2
le-05
4
Fuzzy
x
4
16
le-05
4
dft
Fuzzy
x
2
0.0001
2
dft
Fuzzy
0.0001
dft
Fuzzy
2
8
0.0001
2
dft
Table 11: Downstream hyperparameter configurations for mRFP.
Model
Stiefel
Equiv.
#Layers
Batch
LR
Angle?
Last2?
Hidden
Pooling
Vanilla
2
64
0.0001
mean
Fixed 0
x
2
8
0.0001
X
4
mean
Fixed 8
x
4
32
0.0001
polar
Fixed 0
2
0.0001
so2 mean
Fixed 0
4
0.0001 16
2
lie_avg
Learned 0
X
2
0.0001
mean
Learned 0
X
2
32
0.0001
4
dft
Learned 0
2
0.0001 8
4
polar
Learned 0
2
0.0001 32
4
polar
Fuzzy
x
X
4
32
0.0001
2
polar
Fuzzy
x
2
16
0.0001
4
dft
Fuzzy
V
2
16
0.0001
2
polar
Fuzzy
4
0.0001
dft
A.13.2
Property Retention
Table 18 provides mean squared error (MSE) of predicted properties for generated suffixes at temperature 1.0.
29

---

Table 12: Downstream hyperparameter configurations for Tc-Ribo...
Model
Stiefel
Equiv.
#Layers
Batch
LR
Angle?
Last2?
Hidden
Pooling
Vanilla
4
8
lc-05
2
mean
Fixed
x
4
le-05
x
2
mean
Fixed 0
X
/
2
16
0.0001
x
/
4
polar
Fixed 0
x
2
16
le-05
X
X
4
polar
Fixed
4
le-05
X
4
mean
Learned 0
x
2
0.0001
X
/
2
dft
Learned
/
2
0.0001
2
polar
x
Learned
4
16
Ic-05
4
polar
Learned 0
4
le-05
x
2
polar
Fuzzy 0
X
4
8
le-05
X
X
4
mean
Fuzzy 0
4
16
le-05
X
2
mean
Fuzzy
X
0.0001
polar
Fuzzy
0.0001 4
4
mean
Table 13: Training times for different SO(2)-based models.
Model
Stiefel
Equiv.
Training Time
Vanilla
1h51 m
X
2h m
X
/
4 06 m
Fixed 0
X
h5 m
V
9 45 m
X
4h 15 m
X
6h.55 m
Learned 0 (cyclic)
X
9h m
11 m
4h 18 m
/
X
7h 12 m
Fuzzy 0
X
9h m
/
11h41 m
25M 35M-sequence corpus
EQUI-MRNA (5M) (Mamba-Transfome Transformer Hybrid)
d m
EQUI-MRNA (15M) (GPT-2)
3d 18l1hh2 24
30

---

Table 14; Comparison of vanilla, fixed, and learned 5O(2)-based models across downstream datasets, evaluated using Spearman correlation. Vindicates the presence of component (Stiefel basis, or equivariant loss) in configuration.
Model
Sticfel
Equiv.
E. coli(A)
MLOS(S)
Tc-Ribo.(S)
mRFP(S)
COV Deg(S)
Vanilla
0.580
0.633 + 0.14
0.698
0.797
0.779
X
x
0.615
0.602 + 0.11
0.724
0.834
0.783
Fixed
X
/
0.600
0.633 0.16
0,711
0.841
0.787
x
0.602
0.667 + 0.19
0.688
0.840
0.803
0.592
0.571 + 0.086
0.722
0.838
0.794
X
0.602
0.648 0.16
0.672
0.833
0.799
Learned
X
0.589
0.698 0.15
0.698
0.822
0.787
0.633
0.657 0.10
0.701
0.871
0.790
0.612
0.667 = 0.137
0.741
0,850
0.797
X
x
0.603
0.666 + 0.17
0.680
0.833
0.786
Fuzzy 0
x
/
0.590
0.602 + 0.098
0.682
0.837
0.805
0.619
0.669 0.10
0.729
0.839
0.793
0.605
0.691 0.14
0.736
0.844
0.820
Table 15: Comparison of vanilla, fixed, and learned SO(2)-based models across downstream datasets, evaluated using Pearson correlation. Vindicates the presence of component (Stiefel basis, or cquivariant loss) in configuration.
Model
Stiefel
Equiv.
E. coli(AUC)
MI.OS(P)
Tc-Ribo.(P)
mRFP(P)
COV Deg(P)
Vanilla
0.4335
0.6203 + 0.157
0.6451
0.6875
0.8081
x
0.4290
0.6438 + 0.164
0.6503
0.7231
0.8249
Fixed
0.4768
0.6508 + 0. 0.140
0.6245
0.7162
0.8298
x
0.5934
0.7270 = 0.122
0.6670
0.6979
0.8276
0.6032
0.6579 + 0.089
0.6216
0.6959
0.7939
x
0.4600
0.6412 + 0.139
0.6086
0.7291
0.8252
Learned 0
0.4550
0.6336 + 0.139
0.6183
0.6911
0.8110
x
0.4611
0.6212 = 0.143
0.6463
0.7121
0.8166
0.4749
0.6659 + 0.175
0.6294
0.7071
0.8126
x
0.4549
0.6684 + 0.183
0.6036
0.7307
0.8181
Fuzzy 0
X
/
0.6469
0.6984 + 0.049
0.5910
0.7027
0.8281
x
0.5052
0.6704 + 0.147
0.6194
0.7306
0.8087
0.5452
0.6983 + 0.145
0.6107
0.6975
0.8265
31

---

Table 16: Comparison of vamilla, fixed, and learned SO(2)-based models across downstream datasets, evaluated using coefficient of determination (Rt²). Vindicates the presence of component (Stiefel basis, or equivariant loss) in a configuration.
Model
Stiefel
Equiv.
E. coli(F1)
MLOS(A")
Tc-Ribo.(It")
mRFP(H²))
COV Deg(It²)
Vanilla
0,4763
0.4037 + 0.196
0.2240
0.8028
0.6136
x
0.5340
0.3339 + 0.136
0.2398
0.8314
0.6190
Fixed
d
0,5158
0,3345 + 0.133
0,2467
0,8232
0.6413
x
0,5273
0.3339 + 0.139
0.2506
0.8140
0.6349
0,5133
0,3193 + 0.101
0,2616
0.7977
0.6007
x
0.5211
0.3573 + 0.153
0.2446
0.7788
0.6282
Learned 0
/
0.5036
0.2945 + 0.121
0.2549
0.8038
0.6160
x
0.5024
0.3413 0.134
0,2551
0.8034
0.6215
0.5153
0.3489 0.122
0.2438
0.7931
0.5941
x
0.5265
0.3715 + 0.162
0.2373
0.7908
0.6401
Fuzzy 0
/
0.4933
0.3062 + 0.134
0.2142
0.7851
0.6569
X
0.5101
0.3591 + 0.159
0.2451
0.7572
0.6210
0.4940
0.3320 +.2 0.157
0.2448
0.7799
0.6210
32

---

Table 17: Generation metrics and diversity SanDiv_Gen) for SO(2)-based variants at different sampling temperatures.
Model
Temp
Stiefel
Equiv.
FBD
Prec.
Rec.
F1
IntDiv_Ged
Vanilla
0.2
2561.8 81
0.093
0. 108
0.0999
929.13
2563.64
Fixod
0.2
0.2
/
x
2562.60
2565.27
0.093
0.093
0.104
0.108
0.0982
0.099
929.10
929.08
2563.44
0.1001
929.10
0.2
2565.05
0.093
0.107
0.0995
929.10
Learndd Ø
0.2
X
2563.29
0.094
0.107
0.1001
929.12
0.1
2565,23
0.093
0.109
0.1004
929.11
2563.05
929.10
0.2
X
2564.17
0.093
0.104
0.0982
929.11
0
0.2
2564,33
0.093
0.104
0.0982
929.11
2562.88
0.2
2563.90
0.093
0.103
0.0977
929.12
EQUI-MRNA (5M)
0.2
819,18
0.680
0,745
0.7110
1045.95
EQUI-MRNA (25M)
965.27
Vanilla
0.4
2361 .99
0.093
0.108
0.099
929.15
0.4
2564.09
0.094
0.107
0.1001
929.12
2564.21
Fixed
0.4
2562..99
0.093
0.109
0.1004
929.12 12
0.4
2562.72
0.093
0. 108
0.0999
929.12
Learned
0.4
2562.55
2564.46
0.093
0.094
0. 0.100
0.108
0.1005
0.10044
929.15
929.1 14
2562.82
0.4
2564.21
0.093
0.108
0.0999
929.12
Fuzzy 0
0.1
x
2563..6
0.093
0.106
0.0991
929.13
0.4
2562.74
0.093
0.108
0.0999
929.12
2562.79
0.093
929.13
EqUI-MRNA (5M)
0.4
/
579.21
0.726
0.777
0.7506
1049.83
EQUI-MRNA (25M)
0.1
310.40
0.883
0.877
0,8800
984.58
2562.24
0.6
x
2564.36
0.094
0.107
0.1001
929.13
Fixed
0.6
2563.80
0.093
0.110
0.1008
929.13
0,1008
929.13
0.6
2062.68
0.120
0.134
0.1266
936.00
0.6
2563.25
0.093
0.108
0.0099
929.15
Learned
0.6
/
2564.53
0.093
0.109
0.1004
929,1 15
0.6
/
2562.71
0.093
0. 108
0.0999
929.13
Huzzy 0
0.6
/
2462.77
2563.15
0.093
0.099
0.116
0.108
0,1068
0.0999
929, 12
930.43
2562.95
0.093
0.111
0.1012
929.14
EQUI-MRNA (SM)
/
341.65 68
0.791
0.8038
1049.53
EQUI-MRNA (25M)
0.6
196.15
0.959
0.954
0.9565
1001.13
Vanilla
0.8
2562.67
0.093
0.1016
929.15
0.8
2561.74
0.094
0.107
0.1001
929.14
Fixed
0.8
1714.65
0.142
0.145
0.1435
943.15
0.8
2562,17
0.093
0,110
0.1008
929.15
1039.52
954.47
0.8
X
2564.03
0.093
0.107
0.0995
929 15
Leamedd
0.8
1525.08
0.147
0.152
0.1495
945.01
0.8
2562.60
0.093
0.108
0.0999
929.14
088
2374,21
0.097
0.122
0.1081
930.52
Fuzzy 0
<
2562.80
0.093
0.108
0.0999
929.12 12
0.8
2562.93
0.093
0.111
0.1012
929.15
EQUI-MRNA EQUI-MRNA (25M)
/
0.8
172.02
0.977
0.973
0.9750
1009.09
Vanilla
1.0
2562.78
0.093
0.112
0.1016
929.16
2564.44
Fixed
1.0
2562.25
961.80
0.093
0.209
0.110
0.215
0.2120
0.1008
959.16 16
929.15
653.14
0.3525
969.65
1516.35
0.138
0.144
0.1409
941.34
Leamnd Ø
1.0
1921.42
958.41
0,132
0.204
0.143
0.212
0.1373
0.2079
958. 78
940.06
1807.64
1.0
X
1520.08
0.127
0.150
0.1375
940.55
0
$80.19
0.369
0.367
0.3680
969 57
1652.77
0.136
0.156
0.1453
941.85
EQUI-MRNA (5M)
76.13
0.977
0.976
0.9765
1051.16
1014.56
33

---

Table 18: MSE of predicted property across model variants and datasets at sampling temperature 1.0.
Model
Dataset
Stiefel
MSE Pred. Equiv.
Vanilla
icodon mlosO
0.4640
0.1894
mrfp
1.8885
switch deg
0.4141
0.7868
Fixed 9
icodon
0.4649
mlosO
0.1888
mrlp
X
X
1.8485
switch
X
X
0.4153
deg
0.7921
Fixed 9
icodon
X
0.4346
mlosO)
0.1597
mrlp
1.7898
switch
0,4278
deg
0.6716
Fixed 8
icodon
0.4640
mlos)
0.1888
mrlp
switch
0.4069
deg
0,8017
Fixed 8
icodon
0.4252
mlnsO mrfp
0.1426
1,7786
switch deg
0.6215
Learned
icodon mlosf) mrfp
0.4470
0.1684
1.8193
switch deg
0.4221
0.7086
Learned icodon
mloso mrfp
0.1695
1.7806
switch deg
0.3623
0.6733
Learned
icodon mlosD
0.4509
0.1791
mrfp
1.8256
switch deg
0.4155
0.7357
Learned
icodon mlosO mrfp
0.4498
0.1776
1.8216
switch deg
0.4091
0.7523
Fuzzy
icodon
0.4474
mlosO
0.1642
mrfp
1,8140
switch deg
0.3992
0.7291
Fuzzy
icodon mlosO
0.1361
mrfp
1.7620
switch deg
X
0.3872
Fuzzy
icodon mlosO
0.4582
0.1786
mrlp
1.8332
switch
X
0.4122
dog
0.7707
Fuzzy 0
icodon
0.4497
mlos)
0.1758
mrlp switch
0.4171
deg
0.7149
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction section carefully convey the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mentioned all limitations and future directions in section 4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

We mentioned all the assumptions and proofs in the methodology (Section and the appendix A.6
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All the required details for reproducing this model have been provided.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification: The data are publicly available. We can provide the link to code at any time. For the review version, we did not include the link keep an anonymous review. The supplementary Zip file including code has been attached.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
httpl//guuddes/odeeuubmi.i www.:oubbssssooooioooooooo.... for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.c//pppp
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer section 3.1 and A.10
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We used common metries from the literature that were used for evaluation and comparison. Also, we compared the performances as fair as possible. The significance of our work can be easily interpreted with provided tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to section 3.1 and A.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:///hwaaaa.sccccppppc//tctcc
Answer: [Yes]
Justification:
No possible violation exists in this study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
This work focuses on foundational research for improving language modeling of biological sequences, specifically mRNA. The model is designed to capture codon-level symmetries for better sequence representation and downstream prediction performance. As it is methodological contribution without direct application or deployment pathways at this stage, there are no immediate societal impacts, either positive or negative, associated with this work.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
38

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper proposes methodological advancement for modeling biological sequences, specifically mRNA, using symmetry-aware architectures. The work does not involve the release of general purpose language models, image generators, scraped from the internet. Furthermore, the developed models are specialized for biological sequence analysis and do not pose high risks for misuse dual-use beyond intended scientific applications.
datasets
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us -sal-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: datasets).
All the assets in this work, either are original publicly available (e.g.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
39

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The link to the Github repository including code and datasets will be provided after acceptance, Due to anonymous review, we cannot share the link beforehand. The supplementary Zip file including code has been attached.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
40

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
41


Reviews:
[{"review_id": "3zmMm50Iwj", "reviewer": "Reviewer_uj9Q", "summary": "This manuscript introduces Equi-mRNA, a novel language model for messenger RNA (mRNA) sequences that operates at the codon level.\n\nThe core contribution is a new embedding framework that explicitly models the symmetries inherent in the genetic code, where multiple codons (synonymous codons) encode the same amino acid. The authors represent these synonymous relationships using group theory, specifically by mapping codons within a synonym set to rotations in the 2D special orthogonal group, SO(2). This creates a structured, equivariant representation where synonymous codon substitutions correspond to predictable geometric transformations. The paper proposes several variations of this idea, including fixed, learned, and \"fuzzy\" (distributional) rotation angles, as well as extending rotations to learned 2D subspaces within a higher-dimensional embedding space using the Stiefel manifold. To enforce these symmetries throughout the model, an auxiliary equivariance loss and symmetry-aware pooling mechanisms are introduced.", "questions": "Although tasks such as vaccine degradation is about mRNA sequences, currently most methods are using ncRNA models on them. Given that the proposed method converts the nucleotides to SO(2), is it possible to apply the proposed method on the tasks requiring single-nucleotide resolution?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n### Novel and Principled Approach\n\nThe core idea of using group theory, specifically SO(2) rotations, to model the synonymous codon relationships is highly original and biologically principled. It moves beyond simply treating codons as unrelated tokens or using simple clustering losses, embedding a fundamental biological symmetry directly into the model's architecture. This is a significant conceptual contribution to genomic language modeling.\n\n### Comprehensive Experimental Validation\n\nThe paper's empirical evaluation is thorough and convincing. The authors perform a detailed ablation study across 12 different model variants to dissect the contribution of each component (e.g., learned angles, Stiefel basis, equivariance loss). They test on a diverse set of six downstream tasks, covering expression, stability, and regulatory function. \n\n## Weakness\n\n### Lack of Comparison with Existing mRNA foundation models\n\nSeveral foundation models pre-trained on mRNA sequences have been established in the field of RNA research. This includes [CaLM](https://huggingface.co/multimolecule/calm), [mRNA-FM](https://huggingface.co/multimolecule/mrnafm), and [AIDO.RNA cds variants](https://huggingface.co/multimolecule/aido.rna-1.6b-cds). \nComparing with these methods would significantly improve the validation of the proposed method. \n\nSince a more detailed comparison with other methods is omission, I will not be able to grant a higher score.", "was_revised": true, "final_justification": "> Why 5? \n\nThe proposed method applies group-theoretic principles, and achieves consistent improvements. \n\n> Why not a higher score? \n\nWhile the proposed method should definitely be accepted, but it is not yet award quality. \n\n> Why not a lower score? \n\nNo major limitations are identified.", "rebuttal": "We thank the reviewer for their positive feedback and for recognizing the originality and biological motivation of our approach. We appreciate the acknowledgement of the theoretical contributions, the thorough ablation studies, and the broad applicability of Equi-mRNA across expression, stability, and regulation tasks. Below, we respond point-by-point to the specific concerns raised.\n ## Weakness\n `Lack of Comparison with Existing mRNA foundation models...`\n \nWe have now implemented and evaluated CaLM [1], mRNA-FM [2], and AIDO.RNA [3] (cds variant) as baselines across the same downstream tasks. Each baseline was tuned using a comprehensive grid search, identical to the one used for Equi-mRNA, to ensure a fair comparison.\nWe do note, however, that CaLM was not pre-trained on mRNA sequences, and AIDO.RNA was pre-trained primarily on non-coding RNA (ncRNA). As such, their application to coding sequence tasks may not fully reflect their intended use cases. Additionally, these models are 5–106× larger in parameter count compared to Equi-mRNA, which was deliberately designed to be lightweight and interpretable. Despite this, Equi-mRNA achieves competitive or superior performance, while offering biological interpretability and symmetry-aware modeling.\nFull results are included below. We believe these comparisons reinforce the practical and scientific value of our approach.\n\n**Table: Performance comparison across downstream benchmarks**\n\n| Model | E. coli(A) | MLOS(S) | iCodon(S) | Tc-Ribo.(S) | mRFP(S) | COV Deg(S) |\n|---|---|---|---|---|---|---|\n| **Nucleotide-Based** | | | | | | |\n| RNA-FM [2] (99.5 M) | 0.43 | - | 0.34 | 0.58 | 0.80 | 0.74 |\n| RNABERT [4] (82 M) | 0.39 | - | 0.16 | 0.47 | 0.40 | 0.64 |\n| Aido mRNA [3] (1.6B) | - | 0.461 ± 0.102   | 0.472 | 0.586  | 0.537  | *0.813* |\n| CaLM (86M) [1] | - | 0.430 ± 0.170 | 0.376 | 0.625  | 0.546| 0.773 |\n| mRNA-FM [2] (239M) | - | 0.509 ± 0.154  | 0.458 | 0.690 | 0.564  | 0.714  |\n| **Codon-Based** | | | | | | |\n| CodonBert [5] (82 M) | 0.57 | 0.543 | 0.350 | 0.502 | 0.832 | 0.78 |\n| GPT2 [6] (CLM)(50M)* | - | 0.611 | 0.498 | 0.531 | 0.815 | 0.787 |\n| GPT2 [6] (MLM)(50M)* | - | 0.653 | 0.503 | 0.569 | 0.753 | 0.801 |\n| HELM [6] (CLM)(50M)* | - | 0.592 | *0.529* | 0.619 | 0.849 | 0.789 |\n| HELM [6] (MLM)(50M)* | - | 0.701 | 0.525 | 0.626 | 0.822 | **0.833**|\n| **Equi-mRNA (5M)†** | *0.581* | *0.705 ± 0.12* | 0.519 | **0.764** | *0.853* | 0.756 |\n| **Equi-mRNA (15M)‡** | **0.613** | **0.710 ± 0.13** | **0.537** | *0.737* | **0.855** | 0.791 |\n\n† Mamba-based hybrid Architecture; ‡ GPT-2 Architecture both trained on 25M datapoints; *Trained on Antibody mRNA sequences\n\n\n## Question\n`Although tasks such as vaccine degradation is about mRNA sequences...`\n\nWe agree that certain tasks, such as vaccine degradation prediction, require single-nucleotide resolution, and that many existing models in this space are trained on non-coding RNA (ncRNA).\nEqui-mRNA is designed to model coding sequences (CDS) at the codon level, leveraging SO(2) rotations to capture the biological symmetry among synonymous codons. This structure is well-suited for translation-related tasks, but we acknowledge that it does not currently support per-nucleotide granularity, which is important for structure-sensitive or decay-related predictions.\nAdapting Equi-mRNA to handle such tasks would involve meaningful architectural changes. Specifically, it would require incorporating nucleotide-level embeddings and modifying the input resolution. As an initial direction, we are exploring a hybrid variant in which:\n - Codon-level SO(2) embeddings are retained in the CDS;\n - Nucleotide-level embeddings are used for UTR and regulatory regions;\n - A shared transformer or attention-based architecture integrates both representations.\n\nThis design would allow Equi-mRNA to maintain symmetry-aware modeling while extending its resolution and scope. Given the additional complexity, we have included this extension as part of our future work, and see it as a natural and exciting progression of the framework.\n\n### References\n1. Outeiral, Carlos, and Charlotte M. Deane. \"Codon language embeddings provide strong signals for use in protein engineering.\" Nature Machine Intelligence 6.2 (2024): 170-179.\n2. Chen, Jiayang, et al. \"Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function predictions.\" arXiv preprint arXiv:2204.00300 (2022).\n3. Zou, Shuxian, et al. \"A large-scale foundation model for rna function and structure prediction.\" bioRxiv (2024): 2024-11.\n4. Kalicki, Colin Hall, et al. \"RNAbert: RNA family classification and secondary structure prediction with BERT pretrained on RNA sequences.\" 2020.\n5. Li, Sizhen, et al. \"Codonbert: Large language models for mrna design and optimization.\" bioRxiv (2023): 2023-09.\n6. Yazdani-Jahromi, Mehdi, et al. \"Helm: Hierarchical encoding for mrna language modeling.\" arXiv preprint arXiv:2410.12459 (2024)."}, {"review_id": "toOEMSq1Am", "reviewer": "Reviewer_NetF", "summary": "The paper introduces Equi-mRNA, an mRNA language model that explicitly encodes synonymous codon relationships as cyclic SO(2) rotations in the embedding space. It implements three rotation schemes—fixed, learned, and fuzzy—together with an equivariance loss and SO(2)-aware pooling to enforce these biological symmetries. Across six downstream tasks, Equi-mRNA outperforms vanilla baselines. The interpretability analyses show that the learned rotation angles correlate strongly with GC content and tRNA abundance", "questions": "What biological justification underlies modeling synonymous codons as planar rotations? Have alternative group structures been considered?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n- Applying group theory, specifically SO(2) equivariance, to model the synonymous codon relationships is a novel and theoretically principled approach. \n- The proposed Equi-mRNA model shows consistent performance improvements across most downstream tasks.\n- The interpretability analysis is strong: linking learned geometric representations (codon angles) to known biological features (GC content, tRNA abundance) suggests the model captures biologically meaningful patterns. \n\nWeaknesses\n\nMajor\n- The model treats all synonymous codons for a given amino acid as one uniform cyclic group, which might be an oversimplification. For example, Serine’s six codons split into two distinct families, UCN (UCU, UCC, UCA, UCG) and AGY (AGU, AGC), that are not interconverted by single-nucleotide mutations. Forcing them into a single rotation cycle may impose a geometric constraint at odds with real genetic and evolutionary relationships.\n\nMinor\n- Line 315: an significant leap -> a significant leap\n- Line 367: Stiefel manifold rotations enhances -> Stiefel manifold rotations enhance", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "We thank the reviewer for their thoughtful evaluation and for recognizing the novelty of our group-theoretic framework, the consistent empirical performance of Equi-mRNA, and the strength of our biological interpretability analyses. We are especially grateful for the evolutionarily motivated suggestion regarding codon subgrouping, which directly led to a meaningful improvement in our model. We address the reviewer’s comments and questions point-by-point below:\n## Weakness\n### Major\n`The model treats all synonymous codons for a given amino acid as one uniform cyclic group ...`\n\nWe agree that grouping all synonymous codons under a single cyclic rotation can oversimplify cases where codons form biologically distinct families, such as the Serine codon set (UCN vs. AGY). Based on your observation, we modified the Equi-mRNA embedding framework to assign separate base vectors for each codon subgroup in amino acids with disjoint families. Specifically, we created independent cyclic groups for subfamilies that are not interconvertible by single-nucleotide substitutions, thereby relaxing the shared rotation constraint in such cases.\nThis modification resulted in a significant improvement in model performance, confirming the value of incorporating fine-grained biological structure into the geometric encoding. We are grateful for this insightful suggestion, which led to a meaningful enhancement of our framework. The updated results are shown in the table below.\n\n Model | E. coli(A) | MLOS(S) | Tc-Ribo.(S) | mRFP(S) | COV Deg(S) | Improvement |\n|---|---|---|---|---|---|---|\nVanilla    |  0.580   |  0.633 ±0.14   |  0.698   |  0.797   |  0.779  |  -  |\nEqui-mRNA (AA Groups)    |  *0.605*  | 0.691 ±0.14   | 0.736  |  0.844 |  **0.820**  |  6.00%  |\nEqui-mRNA (AA SubGroups)|  **0.614**  |  **0.697 ± 0.18**  | **0.761** |   **0.861**   |  *0.804*  |  **7.17%**  |\n\nThis adjustment demonstrates that incorporating biologically informed codon subgrouping improves the expressiveness and accuracy of the model. We have updated the manuscript to reflect this change and cited this as a direct response to your feedback.\n\n### Minor\nWe appreciate the careful reading. All such issues have been addressed in the updated manuscript.\n\n## Question\n`What biological justification underlies ...`\n\nModeling synonymous codons as planar rotations is biologically grounded in the fact that codons encoding the same amino acid are functionally interchangeable at the level of protein sequence. However, these synonymous codons are not truly identical in function: they exhibit biases in translation efficiency, mRNA stability, and codon usage preference. Our approach uses planar SO(2) rotations to capture the symmetric structure of synonymous codon sets, while still allowing the model to learn codon-specific preferences through the rotation angles.\nThe SO(2) group provides a natural way to embed codons equivariantly; That is, codons belonging to the same amino acid class are embedded in a way that preserves their relative structure but allows for smooth, learnable variation across the set. This enforces structured parameter sharing and introduces a biologically consistent inductive bias, especially important when training data is limited or biased.\nWe also explored alternative group structures, such as discrete cyclic groups (Cₖ) using frequency-domain embeddings (via blockwise discrete Fourier transforms). This shift-equivariant formulation preserves cyclic symmetry without relying on a geometric manifold. However, in practice, these alternatives underperformed and led to training instability, likely due to the non-local nature of frequency space and sensitivity in optimization. As a result, we focus on SO(2)-based geometric representations, which offered a better trade-off between biological plausibility, interpretability, and empirical performance."}, {"review_id": "TWH7yYb6zA", "reviewer": "Reviewer_jRL6", "summary": "The paper introduces Equi-mRNA, a novel mRNA language model with symmetric codon-level embeddings. The model leverages the SO(2) group to encode synonymous codon relationships in the latent space. This new embedding method helps to improve the biological fidelity and performance of mRNA sequence modeling. The authors demonstrate that the Equi-mRNA model outperforms baseline models in multiple downstream tasks including property prediction tasks and sequence generation. They also show that the trained model angles variability correlates with GC and tRNA availability, which is a good sanity check given the model’s training.", "questions": "The main question that came into our mind when reading this paper is simply \"why go through all this trouble?\" Yes, you can do this, and it's probably very satisfying to make such a sophisticated mathematical construction that appears to work, but it seems a lot of the sophistication comes to solve issues that can easily be addressed by much simpler embeddings such as softmax function over equivalent codons and adding priors on those (e.g. preference for GC or based on codon availability) which can also be learned. \n\nAnother question is how do they think this will combine with UTR elements?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**Strengths:** \n- This paper is well-written with structured flow and solid theoretical inductions. \n- The use of group-theoretic principles to model codon symmetries is a novel and theoretically grounded approach. This method provides a structured way to incorporate biological knowledge into language models.\n- The paper introduced various design of embedding methods, which incorporate fixed or learnable rotation angles to adapt to different biological circumstances. \n- The model shows significant improvements over baseline models in various tasks, including expression prediction and sequence generation. It reported significant gains in accuracy and generative fidelity.\n- The model's ability to reflect known biological patterns, such as GC-content biases and tRNA is a good sanity check. This interpretability is crucial for applications in synthetic biology and mRNA therapeutics.\n- The paper includes a thorough evaluation across multiple datasets and tasks, providing a robust assessment of the model's capabilities.\n\n**Weaknesses:**\n- The introduction of group-theoretic embeddings and the associated computational increment may limit the model's scalability. \n- In the ablation study section, equivariance loss harms the performance of the model in multiple settings across different tasks. The results are suspicious since it contradicts the principle of the paper to design a symmetric embedding.\n- Evaluations and very limited as well the tests against other models. Many of the tests, as the authors admit, are problematic (e.g. no/fixed UTRs).\n- Moreover, some tests are not adaquetly described and their limitations not discussed. For example, the authors use iCodon and claim it's for over 40K human mRNA but the work seems to be based on 1-3K (depending on how you count) sequences in zebrafish used to select their setting for a genetic algorithm for their measure of optimility. We are guessing this was then used to predict for human mRNA? seems a problematic test and not correctly described. Another example is their mRFP. This one is based on ~1.5 codon variants *of the same gene with a fixed UTR*. The authors of this work get great performance (close to 0.8 correlation) just from fitting a random forest on the first 8AA. Yet, this is a test for a 25M parameter model.... It would have been good if the authors at least took the time to look into and acknowledge the how limited these evaluations actually are beyond a general comment at the end.\n- Related to the above it's unclear how train/test are done to prevent information leakage between similar samples. \n- The model is unable to incorporate UTRs, which is a major limitation for usability for the tasks they envision.\n\nIn summary, while we have significant concerns about the evaluations, the comparisons against other models and the usability of this model we believe there is potential (tbd) and the mathematical formulation itself should be of interest to NeurIPS audience.", "was_revised": true, "final_justification": "We read the response and appriciate the additional details provided by the authors, these help answer some of our conerns. We did not see major issues with the paper (hence the original positive score) and concluded no major changes needed to our evaluation after the authors clarified many details. We note that as pointed by us and other reviewers the actual usability of the model is unclear beyond the technical innovation. In response to critisim about the validity of the tests by us and others the authors main point is that this is what was done before and allows comparison to other works. Sure, that's true for comparing ML conference papers, but if the evaluations appear flawed/limited this kind of answer does not alleviate the associated concerns.", "rebuttal": "We thank the reviewer for the thoughtful feedback and recognition of our work’s novelty and clarity, as well as the constructive suggestions on evaluation, modeling, and biological relevance.\n## Weakness\n`The introduction of...`\n\nIt is true that the introduction of group-theoretic embeddings and the equivariance loss adds some computational overhead during training. However, we would like to clarify that the equivariance loss is used only during training to regularize the embedding space; it does not affect inference time or model deployment efficiency.\nThis trade-off between inductive bias and computational cost is common in geometric deep learning, where methods are often not yet optimized for modern accelerators (e.g., GPU/TPU). While our current implementation does incur some training-time overhead, we expect that future versions (leveraging sparse structures or low-rank parametrizations) can reduce this cost significantly. In the meantime, we believe the biological interpretability and improved sample efficiency of symmetry-aware embeddings provide a meaningful justification for this modest overhead.\n\n`In the ablation...`\n\nThe equivariance loss is designed to encourage the model to preserve the SO(2)-based symmetry in the codon embedding space, which reflects the core principle of our framework. However, as the reviewer observed, in some configurations the addition of this loss leads to a modest drop in downstream performance.\nWe believe this is due to two factors:\n(1) In certain tasks, enforcing strict symmetry may conflict with biological asymmetries, such as codon usage bias, tRNA availability, or context-specific effects; and\n(2) More importantly, in some of the simpler or lower-capacity architectures, the model may be too constrained to reconcile the equivariance objective with the task loss. In such cases, the symmetry constraint may dominate optimization and reduce flexibility. We found that in richer architectures (e.g., those with learned subspaces or multi-angle embeddings), the equivariance loss is more stable and often beneficial.\n\n`Evaluations and very limited...`\n\nSince our model is designed to operate on coding sequences (CDS) where codon redundancy is relevant, it was trained without UTRs. Therefore, it is natural and appropriate to evaluate it on tasks where UTRs are fixed or absent. These benchmarks are standard and used by models such as [1], [2], and [3], enabling fair comparison.\nWhile we agree that including models such as mRNA-FM [4], CaLM [5], and AIDO [6] would strengthen the comparison, we have now added these baselines to provide a more complete evaluation. These results are presented in our response to Reviewer 1 (uj9Q).\n\n`Moreover, some tests are not adaquetly...`\n\n**Use of iCodon for Evaluation**\nWe appreciate the opportunity to clarify our use of iCodon. In our experiments, we use the *pretrained* iCodon model as a black-box evaluator to compute codon optimization scores for over 40,000 human mRNA sequences. While the original iCodon model was trained using zebrafish data, it has been adopted in several recent studies (e.g., [1], [2], [5]) to evaluate codon optimality in human sequences due to its ability to capture species-agnostic codon usage patterns. We emphasize that iCodon is not used for training or optimization. Our approach is consistent with common evaluation protocols and serves to support comparability with prior work.\n\n**mRFP Benchmark and Model Performance**\nWe acknowledge that the mRFP dataset is based on synonymous variants of a single gene with a fixed UTR. However, prior studies note that predicting expression levels from codon sequences remains non-trivial. A random forest on the first 8 amino acids yields a Spearman correlation of ~0.73, our model achieves a correlation of **0.855**, demonstrating improved performance in capturing subtle codon-level signals. For additional comparison, we include a shallow one-hot + TextCNN baseline which achieves **0.763**, underscoring the benefit of incorporating codon-aware and symmetry-informed modeling. We view this dataset as a valid benchmark to evaluate generalization over synonymous variation, and our results provide evidence that Equi-mRNA meaningfully improves over prior methods.\n\n**Table: mRFP Benchmark Performance Comparison**\n\n| Model                          | Spearman Corr. |\n|-------------------------------|----------------|\n| One-hot + Lasso (8AA)         | 0.732          |\n| One-hot + Random Forest (8AA) | 0.710          |\n| One-hot Codon + Text CNN      | 0.763          |\n| **Equi-mRNA (Ours)**         | **0.855**      |\n\n`Related to the above it's unclear...`\n\nTo ensure fair and reproducible evaluation, we use the standard train, validation, and test splits as provided by the original benchmarks or accompanying repositories. We avoid custom splits to follow widely accepted protocols. Specifically, for datasets such as mRFP, and the iCodon-based optimization task, we adopt the same partitioning strategies used in [1], [2], [3]. These practices are consistent with the literature and support direct comparison with existing state-of-the-art approaches. We will revise the manuscript to clarify this and ensure readers are informed of our evaluation setup.\n\n`The model is unable to incorporate UTRs...`\n\nIt is true that the current version of Equi-mRNA does not explicitly incorporate untranslated regions (UTRs). Our primary focus in this work was to develop and evaluate codon-level representations that respect the algebraic and biological structure of the coding sequence (CDS), particularly for tasks where synonymous variation in the CDS is known to be the primary determinant of translation efficiency.\nThat said, we fully agree that UTRs play a significant role in post-transcriptional regulation, including translation initiation and mRNA stability. Integrating UTR information is a logical and necessary extension of our framework, particularly for tasks such as whole-transcript translation rate prediction or half-life estimation.\nTo this end, our model is modular and can readily support UTR input by adding a separate embedding path for UTRs, which do not follow codon semantics. In ongoing work, we are exploring hybrid models that combine codon-level embeddings for CDS with nucleotide or convolutional embeddings for UTRs. We believe that combining these modalities will further improve generalization for transcriptome-scale prediction tasks.\n\n## Questions\n`The main question...`\n\nWhile we understand the instinct to prefer simpler solutions such as using a softmax distribution over equivalent codons with biologically motivated priors (e.g., GC content, codon frequency), this specific approach has already been explored and benchmarked in HELM [2], where it consistently underperformed compared to even the vanilla model without explicit symmetry constraints.\nThese results suggest such heuristics fail to capture dependencies across codon sequences, even within synonymous groups. In contrast, Equi-mRNA provides a principled, group-theoretic embedding framework that allows codon-level symmetries to be preserved while still learning context-specific, task-driven patterns.\nImportantly, the benefits of our approach are most apparent in low-resource regimes, which are common in mRNA applications. For example, in the MLOS benchmark, where labeled data is extremely limited, Equi-mRNA significantly outperforms the vanilla model. We also performed additional experiments by subsampling 500 sequences from benchmark datasets. Across all scenarios, Equi-mRNA demonstrates superior performance and stability, reinforcing the value of embedding biologically motivated symmetries when data is scarce.\nThese results, provided below, highlight that while simpler methods may be appealing, they do not match the robustness or expressiveness of a symmetry-aware model like Equi-mRNA.\n\n Model | E. coli(A) | MLOS(S) | iCodon(S) | Tc-Ribo.(S) | mRFP(S) | COV Deg(S) | Improvement |\n|---|---|---|---|---|---|---|---|\nVanilla    |  0.499  |  0.633  |  0.143  |  0.698  |  0.477  |  0.631  |  -  |\nEqui-mRNA  |  0.532  |  0.691  |  0.257  |  0.736  |  0.639  |  0.733  |  **16.47%**  |\n\n`Another question...`\n\n One straightforward integration strategy is to use separate embedding pathways for distinct transcript regions: one designed for codon-level representations in the CDS (where SO(2)-based symmetry is biologically meaningful), and another for raw nucleotide-level inputs from the 5′ and 3′ UTRs, which do not follow codon semantics. These representations can then be fused at higher layers through concatenation, cross-attention, or shared pooling operations to support end-to-end prediction.\nThis modular approach allows us to retain the inductive bias from equivariant codon embeddings while leveraging regulatory features from UTRs. We are actively exploring this extension as part of our ongoing work on full-transcript modeling, and we agree that combining CDS and UTR modalities is a promising direction for improving accuracy in transcript-wide tasks.\n\n### References\n1. Li, Sizhen, et al. \"Codonbert: Large language models for mrna design and optimization.\" bioRxiv (2023): 2023-09.\n2. Yazdani-Jahromi, Mehdi, et al. \"Helm: Hierarchical encoding for mrna language modeling.\" arXiv preprint arXiv:2410.12459 (2024).\n3. Wood, Matthew, et al. \"Helix-mRNA: A Hybrid Foundation Model For Full Sequence mRNA Therapeutics.\" arXiv preprint arXiv:2502.13785 (2025).\n4. Chen, Jiayang, et al. \"Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function predictions.\" arXiv preprint arXiv:2204.00300 (2022).\n5. Outeiral, Carlos, and Charlotte M. Deane. \"Codon language embeddings provide strong signals for use in protein engineering.\" Nature Machine Intelligence 6.2 (2024): 170-179.\n6. Zou, Shuxian, et al. \"A large-scale foundation model for rna function and structure prediction.\" bioRxiv (2024): 2024-11."}, {"review_id": "eMDaPg4VNL", "reviewer": "Reviewer_PmYr", "summary": "In this paper, the authors propose a novel method to encode synonymous codons in mRNA language models. Their method, Equi-mRNA, represents synonymous codons as vector rotations in a base embedding space.", "questions": "- It seems that even the Vanilla model is outperforming competitor models such as CodonBERT (Table 1 and 2). Do you have an explanation for that?\n- Why did you restrict the training set to 512 or less codons? This seems a bit limiting, considering many proteins fall above this threshold. Modern hardware and architectures should easily handle contexts above 512 tokens. \n- Could you also report standard deviations for tasks other than MLOS?", "limitations": "The authors address the limitations of their work", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The presented method is innovative and sound. It’s well motivated by the fact that current codon language models learn embeddings of synonymous codons independently, even though synonymous codons are subject to the same biological constraints at the protein level. \n\nThe authors perform an extensive hyperparameter search and benchmark, evaluating different flavors of their proposed SO(2) embedding and demonstrating that their encoding outperforms competing models on a number of benchmark tasks. \n\nA potential weakness is that the authors do not seem to split the training corpus by sequence homology - this is common practice nowadays, especially for highly constrained sequences such as protein-coding genes. This may dampen the otherwise promising results on sequence generation.", "was_revised": true, "final_justification": "The authors address my concern and to my best judgement, this work provides a significant contribution to the field.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We appreciate the recognition of our method’s novelty, biological motivation, and empirical performance across benchmarks. Below, we address each of the reviewer’s concerns and questions in detail, including the issue of homology-aware splitting, architectural comparisons with CodonBERT, context length limitations, and the reporting of standard deviations.\n## Weakness\n`A potential weakness is that...`\n\nWe respectfully acknowledge the concern regarding the lack of homology-based splitting. However, we would like to clarify that our primary contribution is architectural, rather than focused on achieving state-of-the-art performance through data curation. All models in our study(both vanilla and SO(2) variants) are trained and evaluated on identical data splits, ensuring that any performance improvements are solely attributable to our proposed symmetry-aware embeddings rather than differences in data handling. Since any potential data leakage from homologous sequences would affect all models equally, the observed improvements (up to ~10% in downstream tasks and 4× in generation quality) can be confidently attributed to our architectural innovations. Furthermore, our generation experiments exclusively compare SO(2) variants against our own vanilla baseline rather than external models, making the lack of homology-based splitting less critical for our relative comparisons. While we acknowledge that homology-based splitting would be valuable for establishing absolute performance benchmarks and will add this to our limitations section, this does not diminish our core contribution of demonstrating that encoding codon-level symmetries through group-theoretic priors consistently improves model performance when all other factors are held constant.\n\n## Questions\n`It seems that even the Vanilla model...`\n\nThe performance difference can be attributed to several key architectural and methodological distinctions: CodonBERT [1] uses a BERT-style encoder with masked language modeling (MLM) objective and two-sequence relatedness classification, while our models employ GPT-2's decoder-only architecture with autoregressive language modeling. This unidirectional approach may better capture the biological reality of ribosomal translation, which proceeds directionally from 5' to 3', leading to more biologically meaningful representations. Additionally, our models are trained solely on single sequences, whereas CodonBERT [1] incorporates pairwise sequence relationships, introducing a different inductive bias. These architectural choices, along with differences in pretraining data and modeling objectives, likely explain our vanilla model’s strong baseline performance, which our SO(2) symmetry-aware variants further improve upon.\n\n`Why did you restrict the training set to 512 or less...`\n\nYou're absolutely right that many proteins exceed 512 codons, and modern architectures can handle longer contexts. Our choice was primarily driven by computational constraints specific to our symmetry-aware architecture: the geometric learning components, particularly equivariance loss, require additional forward passes during training. Specifically, the equivariance loss (Section 2.6) necessitates passing both the original base embeddings and rotated codon embeddings through the network, effectively doubling the computational burden. Combined with Riemannian optimization on the Stiefel manifold for learning rotation subspaces, this significantly increases memory usage and training time. As shown in Table 13, our SO(2) variants already require up to 11+ hours for training on just 1M sequences with 512-codon limits, and scaling to 25M sequences took nearly 4 days for the full model. Extending to longer sequences would have made the computational requirements prohibitive for our extensive ablation studies across 12+ model variants. We acknowledge this as a limitation and note that future work could explore more efficient implementations or selective application of symmetry constraints to enable modeling of full-length proteins while maintaining the benefits of our group-theoretic approach.\n\n`Could you also report...`\n\nAmong all the benchmarks used, MLOS was the only dataset that did not provide predefined splits. For this reason, we performed 3-fold cross-validation and reported the standard deviation across folds in the main results.\nFor the remaining datasets, which have fixed train/validation/test splits (as established in prior work), we additionally ran our model three times with different random seeds that affect initialization and dataloader shuffling. Across all these runs, we observed performance variation of less than 0.005 in the evaluation metrics. Given this minimal variability, we omitted reporting standard deviations for those datasets. We are happy to include them in the final version if the reviewers believe it would improve clarity.\n\n### References\n1. Li, Sizhen, et al. \"Codonbert: Large language models for mrna design and optimization.\" bioRxiv (2023): 2023-09."}]

Author rebuttal:

Meta-review: This work introduces Equi‑mRNA, the first codon‑level equivariant mRNA language model that explicitly encodes synonymous codon symmetries as cyclic subgroups of 2D Special Orthogonal matrix (SO(2)). The effectiveness is validated through extensive experiments. Reviewers recognize the merits of the paper including novel and principled method, as well as extensive experimental validation. The weaknesses include lack of comparison with existing mRNA foundation models, oversimplified assumptions for models, model scalability, experiment settings and results. The weaknesses and questions proposed by reviewers are addressed in the rebuttal by more experiments and clarifications, and all reviewers are satisfied with this work. Therefore, AC recommends accepting this paper.
Decision: Accept (poster)