Diffusion on Language Model Encodings for Protein Sequence Generation
Authors: Viacheslav Meshchaninov
Pavel Strashnov
Andrey Shevtsov
Fedor Nikolaev
Nikita Ivanisenko
Olga Kardymon
Dmitry Vetrov
Venue: ICML 2025 poster (2025)
Keywords: continuous diffusion
generative protein design
score matching
denoising models
sequence modeling
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: A continuous latent diffusion framework for protein sequence generation with strong performance and versatile conditional generation capabilities.

Abstract: Protein *sequence* design has seen significant advances through discrete diffusion and autoregressive approaches, yet the potential of continuous diffusion remains underexplored. Here, we present *DiMA*, a latent diffusion framework that operates on protein language model representations. Through systematic exploration of architectural choices and diffusion components, we develop a robust methodology that generalizes across multiple protein encoders ranging from 8M to 3B parameters. We demonstrate that our framework achieves consistently high performance across sequence-only (ESM-2, ESMc), dual-decodable (CHEAP), and multimodal (SaProt) representations using the same architecture and training approach. We conduct extensive evaluation of existing methods alongside *DiMA* using multiple metrics across two protein modalities, covering quality, diversity, novelty, and distribution matching of generated proteins. *DiMA* consistently produces novel, high-quality and diverse protein sequences and achieves strong results compared to baselines such as autoregressive, discrete diffusion and flow matching language models. The model demonstrates versatile functionality, supporting conditional generation tasks including protein family-generation, motif scaffolding and infilling, and fold-specific sequence design, despite being trained solely on sequence data. This work provides a universal continuous diffusion framework for protein sequence generation, offering both architectural insights and practical applicability across various protein design scenarios.  Code is released at [GitHub](https://github.com/MeshchaninovViacheslav/DiMA).

Diffusion on Language Model Encodings for Protein Sequence Generation
Viacheslav Meshchaninoy
Pavel Strashnov *2 Andrey Shevtsov *2 Fedor Nikolaev? Nikita Ivanisenko 2
Olga Kardymon
Dmitry Vetrov
Abstract
1. Introduction
Protein generation is emerging as a key area in academic re- search, with applications spanning bioinformatics, synthetic biology, and protein-based therapeutics (Wu et al., 2021; Ovchinnikov & Huang. 2021). Recent progress in this field has been driven by three main approaches: autoregressive models (Madani et al., 2023; Ferruz et al., 2022; Shin et al., 2021; Lv et al., 2025), which have demonstrated effective- ness in capturing sequential dependencies, diffusion models (Alamdari al., 2023; Wang et al., 2024; 2025), which have shown promise in both sequence and structure generation tasks and flow- based models (Huguet et al., 2024; Campbell et al., 2024; Yim et al., 2024; Lin et al., 2024), which have achieved impressive results on protein conditional structure generation.
Protein sequence design has seen significant ad- vances through discrete diffusion and autore- gressive approaches, yet the potential of con- tinuous diffusion remains underexplored. Here, we present DiMA, a latent diffusion framework that operates protein language model rep- resentations. Through systematic exploration of architectural choices and diffusion compo- nents, we develop a robust methodology that gen- eralizes across multiple protein encoders rang- ing from 8M to 3B parameters. We demon- strate that our framework achieves consistently high performance across sequence-only (ESM- 2, ESMc), dual-decodable (CHEAP), and mul- timodal (SaProt) representations using the same architecture and training approach. We exten- sively evaluate existing methods alongside DiMA using multiple metrics across two protein modali- ties, covering quality, diversity, novelty, and dis- tribution matching of generated proteins. DiMA consistently produces novel, high-quality and di- verse protein sequences and achieves strong re- sults compared to baselines such as autoregres- sive, discrete diffusion and flow matching lan- guage models. The model demonstrates versa- tile functionality, supporting conditional gener- ation tasks including protein motif scaffolding and infilling, and fold-specific sequence design. This work provides a univer- sal continuous diffusion framework for protein sequence generation, offering both architectural insights and practical applicability across vari- ous protein design scenarios. Code is released GitHub.
While discrete diffusion models have been successfully adapted for amino acid sequence generation (Alamdari et al., 2023; Wang et al., 2024), and significant advances have been made in three dimensiimeniioal protein diffusion (Watson et al., 2023; Wu et al., 2024; Lin & AlQuraishi, 2023; Fu et al., 2024), continuous diffusion on protein sequence representa- tions remains underexplored. Previous attempts (Lee et al., 2023; Zhang et al., 2023a) have been limited to specific protein representations or focused primarily on conditional tasks, leaving the potential of continuous diffusion for gen- eral protein generation largely untapped.
Recent advances in protein language models (pLMs) have produced increasingly sophisticated continuous representa- tions of proteins (Lin et al., 2023a; Lu et al., 2024; Su et al., 2023). These representations capture both sequence and structural information, providing natural foundation for latent diffusion approaches. Notably, studies have demon- strated that reconstructing proteins from continuous repre- sentations yields greater accuracy than from discrete ones (Lu et al., 2024; Gaujac et al., 2024), suggesting potential advantages for continuous diffusion models capturing protein properties.
'Equal contribution 'Constructor University, Bremen, Ger- many 2 *AIRI. Moscow, Russia. Correspondence to: Viach- eslav Meshchaninov mmeshhhaninov.vin www.hhhaaniioovvichelllaeee gom,, Pavel Strashnov <strrahnov @ airi.net>> Andrey Shevtsov <shevttov @ airi.net>.
In this study, we develop DiMA, new latent diffusion model that operates pLM representations. We demon- strate that continuous diffusion on protein embeddings en- ables effective sequence and structure generation across mul- tiple tasks and encoder architectures. DiMA addresses the
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Training
Inference
Conditional generation
Reconstructed representations
MNAPL...
Motif scaffolding
Sequence * Decoder
Stucture Decoder
Conditions
Denoiser
mmse
Continuous protein space
Fold Conditioning
Encoder
Denoiser
Famill-specific generation
Sequence
MSSRNNPARV
IPRO33747
WO I)
Structure tokens
vdavdddvvc
Figure 1. DiMA. The framework consists of three main components: (1) a pre-trained protein language model encoder that maps amino acid sequences to continuous latent representations, (2) a diffusion denoiser that generates latent vectors from Gaussian noise, and (3) sequence and structure decoders that reconstruct amino acid sequences and protein structures from the generated latent representations. During training, the model learns to denoise corrupted protein representations. During inference, the framework supports both uncondi- tional generation and conditional generation tasks including motif scaffolding, fold conditioning, family-specific generation. The approach enables joint sequence-structure generation while operating entirely in continuous latent space.
limitations of previous continuous diffusion approaches that have been limited to specific representations by establishing unified framework that generalizes across diverse pro- tein encoders. By operating in the continuous latent space of these pre-trained encoders, our approach circumvents the challenges associated with discrete sequence modeling while maintaining the expressiveness needed for complex protein design tasks. The framework is designed to be encoder-agnostic, allowing it to benefit from advances in protein representation learning without requiring architec- tural modifications. Our key contributions are as follows:
novel proteins, as validated both SwissProt and AFDBv4-90 datasets.
We show that single architecture optimized for uncon- ditional generation can effectively adapt to conditional tasks and backbone generation through lightweight modifications, suggesting a path toward unified protein sequence generation.
2. Continuous diffusion on LM representations of protein sequences
DiMA a latent diffusion model that operates on continu- ous protein representations, DiMA consists of three compo- nents: pre-trained encoder (E) that provides meaningful latent space representation, diffusion model (J) that gen- erates latent vectors from Gaussian noise, and a decoder (D) that maps generated latents back to amino acid sequences.
We develop DiMA, continuous latent diffusion model that operates effectively across different pL. rep- resentations, ranging from sequence-only ESM-2 to dual-decodable CHEAP and structure-aware SaProt encoders.
Through systematic exploration of architectural choices and diffusion components, we establish a ro- bust methodology that generalizes across encoders ranging from 8M to 3B parameters.
Continuous vs. Discrete Diffusion for Proteins. While discrete diffusion may appear more intuitive for sequence generation, continuous representations offer compelling ad- vantages for protein modeling. Continuous encodings from protein language models capture rich semantic and struetural information that has proven effective across diverse protein tasks, from representation learning structure prediction. Recent studies have shown that continuous representations yield superior reconstruction accuracy compared to discrete alternatives (Lu et al., 2024), suggesting their potential for generative modeling.
Despite having only 35M parameters, DiMA demon- strates strong performance on multiple benchmarks, matching exceeding specialized models in uncon- ditional generation, family-specific design, motif scaf- folding and sequence infilling, and fold-conditioned generation.
Our empirical evaluation across protein sequence and structure metries demonstrates that DiMA maintains high structural quality while generating diverse and
Continuous diffusion offers several theoretical and practical
2

---

Diffusion on Language Model Encodings for Protein Sequence Generation
advantages over discrete approaches such as direct appli- cation of established score-based techniques like classifier and classifier-free guidance without requiring discrete ap- proximations. It also provides seamless integration with multimodal representations that jointly encode sequence and structure (e.g. CHEAP, SaProt), Also, continuous dif- fusion more stable and efficient in training compared to discrete ones. However, the sequential and discrete nature of protein sequences presents unique challenges for continu- ous diffusion that necessitate careful adaptation of diffusion components to effectively capture the complexity of protein space.
Reconstruction loss
0t
1.0
linear
0.8
cosine
0.6
tan-10
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Figure 2. Left: the diffusion reconstruction loss of zo from with different noise schedules: ||z0 E0 t)1l Right: vou(I).
Latent Space Adaptation. We utilize a pre-trained transformer-basee pL.M as encoder with ESM-2 (Lin et al., 2023a) being the default choice unless otherwise spec- ified. The encoder maps the sequence of discrete amino acids = [M..... ya] of length 8 to the latent vectors I = [r1...., x2] E Raxd x = Ety) We apply normal- ization 2g = N Normalize(z)) over the hidden dimension d: for each component d;, we precompute the mean and variance over the training data and then apply normalization using these statistics to achieve zero mean and unit variance. This transformation allows us to adapt the discrete protein input to a standard Gaussian diffusion framework.
previous timestep for estimation E0. = 20(2), t, E0,s),
During training, we sample timestep & ~ [[0 1j. In half of the cases, we provide no additional input to the model, setting 20. = 0, where 0 is a zero vector. In the remaining cases, we estimate u0,4 = E0(2,, t, 0). The loss is computed as:
(()) VE.
[| E0 (G (2)
where SG[:] denotes the stop-gradient operation. Unlike (Chen et al., 2022), apply linear transformation to L0, and incorporate it into each transformer block's 's input. The effect of self-conditioning rate on the quality-diversity trade-off is depicted on Figure 12
Noise Schedule Optimization. We have found that the linear and cosine noise schedulers widely employed in the image domain (Song et al., 2021b; Ho et al., 2020; Nichol & Dhariwal, 2021) are sub-optimal for the protein domain. We conjecture that this happens due to the sequential and discrete nature of the protein representations.
Token Reconstruction. Our architecture utilizes the ESM- decoder, which was pre-trained alongside the encoder on masked language modeling objectives. We found that ad- ditional fine-tuning of the decoder, specifically for amino acid reconstruction, improves sequence generation accu- racy from latent representations x during inference. The decoder maintains simple architecture with a single linear layer. For CHEAP and SaProt representations, we fine-tune their corresponding pretrained decoders alongside the dif- fusion model, similar to our approach with ESM-2. We found that low-dimensional embeddings (e.g., ESM-2 8M, d 320) are less robust to small perturbations than higher- dimensional ones (ESM2/SaProt 650M, d = 1280, CHEAP, d = 1024). Fine-tuning the decoders helps minimize these effects during diffusion generation.
The reconstruction loss of diffusion models trained with such schedulers remains minimal at small noise scales. Con- sequently, the reconstruction of zo from 2 Vorz0 vI- ayy becomes quite trivial for the model for long period of time, leading to inefficient training. We adopted the noise schedule from (Hoogeboom et al., 2023):
+ de tan (
where d is hyperparameter reflecting the schedule's 's rate. The larger the value of d, the greater the data corruption rate. In this work we use d = 10, hence the schedule is named t0n-10. We utilize heuristic approach based on the observation that the reconstruction loss should exhibit an approximately linear increase over diffusion time (Figure 2). The rationale behind the tan-10 schedule is discussed in Apeendix C.5.
Length Determination. key challenge in the inference phase. While many discrete diffusion models (Gong et al., 2023; Yuan al., 2022; Li et al., 2022; Mahabadi et al., 2023) generate padding tokens alongside semantic tokens, we found this approach subopti- mal for protein generation. Instead, we employ two distinct strategies for training and inference.
Determining sequence length is
Self- Caff-Conditoning.
Following recent advances in se- quence generation, we apply the self-conditionng technique (Chen et al., 2022). The denoising network predicts &n using the latent variable 2 and timestep as input. Self- conditioning additionally utilizes predicted og,s from the
During training, we use an attention mask to focus the model exclusively on semantic tokens. This masking strategy is
3

---

Diffusion on Language Model Encodings for Protein Sequence Generation
crucial encodings of special tokens (like padding) often contain irrelevant potentially detrimental information to the diffusion model. By excluding these tokens from reconstruction loss computation, we improve both training stability and generation quality.
tance (FD-seq) measures how well generated sequences match the distribution of natural proteins on the held-out test set, predicted Local Distance Difference Test (pLDDT) assesses the structural plausibility of generated proteins, clustering density at 50% sequence identity (CDo.5) quanti- fies sequence diversity by measuring the fraction of distinct protein clusters in generated samples, and novelty scores evaluate similarity to training data to detect potential mem- orization. These metries complement each other, allowing us to evaluate both the quality and diversity of generated proteins across sequence and structure modalities.
During inference, we first sample the target sequence length from the training data distribution to ensure realistic protein lengths. We then sample a random Gaussian vector of the appropriate dimension and apply T steps of iterative refine- ment to generate 2g. Finally, we denormalize the latent representation and decode it into an amino acid sequence. Additional details about length distribution modeling are provided in Appendix C.3.
Our comprehensive evaluation framework also includes structural metrics (TM-score, scPerplexity), additional dis- tributional similarity measures (maximum mean discrep- ancy, I-Wasserstein optimal transport) computed on both se- quence and structure representations, perplexity scores from state-of-the-ar Mss and diversity assessment through mul- tiple sequence identity thresholds. To ensure robustness, we evaluate all metrics on large sample sizes of 2,048 sequences and validate against independent test sets. While we present key findings in the main text for clarity, each experimen- tal result accompanied by a complete evaluation using the full metric suite in the corresponding appendix subsec- tion. Detailed descriptions and implementation details of the evaluation metrics are provided in Appendix B.
Model Architecture. Our diffusion model employs transformer architecture with 12 layers, 16 attention heads, and a hidden size of 320. To adapt this architecture for protein sequence generation, we implement two key modifi- cations (detailed architecture specifications are provided in Appendix C.I).
To incorporate time-dependent information, we integrate time embeddings into the transformer blocks via a linear projection, followed by summation at the input of each block. We extend this mechanism to enable conditional gen- eration tasks, including protein family-specific generation, motif scaffolding, and fold-specific sequence design. For scaffolding and fold-specific sequence design, we further enhance conditioning by introducing cross- pross-attention blocks that process condition encodings, allowing the model to leverage structural constraints effectively.
3.2. Denoiser Component Analysis
Existing protein generation methods based on Gaussian dif- fusion (Lee et al., 2023; Zhang et al., 2023a) insufficiently address the selection of optimal methodologies, largely re- lying on techniques adapted from image diffusion models. In this study, we recognize the need to carefully select the diffusion components, in order to develop a Gaussian diffu- sion model that can effectively capture the complex patterns of protein space.
Inspired by (Bao et al., 2023), we incorporate long skip con- nections throughout the transformer architecture. Our ex- periments demonstrate that these connections significantly accelerate model convergence and enhance training effi- ciency, facilitating the generation of high-quality protein sequences with improved stability.
In this part of our study, we utilize the ESM-8M encoder and DiMA-35M model for our experiments. To assess the con- tribution of the proposed design choices to the performance of DiMA, we train several models from scratch with the fol- lowing modifications: removing the long skip-connections between the shallow and the deep transformer blocks; using time conditioning through admixing the time embeddings to the corrupted latent vectors of amino acids instead of employing dedicated time layer before each transformer block; omitting the transformer encoder (ESM-2), retaining only its embedding matrix; training the model without self- conditioning: training models with linear and cosine noise schedule; training models with padding reconstruction and without prior length sampling: omitting finetuning the decoder; and using flow matching paradigm in our latent generative model.
3. Experiments
In this section, we first identify which architectural com- ponents significantly affect generation quality, diversity, and distributional properties ($3.2). We then show that our architecture maintains strong performance across pro- tein encoders of different sizes and dataset scales, without requiring modifications ($3.4). We conclude by demonstrat- ing practical applications in family-specific design ($3.6.2), motif scaffolding ($3.6.1) 1), and fold-conditioned generation (3.6.3), using quantitative metrics to assess success in each task.
3.1. Evaluation Metrics
We evaluate protein generation using four key metrics that capture essential aspects of the task; sequence Fréchet dis-
Table 1 demonstrates that each proposed feature contributes
4

---

Diffusion on Language Model Encodings for Protcin Sequence Generation
Table 1. Ablation study of key components of DiMA trained on SwissProt dataset using ESM-8M encoder. The complete results are presented in Table 5.
Table 2. Performance comparison between DIMA and alternative generative models for the protein generation of the same parame- ter count trained on SwissProt dataset. The complete results are
Model
FDssey
PLDDT (1)
CDo. (1)
Novelty (*)
presented in Table 6.
Dataset
0.13
80.7
1.000
25.5
Model
FD-seq (1)
pLDDT (1)
CDo. (f)
Novelty (1)
Random sequences
3.97
24.8
1,000
85.1
Dataset
0.13
80.7
1,000
25.3
DIMA
0.34
83.3
0.611
35.7
Randon sequences
3,97
24.8
1,000
85.1
skip connections
0.45
77.3
0.619
43.1
w/o time layers
0.41
79.4
0.550
38.4
Walk-Jump
2.63
32.4
1,000
82.2
w/o ESM encoder
1.07
62.7
0:619
50.0
RITA
1.19
43.9
0.988
60.4
w/o self-condttioinng
0.55
68.2
0.929
56.6
proteinGAN
2.94
30.4
0.955
83,5
w/o finetuned decoder
0.54
80.1
0.589
38.4
SoqDesign
3,53
43.1
0.929
81.2
length sampling
0.67
65.0
0.880
58.4
SeoDiff-OADA
1.49
37.1
0.986
77.6
w linear schedule
0.47
77.0
0.611
45.8
D3PM
1.50
36.7
0.994
78.4
cosine schedule
0.94
54.1
0.878
67.0
DFM
1.46
37.8
0.996
77.2
w flow mwwchiithhn
0.71
63.4
0.960
72.2
DPLM
0.50
84,0
0.494
11.5
nanoGPT
1.24
61.0
0.900
53.7
DIMA
0.34
83.3
0.611
35.7
significantly to the model's performance individually. The most substantial decrease in both the quality and distribu- tion similarity of the generated sequences occurs in the ablated models without the ESM-2 encoder, without length sampling, and when trained without condiiondi....... Re- moving skip-connections and time layers results in less pronounced impact, but still significant decrease in repeti- tions of generated sequences and slight improvement in overall quality.
pared to alternative generative paradigms. DiMA achieves the best proximity to the distribution of real proteins on the held-out test set, while also showeasing high quality in the generated proteins. NanoGPT, an autoregressive model, shows promising results but fails to match datasct-level met- rics, struggling with quality and distribution alignment in protein space. DPLM, a discrete diffusion model, gener- ates structurally plausible proteins but suffers from exces- sive amino acid repetition and whole-sequence duplication, leading to low diversity and novelty. Compared DiMA, DPLM exhibits threefold lower novelty and twice the rep- etition rate, despite close pLDDT scores (see Table 2 and Figure 16). This suggests that while DPLM captures struc- tural features, it lacks the diversity essential for realistic protein generation.
To ablate the impact of the tan-10 noise schedule, we train our diffusion model with standard linear and cosine sched- ules, leaving other parameters intact. We find that tan-10 significantly outperforms the cosine schedule in both quality and distribution similarity. It also achieves less expressed but better results than the linear schedule. The detailed results are provided in the Appendix E.
In comparison, other baselines exhibit notably poorer perfor- mance. SeqDesign and ProteinGAN, initially designed for narrow classes of proteins, may not be suitable for training on diverse datasets. While EvoDiff outperforms SeqDe- sign and ProteinGAN, it still demonstrates metric values closer to a random sample than to the dataset, consistent with observations in the original EvoDiff paper (Table S3 of (Alamdari et al., 2023)).
3.3. Comparison Across Generative Paradigms
We evaluate DiMA, a latent Gaussian diffusion model trained ESM-8M encodings, against various genera- tive models for protein sequence generation in small- scale setup. To ensure fair comparison, all models are trained from scratch with 35M parameters on SwissProt. For methods requiring predefined sequence lengths, we sample lengths from the training set distribution. This comparison focuses on sequence- baseed models with publicly available code.
Detailed results with additional metrics are presented in the Appendix E.2.
We consider five groups of baselines: autoregressive models (RITA (Hesslow et al., 2022), SeqDesign (Shin et al., 2021), nanoGPT (Karpathy, 2023)), score-based models (Walk- Jump (Frey et al., 2024)), generative adversarial networks (ProteinGAN (Repecka et al., 2021)), discrete diffusion models (EvoDitr-OADM. (Alamdari et al., 2023), DPLM (Wang et al., 2024), D3PM (Austin et al., 2021)), and flow- based models (DFM (Campbell et al., 2024)).
3. BIOLOGICAL RELEVANCE
To explore the biological relevance of the generated se- quences, we employ the established protein annotation tool InterProScan L et al., 2023; Jones et al., 2014). We use three different Swissprot-trained models: DPLM, DiMA, and nanoGPT. Our analysis shows that DiMA and DPLM, models exhibiting high-quality metrics, consistently generate sequences with a high degree of anno- tation compared to the lower- wwwrrpperfrmiig nanoGPT (Figure 23A). This pattern is further reflected through the annotation intersections, where DiMA and DPLM demonstrate more significant overlap in their annotations (Figure 23B).
Table 2 presents the result of the comparison of existing methods and DIMA. The evaluation demonstrates that the proposed latent Gaussian diffusion approach performs ex- ceptionally well in unconditional protein generation com-
5

---

Diffusion on Language Model Encodings for Protein Sequence Generation
spaces ($3.4.2).
Table 3. Performance of protein sequence generation using DiMA and different encoders on AFDBy4-90 dataset. The complete results are presented in Table? 7
The SaProt encoder, which integrates structural tokens from FoldSeek's s 3Di vocabulary, introducing hybrid approach to sequence representation with structural awareness ($3.4.3).
Encoder
FD-seq (L)
PLDDT (f)
CDo.s
Novelty
Dataset Random
0.11
83,9
0.994
57.6
2,55
22.16
1,000
84.7
By preserving the core diffusion methodology, we aim to understand how different representation spaces influence protein sequence generation, providing insights into the generative capabilities across diverse protein embedding approaches.
ESM-2 8M
0,560
74,25
0,981
68.0
FSM-2 35M
0.340
75.71
0,986
69.1
ESM-2 150M
0.323
80.07
0,988
65.6
ESM 650M
0.318
0
ESM-2 3B
0,314
K3,40
0.969
63.0
ESMc 300M
0.326
82.70
0,963
64.2
CHEAP-shorten-I
0.346
81,92
0
64.6
CHEAP-sborea-2
0.340
78.81
0,946
66.2
3.4.1. PERFORMANCE SCALING WITH ESM-2 ENCODERS
SaProt 35M
0.366
82.23
0.976
65.5
SaProt 650M
0.411
83.01
0.980
We analyze the latent spaces of the ESM encoder family (ESM-2 8M through ESM-2 3B), investigating how perfor- mance scales with model capacity (Table 3). DiMA success- fully generates designable proteins across the entire encoder spectrum, with generation quality (pLDDT) consistently improving from ESM-2 8M (74.3) to ESM-2 3B (83.4).
While both approaches achieve similar levels of annotated proteins, their domain length characteristics differ. DiMA accurately reproduces dataset domain lengths and tends to generate small domains (50-75 amino acids). In contrast, DPLM frequently produces more extended domains (ap- proaching 254 amino acids in length) (Figure 23C). We hypothesize that the prevalence of long domains in DPLM correlates with its lower generation diversity, as evidenced by our diversity and distribution similarity metrics (Table 6).
The scaling analysis reveals a key trade-off between quality and diversity. While larger encoders demonstrate enhanced precision in capturing protein patterns and improved distri- butional matching (FD-seq decreases from 0.560 to 0.314), they show reduced sequence diversity (CDu.s declines from 0.981 to 0.969). This pattern manifests in the repetition metric smaller encoders generate more repetitive patterns, while larger ones produce more refined but potentially less novel sequences (Table 7).
3.4. Representation Space Scaling
In the previous sections, we established a robust diffusion model through comprehensive ablation studies and demon- strated its strong performance against baseline generative approaches. Building on this foundation, we now explore the model's s adaptability across diverse protein representa- tion spaces using the large-scale AFDBv4-90 dataset.
The recently introduced ESM-C 300M (ESM Team, 2024) matches ESM-2 650M in generation quality (82.7 82.5 PLDDT) but shows slightly reduced coverage of protein space (FD-seq 0.326 vs 0.318, CDp.s 0.963 vs 0.986). The marginal quality improvement from ESM-2 650M to ESM- 2 3B suggests that mid-range architectures may offer an optimal balance between performance and computational efficiency.
The AFDB dataset represents a carefully curated sub- set of UniRef50. Comprising 2.2 million protein sequences with > 90% AlphaFold2-preddiceed structural confidence, this dataset by design excludes intrinsically disordered pro- teins and low-entropy sequences, ensuring high-quality protein corpus. As consequence, this curation allows us to use PLDDT reliable, consistent measure of protein structural quality.
This systematic evaluation demonstrates that DiMA can effectively leverage encoder representations of varying com- plexity while maintaining lightweight deployment foot- print, as the encoder becomes dispensable during inference.
We maintain the core denoising model architecture devel- oped in our ablation studies ($3.2), using only simple linear projection techniques to adapt to different encoder dimen- sions. Here, we examine three representative encoder archi- tectures:
3.4.2. ADAPTATION TO CHEAP REPRESENTATIONS
CHEAP is an encoder that enables efficient dual-modal representations of proteins, providing access both sequence and structure information from sequence in- put alone. Based on ESMFold, it aggregates informa- tion from all layers of ESM-2 3B encoder and com- presses the continuous space, achieving significant dimen- sionality reduction while preserving high-fidelity struc- tural and sequence reconstruction. We explore two compression variants: CHEAP 1horten_1_dim 1024 and
The ESM-2 sequence-only encoder family, spanning models from 8M to 3B parameters, allows us to in- vestigate performance scaling with model complexity 1$3.4.1).
CHEAP representations, which uniquely enable decod- ing into both sequence and 3D-structure from its latent
6

---

Diffusion on Language Model Encodings for Protein Sequence Generation
SHEEAP-shorten, 1iarrten___im_1000 1024, both reducing the channel dimension while the latter additionally compresses the se- quence length dimension.
100
DIMA
DPLM
90
EvoDiff
Quality (pLDDT)
Sequence Generation. DiMA trained with CHEAP en- coder demonstrates strong performance in sequence space (Table 3). Both compression variants maintain high genera- tion quality: CHEAP shorten dim 1024 achieves PLDDT of 81.9 while CHEAP searsessootteiiiii 2_t_m_2ddm_2222---1 reaches 78.8, approaching the dataset quality benchmark of 83.9. The variant without sequence length reduction shows superior performance, suggesting that preserving the full sequence dimension benefits generation quality.
ProLLAMA
ProtGen2
70
PLAID
RFDiffusion
Multiflow
60
ProtGPT2
Chroma
40
0.6
0.7 Diversity (CDo.s)
0.8 0.9
1.0
Structure Generation. We assess structural consistency through two evaluation protocols: co-design, comparing generated backbones against structures reconstructed from predicted sequences, and structure-only evaluation using standart protocol desribed in Yim et al. (2023a) (Table 11). The structure-only approach achieves 92.3% success rate with mean scRMSD of 1.091, while co-design demonstrates strong performance with 88.8% success rate and mean scRMSD of 1 .043A. These results indicate that DiMA effec- tively leverages CHEAP's compressed yet information-rich latent space for both sequence and structure generation. No- tably, the sequence length reduction in CHEAP shorten 2 enables using half-length transformer context for the denois- ing model, offering substantial computational advantages while maintaining generation quality.
Figure 3. Comparison of DiMA with large pretrained protein gen- erative models in quality and diversity. Circle size represents model scale. The complete results are presented in Table 8.
architectural modifications. This adaptability, combined with strong performance in structure-aware tasks like motif- scaffolding (detailed in $3.6.1), indicates that our diffusion framework can effectively capitalize on both sequence and structural information encoded in the latent space.
3.5. Comparison with Large Pretrained Models
In this section we compare DiMA-35M trained over ESM- 3B encodings with existing large protein models, we demon- strate that DiMA achieves performance comparable to exist- ing pre-trained large protein models.
Additional results for the DiMA model using CHEAP encoders for structure generation are available in Ap- pendix E.
We evaluate -iMA-35M, trained on ESM-3B encodings, against of-te-of-thh-arr pre-trained protein models, namely, RITA (Hesslow al., 2022), ProtGPT2 (Ferruz al., 2022), ProGen2 (Madani et al., 2023), Evo (Alamdari et al., 2023), ProLLAMA (Lyv et al., 2025), DPLM (Wang et al., 2024), Chroma (Ingraham et al., 2023), Multiflow (Camp- bell et al., 2024), RFDiffusion (Watson et al., 2023), and PLAID- 100M (Lu et al., 2025). For all models, we use the authors' recommended sampling parameters to ensure a fair comparison. However, for autoregressive models Pro- Gen2 and ProLLAMA, which show suboptimal quality and collapse to highly repetitive sequences on default settings, we performed grid searches to identify optimal temperature and top-p values. We consider only models with publicly available pre-trained weights, ensuring transparency and reproducibility. Figure 3 illustrates the relationship between quality, diversity, and model size.
3.4.3. ADAPTATION TO SAPROT ENCODER
Multimodal SaProt encoder (Su et al., 2023) presents an al- ternative approach to protein representation, integrating both sequence and local structural information. Unlike sequence- only models, SaProt enriches its representations using struc- tural tokens from FoldSeck's (Van Kempen et al., 2023) 3Di vocabulary, providing compact yet informative structural descriptors.
We evaluate SaProt encoders of two sizes (35M and 650M parameters) using the same hyperparameters for the denoiser model that we established through our ablation studies. The quality metrics demonstrate that structural awareness in SaProt's representations translates to improved generation capabilities models achieve higher pL.DDT scores com- pared to same-sized ESM-2 variants (82.23 vs 75.71 for 35M and 83.01 vs 82.48 for 650M architectures, Table 3). This improvement is particularly notable with the smaller 35M parameter encoder, suggesting that structural tokens provide an efficient way to encode protein properties.
DiMA demonstrates high-quality and diverse protein gen- eration, achieving performance on par with much larger models despite using two orders of magnitude fewer param- eters. While DPLM and other diffusion- based models excel in structural plausibility, they lag in diversity. where DiMA outperforms them significantly. Notably, Multiflow, trained
The integration with SaProt demonstrates DiMA's ability to leverage different types of protein embeddings without
7

---

Diffusion on Language Model Encodings for Protein Sequence Generation
with structural data, achieves similar performance to DiMA
Num. of solved problems
despite operating at comparable parameter scale. An im- portant advantage of DiMA is that it achieves performance
ESM3
REDiffusion DIMA
DPLM2
comparable to models trained using structural information, while being trained exclusively on amino acid sequences.
This demonstrates the model's efficiency and ability to ex- tract meaningful representations from sequence data alone,
making it highly versatile and resource-cffiiint solution for protein generation tasks.
DPLM 1
sequence-basee
These findings underscore DiMA's efficiency and scalabil- ity. establishing it as compelling approach for protein sequence generation, even in comparison to large-scale pre- trained models. Full results across model sizes and evalua- tion metrics are provided in Appendix E,4 and Table 8.
no-generation
EvoDiff
structure-bass
-10P
-109
Model size
Figure 4. Motif scaffolcafol performance comparison across dif- ferent model sizes. Methods are colored by input modality: sequence-based (purple), sequence-structure co-generation (green), and structure-basce baattrrre-badddddee (orange). DiMA solves 19/24 benchmark problems while being significantly more compact than other high- performing models. The complete results are presented in Table 13
3.6. Advanced generation tasks
3.6. I. FUNCTIONAL-MOTII SCAFFOLDING
We evaluate the conditional generation capabilities of DiMA on challenging task of cunctional-motf scaffolding us- ing the established RFDiffusion benchmark (Watson al., 2023). This task requires designing entirely new protein structures that incorporate and preserve specific functional motifs.
We investigate two approaches for training DiMA (35M parameters) to perform family generation: classifier guidance and conditional fine-tuning.
Using FSM2-650M encodings and the AFDBv4_90 dataset, train and evaluate model eight diverse protein families, including cRISPR-associated proteins, calmod- ulins, and glycosyl hydrolases. For classifier guidance, we train a lightweight classifier (3 transformer blocks) on noisy protein encodings to predict family membership. For con- ditional fine-tuning, we augment DiMA with family class label embeddings and fine-tune on all families simultane- ously. We compare against significantly larger baselines, including ProLLAMA (7B parameters), ProGen2 (151M parameters), and EvoDiff (640M parameters).
We evaluate DiMA on 24 benchmark problems, where each problem requires generating protein sequences that main- tain precise spatial positioning of functionally important residues. For conditioning, we augment DiMA with an en- coder that provides motif information to each transformer layer. Following established protocols (Alamdari et al., 2023), we sample 100 designs per problem and consider problem solved if at least one design achieves both struc- tural quality (pLDDT 2 70) and motif preservation (RMSD < A for motif residues).
Using the SaProt-650M encoder, DiMA successfully solves 19 out of 24 problems, outperforming other sequence-basse methods, including EvoDiff, DPLM, and DPLM2 (Figure 4). While based methods like RFDiffusion achieve higher overall success rates, DiMA generates more diverse successful scaffolds as measured by unique success rate (0.1 vs 0.06 for RFDiffusion) matching the diversity of ESM-3 while using two orders of magnitude fewer parame- ters. Interestingly, DiMA and 3 show complementary strengths across different scaffold types, with DiMA achiev- ing the highest unique success rates on 6E6R-type problems. Complete results and experimental details are provided in Appendix E.8.
We evaluate generations using multiple complementary met- rics: InterProScan for family membership verification (Fi- delity), pL.DTT for structural quality assessment, and cluster diversity 50% sequence identity (CDo.s) to measure gener- ated sequence diversity. Both DiMA variants achieve high fi- delity to target families while maintaining structural quality, While autoregressive models demonstrate high sequence di- versity but struggle with fidelity and quality (pLDDT Rs 60), discrete diffusion achieves comparable fidelity and quality but generates fewer novel proteins. Notably, the classifier- guided variant achieves competitive performance without requiring model fine-tuning, offering a practical advantage for targeted protein design. Complete results and experi- mental details are provided in Appendix E.5.
3.6.2. FAMILY-SPECINE GENERATION
3.6.3. FOLD-COONITIONED GENERATION
Generating proteins that belong to specific functional fami- lies is key task in protein engineering, enabling targeted exploration of sequence spaces with desired characteristics.
Generation of proteins with specific structural proper- ties represents a distinct challenge from sequence-based
8

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Seq. identity: 46.30%
Seq. identity: 36.08%
lar but non-identical proteins. This performance difference likely stems from DiMA's of richer structural encodings compared to RFDiffusion's secondary structure and block- adjacency representations. Figure 5 illustrates examples of generated proteins alongside their target folds. Complete experimental details are provided in Appendix E.9.
4. Conclusion
Seq. identity: 34.41% %
Seq. identity: 30.13%
In this paper, introduce DiMA, continuous latent dif- fusion framework for protein sequence generation that op- erates effectively across diverse protein language model representations. Through systematic exploration of archi- tectural choices and diffusion components, we establish a robust methodology that generalizes from sequence-only encoders multimodal representations spanning 8M 3B parameters.
DiMA demonstrates versatile functionality across multiple protein design scenarios, including unconditional genera- tion, family-specifie design, motif scaffolding, sequence infilling, and fold-conditioned generation. The framework achieves competitive performance with significantly larger models while maintaining computational efficiency, the encoder becomes dispensable during inference.
Seq. identity: 36.08%
Seq. identity: 38.50%
This work provides both architectural insights and practical applicability for protein sequence generation, offering uni- fied approach that bridges sequence and structure generation through continuous diffusion. The systematic methodology we establish creates a foundation for future developments computational protein design, demonstrating how domain- specific adaptations can unlock the potential of diffusion models for biological sequence generation.
Figure 5. Examples of successful proteins generated via fold- conditioning, aligned on the corresponding target proteins. Se- quenee identity percentage between the target protein and the generated one is reported for each design.
tasks, requiring the model capture complex three- dimensional relationships. We explore DiMA's capabilities in fold-conditioned generation using the CHEAP encoder (shorten=1, 10m=1 which provides access to ESM- Fold's latent space representation of protein structure.
Impact statement
This work advances machine learning approaches for protein sequence generation, introducing a framework that bridges discrete and continuous methodologies. Beyond the core technical contributions to diffusion models and protein rep- resentation learning, our work has potential applications in therapeutic protein design and synthetic biology. While these applications could benefit society through new drug development and biotechnology advances, we acknowledge they require careful consideration of biosafety and ethical implications in deployment. We believe the benefits of ad- vancing protein design capabilities outweigh potential risks when proper safeguards and responsible development prac- tices are followed. Our framework emphasizes transparency through extensive evaluation metrics and validation proto- cols, promoting reproducible and responsible progress in computational biology.
We finetune DiMA on the CATH S40 non-redundant dataset ( (27k proteins) and evaluate performance on hold-out set of 100 structures. For each structure, generate 10 pro- teins and assess their similarity to the target fold using the TM-score. Following the protocol of (Watson et al., 2023), we consider generation successful at least one design achieves TM-score > 0.5, indicating significant structural similarity. For comparison, we evaluate against RFDiffu- sion, specialized structure generation model capable of fold-conditioning.
DiMA achieves a mean TM-score of 0.93 with 100% suc- cess rate across the benchmark set, compared to RFDiffu- sion's 's mean TM-score 0.48 and 41% % success rate. The av- erage RMSD between the best attempts and their target folds 2.6 indicating that DiMA generates structurally simi-

---

Diffusion on Language Model Encodings for Protcin Sequence Generation
References
Consortium, T. U. Uniprot: the universal protein knowl-
edgebase in 2021. Nucleic Acids Research, 49(D1): D480-D489, 11 2020, ISSN 0305-1048, doi: 10.1093/ nar/gkaal 100. URL /cttps://oi.... http:///oooooog///.ooo/oo 00991.113// har/gkaa1100
Alamdari, S., Thakkar, N., van den Berg, R., Lu, A. X., Fusi, N., Amini, A. P., and Yang, K. K. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, 2023. doi: 10.1101/2023.0.111 1..5667..
1093/
URL
www.:/owwww..ogoooteeeeeee content/nnen//////oo early/2023/09/122222
Dauparas, J., Anishchenko, I., Bennett, N., Bai, H., Ragotte, J., Milles, L. F., Wicky, B. I. M., Courbet, A., De Haas, R. J., Bethel, N., Leung, P. J. Y., Huddy, T. F., Pel- lock, S., Tischer, D., Chan, F., Koepnick, B., Nguyen, H., Kang. A., Sankaran, B., Bera, A. K., King, N. P., and Baker, D. Robust deep learning-based protein se- quence design using proteinmpnn. Science, 378(6615): 49-56, 2022. ISSN 0036-8075, 1095-9203. doi: 10.1126/ ccience.add2187. URL https: //ww://wwwww//../// science. org/doi /10. 10.00.126/// science. add2187.
09 11. 556673.
Austin, J., Johnson, D. D., Ho. J.. Tarlow, D., and Van Den Berg, Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14.17981-7799, 2021.
Bao, F. Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22669- 22679, 2023.
Durairaj, J., Waterhouse, A. M., Mets, T., Brodiazhenko, T., Abdullah, M., Studer, G., Akdel, M., Andreeva, A., Bateman, A., Tenson, T., Hauryliuk, V., Schwede, T., and Pereira, What hidden the darkness? deep-learnig assisted large-scale protein family curation uncovers novel protein families and folds. bioRxiv, 2023. doi: 10.1101/2023.0. 14.532539, URL attps://www. http://wwww.ooxii..oo//oooo///// 03.1.53339.
Blum, M., Andreeva, A., Florentino, L. C., Chuguransky, S. R., Grego, T., Hobbs, E., Pinto, B. L., Orr, A., Paysan- Lafosse, T., Ponamareva, I., et al. Interpro: the protein sequence classification resource in 2025;? mode long- meta? Nucleic Acids Research, 53(1) (3(D4):D444 D456, 2025.
Bose, J., Akhound-Sadegh, T., Huguet, G., FATRAS, K., Rector-Brooks, J., Liu, C.-H., Nica, A. C., Korablyov, M., Bronstein, M. M., and Tong, A. SBo3)-stochastic flow matching for protein backbone generation. In The Twelfth International Conference on Learning Representations, 2024. URL https::///// http///eeeeeeeeeeeeeeeeeeeeeeeea netie/eweeeeeeeeeeeeeeeeeeee hnte/foount id=kJF IH23hXb.
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., and Rost, B. ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Patter Analysis and Machine Intelligence, 44(10):111--1227, October 2022. ISSN 0162-8828, 2160-9292, 1939-3539, doi: 10. 10.1109/TPAMII.000 30095381. URL https:// ieeexplore org. http://dcumet//////// 94477085/
Campbell, A., Yim, J.. Barzilay, R., Rainforth, T., and Jaakkola, T. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.044: 04997, 2024.
ESM Team. Esm cambrian: Revealing the mys- teries of proteins with unsupervised learning, 2024. URL esm- cam-cambrian.
Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.). Learning inverse folding from millions of predicted structures, volume 162 of Proceedings of Machine Learning Research, 17-23 Jul 2022. PMLR. URL https: http::/ss///pppeeiee/eemmm mmr. press/val www.sh//hhaae html.
Ferruz, N., Schmidt, S., and Höcker, B. ProtGPT2 is deep unsupervised language model for pro- tein design. Nature Communications, 13(1):4348, July 2022. ISSN 2041-1723. doi: 10.1038/ $41467-022-32007--7 URL http:///:: nature com/artticles/s414555 022-32077-7
Chen, N.. Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., and Chan, W. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. htth:tt/:// tttttps://cpereevieee ope.revien.ee.eee net forum? -adddNaMMLP... NadMNMMMMaaaS FFO880..
Frey, N. C., Berenberg. D., Zadorozhny, K., Kleinhenz, J., Lafrance- Vanicee-Vansse,e,,e J., Hotzel, I., Wu, Y., Ra, S., Bonneau, R., Cho, K., Loukas, A., Gligorijevic, V., and Saremi, S. Protein discovery with discrete walk-jump sampling. In The Twelfth International Conference on Learning Representations, 2024. URL https: httss:/::// /ttp:://pppeeeiissee net: (atorrmmid=zMMPHOOO
Chen, T., Zhang, R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. arXiv preprint arXiv:2208.0422 2022.
10

---

Diffusion on Language Model Encodings for Protcin Sequence Generation
Fu, C., Yan, K., Wang, L., Au, W. Y., McThrow, M. C., Komikado, T., Maruhashi, K., Uchino, K., Qian, X., and Ji, S. A latent diffusion model for protein structure gen- eration. In Learning on Graphs Conference, pp. 29-1. PMLR, 2024.
for conditional protein backbone generation, 2024. URL
https:/ https://arxiv.orrgg http://aaaiii..or//a///////. 20313.
Ingraham. J. B., Baranov, M., Costello, Z., Barber, K. W., Wang, W., Ismail, A., Frappier, V., Lord, D. M., Ng- Thow-Hing, C., Van Vlack, E. R., al. Illuminating protein space with a programmable generative model. Nature, (23(7989):1070-1000 2023.
Gao, Z., Guo, J., Tan, X., Zhu, Y., Zhang, F., Bian, J., and Xu, L. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint 1rXiv:221..09411 2022.
Jones, P., Binns, D., Chang, H.-Y., Fraser, M., Li, W., McAn- M., ulla, C., McWilliam, H., Maslen, J., Mitchell, A., Nuka, G., et al. Interproscan 5: genome-scale protein function classification. Bioinformatics, 300)::33303 2014.
Gaujac, B., Donå, J., Copoiu, L., Atkinson, T., Pierrot, T., and Barrett, T. D. Learning the language of protein structure. arXiv preprint arXiv:2405. 2024.
Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. Dif- fuseq: Sequence sequence text generation with dif- fusion models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL.https: //openreview.nee n formmiid=jjj-_Liiv
Joshua Southern, Arne Schneuing, M. M. B. and Correia, Evaluation metries for protein structure generation. ICML, 12(1), June 2023. ISSN 2041-1723. doi: 10. 0tt1/2223.09.1155550 virtua1/2023/2897
URL https: /ttp///cmm..////////////eeeeeeee cc/
Gulrajani, I. and Hashimoto, T. Likelihood-bassd diffusion language models. In Thirty-seventth Conference on Neural Information Processing Systems, 2023. URL https: / //opeereeiew net. forum? demmmmdddd2220hhbb.mmm
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunkkool, K., Bates, R., Zídek, A., Potapenko. A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A. J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Sil- ver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein struc- ture prediction with AlphaFold. Nature, 596(7873):583- 589, August 2021. ISSN 1476-4687.
Han, X., Kumar, S., and Tsvetkov, Y. Ssd-Im: Semi- autoregressive simplex-based based diffusion language model for text generation and modular control. arXiv preprinl arXiv:2210. 17432, 2022.
Hesslow, D., Zanichelli, N., Notin, P., Poli, and Marks, D. Rita: study on scaling up generative protein sequence models. arXiv preprint arXiv:2205.055 2022.
doi: 10.1038/
s4 $41586-002-003819-22 URL https::tp/// petttt::::/www wwwur...uree... htt.cc/cessss1111 021 021---69------11222
Ho, J., Jain, A., and Abbeel, P. Denoising diffusson proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Kabsch, W. and Sander, C. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12)2577-2633 1983.
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454- 12465, 2021.
Karpathy, A. nanoGPT, 2023. URL https://githitt com Tom/kon httm/karpathy//nanccee
Hoogeboom, E., Heek, J., and Salimans, T. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301.1 /093, 2023.
Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. NV-embed: Improved techniques for training LLMs generalist embedding models. In The Thirteenth International Conference Learning Representations, 2025. URL https: https://pppeeviieeee net/ / forumorummmmmm wttpmmmmmmmmllllssmmmmmmm
Hu, F. J., yelong shen, Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. URL. https: //openreview net/ forum? 1d=
Lee, J. S., Kim, J., and Kim, P. M. Score- based generative modeling for de novo protein de- sign. Nature Computational Science, 3(5)3382-322, May 2023. ISSN 2662-8457. doi: 10.1038/ $43588- 22588-003300444-3, URL https: nature.
Huguet, G., Vuckovic, J., Fatras, K., Thibodcau-Laufer, E., Lemos, P. Islam, R., Liu, C.-H., Rector-Brooks, J., Akhound Akhound-Sadegh, T., Bronstein, M., Tong. A., and Bose, A. aequence-auumenttn sel3)-flow matching
com/articled 0tt.://eeeee//000000-------00000

---

Diffusion on Language Model Encodings for Protcin Sequence Generation
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 -444,,4333200 2022.
J. S., and Naik, N. Large language models gener- ate functional protein sequences across diverse fami- lies. Nature Biotechnology, 41(8):1099-1106, August 2023. ISSN 1087-0156, 1546-1696. doi: 10.1038/ s4 $41587-022-01618-22 URL https: http://www./tttt com/articies/s
Lin, Y. and AlQuraishi, M. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. arXiv preprint arXiv:2301.1248. 2023.
022 01618-2.
Mahabadi, R. K., Ivison, H., Tae, J., Henderson, J., Belt- agy, I., Peters, M. E., and Cohan, A. Tess: Text-to- text self-conditioned simplex diffusion. arXiv preprint arXiv:2305.088 083399, 2023.
Lin, Y., Lee, M., Zhang, Z., and AlQuraishi, M. Out of many, one: Designing and scaffolding proteins the scale of the structural universe with genie 2, may 2024. URL https://arxiv. org/abs/240555 15489, 2024.
Murzin, G., Brenner, E.,, Hubbard, T., and Chothia, Scop: structural classification of proteins database for the investigation of sequences and structures. Journal of molecular biology, 247(4) 1995.
Lin, Z. Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-Zarandi, M., Scrcu, T., Can- dido, S., and Rives, A. Evolutionary-scale prediction of atomic-level protein structure with language model. Science, 379(6637): 1110:1111113000 2023a. doi: 126/ science.ade2574. URL org/doi/abs 10. 1121 hate5/science.. ade2574.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Oates, M. E., Stahlhacke, J., Vavoulis, D. V., Smithers, B., Rackham, O. Sardar, A. J., Zaucha, J., Thurlby, N., Fang. H., and Gough. J. The superfamily 1.75 database in 2014: a doubling of data. Nucleic acids research, 43
Lin, Z., Gong, Y., Shen, Y., Wu, T., Fan, Z., Lin, C., Duan, N., and Chen, W. Text generation with diffusion language models: A pre-training approach with continuous para- graph denoise. In International Conference Machine Learning, pp. 21051 21051-21064 21064. PMLR, 2023b.
(D1):D227-D233, 2015.
Ovchinnikov, S. and Huang, P.-.. Structure-basse protein design with deep learning. Current Opinion in Chemical Biology, 65:136-144, 2021. ISSN 1367-
Lovelace, J., Kishore, V., Wan, C., Shekhtman, E. S., and Weinberger, K. Q. Latent diffusion for language gen- eration. In Thirty-seventh Conference on Neural In- formation Processing Systems, 2023. URL https: adR.
5931. doi: http:://www.scienceediie URL ialemm///attccle/p////1111131111
Mechanistic Biology Machine Learning in Chemical Biology.
//openreviews net
Lu, A. X., Yan, W., Yang, K. K., Gligorijeiic, V., Cho, K., Abbeel, P., Bonneau, R., and Frey, N. Tokenized and continuous embedding compressions of protein sequence and structure. bioRxiv, pp. 2024-08- 08, 2024.
T., Blum, M., Chuguransky, S., Grego, T., Pinto, L.. Salazar, G. A., Bileschi, M. L.,, Bork, P.,
Bridge, A., Colwell, L.,, et al. Interpro in 2022. Nucleic acids research, 51(D1):D418 (111))4D1---22 2023.
Lu, A. X., Yan, W., Robinson, S. A., Kelow, S., Yang, K. K., Gligorijevic, V., Cho, K., Bonneau, R., Abbeel, P., and Frey, N. C. All-atom protein generation with latent diffu- sion. bioRxiv, 2025. doi: 10.1101/2024. 1t.p:/2222.......2.22.2222 12.02.2206666 626353. URL https: www. bioooiiiiiiiioooo org/ contentt///////////////////.. early/20222 00000/221132222 12. 02. 626333.
Piovesan, D., Tabaro, F,, Paladin, L., Necci, M., Mičetić, I., Camilloni, C., Davey, N., Dosztányi, Z., Mészáros, B., Monzon, A. M., al. Mobidb 3.0: more annota- tions for intrinsic disorder, conformational diversity and interactions in proteins. Nucleic acids research, 46(D1): D471-D476, 2018.
Lv, L., Lin, Z., Li, H., Liu, Y., Cui, J., Chen, C. Y.C.,, Yuan, L., and Tian, Prollama: A protein large language model for multi-task protein language processing. IEEE Transactions on Artificial Intelligence, pp. 1-12, 2025. doi: 10. 11100TAA.2225535549999
Popov, V., Vovk, I., Gogoryan, V., Sadekova, T., and Kudi- nov, M. Grad-tts: A diffusion probabilistic model for text-to-speech. International Conference on Machine Learning, pp. 8599-8666 8608. PMLR, 2021.
Repecka, D., Jauniskis, V., Karpus, L., Rembeza, E., Rokaitis, I., Zrimec, J., Poviloniene, S., Laurynenas, A., Viknander, S., Abuajwa, W., Savolainen, O., Meskys, R., Engqvist, M. K. M., and Zelezniak, A. Expanding
Madani, A., Krause, B., Greene, E. R., Subrama- nian, S.. Mohr, B. P., Holton, J. M., Olmos, L., Xiong, C., Sun, Z. Z., Socher, R., Fraser,
12

---

Diffusion on Language Model Encodings for Protein Sequence Generation
functional protein sequence spaces using generative ad- versarial networks. Nature Machine Intelligence, 3(4): 324-333, March 2021. ISSN 2522-5839, doi: 10.1038/ $42256-021-0031004 URL https::/p//.t com/articlln / s42256- 0011222556022100110--02-
2021b. URL U net/ forum? id=PxTIG12RRHS.
Steinegger, M. and Söding, J. MMseqs2 enables sen- sitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, 35(11):1026- 1028, November 2017. ISSN 1546- 1696. doi: 10. 1038/nbt.3988, URL https: www. www.nnatuueoomm com/ articles/nbt 3988.
Ruff, K. M. and Pappu, R. V. Alphafold and implica- tions for intrinsically disordered proteins. Journal of Molecular Biology, 433(20):167208, 2021. ISSN 0022- 2836. doi: http:://oonooo///o.0000jjmmmmm00
URL https://www.seiiencddeeettt com/ actencc/arrlccee//////2222222222
Strudel, R., Tallec, C., Altché, F., Du, Y.. Ganin, Y., Men- sch, A., Grathwohl, W., Savinov, N., Dieleman, S., Sifre, L., et al. Self-conditioned embedding diffusion for text generation. arXiv preprini arXiv:22 2.0.22361... 2022.
From Protein Sequence to Structure at Warp Speed: How Alphafold Impacts Biology.
Salazar, J., Liang, D., Nguyen, T. Q. and Kirchhoff, K. Pseudolikelihood reranking with masked language models. CoRR, abs/1910.14655 2019. URL http:
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F. Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, pp. 2023-10, 2023.
//arxiv. wwooorgg//sa... abs///b////// awaaaaa99990 191...11....1.0
Tunyasuvunakool, K., Adler, J., Wu, Z., Green, T., Zielin- ski, M., Židek, A., Bridgland, A., Cowie, A., Meyer, C., Laydon, A., Velankar, S., Kleywegt, G. Bateman, A.. Evans, R., Pritzel, A., Figurnov, M., Ronneberger, O., Bates, R., Kohl, S. A. A., Potapenko, A., Ballard, A. J., Romera-Paredes, B., Nikolov, S., Jain, R., Clancy, E., Reiman, D., Petersen, S., Senior, A. W., Kavukcuoglu, K., Birney, E., Kohli, P., Jumper, J., and Hassabis, D. Highly accurate protein structure prediction for the hu- man proteome. Nature, 596(7873):590-598 August 2021. ISSN 0028-0836, 1476-4687. doi: 10.1038/ $+1586-021-03828-11 URL https: htwp:///.w//......./e...m
Shabalin, A., Meshchaninov, V., Chimbulatov, E., Lapikov, V., Kim, R., Bartosh, G., Molchanov, D., Markov, S., and Vetrov, D. Tencdm: Understanding the properties of the diffusion model in the space of language model encod- ings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25110-25118, 2025.
Shin, J.-E., Ricsselman, A. J., Kollasch, A. W., McMa- hon, C. Simon, E., Sander, C., Manglik, A., Kruse, A. C., and Marks, D. S. Protein design and variant prediction using autoregressive genera- tive models. Nature Communications, 12(1):2403, April 2021. ISSN 2041-1723. doi: 10.1038/ s41467-021-227322w httpstss:/ www..aatur. com/articlee//4411 0111444447702122272222
Uversky, V. N. Functional roles of transiently and intrin- sically disordered regions within proteins. The FEBS journal, 282(7):1182-1188 2015.
Shukla, V. K., Siemons, L., and Hansen, D. F. Intrin- sic structural dynamics dictate enzymatic activity and inhibition. Proceedings of the National Academy of Sciences, 120(41)::231091011 0027-8424, 1091-6490. doi: 10. 13.10773/pnas.231110000
Van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L. M.. Söding, J., and Steinegger, M. Fast and accurate protein structure search with Foldseek. Nature Biotechnology, May 2023. ISSN 1087-0156, 1546-1696. $41587-023-0177730, URL https: http://wwww....eeeooo//.////..ee ///es///411590 0208877-233000 -
October 2023. ISSN
URL https: /ttp:///aaaaa//e wwwg///oi..... 100.////111011///1// 10a3/pnas. 2310910120.
doi: 10.1038/
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International conference on machine learning, pp. 2256-2265, PMLR, 2015.
Wang, X., Zheng, Z., YE, F., Xue, D., Huang, S., and Gu, Q. Diffusion language models are versatile protein learn- ers. In Forty-first International Conference on Machine Learning, 2024. https: opwprevienreeeeeeeeeeeeeee net/ forum? forum?id-ail Nuwbnndd-NUAbFUUU qyqb.
Song, J., Meng, C., and Ermon, S. Denoising diffu- sion implicit models. In International Conference Learning Representations, 202 URL https:// openreview Spenreview.net/Eorumi ddw.mmiiistts111....1
Wang, X., Zheng, Z., YE, F,, Xue, D., Huang, S., and Gu, Q. DPLM-2: A multimodal diffusion protein lan- guage model. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: // //openrviees net / forum? dG
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- cling through stochastic differential equations. In In- ternational Conference on Learning Representations,
13

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahem, W., Borst, A. J., Ragotte, R. J., Milles, L. F., Wicky, B. I. M., Hanikel, N., Pellock, S. J., Courbet, A., Sheffler, W. Wang, J., Venkatesh, P. Sappington, I., Torres, S. V., Lauko, A., De Bor- toli, V., Mathieu, E., Ovchinnikov, S., Barzilay, R., Jaakkola, S., DiMaio, F., Baek, M., and Baker, D. De novo design of protein structure and function with RFdiffusion. Nature, 620(7976):108- 1100, August 2023. ISSN 0028-0836, 1476-4687. doi: 10.1038/ $415866-023-06415--8 URL httpp:://ww..tttttt/t /wtp://www......mm.///////./ com//rticces///// 0ew.:4448888022006--- 06415-8.
Zhang, S., Jiang, Z., Huang, R., Mo. S., Zhu, L., Li, P., Zhang, Z.,, Pan, E., Chen, X., Long, Y., al. Pro-Idm: Protein sequence generation with a conditional latent diffusion model. bioRxiv, pp. 2023-08, 2023a.
Zhang, Y. and Skolnick, J. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformaties, 57(4):702-710,
December 2004. ISSN 0887-3585, 1097-0134. doi: 10. 1002/prot.20264. URL I wiley.come doi pttp:/.....////p..22222255
Zhang, Y., Gu, J., Wu, Z., Zhai, S., Susskind, J. M., and Jaitly, N. PLANNER: Generating diversified paragraph via latent language diffusion model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. httst:p:: /ttps/////ppeeeieeeeeemm net/ forum? id=SINyBUVSBY.
Wu, K. E., Yang, K. K., den Berg, R., Alamdari, S., Zou, J. Y., Lu, A. X., and Amini, A. P. Protein structure generation via folding diffusion. Nature Communicationss 15(1):1059, Feb 2024. ISSN 2041-1723. doi: 10.1038/ $41467-024-45051-2. URL https:: /tt//di..o/// /rg....o.o.... 1038/
Wu, Z., Johnston, K. E., Amold, F. H., and Yang, K. K. Protein sequence design with deep gen- crative models. Current Opinion Chemical Biology, 65:18-27, 2021. ISSN 1367-5931.
http:///oioo///10000i..hh......0 com/ URL https:
doi:
Mechanistic Biology Machine Learning in Chemical Biology.
Ye, J., Zheng, Z., Bao, Y., Qian, L., and Wang, M. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.1003 2023.
Yim, J., Trippe, B. L., Bortoli, V. D., Mathieu, E., Doucet, A., Barzilay, R., and Jaakkola, T. Se(3) diffusion model with application to protein backbone generation, 2023a. URL https:://// /awxiv.iii..///c/ org/ abs/ 2302. 02277.
Yim, J., Trippe, B. L.,, De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R., and Jaakkola, T. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277 2023b.
Yim, J.. Campbell, A., Mathieu, E., Foong, A. Y. K., Gastegger, M., Jimenez-Luna, J., Lewis, S., Sator- ras, V. G.. Veeling, B. S., Noe, F. Barzilay, R., and Jaakkola, T. Improved motif-scaafoldiing with SE(3) flow matching. Transactions on Machine Learning Re- search, 2024. ISSN 2835-8856. URL https:// openreview met/foorumiiii wtt.forrmmmmaaaaae0aaaaaa
Yuan, H., Yuan, Z., Tan, C., Huang, F., and Huang, S. Seqdif- fuseq: Text diffusion with encoder-decoder transformers. arXiv preprint arXiv:22/2. 2022.
14

---

Appendix
Datasets
16
Evaluation Metrics
16
B.1
Quality
16
B.2
Diversity
17
B.3
Distribution similarity
18
B.4
Novelty
20
Model Details
21
C.1 Model Architecture
21
C.2 Training Details
21
C,3 Length Determination
21
C.4 Parameters for Navigating the
Trade-off
22
C.5 The Rationale Behind the tan-10 Schedule
22
Related Work
24
Additional results
25
E.I
Component Analysis and Architectural Optimization on SwissProt Dataset
25
E.2
Comparison Across Generative Paradigms on SwissProt Dataset
25
E.3
Representation Space Scaling
................................
25
Comparison with Pre-trained Protein Models
................................
28
E.5
Protein Family Controllable Generation
30
................................
E.6
Joint Sequence-Struccure Generation with DiMA
33
E.7
Sequence Infilling
34
Functional-moti Scaffolding
35
E.9
Fold-Conditioned Generation
36
Biological Relevance Analysis
38
15

---

Diffusion on Language Model Encodings for Protcin Sequence Generation
A. Datasets
SwissProt is a dataset that contains a high-quality, manually annotated subset of the UniProt (Consortium, 2020) database. This dataset is small enough and good enough for proof-of-the mroo-offthhe-coneye the-connept studies. After filtering out all sequences shorter than 128 and trimming all sequences longer than 254, we ended up with 470k sequences. MMseqs2 clustering of this dataset (>500 sequence identity and >80% sequence overlap) reveals the presence of clusters of similar sequences with the maximum number of sequences in cluster equal to 1570. Each of those clusters comprises sequences that belong to single protein family. For example, the most populous cluster is 1570 protein sequences of cytochrome of different species, a very abundant protein involved in electron transport in eukaryotic cells. Around 120k sequences do not form clusters under the conditions used.
Another dataset we use is AFDBv4-99 -0 from Durairaj et (2023), a subset of the UniRef50 database. The sequences in this dataset obey two conditions: 1. The sequence identity between all members no more than 50% and The average predicted pLDDT by AlphaFold is no less than 90. After filtering out all sequences shorter than 128 and longer than 254, we ended up with 2.2 million whole sequences of highly diverse proteins of high quality.
B. Evaluation Metrics
B.1. Quality
PLDDT. To assess the foldability of our generated sequences, we utilize ESMfold to predict the three-dimensional structure of the given protein sequence. For each amino acid within the predicted structure, ESMfold provides pLDDT score, which represents the confidence of the model in the predicted positions of amino acids in the 3D structure. We average these PLDDT scores for all amino acids in the sequence to gauge the overall confidence in the predicted protein structure. It is worth noting that, while higher average PLDDT scores indicate reliable structure prediction, lower scores may not necessarily denote poor prediction. In some cases, they can also signify the presence of intrinsically disordered regions in the protein, segments that are inherently flexible and do not conform to a fixed structure but still play vital roles in protein functionality (Ruff& & Pappu, 2021; Shukla et al., 2023).
ProGen perplexity To assess how probable the generated sequences we utilize the ProGen2-base (Madani et al., 2023) model of 764M parameters estimate perplexity.
1
PrroGen = exp
log p(s;)((Ses)) OproGen-bael
(3)
S
ESM-2 pseudoperplexity. To assess how probable the original sequence is under the model's distribution, we used pseudoperplexity (Salazar et al., 2019) using ESM-2 650M encoder transformer-based language model (Lin al., 2023a). Each token (amino acid) in the sequence masked and then predicted, considering all other tokens in the sequence. The final pseudoperplexity value is aggregated using the following equation:
|s|
PESM-2(S) = exp
log pl 8: Su 0ESM-2)
(4)
S
Here, PESM-2(S) represents the pseudoperplexity of sequence S. |S| denotes the length of sequence S, s; is the i-th token in the sequence, Syy represents the sequence without the i-th token, and O F denotes the parameters of the ESM-2 model.
To evaluate the structural relevance of the generated sequences, turned to the TM-score (Zhang & Skolnick, 2004), widely recognized metric for evaluating structural similarity between protein pairs. The TM-score measures the similarity between two protein structures and helps distinguish proteins with similar fold from those with different folds. Unlike many other metrics for 3D-alignment, it does not depend on protein size and always ranges between and 1, where a TM-score above 0.5 indicates a similar fold in structure. The TM-score is given by:
TM-score.
Laguns
1
TM-score
(5)
Liargtt im
1
d dol Lounsn
16

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Here, Inager is the length of the target protein, Liqurry is the number of aligned residues between the two proteins, d; is the distance between the i-th aligned residue pairs, and do is scaling factor to normalize the length difference between query and target proteins. To calculate TM-scores for each sample of generated sequences, we first obtained their 3D structures using ESMFold. For each of these structures, we have found the closest natural protein in the SwissProt and AFDBv4-90 datasets from the AlphaFold Database ( (Tunyasuvunakool et al., 2021) using the FoldSeek (Van Kempen et al., 2023).
BLAST Identity. For each sequence, we ran BLAST with specific parameters (e-value = 0.05 and BLOSUM62 substitution matrix) to identify similar sequences within the training dataset. The number of matching amino acids between the generated sequence and the most identical sequence found in training data was normalized by sequence length and multiplied by 100 to obtain percentages. The BLAST identity metric is the average over batch of 2048 sequences.
B.2. Diversity
Rep. Rep quantifies the internal diversity of gen- erated sequences by assessing the prevalence of re- peated subsequences, it is calculated as Rep(y) = 1
Table 4. Review of the metrics across modalities for evaluating gen- eration quality, diversity, novelty, and distribution similarity.
gen-
[m (46,6666632664
[# unique a-subseq in 21, where y is a set of of n-subseq g) generated proteins. n-subseq means the subsequence of consecutive amino acids of length n.
Sequence
Structure
FD-seq OT-seq
FD-struct
Distributional
CD. To evaluate the model's capacity to generate distinct protein variants while avoiding redundant outputs we em- play the clustering density metric (C),) at two sequence identity thresholds: t = %50 and t = 95%. CD₁ repre- sents the ratio of sequence clusters at threshold t to the to- tal number of generated proteins. Therefore, CDD, ranges from 0 to where indicates that all sequences form individual clusters and the sample is diverse. CDo.5 is an established metric for assessing broad sequence diver- sity (Consortium, 2020), analogous to the widely-adopted TM-score threshold of 0.5 used in structure generation (Yim et al., 2023a). We employ MMseqs2 (Steinegger & Söding, 2017) to perform sequence-based clustering at given thresholds (coverage = 0.8, cov-mode = O, cluster-mode = 1.). While clustering at a moderate threshold (50%) reveals the model's s ability to generate diverse proteins, individual clusters may still contain nearly identical sequences an undesirable characteristic for generative models. Therefore, we complement our analysis with Do.99,, which specifically identifies near-duplicate sequences. This dual-threshold approach provides a more comprehensive assessment of sequence diversity compared to single-metric evaluations.
similarity
OT-struct
MMD-seg
MMD-struct
ProGen-2 ppl
pLDDT
ESM-2 pppl
TM-score
Quality
BLAST
scPerplexity
scPerplexity
Rep CD
Diversity
Novelty
Novelty
PCD and NCD. While CD2 can capture mode collapse in a batch of sequences. it also highly rates random sequences. To evaluate the degree of novelty of the generated sequences we perform co-clustering analysis of generated sequences with the dataset sequences using MMseqs2 (identity = 0.5, coverage 0.8, cov-mode = O, cluster-mode 1). This analysis yields two metrics: DCD and NCDo.s. representing the ratios of "positive" clusters (PC, containing both generated and dataset sequences) and "negative" clusters (NC, containing only generated sequences) to the total number of sequences, respectively. The desired values of PCDo.s and NCD should be close to reference ones.
Notably, that generation out of distribution is also very important, so we evaluate the quality of generated sequences from other ("negative") clusters. We found that the average PLDDT of these sequences from DiMA (SwissProt and AFDBv4-90 90: 65 +14 and 63 +12, respectively). which is significantly higher than that of other models (nanoGPT: SwissProt and AFDBv4-90 43 +12 and 52 +16). This indicates that the model generalizes beyond the training data.
UMAP. To visually represent the distribution of generated sequences across PC, we trained UMAP on all sequences from PC for all models (parameters- neighbors- 25 and min_dict- 0.5). The UMAP plots in Figures 6 and 7 show that despite the fact that the diversity metrisc of the DIMA w/o celf-conditioning are higher, DIMA with self-condition has the same coverage on the SwissProt (and even more coverage on AFDBv4-90) This and the fact that DIMA is closer to the dataset in terms of distribution learning metrics shows that DIMA w/o self condition achieved better diversity by generating sequences that greatly differ from those from the dataset.
17

---

Diffusion on Language Model Encodings for Protein Sequence Generation
68
DiMA
DiMA SC
Dataset
nanoGPT
EvoDiff
SeqDesign
Figure UMAP projection sequences from PC, Training dataset SwissProt. Grey background points datasct sequences from PC.
DIMA
DiMA no SC
Dataset
nanoGPT
Figure 7. UMAP projection of sequences from PC. Training dataset AFDBy4-90. Grey background points dataset sequences from PC.
B.3. Distribution similarity
Fréchet ProtT5 Distance (FD-seq) and Fréchet ProteinMPNN Distance (FD-struct). The Fréchet distance, also known as the 2-ssssessniinn distance, quantifies the dissimilarity between samples drawn from multivariate Gaussian distributions, denoted as X2 N(4, E1) and X2 (N2,, Σz). and is defined as follows:
d(Xi, X2) || 22ll2 + Tr(E, + E₂ 2 Σ Σ2)
(6)
Maximum mean discrepancy (MMD). The idea behind MMD involves assessing the distance between two samples by measuring the difference in mean values resulting from applying smooth function to the samples. A biased empirical estimate of MMD between two samples X {I1, Ind and Y (yI...., "...,.. using kernel k is defined follows:
MMDi(XXY)
(k(2,, + k(y, y1) - 2k(x, (1(x)))))
As a kernel function, we used the radial basis function kernel (RBF). We evaluated the distance between batches of sequences, each of size n equal to 2048, sampled from the dataset and generated by the respective models. Following the methodology
18

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Dataset
casine sheduler
Dataset
linear sheduler
0.7
sc
DIMA
FM
DIMA
0.6
DFM
no encoder
DPLM
conditioning
0.6
MoDDiI--DADM
skip connections no time layers
Random
0.5
nanoGPT
0.4
0.5
proteinGAN
decoder
Densty
0.4
Random
Denetry
0.3
no length sampling
Density
0.4
SegDesign
0.3
Walk Walk-jump
0.2
0.2
0.1
0.1
0.1
Figure The distribution optimal transport distances between pairs of generated dataset sequences. For each model, we compute pairwise Levenshtein distances between generated sequences and datasct sequences, then find optimal matching pairs using Earth Mover Distance with uniform distribution of samples. Left: Comparison DiMA against baseline models on SwissProt dataset. Center: Analysis of DiMA's architectural components through ablation studies. Right: Performance comparison on the larger AFDBx4-90 dataset. The dashed blue line represents the reference distribution obtained by matching samples within the dataset (optimal transport distance to itself), while the dashed orange line shows DiMA's 's distribution.
proposed for 3D structures in (Joshua Southern & Correia, 2023), we utilized ProtT'S sequence representations to calculate MMD.
1- I-Wasserstein optimal transport (OT). The BLAST identity metric effectively evaluates the similarity between generated and natural sequences. However, its limitation lies in assessing the model's capability to produce diverse sequences, it may identify the same dataset sequence as the closest match for every generated sequence. To overcome this limitation, we employ transportation theory to establish optimal pairs between generated sequences and the dataset.
Optimal transport theory, initially devised for solving economic problems, has found applications in various fields, including physies, biology, and tomography. To implement our approach, we calculate pairwise Levenshtein distances and use them as transportation costs. Subsequently, we determine optimal sequence pairs using the Earth Mover Distance (EMD) solver with uniform distribution of the samples. We use the average distance between these optimal pairs, measuring both the diversity and proximity of generated samples to the dataset.
The inherent diversity of the dataset, i.e., when a sample from the dataset pairs with itself, gives zero distances (datdaasset) = 0). contrast, random sequences form optimal pairs with the highest mean distances, as illustrated in Figure The optimal transport distance distributions reveal differences in how models capture the protein sequence space. Most ablation studies (Figure 8, center) show distributions similar the reference, except for flow matching, cosine scheduler, and no encoder variants, indicating these components most critical for DiMA's performance. Several baseline models (D3PM, DFM, EvoDiff-OADM, RITA) cluster around similar mode between random and reference distributions, suggesting they mainly learn basic patterns like amino acid frequencies while capturing only limited set of protein families, as evidenced by their left tail behavior (Figure 8, left).
DiMA's distribution (dashed orange) most closely matches the dataset reference (dashed blue) across both datasets. On SwissProt, DPLM shows sharp, concentrated peak indicating high-quality but limited diversity, while other baselines show broader, right-shifted distributions indicating greater deviation from natural sequences. On the larger AFDBv4-90 dataset, while nanoGPT's distribution mode is closer to the reference, DiMA generates fewer distant proteins (smaller right tail) and better maintains the overall distribution shape, demonstrating robust performance even with increased dataset complexity (Figure 8 right).
Although our OT implementation offers advantages over BLAST, it has a special feature: the EMD solver identifies an exact pair for each sequence. This poses a challenge when dealing with two query sequences that are similar to one dataset sequence but distant from others, resulting in one close pair and one distant pair. However, we employ EMD precisely to
19

---

Diffusion on Language Model Encodings for Protein Sequence Generation
penalize such cases, reinforcing the generation of diverse rather than similar sequences.
Structural analogues. To measure structural distribution similarity, we calculate analogous FD, MMD, OT metrics using structural encoder ProteinMPNN. ProteinMPNN is a powerful graph neural network (GNN) model pretrained on massive dataset of protein structures.
B.4. Novelty
To directly evaluate the potential memorization of the training data, we measure novelty by calculating the mean sequence identity between each generated sequence and its nearest neighbor in the training dataset.
We assume that novel proteins should be far from the train dataset, so for each generated sequence, we computed distance to the nearest train sequence. The golden standard for pairwise distance measure between amino acid sequences is an aligmment score using Needleman- -unnshh (NW) algorithm. However, due to O calculation cost we use BLAST to find the nearest sequence in the training set and only then align these sequences using NW. (We employ BLAST and NW with the following parameters: evalue = 15.05, matrix = BLOSUM62, word_size- 2; matrix= BLOSUM62. gap open= -00, gap extend-0.5). The novelty value of a batch of generated sequences is defined as Novelty(y) = where y of generated proteins
| of same letters in alignment]
aligoment length
20

---

Diffusion on Language Model Encodings for Protein Sequence Generation
C. Model Details
C.1. Model Architecture
We employ a 12-layer Transformer model with 16 atten- tion heads and a hidden size of 320 as the backbone for diffusion model, incorporating several modifications specifically designed optimize the denoising diffusion process in the context of protein-related data. To enhance the model's performance, we first introduce trainable po- sitional encodings to the noisy protein latents, allowing the model to better capture the sequential nature of the data, The input for each transformer block constructed as sum of the output from the previous block, time embeddings, and self-condition predictions, which are projected through linear layers. This approach facilitates the integration of temporal information and improves the model's 's ability to learn complex patterns. Additionally, we implement long skip connections, recognizing that for time steps close zero, the model's output closely resembles the input. This modification is crucial aids in learning identity transformation, thereby stabilizing the training process and enhancing the model's overall efficacy. The architecture of our model is illustrated in Figure 9.
+
2
Linear
Transformer Tllack
Transformer Block
Long skip
connoctions
Transformer Block
Transformer Block
C.2. Training Details
Simusoidal embeddings
Positional Ficodding
All models were trained with batch size of 512 and learning rate of 1e- to convergence. We clip our gradient norm to 2 and have linear warmup schedule for the first 5000 iterations. We also use a 0.9999 EMA.
Noizy
Saz
The experiments were conducted using 4 A100 80GB GPUs. Each training session lasts approximately 10 days
Figure 9, The architecture of the denoising model.
C.3. Length Determination
Length distribution on AFDB
800
Generated
700
Train
600
500
400
300
200
100
75
100
125
150
175
200
225
250
Figure 10.
The distribution of lengths in the training and generated sets for models trained on AFDB.
21

---

The key issue with standard noise schedules (linear, cosine) is that they corrupt data very gradually at small timesteps (Figure 2). This means the model spends significant training resources on nearly trivial denoising tasks where input and target are almost identical. Our approach ensures the difficulty of the denoising task increases steadily with each timestep. By designing a noise schedule where reconstruction loss grows linearly with diffusion time, we create learning problems of Diffusion on Language Model Encodings for Protein Sequence Generation
During inference, the model must determine the length of the generated sequence. We explore strategies address this challenge: training diffusion models with and without padding masks.
Diffusion with padding masks: During training, we provide the attention mask for pad tokens along with the cornupted latents and exclude pad tokens when computing the diffusion loss. At inference time, the sequence length is sampled from the empirical length distribution observed in the training set.
Diffusion without padding masks: In this setting, no explicit information about padding is provided during training, and the loss is computed over the entire sequence, including pad tokens. Consequently, during inference, the model must implicitly infer the appropriate sequence length without external guidance.
Figure 10 compares the length distributions of sequences from the training set and those generated using the second approach. Notably, the generated sequences exhibit distribution shift, deviating from the expected length profile of the training data.
To mitigate this mismatch, adopt the invention-masked approach, incorporating explicit length information during training and sampling lengths during inference to ensure closer alignment with the training distribution.
C.4. Parameters for Navigating the
Trade-off'
90
Natural
Natural
1.00
Natural
1.00
Natural
0.90
0.90
0 0.97
0.97
0.83
0.85
60
66
Number Steps
Figure 11. Effect of the number of sampling steps on DiMA-35M generation quality. Increasing the number of diffusion steps from 250 to 2000 shows a trade-off' between quality and diversity. Top: structural quality (pLDDTT) improves with more sampling steps, approaching natural protein level (gray dashed line). Middle: se- quence diversity (CDo.s) slightly decreases with more steps but remains consistently high (>0.97). Bottom: novelty decreases with increased sampling steps, indicating convergence toward more con- servative protein designs. Each point represents evaluation 2048 protein sequences generated with specified number of steps.
Figure 12. Effect of self-conditioning rate parameter (w) on DiMA- 35M performance metrics. In self-conditioned generation zo fig + w * (æo - foo). the parameter w controls how strongly the model relies its own previous predictions during the diffusion process. Top: structural quality (pI.DDT) increases with higher values, approaching natural protein levels (gray dashed line) > 0.9. Middle:
sequence diversity (CDo.s) decreases as w increases, with noticeable reduction follows similar trend, showing higher values at lower Each point represents evaluation on 2048 protein sequences generated with 2000 sampling steps and the specified w value.
1.0. Bottom: novelty
C.5. The Rationale Behind the tan-10 Schedule
22

---

Diffusion on Language Model Encodings for Protein Sequence Generation
incrementally increasing difficulty, allowing the model make consistent progress throughout training rather than facing an abrupt jump in task complexity.
When working with protein encodings, reconstruction at small noise levels turns out to be quite robust. For example, testing DiMA with CHEAP representations (which enables dual-decoding into both sequence and structure) we observe that at t == 0.05, sequence reconstruction accuracy remains 100%, and structural RMSD stays below 0.2.A (Figures 13 and 14). This shows that unlike for example pixel-based image diffusion models, DiMA does not have to spend much effort at small noise steps learn the fine- mne-grained denoising capability. Our schedule leverages this robustness by allocating more training challenging noise levels rather than the nearly lossless stages with 0.
033
Accuracy
05
6.0
Figure Structural stability of protein representations under noise. Root Mean Square Deviation (RMSD) between original protein structure structure reconstructed from noisy CHEAP represen- tations at different diffusion timesteps.
Figure Sequence reconstruction accuracy noise. Percent- age of amino acids correctly reconstructed from noisy CHEAP representations at different diffusion timesteps.
23

---

Diffusion on Language Model Encodings for Protein Sequence Generation
D. Related Work
Diffusion generative models, introduced by Sohl-Dickstein et al. (2015), have gained attention for their remarkable results in image (Ho et al., 2020; Song al., 2021b;a), and speech generation (Chen al., 2021; Popov al., 2021). Due to their impressive generative quality, some studies have extended the application of diffusion models the text domain. Hoogeboom et al. (2021) and Austin al. (2021) proposed multinomial diffusion for discrete data corruption. Subsequently, other works (Li et al., 2022; Lin et al., 2023b; Gulrajani & Hashimoto, 2023; Han et al., 2022; Strudel et al., 2022; Gao et al., 2022: Shabalin et al., 2025) adapted Gaussian diffusion to sequence learning by embedding discrete data into continuous space. Yuan et al. (2022) extended the text diffusion model to the sequence-to--equeee setting. Ye et al. (2023) conducted a study on the discrepancy of the text embedding space, demonstrating that the diffusion task at small noise scales is trivial. Zhang al. (2023b) implemented latent text diffusion inside a VAE with an autoregressive decoder. Lovelace et al. (2023) utilized diffusion models to generate fixed-length latent representation, mapped into high-dimensional space with the reconstruction network before being fed into autoregressive decoder to generate text.
In protein science, deep learning has emerged transformative tool. Pre-trained on extensive protein sequence datasets, it provides representations widely employed in various tasks (EInaggar et al., 2022; Lin et al., 2023a; Lu et al., 2024), Generative models for protein sequences, exemplified by recent advancements, enhance predictions of proteins with improved properties and functions (Wu et al., 2021; Ovchinnikov & Huang, 2021). Simultancously, progress in the loquence-to-strutuue domain, as seen in models like AlphaFold (Jumper et al., 2021) and ESMFold (Lin et al., 2023a), enables the prediction of 3D protein conformation from amino acid sequences. Models such ProteinMPNN (Dauparas et al., 2022) or ESM-IFI (Chaudhuri et al., 2022) predict an amino acid sequence given specific 3D structure, effectively reverse engineering the process.
In the realm of protein generation, a diverse array of autoregressive models has been developed, establishing sophisticated baseline for subsequent model classes (Madani et al., 2023; Ferruz et al., 2022; Shin et al., 2021; Ly et al., 2025; Hesslow et al., 2022; Shin et al., 2021). Beyond autoregressive approaches, both categorical and continuous diffusion methods have emerged as promising techniques for sequence generation (Alamdari al., 2023; Wang al., 2024: Lee et al., 2023; Zhang al., 2023a). Additionally, three-dimensional diffusion models have been successfully utilized for the generation of protein structures (Watson et al., 2023; Wu et al., 2024; Lin & AlQuraishi, 2023; Fu et al., 2024). Notably, there models that facilitate the simultaneous generation of both sequence and structure, providing a more integrated approach to protein design (Campbell et al., 2024; Ingraham et al., 2023). Furthermore, the field has seen the introduction of energy-based models (Frey et al., 2024) and generative adversarial networks (GANs) (Repecka et al., 2021), which offer alternative frameworks for protein generation.
24

---

Diffusion on Language Model Encodings for Protein Sequence Generation
E. Additional results
E.1. Component Analysis and Architectural Optimization on SwissProt Dataset
In this section, we conduct a detailed ablation study to as- sess the impact of key architectural components DiMA, trained the SwissProt dataset with the lightweight ESM-8M encoder. Our goal is evaluate how different design choices influence generation quality, distribution alignment, novelty, and diversity.
0.32
83
82
0.30
81
80
00.8
6
To achieve this, we compare multiple model variants across a comprehensive set of metrics, ensuring a rigorous analysis of their ability to generate high-quality, diverse, and novel proteins while maintaining fidelity real pro- tein distributions. All models were evaluated using 250 diffusion steps to ensure consistency in generation. The results are summarized in Table 5. This study ultimately aims to refine DiMA's capacity for robust and biologically meaningful protein generation.
79
0..6
Rep
0
78
77
00.2
Q
76
75
0.22
74
0..0
73
72
0.18
50
100
250 500 1000 2000
The proposed model also establishes a trade-off between structural plausibility and diversity illustrated in Fig- ure 15. Increasing the number of diffusion generation steps in protein production leads to higher protein quality; however, this improvement is accompanied by a slight decrease in protein diversity. This trade-off between qual- ity and diversity provides ficxibility during generation, allowing for the selection of proteins based on desired characteristics.
# generation steps
Figure 15. The dependence between the structural plausibility of generation, the degrce of repetition, and the number of generation steps.
E.2. Comparison Across Generative Paradigms on SwissProt Dataset
In this section, we present comprehensive evaluation diverse generative paradigms for protein sequence design, including latent Gaussian diffusion (DiMA), autoregressive models, score-based models, GANs, and discrete diffusion models. All models trained on the SwissProt dataset ensure consistency and comparability.
Our primary objective is to investigate how these generative approaches impact key aspects of protein generation, such as quality, distribution alignment, novelty, and diversity. By analyzing these dimensions, we aim to uncover the strengths and limitations each paradigm in capturing the complexity of protein sequences.
To achieve this, we employ a robust set of evaluation metrics, enabling rigorous and multidimenssionl comparison the models' performance. The results of this evaluation are summarized in Table 6.
E.3. Representation Space Scaling
In this section, we explore the impact of encoder choice on the performance of latent diffusion models for protein sequence generation. All models in this analysis are contigured with approximately 35 million parameters and are trained on the AFDBv4-90 dataset ensure consistency and comparability. The results of this analysis, including detailed performance metrics and comparisons, are presented in Table 7, providing valuable insights into the role of encoder design in latent diffusion models.
CHEAP Architecture CHEAP (Lu et al., 2024) is hierarchical representation learning framework for proteins, composed of six key components:
1. ESM-3B Encoder - Aggregates outputs from multiple transformer blocks into a sequence of 1024-dimensional embeddings, maintaining the length of the amino acid sequence.
25

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Table Ablation study evaluating key components of DiMA trained on the SwissProt dataset using the ESM-&M encoder. The study examines the impact architectural choices from multiple perspectives, employing comprehensive set metrics assess generation quality, distribution alignment, novelty, and diversity of the produced proteins. All models were evaluated using 250 diffusion steps during
generation.
Model
PLDDT (1)
Progen ppl ())
ESM-2 pppl (1)
scPpl (1)
TM-score
BLAST (1)
Dataset
80.7
6.03
5.35
1.88
0.80
100
Random sequences
24.8
21.91
21.53
2.77
0.33
0
DIMA
5.07
4.68
1.17
w/o skip connections
77.3
6.79
5.84
1.87
0.82
61
w/o time layers
79.4
6.42
5.49
1.83
0.85
66
Quality
w/o ESM encoder
62.7
10.42
9.22
2.09
0.71
48
w/o self-conditionin
68.2
10.45
9.18
2.08
0.74
46
w/o finetuned decoder
80.1
6.66
5.59
1.78
0.85
65
w/o length sampling
65.0
11.36
9.84
2.12
0.72
44
w linear schedule
77.0
7.66
6.29
1.89
0.82
58
cosine schedule
54.1
13.11
10.86
2.16
0.60
34
flow-matching
63.4
11.44
8.97
2.08
0.68
40
FD-seq (1)
MMDD-eq (1)
OT-seq
FD-struct (J)
MMD-struel (1)
OT-struct (1)
Dataset
0.13
0.000
1.08
0.000
0.000
0.053
Random sequences
3.97
0.200
3.88
1.231
0.412
1.313
Distributional Similarity
DiMA
0.34
0.020
1.26
0.030
0.014
0.090
w/o skip connections
0.45
0.021
1.36
0.029
0.002
0.081
w/o time layers
0.41
0,022
1.29
0.035
0.004
0.097
w/o ESM encoder
1.07
0,068
2.04
0.069
0.010
0.153
w/o self-conditoniin
0.55
0,047
1.51
0.031
0.005
0,093
w/o finetuned decoder
0.54
0,031
1.44
0.042
0.004
0.108
w/o length sampling
0.67
0.058
1.67
0.048
0.007
0.139
linear schedule
0.47
0,026
1.37
0.031
0.003
0.092
cosine schedule
0.94
0.091
1.90
0.122
0.019
0.215
flow-mn matching
0.71
0,063
1.75
0.049
0.008
0.130
Rep (1)
CDo.s (f)
CDp.95 (t)
PCDo.5
NCDo.5
Novelty (1)
Dataset
0.045
1,000
0.943
0.990
0.304
25.3
Random sequences
0.000
1,000
1.000
1.000
0.000
85.1
Diversity and Novelty
DiMA
0.320
0.611
0.992
0.246
0.392
35.7
w/o skip connections
0.274
0.619
0,990
0.187
0.439
43.1
w/o time layers
0.256
0.550
1.000
0.246
0.347
38.4
w/o ESM encoder
0.346
0.619
1.000
0.107
0,507
50.0
w/o self-conditioning
0.043
0,929
1.000
0.146
0,779
56.6
w/o finetuned decoder
0.266
0,589
0.996
0.255
0,357
38.4
w/o length sampling
0.050
0,880
1.000
0.089
0.726
58.4
linear schedule
0.208
0.611
1.000
0.181
0.431
45.8
cosine schedule
0.046
0.878
1.000
0.017
0.798
67.0
flow Now-matching
0.041
0.960
1.000
0.214
0.945
72.2
26

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Table Performance comparison of different generative paradigms for protein sequence generation, each trained from scratch with approximately 35M parameters on the SwissProt dataset. The comparison assesses generation quality, distribution alignment, novelty, and diversity.
Model
PLDDT (1)
Progen ppl (J)
ESM-2 pppl (1)
scPpl (1)
TM-score (1)
BLAST (1)
Dataset
80.7
6.03
5.35
1.88
0.80
100
Random sequences
24.8
21.91
21.53
2.77
0.33
0
Walk-Jump
32.4
15.47
14.72
2.41
0.35
1
RITA
43.9
14,99
13.77
2.36
0.48
28
proteinGAN
30.4
17.58
16.48
2.57
0.00
0
Quality
SeqDesign
43.1
12.78
11.89
2.35
0.41
17
EvoDiff-OADM
37.1
16.42
15.77
2.44
0.42
12
D3PM
36.7
16.83
16.52
2.36
0.48
9
DFM
16.48
15.25
2.44
0.40
DPLM
84.0
3,57
3,50
1.68
0.93
88
nanoGPT
61.0
8.87
8.18
2.04
0.63
43
DiMA
83.3
5.07
4.68
1.17
0.87
68
FD-seq (L)
MMD-seg (.)
OT-see (L)
FD-struct (1)
MMD-struct (L)
OT-struct (L)
Dataset
0.13
0.00
1.08
0.00
0.00
0.05
Random sequences
3.97
0.20
3.88
1.23
0.41
1.31
Distributional Similarity
Walk-Jump
0.33
3.56
0.61
0.69
RITA
1.19
0.14
2.28
0.37
0,03
0.52
proteinGAN
2.94
0.17
3,98
0.93
0.34
1.02
SeqDesign
3.53
0.19
5.12
0.95
0,25
1.11
EvoDiff-OADM.
1.49
0.11
2.63
0.52
0.20
0.66
D3PM
1,50
0.19
2.56
0.57
0.05
0.72
DFM
1.46
0.19
2.49
0.52
0.04
0.68
DPLM
0.50
0.02
3.50
1.68
0.93
0.88
nanoGPT
1.24
0.06
2.53
0.15
0.04
0.26
DIMA
0.34
0.02
1.26
0.03
0.01
0.09
Rep ())
CDo.s (1)
CDo.95 (t)
PCDo.5
NCDa.5
Novelty (1)
Datasct
0.045
1,000
0.943
0.990
0.304
25.3
Random sequences
0.000
1.000
1.000
1.000
0.000
85.1
Diversity and Novelty
Walk-Jump
0.001
1.000
1,000
0.000
1.000
82.2
RITA
0.028
0,988
0.998
0.125
0.861
60.4
proteinGAN
0.042
0.955
1.000
0.000
0.955
83.5
SeqDesign
0.210
0,929
1.000
0.009
0.929
81.2
EvoDiff-OADM
0.006
0,986
1.000
0.058
0.929
77.6
D3PM
0.003
0,994
1.000
0.025
0.968
78.4
DFM
0.004
0.996
1.000
0.048
0.947
77.2
DPLM
0.781
0.494
0.812
0.267
0.236
11.5
nanoGPT
0.228
0.900
0.994
0.226
0.679
53.7
DiMA
0.320
0.611
0,992
0.246
0.392
35.7
27

---

Diffusion on Language Model Encodings for Protein Sequence Generation
2. Normalizer Perffrrm per-component normalization, removing high-activation components to enhance stability.
3. CHEAP Encoder - Compresses the sequence embeddings into a lower-dimensional latent space, reducing both dimensionality and sequence length by predefined factor.
4. CHEAP Decoder- -
Maps the CHEAP latent representations back to ESMFold embeddings.
5. Sequence Decoder- Reconstructs the amino acid sequence from the ESMFold embeddings.
6. Structure Decoder- - Recovers the protein backbone from the ESMFold embeddings.
Our diffusion model operates on CHEAP latents. In the and decoder are omitted, meaning diffusion is trained directly on the normalized aggregated embeddings from ESM-3B.
-etorten___di__lele _dimm_024 configuration, the CHEAP encoder
E.4. Comparison with Pre-trained Protein Models
In this section, we compare DiMA with existing large protein models, including RITA (Hesslow et al., 2022), Prot- GPT2 (Ferruz al., 2022), ProGen2 (Madani et al., 2023), EvoDiff (Alamdari et al., 2023), ProLLAMA (Lv et al., 2025), DPLM (Wang et al., 2024), Chroma (Ingraham et al., 2023), Multiflow (Campbell et al., 2024), RFDiffusion (Watson et al., 2023) in different configurations. For all models, we adhere to the sampling parameters recommended by the authors. This experiment specifically focuses on methods that provide publicly accessible pre-trained weights, ensuring transparency and reproducibility in our evaluation.
The majority of models were pre-trained on distinct versions of the UniProt (Consortium, 2020) dataset. As a result, the application of distributional similarity metries in the current experiment is rendered unfeasible. Consequently, we focused solely on evaluating quality and diversity metries. Given that RFDiffusion generates protein structures, employed ProteinMPNN, a neural network trained to predict amino acid sequences from 3D protein structures, to infer sequences from the generated structures. The authors of RFDiffusion ran ProteinMPPNN multiple times for each generated structure and selected the sequence with the lowest perplexity as the final prediction. In contrast, we performed a single ProteinMPNN prediction for each generated protein, using the output of the first run to represent the inferred sequence. This approach was chosen to accelerate the inference process of the model and to ensure that the final perplexity metric not artificially inflated.
We conduct comprehensive comparison of DiMA with suite of existing pre-trained models for generating proteins of varying sizes. Due to the absence of a reference sample in this experiment, we focus on evaluating protein quality and diversity. The results are presented in the Table 8. DiMA, DPLM, ProtGPT2. and RFdiffusion models demonstrated the strongest performance in protein structural plausibility and foldability assessment. The remaining baselines exhibit significantly lower quality in terms of perplexity and structural plausibility.
Notably, RFDiffusion, trained on structural representations of proteins, exhibits a high degree of protein structural plausibility, potentially attributed to its structural bias. However, RF-Diffusion exhibits high perplexity value, suggesting low quality of predicted amino acid sequences and potentially indicating limitations in the performance of ProteinMPNN. While the combined use of these models for protein sequence generation yields promising results, it does not achieve state-of-the-art performance.
The family of DPLM models demonstrates high protein foldability quality with low perplexity scores, indicating the generation of high-quality proteins. However, DPLM models exhibit a significant drawback terms of diversity (see Table 8). A substantial portion of subsequences are repeated across multiple generated proteins, negatively impacting the representativeness of the generated proteins. Notably, DPLM-3B generates 27% duplicate sequences, highlighting the challenge of balancing quality with diversity in this model.
DiMA is capable of generating high-quality and diverse protein sequences with reasonable predicted structures. Using 100 times fewer parameters, it achieves comparable quality to other models, like DPLM models, while surpassing them in the diversity of proteins generated. These findings highlight DiMA's potential as a promising approach for protein sequence generation. It balances computational efficiency with the generation of diverse and high-quality proteins.
28

---

Diffusion on Language Model Encodings for Protein Sequence Generation
0.8
80
0.6
PLDDT
Rep
0.4
40
20
Random
Walk-Jump
D3PM
DFM
EvoDiff- EvoDiff-OADM. -OT--ADMMM
RITA
proteinGAN
Dataset
SeqDesign
nanoGPT
DiMA
DPLM
Random
Walk-Jump
D3PM
DFM
EvoDiff-OADM
RITA
proteinGAN
Dataset
SeqDesign
nanoGPT
DiMA
DPLM
Figure 16. Comparison of Rep (diversity) and pLDDT (structural quality) values for different protein generation models trained on the SwissProt dataset.
Table 7. Perfomance of protein sequence generation using DiMA and different encoders AFDBv4-90 90.
The comparison assesses
generation quality, distribution alignment, novelty, and diversity from different perspectives.
Encoder
PLDDT (1)
Progen ppl (L)
ESM-2 pppl ())
sePpl (L)
BLAST (t)
ESM2-8M
10.71
8,19
1.89
50
ESM2-35M
75.7
10.16
8.00
1.95
44
ESM2-150M
80.1
9.92
7.01
1.86
48
Quality
ESM2-650M
82.5
9.00
5,90
1.79
50
ESM2-3B
83.4
9.00
5,80
1.78
50
ESMC-300M
82.7
9.41
6.05
1.78
49
CHEAP shorten dim 1024
81.92
9.25
6.92
1.81
50
CHEAP.shorten. www.sthn....im.......
78.81
9.77
7.81
1.86
47
SaProt-35M
82.23
9.93
6.90
1.82
50
SaProt-650M
83.01
9.57
6.65
1.79
50
FD-seq (L)
MMD-seq (L)
OT-seq (L)
FD-struct (L)
MMD-struct (L)
Distributional Similarity
ESM2-8M
0.560
0.0389
1.488
0.027
0.100
ESM2-35M
0.340
0.0204
1.294
0.029
0.098
ESM2-150M
0,323
0.0152
1.232
0.018
0,081
ESM2-650M
0.318
0.0151
1.247
0,018
0.081
ESM2-3B
0.314
0.0143
1.246
0.022
0.077
ESMC-300M
0.326
0.0155
1.259
0.020
0.086
CHEAP shorten. 1
0.346
0.0178
1.289
0.013
0.067
CHEAP shorten.22 1aweee...ddii. 2.iim_dd22--
0.340
0.0201
1.328
0.014
0.092
SaProt-35M
0.366
0.0188
1.269
0.021
0.071
SaProt-650M
0.411
0.0207
1.326
0.017
0.078
Rep (1)
CDo.s (1)
CDo (f)
Novelty (t)
Diversity and Novelty
ESM2-8M
0.029
0.981
1.0
68.0
ESM2-35M
0.019
0,986
1.0
69.1
ESM2-150M
0.015
0.988
65.6
ESM2-650M
0.01
0,986
1.0
64.1
ESM2-3B
0.01
0.969
1.0
63.0
ESMC-300M
0.012
0.863
1.0
64.2
CHEAP CHEAP.shorten. 1_iim_dddm_2----1
0.017
0.951
1.0
64,6
CHEAP [HEEAP_shorten_2 shhorten.2.dim_1022
0.012
0.946
1.0
66.2
SaProt-35M
0.018
0.976
1.0
65.5
SaProt-650M
0.010
0.980
1.0
65.7
29

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Table 8. Comparison of the DiMA model with established pre-trained large protein models.
Encoder
pLDDT (1)
Progen ppl (1)
ESM-2 pppl (1)
scPpl ())
Rep ())
CDo.s (1)
CD9.9 (t)
Multiflow-22 21M
82,8
8.67
4.87
1.00
0.181
0.990
1.000
Chroma-33M
66.8
12.09
7.64
1.55
0.022
1.000
1.000
RFDiffusion-800
76.7
12.07
8.05
1.25
0.018
1.000
1.000
ProtGPT2-738M
63.0
7.79
5.70
2.20
0.096
0.998
1.000
ProGen2-1510
46.2
12.78
11.33
2.39
0.084
0.998
1.000
ProGen2-764N)
50.3
12.05
10.94
2.37
0.066
0.996
0.996
ProGen2-2.7B
52.3
11.78
10.57
2.35
0.044
0.992
0.994
ProGen2-6.4B
57.2
9.71
8.67
2.26
0.087
0.976
1.000
EvoDiff-38M
40.2
17.46
15.61
2.53
0.005
1.000
1.000
MoDiff-6440M.
40.5
17.35
15.38
2.52
0,000
1.000
1.000
Prol.LAMA-7B
53.1
10.50
7.46
2.26
0.133
0.982
1.000
RITA-85M
40.3
18.34
16.16
2.55
0.000
1.000
1.000
RITA-300M
41.5
19.10
15.73
2.57
0,000
0.990
0,990
RITA-680M
42.5
20.48
15.31
2.63
0.000
0.958
0.958
RITA-1.2B
42.6
19.39
15.22
2.64
0.000
0.966
0.966
DPLM-150M
81.8
3,90
2.82
1.60
0.658
0.654
0.917
DPLM-650M
81.8
4.36
2.41
1.60
0.533
0.746
0,943
DPLM-3B
83.1
4.16
2.75
1.57
0.911
0.568
0.732
PLAID-100M
53.5
14.98
13.46
2.29
0.001
1.000
1.000
DiMA-35M
83.4
9.00
5.80
1.78
0.010
0.969
1.000
E.5. Protein Family Controllable Generation
In this section, we evaluate the model's capability to generate novel, high-quality, and diverse proteins by leveraging protein family labels. Specifically, we focus on eight distinct protein families:
CRISPR-associated proteins (CRISPR).
Calmodulins,
Glycosyl hydrolase family 12 (GH12),
LexA DNA-binding domain proteins (LEXA).
Phage lysozymes (Lysozyme).
Transcriptional repressor NrdR proteins (NRDR),
Phosphosugar- binding domain proteins (PHT), and
carboxylase (PurE).
We filter the proteins, removing any duplicate entries. The sizes of the datasets for these protein families detailed in Table 9.
Table 9. The dataset sizes for the protein families used in controllable generation.
CRISPR Calmodulins
GH12
LexA
Lysozyme
NrdR
PHI
PurE
2818
551
2061
4516
11395
4515
23359
30817
We investigate two approaches for conditional protein generation: (1) classifier guidance, which avoids additional diffusion model training, and (2) conditional fine-tuning of the diffusion model across all protein families simultaneously.
Classifier Guidance. In the classifier guidance approach, train classifier that takes input noisy, normalized encoding of ESM2-650M along with a time label and predicts logits for 9 classes. Label 0 represents the absence of class, where a protein is randomly sampled from the AFDB in 50% of cases, ensuring it does not belong to any of the specified protein families. The remaining labels correspond to specific protein families. The classifier architecture consists of three transformer blocks with latent attention (Lee et al., 2025), followed by a linear layer. Training is performed with batch size of 1024 and learning rate of 2e over 300 iterations.
30

---

Diffusion on Language Model Encodings for Protcin Sequence Generation
Conditional Fine-tuning. In the conditional fine-tuning approach, we also utilize labels and randomly sample a protein from the AFDB in 50% of cases for the absence of specific family. To condition the diffusion model on family labels, we introduce embedding layer. The label embeddings, after passing through a linear layer, are incorporated into the input of each transformer block within the diffusion model. The fine-tuning process is conducted with batch size of 1024 over 2500 iterations, using a learning rate of 5e - 5.
In both approaches, we utilize DiMA 35M, trained on ESM2-650M encodings, as the backbone for protein representations on AFDB dataset.
Baselines, We compare DiMA against groups of baselines that enables controllable generation based family tag:
Autoregressive models: ProLLAMA (Lv et al., 2025) (7B), ProGen2 (Madani et al., 2023) (151 M), NanoGPT (Karpa- thy, 2023) (35M).
Discrete diffusion model: EvoDiff (Alamdari et al., 2023) (640M)
For training the baseline models, we primarily adhere to the authors' recommendations, training each model until convergence while monitoring the loss a validation set.
For training ProLLAMA, we initialize the model with pre-trained weights available at https: //wpg//hugggggg///e wggiagggiggaaan face. co/ Great i nnemo/ /aceem//Proliee PrOLLLMA. Fine-tuning is performed using LoRA (Hu et al., 2022) on an annotated dataset comprising eight protein families, following the format recommended by the original authors. During our experiments, we observed that the generation parameters suggested by the authors were suboptimal for producing family-specine protein data. Most generated proteins did not correspond to the specified family. To address this issue, we adjusted the parameters and identified a of values that yielded the highest generation fidelity. The optimized parameters are: tops- = 40, top,- - 0.9, repetition penalty= 1.2, temperature= 0.5. The most significant modification was in the temperature parameter, where replaced the originally suggested value of 0.2 with 0.5, resulting in improved family-specific protein generation.
For fine-tuning ProGen2, we use the code provided in the repository https://gg /ttt:///gttubb..///// com hww.m//hhghraaa////ooooo ProGen2- fine2-fineetee finten2-finetniiie ng and initialize the model with pre-trained weights from https::/ge / thub. com/ salesforce/ wwtoorrcepprggn/treeeeee wwwrre/mae//progeeeeeeeeee ily tag the beginning of each sequence. For sampling, we use the following parameters: temperature 0.5 and lopk 40.
To condition the model on the family label, we prepend the fam-
For NanoGPT fine-tuning, we apply the same conditioning and sampling schemes used with ProGen2. The model is initialized with weights pre-trained on the AFDB dataset.
For EvoDiff, we initialize the model using the pre-trained weights available at https: ttb..b.tthhgggogggg com/mi croso evodi ff ning of each sequence for conditioning, and the model is then fine-tuned. During sampling, we generate sequences starting with an empty sequence that includes the family label the beginning.
specifically the tag d33m_uni Muniforme form_640M. The family tag is added to the begin-
Evaluation Metrics. We use the following metries evaluate the generated protein sequences. First, ProScan (Blum et al., 2025) to determine whether the generated proteins belong to the specified family. The proportion of correctly classified proteins among all generated sequences is referred to as Fidelity. Next, we analyze only the generated proteins that are confirmed to belong to the target family. For these proteins, we compute the following metrics: quality (pLDDT), novelty (novelty), and diversity (CDo.s). The detailed calculation of the metrics is presented in Appendix The aggregated values, calculated as the arithmetic mean for each family and each model, are presented in Table 10.
Inter-
31

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Table 10. The performance of DiMA protein family generation against baseline methods. DiMA represents conditional fine-tuming inn-tuning, while DiMAce corresponds to classifier guidance. The best- perfornnngn scores are highlighted in bold, and the second- best scores are indicated with underlined formatting.
Model
CRISPR
Calmodulins
GH12
LexA
Lysozyme
NrdR
PHI
PurE
ProLLAMA
0.591
0.531
0.611
0.583
0.706
0.543
0.642
0.576
ProGen2
0.622
0.088
0.545
0.531
0.578
0.520
0.561
0.514
NanoGPT
0,520
0.501
0.640
0.612
0.679
0.528
0.652
0.542
Evodiff
0,591
0.648
0.535
0,547
0.591
0,515
0.649
0.572
DiMA
0.765
0.614
0.674
0.679
0.736
0.552
0.658
0.590
DiMAcg
0.689
0.629
0.604
0.584
0.720
0.536
0.634
0.578
Fidelity
Fidelity
Fidelity
CRISPR
Calmodulins
GH12
100
60
CDus
PLDDT CDas
PLDDT CD.s
PLDDT
DiMAconn
DIMAng
ProLLAMA
ProGen2
NanoGPT
Novelty
Evodiff
Novelty
Novelty
Fidelity
Fidelity
Fidelity
LEXA
Lysozyme
PHI
100
100
130
CDas
LLDTT CDas
OLDDT CD.s
PLDDT
Novelty
Novelty
Novelty
Figure 17. The comparison of the characteristics of generated proteins in the protein family controllable generation task. Solid lines represent DiMA models, including conditional fine-tuning (DiMAcoma) and classifier guidance (DiMAay). Dashed lines correspond to autoregressive baselines, while the dash-dot line indicates the performance of the discrete diffusion approach.
Results. The results presented in Figure 17 highlight the remarkable ability of DiMA to generate novel, high-quality, and diverse proteins guided by family class labels. Unlike autoregressive baselines, which frequently deviate from the specified class labels, DiMA demonstrates a strong adherence to the intended protein family. A significant drawback of the autoregressive models is that substantial proportion of their generated proteins fail to belong the desired class. Even when they do, the quality of these proteins is notably lower, with an average pLDDT score hovering around 60 across many families. Despite this limitation, autoregressive models excel in producing a diverse range of proteins, showcasing their strength in variability.
The discrete diffusion model, on the other hand, achieves comparable performance to DiMA in terms of fidelity and protein quality. However, struggles to match DiMA's ability to generate truly novel proteins while maintaining diversity. This
32

---

Diffusion on Language Model Encodings for Protein Sequence Generation
limitation underscores the unique advantage of DiMA in balancing quality, novelty, and diversity in protein generation.
Interestingly, the classifier-guidanee approach (DiMAcg) emerges as compelling alternative. While it does not outperform the fine-tuned model (DiMAcoma) it still delivers robust results, as evidenced by Table 10. Notably, DiMA.c has an undeniable advantage over all other approaches, it achieves competitive performance without requiring additional fine-tuning of the generative model.
E.6. Joint Sequence-Structuue Generation with DiMA
In this section, we evaluate the DiMA model trained on the latent representations of the CHEAP encoder (Lu et al., 2024), The CHEAP framework has demonstrated that vector representations derived from input sequences can be successfully decoded into both the amino acid sequence and the corresponding protein structure, This capability enables the construction of latent diffusion model that jointly generates structurally consistent amino acid sequences and backbone conformations.
To assess the self-consistency of the generated proteins, we follow the methodology of Yim et al. (2023b) and compute two key metrics:
Self-consistent Root Mean Square Deviation (scRMSD): Measures the deviation between the predicted backbone structure and the backbone reconstructed from the predicted sequence.
Self-consistent TM-score (scTM): Quantifies the structural similarity between the predicted and reconstructed back- bones.
A generated protein is considered designable if it satisfies the criterion scRMSD < 2Å and scTM > 0.5. Following established literature (Yim et al., 2023b; Lin et al., 2024; Camphell al., 2024; Bose et al., 2024), we also report the proportion of designed proteins that meet this self-consistency threshold.
50-
200
Count
Count
150
30-
100
50
10-
0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
scRMSD
SCTM
Figure 18. Distributions of the self-consistency metrics between generated structure and structure reconstructed from the generated sequence.
We present the distributions of self-consistency metrics in Fig. 18, demonstrating the high designability of the generated sequences and structures. The results indicate that the predicted amino acid sequences and backbone conformations are highly consistent, with 89% of proteins achieving scRMSD < 2 and 99% exhibiting scTM > 0.5. further assess the designability of DiMA, we compare its performance against structure-based generative models (Watson et al., 2023; Yim al., 2023b; Lin et al., 2024; Bose et al., 2024; Ingraham et al., 2023) in Table 11. DiMA's performance is evaluated using two complementary approaches:
Co-design evaluation: The mean scRMSD and the fraction of designable structures are computed by comparing the generated protein backbone with the structure reconstructed from its predicted sequence. This corresponds to the DiMA [co-design] approach.
Structure-only evaluation: The generated structures are assessed independently using ProteinMPNN, which generates
33

---

Diffusion on Language Model Encodings for Protein Sequence Generation
eight candidate sequences for each structure. The sequence yielding the best RMSD is selected, following the standard evaluation protocol (Yim et al., 2023b). This corresponds to the DiMA [structure-only] approach.
Table 11. The performance of DiMA in structure generation compared structure-based protein models. DiMA's quality is assessed through two approaches. In the co-design evaluation, the mean scRMSD and the fraction of designable structures are measured by comparing the RMSD between the generated structure and the structure reconstructed from the generated sequence. This corresponds the DiMA [co-design] approach. In the DiMA peructure-only case we perform standard structure evaluation approach, the generated structure assessed independently by using ProteinMPNN to produce eight plausible sequences each design, selecting sequence with the best RMSD.
Model
Fraction (t)
scRMSD (L)
RFDiffusion
0.969
0.650
Genie 2
0.960
FoldFlow-OT
0.820
1.806
Chroma
0,700
Genie
0,581
2.968
Fifme
0.555
2.929
DiMACHEAP shorten 1 dim 1024
[structure only]
0.923
1.091
DiMACHEAP.Honm _dim _02__10244 (co design]
0.888
1.043
DiMACHEAP shorten 2 dim 1024 [structure only]
0.845
1.211
DiMACHEAP MAMMAAAHAA-.hootunn2........0 [co design]
0.782
1.496
Tables 11, 7 demonstrate that DiMA generates high-quality, diverse, novel, and designable protein structures despite being trained exclusively on sequence data. Notably, DiMA outperforms Chroma, Genie, and FrameDiff by a substantial margin, while achieving results slightly worse to state-oftthe the-art models like RFDiffusion and Genie2. Although DiMA slightly lags behind the best 3D protein models, its strong performance-dessee ability to leverage sequence-barn -aaee representations for structure generation. This reinforces the power of sequence driven generative models, which benefit from the extensive coverage of the protein sequence landscape, even in the absence of explicit structural supervision.
lacking direct training on structural data highlights its
E.7. Sequence Infilling
Sequence infilling requires generating missing segment of amino acids that maintains structural and functional coherence with the surrounding sequence context. This task serves a test of conditional generation capabilities, demanding the model to generate novel sequence segments that integrate naturally with the given context while maintaining both global structural properties and local features at the boundaries of infilled regions.
We construct our evaluation set from SwissProt sequences with strict filtering criteria to ensure a challenging benchmark. The final test set comprises 180 sequences, each having at most 50% sequence identity to the training set and containing no ambiguous amino acids or non-standard residues. For each sequence, we mask region of random length, uniformly sampled between 8 and 50 amino acids, a random position. This variation in mask length tests the model's ability to handle different infilling scenarios.
We augment the base DiMA architecture (35M parameters) with conditional adapter for infilling. The adapter consists 3 3 transformer blocks with architecture matching the base model. It processes concatenated ESM2-650M encodings of
Table 12, Performance comparison of DiMA and DPLM on the protein infilling task, measured by three metrics: success rate, average quality (Region pIDDDT), and average novelty (Region identity).
Model
Success rate, % (1)
Region pL.DT (1)
Region Novelty (t)
DiMA
42.2
66.9
80
DPLM
40.0
59.3
75
Random
21.1
50.9
92
34

---

Diffusion on Language Model Encodings for Protein Sequence Generation
unmasked regions and mask tokens, with its outputs added to all diffusion transformer blocks via residual connections, The adapter is trained unconditional training set using AdamW optimizer (/1 0.9, A2 0.999) with batch size of 1024 and learning rate of 5e-5 with cosine decay. Random masking is applied during the training process, which continues for 10,000 steps.
We define success metric based on three criteria that must be simultaneously satisfied: (1) the complete sequence
A
B
must achieve pLDDT 2 80, (2) the infilled region must maintain pLDDT 2 80, and (3) the unmasked regions must preserve their structure with RMSD < 1Â compared reference. For each test sequence, we generate 10 infilling attempts to account for generation stochasticity, consider- ing success if any attempt satisfies all the criteria. We use ESMFold for pLDDT calculation and structure prediction. We compare DiMA against random baseline (randomly sampled amino acids for the masked region) and DPLM.
The quantitative evaluation demonstrates DiMA's strong performance, achieving 42.2% success rate compared to 40.0% for DPLM and 21.1% for the random baseline. DiMA generates infilled regions with higher average qual- ity (pLDDT 66.9 vs 59.3 for DPLM) while maintaining high sequence novelty (mean 80.0). Table 12 presents the complete results across all evaluation metries.
Several key observations emerge from experiments. DiMA maintains high structural quality while achieving greater sequence novelty compared to baselines. The suc- cess rate correlates positively with the length of the infilled region up to approximately 30 amino acids. Failed cases often involve regions near critical structural elements or domain boundaries. The adapter-based approach proves effective for conditioning without requiring full model re- training.
Figure 19 shows selected examples of sequence infilling where the generated regions maintain structural coherence with their surroundings. These results suggest that DiMA's diffusion-based architecture can be extended handle conditional generation tasks while maintaining its core strengths in protein sequence modeling.
Figure 19. Examples of successfully solved infilling cases. A- refer- ence proteins, B- DiMA generated proteins. DiMA produces a novel region (purple), being conditioned on its surroundings (green).
E.8. Functional-motif Scaffolding
Functional motif scaffolding requires generating protein
sequences that incorporate predefined structural motif's while maintaining their spatial configuration. Following the RFDiffusion benchmark protocol, we evaluate performance on 24 distinct scaffolding problems. Each problem consists of a structural motif that must be preserved within newly designed protein structure.
For this task, we use DiMA with the SaPror-650M encoder and augment it with a conditional encoder. The conditional encoder processes motif information and passes it each transformer layer the diffusion model through additive conditioning. We fine-tune the model randomly masked proteins, where the masking pattern mimics the scaffolding task setup. During inference, sample 100 designs per problem.
We employ two complementary success metrics:
Success Rate (SR). A design is considered successful if it achieves PLDDT 2 70 (indicating overall structural quality)
35

---

Diffusion on Language Model Encodings for Protein Sequence Generation
0.7
DIMA
ESM3
DPLM
0.5
Success rate
0.4
0.3
0.1
0.0
STRV STRV_shorr short +
1BCF
1PRW
1Q/G
1YCR
2KL8
3IXT
4J-M
42
5I0S
5TPN
5TRV long
5TRV _medd
5WN9
5YU1
6E6R, llng
6E6R, _0d
6E6R_ short
6EXZ long
6EXZ 6EXZ_men med
6EXZ 6EXZ_shon short
7MRX 128
7MRX_ (66600
7MRX_8 85
Figure Motif-scaffolding: models performances across benchmark problems, measured by unique success rates. DPLM model
sequence based, ESM3 and DiMA- are s0-generation models variants.
and maintains motif RMSD < 1Å (indicating preservation the functional motif). The SR for each problem the fraction of successful designs among 100 samples.
Unique Success Rate (USR). To account for potential mode collapse, we measure CDo.5 diversity by cluster successful designs using the 50% sequence identity threshold and count only cluster centroids. This metric reflects the diversity of viable solutions.
As shown in Table 13, DiMA successfully solves 19 out of 24 benchmark problems, positioning it among the top-performing methods. Detailed analysis reveals several key findings:
Performance by Problem Type. DiMA shows particularly strong performance on 6E6R variants (SR: 0.66-0.79) but struggles with certain challenging cases like 4JHW and SYUI where most methods fail.
Solution Diversity. DiMA achieves a mean unique SR 0.1, matching ESM-3 and outperforming RFDiffusion (0.06), indicating greater diversity successful designs.
Model Size Efficiency. DiMA (35M parameters) achieves competitive performance with significantly larger models like ESM-3 and RFDiffusion, as visualized in Figure 4.
Figure presents a detailed comparison of unique success rates across methods, highlighting DiMA's ability to generate diverse successful scaffolds. DiMA and ESM-3 show different strengths across problem types, with DiMA performing better 6E6R variants while ESM-3B excels at 6EXZ problems (Table 13).
E.9. Cad--Conditioned Generation
Fold-conditioned protein generation aims produce novel protein sequences that adopt specific three-dimensional structural folds. Unlike inverse folding, where the protein backbone is fixed, or fold-class conditioning via discrete labels, fold- conditioned generation allows exploration of the structural neighborhood of target fold through generation of diverse but structurally similar proteins. This capability is particularly valuable for protein engineering applications that require structural similarity while maintaining sequence flexibility.
We utilize DiMA (35M parameters) with the CHEAP encoder (shorten=1, dim=1024 configuration), which provides access ESMFold's 's latent space representations. This choice enables direct encoding of structural information into the continuous
36

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Table 13. Detailed results scaffolding benchmark for each task separately.
sequence
structure
cogeneration
EvoDiff
DPLM
ESM3
DPLM
RFDiffusion
DPLM2
ESM3
DPLM
DIMA
IBCF
0
0
0.89
0.01
0.07
0.23
0.01
0.01
IRRW
0.61
0.83
0.96
0.86
0.08
0.96
0.54
0.84
0.7
QJG
0
0
0.02
0.03
0
0.03
0.02
TYCR
0.02
0.38
0.41
0.77
0.74
0.93
0.18
0.53
0.37
2KL8
0.04
0.08
0.11
0.47
0.88
0.94
0.11
0.57
0.03
3IXT
0.06
0.17
0.18
0.67
0.25
0.77
0.02
0.41
0.44
4JHW
0
0
0
0
0
0
0
4ZYP
0
0
0.03
0.16
0.4
0.51
0.08
0.1
0.02
SIUS
0
0
0
0
0.02
0
0
0
STPN
0
0.03
0
0.61
0.06
0.01
0.01
STRV_log _lon
0
0.19
0
0.37
0.08
0.19
0.05
5TRV med
0.16
0.03
0.24
0.07
0.16
0.02
0.05
STRV
0.01
0.07
0.04
0.1
0.01
0.03
0.01
5WN9
0.02
0
0
0.2
0
0
0.1
SYUI
0
0
0
0
0.02
0
0
0
0
6F6R long
0.01
0.65
0.07
0.91
0.86
0.92
0.04
0.78
0.67
6E6R_me _ed
0,03
0.94
0.24
0.93
0.89
0.88
0.14
0.77
0.79
short
0.07
0.87
0.09
0.86
0.39
0.78
0.06
0.64
0.66
6EX7.Jong
0.01
0.32
0.61
0.76
0.63
0.13
0.44
0.01
6EXZ med
0
0.31
0.66
0.49
0.63
0.31
0.55
0.01
6EXZ.short
0
0.31
0.66
0.39
0.41
0.28
0.58
0.02
7MRX.128
0
0.02
0.36
0.23
0.09
0.32
0.37
0.2
0
7MRX660
0
0.31
0.65
0.28
0.11
0.31
0.59
0.22
0.01
7MRX 85
0
0.34
0.68
0.26
0.02
0.41
0.74
0.24
0.02
Total Success
7
11
21
18
21
20
20
18
19
mean SR
0.04
0.19
0.25
0.35
0.36
0.42
0.18
0.29
0.17
mean unique SR
0.06
0.06
0.07
0.1
0.1
Based on the analysis from (Yim et al., 2024)
latent space used by our diffusion model. The model architecture remains unchanged from the base configuration, with structural information incorporated through the same conditioning mechanisms developed for sequence tasks.
The model is finetuned on the CATH non-redundant S40 dataset, comprising approximately 27,000 non-redundant protein structures. We employ the following training parameters: training duration 20,000 steps with batch size of 256 and learning rate of le-4.
We conduct evaluations two benchmarks. The first consists of 100 diverse protein structures from CATH S40 hold-out set. The second uses specific protein folds previously studied in the literature (Watson et al., 2023): a de novo designed TIM barrel fold (PDB: 6WVS) and NTF2 fold (PDB; 1GY6). For each target structure, we generate 10 protein designs and assess their structural similarity using TM-score and RMSD metrics.
Success criteria are defined as follows:
Per-protein Success, At least one generated design achieves TM-score 2 0.5
Overall Success Rate. Percentage of target proteins meeting per-protein success criterion
Mean Maximum TM--core. Average of best TM-scores across 10 attempts for each target
We compare DiMA against RFDiffusion, a structure generation model that conditions on secondary structure and block- adjacency information. The comparison uses identical evaluation protocols and success criteria for both models.
the CATH benchmark, DiMA achieves mean TM-score 0.93 with 100% success rate, compared RFDiffusion's
37

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Table 14, Performance comparison fold-conditioned generation tasks.
Model Benchmark Mean max TM-score
Success rate (%)
DiMA (35M)
0.93
100
RFDiffusion (90M)
0.48
41
B
PS5343 (58%)
(57%)
Q76SE8 (32161
Q2X248 (36%)
Figure 21. ESMFold predicted representative examples of proteins generated by DiMA (A) and the closest hit SwissProt (B) with UniProt IDs and the homology %. colored by pLDDT.
mean TM-score of 0.48 and 41 success rate. For TIM barrel and NTF2 folds from the RFDiffusion paper DiMA maintains consistent performance with 100% success rate (Table 14). The high TM-scores achieved by DiMA indicate significant structural similarity between generated proteins and their target folds. However, the non-zero RMSD) values (mean 2.6A) demonstrate that the model generates structurally similar but non-identical proteins, confirming true fold-conditioned generation rather than simple structure copying. Figure 5 presents representative examples of generated structures alongside their target folds.
The performance difference between DiMA and RFDiffusion can be attributed to their distinct approaches to structural conditioning. DiMA leverages rich structural encodings from the CHEAP encoder, whereas RFDiffusion relies on more abstract representations through secondary structure and block-adjacency information. This difference in conditioning information likely contributes to DiMA's higher success rates despite its smaller parameter count.
F. Biological Relevance Analysis
Superfamily Annotation. For proteins annotation we utilized the established protein annotation tool InterProScan (Paysan- Lafosse et al., 2023; Jones et al., 2014). InterProScan includes a set of pre-trained models based on hidden Markov models (HMMs), which allow for assigning potential folds and functions. This analysis involves annotating the generated protein sequences using the SUPERFAMILY HMM library (Oates al., 2015), which provides sequence homology to SCOP structural domains (Murzin et al., 1995). IDR Exploration. Natural proteins encompass both structured regions and IDRs that lack regular structure but still play functional roles (Uversky. 2015) (Figure 25). To annotate these regions, we employ the MobiDB model within the InterProScan tool, which predicts IDRs in protein sequences using multiple classifiers (Piovesan et al., 2018). Sequences generated by DiMA exhibit a natural-like profile of IDR length distribution (Figure 25). Generation of both folded and unfolded structural regions provides distinct advantage for sequence diffusion models over models exclusively trained on folded protein domains. Secondary Structure Exploration. Finally, we calculate the frequency of secondary structure elements within the folded regions using the DSSP tool (Kabsch & Sander, 1983) against protein structures predicted via ESMFold. DiMA mirrors the amount of secondary struetural elements of natural proteins. DiMA generates seuqnces with number of secondary elements close to relevant number in validation dataset (Figure 26).
38

---

Diffusion on Language Model Encodings for Protein Sequence Generation
FAD/NAD(P)-binding domain
Dataset
Winged helix DNA-binding domain
DIMA
DPLM
Translation proteins
nanoGPT
Ribosomal protein 55 domain 2-like
PLP-dependent transferases
Protein kinase-like (PK-like)
beta-beta-alpha zinc fingers
Actin-like ATPase domain
Adenine nucleotide alpha hydrolases-like
Nucleic acid-binding proteins
Nucleotidylyl transferase
Class aaRS and biotin synthetases
Eeddensyl-mmehhienneeddddeeet methyltransferases
NAD(P)-binding Rossmann-fold domains
P-loop containing nucleoside triphosphate hydrolases
0.00
0.02
0.04
0.06
0.08
0.10
0.12
annotation fraction
Figure 22. Histogram depicting the occurrence of the top 15 frequent SUPERFAMILYY domains in the SwissProt dataset pool. (Oates et al., 2015; Jones et al., 2014). x- Fraction of each annotation per model.
39

---

Diffusion on Language Model Encodings for Protein Sequence Generation
A
SUPERFAMILY annotation
B
DIMA
DPLM
0.8
0.6
145
21
21
nanoGPT
Dataset
DIMA
DPLM
nenoGPT
annotation fraction
C
350
200
100
50
100
150
200
250
0
50
100
150
200
250
0
50
100
150
200
250
Domein
Domein
Domain length
Figure 23. Analysis of protein domain characteristics in generated sequences, (A) Fraction of sequences containing InterPro SUPERFAM- ILY domains shows comparable annotation rates between models and natural proteins. (B) Venn diagram revealing DiMA's superior domain coverage, generating both more unique domains (67) and higher total domain count compared to DPLM and nanoGPT. (C) Domain length distributions demonstrating that DiMA accurately preserves natural domain length patterns, while DPLM shows bias toward full-sequence domains (240 250 aa), and nanoGPT exhibits non-native uniform distribution. These results indicate DiMA's capacity capture diverse and biologically relevant domain architectures.
40

---

Diffusion on Language Model Encodings for Protein Sequence Generation
53383
SSF52314
SSF52540
S8F102214
SSP51905
SSP51 735
SSF53067
SSP35681
SSF46785
SSF50447
SSF54211
S5F57067
55/56112
55F53355
S5P50249
55F524002
aboo
Figure 24. Sequence annotation into known structural domains using SUPERFAMILY tool within InterProScan (Oates et al., 2015; Jones al., 2014). Eseeeeedpreeiited structures of representative SUPERFAMILY domains generated by DiMA.

---

Diffusion on Language Model Encodings for Protein Sequence Generation
Intrinsic disorder DIMA, final pwiesprol
Intrinsic discrder DIMA, base, swissprot
Intrinsic disorder nanoGPT swissprot
Intrinsic disorder Evodiff swissprol
Intrinsic disorder SeqDesign, swissprot
Intrinsic disorder proteinGAN swissprot
Intrinsic disorder random, swissprot
100
160
Count
100
100
50
50
50
100
100
domain longth
domein langth
domein length
domain longth
domain longth
domein length
domain longth
IDR
BAN (00)
IDR
IDR
Figure 25. Prediction intrinsic disorder regions (IDR) using the MobiDBLite tool (Piovesan et al., 2018). (A) Histogram depicting the
lengths of intrinsic disorder regions. The blue color represents the dataset, while the red color represents the generated sequences. No hits were found for random sequences. (B) Representative examples of proteins generated by DiMA, highlighting intrinsic disorder regions red and folded structural domains in blue.
dataset swissprot
200000
DIMA final swissprat
DIMA base swissprot
nenoOPT swissprol
150000
Exodiff_sissppd
SeqDesign swissprot
Counts
protainGAAN ssprot
100000
mindom swissprot
50000
E
T
S
G
B
DSSP class
Figure 26. The number of secondary structure elements calculated per residue from ESMFold predicted structures using DSSP (Kabsch Sander, 1983) software. H = achelix; B residue in isolated 3 3-bridge; E = extended strand, participates in & ladder; G 3-helix (310 helix); == helix (π--hllix); T hydrogen bonded turn; S bend.
42


Reviews:
[{"review_id": "T0FsfPxwob", "reviewer": "Reviewer_YSrx", "summary": "The authors have proposed a continuous diffusion framework, named DiMA. DiMA consists of three modules, i.e. 1) frozen pLMs like ESM2 to extract latent embedding for a given protein sequence, 2) a continuous diffusion module to generate latent embedding for noise and 3) a decoder that maps the latent embeddings to amino acid sequences.  It can be seen as a kind of knowledge distillation that injects information from pre-trained large pLMs, like ESM or SaProt, into DiMA.  Through comprehensive and systematic experiments, the authors claim that the performance of DiMA is consistent with (or better than) specialized baselines in different scenarios like unconditional generation and conditional generation, such as motif scaffolding and fold-conditioned generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful and detailed feedback. We appreciate the time you have taken to review our work and your positive comments about our paper. We aim to address your concerns and questions below.\n\n> **W1.** On technical novelty.\n\nWhile diffusion models are not new, our work shows that naive application of continuous diffusion to proteins yields poor results (~60 vs 83.3 pLDDT, Table 1). Our systematic ablations quantify the impact of each component. As reviewer *Y6i7* notes, we contribute by 'exploring training and inference strategies instead of directly copying parameterization from image domain'. This enables DiMA to achieve strong performance across diverse representation spaces (from 8M to 3B parameter sequence-based and multimodal encoders) and multiple tasks with a single architecture - outperforming specialized models like RFDiffusion in fold conditioning (TM-score 0.93 vs 0.48).\n\n>**C1.** On the meaning of \"sd-10\".\n\nThank you for noting the unclear meaning of \"sd-10\" in Figure 1. This refers to our noise schedule based on the formula $\\alpha_t = \\frac{1}{1 + d^2tan^2(\\frac{\\pi t}{2})}$, where d=10 determines the rate of information decay. We will rename this to \"tan-10\" in the revised manuscript for better clarity.\n\n>**C2.** On padding omitting.\n\nThe \"w/o length sampling\" entry in Table 1 corresponds to the model trained with padding and without length sampling. We will clarify this connection in the revision.\n\n>**C3.** On sequence length determination.\n\nFor fair comparison across all methods, we apply the same length sampling strategy during inference: sequence lengths are sampled from the empirical distribution of the training set. \n\nThe difference noted on line 236 refers to protein domains (distinct structural/functional units within proteins), not overall sequence length. \nWhile we control overall sequence length distribution across methods, domain length distributions emerge naturally from each model's generation process. Our analysis shows DiMA generates domain length distributions closely matching natural proteins, whereas DPLM skews toward longer domains (Figure 18C).\n\n>**Q1.** On the necessity of continuous vs. discrete diffusion.\n\nThank you for this important question. While discrete diffusion may seem more intuitive for sequences, continuous representations have proven highly effective in protein domains including representation learning (ESM, ProtT5), structure prediction (AlphaFold, ESMFold), and backbone generation. Recent work (CHEAP [1]) confirms continuous representations capture richer protein features.\n\nContinuous diffusion offers several advantages:\n\n1. Direct application of established score-based techniques like classifier and classifier-free guidance without requiring discrete approximations\n2. Seamless integration with multimodal representations (CHEAP, SaProt) that jointly capture sequence and structure\n3. More stable and efficient training compared to discrete spaces\n4. Fine-grained optimization of diffusion parameters, as shown in our ablation studies\n\nOur experiments demonstrate that this approach not only achieves strong performance but also enables structure-aware generation and fold-conditioning that are challenging in purely discrete frameworks.\n\nWe believe both approaches have merits, and our work contributes to understanding how continuous diffusion can be effectively applied to protein design. We will expand this discussion in our final manuscript.\n\n>**Q2.** On sequence length preference.\n\nThank you for this question. We conducted additional experiments and our results show that DiMA maintains consistent performance across all sequences length ranges, with quality metrics closely tracking the natural protein distribution (figure 1 at https://tinyurl.com/icml25re). DiMA achieves more stable diversity than DPLM-3B, which shows a drop in diversity for longer sequences.\n\n>**Q3.** On control over diversity and novelty.\n\nThank you for raising this point on balancing quality, diversity, and novelty. Our framework offers two knobs to control this trade-off without changing the encoder architecture (figures 2, 3 at https://tinyurl.com/icml25re):\n\n1. The number of sampling steps. Reducing the number of steps increases diversity and novelty and maintains reasonable quality.\n2. The self-conditioning rate parameter (w), which controls how much the model relies on its previous predictions during sampling. Lower values of w (0.6-0.8) yield higher diversity and novelty with a modest quality trade-off.\n\n\n>**Q4.** On the code release.\n\nWe are currently preparing our codebase for public release and plan to make it available upon publication.\n\n---\n\nWe are grateful for your detailed feedback and will incorporate your suggestions in the final version of our manuscript. Please let us know if any questions still need clarification.\n\n\n[1] https://www.biorxiv.org/content/10.1101/2024.08.06.606920v2", "claims_and_evidence": "The claims are supported by clear evidence. The authors claimed the performance of DiMA  matchs or exceeds the baselines. Table 2 compares DiMA with other generative baselines in unconditional scenario, Figure 3 in motif scaffolding-condtioned, Table 10 in family-specific conditioned and Table 14 in fold-conditioned scenario. Besides, the authors have conducted a detailed ablation study to identify key components in Table 1", "methods_and_evaluation_criteria": "The proposed continuous diffusion model for generating amino acid sequences is logical. The evaluation criteria, such as pLDDT and TM-score for structural quality and perplexity for sequence quality, are appropriate. These criteria comprehensively address quality, diversity, and novelty in generation tasks.", "theoretical_claims": "The model illustration in Session 2 and Appendix C.1 includes details on Noise Schedule Optimization and Self-Conditioning.", "experimental_designs_or_analyses": "The overall experimental designs are completed and comprehensive. For benchmark, DiMA are compared with 5 groups of baselines. An ablation study is conducted to find key contributed module in DiMA. For illustrate the performance of DiMA, the authors have conducted extensive experiments both on unconditional generation and conditional generation including motif-scaffolding, family-specific and fold-conditioned.", "relation_to_broader_scientific_literature": "Essential prior works like EvoDiff(Alamdarietal.,2023), DPLM(Wangetal.,2024), Multiflow(Campbelletal.,2024), are appropriately cited and compared with proposed method as baseline methods.", "essential_references_not_discussed": "The discussion of related works is comprehensive.", "other_strengths_and_weaknesses": "Strengths:\n1. The experiment designs are well-done, demonstrating the ability of DiMA in different generation scenarios.\n2. The manuscript is well-written and easy to read.\n\n\nWeaknesses:\n1. The paper is not so technical novel. Diffusion is widely applied for protein sequence in the community, as is the pretrained pLMs. Techniques like noise schedule optimization and long-skip connection are mainly engineering improvements.", "other_comments_or_suggestions": "1.\tThe authors should explain the meaning of ‘sd-10” in Figure 1.\n2.\tOn line 208 of page 4, the authors have mentioned “padding omitting” decreases the performance, but the detail influence is not shown in Table 1. \n3.\tOn line 236 of page 5, the author asserts that DPLM generates longer sequences compared to DiMA. What Length Determination strategy (Appendix C.3) is employed for DiMA inference here? Is the same strategy applied to DPLM to ensure a fair comparison?", "questions_for_authors": "1.\tThe authors introduced DiMA, a continuous diffusion method for protein sequence generation. However, they did not adequately demonstrate the necessity of using continuous diffusion compared to the discrete diffusion approach employed in DPLM and multiflow, which appears more intuitive for sequence generation. \n2.\tDo DiMA have a sequence length preference? Say we separate protein sequences by their length into different bins, like [0-50],[50-100],…,  what’s the performance of DiMA in each group, in terms of quality, diversity and novelty?\n3.\tAs shown in Table 3, DiMA archives consistent quality (i.e. pLDDT) comparing with other baselines while sacrificing diversity and novelty (Table 3). Is there any approach to elevating diversity and novelty without reducing the size of ESM encoder (so that the generation quality may keep at the same level) ?\n4.\tDo you have a plan to release code?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4sjfeLGD4d", "reviewer": "Reviewer_QrCA", "summary": "The paper introduces DiMA, a latent diffusion approach for protein sequence generation leveraging pre-trained embeddings. The authors consider sequence-only, structural, and sequence-structure joint embeddings. DiMA produces novel and high pLDDT samples. Conditional generation tasks based on protein family, motif scaffolding, and infilling are shown.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and the valuable suggestions. We appreciate your recognition of DiMA's thorough evaluation and benchmarking approach. Below, we address each point raised.\n\n> Concurrent work that the authors will be interested in.\n\nRegarding the work by Lu et al. (2024), we became aware of PLAID and evaluated it after our submission, just before receiving the reviews. We evaluated PLAID-100M (https://github.com/amyxlu/plaid) using the same protocol, we apply to all the models in our work. Our analysis shows that DiMA substantially outperforms PLAID, especially in terms of protein quality:\n\n| Model | pLDDT ↑ | ProGen-ppl ↓ | ESM2-pppl ↓ | scPerplexity ↓ | Rep | CD₀.₅ | CD₀.₉₅ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| PLAID-100M | 53.48 | 14.982 | 13.46 | 2.294 | 0.0007 | 1.0 | 1.0 |\n| DiMA-35M | 83.4 | 9.00 | 5.80 | 1.78 | 0.010 | 0.969 | 1.000 |\n\nWe will include these results in the final manuscript, specifically we will update Section 3.5 ('Comparison with Large Pretrained Models') and Figure 2.\n\n> Can the authors comment on the 4JHW and 5YUI PDB IDs as failure modes? What causes DiMA and other methods to struggle with these particular cases?\n \nThese cases highlight limitations in the current benchmark rather than specific model shortcomings. Our analysis reveals that even when using reference sequences, the resulting structures after ESMFold prediction fail to meet the benchmark success criteria. For 4JHW, the reference sequence yields motif RMSD exceeding 6.0Å with pLDDT around 30, far below the thresholds for success (RMSD ≤ 1.0Å, pLDDT ≥ 70). Similarly, 5YUI produces motif RMSD above 3.0Å. The detailed results of our analysis on these challenging cases are presented in Table 1 at [https://tinyurl.com/icml25re]. \n\nRecent work [1] has recognized these benchmark limitations and is developing improved evaluation protocols, specifically addressing them in Table 4 of their preprint.\n\n\n>It would be more informative to include a graphic overview of the model and approach as Fig 1, rather than the noise schedule.\n\nWe appreciate your suggestion about including a graphic overview of our model approach. In the final version, we are commited to add a figure to illustrate DiMA's architecture and workflow.\n\n>Distance metrics like FD-seq and other metrics should be introduced and defined in the main text.\n\nWe will also ensure that distance metrics like FD-seq and other evaluation metrics are properly introduced and defined in the main text to improve clarity.\n\n---\n\nThank you again for your constructive feedback, which will help us improve the final manuscript.\n\n[1] https://arxiv.org/pdf/2502.12479", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper is contextualized well within the literature, both with respect to different pre-trained embeddings and generative sequence models.", "essential_references_not_discussed": "Concurrent work that the authors will be interested in:\n\nLu, A. X., Yan, W., Robinson, S. A., Yang, K. K., Gligorijevic, V., Cho, K., ... & Frey, N. (2024). Generating All-Atom Protein Structure from Sequence-Only Training Data. bioRxiv, 2024-12.", "other_strengths_and_weaknesses": "Latent diffusion over pre-trained embeddings is an interesting and timely idea. The DiMA evaluation is thorough, as are the benchmarks and baselines. \n\nCan the authors comment on the 4JHW and 5YUI PDB IDs as failure modes? What causes DiMA and other methods to struggle with these particular cases? \n\nIt would be more informative to include a graphic overview of the model and approach as Fig 1, rather than the noise schedule. \n\nDistance metrics like FD-seq and other metrics should be introduced and defined in the main text.", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zW7vhxoRHf", "reviewer": "Reviewer_2rVp", "summary": "This paper introduces DiMA, a continuous latent diffusion model that creates (novel) protein sequences using protein language model (PLM) hidden representations. Unlike other approaches that use discrete diffusion or step-by-step generation, DiMA explores continuous diffusion to make better sequences. It works well with different protein encoders (ESM-2, CHEAP, and SaProt to name a few) and performs strongly in tasks such as adding motifs, creating fold-specific sequences, and designing protein families. Experimental (in silico) results show that DiMA can make diverse, new, and structurally sound protein sequences that often beat existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive comments. We appreciate your recognition of our work's comprehensive experimental design and clear methodology.\n\n> **W1.** Limited theory behind the proposed method, making is a straightforward application practice of well-established method\n\nWhile latent diffusion models have been established in other domains, our work demonstrates that naive application of continuous diffusion to protein sequences yields poor results. Our ablation studies in Table 1 show that standard diffusion implementations struggle with protein generation, achieving only ~60 pLDDT compared to DiMA's 83.3 pLDDT. We demonstrate that domain-specific engineering is essential for effective latent diffusion models. \n\nThe challenge of applying diffusion to proteins is further evidenced by existing approaches requiring custom modifications. For example, discrete model DPLM [1] requires specialized initialization and sampling strategies [1, Appendix D.1], MultiFlow requires distillation steps, and so on.\n\nOur work contributes by establishing a principled framework for latent diffusion that generalizes across different representation spaces (from 8M to 3B parameter sequence-based and multimodal encoders) and tasks with a single architecture. As shown in our fold-conditioning experiments, this enables DiMA to achieve stronger performance than specialized structure-generation models like RFDiffusion (TM-score 0.93 vs 0.48).\n\n> **W2.** Not comprehensive discussion of potential limitations or failure cases\n\nThank you for this suggestion. In our revision, we will add a dedicated discussion section addressing current challenges and future opportunities. Structure generation in our approach relies on heavyweight folding models like ESMFold. We see significant potential for enhancing performance through joint training of the diffusion model with lightweight structure prediction components. This approach could both improve computational efficiency and structural awareness of the model.\n\n> **W3.** Not strong performance compared to existing methods which may weaken the contribution of this work\n\nAs shown in Figure 2, DiMA, MultiFlow, RFDiffusion, and DPLM each demonstrate Pareto-optimal performance on the quality-diversity tradeoff, with each dominating in different aspects. Considering model sizes, DiMA shows remarkable performance, using two orders of magnitude fewer parameters than e.g. DPLM. Table 8 demonstrates that only DiMA and MultiFlow achieve balanced metrics on this tradeoff, with the notable advantage that DiMA trains exclusively on sequences while MultiFlow requires both sequence and structure data, making DiMA more scalable due to its independence from limited 3D structural data.\n\n> **Q1.** How does DiMA compare in speed (training/inference) to discrete diffusion models such as ESM3, DPLM?\n\nWe analyzed inference speed by generating 100 sequences of length 250 with 5 repetitions (for details see Figure 4 at https://tinyurl.com/icml25re). DiMA-35M is ~10x faster than DPLM-150M, which is slowed by using a larger model at each generation step. Structure models are notoriously computationally demanding. We have not measured training speed, but we expect DiMA to be faster as it requires only forward encoder passes, whereas discrete models need backward encoder passes as well.\n\n> **Q2.** Can you train or fine-tune DIMA with structure-based tasks and data?\n\nYes, DiMA incorporates structural information in two ways:\n\n1. We use local structural tokens for each amino acid passed with the sequence into the SaProt encoder. This allows explicit use of 3D structure in the scaffolding task, achieving strong results (details in section 3.6.1, appendix E.8, and Figure 3).\n\n2. We employ DiMA with the CHEAP encoder based on ESMFold, enabling dual decoding (sequence and structure) from sequence alone. This approach outperforms RFDiffusion in fold-specific generation using explicit structure guidance (details in sections 3.6.3 and appendix E.9, and Figure 4).\n\n> **C1.** Line 187-202. the long list of bullet points is not reader-friendly. Please consider re-organize it.\n\nThank you for this feedback. We will reorganize this section in the revision to improve readability.\n\n> **C2.** A discussion on how to scale up DiMA (maybe by combining it with structural models like AlphaFold) would be valuable.\n\nThank you for this suggestion. We are currently exploring the integration of structural generation models with our latent diffusion approach for co-generation. In the revision, we will add a discussion section highlighting future directions and the potential benefits of combining DiMA with structural models.\n\n---\n\nWe thank you for your thoughtful feedback and suggestions. We would be grateful if you would consider raising your score, in case we have addressed your concerns. Please let us know if any aspects still need clarification.\n\n[1] https://arxiv.org/abs/2402.18567", "claims_and_evidence": "The paper explore the proposed paradigm (DiMA) to generate protein sequences comparable to larger models, where the major claim is that DiMA demonstrates strong performance on multiple sequence-design based benchmarks, which i find is however not well supported by the experiment (in table 2, not yet outperform other baseline models).", "methods_and_evaluation_criteria": "The proposed method, as an application of the well-established latent diffusion models, is straightforward and makes sense. In regard of methodology, there is no novel theory or method is proposed. The evaluation criteria basically make sense with authors’ justification but not following previous practice.", "theoretical_claims": "No theoretical claim is found in this application-oriented paper.", "experimental_designs_or_analyses": "The experiments are comprehensive by design, covering several useful ablation studies across different tasks for sequence generation. The extended results in the appendix also demonstrate the solidness of experiments.\nPersonally speaking, the use of swissprot and AFDBv4-90 is not very typical for sequence-based protein language models.", "relation_to_broader_scientific_literature": "This work is situated at the intersection of (latent) diffusion models, protein language models and the protein design. It involves engineering the foundational protein language models (related: ESM, DPLM, etc.) and use denoising diffusion models over the latent space.", "essential_references_not_discussed": "No, but I recommend the authors to explicitly set a “related work” section in the main text rather than in the appendix, which is important for general readers to grasp a proper context of this paper.", "other_strengths_and_weaknesses": "Strengths: \n- Clear problem and methodology \n- Comprehensive testing/benchmarking experiments and comparison\n\nWeaknesses: \n- Limited theory behind the proposed method, making is a straightforward application practice of well-established method\n- Not comprehensive discussion of potential limitations or failure cases \n- Not strong performance compared to existing methods which may weaken the contribution of this work", "other_comments_or_suggestions": "- Line 187-202. the long list of bullet points is not reader-friendly. Please consider re-organize it.\n- A discussion on how to scale up DiMA (maybe by combining it with structural models like AlphaFold) would be valuable.", "questions_for_authors": "- How does DiMA compare in speed (training/inference) to discrete diffusion models such as ESM3, DPLM?\n- Since the sequence representation can also share some merits with the structure prediction models (such as Alphafold2) such that people repurpose the PLM to folding (eg., ESMFold), can you train or fine-tune DIMA with structure-based tasks and data? i.e. explicitly incorporating structure information into account.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOlhbGcXdF", "reviewer": "Reviewer_Y6i7", "summary": "The paper introduces DiMA, a latent diffusion framework that works on protein language model representations. While protein sequence design has advanced with discrete and autoregressive methods, the potential of continuous diffusion has been under-explored. DiMA is developed through a systematic exploration of architectural choices and diffusion components, enabling it to generalize across multiple protein encoders with 8M to 3B parameters. It achieves high performance across different types of protein representations like sequence-only, dual-decodable, and multimodal ones using the same architecture and training approach. DiMA is extensively evaluated against existing methods using multiple metrics across two protein modalities. It generates novel, high-quality, and diverse protein sequences, outperforming baselines in various tasks such as unconditional generation, family-specific design, motif scaffolding, and fold-conditioned generation. With only 35M parameters, it shows strong performance on multiple benchmarks. DiMA also demonstrates versatile functionality in conditional generation tasks. This work offers a universal continuous diffusion framework for protein sequence generation, providing both architectural insights and practical applications across diverse protein design scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and positive assessment of our work. We appreciate your recognition of our systematic ablation studies and approach to developing protein-specific diffusion parameterizations rather than simply adopting techniques from the image domain. Below, we address each point raised.\n\n> **C1, C2.** Formatting and typos.\n\nThank you for pointing out the formatting issues and typos. We will fix the formatting inconsistencies, including proper capitalization of titles and section headings.\n\n> **Q1.** Line 98: I'm wondering whether the normalize is applied over the length dimension $s$ or the hidden dimension $d$. From the last sentence, it seems that the normalization is over the length dimension, which seems weird for generation.\n\nWe apologize for the unclear description. The normalization is applied over the hidden dimension $d$ rather than the sequence length dimension $s$. Specifically, for each component $d_i$, we precompute the mean and variance over the training data and then apply normalization using these statistics to achieve zero mean and unit variance. We will clarify this in the revised manuscript.\n\n> **Q2.** Do the authors also use ESM-2 decoder for diffusion on CHEAP and SaProt representations?\n\nWe use the same approach across all encoder architectures. For CHEAP and SaProt representations, we fine-tune their corresponding pretrained decoders alongside the diffusion model, similar to our approach with ESM-2. We found that low-dimensional embeddings (e.g., ESM-2 8M, $d$=320) are less robust to small perturbations than higher-dimensional ones (ESM2/SaProt 650M, $d$=1280, CHEAP, $d$=1024). Fine-tuning the decoders helps minimize these effects during diffusion generation.\n\n> **Q3.** What's the rationale behind the heurisitc approach that the reconstruction loss should exhibit an approximately linear increase over diffusion time?\n\nThe key issue with standard noise schedules (linear, cosine) is that they corrupt data very gradually at small timesteps. This means the model spends significant training resources on nearly trivial denoising tasks where input and target are almost identical. Our approach ensures the difficulty of the denoising task increases steadily with each timestep. By designing a noise schedule where reconstruction loss grows linearly with diffusion time, we create learning problems of incrementally increasing difficulty, allowing the model to make consistent progress throughout training rather than facing an abrupt jump in task complexity.\n \n> **Q4.** Typically, the quality of generated samples from diffusion models can be controlled by adjusting the temperature. I'm wondering how the authors set the temperature values in different experiments and for different baselines.\n\nFor baseline comparisons, we use the sampling parameters recommended by the original authors of each method. For autoregressive models like ProGen2 and ProLLAMA, which show suboptimal quality and collapse to highly repetitive sequences on default settings, we performed grid searches to identify optimal temperature and top-p values. DiMA has two analogous parameters for navigating the quality-diversity trade-off: the number of generation steps and self-conditioning rate. The dependencies of quality, diversity and novelty on these parameters are shown in (figures 2 and 3 on https://tinyurl.com/icml25re).\n\n---\n\nWe look forward to incorporating your feedback in our final version and thank you again for your time and insightful comments.", "claims_and_evidence": "All claims are well supported by convincing evidence and experiments.", "methods_and_evaluation_criteria": "Although the method is directly built on existing latent diffusion methods and protein language models, which may limit its novelty, I find it conceptually strightforward and useful in practice, even generalizable to sequence-structure co-design. I especially like the author's comprehensive ablation study and exploring the training and inference strategies for diffusion models instead of directly copying parameterization from image domain, which was commonly done in previous works.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The evaluation of the method is thorough and convincing, covering different aspects of the generated samples and extending to more interesting conditional tasks. I believe the experiments are convicing enough to demonstrate the advantages of the proposed method.", "relation_to_broader_scientific_literature": "See Summary.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "1. The formatting of the paper requires improvement. For instance, the titles of the paper and its subsections should start with uppercase letters. An example would be \"Diffusion on Language Model Encodings for Protein Sequence Generation.\" \n2. Line 51: \"We demonstrate that continuous diffusion on protein embeddings enables effective sequence redand structure generation across multiple tasks and encoder architectures.\" typo: \"redand\" -> \"and\"", "questions_for_authors": "1. Line 98: I'm wondering whether the normalize is applied over the length dimension s or the hidden dimension d. From the last sentence, it seems that the normalization is over the length dimension, which seems weird for generation.\n2. Do the authors also use ESM-2 decoder for diffusion on CHEAP and SaProt representations?\n3. What's the rationale behind the heurisitc approach that the reconstruction loss should exhibit an approximately linear increase over diffusion time?\n4. Typically, the quality of generated samples from diffusion models can be controlled by adjusting the temperature. I'm wondering how the authors set the temperature values in different experiments and for different baselines.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present a novel continuous diffusion framework, termed DiMA. DiMA is composed of three key modules: Frozen pre-trained language models (pLMs), a continuous diffusion module and a decoder. DiMA demonstrates the ability to generate novel protein samples with high pLDDT scores. Furthermore, it supports conditional generation tasks, including protein family-specific design, motif scaffolding, and sequence infilling. All reviewers vote for acceptance.
Decision: Accept (poster)