Protein Design with Dynamic Protein Vocabulary
Authors: Nuowei Liu
Jiahao Kuang
Yanting Liu
Tao Ji
Changzhi Sun
Man Lan
Yuanbin Wu
Venue: NeurIPS 2025 spotlight ()
Keywords: function-based protein design
Primary area: machine_learning_for_sciences

Abstract: Protein design is a fundamental challenge in biotechnology, aiming to design novel sequences with specific functions within the vast space of possible proteins. Recent advances in deep generative models have enabled function-based protein design from textual descriptions, yet struggle with structural plausibility. Inspired by classical protein design methods that leverage natural protein structures, we explore whether incorporating fragments from natural proteins can enhance foldability in generative models. Our empirical results show that even random incorporation of fragments improves foldability. Building on this insight, we introduce ProDVa, a novel protein design approach that integrates a text encoder for functional descriptions, a protein language model for designing proteins, and a fragment encoder to dynamically retrieve protein fragments based on textual functional descriptions. Experimental results demonstrate that our approach effectively designs protein sequences that are both functionally aligned and structurally plausible. Compared to state-of-the-art models, ProDVa achieves comparable function alignment using less than 0.04% of the training data, while designing significantly more well-folded proteins, with the proportion of proteins having pLDDT above 70 increasing by 7.38% and those with PAE below 10 increasing by 9.62%.

Protein Design with Dynamic Protein Vocabulary
Nuowei Liu'
Jiahao Kuang"*
Yanting Liu'
Tao Ji
Changzhi Sun
Man Lan'
Yuanbin Wu!
School of Computer Science and Technology, East China Normal University English Department, College of Foreign Languages and Literatures, Fudan University Institute Artificial Intelligence (TeleAD,, China Telecom
(nwliudstu, jhkuangdstu, ybwuêcs) ecnu. edu...dd.ccr cn, taoji@fudan. edu.
Abstract
Protein design is fundamental challenge in biotechnology, aiming to design novel sequences with specific functions within the vast space of possible proteins. Recent advances in deep generative models have enabled function-based protein design from textual descriptions, yet struggle with structural plausibility. Inspired by clas- sical protein design methods that leverage natural protein structures, we explore whether incorporating fragments from natural proteins can enhance foldability in generative models. Our empirical results show that even random incorporation of fragments improves foldability. Building on this insight, we introduce PRODVA, a novel protein design approach that integrates text encoder for functional descrip- tions, protein language model for designing proteins, and fragment encoder dynamically retrieve protein fragments based on textual functional descriptions. Experimental results demonstrate that our approach effectively designs protein sequences that are both functionally aligned and structurally plausible. Compared to ootte-of-the-rt models, PRODVA achieves comparable function alignment using less than 0.04% of the training data, while designing significantly more well-folded proteins, with the proportion of proteins having PLDDT above 70 increasing by 7.38% and those with PAE below 10 increasing by 9.62%.
Introduction
The natural world presents remarkably intricate landscape of proteins that have evolved over billions of years to perform diverse biochemical functions, such catalyzing reactions and binding specific molecules 2 3. These natural proteins, found across all forms of life, represent only small fraction of the vast sequence space of possible proteins. Designing novel proteins that exhibit user-specified functions remains longstanding challenge in biotechnology.
Recent deep generative models have demonstrated promising potential in function-based protein design. They accept texts as input, either in the form of simple control tags (e.g., ProGen 4
ESM3
(5)) or complex function descriptions containing rich and nuanced information (e.g., ProteinDT [6], PAAG 77. Pinal (8). and generate new protein sequences. One challenge (and main shortcoming) of current models is that, besides satisfying the requirements of input texts, the designed protein should be able to fold into stable diree-dimessional structures.
Equal Contribution.
Corresponding authors are Yuanbin Wu, Tao Ji, Changzhi Sun and Lan. Datasets and codes are publicly available at httpttps:///giiee htt.ttpe//giicbb..coo///e/e/ com/sornkL//ProDVa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To improve foldability, we notice that, though being expert- intert-ittensive and time- time-consuming, classical methods for protein design usually take inspiration from natural structures. For example, rational design
10 leverages physical principles applied to known structures, and directed evolution relies on iterative cycles of mutation based on known structures. Therefore, for generative protein design models, natural question is whether well-folded novel proteins with user-specified functions can be directly assembled by utilizing fragments of natural proteins (e.g. motifs, functional sites, etc.) and their extensive functional annotations.
9
11
12
Before jumping into learned generative models, we first consider two random generative models. Figure I presents a primary empirical analysis, illustrating that even the random incorporation of fragments into amino acid sequences can expand the landscupe of generated proteins and enhance their foldability.
Very high 1PLDOT 201
pLDDT
Vary low (SLDOT x 5.0
rAE:
0.008
DOT 93.08 1AE:
Randam Random
Fral
Dira
Naturdl
-0.006
(b)
-0.004
15-
-0.002
GLDOT: 29 92
ALOUT 20.11
PKC: 23.48
PAE: 27.59
(a)
(a) Visualization of proteins designed by our method, embedded using ESM C and projected with UMAP [13). Random refers to proteins generated by randomly selecting amino acids according to their empirical distribution SwissProt [14). Random+ refers to proteins generated by selecting amino acids and incorporating fragments. Natural proteins randomly sampled from SwissProt (in gray) form a broad distribution, representing the diverse landscape of natural proteins. Proteins generated by Random sampling from all possible sequences (in red scatter points) cluster tightly at the periphery of the natural protein distribution, suggesting that random proteins are less diverse. Proteins generated by Random+ (in yellow scatter points) exhibit a much more diverse distribution than those generated by Random. (b) Performance on PLDDT (t). Our method improves PLDDT by 12% over Randomt, exceeding the well-folded threshold. (c) Performance on PAE (L). Our method reduces PAE by 9% compared to Random+ and is the only model to surpass the well- folded threshold. Notably, it outperforms the state-of-the-art baseline model Pinal in both metrics.
Figure 1:
To further improve the random generative models, inspired by research on dynamic vocabulary we propose a novel approach, PRODVA. Our method integrates text language model for encoding functional descriptions and annotations, a protein language model backbone for generating de novo protein sequences, and fragment encoder for learning protein fragment representations. Similar to other GPT-style models, PRODVA can be trained in a self-supervised manner, using protein sequences split into sets of tokens (amino acids) and fragments. During inference, protein fragments are dynamically retrieved as candidates according to various textual functions provided as inputs. In Figure compared to the two random generative models, Random and Randomt, our method (in blue scatter points) effectively spans the landscape of natural proteins and demonstrates robust ability to design well-folded proteins. Furthermore, in comparison to the state-of-the-e model Pinal, PRODVA achieves competitive alignment with textual descriptions (a
[15 16. [7,,

---

marginal gap of 0.1%) using less than 0.04% of the training data, and significantly outperforms in foldability, generating 7.38% and 9.62% more structurally plausible proteins with pLDDT' > 70 and PAE< 10, respectively.
2
Background
Proteins are macromolccules composed of linear chains of amino acids, with 20 standard amino acids serving as their fundamental building blocks. Continuous subsequences S = 82, 82, within an amino acid sequence P. often referred as fragments segments, play crucial roles in determining the structure and function of proteins. In Figure 2 we utilize InterPro [18], integrated database and diagnostic tool, identify these fragments within proteins. These fragments can be classified into 8 categories: Domain, Family, Homologous Superfamily, Repeat, Conserved Site, Active Site, Binding Site, and most-Trannlational Modification (PTM). Details fragment types are discussed in Appendix B The fragments, along with their corresponding types and descriptions, are referred as functional annotations, denoted by Aype (S) and Adese (5).
8m
Agps Conserved Site Aaye Conserved Site Rix SucY cansareed via
Aasz Seic comsaree sia
Conserve
GIVE LOEL
LOKGYGLOSG
ISL PMATNIC
EQJ PMKAFAP
TIMIGRGDE
FEGAVVAL PH
LL LSVAKIRRA
AL ALVEARYRON
Figure 2:
An example of Q96TW8, illustrating how an amino acid sequence is divided into sets of is used, a single token may represent
tokens and fragments. Note that when the BPE tokenizer [19 multiple amino acids.
The function-based protein design problem is to generate a novel protein P given the textual function description I. The problem be formalized conditional generation task, denoted p(P (). Our goal is to assemble a novel protein using the 20 standard amino acids = {41, 02 .. 020 and set of protein fragments S.
p(P 1) p((11, ".,
Vi, ,Ix) ",,)")
AUS)
3 Method
3.1 Model Architecture
As illustrated Figure PRODVA consists of three components: (1) a Text Language Model (TexiLM), (2) a Protein Language Model (PLM), and (3) Fragment Encoder (FE).
Text Language Model encodes the input textual function description t. To align the text represen- tations Z, TextLM(t) with the embedding space of protein tokens, projection layer is employed to transform Z, into protein tokens H,,
Protein Language Model sively designing novo protein sequences. The PLM employs a fixed static vocabulary Vokens with a tokenizer that divides the protein sequences into discrete tokens. We denote the token embed- ding matrix and the language modeling head matrix of the PLM as W Wokens, E IR. Viaen x and W lokens, E R deux x |Vaaanl respectively, where dPLM is the hidden dimension of the PLM.
serves as a decoder-only architecture backbone, capable of autoregres-
Fragment Encoder Given a set fragment candidates S = {51, 82, "........ Sm.) where m denotes
the number of distinct fragments, the fragment representations can be trivially obtained using the Fragment Encoder, denoted as FE(S1, $2, )... By embedding the fragment representations through a projection layer, we obtain the fragment embedding matrix W Imagments E d Note that the fragment representations mapped the same embedding space as the PLM.

---

Fragroent
Probability Distmbbttio
Probability
Distibution
MLL
NK
WVLG..
[EOS]
ANTP
PLM LM Head
Static Vocabulary
Dynamic Protein Vocahulary
Protein Langunge Model Transformer Layers
[805]
Projection Layer
Projection Layer
Aprsst
TYPR
Language Model
Fragment Ficoddr
LYLONAGTAARFLT
YQISPGN...
Design functional protein sequence with the following characteristies: 1. The designed protein have Ribose-phosphate pymphosphokinuee N-terminal domains [...]
dosc
typo
KLKYCFT... ..CCFFT...HCP
Token
Fregmant
Fragment Candidates
Token/ragment Embedding
Dynamic Selection
Training Loss
InterPro Database
Figure 3: Overview of our model architecture.
3.2 Training
During training, given a training set D' = ((t, P consisting of text-protein pairs, first obtain all functional annotations using InterPro. For each minibatch M {(t,, Aupe (uec()))) variable number of segments S, the corresponding fragment types and descriptions are included.
where B the inner batch size and each protein P; contains
Learning Next Token/Fragment Prediction To generate novel amino acid sequences, we adopt the standard language modeling objective of predicting the next token. There are two key differences.
First, the embedded input functional description H, is incorporated as prefix, serving as a condi- tioning context. Therefore, the preceding context, consisting of the functional description and the previous tokens fragments, denoted Hpre: can be represented
Win Concal (W inkens, in: W frigments)
H pre Concat Win(zee)
where F<i denotes the previous tokens fragments.
Second, the prediction of the next token fragment is determined by the joint distribution of the token and fragment probability distributions. Given the preceding context Hpre defined in Equation the prediction of the i-th token or fragment be computed follows:
W out Concat W tokens, out WT
fragments
W W-(k), out
exp H
p(a, k
H
exp (H W (k*)
out
Therefore, the learning objective of predicting the next token or fragment is to minimize the negative log-likelihood of Equation 2 as the following:
CNT
1 TI
log
Σ
72
p(:;
Hpre)
Learning Functional Annotations tional annotations, Atyee-) and Adese (-), propose type loss LTYPE, and description loss CDesc-
To fully leverage the fragments and their corresponding func-

---

The type loss is designed to capture the intrinsic relationships between fragments and their cor- responding types by classifying the type. To this end, we add a MLP as the classification head, consisting of linear layer with dropout (20 to the Fragment Encoder. Note that the fragment types may be unbalanced. Therefore, we assign a weight w each type across the entire training set D instead of within each minibatch. The weighted type loss LYYPE for the minibatch M can be defined as follows:
exp (q. (q((AAn((()))))
B
TYPEE
W (S,), log
1
Exx (1ir)
Σ
[
where q, MLP(FE(S,)) denotes the logits computed by the classification head and the Fragment Encoder.
Additionally, define the description loss using the InfoNCE loss 21]. Similar to the type loss, introduce an additional MLP as the description projection layer to the Text Language Model. We use the fragment representations u, FE(S,) and their projected description representations (Adee
as positive samples. The description loss aims to learn the relationships between fragments and their descriptions. Therefore, select the remaining nngment-description pairs in a minibatch M as negative samples. The description loss DDSSC is defined as follows:
exp(sim(Ui) ViT)/T)
DESCC
log
B Σ 1=1
S
exp(sim(urj VW//TT
where 7 is the temperature and sim(u, e) | denotes the cosine similarity, which measures the similarity between the fragment representations and the description representations.
Note that the two additional MLP modules (i.e., the classification head and the description projection layer) are introduced during training to facilitate effective learning of the functional annotations and are discarded during inference. The final training objective is defined as
C = NTP + aLyype + DDDSC
(6)
where a and are two significant hyperparameters used to adjust the weights of the loss function.
3.3 Inference
Compared the training paradigm, the strategy for constructing fragmme candidates during inference differs. Assume a set of supporting documents D' = {4,, P.) consisting of text-protein pairs similar to those in the training set. Given a textual function description / as input, the top K most relevant descriptions are retrieved from the supporting documents based on the similarity of their description embeddings. InterPro is then used to identify the fragments of the K protein sequences,
resulting in the fragment candidates {S KK for inference. Additionally, we employ a top-k sampling decoding strategy to further enhance the diversity of the designed novel proteins.
4 Experiments
4.1 Experimental Setups
Implementation Details The Text Language Model initialized with GPT-2 [22] and its pre- trained weights. Both the Protein Language Model and the Fragment Encoder are initialized with ProtGPT2 (23, which GPT-2 based PLM capable of generating de novo protein sequences under an unconditional generation setting. We employ PubMedBERT 24] as the embedding model to retrieve the top K most similar descriptions using the txtai framework, which supported by the Faiss [25] backend. Unless otherwise specified, the supporting documents used in our experiments are identical to the training sets to ensure fair comparison, and we select K = 16 during inference. The results are averaged across three runs with different random seeds. Details of the training implementation are provided in Appendix C.T An ablation study to investigate the effect of loss weighting is presented in Appendix E.4
5

---

Baselines We compare PRODVA with the following function- based protein design methods. Pro- teinDT 6 a multimodal framework consisting of a facilitator that generates protein representations and a decoder that creates the protein sequences from the representations. Pinal g first gener- ates protein structures based on textual descriptions and then designs protein sequences from the structures D flexible combinations of domain annotations, ESM3 [5] is a multimodal generative language model that generates proteins based on multi-hot function keywords. Chroma [26) is diffusion model for protein generation that leverages the ProCap model for natural language understanding, which implement connects GNN backbone with autoregressive language model. Additionally,
PAAG (7] integrates the textual annotations and designs the proteins conditioned on
three random baselines. Random (U) refers to the random selection of amino acids with a uniform distribution. Random (E) involves random selection based on the empirical amino acid distribution observed in SwissProt. Random+ (E) selects amino acids according to the empirical distribution with 90% probability, while incorporating random fragments with a 10% probability.
Metrics To comprehensively evaluate our approach, we consider the following metrics from four perspectives. Detailed definitions and implementations provided in Appendix D Sequence Plausibility, For rapid validation of designed protein sequences, we employ Perplexity (PPL) and
Repetitiveness (Rep)
27 evaluate sequence-level plausibility. Foldability. Following 88)
utilize ESMFold (8 to predict the 3D structures of designed proteins and compute the predicted Local Distance Difference Test (pLDDT) [29] as a per-residue measure of local confidence (0-100, higher is better). Additionally, we also compute the Predicted Aligned Error (PAE) [29]. which measures the confidence in the relative position of two residues within the predicted structure (lower is better). Consistent with previous researches [29 30, 31 8
we adopt pLDDT above 70 and PAE below 10 to define well-folded predicted structures. Language Alignment. To mitigate the high cost of wet-lab experiments, we employ oracle model-based metrics to evaluate the alignment between proteins and functions. Following [8] we use ProTrek (32, which evaluates alignment using cosine similarity, referred to as the ProTTrek Score, within joint embedding space. We also use Evol 33.. fine-tuned on our downstream datasets, computing the Evol. Score as the cosine similarity between textual descriptions and predicted functions. Higher scores from both models indicate better alignment with the specified function. Additionally, we employ two retrieval-based metrics. The first is Keyword Recovery [5), which measures the recovery of function keywords in the designed proteins using InterProScan [34]. And the second is Retrieval Accuracy [6], which assesses how well the designed proteins aligns with their funetional descriptions by evaluating whether their representations exhibit the highest similarity among all candidate pairs. Sequence Diversity. MMseqs2 (35) is used to compute sequence-leve similarity between each pair of proteins within a batch, where all sequences are generated based on the same textual description. A higher score indicates a greater ability to design diverse protein sequences.
4.2 Designing Proteins from Function Keywords
Datasets To ensure a fair comparison with ESM3, we use the same test set employed by ESM3. This test set is derived from the Continuous Automated Model EvaluatiOn (CAMEO) targets released between May 1, 2020 and August 1, 2023 [36]. Proteins lacking function keywords or containing only Family or Homologous SuperFamily keywords are excluded, resulting in a final test set comprising 507 proteins, referred to as the CAMEO subset. For training and validation, we construct the dataset by annotating the function keywords of proteins in SwissProt using InterPro, resulting in approximately 412K proteins. We randomly sample 5% of this dataset as the validation set, with the remaining used for training. Details are included in Appendix C.2
Results Pinal is a state-of-the-art baseline model capable of designing well-folded, user-specified amino acid sequences. As shown in Table I Pinal consistently outperforms other baselines across most evaluation metrics. We highlight our key findings as follows:
(1) Under the same training data setting, PRODVA consistently surpasses both ProteinDT and PAAG. ProteinDT and PAAG show inferior performance even after fine-tuning with additional data consisting of function keywords. In contrast, PRODVA, trained on the same dataset, consistently surpasses these two models, achieving improvements of 34.35% in PLDDT and 17.51% % in PAE.
2 Pinal was trained on 1.76B pairs from SwissProt and UniProtKB, and has likely encountered most proteins and annotations in various forms. Thus, we do not distinguish baselines by zero-shot settings.

---

Table 1: Performance on the CAMEO subset (Best, Second Best). "Pairs" refers the text- text- proteen pairs used during training and "Params" " denotes the model's parameter size. Violin plots are provided Appendix E.6 7 indicates that the training scripts are not publicly available.
Sequence Plausibiliay
Foldability
Language Alignment (in
Sequence Diversity (t)
Models
#Paits #Params
PPL.
Rep
pl.D1I
S
PAR
10
ProTrek Score (1)
Keyaned
Rernivall Accuracy (1)
(1)
(.)
(1)
(1)
(2)
(1)
Recovrry (1)
Natural
467.64
0.02
81.21
90.53
7.08
$2.05
21.09
100.00
70.81
2/71.95
7.50
6.05
Randox (E)
3046.64
0.01
27.46
0.00
23.70
0.00
6.59
0.00
5.13
99.78
Bandox+ (6)
966.24
0.01
62.38
32.65
17:23
9.28
3.29
0.00
5.79
98.97
ProteinDT
S41K/22M
1405,70
0.11
38.70
0.20
26.25
0.00
3.89
0.05
7.43
99.72
ProteinDTTr
392K/129M
23.90
0.42
6.28
16.57
Pinal
1.768/2B
584.22
0.15
66.50
47.21
[4.57
33.53
14,57
30.46
$1.68
82.72
PAAG
130K/L.3B 3B
2571.10
0.02
33.14
0.00
23.31
0.00
5.21
0.23
7.10
99.02
PAAGTr
392K/1.3B
2004.01
7.82
Chroma
4SK/334M
1322.37
0.03
61.66
28 96
13.01
39,03
2.97
D.11
6.57
97.21
ESME!
53MM/1.48
279.78
0.33
59.79
31
17
21.37
3.76
5.49
11.97
96.77
PRODVA
392K/1.8B
656.04
0.01
75.88
77.00
6.39
83.88
14.43
30.31
44.22
98.58
Furthermore, our approach surpasses ProteinDT by an average of 8.15% on the ProTrek Score and approximately 30% on Keyword Recovery, demonstrating the effectiveness of our proposed learning methods in low-resources settings.
(2) PRODVA remains within a reasonable PPL. range and demonstrates the capability to design well-folded proteins. Although ESM3 achieves the lowest PPL, the sequences it generates display repetitive patterns, leading to relatively lower pLDDT scores and higher PAE values. Compared to PRODVA, the number of well-folded proteins is lower by 45.51% 51% and 66.48% in terms of pLDDT and PAE, respectively.
(3) PRODVA uses only 0.02% of the text-protein pairs used to train Pinal, yet achieves competi- tive performance. Compared to PRODVA, Pinal shows marginal improvements, with increase of 0.14% and 0.12% ProTrek Score and Keyword Recovery, respectively. Moreover, method demonstrates significantly superior performance in foldability, outperforming Pinal by 9.38% in pLDDT and 8.18% in PAE, highlighting the effectiveness in designing structurally plausible proteins.
4.3 Designing Proteins from Textual Descriptions
Unlike ESM3, which accepts function keywords but lacks the understanding of natural language, PRODVA accepts arbitrary textual descriptions as input. Therefore, we utilize the Mol- Instructions 37 protein design-related instructions as the test set. The Mol-Instructions dataset comprises approximately 200K instruction- pnerrcttio--roteinn pairs, with each instruction generated from 20 UniProtKB-deriver features (8 that capture both functional (e.g., catalytic activities, Gene Ontology annotations
Datasets
(39]) and structural properties (e.g., helix, turn). These structured annotations are converted into instructions using diverse templates enriched by LLMs. We combine the dataset with text-protein pairs derived from SwissProt, resulting in total of 712K text-protein pairs used as the training set. Additionally, we allocate 10% of the Instructttss protein design-related training set for validation. Details are included Appendix C.2
Results Table 2 presents comparison between PRODVA and other baseline models Mol- Instructions. Our key observations are highlighted as follows:
(1) Most baseline models struggle to design proteins that are both well-folded and well-aligned, whereas PRODVA successfully accomplishes this. Random+ (E) serves as a strong baseline, demonstrating that incorporating even random fragments can substantially improve the structural plausibility of proteins. Among the other baselines, only Pinal and Chroma are capable of designing well-folded proteins when compared to randomly generated sequences. However, the sequences generated by Chroma are not well-aligned with the functional descriptions. A possible explanation is that Chroma trained a significantly smaller number of caption-protein pairs compared to other models, resulting in inferior performance in understanding textual descriptions. In contrast, PRODVA consistently demonstrates the ability design well-folded proteins with user-specified functions.
(2) Incorporating additional data may potentially improve performance, particularly in terms of language alignment. After fine-tum linettuning ProteinDT and PAAG, these models demonstrate relatively
7

---

Table 2: Performance on the Mol- iol-Inssructions protein design task (Best, Second Best). "Pairs" refers to the text-protein pairs used during training and "Params" denotes the model's parameter size. Violin plots are provided in Appendix E.7 indicates that the training scripts are not publicly available.
Sequence Plausibllity
Foldability
Language Alignment (in
Sequence
Models
#Pairs *Params
PPl.
Rep
PLDD1
%/.
PAR (3)
% <
ProTrek Sourc (1)
Evol Score (1)
Retrieval
Diversity (1)
(L)
(4)
<T)
(1)
(1)
Accuracy (1)
Natural
318.15
0.02
80.64
81.27
9.20
65.73
27.00
60.33
84.85
Sandon (1)
2484.03
0.01
22.96
0.16
24.85
0.56
1.03
36.23
6.89
97.01
Sandon (E)
3136.88
0.01
25.77
0,20
24.71
0.60
1.04
34.11
6.78
99.56
Sandon+
846.01
0.01
37.03
17.91
7.52
38.65
6.13
ProteiaDT
51LK/729M
1576.23
0.07
38.29
0.98
25.13
0.10
1,20
10.57
9.28
99.23
ProteinDTyr
712K/729M
1213.38
0.04
51.42
25.61
18.57
23.92
13.89
52.84
47.29
79.87
Pinal!
1.76B/2B
308.97
68.97
58.44
PAAG
130K/1.38
2782.70
0.02
28.39
0.37
25.38
0.10
1.29
34.39
7,06
99.15
PAAG
712K/1.3B
1332.35
0.04
50.37
23.86
19.96
21.99
10.04
49.69
33.66
$6.09
Chroma!
45K/33IM
1370.21
20.17
28.62
PRODVA
712K/188
415.63
0.02
76.86
76.35
8.66
68.06
17:40
51.10
59.07
83.29
improved performance across all metrics, with improvements of 12.69% and 8.75% on the ProTrek Score, and 12.27% and 15.30% on the Evol Jama Score, respectively. These results indicate that training with more data can significantly enhance model's ability understand textual functional descriptions written in natural language. In comparison, Pinal has been trained on approximately 1.76 billion text-protein pairs using both manually annotated and synthetic annotations, and demonstrates competitive performance on foldability and language alignment metrics. However, the large-scale datasets and training scripts used by Pinal have not yet been open-sourced. In contrast, our approach utilizes only 0.04% of the data and surpasses Pinal by generating 7.38% and 9.62% more structurally plausible proteins with pLDDT > 70 and PAE < 10, respectively. For language alignment, PRODVA also demonstrates highly comparable performance, being only marginally outperformed by Pinal with gap of 0.1 on the ProTrrek Score and 2.32% on the Evol. Score.
(3) PRODVA demonstrates competitive sequence diversity compared to other baselines. We find that baselines that struggle to design well-folded or well-aligned proteins consistently exhibit higher sequence diversity, comparable to that of random methods. Furthermore, textual descriptions containing more constraints than simple keywords generally result in lower sequence diversity, suggesting that complex function descriptions may restrict the design space. By leveraging the top-k sampling strategy during decoding, PRODVA achieves slightly better performance than Pinal on Mol-Instructions and surpasses it by 15.86% on the CAMEO subset.
4.4 Unconditional Protein Generation
Although PRODVA is designed for conditional protein design tasks, we demonstrate that it can be adapted for unconditional protein generation. By fixing the input instruction to Design a novel protein sequence, the problem addressed by PRODVA effectively transforms into an unconditional protein generation task. To further ensure that the input acts as a no-op, we replace the retrieval method with the random selection of fragments For fair comparison, each model generates the same number of protein sequences as the size of the Mol-Instructions test set. The results are shown in Table 3 PRODVA outperforms all baseline models on the unconditional protein generation task. baseline, Pinal, PRODVA generates 22.7 22.76% and 11.81 % more
Table Performance on the unconditional protein generation task (Best, Second Best). The fine-tuned models and PRODVA are initialized with the weights obtained in Section 4.3 without additional fine-tuning for this task.
PPL (1)
Rep (4)
PLDDT
70
PAE
%
Models
(t)
(1)
(1)
(1)
Prr
593.06
17.92
47.79
0.02
26.56
0.00
PAAGFT
1327.98
3.55
50.32
23.83
19.95
22.24
Pinal
411.93
14.05
70.11
57.02
12.76
48.44
PRODVA
476.02
1.47
77.52
79.78
9.32
60.25
from the training set described in Section 4.3
Compared to the state-of-the-a well-folded proteins in terms of PLDDT and PAE, respectively. Furthermore, compared to other fine-tuned models, PRODVA achieves substantially superior performance. These findings highlight the effectiveness of our approach in both conditional and unconditional protein generation settings.
8

---

4.5 Comparison with a Vanilla Multimodal Approach
To further assess the effectiveness of our approach, we adopt methods 40 other fields. such as computer vision, to train vanilla multimodal baseline. This baseline model integrates GPT-2 for embedding, which serves as the imput to ProtGPT2. Compared to PRODVA, all dynamic protein vocabulary components are excluded, making next token prediction the only learning objective. 'he baseline is trained using the same experimental setups described Section
commonly used in
4.1 and the same dataset used in Section 43
6000
PTL
10
0.8
70
60
50
40
30
1000
0.2-
20
10
Netural
VenTa
ProDiss
Naturi
Venils
ProDis
Barual
Veniil
PrD
Neturll
Vari
ProDAs
(a)
(b)
50-
20
Venila
Natural
Werill
Prodve
Natural
Variaa
Provee
-10
28
25 56
206
(c)
(d)
Figure 4: Experimental results comparing PRODVA with vanilla multimodal baseline Mol- Instructions. (a) illustrates performance on sequence plausibility metrics. (b) and (c) show perfor- mance on foldability metries. (d) presents performance language alignment metries, where deeper colors in the upper right corner indicate better performance.
Figure illlstratts that the vanilla multimodal baseline outperforms most models presented in Table 2. indicating that even simple integration of multimodal representations from textual descriptions
convey strong and detailed information. However, compared PRODVA, the proteins designed by the baseline exhibit more repetitive patterns, resulting in a 4.63% decrease in PLDDT scores and 2.71% increase in PAE scores. Furthermore. PRODVA achieves a 13.66% higher proportion of proteins with PLDDT > 70 and a 9.57% higher proportion with PAE < 10. These results underscore the importance of incorporating fragments and functional annotations, which enable the model to design proteins with more plausible structures. Furthermore, PRODVA significantly outperforms the baseline on the ProTrek Score with an average improvement of 7.77%, demonstrating its superior ability to design well-aligned proteins that more effectively meet specific user requirements.
4.6 Retrieving the Top K Most Relevant Descriptions
The value of K affects the size the fragment candidate set during inference. Therefore, selecting an optimal K has a significant impact on the model's performance. As shown in Figure 5 larger fragment candidate descriptions are retrieved. Although the model's performance on PPL and PAE deteriorates with increasing K, the average PAE scores remain within reasonable range (i.e., PAE 10), For language alignment metrics, we observe that the ProTrrek Score decreases when selecting more than the 16 most relevant descriptions. A possible explanation that retrieving the top most relevant descriptions helps filter out irrelevant proteins and fragments, thereby enhancing the model's performance in designing aser-specified proteins. These results demonstrate that simply increasing the number of fragment candidates during inference does not consistently yield positive outcomes, highlighting the
consistently lead to higher pLDDT scores, until more than 64 relevant

---

importance of selecting an optimal Top K value. We provide a more detailed analysis of this pattern
with additional experiments Appendix E.9
7L(1
PLDOT Netaio37
P41(11
PAC
Score (?)
Score (1)
150.0
17.0
51.5
9.0
70.2
16.5
51.0
70.0
65.
50.5
200.0 0
67.
0.0
FFL
of PLDOT
PLDDT
13
% of PAE
PA5
00.0
15.5
50.0
2
16
8 16 M4 150
8 32
61
128
Top
Top K
Top K
Tap
(a)
(b)
(c)
(d)
Figure 5: Analysis regarding the selection of Top K most relevant descriptions during inference Mol-Instructions. Results on the CAMEO subset are provided in Appendix E.8
5
Related Work
The protein design problem can be broadly categorized into two paradigms: unconditional and conditional protein design. Unconditional methods aim to generate novel protein sequences or structures without specific constraints. Representative sequence-basee
approaches include DARK RITA 43,, ProtGPT2 [23], and DPLM 44 while structure- based methods include Multiflow
42
and DPLM-2 [46]. In contrast, conditional protein design seeks to generate proteins under A key of conditional generation function-basee protein design. Early models such as ProGen [4], ProGen2 [51]. and ESM-3 (S condition generation control tags multi-hot encodings. However, these approaches limited in expressiveness and cannot fully interpret natural language. Recent methods, including ProteinDT [6]. Pinal [8), PAAG [7], and Chroma (26), aim to support arbitrary textual functional descriptions. Despite this, they often underexploit rich functional annotations or employ less effective learning strategies, resulting in protein sequences that poorly folded misaligned with the user-specified functions. We discuss more related work in Appendix A
45
desired functional structural constraints 47
48
49
50
Conclusion
In this paper, motivated by classical approaches protein design, we propose a novel method, PRODVA, which integrates a text language model, protein language model backbone, and fragment encoder. On protein design tasks based on both function keywords and textual descriptions, our approach demonstrates competitive performance in language alignment and significantly outperforms slate-of-the-art baselines in designing well-folded proteins.
Acknowledgement
The authors wish to thank all reviewers for their helpful comments and suggestions. We also thank Jie Wang for his insightful discussions. The corresponding authors are Yuanbin Wu, Tao Ji, Changzhi Sun and Man Lan. The computations in this research were performed using the CFFF platform of Fudan University.
References
[1] Joseph Kraut. How do enzymes work? Science, 242(4878):533-5 540. 1988.
[2] Bruce Alberts. The cell collection of protein machines: preparing the next generation of molecular biologists. cell, 92(3):291-2944 1998.
[3] Sharon Saammes-Schiffed and Stephen J Benkovic, Relating protein motion to catalysis. Annu. Rev. Biochem.. 15(1) 2006.
10

---

[4] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprini arXiv:2004.03497, 2020.
[5] Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf A. Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with language model. bioRxiv, 2024.
[6] Shengchao Liu, Yanjing Li, Zhuoxinran Li, Anthony Gitter, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Arvind Ramanathan, Chaowei Xiao, et al. text-guided protein design framework. Nature Machine Intelligence, pages 1-12, 2025.
[7] Chaohao Yuan, Songyou Li, Geyan Ye, Yikun Zhang, Long-Kai Huang, Wenbing Huang, Wei Liu, Jianhua Yao, and Yu Rong. Annotation- guiided protein design with multi-level domain alignment. arXiv preprint arXiv:2404. 16866, 2024.
[8] Fengyuan Dai, Shiyang You, Chentong Wang, Yuliang Fan, Jin Su, Chenchen Han, Xibin Zhou, Jianming Liu, Hui Qian, Shunzhi Wang, et al. Toward de novo protein design from natural language. bioRxiv, pages 2024-08, 2024.
[9] HW Hellinga. Rational protein design: combining theory and experiment. Proceedings of the National Academy of Sciences, 94(19):10015-10017, 1997.
[10] Ivan V Korendovych. Rational and semirational protein design. Protein engineering: methods and protocols, pages 15-23, 2018
[11] Christian Jäckel, Peter Kast, and Donald Hilvert. Protein design by directed evolution. Annu. Rev. Biophys., 37(1):153-177 2008.
[12] Michael J Dougherty and Frances H Amold. Directed evolution: new parts and optimized function. Current opinion in biotechnology. 20(4):486-491, 2009.
[13] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018
[14] Amos Bairoch and Rolf Apweiler. The swiss-prot protein sequence database and its supplement trembl in 2000. Nucleic acids research, 28(1):45-48, 2000.
[15] Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. Copy is all you need. arXiv preprint arXiv:2307.00662, 2023.
[16] Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi, Yuexian Zou, and Shuming Shi. Retrieval is accurate generation. arXiv preprini arXiv:2402.17532, 2024.
[17] Yanting Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Xiaoling Wang. Generation with dynamic vocabulary. arXiv preprint arXiv:2410.0848/ 2024.
[18] Matthias Blum, Antonina Andreeva, Laise Cavalcanti Florentino, Sara Rocio Chuguransky, Tiago Grego, Emma Hobbs, Beatriz Lazaro Pinto, Ailsa Orr, Typhaine Paysan-Lafosse, Irina Ponamareva, et al. Interpro: the protein sequence classification resource in 2025. Nucleic Acids Research, 53(D1))D444 2025.
[19] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: /00iv:/5008.70000 2015.
[20] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1999- 2014.
[21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0377 2018.

---

[22] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[23] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications, :3(14),348, 2022.
[24] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specifie language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23, 2021.
[25] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):335-545 05755557,440 2019.
[26] John B Ingraham, Max Baranov, Zak Costello, Karl W Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M Lord, Christopher Ng-Thow-Hing, Erik R Van Vlack, al. Illumi- nating protein space with a programmable generative model. Nature, (23(7989)11070-1077 2023.
[27] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.043) 2019.
[28] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
[29] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-588) 2021.
[30] Mehmet Akdel, Douglas EV Pires, Eduard Porta Pardo, Jürgen Jänes, Arthur O Zalevsky, Bálint Mészáros, Patrick Bryant, Lydia Good, Roman Laskowski, Gabriele Pozzati, et al. A structural biology community assessment of alphafold2 applications. Nature Structural & Molecular Biology, 29(1
2022.
[31] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D439-D444 2022.
[32] Jin Su, Xibin Zhou, Xuting Zhang, and Fajie Yuan. Protrek: Navigating the protein universe through tri-modal contrastive learning. bioRxiv, pages 2024-05, 2024.
[33] Nuowei Liu, Changzhi Sun, Tao Ji, Junfeng Tian, Jianxin Tang, Yuanbin Wa, and Man Lan. Evollama: Enhancing Ilms' understanding of proteins via multimodal structure and sequence representations. arXiv preprint arXiv:2412.1 16118, 2024.
[34] Philip Jones, David Binns, Hsin-Ya Yu Chang, Matthew Fraser, Weizhong Li, Craig McAnulla, Hamish McWilliam, John Maslen, Alex Mitchell, Gift Nuka, et al. Interproscan 5; genome-scale protein function classification. Bioinformatics, 30(9): 2014.
[35] Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology, 15261):002- 2017.
[36] Jürgen Haas, Alessandro Barbato, Dario Behringer, Gabriel Studer, Steven Roth, Martino Bertoni, Khaled Mostaguir, Rafal Gumienny, and Torsten Schwede. Continuous automated model evaluation (cameo) complementing the critical assessment of structure prediction in casp12. Proteins:
Structure, Function, and Bioinformatiics 86:387-398, 2018.
[37] Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: scale biomolecular instruction dataset for large language models. arXiv preprint 2306.00000 2023.
12

---

[38] Uniprot: the universal protein knowledgebase in 2023. Nucleic acids research, 51(D1): D531, 2023.
[39] Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Budler, Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al. Gene ontology: tool for the unification of biology. Nature genetics, 25(1):25-22 29, 2000.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36:34892-34916, 2023.
[41] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[42] Lewis Moffat, Shaun M Kandathil, and David T Jones. Design in the dark: learning deep generative models for de novo protein design. bioRxiv, pages 2022-01, 2022.
[43] Daniel Hesslow, Niccoló Zanichelli, Pascal Notin, Iacopo Poli, and Debora Marks. Rita: study on scaling up generative protein sequence models. arXiv preprint arXiv:2205.05789 2022.
[44] Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXhv:2402.18567, 2024.
[45] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Gener- ative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint urXiv:2402.04997, 2024.
[46] Xinyou Wang. Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410.13782, 2024.
[47] Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative re- finement graph neural network for antibody sequence-structure co-design. arXiv preprint arXiv:21 10.0424 2021.
[48] Joseph L Watson, David Juergens, Nathaniel Bennett, Brian Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976): 2023.
[49] Jason Yim, Andrew Campbell, Emile Mathieu, Andrew YK Foong, Michael Gastegger, José Jiménez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Frank Noé, et al. Improved motif-scaffoo scaffo--ffoll with se (3) flow matching. ArXiv, pages arXiv-2401, 2024.
[50] Xiangxin Zhou, Dongyu Xue, Ruizhe Chen, Zaixiang Zheng, Liang Wang, and Quanquan Gu. Antigen-specific antibody design via direct energy-based preference optimization. Advances in 2024.
Neural Information Processing Systems,
[51] Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11) (111)))6697758 2023.
[52] Typhaine Paysan-Lafosse, Antonina Andreeva, Matthias Blum, Sara Rocio Chuguransky, Tiago Grego, Beatriz Lazaro Pinto, Gustavo A Salazar, Maxwell L Bileschi, Felipe -linares-óppz, Laetitia Meng- Pangg-Papaaatthos, et al. The pfam protein families database: embracing ai/ml. Nucleic acids research, 53(DD))
2025.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[54] Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. lddt: a local superposition-frre free score for comparing protein structures and models using distance difference tests. Bioinformatics, 29(21): 2013.
[55] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. BioRxiv, pages 2023-10, 2023,
13

---

A More Related Work
explores augmenting the fixed vocabulary of lan- guage models with phrases extracted from retrieved documents, typically using heuristics such as Forward Maximum Matching or N-gram extraction. While effective for natural language tasks, these approaches differ from ours in three key points. First, function- based protein design a multimodal problem that requires aligning natural language descriptions with amino acid sequences, making cross-modal retrieval and generation more challenging. Second, prior methods extract phrases based on token-level or word-level heuristics, often resulting in semantically shallow units that serve mainly as grammatical components. Third, our method leverages InterPro [8] to identify biologically meaningful fragments with functional annotations, enabling more interpretable protein vocabulary construction, leading to better generation user-specified protein sequences.
Prior work on dynamic vocabulary [1 16
B Details of InterPro and Fragment Types
InterPro [8] is an integrated database and diagnostic tool that facilitates the functional analysis of proteins by classifying them into families and predicting domains and functional sites. It achieves this by utilizing predictive models from several member databases, such as Pfam [52]. While InterPro offers user-friendly GUI for searching, in our experiments, we may utilize InterProScan (34), software package that can be executed as a command-line tool sean sequences against the InterPro database.
Table
4 presents the details of the 8 fragment types identified by InterPro.
Table 4: Descriptions of 8 fragment types identified by InterPro.
Types
Descriptions
Domains
Distinct functional, structural or sequence units that may exist in variety of biological contexts.
Family
A group of proteins that share a common evolutionary origin reflected by their related functions, similarities in sequence, or similar primary, secondary
tertiary structure.
Homologous Superfamily
A group of proteins that share a common evolutionary origin reflected by their related functions, similarities in sequence, similar primary, secondary or tertiary structure. (usually predicted by Hidden Markov Models)
Repeat Conserved Site
A short sequence that is typically repeated within a protein. A short sequence that contains one more conserved residues.
Active Site
A short sequence that contains one or more conserved residues, which allow the protein bind to a ligand.
Binding Site
A short sequence that contains one or more conserved residues, which form protein interatcion site.
PTM
A short sequence that contains one more conserved residues, Post- Traditional Modification site.
C
Details of the Experimental Setups
C.1
Training Details
PRODVA is trained using the AdamW optimizer 53 with 3 = 0.9, 32 0.95, and a gradient clipping of 1.0. The maximum learning rate is 5% of training steps. The learning rate remains constant after warmup and follows "I-sqrt" decay schedule during the last 10% of training steps. The minibatch size set to to ensure memory efficiency, with overall batch size of 64. During training, the weights of the Text Language Model are frozen. Training is conducted for 10K steps on the CAMEO subset and 20K steps on The loss weights are set to 3 = 0.2.
x 10 10-4, with linear warmup over the first
14

---

C.2
Dataset Construction Details
Table 5: Statistics of the datasets.
Datasets
#Training
#Validation
#Test
CAMEO subset 36
391,933
20,629
507
SwissProCCIAP 6
541,159
Mol-Instructions (Protein Design)
171,089
19,010
5,876
The statistics of the datasets used in Sections 4.2 and 4.3 are presented in Table
CAMEO subset
Input (Textual Function Keywords)
Generate sequence for protein that integrates the following function keywords: SUMF_dom. The designed protein sequence
Output (Designed Protein)
Region(s)
keywordis)
W.TEW//WW/.AAAAAAAAAAAAAAAAAAAAA BANTVEAGPLSTTPVGGING
SATTT//STTTGGRRRREOOOOOOOOOOOAOD LACCARRHGGYORDVAMMGFRRREEET
Mol-Instructions (Protein Design)
Input (Textual Function Description)
Design
sequence that exhibits desired activity and specificity. The presence of Mg(2+) is necessary for the protein function the desired environment.
2. The AMP, (6S)-NADPHX binding site should be located in region of the protein that accessible to the ligand.
3. The designed protein should have ATP binding [...]
The designed protein sequence is
Output [Designed Protein)
Regionts) associated with userspecified function(s)
LEAUUU/SSSSGGGRRAAOOGGGGGGAAAAAD CANGIAAGAAPAAISSHHHAAAAAAAARRRRA
Figure 6: Examples from the CAMEO subset and Mol-Instructions Regions associated with the input functions are highlighted both the protein structures and sequences. These regions include specific domains or arbitrary domains related to user-specified functions.
The CAMEO subset consists of function keyword wwwweddproteinn pairs, where the function keywords formulated as simple instructions tailored to our use cases. Fach protein may be associated with one more function keywords. In contrast, Mol-Innttuttions consists of instruction-prote pairs, where each instruction may include one or more diverse functional descriptions written in natural language. Examples of these two datasets are illustrated in Figure 6
D Details of the Metrics
In this section, we present detailed definitions and implementations of the metries introduced in Section 4.1
15

---

D.1 Sequence Plausibility Metrics
Perplexity Perplexity (PPL) a commonly used metric in Natural Language Processing to evaluate the quality of sentences generated by language model. In our experiments, we use PPL to assess how effectively a model has learned intricate patterns in protein sequences and to provide a rapid validation of protein structures. Lower PPL generally indicates more accurately folded structure. Specifically, we employ ProtGPT2 23
to evaluate the PPL as follows:
PPL = exp
log p(a: xess)
where 22, , In. sequence of tokens tokenized using the ProtGPT2 tokenizer. Additional PPL evaluations using alternative language models are provided in Appendix E.2
Repetitiveness Some previous studies 22 have suggested that repetitive patterns in amino acid sequences may result in regions of intrinsic disorder, which are typically associated with lower PLDDT scores. Inspired by Rep-N [27). which measures repetition at the N-gram level, we adopt Repetitiveness, a metric that calculates the proportion of repeated subsequences within the entire sequence.
D.2 Foldability Metrics
PLDDT The predicted Local Distance Difference Test (pLDDT) is a per-residue evaluation metric that produces a pLDDT score array of size IR 1xn where denotes the length of the amino acid sequence. Each score in the array ranges from 0 to 100 and estimates the agreement between the predicted and experimental structures. It is based on the Local Distance Difference Test Co (54), which evaluates the correctness of the local atomic distances without requiring superposition. We report the average PLDDT score across all residues.
The Predicted Aligned Error (PAE), computed by ESMFold, quantifies the model's 's confidence in the relative positioning residue pairs within a predicted structure. It represents the expected positional error, in Å, at residue X when the predicted and actual structures are aligned on residue Y, thereby indicating the reliability of domain packing and inter-domain arrangement. Consequently, the PAE is represented as a matrix of size Rax", where n is the length of the amino acid sequence. We report the average PAE score across this matrix.
PAE
Very high (pLDDT > 90)
350
Confident (90 > PLDDT 70)
Low (70 pLDDT > 50)
300
Very low (pLDDT 50)
Alined Ressivue
Pradicted Afg aed Error
250
15
100
30
50
100
159
200
25:
300
350
400
Scored Residue
(a)
(b)
Figure 7: An example of G9WEQ2. (a) The PLDDT score the protein, where blue regions indicate well-folded areas. (b) The PAE matrix of the protein, with darker green indicating higher predicted structural correctness (i.e., lower predicted error).
16

---

D.3 Language Alignment Metrics
ProTreek Score The ProTrek Score defined as the cosine similarity between a textual function description / and designed protein sequence P. Since PRODVA generates protein sequences directly, without an intermediate step for designing structure backbones, only the text and sequence encoders of ProTreek [32] are utilized. Specifically, we employ the pre-trained ProTrrek 650M P as the oracle model. The ProTrek Score is formally defined as:
ProTrek Score (im)(T(t), Tp())
where sim(..-) denotes the cosine similarity function, and (t) and (P(P) represent the embeddings of the textual description and the protein sequence, respectively. Note that we follow the original implementation of the ProTrek Score as described in [8]. In contrast, the latest version of the paper defines the ProTrek Score with an additional division by the model temperature.
EvoLlama Score Unlike the ProTrek Score, which is directly defined based on the similarity between the function description t and the protein sequence P, propose generative approach to compute the alignment using EvoLlama (33] Specifically, we adopt EvoLlama with a 650M ESM2 protein sequence encoder and 3B Llama-3.2 text decoder. The model is randomly initialized and trained from scratch using the Mol-Instructions de novo design datasets described in Section
During evaluation, the fine-tuned EvoLlama takes the protein sequence P and simple prompt. The function of the protein is, as inputs and generates predicted function description t'. Assume that the ground truth description t and the generated description t' be tokenized into and ' tokens, respectively. The EvoLlama Score is then defined as:
1 k
1 R.
EvoLlama Score
sim(
Embed (t),
Embed(t"))
where Embedd.) denotes using PubMedBERT 24 as the embedding model.
Keyword Recovery ESM3 (5] lacks the capability to understand arbitrary natural languages,
limiting it to accepting only multi-hot function keywords as inputs. Keyword Recovery is a protein- level metric that measures the percentage of function keywords identified in the designed proteins using InterProScan [34] Let the reference set of function keywords (ground truth) be denoted as Knts and the set of function keywords associated with the designed sequence be denoted as Kpred-
The Keyword Recovery metric can then be defined as follows:
pree n K
Keyword Recovery
Kgatl
Retrieval Accuracy Retrieval Accuracy 66 evaluates model's ability to correctly associate a textual function description with its corresponding protein sequence P. is computed by comparing the similarity between the text description embedding and multiple protein embeddings, including one correct match and T randomly retrieved negative samples. The metric is defined as the proportion of instances in which the correct protein sequence exhibits the highest similarity to the function description among all candidates. Note that the original implementation relies on ProteinCLAP 6 for both text and protein embeddings, whereas we adopt ProTTrek (32 to achieve faster and more accurate representations, denoted as Embed. our evaluation, T is set to 20, representing the most challenging setting in 66. The Retrieval Accuracy for a text-protein pair (1, Pi) is defined as:
T
Retrieval Accuracy@T = sim (Embed(t), (mbee((P)) = max {sim (Embed(t). Embed (P))} J=l
where 1(:) is the indicator function.
August 2024 version:
08. 00t.e4........666681188
full
pdf
17

---

D.4 Sequence Diversity Metric
Sequence Diversity Sequence Diversity measures a model's ability generate diverse protein sequences under identical conditions (i.e.. given the same function as inputs). Given a textual descrip- tion, the model is tasked with designing a batch of protein sequences. denoted {P1. P2. .NN). Formally, Sequence Diversity is defined as:
Sequence Diversity
2 N(N - ISISISN
Σ
sim PJ)
where MMseqs2 (35] is used to compute the sequence-level similarity, sim(:,). between each pair of proteins P. and Pj.
E
Additional Evaluations
E.1
Additional UMAP Visualization of Designed Proteins
In Figure [ we present a visualization of proteins embedded using ESM C, a state-of-the-ar foundation model designed for computing protein representations. To further validate the intuition behind PRODVA, we utilize Pinal to compute the protein representations. Although Pinal, like PRODVA, is not specifically designed for protein representation, consists of two primary modules: a text-tosttructure module and a SaProt (55) module, the latter of which can serve as an embedding model. The weights of the SaProt module are initialized using the Pinal checkpoint [
Following the approach used Figure Ta). the additional UMAP visualization is presented in Figure 8 The results are highly consistent with those obtained using ESM C. Proteins generated by Random form distinct clusters, while those from Random+ exhibit significantly more diverse distribution. As expected, proteins designed by PRODVA span the landscape of natural proteins.
-c0155
Natunall Protern
..00.
Dupt
Figure 8: Additional visualization of proteins designed by our method, Random, and Random+, embedded using Pinal (SaProt module) and projected with UMAP.
E.2 PPL Evaluation Using Alternative Protein Language Models
To provide a more comprehensive evaluation of the PPI. metric, we select two additional language models (i.e., ProGen2 [51] and RITA 43) pre-trained on large scale of protein sequences. The evaluations conducted Mol-Instructions are displayed in Table The results are consistent with the PPL values computed by Pro1GPT2, despite differences absolute scores. PRODVA consistently remains within low PPL range and is closer to natural proteins.
18

---

Table 6: Comparison of PPL performance computed using ProtGPT2, ProGen2, and RITA.
Models
PPLpmarr12 (4)
PPLproGien2 (4)
PPLRIIA (4)
Natural
318.15
5.99
5.52
Random (U)
2484.03
21.71
22.
Random (E)
3136.88
18.68
19.04
Random+ (E)
846.01
10.08
9.32
ProteinDT
1576.23
12.41
12.44
ProteinDTPr
1213.38
10.80
10.69
Pinal
308.97
5.81
5.77
PAAG
2782.70
17.84
18.05
PAAGET
1332.35
11.09
11.03
Chroma
1370.21
12.22
12.42
PRODVA
415.63
7.63
8.82
E.3 Further Analysis of Foldability Metrics
The pLDDT and PAE metrics in our experiments are averaged across all designed proteins. To mitigate the potential bias of foldability metries being dominated by few clusters, we first exclude well-folded proteins. The remaining proteins are then clustered using MMseqs2 based on sequence identity. Finally, the average pLDDT and PAE computed across the filtered clusters. Assuming the filtered proteins are grouped into k clusters, the metric can be defined as follows:
PLDDT clusters
PLDDT
k
n2
where ny denotes the number of proteins in the j-th cluster. The PAE can be defined a similar way. The results shown in Table 7 For these filtered proteins that not well-folded (pLDDT below 70 or PAE above 10), PRODVA outperforms other models by substantial margin, with average scores across clusters approaching the well-folded threshold. This highlights the robustness of our method in designing structurally plausible proteins.
Table 7: The average pLDDT and PAE scores across clusters (denoted pLDDT clusters and P...clustess respectively) are evaluated on Mol-Instructions. The percentage represents the sequence identity threshold. For clarity of comparison, the results for unfiltered proteins (i.e., including well-folded proteins) are presented in the parentheses.
PLDDT clusters (1)
PAE el (L)
Models
PLDDT (t)
PAE (L)
30%
50%
30%
50%
Natural
80.64
67.04
(77.24)
67.79 (80.44)
9.20
17.83 (11.25)
17.59 (9.61)
Random (U)
22.96
22.87 (22.96)
22.87 (22.96)
24.85
24.88 (24.85)
24.88 (24.85)
Random (E)
25.77
25.68 (25.77)
25.68 (25.77)
24.71
24.74 (24.71)
24.74 (24.71)
Random+ (E)
64.47
63.19
(64.32)
63.20 (64.40)
17.91
18.73 (18.01)
18.72 (17.96)
ProteinDT
38.29
38.04
(38.10)
38.10 (38.16)
25.13
25.22 (25.19)
25.21 (25.19)
ProteinDTPr
51.42
39.47
(40.53)
40.17
(43.57)
18.57
22.93 (22.50)
22.73 (21.4)
Pinal
75.25
57.71
(66.90)
57.59 (67.87)
10.96
19.57
(14.81)
19.73 (14.41)
PAAG
28.39
28.37
(28.39)
28.37
(28.39)
25.38
25.39
(25.38)
25.39
(25.38)
PAAGIT
50.37
39.14
(39.93)
39.87 (43.17)
19.96
24.48 (24.14)
24.23
(22.85)
Chroma
59.18
55.36 (59.13)
55.37
(59.17)
15.03
17.13 (15.06)
17.13
(15.03)
PRODVA
76.86
64.59 (74.51)
66.49 (76.68)
8.66
14.19 (9.09)
14.37 (8.74)
19

---

E.4 Effect of Loss Weighting
As discussed in Section 3.2 and [3 are hyperparameters adjusting the loss weights. To determine their optimal values, we conduct experiments on the validation set of Mol-Instructionss Figure ? demonstrates that incorporating both LTYPE and D significantly improves the model's perfor- mance across all metrics. Specifically, we observe that continuously increasing the weight of either or 33 leads decreased PL.DTT and ProTrrek Score. To balance foldability and language alignment, we further explore loss weights around 0. and 0.2, ultimately selecting a = 3 0.2 to achieve an optimal trade-off.
51.0
17.0
30.5
50.0
69.5
018
68.0
0.0
0.4 Weights
0.2
0.4
0.2
15
Weights
Weights
Weights
(a)
(b)
(c)
(d)
Figure 9: Evaluations on the effect of loss weighting. The blue lines indicate the effect a with ==0 O, while the red lines indicate the effect of / with 0. The triangle and star markers indicate configurations where both a and / are set to 0.1 and 0.2, respectively. The grey dashed line denotes the performance without incorporating type loss and description loss during training.
E,5
Varying the Size of the Supporting Documents
The size of the supporting documents constrains the number of text-protein pairs that can be retrieved or utilized during inference. To investigate the effect varying the supporting document size, we present experimental results obtained by both decreasing and increasing the size of the supporting documents, as shown in Figure TO The size of the supporting documents appears have minimal impact PPL and foldability metrics, with significant differences observed as the size increases. In contrast, for the language alignment metries, PRODVA generally performs better with larger supporting document sizes. A possible explanation is that a larger set of supporting documents increases the likelihood of retrieving more relevant descriptions and sequences, thereby facilitating the construction of more relevant fragment candidates.
Protrsk
04021 Sacaali2l
PLS 10.111
18.0
Score
53.0
600.0
71.3
78.0
76.0
400.0 0
30.0
PLDD1
% PLDOT 30
9.5
70.0
15.0
51.0
4.0
72.0
Score
100.0 0
10.0
66.3
EveLiama Score
14.0
40.0
2110
10%L (#Pairs)
100%
337h
17%.
18s, 49
77%
480
25%
(143) aae Hare comn buo 1900
Size
Size (*Pairs)
(a)
(c)
(d)
Figure 10: Analysis of the variation in supporting document size during inference Mol-Instructions "Size" denotes the proportion the training set used as supporting documents, with 100% (i.e., the complete training set) serving the default in Table Additionally, the validation set is included to further expand the supporting document size, indicated as 100%+
E.6 Additional Evaluations the CAMEO Subset
Violin plots illustrating the results on the CAMEO subset are shown in Figure TT Pinal consistently outperform the baseline models across most metrics. Specifically, the majority of
PRODVA and
20

---

proteins designed by our approach remain within a reasonable PPL range. Furthermore, PRODVA generates greater number of well-folded proteins compared to Pinal, demonstrating the effectiveness of our method in designing structurally plausible proteins. Additionally, PRODVA demonstrates distribution closely matching that of Pinal on both the ProTrek Score and Keyword Recovery metrics, despite using only 0.02% of the training data.
3500
0.5
2500
0.6
2000
0.4-
1000
0.2
Chroma ProteinDT, PAGG,,
E5.45
Pinal
ProDMe
Natunal
Cleuma ProteinDT, by PAAG
ESAB
Pinal
ProDWi
Natural
(a)
(LDET(11
(A)
30
20
35
50
10-
Chroma ProteinOTr RAAGrT
ESM3
Pinal
ProDWb
Natural
Chroms ProteinDTyr PAAGr
ESM3
Final
Protive
Natural
Profikk Scone
Soyword Recovery (1)
40
100
80-
60
30
40
-10
Cheema ProteinDT. PMAG
ESM3
Pinal
ProdNe
Natural
Chroma ProteinOT, PAAGa
ESM3
Pinal
Natural
(e)
(I)
Figure 11: Violin plots illustrating the results on the CAMEO subset.
E.7
Additional Evaluations Mol-Instructions
Violin plots illustrating the results on the Mol-Instruuciios protein design task are shown in Figure A similar conclusion can be drawn that PRODVA demonstrates consistent capability in designing well- folded and alll-aligned proteins compared to other baseline models, including the ofate-of-thh-art Pinal.
21

---

1.0
1000
0-
0.0-
Chroma
Protein DTyy
PAAGOT
Hinal
ProDNb
Natural
Chroma
ProteinOTiy
PMAG
Pihal
ProDva
Natural
OLDOT(11
PAE
100
30
90
60
15
10-
20
0-
Chroma
ProteintTiv
PAdG,
Araal
Pmotie
Chroma
ProtwnDTer
FAAGe
Miall
ProDMi
(c)
(d)
Score
Sore
100
30
60
10
40
-10
Chroma ProteinOTir
PANGIT
Phot
ProDVo
Natural
Chrima
ProtainDTi
PARGTT
Rinal
ProDia
Natural
(c)
Figure 12: Violin plots illustrating the results on the Mol-Instructions protein design task.
E.8
Additional Results on Retrieving the Top K Most Relevant Descriptions
Further analysis conducted on the CAMEO subset to determine the optimal value of for retrieval during inference. The experimental results, illustrated in Figure 13 exhibit a consistent pattern with the findings discussed Section 4.6, Both PPL and PAE scores increase with larger values of K, remain within an acceptable range. The model's performance measured by the pLDDT score, initially improves but declines when more than 16 of the most relevant descriptions are retrieved. Additionally, the ProTreek Score achieves its highest performance when retrieving the top 16 most relevant descriptions. To balance well-folded structures and well-aligned proteins, we select 16 and K 32 throughout all experiments.
E.9 Inference with Additional Fragments
argue that retrieving the top K most relevant descriptions helps filter out irrelevant proteins and fragments, thereby improving the model's performance. To further investigate this, we conduct two additional experiments. The first involves incorporating randomly sampled fragments
In Section 4.6
22

---

MLCH
SLE0T PACTT 20113
(RE
of IA A0.11
Po Sonez11
775
14.5
75.-
75.0
14.0
000.0
650.0
75.0
675
700
725
72-
99.5
76.3
130
135
550.0
73.0
0.007
650
6.-
PAE
FRL
of > 70
% FAE <0
PoT Score
72.5
32
64
128
2 4
32
04 326
16
64
128
8 15
61
12a
T0p
top
Top
Topk
(a)
(b)
(c)
(d)
Figure 13: Analysis regarding the selection of Top K most relevant descriptions during inference the CAMEO subset.
that are not among those from the top K relevant protein sequences. The second to add N-gram subsequences of random lengths. Both cases be considered introducing noise either through additional irrelevant fragments or randomly generated N-gram subsequences.
Figure 14 presents several interesting findings, summarized as follows:
(1) The Fragment Encoder learns to differentiate between biologically meaningful fragments and biologically meaningless N-gram subsequences. The results in Figures [4[g) and (h) demon- strate that incorporating additional N-gram subsequences has marginal negative impact on both the ProTTrek Score and the EvoL.ama Score. In contrast, incorporating irrelevant fragments during inference significantly decreases the model's performance on language alignment. This indicates that our Fragment Encoder exhibits robustness to N-gram noise.
(2) The retrieval method further filters out portions of the fragment noise. Figures 14 af)(() show results consistent with those in Figures 5 and [3 indicating that introducing additional noise during inference has minimal impact on designing structurally plausible proteins. Therefore, the retrieval method effectively filters out irrelevant fragments as noisc.
#LBB 23
$30
Andt:ina Ine Prggara
08223
Addiccal Prygmier
AAtOIra Frigoott
38575
430
00.52
330
330
330
38325
4/0
2.0
200
4.8%
3,0x
38300
90
26
26%
4.6%
007
3.0
4.7
38x
16%
28%
4.%
03x
(a)
(b)
(c)
(d)
7AB 153
of XA 121
hohsk Scam (T)
Eveilama StretTi
14
373
220
50
254
1.54
4.54
0.
1.54
1.5
25
(c)
(1)
(g)
(h)
Figure 14: Results of incorporating additional fragments or N-gram subsequences during inference. The experiments are conducted under the setting of K 32. Gray bars indicate the performance of retrieving the top 32 relevant descriptions without incorporating any additional fragments or N- N-grams. The x-axis represents the multiple of noise introduced relative the original number of fragments during inference.
23

---

To further validate our first finding regarding the Fragment Encoder, we utilize UMAP [3] in Figure 13 visualize the embeddings of fragments and N-grams represented by the Fragment Encoder. Additionally, we train linear probe to classify the fragment and N-gram embeddings. The results demonstrate that, in a high-dimensional embedding space, simple linear prohe can accurately distinguish between the two types of subsequences, achieving an average classification accuracy of 93.87%. These findings support our conclusion that the Fragment Encoder effectively learns to differentiate between N-grams and fragments.
N N-grams Fragments
Figure 15: Visualization of fragment and N-gram embeddings represented by the Fragment Encoder.
Case Study
An example of protein designed by PRODVA shown in Figure T6 PRODVA successfully designs a MobA-like NTP transferase domain that meets the input specifications. Additionally, the GlmU domain designed exhibit functions associated with the aDDNN biosynthesis pathway.
G Limitations
In this paper, we employ four widely adopted language alignment metrics, including both oracle model-based and retrieval-based approaches, to evaluate the alignment between the designed proteins and their functional deseriptions. While these metrics offer valuable insights and have been commonly used in prior work, they may not fully capture the biological validity functional efficacy of the designed proteins. To address this limitation, future work will focus on conducting wet-lab experiments to empirically validate the functions of the designed proteins.
24

---

Regionls) associated function keyword(s) Input (Textual Function Description)
Synthesize a protein sequence with the appropriate folding and stability properties for the desired function. The designed protein must have Proton acceptor active site that highly conserved among related enzymes. The designed protein must be able regulate peptidoglycan biosynthetic process, cell wall organization. MobA-like compete with endogenous proteins inhibitors for UDP-N-acety-aalha---
designed protein have
The designed protein should
glucosamine binding.
The protein design should take into account the physiological conditions of the Nucleotide-sugar biosynthesis, UDP-N- acetyl-alpha-D- glucosamine biosynthesis pathway. The designed protein sequence
Reference (Ground Truth)
Designed Protein (ProDVa)
WWA.G//WWW...ONNOOOOOOOO/EEEOOOO PATHLRGKTTICGGSR TIQAGSRI
[ümU_ C
This entry represints the C-teminal Lbk
domaln that possesses the
acstyltrandferase
This entry represents wide range of NTP translerase domains, including the NTP transfurase domain of (Mbiyydapperrn dinuclesside blosyrthesis protein
Superfamily
11001200
Trimeric Lpok like superfamily
[-.] structures several proteins with this domein have been determined, including UDP N-
PAE 7.81
EvoLiama Score 71.40
scoty/gluosemmin acyltrandeerase [..]
Retrieval Accuracy@20 pass
Figure 16:
An example of a protein designed by PRODVA. The annotations are identified using InterPro, with two selected for illustration.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

reproducible, they Answer: [No]
Justification: All the datasets and codes will be made publicly available the final version. We are committed continuously maintaining and updating these resources support ongoing research and contribute to the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pub.e httppss.ccpublli/guuueeee Sttp:ee///oooeSeeesseiiiiiii
ssioiiooiiiiioiociicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 4.1 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the results in our experiments are averaged across three runs with different random seeds and the error bars are properly reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
28

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1 and Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no potential societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets used in the paper are cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See Appendix C.2
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
31

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
32


Reviews:
[{"review_id": "m14N9sS8Q4", "reviewer": "Reviewer_Sz6U", "summary": "The manuscript focuses on protein design, specifically on generating sequences with desirable properties specified through text inputs. With common methods, such sequences often suffer from poor in silico folding metrics. The authors address this issue by incorporating protein fragments (e.g., motifs, functional sites), which helps improve folding quality.", "questions": "- The embeddings of random protein sequences using ESM-C might be misleading, or at least should be interpreted cautiously. ESM-C may not have encountered such proteins during training and could project them into similar embedding regions. Using embeddings from Pinal might provide a more accurate comparison and a better sense of the embedding space's range.\n\n- In Figure 1a, the orange color is used for both random and very low pLDDT sequences. I understand that one corresponds to structures and the other to models, but using distinct colors would help avoid confusion.\n\n- In Equations (1) and (2), is something missing on the left side of the curly bracket?\n\n- On line 86, are the representations mapped to the same embedding space as the protein language model via an adapter, or do they already share the same dimensional space?", "limitations": "The limitations of the proposed method should be discussed explicitly in the conclusion or highlighted through specific experiments.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The manuscript is overall easy to follow, and the method is clearly described. The proposed approach appears to outperform existing baselines across different tasks.\n\nWeaknesses\n\nSequence diversity should be reported alongside other metrics. Additionally, It would be interesting to report pAE (or other structural metrics) in terms of clusters of sequences.\n\nI may have missed this, but is the proposed method restricted to single-chain proteins? If not, evaluating interaction predictions (e.g., by reporting pae_interaction on known interacting chains) would strengthen the work.", "was_revised": true, "final_justification": "My main concern was around the evaluation, especially sequence diversity paired with other metrics. The authors provided additional quantitative metrics and addressed my questions.", "rebuttal": "Dear Reviewer Sz6U,\n\nFirst and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:\n\nQ1: Evaluations on Sequence Diversity and structure-based metrics in terms of clutsters of sequences.\n\nA1: Thanks for your valuable feedback.\n\n> Sequence diversity should be reported alongside other metrics.\n\nWe first present the experimental results on Sequence Diversity. MMseqs2 [1] is used to compute the similarity $\\mathrm{sim}$ between each pair of sequences in a batch of size $N$, where all sequences in the batch are designed based on the same textual description (i.e., with different random seeds). Sequence Diversity is formalized as follows:\n\n$$\n\\text{Sequence Diversity} = \\frac{\\sum_i (1 - \\mathrm{sim}_i)}{N(N - 1)}\n$$\n\nThe results (in %) are presented in the table below. Key observations are summarized as follows:\n- **ProDVa exhibits comparable sequence diversity to all other baseline models.** Compared to the state-of-the-art baseline model Pinal, our method achieves similar performance on the Mol-Instructions dataset and outperforms it by over 15% on the CAMEO subset. **This demonstrates that ProDVa is capable of designing diverse sequences that satisfy the same input descriptions.**\n- On the Mol-Instructions dataset, we observe that both ProteinDT and PAAG exhibit significantly reduced sequence diversity after fine-tuning, with decreases of 19.36% and 13.06%, respectively. **In contrast, under the same training settings, ProDVa maintains high sequence diversity, outperforming these two fine-tuned baselines by 14.14% and 7.92%, respectively. This underscores ProDVa’s ability to design protein sequences that satisfy user-specified requirements while preserving diversity in the generated sequences.**\n\n| Models         | Diversity $\\uparrow$ (CAMEO Subset) | Diversity $\\uparrow$ (Mol-Instructions) |\n| -------------- | :----------------------: | :--------------------------: |\n| Random (U)     |          97.46           |            97.01             |\n| Random (E)     |          99.78           |            99.56             |\n| Random+ (E)    |          98.85           |            98.63             |\n| ProteinDT      |          99.72           |            99.23             |\n| ProteinDT (FT) |          99.32           |            79.87             |\n| Pinal          |          82.72           |            95.28             |\n| PAAG           |          99.02           |            99.15             |\n| PAAG (FT)      |          99.87           |            86.09             |\n| Chroma         |          98.12           |            97.00             |\n| ESM3           |          96.75           |              -               |\n| **ProDVa**     |        **98.58**         |          **94.01**           |\n\n> Additionally, It would be interesting to report pAE (or other structural metrics) in terms of clusters of sequences.\n\nTo evaluate foldability in terms of clusters of sequences, we first use MMSeqs2 to cluster sequences at various identity thresholds (indicated in parentheses). The average pLDDT across clusters is defined as follows (similarly for PAE across clusters):\n\n$$\n\\text{pLDDT across clusters} = \\frac{1}{k} \\sum_{j=1}^{k} \\left( \\frac{1}{n_j} \\sum_{i=1}^{n_j} \\text{pLDDT}_{ij} \\right)\n$$\n\nwhere $k$ is the number of clusters, and $n_j$ denotes the number of proteins in the $j$-th cluster.\n\n| Models         |   pLDDT $\\uparrow$   | pLDDT (30%) $\\uparrow$ | pLDDT (50%) $\\uparrow$ |   PAE $\\downarrow$   | PAE (30%) $\\downarrow$ | PAE (50%) $\\downarrow$ |\n| -------------- | :-------: | :---------: | :---------: | :------: | :-------: | :-------: |\n| Natural        |   80.64   |    77.24    |    80.44    |   9.20   |   11.25   |   9.61    |\n| Random (U)     |   22.96   |    22.96    |    22.96    |  24.85   |   24.85   |   24.85   |\n| Random (E)     |   25.77   |    25.77    |    25.77    |  24.71   |   24.71   |   24.71   |\n| Random+ (E)    |   64.47   |    64.32    |    64.40    |  17.91   |   18.01   |   17.96   |\n| ProteinDT      |   38.29   |    38.10    |    38.16    |  25.13   |   25.19   |   25.19   |\n| ProteinDT (FT) |   51.42   |    40.53    |    43.57    |  18.57   |   22.50   |   21.41   |\n| Pinal          |   75.25   |    66.90    |    67.87    |  10.96   |   14.81   |   14.41   |\n| PAAG           |   28.39   |    28.39    |    28.39    |  25.38   |   25.38   |   25.38   |\n| PAAG (FT)      |   50.37   |    39.93    |    43.17    |  19.96   |   24.14   |   22.85   |\n| Chroma         |   59.18   |    59.13    |    59.17    |  15.03   |   15.06   |   15.03   |\n| **ProDVa**     | **76.86** |  **74.51**  |  **76.68**  | **8.66** | **9.09**  | **8.74**  |\n\n**The results indicate that ProDVa consistently outperforms all baseline models in both overall foldability and foldability across clusters.** At lower identity thresholds, the pLDDT decreases slightly compared to fine-tuned ProteinDT and PAAG, demonstrating ProDVa's robustness across diverse designed protein sequences.\n\nWe will include the results above in the final version.\n\n---\n\nQ2: Clarifications on the applicability of ProDVa.\n\nA2: Thanks for your question. ProDVa is proposed for the design of single-chain proteins, consistent with previous work such as ProteinDT, PAAG, and Pinal.\n\n---\n\nQ3: Explanations of using ESM C as the embedding model in Figure 1.\n\nA3: Thanks for your question. There are two main reasons why we prefer to use ESM C instead of Pinal.\n\n**First, Pinal is a text-to-protein model that takes natural language as input and is not specifically designed to compute protein embeddings.** It comprises two main modules: a text-to-structure module and a SaProt [2] module, of which only the latter can be used as an embedding model.\n\n**Second, technically speaking, computing protein embeddings using ESM C and Pinal (the SaProt module) exhibits very similar behavior, including for those generated by Random(+) and ProDVa.** This similarity arises because Pinal was trained on proteins from SwissProt, UniRef50, and UniProtKB. Consequently, both ESM C and Pinal are unlikely to have encountered the remaining proteins, whether random or novel.\n\nTherefore, we believe that ESM C is a more suitable choice for embedding the landscape of random proteins and novel proteins designed by ProDVa, as it can offer more meaningful insights for comparison with natural proteins.\n\n**To validate this, we use Pinal (the SaProt module) as the embedding model to visualize the landscape, following the approach used in Figure 1(a). The visualization results are highly consistent with those obtained using ESM C. Proteins generated by Random are clustered, while those from Random+ exhibit a significantly more diverse distribution. As expected, proteins designed by ProDVa span the landscape of natural proteins. The visualization results using Pinal will be updated in the final version.**\n\n---\n\nQ4: Clarifications on the colors chosen for legends used in Figure 1(a).\n\nA4: Thanks for your valuable feedback. The legend colors in the upper right are consistent with AlphaFold's color scheme. We will adjust the legend color scheme for scatter points in the final version.\n\n---\n\nQ5: Clarifications on the Equations (1) and (2).\n\nA5: Sorry for misunderstanding. There is nothing missing on the left side of the curly brackets in Equations (1) and (2). They denote a group of variables.\n\n---\n\nQ6: Further explanations of the representations on Line 86.\n\nA6: Sorry for misunderstanding. The projection layer is optional. If the Fragment Encoder shares the same dimension as the PLM (i.e., $d_{\\text{PLM}}$), the use of the adapter is determined by the user. Otherwise, the projection layer is employed to map the fragment embedding matrix (representing $m$ fragments) to the same embedding space as the PLM.\n\nReferences:\n\n[1] Steinegger, Martin, and Johannes Söding. \"MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets.\" Nature biotechnology 35.11 (2017): 1026-1028.\n\n[2] Su, Jin, et al. \"Saprot: Protein language modeling with structure-aware vocabulary.\" BioRxiv (2023): 2023-10."}, {"review_id": "atyYdXgPaA", "reviewer": "Reviewer_Q2yf", "summary": "The authors propose PRODVA, a text-guided protein design model that has a fragment encoder to retrieve protein fragments based on textual descriptions. The advantages of PRODVA is it achieves similar functional alignment with 0.04% training data, and achieving high pLDDT and PAE.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The PRODVA achieves high foldability and comparable 'language alignment' with limited size of training set.\n\n2. The presentation is easy to understand.\n\n3. The paper introduces fragments into protein design, which is technically sound and interesting.\n\nWeaknesses:\n\n1. Since PRODVA designs proteins by retrieval from existing fragments, e.g., motifs and functional sites, one main concern is, it seems PRODVA cannot designs proteins with novel fragments. It raises the concern that the proteins generated by PRODVA may has not satisfactory novelty.\n\n2. In Table 1, because the metrics are evaluated by other deep learning model, these metrics are not the higher/lower the better. These metrics need to be close to the distribution of natural proteins.\n\n3. As PRODVA is initilized by ProtGPT2, I am not sure if that's appropriate to also employ ProtGPT2 to test the PPL. Could the authors have more discussions on this point?\n\n4. The work defines 7 fragment types, including family. Does it more like a property of protein instead of a fragment?\n\n5. In 'Language Alignment' metrics, PRODVA excels at retrieval accuracy. Is this because PRODVA itself is retrieving fragments from existing protein dataset, and these fragments are easier to be identified by protein encoder than the proteins generation from scratch?", "was_revised": true, "final_justification": "I do not have any other concern now.", "rebuttal": "Dear Reviewer Q2yf,\n\nFirst and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:\n\nQ1: Explanations of designing proteins with novel fragments.\n\nA1: Thanks for your question. The novel protein sequences designed by ProDVa comprise two components: sets of tokens (amino acids) and sets of fragments. The novelty of these sequences is determined by both factors. In terms of tokens, ProDVa possesses capabilities comparable to those of ProtGPT2. In terms of fragments, fragment candidates are dynamically retrieved from a database (supporting documents) during inference. **In our experimental settings, the retrieval database contains over 1 million unique fragments (1.16M for the function keyword setting and 1.43M for the textual description setting). Consequently, ProDVa is capable of designing novel protein sequences.**\n\nAdditionally, we will explain our choice of using a Fragment Encoder instead of a Fragment Decoder, which might otherwise generate novel fragments. We randomly mutate the amino acids in fragments of the protein sequences designed by ProDVa with a 40% probability, replacing each amino acid with a different one. The table below presents a comparison between ProDVa and simple mutation on fragments. **Both foldability and language alignment perform significantly worse under random mutation. These results highlight the challenges of incorporating novel fragments and indicate that generating novel fragments may not be suitable for designing user-specified protein sequences.**\n\n| Models       |  PPL $\\downarrow$   | pLDDT $\\uparrow$ |  PAE $\\downarrow$  | ProTrek Score $\\uparrow$ | EvoLlama Score $\\uparrow$ |\n| :----------- | :----: | :---: | :---: | :-----------: | :------------: |\n| ProDVa       | 415.63 | 76.86 | 8.66  |     17.40     |     51.10      |\n| ProDVa (Mut) | 4168.9 | 44.34 | 17.55 |     7.71      |     41.03      |\n\n---\n\nQ2: Clarifications on how to interpret the results and metrics in Table 1.\n\nA2: Thanks for your valuable feedback. The metrics presented in Tables 1 and 2 are used to evaluate the quality of the designed proteins and to compare them with various baseline models. Admittedly, interpreting these metrics solely based on whether higher or lower values are better may not always be appropriate. To more intuitively illustrate performance differences among models, we report the absolute values of each metric rather than their relative differences (i.e., delta values) from natural proteins. This approach is consistent with prior studies (e.g., ProteinDT, Pinal), which also present absolute scores. Due to space constraints and to maintain the simplicity of our table presentations, delta values were not included in the current version. We will consider incorporating them in the final version. In this way, we believe that comparisons with other models using absolute values, as well as comparisons with natural protein distributions using delta values, are both intuitive.\n\n---\n\nQ3: Discussions and evaluations on computing the PPL metric with the alternative Protein Language Model.\n\nA3: Thanks for your valuable feedback. To evaluate the impact of different Protein Language Models on the computation of the PPL metric, we selected three additional models: ProGen2 [1], RITA [2], and ProteinGLM [3]. The table below presents supplementary results for the PPL metric (i.e., PPL (ProtGPT2)) as referenced in Table 2.\n\n| Models         | PPL (ProtGPT2) $\\downarrow$  | PPL (ProGen2) $\\downarrow$ | PPL (RITA) $\\downarrow$  | PPL (ProteinGLM) $\\downarrow$  |\n| :------------- | :------------: | :-----------: | :--------: | :--------------: |\n| Natural        |     318.15     |     5.99      |    5.52    |      16.86       |\n| Random (U)     |    2484.03     |     21.71     |   22.14    |      40.87       |\n| Random (E)     |    3136.88     |     18.68     |   19.04    |      32.95       |\n| Random+ (E)    |     846.01     |     10.08     |    9.32    |       8.77       |\n| ProteinDT      |    1576.23     |     12.41     |   12.44    |      18.41       |\n| ProteinDT (FT) |    1213.38     |     10.80     |   10.69    |      19.98       |\n| Pinal          |     308.97     |     5.81      |    5.77    |      17.77       |\n| PAAG           |    2782.70     |     17.84     |   18.05    |      24.44       |\n| PAAG (FT)      |    1332.35     |     11.09     |   11.03    |      19.28       |\n| Chroma         |    1370.21     |     12.22     |   12.42    |      38.59       |\n| **ProDVa**     |   **415.63**   |   **7.63**    |  **8.82**  |    **31.39**     |\n\n**The results are consistent with the PPL values computed by ProtGPT2, despite differences in absolute scores. ProDVa consistently remains within a low PPL range and is closer to natural proteins.**\n\n---\n\nQ4: Explanations of fragment types.\n\nA4: Thanks for your question. In our manuscript, we define fragments of a protein as continuous subsequences (see Lines 58–59), implying that any arbitrary subsequence can be considered a fragment. However, fragments lacking functional annotations are not significant for designing proteins with user-specified requirements. Therefore, identifying usable fragments is a central challenge in our work. In Section 2 and Appendix B, we describe how InterPro is employed to extract fragments with functional annotations. Moreover, a single protein may contain multiple fragments, each potentially associated with different properties (or functions). Thus, we consider \"fragment\" to be the most appropriate term, as the fragments used in our work possess diverse functional annotations.\n\n---\n\nQ5: Further discussions on the results of Retrieval Accuracy metric.\n\nA5: Thanks for your question. We argue that ProDVa’s high Retrieval Accuracy is unrelated to retrieving fragments from existing protein datasets. The metric used (see Lines 525–534 for details) evaluates whether the designed protein shows the highest similarity to the textual description among a set of randomly selected candidates. Similarity is computed using the ProTrek model, which is not involved in the training or inference of ProDVa, thereby avoiding evaluation bias. Moreover, whether the sequences are designed from scratch or generated by ProDVa, they more or less exhibit features of natural proteins (e.g., evolutionary information). Importantly, the candidate pool for Retrieval Accuracy consists entirely of natural proteins, ensuring that the metric provides an unbiased comparison of protein sequences generated by different models.\n\nReferences:\n\n[1] Nijkamp, Erik, et al. \"Progen2: exploring the boundaries of protein language models.\" Cell systems 14.11 (2023): 968-978.\n\n[2] Hesslow, Daniel, et al. \"Rita: a study on scaling up generative protein sequence models.\" arXiv preprint arXiv:2205.05789(2022).\n\n[3] Chen, Bo, et al. \"xTrimoPGLM: unified 100B-scale pre-trained transformer for deciphering the language of protein.\" arXiv preprint arXiv:2401.06199 (2024)."}, {"review_id": "W3YKZF5nxI", "reviewer": "Reviewer_Dwmc", "summary": "The paper proposes a novel approach for function-based protein design, named PRODVA, that integrates (1) a text encoder, (2) a protein language model (PLM), and (3) a fragment encoder. Unlike prior methods, PRODVA dynamically incorporates fragments of natural proteins—retrieved based on functional annotations—to enhance structural plausibility and language alignment. The model demonstrates strong performance across multiple benchmarks (CAMEO and Mol-Instructions), significantly outperforming previous approaches (e.g., ProteinDT, PAAG, and even Pinal) in terms of foldability (pLDDT, PAE) while using a fraction of the training data.", "questions": "Have the authors considered weighting fragments differently based on type (e.g., active site vs. repeat)? Does one category contribute more to structural accuracy?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tThe integration of dynamic fragment vocabulary based on functional annotations is both novel and biologically grounded.\n2.\t PRODVA achieves superior foldability and competitive functional alignment, outperforming state-of-the-art models despite using substantially less training data.\nWeaknesses\n1.\tSome figure captions are insufficiently descriptive. Consider expanding them (e.g., Figures 1b/c).\n2.\tIt is recommended to evaluate the model on out-of-distribution settings, such as unseen protein families or novel synthetic functions.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Dwmc,\n\nFirst and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:\n\nQ1: Expanding the figure captions to provide more descriptive information.\n\nA1: Thanks for your valuable feedback. We will expand the figure captions in the final version as follows:\n\n- Figure 1(b): Performance on pLDDT (higher is better). Our method significantly improves the pLDDT of proteins generated by Random+ by a large margin of 12%, with the average pLDDT exceeding the well-folded threshold.\n- Figure 1(c): Performance on PAE (lower is better). Compared to Random+, our method achieves a 9% improvement in PAE and is the only model to surpass the well-folded threshold. Notably, our method outperforms the state-of-the-art baseline model Pinal in both pLDDT and PAE.\n\n---\n\nQ2: Evaluations on designing proteins under out-of-distribution settings (e.g., evaluating on novel synthetic functions).\n\nA2: Thanks for your valuable feedback. To evaluate performance in designing novel proteins under out-of-distribution (OOD) settings, we first construct a test set comprising 1,000 randomly synthesized novel functions. The Mol-Instructions test set is utilized, and all unique functions are extracted. For each synthetic function, 2–5 different functions are randomly selected and concatenated into a single composite function. Due to the random selection, combination, and arrangement of functions, the resulting synthetic functions are guaranteed to be novel and under OOD settings.\n\nSome key findings can be summarized as follows:\n\n- **ProDVa outperforms other baseline models under OOD settings, including the state-of-the-art baseline Pinal, across both foldability and language alignment metrics.**\n- In terms of foldability, both ProDVa and Pinal demonstrate promising performance, indicating that OOD settings do not affect a model’s ability to design structurally plausible proteins.\n- For language alignment, all models perform worse than on the Mol-Instructions test set (see Section). This may be attributed to two main factors. First, the OOD settings pose significant challenges for current models. Understanding synthetic functions that may not exist in the real world likely requires further investigation in future work. Second, the OOD settings may not be properly validated, as verifying the plausibility of these synthetic functions is inherently difficult (e.g., may require further wet-lab experiments). Moreover, natural proteins with such properties may not exist, potentially rendering the design of proteins with novel synthetic functions unfeasible.\n\n| Models         |    PPL $\\downarrow$     |   pLDDT $\\uparrow$  |   PAE $\\downarrow$    | ProTrek Score $\\uparrow$ | EvoLlama Score $\\uparrow$ |\n| :------------- | :--------: | :-------: | :------: | :-----------: | :------------: |\n| Pinal          |   326.79   |   74.75   |  11.29   |     9.23      |     48.47      |\n| ProteinDT (FT) |  1877.44   |   34.38   |   24.3   |     4.03      |     43.06      |\n| PAAG (FT)      |  1406.22   |   48.65   |  20.88   |     4.19      |     45.89      |\n| **ProDVa**     | **412.27** | **77.41** | **9.11** |   **9.79**    |   **49.17**    |\n\n---\n\nQ3: Explanations of weighting fragment types.\n\nA3: Thanks for your question. As mentioned in our manuscript (Lines 105–110), the fragment types in our training set (CAMEO subset and Mol-Instructions) are inherently unbalanced. Enforcing a balanced distribution of fragment types would require removing up to 80% of the training data, potentially resulting in insufficient data for training. To address this issue, we introduce a weighted type loss $\\mathcal{L}_{\\text{TYPE}}$, as described in Equation (4) (see Lines 105–110)."}, {"review_id": "pd9fsPhto4", "reviewer": "Reviewer_VmDi", "summary": "This paper proposes a method to improve the plausibility of protein sequences generated by a protein language model by expanding the vocabulary from the 20 standard amino acids to fragments composed of multiple adjacent amino acids. The approach is designed to address the problem of generating protein sequences conditioned on textual descriptions.", "questions": "Q1. Why is the amount of training data used in this work smaller than that used in Pinal?\n\nQ2. How does the choice of different tokenization methods affect the results?\n\nQ3. Does using fragments to represent proteins impact the diversity of the generated sequences?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nS1. The algorithm is clearly described and reasonably designed.\n\nS2. The experimental metrics are well constructed and support the conclusions of the paper.\n\nS3. Introducing protein fragments into the generation vocabulary is an interesting and novel attempt, offering new ideas for future research.\n\n**Weaknesses**\n\nW1. The paper mentions that the baseline method Pinal, which achieves comparable performance, uses more training data. However, it does not explain why the training data used in this work is smaller than that of Pinal, nor does it analyze whether increasing the training data might lead to better results than Pinal.\n\nW2. The paper adopts existing tokenization methods but lacks a discussion on whether the choice of tokenizer affects the results.", "was_revised": true, "final_justification": "Their responses addressed most of my concerns. I'll keep my initial positive score.", "rebuttal": "Dear Reviewer VmDi,\n\nFirst and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:\n\nQ1: Explanations of using significantly less training data compared to Pinal.\n\nA1: Thanks for your questions. For the first question:\n\n> Why is the training data used for ProDVa smaller than that used for Pinal?\n\nDuring training, ProDVa utilizes approximately 712K text-protein pairs, whereas Pinal employs 1.76B pairs. **Despite relying solely on annotated datasets (e.g., SwissProt), Pinal achieves this larger dataset through various data augmentation methods, including the generation (automatic annotation) of textual descriptions for proteins.** For example, it uses models such as GPT-4 and GLM-4-Flash to convert records from SwissProt into complete sentences and domain-specific question-answer pairs that serve as descriptive annotations for different proteins. Additionally, Pinal leverages ProTrek to annotate proteins from UniProtKB and UniRef50, resulting in a large-scale synthetic dataset. **In contrast, our approach primarily focuses on SwissProt and its corresponding annotations, which are manually curated and reviewed by experts.** Our experiments yield the following observations:\n\n- High-quality data (reviewed and manually annotated) is sufficient to train ProDVa to achieve performance comparable to Pinal, which is trained on a large-scale synthetic dataset (model-predicted, unreviewed annotations).\n- ProDVa demonstrates data efficiency, utilizing only 0.02%–0.04% of the data used by Pinal, as it does not rely on synthetic annotations.\n- The dataset used for ProDVa is entirely open-source, and its acquisition is straightforward and time-efficient, in contrast to Pinal, which depends on LLMs or other oracle models for data generation.\n\nFor the second question:\n\n> Whether increasing the training data might lead to better results than Pinal?\n\nWe have discussed the question in our manuscript (Lines 229-241). Here is a summary:\n\n- Unfortunately, the large-scale datasets used by Pinal have not yet been open-sourced. Reproducing the synthetic annotations described above is challenging for us, as the process is both costly (e.g., requiring the use of LLM APIs and deployment of oracle models) and time-consuming (e.g., UniRef50 alone has over 41M proteins, yet UniProtKB includes more than 173M).\n- In Table 2, we demonstrate that ProteinDT and PAAG perform significantly better after fine-tuning with additional data (the same dataset used by ProDVa), particularly with respect to Language Alignment metrics. Therefore, we carefully use the phrase \"may potentially\" to conclude that incorporating more data may improve performance.\n\n---\n\nQ2: Clarifications on choosing the tokenizer (tokenization method).\n\nA2: Thanks for your question. Tokenization in our work involves two aspects: tokenizing amino acids and tokenizing fragments.\n\nFirst, for amino acid tokenization, since ProDVa utilizes ProtGPT2 as the PLM backbone, we employ the ProtGPT2 tokenizer (a BPE tokenizer) for both tokenizing amino acids and constructing the static protein vocabulary. Alternative tokenization methods (e.g., per-amino-acid tokenization) are not suitable for our purposes.\n\nSecond, for fragment tokenization, we introduce our methods in Section 2, based on the intuition and empirical study presented in Section 1 (see Figure 1). Additional experiments in Appendix E.5 demonstrate that our tokenization method exhibits a degree of robustness to noise when encountering irrelevant fragments or random N-gram subsequences.\n\n---\n\nQ3: Evaluations on Sequence Diversity.\n\nA3: Thanks for your valuable feedback. Using fragments to represent proteins has minimal impact on the diversity of designed sequences. In contrast, our training paradigm enables ProDVa to design user-specified proteins while maintaining diversity in the generated sequences.\n\nWe have included experiments and analyses on Sequence Diversity **in our response to Reviewer Sz6U in Q1**. Please refer to that response for more details. A summary of the key observations is provided below:\n\n- ProDVa exhibits comparable sequence diversity to all other baseline models. This demonstrates that ProDVa is capable of designing diverse sequences that satisfy the same input descriptions.\n- Under the same training settings, ProDVa maintains high sequence diversity, outperforming these two fine-tuned baselines by 14.14% and 7.92%, respectively. This underscores ProDVa’s ability to design protein sequences that satisfy user-specified requirements while preserving diversity in the generated sequences."}, {"review_id": "JazYKM7g7c", "reviewer": "Reviewer_Vivg", "summary": "This paper proposes PRODVA, a protein sequence generation model based on a dynamic vocabulary of protein fragments. Specifically, the model constructs a database of fragment candidates. During training, it aligns the representations of a text language model and a fragment encoder on paired text-fragment data. During generation, the model is required to utilize the dynamically constructed fragments (in a manner similar to CopyNet[1] in text generation). In the inference stage, relevant fragments are retrieved based on the input text description and incorporated into the generation process.\n\n\n[1] Gu, Jiatao, et al. \"Incorporating copying mechanism in sequence-to-sequence learning.\" arXiv preprint arXiv:1603.06393 (2016).", "questions": "1. Please provide more ablation studies to validate the effectiveness of the model design. Specifically:\n\n   * What is the effect of removing the dynamic vocabulary mechanism while still allowing the model to access top-k fragment candidates selected using the text encoder from a static fragment pool?\n   * What is the effect of removing the fragment retrieval mechanism entirely (i.e., without any top-k fragment candidates)?\n   * How does varying the size of the Fragment Candidates database affect model performance?\n\n2. Recent diffusion-based language models (e.g., DPLM[1]) have shown strong potential in protein design. Can PRODVA’s approach be applied or adapted to diffusion models?\n\n3. Since PRODVA relies on aligning textual descriptions with known fragment candidates, how well does it generalize to low-resource or novel protein design scenarios where such fragment annotations are unavailable or sparse?\n\n\n[1] Wang, Xinyou, et al. \"Diffusion language models are versatile protein learners.\" arXiv preprint arXiv:2402.18567 (2024).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper addresses an interesting and meaningful problem—how to efficiently leverage protein fragment libraries for protein design, which is both non-trivial and highly relevant.\n2. The writing is clear and well-structured, making the paper easy to follow.\n3. PRODVA demonstrates strong performance across multiple tasks, showcasing the potential of this class of models.\n\nWeaknesses\n\n1. The paper lacks key ablation studies to justify and validate its design choices.\n2. The method relies on text-based conditioning, which may limit its applicability in settings like unconditional protein generation.", "was_revised": true, "final_justification": "In their latest response, they provided additional ablation study results, which demonstrate the rationality and effectiveness of the model architecture. This was my primary concern previously, and it has now been addressed. Therefore, I believe the paper can be accepted.", "rebuttal": "Dear Reviewer Vivg,\n\nFirst and foremost, we extend our gratitude for your thorough review. Here are our responses to the key points you've raised:\n\nQ1: More ablation studies.\n\nA1: Thanks for your valuable feedback.\n\n> What is the effect of removing the dynamic vocabulary mechanism while still allowing the model to access top-k fragment candidates selected using the text encoder from a static fragment pool?\n\nAs shown in Figure 3, ProDVa relies on both the static (tokens) and dynamic (fragments) protein vocabulary to determine: 1) whether to output a token or fragment next, and 2) which token or fragment to select. Therefore, without the dynamic vocabulary mechanism, these two mechanisms cannot operate as intended. In other words, the static fragment pool cannot be utilized by ProDVa without the dynamic vocabulary mechanism.\n\n> What is the effect of removing the fragment retrieval mechanism entirely (i.e., without any top-k fragment candidates)?\n\nThe table below compares two scenarios (i.e., the vanilla multimodal approach and the Top $K=0$ setting) that removing the fragment retrieval mechanism entirely. For further details on the vanilla multimodal approach, please refer to Section 4.4 in our manuscript.\n\n| Models  | Top K |  PPL $\\downarrow$   | RepN  $\\downarrow$ | PLDDT $\\uparrow$ |  PAE  $\\downarrow$  | ProTrek Score $\\uparrow$ | EvoLlama Score  $\\uparrow$ |\n| ------- | :---: | :----: | :--: | :---: | :---: | :-----------: | :------------: |\n| Vanilla |   -   | 109.66 | 0.10 | 72.23 | 11.37 |     9.63      |     50.99      |\n| ProDVa  |   0   | 107.63 | 6.25 | 62.83 | 7.89  |     3.80      |     40.77      |\n| ProDVa  |   1   | 239.52 | 0.03 | 74.00 | 9.22  |     15.47     |     49.90      |\n| ProDVa  |  16   | 415.63 | 0.02 | 76.86 | 8.66  |     17.40     |     51.10      |\n\nThe results indicate that, compared to the vanilla multimodal approach, ProDVa without the use of any Top $K$ fragments performs worse across most metrics. However, when utilizing the Top 1 fragments, ProDVa improves its performance by 11.17% in pLDDT and 11.67% in the ProTrek Score. This highlights the significance of incorporating Top $K$ fragments. **The relatively poor performance without any Top $K$ fragments can be attributed to the fact that, during training, most sequences are split into tokens and fragments. Thus, relying solely on tokens (without incorporating any fragments) may lead to failure.**\n\n> How does varying the size of the Fragment Candidates database affect model performance?\n\nIn our manuscript, the size of the supporting documents is identical to that of the training dataset for a fair comparison (see Lines 140–141). To investigate the effect of varying the size of the supporting documents (fragment database), we retain 20%, 40%, 60%, and 80% of the training dataset as supporting documents. Additionally, we incorporate the validation set to further increase the size of the supporting documents. For the statistics of the training and validation sets, please refer to Appendix C.2.\n\nThe table below presents the experimental results obtained by decreasing or increasing the size of the supporting documents. \"T\" denotes the training set (by default, \"100% T\" is used in our manuscript, see Table 2), and \"V\" denotes the validation set. We summarize the key observations as follows:\n\n- **The size of the supporting documents has minimal impact on PPL and foldability, with no significant differences observed as the size increases.**\n- **Regarding the language alignment metrics, ProDVa generally performs better as the size of the supporting documents increases.** A possible explanation is that a larger set of supporting documents enhances the likelihood of retrieving more relevant descriptions and corresponding sequences, thereby enabling the construction of more relevant fragment candidates.\n\n|   Size (#pairs)   |  PPL $\\downarrow$  | pLDDT $\\uparrow$ | PAE $\\downarrow$  | ProTrek Score $\\uparrow$ | EvoLlama Score $\\uparrow$ |\n| :---------------: | :----: | :---: | :--: | :-----------: | :------------: |\n|   20% T (142K)    | 412.37 | 77.84 | 8.40 |     15.93     |     52.36      |\n|   40% T (284K)    | 423.45 | 77.37 | 8.43 |     16.43     |     51.68      |\n|   60% T (427K)    | 419.04 | 77.29 | 8.34 |     16.72     |     51.78      |\n|   80% T (569K)    | 407.95 | 77.23 | 8.39 |     16.96     |     51.85      |\n|   100% T (712K)   | 415.63 | 76.86 | 8.66 |     17.40     |     51.10      |\n| 100% T + V (731K) | 410.46 | 77.15 | 8.27 |     17.02     |     51.99      |\n\n---\n\nQ2: Clarifications on function-based protein design problems (i.e., conditional generation).\n\nA2: Sorry for misunderstanding. In our manuscript, we state that the problem addressed by ProDVa is function-based protein design (i.e., conditional generation). This problem can be formalized as follows (see Section 2, Lines 66–69):\n\n$$\np(P \\mid t) = p((x_1, x_2, \\cdots, x_k) \\mid t, \\forall i, x_i \\in A \\cup S)\n$$\n\nwhere $t$ denotes the input textual description (i.e., the condition).\n\nHere, we provide a more detailed clarification as to why ProDVa is specifically designed for function-based protein design problems, as opposed to unconditional protein generation.\n\n- First, ProDVa leverages a Text Language Model in three key ways: **(1) to interpret the input textual description, (2) to represent protein fragment descriptions during training, and (3) to retrieve relevant proteins based on functional descriptions during inference.** In the context of unconditional protein generation, none of these functionalities are applicable.\n- Second, the backbone Protein Language Model (ProtGPT2) employed in ProDVa is specifically designed for unconditional generation. Therefore, **ProDVa inherently possesses the capability for unconditional protein generation.**\n\nDespite the above clarifications, we demonstrate how ProDVa can be adapted for unconditional protein generation. By simply fixing the input instruction to \"Design a novel protein sequence,\" the problem addressed by ProDVa effectively reduces to an unconditional protein generation task. To further ensure that the text input as a no-op, we replace the retrieval method with the random selection of fragments. The table below presents the results on unconditional generation. Note that the language alignment metrics are not considered under unconditional settings.\n\n| Models         |    PPL  $\\downarrow$  |   pLDDT $\\uparrow$  |  % > 70 $\\uparrow$  |   PAE  $\\downarrow$  |  % < 10 $\\uparrow$  |\n| :------------- | :--------: | :-------: | :-------: | :------: | :-------: |\n| Pinal          |   411.93   |   70.11   |   57.02   |  12.76   |   48.44   |\n| ProteinDT (FT) |   593.06   |   47.79   |   0.02    |  26.56   |   0.00    |\n| PAAG (FT)      |  1327.98   |   50.32   |   23.83   |  19.95   |   22.24   |\n| **ProDVa**     | **476.02** | **77.52** | **79.78** | **9.32** | **60.25** |\n\n**ProDVa outperforms all baseline models on the unconditional protein generation task.** Compared to the state-of-the-art baseline, Pinal, ProDVa generates 22.76% and 11.81% more well-folded proteins in terms of pLDDT and PAE, respectively. Compared to other fine-tuned models, ProDVa outperforms them by a large margin. This demonstrates the effectiveness of our method in both conditional and unconditional protein generation settings.\n\n---\n\nQ3: Explanations of applying ProDVa to Diffusion-based Language Models.\n\nA3: Thanks for your question. ProDVa is specifically designed for autoregressive Transformer architectures. It employs a decoder-only backbone (see Lines 46 and 77) and is trained using curated learning objectives rather than simple next-token prediction (see Lines 92–102). Applying ProDVa to recent Diffusion-based language models remains a non-trivial and challenging task. Nonetheless, we believe this work establishes a foundation for future research on integrating ProDVa with diffusion models.\n\n---\n\nQ4: Explanations of the generalization under low-resource or novel protein design scenarios where related fragment annotations are unavailable or sparse.\n\nA4: Thanks for your question. In fact, during inference, ProDVa does not rely on any fragment annotations (see Lines 121–123). The training paradigm involves learning functional annotations that are not required during inference. ProDVa learns to select the appropriate token or fragment at each step of decoding. Therefore, under low-resource settings, there is minimal impact on ProDVa's training paradigm. During inference, if no related fragments are provided, ProDVa generates tokens to design a novel protein sequence, similar to ProtGPT2."}]

Author rebuttal:

Meta-review: This paper presents ProDVa, a protein design framework that integrates text-based functional descriptions with fragment information from natural proteins to improve structural plausibility. By combining a text encoder, a protein language model, and a fragment encoder, the method achieves comparable function alignment to state-of-the-art models while using less than 0.04% of the data, and significantly improves foldability (e.g., +7.38% pLDDT > 70). 
During the review process, all reviewers gave a positive score to this paper, recognizing its novelty, technical contribution, performance, and writing. From my perspective, integrating functional fragments through a dynamic vocabulary is novel. It may greatly improve foldability and functional alignment. The comprehensive experiments validate this point. Therefore, I recommend the acceptance of this paper for a spotlight presentation.
Decision: Accept (spotlight)