SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding
Authors: Thomas Walton
Darin Tsui
Aryan Musharaf
Amirali Aghazadeh
Venue: NeurIPS 2025 spotlight ()
Keywords: speculative decoding
protein design
autoregressive
PLM
sampling
structure
Primary area: machine_learning_for_sciences
TL;DR: We develop a novel speculative decoding framework for protein generation, using structure aware guidance from k-mers to generate proteins with higher likelihood and structural confidence.

Abstract: Autoregressive models have transformed protein engineering by enabling the generation of novel protein sequences beyond those found in nature. However, their sequential inference introduces significant latency, limiting their utility in high-throughput protein screening. Speculative decoding accelerates generation by employing a lightweight draft model to sample tokens, which a larger target model then verifies and refines. Yet in protein sequence generation, draft models are typically agnostic to the structural and functional constraints of the target protein, leading to biologically implausible outputs and a shift in the likelihood distribution of generated sequences. We introduce SpecMER (Speculative Decoding via k-mer Guidance), a novel framework that incorporates biological, structural, and functional priors using k-mer motifs extracted from multiple sequence alignments. By scoring candidate sequences in parallel and selecting those most consistent with known biological patterns, SpecMER significantly improves sequence plausibility while retaining the efficiency of speculative decoding. SpecMER achieves 24–32% speedup over standard autoregressive decoding, along with higher acceptance rates and improved sequence likelihoods.

SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding
Thomas A. Walton
Darin Tsui Georgia Institute of Technology darint @ gatech..edduuu
Georgia Institute of Technology twalton42 gatech.edu
Aryan Musharaf
Amirali Aghazadeh
Georgia Institute of Technology amusharaf3 gatech.edu
Georgia Institute of Technology amiralia @gatech.edll
Abstract
Autoregressive models have transformed protein engincering by enabling the gen- eration of novel protein sequences beyond those found nature. However, their sequential inference introduces significant latency, limiting their utility in high- throughput protein screening. Speculative decoding accelerates generation by employing a lightweight draft model sample tokens, which larger target model then verifies and refines. Yet, in protein sequence generation, draft models are typically agnostic to the structural and functional constraints of the target protein, leading to biologically implausible outputs and a shift in the likelihood distribution of generated sequences. We introduce SpecMER (Speculative Decoding via k-mer Guidance), a novel framework that incorporates biological, structural, and func- tional priors using k-mer motif's extracted from multiple sequence alignments. By scoring candidate sequences in parallel and selecting those most consistent with known biological patterns, SpecMER significantly improves sequence plausibility while retaining the efficiency of speculative decoding. SpecMER achieves 24-32% speedup over standard autoregressive decoding, along with higher acceptance rates and improved sequence likelihoods.
Introduction
Designing proteins with enhanced or novel biological functions an important problem with wide- ranging applications in therapeutics, sustainability, and drug discovery [] Protein design is challeng- ing due to the astronomically large space possible protein sequences, which only a tiny fraction is likely exhibit the desired functions (2). Generative models trained on the vast landscape of evolved protein sequences have emerged as powerful tools for navigating this combinatorial sequence space. Among them, autoregressive models have been especially successful in the design of proteins with functional properties comparable to natural proteins 3 Their autoregressive nature integrates well with textual information 4 ensures training stability [.) enables variable-length sequence generation
[ and effectively captures long-range dependencies that encode complex amino acid interactions Despite these advantages, autoregressive models struggle with pathological repe- tition (6-8), early termination 77), out-of-distribution generalization [8]. and most consequential, high computational
during inference, resulting in slow sequence generation. This computational cost is debilitating limitation for high-throughput protein screening, where thousands of protein sequences need be generated and screened create large-scale libraries of functional and struc- turally stable proteins [9]. For instance, generating 20,000 protein sequences of length 200 amino acids using ProGen2-XL. a H-billion-parameet transformer-bae based autoregressive model, takes approximately 65 hours using a single NVIDIA A6000 GPU. Such slow sequence generation can
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Speculative decoding
Speculative
MSKGE
Diahi
MSKGEELE
Noma bod Freqereey
Normattedd Frequentoy
0.03
decoding
0.00
0.00
Targei model
0.00
0.00
MSKGFELF
MSKGFELEF
p0g
-00
0.00
3.0
2.0
10
45
SpecMER
C=
MSKGE
MSKGEVILL MSKGLAAA
047
SpecMER
model
0.08
0.03
MBKAGL 6%..
Sequence
Multiple
MS0 ELEVS.
NEGGGELAAA
nitering
Nomraaod
043-
0.33
Aignment
MSGER VK-1
NBQWALVXX.
Target model
001
0.11
-.0
18
MSKGEVLK
MSKGEVLLK
Logs LoghLkdloood
Log fhodd
Figure 1: Overview of speculative decoding and SpecMER. a. Given context sequence, spec- ulative decoding uses a draft model to sample tokens. The target model verifies generated tokens, accepting or rejecting them. In this example, two tokens were accepted from the draft model, and the token "E" was corrected to "F".. b. SpecMER augments the drafting process, generating multiple sequences which are then filtered by k-mers at near-zero cost. K-mers are formed through multiple sequence alignment, capturing evolutionary patterns through homologous protein sequences. In this example, "VLL" was selected from the proposed candidates, where two tokens were accepted and the token "L" was corrected to "K" c, Speculative decoding undersamples high log- ligliikli sequences, missing the tail distribution of the target model. K-mers act as guidance for SpecMER, resulting in distribution of generated sequences that more adequately samples high-likelihood, high-plausibilityy protein sequences. Additionally, SpecMER retains nearly the same generation speed as speculative decoding, with only 3% overhead observed at 3.
delay toththhroughput design workflows by days, not accounting for the additional time required for iterative design processes.
Recently, speculative decoding 10] has proven effective in accelerating autoregressive generation in natural language processing. Speculative decoding utilizes lightweight draft model to propose tokens, which are then verified by a larger, more expressive larget model. This speculate-then-verrs framework of generation has demonstrated wall-time speedup factors exceeding 2x by selecting outputs consistent with the target model distribution [II].
In protein sequence generation, however, target-consiseent outputs may not align with the most biologically plausible proteins, particularly those that best satisfy structural functional constraints. This misalignment is amplified in protein design workflows, where satisfying biochemical constraints essential. While draft models may capture broad structural patterns. small deviations local structure can lead to misfolded or nonfunctional proteins. Among target-consistent outputs, some may result in proteins with more plausible biological properties; however, such distinctions are not considered during speculative decoding. To address this, incorporating biologically meaningful local priors offers a principled way to guide generation. Notably, k-mers, or short contiguous amino acid subsequences, encode local structural motif's that correlate with both folding and function, and can be extracted with minimal computational cost.
We hypothesized that incorporating structural cues from k-mers could guide speculative decoding toward more biologically plausible proteins, while retaining or even enhancing generation speed. To test this hypothesis, we develop speculative decoding via k-mer guidance (SpecMER), novel decoding framework that leverages structural motifs from homologous protein sequences to improve both generation speed and quality. By incorporating structural priors into the decoding process, SpecMER provably selects tokens that are not only consistent with the target model but also biased toward completions likely to yield structured, biologically plausible proteins. This motif infir-in

---

Algorithm Token-level maximal coupling
Input: Distributions p. q, Draft sample X p.
Compute the residual distribution Pres where Vx E V. Pres(x) I- --ver min {p(z )a('')}}
g(z) و
Sample 17 (0, 1)
if S min (1, g(x) p(X]
then
Return X. (Accept the draft token.}
else
Return Y ~ Pres. (Correct the token by sampling from the residual distribution.)
end if
guidance increases the likelihood that generated sequences will fold into stable structures. Our contributions are as follows:
We develop the first speculative decoding framework for protein generation. Using ProGen2-S the draft model and ProGen2-M as the target model, we demonstrate average increase in generation speed by 32%.
We develop SpecMER, novel framework that incorporates k-mer frequencies from natural proteins to guide draft token selection. SpecMER produces sequences more likely to resemble natural proteins while retaining increased generation speed from speculative decoding, enabling fast generation of plausible proteins. Software for SpecMER is available at /ttpss:/ggtthe com/ mmiirrroup-cods//Speeeeee
git
We establish theoretical bounds on wall-time speedup and characterize how guidance from k-mers improves draft token selection, offering practical guidance for estimating speedups under varying system configurations and sequence constraints.
2 Preliminary
Notation. Let draft model P and target model Q be autoregressive models with probability density functions and q. A protein sequence x ~ P denoted by sequence sri), xff+ 1),...,,, x(j). where x(1) is the jth token sampled from p. Tokens are defined by the shared vocabulary V of and Q consisting of amino acids and special tokens.
2.1 Speculative Decoding
Speculative decoding has recently emerged as highly effective technique for accelerating autoregres- sive generation. Speculative decoding employs smaller draft model to propose candidate tokens, which are then refined by a larger, more expressive target model [10] 11. This approach generates tokens consistent with the target model distribution while significantly reducing decoding latency. Various extensions to speculative decoding have explored architectural modifications [12], tree-based verification [33, distillation of the draft model and optimized scheduling reference-frree
inference 15
Given context sequence x(t) and draft model P, speculative decoding efficiently samples L tokens, ä( + 1).... ø(t + L). Speculative decoding can be broken down into three steps: 1) Draft construction, 2) Conditional probability computation, and 3) Draft selection (Figure TTp.
1. Draft construction. The draft model first samples tokens, ă(L+ .), .(t + L). For every the draft model compute the conditional probability of the next token y.
token i < L., P(y xx(t), ă(t + 1: i)), where V.
Conditional probability computation. For every token < L, we use the target model Q to compute the conditional probability of the next token Q(y xxt), :
:t+++)) i)). 3. Draft selection. We validate the first L' out of L tokens based on the conditional probabilities of the draft and target models. We set sr1 + i) = 7(t + i) for S T.. For the rejected token a(t + T/' 1), sample from residual distribution and correct it.
After steps through 3, we use æ²| T/ 4- 1) as the next context and continue the speculative decoding process iteratively. Algorithm [ details the acceptance and rejection process for each
3

---

token generated by the draft model. Rejected tokens are corrected by sampling from the residual distribution (r))
The correctness guarantees of speculative decoding are formally defined for ancestral sampling, where outputs drawn directly from the full target distribution. In protein language models, however, ancestral sampling is rarely used in practice, as it often produces implausible or repetitive sequences [8. Instead, decoding with nucleus sampling (top-p) is common balance plausibility and diversity, and is utilized with both ProGen and ProGen2 [4] [8] [6] I7. When paired with speculative decoding, nucleus sampling truncates the probabbibill tail of the target distribution, as observed in Figure Since this tail accounts for a small fraction of probability mass (p = 0.95 in this study), outputs remain closely aligned with the target model, with differences confined rare tail events.
Speculative decoding has demonstrated impressive speedups in natural language generation, where structural constraints minimal. However, existing methods are agnostic domains like protein design, where sequences must satisfy intricate biochemical and structural requirements. In this setting, small changes can disrupt folding, stability, functiin, and distributional drift between the draft and target models can accumulate over time, degrading generation quality. To the best of our knowledge, no prior work in speculative decoding incorporates sequence-lever structural information to guide draft generation, limiting its applicability in this domain. This work addresses that gap by introducing a structure-ave aware drafting strategy that leverages local sequence motifs to guide speculative decoding toward more plausible protein sequences.
Which tokens are optimal? Assume that the acceptance ratio of token srt + i) sampled from p is given by [t + i). If we assume each (( + i) is i.i.d., then [[3] = o. where a indicates the overall acceptance ratio for given configuration of speculative decoding [T0]. It follows from [1] that is the complement to the expected total variation distance between and g. Therefore, describes exactly how well p approximates q. If the generation cost coefficient ce M, is known, where Mp and Mg represent the time it takes for P and Q generate one full-length sequence, the wall-time speedup for drafts of length can be computed as:
MMp
a+
(I)
()... +1)
Therefore, an effective speculative decoder should aim to maximize a or minimize c.. Importantly, the set of tokens that increase a is not limited to the highest likelihood token under the target model. Rather, there exists a set of completions E which satisfy the requirements of Algorithm T While speculative decoding excels proposing any given token from E, only a subset of these completions may align with biologically meaningful constraints. For instance, two candidate sequences "EEL" and "VLL" (as in Figure IT] might both pass the maximal coupling acceptance check, yet "VLL" may correspond to a conformation with higher predicted structural stability. Current speculative decoding frameworks lack a mechanism to prioritize such candidates, they do not account for auxiliary objectives, which are essential in protein design but not explicitly modeled during generation.
2.2 K-mers
K-mers are contiguous substrings of length extracted from biological sequences. In proteins, k-mers often correspond to recurring structures such as alpha-helices or beta-strands, capturing local features that are critical to folding and function. Compared to analogous representations in other domains, protein m-mers encode a dense combination of physicochemical properties, structural motifs, and functional sites, all minimal computational cost. In contrast, n-grams in natural language primarily reflect syntactic or statistical patterns without necessarily encoding physical or functional constraints. Due their high information content, k-mers have been used across many applications in computational biology, including tasks such as classification, motif discovery, or alignment- free sequence comparison (18). Notably, Tranception [19] utilizes k-mer attention to better capture structural dependencies during the generation process.
3 Methods
3.1 SpecMER: K-mer Guided Protein Generation
SpecMER leverages k-mer statisties to guide protein sequence generation. Figure [ details the modified sampling procedure of SpecMER. Instead of sampling one completion at time from

---

the draft model, SpecMER batch generates multiple completions, which we refer to candidate sequences. A k-mer scoring function then assesses each length candidate sequence, providing a score based on the selected values of k. The highest-scoring candidate is then scored by the draft model and verified by the target model. To summarize, SpecMER decodes protein sequences using the following steps, which repeat until generation is complete:
1. Candidate construction. The draft model first samples L tokens times, producing batch with dimensions с x of candidate sequences.
2. K-mer scoring. A k-mer scoring function then selects the candidate sequence [ - 1)....,5(()( L)] which yields the highest score.
Conditional probability computation. For every token i L, the selected candidate sequence scored using the draft and target models.
4. Draft selection. Each proposed token in the candidate sequence is accepted or rejected in accordance with Algorithm The new context set to z(t + i) a(t + i). where is the number of accepted tokens to add.
3.2 K-mer Scoring
After the draft model samples candidate sequences, each sequence is evaluated based on its k-mer frequency. K-mers are computed prior to generation using a multiple sequence alignment (MSA) constructed for a target wild-type protein sequence. MSA data consists of homologous sequences identified based on similarity to the wild-type protein. For each wild-type sequence, we retrieve its corresponding MSA from ProteinGym [20]. The resulting alignment encapsulates evolutionary and functional constraints, making MSA-derived data particularly well-suited for k-mer analysis.
We extract k-mers by applying a sliding window of size kt across all sequences in the MSA. We selected k values of 1, 3, and 5, in accordance with [19]. We chose not to exceed k: = 5 as the number of possible k-mers can grow exponentially relative to the vocabulary size, increasing access cost during inference. K-mers are tracked over all sequences in the MSA as they appear, enabling better memory and computational efficiency during retrieval. The resulting k-mer frequencies serve as proxy for biological plausibility and are normalized to form probability distribution. During the decoding process, k-mers are used to score candidate sequences 8:
1 (core(s)
-
Σ ( KEK 1=0
where I. is the length of the candidate sequence, K is the set of k-mers to evaluate, and PP is the probability of a k-mer given the normalized distribution from the MSA. The scoring additive rather than multiplicative to avoid zero score in the case of unseen k-mers and to promote exploration of sequences with partially formed motifs. The candidate with the highest k-mer score is selected to continue the decoding process. A significant advantage of k-mer scoring is that it can be pre- computed and is easy to access during generation, contributing negligible computational overhead. Consequently, SpecMER has the same computational complexity speculative decoding: O(L?).
3.3 Performance Gain from K-mers
Vanilla speculative decoding drafts individual sequences and verifies each token in the sequence independently. The maximal coupling acceptance algorithm (Algorithm ensures that selected tokens do not diverge from the target model distribution, increasing generation speed with no sacrifice to the quality of the generated sequences. However, there could be many tokens that consistent Q(2)
with the target distribution at each verification step, passing the acceptance criteria 1) S min(1, P(z) )- To quantify the looseness of this criterion and the efficacy of external scoring functions, we describe the conditions for selecting tokens consistent with the target model and optimal under the scoring function.
Proposition 4.4. The expected batch-and-selech acceptance satisfies:
(1) -
(3)
where [[[A)] is the acceptance ratio of the speculative decoder, or is the acceptance ratio of vanilla speculative decoding, and m is the number of batch generated candidate sequences.
5

---

Proof. Let P be the draft model and Q the target as before. For a drafted (candidate) sequence let (s) be an indicator the acceptance or rejection of s. That is, (s) if Q accepts s, and
M(s) = 0 otherwise. For a given context sequence
Q(s)ar) ([M(s) = 8) r(s) min (1, P(s]a))
If is single draw, in vanilla speculative decoding. then the acceptance rate is defined as (MM = 1] = [r(s)] In the case batch generation, m i..d. candidates are selected for each draft: $1,..... Sm P. Following drafting, the scoring lunction S selects the highest scoring candidate s argmax, (s:)) Let acceptance indicator A = M(s"), ) and define = M ($t) 1) the event in which at least 8; is acceptable candidate under Q. The probability that an acceptable candidate s; exists but is not selected by S is defined by the misranking loss E E A (A* 0)]. Consider the case where no candidates satisfy the acceptance criteria of P = P(M(8)),
(smm )] = a1) Intuitively, (1 a)) is the probability that among m candidates, none pass the criteria of Q. The probability that selected candidate will be rejected is then defined by P[A* = 0] [P[F9] + IP[EE (A* = =0] = =1 - a) t + c. The overall acceptance of
the decoder is then:
E[A*] = FIA" = 00 (1
In practice, E[A"] is measured empirically the average acceptance ratio of the speculative decoder, and a is the average acceptance ratio of vanilla speculative decoding (no external scoring). e the probability that the scoring function chooses the wrong candidate, given that there was acceptable candidate choose, A good scoring function should yield < 1, indicating that it excels in selecting candidates that will be accepted by Q. Additional bounds expected speedups for given hardware configurations can found in Appendix A
Experiments
In this section, we detail experiments across seven different proteins. We tested SpecMER with ProGen2 [4), autoregressive protein language model. For each experiment, conditioned generation on fixed context length from a given wild-type protein as detailed in Table [ We selected proteins with varying molecular functions and lengths to ensure robustness of testing. Each experiment consisted of generating 200 sequences on an NVIDIA RTX A6000 GPU. Sequences were generated up until the length of the wild-type protein until stop token generated. We swept across a set of hyperparameters for each decoding method to determine the configuration with the highest acceptance ratio and generation speed. Results of full hyperparameter sweep are located in Appendix B.3
4.1
Datasets
We focus on the conditional generation task, where the objective is to generate starting from context sequence. We selected seven proteins with varying functions and collected their MSA from ProteinGym 22) For each protein, we set the context length to roughly 10% of the wild-type sequence, balancing exploration of sequence space while avoiding common pitfalls in autoregressive generation such as pathological repetition. Table I details each protein and respective molecular function.
Table Summary of proteins and context length used.
Protein
Description
Molecular Function
Length
Context
MSA Sequenees
Citation
GFP
Green fluorescent protein
Fluorescence
238
211
RBP1
RulA-binding protein I
Stability
52
10
135922
223
ParD3
Antitoxin ParD3
Growth enrichment
93
15
38613
28
GB1
IgG-binding domuin of protein G
Binding
10
44
24
Bg13
3-glucosidase
Enzyme function
105913
ADRB2
Beta-2 adrenergic receptor (GPCR)
Receptor activity
413
40
204722
26
CBS
Cystathionine belu-synthase
Growth
551
50
19563
27
We opt place of to avoid confusion with set complement.

---

4.2 Experimental Setup
Hyperparameters. Each experiment consisted of generating 200 protein sequences given the starting context sequence (Table D]. We swept over the following hyperparameters: draft tokens e {5, 10, 15), temperatures T E {0.7, 1, 1.4), and k-mers E {11, (3), (1,3), 3)). Sequences were sampled using nucleus (top-p) sampling, setting p 0.95. We opted for a large value of p to enable diverse sampling, controlling how deterministic the output is instead with temperature.
Evaluation metries. We measured four categories across all experiment configurations: acceptance ratio, negative log- oog-likelhhodd (NLL), top-20 NLJ., and top-5 NL.L. Acceptance ratio a is defined as:
No. accepted tokens
(6)
No. accepted tokens + No. rejected tokens
where a higher acceptance ratio is better. After generation, we computed NLLL, top-20 NI.I, and top-5 NI.L under the target model, normalizing for length. We assessed the top-20 and top-5 II. as protein design experiments often create a library of sequences and filter for only the most plausible candidates. Sequences with higher likelihoods are more likely to resemble natural proteins, making likelihood proxy for sequence quality (28 22]. In addition to likelihood, we computed predicted local- pocal-ditaacce difference test (pLDDT) scores 30 using ESMFold B1. The pLDDT score measures how confident the structure prediction is for given protein sequence, and has been demonstrated to correlate with the likelihood that a protein sequence will fold into a stable structure [30]. We evaluated biological plausibility using both sequence likelihood and pLDDT, under the assumption that sequences achieving high scores on both metrics are more likely to adopt stable, protein-like structures consistent with natural biological constraints. Sequence likelihoods and pLDDT scores for each wild-type sequence can be found in Appendix [33
We generated embeddings for both MSA and sampled sequences (c 1,2,3, 5) using ESM2 [31], and used PCA to qualitatively assess alignment between natural and generated sequences. For each decoding method, selected the top three configurations by average NLL, further filtering for the 100 sequences with the highest pLDDT scores. Bg13, ADRB2, and CBS were excluded from the PLDDT score analysis due to sequence length limitations of ESMFold.
Generation speed experiments were conducted using ProGen2-S and ProGen2-M as baselines. We tested vanilla speculative decoding (c = 1) and SpecMER (c = 2, 3, 5) against these baselines, tracking tokens per second. We swept over range of hyperparameters find the fastest configuration, averaging over 20 generated sequences.
4.3 Results
Acceptance and logeiiklioood. The best results per experimental configuration are reported in Table 2 We observed that acceptance ratios were marginally higher on average for SpecMER compared to speculative decoding and increased with e. Sequences generated with SpecMER had significantly lower NLL, including lower top-20 and top-5 NLL. That is, while k-mer scoring increased acceptance for some configurations, it nearly unanimously produced protein sequences with higher likelihood.
We observed that the acceptance ratio was sensitive T and k. Specifically, strongly influenced the acceptance ratio for some proteins, while others were relatively unaffected. For example, GFP exhibited the best performance with i: = 3). whereas RBP1 had no preference toward k. A full discussion on important hyperparameters can be found in Appendix B.3
Generation of plausible sequences. SpecMER yielded sequences with higher likelihoods compared to speculative decoding, indicating that sequences generated by this framework are more likely to be biologically plausible. To corroborate this claim, we computed PLDDT scores, a measure that correlates with the likelihood that a protein will fold into a stable conformation. Table 3 details the average PLDDT scores across 300 sequences per configuration. We observed that on average, SpecMER produces sequences with higher pLDDT scores. GFP was an exception to this trend, exhibiting higher scores for vanilla speculative decoding. However, top-5 pLDDT scores for GFP demonstrated that top-end performance was comparable between all decoding methods (Table [0 in Appendix D). The PLDDT scores did not increase linearly with e; instead, the best value of e varied across proteins. This suggests that the optimal value of reflects a trade-off between exploration of diverse sequence space and adherence to the likelihood distribution of the target model. Furthermore, this trade-off may depend on structural characteristics of the target protein.
7

---

Table 2: Decoding results using ProGen2. Each metric is averaged over 200 generated sequences. The value of с chosen for SpecMER experiments represents the number of batch-gene aatch-generated candidate sequences, with eters conducted as described in Section The best results in each category are reported. A full
indicating speculative decoding. For each method, sweep over hyperparam-
summary of results is located in Appendix B.3
Decoding Method
Protein
Accept Ratio 1
NLL +
Top-20 NLL +
Top-5 NLL +
GFP
0,911 + 0.029
2.45 +: 0.42
1.38 = 0.32
0.98 + 0.11
RBP1
0.938 + 0.043
2.73 0.19
0.19
2.01 + 0.44
ParD3
0,902 + 0.036
1.93 0.59
0.20
0.50 0.09
Speculative Decoding
GB1
0.927 0.040
0.19
2.20
0.33
BgI3
0,852 + 0.025
0.11
0.76
0.67 + 0.08
ADRB2
0.866 + 0.079
0.90
1.18
0.78 0.16
CBS
0,910 = 0.051
2.42
2.06
1,43 143:0.3
GFP
0,937 = 0.076
1.68
0.44
0.35 + 0.04
RBP1
0,926 + 0.038
0.32
+
1,48
ParD3
+ 0.919
1.46
0.15
0.06
SpecMER (c =
GB1
= 0.043
2.71
2.43
0.57
Bgl3
0.869 + 0.030
0.11
+
0.54
ADRB2
0.857 0.062
1.50
0.77
0.01
CBS
0.902 = 0.081
2.17
1.47 0.50
0.16
GFP
0.945 + 0.084
1.09 0.64
0.41 0.07
0.33 0.02
RBP1
0.926 = 0.047
2.41
1.25
ParD3
0.942 0.063
0.12
0.52 0.05
SpecMER (c :C=5) 5)
GB1
0.925 = 0.038
2.61 2.61
2.20
0.15
Bgl3
0.867 + 0.022
0.17
0.63 0.11
0,46 0.05
ADRB2
0.869 1 0.082
1.03 + 0.60
0.57 1 0.11
0.43 1 0.04
CBS
0.908 = 0.066
1.87 0.68
+.14
0.74 + 0.07
Table 3: Average PLDDT scores across four different proteins. Sequences are collected from the three best configurations for each decoding method, determined by the highest average log likelihood under the target model (ProGen2-M)
Protein
Speculative Decoding (c = SpecMER (c = 2)
SpecMER (c = 3)
SpecMER (c = 5)
GFP (t)
0.493 + 0.156
0.449 = 0.160
0.479 0.191
0.426 + 0.176
RBPI (f)
0.571 + 0.1
0.664 + 0.127
0.700 + 0.115
0.740 + 0.105
ParD3 (f)
0.638 + 0.206
0.650 = 0.202
0.584
0.201
0.519 + 0.176
GBI (1)
0.465 + 0.084
0.464 = 0.071
0.477 0.075
0.504 0.09
When compared to speculative decoding (c 1), SpecMER (c 5) generated sequences that more closely aligned with homologous sequences from an MSA, while also exploring beyond the MSA centroid. Figure 2a illustrates this result for RBP1, where SpecMER captures both MSA- like sequences and novel variants. Notably, many of these divergent sequences exhibited high likelihood under the target model and achieved higher pL.DDT scores average (Figure 2 and Table 3 indicating greater degree of biological plausibility. Beyond structural plausibility, we also assessed sequence diversity and found that both speculative decoding and SpecMER generated novel sequences far from the wild-type while maintaining high inter-sequence diversity (see Appendix [] Taken together, these results suggest that k-mer guidance preserves evolutionary plausibility while facilitating exploration of high-quality regions of sequence space.
To investigate the role of deriivv k-mers, we conducted two ablation experiments (see Ap- pendix for full details). First, we tested SpecMER under cross-protein mismatches, conditioning on GFP while using GB 1-deiieee k-mers to select continuations, and conditioning GB1 while using Rgl3-derived k-mers. In both cases, likelihoods dropped relative to protein-specific k-mers, indicating that observed gains depend on the correct evolutionary context. Second, reduced the

---

b
PC2 (18 variance)
PC2 (16 10% variance)
Log Likellhood (Cmr)
-13
PC1 (23.27% variance)
PC1 (23.27% variance)
Average PLDDT Score
biological plausibility of sequences generated by vanilla speculative decoding (c = 1) versus SpecMER for RBP1. a, PCA of embeddings from sequences generated by speculative decoding (c 1) (highlighted the left plot) and SpecMER (highlighted on the right plot), compared to embeddings from the MSA of RBP1. Each sequence is shaded by its likelihood under the target model (ProGen2-M). Sequences generated by SpecMER cluster more closely to the MSA embeddings and demonstrate a higher likelihood, indicating a higher degree of biological plausibility. b. PLDDT distributions for different configurations SpecMER, with e = representing vanilla speculative decoding. SpecMER demonstrates higher structural confidence scores, indicating that generated sequences are more likely to fold.
Figure 2:
Impact of candidates
5)
depth of the Bg13 MSA from 100k to 1k sequences. The resulting likelihoods steeply declined, supporting our conclusion that MSA quality impacts the performance of SpecMER.
While vanilla speculative decoding (c routinely undersamples the high-likelihood tail of the target distribution (Figure Te: see Section 2.1), , our experiments demonstrate that SpecMER overcomes this limitation. By using k-mers extracted from the MSA to bias the draft model toward structurally grounded completions, SpecMER achieves improved coverage of biologically plausible sequences, often exceeding the likelihoods obtained by target-only decoding. Table 4 compares the top-20 NLL values from SpecMER (c = 5) compared to those from the target model, highlighting the improved coverage of high-likelihood sequences.
Table 4: Comparison top-20 NLL from sequences generated by the target model (ProGen2-M) versus SpecMER using the same temperature values. Sequences are scored using ProGen2-M.
Method
Bgl3
GFP
RBP1
GB1
ParD3
Target
0.78 + 0.02
0.51 + 0.04
1.62 + 0.15
2.27 + 0.24
0.69 + 0.11
SpecMER (c
0.63 0.11
0.41 = 0.07
1.72 0.30
2.20 + 0.31
0.67 + 0.12
Wall-time speedups. We observed an increase in sequence likelihoods as the number of candidates increased (Table[Bh. This improvement, however, incurred slight cost to generation speed, indicating a trade-off between sequence quality and computational efficiency. As demonstrated in Table generating more candidates slowed decoding, though the decline in speed was minimal. Speculative decoding (c = 1) achieved the fastest generation speed with an average speedup of 32%, while SpecMER (c 5) yielded the slowest configuration, achieving modest 24% speedup. Notably, e= = 3 yielded the largest increase sequence likelihodo for the smallest decrease in speed. We noticed increase in variance increased due to batch generation, which we discuss further in Appendix B.I.
We also tested the generation speed with ProGen2-XL. as the target model and ProGen2-M as the draft, finding that with e 3, SpecMER demonstrated a speed increase of 38% over target-only decoding. In reference to the example from the introduction, SpecMER (c 3) would have taken 40 hours to generate 20,000 protein sequences of length 200 amino acids, saving an entire day of computing.

---

Table 5: Generation speed (tokens/sec) over the best configuration per method. Results are averaged over GFP, RBPI, and GB1. The draft and target models are ProGen2-S and ProGen2-M, and the baseline is speculative decoding (c = 1). Speedup is calculated with respect to the target model.
Draft
Target
Baseline
SpecMER (c 2)
SpecMER (c = 3)
SpecMER (c 5)
Toks/sec
74.11
31.48
41.62 + 0.88
41.43 - 1.24
40.57
2.83
39.11 + 4.50
Speedup
32%
32%
29%
24%
While increasing the number of candidates decreased tokens per second, it resulted in selecting better candidates the majority of the time. SpecMER (c = 5) had the lowest misranking error (Equation acceptable sequence from a pool of candidates 92% of the time. Of the set of tokens consistent with the target distribution, SpecMER was able to correctly discern between tokens that increased acceptance and tokens that increased both acceptance and biological plausibility. To summarize the trade-off space, increasing decreases tokens per second, and negative logllikelhood, while increasing acceptance ratio and pLDDT scores. Further discussion on this trade-off space can be found in Appendix B.2
3 and Figure 3) selecting
5 Conclusion
Discussion. In this study, we developed the first speculative decoding framework for protein generation. Using ProGen2-S the draft model and ProGen2-M the target model, we achieved an average speedup 32% across functionally diverse proteins. Building on this foundation, we developed SpecMER, novel framework that incorporates k-mer guidance derived from multiple sequence alignments to guide generation toward biologically meaningful regions of sequence space.
SpecMER drafts multiple candidate sequences and uses evolutionary patterns from k-mers to select completions that are more likely to fold into stable, biologically plausible proteins. As illustrated in Figure D speculative decoding often undersamples high- high-likeihhood sequences, despite proposing tokens consistent with the target model distribution. SpecMER addresses this deficiency by guiding generation toward sequences that both align with MSA-derived motifs and exhibit improved log- likelihood and pLDDT scores. In doing so, it preserves the speed advantages of speculative decoding while producing sequences that more likely to fold into stable conformations, achieving the best of both efficiency and quality.
Beyond empirical gains, we also provide bounds wall-time speedups and define the criteria for selecting tokens that conform to guidance and acceptance objectives. Furthermore, we offer practical guidance for tailoring hyperparameter configurations to maximize performance improvements for protein design workflows. As speculative decoding techniques continue to evolve and more protein data becomes available, we anticipate larger speedups in automated protein generation. SpecMER marks a step forward in accelerating protein design, paving the way for high-throughput design workflows and beyond.
The effectiveness of SpecMER depends on the quality of the input alignment (MSA); performance may degrade when informative motifs are sparse or unavailable. For example, target proteins with extensive disordered regions or lacking discernible structural patterns may benefit less from k-mer guidance. While k-mers impose minimal computational overhead, increasing the number of drafted candidates e increases compute and therefore energy cost, which may limit scalability in certain settings. SpecMER utilizes batch generation to efficiently draft multiple candidates, though this is not strictly parallel. In practice, a fully parallel implementation could yield even greater speedups, but was not attainable given the limitations of our hardware setup.
Limitations.
Future work. While our method was developed for conditional generation of protein sequences, its principles could extend to natural language, where low-cost scoring functions can guide generation under constraints such as style or safety.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Parker H. Petit Institute for Bioengineering and Biosciences (IBB) interdisciplinary seed grant, the Institute of Matter and Systems (IMS) seed grant, the National Science Foundation (NSF) Graduate Research Fellowship Program (GRFP), and Georgia Institute of Technology spart-up funds.
References
[1] Xiangru Tang, Howard Dai, Elizabeth Knight, Fang Wu, Yunyang Li, Tianxiao Li, and Mark Gerstein. A survey of generative AI for de novo drug design: new frontiers in molecule and protein generation. Briefings in Bioinformatics, 25(4):bbae338, 07 2024. ISSN 1477-4054. doi: 10. 10.1093/bib/bbaee338.
[2] Pascal Notin, Nathan Rollins, Yarin Gal, Chris Sander, and Debora Marks. Machine learning for functional protein design. Nature Biotechnology, 42(2):216-228, February 2024. ISSN 1546-1696. doi: 10. 00.1088641158--02--222222
[3] Jung-Eun Shin, Adam J. Riesselman, Aaron W. Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C. Kruse, and Debora S. Marks. Protein design and variant prediction using autoregressive generative models. Nature Communicationss, 12(1):2403, 2021.
[4] Erik Nijkamp, Jeffrey Ruffolo, Eli N. Weinstein, Nikhil Naik, and Ali Madani. ProGen2: exploring the boundaries of protein language models. Cell Systems, 14((1):968-) 9788 2023.
[5] Alexey Strokach and Philip M. Kim. Deep generative modeling for protein design. Current Opinion in Structural Biology, 72:226-2555 2022.
[6] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 8946-88 9946-8970. PMLR, 17-23 Jul 2022. URL httts:://proceedings mlr wwwpppresssseee https//v11/hhsssu2a html
[7] Han Spinner, Aaron W Kollasch, and Debora Susan Marks. How well do generative protein models generate? In ICLR 2024 Workshop on Generative and Experimental Perspectives for Biomolecular Design, 2024. URL https: http//ppwwwvvveeeeeeeeeeoorr//aa
[8] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL
https://openreviae wtwpeevveeveeeeefforeeeeeeeee
net. wttp://eooooodddddddgeereeeeeee
[9] Zachary Wu, S. B. Jennifer Kan, Russell D. Lewis, Bruce J. Wittmann, and Frances H. Amool. Machine learning-assisted directed protein evolution with combinatorial libraries. Proceedings of the National Academy of Sciences, 116(18 (6618))8852-888 April 2019. ISSN 1091-6490. doi:
10. 10.1073/pnas.. 19019791999991911
[10] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. Proceedings of the 40th International Conference on Machine Learning. ICML'23. JMLR, 2023.
[11] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. SpecTr:: Fast speculative decoding via optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[12] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 4 st International Conference on Machine Learning, 24 JMLR,
2024.

---

[13] Xupeng Miao. Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chuman Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. SpecInfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24, page 932-949. ACM, April 2024. doi: 10.1 09.1145/6620666.3555533
[14] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Menon, Afshin Rostamizadeh, Sanjiv Kumar, lean-François Kagy, and Rishabh Agarwal. DistillSpec: Improving speculative decoding via knowledge distillation. In International Conference on Learning Representations (ICLR), 2024.
[15] Nan Yang, Tao Ge, Liang Wang. Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless accelcration of large language models, 2023. URL, httppt:tt://arxii org/ /204. 04487
[16] Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik. Large language models generate functional protein sequences across diverse families. Nature Biotechnology. 41(8):1099-11 110,, January 2023. ISSN 1546-1696 doi: 10.
(188/411577-022-06611 URL http: doi. org 10. 1038/ #4150//418777020 -22-88888-72------222-
[17] Jeremie Theddy Darmawan, Yarin Gal, and Pascal Notin. Sampling protein language models for functional protein design. In NeurIPS 2023 Generative Al and Biology (GenBio) Workshop,
2023. URL
[18] Camille Moeckel, Manvita Marehoina, Maxwell A. Konnaris, Candace S.Y. Chan, Ioan- nis Mouratidis, Austin Montgomery, Nikol Chantzi, Georgios A. Pavlopoulos, and Ilias Georgakopoulos-Sonee A survey of k-mer methods and applications in bioinformatics. Com- putational and Structural Biotechnology Journal, 23:2289-2300 2001-0370. doi: 1.1006/...sbj.2024..00
December 2024. ISSN
[19] Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena- Marchena-Hurtado, Aidan N. Gomez, Debora Marks, and Yarin Gal. Tranception: Protein fitness prediction with autoregressive transformers and inference-ting -ieneee--mmeeeeee retrieval. In Proceedings of the International Conference Machine Learning, pages 16990-17017. PMLR, 2022.
[20] Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Nickerk, Steffanic Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and design. In Advances in Neural Information Processing Systems, volume 36, pages 64331-64379. Curran Associates, Inc., 2023.
[21] Karen S. Sarkisyan, Dmitry A. Bolotin, Margarita Meer, Dinara R. Usmanova, Alexander Mishin, George V. Sharonov, Dmitry N. Ivankov, Nina G. Bozhanova, Mikhail S. Baranov, Onuralp Soylemez, Natalya S. Bogatyreva, Peter K. Vlasov, Evgeny Egorov, Maria D. Logacheva, Alexey S. Kondrashov, Dmitry M. Chudakov. Ekaterina V. Putintseva, Ilgar Z. Mamedov, Dan S. Tawfik, Konstantin A. Lukyanov, and Fyodor A. Kondrashov. Local fitness landscape of the green fluorescent protein. Nature, 533(7603):397-001 May 2016. ISSN 1476-4687. doi: 10. 10.1038//a 17995.
[22] Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani, Jonathan J. Weinstein, Niall M. Mangan, Sergey Ovchinnikov, and Gabriel J. Rocklin. Mega- scale experimental analysis of protein folding stability in biology and design. Nature, 620 (7973):434 444, 2023. doi:
[23] David Ding, Ada Y. Shaw, Sam Sinai, Nathan Rollins, Noam Prywes, David F. Savage, Michael Laub, and Debora S. Marks. Protein design using structure-based residue preferences. Nature Communications, 15(1):1639, 2024.
12

---

[24] C. Anders Olson, Nicholas C. Wu, and Ren Sun. A comprehensive biophysical description of pairwise epistasis throughout an entire protein domain. Current Biology, 24(22):2643-2666 2014. doi: 10. (0.1016/j.cub.2011 20140.00...011.000007
[25] Philip A. Romero, Tuan M. Tran, and Adam R. Abate. Dissecting enzyme function with microfluidic-bast deep mutational scanning. Proceedings of the National Academy of Sciences, 112(23):7159-711 7164, 2015. doi: 10. 10..1033pnas.1422220 1422285 112.
[26] Eric M Jones, Nathan B Lubock, AJ Venkatakrrishnan, Jeffrey Wang. Alex M Tseng, Joseph M Paggi, Naomi R Latorraca, Daniel Cancilla, Megan Satyadi, Jessica E Davis, M Madan Babu, Ron O Dror, and Sriram Kosuri. Structural and functional characterization of g protein-coupled receptors with deep mutational scanning. eLife, 9, October 2020. ISSN 2050-084X. doi: 10.7554/elife.54895 URL http://dx doi. org/ 10.111...55544 7554/eLife. S.....
[27] Song Sun, Jochen Weile, Marta Verby, Yingzhou Wu, Yang Wang. Atina G. Cote, Iosifina Fotiadou, Julia Kitaygorodsky, Mare Vidal, Jasper Rine, Pavel Ješina, Viktor Kožich, and Fred- erick P. Roth. A proactive senotype-to-patien---oe phenotype map for cystathionine beta-synthase. Genome Medicine, 12(1), January 2020. ISSN 1756-994X. doi: 10. 181116663----000777111 URI. htRl./ttp://áxx.oo doi http:///xxddoo.or///// 10.
[28] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. ProtGPT2 is deep unsupervised language model for protein design. Nature Communications, 13(1), July 2022. ISSN 2041-1723. doi: 10. 10.1038/s41467-022 -0000-2--777
[29] Zhidian Zhang, Hannah K. Wayment-Steele, Garyk Brixi, Haobo Wang, Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary statistics of interacting sequence motifs. Proceedings of the National Academy of Sciences, 121(45), October 2024.
ISSN 1091-6490. doi: 10.10733pnas.224668555 2406285121.
[30] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler. Trevor Back. Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure pre- diction with AlphaFold. Nature, 596(7873):583-555 589, July 2021. ISSN 1476-4687. doi: 10. 10310/8/s4 (1338/411566-22 586-021-03819-
[31] Zeming Lin, Halil Akin, Roshan Rao, Brian Hic, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarande Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-sctte prediction of atomic- level protein structure with language model. Science, 379(6637): 1123 1130, March 2023. ISSN 1095-9203. doi: 10.1 26/sscience.ade2574.
13

---

Appendix
A Bounding Speedups
Definition A.1. Let ce be the cost coefficient for one iteration of SpecMER. Then,
Ge c.Vp + MR.
(7)
Mg
where M₁ is the time it takes the k-mer scoring function to score candidates. Assuming M, can batch generate candidates, then the cost of M, grows sublinearly with respect to c. Additionally, M₁ < Mp. and it follows that:
E
Ce Mg
(8)
where 5 is the batch generation cost. Under true parallelism, & 1 and the expected improvement factor remains the same as Equation (T)
Proposition A.2. The expected (batch) wall-time speedup factor of batch generation is
S(7) 1 1-a
-1-0 a)le +]
Proof. Let Tieration()) (7) be the time takes for one iteration of draft selection and evaluation to complete on draft tokens. Similarly, let (p(7), (7). and Ti be the time it takes for the draft model (single candidate), target model, and k-mer scoring function to complete one iteration.
Since (7) can batch generate, TT(7) = (7), whereas in the serial case, Tp(>) (7)) The total time for an iteration is then T Inceationn (7) = (ET,(7) Tx] + (T))), with an expected speedup of:
a a++1
S(7) = (1 - a) Tim (7)
(10)
Let us define the cost coefficient Caran (7)
+T)())+Ta Ta(2)
By Equation 8), Caeat (?) = Ce. simplifying
the ratio between total iteration time and target iteration time Thmma
Ca + 1. Consequently,
T,(7)
a++
S(7) (1 (1-a) (1-a)lce +1]
(11)
In the case that generation is serial, the wall-time speedup results in a similar equation. Corollary A.3. The expected serial wall-time speedup factor is:
7+1
5(7) ~ ax) (6 +1]
(12)
We note that (y) and (7) are functions of y as the time it takes to decode a sequence of length ? does scale linearly. In practice, generating short sequences approximately scales linearly with 7, in which near exact equality is achieved in Equation (9) and Equation (12). Increasing the acceptance ratio leads to large gains in time saved across all configurations of 7.
B Hyperparameters
B.1
Implementation Details
Code for SpecMER is publicly available here:
SpecMER git
ProGen2 4 is decoder-only Transformer architecture with four different parameter sizes: small
115M;; medium/base 764M; large 2.7B; xlarge 6.4B. No model training is performed this work: inference is on server with eight NVIDIA A6000 GPUs.
14

---

We conducted extensive testing to find the correct parallel implementation of candidate sequence generation. Batch generation, while not exactly parallel. yielded the best results in regard tokens per second. We estimate that batch generation took about 1.2 1.3x longer to generate = 5 sequences as compared to e = sequences, incurring additional latency due larger batch operations. This discrepancy in generation time resulted in larger variance in tokens per second as c increased. When a speculated token is rejected, decoding takes longer, and the added cost of batch generation appears in the variance. Due to the non-deterministic nature of sampling, some generations have higher acceptance rates than others; therefore, occasional rises and dips in acceptance have a larger impact on tokens per second than producing single sequence at a time. We implemented SpecMER on top of the sample method implementation provided by ProGen2. We considered attempting a fully parallel implementation of batched sequence generation; however, we ultimately decided to defer to the default implementation. This was due to hardware limitations and practical considerations. A fully parallel multi-GPU system, while likely yield a higher tokens per second, is significantly more demanding for practitioners who wish to use SpecMER. We implemented SpecMER with single GPU instances mind.
In addition to testing configurations for parallel implementations, we tested KV caching versus full rescoring during each iteration of SpecMER. In normal setting, KV caching would almost always be preferable. However, due to the maximal coupling acceptance logic, full rescoring versus KV caching depended the expected performance of the decoder. We stored KV caches for the draft model and target model, updating at each iteration; however, if any token was rejected and resampled using the logic in Algorithm [T] that token would require full forward pass. Since correction tokens are resampled from the residual distribution, the corresponding logits must be recomputed with a full forward pass. This poses tradeoff: either compute full uncached forward pass for both the draft and target model at each iteration, or compute the cached forward pass at the risk of incurring an extra forward pass for every rejected token. We observed that as long as the speculative decoder achieved an acceptance ratio of x 80% or higher, the KV cache implementation was faster. For this reason, we chose use custom implementation of KV caching for SpecMER. We recognize that our implementation of SpecMER is not fully optimized and could be improved to realize even faster speedups. We leave this task for future work, or for specialized implementations with access to better computing hardware.
B.2 Trade-off Space
Selecting the number of candidates to draft e comes at the cost of a trade-off between generation speed and generation quality. Figure 3 details how e influences generation speed and misranking error e. Setting e = 3 yielded the best tradeoff in this space, demonstrating the largest gain in likelihood improvements for the least decrease in tokens per second. We observed an improvement in misranking eтror for SpecMER, indicating that structural cues from SpecMER not only increase structural plausibility but also help close the distribution gap between the target and draft model. The target model in protein generation often exhibits higher correlation with functional and structural benchmarks, learning these auxiliary objectives implicitly from millions of protein sequences. This correlation increases as a function of the number of parameters. SpecMER illustrates that this auxiliary gap between the target and draft can be supplanted with external functional or structural information.
B.3 Results From Hyperparameter Sweep
This section details how each hyperparameter impacted acceptance and likelihood. As stated in the main text, we swept over the following hyperparameters: draft tokens (7), temperature (T), and k-mers (k). We tested every combination of the following configurations: 7 {5, (5,10, 15). T E 1, & E {(1), (3), (1, (1, 5)}. We chose this consolidated list to encompass realistic hyperparameters used for protein generation. For each configuration, we generated 200 protein sequences, stopping generation at max sequence length (length of the wild-type for the context being used) or upon generation of a stop token ("2" in the case of ProGen2). Depending on the protein selected, testing a single configuration could last anywhere between 2 30 minutes to generate 200 full-length sequences. The final hyperparameter set used to report results in Table is listed in Table [
15

---

a
b
Average Negalive Log-Likelhhodd
2.25
2.00
(e)
0.090
Candidates
0.088
1.50
0.006
1.25
1.00
0.060
40.0
40.5
41.0
41.5
Generation Speed (tokens/sec)
Number of Candidates
Figure 3: Trade-off space between number of candidates, misranking error, and negative log- likelihood (NLL). a, Generation speed versus NLL. As increases, NLL drastically increases at the penalty of reduced generation speed. b, Number of candidates versus misranking increases, misranking error decreases, indicating the influence k-mers on selecting candidates that maximize acceptance while also maximizing the k-mer score.
As
Table 6: Final hyperparameter configurations for each protein corresponding to results in Table
Protein
Temperature
Draft Tokens
l value
Candidates
Bgl3
1.0
3
5
GFP
0.7
1.3
RBPI
1.0
10
3
GB1
1.4
10
1,3,5
ParD3
1,3,5
CBS
1,3,5
ADRB2
1.3
We observed variance generated sequence lengths, largely dependent on temperature. Sharper temperatures (T 0.7) led to longer sequences average, shifting probability mass away from producing a stop token early. Sequence likelihoods were heavily influenced by T, with likelihood increasing average when temperature decreased. This is not unexpected result: decreasing temperature with top-p sampling results in a more deterministic sampling procedure. With this in mind, we chose generate sequences over range of temperature values, as increased sequence likelihood does not always result in more plausible structures.
We noticed that each protein had subtle preferences toward the selection of Shorter proteins, such as ParD3 and GB 1, preferred = 5, while longer proteins, such as GFP, saw improvements with y 15. Bgl3 and RBP1 showed significant preference toward this hyperparameter. Plots pertaining to the effect of T, k, and c are available at the conclusion the appendix.
For additional context to the main results, we include the PLDDT and NLL measurements for each protein Table
Table 7:
Negative logelikel (NLL) and pLDDT scores for each wild-type sequence. Missing values indicate proteins for which structural predictions were not available.
Protein
NLLL
PLDDT
CBS
0.75
Bg13
0.92
ADRB2
1.31
ParD3
2.11
0.79
GBI
2.52
0.82
RBPI
2.63
0.83
GFP
2.93
0.42
16

---

C MSA Ablations
We performed two ablation studies to test the efficacy of k-mers in SpecMER.
Cross-protein k-mers. We tested SpecMER using mismatched evolutionary signals: (1) generate conditioned on GFP with Idl-evveeed k-mers, and (2) generate conditioned on GB1 with Bgl3- derived k-mers. The results are shown in Table 8 In both cases, we observed a drop in likelihoods, both on average and for the top-20 highest-likefhoon sequences, compared to the protein- results in Table 4 This suggests that protein-specifee k-mers are responsible for the increase in likelihood observed with SpecMER.
Table 8: Cross-protein k-mer ablation results.
Condition
Mean NLL
Top-20 NLL
GFP + GB1 k-mers
2.52 + 0.27
1.78 = 0.23
GB1 + Bgl3 k-mers
2.79 0.10
2.59 + 0.11
MSA depth. We next tested the importance of MSA depth by generating Bgl3 proteins using only 1,000 sequences from the MSA instead of the full set of ~130 sequences. This reduction limited the number of k-mers available to guide generation. The top-20 sequences yielded an average NLL of 1.56 + 0.20, compared to 0.63 = 0.11 for SpecMER (c 5) with the full- MSA. These results support our conclusion that SpecMER's performance can degrade when informative motifss are sparse or unavailable.
D Structural Scores and Embeddings
This section describes experiments pertaining to structural scores, diversity metrics, and embeddings for generated protein sequences. For each section, metrics are computed over all tested configurations of vanilla speculative decoding and SpecMER.
D.1 Sequence Diversity
Novelty is key metric for determining how well generative models sample from the space of learned protein sequences. We computed two diversity metrics to assess how SpecMER affects sequence novelty: WT Hamming distance and inter-sequence Hamming distance. WT Hamming distance illustrates how many edits away the generated protein is from the wild-type. Inter- sequenee illustrates how similar sequences generated under the same configuration are. Results for both metrics are reported in Table D.I
Table 9: Wild-type (WT) distance and inter-sequence (Inter-Seq) distance across proteins. SpecMER generates sequences further from the WT while maintaining inter-sequence diversity, thereby explor- ing sequence space while producing plausible protein sequences.
Protein
WT Dist. (SpecMER)
WT Dist. (Spec. Dec.)
Inter-Seq (SpecMER)
Inter-Seq (Spec. Dec.)
GFFP
208.35 4: 5.76
208.49 4,91
181.78 27.14
184,56 27.14
RBPI
41.27 :: 3.48
42.81 1: 3.43
42.60 1 3.87
44,88 4,00
ParD3
75.97 3.01
78.68 + 2.41
67.39 6.87
70.00 + 5.41
GBI
44.70 = 3.25
45,27 1: 3,27
46.47 4,46
46.99 +: 3.86
Bg13
324.88 30.88
333.12 + 33.96
261.02 + 42.28
284.97 + 40.44
CBS
378.64 + 140.60
431.00 + 106.05
291.84 = 161.60
457.27 + 40.98
ADRB2
263.80 1 120.22
290.18 109.01
270.82 93.46
340.28 47.66
We observed that SpecMER followed the same trends as speculative decoding for both WT and inter-sequence distance. Both methods generate many dissimilar designs far away from the WT, ensuring that sequence generation explores novel sequence space, a key advantage of autoregressive PLM generation.
17

---

D.2 Embeddings
We assessed embeddings of sequences generated from vanilla speculative decoding and SpecMER. Embeddings were generated using ESM-2 8M [31] with mean pooling and unaligned sequences. ESM-2 is a protein language model that embeds sequences that are functionally or structurally similar closer together. For each protein used context for conditional generation, we computed embeddings for its respective MSA. Sequences from the MSA are expressed in nature, providing an anchor point for which sequences are likely to be biologically plausible. We used PCA to compare embeddings over two principal components, finding that sequences generated by SpecMER were closer in proximity to the MSA sequences than vanilla speculative decoding. We deduced from this analysis that k-mers formed from the MSA did guide generation toward biologically plausible sequences. However, this analysis alone does not provide enough evidence to support this claim, embedding distance and PCA are approximate metrics for sequence plausibility. To further corroborate claims of plausibility, we computed structural confidence scores detailed in the next section. Embedding plots for each protein can be found at the conclusion of the appendix.
D.3 PLDDT Scores
We assessed pLDDT 30 scores as a structural confidence measure. pLDDT ranges from 0 100 at each amino acid site, with 0 indicating structure very unlikely to fold, and 100 indicating a structure very likely to fold. We computed PLDDT scores using ESMFold [31, a structure prediction model that provides per-residue pL.DDT scores. To assess the overall structural confidence generated proteins, we averaged the pL.DT scores across all amino acid positions. Figure 2b details pL.DTT scores from top configurations of vanilla speculative decoding and SpecMER for different values, with generation conditioned on RBPI. For each candidate we identified the three best configurations for each method, determined by average likeliiddoood. Furthermore, we selected the top- 100 most likely sequences from these three configurations, resulting in set of 300 data points for each value. This was done to ensure diversity in the set of sampled sequences. We observed that as c increased, so did the average pLDDT score. This indicates clear relationship between increasing the influence of k-mers and generating sequences with higher biological plausibility.
We computed average (Table pLDDT scores for all proteins with the exception of Bgl3, This is due to the sequence length, which exceeds 400, and is therefore incompatible with ESMFold. For the remaining proteins, we selected the top 300 sequences across the three best configurations for each method as described before. We observed that sequences generated using SpecMER yielded the highest pLDDT scores, with the exception of GFP, which demonstrated the highest PLDDT scores under vanilla speculative decoding. This was a surprising observation, as conditional generation using GFP the context sequence yielded higher log- lgelikklisoods when using SpecMER. However, top-5 statistics indicated higher-end scores were comparable for GFP. A summary of top-5 PLDDT scores can be found in Table [0] We observed similar trend with top-5 and average pLDDT scores. That is, increasing led to higher pLDDT scores, with the exception of GFP, which was competitive across all values of c.
Table 10: Top-5 pLDDT scores across four different proteins. Sequences are collected from the three best configurations for each decoding method, determined by the highest average log-likelihood under the target model (ProGen2-Mr -M).
Protein
Speculative Decoding (c 1) SpecMER (c 2)
SpecMER (c = 3)
SpecMER (c = 5)
GFP
0.892 +: 0.020
0.838 +:2 0.062
0.888 + 0.008
0.850 +: 0.035
RBP1
0.864 0.019
0.884 1 0.016
0.876 1 0.016
0.888 0.011
ParDa
0.912 + 0.004
0.918 + 0.008
0.922 + 0.004
0.926 + 0.005
GB1
0.790 0.030
0.704 +:: 0.010
0.760 + 0.059
0.794 0.019
E
Multiple Sequence Alignment and K-mer Generation
We collect MSAs of target proteins from ProteinGym 20 As described in Section 3.2 k-mers are
formed by applying a length sliding window over all sequences the MSA of a respective protein. Gap characters 5_" are ignored.
18

---

We observed that the best & value for each protein varied substantially. Some proteins demonstrated little no preference toward k, such as RBPI and GBI, whereas GFP was heavily influenced by k = {1.3} and & {1,3, 5). It is possible that for proteins comprised of large disordered regions, SpecMER would show reduced effectiveness, or even detrimental generation. Plots describing how k values affected log-likellhood can be found at the conclusion of the appendix.
19

---

Log-Likelihood by K Valuue
Type:
-053
MSA Aug: -2224
-100
Log- Li-Liielhood
-1.5
-3.0
1.3
135
K Values
Figure 4: Log-likelihood versus selection of is for ParD3.
Log-Liken Likelihood by Number Candidates
-1.0
Log-Likelhood
-1.5
-23
Wild Type: 2211
MSA Aug: -2.24
1.0
3.0
5.0
Number of Candidates
Figure 5: Log-likelihood versus selection of candidates for ParD3.
20

---

Log Likelihood by Temperature
Type: MSA Avg: -2.24
-0.5
-10
Log-Likelihooz
-2.0
-25
0.7
144
Temperature
Figure 6: Log-likelihood versus selection of temperature for ParD3.
Distribution of Sequence Log-Likelihoods
Generated
1400
1700
1000
Count
400
200
0
Log-Likelihood
Figure 7: Likelihood distribution ParD3 sequences generated using SpecMER compared to the
likelihood distribution of its collected MSA (scored using ProGen2-M).
21

---

PCA of Protein Embeddiings By Number Candidates vs MSA
Nambis of Carelccetao))
Number 2
Number Cardicateen
Cardicatee
PC 118 61% valianced
Figure 8: PCA plot for sequence embeddings generated with varying c, with c 1 denoting speculative decoding, and >> 1 denoting SpecMER. Sequences are generated using the context of ParD3.
22

---

Log-Likelihood by K Values
Wild
MSA Avg: 2.15
-1.0
-1.5
Log-L Log-Likelhood
-3.0
1.3
1.3.5
K Values
Figure 9: Log-likelihood versus selection of ls for GBI.
Log-Like Lig-eiieldhood by Number Candidates
MSA Aug: 2.15
-1.0
-1.5
Log-Likelhood
-2.0
-25
1.0
3.0
5.0
Number of Candidates
Figure 10: Log-likelihood versus selection of candidates for GBI.
23

---

Log Likelihood by Temperature
Type: MSA Avg: -2.15
-1.0
Log-Likelihood
-.0
-25
-3.0
0.7
144
Temperature
Figure 11: log-likelihood versus selection of temperature for GBI.
Distribution Sequence Log-Likelihoods
Generated
400
Count
300
100
-1.0
Log-Likelhhood
Figure 12: Likelihood distribution of GB1 sequences generated using SpecMER compared to the likelihood distribution of its collected MSA (scored using ProGen2-M).
24

---

PCA of Protein Embeddings By Number of Candidates vs MSA
Numbor ol Cardhdelas
Nummer of 2artBdates=
Number Cardidamen
Cardidatss-d
PC2 111 .45 valiancel
Figure 13: PCA plot for sequence embeddings generated with varying with 1 denoting speculative decoding, and >> 1 denoting SpecMER. Sequences are generated using the context of GBI.
25

---

Log-Likelihood by K Values
-0.5
-1.0
Likeeiod
-1.5
Wild Type:
--- MSA Aug: -2.89
-20
-3.0
1,3
135
K Values
Figure 14: logeliiel versus selection of ie for GFP.
Log-Liken Li-LLiielihodd by Number Candidates
MSA Aug: 2.89
-0.5
-1.0
Log-Likelhood
-2.0
-25
1.0
3.0
Number of Candidates
Figure 15: Log-likelihood versus selection of candidates for GFP.
26

---

Log Likelihood by Temperature
-0.5
-10
Log-Likelihood
-20
-2.5
Wild Type: 2.91
2.89
0.7
14
Temperature
Figure 16: Log-likelihood versus selection of temperature for GFP.
Distribution Sequence Log- LigeLiiklihodds
1600
sequence
Generated
Count
400
200
Log-Likelihood
Figure 17: Likelihood distribution GFP sequences generated using SpecMER compared to the
likelihood distribution of its collected MSA (scored using ProGen2-M).
27

---

Embeddiings
By Number
Sequences Centrold Type
varfancel
PC1 (44.07% variance)
Figure 18: PCA plot for sequence embeddings generated with varying with 1 denoting speculative decoding, and >> 1 denoting SpecMER. Sequences are generated using the context of GFP.
28

---

Log-Likellnn Log-Likellhood by K Values
-0.5
hi-Liiellhood
-1.5
-2.
Wild Type: 0.92 MSA
-3.0
1.3
1.3.5
K Values
Figure 19: Log-likelihoo ligeliidl versus selection of & for Bgl3.
Log-Like Ligeeikklihodd by Number Candidates
MSA Avg: 1.38
-0.5
-1.0
Log-Likelhood
1.0
3.0
Number of Candidates
Figure 20: Lik-likelihood versus selection of candidates e for Bgl3,
29

---

Log Likelihood by Temperature
Type: MSA Avg: -1.38
-0.3
Log-Likelihoop
-2.
-3.0
0.7
1.4
Temperature
Figure 21: Log-likelihood versus selection of temperature for Bgl3.
Distribution of Sequence Log-Likelihoods
Generated
2500
2000
Count
500
Log-Likelihood
Figure 22: Likelihood distribution of Bgl3 sequences generated using SpecMER compared to the likelihood distribution of its collected MSA (scored using ProGen2-M).
30

---

Embeddings Candidates
aaCrdiictess
MSA Sequanoess
MSA Centrald
WMl Tyne
PC2 48%48% variano))
PCI (56.03% varience)
Figure 23: PCA plot for sequence embeddings generated with varying with 1 denoting speculative decoding, and >> denoting SpecMER. Sequences are generated using the context of Bgl3.
31

---

Log-Likelihood by K Values
Wild Type: -2.63
MSA Aug: 2244
-1.0
-15
Li-Likklhood
-3.0
1.3
133.5
K Values
Figure 24: logeliiel versus selection of ls for RBP1.
Log- Likelihood by Number Candidates
MSA Aug: 2.45
-1.0
-1.5
Likeliodl
-2.0
-2.5
1.0
3.0
Number of Candidates
Figure 25: Log-likelihood versus selection of candidates for RBP1.
32

---

Log- Likeeiidddood by Temperature
Type: MSA Avg: -2.45
-1.0
-15
wog-Likeihhodd
-2.0
-25
-3.0
0.7
14
Temperature
Figure 26: Log-likeline aigllike versus selection of temperature for RBP1.
Distribution Sequence log-Likelshoods
Generated
Count
400
200
-3.0
-2.0
Log-Likelihood
Figure 27: Likelihood distribution of RBP1 sequences generated using SpecMER compared to the likelihood distribution of its collected MSA (scored using ProGen2-M).
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Generation speed claims are supported in Section 4.3 and Tables 2 and 5 Claims of sequence plausibility are corroborated in Section 4.3 Figure 2 and Tables and
Theoretical results are proven in Section 3.3 with additional results proven in Appendix
The scope claims is centered around protein generation, which is supported by over five different functionally diverse protein sequences.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations discussed thoroughly in the conclusion (Section 5 addressing data, scalability, and hardware limitations.
Guidelines:
The answer NA means that the paper has limitations, while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
34

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are described in Section 3.3 as well as Section 2 Proofs can
found Section 3.3 and Appendix [A] Additional assumptions are supported by related works and are properly attributed credit.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The presented algorithm implementation details are described in Section - Section 3 Figure I and Algorithm Code is released for exact implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
35

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code made available on GitHub, and can be found Section I and Appendix B.1 Workflow details are available in the GitHub repository README file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guidess
onllicyyy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips //nips.cc/ppp httpol/i/ggiidee//ooooooo
sssiisssiniiii ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Training details and evaluation metrics can be found in Section 4 details required to reproduce results are located in Appendix B.3
Additional
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Standard deviation for experimental results are presented in each table. No figures were presented that required error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Hardware and compute details are presented in Section Expected times for reproducing results can be ascertained from Section 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
This work conforms the guidelines outlined in the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
37

---

Justification: While the broader field of protein design has clear societal implications, this work presents a methodological contribution that is tangential applications with direct positive or negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not pose any risks that require safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use the publicly available ProGen2 model and existing protein sequence datasets. We cite the original works for each of these assets and follow usage terms as provided by these works.
Guidelines:
The answer NA means that the paper does not existing assets.
38

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code for new assets is released publicly and includes extensive documentation both within this work as well the repository where the code stored.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing human subjects.
39

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Method development did not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
40


Reviews:
[{"review_id": "H9QICKyEIp", "reviewer": "Reviewer_iBwT", "summary": "Authors note that a limiting step in ML protein engineering is generation. Thus, they introduce SpecMER - a novel protein generation framework that accelerates autoregressive generation using speculative decoding guided by biologically-informed k-mer motifs from MSAs. By integrating these k-mer patterns, SpecMER prioritizes biologically plausible sequences during generation, improving both structural fidelity and functional plausibility without sacrificing speed. Across five diverse proteins, SpecMER achieves up to 32% faster generation compared to standard methods and consistently improves log-likelihoods and predicted structural confidence scores (pLDDT).", "questions": "1. How does ProGen-XL, -S, -M compare in speed to each other and how does that compare to other autoregressive models that exist? Having a 32% speedup is great, but if other AR models are already faster then wouldn't the protein designer just want to use those? No cross-model speed benchmarks are included, and no papers are cited that provide such benchmarks either.\n2. Can you provide more experiments to back up the claim \"The effectiveness of SpecMER depends on the quality of input alignment (MSA);\nperformance may degrade when informative motifs are sparse or unavailable\"? It is intuitive, but not actually supported by your work. Right now, you have described the size of the MSAs for your different proteins of interest but there is not information on the \"quality\" of the MSA - are you talking about how well aligned everything is? the depth of the MSA? how many relevant proteins are included/excluded? I believe you just pulled the MSAs from ProteinGym, so perhaps consider assessing the quality and seeing if it leads to changes in performance in the model. Also consider testing multiple MSAs per protein (e.g. at different bitscores) and see what impacts performance.\n3. There is a lot of talk about the tradeoffs of \"cheap\" generators generating a ton TON of sequences and then having a good filtering set that can quickly weed out the bad sequences. Have the authors considered this? Have you dont any comparison between this paradigm and the speculative decoding approach?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality:\n\nStrengths\n\n- The authors build a robust and thoughtful generative framework for protein sequence design that balances speed, plausibility, and computational cost.\n- SpecMER is clearly motivated and well-engineered, combining speculative decoding with biological priors (k-mers) in a way that addresses their motivation\n- The paper includes some empirical support: results across five functionally diverse proteins, ablation over multiple hyperparameters (e.g. number of candidates, k-mer size, temperature), and evaluation using both likelihood and structural metrics like pLDDT.\n\nWeaknesses\n\n- Authors should include a few additional citations to increase the quality of their scholarship and expand upon why generation from autoregressive models is challenging. Yes, it can take a while, but more than that there are some pathologies with that kind of generation that have been discussed in the literature:\n(A) Cite Hsu et al. 2022, “Learning inverse folding from millions of predicted structures” (ESMF1) — they mention common autoregressive sampling failures, like pathological repetition (e.g., “EEEEEEE”).\n(B) Additionally, Spinner et al. 2023, “How well do generative protein models generate?” outlines deeper issues with autoregressive models drifting from biologically meaningful sequences.\n- The evaluation framework is lacking biological relevance. Have the authors checked if key residues in the proteins they design are preserved or accurately sampled? For example, the chromophore in GFP, catalytic residues in enzymes, or critical binding residues in GB1 and RBP1. Including this would make the biological quality claims more compelling.\n\n\nClarity: \n\nStrengths \n- Generally the writing in the paper is very clear. Authors do a good job of describing relevant hyper parameters \n\nWeaknesses \n- Figure text is very small and impossible read without magnification. Figure 2 is especially difficult to read without zooming in a lot. \n- Figure 1B is important and difficult to compare the different plots - is there a way for the authors to overlay the traces on a single plot so that it's easier to compare by eye? \n\n\nSignificance:\n\nStrengths \n- This framework is very significant for the field of ML protein design. \n- Authors highlight theoretical and architectural benefits of this speculative decoding. It's clear that this will impact many peoples work and potentially benefit generative protein design\n\nWeaknesses \n- But they do not provide concrete biological or design throughput outcomes...Authors could be a bit more clear about how important a 32% speedup is. If their argument is that 65 hours is a LONG time to generate 20K sequences (\"Such slow sequence generation can delay high-throughput design workflows by days\"), and a 32% speedup would mean that they could generate the same number of sequences in 40 hours (still days), does that matter/what does that actually get us? What is the tradeoff between 13K sequences at original speed vs 20K sequences at faster speeds? perhaps some simulation of the quality of the \"best\" top sequences from both experiments would be helpful to see that you are actually pushing the distrusting further to the right means\n\n\nOriginality:\n\nStrengths\n- The paper brings a clever and original twist to speculative decoding by incorporating biologically grounded priors (k-mers) from MSAs. People talk about generating more sequences and having better filtering as a way of getting around the time issue, and this approach more more creative\n\nWeaknesses\n\n- The speculative decoding setup itself is largely borrowed from prior work in NLP, and while applying it to protein generation is useful, it’s not a conceptual leap. Much of the novelty comes from the domain-specific k-mer filtering rather than a fundamentally new decoding strategy.", "was_revised": true, "final_justification": "I recommend acceptance, and encourage the authors to expand on biological evaluation in future work or the camera-ready version.", "rebuttal": "Thank you for your thoughtful and detailed feedback. Your constructive comments have helped us clarify key points and strengthen the paper.\n\n> Authors should include a few additional citations to increase the quality of their scholarship and expand upon why generation from autoregressive models is challenging.\n\nWe will include the following references:\n(Hsu et al., 2022), \n(Spinner et al., 2023), \n(Holtzman et al., 2020).\nAdditionally, we will expand upon challenges in autoregressive decoding in the introduction, including pathological repetition, early termination, and OOD generation.\n\n> Have the authors checked if key residues in the proteins they design are preserved or accurately sampled?\n\nOur work does not explicitly evaluate the preservation of individual residues, as the core objective of SpecMER is to accelerate protein generation while maintaining or improving overall sequence quality. Our results demonstrate that proteins generated by SpecMER achieve higher likelihoods and pLDDT scores than those of speculative decoding, suggesting that the generated sequences align with commonly used design proxies.\n\n> What is the tradeoff between 13K sequences at original speed vs 20K sequences at faster speeds?\n\nSpecMER, being a type of speculative decoding, increases the generation speed with no sacrifice to the quality of the sequences; therefore, there is no inherent tradeoff (Leviathan et al. (2021), Sun et al. (2022)). We discuss this property and expand on its result in section 2 and 3, but are happy to discuss in more detail in the camera ready version.\nWe conducted additional experiments generating sequences using only the target model (ProGen2-M) to compare sequence likelihoods to SpecMER (c=5). The resulting sequences were scored using ProGen2-M, and we report the top 20 highest likelihood averages:\nProtein|Target Model|SpecMER (c=5)\n-|-|-\n|Bgl3|0.78 +/- 0.02|0.63 +/- 0.11  \n|GFP|0.51 +/- 0.04| 0.41 +/- 0.07\n|RBP1| 1.62 +/- 0.15| 1.72 +/- 0.30\n|GB1| 2.27 +/- 0.24| 2.20 +/- 0.31 \n|ParD3| 0.69 +/- 0.11|0.67 +/- 0.12\nWe found that sequences generated by SpecMER (c=5) had comparable or better likelihoods compared to standard autoregressive generation from the target model, a key property of speculative decoding discussed in Sections 2.1 and 3.3.\n\n> How does ProGen-XL, -S, -M compare in speed to each other and how does that compare to other autoregressive models that exist? Having a 32% speedup is great, but if other AR models are already faster then wouldn't the protein designer just want to use those? No cross-model speed benchmarks are included, and no papers are cited that provide such benchmarks either.\n\nWe tested the generation speed of ProGen2-S, M, and XL. ProGen2-S: 74.11 toks/s, ProGen2-M: 31.48 toks/s, ProGen2-XL: 9.72 toks/s. We observed a sharp decline in speed, reinforcing the difficulties of generating large amounts of sequence with autoregressive generation. We then performed experiments with Tranception and found that it generated sequences more slowly than ProGen2 and with lower likelihoods. As SpecMER is a decoding strategy for accelerating autoregressive generation, it is applicable to any AR model given a compatible vocabulary size. \n\n> Can you provide more experiments to back up the claim \"The effectiveness of SpecMER depends on the quality of input alignment (MSA); performance may degrade when informative motifs are sparse or unavailable\"?\n\nWe performed two ablation studies to test the efficacy of k-mers. We first tested SpecMER using two configurations: generate GFP, use GB1-derived k-mers to select continuations; generate GB1, use Bgl3-derived k-mers to select continuations.\nCondition          | Mean NLL ± Std | Top‑20 Avg. NLL ± Std\n-|-|-\n| GFP + GB1 k‑mers   | −2.52 ± 0.27   | −1.78 ± 0.23         |\n| GB1 + Bgl3 k‑mers  | −2.79 ± 0.10   | −2.59 ± 0.11         |\n\nThis mismatch in evolutionary signal led to lower likelihoods both on average and for top end generation (top-20) when compared to Table 2, indicating that protein specific k-mers are responsible for the increase in likelihood. \nThe second ablation tested the importance of MSA depth. We generated Bgl3 proteins using 1,000 sequences from the MSA instead of the full depth MSA (130k sequences), limiting the number of k-mers to guide generation. We observed a sharp decline in likelihoods, with the top-20 sequences having a likelihood of 1.56 +/- 0.20, whereas SpecMER (c=5) with the full depth MSA had an average of 0.63 +/- 0.11. This further supports our claim in the conclusion that performance may degrade when informative motifs are sparse or unavailable. We will include an updated section in the appendix with this ablation.\n\n> There is a lot of talk about the tradeoffs of \"cheap\" generators generating a TON of sequences and then having a good filtering set that can quickly weed out the bad sequences. Have the authors considered this?\n\nWe have not considered this filtering strategy. The goal of SpecMER is to design many high quality sequences, using k-mers as a form of filtering during generation, and we view our method as adjacent to this line of work.\nWe will address all of your formatting concerns while making revisions. Thank you again for your feedback."}, {"review_id": "Izk6F8S7em", "reviewer": "Reviewer_QzzJ", "summary": "This paper presents SpecMER, a novel framework that accelerates autoregressive protein generation by enhancing speculative decoding. The method generates multiple draft sequences and uses k-mer frequencies from a Multiple Sequence Alignment (MSA) to select the most biologically plausible candidate for verification by a larger target model.", "questions": "Novelty is a key criterion in protein generation. Does incorporating MSA-derived information make the generated proteins more similar to existing sequences?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- To the best of my knowledge, this is the first work to combine k-mer evolutionary priors with speculative decoding for protein language models.\n- On five protein targets, the method reports 24–32 % wall-time speed-ups over plain autoregressive decoding.\n- All hyperparameters are disclosed, and the authors provide open-source code, enabling full reproducibility.\n\nWeaknesses\n\nMajor\n- Because computational sequence generation already outpaces wet-lab synthesis and screening by several orders of magnitude, the claim that “slow sequence generation can delay high-throughput design workflows by days” appears overstated and insufficiently justified.\n- Improvements in negative log-likelihood and pLDDT are modest, and their biological significance is not demonstrated.\n- The study lacks ablations with random k-mer tables or shuffled MSAs (or MSA from another protein), making it hard to confirm that observed gains come from genuine evolutionary signal rather than extra sampling.\n\nMinor\n- Figure 8 title: embeddings - > embeddings\n- Line 161:  in accordance to ->  in accordance with", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "Thank you for your constructive feedback. After reviewing your comments, we have touched on some key points of clarification. \n\n> Because computational sequence generation already outpaces wet-lab synthesis and screening by several orders of magnitude, the claim that “slow sequence generation can delay high-throughput design workflows by days” appears overstated and insufficiently justified.\n\nThe quality of designs chosen for wet-lab synthesis can drastically speed up the design process by proposing better variants, shortening or skipping design cycles altogether (Biswas et al., 2021). SpecMER addresses this by generating biologically plausible sequences with higher likelihoods and pLDDT scores than standard autoregressive decoding (Section 4.3). By producing more high-quality sequences in the same compute time, SpecMER increases the likelihood of finding stronger candidates earlier, ultimately reducing the experimental effort required during wet-lab synthesis.\n\n> Improvements in negative log-likelihood and pLDDT are modest, and their biological significance is not demonstrated.\n\nThe goal of this work is to introduce a faster decoding method that can be applied to existing PLMs. We demonstrate that we can increase decoding speedups without degrading output quality, and in many instances, produce a better output. We believe this is a valuable contribution to the design process, particularly in scenarios where speed is critical (i.e. producing large libraries of candidate sequences). \nWe note that even modest increases in negative log-likelihood and pLDDT scores will result in improved structural quality (Table 3) and can result in drastic improvement in function. Our results demonstrate that SpecMER achieves these improvements while enabling faster protein generation and accelerating protein design workflows.\n\n> The study lacks ablations with random k-mer tables or shuffled MSAs (or MSA from another protein), making it hard to confirm that observed gains come from genuine evolutionary signal rather than extra sampling.\n\nWe performed two ablation studies to test the efficacy of k-mers. We first tested SpecMER using two configurations: generate conditioned on GFP, use GB1-derived k-mers to select continuations; generate conditioned on GB1, use Bgl3-derived k-mers to select continuations.\n\nCondition          | Mean NLL ± Std | Top‑20 Avg. NLL ± Std\n-|-|-\n| GFP + GB1 k‑mers   | −2.52 ± 0.27   | −1.78 ± 0.23         |\n| GB1 + Bgl3 k‑mers  | −2.79 ± 0.10   | −2.59 ± 0.11         |\n\nThis mismatch in evolutionary signal led to lower likelihoods both on average and for top end generation (top-20) when compared to Table 2, indicating that protein specific k-mers are responsible for the increase in likelihood. \nThe second ablation tested the importance of MSA depth. We generated Bgl3 proteins using 1,000 sequences from the MSA instead of the full depth MSA (130k sequences), limiting the number of k-mers to guide generation. We observed a sharp decline in likelihoods, with the top-20 sequences yielding a likelihood of 1.56 +/- 0.20, whereas SpecMER (c=5) with the full depth MSA yielded an average of 0.63 +/- 0.11. This further supports our claim in the conclusion that performance may degrade when informative motifs are sparse or unavailable. We will include an updated section in the appendix with this ablation.\n\n> Novelty is a key criterion in protein generation. Does incorporating MSA-derived information make the generated proteins more similar to existing sequences?\n\nTo evaluate novelty, we computed the Hamming distance between each generated sequence and the wild-type, as well as the pairwise Hamming distance among generated sequences (see table below). \nProtein | WT Distance (SpecMER) | WT Distance (Spec. Decoding) | Inter‑Seq (SpecMER) | Inter‑Seq (Spec. Decoding)\n-|-|-|-|-\n | GFP | 208.35 ± 5.76 | 208.49 ± 4.91 | 181.78 | 184.56 ± 27.14 |\n | RBP1 | 41.27 ± 3.48 | 42.81 ± 3.43 | 42.60 ± 3.87 | 44.88 ± 4.00 |\n | ParD3 | 75.97 ± 3.01 | 78.68 ± 2.41 | 67.39 ± 6.87 | 70.00 ± 5.41 |\n | GB1 | 44.70 ± 3.25 | 45.27 ± 3.27 | 46.47 ± 4.46 | 46.99 ± 3.86 |\n | Bgl3 | 324.88 ± 30.88 | 333.12 ± 33.96 | 261.02 ± 42.28 | 284.97 ± 40.44 |\n\nSpecMER generates sequences far from the wild-type while maintaining inter-sequence diversity, exploring novel sequence space. Additionally, Appendix D illustrates embedding plots comparing generated sequence to MSA sequences, indicating that SpecMER generates novel, high-likelihood sequences that are distinct from the wild-type yet remain grounded near homologous sequence space. We will update the appendix to include these results."}, {"review_id": "i0TPhDsALP", "reviewer": "Reviewer_icHu", "summary": "In this paper, Speculative Decoding is introduced as a novel approach to sequence generation that accelerates standard autoregressive decoding techniques. Additionally, Speculative Decoding via k-mer Guidance is introduced to improve the efficiency of standard speculative decoding.", "questions": "Could you add more specific details relevant to the fact that this work is being submitted to a computer science conference? It currently feels like it lacks a clear description of the model architecture, training protocol, and computational resources used—I'd love to see that information, perhaps included in the appendix.\n\nAlso, looking at Table 4, it seems that increasing the c parameter in SpecMER leads to a noticeable increase in variance. Could you comment on why you think that might be the case? And what might be the concerns of using high c?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nStrong empirical gains: 24–32% faster generation while producing sequences with higher likelihoods and better structural plausibility, validated across five diverse proteins.\n\nTheoretical analysis of acceptance ratios and speedups is provided, and the open-source implementation makes the method accessible and reproducible.\n\nWeaknesses:\n\nIt’s unclear what the main contribution of the paper is. From the abstract, it feels like SpecMER is the main contribution, which is built on top of existing speculative decoding methods. However, while reading the paper, if I understood correctly, speculative decoding was not used at all for protein generation. If this is the case, I think it should be emphasized more clearly in the abstract.\n\nThe following drawback follows from my previous comment: there are two metrics to evaluate model performance—speed and accuracy. However, there is a kind of inconsistency in the comparison. To show speed-up, you compare SpecMER with the target model and demonstrate improved performance, which is fair. However, in terms of accuracy, you compare only speculative decoding against SpecMER. If both of these methods are introduced by you, I don’t think this proves SpecMER’s efficiency convincingly. A comparison should also be made with existing methods—for instance, the one you used as a baseline for demonstrating the speed-up.", "was_revised": true, "final_justification": "The authors have addressed my questions and provided the exact locations in the appendix where the needed information can be found", "rebuttal": "Thank you for your constructive feedback. We have touched on some key points of clarification to address your comments.\n\n> It’s unclear what the main contribution of the paper is. From the abstract, it feels like SpecMER is the main contribution, which is built on top of existing speculative decoding methods. However, while reading the paper, if I understood correctly, speculative decoding was not used at all for protein generation. If this is the case, I think it should be emphasized more clearly in the abstract.\n\nTo clarify our contributions:\n\nWe adapt speculative decoding to protein language models and demonstrate acceleration over standard autoregressive decoding (Table 4, Baseline). Prior works in speculative decoding have focused on natural language. To the best of our knowledge, speculative decoding has not previously been applied to protein generation, so this is a key contribution of our work.\n\nWe introduce SpecMER, maintaining speedups from speculative decoding, while generating protein sequences with higher biological plausibility. \nWe discuss these contributions in lines 60-70, but we are happy to make it more clear in the abstract.\n\n> The following drawback follows from my previous comment: there are two metrics to evaluate model performance—speed and accuracy. However, there is a kind of inconsistency in the comparison. To show speed-up, you compare SpecMER with the target model and demonstrate improved performance, which is fair. However, in terms of accuracy, you compare only speculative decoding against SpecMER. If both of these methods are introduced by you, I don’t think this proves SpecMER’s efficiency convincingly. A comparison should also be made with existing methods—for instance, the one you used as a baseline for demonstrating the speed-up.\n\nSpecMER, being a type of speculative decoding, increases the generation speed with no sacrifice to the quality of the sequences (Leviathan et al. (2021), Sun et al. (2022)). We discuss this property and expand on its result in section 2 and 3, but are happy to discuss in more detail in the camera ready version.\nTo demonstrate this property, we conducted additional experiments, generating sequences using only the target model (ProGen2-M) for 4 hours to compare sequence likelihoods to SpecMER (c=5). The resulting sequences were scored using ProGen2-M, and we report the top 20 highest likelihood averages:\nProtein|Target Model|SpecMER (c=5)\n-|-|-\n|Bgl3|0.78 +/- 0.02|0.63 +/- 0.11  \n|GFP|0.51 +/- 0.04| 0.41 +/- 0.07\n|RBP1| 1.62 +/- 0.15| 1.72 +/- 0.30\n|GB1| 2.27 +/- 0.24| 2.20 +/- 0.31 \n|ParD3| 0.69 +/- 0.11|0.67 +/- 0.12\n\nWe found that protein sequences generated by SpecMER (c=5) had likelihoods comparable to, or exceeding, those from the target model, a result we discuss in Section 3.3.\n\n> Could you add more specific details relevant to the fact that this work is being submitted to a computer science conference? It currently feels like it lacks a clear description of the model architecture, training protocol, and computational resources used—I'd love to see that information, perhaps included in the appendix.\n\nProGen2 is a decoder-only Transformer architecture with four different parameter sizes: small - 151M; medium/base - 764M; large - 2.7B; xlarge - 6.4B. No model training is performed in this work; inference is run on an NVIDIA A6000 GPU. More details can be found in Nijkamp et al. (2022). We discuss some of these details in Section 4, but are happy to add more details in the appendix.\n\n> Also, looking at Table 4, it seems that increasing the c parameter in SpecMER leads to a noticeable increase in variance. Could you comment on why you think that might be the case? And what might be the concerns of using high c?\n\nThis increase in variance is due to batch generation with ProGen2. Batch generation is slightly slower than producing a single sequence continuation, and the added time to generate increases with c. Therefore, when a speculated token is rejected, decoding takes longer and the added cost of batch generation appears in the variance. Due to the non-deterministic nature of sampling, some generations have higher acceptance rates than others; therefore, occasional rises and dips in acceptance have a larger impact on tokens per second than producing a single sequence at a time. Further discussion on this trade-off space can be found in Appendix B.2."}, {"review_id": "cw4W2oa6Cn", "reviewer": "Reviewer_R8tH", "summary": "The paper introduces SpecMER: an extension of speculative decoding from natural language processing to protein sequences. In addition to using a combination of draft and target models, SpecMER uses k-mer-based filtering to select candidate sequences generated by the draft model. The method is applied to five proteins from different functional categories and consistently shows a generation quality increase over the standard speculative decoding. Depending on the number of candidate sequences generated by the draft model, SpecMER obtains a speed up w.r.t. the target model of between 25-30%. The paper presents a thorough theoretical model analysis.", "questions": "1. L146-147: As per L140, the k-mer scoring function scores $c$ sequences of length $L$. In that case, shouldn’t the list be over the length $L$ sequences, i.e., $[\\tilde{x}^{(1)}(t+L), \\tilde{x}^{(2)}(t+L), …, \\tilde{x}^{(c)}(t+L)]$? The shown list suggests to me that the function scores $c\\times L$ sequences but perhaps I’m misunderstanding.\n2. L178: Should be lowercase Q and P according to Algorithm 1.\n3. Section 4: It is not immediately clear to me what the final hyperparameter configuration actually is. In section 4.2 and in the appendix, it is stated that all combinations of hyperparameters are evaluated. Does this mean that all 36 configurations are used to generate 200 sequences each, and the single configuration with the best results are then chosen and presented in the tables?  \n4. Regarding the hyperparameter sweep, I think it would benefit the reader to present the results quantitatively instead of in many different figures which can make it difficult to track the results.\n5. SpecMER is intended to replace the standard autoregressive generation mechanism. While the speed-up is demonstrated in Table 4, I do not see any results comparing the *quality* of the generated sequences of SpecMER to the “vanilla” models. I think that experiments should be conducted and presented where the vanilla draft and target models are used generate a similar number of sequences (e.g, 36x200 sequences or in a similar amount of time) as SpecMER. Quality metrics like pLDDT and likelihoods could then be used to directly compare the model outputs. While a speed-up of ~30% is neat, it is not by itself convincing without proper comparison.\n6. Is the termination criteria of stopping when the wild-type sequence length has been reached sensible? Or could this have negative effects on sequence (and the resulting structure) quality?\n7. What would the metrics look like if the exact wild-type sequence was sampled? \n8. Often when generating sequences conditioned on a reference (e.g., wild type), the aim is to generate diverse sequences with similar properties.  I think it could be interesting to include a diversity metric to see how far from the wild type and how far from each other the sampled sequences are. It could be as simple as computing the sequence identity between the reference and samples.\n9. Three of the five proteins/domains examined are shorter than 100 amino acids. I think the inclusion of more sequence diversity would more clearly demonstrate the benefits of SpecMER.\n10. What is the complexity of SpecMER  w.r.t. sequence length and hyperparameters?\n11. How does MSA depth affect the quality of the generated sequences? Since only five proteins (three of which are very short) are examined, I’m not yet convinced that MSA depth and quality is independent.\n12. Which ESM-2 version was used to compute the embeddings for the PCA? Were the embeddings mean-pooled? Are the aligned sequences embedded, or are they converted into unaligned sequences? I think some additional details should be included in appendix C.1.\n13. How does context length affect generation quality? The context length is chosen to be approximately 10 % of the wild-type sequence length. Why is that?\n14. In the introduction, ProGen2-XL is used as an example, yet the chosen target model is ProGen2-M. What would the speed up using SpecMER be if ProGen2-XL was used instead?", "limitations": "Provided that my raised weaknesses are addressed and questions answered, I think that the authors adequately address their work's limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Biologically motivated extension of speculative decoding.\n2. Thorough theoretical analyses.\n3. Increased autoregressive generation speed. \n4. Well-written paper.\n5. Inclusion of code and data for reproducibility.\n\nWeaknesses\n\n1. Limited evaluation. The method is evaluated on five proteins, three of which are shorter than 100 amino acids. I think that a more diverse selection of proteins should be examined. I also think that the inclusion of a diversity metric (such as sequence identity or similarity to the wild type) or the inclusion of additional sequence quality metrics would further demonstrate the improved generation.\n2. Limited comparison. The quality of the generated sequences are only compared between different hyperparameter configurations of SpecMER. Since the intention is to replace standard autoregressive decoding, I think that the generation quality should be directly compared to the sequences generated by the draft and target models, potentially at different temperature values. \n3. Limited impact. Without the above analyses, the model presumably yields similar quality sequences ~30% faster. While this is certainly useful, the comprehensive hyperparameter sweep generates a total of 7200 sequences which is not insubstantial. I think that the benefits of using SpecMER compared to simply sampling from the target model at different temperatures for a comparable amount of time should be clearly demonstrated in order to convincingly show the impact.\n\nI like the paper and would like to see it published, but the above weaknesses and related questions should be addressed and adequately and convincingly answered for me to raise my score.", "was_revised": true, "final_justification": "The authors have adequately addressed my raised questions, weaknesses, and limitations. I believe that the concept of speculative decoding is highly useful for protein sequence generation. I appreciate that the authors have gone beyond simply applying the method to protein data to adapting it to biological sequences with biological priors. I believe that the methods and results presented are of interest to the community and would therefore like to see the paper featured at this year's NeurIPS.", "rebuttal": "We thank you for your feedback. Your constructive comments have helped us clarify key points and strengthen our paper.\n\n> Limited evaluation. The method is evaluated on five proteins, three of which are shorter than 100 amino acids. I think that a more diverse selection of proteins should be examined. I also think that the inclusion of a diversity metric (such as sequence identity or similarity to the wild type) or the inclusion of additional sequence quality metrics would further demonstrate the improved generation.\n\nWe have performed additional experiments on two longer proteins: CBS_HUMAN (551 amino acids) and ADRB2_HUMAN (413 amino acids). We report the average, top-20 and top-5 NLL:\n\nADRB2_HUMAN\nDecoding Method|Average NLL|Top-20 NLL|Top-5 NLL\n-|-|-|-\n|Spec. Decoding| 1.90 +/- 0.65 | 1.18 +/- 0.30 | 0.78 +/- 0.16\n|SpecMER (c=3)| 1.33 +/- 0.50 | 0.77 +/- 0.13 | 0.61 +/- 0.04\n|SpecMER (c=5)| 1.03 +/- 0.60 | 0.57 +/- 0.11 | 0.43 +/- 0.04\n\nCBS_HUMAN\nDecoding Method|Average NLL|Top-20 NLL|Top-5 NLL\n-|-|-|-\n|Spec. Decoding| 2.42 +/- 0.42 | 2.06 +/- 0.44 | 1.43 +/- 0.38\n|SpecMER (c=3)| 2.17 +/- 0.66 | 1.47 +/- 0.50 | 0.84 +/- 0.16\n|SpecMER (c=5)| 1.87 +/- 0.68 | 1.14 +/- 0.32 | 0.74 +/- 0.07\n\nWe found that SpecMER outperformed speculative decoding, further supporting our existing results. We will include these results in the main text.\n\nWe agree that diversity metrics help strengthen our findings. For each protein, we computed the average Hamming distance between generated sequences and the wild-type sequence. Additionally, we computed the average pairwise Hamming distance between generated sequences to measure inter-sequence diversity:\n\nProtein | WT Distance (SpecMER) | WT Distance (Spec. Decoding) | Inter‑Seq (SpecMER) | Inter‑Seq (Spec. Decoding)\n-|-|-|-|-\n | GFP | 208.35 ± 5.76 | 208.49 ± 4.91 | 181.78 | 184.56 ± 27.14 |\n | RBP1 | 41.27 ± 3.48 | 42.81 ± 3.43 | 42.60 ± 3.87 | 44.88 ± 4.00 |\n | ParD3 | 75.97 ± 3.01 | 78.68 ± 2.41 | 67.39 ± 6.87 | 70.00 ± 5.41 |\n | GB1 | 44.70 ± 3.25 | 45.27 ± 3.27 | 46.47 ± 4.46 | 46.99 ± 3.86 |\n | Bgl3 | 324.88 ± 30.88 | 333.12 ± 33.96 | 261.02 ± 42.28 | 284.97 ± 40.44 |\n| CBS | 378.64 +/- 140.60 | 431 +/- 106.05 | 291.84 +/- 161.60 | 457.27 +/- 40.98 |\n| ADRB2 | 263.80 +/- 120.22 | 290.18 +/- 109.01 | 270.82 +/- 93.46 | 340.28 +/- 47.66\n\nSpecMER generates sequences far away from the wild-type while maintaining inter-sequence diversity, exploring sequence space while still generating plausible protein sequences.\n\n> Limited comparison. The quality of the generated sequences are only compared between different hyperparameter configurations of SpecMER. Since the intention is to replace standard autoregressive decoding, I think that the generation quality should be directly compared to the sequences generated by the draft and target models, potentially at different temperature values. \n\nSpecMER, being a type of speculative decoding, increases the generation speed with no sacrifice to the quality of the sequences (Leviathan et al. (2021), Sun et al. (2022)). We discuss this property and expand on its result in section 2 and 3, but are happy to discuss in more detail in the camera ready version.\n\nTo demonstrate this property, we conducted additional experiments generating sequences using only the target model (ProGen2-M) for 4 hours to compare sequence likelihoods to SpecMER (c=5) using the same temperatures as the results in Table 2. The resulting sequences were scored using ProGen2-M, and we report the top 20 highest likelihood averages:\n\nProtein|Target Model|SpecMER (c=5)\n-|-|-\n|Bgl3|0.78 +/- 0.02|0.63 +/- 0.11  \n|GFP|0.51 +/- 0.04| 0.41 +/- 0.07\n|RBP1| 1.62 +/- 0.15| 1.72 +/- 0.30\n|GB1| 2.27 +/- 0.24| 2.20 +/- 0.31 \n|ParD3| 0.69 +/- 0.11|0.67 +/- 0.12\n\nWe found that protein sequences generated by SpecMER (c=5) had likelihoods comparable to, or exceeding, those from the target model, a result we discuss in Section 3.3.\n\n> Limited impact. Without the above analyses, the model presumably yields similar quality sequences ~30% faster. While this is certainly useful, the comprehensive hyperparameter sweep generates a total of 7200 sequences which is not insubstantial. I think that the benefits of using SpecMER compared to simply sampling from the target model at different temperatures for a comparable amount of time should be clearly demonstrated in order to convincingly show the impact.\n\nWe perform an extensive hyperparameter sweep, which we detail in Appendix D, to identify the best configuration per protein, a one time cost that results in accelerated generation. As noted in our previous response, we performed additional experiments comparing SpecMER (c=5) to standard autoregressive decoding, using the same temperatures of the results in Table 2. SpecMER produced sequences with likelihoods comparable to or exceeding standard autoregressive generation, supporting that generation quality is maintained while achieving significant speedups.\n\n> L146-147: As per L140, the k-mer scoring function scores c sequences of length L. In that case, shouldn’t the list be over the length L  sequences, i.e., …,\n\nYou are correct. The way it is described in the paper could be clarified. c x L indicates the batch dimension, i.e. c sequences each of length L. \n\n> Section 4: It is not immediately clear to me what the final hyperparameter configuration actually is. In section 4.2 and in the appendix, it is stated that all combinations of hyperparameters are evaluated. \n\nThe final hyperparameter configurations for each protein reported in Table 2 are:\nProtein | Temperature | Draft Tokens | k value | Candidates\n-|-|-|-|-\nBgl3| 1.0 | 5 | 3 | 5\nGFP| 0.7 | 5 | 1,3 | 5\nRBP1| 1.0 | 10 | 3 | 5\nGB1| 1.4 | 10 | 1,3,5 | 5\nParD3| 1.0 | 5 | 1,3,5 | 5\nCBS| 0.7 | 5 | 1,3,5 | 5\nADRB2| 0.7 | 5 | 1,3 | 5\n\nWe will include these values in the Appendix.\n\n> Does this mean that all 36 configurations are used to generate 200 sequences each, and the single configuration with the best results are then chosen and presented in the tables?\n\nYes.\n\n> Is the termination criteria of stopping when the wild-type sequence length has been reached sensible? Or could this have negative effects on sequence (and the resulting structure) quality?\n\nYes. Our aim is to design variants around the wild-type sequence, a common practice in protein engineering (Hayes et al., 2025). In practice, we observed no degradation in sequence quality - sequences matched or exceeded the quality of those generated with standard autoregressive decoding.\n\n> What would the metrics look like if the exact wild-type sequence was sampled?\n\nHere is the negative log-likelihood and pLDDT for each wild-type sequence:\nProtein| NLL | pLDDT\n-|-|-\nCBS | 0.75 | -\nBgl3 | 0.92 | -\nADRB2 | 1.31 | -\nParD3 | 2.11 | 0.79\nGB1 | 2.52 | 0.82\nRBP1 | 2.63 | 0.83\nGFP | 2.93 | 0.42\n\n > What is the complexity of SpecMER w.r.t. sequence length and hyperparameters?\n\nSpecMER has the same computational complexity as speculative decoding, O(L^2), with respect to sequence length L. It does not introduce additional overhead beyond speculative decoding, whose accelerated generation is achieved without increasing asymptotic complexity relative to standard autoregressive decoding (Leviathan et al., 2021). The additional k-mer scoring in SpecMER is constant time and does not affect overall complexity. We will make this clear in the main text.\n\n> How does MSA depth affect the quality of the generated sequences? Since only five proteins (three of which are very short) are examined, I’m not yet convinced that MSA depth and quality is independent.\n\nWe ran additional experiments to test the importance of MSA depth for Bgl3, forming k-mers using 1,000 sequences from the MSA instead of the full depth MSA (130k sequences), limiting the guidance signal from the MSA. We observed a sharp decline in likelihoods, with the top-20 sequences yielding a likelihood of 1.56 +/- 0.20, whereas SpecMER (c=5) with the full depth MSA yielded an average of 0.63 +/- 0.11. This supports our claim in the conclusion that performance may degrade when informative motifs are sparse or unavailable. We will include an updated section in the appendix with this ablation.\n\n> Which ESM-2 version was used to compute the embeddings for the PCA? Were the embeddings mean-pooled? Are the aligned sequences embedded, or are they converted into unaligned sequences? I think some additional details should be included in appendix C.1.\n\nWe utilized ESM-2 8M with mean-pooled embeddings and unaligned sequences in accordance with standard practice. We will update Appendix C.1 to reflect this.\n\n> How does context length affect generation quality? The context length is chosen to be approximately 10 % of the wild-type sequence length. Why is that?\n\nWe chose a context length of ~10% based on empirical observation: shorter contexts often led to pathological repetition of amino acids (Hsu et al., 2022), while longer contexts restricted exploration of sequence space. A 10% context provided a balance, enabling high-quality, diverse sequence generation.\n\n> In the introduction, ProGen2-XL is used as an example, yet the chosen target model is ProGen2-M. What would the speed up using SpecMER be if ProGen2-XL was used instead?\n\nWe ran additional experiments with ProGen2-XL as the target model and ProGen2-M as the draft. The tokens per second jump from 7.03 to 9.72 with SpecMER (c=3), a 38% increase in speed.\n\nWe will address your formatting concerns while making revisions. Thank you again for your feedback."}]

Author rebuttal:

Meta-review: This submission introduces SpecMER, a speculative decoding framework for accelerating autoregressive protein generation, integrating biological priors (k-mer motifs extracted from MSAs) into the decoding process, guiding a lightweight draft model to propose biologically plausible sequences that a larger target model then verifies. This balances speed (via speculative decoding) and sequence quality (via MSA-derived k-mer filtering), addressing the latency bottleneck of standard autoregressive models in high-throughput protein design. 

The strengths noted by the reviewers include its first domain adaptation of speculative decoding to protein generation, strong empirical performance (24–32% speedups across five initial proteins, extended to two longer proteins >400 amino acids during rebuttal), and extensive validation metrics. The method’s biological grounding by using MSA k-mers to avoid implausible sequences was also emphasized as a key advantage over generic speculative decoding. During the rebuttal period, the authors also addressed reviewer concerns thoroughly. Although limitations such as the need for more functional validation (e.g., key residue preservation) and broader testing on diverse protein families remain, all the reviewers are in favor of this submission and I also agree. We strongly encourage the authors to incorporate the additional results and valuable discussions in the revised manuscript.
Decision: Accept (spotlight)