Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Authors: Xiang Zhang
Jiaqi Wei
Zijie Qiu
Sheng Xu
Nanqing Dong
ZhiQiang Gao
Siqi Sun
Venue: ICML 2025 poster (2025)
Keywords: De Novo
protein language modelling
sequence modelling
Primary area: applications->everything_else
TL;DR: We introduce a curriculum learning approach and a self-refining inference module to improve non-autoregressive peptide sequencing, achieving state-of-the-art accuracy across nine species.

Abstract: Peptide sequencing—the process of identifying amino acid sequences from mass spectrometry data—is a fundamental task in proteomics. Non-Autoregressive Transformers (NATs) have proven highly effective for this task, outperforming traditional methods. Unlike autoregressive models, which generate tokens sequentially, NATs predict all positions simultaneously, leveraging bidirectional context through unmasked self-attention.
However, existing NAT approaches often rely on Connectionist Temporal Classification (CTC) loss, which presents significant optimization challenges due to CTC's complexity and increases the risk of training failures. To address these issues, we propose an improved non-autoregressive peptide sequencing model that incorporates a structured protein sequence curriculum learning strategy. This approach adjusts protein's learning difficulty based on the model’s estimated protein generational capabilities through a sampling process, progressively learning peptide generation from simple to complex sequences. 
Additionally, we introduce a self-refining inference-time module that iteratively enhances predictions using learned NAT token embeddings, improving sequence accuracy at a fine-grained level. Our curriculum learning strategy reduces NAT training failures frequency by more than 90% based on sampled training over various data distributions. Evaluations on nine benchmark species demonstrate that our approach outperforms all previous methods across multiple metrics and species. Model and source code are available at https://github.com/BEAM-Labs/denovo.

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Xiang Zhang 11 Jiaqi Wei 34 Zijie Qiu 31 Sheng Xu 31
Nanqing Dong Zhiqiang Gao Siqi Sun 13
Abstract
Aj
Ag
Peptide sequencing-ther process of identifying amino acid sequences from mass spectrometry data-is fundamental task in proteomies. Non- Autoregressive Transformers (NATs) have proven highly effective for this task, outperforning tra- ditional methods. Unlike autoregressive models, which generate tokens sequentially, NATs predict all positions simultaneously, leveraging bidirec- tional context through unmasked self-attention. However, existing NAT approaches often rely Connectionist Temporal Classification (CTC) loss, which presents significant optimization chal- lenges due to CTC's complexity and increases the risk of training failures. To address these issues, we propose an improved non-autoreerressvve pep- tide sequencing model that incorporates a struc- tured protein sequence curriculum learning strat- egy. This approach adjusts protein's 's learning diffi- culty based on the model's estimated protein gen- erational capabilities through sampling process, progressively learning peptide generation from simple to complex sequences. Additionally, we introduce a self-refining inference-tine module that iteratively enhances predictions using learned NAT token embeddings, improving sequence ac- curacy fine- fineggrineed level. Our curriculum learning strategy reduces NAT training failures frequency by more than 90% based on sampled training over various data distributions. Evalua- tions on nine benchmark species demonstrate that our approach outperforms previous methods across multiple metrics and species. Model and source code are available at Github.
Iterative Musking
Nor- sutoegreeeeiee
Non senn-auttagressive
Peptide Decoder
Peptide Decoder
Spectro
Spectrum Memory
M
CTC Path
A
Spectrum
Spectrum
Encoder
Encoder
(a) Vanilla poptide de novo architect
(b) NAT architect with structured curriculum learning
Figure Comparison between (a) Vanilla NAT and (b) our pro- posed NAT peptide sequencing architectures, which integrate cur- riculum learning and self-refining module.
1. Introduction
Peptide sequencing via tandem mass spectrometry plays a pivotal role in proteomics research, with significant impli- cations for fundamental and applied studies in chemistry, biology, medicine, and pharmacology (Aebersold & Mann, 2003; Ng al., 2023). The principal aim of peptide sequenc- ing, as illustrated in Figure 2, is to deduce the amino acid sequences of segmented short protein sequences from mass spectra derived from specific biological samples. Database- searching- based sequencing and de novo peptide sequencing are the two most widely used methods for identifying pep- tide sequences (Chen et al., 2020). Traditional database search methods have inherent limitations, such as the inabil- ity to identify peptide sequences absent from the database and the dramatic increase in computational costs and pro- cessing times as the database expands. In contrast, de (from) novo (scratch) peptide sequencing directly deduces peptide sequences from the spectra, overcoming the limitations of the static search space in database-based algorithms (Muth et al., 2018).
'Equal contribution work done while interning at Fudan Univer- sity. 'Fudan University UUniversity of British Columbia 'Shanghai Artificial Intelligence Laboratory *Zhejiang University. Corre- spondence to: Xiang Zhang <xzhang23 @aaten222aabbaaccccaa ualberta.ca>> Siqi Sun <siqisun@ fudan.cdu... edu..............
Deep learning based models (LeCun et al., 2015; Liu et al., 2022; Zhang et al., 2024) have revolutionized the scien- tific field, including de novo sequencing. Notably, non- autoregressive Transformer (Gu et al., 2017; Xiao et al., 2023) models have demonstrated superior performance
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
among all deep learning-based methods in protein sequence predictions (Lin al., 2023; Hayes al., 2024; Zhang et al., 2025). Unlike autoregressive models, which rely on "next token prediction" for generation, non-autoregreessive models compute the token probabilities at each position simultaneously. This parallel prediction approach enables bidirectional information flow during generation through self-attention mechanism. Specifically, each position ac- cesses information from all surrounding positions when generating its token, as opposed to relying solely the pre- ceding ones in an autoregressive model. Such an approach closely aligns with the nature of protein formation and has proven to be more accurate and efficient in de novo se- quencing as well as some of the other sio-sequence-related tasks (Eloff et al., 2023: Zhang et al., 2025).
Protein Samples
Mass Spectromatry
M5/ /MS Spectra
De Novo Poptide
SNCURLLARR
CRSSS
Protein Identification Workflow
Figure 2. Overview of the protein identification workflow. Protein samples are digested into peptides, which are then analyzed using mass spectrometry to generate MS/MS spectra. These spectra are subsequently used in de novo peptide sequencing to determine the peptide sequences.
Given that straightforward eaken-by-token optimization with cross-entropy loss in Nut-Autoreereessvve (NAT) modeling often leads to poor global sequence-level connectivity (Gu et al., 2017) (explained in later section), many NAT-based models employ alternative loss objectives such as Connec- tionist Temporal Classification (CTC) loss (Graves et al., 2006; Graves& & Jaitly, 2014) and Directed Acyelic Trans- former (DAT) loss (Huang et al., 2022b). The previous NAT-based de novo sequencing model (Figure Ia) also uti- lized the CTC objective and has achieved state-of-the-ee ahe- performance (by 10% improvement compared to AT mod- els). However, the naive CTC objective is far from flawless. involves complex reduction rules (detailed in later section) and creates a vast search space during optimization, leading to an unstable learning curve and slow convergence during training (Figure 4). This optimization challenge adverscly affects overall performance.
predicted sequences as input, models can iteratively refine their predictions, improving generation quality. However, traditional NAT models (Zhang et al., 2025) struggle with this approach, they are trained without conditioning on input sequences beyond positional encoding. Our method- ology overcomes this limitation by incorporating input se- quence information into the curriculum learning process. By leveraging learned token embeddings, we seamlessly integrate iterative refinement during inference, significantly enhancing the accuracy of protein sequence generation.
Experiments and case studies demonstrate that our train- ing framework effectively mitigates three common failure modes in NAT protein models: loss explosion, extreme over- fitting, and unstable loss convergence. Our approach ensures consistently smooth training curve across diverse datasets, including both well-distribtted and poorly distributed sam- ples, achieving over 90% reduction in training failures.
In this work, we enhance standard CTC training and in- troduce the first protein curriculum learning model for im- proved sequence prediction. Our method exposes the NAT model to proteins of varying difficulty levels, guiding it to progressively learn from simple to complex sequences, mir- roring the human learning process by starting with easier prediction targets and gradually increasing difficulty.
Experiments conducted on the widely recognized 9-species- V1 (Tran et al., 2017) and 9-species- V2 (Yilmaz et al., 2024) benchmark datasets demonstrate the superiority of the pro- posed RetineNovo over all previous models across various evaluation metrics, establishing a new state-of-the field. Additionally, we investigate RefineNovo's 's capacity to differentiate between amino acids with similar masses, showcasing its adeptness in discerning subtle distinctions among challenging amino acids. The overall performance again highlights the exceptional capabilities of RefineNovo, positioning it as a promising and innovative tool in the field of proteomics.
the
Specifically, we integrate curriculum learning with a CTC- based training objective, sampling from the model's 's own CTC output path during training to determine learning tar- gets and adjust difficulty based on its current performance (Figure 1b). By exposing the model to simpler targets first, the curriculum learning approach enables a more gradual and smooth learning process, eliminating the need for the model to grapple with complex protein generation rules from the very beginning.
2. Related Work
Autoregressive v.s. Non-Autoregressive. Autoregressive generation with Transformer models, where tokens are pre- dicted sequentially, has demonstrated outstanding perfor- mance across various scenarios (Xiao et al., 2023). How- ever, this approach can be time-consuming, especially for
Furthermore, iterative refinement has proven highly effec- tive in various protein modeling tasks, including protein language modeling and structure prediction (Jumper al., 2021; Abramson et al., 2024). By incorporating previously
2

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
long sequences. To address this limitation, Gu et al. (2017) introduced the first NAT model for neural machine trans- lation, which facilitates parallel decoding and significantly accelerates the generation process. This substantial increase in inference speed has attracted much attention to NAT methods, leading to impressive progress through techniques such as knowledge distillation (Zhou& & Keung, 2020; Ding et al., b0 Shao et al., 2022), innovative learning strate- gies (Qian al., 2020; Ding al., 2021a; Zhu et al., 2022), iterative methods (Stern et al., 2019; Guo et al., 2020; Savi- nov et al., 2021; Huang et al., 2022c), latent variable-based techniques (Ma et al., 2019; Shu et al., 2020; Song et al., 2021; Bao et al., 2021), enhancement techniques ( (Wang et al., 2019; Yin et al., 2023; Guo et al., 2019; Ding et al.,
decoding in de novo peptide sequencing. In this study, we aim to build on this potential by refining the NAT-based method to further enhance de novo sequencing performance. Our focus is on increasing the reliability of the decoding process, which we believe will lead to advancements in the field.
3. Method
3.1. Problem Formulation
We first formally define the task of de novo peptide se- quencing. Our goal is to translate given mass spec- trometry spectrum into the sequence of amino acids it encodes (Figure 2). The spectrum consists of a set of paired mass-to-charge ratio values (x-axis) with corre- sponding peak intensity values (y-axis), denoted as = { (onz())p'))) (mz()))p())) ...., (*), ( (())). Addition- ally, two more pieces of information about the target peptide are provided by the mass spectrometer: the overall peptide mass (precursor mass) m and the entire peptide charge z. Using all these inputs, we aim to predict the correct amino acid sequence A = (a1, 02, O,.).........
2020; Liu al., 2022: Huang al., 2022a), and different criterion (Saharia et al., 2020; Shao et al., 2020; Ghazvinine- jad et al., 2020), all of which have further improved the performance of NAT models.
Building on these advancements, NAT models have also shown substantial benefits in biology-related sequence gen- eration tasks, such as protein generative modeling (Lin et al., 2023; Hayes et al., 2024) and de novo peptide se- quencing (Zhang al., 2025), demonstrating the powerful generative ability brought by the bi-directional scheme.
3.2. Non-autoregressive Transformer BackBone
Deep Bearning-Based De Novo Peptide Sequencing. With the advent of deep learning (LeCun et al., 2015; Gao et al., 2023; Jin et al., 2023), the performance of de novo pep- tide sequencing has markedly improved. A notable pioneer in this domain is DeepNovo (Tran et al., 2017), the first deep dearning-based method leveraging CNNs and LSTMs to model spectra and peptide sequences. Following this innovation, numerous deep soaming--bsed approaches have emerged for de novo peptide sequencing (Zhou et al., 2017; Karunratanakul et al., 2019; Yang et al., 2019; Liu et al., 2023; Mao et al., 2023). However, these methods often involve complex modeling techniques, including the com- bination of multiple neural networks and intricate post- processing steps (Yilmaz et al., 2022). Casanovo (Yilmaz et al., 2022; 2024) and its derivatives (e.g., AdaNovo (Xia et al., 2024a). HelixNovo (Yang et al., 2024), InstaNovo (Eloff et al., 2023), SearchNovo (Xia et al., 2024c)), and Nonk (Qiu et al., 2025) introduced the Transformer architecture to directly model the de novo peptide sequenc- ing problem analogously to machine translation in natural language processing. Among these, ContraNovo (Jin et al., 2024) implemented a contrastive learning strategy and inte- grated additional amino acid mass information, thereby en- hancing sequencing accuracy. Recently, PrimeNovo (Zhang et al., 2025) presented the first non non-autoreeressive Trans- former architecture, achieving sfate-of-the -hr results in this task.
Our model utilizes an encoder-decoder transformer archi- tecture, similar to the previous NAT-based de novo model (Zhang et al., 2025). To incorporate the curriculum learning strategy, we make several architectural changes, outlined in detail below.
The encoder compresses the spectrum imput I into meaningful embedding E in latent space. We treat the values in as a sequence and encode each float value of mz(") using a sinusoidal encoding as follows::
Spectrum Encoder.
c/ (mz) = (iin(mz)/( (
(mz) Jano (me))mi 2x
(mz)aain 3 )), for i S
(1)
coo(((mm (oos(m))/((
[me)max
(mmz)ant
# }
otherwise
2m
where d is the hidden dimension of our model. Similarly, the peak intensity p(9) is encoded with the same function into dimensions and then added e(mz) each position. The encoded spectrum embedding E (e%, eg. ..... .e() is then fed into the Transformer Encoder with layers, where the jth encoder layer updates the last layer embedding E as follows:
E = SelfAttention((e) ef ... e
(2)
The output from the last layer, E(m), is used as the feature representation of the input spectrum and will be used in the decoding process.
PrimeNovo highlights the potential of non non-autoreeressive
Peptide Decoder. Unlike autoregressive decoders, which use self-attention for next token prediction, NAT decoders
3

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
(A) Training Phase
(B) Inference Phase
Path Sampling Module
Candidate Paths
Rerank Paths
Spectrum Encoder
Spectrum Encoder
Path
Path 2
Path 3
Non autoagreesive Peptide Decoder
Path 3
Path
Non-autoagreesiee
Peptide Decoder
Spectrum Memory
Probablyy
Path n
Path
Spectruu
Matrix
Probably
Argmax Path Sampling
Matrix
Curriculum Learning Module
Heavy Mask
Moderate Mask
Non aunaaaraageeeevve
Mi
Ao
Peptide Decoder
A
Non autoagr eesive
Peptide Decoder
Aa
Ag
A
Ac
Intermediate
Advanced
Seff-refine Itertively
Aa
Self-refine Ieertivaly
Train with CTC Loss
Curriculum Training Phases
Peak Token
A
Amino Acid Token
Blank Token
Figure The architecture of RefineNovo. (A) The training phase of RevineNovo begins with Spectrum Encoder that processes the input spectra. Following encoding, the Son-eeetreeressvvve Peptide Decoder leverages the encoded spectra and blank tokens generate probability matrix. To predict potential sequences, CTC path sampling identifies all candidate paths, which are then re-ranked for optimal selection. The model employs Masking strategy, progressively masking tokens support curriculum training approach that advances from basic to advanced stages. Further refining the learning process, the model iteratively adjusts based on previously predicted sequenees and is trained using the CTC loss function.
(B) In the inference phase, the model operates similarly the training phase, with the primary difference being the use of Argmax CTC path sampling. To maintain accuracy, the model continues to iteratively refine its predictions. ensuring precise peptide sequence generation.
generate token probabilities for each position indepen- dently, using a self-attention-basee decoder module. Un- like previous NAT designs that only take positional en- codings input,
I) consecutive identical tokens are merged; 2) placeholder token is removed; 3) identical tokens adjacent are not merged. For example, the sequence AABCeC is reduced ABCC.
model incorporates calculated quences as input for curriculum learning purpose. To accommodate this, we add an embedding layer, denoted as hy EmbeeddingLayer(()). where y, is the input to- ken at the ith position. The encoded input is then passed through both self-attention and cross-attention layers with the spectrum features ET Finally, the output of the last layer, h(),, is mapped the probability of tokens P.(. Iz) = softmax(W77 ( for decoding position s.
During training, instead of maximizing the generation prob- ability of the true token at position i, CTC maximizes the probability of all generation paths = V2--2...... such that [(y) = A. For example, with the target sequence ATC and generational length of 5, decoding paths such as AATTC and AATTC will be assigned higher probabilities as they be reduced to the true sequence. Conversely, the path A ATTC will be discouraged since does not reduce to the target sequence.
CTC Training. Using naive cross-entropy loss parallel prediction models can lead to the multi-modal problem! due to the lack of token dependencies (Gu et al., 2017). To address this, we use the Connectionist Temporal Classifica- tion (CTC) loss as our optimization objective. In CTC, we first define maximum generational length T, which can be reduced to the target length using the following rules I'(-):
Therefore, the objective for CTC is to maximize the total probability P(A[T) of all valid paths (those that map to the target sequence A), or equivalently, to minimize the negative logarithm of this probability:
Lore = log P(A =- -log Σ
P
For example, translation of aau revoir" to english might result in sentence such "good you" or "see bye".
y=
The term P[y[Z) represents the probability single align-

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
ment path y. If this path probability is computed as product of individual token probabilities (e-g.. P(y|Z)= Iyvey ((y for tokens y; in path y), then its logarithm, log P(y[Z). is indeed equal to the sum of the individual log token probabilities, Ey.ey log (Z) This is the prop- erty that 'the log of the product of all token probabilities equals the sum of the log of each token's probability.' It's important to note that this property applies to the calculation of the log- probaabbbiltyy of single path, whereas the CTC loss involves summing the actual probabilities P(y[Z) of all valid paths before taking the logarithm. The detailed explanation of CTC loss calculation is in Appendix Section C.
where y(A,y) represents the selected unmasked tokens. By "leaking" some ground-truth tokens, prediction becomes easier, and the search space is effectively reduced as:
SPACE(N AT) 7 TYPE(Loss) Pratio
(4)
In this case. SPACE(NAT)) reaches zero when no masking is applied (i.e. Peatio = 0. the true label is fully provided) and is maximized when all positions are masked (i.e. Naive NAT model, Pratio = 1). The TYPE(cte) yield much larger search space than TYPESSTOSS entropy) since each target se- quence requires searching through On'T) pre-ete-reduction paths, rather than single path in cross-entropy loss.
PMC Unit. Following the work of Zhang et al. (Zhang al., 2025), we apply a precise mass control post-decoding unit Since the precursor mass m provided as input is strict constraint for the generated sequence y, we use dynamic programming solver to find the path that satisfies this con- straint. Specifically, this is modeled as a knapsack problem where the total generation mass m is the maximum capacity of the knapsack. We select items (amino acid tokens) from each position fill the bag, with each token having a value (predicted log probability) and a weight (molecular mass). The most valuable (probable) path that meets the mass con- straint is determined through a 2D dynamic programming table. The detailed decoding algorithm can be referred to in Appendix Section D.
However, while this strategy is straightforward for non-CTC- based learning objectives, it becomes significantly more complex when CTC reduction rules are involved. In CTC, position t in the generated sequence does not necessarily correspond a in the true label as the CTC path is often much longer before reduction. This makes it impractical to create our conditional input p(A, y). To address this, we propose CTC-based curriculum strategy.
First, we perform a forward pass with an empty decoder input to obtain the probability distribution for each position E (1, (1..... -T). We then calculate the sequence probability for all valid CTC decoding paths y such that I'(y) = -A. We re-rank all such paths y according to their total sequence probability and choose the most likely path y' our to- be- input y' (Figure 3 Yellow). We apply masking to obtain the input sequence y(A,y') which is fed into the decoder for conditioned generation and backpropagation update.
3.3. Protein Curriculum Learning with CTC-based NAT
Parallel generative models face more complex optimiza- tion landscapes than autoregressive models due to their larger search space. In autoregressive models, predic- tions at time step are conditioned on previous tokens, i.e., 1,, I). effectively constraining the search space based on prior information T1t-1 and simplifying op- timization. In contrast, NAT models predict P(x,) indepen- dently, without conditioning on previous outputs, leading to a significantly larger search space that makes convergence to the global optimum more challenging. Additionally, the implicit reduction rule in CTC further complicates learning, making target path discovery more difficult than next-token prediction in autoregressive models. As a result, we fre- quently observe instability in loss convergence and high training failure rates in NAT models (Figure 4).
This approach exposes the model to partial information from one valid decoding path, allowing it infer the structure of the true CTC path while learning to predict masked tokens and generalize to unobserved paths. Intuitively, this partial sequence exposure lowers learning difficulty by strategically "leaking" information, guiding the model to internalize the alignment patterns inherent in CTC decoding.
Difficulty Annealing. Naively applying masking (e.g. Pratio = 0.5) reduce learning diffieulty can result in inef- fective learning due to excessive information leakage. To address this, we adopt difficulty annealing approach to adaptively reduce the masking ratio over the course of train- ing based on model's per-epoch performance. Specifically, during the forward pass in training time, we calculate the model's prediction accuracy acc(A, y"rgmus) based on CTC- argmax decoding. The masking ratio is then calculated as Pratio = ax( acc y aargmax)) ). Consequently, most tokens will be unmasked at the beginning of training, with more tokens being masked as the model's predictions become more accurate over time.
To address these challenges, we introduce a curriculum learning strategy tailored for CTC-based protein prediction. Unlike naive NAT models, which learn to predict the entire sequence from scratch, our model selectively masks parts of the target sequence with a special masking token. This modifies the learning objective from independent sequence prediction to conditioned probability estimation:
f P(Alp(A,)),
We present the curriculum learning method in Algorithm 1.
5

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Algorithm 1 Curriculum Learning with CTC-based NAT
CTC path y. Specifically, at the i-th iteration, the previously decoded pseudo path using argmax, y(i-1) is used as input for the decoder again for the conditional generation:
Require: Training batch batch, decoder decoder, peek
factor a
Ensure: Masked oracle sequence glat.prey
1: if TTriiiing mode then
2: # Forward pass without gradient updates
y( argmax P(-IZ, Embeddinglaayer(y
(5)
3: word_insouut, tgt.tokens
sorward_step( (orward_step(batcch) batch)
4: pred_tokent f- arg muxx(word ins)
This process is iterated for N times, with the final output P(-IZy(NNI)) sent to the PMC unit for final decoding. We perform whole-sequence recyeling instead of decoding partial tokens in each pass because the PMC unit requires probability distribution for all positions from a single pass to avoid distributional shifts. The algorithm detail is in Appendix Section B.
5: # Compute best alignment paths via CTC
6: best_aligns
CTC_aligní (word_wong .out, tgk.,t
7: oracle_pos t-
8: oracle t gather(tgt tokens, oracle pos)
9:
masked
mask( (oracle, blank where needed)
10: # Compute model accuracy via CTC-argmax decod-
4. Experiments
ing same num oracle_masked
11:
[prred tokens
4.1. Datasets
To conduct our study, utilized three distinct datasets
12:
seq lens +- pred tokens
in alignment with previous research for fair comparisons: MassIVE-KB (Wang et al., 2018), 9-species-V V1 (Tran et al., 2017), and 9-species- V2 (Yilmaz et al., 2024). MassIVE- KB is a comprehensive collection of human proteomic data, including a high-quality subset of more than 30 million peptide spectrum matches (PSMs). We then evaluated our model on the 9-species benchmark test set, which has been used in all previous de novo sequencing work and serves as comprehensive evaluation dataset with diverse spectrum- peptide data distributions across nine species. The revised version, 9-specics-V2 V2, includes more data samples for each species and enforces stricter annotation process for higher data quality.
13:
acc - 1 same num
seq lens
14:
# Compute adaptive masking probability based on
performance
15:
Pratio a acc)
16:
# Apply curriculum masking
17:
mask +- random() < Pratio
18:
glat prreppree t mask(oracle masked, mask)
19:
end if
Return glat. prev
This algorithm outlines the key steps for computing align- ment paths, selecting oracle tokens, and applying adaptive masking for training.
Evaluation Metrics. To evaluate model's prediction accuracy, we used metrics at both amino acid and peptide levels as established by previous research. At the amino acid level, we calculated the count of correctly predicted amino acids, N using criteria based on mass deviations (Yil- maz al., 2022). Amino acid Precision was determined by No /manch/ /Npa At the peptide level, we considered pep- tide accurately predicted if all its constituent amino acids matched their ground truth counterparts. We denote NPT as the number of peptides with all amino acids correctly matched in given dataset. Peptide recall was then defined where NP9 represents the total number of
The python implementation well pseudo-code of this learning can be seen in Appendix Sec. A and Algorithm 1.
3.4. CTC-Currieulmm-based Protein Iterative Refinement
Our trained model includes curriculum token embedding layer designed to encode conditional inputs, p(A, How- ever, during inference, this layer is largely unused since there is no ground-truth label, and all conditional inputs are mask-tokens. To maximize its utility, we adopt a multi- pass forward approach to iteratively refine the generated sequence. Notably, prior studies have demonstrated that iterative refinement enhances prediction accuracy in protein- related tasks.
as N peptides the dataset. These metrics are essential for quan- tifying the performance of our predictive algorithms in mass spectrometry data analysis.
Baselines. To rigorously assess our model's performance, we conducted a comparative analysis against several state- of-the-art methods. The baselines are categorized by archi- tecture: Peaks (Ma et al., 2003), representing the Database (DB) approach. The Autoregressive (AR) methods in-
To this end, we propose a whole-sequence refinement infer- ence scheme using the previously trained CTC embedding layer, which can encode meaningful information from any
6

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Metries
Category
Methods
Mouse
Human
Yeast
M. mazei
Honeybee
Tomato
Rice bean Bacillus
C. bacteria
Average
DB
Peaks Novo (Ma ct al., 2003)
0.600
0.639
0.748
0.673
0.633
0.728
0.644
0.719
0.586
0.663
Deep. (Tran et al., 2017)
0.623
0.610
0.750
0.694
0.630
0.731
0.679
0.742
0.602
0.673
Point. (Qiao al., 2021)
0.626
0.606
0.779
0.712
0.644
0.733
0.730
0.768
0.589
0.687
Casa. (Yilmaz et al., 2022)
0.689
0,586
0.684
0.679
0.629
0.721
0.668
0.749
0,603
0.667
AA Precision
AR
Ada. (Xia el al., 2024b)
0.646
0.618
0.793
0.728
0.650
0.740
0.719
0.739
0.642
0.697
Casa. (Yilmaz et al., 2024)
0.760
0.676
0.752
0.755
0.706
0.785
0.748
0.790
0.681
0.739
Prime. (Zhang et al., 2025)
0.784
0.729
0.802
0.801
0.763
0.815
0.822
0.846
0.734
0.788
NAR
Ours
0.800
0.730
0,818
0.819
0.780
0.825
0.835
0.854
0.742
0.800
DB
Peaks Novo (Ma et al., 2003)
0.197
0.277
0.428
0.356
0.287
0,403
0.362
0.387
0.203
0.322
Deep. (Tran al., 2017)
0.286
0.293
0.462
0.422
0.330
0.454
0.436
0.449
0.253
0.376
Point. (Qiao al., 2021)
0.355
0.351
0.534
0.478
0.396
0.513
0.511
0.518
0.298
0.439
Casa, (Yilmaz et al,, 2022)
0.426
0.341
0.490
0.478
0.406
0,521
0.506
0.537
0.330
0.448
Peptide Recall
AR
Ada. (Xia et al., 2024b)
0.467
0.373
0.593
0.496
0.431
0.530
0.546
0.528
0,372
0.481
Casa. (Yilmaz ct al., 2024)
0.483
0.446
0.599
0.557
0.493
0.618
0.589
0.622
0.446
0.539
Prime. (Zhang et al.. 2025)
0.567
0.574
0.697
0,650
0.603
0.697
0.702
0.721
0,531
0.638
NAR
Ours
0.583
0.581
0.709
0.667
0.616
0.705
0.720
0.736
0.549
0.653
Table Comparison the performance on the 9-species-V benchmark datasets. The models are categorized by their architecture type: DB represents Database, AR stands for Autoregressive Generation, and NAR denotes Non-Autoregressive Generation. The bold font indicates the best performance.
clude DeepNovo (Tran et al., 2017), which integrates CNN and LSTM architectures; PointNovo (Qiao et al., 2021), which processes mass spectrometry data across varying resolutions without increased computational com- plexity; Casanovo (Yilmaz et al., 2022), a transformer- based model; Casanovo V2 (Yilmaz et al., 2024), which en- hances sequencing accuracy by incorporating beam search; and AdaNovo (Xia et al.. 2024b), which leverages Con- ditional Mutual Information to achieve better AR perfor- mance. Lastly, PrimeNovo (Zhang et al., 2025), the first Non-Autoreegressive Generation (NAR) method, sets a new benchmark for peptide sequencing precision.
bedding dimensions, attention heads count, and learning rate scheduling strategy, were consistently applied as de- fault settings for all subsequent downstream experiments unless modifications were necessary. More information on its implementation details can be referred to the code in our provided link.
4.2. Results
Performance on 9-species-VV V1 Benchmark Dataset. In this study, we assess the performance of various models on the 9-species-VI benchmark dataset. The experimental results, summarized in Table 1, indicate that our proposed model demonstrates superior performance in amino acid precision across most tested species, achieving an average precision of 0.800. Furthermore, our model excels in peptide recall, with an average recall rate of 0.653.
Model Details. We starts by transforming all inputs, such as peaks, precursors, peptides, and amino acids, into 400- dimensional embedding space, which serves as the foun- dation for further processing within the model. Built upon this embedding space, the model's architecture features a 9-layer Transformer, where each layer contains eight atten- tion heads. The feedforward network across all attention layers has dimension of 1024. To optimize the model's performance, spectra were processed with batch size of 1600 during the training phase. An initial learning rate of 4e-4 was implemented, which was gradually increased to the target peak within the first epoch and then followed a cosine decay schedule to ensure a controlled reduction over time. Model parameters were optimized using the AdamW optimizer (Kingma & Ba, 2014). This strategic learning rate adjustment was crucial during the 30 epochs of train- ing, which were conducted using eight A100 GPUs. The model's s hyperparameters, such as the number of layers, em-
Notably, our model surpasses all other methods in both amino acid precision and peptide recall for 8 out of the 9 species tested. In these species, our model achieves the highest amino acid precision, ranging from 0.780 to 0.854, and the highest peptide recall, ranging from 0.549 to 0.736. This performance underscores the robustness and generaliz- ability of our approach across a diverse set of species. For the human species, our NAR model achieves an amino acid precision of 0.730 and peptide recall of 0.581. Although these results are slightly lower than those of the state-of-the- art AR model ContraNovo, they significantly narrow the gap between NAR and AR models.
Overall, the superior performance of our model demon-
7

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Case Study
Case A
Same value Bettrr generallzation
Case
Easker opáriaziinn Lower makk of traiiig fallurs
Better performance
COF1
Out
Toring Steplpech
Trinig Stepppech
Step
Figure 4. A case study showing three types of training failures that frequently happened during the training of NAT peptide sequencing models. RefineNovo can successfully mitigate training problems in traditional NAT models. Note that the plots show training different datasets rather than one single training for RefineNovo, and validation does not involve looking at true tokens. The training data logging was done and supported by Neptune.AI.
strates its capability to effectively identify peptides in com- plex mass spectrometry data, thereby highlighting the po- tential of our approach to advance proteomies research.
When both models ensure smooth loss convergence (Case A), PrimeNovo often suffers from overfitting issues with oscillating validation accuracy, suggesting the model learns to memorize rather than generalize. RefineNovo's 's gradual training strategy leads to much smoother validation accuracy. Moreover, Refine often exhibits higher learning speed and generalization outcomes (Case C), demonstrating the superiority of the proposed approach in training.
Performance on 9-species-V V2 Benchmark Dataset. We comprehensively evaluate our NAT model's performance against state-of-the-art baselines on the 9-species-V2 V2 dataset, as detailed in Table 2. Our model consistently excels in amino acid precision and peptide recall across most species. Specifically, it achieves the highest amino acid precision for all 9 species, averaging 0.907, surpassing previous NAT and AT models. In peptide recall, our model leads in 8 species, with an average of 0.790, also outper- forming NAT baselines. Although ContraNovo marginally outperforms us in human recall, our model demonstrates its generalizability and overall superiority. These results confirm our model's 's robustness in peptide identification.
Lastly, we systematically evaluate training stability by ran- domly selecting 20 different subsets from the MassiveKB dataset and training both PrimeNovo and our model each split. Among the 20 training runs, PoimeNovo fails in 18 cases due to loss explosion or extreme overfitting, while our model fails only once. This demonstrates that our ap- proach is 90% more likely to result in successful training, establishing significantly more stable and reliable training paradigm.
Training Success Rate and Case Study, Our approach aims to ease the learning difficulty in carly-stage training and improve overall learning outcomes. We conducted comprehensive case study demonstrating how RefineNovo assists with the training process and facilitates generaliza- tion. Specifically, we showcase three types of training ob- stacles frequently encountered during the training of the baseline NAT model, PrimeNovo, and compare the training loss and validation accuracy with ours. We demonstrating training two different training datasets as adopted by Mao et al. (2023) and Yilmaz et al. (2022). All model configura- tions were based on the optimal settings and were trained under the same environment, batch size, and learning rate. As shown in 4, PrimeNovo frequently (up to 80% of the time on some datasets) suffers from loss explosion (Case B), heavily depending on the choice of random seed for good convergence. In contrast, our model eliminates the problem of loss explosion due to its easy learning objective at the beginning stage by peeking at true tokens.
Performance of RefineNovo for Similar Mass Amino Acids. Distinguishing amino acids with similar molecular weights is crucial for accurate peptide sequencing. Glu- tamine (Q) and Lysine (K) differ by just 0.036385 Da while oxidized Methionine (Met(O)) and Phenylalanine (F) have nearly identical weights, challenging algorithmic differenti- ation. We rigorously evaluated RefineNovo's 's proficiency in predicting these amino acids, aiming to ascertain its effec- tiveness in intricate cases. Figure 3 highlights RefineNovo's outstanding performance, consistently achieving superior accuracy in identifying amino acids with minimal mass differences, demonstrating its unparalleled precision.
Ablation Study. Table 4 presents an ablation study assess- ing the impact of three key components on our model's performance. Our baseline is naive NAT model, PrimeNovo, using simple etc loss. Introducing the curriculum learning with a fixed mask ratio of 0.7 results in a performance de- cline, indicating that fixed mask ratio may not be optimal.
8

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Metries
Architect
Methods
Mouse Human
Yeast
M.mazei Honeybee
Tomato
Rice bean Bacillus
C.bacteria
Average
AT
Casa. (Yilmaz et al., 2024)
0.813
0,872
0.915
0.877
0.823
0.891
0.891
0.888
0.791
0.862
AA Precision
Prime. (Zhang et al., 2025)
0.839
0.893
0.932
0,908
0.862
0.909
0.931
0.921
0.827
0.891
NAT
Ours
0.850
0.921
0.941
0.921
0.879
0.916
0.931
0.942
0.841
0.907
AT
Casa. V2 (Yilmaz al., 2024)
0.555
0.712
0.837
0.754
0.669
0.783
0.772
0.793
0.558
0.714
Peptide Recall
Prime. (Zhang et al., 2025)
0.627
0,795
0.884
0.812
0.742
0.824
0.837
0.849
0.626
0.777
NAT
Ours
0.637
0.805
0.895
0.827
0.762
0.829
0.862
0.856
0.637
0.790
Table Comparison the performance on the 9-species- V2 benchmark datasets. AT stands for Autoregressive Transformer and NAT stands for non-autoro autoregressive Transformer.
Amino Acid Precision
The model's enhanced training stability and the innovative integration of iterative refinement with the learnt curriculum embedding are key contributors to its effectiveness. These advancements establish RefineNovo valuable tool for proteomics research and other downstream tasks.
Methods
M(O)
Q
F
K
Casa. V2
0.463
0.648
0.678
0.689
Prime.
0.578
0,770
0.806
0.800
Ours
0.600
0.782
0.816
0.810
Impact Statement
Table Comparison of precision for amino acids with similar masses,
Peptide sequencing plays fundamental role in proteomics, yet existing computational methods face significant chal- lenges in efficiency, accuracy, and robustness. Our work introduces a novel curriculum learning framework tailored for non-autoregreesive Transformers (NATs), addressing the longstanding difficulties of training stability and con- vergence in CTC-based models. By dynamically adjusting learning difficulty and integrating an iterative refinement strategy, our approach not only enhances model generaliza- tion but also significantly reduces training failures, improv- ing both sequence prediction accuracy and reliability.
Curriculum Iterative
Difficulty Annealing
Amino Acid
Peptide Recall
Learning
Refinement
Precision
0.788
0.638
0.733
0.558
0.742
0.571
0.793
0.645
0.800
0.653
Table Results of ablation study showing the effects three key components RefineNovo's 's final performance.
This research bridges the gap between structured learning paradigms and modern deep learning architectures, paving the way for more efficient peptide sequencing pipelines. Our methodology has the potential accelerate discoveries in proteomics, drug development, and biomolecular analysis, where high-throughput, accurate sequencing is crucial. By releasing our implementation and trained models, we aim to provide a scalable, reproducible, and widely applicable solution to the broader scientific community.
Adding iterative refinement with fixed mask ratio yields a slight improvement, suggesting that iterative refinement can help correct errors. When combining the curriculum learning with difficulty annealing (dynamic masking ratio), we observe significant enhancement over baseline model. This proves the importance of annealing ratio in adjusting learning curves. When all three components are integrated, our model achieves the highest performance. These results underscore the importance of each component and their synergistic effect, allowing our model to generate highly accurate peptide sequences.
A Acknowledgement
This project partially supported by Shanghai Artificial Intelligence Laboratory (S.S.). This work is partially sup- ported by Netmind.Al and ProtagoLabs Inc. This work is also partially supported by CURE (Hui-Chun Chin and Tsung- -Do Lee Chinese Undergraduate Research Endow- ment) (24924), and National Undergraduate Training Pro- gram on Innovation and Entrepteneurship grant(24924).
5. Conclusion
In conclusion, we introduce RefineNovo, novel and ef- fective approach to de novo peptide sequencing that sets a new performance benchmark in the field. Our model achieves superior performance and consistently surpasses previous deep learning models across all evaluated metries.

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
References
Graves, A. and Jaitly, N. Towards end-to-end speech recog- nition with recurrent neural networks. In International conference machine learning, pp. 1764-1772. PMLR, 2014.
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J.. et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp- 1-3, 2024.
Graves, A., Fernández, S., Gomez, F,, and Schmidhuber, J. Connectionist temporal classification: labelling un- segmented sequence data with recurrent neural networks. Proceedings of the 23rd international conference on Machine learning. pp.
Aebersold. R. and Mann, M. Mass spectrometry-basee proteomics. Nature, 422(6928) 2003.
2006.
Bao, Y., Huang, S., Xiao, T., Wang, D., Dai, X., and Chen, J. von-autoregreessve translation by learning target cate- gorical codes. arXiv preprint arXiv:2 103. 1405, 2021.
Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. Non- con-autoregressive neural machine translation. arXiv preprint arXiv: 1711:011120000 2017.
Chen, C., Hou, J., Tanner, J., and Cheng, J. Bioinformatics methods for mass spectrometry- bassd proteomics data analysis. International journal of molecular sciences, 21 (8):2873, 2020.
Guo, J., Tan, X., He, D., Qin, T., Xu, L., and Liu, T.-Y. Non- autoregressive neural machine translation with enhanced decoder input. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3723-3730, 2019.
Ding, L., Wang, L.. Wu, D., Tao, D., and Tu, Z. Context- aware cross-attention for con-autoregressive translation. arXiv preprint orXiv:2011.0077 1.00770, 2020.
Guo, J., Xu, L., and Chen, E. Jointly masked sequence-to- sequence model for non- non-autoreeressvve neural machine translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 376- 385, 2020.
Ding, L., Wang, L., Liu, X., Wong, D. F., Tao, D., and Tu, 7. Progressive multi-granularity training for non-autoregressive translation. arXiv preprint arXiv:2106.05546, 2021a.
Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., al. Simulating 500 million years of evolution with language model. bioRxiv, pp. 2024 2024.
Ding, I.,, Wang, I.., Liu, X., Wong, D. F., Tao, D., and Tu, Z. Rejuvenating low-frequency words: Making the most of parallel data non-autoregressive translation. arXiv preprint arXiv:2106.0090 2021b.
Huang, C. Zhou, H., Zaiane, O. R., Mou, L.,, and Li, L. Non- autoregressive translation with layer-wise wise prediction and deep supervision. In Proceedings of the AAAI conference on artificial intelligence, pp. 10776-10784, 2022a.
Eloff, K., Kalogeropoulos, K., Morell, O.,, Mabona, A., Jes- persen, B., Williams, W., Beljouw, S. P. v., Skwark, M., Laustsen, A. H., Brouns, S. J., et De novo peptide sequencing with instanovo: Accurate, tatabase-free pep- tide identification for large scale proteomics experiments. bioRxiv, pp. 2023-08, 2023.
Huang, F., Zhou, H., Liu, Y., Li, H., and Huang, M. Directed acyclic transformer for non-autoregressive machine trans- lation. In International Conference on Machine Learning, pp. 9410-9428, PMLR, 2022b.
Gao, J., Chen, J., Wei, J., Jiang, B., and Luo, A.-L. Deep multimodal networks for m-type star classification with paired spectrum and photometric image. Publications of the Astronomical Society of the Pacific, 135(1046): 044503, 2023.
Huang, X. S., Perez, F., and Volkovs, M. Improving non- autoregressive translation models without distillation. In International Conference on Learning Representations, 2022c.
Jin, Z., Xu, S., Zhang, X., Ling, T., Dong, N., Ouyang, W., Gao, 7., Chang, C., and Sun, S. Contranovo: contrastive learning approach to enhance de novo peptide sequencing. arxiv. arXiv preprint 22Xiv:2312.114: 15884, 2023.
Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L.. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint 19Xvv:9904.09324 2019.
Jin, Z., Xu, S., Zhang, X., Ling, T., Dong, N., Ouyang, W., Gao, Z., Chang, C., and Sun, S. Contranovo: A contrastive learning approach to enhance de novo peptide sequencing. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 144-152, 2024.
Ghazvininejad, M., Karpukhin, V., Zettlemoyer, L., and Levy, O. Aligned cross entropy for non-autoregressive machine translation. In International Conference on Ma- chine Learning, pp. 3515-3522 PMLR, 2020.
10

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneherger, O., Tunyasuvunakool,, K., Bates, R., Zidek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588 2021.
Qian, L.,, Zhou, H., Bao, Y., Wang, M., Qiu, L., Zhang, W., Yu, Y., and Li, L. Glancing transformer for non- autoregressive neural machine translation. arXiv preprint arXiv:2008.07900 2020.
Qiao, R. Tran, N. H., Xin, I.., Chen, X., Li, M., Shan, B., and Ghodsi, A. Computationally instrument- resolution-indeeendenn de novo peptide sequencing for high-resolution devices. Nature Machine Intelligence, (5):420-44) 425, 2021.
Karunratanakul, K., Tang, H.-Y., Speicher, D. W., Chuang- suwanich, F.,, and Sriswasdi, S. Uncovering Thousands of New Peptides with Sequence-Mask-Seenn Search Hybrid De Novo Peptide Sequencing Framework. Molecular & cel- lular proteomics MCP. 18(12):2478-244 2491, dec 2019. ISSN 1535-9484 (Electronic).
Qiu, Z., Wei, J., Zhang, X., Xu, S., Zou, K., Jin, Z., Gao, Z., Dong, N., and Sun, S. Universal biological sequence reranking for improved de novo peptide se- quencing. arXiv preprint arXiv:2505.17552: 2025.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
LeCun, Y., Bengio, Y., and Hinton, G. Deep leaming. nature, 521(7553):436 444, 2015.
Saharia, C., Chan, W., Saxena, S., and Norouzi, M. Non- autoregressive machine translation with latent alignments. arXiv preprint arXiv:2004.0744 2020.
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., al. Evolutionary scale prediction of atomic-level protein structure with a language model. Science, 379(6637): 1123-1130, 2023.
Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136 130184, 2024.
Liu, K., Ye, Y., Li, S., and Tang, H. Accurate de novo pep- tide sequencing using fully convolutional neural networks. Nature Communications, 14(1):7974, 2023.
Savinov, N., Chung, J., Binkowski, M., Elsen, E., and Oord, v. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749) 2021.
Liu, P. Zhang. X., and Mou, L. A character-level length- control algorithm for non-autoregressive sentence sum- marization. Advances in Neural Information Processing Systems, 15:29101-299
Shao, C.. Zhang, J., Feng. Y., Meng, F., and Zhou, J. Minimizing the bag-of-ngrams difference for non- autoregressive neural machine translation. In Proceed- ings of the AAAI conference artificial intelligence, volume 34, pp. 198-205, 2020.
2022.
Ma, B., Zhang, K., Hendrie, C., Liang, C., Li, M., Doherty- Kirby, A., and Lajoie, G. Peaks: powerful software for peptide de novo sequencing by tandem mass spectrome- try. Rapid communications in mass spectrometry, 17(20): 2337-2342, 2003.
Shao, C., WU, X., and Feng, Y. One reference is not enough: Diverse distillation with reference selec- tion for non-autoregressive translation. arXiv preprint arXiv:2205.14:3 2022.
Ma, X., Zhou, C., Li, X., Neubig, G., and Hovy, E. Flowseq: Non-autoregreesive conditional sequence generation with generative flow. arXiv preprint arXiv: 024400 2019.
Shu, R., Lee, J., Nakayama, H., and Cho, K. Latent-variable non-autoregreessvve neural machine translation with de- terministic inference using a delta posterior. In Proceed- ings of the aaai conference on artificial intelligence, vol- ume 34, pp. 8846 8853, 2020.
Mao, Z., Zhang, R., Xin, L., and Li, M. Mitigating the miising--ragmentation problem in de novo peptide se- quencing with two-stage graph-based deep learning model. Nature Machine Intelligence, 5(11):1250-1260, 2023.
Muth, T., Hartkopf, F., Vaudel, M., and Renard, B. Y. A potential golden age to come-cuee cureett tools, recent use cases, and future avenues for de novo sequencing in pro- teomics. Proteomics, (8(18):1700150 2018
Song, J. Kim, S., and Yoon, S. Alignart: Non- autoregressive neural machine translation by jointly learn- ing to estimate alignment and translate. arXiv preprint 10N:::210.00444 2021.
Ng. C. C. A., Zhou, Y., and Yao, Z.P.. Algorithms for de-novo sequencing of peptides by tandem mass spec- trometry: review. Analytica Chimica Acta, pp. 341330, 2023.
Stern, M., Chan, W., Kiros, J., and Uszkoreit, J. Inser- tion transformer: Flexible sequence generation via inser- tion operations. In International Conference on Machine Learning, pp. 5976-5985. PMIRR 2019.

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Tran, N. H., Zhang, X., Xin, L., Shan, B., and Li, M. De novo peptide sequencing by deep learning- Proceedings of the National Academy of Sciences, 114(31):247-82552 2017.
Yin, Y., Kaddour, J., Zhang, X., Nie, Y., Liu, Z., Kong, L., and Liu, Q. Trida: Controllable generative data augmen- tation via text-to-text and text-to-image models, arXiv preprint arXiv:2304.0882// 2023.
Wang, M., Wang, J., Carver, J., Pullman, B. S., Cha, S. W., and Bandeira, N. Assembling the community-scale dis- coverable human proteome. Cell systems, 7(4):412-421, 2018.
Zhang, X., Ling, T., Jin, Z., Xu, S., Gao, Z., Sun, B., Qiu, Z., Dong, N., Wang, G., Wang, G., et al. 7-primenovo: An accurate and efficient non-autoregressive deep learning model for de novo peptide sequencing. bioRxiv, pp. 2024- 05, 2024.
Wang. Y., Tian, F., He, D., Qin, T., Zhai, C., and Liu, T.-Y. Non-autoreeressive machine translation with auxiliary regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp- 5377-5384, 2019.
Zhang, X., Ling, T., Jin, Z., Xu, S., Gao, Z., Sun, B., Qiu, Z.., Wei, J., Dong, N., Wang, G., et al. x-primenovo: an accu- rate and efficient -ooreevoreeeesivve deep learning model for de novo peptide sequencing. Nature Communications, 16(1):267, 2025.
Xia, J., Chen, S., Zhou, J., Ling, T., Du, W. Liu, S., and Li. S. Z. Adanovo: Adaptive emph {De Novo] peptide sequencing with conditional mutual information. arXiv preprint arXiv:2403.07013 2024a.
Zheng. Z., Deng, Y., Xue, D., Zhou, Y., Ye, F., and Gu, Q. Structure-informed language models are protein design- ers. In International conference on machine learning, pp. 42338.4333. PMLR, 2023.
Xia, J., Chen, S., Zhou, J., Ling, T., Du W., Liu, S., and Li, S. Z. Adanovo: Adaptive De Novo peptide sequenc- ing with conditional mutual information, 2024b. URL. https: //tp///axvvv/////////a//eeeeaaav org/ abs 2403. 07013.
Zhou, J. and Keung, P. Improving non oon-autorerressvve neu- ral machine translation with monolingual data. arXiv preprint arXiv:2005.0000 2020.
Xia, J., Liu, S., Zhou, J., Chen, S., Xiang, H., Liu, Z., Liu, Y., and Li, S. Z. Bridging the gap between database search and de novo peptide sequencing with searchnovo. bioRxiv, pp. 2024-10, 2024c.
Zhou, Chen, S., Xia, J., Liu, S., Ling, T., Du, W., Liu, Y.. Yin, J., and Li, S. Z. Novobench: Benchmarking deep learning-based de novo peptide sequencing methods in proteomics, arXiv preprint arXiv:2406. 11926 2024.
Xiao, Y., Wu, L.. Guo, J., Li, J., Zhang, M., Qin, T., and Liu, T.yy A survey non-autoregressive generation for neural machine translation and beyond. IEEE Transac- tions Puttern Analysis and Machine Intelligence, 45 (10): 1407-114777 2023.
Zhou, X.-X., Zeng, W.-F., Chi, H., Luo, C., Liu, C., Zhan, J., He, S.-M., and Zhang, Z. pdeep: predicting ms/ms spectra of peptides with deep learning. Analytical chemistry, 89 023:12690-12661 2017.
Zhu, M., Wang. J., and Yan, C. Non-aattoregressive neural machine translation with consistency regularization opti- mized variational framework. In Proceedings of the 2022 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 607-617, 2022.
Yang, H., Chi, H., Zeng, W.-F., Zhou, W.-J., and He, S.- M. pnovo 3: precise de novo peptide sequencing using a lcarning-to- to-rankk framework. Bioinformatics, 35(14): i183-i190, 2019.
Yang, T., Ling, T., Sun, B., Liang, Z., Xu, F., Huang, X., Xie, L., He, Y., Li, L., He, F., Introducing --helixnovo for practical large-scale de novo peptide sequencing. Brief- ings in Bioinformatics, 25(2)bbbae021, 2024.
Yilmaz, M., Fondrie, W., Bittremieux, W., Oh, S., and No- ble, S.S. De novo mass spectrometry peptide sequencing with a transformer model. In International Conference on Machine Learning, pp.
PMLR, 2022.
Yilmaz, M., Fondrie, W. E., Bittremieux, W., Melendez, C. F., Nelson, R., Ananth, V., Oh, S., and Noble, W. S. Sequence-to-ssquuence sequence translation from mass spectra to peptides with a transformer model. Nature communica- tions, 15(47, 2024.
12

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
A. Curriculum Learning
Curriculum learning improves training stability and convergence CTC-based non-autooregressive (NAT) models by dynamically adjusting the learning difficulty based on model performance. Instead of forcing the model to learn complex sequences from the start, our method progressively increases the difficulty, ensuring a smoother optimization process.
The approach consists of four main stages. First, we compute the best alignment path using the CTC loss function, which determines the optimal token mapping between the predicted and target sequences. Next, we extract oracle sequence, selecting key reference tokens that serve as easier learning targets for the model. To control learning difficulty, we introduce a dynamic masking ratio, where the probability of masking increases as the model's ' accuracy improves. Finally, we apply conditional masking, feeding the model partially masked sequences to facilitate structured learning process that transitions from high supervision to a more generalized sequence prediction task.
The following code provides Python implementation of our curriculum learning approach, maintaining consistency with Algorithm 1. We use Py Torch to efficiently compute CTC alignments.
if mode == "train" ::
with torch no -o-grd ():
# Forward pass without gradient updates
word ins out tgt tokens, = self. forward step (s batch)
nonpad_positions
tgt_tokens ne self. decoder get_pad_idx ())
target_lens nonpad-positions.. s1) (um())
pred tokens = word ins out argmax((1) out_lprobs = F. log.softmax, www.tttaawwooddiins.ooo
(word
dim=-1)
# Compute sequence lengths seq lens torch. full
(pred_tokens, size (0).), pred_tokens. size (1)
). (self. device)
# Compute best alignment using CTC
Best_alignt = best.alignment(
out-Iprobs transpose (0, 1), tgt_tokens, seq-lens, target_lens, self.decodee get.blank.idx()
zero_infinity=Truee
# Generate oracle sequence
best_aligns = torch tensor ( [a for a in best aligns], device=word ins out device
oracle pos abest.aligns.pad // 2). clip
max=tgt_tokens shape [1]
oracle tot_ttoens... gather(-1, totens.gather((1... oracle.por oracle.empty = oracle. masked-fill
best aligns pad % = 0, self decoder get.blank.idx ()
# Compute dynamic masking ratio same_num (pred_tokens pracle-empet ). sum keep prob = ((seq lens same num) seq lens peek factor) keep-prob = keep-prob. unsqueeze(-I)
# Generate curriculum learning mask
keep word mask = (
13

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
torch rand (pred_tokens shape, duvice=word.ins.our,, devieeeeee
keep prob
). ()ol
glat.prev = oracle_empty masked-fill
waap self decoder. get_mask_idx ()
B. Iterative Refinement
B.1. Pseudo Code
Algorithm 2 outlines our iterative refinement approach. Instead of relying solely on a single-pass decoding, our method iteratively refines the generated peptide sequence by reintroducing previously decoded outputs as conditional input. At each iteration, the decoder takes the spectra and precursor information as input, along with the pseudo-label generated from previous iteration. The model then refines its predictions by leveraging learned token embeddings, progressively improving sequence accuracy. This iterative process continues for fixed number of steps, ensuring better alignment between the predicted and true sequences while mitigating common decoding errors.
Algorithm 2 Curriculum- Embedding
Murriculum-Embeeddingg based Iterative Refinement
Require: Spectra spectra, Precursors precursors, Decoder decoder, Encoder encoder, Iterations N Ensure: Refined output sequence output .dccoded Initialize prev <- None
2: Initialize output.decoded decoded +
to 3: for 1 N do
4: # Forward pass with previously decoded sequence input
5: output logits, output list + decoder(None, precursors, =encoder precursors), prev)
6: # Decode using argmax and update previous predictions
7: prev < arg max(output logits.-1)
8: end for
RETURN prev
B.2. Impact of Iterative Steps
Table illustrates the impact iterative steps on performance metrics. The first iteration establishes baseline with a peptide recall of 0.728 and an amino acid (AA) precision of 0.848. By the third iteration, peptide recall improves 0.736 and AA precision 0.854, demonstrating that iterative optimization strategy results in more accurate outcomes. However, beyond the third iteration, both metrics plateau 0.737 for recall and 0.855 for precision. This plateau suggests that while initial iterations yield significant improvements, further iterations provide minimal additional benefits. Therefore, we select three iterations to balance inference cost with model accuracy.
Iterative Steps
AA Precision
Peptide Recall
0.848
0.728
2
0.853
0.731
3
0,854
0.736
0.855
0.737
0.855
0.737
10
0.855
0.737
Table 5. Effect of different beam sizes on RefineNovo.
14

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
C. CTC Loss Calculation for Protein Sequence
Connectionist Temporal Classification (CTC) loss a widely used objective function sequence-to-ssequene models, particularly for tasks where the alignment between input and output sequences is unknown. In this section, we provide detailed explanation of how CTC loss calculated efficiently using dynamic programming (Zhang et al., 2025; Gu et al., 2017).
C.I. Problem Definition
The goal of CTC is to compute the total probability of all valid aligmment paths y that reduce to the target sequence A, denoted as [(y) A. Let I represent the input spectrum and A (a1, 02,.... - ,O,) be the target sequence of amino acids. Each alignment path y (y1, ................................ WT) must satisfy the CTC reduction rules: 1. Consecutive identical tokens in are merged. 2. Placeholder token e is removed. 3. Identical tokens adjacent to are not merged.
CTC maximizes the total probability of all valid paths, given by:
P(A[Z) = Σ
P(y
(6)
y-
where P(y[z)=II'=
P(y|IT under the independence assumption. However, directly enumerating all possible paths is computationally infeasible, as the number of valid paths grows exponentially with the length of .A and y. To address this, we use dynamic programming to efficiently compute P(A[Z).
C.2. Dynamic Programming Formulation
We define a(T, r) as the probability of generating the first r amino acids of the target sequence A using the first t tokens in the alignment path y:
a(T,r) = P(A
Σ
P(y|S),
y: A
where S represents the input spectrum I. This recursive relationship allows us to compute P(A[I) efficiently.
The initialization of a(T, r) is as follows:
a(r, 0) P(v =)) P(32 =)) ... .-P(yr c),
VI
(8)
a(1, 1) = P(y = =a1),
(9)
rl,,r) 0,
for r > 1.
(10)
C.3. Recursive Calculation
To compute a(T,r) r) for r and 2 1, we decompose it based on whether the current token y- matches the target amino acid ar, or if contributes to a blank or repeated token. Using the law of total probability:
a(T - 1,r) P(yr = ar), a(T 1, 1) P(y+ = =,,, a(T a(T-1,7) 1,r) PPy, E),
if a, ar-1s
a(T,r)-
a, # ar-1; otherwise.
(1I)
This recurrence efficiently aggregates the probabilities of all valid paths reducing to A
C.4. Loss Function
The final CTC loss is defined as the negative log probability of the target sequence A:
Lere log P(A = log (T, (A]).
(12)
C.5. Practical Considerations
To ensure numerical stability during training, the log-space version of the dynamic programming equations is often used. Additionally, dynamic programming reduces the computational complexity from exponential to linear in terms of T and n. This makes CTC loss feasible for large-scale peptide sequencing tasks.
15

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
C.6. Illustration of CTC Path
An example of the valid paths for a target sequence (A,T, C) and alignment length T is shown below:
Path C
Path 2: A
Path e AT
Only paths that reduce the exact target sequence according to r(.) are considered valid.
Dynamic programming significantly simplifies the computation of CTC loss by avoiding explicit enumeration of paths. This approach ensures efficiency and scalability in de novo peptide sequencing tasks, enabling the model to focus on meaningful aligmment paths while discarding invalid ones. The detailed recursive formulation provided here serves as the foundation for implementing robust CTC-based training for sequence generation models.
D. Precise Mass Control (PMC) Method
D.1. Overview
Precise Mass Control (PMC) (Zhang et al., 2025) is knapsack-like dynamic programming algorithm designed to enforce mass constraints during peptide decoding. The goal of PMC is to ensure that the total mass of the generated peptide sequence aligns with the experimentally measured precursor mass mpr. within a predefined error tolerance a. This ensures that the generated peptide both accurate and physically valid, addressing challenges non-autoregressive sequence generation where no direct mechanism exists to enforce mass constraints during decoding.
D.2. Problem Formulation
The PMC problem can be formalized as maximizing the total log probability of generating a peptide sequence A (01, 02,..., a,), subject mass constraint:
max
log PT),
(13)
A
where I represents the input spectrum, Ply (I) is the model's predicted probability for amino acid ys at position and the mass constraint is given by:
Mpr
u(a:)) S Mp + a,
(14)
a.EA
where u(a;) is the mass of amino acid a, The goal is to find a sequence AA that maximizes the probability while satisfying the mass constraint.
D.3. Dynamic Programming Approach
To solve this optimization problem, we employ a dynamic programming (DP) table df where denotes the step index and denotes the mass at that step. The DP table stores the most probable sequence of amino acids that satisfies the mass constraint at each step.
D. INITIALIZATION
For the first decoding step (t 1), initialize the DP table follows:
c,
if f O,
df = Vı, if u(l() e [e o, o,
(15)
0. otherwise.
Here, é represents the empty sequence, //1 is the first amino acid in the sequence, and u(y1) denotes its mass.
16

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
D.3.2. RECURSION
At each decoding step > I, the DP table is updated by considering three cases, depending on the nature the new token yr 1. If y the mass remains unchanged due to CTC reduction. In this case:
de
U 7Oc, 7Ed
(16)
where denotes sequence concatenation.
2. If y/ is identical to the previous token, the sequence remains the same, but the mass remains constant:
de
U 7 1-
(17)
7Ed
3. If ys is a new token, the mass increases, and the potential sequences are updated as:
d
(-Ve) 7o yt-
(18)
7Ed
D.3.3. MASS-CONSTR UPDATE
To ensure the DP table does not grow excessively large, we retain only the top B sequences with the highest probabilities at each step:
U E )
d = top
(19)
YLEY 7Ed(1
where y represents the of all amino acid tokens.
D.4. Final Sequence Selection
After completing all decoding steps, the final sequence is selected the most probable sequence stored in df, where T is the total decoding length:
A arg max P(y). yEd
(20)
E. Performance Comparison on NovoBench
To rigorously benchmark model performance, we evaluated RefineNovo and all baseline models on NovoBench (Zhou et al., 2024), recently released de novo sequencing benchmark dataset. We adhered to the experimental setup and evaluation protocol outlined in the NovoBench publication, consistently utilizing Saccharomyces cerevisiae (yeast)) the test species for all comparative analyses. We directly evaluated our existing pretrained model, which was trained on MassiveKB, against the NovoBench yeast test sets. This approach, while not leveraging benchmark-specifie training, provides a robust assessment of our model's generalization capabilities.
For direct and equitable comparison with PrimeNovo, the prior state-of- oftte-of-theant Non-Autoreereessive Transformer (NAT)- based model, we utilized its publicly available weights. Both RefineNovo and PrimeNovo were evaluated under identical conditions on the NovoBench test data ensure comparability.
We notice the relatively low performance the 7-species dataset when directly testing with all pretrained models such as Casanovo, PrimeNovo, and RefineNovo. Upon further investigation, we found that this dataset was generated using MS equipment with precision levels significantly different from those in the MassiveKB training data. This results in notable distribution mismatch. Nevertheless, despite this domain shift, the pretrained RefineNovo model still demonstrates clearly better performance compared to other models trained on MassiveKB. This highlights the robustness of our method under distributional variation.
17

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
Table Performance comparison the NovoBench benchmark (yeast test species). Scores for models marked with quoted from the NovoBench paper or original publications. CV denotes cross-validation results from the original PrimeNovo paper. indicates data not available.
Model
9Species (yeast)
7Species (yeast)
HC-PT
Casanovo
0.48
0.12
0.21
InstaNovo
0.53
0.57
AdaNovo
0.50
0.17
0.21
HelixNovo
0.52
0.23
0.21
SearchNovo *
0.55
0.26
0.45
PrimeNovo-CV
0.58
Casanovo-pretrained
0.60
0.05
PrimeNovo
0.70
0.09
0.85
RefineNovo (ours)
0.71
0.09
0.88
F. Relationship to Prior Work in Iterative Refinement and Difficulty Annealing
The development of robust generative models for sequences, particularly in complex domains like peptide generation, benefits significantly from strategies that enhance output quality and training stability. In this context, our proposed self-refining module and difficulty annealing strategy build upon established concepts while introducing specific innovations tailored to Non-Autoregreesive Transformer (NAT) models and Connectionist Temporal Classification (CTC) based decoding.
E1. Iterative Self-Refinement
The principle of iteratively refining generated outputs to improve quality is a powerful paradigm. Our post-training self-refining module, integrated into the main NAT architecture, draws inspiration from multi-pass generation techniques employed in notable protein-related models like ESM-3 and AlphaFold2, which leverage iterative processing for enhanced prediction accuracy.
We acknowledge that the motivation for such refinement-improvie generation quality through successive rounds of error correction and adjustment- is shared with several existing lines of research. For instance. masked language modeling approaches, including conditional masked language modeling, and discrete or masked diffusion models also employ iterative refinement (Ghazvininejad et al., 2019; Zheng et al., 2023; Sahoo et al., 2024), often by re-predicting masked or noisy portions of a sequence.
Our method, however, introduces a distinct mechanism specific NAT models utilizing CTC. Instead of directly refining the generated token sequence, our self-refinement module operates the CTC path. A CTC path represents one of many possible alignment and reduction outcomes that can produce the target label sequence. In our framework, an initial forward pass yields a CTC path. This path is then fed back into the model, allowing for iterative adjustment and refinement of this alignment representation in subsequent passes. The final, refined CTC path is then used for the reduction to the ultimate output sequence. This CTC-path-centric refinement allows the model to explore and optimize the alignment space more effectively within the NAT framework, differentiating it from methods that directly manipulate the sequence tokens during refinement.
F.2. Difficulty Annealing in Sequence Generation
Curriculum learning, or "easy-to-hard" training strategies, has demonstrated efficacy in various machine learning tasks, including sequence generation. Prior work has explored difficulty annealing primarily at a coarser granularity, such as the task level (learning simpler tasks before more complex ones) at the inter-sequence level (e.g., training shorter structurally simpler sequences before progressing to longer or more complex ones, as explored in some protein generation contexts (Ghazvininejad et al., 2019).
To the best of our knowledge, our proposed difficulty annealing strategy is among the first to implement annealing at a within-sequence granularity for NAT models with CTC. Our method defines and modulates difficulty within each training sequence by controlling the amount of information exposed from its chosen CTC path during training. Each sequence
18

---

Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing
effectively starts an "easier" instance by revealing greater portion of its CTC path. As training progresses, the visibility of this path information is gradually reduced. This reduction exponentially increases the learning difficulty due the combinatorial explosion in the number of valid CTC paths that could correspond to the same target label sequence.
This fine- aeneggraiedd within-sequence and within-path difficulty annealing is uniquely enabled by our CaC-sampling mecha- nism, which is specifically designed for this purpose. This approach plays a crucial role in stabilizing the training dynamics of our NAT model and guiding it towards more robust representations.
19


Reviews:
[{"review_id": "g4IBAfKLra", "reviewer": "Reviewer_V8fu", "summary": "This paper proposed a new method, named RefineNovo, that improves non-autoregressive transformers (NATs) for peptide sequencing by introducing a curriculum learning strategy and a self-refinement module (including a \"difficulty annealing\" strategy). It reduces training failures by over 90% and enhances sequence accuracy. Authors evaluated on 9-species-v1 and 9-species-2 benchmarks that show improved performance over existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and for recognizing the novelty and effectiveness of our proposed method. We sincerely appreciate your effort and provide below point-by-point responses to your comments.\n\n> The self-refining module has been explored for peptide generation to some extent\n\nWe thank the reviewer for their interest in comparing our \"difficulty annealing\" and \"self-refinement\" strategies with related prior work ```(Marjan et al., 2019; Zaixiang et al., 2023; Subham et al., 2024)```.\n\nRegarding self-refinement, our method introduces a **post-training refining module** integrated into the main NAT architecture. As noted in our paper, it is **inspired by prior work such as ESM-3 and AlphaFold2**, which leverage multi-pass generation for more accurate predictions in protein-related tasks. We reviewed the works mentioned by the reviewer and **agree that masked modeling and diffusion-based approaches share a similar motivation**—improving generation quality through iterative refinement.\n\nWe acknowledge that we did not adequately cite these related works in the current draft and **will revise the manuscript to include and discuss them**. Specifically, our refinement method is designed as a CTC-path refinement, specific to **NAT models**. Unlike prior methods that directly refine the generated sequence, our approach refines the CTC path obtained from a previous forward pass. This CTC path represents _**one**_ possible reduction outcome under CTC alignment, and our refinement allows the model to iteratively adjust it in future passes. The final path will be used for reduction of final sequence. **The core motivation remains the same with many prior work**: to improve final predictions through successive rounds of error correction and adjustment, as mentioned prior work. **We will make this clear in final paper with proper reference to all above mentioned paper and others**!\n\n> Difficulty annealing strategy has been explored is some studies for sequence generation task e.g., conditional masked language modeling\n\nRegarding difficulty annealing, to the best of our knowledge, we are **among the first to apply _within-sequence_ difficulty annealing**. Most prior work using \"easy-to-difficult\" curriculum strategies does so at a **coarser granularity**—typically at the _task level_, where simpler tasks are learned before more challenging ones. In some protein-related work, including those mentioned by the reviewer, the approach is _intra-sequence_—e.g., learning shorter or simpler sequences before longer, more complex ones.\n\nIn contrast, our method defines difficulty **within each sequence** by modulating the **amount of CTC path information exposed during training**. Each sequence can start with an easier version—by revealing more information in its chosen CTC path—and gradually become harder by reducing path visibility. This increases difficulty exponentially due to the combinatorial number of valid CTC paths per label. Our design thus enables _within-sequence_ and _within-path_ difficulty annealing, made possible by our CTC-sampling mechanism tailored for this purpose. This fine-grained annealing plays a key role in our model’s training dynamics. We will further discuss this and difference with previous work ```(Marjan et al., 2019; Zaixiang et al., 2023; Subham et al., 2024)```\n in our final writing! \n\n> In Equation 3 (page 4), there is a wrong math notation: sum over probabilities is written to be directly equal to the sum over log probabilities.\n\nThank you for the careful reading! We will ensure this error is corrected in the final version. It should read: the log of the product of all token probabilities equals the sum of the log of each token’s probability. Thanks again for catching this!\n\n> There is notation mismatch between Equations 3 and 4. Equation 4 uses L as the likelihood, and Equation 3 used L as the negative log likelihood. I would suggest using different symbols for clarity.\n\nApologies for the confusion—we inadvertently reused the same notation without realizing it. We will revise the notation to use distinct symbols to avoid ambiguity. Thank you again for the careful reading!\n\n> a self-refinement module, which is much related to discrete or masked diffusion language modeling (e.g., Subham et al. 2024) and conditional masked language modeling (Marjan et al. 2019) that the authored did not cite or discuss.\n\n we will thoroughly discuss the similarities and differences between their self-refinement modules  (Marjan et al., 2019; Zaixiang et al., 2023; Subham et al., 2024) and ours based on above discussion, and ensure all mentioned and related works are properly cited in the Related Work section. Thank you for pointing this out!\n\n\n**Please let us know if you have any further questions—we’d be happy to address them. Thanks again for your time and thoughtful review!**", "claims_and_evidence": "- The idea of introducing a curriculum learning strategy to peptide sequencing is indeed novel. \n\n- The method is properly explained, with clear depiction, discussion of the components, and mathematical derivations in the appendix.\n\nConcerns:\n- The self-refining module has been explored for peptide generation to some extent, e.g., with discrete or masked diffusion language modeling, conditional masked language modeling, etc. As far as I know, the difficulty annealing strategy has been explored is some studies for sequence generation task e.g., conditional masked language modeling (Marjan et al. 2019), LM-design (Zaixiang et al,. 2023). I wonder how the proposed method RefineNovo is different from these approaches.", "methods_and_evaluation_criteria": "- The proposed method, focusing on three components, is indeed relevant for the problem at hand.\n\n- The authors show results on ed 9-species-v1 (Tran et al., 2017) and 9-species-v2 (Yilmaz et al., 2024) benchmark datasets. The experimental results reported in Tables 1, 2, and 3 support their claim. \n\n- The author also showed ablation of the effect of three different components in Table 4, which is useful to see get a clearer idea how each component and their combinations are affecting the generation task.\n\n- In Figure 4, they also show a case study on training failures, where RefineNovo is clearly showing promising improvements.", "theoretical_claims": "- The main paper does not have any theoretical claims.\n\n- The appendix has discussion on CTC loss computation and PMC methods, which looks okay. Although there are not the contribution of this paper, I appreciate the authors including that in the appendix for a clearer picture. \n\n- The appendix also include the algorithms for Curriculum Learning and Iterative Refinement, which also are properly written. \n\nMinor concerns:\n- In Equation 3 (page 4), there is a wrong math notation: sum over probabilities is written to be directly equal to the sum over log probabilities.\n\n- There is notation mismatch between Equations 3 and 4.  Equation 4 uses L as the likelihood, and Equation 3 used L as the negative log likelihood. I would suggest using different symbols for clarity.", "experimental_designs_or_analyses": "The experimental design with the two dataset 9-species-v1 and 9-species-v2 as well as the analyses on training failures are clearly discussed.", "relation_to_broader_scientific_literature": "The main contribution of this paper is related to the research on bio-sequencing (especially for proteins), as well as machine learning research on bio-sequence generation. There are prior works on protein sequence generation, especially in the regime of inverse folding, where ideas closely related to what was proposed here has been explored to some extend.", "essential_references_not_discussed": "As mentioned before, the main contribution is introducing a curriculum learning strategy and a self-refinement module, which is much related to discrete or masked diffusion language modeling (e.g., Subham et al. 2024) and conditional masked language modeling (Marjan et al. 2019) that the authored did not cite or discuss.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyY4KtHgjK", "reviewer": "Reviewer_NQdu", "summary": "The work proposes a non-autoregressive transformer (NAT)-based curriculum learning framework to deduce the animo acid sequence from tandem mass spectrometry signals. The input peak signals are encoded with an transformer encoder, which is then used to predict the probability of tokens on each position. The decoding is trained on the sampled decoding paths from the predicted probability with increasingly higher masked ratios. The predictions are iteratively refined during training. Results indicate higher performance across multiple species than the baseline methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and effort in providing thoughtful comments and feedback. Below, we provide a point-by-point response to your questions and concerns.\n\n> Is the masking performed on $y'$ or $A$.  Also, how is the CTC objective defined with masking?\n\n\nThe masking is performed on a selected CTC path $y'$, which is one of the possible alignments (CTC paths) derived from the true label sequence $A$.\n\nFor example, if the true label sequence $A$ is **ACTC** and the generation length is 5, there can be multiple valid CTC paths $y$, such as:\n\n- `A C T T C`  \n- `A C T C C`  \n- `A C ε T C`  \n- *...and many others*\n\nOur algorithm proceeds as follows:\n\n **Forward Pass for CTC Path Selection:**  \n   We first perform a forward pass to obtain the most likely CTC path $y'$ corresponding to the ground truth $A$. For instance, the model may choose `A C T T C` (all $y$ are ranked by their total probability for selection).\n\n **Masking:**  \n   We then apply masking to this specific path $y'$. For example, it may become:  \n   `A <mask> T <mask> C`\nafter masking.\n\n\n**Prediction Condition:**  \n   This masked sequence is then used as a conditioning input. The model is tasked with predicting the full distribution over all valid CTC paths for $A$—such as `ACTTC`, `ACTCC`, `AC ε TC`, etc.—given the partially masked version of one such path.\n\nThe CTC objective **remains unchanged**. The model is still trained to maximize the total probability over ```all``` valid CTC paths corresponding to the true label $A$. Intuitively, the model sees  ```one```  **partially** revealed  CTC path and is asked to infer and generalize over the ```full``` space of valid CTC paths.\n\n\n> As the path $y$ does not necessarily have the same length as the ground truth sequence $A$, how is the masking performed? More specifically, how are ground truth tokens from $A$ incorporated into $y$?\n\nAs illustrated in the example above, masking is applied to **one** single sampled CTC path $y'$ derived from the ground truth sequence $A$. Each $A$ can correspond to $\\mathcal{O}(b^n)$ possible paths $y$, where $n$ is the generation length in NAT (40 in our case), and $b$ is the vocabulary size.\n\nRegarding your question: while $A$ and $y$ may differ in length, all sampled $y$ from $\\mathcal{O}(b^n)$ have the same fixed generation length (e.g., 40). We sample ```one``` such $y'$, apply masking to it, and use the result as a condition for predicting ```all``` valid paths. In this way, some of the tokens from $A$ naturally appear in the unmasked positions of $y$, providing partial supervision.\n\nFor example, let the ground truth be $A = \\text{ACTC}$, and a few possible CTC paths $y$ could be:\n\n- `A C T T C`  \n- `A C T C C`  \n- `A C ε T C`  \n- *...among others*\n\nWe might select $y'$ = ```ACTCC```, and mask it to get:  \n`A <mask> T <mask> C`\n\nThis masked sequence includes some ground truth tokens from $A$ and serves as a partial observation. The model is then trained, under the CTC loss, to recover the full output distribution over all valid paths $y$ corresponding to $A$. This encourages it to learn both how to complete the sequence and how to generalize across CTC paths (all of length 40).\n\n> In the earlier stages of training, the predicted token probabilities are inaccurate and all paths that satisfy the CTC constraints would have low probability, which could lead to biases in training. Does the proposed method take into account the probability of the sampled path?\n\nYes, this is a very insightful question—thank you for raising it.\n\nWhat we observed is that, during the early stages of training, the model tends to assign disproportionately high probabilities to the $\\epsilon$ (blank) token. As a result, the selected CTC path $y$ (via greedy decoding) often includes many $\\epsilon$ tokens. These $\\epsilon$-heavy paths are selected simply because they appear most probable under the model’s initial, untrained predictions.\n\nTo address this, we experimented with **top-$k$ sampling** during early training to encourage more diverse path selection. However, in practice, we observed that this made little difference. As training progresses, the model rapidly learns meaningful path patterns, and the issue of degenerate $\\epsilon$-dominated paths diminishes.\n\nEventually, greedy selection (top-1) naturally yields well-formed CTC paths with fewer consecutive $\\epsilon$ tokens. We found almost no performance difference between top-$k$ sampling and greedy choice, especially beyond the initial training phase.\n\nThank you again for highlighting this point!\n\n**We thank the reviewer for the careful and detailed reading—masking in the context of CTC can indeed be tricky. We will include a more detailed explanation with the examples to avoid any confusion.**\n\n**Please don’t hesitate to reach out with further questions!**\n\n**Thank you again for your time!**", "claims_and_evidence": "The authors provide comprehensive results to support their claims.\n\n1. In the results section, ContraNovo is compared to several times, producing slightly higher performance. However, the actual values are not shown in the main text or supplementary.", "methods_and_evaluation_criteria": "The methods and evaluation make sense for the application, despite a few points that needs clarification:\n\n1. Is the masking performed on $y'$ or $A$? Also, how is the CTC objective defined with masking? \n\n2. As the path $y$ does not necessarily have the same length as the ground truth sequence, how is the masking performed? More specifically, how are ground truth tokens from $A$ incorporated into $y$?\n\n3. In the earlier stages of the training, the predicted token probabilities are inaccurate and all paths that satisfy $\\Gamma(y)=A$ would have low probability, which would lead to biases in the training. Does the proposed method take into account the probability of the sampled path $y'$?", "theoretical_claims": "I checked the definition of the CTC objective and the PMC.", "experimental_designs_or_analyses": "The experiments and analyses are sound and valid.", "relation_to_broader_scientific_literature": "The proposed framework combines well-established techniques in the field as well as broader machine learning to enhance the effectiveness of the inference and address several technical challenges, such as the optimization of the CTC objective and the iterative refinement.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "See previous sections", "other_comments_or_suggestions": "See previous sections", "questions_for_authors": "I only have some minor questions and concerns. See previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SWpZ1rL7cb", "reviewer": "Reviewer_Qeo9", "summary": "This paper proposes an improved non-autoregressive peptide sequencing model incorporating a structured protein sequence curriculum learning strategy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful comments. We have addressed all of your concerns with point-by-point responses below:\n\n> This field has an open benchmark dataset, and the method should be tested on it.\n\nThank you for pointing this out. We apologize for missing this benchmark dataset earlier, as it is relatively new. In response, we have conducted additional experiments using the **NovoBench** dataset and evaluated our method, **RefineNovo**, alongside previously reported baselines.\n\nFollowing the experimental setup from the NovoBench paper, we downloaded the test data (from multiple sources as specified by the paper's data source instruction) and compared our method against the results reported in the benchmark, including those of **PrimeNovo** and other baselines.\n\nConsistent with NovoBench’s evaluation protocol, we used **yeast** as the test species. Due to time constraints during the rebuttal period, we were unable to retrain our model on the NovoBench training data. Instead, we directly evaluated our pretrained model on the benchmark test set.\n\nTo highlight the effectiveness of our method, we include a direct comparison between **RefineNovo** and **PrimeNovo**, the previous best-performing NAT-based model. Using the publicly available weights of PrimeNovo from GitHub, we evaluated both models under identical conditions. Our results show that **RefineNovo outperforms PrimeNovo across all test datasets**, as Seen in Table below.\n\nAdditionally, to emphasize the performance advantages of NAT-based architectures, we referenced the **cross-validation (CV)** results from the PrimeNovo paper, where the model was trained on nine species (excluding yeast). These results demonstrated the strength of NAT designs relative to other architectures on the same task.\n\nWe notice the relatively low performance on the 7-species dataset when directly testing with all pretrained models such as **Casanovo**, **PrimeNovo**, and **RefineNovo**. Upon further investigation, we found that this dataset was generated using MS equipment with precision levels significantly different from those  in the **MassiveKB** training data. This results in a notable distribution mismatch.\nNevertheless, despite this domain shift, the pretrained **RefineNovo** model still demonstrates clearly better performance compared to other models trained on MassiveKB. This highlights the robustness of our method under distributional variation.\n\nWe thank the reviewer again for bringing this benchmark to our attention. ```We will incorporate all experimental results, dataset references, and baseline comparisons into the final version of the manuscript and ensure all relevant works are properly cited.```\n\n\n| **Model**            | **9Species (yeast)** | **7Species (yeast)** | **HC-PT** |\n|----------------------|----------------------|-----------------------|-----------|\n| Casanovo *           | 0.48                 | 0.12                  | 0.21      |\n| InstaNovo *          | 0.53                 | –                     | 0.57      |\n| AdaNovo *            | 0.50                 | 0.17                  | 0.21      |\n| HelixNovo *          | 0.52                 | 0.23                  | 0.21      |\n| SearchNovo *         | 0.55                 | 0.26                  | 0.45      |\n| PrimeNovo-CV *       | 0.58                 | –                     | –         |\n| Casanovo-pretrained | 0.60                 | 0.05                | –         |\n| PrimeNovo    | 0.70                 | 0.09                  | 0.85      |\n| **RefineNovo**       | 0.71            | 0.09              | 0.88  |\n\n\\* **marks numbers are quoted from benchmark/original paper**\n\n\n> The experimental section of this paper needs to incorporate more baseline methods [1][2][3] for comparison.\n\nThank you for the suggestion. We will incorporate the **full set of baseline comparisons using the NovoBench benchmark**, as shown in the table above. This includes **InstaNovo**, **AdaNovo**, **HelixNovo**, **SearchNovo**, and **PrimeNovo**. ```These methods will be discussed and properly referenced in both the <Related Work> and <Experiments> sections of the revised manuscript```.\n\nRegarding **PowerNovo**, we have reviewed the published work and found that the dataset used appears to be inconsistent with the NovoBench benchmark. Nonetheless, we will discuss PowerNovo separately in both the *Related Work* and *Experiments* sections to ensure a comprehensive comparison and fair contextualization of our approach.\n\n\n**We sincerely thank the reviewer again for highlighting these points—your feedback has helped us significantly improve the completeness and clarity of our experimental evaluation, let us know if you have further questions after reading our responses!!**", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "This field has an open benchmark dataset, and the method should be tested on it.\n\n\n[1] NovoBench: Benchmarking Deep Learning-based De Novo Peptide Sequencing Methods in Proteomics", "theoretical_claims": "This paper has no theoretical claims.", "experimental_designs_or_analyses": "The experimental section of this paper needs to incorporate more baseline methods[1][2][3] for comparison.\n\n[1] De novo peptide sequencing with InstaNovo: Accurate, database-free peptide identification for large scale proteomics experiments\n\n[2] Bridging the Gap between Database Search and De Novo Peptide Sequencing with SearchNovo\n\n[3] PowerNovo: de novo peptide sequencing via tandem mass spectrometry using an ensemble of transformer and BERT models", "relation_to_broader_scientific_literature": "This paper introduces the NAT decoding technology to the task of de novo peptide sequencing.", "essential_references_not_discussed": "This paper has no essential references not discussed.", "other_strengths_and_weaknesses": "This paper is well-written, and the proposed method is also novel.", "other_comments_or_suggestions": "I have no further comments or suggestions.", "questions_for_authors": "1. Using the latest benchmark datasets to test the proposed model.\n2.  Add more baseline methods as comparison methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents RefineNovo, a non-autoregressive Transformer model (RefineNovo) for de novo peptide sequencing, enhanced with a curriculum learning strategy and a self-refinement module. All three reviewers recommend acceptance, citing the method’s novelty, sound design, and strong experimental results across multiple datasets.

While some related work in curriculum learning and refinement is not deeply discussed, and additional baseline comparisons (e.g., from NovoBench) could further strengthen the study, the reviewers were satisfied by the author's rebuttal and overall contributions. The paper is clearly written, well-motivated, and offers a practical improvement for peptide sequencing.
Decision: Accept (poster)