Title: Bidirectional Representations Augmented Autoregressive Biological Sequence Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Protein Sequencing
Abstract: Autoregressive (AR) models, common in sequence generation, are limited in many biological tasks like de novo peptide sequencing and protein modeling by their unidirectional nature, failing to capture crucial global bidirectional token dependencies. Non-Autoregressive (NAR) models offer holistic, bidirectional representations but face challenges with generative coherence and scalability.
To transcend this, we propose a hybrid framework enhancing AR generation by dynamically integrating rich contextual information from non-autoregressive mechanisms. Our approach couples a shared input encoder with two decoders: a non-autoregressive one learning latent bidirectional biological features, and an AR decoder synthesizing the biological sequence by leveraging these bidirectional features. A novel cross-decoder attention module enables the AR decoder to iteratively query and integrate these bidirectional features, enriching its predictions. This synergy is cultivated via a tailored training strategy with importance annealing for balanced objectives and cross-decoder gradient blocking for stable, focused learning.
Evaluations on a demanding 9-species benchmark of de novo peptide sequencing task show our model substantially surpasses AR and NAR baselines. It uniquely harmonizes AR stability with NAR contextual awareness, delivering robust, superior performance on diverse downstream data. This research advances biological sequence modeling techniques and contributes a novel architectural paradigm for augmenting AR models with enhanced bidirectional understanding for complex sequence generation.
Our code is available on GitHub: https://github.com/BEAM-Labs/denovo


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce **CROSSNOVO**, a hybrid framework designed for de novo peptide sequencing from mass spectrometry data. Traditional autoregressive (AT) models capture strong sequential dependencies but miss important global bidirectional context, whereas non-autoregressive transformers (NATs) capture holistic context but face optimization challenges. CROSSNOVO combines these two paradigms by using a shared encoder and two decoders: an NAT decoder that generates rich, bidirectional context features, and an AT decoder that leverages these features via a novel cross-decoder attention module to improve sequential prediction. To stabilize training, the authors introduce a tailored multitask loss strategy incorporating importance annealing and gradient blocking. Empirical evaluations on comprehensive 9-species peptide sequencing benchmarks demonstrate significant improvements over both autoregressive and non-autoregressive baselines, validating CROSSNOVO's capability to integrate the strengths of both modeling paradigms.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The authors propose a sophisticated and mathematically rigorous framework, CROSSNOVO, that effectively integrates autoregressive (AT) and non-autoregressive (NAT) approaches. The mathematical formulations, especially the cross-decoder attention mechanism (Eq. 6, Section 3.4) and Connectionist Temporal Classification (CTC) loss derivations (Eqns. 5, 8–15 in Appendix), are detailed, clearly explained, and well-justified. I think this is a strong innovation.\n- The joint training strategy with importance annealing (Eq. 3, Section 3.3) and cross-decoder gradient blocking (Eq. 7, Section 3.4) is really cool and clearly justified. The authors thoroughly validate these choices with rigorous ablation experiments (Table 4 on page 9).\n- The empirical validation is super strong and extensive, evaluating on challenging, multi-species benchmarks (9-species-v1 and v2, Tables 1 and 2), and explicitly demonstrating superiority over well-established baselines (CasanovoV2, ContraNovo). The rigorous downstream evaluation on antibody sequencing tasks (Figures 5 and 7, Tables 6 and 7) further emphasizes practical significance.\n- The authors present detailed evaluations of hyperparameter sensitivity (beam size analysis in Table 3) and model generalization to biologically important cases (amino acids with similar masses, Figure 4). These analyses demonstrate thoughtful, thorough experimental design and validation.\n\n**Weaknesses:**\n- While the paper provides a detailed mathematical exposition, for me, some critical aspects of the cross-decoder attention mechanism (Eq. 6) could benefit from additional clarity/info. Specifically, I wonder how the distinct positional encodings are exactly defined or computed for combining NAT decoder representations and original spectral features. A more explicit mathematical formulation or intuitive explanation would significantly enhance understanding.\n- The choice and sensitivity of the importance annealing schedule (Eq. 3, λ_AT parameter) are only briefly discussed. It would greatly improve the clarity and robustness of the training strategy if the authors explicitly provided a sensitivity analysis or mathematical rationale for the chosen linear schedule. Why not alternative schedules such as exponential or piecewise-linear annealing?\n- The dynamic programming-based PMC decoding described extensively in the appendix (Section A.3) appears critical for practical performance. However, I was confused why the interplay between PMC decoding and cross-decoder attention is not clearly defined mathematically. Clarifying this interaction explicitly (perhaps via an integrated algorithm or explicit joint mathematical formulation?) would significantly improve methodological rigor and reproducibility.\n\nIn summary, while CROSSNOVO demonstrates strong methodological rigor, empirical validation, and clear practical significance, I would help to explicitly address the detailed mathematical clarifications, rigorous comparison with key peptide-generation baselines, and thorough computational efficiency analysis would greatly enhance the overall stre\n\nQUESTIONS:\n1. The cross-decoder attention mechanism (Eq. 6, page 5) is intriguing, but its formulation isn't super clear to me. Specifically, how exactly are the positional encodings for the concatenation of NAT-derived features \\( V^{(L')}_{p\\{1:T_{max}\\}} \\) and original spectral embeddings \\( E^{(b)}_{p\\{T_{max}+1:T_{max}+k\\}} \\) computed and differentiated mathematically? It would greatly strengthen the paper if you explicitly clarified the mathematical or algorithmic steps used to distinguish these positional encodings, perhaps providing explicit equations or pseudocode in the appendix.\n\n2. Your importance annealing schedule (Eq. 3, page 5) linearly increases the weighting coefficient \\( \\lambda_{AT}(i) = \\frac{i}{T_{total}} \\). I wonder why you specifically chose a linear schedule over alternatives like exponential decay, sigmoid, or stepwise annealing. Could you provide either an empirical sensitivity analysis comparing these alternative schedules or a theoretical justification explaining why a linear schedule is optimal or preferable for this training strategy?\n\n3. Your Precise Mass Control (PMC) decoding algorithm (detailed in Appendix A.3) is quite sophisticated. However, it's unclear exactly how PMC decoding integrates mathematically with the cross-decoder attention mechanism during inference. Could you provide a more rigorous mathematical integration or pseudocode explicitly detailing the interaction between PMC decoding and cross-decoder attention during sequence inference? \n\n5. Considering the computational complexity of your dual-decoder architecture and PMC decoding, explicit computational benchmarking is missing. Could you provide runtime benchmarks, inference memory usage, and scalability comparisons against baseline methods? \n\nAddressing these concerns will help me move my decision to a solid accept.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel hybrid architecture for de novo peptide sequencing, which innovatively combines the advantages of autoregressive and non-autoregressive models. The core contribution of the paper lies in the design of a dual decoder architecture. Experiments have shown that it outperforms existing methods significantly on 9 species benchmark tests.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper clearly explains the limitations of AT and NAT models in de novo peptide sequencing.\n\n2. The model architecture, training strategy, and knowledge transfer mechanism are described in detail and organized clearly.\n\n3. The validation of this method in downstream tasks such as antibody sequencing has demonstrated its potential for practical biological research.\n\nWeaknesses:\n\n1. The paper did not fully explore the impact of model size on performance, as well as the sensitivity analysis of different parameter settings.\n\n2. Although the experimental results are impressive, a theoretical analysis can be conducted on why NAT to AT knowledge flow is more effective than two-way.\n\n3. The proposed method seems to be a universal architecture. Although models have advantages in solving de novo peptide sequencing, they should also be compared to similar general architectures.\n\n4. If the writing of the paper is corrected again, it will bring a good reading experience. For example, the first reference is duplicated with the second one.\n\n5. Although quantitative ablation experiments are conducted, qualitative results are more convincing.\n\nQUESTIONS:\n1. How sensitive are the parameters?\n\n2. Will the proposed dual decoder architecture significantly increase computational costs? Can you provide a comparison of inference time with existing methods?\n\n3. Can CROSSNOVO's hybrid architecture design principles be extended to other sequence modeling tasks, such as RNA sequence analysis?\n\n4. Can the author analyze the performance of CROSSNOVO on peptide sequences of different lengths?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposed a novel framework combining both autoregressive (AT) and non-autoregressive (NAT) tranformers for de novo peptide sequencing. The two-stage training firstly pretrain both AT and NAT decoder jointly, and then finetune the AT decoder with NAT features to improve the overall performance of the AT model. The proposed CrossNovo model achieves strong performance compared with baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengh**\n1. Clear explanation of the proposed model architecture. \n1. Strong performance in benchmark compared to comprehensive list of baselines.\n\n**Weakness**\n1. Discussion of baselines is lacking for readers who are not familiar with the specific task.\n1. Ablation study can be improved.\n1. Formatting of manuscript and figures can be improved. \nSee *Questions* section for more details about weakness.\n\nQUESTIONS:\n1. Related work should probably not include the model proposed in this work (line 92-97).\n1. The problem formulation is clear but can be improved for readers who are not familiar with the domain knowledge. From my understanding, the length of the amino acid sequence $\\mathbf{A}$ to be predicted is unknown, which is why AT is preferred over NAT method for this problem. It would be nice if the authors can mention that.\n1. The authors mention augmenting the AT decoder with prefix and suffix mass information at each generation step (line 140-143). Can the authors elaborate?\n1. Figure 3 and Figure 6: missing x and y axis labels.\n1. Ablation study can be improved. First of all, the author should elaborate more about different ablation configurations. The shared encoder config is not explained clearly in the paper. More importantly, I am specifically curious about one possible configuration: if you train the NAT decoder first with spectrum encoder, and then train the AT decoder from scratch as in the second stage of training (cross attending to both spectrum encoder and NAT decoder features), how would the performance be. To me, the NAT decoder functions more as another “encoder” or “feature extractor” that provides bi-directional features to the AT decoder, hence I’m curious about the ablation of such configuration. \n1. Can authors provide comparison of number of parameters of the proposed CrossNovo model and the baselines (such as CasaNovo V2 and ContraNovo)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study addresses the de novo peptide sequencing problem in mass spectrometry-based proteomics, a critical challenge in proteomic research. Prior approaches have relied primarily on autoregressive (AR) or non-autoregressive (NAR) models, which face limitations in capturing bidirectional sequence dependencies (AR) or ensuring generative coherence and scalability (NAR). To overcome these issues, we introduce a hybrid framework integrating AR and NAR paradigms through: (1) A shared spectral encoder coupled with dual decoders (AR and NAR) featuring cross-decoder attention mechanisms to leverage complementary strengths. (2) Importance annealing to dynamically balance contributions from the two decoding strategies and cross-decoder gradient blocking to prevent interference during training.\nThis study provides validation across a 9-species benchmark dataset and antibody-specific peptide datasets, demonstrating significant performance improvements over state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method undergoes rigorous evaluation and demonstrates significant performance improvements in de novo peptide sequencing tasks.\n2. The study is overall well-written and clearly presented.\n3. This study can benefit the audience in the peptide sequencing domain.\n\nWeakness:\n1. Previously, hybrid autoregressive and non-autoregressive decoder was studied by many other papers for various traditional NLP tasks, including speech recognition. It is not clear what is the uniqueness of the method for peptide sequencing task?\n2. It would be interesting to look into the post-translational modified peptide task as PrimeNovo did (https://d-nb.info/1360284540/34)\n\nQUESTIONS:\n1. Previous studies have explored hybrid autoregressive (AR) and non-autoregressive (NAR) decoders for various traditional NLP tasks, such as speech recognition. However, the unique advantages or innovations of this hybrid approach in the specific context of peptide sequencing remain unclear. What distinct features does the proposed method offer to address the challenges inherent to peptide sequencing that set it apart from existing hybrid frameworks?\n2. Extending the analysis to post-translationally modified (PTM) peptides—similar to the work in PrimeNovo (https://d-nb.info/1360284540/34)—would be a valuable addition. Investigating how the method performs on PTM peptides could reveal its applicability to more complex proteomic scenarios.\n3. Attention map and feature vector similarity study would benefit the study.\n\nLIMITATIONS:\nIn summary, the paper’s key strength lies in its presentation of an effective approach that integrates autoregressive (AR) and non-autoregressive (NAR) architectures within a single model, leading to significant performance gains. A notable weakness, however, is the lack of clarity regarding the uniqueness or novelty of this hybrid AR/NAR framework when applied specifically to the peptide sequencing task.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}