PepBenchmark: A Standardized Benchmark for Peptide Machine Learning
Authors: Jiahui Zhang
Rouyi Wang
Kuangqi Zhou
Tianshu Xiao
Lingyan Zhu
Yaosen Min
Yang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: peptide machine learning
benchmark
protein language models
Primary area: datasets and benchmarks
TL;DR: We introduce PepBenchmark, a standardized framework with datasets, preprocessing, and evaluation tools that establish the first reproducible benchmark for peptide machine learning.

Abstract: Peptide therapeutics are widely regarded as the “third generation” of drugs, yet progress in peptide Machine Learning (ML) are hindered by the absence of standardized benchmarks. Here we present \textbf{PepBenchmark}, which unifies datasets, preprocessing, and evaluation protocols for peptide drug discovery. PepBenchmark comprises three components: (1) \textbf{PepBenchData}, a well-curated collection comprising 29 canonical-peptide and 6 non-canonical-peptide datasets across 7 groups, systematically covering key aspects of peptide drug development—representing, to the best of our knowledge, the most comprehensive AI-ready dataset resource to date; (2) \textbf{PepBenchPipeline}, a standardized preprocessing pipeline that ensures consistent dataset cleaning, construction, splitting, and feature transformation, mitigating quality issues common in ad hoc pipelines; and (3) \textbf{PepBenchLeaderboard}, a unified evaluation protocol and leaderboard with strong baselines across 4 major methodological families: Fingerprint-based, GNN-based, PLM-based, and SMILES-based models. Together, PepBenchmark provides the first standardized and comparable foundation for peptide drug discovery, facilitating methodological advances and translation into real-world applications. The data and code are publicly available at \href{https://github.com/ZGCI-AI4S-Pep/PepBenchmark/}{\texttt{https://github.com/ZGCI-AI4S-Pep/PepBenchmark/}}.


Reviews:
[{"review_id": "dTG2x21nLq", "reviewer": "Reviewer_HNKA", "summary": "This paper introduces PepBenchmark, a standardized benchmark for peptide machine learning that unifies diverse datasets(PepBenchData), standardized preprocessing(PepBenchPipeline), and unified evaluation protocols(PepBenchLeaderboard) across diverse peptide-related tasks.", "questions": "1. How are graphs built for the GNN-based models?\n2. How does distribution-controlled sampling change training and test results? Does matching five key properties between positive and negative sets (length, charge, hydrophobicity, molecular weight, isoelectric point) improve early enrichment and calibration?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper presents the largest AI-ready peptide database, integrating 23 datasets across 7 pharmacological tasks, which supports a wide range of predictive applications in peptide therapeutics.\n- The biologically-informed, distribution-controlled negative sampling strategy ensures that decoy peptides are generated in a realistic and un-biased manner.\n- The benchmark systematically evaluates multiple model families and provide detailed insights into how each type of model performs across different peptide-related tasks.\n- The benchmark explicitly addresses k-mer leakage and sequence redundancy", "weaknesses": "1. The benchmark is sequence-centric, which can understate GNN baselines that need reliable 3D/contact graphs.\n2. More strong PLM embedders should be benchmarked (for example, ESM-2, ProtT5, and MSA Transformer), since they output fixed embeddings for prediction tasks. [1]\n3. Add language models that can process non-canonical peptides (for example, GPepT) to cover tasks with modified residues. [2]\n4. An ablation of the biologically informed, distribution-controlled negative sampling would show how much this strategy drives the gains.\n\n\n[1] Zhang, R., et al. “Evaluating the advancements in protein language models for encoding protein sequences.” *Frontiers in Bioengineering and Biotechnology*, 2025. \n\n[2] Oikawa, Y., et al. “GPepT: A Foundation Language Model for Peptidomimetics Incorporating Non-canonical Amino Acids.” *ACS Medicinal Chemistry Letters*, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBhblKtjCW", "reviewer": "Reviewer_26PZ", "summary": "The paper presents PepBenchmark, a standardized benchmark for peptide machine learning. It integrates three components: PepBenchData, a curated collection of 35 datasets (29 canonical, 6 non-canonical) across 7 pharmacological tasks; PepBenchPipeline, a preprocessing framework featuring biologically informed negative sampling and hybrid data splitting to prevent leakage; and PepBenchLeaderboard, a unified evaluation across four model families: fingerprint, GNN, SMILES, and PLM. Experiments show that Protein Language Models (PLMs) outperform others, with peptide-specific finetuning and scaling improving performance. Fingerprint models remain strong for small and non-canonical datasets, while GNN and SMILES models excel in peptide–protein interaction tasks. PepBenchmark provides the first reproducible foundation for systematic peptide ML evaluation and model comparison.", "questions": "1. In Lines 185–186, the authors state that all non-canonical peptides are converted to SMILES. How exactly was this conversion performed, and does it result in any **loss of structural or chemical information**?\n\n2. In Lines 234–245, regarding **MMSeq**, what **exact command** and **sensitivity parameter** were used during clustering or filtering?\n\n3. For **“Task correlation filtering,”** the paper mentions using expert prior knowledge and statistical analysis to estimate correlations among tasks and exclude closely related ones. Could the authors clarify **what specific expert knowledge** and **which statistical methods** were applied in this process?\n\n4. The paper notes that the model takes a canonical peptide as input and generates a chemically modified non-canonical peptide, allowing transformation of a canonical negative set into a non-canonical one. Please provide **more details about this generative model**, including its architecture, training data, and how the modifications are controlled or validated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Comprehensive scale: 35 datasets covering over 78k sequences across canonical, non-canonical, and interaction tasks enable unified evaluation.\n\n2. Rigorous preprocessing: New negative sampling and hybrid-split methods reduce leakage and overestimation of model performance.\n\n3. Empirical insight: Systematic benchmarking across 4 model types and 30 tasks reveals clear scaling laws and practical model guidance for peptide ML.", "weaknesses": "1. Authors might want to inlcude these works in the related works section\n- https://www.nature.com/articles/s42256-023-00691-9\n- https://www.nature.com/articles/s42004-025-01601-3\n- https://www.nature.com/articles/s41587-025-02761-2\n- https://arxiv.org/abs/2410.19222?\n\n2. For data splitting, the authors use sequence-level splits. However, since some evaluated methods are graph-based or structure-based, a more rigorous approach might consider structural or graph similarity during splitting. At minimum, the authors could quantify the graph or structural similarity within and across splits or clusters to assess potential data leakage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eeg0a7bfDA", "reviewer": "Reviewer_XKRu", "summary": "The paper provides a collection of standard benchmarks and leaderboards for peptide representation learning. The authors collected and clean a large set of peptides including both canonical and non-canonical peptides. Negative sampling was proposed to extend the peptide datasets for machine learning.  Different splitting strategies were considered, including spliting by kmers and then spltting by sequence similarity using clustering methods. The authors consider different representations of the peptides including simple fingerprints, pretrained smiles representation, pretrained PLM representation (for canonical sequences) and difefrent learning approaches including tree-based ML methods like RF and LightGBM, graph neural networks and finetuning pretrained smiles or PLM models. The datasets, the splits and the benchmark results are reported on the rich datasets. Some conclusions were drawn from the experimental results based on the benchmark experiments.", "questions": "Could you please explain how the proposed methods of negative sampling would be better in related approach in resolving the false negative issues in the benchmark data?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In peptide representation, there is lack of standard leaderboards with fixed splits so that research in the field can rely on whenever there is a requirement for comparison between different methods. The proposed benchmark and frameworks are useful for that purpose.\n\nThe dataset cover broad types of tasks, even the datasets were collected elsewhere, the collection and bringing them to the same place will trigger standard benchmarking for the research field.\n\nThe conclusion and benchmarking results are interesting even it is known in the field that current not only limitted to non-canonical peptide in general for small molecules, fingerprint-based approaches are the best representation. For canonical representation PLM is the best. This is not new as previous work that the authors cited also had such similar results.", "weaknesses": "Although the datasets and the benchmark is useful, I think the technical contribution of the work is limited for the machine learning community. The paper may fits better a specific dataset and benchmark tracks rather a research main track.\n\nRegarding the methods for collecting negative samples, the authors criticizes related work regarding false negatives possibility but their own method neither resolve that issue, I don't see how the proposed approach in the paper can help resolving the false negative may happen in the collected data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)