Package: ibench
iBenchBenchmarking Mass Spectrometry Identification Methods.iBench is a tool to help you understand the performance of an mass spectrometry identification methodTo use iBench you will need:A source of ground truth identifications which you are confident in.MS data for the ground truth identifications (mgf or mzML format)A proteome fasta file to modify.An identification method or set of method which you wish to benchmark.In the example used in this data we haveGround Truth Identifications : Synthetic Peptides identified at 1% FDR by both MaxQuant and PEAKS.MS data : mgf files for the Synthetic Peptide MS measurments.Fasta File : This file comes from the expressed proteome of K562 cell line.Identification Method(s) : We provide output files for Mascot search and percolator rescoring for 3 different feature sets.CitePlease cite the following article if you are using iBench in your research:Cormican, J. A., Soh, W. T., Mishto, M., and Liepe, J. (2022) iBench: A ground truth approach for advanced validation of mass spectrometry identification method. Proteomics, e2200271.doi.org/10.1002/pmic.202200271Set UpBefore DownloadingYou will also require conda to use the iBench software.Setting up your environment:To start with create a new conda environment with python version 3.8:conda create --name ibench python=3.8Activate this environmentconda activate ibenchYou will then need to install the iBench package:pip install ibenchTo check your installation, run the following command (it is normal for this call to hang for a few seconds on first execution)ibench -hOnce you have successfully installed iBench you must run it specifying your pipeline and a config file.iBench ExecutioniBench is executed with two command line arguments --pipeline which specifies the pipeline to be executed.iBench Pipeline: createDBThe first option for the pipeline is "createDB" which processes the ground truth identifications to produce the ground truth datasets, artificial reference database and reindexed MS files needed to benchmark performance against.iBench Pipeline: analysisAfter running the first pipeline the user should apply their identification method to the reindexed MS files and artificial reference database. These identifications can be provided to iBench for the "analysis" pipeline. This processes the data and produces a report containing multiple figures describing each method's performance.iBench Config FileThe second argument provided to iBench is --config_file. This specifies the location of a yaml file which contains all the meta data needed to run the iBench software. Such data includes locations of search results, original MS data, as well as user preferences on the creation of the database and results output. Full details of the configuration settings possible are given in the table at the bottom of this README and an example config file is provided used in the "Running a Small Example" section.Running a Small ExampleAs an example we will demonstrate how iBench could be used to benchmark Mascot with Percolator Rescoring with 3 different feature sets. This example will use ground truth identifications from PEAKS and MaxQuant searches of our synthetic peptide library. Before running this example you will need to download the necessary data with:ibench --pipeline downloadExampleThis will create a folder called example in your working directory which will allow you to run the iBench example. Alternatively, you could download the required data fromfigshare.Create the Ground Truth DatasetTo create the ground truth datasets:ibench --config_file example/config.yml --pipeline createDBNext StepsTo provide a working example we have performed a Mascot search on the output data and providedRunning the AnalysisYou will nowibench --config_file example/config.yml --pipeline analysisThis will create a html report with all relevant plots to compare the performance of the 3 pipelines.Full List of iBench Config SettingsThis provides all of the configuration setting both optional and required for each pipeline.Required for any ExecutioniBench will always require an identifier for your experiment and an output folder.KeyDescriptionidentifierSome identifier or experiment title for your iBench analysis.outputFolderThe folder into which all iBench outputs will be written.Required for createDB ExecutionThe following config settings can be usedKeyDescriptionscanFolderA folder containing all of the MS files.scanFormatThe format of the MS files (either mgf or mzML).searchResultsA list of all of the search results used. The search results should be a list of outputs from PEAKS, MaxQuant, Mascot, or Percolator, see more details below.canonicalFractionThe fraction of the ground truth identifications which should be embedded as canonical sequences in the artificial reference.cissplicedFractionThe fraction of the ground truth identifications which should be embedded as cisspliced sequences in the artificial reference.transsplicedFractionThe fraction of the ground truth identifications which should be transspliced or trapping sequences in the artificial reference (not discoverable as canonical or cisspliced).ms2AccuracyThe m/z accuracy on the measurement of the MS2 spectrum (needed for calculating coverage and signal to noise features).enzymeThe enzyme used to produce the peptides, can be set to trypsin, otherwise the default value is None (unspecific digestion).proteomeThe location of a proteome fasta file which will be modified to produce the artificial reference database.Optional for createDB ExecutionThe following config settings can be used for specific user requirements in the createDB pipeline.KeyDescriptionclosenessCutOffThe minimum number of residues difference between the shorter of any pair of peptides and their longest shared subsequence (default is 1 if no cisspliced peptides are required and 3 if cisspliced peptides are being used.randomSeedThe random seed to use to ensure reproducibility of iBench execution (default=42).maxSequenceLengthDefine the maximum sequence length of peptides to be used in the ground truth identifications.minSequenceLengthDefine the minimum sequence length of peptides to be used in the ground truth identifications.filterPTMsWhether to filter modified peptides from the ground truth identificaitons, by default this will be True to ensure maximum confidence in the ground truth dataset, however it can be set to False if the user wishes to investigate PTMs.maxInterveningIf iBench is being used to embed spliced peptides this defines the maximum allowed intervening sequence length between splice reactants (default=25).Required for analysis ExecutionThe following config settings can be usedKeyDescriptionscanFolderA folder containing all of the MS files.scanFormatThe format of the MS files (either mgf or mzML).benchmarkResultsA list of all of the search results to benchmark. The search results should be a list of outputs from PEAKS, MaxQuant, Mascot, or Percolator, see more details below.Specifying Search ResultsWhen providing search results, iBench requires more information than a location. To see how this should be formated in yaml see the example/config.yml file.For ground truth identifications you will require:KeyDescriptionsearchEnginemascot, peaks, percolator, or maxquantresultsLocationThe location of the results file.scoreLimitThe score above which we should consider PSMs for ground truth identifications.qValueLimitOnly for mascot or percolator. The q-value below which we should consider PSMs for ground truth identifications.identificationGroupIf you are using multiple search engines for the original search and only wish to use PSMs identified by both, assign them to the same identificationGroup. Otherwise ensure that all results belong to separate identification groups.For benchmarking results you will require:KeyDescriptionnameThe name of your identification method.searchEnginemascot, peaks, percolator, or maxquant.resultsLocationThe location of the results file.decoyLocation(optional) The location of your decoy PSMs if using Percolator or Mascot.colourThe colour you want to identify the method with in the iBench output plots. Can be any CSS colour.