BACKGROUND
existing methods for differential proteomics  fall into two categories: spectral counting methods that rely on counting the number of spectra that map to a given protein across multiple experiments, and peptide chromatographic peak intensity methods that use the area under the peptide precursor ion peak as a measure of peptide abundance.
in principle, methods based on mass spectrometry peak areas are potentially much more accurate, but these methods require highly reproducible liquid chromatography as well as accurate methods for chromatographic alignment and identification of peaks within the profile spectra.
in contrast, spectral counting methods are straightforward to employ and have been shown to correctly detect known differences between samples
<cit> , which contributes to their wide use.
the command line tool crux spectral-counts implements four popular spectral counting methods: the spectral index
<cit> , the exponentially modified protein abundance index
<cit> , the normalized spectral abundance factor
<cit> , and the distributed normalized spectral abundance factor
<cit> .
the crux spectral-counts command is integrated within the crux software toolkit, which provides actively maintained open-source methods to identify and now quantify peptides and proteins from shotgun mass spectrometry datasets.
crux supports a variety of input spectra formats, and the tools can easily be incorporated into proteomic analysis pipelines, such as the trans-proteomic pipeline
<cit> .
finally, the modular design of crux allows improvements to one part of the toolkit to be propagated through downstream analyses.
currently, several software packages offer spectral counting protein quantification methods
<cit> .
proteoiq  and scaffold
<cit>  are commercial software products that post-process results from a variety of database search programs.
freely available tools such as apex
<cit> , empai calc
<cit> , and pepc
<cit>  each offer a single spectral counting method.
table
<dig> compares the features of six software spectral counting tools.
crux offers more spectral counting methods than other tools and is the only method to provide peptide-level in addition to protein-level counts.
this table summarizes the features of various spectral counting software methods.
using crux spectral-counts, we compared and contrasted the reproducibility and linearity of the four spectral counting methods.
our experiments suggest that the nsaf metric provides the most reproducible protein quantification.
in contrast, our linearity experiments show that sin and nsaf provide the best performance, with dnsaf providing intermediate performance and empai yielding the worst linearity.
the contributions of this paper are thus two-fold: we describe a performance comparison of the reproducibility and linearity of the sin, empai, nsaf, and dnsaf protein quantification methods, and we provide to the proteomics community a flexible, open source spectral counting software tool.
implementation
software
the crux spectral-counts command is implemented as part of the crux proteomics software toolkit
<cit> .
the toolkit is implemented in c++ as a single binary that supports commands for database searching and a variety of downstream analyses
<cit> .
the crux spectral-counts command takes as input a protein database in fasta format and a collection of peptide-spectrum matches  produced by a database search procedure.
the psms may be in crux’s tab-delimited text format, peptideprophet’s pepxml or mzidentml
<cit> .
to compute the sin score, a set of spectra must also be provided as input in ms <dig>  mzxml, or mgf format.
by default, crux spectral-counts will select the psms in the input by a user modifiable threshold of q-value ≤  <dig> .
for each protein with at least one spectral count, the program then computes the nsaf, dnsaf, empai, or the sin score.
the nsaf metric is defined as
nsafn=sn/ln∑i=1n
where n is the protein index, sn is the number of spectra matched to protein n, ln is the length of protein n, and n is the total number of proteins in the input database.
the dnsaf metric is given by
dnsafn=snu+∑j=1kdj,nsj,nsln∑i=1nsiu+∑j=1kdj,isj,isli
where
snu is the spectral count for the peptides uniquely mapping to protein n,
sj,ns is the spectral count of degenerate peptide j  mapped to protein n, and dj,n is the distribution factor of peptide shared counts, defined by the equation
dj,n=snu∑i=1nsiu
the metric empai is defined as
empain=10pnobservedpnobservable−1∑i=1n10piobservedpiobservable− <dig>
where
pnobservable is the number of unique peptides observable for protein n and
pnobserved is the number of unique peptides observed for protein n.
finally, the sin score is calculated using
sin=∑j=1pn∑k=1sjikln∑j=1nsij
where pn is number of unique peptides in protein n, sj is the number of spectra assigned to peptide j, and ik is the total fragment ion intensity of spectrum k. analogous scores can also be computed for each peptide, rather than for each protein.
a detailed description of the peptide-level scoring metrics is available in the on-line documentation.
as output, crux spectral-counts produces a tab-delimited file listing proteins and their corresponding counts, in reverse sorted order.
the crux spectral-counts command also computes a parsimonious set of proteins, using the greedy set cover approach used by idpicker
<cit> .
users thus have the option of considering spectral counts only for proteins within the parsimonious set.
data collection
for the reproducibility experiments, proteins were extracted from the cochlear nucleus of the developing mouse brain at postnatal day  <dig> and postnatal day  <dig>  two biological replicates were generated for each age by dissecting out the cochlear nuclei from two separate mice at each age.
one of the 21-day mice was used to generate two samples, thereby providing a technical replicate in addition to a biological replicate.
the samples prepared from the chicken brain were derived from nucleus laminaris, an auditory region in the brain stem.
samples were taken from the dorsal  and ventral  regions of this area.
for each region, two biological replicates were generated, and one of those replicates was also subjected to technical replication.
each sample was digested with trypsin and subjected to liquid chromatography followed by tandem mass spectrometry.
for the linearity experiments, we used eight samples that represent a dilution curve of  <dig> known proteins synthesized by sigma .
these data sets are mixtures  of the c. elegans lysate at equal concentrations and the  <dig> proteins, diluted by a two-fold in each successive standard.
std  <dig> has the lowest concentration of the known proteins  and std  <dig> has the highest concentration .
all three data sets are publicly available at
http://noble.gs.washington.edu/proj/crux-spectral-counts.
data analysis
the fragmentation spectra from the experiments were searched against their respective mouse, chicken, or the c. elegans+ups <dig> protein database using crux sequest-search followed by crux q-ranker, with the default parameters.
crux spectral-counts was applied to the peptide-spectrum matches  that received q-values ≤  <dig> .
the resulting data sets for the mouse and chicken replicates are summarized in additional file
1: table s <dig>  and the ups <dig> dilution curve data sets are summarized in additional file
1: table s <dig>
RESULTS
testing reproducibility between replicates
to investigate the reproducibility of the four spectral count methods, we analyzed mass spectrometry data from technical and biological replicates from chicken and mouse samples.
we then produced a scatter plot for each pair of biological or technical replicates and computed the corresponding spearman correlation.
for these comparisons, proteins identified in only one of the two datasets were ignored.
figure
<dig> shows sixteen such plots, corresponding to one biological and one technical replicate for chicken and mouse, respectively.
the complete collection of  <dig> plots is provided as additional file
1: figures s1–s <dig>  from these analyses, as summarized in table
<dig>  we draw two primary conclusions.
first, the spectral counts are generally reproducible: the mean correlation value across all  <dig> pairs is  <dig> , and the minimum correlation is  <dig> .
second, reassuringly, the technical replicates produce higher correlations than the biological replicates: the mean correlation among  <dig> pairs of technical replicates is  <dig> , whereas the corresponding value for the  <dig> pairs of biological replicates is  <dig>  .
this table summarizes the average correlation of the spectral-counting metrics across the technical and biological replicates.
to test whether the observed differences in correlations among the four metrics are significant, we applied a wilcoxon signed-rank test to paired sets of correlations.
with four metrics, there are six possible paired comparisons.
figure
<dig> shows the results of this analysis, where one metric attaining a significant increase  over another is indicated by a directed edge.
from this graph we conclude that, for the biological and technical replicates, nsaf yields significantly more reproducible quantification values than sin, dnsaf and empai.
our reproducibility results agree with colaert et al., who claim that nsaf is more reproducible than sin and empai
<cit> .
however, in contrast to our results, griffen et al.
report better reproducibility across replicates for sin compared to nsaf
<cit> .
testing linear response for protein abundance across samples
to determine the linear response of each of the spectral count metrics, we analyzed mass spectra from a dataset of samples that form a dilution curve of forty-eight proteins with known amounts spiked into a c. elegans lysate.
we performed linear regression between each protein spectral count and the associated amounts across the dilution curve samples.
for this analysis, we only included proteins that obtain a positive spectral count in three or more of the data sets, which results in a comparison of forty-two proteins across the four metrics.
we then carried out a wilcoxon signed rank test analysis separately on the average correlation, r <dig>  and the mean percent error .
the results of these tests  are fairly consistent with one another: nsaf significantly outperforms dnsaf, and dnsaf and sinsignificantly outperform empai.
colaert et al.
claim that sin is more accurate than both nsaf and empai
<cit> , but we find evidence only to support the former claim, even though our experiments employ a wider dynamic range of protein abundance  and more data sets .
based on our experiments, we conclude that nsaf or sin are the methods of choice for ensuring an accurate linear response between a protein’s change in abundance across different samples.
it is worth noting that griffin et al.
observe a good linear fit between sin and protein quantification.
however, their evaluation methodology fits a single line to all of the sin values from many proteins, whereas we have fit a separate line for each protein.
this difference reflects our belief that spectral counting methods are most useful as measures of the relative abundance of a single protein between two experiments.
we did not test the claim that sin provides an accurate absolute protein abundance metric.
discussion
overall, our experiments suggest a relative ordering of spectral counting methods according to their reproducibility and the linearity of their response, but we can only speculate as to the reasons for the ranking that we observe.
for example, we note that nsaf outperforms the empai metric in both of our experiments.
the empai measure takes into account the least information—not only does it ignore fragment ion intensities, but empai also fails to account for the length of the protein.
apparently, this relatively simple approach is insufficient to accurately estimate protein abundance.
the relative performance of nsaf and sin, on the other hand, is less clear: nsaf yields more reproducible results than sin but the two methods are statistically indistinguishable with respect to linearity.
the main difference between sin and the other three metrics is that sin is the only metric that takes into account the intensities of the fragment ion peaks.
in this sense, sin goes a bit beyond the strict definition of “spectral counting.” our experiments do not support the claim that such intensity information is valuable for quantification.
however, the conflicting results of our study and collaert et al., on the one hand, versus griffin et al.
on the other hand, suggests perhaps that further comparison of these methods is warranted.
an additional direction for future work involves quantifying the linearity and reproducibility of proteins in a segregated fashion according to protein abundance.
for example, visual inspection of figure
<dig> suggests that perhaps the sin measure yields more reproducible counts for high abundance proteins, with a corresponding decrease in reproducibility as the abundance drops.
arguably, in many studies, such low abundance proteins are of the greatest interest; hence, it may be worthwhile to investigate in a systematic fashion the extent to which either the linearity or the reproducibility of a given spectral counting measure varies as a function of protein abundance.
CONCLUSIONS
quantifying protein amounts in mass spectrometry by spectral counting is a simple and robust method for measuring the relative change of protein amounts across different samples; however, many different algorithms exist for assigning a score to each identified protein.
using crux spectral-counts, we compared and contrasted four spectral counting methods with respect to their reproducibility across replicates and their linear response relative to protein abundance.
crux provides a flexible, easy to use open source tool for performing protein quantification using spectral counting.
availability and requirements
project name: crux tandem mass spectrometry analysis software
project home page:http://noble.gs.washington.edu/proj/crux
operating systems: linux, macos, windows + cygwin
programming language: c++
other requirements: crux has no requirements to install the binary version under linux or macos.
on windows, crux requires cygwin.
to compile crux requires a c++ compiler, cmake, and subversion.
license: apache
any restrictions to use by non-academics: none
competing interests
the authors declare that they have no competing interests.
authors’ contributions
the chicken and mouse samples were provided by er’s lab, and the lc-ms/ms data were collected by members of the mm lab.
mb prepared and collected the ups <dig> + c. elegans dilution sample datasets.
mm wrote the initial code for crux spectral-counts and the initial draft of the manuscript.
sm finished the coding of crux spectral-counts and the final draft with wsn.
all authors revised and approved the final manuscript.
supplementary material
additional file 1
supplementary information.
supplementary tables  <dig> and  <dig> and suplementary figures  <dig> and  <dig> are provided as quantify-supplement.pdf.
click here for file
acknowledgements
nih awards r <dig> eb <dig>  p <dig> gm <dig> and r <dig> dc <dig>  the authors acknowledge karl schweighofer for his input on the crux spectral-counts tool and the anonymous reviewers for many helpful suggestions.

 existing methods for differential proteomics fall into two categories: spectral counting methods that rely on counting the number of spectra that map to a given protein across multiple experiments, and peptide chromatographic peak intensity methods that use the area under the peptide precursor ion peak as a measure of peptide abundance. in principle, methods based on mass spectrometry peak areas are potentially much more accurate, but these methods require highly reproducible liquid chromatography as well as accurate methods for chromatographic alignment and identification of peaks within the profile spectra. in contrast, spectral counting methods are straightforward to employ and have been shown to correctly detect known differences between samples, which contributes to their wide use. the command line tool crux spectral-counts implements four popular spectral counting methods: the spectral index, the exponentially modified protein abundance index, the normalized spectral abundance factor, and the distributed normalized spectral abundance factor. the crux spectral-counts command is integrated within the crux software toolkit, which provides actively maintained open-source methods to identify and now quantify peptides and proteins from shotgun mass spectrometry datasets. crux supports a variety of input spectra formats, and the tools can easily be incorporated into proteomic analysis pipelines, such as the trans-proteomic pipeline. finally, the modular design of crux allows improvements to one part of the toolkit to be propagated through downstream analyses. currently, several software packages offer spectral counting protein quantification methods. proteoiq and scaffold are commercial software products that post-process results from a variety of database search programs. freely available tools such as apex, empai calc, and pepc each offer a single spectral counting method. table compares the features of six software spectral counting tools. crux offers more spectral counting methods than other tools and is the only method to provide peptide-level in addition to protein-level counts. this table summarizes the features of various spectral counting software methods. using crux spectral-counts, we compared and contrasted the reproducibility and linearity of the four spectral counting methods. our experiments suggest that the nsaf metric provides the most reproducible protein quantification. in contrast, our linearity experiments show that sin and nsaf provide the best performance, with dnsaf providing intermediate performance and empai yielding the worst linearity. the contributions of this paper are thus two-fold: we describe a performance comparison of the reproducibility and linearity of the sin, empai, nsaf, and dnsaf protein quantification methods, and we provide to the proteomics community a flexible, open source spectral counting software tool. implementation software the crux spectral-counts command is implemented as part of the crux proteomics software toolkit. the toolkit is implemented in c++ as a single binary that supports commands for database searching and a variety of downstream analyses. the crux spectral-counts command takes as input a protein database in fasta format and a collection of peptide-spectrum matches produced by a database search procedure. the psms may be in crux’s tab-delimited text format, peptideprophet’s pepxml or mzidentml. to compute the sin score, a set of spectra must also be provided as input in ms mzxml, or mgf format. by default, crux spectral-counts will select the psms in the input by a user modifiable threshold of q-value ≤. for each protein with at least one spectral count, the program then computes the nsaf, dnsaf, empai, or the sin score. the nsaf metric is defined as nsafn=sn/ln∑i=1n where n is the protein index, sn is the number of spectra matched to protein n, ln is the length of protein n, and n is the total number of proteins in the input database. the dnsaf metric is given by dnsafn=snu+∑j=1kdj,nsj,nsln∑i=1nsiu+∑j=1kdj,isj,isli where snu is the spectral count for the peptides uniquely mapping to protein n, sj,ns is the spectral count of degenerate peptide j mapped to protein n, and dj,n is the distribution factor of peptide shared counts, defined by the equation dj,n=snu∑i=1nsiu the metric empai is defined as empain=10pnobservedpnobservable−1∑i=1n10piobservedpiobservable− where pnobservable is the number of unique peptides observable for protein n and pnobserved is the number of unique peptides observed for protein n. finally, the sin score is calculated using sin=∑j=1pn∑k=1sjikln∑j=1nsij where pn is number of unique peptides in protein n, sj is the number of spectra assigned to peptide j, and ik is the total fragment ion intensity of spectrum k. analogous scores can also be computed for each peptide, rather than for each protein. a detailed description of the peptide-level scoring metrics is available in the on-line documentation. as output, crux spectral-counts produces a tab-delimited file listing proteins and their corresponding counts, in reverse sorted order. the crux spectral-counts command also computes a parsimonious set of proteins, using the greedy set cover approach used by idpicker. users thus have the option of considering spectral counts only for proteins within the parsimonious set. data collection for the reproducibility experiments, proteins were extracted from the cochlear nucleus of the developing mouse brain at postnatal day and postnatal day two biological replicates were generated for each age by dissecting out the cochlear nuclei from two separate mice at each age. one of the 21-day mice was used to generate two samples, thereby providing a technical replicate in addition to a biological replicate. the samples prepared from the chicken brain were derived from nucleus laminaris, an auditory region in the brain stem. samples were taken from the dorsal and ventral regions of this area. for each region, two biological replicates were generated, and one of those replicates was also subjected to technical replication. each sample was digested with trypsin and subjected to liquid chromatography followed by tandem mass spectrometry. for the linearity experiments, we used eight samples that represent a dilution curve of known proteins synthesized by sigma. these data sets are mixtures of the c. elegans lysate at equal concentrations and the proteins, diluted by a two-fold in each successive standard. std has the lowest concentration of the known proteins and std has the highest concentration. all three data sets are publicly available at http://noble.gs.washington.edu/proj/crux-spectral-counts. data analysis the fragmentation spectra from the experiments were searched against their respective mouse, chicken, or the c. elegans+ups protein database using crux sequest-search followed by crux q-ranker, with the default parameters. crux spectral-counts was applied to the peptide-spectrum matches that received q-values ≤. the resulting data sets for the mouse and chicken replicates are summarized in additional file 1: table s and the ups dilution curve data sets are summarized in additional file 1: table s  testing reproducibility between replicates to investigate the reproducibility of the four spectral count methods, we analyzed mass spectrometry data from technical and biological replicates from chicken and mouse samples. we then produced a scatter plot for each pair of biological or technical replicates and computed the corresponding spearman correlation. for these comparisons, proteins identified in only one of the two datasets were ignored. figure shows sixteen such plots, corresponding to one biological and one technical replicate for chicken and mouse, respectively. the complete collection of plots is provided as additional file 1: figures s1–s from these analyses, as summarized in table we draw two primary conclusions. first, the spectral counts are generally reproducible: the mean correlation value across all pairs is, and the minimum correlation is. second, reassuringly, the technical replicates produce higher correlations than the biological replicates: the mean correlation among pairs of technical replicates is, whereas the corresponding value for the pairs of biological replicates is. this table summarizes the average correlation of the spectral-counting metrics across the technical and biological replicates. to test whether the observed differences in correlations among the four metrics are significant, we applied a wilcoxon signed-rank test to paired sets of correlations. with four metrics, there are six possible paired comparisons. figure shows the results of this analysis, where one metric attaining a significant increase over another is indicated by a directed edge. from this graph we conclude that, for the biological and technical replicates, nsaf yields significantly more reproducible quantification values than sin, dnsaf and empai. our reproducibility results agree with colaert et al., who claim that nsaf is more reproducible than sin and empai. however, in contrast to our results, griffen et al. report better reproducibility across replicates for sin compared to nsaf. testing linear response for protein abundance across samples to determine the linear response of each of the spectral count metrics, we analyzed mass spectra from a dataset of samples that form a dilution curve of forty-eight proteins with known amounts spiked into a c. elegans lysate. we performed linear regression between each protein spectral count and the associated amounts across the dilution curve samples. for this analysis, we only included proteins that obtain a positive spectral count in three or more of the data sets, which results in a comparison of forty-two proteins across the four metrics. we then carried out a wilcoxon signed rank test analysis separately on the average correlation, r and the mean percent error. the results of these tests are fairly consistent with one another: nsaf significantly outperforms dnsaf, and dnsaf and sinsignificantly outperform empai. colaert et al. claim that sin is more accurate than both nsaf and empai, but we find evidence only to support the former claim, even though our experiments employ a wider dynamic range of protein abundance and more data sets. based on our experiments, we conclude that nsaf or sin are the methods of choice for ensuring an accurate linear response between a protein’s change in abundance across different samples. it is worth noting that griffin et al. observe a good linear fit between sin and protein quantification. however, their evaluation methodology fits a single line to all of the sin values from many proteins, whereas we have fit a separate line for each protein. this difference reflects our belief that spectral counting methods are most useful as measures of the relative abundance of a single protein between two experiments. we did not test the claim that sin provides an accurate absolute protein abundance metric. discussion overall, our experiments suggest a relative ordering of spectral counting methods according to their reproducibility and the linearity of their response, but we can only speculate as to the reasons for the ranking that we observe. for example, we note that nsaf outperforms the empai metric in both of our experiments. the empai measure takes into account the least information—not only does it ignore fragment ion intensities, but empai also fails to account for the length of the protein. apparently, this relatively simple approach is insufficient to accurately estimate protein abundance. the relative performance of nsaf and sin, on the other hand, is less clear: nsaf yields more reproducible results than sin but the two methods are statistically indistinguishable with respect to linearity. the main difference between sin and the other three metrics is that sin is the only metric that takes into account the intensities of the fragment ion peaks. in this sense, sin goes a bit beyond the strict definition of “spectral counting.” our experiments do not support the claim that such intensity information is valuable for quantification. however, the conflicting results of our study and collaert et al., on the one hand, versus griffin et al. on the other hand, suggests perhaps that further comparison of these methods is warranted. an additional direction for future work involves quantifying the linearity and reproducibility of proteins in a segregated fashion according to protein abundance. for example, visual inspection of figure suggests that perhaps the sin measure yields more reproducible counts for high abundance proteins, with a corresponding decrease in reproducibility as the abundance drops. arguably, in many studies, such low abundance proteins are of the greatest interest; hence, it may be worthwhile to investigate in a systematic fashion the extent to which either the linearity or the reproducibility of a given spectral counting measure varies as a function of protein abundance.  quantifying protein amounts in mass spectrometry by spectral counting is a simple and robust method for measuring the relative change of protein amounts across different samples; however, many different algorithms exist for assigning a score to each identified protein. using crux spectral-counts, we compared and contrasted four spectral counting methods with respect to their reproducibility across replicates and their linear response relative to protein abundance. crux provides a flexible, easy to use open source tool for performing protein quantification using spectral counting. availability and requirements project name: crux tandem mass spectrometry analysis software project home page:http://noble.gs.washington.edu/proj/crux operating systems: linux, macos, windows + cygwin programming language: c++ other requirements: crux has no requirements to install the binary version under linux or macos. on windows, crux requires cygwin. to compile crux requires a c++ compiler, cmake, and subversion. license: apache any restrictions to use by non-academics: none competing interests the authors declare that they have no competing interests. authors’ contributions the chicken and mouse samples were provided by er’s lab, and the lc-ms/ms data were collected by members of the mm lab. mb prepared and collected the ups + c. elegans dilution sample datasets. mm wrote the initial code for crux spectral-counts and the initial draft of the manuscript. sm finished the coding of crux spectral-counts and the final draft with wsn. all authors revised and approved the final manuscript. supplementary material additional file 1 supplementary information. supplementary tables and and suplementary figures and are provided as quantify-supplement.pdf. click here for file acknowledgements nih awards r eb p gm and r dc the authors acknowledge karl schweighofer for his input on the crux spectral-counts tool and the anonymous reviewers for many helpful suggestions.
the crux spectral-counts command, implemented as part of the crux software toolkit, implements four previously reported spectral counting methods, the spectral index, the exponentially modified protein abundance index, the normalized spectral abundance factor, and the distributed normalized spectral abundance factor. our analysis suggests that nsaf yields the most reproducible counts across technical and biological replicates, and both sin and nsaf achieve the best linearity. with the crux spectral-counts command, crux provides open-source modular methods to analyze mass spectrometry data for identifying and now quantifying peptides and proteins.
 spectral counting methods provide an easy means of identifying proteins with differing abundances between complex mixtures using shotgun proteomics data. the crux spectral-counts command, implemented as part of the crux software toolkit, implements four previously reported spectral counting methods, the spectral index, the exponentially modified protein abundance index, the normalized spectral abundance factor, and the distributed normalized spectral abundance factor.  we compared the reproducibility and the linearity relative to each protein’s abundance of the four spectral counting metrics. our analysis suggests that nsaf yields the most reproducible counts across technical and biological replicates, and both sin and nsaf achieve the best linearity.  with the crux spectral-counts command, crux provides open-source modular methods to analyze mass spectrometry data for identifying and now quantifying peptides and proteins. the c++ source code, compiled binaries, spectra and sequence databases are available at http://noble.gs.washington.edu/proj/crux-spectral-counts. 