BACKGROUND
in recent years, the non-parametric wilcoxon rank sum test  based statistical analysis of urine proteome profiles provided by capillary zone electrophoresis on-line coupled to electrospray ionization time-of-flight mass spectrometry  lead to the discovery of biomarker patterns e.g.
for the diagnosis of coronary artery disease   <cit> .
in these studies, support vector machine  modelling was used to establish disease classifiers based on these biomarker patterns.
however, it is unclear if this is the best possible approach.
besides critical technical aspects using ce-ms and study design  <cit> , biomarker detection in these urine proteome profiles depends on computing statistical analysis of high-dimensional datasets while dealing with often limited statistical power due to rather small sample sizes.
a small sample size in comparison to the number of variables causes statistical algorithms to overfit the data.
this presents challenges for the statistical analysis that must be addressed as good as possible to yield effective and reliable biomarkers.
the purpose of this study was to compare the effectiveness of different statistical methods for urinary biomarker discovery as well as the performance of classifiers established by different modelling approaches in respect of the diagnosis of coronary artery disease in  <dig> study subjects and the prognostication of acute coronary syndromes in  <dig> study subjects.
for biomarker discovery and modelling as well as validation of the classifier performance, study subjects were assigned to sub-cohorts.
the statistical methods for biomarker discovery included correlation-adjusted t-scores , introduced by zuber and strimmer  <cit> , to address potential correlations among peptides/proteins.
methods
ethics approval and consent to participate
all studies contributing samples to this new study were originally approved by local ethics committees, are in keeping with the principles of the declaration of helsinki and all participants originally gave written informed consent.
all datasets derived from studies mentioned above were pooled from the database at mosaiques diagnostics gmbh, hanover, germany.
the current study was approved by the local ethics committee at the medical school hanover, approval number 3184- <dig>
study population
the different computational statistical methods applied for the discovery of proteomic biomarkers were compared in a combined multi-centre cohort.
this cohort included individuals with known symptomatic or unknown asymptomatic cad with and without an incident of acute myocardial infarction  within up to 11 years post urine sampling and randomly selected suitable controls without cad or an ami from separate studies conducted in australia, europe and north america.
urine proteome datasets of cases and controls were compiled from four cohorts.
the first cohort comprised  <dig> patients with cad characterized by stable angina including chest pain and with at least one coronary artery stenosis ≥75% of the artery lumen and  <dig> controls without any angiographic evidence of cad.
it originated from the diagnosis of coronary artery disease with urine proteomics  study performed at the golden jubilee national hospital, clydebank, uk  <cit> .
the second cohort consisted of  <dig> proteome profile datasets from patients with severe cad requiring elective coronary artery bypass grafting  and  <dig> healthy volunteers without evidence of cad as controls from the vascular function in coronary artery bypass  study performed at the glasgow bhf cardiovascular research centre  <cit> .
the third cohort included  <dig> patients with an incident ami and  <dig> subjects without a cv event during a follow-up period of 11 years from the australian diabetes, obesity and lifestyle  study  <cit> .
the fourth cohort comprised  <dig> patients with an incident ami and  <dig> subjects without a cv event during a follow-up period of 6 years from the coronary artery calcification in type  <dig> diabetes  study  <cit> .
from the ausdiab and cacti cohorts,  <dig>  out of the  <dig> patients with an incident ami event during the observation period  had a previous history of angina pectoris and/or ami.
in the other  <dig> patients, ami was the first cardiac event.
out of the  <dig> subjects without an event during follow-up ,  <dig>  had previous symptoms of angina pectoris and/or an ami.
definition of coronary artery disease and assessment of outcome
cad was confirmed by coronary angiography.
in addition, four incident outcomes were considered, non-fatal ami , fatal ami , ami without information on fatality  and no ami  during a follow-up time up to 11 years after urine sampling.
ami was defined as having at least two of the following three features:  a typical clinical presentation,  electrocardiography changes and  cardiac enzymes rises  compliant with world health organisation monica criteria for myocardial infarction.
fatal ami was defined from death certificate coding, using international classification of diseases version  <dig>  codes i20-i <dig>
proteome profiles
all urine proteome profiles originating from previous studies  were pooled from our database at mosaiques diagnostics gmbh, hanover, germany.
the proteome profiles were based on ce-ms analysis performed by mosaiques diagnostics gmbh, and they had passed all quality control criteria  <cit> .
briefly, sample storage and preparation followed established standard operating procedures as described previously  <cit> .
ce-ms analyses also followed standard operating procedures using a p/ace mdq capillary electrophoresis system  on-line coupled to a microtof ms  as described previously  <cit> .
accuracy, precision, selectivity, sensitivity, reproducibility, and stability of the ce-ms measurements were demonstrated elsewhere  <cit> .
mass spectral peaks representing identical molecules at different charge states were deconvoluted into single masses using mosaiquesvisu software  <cit> .
for normalization of analytical and urine dilution variances, signal intensities were normalized relative to  <dig> “housekeeping” peptides  <cit> .
all detected peptides were deposited, matched, and annotated in a microsoft sql database allowing further statistical analysis  <cit>  and sequenced as described elsewhere  <cit> .
cohorts for the detection of biomarkers
in each of the  <dig> study cohorts, subjects were assigned either to peptide biomarker discovery or validation set by a  <dig> to  <dig> ratio, respectively, as presented in table  <dig>  the discovery datasets were further grouped for the discovery of potential biomarkers for the diagnosis of significant to severe cad , the prognostication of future incidents of ami  and the combined diagnosis of cad and prognostication of its outcome as an ami  .
none of the subjects used as controls for peptide biomarker discovery had a known cardiovascular condition.table  <dig> numbers of study subjects in cohorts for biomarker discovery and validation
fig.
<dig> study design for biomarker identification and validation.
cadd, coronary artery disease diagnosis; amip, acute myocardial infarction prognostication; acd, combined coronary artery disease diagnosis and outcome  prognostication; wt, non-parametric wilcoxon rank sum test, bda, binary discriminant analysis; rf, random forests; svm, support vector machine; dda, diagonal discriminant analysis; lda, linear discriminant analysis
statistical procedures for the discovery of biomarkers
non-parametric wilcoxon rank sum test
: in wt analysis only peptides that were present at a frequency of 70% or higher in either case or control group were considered as potential biomarkers.
the false discovery rate adjustments of benjamini-hochberg  <cit>  were employed to correct for multiple testing.
a p-value <  <dig>  was considered to be statistically significant.
t-score: the t-statistic was used to identify differentially expressed peptides.
local false discovery rate was applied to account for multiple testing.
correlation adjusted t-score: the previous two scores do not take into account that different peptides in the high-dimensional proteomic datasets may not be independent of each other, e.g.
if peptides originate from the same protein.
to this end,  <cit>  we introduced the correlation-adjusted t-score .
the p-values were adjusted using the local false discovery rate.
random forests : random forests are used extensively in the literature.
diaz-uriarte and de andres  <cit>  introduced a method that combines rf and variable selection.
this method was applied in the gene selection context and is extended in the proteomics context in this paper.
the contribution of the work of diaz-uriarte and de andres was two-fold: first, they introduce an iterative procedure to perform variable selection.
at each step and for each tree, they discarded the features with the smallest variable importance factor.
then, they re-grew their tree with the remaining variables.
this can also be seen as a backward elimination procedure.
their second contribution was to evaluate the stability of their results using bootstrap.
calle et al.
<cit>  stated that the method of diaz-uriarte and de andres might not work well when the number of control and cases is unbalanced.
classifier modelling
in addition to the different statistical attempts for biomarker discovery, we also assessed different methods for disease classifier modelling in respect to the diagnosis of significant to severe cad , the prognostication of future incidents of ami  and the combined diagnosis of cad and prognostication of its outcome as an ami .
linear discriminant analysis
: linear discriminant analysis reduces the dimensionality of the data, while preserving the structure that discriminates between the different groups.
lda assumes that the data have been generated from a mixture of multivariate normal distributions, where the covariance matrix is the same across the different components.
however the assumption of normality is not restrictive.
since the number of proteins is much larger than the number of samples, regularisation methods have to be applied.
this is done in order to avoid computational problems with the matrix operations.
ahdesmaeki and strimmer  <cit>  utilised james-stein shrinkage methods to address the problem.
an additional benefit from using this procedure is that one can obtain analytically the regularisation parameters without having to employ cross-validation.
feature selection is performed using the cat-score.
diagonal discriminant analysis
: diagonal discriminant analysis is a special case of lda.
dda assumes that each feature is independent .
this assumption, although simplistic, has been proven to work well even in high-dimensional settings.
t-score is the optimal statistic to perform feature selection in this setting.
binary discriminant analysis
: gibb and strimmer  <cit>  developed a binary discriminant analysis method for the identification of differential protein expression.
as the name suggests, the protein intensities were converted to binary according to a data-dependent thresholding procedure.
since the task is linked with the construction of a classifier, informative proteins were selected based on their ability to distinguish between the two groups.
a limitation of the method is that no multiple testing corrections can be applied.
random forests
: another popular machine learning method used for classification is random forests .
the method was introduced by  <cit>  and belongs to the class of ensemble learning classifiers.
a large number of classification trees are grown and their results are averaged.
in this context, all the variables obtained from the method of diaz-uriarte and de andres are plugged in a rf classifier.
support vector machine modelling
: support vector machine  is a supervised learning algorithm for two-group classification belonging to the family of margin-based classifiers <cit> .
compared with other supervised classification algorithms , svm has the highest potential to perform well in terms of average classification accuracy, time required for training and memory usage on high dimensional noisy data typical of biomedical datasets  <cit> .
as a result, svm is widely used for a number of machine learning applications in the life sciences, including for biomarker discovery in proteomics datasets  <cit> .
thus, identified peptide biomarkers were combined into single multi-dimensional classifiers, using the support-vector machine based mosacluster software, version  <dig> .
<dig>  <cit> .
mosacluster calculates classification scores based on the amplitudes of the biomarkers selected.
classification is performed by determining the euclidian distance  of the vector to a maximal margin hyperplane.
library for support vector machines
for this work, the library for support vector machines  was used  <cit> .
libsvm is an integrated, open source machine-learning library that implements the sequential minimal optimization  algorithm for svm training  <cit> .
input datasets consisted of urinary peptidomic readouts from combined patient cohorts as described in the methods section above.
the steps followed to perform the svm modelling from these input datasets were as follows: a) for a given patient cohort, define the binary response variable against which the model is developed; b) assign cohort members into balanced discovery  and validation  sets; c) prepare the peptidomics data  through transformation into the correct non-categorical format , and scaling to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges; d) configure the svm software through selection of an appropriate kernel, and optimization of the kernel parameters; radial basis function kernel was used because of its accurate and reliable performance, and kernel parameters  run the svm software on the training set to generate an initial classifier; f) iteratively exclude peptides features with the lowest contribution  to the svm until the optimal classifier in terms of receiver operating characteristic area under the curve  with the smallest number of peptides is obtained.
this is a “wrapper type” approach to feature selection  <cit> .
the accuracy of the initial svm classifier from all peptide features and the final classifier following feature selection were calculated using the validation  sets.
statistical methods and sample classification
we determined optimal threshold criterion for the classifiers to differentiate control individuals from individuals with cad and/or individuals who experienced an incident ami based on maximized younden’s index determined by receiver operating characteristic  analysis carried out in medcalc version  <dig> .
<dig>  .
the roc plot was obtained and the area under the roc curve  was evaluated.
areas under the curve  and their 95% confidence intervals  were determined based on the optimal threshold criterion.
RESULTS
biomarker discovery
biomarkers were detected by computing the grouped proteomic biomarker discovery datasets for cadd, amip and acd mentioned in the methods section  based on wt, t-score, cat score, bda or rf .
demographic and clinical characteristics of these selected cases and controls are presented in table  <dig>  the biomarkers detected by the different statistical approaches are listed in additional file 1: table s <dig> for cadd, table s <dig> for amip and table s <dig> for acd.
these biomarkers were selected based on a p-value <  <dig>   or local false discovery rate < <dig>  , while the stability of the results obtained with rf was checked by performing bootstrap .table  <dig> demographic and clinical data of subjects in the biomarker discovery cohort
n.a.
not available; diabetes, type  <dig> except in cacti where it is type i
firstly, we observed that the numbers of potential peptide biomarkers for cadd, amip and acd differed greatly between the different discovery methods.
besides the numbers of potential peptide biomarkers also the kind of potential biomarkers differed.
out of in total  <dig> peptide biomarkers discovered for cadd, only four were detected by all biomarker discovery approaches  .
for  <dig> of these sequence information was available revealing that two peptides originated from type iii collagen and one originated from type i collagen.
the biggest overlap was observed between t-score  and bda  based biomarker discovery reaching an overlap of  <dig> peptides.
in amip analysis, the numbers of discovered peptide biomarkers were much lower.
the t-score analysis did not identify any biomarkers.
moreover, in amip no biomarkers were commonly discovered with the remaining  <dig> methods.
in acd only one potential biomarker was detected by all discovery approaches out of  <dig> potential biomarkers discovered in total.
this biomarker, a type iii collagen fragment with the sequence spgergetgppgp, was also one of the four biomarkers commonly discovered for cadd.fig.
<dig> identified biomarkers using different statistical approaches.
cadd, coronary artery disease diagnosis; amip, acute myocardial infarction prognostication; acd, combined coronary artery disease diagnosis and outcome  prognostication
classifier modelling
the peptide biomarker patterns discovered by wt, cat score, t-score, bda or rf were used in svm, lda, dda, bda and rf modelling of classifiers for cadd, amip and acd.
while in lda, dda, bda and rf modelling specific biomarkers were only combined with the directly related urinary proteomic profiles  to model a specific classifier, in svm all possible combinations of the specific biomarker patterns have been used for modelling different classifiers.
this included patterns generated by selecting all peptides present in either at least three  or two  out of the wt, cat scores, t-score and rf patterns for each diagnostic/prognostic purpose.
however, for amip no classifier could be modelled for the wt and t-score due to the low number, or absence, of significant peptides.
since the number of biomarkers discovered for amip was limited regardless of the statistical approach, we also generated a svm-modelled classifier utilizing a biomarker pattern including all the biomarkers discovered by the different statistical approaches.
the same was done for acd.
based on libsvm, three classifiers were built from the urinary peptidomic dataset for the following multi-centre cohorts: 1) patients already diagnosed with coronary artery disease .
2) patients diagnosed with acute coronary syndrome  within 5 years after sample collection.
3) patients belonging to the previous two groups .
validation of classifiers
all modelled classifiers were validated by assessing the proteomic validation datasets mentioned in the methods section .
the discriminatory power characterized by the area under the receiver operating characteristics curve  for the different classifiers modelled by svm, dda, lda, bda and rf based on the biomarkers patterns for cadd, amip and acd are shown in table  <dig>   <dig> and  <dig>  respectively.
when comparing the performance of classifiers established by different modelling methods utilizing the same biomarker patterns discovered by either t-score, cat-score, bda or rf, no significant differences were observed between classifiers modelled by dda, lda, bda or rf compared to classifiers modelled by svm.
however, when comparing the performance of classifiers established by svm for amip utilizing the different biomarker patterns discovered by either cat-score, bda or rf, the classifier based on the pattern discovered by bda was significantly superior .
moreover wt and t-score did not provide a usable pattern at all.
such superiority of a method was not observed for cadd and acd.
the classifiers modelled by libsvm did, when applied to the appropriate matched validation cohorts, not provide superior performance .table  <dig> diagnostic performance of classifiers modelled by svm, dda, lda, bda and rf for cadd
the values shown are the areas under the curve of receiver operating characteristic  curve analyses
cadd coronary artery disease diagnosis, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests
*p <  <dig>  for cadd wt + rf + bda vs. cadd t-score and cadd ≥ 3
the values shown are the areas under the curve of receiver operating characteristic  curve analyses
amip acute myocardial infarction prognostication, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests, bm biomarker; ≥  <dig>  biomarkers present in at least  <dig> out of the  <dig> biomarker patterns resulting from the different discovery approaches; ≥  <dig>  biomarkers present in at least  <dig> out of the  <dig> biomarker patterns resulting from the different discovery approaches
* p <  <dig>  for amip bda vs. amip cat-score, amip ≥  <dig> and amip ≥ 2
the values shown are the areas under the curve of receiver operating characteristic  curve analyses
acd combined coronary artery disease diagnosis and outcome  prognostication, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests, bm biomarker; ≥  <dig>  biomarkers present in at least  <dig> out of the  <dig> biomarker patterns resulting from the different discovery approaches; ≥  <dig>  biomarkers present in at least  <dig> out of the  <dig> biomarker patterns resulting from the different discovery approaches
the values shown are the areas under the curve of receiver operating characteristic  curve analyses
cadd coronary artery disease diagnosis, amip acute myocardial infarction prognostication, acd combined coronary artery disease diagnosis and outcome  prognostication, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests, bm biomarker; ≥  <dig>  biomarkers present in at least  <dig> out of the  <dig> biomarker patterns resulting from the different discovery approaches; ≥  <dig>  biomarkers present in at least  <dig> out of the  <dig> biomarker patterns resulting from the different discovery approaches
y years, auc area under the curve of a receiver operating characteristic  curve analysis
of the svm-modelled classifiers based on the integration of biomarkers discovered by different computational approaches for a specific diagnostic approach  into combined patterns, only the ones providing an auc above  <dig>  in respect of their diagnostic/prognostic performance are shown.
as the aucs shown below the dashed lines in tables  <dig>   <dig> and  <dig> illustrate, the discriminatory power of these classifiers was mostly similar to the performance of classifiers based on single biomarker patterns.
only for cadd the classifier based on the pattern combining the patterns of wt, bda and rf significantly outperformed some of the other classifiers .
we further assessed the performance of classifiers based on the integration of biomarkers discovered by different computational approaches for cadd and amip  as an alternative to the biomarker patterns discovered for acd.
while we observed trends towards better diagnostic and prognostic performance for cadd, amip and acd, none of the differences in performance were significant.
however, independently of the classifier, the discriminatory power in respect of ami prediction is much better for the period of  <dig> to 5 years than the one for the period of  <dig> to 11 years.
discussion
the results of this study showed clearly how challenging the statistical analysis of complex high-dimensional proteomic datasets for the identification of reliable disease-associated biomarkers is, even based on only one outcome variable.
this was first illustrated by the observed differences in the patterns of potential biomarkers and the resulting low overall overlap between the biomarker patterns discovered based on wt, t-score, cat-score, rf and bda .
the observation that almost no single statistical computational approach for biomarker discovery provided a biomarker pattern and thus a disease classifier that allowed for a robust significantly superior diagnostic/prognostic classification of patients, further underlined this.
this was true even when taking correlations between peptides into account by selecting biomarkers based on correlation-adjusted t-scores followed by classifier modelling by linear discriminant analysis.
even the integration of biomarkers discovered by different statistical approaches into master-patterns did not provide clearly superior classifiers.
interestingly, the only peptides of the cadd and acd biomarker patterns that were detected by all statistical discovery approaches were fragments of the fibrillar type i and iii collagens.
these collagens are part of the extracellular matrix  surrounding endothelial cells in the tunica intima of blood vessels walls and contribute to the composition of the three dimensional network of vascular smooth muscle cell , fibronectin and proteoglycan-rich layers of the tunica media as well as the composition of the fibroblast-rich tunica adventitia  <cit> .
both, type i and iii collagens are further responsible for the strength and integrity of the fibrous cap of atherosclerotic plaques and contribute to the modulation of cellular responses within it .
initial accumulation of collagens as part of the fibrotic remodelling associated atherosclerosis  <cit>  thus confers stability to the whole plaque.
type i collagen can thereby comprise approximately 60% of the total protein content of an atherosclerotic plaque and plays, in addition to proteoglycans, an active role in lipid retention  <cit> .
later atherogenic alterations of the ecm weaken the fibrous cap and thus contribute to plaque destabilization  <cit> .
CONCLUSIONS
in conclusion, our study revealed a very considerable dependence of peptide biomarker discovery on statistical computing of urinary peptide profiles while the observed diagnostic and/or prognostic reliability of classifiers was widely independent of the modelling approach.
this may however be due to the limited statistical power in classifier testing.
nonetheless, our study showed that urinary proteome analysis has the potential to provide valuable biomarkers for coronary artery disease mirroring especially alterations in the extracellular matrix.
it further shows that for a comprehensive discovery of biomarkers and thus of pathological information, the results of different statistical methods should be combined into a master pattern that then can be used for classifier modelling.
additional file
additional file 1:
table s <dig>  urinary peptides identified for cad diagnosis .
table s <dig>  urinary peptides identified for ami prediction .
table s <dig>  urinary peptides identified for the combined diagnosis of cad and prognostication of its outcome as an ami .
abbreviations
acdcombined diagnosis of cad and prognostication of its outcome as an ami
amiacute myocardial infarction
amipacute myocardial infarction prediction
aucarea under the curve
ausdiabaustralian diabetes, obesity and lifestyle study
bdabinary discriminant analysis
cacticoronary artery calcification in type  <dig> diabetes study
cadcoronary artery disease
caddcoronary artery disease diagnosis
cecapillary zone electrophoresis
ce-mscapillary zone electrophoresis on-line coupled to electrospray ionization time-of-flight mass spectrometry
ddadiagonal discriminant analysis
dicadudiagnosis of coronary artery disease with urine proteomics study
ecmextracellular matrix
ldalinear discriminant analysis
libsvmlibrary for support vector machine
monicamonitoring trends and determinants of cardiovascular disease
rfrandom forests
rocreceiver operating characteristic
svmsupport vector machine
vascabvascular function in coronary artery bypass study
vsmcvascular smooth muscle cells
wtnon-parametric wilcoxon rank sum test

 in recent years, the non-parametric wilcoxon rank sum test based statistical analysis of urine proteome profiles provided by capillary zone electrophoresis on-line coupled to electrospray ionization time-of-flight mass spectrometry lead to the discovery of biomarker patterns e.g. for the diagnosis of coronary artery disease. in these studies, support vector machine modelling was used to establish disease classifiers based on these biomarker patterns. however, it is unclear if this is the best possible approach. besides critical technical aspects using ce-ms and study design, biomarker detection in these urine proteome profiles depends on computing statistical analysis of high-dimensional datasets while dealing with often limited statistical power due to rather small sample sizes. a small sample size in comparison to the number of variables causes statistical algorithms to overfit the data. this presents challenges for the statistical analysis that must be addressed as good as possible to yield effective and reliable biomarkers. the purpose of this study was to compare the effectiveness of different statistical methods for urinary biomarker discovery as well as the performance of classifiers established by different modelling approaches in respect of the diagnosis of coronary artery disease in study subjects and the prognostication of acute coronary syndromes in study subjects. for biomarker discovery and modelling as well as validation of the classifier performance, study subjects were assigned to sub-cohorts. the statistical methods for biomarker discovery included correlation-adjusted t-scores, introduced by zuber and strimmer, to address potential correlations among peptides/proteins. methods ethics approval and consent to participate all studies contributing samples to this new study were originally approved by local ethics committees, are in keeping with the principles of the declaration of helsinki and all participants originally gave written informed consent. all datasets derived from studies mentioned above were pooled from the database at mosaiques diagnostics gmbh, hanover, germany. the current study was approved by the local ethics committee at the medical school hanover, approval number 3184- study population the different computational statistical methods applied for the discovery of proteomic biomarkers were compared in a combined multi-centre cohort. this cohort included individuals with known symptomatic or unknown asymptomatic cad with and without an incident of acute myocardial infarction within up to 11 years post urine sampling and randomly selected suitable controls without cad or an ami from separate studies conducted in australia, europe and north america. urine proteome datasets of cases and controls were compiled from four cohorts. the first cohort comprised patients with cad characterized by stable angina including chest pain and with at least one coronary artery stenosis ≥75% of the artery lumen and controls without any angiographic evidence of cad. it originated from the diagnosis of coronary artery disease with urine proteomics study performed at the golden jubilee national hospital, clydebank, uk. the second cohort consisted of proteome profile datasets from patients with severe cad requiring elective coronary artery bypass grafting and healthy volunteers without evidence of cad as controls from the vascular function in coronary artery bypass study performed at the glasgow bhf cardiovascular research centre. the third cohort included patients with an incident ami and subjects without a cv event during a follow-up period of 11 years from the australian diabetes, obesity and lifestyle study. the fourth cohort comprised patients with an incident ami and subjects without a cv event during a follow-up period of 6 years from the coronary artery calcification in type diabetes study. from the ausdiab and cacti cohorts, out of the patients with an incident ami event during the observation period had a previous history of angina pectoris and/or ami. in the other patients, ami was the first cardiac event. out of the subjects without an event during follow-up, had previous symptoms of angina pectoris and/or an ami. definition of coronary artery disease and assessment of outcome cad was confirmed by coronary angiography. in addition, four incident outcomes were considered, non-fatal ami, fatal ami, ami without information on fatality and no ami during a follow-up time up to 11 years after urine sampling. ami was defined as having at least two of the following three features: a typical clinical presentation, electrocardiography changes and cardiac enzymes rises compliant with world health organisation monica criteria for myocardial infarction. fatal ami was defined from death certificate coding, using international classification of diseases version codes i20-i proteome profiles all urine proteome profiles originating from previous studies were pooled from our database at mosaiques diagnostics gmbh, hanover, germany. the proteome profiles were based on ce-ms analysis performed by mosaiques diagnostics gmbh, and they had passed all quality control criteria. briefly, sample storage and preparation followed established standard operating procedures as described previously. ce-ms analyses also followed standard operating procedures using a p/ace mdq capillary electrophoresis system on-line coupled to a microtof ms as described previously. accuracy, precision, selectivity, sensitivity, reproducibility, and stability of the ce-ms measurements were demonstrated elsewhere. mass spectral peaks representing identical molecules at different charge states were deconvoluted into single masses using mosaiquesvisu software. for normalization of analytical and urine dilution variances, signal intensities were normalized relative to “housekeeping” peptides. all detected peptides were deposited, matched, and annotated in a microsoft sql database allowing further statistical analysis and sequenced as described elsewhere. cohorts for the detection of biomarkers in each of the study cohorts, subjects were assigned either to peptide biomarker discovery or validation set by a to ratio, respectively, as presented in table  the discovery datasets were further grouped for the discovery of potential biomarkers for the diagnosis of significant to severe cad, the prognostication of future incidents of ami and the combined diagnosis of cad and prognostication of its outcome as an ami. none of the subjects used as controls for peptide biomarker discovery had a known cardiovascular condition.table numbers of study subjects in cohorts for biomarker discovery and validation fig. study design for biomarker identification and validation. cadd, coronary artery disease diagnosis; amip, acute myocardial infarction prognostication; acd, combined coronary artery disease diagnosis and outcome prognostication; wt, non-parametric wilcoxon rank sum test, bda, binary discriminant analysis; rf, random forests; svm, support vector machine; dda, diagonal discriminant analysis; lda, linear discriminant analysis statistical procedures for the discovery of biomarkers non-parametric wilcoxon rank sum test: in wt analysis only peptides that were present at a frequency of 70% or higher in either case or control group were considered as potential biomarkers. the false discovery rate adjustments of benjamini-hochberg were employed to correct for multiple testing. a p-value <  was considered to be statistically significant. t-score: the t-statistic was used to identify differentially expressed peptides. local false discovery rate was applied to account for multiple testing. correlation adjusted t-score: the previous two scores do not take into account that different peptides in the high-dimensional proteomic datasets may not be independent of each other, e.g. if peptides originate from the same protein. to this end, we introduced the correlation-adjusted t-score. the p-values were adjusted using the local false discovery rate. random forests: random forests are used extensively in the literature. diaz-uriarte and de andres introduced a method that combines rf and variable selection. this method was applied in the gene selection context and is extended in the proteomics context in this paper. the contribution of the work of diaz-uriarte and de andres was two-fold: first, they introduce an iterative procedure to perform variable selection. at each step and for each tree, they discarded the features with the smallest variable importance factor. then, they re-grew their tree with the remaining variables. this can also be seen as a backward elimination procedure. their second contribution was to evaluate the stability of their results using bootstrap. calle et al. stated that the method of diaz-uriarte and de andres might not work well when the number of control and cases is unbalanced. classifier modelling in addition to the different statistical attempts for biomarker discovery, we also assessed different methods for disease classifier modelling in respect to the diagnosis of significant to severe cad, the prognostication of future incidents of ami and the combined diagnosis of cad and prognostication of its outcome as an ami. linear discriminant analysis: linear discriminant analysis reduces the dimensionality of the data, while preserving the structure that discriminates between the different groups. lda assumes that the data have been generated from a mixture of multivariate normal distributions, where the covariance matrix is the same across the different components. however the assumption of normality is not restrictive. since the number of proteins is much larger than the number of samples, regularisation methods have to be applied. this is done in order to avoid computational problems with the matrix operations. ahdesmaeki and strimmer utilised james-stein shrinkage methods to address the problem. an additional benefit from using this procedure is that one can obtain analytically the regularisation parameters without having to employ cross-validation. feature selection is performed using the cat-score. diagonal discriminant analysis: diagonal discriminant analysis is a special case of lda. dda assumes that each feature is independent. this assumption, although simplistic, has been proven to work well even in high-dimensional settings. t-score is the optimal statistic to perform feature selection in this setting. binary discriminant analysis: gibb and strimmer developed a binary discriminant analysis method for the identification of differential protein expression. as the name suggests, the protein intensities were converted to binary according to a data-dependent thresholding procedure. since the task is linked with the construction of a classifier, informative proteins were selected based on their ability to distinguish between the two groups. a limitation of the method is that no multiple testing corrections can be applied. random forests: another popular machine learning method used for classification is random forests. the method was introduced by and belongs to the class of ensemble learning classifiers. a large number of classification trees are grown and their results are averaged. in this context, all the variables obtained from the method of diaz-uriarte and de andres are plugged in a rf classifier. support vector machine modelling: support vector machine is a supervised learning algorithm for two-group classification belonging to the family of margin-based classifiers. compared with other supervised classification algorithms, svm has the highest potential to perform well in terms of average classification accuracy, time required for training and memory usage on high dimensional noisy data typical of biomedical datasets. as a result, svm is widely used for a number of machine learning applications in the life sciences, including for biomarker discovery in proteomics datasets. thus, identified peptide biomarkers were combined into single multi-dimensional classifiers, using the support-vector machine based mosacluster software, version.. mosacluster calculates classification scores based on the amplitudes of the biomarkers selected. classification is performed by determining the euclidian distance of the vector to a maximal margin hyperplane. library for support vector machines for this work, the library for support vector machines was used. libsvm is an integrated, open source machine-learning library that implements the sequential minimal optimization algorithm for svm training. input datasets consisted of urinary peptidomic readouts from combined patient cohorts as described in the methods section above. the steps followed to perform the svm modelling from these input datasets were as follows: a) for a given patient cohort, define the binary response variable against which the model is developed; b) assign cohort members into balanced discovery and validation sets; c) prepare the peptidomics data through transformation into the correct non-categorical format, and scaling to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges; d) configure the svm software through selection of an appropriate kernel, and optimization of the kernel parameters; radial basis function kernel was used because of its accurate and reliable performance, and kernel parameters run the svm software on the training set to generate an initial classifier; f) iteratively exclude peptides features with the lowest contribution to the svm until the optimal classifier in terms of receiver operating characteristic area under the curve with the smallest number of peptides is obtained. this is a “wrapper type” approach to feature selection. the accuracy of the initial svm classifier from all peptide features and the final classifier following feature selection were calculated using the validation sets. statistical methods and sample classification we determined optimal threshold criterion for the classifiers to differentiate control individuals from individuals with cad and/or individuals who experienced an incident ami based on maximized younden’s index determined by receiver operating characteristic analysis carried out in medcalc version.. the roc plot was obtained and the area under the roc curve was evaluated. areas under the curve and their 95% confidence intervals were determined based on the optimal threshold criterion.  biomarker discovery biomarkers were detected by computing the grouped proteomic biomarker discovery datasets for cadd, amip and acd mentioned in the methods section based on wt, t-score, cat score, bda or rf. demographic and clinical characteristics of these selected cases and controls are presented in table  the biomarkers detected by the different statistical approaches are listed in additional file 1: table s for cadd, table s for amip and table s for acd. these biomarkers were selected based on a p-value <  or local false discovery rate <, while the stability of the results obtained with rf was checked by performing bootstrap.table demographic and clinical data of subjects in the biomarker discovery cohort n.a. not available; diabetes, type except in cacti where it is type i firstly, we observed that the numbers of potential peptide biomarkers for cadd, amip and acd differed greatly between the different discovery methods. besides the numbers of potential peptide biomarkers also the kind of potential biomarkers differed. out of in total peptide biomarkers discovered for cadd, only four were detected by all biomarker discovery approaches. for of these sequence information was available revealing that two peptides originated from type iii collagen and one originated from type i collagen. the biggest overlap was observed between t-score and bda based biomarker discovery reaching an overlap of peptides. in amip analysis, the numbers of discovered peptide biomarkers were much lower. the t-score analysis did not identify any biomarkers. moreover, in amip no biomarkers were commonly discovered with the remaining methods. in acd only one potential biomarker was detected by all discovery approaches out of potential biomarkers discovered in total. this biomarker, a type iii collagen fragment with the sequence spgergetgppgp, was also one of the four biomarkers commonly discovered for cadd.fig. identified biomarkers using different statistical approaches. cadd, coronary artery disease diagnosis; amip, acute myocardial infarction prognostication; acd, combined coronary artery disease diagnosis and outcome prognostication classifier modelling the peptide biomarker patterns discovered by wt, cat score, t-score, bda or rf were used in svm, lda, dda, bda and rf modelling of classifiers for cadd, amip and acd. while in lda, dda, bda and rf modelling specific biomarkers were only combined with the directly related urinary proteomic profiles to model a specific classifier, in svm all possible combinations of the specific biomarker patterns have been used for modelling different classifiers. this included patterns generated by selecting all peptides present in either at least three or two out of the wt, cat scores, t-score and rf patterns for each diagnostic/prognostic purpose. however, for amip no classifier could be modelled for the wt and t-score due to the low number, or absence, of significant peptides. since the number of biomarkers discovered for amip was limited regardless of the statistical approach, we also generated a svm-modelled classifier utilizing a biomarker pattern including all the biomarkers discovered by the different statistical approaches. the same was done for acd. based on libsvm, three classifiers were built from the urinary peptidomic dataset for the following multi-centre cohorts: 1) patients already diagnosed with coronary artery disease. 2) patients diagnosed with acute coronary syndrome within 5 years after sample collection. 3) patients belonging to the previous two groups. validation of classifiers all modelled classifiers were validated by assessing the proteomic validation datasets mentioned in the methods section. the discriminatory power characterized by the area under the receiver operating characteristics curve for the different classifiers modelled by svm, dda, lda, bda and rf based on the biomarkers patterns for cadd, amip and acd are shown in table  and respectively. when comparing the performance of classifiers established by different modelling methods utilizing the same biomarker patterns discovered by either t-score, cat-score, bda or rf, no significant differences were observed between classifiers modelled by dda, lda, bda or rf compared to classifiers modelled by svm. however, when comparing the performance of classifiers established by svm for amip utilizing the different biomarker patterns discovered by either cat-score, bda or rf, the classifier based on the pattern discovered by bda was significantly superior. moreover wt and t-score did not provide a usable pattern at all. such superiority of a method was not observed for cadd and acd. the classifiers modelled by libsvm did, when applied to the appropriate matched validation cohorts, not provide superior performance.table diagnostic performance of classifiers modelled by svm, dda, lda, bda and rf for cadd the values shown are the areas under the curve of receiver operating characteristic curve analyses cadd coronary artery disease diagnosis, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests *p <  for cadd wt + rf + bda vs. cadd t-score and cadd ≥ 3 the values shown are the areas under the curve of receiver operating characteristic curve analyses amip acute myocardial infarction prognostication, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests, bm biomarker; ≥ biomarkers present in at least out of the biomarker patterns resulting from the different discovery approaches; ≥ biomarkers present in at least out of the biomarker patterns resulting from the different discovery approaches * p <  for amip bda vs. amip cat-score, amip ≥  and amip ≥ 2 the values shown are the areas under the curve of receiver operating characteristic curve analyses acd combined coronary artery disease diagnosis and outcome prognostication, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests, bm biomarker; ≥ biomarkers present in at least out of the biomarker patterns resulting from the different discovery approaches; ≥ biomarkers present in at least out of the biomarker patterns resulting from the different discovery approaches the values shown are the areas under the curve of receiver operating characteristic curve analyses cadd coronary artery disease diagnosis, amip acute myocardial infarction prognostication, acd combined coronary artery disease diagnosis and outcome prognostication, svm support vector machine, dda diagonal discriminant analysis, lda linear discriminant analysis, bda binary discriminant analysis, rf random forests, bm biomarker; ≥ biomarkers present in at least out of the biomarker patterns resulting from the different discovery approaches; ≥ biomarkers present in at least out of the biomarker patterns resulting from the different discovery approaches y years, auc area under the curve of a receiver operating characteristic curve analysis of the svm-modelled classifiers based on the integration of biomarkers discovered by different computational approaches for a specific diagnostic approach into combined patterns, only the ones providing an auc above in respect of their diagnostic/prognostic performance are shown. as the aucs shown below the dashed lines in tables  and illustrate, the discriminatory power of these classifiers was mostly similar to the performance of classifiers based on single biomarker patterns. only for cadd the classifier based on the pattern combining the patterns of wt, bda and rf significantly outperformed some of the other classifiers. we further assessed the performance of classifiers based on the integration of biomarkers discovered by different computational approaches for cadd and amip as an alternative to the biomarker patterns discovered for acd. while we observed trends towards better diagnostic and prognostic performance for cadd, amip and acd, none of the differences in performance were significant. however, independently of the classifier, the discriminatory power in respect of ami prediction is much better for the period of to 5 years than the one for the period of to 11 years. discussion the results of this study showed clearly how challenging the statistical analysis of complex high-dimensional proteomic datasets for the identification of reliable disease-associated biomarkers is, even based on only one outcome variable. this was first illustrated by the observed differences in the patterns of potential biomarkers and the resulting low overall overlap between the biomarker patterns discovered based on wt, t-score, cat-score, rf and bda. the observation that almost no single statistical computational approach for biomarker discovery provided a biomarker pattern and thus a disease classifier that allowed for a robust significantly superior diagnostic/prognostic classification of patients, further underlined this. this was true even when taking correlations between peptides into account by selecting biomarkers based on correlation-adjusted t-scores followed by classifier modelling by linear discriminant analysis. even the integration of biomarkers discovered by different statistical approaches into master-patterns did not provide clearly superior classifiers. interestingly, the only peptides of the cadd and acd biomarker patterns that were detected by all statistical discovery approaches were fragments of the fibrillar type i and iii collagens. these collagens are part of the extracellular matrix surrounding endothelial cells in the tunica intima of blood vessels walls and contribute to the composition of the three dimensional network of vascular smooth muscle cell, fibronectin and proteoglycan-rich layers of the tunica media as well as the composition of the fibroblast-rich tunica adventitia. both, type i and iii collagens are further responsible for the strength and integrity of the fibrous cap of atherosclerotic plaques and contribute to the modulation of cellular responses within it. initial accumulation of collagens as part of the fibrotic remodelling associated atherosclerosis thus confers stability to the whole plaque. type i collagen can thereby comprise approximately 60% of the total protein content of an atherosclerotic plaque and plays, in addition to proteoglycans, an active role in lipid retention. later atherogenic alterations of the ecm weaken the fibrous cap and thus contribute to plaque destabilization.  in conclusion, our study revealed a very considerable dependence of peptide biomarker discovery on statistical computing of urinary peptide profiles while the observed diagnostic and/or prognostic reliability of classifiers was widely independent of the modelling approach. this may however be due to the limited statistical power in classifier testing. nonetheless, our study showed that urinary proteome analysis has the potential to provide valuable biomarkers for coronary artery disease mirroring especially alterations in the extracellular matrix. it further shows that for a comprehensive discovery of biomarkers and thus of pathological information, the results of different statistical methods should be combined into a master pattern that then can be used for classifier modelling. additional file additional file 1: table s urinary peptides identified for cad diagnosis. table s urinary peptides identified for ami prediction. table s urinary peptides identified for the combined diagnosis of cad and prognostication of its outcome as an ami. abbreviations acdcombined diagnosis of cad and prognostication of its outcome as an ami amiacute myocardial infarction amipacute myocardial infarction prediction aucarea under the curve ausdiabaustralian diabetes, obesity and lifestyle study bdabinary discriminant analysis cacticoronary artery calcification in type diabetes study cadcoronary artery disease caddcoronary artery disease diagnosis cecapillary zone electrophoresis ce-mscapillary zone electrophoresis on-line coupled to electrospray ionization time-of-flight mass spectrometry ddadiagonal discriminant analysis dicadudiagnosis of coronary artery disease with urine proteomics study ecmextracellular matrix ldalinear discriminant analysis libsvmlibrary for support vector machine monicamonitoring trends and determinants of cardiovascular disease rfrandom forests rocreceiver operating characteristic svmsupport vector machine vascabvascular function in coronary artery bypass study vsmcvascular smooth muscle cells wtnon-parametric wilcoxon rank sum test 
in conclusion, our study revealed a very considerable dependence of peptide biomarker discovery on statistical computing of urinary peptide profiles while the observed diagnostic and/or prognostic reliability of classifiers was widely independent of the modelling approach. the purpose of this study was thus to evaluate the effectiveness of different statistical methods applied for urinary proteomic biomarker discovery and different methods of classifier modelling in respect of the diagnosis of coronary artery disease in study subjects and the prognostication of acute coronary syndromes in study subjects. nonetheless, our study showed that urinary proteome analysis has the potential to provide valuable biomarkers for coronary artery disease mirroring especially alterations in the extracellular matrix. computing the discovery sub-cohorts comprising of the study subjects based on the wilcoxon rank sum test, t-score, cat-score, binary discriminant analysis and random forests provided largely different numbers of potential peptide biomarkers. it further showed that for a comprehensive discovery of biomarkers and thus of pathological information, the results of different statistical methods may best be combined into a master pattern that then can be used for classifier modelling. however, these differences in biomarker patterns did mostly not translate into significant differently performing diagnostic or prognostic classifiers modelled by support vector machine, diagonal discriminant analysis, linear discriminant analysis, binary discriminant analysis and random forest. this may however be due to the limited statistical power in classifier testing. moreover, these biomarker patterns showed very little overlap limited to fragments of type i and iii collagens as the common denominator. this was even true when different biomarker patterns were combined into master-patterns.
 when combined with a clinical outcome variable, the size, complexity and nature of mass-spectrometry proteomics data impose great statistical challenges in the discovery of potential disease-associated biomarkers. the purpose of this study was thus to evaluate the effectiveness of different statistical methods applied for urinary proteomic biomarker discovery and different methods of classifier modelling in respect of the diagnosis of coronary artery disease in study subjects and the prognostication of acute coronary syndromes in study subjects.  computing the discovery sub-cohorts comprising of the study subjects based on the wilcoxon rank sum test, t-score, cat-score, binary discriminant analysis and random forests provided largely different numbers of potential peptide biomarkers. moreover, these biomarker patterns showed very little overlap limited to fragments of type i and iii collagens as the common denominator. however, these differences in biomarker patterns did mostly not translate into significant differently performing diagnostic or prognostic classifiers modelled by support vector machine, diagonal discriminant analysis, linear discriminant analysis, binary discriminant analysis and random forest. this was even true when different biomarker patterns were combined into master-patterns.  in conclusion, our study revealed a very considerable dependence of peptide biomarker discovery on statistical computing of urinary peptide profiles while the observed diagnostic and/or prognostic reliability of classifiers was widely independent of the modelling approach. this may however be due to the limited statistical power in classifier testing. nonetheless, our study showed that urinary proteome analysis has the potential to provide valuable biomarkers for coronary artery disease mirroring especially alterations in the extracellular matrix. it further showed that for a comprehensive discovery of biomarkers and thus of pathological information, the results of different statistical methods may best be combined into a master pattern that then can be used for classifier modelling. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords statistical proteome analysisbiomarker detectionclassifier modellingfp- healthfp- health 603228fp- health 603228kolch walter schanstra joost p. fp- health fp- health 603228mischak harald issue-copyright-statement© the author 2016 