BACKGROUND
the use of gene expression profiling for the classification of human cancers has been widely investigated.
previous works were successful in predicting tumor types in the context of binary problems.
many algorithms for feature extraction and sample classification have been proposed  <cit> .
more recently, a method for addressing the potential mislabeling in the training set was proposed for binary classification of cancer samples  <cit> .
as there are over a hundred types of cancers, and potentially even more subtypes  <cit> , it is essential to develop multi-category methodologies for molecular classification for any practical application  <cit> .
multi-category prediction can be achieved using binary classification algorithms via the one-versus-one  and/or one-versus-rest  partition of the training data set.
however, in a cancer type prediction, multi-category problems proved to be more challenging than simple binary problems, and the reported results were less than satisfactory  <cit> .
on one hand, when the available resource is limited and the sample size of a given category  is small, classifiers based on the ovr partition of the data set potentially suffer from severe over-fitting, leading to low predictive ability and robustness.
furthermore, the substantial noise introduced by implementing the numerous classifiers under an ovo scheme and the asymmetric training sets caused by ovr partitioning of the data will inevitably weaken the classification system.
on the other hand, the effects of biological and technical noise together with the genetic heterogeneity of samples within a clinically defined tumor class decrease the predictive power in a multiple setting  <cit> .
in disease diagnostic, a measurement of confidence or uncertainty reported with the type determination is always desirable  <cit> .
however, some well-established statistical criteria  often become less credible and of little biological meaning for highly heterogeneous cancer types, especially in the context of multiple cancer types.
a potential reason is that the winning classifier used to discriminate one cancer type from others could be weak or unstable due to limited training samples.
although this phenomenon was alluded to in previous studies  <cit> , it has not received appropriate attention.
figure  <dig> presents a graphical illustration of the problem.
using an ovr binary classifier, all samples of a homogeneous cancer type  were classified correctly and with high confidence.
all other cancer type samples in the group have probabilities of being cancer type a close to zero .
however, the situation was very different when a heterogeneous cancer class  was considered.
in fact, some samples of cancer b type had classification probability lower than  <dig>  .
such low classification probability could lead to misdiagnosis if a hard classification rule is applied.
it is possible that such low probability is due to the weakness of the classifier that is established with a highly heterogeneous training set.
the jackknife is a well known, non-parametric method often used for estimating the sampling distribution of a statistic.
given a sample dataset and a desired statistic , the jackknife works by computing the desired statistic with an element  removed from the equation.
the process is repeated for each element in the dataset.
the application in cancer classification with gene expression profiling has been reported in the context of binary problems  <cit> .
in that study, the individual maximum difference subsets  of genes identified from a set of jackknife subsets of samples were aggregated to generate the "overall mdss" in order to return the expected classification.
in other words, jackknife was used for feature selection rather than for training multiple sub-classifiers.
in this study, a new learning method called paired-samples test algorithm , which is based on the jackknife method, was used to classify multiple tumor types using gene expression data.
the proposed method is designed for solving multi-category problems under an ovr scheme with a very limited training data set, and it is similar to the bootstrap aggregating  procedure, which proved to be helpful in improving weak classifiers  <cit> .
in order to get a relative measurement of uncertainty in the prediction of a sample category , the training sample being removed  each time was predicted together with the training samples.
the procedure was implemented in a parsimonious way, making its integration with a computationally intensive algorithm, such as the stochastic, regulation-based binary regression  <cit> , feasible.
the performance of the proposed method was evaluated under several scenarios of gene selection criteria using two well known and challenging datasets: the gcm and nci <dig> datasets containing  <dig> and  <dig> cancer tumor types, respectively.
RESULTS
determination of the optimum number of genes  to be used by the classification algorithm is usually a difficult task that depends on several factors, including the classification algorithms and the complexity of the data set.
for the used binary regression algorithm, previous studies have shown that a feature set of one to two hundred top genes is adequate for a simple two category problem  <cit> .
in this study, the size of the feature set used was  <dig>   <dig>   <dig> or  <dig> genes for the gcm dataset and  <dig>   <dig>   <dig> or  <dig> genes for the nci <dig> dataset.
gcm data
the prediction accuracy of the  <dig> validation samples, using different gene selection procedures, is summarized in figure  <dig>  the results showed that fold change and penalized t-statistic based methods for feature selection outperformed the t-statistic-based procedure.
in most cases, the application of pst improved the prediction accuracy or maintained the high accuracies that had been obtained prior to its application, except in the scenario of  <dig> genes and penalized t-statistic.
the largest improvement occurred when  <dig> genes were considered using different feature selection criteria, resulting in an increase in accuracy ranging from  <dig> % to  <dig> %.
the combination of  <dig> genes, fold change-based feature selection and pst had the highest prediction accuracy of  <dig> %.
additionally, almost 50% of the  <dig> % incorrectly classified samples had their true tumor type predicted as the second possible classification in this scenario.
it should be noted that while the largest improvements were seemingly coming from the weaker gene selection mechanisms, the application of pst made the binary regression algorithm more robust in relation to the gene selection methods and the size of the gene set to be used.
these prediction results are, in general, better than those obtained by several previous studies using the same data set .
using a recursive feature elimination procedure and a support vector machine  classification algorithm, ramaswamy et al.
obtained their best result with  <dig> tumors correctly predicted among the  <dig> test samples, corresponding to an accuracy of 78%  <cit> .
using a feature selection algorithm based on the overlaps of gene expression values between different classes in conjunction with the covering classification algorithm , a modification of the k-nn method, bagirov et al.
achieved a prediction accuracy of around 80%  <cit> .
based on the concept of gene interaction, antonov et al  proposed a maximal margin linear programming  procedure that combines linear programming  and svm  <cit> .
using mama, only eight test samples were misclassified.
although slightly superior to our method  in the overall accuracy, the lack of information about confusion profiles of the prediction and the secondary classification of non-correctly predicted samples make the direct comparison between both methods difficult.
recently, sheng and tan  reported a prediction accuracy of around 83% by using error correcting out codes , svm and a recursive feature elimination procedure  <cit> .
the output coding based approach is very costly in implementation and the result was highly sensitive to the decoding functions and the length of the random code.
a in parenthesis are the assigned tumor types for the misclassified samples.
it is possible that the superiority of the proposed method over svm and other learning algorithms could be related to the difference in gene selection methods used in this study and by ramaswamy et al  and bagirov et al   <cit> .
however, our preliminary work as well as readily available information  <cit>  demonstrated that svm outperformed k-nn, nn , pnn  and the decision tree in general does not support such a claim.
in fact, the highest accuracies obtained using svm occurred when 200- <dig> genes were selected based on fc, t-statistics, penalized t-statistics and non-parametric anova, ranging from  <dig> % to  <dig> %.
these were well below the results obtained using our approach.
as indicated in table  <dig>  it seems that some tumor types are easily predicted.
for example, ly, ut, me and csn tumors had 100% prediction accuracy using all three methods.
meanwhile, other types, such as br, had a high misclassification rate ranging from  <dig> to 75%, indicating potential excess heterogeneity.
additionally, the profile of misclassified samples was very different between the four studies.
in fact, among the four br tumors, two were misclassified as ov and pa in ramaswamy et al   <cit> , three were misclassified as lu, lu and bl in bagirov et al   <cit> , and three were misclassified as le, pa, and ut in the current study.
to further validate the results behind the use of the independent  <dig> test samples, a four-fold cross validation was conducted for the  <dig> training samples.
the results of this validation are presented in figure  <dig>  in most scenarios, the prediction accuracy was improved when the proposed jackknife method was used.
the highest value was  <dig> %, which was achieved from several combinations of the gene selection method and gene number, including the case of fc-based gene selection and  <dig> genes.
this accuracy value was similar to the best performance of  <dig> % obtained using the independent  <dig> test samples, and it is  <dig> % higher than the accuracy obtained by ramaswamy et al   <cit> .the np-anova feature selection performed marginally better in the cross validation than in the independent test with the highest prediction accuracy of  <dig> %.
ppp rediction uncertainty
uncertainty of the  <dig> correctly classified test samples from the best result is graphically presented in figure  <dig>  among these  <dig> samples, eight tumors  had high f  and nearly 3/ <dig> of the tumors had their prediction confidence <  <dig> .
for the classification algorithm used in this study, f was defined as the aggregate probability that the test sample t belongs to the assigned tumor type.
in this context, considering f alone makes the current prediction results seem unexpected.
however, when taking r values into account, confidence measurement, or f, appears to be in better agreement with the results of this study.
of those samples with lower prediction confidence, the majority had their r between - <dig>  and  <dig> , suggesting that their lower prediction confidences were mainly due to the potential weakness of the classifiers and/or some moderate heterogeneity.
in addition, the profile of the four metastatic prostate  samples was interesting.
although they were predicted with 100% accuracy, their relative confidences were low.
this suggests the metastatic tumors can be distinguished from the primary tumors of the same type by using the proposed relative confidence criterion.
nci <dig> data
there were no independent test  samples in the nci <dig> dataset.
consequently, ten-fold cross validation was conducted as in statnikov et al   <cit> .
the results are summarized in figure  <dig>  in most scenarios, the prediction accuracy ranged from  <dig> % to  <dig> %.
the improvement due to the use of the pst algorithm was not as significant as with the gcm data.
a modest improvement was observed when the number of used genes was relative small .
one explanation is that, because some tumor types had a very limited number of samples  available for training the classifiers, holding out one sample from the training set as is required for the implementation of pst sharpened sample shortage and weakened the trained classifiers.
nevertheless, the prediction accuracy obtained was comparable to the best reported results using this dataset.
according to statnikov et al.
<cit> , svm-based methods performed much better than k-nn, pnn  and other non-svm methods with an accuracy ranging from  <dig> % to  <dig> %.
furthermore, it was evident from our study that breast cancer  samples were unpredictable.
this result is consistent with ross et al , in which the br samples could not be clustered together  <cit> .
the reason could be that the br samples contained estrogen positive  and estrogen negative  subtypes  <cit> .
CONCLUSIONS
in cancer type predictions, multi-category problems have proven to be more challenging than binary cases, not only in the classification accuracy but also in the assessment of uncertainty.
in this paper, a jackknife-like classification method, called paired-samples test algorithm , was proposed and applied to two bench datasets of multiple tumor types  <cit> .
the results showed that the proposed method has improved the prediction accuracy of test samples in the gcm dataset, especially when t-statistics were used for primary feature selection.
for the nci <dig> dataset, improvement was observed only when the number of used genes was relative small.
these improvements made the binary regression algorithm more robust to gene selection and the number of genes used.
the core idea of the proposed method is to repeatedly test a certain known tumor type with a blind test sample while withholding an associated training sample; in this way, not only can the prediction be made but also the relative confidence r of the prediction can be accessed by measuring the difference between the prediction probability of the test sample and the corresponding value of the withheld training sample.
r provided insight into the sources of the uncertainty in the statistical classification by revealing the loss in confidence due to the utilization of weak classifiers or heterogeneity in a given tumor type.
it is possible to combine the measurement f and r to make a better score for type determination.
our continuous work will consider this possibility in regards to penalizing a negative r value.

 the use of gene expression profiling for the classification of human cancers has been widely investigated. previous works were successful in predicting tumor types in the context of binary problems. many algorithms for feature extraction and sample classification have been proposed. more recently, a method for addressing the potential mislabeling in the training set was proposed for binary classification of cancer samples. as there are over a hundred types of cancers, and potentially even more subtypes, it is essential to develop multi-category methodologies for molecular classification for any practical application. multi-category prediction can be achieved using binary classification algorithms via the one-versus-one and/or one-versus-rest partition of the training data set. however, in a cancer type prediction, multi-category problems proved to be more challenging than simple binary problems, and the reported results were less than satisfactory. on one hand, when the available resource is limited and the sample size of a given category is small, classifiers based on the ovr partition of the data set potentially suffer from severe over-fitting, leading to low predictive ability and robustness. furthermore, the substantial noise introduced by implementing the numerous classifiers under an ovo scheme and the asymmetric training sets caused by ovr partitioning of the data will inevitably weaken the classification system. on the other hand, the effects of biological and technical noise together with the genetic heterogeneity of samples within a clinically defined tumor class decrease the predictive power in a multiple setting. in disease diagnostic, a measurement of confidence or uncertainty reported with the type determination is always desirable. however, some well-established statistical criteria often become less credible and of little biological meaning for highly heterogeneous cancer types, especially in the context of multiple cancer types. a potential reason is that the winning classifier used to discriminate one cancer type from others could be weak or unstable due to limited training samples. although this phenomenon was alluded to in previous studies, it has not received appropriate attention. figure presents a graphical illustration of the problem. using an ovr binary classifier, all samples of a homogeneous cancer type were classified correctly and with high confidence. all other cancer type samples in the group have probabilities of being cancer type a close to zero. however, the situation was very different when a heterogeneous cancer class was considered. in fact, some samples of cancer b type had classification probability lower than. such low classification probability could lead to misdiagnosis if a hard classification rule is applied. it is possible that such low probability is due to the weakness of the classifier that is established with a highly heterogeneous training set. the jackknife is a well known, non-parametric method often used for estimating the sampling distribution of a statistic. given a sample dataset and a desired statistic, the jackknife works by computing the desired statistic with an element removed from the equation. the process is repeated for each element in the dataset. the application in cancer classification with gene expression profiling has been reported in the context of binary problems. in that study, the individual maximum difference subsets of genes identified from a set of jackknife subsets of samples were aggregated to generate the "overall mdss" in order to return the expected classification. in other words, jackknife was used for feature selection rather than for training multiple sub-classifiers. in this study, a new learning method called paired-samples test algorithm, which is based on the jackknife method, was used to classify multiple tumor types using gene expression data. the proposed method is designed for solving multi-category problems under an ovr scheme with a very limited training data set, and it is similar to the bootstrap aggregating procedure, which proved to be helpful in improving weak classifiers. in order to get a relative measurement of uncertainty in the prediction of a sample category, the training sample being removed each time was predicted together with the training samples. the procedure was implemented in a parsimonious way, making its integration with a computationally intensive algorithm, such as the stochastic, regulation-based binary regression, feasible. the performance of the proposed method was evaluated under several scenarios of gene selection criteria using two well known and challenging datasets: the gcm and nci datasets containing and cancer tumor types, respectively.  determination of the optimum number of genes to be used by the classification algorithm is usually a difficult task that depends on several factors, including the classification algorithms and the complexity of the data set. for the used binary regression algorithm, previous studies have shown that a feature set of one to two hundred top genes is adequate for a simple two category problem. in this study, the size of the feature set used was or genes for the gcm dataset and or genes for the nci dataset. gcm data the prediction accuracy of the validation samples, using different gene selection procedures, is summarized in figure the results showed that fold change and penalized t-statistic based methods for feature selection outperformed the t-statistic-based procedure. in most cases, the application of pst improved the prediction accuracy or maintained the high accuracies that had been obtained prior to its application, except in the scenario of genes and penalized t-statistic. the largest improvement occurred when genes were considered using different feature selection criteria, resulting in an increase in accuracy ranging from % to %. the combination of genes, fold change-based feature selection and pst had the highest prediction accuracy of %. additionally, almost 50% of the % incorrectly classified samples had their true tumor type predicted as the second possible classification in this scenario. it should be noted that while the largest improvements were seemingly coming from the weaker gene selection mechanisms, the application of pst made the binary regression algorithm more robust in relation to the gene selection methods and the size of the gene set to be used. these prediction results are, in general, better than those obtained by several previous studies using the same data set. using a recursive feature elimination procedure and a support vector machine classification algorithm, ramaswamy et al. obtained their best result with tumors correctly predicted among the test samples, corresponding to an accuracy of 78%. using a feature selection algorithm based on the overlaps of gene expression values between different classes in conjunction with the covering classification algorithm, a modification of the k-nn method, bagirov et al. achieved a prediction accuracy of around 80%. based on the concept of gene interaction, antonov et al proposed a maximal margin linear programming procedure that combines linear programming and svm. using mama, only eight test samples were misclassified. although slightly superior to our method in the overall accuracy, the lack of information about confusion profiles of the prediction and the secondary classification of non-correctly predicted samples make the direct comparison between both methods difficult. recently, sheng and tan reported a prediction accuracy of around 83% by using error correcting out codes, svm and a recursive feature elimination procedure. the output coding based approach is very costly in implementation and the result was highly sensitive to the decoding functions and the length of the random code. a in parenthesis are the assigned tumor types for the misclassified samples. it is possible that the superiority of the proposed method over svm and other learning algorithms could be related to the difference in gene selection methods used in this study and by ramaswamy et al and bagirov et al. however, our preliminary work as well as readily available information demonstrated that svm outperformed k-nn, nn, pnn and the decision tree in general does not support such a claim. in fact, the highest accuracies obtained using svm occurred when 200- genes were selected based on fc, t-statistics, penalized t-statistics and non-parametric anova, ranging from % to %. these were well below the results obtained using our approach. as indicated in table it seems that some tumor types are easily predicted. for example, ly, ut, me and csn tumors had 100% prediction accuracy using all three methods. meanwhile, other types, such as br, had a high misclassification rate ranging from to 75%, indicating potential excess heterogeneity. additionally, the profile of misclassified samples was very different between the four studies. in fact, among the four br tumors, two were misclassified as ov and pa in ramaswamy et al, three were misclassified as lu, lu and bl in bagirov et al, and three were misclassified as le, pa, and ut in the current study. to further validate the results behind the use of the independent test samples, a four-fold cross validation was conducted for the training samples. the results of this validation are presented in figure in most scenarios, the prediction accuracy was improved when the proposed jackknife method was used. the highest value was %, which was achieved from several combinations of the gene selection method and gene number, including the case of fc-based gene selection and genes. this accuracy value was similar to the best performance of % obtained using the independent test samples, and it is % higher than the accuracy obtained by ramaswamy et al.the np-anova feature selection performed marginally better in the cross validation than in the independent test with the highest prediction accuracy of %. ppp rediction uncertainty uncertainty of the correctly classified test samples from the best result is graphically presented in figure among these samples, eight tumors had high f and nearly 3/ of the tumors had their prediction confidence <. for the classification algorithm used in this study, f was defined as the aggregate probability that the test sample t belongs to the assigned tumor type. in this context, considering f alone makes the current prediction results seem unexpected. however, when taking r values into account, confidence measurement, or f, appears to be in better agreement with the results of this study. of those samples with lower prediction confidence, the majority had their r between - and, suggesting that their lower prediction confidences were mainly due to the potential weakness of the classifiers and/or some moderate heterogeneity. in addition, the profile of the four metastatic prostate samples was interesting. although they were predicted with 100% accuracy, their relative confidences were low. this suggests the metastatic tumors can be distinguished from the primary tumors of the same type by using the proposed relative confidence criterion. nci data there were no independent test samples in the nci dataset. consequently, ten-fold cross validation was conducted as in statnikov et al. the results are summarized in figure in most scenarios, the prediction accuracy ranged from % to %. the improvement due to the use of the pst algorithm was not as significant as with the gcm data. a modest improvement was observed when the number of used genes was relative small. one explanation is that, because some tumor types had a very limited number of samples available for training the classifiers, holding out one sample from the training set as is required for the implementation of pst sharpened sample shortage and weakened the trained classifiers. nevertheless, the prediction accuracy obtained was comparable to the best reported results using this dataset. according to statnikov et al., svm-based methods performed much better than k-nn, pnn and other non-svm methods with an accuracy ranging from % to %. furthermore, it was evident from our study that breast cancer samples were unpredictable. this result is consistent with ross et al, in which the br samples could not be clustered together. the reason could be that the br samples contained estrogen positive and estrogen negative subtypes.  in cancer type predictions, multi-category problems have proven to be more challenging than binary cases, not only in the classification accuracy but also in the assessment of uncertainty. in this paper, a jackknife-like classification method, called paired-samples test algorithm, was proposed and applied to two bench datasets of multiple tumor types. the results showed that the proposed method has improved the prediction accuracy of test samples in the gcm dataset, especially when t-statistics were used for primary feature selection. for the nci dataset, improvement was observed only when the number of used genes was relative small. these improvements made the binary regression algorithm more robust to gene selection and the number of genes used. the core idea of the proposed method is to repeatedly test a certain known tumor type with a blind test sample while withholding an associated training sample; in this way, not only can the prediction be made but also the relative confidence r of the prediction can be accessed by measuring the difference between the prediction probability of the test sample and the corresponding value of the withheld training sample. r provided insight into the sources of the uncertainty in the statistical classification by revealing the loss in confidence due to the utilization of weak classifiers or heterogeneity in a given tumor type. it is possible to combine the measurement f and r to make a better score for type determination. our continuous work will consider this possibility in regards to penalizing a negative r value. 
previous studies were successful in distinguishing several tumor types in binary problems. the overall prediction accuracies were competitive in comparison to the most accurate results obtained by several previous studies on the same datasets and with other methods. a jackknife-based supervised learning method called paired-samples test algorithm, coupled with a binary classification model based on linear regression, was proposed and applied to two well known and challenging datasets consisting of and tumor types. the results showed that the proposed method improved the prediction accuracy of the test samples for the gcm dataset, especially when t-statistic was used in the primary feature selection. for the nci dataset, the application of pst improved prediction accuracy when the numbers of used genes were relatively small. the strengths were demonstrated in the application to two bench datasets. furthermore, the relative confidence r provided a unique insight into the sources of the uncertainty shown in the statistical classification and the potential variants within the same tumor type. as there are over a hundred types of cancers, and potentially even more subtypes, it is essential to develop multi-category methodologies for molecular classification for any meaningful practical application. the use of gene expression profiling for the classification of human cancer tumors has been widely investigated. we proposed a novel bagging method for the classification and uncertainty assessment of multi-category tumor samples using gene expression information.
 the use of gene expression profiling for the classification of human cancer tumors has been widely investigated. previous studies were successful in distinguishing several tumor types in binary problems. as there are over a hundred types of cancers, and potentially even more subtypes, it is essential to develop multi-category methodologies for molecular classification for any meaningful practical application.  a jackknife-based supervised learning method called paired-samples test algorithm, coupled with a binary classification model based on linear regression, was proposed and applied to two well known and challenging datasets consisting of and tumor types. the results showed that the proposed method improved the prediction accuracy of the test samples for the gcm dataset, especially when t-statistic was used in the primary feature selection. for the nci dataset, the application of pst improved prediction accuracy when the numbers of used genes were relatively small. these improvements made the binary classification method more robust to the gene selection mechanism and the size of genes to be used. the overall prediction accuracies were competitive in comparison to the most accurate results obtained by several previous studies on the same datasets and with other methods. furthermore, the relative confidence r provided a unique insight into the sources of the uncertainty shown in the statistical classification and the potential variants within the same tumor type.  we proposed a novel bagging method for the classification and uncertainty assessment of multi-category tumor samples using gene expression information. the strengths were demonstrated in the application to two bench datasets. 