BACKGROUND
biomarker discovery and prognosis prediction are essential for improved personalized cancer treatment.
microarray technology is a significant tool for gene expression analysis and cancer diagnosis.
typically, microarray data sets are used for class discovery  <cit>  and prediction  <cit> .
the high dimensionality of the input feature space in comparison with the relatively small number of subjects is a widespread concern; hence some form of dimensionality reduction is often applied.
feature selection and feature transformation are two commonly used dimensionality reduction techniques.
the key difference between feature selection and feature transformation is that, in the former only a subset of original features is selected while the latter is based on generation of new features.
in this genomic era, several classification and dimensionality reduction methods are available for analyzing and classifying microarray data.
prediction analysis of microarray   <cit>  is a statistical technique for class prediction from gene expression data using nearest shrunken centroid .
pam identifies subsets of genes that best characterize each class.
ls-svm is a promising method for classification, because of its solid mathematical foundations which convey several salient properties that other methods hardly provide.
a commonly used technique for feature selection, t-test, assumes that the feature values from two different classes follow normal distributions.
several studies, especially microarray analysis, have used t-test and ls-svm together to improve the prediction performance by selecting key features  <cit> .
the least absolute shrinkage and selection operator   <cit>  is often used for gene selection and parameter estimation in high-dimensional microarray data  <cit> .
the lasso shrinks some of the coefficients to zero, and extend of shrinkage is determined by the tuning parameter, often obtained from cross validation.
inductive learning systems were successfully applied in a number of medical domains, e.g.
in localization of primary tumors, prognostic of recurring breast cancer, diagnosis of thyroid diseases, and rheumatology  <cit> .
an induction algorithm is used to learn a classifier, which maps the space of feature values into the set of class values.
this classifier is later used to classify new instances, with the unknown classifications .
researchers and practitioners realize that the effective use of these inductive learning systems requires data preprocessing, before a learning algorithm could be applied  <cit> .
due to the instability of feature selection techniques, it might be difficult or even impossible to remove irrelevant and/or redundant features from a data set.
feature transformation techniques, such as kpca, discover a new feature space having fewer dimensions through a functional mapping, while keeping as much information, as possible in the data set.
kpca, which is a generalization of pca, a nonlinear dimensionality reduction technique that has proven to be a powerful pre-processing step for classification algorithms.
it has been studied intensively in the last several years in the field of machine learning and has claimed success in many applications  <cit> .
an algorithm for classification using kpca was developed by liu et al.
<cit> .
kpca was proposed by schölkopf and smola  <cit> , by mapping features sets to a high-dimensional feature space  and applying mercer’s theorem.
suykens et al.
<cit>  proposed a simple and straightforward primal-dual support vector machine formulation to the pca problem.
to perform kpca, the user first transforms the input data x from the original input space f <dig> into a higher-dimensional feature space f <dig> with a nonlinear transform x→Φ where Φ is a nonlinear function.
then a kernel matrix k is formed using the inner products of new feature vectors.
finally, a pca is performed on the centralized k, which is an estimate of the covariance matrix of the new feature vectors in f <dig>  one of the commonly used kernel function is radial basis function  kernel: k=exp−∥xi−xj∥22h <dig> .
traditionally the optimal parameters  of rbf kernel function are selected in a trial and error fashion.
pochet et al.
<cit>  proposed an optimization algorithm for kpca with rbf kernel followed by fisher discriminant analysis  to find the parameters of kpca.
in this case, the parameter selection is coupled with the corresponding classifier.
this means that the performance of the final procedure depends on the chosen classifier.
such a procedure could produce possible inaccurate results in the case of weak classifiers.
in addition, this appears to be a time consuming procedure, while tuning the parameters of kpca.
most classification methods have inherent problem with high dimensionality of microarray data and hence require dimensionality reduction.
the ultimate goal of our work is to design a powerful preprocessing step, decoupled from the classification method, for large dimensional data sets.
in this paper, initially we explain an svm approach to pca and ls-svm approach to kpca.
next, by following the idea of least squares cross-validation in kernel density estimation, we propose a new data-driven bandwidth selection criterion for kpca.
the tuned ls-svm formulation to kpca is applied to several data sets and serves as a dimensionality reduction technique for a final classification task.
in addition, we compared the proposed strategy with an existing optimization algorithm for kpca, as well as with other preprocessing steps.
finally, for the sake of comparison, we applied ls-svm on whole data sets, pca+ls-svm, t-test + ls-svm, pam and lasso.
randomization on all data sets are carried out in order to get a more reliable idea of the expected performance.
data sets
in our analysis, we collected  <dig> publicly available binary class data sets .
the data sets are: colon cancer data  <cit> , breast cancer data  <cit> , pancreatic cancer premalignant data  <cit> , cervical cancer data  <cit> , acute myeloid leukemia data <cit> , ovarian cancer data  <cit> , head & neck squamous cell carcinoma data  <cit> , early-early stage duchenne muscular dystrophy  data  <cit> , hiv encephalitis data  <cit> , high grade glioma data  <cit> , and breast cancer data  <cit> .
in breast cancer data  <cit>  and high grade glioma data, all data samples have already been assigned to a training set or test set.
the breast cancer data in  <cit>  contains missing values; those values have been imputed based on the nearest neighbor method.
an overview of the characteristics of all the data sets can be found in table  <dig>  in all the cases, 2/3rd of the data samples of each class are assigned randomly to the training and the rest to the test set.
these randomizations are the same for all numerical experiments on all data sets.
this split was performed stratified to ensure that the relative proportion of outcomes sampled in both training and test set was similar to the original proportion in the full data set.
in all these cases, the data were standardized to zero mean and unit variance.
methods
the methods used to set up the case studies can be subdivided into two categories: dimensionality reduction using the proposed criterion and subsequent classification.
svm formulation to linear pca
given training set{xi}i=1n, xi∈ℝd  and n given data points for which one aims at finding projected variables v
t
x
i
with maximal variance.
svm formulation to pca problem is given in  <cit>  as follows:
maxv∑i=1n0−vtxi <dig>
where zero is considered as a single target value.
this interpretation of the problem leads to the following primal optimization problem
maxv,ejp=γ12∑i=1nei2−12vtv
such that
ei=vtxi,i= <dig> …,n.
this formulation states that one considers the difference between v
t
x
i
and the value  <dig> as error variables.
the projected variables correspond to what one calls the score variables.
these error variables are maximized for the given n data points while keeping the norm of v small by the regularization term.
the value γ is a positive real constant.
the lagrangian becomes
ℒ=γ12∑k=1nek2−12vtv−∑k=1nαkek−vtxk
with conditions for optimality
∂ℒ∂v=0→v=∑k=1nαkxk∂ℒ∂ek=0→αk=γekk= <dig> …,n∂ℒ∂αk=0→ek−vtxk,k= <dig> …,n.
by elimination of the variables e, v one obtains the following symmetric eigenvalue problem:
x1tx1…x1txn⋮⋮xntx1…xntxnα1⋮αn=λα1⋮αn
the vector of dual variables α=[α1;…;α
n
] is an eigenvector of the gram matrix and λ=1γ is the corresponding eigenvalue.
the score variable, znpca of sample x on nth eigenvector α
n
becomes
znpca=vtx=Σi=1nαixitx
ls-svm approach to kpca
the pca analysis problem is interpreted as a one-class modeling problem with a target value equal to zero around which the variance is maximized.
this results into a sum of squared error cost function with regularization.
the score variables are taken as additional error variables.
we now follow the usual svm methodology of mapping the d-dimensional data from the input space to a high-dimensional feature space ϕ:ℝd→ℝnh, where n
h
can be infinite, and apply mercer’s theorem  <cit> .
our objective is the following
maxv∑k=1n0−vt−μ^ϕ)) <dig>
with μ^ϕ=∑k=1nϕ and v is the eigenvector in the primal space with maximum variance.
this formulation states that one considers the difference between vt−μ^ϕ)  and the value  <dig> as error variables.
the projected variables correspond to what is called score variables.
these error variables are maximized for the given n data points.
next, by adding a regularization term we also want to keep the norm of v small.
the following optimization problem is formulated now in the primal weight space
maxv,ejp=γ12∑k=1nek2−12vtv
such that
ek=vt−μϕ),k= <dig> …,n.
the lagrangian yields
ℒ=γ12∑k=1nek2−12vtv−∑k=1nαkek−vtϕxk−μ^ϕ
with conditions for optimality
∂ℒ∂v=0→v=∑k=1nαkϕxk−μ^ϕ∂ℒ∂ek=0→αk=γekk= <dig> …,n∂ℒ∂αk=0→ek−vtϕxk−μ^ϕ= <dig> k= <dig> …,n.
by elimination of variables e and v, one obtains
1γαk−∑l=1nαlϕxl−μ^ϕtϕxk−μ^ϕ=0k= <dig> …,n.
defining λ=1γ, one obtains the following dual problem
Ωcα=λα
where Ω
c
denotes the centered kernel matrix with ijth entry:Ωc,i,j=k−1n∑r=1nk−1n∑r=1nk+1n2∑r=1n∑s=1nk.
data-driven bandwidth selection for kpca
model selection is a prominent issue in all learning tasks, especially in kpca.
since kpca is an unsupervised technique, formulating a data-driven bandwidth selection criterion is not trivial.
until now, no such data-driven criterion was available to tune the bandwidth  and number of components  for kpca.
typically these parameters are selected by trial and error.
analogue to least squares cross validation  <cit>  in kernel density estimation, we propose a new data driven selection criterion for kpca.
let
zn=Σi=1nαik
where k=exp−∥xi−xj∥22h <dig>  and set the target equal to  <dig> and denote by z
n
the score variable of sample x on n
t
h
eigenvector α. here, the score variables are expressed in terms of kernel expressions in which every training point contributes.
these expansions are typically dense .
in equation  <dig>  the kpca uses l <dig> lose function.
here we have chosen the l <dig> loss function to induce sparsness in kpca.
by extending the formulation in equation  <dig> to l <dig> loss function, the following problem is formulated for kernel pca  <cit> .
maxv,ejp=γ12∑k=1nl1−12vtv
such that
ek=vt−μϕ),k= <dig> …,n.
we propose the following tuning criterion for the bandwidth h which maximizes the l <dig> loss function of kpca:
j=argmaxh∈ℝ0+e∫|zn|dx,
where e denotes the expectation operator.
maximizing eq.
<dig> would lead to overfitting since we used all the training data in the criterion.
instead, we work with leave-one-out cross validation  estimation of z
n
to obtain the optimum bandwidth h of kpca, which gives projected variables with maximal variance.
a finite approximation to eq.
<dig> is given by
j=argmaxh∈ℝ0+1n∑j=1n∫|zn|dx
where n is the number of samples and zn denotes the score variable with the jth observation is left out.
in case the leave-one-out approach is computationally expensive, one could replace it with a leave v group out strategy .
integration can be performed by means of any numerical technique.
in our case, we have used trapezoidal rule.
the final model with optimum bandwidth is constructed as follows:
Ωc,ĥmaxα=λα,
where ĥmax=maxh∈ℝ0+1n∑j=1n∫|zn|dx.
figure  <dig> shows the bandwidth selection for cervical and colon cancer data sets for fixed number of components.
to also retain the optimum number of components of kpca, we modify eq.
<dig> as follows:
j=argmaxh∈ℝ0+,k∈ℕ01n∑n=1k∑j=1n∫|zn|dx
where k= <dig> …,n.
figure  <dig> illustrate the proposed model.
figure  <dig> shows the surface plot of eq.
<dig> for various values of h and k.
thus, the proposed data-driven model can obtain the optimal bandwidth for kpca, while retaining minimum number of eigenvectors which capture the majority of the variance of the data.
figure  <dig> shows a slice of the surface plots.
the values of the proposed criterion were re-scaled to be maximum  <dig>  the parameters that maximize eq.
<dig> are h= <dig>  and k= <dig> for cervical cancer data and h= <dig>  and k= <dig> for colon cancer data.
classification models
the constrained optimization problem for an ls-svm  <cit>  for classification has the following form:
minw,b,e12wtw+γ12Σk=1nek <dig>
subject to:
ykwtϕ+b=1−ek,k= <dig> …,n
where ϕ: ℝd→ℝdh is a nonlinear function which maps the d-dimensional input vector x from the input space to the d
h
-dimensional feature space, possibly infinite.
in the dual space the solution is given by
0ytyΩ+iγbβ=01v
with y=[ y <dig> …,y
n
]
t
,1
n
=
t
,e=[ e <dig> …,e
n
]
t
, β=[ β <dig> …,β
n
]
t
and Ω
i,j
=y
i
y
j
k where k is the kernel function.
the classifier in the dual space takes the form
y=sign∑k=1nβkykk+b
where β
k
are lagrange multipliers.
RESULTS
first we considered nine data sets described in table  <dig>  we have chosen the rbf kernel k=exp−||xi−xj||22h <dig> for kpca.
in this section all the steps are implemented using matlab r2012b and ls-svmlab v <dig>  toolbox  <cit> .
next, we compared the performance of the proposed method with classical pca and an existing tuning algorithm for rbf-kpca developed by pochet et al.
<cit> .
later, with the intention to comprehensively compare pca+ls-svm and kpca+ls-svm with other classification methods, we applied four widely used classifiers to the microarray data, being ls-svm on whole data sets, t-test + ls-svm, pam and lasso.
to fairly compare kernel functions of the ls-svm classifier; linear, rbf and polynomial kernel functions are used .
the average test accuracies and execution time for all these methods when applied to the  <dig> case studies are shown in table  <dig> and table  <dig> respectively.
statistical significance test results  are given in table  <dig> which compares the performance of kpca with other classifiers.
for all these methods, training on 2/3rd of the samples and testing on 1/3rd of the samples was repeated  <dig> times.
p-value: false discovery rate  corrected.
p-values of two-sided signed test are given.
p-value: false discovery rate  corrected.
comparison between the proposed criterion and pca
for each data set, the proposed methodology is applied.
this methodology consists of two steps.
first, eq.
<dig> is maximized in order to obtain an optimal bandwidth h and corresponding number of components k. second, the reduced data set is used to perform a classification task with ls-svm.
we retained  <dig> and  <dig> components respectively for cervical and colon cancer data sets.
for pca, the optimal number of components were selected by slightly modifying the equation  <dig>  i.e., which performed only for the components k as follows:
j=argmaxk∈ℕ01n∑n=1k∑j=1n∫znpcadx
where znpca the score corresponding to the varibale x on pca problem.
.
the score variables  are used to develop an ls-svm classification model.
the averaged test auc values over the  <dig> random repetitions were reported.
the main goal of pca is the reduction of dimensionality, that is, focusing on a few principal components  versus many variables.
there are several criteria have been proposed for determining how many pc should be investigated and how many should be ignored.
one common criteria is to include all those pcs up to a predetermined total percent variance explained, such as, 95%.
figure  <dig> depicts the prediction performances on colon cancer data, with pca+ls-svm, at different fractions of explained total variance.
it shows the results vary with the selected components.
here the number of retained components, depends on the chosen fraction of explained total variance.
the proposed approach offers a data-driven selection criterion for pca problem, instead of a traditional trial and error pc selection.
comparison between the proposed criterion and an existing optimization algorithm for rbf-kpca
we selected two experiments from pochet et al.
<cit>  , being high-grade glioma and breast cancer ii data sets.
we repeated the same experiments as reported in pochet et al.
<cit>  and compared with the proposed strategy.
the results are shown in table  <dig>  the three dimensional surface plot of loocv performance of the method proposed by  <cit>  for the high-grade glioma data set is shown in figure  <dig>  with the optimal h= <dig>  and k= <dig>  the optimum parameters are h= <dig>  and k= <dig> obtained by the proposed strategy  for the same data set.
et al.
: averaged test auc over  <dig> iterations and execution time in minutes
et al.
when looking at test auc in table  <dig>  both case studies applying the proposed strategy, perform better than the method proposed by pochet et al.
<cit>  with less variability.
in addition, the tuning method pochet et al.
<cit>  appears to be quite time consuming, whereas the proposed model enjoys a distinctive advantage with its low time complexity to carry out the same process.
comparison between the proposed criterion and other classifiers
in table  <dig>  we have highlighted the comparisons in which the proposed method was significantly better.
when looking specifically on the performance of each of the discussed methods, we note that ls-svm performance was slightly low on pca.
on data sets iv, vi, vii proposed approach performs better than, ls-svm with rbf kernel and ls-svm with linear kernel.
the proposed approach is outperformed, by the t-test + ls-svm on data sets v and vi and, by both pam and lasso on most of the data sets.
discussions
the obtained test auc of different classifiers on nine data sets, do not direct to a common conclusion that one method outperforms the other.
instead, it shows that each of these methods have its own advantage in classification tasks.
when considering classification problems without dimensionality reduction, the regularized ls-svm classifier shows a good performance on  <dig> percentage of data sets.
up till now, most microarray data sets are smaller in the sense of number of features and samples, but it is expected that these data sets might become larger or perhaps represent more complex classification problems in the future.
in this situation, dimensionality reduction processes  are the essential steps for building stable, robust and interpretable classifiers on these kind of data.
the selected features of feature selection method such as t-test, pam and lasso widely vary for each random iteration.
moreover, the classification performance of these methods on each iteration depends on the number of features selected.
table  <dig> shows the range, i.e.
minimum and maximum number of features selected on  <dig> iterations.
however pam is a user friendly toolbox for gene selection and classification tasks, its performance depends heavily on the selected features.
in addition, it is interesting that the lasso selected only very small subsets of the actual data sets.
but, in the lasso, the amount of shrinkage varies, depending on the value of the tuning parameter, which is often determined by cross validation  <cit> .
the number of genes selected as the outcome-predictive genes, generally decrease as the value of the tuning parameter increases.
the optimal value of the tuning parameter, that maximizes the prediction accuracy is determined; however, the set of genes identified using the optimal value contains the non-outcome-predictive genes  in many cases  <cit> .
p-value: false discovery rate  corrected.
the test auc on all nine case studies shows that kpca performs better than classical pca.
but the parameters of kpca need to be optimized.
here we have used loocv approach for parameters selection  of kpca.
in the optimization algorithm proposed by pochet et al.
<cit> , the combination of kpca with rbf kernel followed by fda tends to result in overfitting.
the proposed parameter selection criterion of kpca with rbf kernel, often results in test set performances  that is better than using kpca with a linear kernel, which reported in pochet et al.
it means that loocv in the proposed parameter selection criterion does not encounter an overfitting for kpca with rbf kernel function.
in addition, the optimization algorithm proposed by pochet et al.
is completely coupled with the subsequent classifier and thus it appears to be very time-consuming.
in combination with classification methods, microarray data analysis can be useful to guide clinical management in cancer studies.
in this study, several mathematical and statistical techniques were evaluated and compared in order to optimize the performance of clinical predictions based on microarray data.
considering the possibility of increasing size and complexity of microarray data sets in future, dimensionality reduction and nonlinear techniques have its own significance.
in many cases, in a specific application context the best feature set is still important .
while considering the stability and performance  of classifiers, the proposed methodology has its own importance to predict classes, of future samples of known disease cases.
finally this work could be extended further to uncover key features from biological data sets.
in several studies, kpca have used to obtain biologically relevant features such as genes  <cit>  or detect the association between multiple snps and disease  <cit> .
in all these cases, one needs to address the parameter optimization of kpca.
the available bandwidth selection techniques of kpca are time-consuming with high computational burden.
this could be resolved with the proposed data-driven bandwidth selection criterion for kpca.
CONCLUSIONS
the objective in class prediction with microarray data is an accurate classification of cancerous samples, which allows directed and more successful therapies.
in this paper, we proposed a new data-driven bandwidth selection criterion for kpca .
in particular, we optimize the bandwidth and the number of components by maximizing the projected variance of kpca.
in addition, we compared several data preprocessing techniques prior to classification.
in all the case studies, most of these preprocessing steps performed well on classification with approximately similar performance.
we observed that in feature selection methods selected features widely vary on each iteration.
hence it is difficult, even impossible to design a stable class predictor for future samples with these methods.
experiments on nine data sets show that the proposed strategy provides a stable preprocessing algorithm for classification of high dimensional data with good performance on test data.
the advantages of the proposed kpca+ls-svm classifier were presented in four aspects.
first, we propose a data-driven bandwidth selection criterion for kpca by tuning the optimum bandwidth and the number of principal components.
second, we illustrate that the performance of the proposed strategy is significantly better than an existing optimization algorithm for kpca.
third, its classification performance is not sensitive to any number of selected genes, so the proposed method is more stable than others proposed in literature.
fourth, it reduces the dimensionality of the data while keeping as much information as possible of the original data.
this leads to computationally less expensive and more stable results for massive microarray classification.
competing interests
the authors declare that they have no competing interests.
authors’ contributions
mt performed bandwidth selection, subsequent classification and drafted the paper.
kdb participated in the design and implementation of framework.
kdb and bdm helped draft the manuscript.
all authors read and approved the final manuscript.

 biomarker discovery and prognosis prediction are essential for improved personalized cancer treatment. microarray technology is a significant tool for gene expression analysis and cancer diagnosis. typically, microarray data sets are used for class discovery and prediction. the high dimensionality of the input feature space in comparison with the relatively small number of subjects is a widespread concern; hence some form of dimensionality reduction is often applied. feature selection and feature transformation are two commonly used dimensionality reduction techniques. the key difference between feature selection and feature transformation is that, in the former only a subset of original features is selected while the latter is based on generation of new features. in this genomic era, several classification and dimensionality reduction methods are available for analyzing and classifying microarray data. prediction analysis of microarray is a statistical technique for class prediction from gene expression data using nearest shrunken centroid. pam identifies subsets of genes that best characterize each class. ls-svm is a promising method for classification, because of its solid mathematical foundations which convey several salient properties that other methods hardly provide. a commonly used technique for feature selection, t-test, assumes that the feature values from two different classes follow normal distributions. several studies, especially microarray analysis, have used t-test and ls-svm together to improve the prediction performance by selecting key features. the least absolute shrinkage and selection operator is often used for gene selection and parameter estimation in high-dimensional microarray data. the lasso shrinks some of the coefficients to zero, and extend of shrinkage is determined by the tuning parameter, often obtained from cross validation. inductive learning systems were successfully applied in a number of medical domains, e.g. in localization of primary tumors, prognostic of recurring breast cancer, diagnosis of thyroid diseases, and rheumatology. an induction algorithm is used to learn a classifier, which maps the space of feature values into the set of class values. this classifier is later used to classify new instances, with the unknown classifications. researchers and practitioners realize that the effective use of these inductive learning systems requires data preprocessing, before a learning algorithm could be applied. due to the instability of feature selection techniques, it might be difficult or even impossible to remove irrelevant and/or redundant features from a data set. feature transformation techniques, such as kpca, discover a new feature space having fewer dimensions through a functional mapping, while keeping as much information, as possible in the data set. kpca, which is a generalization of pca, a nonlinear dimensionality reduction technique that has proven to be a powerful pre-processing step for classification algorithms. it has been studied intensively in the last several years in the field of machine learning and has claimed success in many applications. an algorithm for classification using kpca was developed by liu et al.. kpca was proposed by schölkopf and smola, by mapping features sets to a high-dimensional feature space and applying mercer’s theorem. suykens et al. proposed a simple and straightforward primal-dual support vector machine formulation to the pca problem. to perform kpca, the user first transforms the input data x from the original input space f into a higher-dimensional feature space f with a nonlinear transform x→Φ where Φ is a nonlinear function. then a kernel matrix k is formed using the inner products of new feature vectors. finally, a pca is performed on the centralized k, which is an estimate of the covariance matrix of the new feature vectors in f one of the commonly used kernel function is radial basis function kernel: k=exp−∥xi−xj∥22h. traditionally the optimal parameters of rbf kernel function are selected in a trial and error fashion. pochet et al. proposed an optimization algorithm for kpca with rbf kernel followed by fisher discriminant analysis to find the parameters of kpca. in this case, the parameter selection is coupled with the corresponding classifier. this means that the performance of the final procedure depends on the chosen classifier. such a procedure could produce possible inaccurate results in the case of weak classifiers. in addition, this appears to be a time consuming procedure, while tuning the parameters of kpca. most classification methods have inherent problem with high dimensionality of microarray data and hence require dimensionality reduction. the ultimate goal of our work is to design a powerful preprocessing step, decoupled from the classification method, for large dimensional data sets. in this paper, initially we explain an svm approach to pca and ls-svm approach to kpca. next, by following the idea of least squares cross-validation in kernel density estimation, we propose a new data-driven bandwidth selection criterion for kpca. the tuned ls-svm formulation to kpca is applied to several data sets and serves as a dimensionality reduction technique for a final classification task. in addition, we compared the proposed strategy with an existing optimization algorithm for kpca, as well as with other preprocessing steps. finally, for the sake of comparison, we applied ls-svm on whole data sets, pca+ls-svm, t-test + ls-svm, pam and lasso. randomization on all data sets are carried out in order to get a more reliable idea of the expected performance. data sets in our analysis, we collected publicly available binary class data sets. the data sets are: colon cancer data, breast cancer data, pancreatic cancer premalignant data, cervical cancer data, acute myeloid leukemia data, ovarian cancer data, head & neck squamous cell carcinoma data, early-early stage duchenne muscular dystrophy data, hiv encephalitis data, high grade glioma data, and breast cancer data. in breast cancer data and high grade glioma data, all data samples have already been assigned to a training set or test set. the breast cancer data in contains missing values; those values have been imputed based on the nearest neighbor method. an overview of the characteristics of all the data sets can be found in table in all the cases, 2/3rd of the data samples of each class are assigned randomly to the training and the rest to the test set. these randomizations are the same for all numerical experiments on all data sets. this split was performed stratified to ensure that the relative proportion of outcomes sampled in both training and test set was similar to the original proportion in the full data set. in all these cases, the data were standardized to zero mean and unit variance. methods the methods used to set up the case studies can be subdivided into two categories: dimensionality reduction using the proposed criterion and subsequent classification. svm formulation to linear pca given training set{xi}i=1n, xi∈ℝd and n given data points for which one aims at finding projected variables v t x i with maximal variance. svm formulation to pca problem is given in as follows: maxv∑i=1n0−vtxi where zero is considered as a single target value. this interpretation of the problem leads to the following primal optimization problem maxv,ejp=γ12∑i=1nei2−12vtv such that ei=vtxi,i= …,n. this formulation states that one considers the difference between v t x i and the value as error variables. the projected variables correspond to what one calls the score variables. these error variables are maximized for the given n data points while keeping the norm of v small by the regularization term. the value γ is a positive real constant. the lagrangian becomes ℒ=γ12∑k=1nek2−12vtv−∑k=1nαkek−vtxk with conditions for optimality ∂ℒ∂v=0→v=∑k=1nαkxk∂ℒ∂ek=0→αk=γekk= …,n∂ℒ∂αk=0→ek−vtxk,k= …,n. by elimination of the variables e, v one obtains the following symmetric eigenvalue problem: x1tx1…x1txn⋮⋮xntx1…xntxnα1⋮αn=λα1⋮αn the vector of dual variables α=[α1;…;α n ] is an eigenvector of the gram matrix and λ=1γ is the corresponding eigenvalue. the score variable, znpca of sample x on nth eigenvector α n becomes znpca=vtx=Σi=1nαixitx ls-svm approach to kpca the pca analysis problem is interpreted as a one-class modeling problem with a target value equal to zero around which the variance is maximized. this results into a sum of squared error cost function with regularization. the score variables are taken as additional error variables. we now follow the usual svm methodology of mapping the d-dimensional data from the input space to a high-dimensional feature space ϕ:ℝd→ℝnh, where n h can be infinite, and apply mercer’s theorem. our objective is the following maxv∑k=1n0−vt−μ^ϕ)) with μ^ϕ=∑k=1nϕ and v is the eigenvector in the primal space with maximum variance. this formulation states that one considers the difference between vt−μ^ϕ) and the value as error variables. the projected variables correspond to what is called score variables. these error variables are maximized for the given n data points. next, by adding a regularization term we also want to keep the norm of v small. the following optimization problem is formulated now in the primal weight space maxv,ejp=γ12∑k=1nek2−12vtv such that ek=vt−μϕ),k= …,n. the lagrangian yields ℒ=γ12∑k=1nek2−12vtv−∑k=1nαkek−vtϕxk−μ^ϕ with conditions for optimality ∂ℒ∂v=0→v=∑k=1nαkϕxk−μ^ϕ∂ℒ∂ek=0→αk=γekk= …,n∂ℒ∂αk=0→ek−vtϕxk−μ^ϕ= k= …,n. by elimination of variables e and v, one obtains 1γαk−∑l=1nαlϕxl−μ^ϕtϕxk−μ^ϕ=0k= …,n. defining λ=1γ, one obtains the following dual problem Ωcα=λα where Ω c denotes the centered kernel matrix with ijth entry:Ωc,i,j=k−1n∑r=1nk−1n∑r=1nk+1n2∑r=1n∑s=1nk. data-driven bandwidth selection for kpca model selection is a prominent issue in all learning tasks, especially in kpca. since kpca is an unsupervised technique, formulating a data-driven bandwidth selection criterion is not trivial. until now, no such data-driven criterion was available to tune the bandwidth and number of components for kpca. typically these parameters are selected by trial and error. analogue to least squares cross validation in kernel density estimation, we propose a new data driven selection criterion for kpca. let zn=Σi=1nαik where k=exp−∥xi−xj∥22h and set the target equal to and denote by z n the score variable of sample x on n t h eigenvector α. here, the score variables are expressed in terms of kernel expressions in which every training point contributes. these expansions are typically dense. in equation the kpca uses l lose function. here we have chosen the l loss function to induce sparsness in kpca. by extending the formulation in equation to l loss function, the following problem is formulated for kernel pca. maxv,ejp=γ12∑k=1nl1−12vtv such that ek=vt−μϕ),k= …,n. we propose the following tuning criterion for the bandwidth h which maximizes the l loss function of kpca: j=argmaxh∈ℝ0+e∫|zn|dx, where e denotes the expectation operator. maximizing eq. would lead to overfitting since we used all the training data in the criterion. instead, we work with leave-one-out cross validation estimation of z n to obtain the optimum bandwidth h of kpca, which gives projected variables with maximal variance. a finite approximation to eq. is given by j=argmaxh∈ℝ0+1n∑j=1n∫|zn|dx where n is the number of samples and zn denotes the score variable with the jth observation is left out. in case the leave-one-out approach is computationally expensive, one could replace it with a leave v group out strategy. integration can be performed by means of any numerical technique. in our case, we have used trapezoidal rule. the final model with optimum bandwidth is constructed as follows: Ωc,ĥmaxα=λα, where ĥmax=maxh∈ℝ0+1n∑j=1n∫|zn|dx. figure shows the bandwidth selection for cervical and colon cancer data sets for fixed number of components. to also retain the optimum number of components of kpca, we modify eq. as follows: j=argmaxh∈ℝ0+,k∈ℕ01n∑n=1k∑j=1n∫|zn|dx where k= …,n. figure illustrate the proposed model. figure shows the surface plot of eq. for various values of h and k. thus, the proposed data-driven model can obtain the optimal bandwidth for kpca, while retaining minimum number of eigenvectors which capture the majority of the variance of the data. figure shows a slice of the surface plots. the values of the proposed criterion were re-scaled to be maximum the parameters that maximize eq. are h= and k= for cervical cancer data and h= and k= for colon cancer data. classification models the constrained optimization problem for an ls-svm for classification has the following form: minw,b,e12wtw+γ12Σk=1nek subject to: ykwtϕ+b=1−ek,k= …,n where ϕ: ℝd→ℝdh is a nonlinear function which maps the d-dimensional input vector x from the input space to the d h -dimensional feature space, possibly infinite. in the dual space the solution is given by 0ytyΩ+iγbβ=01v with y=[ y …,y n ] t,1 n = t,e=[ e …,e n ] t, β=[ β …,β n ] t and Ω i,j =y i y j k where k is the kernel function. the classifier in the dual space takes the form y=sign∑k=1nβkykk+b where β k are lagrange multipliers.  first we considered nine data sets described in table we have chosen the rbf kernel k=exp−||xi−xj||22h for kpca. in this section all the steps are implemented using matlab r2012b and ls-svmlab v toolbox. next, we compared the performance of the proposed method with classical pca and an existing tuning algorithm for rbf-kpca developed by pochet et al.. later, with the intention to comprehensively compare pca+ls-svm and kpca+ls-svm with other classification methods, we applied four widely used classifiers to the microarray data, being ls-svm on whole data sets, t-test + ls-svm, pam and lasso. to fairly compare kernel functions of the ls-svm classifier; linear, rbf and polynomial kernel functions are used. the average test accuracies and execution time for all these methods when applied to the case studies are shown in table and table respectively. statistical significance test results are given in table which compares the performance of kpca with other classifiers. for all these methods, training on 2/3rd of the samples and testing on 1/3rd of the samples was repeated times. p-value: false discovery rate corrected. p-values of two-sided signed test are given. p-value: false discovery rate corrected. comparison between the proposed criterion and pca for each data set, the proposed methodology is applied. this methodology consists of two steps. first, eq. is maximized in order to obtain an optimal bandwidth h and corresponding number of components k. second, the reduced data set is used to perform a classification task with ls-svm. we retained and components respectively for cervical and colon cancer data sets. for pca, the optimal number of components were selected by slightly modifying the equation i.e., which performed only for the components k as follows: j=argmaxk∈ℕ01n∑n=1k∑j=1n∫znpcadx where znpca the score corresponding to the varibale x on pca problem.. the score variables are used to develop an ls-svm classification model. the averaged test auc values over the random repetitions were reported. the main goal of pca is the reduction of dimensionality, that is, focusing on a few principal components versus many variables. there are several criteria have been proposed for determining how many pc should be investigated and how many should be ignored. one common criteria is to include all those pcs up to a predetermined total percent variance explained, such as, 95%. figure depicts the prediction performances on colon cancer data, with pca+ls-svm, at different fractions of explained total variance. it shows the results vary with the selected components. here the number of retained components, depends on the chosen fraction of explained total variance. the proposed approach offers a data-driven selection criterion for pca problem, instead of a traditional trial and error pc selection. comparison between the proposed criterion and an existing optimization algorithm for rbf-kpca we selected two experiments from pochet et al., being high-grade glioma and breast cancer ii data sets. we repeated the same experiments as reported in pochet et al. and compared with the proposed strategy. the results are shown in table the three dimensional surface plot of loocv performance of the method proposed by for the high-grade glioma data set is shown in figure with the optimal h= and k= the optimum parameters are h= and k= obtained by the proposed strategy for the same data set. et al.: averaged test auc over iterations and execution time in minutes et al. when looking at test auc in table both case studies applying the proposed strategy, perform better than the method proposed by pochet et al. with less variability. in addition, the tuning method pochet et al. appears to be quite time consuming, whereas the proposed model enjoys a distinctive advantage with its low time complexity to carry out the same process. comparison between the proposed criterion and other classifiers in table we have highlighted the comparisons in which the proposed method was significantly better. when looking specifically on the performance of each of the discussed methods, we note that ls-svm performance was slightly low on pca. on data sets iv, vi, vii proposed approach performs better than, ls-svm with rbf kernel and ls-svm with linear kernel. the proposed approach is outperformed, by the t-test + ls-svm on data sets v and vi and, by both pam and lasso on most of the data sets. discussions the obtained test auc of different classifiers on nine data sets, do not direct to a common conclusion that one method outperforms the other. instead, it shows that each of these methods have its own advantage in classification tasks. when considering classification problems without dimensionality reduction, the regularized ls-svm classifier shows a good performance on percentage of data sets. up till now, most microarray data sets are smaller in the sense of number of features and samples, but it is expected that these data sets might become larger or perhaps represent more complex classification problems in the future. in this situation, dimensionality reduction processes are the essential steps for building stable, robust and interpretable classifiers on these kind of data. the selected features of feature selection method such as t-test, pam and lasso widely vary for each random iteration. moreover, the classification performance of these methods on each iteration depends on the number of features selected. table shows the range, i.e. minimum and maximum number of features selected on iterations. however pam is a user friendly toolbox for gene selection and classification tasks, its performance depends heavily on the selected features. in addition, it is interesting that the lasso selected only very small subsets of the actual data sets. but, in the lasso, the amount of shrinkage varies, depending on the value of the tuning parameter, which is often determined by cross validation. the number of genes selected as the outcome-predictive genes, generally decrease as the value of the tuning parameter increases. the optimal value of the tuning parameter, that maximizes the prediction accuracy is determined; however, the set of genes identified using the optimal value contains the non-outcome-predictive genes in many cases. p-value: false discovery rate corrected. the test auc on all nine case studies shows that kpca performs better than classical pca. but the parameters of kpca need to be optimized. here we have used loocv approach for parameters selection of kpca. in the optimization algorithm proposed by pochet et al., the combination of kpca with rbf kernel followed by fda tends to result in overfitting. the proposed parameter selection criterion of kpca with rbf kernel, often results in test set performances that is better than using kpca with a linear kernel, which reported in pochet et al. it means that loocv in the proposed parameter selection criterion does not encounter an overfitting for kpca with rbf kernel function. in addition, the optimization algorithm proposed by pochet et al. is completely coupled with the subsequent classifier and thus it appears to be very time-consuming. in combination with classification methods, microarray data analysis can be useful to guide clinical management in cancer studies. in this study, several mathematical and statistical techniques were evaluated and compared in order to optimize the performance of clinical predictions based on microarray data. considering the possibility of increasing size and complexity of microarray data sets in future, dimensionality reduction and nonlinear techniques have its own significance. in many cases, in a specific application context the best feature set is still important. while considering the stability and performance of classifiers, the proposed methodology has its own importance to predict classes, of future samples of known disease cases. finally this work could be extended further to uncover key features from biological data sets. in several studies, kpca have used to obtain biologically relevant features such as genes or detect the association between multiple snps and disease. in all these cases, one needs to address the parameter optimization of kpca. the available bandwidth selection techniques of kpca are time-consuming with high computational burden. this could be resolved with the proposed data-driven bandwidth selection criterion for kpca.  the objective in class prediction with microarray data is an accurate classification of cancerous samples, which allows directed and more successful therapies. in this paper, we proposed a new data-driven bandwidth selection criterion for kpca. in particular, we optimize the bandwidth and the number of components by maximizing the projected variance of kpca. in addition, we compared several data preprocessing techniques prior to classification. in all the case studies, most of these preprocessing steps performed well on classification with approximately similar performance. we observed that in feature selection methods selected features widely vary on each iteration. hence it is difficult, even impossible to design a stable class predictor for future samples with these methods. experiments on nine data sets show that the proposed strategy provides a stable preprocessing algorithm for classification of high dimensional data with good performance on test data. the advantages of the proposed kpca+ls-svm classifier were presented in four aspects. first, we propose a data-driven bandwidth selection criterion for kpca by tuning the optimum bandwidth and the number of principal components. second, we illustrate that the performance of the proposed strategy is significantly better than an existing optimization algorithm for kpca. third, its classification performance is not sensitive to any number of selected genes, so the proposed method is more stable than others proposed in literature. fourth, it reduces the dimensionality of the data while keeping as much information as possible of the original data. this leads to computationally less expensive and more stable results for massive microarray classification. competing interests the authors declare that they have no competing interests. authors’ contributions mt performed bandwidth selection, subsequent classification and drafted the paper. kdb participated in the design and implementation of framework. kdb and bdm helped draft the manuscript. all authors read and approved the final manuscript. 
due to the dynamic selection property of feature selection, it is hard to define significant features for the classifier, which predicts classes of future samples. the main input to this class predictors are high dimensional data with many variables and few observations. in this paper, we propose a new data-driven bandwidth selection criterion for kpca, which is related to least squares cross-validation for kernel density estimation. then, we compare its performances and computational time) with other well known techniques such as whole data set + ls-svm, pca + ls-svm, t-test + ls-svm, prediction analysis of microarrays and least absolute shrinkage and selection operator. both feature selection and feature transformation perform well on classification tasks. we propose, evaluate, and compare several mathematical/statistical techniques, which apply feature transformation/selection for subsequent classification, and consider its application in medical diagnostics. moreover, the proposed strategy enjoys a distinctive advantage with its relatively lesser time complexity. disease class predictors can be designed for known disease cases and provide diagnostic confirmation or clarify abnormal cases. dimensionality reduction of these features set significantly speeds up the prediction task. studies show that a well tuned kernel pca is an efficient preprocessing step for dimensionality reduction, but the available bandwidth selection method for kpca was computationally expensive.
 dna microarrays are potentially powerful technology for improving diagnostic classification, treatment selection, and prognostic assessment. the use of this technology to predict cancer outcome has a history of almost a decade. disease class predictors can be designed for known disease cases and provide diagnostic confirmation or clarify abnormal cases. the main input to this class predictors are high dimensional data with many variables and few observations. dimensionality reduction of these features set significantly speeds up the prediction task. feature selection and feature transformation methods are well known preprocessing steps in the field of bioinformatics. several prediction tools are available based on these techniques.  studies show that a well tuned kernel pca is an efficient preprocessing step for dimensionality reduction, but the available bandwidth selection method for kpca was computationally expensive. in this paper, we propose a new data-driven bandwidth selection criterion for kpca, which is related to least squares cross-validation for kernel density estimation. we propose a new prediction model with a well tuned kpca and least squares support vector machine. we estimate the accuracy of the newly proposed model based on case studies. then, we compare its performances and computational time) with other well known techniques such as whole data set + ls-svm, pca + ls-svm, t-test + ls-svm, prediction analysis of microarrays and least absolute shrinkage and selection operator. finally, we assess the performance of the proposed strategy with an existing kpca parameter tuning algorithm by means of two additional case studies.  we propose, evaluate, and compare several mathematical/statistical techniques, which apply feature transformation/selection for subsequent classification, and consider its application in medical diagnostics. both feature selection and feature transformation perform well on classification tasks. due to the dynamic selection property of feature selection, it is hard to define significant features for the classifier, which predicts classes of future samples. moreover, the proposed strategy enjoys a distinctive advantage with its relatively lesser time complexity. 