BACKGROUND
various machine learning methods have been applied in the diagnosis or prognosis of cancer research, such as, artificial neural networks, fuzzy logic, genetic algorithm, support vector machine and other hybrid techniques  <cit> .
from the medical perspective, diagnosis is to identify a disease by its signs and symptoms while prognosis is to predict the outcome of the disease and the status of the patient, whether the patient will survive or recover from the disease or vice versa.
in some studies, the researchers have proven that machine learning methods could generate more accurate diagnosis or prognosis as compared to traditional statistical methods  <cit> .
usually, clinicopathologic data or genomic data are used in researches either involving diagnosis or that with respect to prognosis.
currently, there are some researches that have shown that prognosis results are more accurate when using both clinicopathologic and genomic data.
examples of these are the works in  <cit>  in diffuse large b-cell lymphoma  cancer, the works in  <cit>  in breast cancer,  <cit>  in oral cancer, and  <cit>  in bladder cancer.
however, the number of published articles on researches that combine both clinicopathologic and genomic data are few as compared to that using only clinicopathologic data  <cit> .
in the oral cancer domain,  <cit>  used machine learning techniques in the oral cancer susceptibility studies.
they proposed a hybrid adaptive system inspired from learning classifier system, decision trees and statistical hypothesis testing.
the dataset includes both demographic data and  <dig> types of genes.
their results showed that the proposed algorithm outperformed the other algorithms of naive bayes, c <dig> , neural network and xcs .
however, they did not validate their results against the traditional statistical methods.
<cit>  focused on the 5-year overall survival in a group of oral squamous cell carcinoma  patients and investigated the effects of demographic data, clinical data and genomic data, and human papillomavirus on the prognostic outcome.
they used the statistical methods for the prediction and their results showed that the 5-year overall survival was  <dig> % and highlighted the influence of p <dig> immunoexpression, age and anatomic localization on ocss prognosis.
however, in this research, no machine learning methods were used and compared.
another oral cancer research that was done by  <cit>  was in the oral cancer reoccurrence.
bayesian network was used and compared with ann, svm, decision tree, and random forests.
they used multitude of heterogeneous data which included clinical, imaging, tissue and blood genomic data.
they built a separate classifier for different types of data and combined the best performing classification schemes.
they claimed that they had achieved an accuracy of 100% with the combination of all types of data and proved that the prediction accuracy was the best when using all types of data.
however, more than  <dig> markers were required for their final combined classifier.
for the genomic domain,  <cit>  showed that p <dig> over expression associates with poor prognosis in oral cancer.
their study showed that cases with diffuse p <dig> expression were more aggressive and poorly differentiated and related to a poorer prognosis, these findings supporting the use of p <dig> as an additional marker for diagnostic use in oral scc.
in  <cit> , immunohistochemical analysis of protein expression for p <dig>  p <dig> and p <dig> was performed for  <dig> samples of well-differentiated human buccal squamous-cell carcinomas, with  <dig> specimens of normal buccal mucosa employed as controls.
their results indicated that both p <dig> and p <dig> may be involved in the development of human buccal squamous-cell carcinoma, perhaps in concert with p <dig>  similar results were obtained by  <cit> , they have showed that in head and neck squamous carcinomas , p <dig> was the most frequently expressed , followed by p <dig>  and p <dig> .
their study indicated that p <dig> and p <dig> expression may represent an early event in hnsc tumorigenesis and p <dig> and p <dig> may function as oncogenes in the development of these tumors.
in this research, an oral cancer prognostic model is developed.
this research used real-world oral cancer dataset which is collected locally at the oral cancer research and coordinating centre , faculty of dentistry, university of malaya, malaysia.
the model takes both clinicopathologic and genomic data in order to investigate the impact of each marker or combination of markers to the accuracy of the prognosis of oral cancer.
five feature selection methods are proposed with the objectives to reduce the number of input variables to avoid over-fitting and to find out an optimum feature subset for oral cancer prognosis.
this is followed by the classification procedures which are used to classify the status of the patient after 1–3 years of diagnosis .
four classification methods, from both machine learning and statistical methods, are tested and compared.
the objective of this research is to prove that the prognosis is better by using both types of clinicopathologic and genomic markers, and to identify the key markers for oral cancer prognosis using the hybrid of feature selection and machine learning methods.
methods
the framework for the oral cancer prognostic model is shown in figure  <dig>  clinicopathologic variables from the ocrcc database and genomic variables from immunohistochemistry  staining are fed into the model.
basically, there are three main parts for the oral cancer prognostic model, which are wet-laboratory testing for genomic variables, feature selection methods and the classification models.
this research was approved by medical ethics committee, faculty of dentistry, university of malaya.
clinicopathologic data
a total of  <dig> oral cancer cases were selected from the malaysian oral cancer database and tissue bank system  coordinated by the oral cancer research and coordinating centre , faculty of dentistry, university of malaya.
the selection was based on the completeness of the clinicopathologic data, the availability of tissues and the availability of data .
the selected cases were based on the oral cancer cases seen in the faculty of dentistry, university of malaya and hospital tunku ampuan rahimah, klang, a malaysian government hospital, from the year  <dig> to  <dig>  these cases were diagnosed and followed up and the data were recorded in the standardised forms prepared by the mocdtbs.
later, mocdtbs transcribed all the data from paper to an electronic version and stored in the database.
all the cases selected were diagnosed as squamous cell carcinomas .
table  <dig> shows the  <dig> to 3-year survival for these  <dig> cases.
basically, three types of data are available for each oral cancer case, namely, social demographic data , clinical data , and pathological data .
pathological data were obtained from the biopsy reports before and after surgical procedures.
in this research, we referred to the clinical and pathological data as clinicopathologic data.
based on the discussions with two oral cancer clinicians, prof. rosnah binti zain and dr thomas george kallarakkal,  <dig> key variables had been identified as important prognostic factors of oral cancer.
the selected clinicopathologic variables are listed in table  <dig>
genomic data
two genomic variables had been identified through the literature studies and discussions with oral pathologists, from the department of oral pathology and oral medicine and periodontology, faculty of dentistry, university of malaya.
both of these variables are tumour suppressor genes, namely, p <dig> and p <dig>  p <dig> is the most frequently associated marker in the head and neck cancers  <cit> .
p <dig> is called the “guardian of the genome”, it is important in maintaining genomic stability, progression of cell cycle, cellular differentiation, dna repair and apoptosis.
it is difficult to demonstrate p <dig> protein in normal tissues using immunohistochemistry procedures due to its high catabolic rate; however mutated p <dig> exhibits a much lower catabolic rate and accumulates in the cells  <cit> .
in addition, p <dig> gene, a homolog of the p <dig> is located in chromosome 3q21- <dig>  and its amplification has been associated with prognostic outcome in oral cancer  <cit> .
the p <dig> gene is highly expressed in the basal or progenitor layers of many epithelial tissues.
the cases selected were the same as in the clinicopathologic data.
immunohistochemistry  staining was performed on the selected formalin-fixed paraffin embedded oral cancer tissues to obtain the results for the selected genomic variables.
the archival formalin-fixed paraffin embedded tissues were obtained from the oral pathology diagnostic laboratory, faculty of dentistry, university of malaya.
the tissues containing the tumour were cored and re-embedded and made into tissue macroarray blocks .
a total of 4-μm-thick sections of the resulting tmaa blocks were cut and placed on the poly-l-lysine-coated glass slides for ihc staining.
the samples were mounted on the glass slides and ready for ihc staining.
in this research, the dako real™ envision™ detection kit was used.
in total,  <dig> tmaa slides with  <dig> oral cancer cases were stained.
two types of antibodies were used namely monoclonal mouse anti-human p <dig> protein, clone 318-6- <dig> for p <dig> and monoclonal mouse anti-human p <dig> protein, clone 4a <dig> for p <dig>
the results of the staining were analyzed and the images were captured by using an image analyzer system which included nikon eclipse e <dig> microscope with cfi plan fluor 40x objective for measurements, qimaging evolution digital colour cooled camera with  <dig>  megapixels, a personal computer  and mediacybernatics image pro plus version  <dig>  image analysis software.
each slide was first examined under the microscope with lower objective, that is, the 4x objective.
cases were considered sufficient for evaluation if there were tumour cells presented in the sections.
next, the slide was divided into  <dig> grid cells and numbered accordingly from left to right.
a simple randomization program was used to generate random numbers.
for each case, five tumour representative areas were selected.
if the number falls on the non-tumour representative area, the next number  was chosen until all five areas were selected.
next, the five selected areas were examined under the microscope using a higher objective, that is, the 40x objective and the images were captured.
the percentage of the positive nuclear cells for each area was counted and the average for five areas was calculated.
the staining result is considered positive if there is more than 10% positive nuclear stained, in accordance with the practice used in the previous studies  <cit> .
figure  <dig> shows the flowchart for the ihc results analysis and scoring process.
the results obtained from the ihc staining are combined with the clinicopathologic variables and served as the inputs for the feature selection module.
the combined dataset is further divided into two groups, which are group  <dig> with clinicopathologic variables  only and group  <dig> with both of the clinicopathologic and genomic variables .
we need to emphasize that the genomic variables were obtained from the same corresponding cases from which the clinicopathologic variables  were obtained.
thus, if the clinicopathologic variables were that of case  <dig>  then, the genomic variables were from the same case.
feature selection
in this research, the purpose of feature selection is to find an optimal number of features for the small sample of oral cancer prognosis data.
five feature selection methods have been selected and compared, which are, pearson’s correlation coefficient   <cit> , and relief-f  <cit>  as the filter approach, genetic algorithm   <cit>  as the wrapper approach, cc-ga and relieff-ga as the hybrid approach.
genetic algorithm
in the feature subset selection problem, a solution is a specific feature subset that can be encoded as a string of binary digits .
each feature is represented by binary digits of  <dig> or  <dig>  for example, in the oral cancer prognosis dataset, if the solution is a  <dig> string of  <dig> binary digits, it indicates that features  <dig>   <dig>   <dig>  and  <dig> are selected as the feature subset  <cit> .
the initial population is generated randomly to select a subset of variables.
in this proposed ga feature selection method, if the features are all different, the subset is included in the initial population.
if not, it is regenerated until an initial population with the desired size for the feature subset  is created.
the fitness function used in this method is a classifier to discriminate between two groups, which are alive and dead after 3-year of diagnosis.
the mean square error rate of the classification is calculated using a 5-fold cross-validation.
the fitness function is the final mean square error rate obtained.
the subset of variables with the lowest error rate is selected.
figure  <dig> shows the flowchart and the criteria used for the ga feature selection approach.
pearson’s correlation coefficient
pearson’s correlation coefficient, r, is use to see if the values of two variables are associated.
in this research, r is calculated and ranked for each of the feature input and the one with the highest r is selected.
for example, for the 3-input model, the top three inputs with the highest r value are selected.
this is repeated for the n-input models for both group  <dig> and group  <dig>
relief-f
relief-f is the extension to the original relief algorithm, which is able to deal with noisy and incomplete datasets as well as multi-class problems.
the key idea of relief is to estimate attributes according to how well their values distinguish among instances that are near to each other  <cit> .
in this research, each feature input is ranked and weighted using the k-nearest neighbours classification, in which k =  <dig>  the top features with large positive weights are selected for both groups of dataset.
pearson’s correlation coefficient and genetic algorithm
this is the hybrid feature selection approach which consists of two stages: first, it is a filter approach which calculates the correlation coefficient, r, and second, it is a wrapper approach of ga. in the first stage,  <dig> features with the highest r are selected and fed into the second stage of the ga approach.
the procedures of the ga are the same as that described in the previous section.
relief-f and genetic algorithm
this hybrid feature selection approach consists of two stages: first, it is the filter approach of relief-f, and second, it is a wrapper approach of ga. in the first stage,  <dig> features with the highest weights are selected and fed into the second stage of the ga approach.
in the second stage, n-input is selected for both group  <dig> and group  <dig>
selection of n-input models
before the implementation of the feature selection method, a simple ga was run to find out the optimal number of inputs  from the  <dig> inputs of clinicopathologic and genomic data.
the number of inputs with lower mean square error rate was chosen.
the error rate for each n-input model is shown in table  <dig>  which shows that for group  <dig>  there are four models with the lowest error rate of  <dig> , which are the 3-input, 4-input, 5-input, and 6-input model.
meanwhile, for group  <dig>  the model with the lowest error rate is the 3-input model with an error rate of  <dig> .
in this case, for comparison purposes, the number of inputs between 3-input to 7-input are chosen.
hence n is set as n =  <dig>   <dig>   <dig>   <dig>   <dig> for the feature selection methods.
classification
next, the data with n selected features are fed into the classification models.
the final output is the classification accuracy for oral cancer prognosis, which classifies the patients as alive or dead after subsequent years of diagnosis with the optimum feature of subset.
four classification methods were experimented with and their results were subsequently compared, these are anfis, artificial neural network , support vector machine and logistic regression.
in order to obtain accurate estimate results, cross-validation  was used.
cv provides an unbiased estimation, however cv presents high variance with small samples in some studies  <cit> .
in this research, a 5-fold cross-validation was implemented with each of the classifiers.
5-fold cross-validation was chosen over the commonly use 10-fold cross-validation due to the small sample size; hence, it will leave more instances for validation and has lower variance  <cit> .
in 5-fold cross-validation, the  <dig> samples of oral cancer prognosis data were divided into  <dig> subsets of equal size and trained for  <dig> times, each time leaving out a sample as validation data.
adaptive neuro-fuzzy inference system
anfis implements the takagi-sugeno fuzzy inference system.
the details for anfis can be found in  <cit>  respectively.
in the input layer, the number of input is defined by n, with n =  <dig>   <dig>   <dig>   <dig>   <dig>  in the input membership  layer, the number of membership function is defined by mi, with i =  <dig>   <dig>   <dig>  the rules generated are based on the number of input and the number of input membership functions, and it is represented as  rules, in which n <dig>  n <dig>  and n <dig> represent the number of input with mi membership functions respectively, and n1 + n2 + n3 = n. for example, in the anfis with 3-input, x, y, and z, in which input x has  <dig> membership functions, input y has  <dig> membership functions, and input z has  <dig> membership functions, hence the number of rules generated is  =  <dig> rules, as shown in figure  <dig>
the rules generated are the output membership functions which will be computed as the summation of the contribution from each rule towards the overall output.
the output is the survival condition, either alive or dead after 3-year of diagnosis.
the output is set as  <dig> for dead and − <dig> for alive; the pseudo-code is as below:
if output ≥ 0
then set output =  <dig>  classify as dead
else output <  <dig>
then set output = − <dig>  classify as alive
the membership functions were obtained according to the categorical variables that has been set through the discussions with two oral cancer clinicians as mentioned in section clinicopathologic data.
the type of membership function used was the gaussian and the name and parameters of membership functions for each input variable are shown in table  <dig> for the clinicopathologic variables and  <dig> for the genomic variables.
each anfis was run for  <dig> epochs for the optimum result.
artificial neural network
the ann employed in this research is the multi-layered feed forward  neural network, which is the most common type of ann  <cit> .
the ff neural network was trained using the levenberg-marquardt algorithm.
in this research, one hidden layer with five neurons  was used and ff neural network was run for  <dig> epochs .
the training stopped when there was no improvement on the mean squared error for the validation set.
support vector machine
for the purpose of this research, a widely used svm tool which is libsvm  <cit>  was used.
there are  <dig> steps involved in the libsvm;  the dataset was trained to obtain a model and  the model was used to predict the information for the testing dataset.
the details for libsvm can be found in  <cit>  respectively.
linear kernel was used in this research.
logistic regression
logistic regression  was selected as the benchmark test for the statistical method.
lr is the most commonly used statistical method for the prediction of diagnosis and prognosis in medical research.
lr is the prediction of a relationship between the response variable y and the input variables xi <cit> .
in this research, multiple logistic regression is used.
experiment
the oral cancer dataset with 3-year prognosis is used in this experiment.
first, the oral cancer prognosis dataset was divided into two groups; group  <dig> consisted of clinicopathologic variables only  and group  <dig> consisted of clinicopathologic and genomic variables .
next, feature selection methods were implemented on both groups to select the key features for the n-input model.
lastly, the classifiers with 5-fold cross-validation were tested on the n-input model.
the results obtained from the 5-fold cross-validation were averaged in order to produce the overall performance of the algorithm.
the measures used to compare the performance of the proposed methods were sensitivity, specificity, accuracy and area under the receiver operating characteristic  curve .
RESULTS
group  <dig>
table  <dig> shows the features selected for the proposed feature selection methods for group  <dig>  next, the n-input models generated from each feature selection methods were tested with the proposed classification methods.
table  <dig> shows the classification results for anfis, ann, svm and lr.
from table  <dig>  it can be seen that anfis with the cc-4-input model, obtained the best accuracy of  <dig> % and an auc of  <dig> .
for the ann results, the model with the highest accuracy is the relieff-ga-5-input model with an accuracy of  <dig> % and an auc of  <dig> .
whereas, for the svm classifier, the models with the best accuracy are relieff-ga-3-input to 6-input models with an accuracy of  <dig> % and an auc of  <dig> .
as for lr classification, the best model is the cc-ga-6-input model with an accuracy of  <dig> % and an auc of  <dig> .
the results obtained from both ann and svm showed low accuracy  and low auc , hence, indicated that these two are not the suitable classifiers to use for group  <dig>
group  <dig>
the same experiments were carried out on group  <dig>  which is the combination of clinicopathologic and genomic variables.
the selected features for each n-input model are listed in table  <dig>  table  <dig> shows that almost all the feature selection methods included the genomic variable as one of the key features, except for the relieff-3-input and relieff-4-input.
for group  <dig> using the anfis classification , there are five models with an accuracy of above 70%, these are namely, ga-3-input, cc-ga-3-input, cc-ga-4-input, relieff-ga-3-input and relieff-ga-4-input.
the best results were obtained from the relieff-ga-3-input and relieff-ga-4-input with an accuracy of  <dig> % and an auc of  <dig>  and the features selected for the relieff-ga-3-input are drink, invasion, and p <dig> while the features selected for the relieff-ga-4-input are drink, invasion, treatment and p <dig> .
as shown in table  <dig>  the ff neural network together with the relieff-ga-3-input model achieved the best result with an accuracy of  <dig> % and an auc of  <dig> .
for svm classification, the classification results are generally better in group  <dig> when compared to group  <dig>  with some exceptions .
the best accuracy in group  <dig> is obtained by the ga-3-input, cc-ga-3-input, relieff-ga-3-input, and relieff-ga-5-input with an accuracy of  <dig> % and an auc of  <dig> .
whereas, for lr classification in group  <dig>  ga-3-input, cc-ga-3-input, relieff-ga-3-input and relieff-ga-4-input achieved the best classification accuracy of  <dig> % and an auc of  <dig> .
comparison of the results for group  <dig> and group 2
this section discusses and compares the results generated from different classification methods for both group  <dig> and group  <dig>  table  <dig> summarizes the best accuracy for the n-input model based on the feature selection method for group  <dig> and group  <dig>  the summary is also depicted in the graph as shown in figure  <dig> and figure  <dig> respectively.
for group  <dig> , the correlation coefficient  feature selection method performed better than the other methods with the highest accuracy of  <dig> % in the 4-input model.
there are three models that achieved accuracies of above 70%; the other two are ga-3-input and cc-ga-6-input.
relieff feature selection method obtained the worst results when compared to the other methods
as regards to group  <dig> , the relieff-ga feature selection method outperformed the others in all the n-input models, with the highest accuracy of  <dig> %.
there are ten models with accuracies above 70% as shown in table 8; this confirms that group  <dig> which includes genomic variables achieved higher accuracy with feature selection methods.
in addition, most of the models with higher accuracy are the lower input models with  <dig> or 4-input only.
next, table  <dig> lists the best accuracy by classification method and the graphs are depicted in figures  <dig> and  <dig> for both group  <dig> and group  <dig> respectively.
from figure  <dig>  anfis performed the best in group  <dig> when compared to other classification methods for all types of feature selection methods except cc-ga method.
all the classification methods performed worst in relieff feature selection method except for ann.
ann had the lowest accuracy rate if compared to other methods.
whereas, in group  <dig> as shown in figure  <dig>  anfis outperformed the other classification methods except in cc feature selection method.
the best accuracy is achieved by anfis in relieff-ga method with the accuracy of  <dig> % .
in general, all classification methods performed better in cc-ga and relieff-ga hybrid feature selection methods except for svm and lr.
as with group  <dig>  ann had the lowest classification rate except in relieff-ga method.
overall, the performance of the classification method is better in group  <dig> as compared to group  <dig>  table  <dig> summarizes the best models with their selected features for both group  <dig> and group  <dig>  all the models with the accuracy of 70% and above are selected.
from table  <dig>  the models with the highest accuracy are relieff-ga-3-input and relieff-ga-4-input from group  <dig> with anfis classification, the accuracy is  <dig> % and auc of  <dig> .
the features selected are drink, invasion and p <dig> and drink, invasion, treatment, and p <dig> respectively.
this is followed by the relieff-ga-3-input model from group  <dig> with ann classification, with the accuracy of  <dig> % and auc of  <dig> .
most of the best models are generated from the relieff-ga feature selection method; this proves that the features selected by this method are the optimum features for the oral cancer prognosis dataset.
discussions
the results shown meets the objective of this research, namely, the classification performance is much better with the existence of genomic variables in group  <dig>  from the results in table  <dig>  the best feature selection method for oral cancer prognosis is relieff-ga with anfis classification.
this shows that the anfis is the most optimum classification tool for oral cancer prognosis.
since there are two top models with the same accuracy, hence, the simpler one is recommended for further works in similar researches which is the relieff-ga-3-input model with anfis classification, and the optimum subset of features are drink, invasion and p <dig>  these findings confirmed that of some previous studies.
alcohol consumption has always been considered as a risk factor and one of the reasons for poor prognosis of oral cancer  <cit> .
walker d et al.
<cit>  showed that the depth of invasion is one of the most important predictors of lymph node metastasis in tongue cancer.
the different researches done by  <cit> , discovered a significant link between the depth of invasion and oral cancer survival.
as regards to p <dig>   <cit>  showed that p <dig> over expression associates with poor prognosis in oral cancer.
a comparison between the current methodology and the other methods in the literature was done and shown in table  <dig>  nevertheless, direct comparisons cannot be performed since different datasets have been employed in each case.
in this comparison, we compared the studies which utilized at least both types of clinical and genomic data in oral cancer.
in general, the proposed methodology exhibits superior results compared to the other methods except the work done by  <cit>  which claimed to achieve an accuracy of 100%.
however, they employed different classifiers for different source of data and more than  <dig> markers were required for their final combined classifier.
a significant advantage of our proposed methodology is only three optimum markers are needed with a single classifier for both types of clinicopathologic and genomic data to obtain high accuracy result.
it is hope that the proposed methodology is feasible to expedite oral cancer clinicians in the decision support stage and to better predict the survival rate of the oral cancer patients based on the three markers only.
a common problem associated with medical dataset is small sample size.
it is time consuming and costly to obtain large amount of samples in medical research and the samples are usually inconsistent, incomplete or noisy in nature.
the small sample size problem is more visible in the oral cancer research since oral cancer is not one of the top ten most common cancers in malaysia, hence there are not many cases.
for example, in peninsular malaysia, there are only  <dig>  new oral cancer cases from  <dig> to  <dig>  <cit>  and  <dig> new oral cancer cases in the year  <dig>  <cit>  as compared to breast cancer, where the incidence between  <dig> and  <dig> is  <dig>   <cit>  and the incidence for  <dig> is  <dig>   <cit> .
out of these oral cancer cases, some patients are lost to follow-up, some patients seek treatments in other private hospitals and thus, their data are not available for this research.
another reason for small sample size is caused by the medical confidentiality problems.
this can be viewed from two aspects, namely, patients and clinicians.
some patients do not wish to reveal any information about their diseases to others, and are not willing to donate their tissues for research/educational purposes.
as for clinicians, some may not want to share patients’ data with others especially those from the non-medical fields, while some do not keep their medical records in the correct medical form.
from those available cases, some patients’ clinicopathologic data are incomplete, some tissues are missing due to improper management and some are duplicated cases.
due to that, the number of cases that can actually be used for this research is very limited.
in order to overcome the problem of small sample size, we employed the feature selection methods on our dataset to choose the most optimum feature subsets based on the correlations of the input and output variables.
the features selected are fed into the proposed classifier and the results showed that the relieff-ga-anfis prognostic model is suitable for small sample size data with the proposed optimum feature subset of drink, invasion and p <dig>
significance testing
the significance test used in this research was the kruskal-wallis test.
kruskal-wallis is a non-parametric test to compare samples from two or more groups and returns the p-value.
for this research, we wanted to test if there is any statistical significant difference between the accuracy results generated for the 3-input model of group  <dig> for the different feature selection methods.
thus, the null hypothesis is set as: h0 = there is no difference between the results of the different feature selection models.
if the p-value computed from the test is  <dig>  or less, the h <dig> is rejected, which means there is a difference between the results of the different feature selection methods.
the p-value that generated was  <dig> , which is less than  <dig> , this means the h <dig> is rejected and there is a significant difference between the feature selection methods.
validation testing
in this section, the best model of relieff-ga-3-input model is compared with other models with a random permutation of three inputs.
the purpose is to validate that the features selected by the relieff-ga method are the optimum subset for oral cancer prognosis.
in addition, the full-input model  will be tested as well in order to verify that the reduced model can achieve the same or better results than the full model.
in this testing, the classification method used is anfis due to its best performance in the previous section and the results are tabulated in table  <dig>
*n.a.
- results not available due to over-fitting problem as the rule-base generated was too large.
table  <dig> presents the results from different permutation of the 3-input models using anfis and that of the full model with all the  <dig> variables using the different classification methods.
the three inputs are generated randomly and the best accuracy obtained is  <dig> % with an auc of  <dig> .
the features selected are drink, p <dig> and p <dig>  the results of the full model are not promising and the results of full model using anfis cannot be generated due to over-fitting problems as the rule base generated is too large.
finally, the selected features are tested on the oral cancer dataset for 1-year and 2-year with anfis classification and the results are very promising with an accuracy for 1-year prognosis of  <dig> % and 2-year prognosis observed at  <dig> %, as compared to the 3-year prognosis of  <dig> %.
the results are shown in table  <dig>
findings
the analyses and findings from this research are:
the performance of group  <dig>  is better than group  <dig> .
this is in accordance with the objective of this research, which shows that the prognostic result is more accurate with the combination of clinicopathologic and genomic markers.
the model with the best accuracy is the relieff-ga-3-input model with the anfis classification model from group  <dig> and the kruskal-wallis test showed a significant difference as compared to the 3-input model of ga, cc, relieff and cc-ga.
the optimum subset of features for oral cancer prognosis is drink, invasion and p <dig> and this finding is in accordance with similar studies in the literature.
the anfis classification model achieved the best accuracy in oral cancer prognosis when compared to artificial neural network, support vector machine and statistical method of logistic regression.
the prognostic result is more accurate with fewer inputs in comparison with the full model.
as a summary, the hybrid model of relieff-ga-anfis with 3-input features of drink, invasion and p <dig> achieved the best accuracy.
through the identification of fewer markers for oral cancer prognosis, it is hoped that this will aid clinicians in carrying out prognostic procedures, and thus help them in making a more accurate prognosis in a shorter time at lower costs.
furthermore, the results of this research helps patients and their family plan their future and lifestyle through a more reliable prognosis.
CONCLUSIONS
in this research, we presented a prognostic system using the hybrid of feature selection and machine learning methods for the purpose of oral cancer prognosis based on clinicopathologic and genomic markers.
as a conclusion, the hybrid model of relieff-ga-anfis resulted in the best accuracy  with the selected features of drink, invasion and p <dig>  the results proved that the prognosis is more accurate when using the combination of clinicopathologic and genomic markers.
however, more tests and experiments needed to be done in order to further verify the results obtained in this research.
our future works include increasing the sample size of the dataset by providing more medical samples thus making it closer to the real population and including more genomic markers in our study.
competing interests
the authors declare that they have no competing interests.
authors’ contributions
swc developed the prognostic model, performed the experiments and drafted the manuscript.
swc and sak conceived with the study and contributed to the experimental design.
afm and rbz contributed in the analysis and interpretation of oral cancer prognostic dataset.
all authors read and approved the final manuscript.
acknowledgment
this study is supported by the university of malaya research grant  with the project number rg026-09ict.
the authors would like to thank dr mannil thomas abraham from the tunku ampuan rahimah hospital, ministry of health, malaysia, dr thomas george kallarakkal from the department of oral pathology and oral medicine and periodontology, the staff from the oral & maxillofacial surgery department, the oral pathology diagnostic laboratory, the ocrcc, the faculty of dentistry, and the ent department, faculty of medicine, university of malaya for the preparation of the dataset and the related data and documents for this project.

 various machine learning methods have been applied in the diagnosis or prognosis of cancer research, such as, artificial neural networks, fuzzy logic, genetic algorithm, support vector machine and other hybrid techniques. from the medical perspective, diagnosis is to identify a disease by its signs and symptoms while prognosis is to predict the outcome of the disease and the status of the patient, whether the patient will survive or recover from the disease or vice versa. in some studies, the researchers have proven that machine learning methods could generate more accurate diagnosis or prognosis as compared to traditional statistical methods. usually, clinicopathologic data or genomic data are used in researches either involving diagnosis or that with respect to prognosis. currently, there are some researches that have shown that prognosis results are more accurate when using both clinicopathologic and genomic data. examples of these are the works in in diffuse large b-cell lymphoma cancer, the works in in breast cancer, in oral cancer, and in bladder cancer. however, the number of published articles on researches that combine both clinicopathologic and genomic data are few as compared to that using only clinicopathologic data. in the oral cancer domain, used machine learning techniques in the oral cancer susceptibility studies. they proposed a hybrid adaptive system inspired from learning classifier system, decision trees and statistical hypothesis testing. the dataset includes both demographic data and types of genes. their results showed that the proposed algorithm outperformed the other algorithms of naive bayes, c, neural network and xcs. however, they did not validate their results against the traditional statistical methods. focused on the 5-year overall survival in a group of oral squamous cell carcinoma patients and investigated the effects of demographic data, clinical data and genomic data, and human papillomavirus on the prognostic outcome. they used the statistical methods for the prediction and their results showed that the 5-year overall survival was % and highlighted the influence of p immunoexpression, age and anatomic localization on ocss prognosis. however, in this research, no machine learning methods were used and compared. another oral cancer research that was done by was in the oral cancer reoccurrence. bayesian network was used and compared with ann, svm, decision tree, and random forests. they used multitude of heterogeneous data which included clinical, imaging, tissue and blood genomic data. they built a separate classifier for different types of data and combined the best performing classification schemes. they claimed that they had achieved an accuracy of 100% with the combination of all types of data and proved that the prediction accuracy was the best when using all types of data. however, more than markers were required for their final combined classifier. for the genomic domain, showed that p over expression associates with poor prognosis in oral cancer. their study showed that cases with diffuse p expression were more aggressive and poorly differentiated and related to a poorer prognosis, these findings supporting the use of p as an additional marker for diagnostic use in oral scc. in, immunohistochemical analysis of protein expression for p p and p was performed for samples of well-differentiated human buccal squamous-cell carcinomas, with specimens of normal buccal mucosa employed as controls. their results indicated that both p and p may be involved in the development of human buccal squamous-cell carcinoma, perhaps in concert with p similar results were obtained by, they have showed that in head and neck squamous carcinomas, p was the most frequently expressed, followed by p and p. their study indicated that p and p expression may represent an early event in hnsc tumorigenesis and p and p may function as oncogenes in the development of these tumors. in this research, an oral cancer prognostic model is developed. this research used real-world oral cancer dataset which is collected locally at the oral cancer research and coordinating centre, faculty of dentistry, university of malaya, malaysia. the model takes both clinicopathologic and genomic data in order to investigate the impact of each marker or combination of markers to the accuracy of the prognosis of oral cancer. five feature selection methods are proposed with the objectives to reduce the number of input variables to avoid over-fitting and to find out an optimum feature subset for oral cancer prognosis. this is followed by the classification procedures which are used to classify the status of the patient after 1–3 years of diagnosis. four classification methods, from both machine learning and statistical methods, are tested and compared. the objective of this research is to prove that the prognosis is better by using both types of clinicopathologic and genomic markers, and to identify the key markers for oral cancer prognosis using the hybrid of feature selection and machine learning methods. methods the framework for the oral cancer prognostic model is shown in figure  clinicopathologic variables from the ocrcc database and genomic variables from immunohistochemistry staining are fed into the model. basically, there are three main parts for the oral cancer prognostic model, which are wet-laboratory testing for genomic variables, feature selection methods and the classification models. this research was approved by medical ethics committee, faculty of dentistry, university of malaya. clinicopathologic data a total of oral cancer cases were selected from the malaysian oral cancer database and tissue bank system coordinated by the oral cancer research and coordinating centre, faculty of dentistry, university of malaya. the selection was based on the completeness of the clinicopathologic data, the availability of tissues and the availability of data. the selected cases were based on the oral cancer cases seen in the faculty of dentistry, university of malaya and hospital tunku ampuan rahimah, klang, a malaysian government hospital, from the year to these cases were diagnosed and followed up and the data were recorded in the standardised forms prepared by the mocdtbs. later, mocdtbs transcribed all the data from paper to an electronic version and stored in the database. all the cases selected were diagnosed as squamous cell carcinomas. table  shows the to 3-year survival for these cases. basically, three types of data are available for each oral cancer case, namely, social demographic data, clinical data, and pathological data. pathological data were obtained from the biopsy reports before and after surgical procedures. in this research, we referred to the clinical and pathological data as clinicopathologic data. based on the discussions with two oral cancer clinicians, prof. rosnah binti zain and dr thomas george kallarakkal, key variables had been identified as important prognostic factors of oral cancer. the selected clinicopathologic variables are listed in table  genomic data two genomic variables had been identified through the literature studies and discussions with oral pathologists, from the department of oral pathology and oral medicine and periodontology, faculty of dentistry, university of malaya. both of these variables are tumour suppressor genes, namely, p and p p is the most frequently associated marker in the head and neck cancers. p is called the “guardian of the genome”, it is important in maintaining genomic stability, progression of cell cycle, cellular differentiation, dna repair and apoptosis. it is difficult to demonstrate p protein in normal tissues using immunohistochemistry procedures due to its high catabolic rate; however mutated p exhibits a much lower catabolic rate and accumulates in the cells. in addition, p gene, a homolog of the p is located in chromosome 3q21- and its amplification has been associated with prognostic outcome in oral cancer. the p gene is highly expressed in the basal or progenitor layers of many epithelial tissues. the cases selected were the same as in the clinicopathologic data. immunohistochemistry staining was performed on the selected formalin-fixed paraffin embedded oral cancer tissues to obtain the results for the selected genomic variables. the archival formalin-fixed paraffin embedded tissues were obtained from the oral pathology diagnostic laboratory, faculty of dentistry, university of malaya. the tissues containing the tumour were cored and re-embedded and made into tissue macroarray blocks. a total of 4-μm-thick sections of the resulting tmaa blocks were cut and placed on the poly-l-lysine-coated glass slides for ihc staining. the samples were mounted on the glass slides and ready for ihc staining. in this research, the dako real™ envision™ detection kit was used. in total, tmaa slides with oral cancer cases were stained. two types of antibodies were used namely monoclonal mouse anti-human p protein, clone 318-6- for p and monoclonal mouse anti-human p protein, clone 4a for p the results of the staining were analyzed and the images were captured by using an image analyzer system which included nikon eclipse e microscope with cfi plan fluor 40x objective for measurements, qimaging evolution digital colour cooled camera with megapixels, a personal computer and mediacybernatics image pro plus version image analysis software. each slide was first examined under the microscope with lower objective, that is, the 4x objective. cases were considered sufficient for evaluation if there were tumour cells presented in the sections. next, the slide was divided into grid cells and numbered accordingly from left to right. a simple randomization program was used to generate random numbers. for each case, five tumour representative areas were selected. if the number falls on the non-tumour representative area, the next number was chosen until all five areas were selected. next, the five selected areas were examined under the microscope using a higher objective, that is, the 40x objective and the images were captured. the percentage of the positive nuclear cells for each area was counted and the average for five areas was calculated. the staining result is considered positive if there is more than 10% positive nuclear stained, in accordance with the practice used in the previous studies. figure  shows the flowchart for the ihc results analysis and scoring process. the results obtained from the ihc staining are combined with the clinicopathologic variables and served as the inputs for the feature selection module. the combined dataset is further divided into two groups, which are group with clinicopathologic variables only and group with both of the clinicopathologic and genomic variables. we need to emphasize that the genomic variables were obtained from the same corresponding cases from which the clinicopathologic variables were obtained. thus, if the clinicopathologic variables were that of case then, the genomic variables were from the same case. feature selection in this research, the purpose of feature selection is to find an optimal number of features for the small sample of oral cancer prognosis data. five feature selection methods have been selected and compared, which are, pearson’s correlation coefficient, and relief-f as the filter approach, genetic algorithm as the wrapper approach, cc-ga and relieff-ga as the hybrid approach. genetic algorithm in the feature subset selection problem, a solution is a specific feature subset that can be encoded as a string of binary digits. each feature is represented by binary digits of or for example, in the oral cancer prognosis dataset, if the solution is a string of binary digits, it indicates that features and are selected as the feature subset. the initial population is generated randomly to select a subset of variables. in this proposed ga feature selection method, if the features are all different, the subset is included in the initial population. if not, it is regenerated until an initial population with the desired size for the feature subset is created. the fitness function used in this method is a classifier to discriminate between two groups, which are alive and dead after 3-year of diagnosis. the mean square error rate of the classification is calculated using a 5-fold cross-validation. the fitness function is the final mean square error rate obtained. the subset of variables with the lowest error rate is selected. figure  shows the flowchart and the criteria used for the ga feature selection approach. pearson’s correlation coefficient pearson’s correlation coefficient, r, is use to see if the values of two variables are associated. in this research, r is calculated and ranked for each of the feature input and the one with the highest r is selected. for example, for the 3-input model, the top three inputs with the highest r value are selected. this is repeated for the n-input models for both group and group relief-f relief-f is the extension to the original relief algorithm, which is able to deal with noisy and incomplete datasets as well as multi-class problems. the key idea of relief is to estimate attributes according to how well their values distinguish among instances that are near to each other. in this research, each feature input is ranked and weighted using the k-nearest neighbours classification, in which k =  the top features with large positive weights are selected for both groups of dataset. pearson’s correlation coefficient and genetic algorithm this is the hybrid feature selection approach which consists of two stages: first, it is a filter approach which calculates the correlation coefficient, r, and second, it is a wrapper approach of ga. in the first stage, features with the highest r are selected and fed into the second stage of the ga approach. the procedures of the ga are the same as that described in the previous section. relief-f and genetic algorithm this hybrid feature selection approach consists of two stages: first, it is the filter approach of relief-f, and second, it is a wrapper approach of ga. in the first stage, features with the highest weights are selected and fed into the second stage of the ga approach. in the second stage, n-input is selected for both group and group selection of n-input models before the implementation of the feature selection method, a simple ga was run to find out the optimal number of inputs from the inputs of clinicopathologic and genomic data. the number of inputs with lower mean square error rate was chosen. the error rate for each n-input model is shown in table  which shows that for group there are four models with the lowest error rate of, which are the 3-input, 4-input, 5-input, and 6-input model. meanwhile, for group the model with the lowest error rate is the 3-input model with an error rate of. in this case, for comparison purposes, the number of inputs between 3-input to 7-input are chosen. hence n is set as n =  for the feature selection methods. classification next, the data with n selected features are fed into the classification models. the final output is the classification accuracy for oral cancer prognosis, which classifies the patients as alive or dead after subsequent years of diagnosis with the optimum feature of subset. four classification methods were experimented with and their results were subsequently compared, these are anfis, artificial neural network, support vector machine and logistic regression. in order to obtain accurate estimate results, cross-validation was used. cv provides an unbiased estimation, however cv presents high variance with small samples in some studies. in this research, a 5-fold cross-validation was implemented with each of the classifiers. 5-fold cross-validation was chosen over the commonly use 10-fold cross-validation due to the small sample size; hence, it will leave more instances for validation and has lower variance. in 5-fold cross-validation, the samples of oral cancer prognosis data were divided into subsets of equal size and trained for times, each time leaving out a sample as validation data. adaptive neuro-fuzzy inference system anfis implements the takagi-sugeno fuzzy inference system. the details for anfis can be found in respectively. in the input layer, the number of input is defined by n, with n =  in the input membership layer, the number of membership function is defined by mi, with i =  the rules generated are based on the number of input and the number of input membership functions, and it is represented as rules, in which n n and n represent the number of input with mi membership functions respectively, and n1 + n2 + n3 = n. for example, in the anfis with 3-input, x, y, and z, in which input x has membership functions, input y has membership functions, and input z has membership functions, hence the number of rules generated is  =  rules, as shown in figure  the rules generated are the output membership functions which will be computed as the summation of the contribution from each rule towards the overall output. the output is the survival condition, either alive or dead after 3-year of diagnosis. the output is set as for dead and − for alive; the pseudo-code is as below: if output ≥ 0 then set output =  classify as dead else output <  then set output = − classify as alive the membership functions were obtained according to the categorical variables that has been set through the discussions with two oral cancer clinicians as mentioned in section clinicopathologic data. the type of membership function used was the gaussian and the name and parameters of membership functions for each input variable are shown in table  for the clinicopathologic variables and for the genomic variables. each anfis was run for epochs for the optimum result. artificial neural network the ann employed in this research is the multi-layered feed forward neural network, which is the most common type of ann. the ff neural network was trained using the levenberg-marquardt algorithm. in this research, one hidden layer with five neurons was used and ff neural network was run for epochs. the training stopped when there was no improvement on the mean squared error for the validation set. support vector machine for the purpose of this research, a widely used svm tool which is libsvm was used. there are steps involved in the libsvm; the dataset was trained to obtain a model and the model was used to predict the information for the testing dataset. the details for libsvm can be found in respectively. linear kernel was used in this research. logistic regression logistic regression was selected as the benchmark test for the statistical method. lr is the most commonly used statistical method for the prediction of diagnosis and prognosis in medical research. lr is the prediction of a relationship between the response variable y and the input variables xi. in this research, multiple logistic regression is used. experiment the oral cancer dataset with 3-year prognosis is used in this experiment. first, the oral cancer prognosis dataset was divided into two groups; group consisted of clinicopathologic variables only and group consisted of clinicopathologic and genomic variables. next, feature selection methods were implemented on both groups to select the key features for the n-input model. lastly, the classifiers with 5-fold cross-validation were tested on the n-input model. the results obtained from the 5-fold cross-validation were averaged in order to produce the overall performance of the algorithm. the measures used to compare the performance of the proposed methods were sensitivity, specificity, accuracy and area under the receiver operating characteristic curve.  group table  shows the features selected for the proposed feature selection methods for group next, the n-input models generated from each feature selection methods were tested with the proposed classification methods. table  shows the classification results for anfis, ann, svm and lr. from table  it can be seen that anfis with the cc-4-input model, obtained the best accuracy of % and an auc of. for the ann results, the model with the highest accuracy is the relieff-ga-5-input model with an accuracy of % and an auc of. whereas, for the svm classifier, the models with the best accuracy are relieff-ga-3-input to 6-input models with an accuracy of % and an auc of. as for lr classification, the best model is the cc-ga-6-input model with an accuracy of % and an auc of. the results obtained from both ann and svm showed low accuracy and low auc, hence, indicated that these two are not the suitable classifiers to use for group group the same experiments were carried out on group which is the combination of clinicopathologic and genomic variables. the selected features for each n-input model are listed in table  table  shows that almost all the feature selection methods included the genomic variable as one of the key features, except for the relieff-3-input and relieff-4-input. for group using the anfis classification, there are five models with an accuracy of above 70%, these are namely, ga-3-input, cc-ga-3-input, cc-ga-4-input, relieff-ga-3-input and relieff-ga-4-input. the best results were obtained from the relieff-ga-3-input and relieff-ga-4-input with an accuracy of % and an auc of and the features selected for the relieff-ga-3-input are drink, invasion, and p while the features selected for the relieff-ga-4-input are drink, invasion, treatment and p. as shown in table  the ff neural network together with the relieff-ga-3-input model achieved the best result with an accuracy of % and an auc of. for svm classification, the classification results are generally better in group when compared to group with some exceptions. the best accuracy in group is obtained by the ga-3-input, cc-ga-3-input, relieff-ga-3-input, and relieff-ga-5-input with an accuracy of % and an auc of. whereas, for lr classification in group ga-3-input, cc-ga-3-input, relieff-ga-3-input and relieff-ga-4-input achieved the best classification accuracy of % and an auc of. comparison of the results for group and group 2 this section discusses and compares the results generated from different classification methods for both group and group table  summarizes the best accuracy for the n-input model based on the feature selection method for group and group the summary is also depicted in the graph as shown in figure  and figure  respectively. for group, the correlation coefficient feature selection method performed better than the other methods with the highest accuracy of % in the 4-input model. there are three models that achieved accuracies of above 70%; the other two are ga-3-input and cc-ga-6-input. relieff feature selection method obtained the worst results when compared to the other methods as regards to group, the relieff-ga feature selection method outperformed the others in all the n-input models, with the highest accuracy of %. there are ten models with accuracies above 70% as shown in table 8; this confirms that group which includes genomic variables achieved higher accuracy with feature selection methods. in addition, most of the models with higher accuracy are the lower input models with or 4-input only. next, table  lists the best accuracy by classification method and the graphs are depicted in figures  and for both group and group respectively. from figure  anfis performed the best in group when compared to other classification methods for all types of feature selection methods except cc-ga method. all the classification methods performed worst in relieff feature selection method except for ann. ann had the lowest accuracy rate if compared to other methods. whereas, in group as shown in figure  anfis outperformed the other classification methods except in cc feature selection method. the best accuracy is achieved by anfis in relieff-ga method with the accuracy of %. in general, all classification methods performed better in cc-ga and relieff-ga hybrid feature selection methods except for svm and lr. as with group ann had the lowest classification rate except in relieff-ga method. overall, the performance of the classification method is better in group as compared to group table  summarizes the best models with their selected features for both group and group all the models with the accuracy of 70% and above are selected. from table  the models with the highest accuracy are relieff-ga-3-input and relieff-ga-4-input from group with anfis classification, the accuracy is % and auc of. the features selected are drink, invasion and p and drink, invasion, treatment, and p respectively. this is followed by the relieff-ga-3-input model from group with ann classification, with the accuracy of % and auc of. most of the best models are generated from the relieff-ga feature selection method; this proves that the features selected by this method are the optimum features for the oral cancer prognosis dataset. discussions the results shown meets the objective of this research, namely, the classification performance is much better with the existence of genomic variables in group from the results in table  the best feature selection method for oral cancer prognosis is relieff-ga with anfis classification. this shows that the anfis is the most optimum classification tool for oral cancer prognosis. since there are two top models with the same accuracy, hence, the simpler one is recommended for further works in similar researches which is the relieff-ga-3-input model with anfis classification, and the optimum subset of features are drink, invasion and p these findings confirmed that of some previous studies. alcohol consumption has always been considered as a risk factor and one of the reasons for poor prognosis of oral cancer. walker d et al. showed that the depth of invasion is one of the most important predictors of lymph node metastasis in tongue cancer. the different researches done by, discovered a significant link between the depth of invasion and oral cancer survival. as regards to p showed that p over expression associates with poor prognosis in oral cancer. a comparison between the current methodology and the other methods in the literature was done and shown in table  nevertheless, direct comparisons cannot be performed since different datasets have been employed in each case. in this comparison, we compared the studies which utilized at least both types of clinical and genomic data in oral cancer. in general, the proposed methodology exhibits superior results compared to the other methods except the work done by which claimed to achieve an accuracy of 100%. however, they employed different classifiers for different source of data and more than markers were required for their final combined classifier. a significant advantage of our proposed methodology is only three optimum markers are needed with a single classifier for both types of clinicopathologic and genomic data to obtain high accuracy result. it is hope that the proposed methodology is feasible to expedite oral cancer clinicians in the decision support stage and to better predict the survival rate of the oral cancer patients based on the three markers only. a common problem associated with medical dataset is small sample size. it is time consuming and costly to obtain large amount of samples in medical research and the samples are usually inconsistent, incomplete or noisy in nature. the small sample size problem is more visible in the oral cancer research since oral cancer is not one of the top ten most common cancers in malaysia, hence there are not many cases. for example, in peninsular malaysia, there are only new oral cancer cases from to and new oral cancer cases in the year as compared to breast cancer, where the incidence between and is and the incidence for is. out of these oral cancer cases, some patients are lost to follow-up, some patients seek treatments in other private hospitals and thus, their data are not available for this research. another reason for small sample size is caused by the medical confidentiality problems. this can be viewed from two aspects, namely, patients and clinicians. some patients do not wish to reveal any information about their diseases to others, and are not willing to donate their tissues for research/educational purposes. as for clinicians, some may not want to share patients’ data with others especially those from the non-medical fields, while some do not keep their medical records in the correct medical form. from those available cases, some patients’ clinicopathologic data are incomplete, some tissues are missing due to improper management and some are duplicated cases. due to that, the number of cases that can actually be used for this research is very limited. in order to overcome the problem of small sample size, we employed the feature selection methods on our dataset to choose the most optimum feature subsets based on the correlations of the input and output variables. the features selected are fed into the proposed classifier and the results showed that the relieff-ga-anfis prognostic model is suitable for small sample size data with the proposed optimum feature subset of drink, invasion and p significance testing the significance test used in this research was the kruskal-wallis test. kruskal-wallis is a non-parametric test to compare samples from two or more groups and returns the p-value. for this research, we wanted to test if there is any statistical significant difference between the accuracy results generated for the 3-input model of group for the different feature selection methods. thus, the null hypothesis is set as: h0 = there is no difference between the results of the different feature selection models. if the p-value computed from the test is or less, the h is rejected, which means there is a difference between the results of the different feature selection methods. the p-value that generated was, which is less than, this means the h is rejected and there is a significant difference between the feature selection methods. validation testing in this section, the best model of relieff-ga-3-input model is compared with other models with a random permutation of three inputs. the purpose is to validate that the features selected by the relieff-ga method are the optimum subset for oral cancer prognosis. in addition, the full-input model will be tested as well in order to verify that the reduced model can achieve the same or better results than the full model. in this testing, the classification method used is anfis due to its best performance in the previous section and the results are tabulated in table  *n.a. - results not available due to over-fitting problem as the rule-base generated was too large. table  presents the results from different permutation of the 3-input models using anfis and that of the full model with all the variables using the different classification methods. the three inputs are generated randomly and the best accuracy obtained is % with an auc of. the features selected are drink, p and p the results of the full model are not promising and the results of full model using anfis cannot be generated due to over-fitting problems as the rule base generated is too large. finally, the selected features are tested on the oral cancer dataset for 1-year and 2-year with anfis classification and the results are very promising with an accuracy for 1-year prognosis of % and 2-year prognosis observed at %, as compared to the 3-year prognosis of %. the results are shown in table  findings the analyses and findings from this research are: the performance of group is better than group. this is in accordance with the objective of this research, which shows that the prognostic result is more accurate with the combination of clinicopathologic and genomic markers. the model with the best accuracy is the relieff-ga-3-input model with the anfis classification model from group and the kruskal-wallis test showed a significant difference as compared to the 3-input model of ga, cc, relieff and cc-ga. the optimum subset of features for oral cancer prognosis is drink, invasion and p and this finding is in accordance with similar studies in the literature. the anfis classification model achieved the best accuracy in oral cancer prognosis when compared to artificial neural network, support vector machine and statistical method of logistic regression. the prognostic result is more accurate with fewer inputs in comparison with the full model. as a summary, the hybrid model of relieff-ga-anfis with 3-input features of drink, invasion and p achieved the best accuracy. through the identification of fewer markers for oral cancer prognosis, it is hoped that this will aid clinicians in carrying out prognostic procedures, and thus help them in making a more accurate prognosis in a shorter time at lower costs. furthermore, the results of this research helps patients and their family plan their future and lifestyle through a more reliable prognosis.  in this research, we presented a prognostic system using the hybrid of feature selection and machine learning methods for the purpose of oral cancer prognosis based on clinicopathologic and genomic markers. as a conclusion, the hybrid model of relieff-ga-anfis resulted in the best accuracy with the selected features of drink, invasion and p the results proved that the prognosis is more accurate when using the combination of clinicopathologic and genomic markers. however, more tests and experiments needed to be done in order to further verify the results obtained in this research. our future works include increasing the sample size of the dataset by providing more medical samples thus making it closer to the real population and including more genomic markers in our study. competing interests the authors declare that they have no competing interests. authors’ contributions swc developed the prognostic model, performed the experiments and drafted the manuscript. swc and sak conceived with the study and contributed to the experimental design. afm and rbz contributed in the analysis and interpretation of oral cancer prognostic dataset. all authors read and approved the final manuscript. acknowledgment this study is supported by the university of malaya research grant with the project number rg026-09ict. the authors would like to thank dr mannil thomas abraham from the tunku ampuan rahimah hospital, ministry of health, malaysia, dr thomas george kallarakkal from the department of oral pathology and oral medicine and periodontology, the staff from the oral & maxillofacial surgery department, the oral pathology diagnostic laboratory, the ocrcc, the faculty of dentistry, and the ent department, faculty of medicine, university of malaya for the preparation of the dataset and the related data and documents for this project.
the hybrid model of relieff-ga-anfis with 3-input features of drink, invasion and p achieved the best accuracy for the oral cancer prognosis. however, it is not easy for the most skilful clinician to come out with an accurate prognosis by using these markers alone. the main aim of this research is to apply a hybrid of feature selection and machine learning methods in oral cancer prognosis based on the parameters of the correlation of clinicopathologic and genomic markers. in the first stage of this research, five feature selection methods have been proposed and experimented on the oral cancer prognosis dataset. thus, there is a need to use genomic markers to improve the accuracy of prognosis. the results revealed that the prognosis is superior with the presence of both clinicopathologic and genomic markers. four types of classifiers are chosen; these are namely, anfis, artificial neural network, support vector machine and logistic regression. machine learning techniques are becoming useful as an alternative approach to conventional medical diagnosis or prognosis as they are good for handling noisy and incomplete data, and significant results can be attained despite a small sample size. a k-fold cross-validation is implemented on all types of classifiers due to the small sample size. in the second stage, the model with the features selected from each feature selection methods are tested on the proposed classifiers.
 machine learning techniques are becoming useful as an alternative approach to conventional medical diagnosis or prognosis as they are good for handling noisy and incomplete data, and significant results can be attained despite a small sample size. traditionally, clinicians make prognostic decisions based on clinicopathologic markers. however, it is not easy for the most skilful clinician to come out with an accurate prognosis by using these markers alone. thus, there is a need to use genomic markers to improve the accuracy of prognosis. the main aim of this research is to apply a hybrid of feature selection and machine learning methods in oral cancer prognosis based on the parameters of the correlation of clinicopathologic and genomic markers.  in the first stage of this research, five feature selection methods have been proposed and experimented on the oral cancer prognosis dataset. in the second stage, the model with the features selected from each feature selection methods are tested on the proposed classifiers. four types of classifiers are chosen; these are namely, anfis, artificial neural network, support vector machine and logistic regression. a k-fold cross-validation is implemented on all types of classifiers due to the small sample size. the hybrid model of relieff-ga-anfis with 3-input features of drink, invasion and p achieved the best accuracy for the oral cancer prognosis.  the results revealed that the prognosis is superior with the presence of both clinicopathologic and genomic markers. the selected features can be investigated further to validate the potential of becoming as significant prognostic signature in the oral cancer studies. oral cancer prognosisclinicopathologicgenomicfeature selectionmachine learning 