BACKGROUND
the 'omics'  are making significant contributions to the study of chronic diseases, especially the identification of novel biomarkers.
a biomarker is defined as a characteristic that may be objectively measured and evaluated as an indicator of normal biologic processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention  <cit> .
biomarkers are actively investigated in the areas of clinical rheumatology, orthopaedics and sports medicine.
osteoarthritis  is a degenerative joint disease that affects the entire joint structure  <cit> .
it is characterised by progressive degeneration of cartilage, menisci, ligaments and subchondral bone  <cit> .
synovial inflammation  is a major contributor to disease progression  <cit>  and is responsible for the increased production of catabolic and pro-inflammatory mediators that alter the balance of cartilage matrix degradation and repair, leading to excess production of the proteolytic enzymes responsible for cartilage breakdown  <cit> .
oa is currently diagnosed by radiography, once clinical signs of pain and loss of mobility have already appeared, and therefore biomarkers that could identify early signs of oa would significantly aid in diagnosis  <cit> .
current research is aimed at identifying panels of clinically useful biochemical and imaging markers into single diagnostic algorithms that can be used for diagnostic and prognostic applications and for testing the efficacy of new drugs  <cit> .
applying ‘omics’ results in the generation of large datasets that are suitable for bioinformatic analysis using machine learning, to extract important information  <cit> .
bioinformatics tools play an important role in the analysis of data from omics technologies, such as microarrays, next generation sequencing and mass spectrometry , and as a result a wide range of methods have been developed  <cit> .
such methods include supervised machine learning  techniques, which are used to build classification models.
models are used to automatically label samples of unknown class by using a training set of known labelled samples.
there are many different types of ml methods, some of which can be used to identify putative biomarkers from data by observing the attributes  used to build the models.
rule-based methods are an example of this, as it is possible to read the rules generated to form the model  <cit> .
biohel is a rule based machine learning method which has been used for sample classification in highly dimensional datasets because of its fine-grained embedded feature selection  <cit> .
it has been successfully applied to accurately classify many different types of biological data  <cit> .
rule-based methods construct rule sets that contain at least one rule for each sample group, based on the values associated with the attributes, for example the expression value of the genes.
an example of a rule set is shown in figure  <dig>  ml can also be used to identify possible biomarkers in the form of feature selection , a method of data reduction.
fs techniques identify a subset of attributes, for example genes or proteins, which could be used to build a more successful model, compared to using the whole dataset.
supervised fs methods analyse data with known class labels, with the aim to remove irrelevant or redundant features.
using fs techniques can improve model performance, increase classification accuracy, and provide a clearer understanding of attributes, which are useful for generating an effective classification model.
their key challenge is to avoid a loss of information by dropping features that contain crucial information for the data being analysed.
the application of fs methods is particularly relevant for transcriptomic datasets that often have very large numbers of attributes, in relation to the number of samples.
there are three main forms of feature selection: filter, wrapper and embedded methods  <cit> .
filter methods, such as correlation-based feature selection  <cit> , are used to estimate the effectiveness of a reduction in attributes.
wrapper methods, including genetic algorithms  <cit> , use classifiers to determine if the subset of attributes gives successful classifications.
the third method, embedded, includes feature reduction within the process of classification and uses such machine learning techniques as naïve bayes  <cit>  and support vector machines   <cit> .
the aim of this study was to develop an fs heuristic specialized on identifying very reduced sets of variables built on top of the biohel rule-based ml method, in order to increase the accuracy of classification models and to identify putative biomarkers.
this heuristic performs an iterative feature elimination process that is guided  by an analysis of the rule sets generated by biohel.
this guiding process avoids numerous iterations of blind trial-and-error attempts at removing features from the dataset, and quickly finds, in most datasets, very reduced subsets of features.
we call this method rgife .
other work in this area has involved iterative processes for feature reduction, however these have been based on different methods  <cit> .
one example of such a process for feature reduction is support vector machines recursive feature elimination , which is a form of backward feature elimination.
svm rfe has three main steps.
firstly it trains the svm classifier, followed by computing a specific ranking criterion for all features.
this criterion is based on the weights of a linear svm.
the features are ranked based on this value and the feature with the smallest ranking criterion are removed.
these steps are then repeated in an iterative process, resulting in a reduced number of features.
the rgife method is tested on several proteomics and transcriptomics oa datasets to demonstrate its suitability for multiple data types.
its performance is compared to several combinations of feature selection and classification algorithms.
the results show that rgife improves biohel’s performance in all datasets, is able to identify very reduced sets of variables in most of them and shows competitive performance to other methods from the literature.
methods
proteomics and transcriptomics datasets
a proteomics dataset was selected, which had been previously analysed by biohel to identify putative biomarkers  <cit> .
the proteomics dataset analysed here consisted of  <dig> samples analysed by mass spectrometry to identify the proteins present in each sample  <cit> .
the dataset included six samples treated with the pro-inflammatory cytokine interleukin- <dig> beta , to simulate oa in culture, and five samples treated with il-1β followed by carprofen, which is a non-steroidal anti-inflammatory drug  used to treat oa.
also included in the study were six control samples and six samples treated with only carprofen.
mascot was applied, using the uniprot database, to determine proteins present in the samples.
from this two datasets were generated.
the first included an empai quantitation value assigned to each protein present and alternatively, the second provided each protein with a proteinprophet probability  <cit> .
this probability is a measure of how likely it is each protein is present in a sample.
to assess the method developed on transcriptomics data, datasets were identified from arrayexpress  <cit>  and ncbi geo  <cit>  by searching for the term ‘osteoarthritis’.
of the datasets that this search term returned, those that had more than five samples per group  were analysed.
this resulted in five datasets  that vary in size, but which all contain many more attributes than the proteomics dataset.
the sample numbers also vary between  <dig> samples over  <dig> classes to  <dig> samples over  <dig> classes.
these datasets were classified using biohel and other machine learning methods, as the canine proteomic dataset was, reported in a recent paper by swan et al.,  <cit> .
all datasets were partitioned into training and test sets following the leave-one-out cross-validation methodology.
the rule-guided iterative feature elimination heuristic
with the goal of identifying very reduced and highly accurate sets of variables we propose a feature selection heuristic built on top of the biohel rule-based machine learning method and based on the iterative feature elimination  principle.
the basic ife method would pick an attribute, remove it from the dataset and train a classifier  with the remaining attributes.
if the prediction capacity of the new model is equal or better than with the whole set of attributes , the attribute is eliminated.
otherwise it is inserted back into the dataset.
the basic ife process becomes extremely computationally demanding in high dimensional datasets, such as the case of omics data.
therefore, in order to make such ife heuristic feasible it needs to be improved in several directions.
the main change is that, rather than picking attributes to be removed at random, we will pick them based on their relevance, estimated from the rule sets generated by biohel from the whole set.
then, the first attribute to be picked for elimination are those at the bottom of biohel’s ranking.
the second change is remove attributes in blocks, rather than one by one.
at the start of the heuristic the block size is set up to a very large number, 25% of the problem’s attributes.
if a block cannot be removed because it would decrease the prediction capacity of the classification model then the next block  is tested.
the block size is reduced to be 25% of the previous block size whenever  all attributes have been tested or  five consecutive unsuccessful trials have been attempted.
as a final change the acceptance criteria of a trial is relaxed in certain scenarios: whenever five consecutive unsuccessful trials have been performed, before reducing the block size the heuristic checks if one of these trials suffered an accuracy drop corresponding to just one sample.
if so, this trial is accepted.
the iterative process then finishes when the attributes are being removed one at a time and either the removal of all attributes left in the dataset had been tested or five iterations in a row resulted in a drop in the percentage classification accuracy equivalent to more than one sample.
the overall workflow of the heuristic is represented in figure  <dig>
experimental design
for the experiments of this paper our aim is to show that the rgife heuristic is both able to find feature subsets that are both  small and  highly accurate.
to this aim we have designed a series of experiments that firstly compares the heuristic with other fs methods, and then compares the accuracy obtained by biohel with the reduced feature subsets with other machine learning methods.
the feature selection techniques used were correlation-based , svm rfe, random forest, naïve bayes and chi squared, some of which are feature selection methods based on the machine learning methods used.
all methods used were those implemented in the machine learning software, weka  <cit> .
for those methods that ranked their selected attributes, rather than identifying a small selection of attributes, the top  <dig> features were used.
the machine leaning methods included in the comparison are naïve bayes, support vector machines , k-nearest neighbour , jrip , j <dig> and random forest , also using weka implementations  <cit> .
for all classifications performed, the true positive rate  and true negative rate  were calculated.
the tpr is the measure of the proportion of correctly classified samples and multiplying it by  <dig> gives the percentage classification accuracy.
tpr is calculated by:
in contrast, the tnr is a measure of the proportion of negatives that are correctly classified.
tnr is calculated by:
the combinations of methods were compared in two ways.
firstly, the classification accuracies were compared, to determine which ones gave the highest accuracy.
then, the attributes selected by the fs techniques, were analysed using david bioinformatics resource to identify the genes or proteins in each reduced dataset, from their microarray identifiers  <cit> .
the genes and proteins included in the biohel reduced datasets were assessed by literature searches with pubmed, to determine any known relevance of the attributes to oa and any other disease or treatment classes included in the datasets.
RESULTS
the rgife heuristic increased the classification accuracy achieved for both proteomics and transcriptomics data
by applying the rgife fs heuristic to the canine proteomics dataset, the classification accuracies achieved were considerably higher compared to the accuracy achieved without any feature selection .
this result is robust to choice of protein quantification .
in previous work  <cit>  we showed that biohel performed favourably as compared with other classification techniques, best performing with proteinprophet probabilities , but out-performed by jrip with empai values .
thus our new feature reduction method, with tpr and tnr of 96% and 99% respectively, has considerably outperformed all other methods for classification accuracy on this dataset.
all five transcriptomics datasets were analysed using the seven ml methods and biohel+rgife .
for three of the five transcriptomics datasets, biohel without feature reduction gave a classification that was higher or equal to the other methods.
for the other two datasets, gse <dig> and e-geod- <dig>  the best methods were ibk and svm.
the application of rgife increased the classification of all five datasets, except for e-geod- <dig>  which was already at 100%.
after the feature reduction, only one dataset, e-geod- <dig>  was classified better using an alternative method, svm.
rgife compares favourably with other feature reduction methods
the comparison shown above could be construed as unfair, as it is comparing a classification using the attributes identified by a feature selection method with classifications using the whole dataset.
therefore, it is plausible that application of other feature selection methods could provide improvements comparable to the rgife technique.
to test this, we have analysed both the proteomics and the transcriptomics datasets using all combinations of the fs and classification method methods.
detailed results are given in supplementary tables 1- <dig> .
for the proteomics dataset, rgife is the best feature reduction method, both with empai and proteinprophet values.
using empai values , the highest accuracy was achieved by biohel for both fs and classification .
this is compared to the highest achieved prior to fs, of  <dig> %, using jrip, a rule-based classifier.
using proteinprophet values , the accuracy rose from the highest accuracy of  <dig> % using biohel to  <dig> % after application of fs.
the two fs and classification method combinations that gave this highest classification are: rgife with ibk  and naïve bayes for both fs and classification .
the combination of rgife+biohel compares reasonably favourably with these scores .
the five transcriptomics datasets were also analysed using the different fs and classification method combinations, with mixed results.
for gse <dig> , classification accuracies of 100% were achieved using rgife+biohel, as well as with a combination of svm rfe fs with either svm or ibk.
for gse <dig> , svm rfe fs again gave the highest accuracy of 100%, when used with nb, svm or ibk.
for this dataset, rgife misclassified only one of the  <dig> samples.
only one sample was also misclassified when rgife was applied to e-geod- <dig> , and again svm rfe combined with nb, svm or ibk gave the highest accuracy of 100%.
for dataset e-geod- <dig> , a number of fs and classification methods resulted in classifications of 100% accuracy.
this included the combination of rgife with either biohel or rf.
the fs methods cfs and rf also resulted in classifications of 100% when used with various classification methods.
dataset e-geod- <dig>  was generally less well classified by the various methods, with only two fs and classification method combinations resulting in classifications of 100% accuracy.
these were svm rfe with svm and rf for fs with rf.
thus for these datasets, no single method stands out as being best.
for each combination of classifier and dataset, the best and worse methods were recorded, considering both the tprs and tnrs.
tables  <dig> and  <dig> compare the six fs methods; they show that the method that most frequently resulted in the highest accuracy was svm rfe, however it also frequently resulted in the lowest tpr and tnr as well, showing it to be very unstable.
the comparison showed rgife to be the second best for the highest accuracy, and the second worst for the lowest accuracy, showing it to be the most stable across the datasets tested.
feature reduction identifies inflammation-associated proteins, which have been associated previously with cartilage matrix degradation
rgife identified ten proteins: six using the proteinprophet quantification and five using empai values, with one protein  in common .
six of these proteins found have been previously associated with cartilage inflammation and oa: mmp- <dig>  <cit> , il- <dig>  <cit> , thrombospondin- <dig>  <cit> , hyaluronan and proteoglycan link protein  <dig>  <cit> , clusterin  <cit>  and fetuin-a  <cit> .
the reduced datasets generated from the transcriptomics datasets vary in size and utility
all the transcriptomics datasets were vastly reduced in the number of genes using biohel reduction.
table  <dig> shows the number of genes in each dataset both before and after application of rgife.
the numbers of genes the datasets were reduced to varied: four out of five were between five and twenty-four genes .
however, e-geod- <dig> gave  <dig> genes , a much larger number.
the genes present in all the reduced subsets identified by rgife are listed in supplementary tables  <dig> –  <dig> , except for e-geod- <dig>  due to the large number of genes included.
of the genes included in the gse <dig> reduced dataset, two genes, rsad <dig> and cxcl <dig>  were found to be associated with ra, one of the disease groups included in the datasets  <cit> .
in the gse <dig> reduced dataset, four genes are clearly relevant to the diseases analysed with this dataset.
these genes are fn <dig>  ddr <dig>  mmp- <dig> and notch <dig>  all of which have been associated with either oa or ra  <cit> .
dataset e-geod- <dig> was reduced to only five genes.
included in this small subset of genes was cxcl <dig>  which has previously been suggested as a possible biomarker for ra  <cit> .
none of the genes in the e-geod- <dig> reduced dataset were found to be specifically related to oa or ra, the two disease classes included in the dataset.
however, there were a number of genes whose functions are currently unknown.
discussion
comparison of classification accuracies achieved by feature selection methods
from the results shown, it can be seen that there is not one fs and classification method combination that is best for all datasets.
for the proteomics data, the rgife+biohel combination compared very well with other methods.
for all five of the transcriptomics datasets, at least one combination gave a classification accuracy of 100%.
however, there was a lot of variation both in the accuracies of the classifications for each dataset, using the different methods, and the classifications performed by each method across the different datasets.
rgife gave the equal highest accuracies for two of the five transcriptomics datasets and was close to the highest accuracies for another two datasets.
however, for dataset e-geod- <dig> , other methods were better.
proteins identified from the proteomics dataset by rgife
using the two different values for the proteins, empai and proteinprophet, did result in different proteins being selected, however mmp- <dig> was included in both.
this is because empai is a measure of relative protein quantitation and proteinprophet gives a probability based on how likely it is a protein is present in a sample.
therefore, whilst these measures are related, they are not the same.
included in the proteomics empai-reduced dataset are two bacterial proteins, genome polyprotein  and atp synthase subunit b' .
these proteins have fairly low mascot scores, indicating a lack of confidence in this identification and so, based on this, it may be suitable to increase the mascot score threshold to reduce the likelihood of false positives.
also, polg was found in only three samples and atpx was present in only one sample.
through analysis of the rules generated by biohel it is also clear that these proteins were used to distinguish between the control and carprofen-only treated samples, indicating that carprofen treatment has no detectable direct physiological effect.
supplementary tables  <dig> and  <dig>  show the proteins selected by the other feature selection methods tested.
all the proteins identified by rgife for the ms dataset, using proteinprophet probabilities, were also identified by at least one of the other fs techniques tested.
for the empai values dataset, two of the five proteins, clus & mmp- <dig>  were identified by other fs methods.
these proteins have previously been identified in oa  <cit> .
for the dataset using empai values, other methods did also select bacterial proteins.
biohel reduction method applied to transcriptomics datasets
the genes identified by rgife were also compared to the genes selected by the other fs techniques used.
supplemental tables 14- <dig>  list the genes identified by each method, for comparison with the genes included by rgife.
for datasets e-geod- <dig> and e-geod- <dig> no genes selected by rgife were included in the lists of genes identified by the other fs methods.
there was some overlap between the genes selected by the other methods, however only very few.
dataset gse <dig> had one gene, rsad <dig>  which was included in the reduced dataset of another fs method.
rsad <dig> has been found to be up-regulated in ra  <cit> , which was a disease assessed by the gse <dig> investigation.
five of the genes selected by rgife for this dataset were also identified by other fs methods.
when faced with such high dimensional datasets, the results show that the natural bias of each fs method produces very divergent results.
perhaps this is not surprising since all of them are heuristic.
the transcriptomics reduced-datasets were generally larger than those of the proteomics datasets.
quite likely this is due to proteomics datasets having far fewer features than transcriptomics ones, and the features present are less noisy  <cit> .
CONCLUSIONS
the rgife feature-reduction method has been shown to be suitable for the analysis of both transcriptomics and proteomics data.
the classification accuracies achieved by this method in combination with the biohel rule-based machine learning method were better than other machine learning methods used  for all datasets and better or equal for the majority of datasets with feature selection also applied.
the feature reduction method resulted in the selection of a subset of genes for all datasets, some of which had clear links to the diseases related to the datasets.
a number of genes were identified that may be suitable as possible biomarkers, however they require further individual analysis to determine their relevance and suitability.
given that this is a generic supervised machine-learning technique, this method should also be suitable for analysis of other forms of complex data, including data from other omics areas, such as metabolomics and lipidomics, however testing in this area is still required.
some cytokines and chemokines in the joint, such as il <dig> , have increased activity during oa,  <cit>  which drive the increased production and secretion of matrix degrading enzymes such as matrix metalloproteinases   that mediate the destruction of articular cartilage.
therefore, the feature-reduction methods used in this study identified biologically relevant proteins.
in summary, to our knowledge bioinformatics feature-reduction tools have never been applied to ‘omics’ data in the area of rheumatology.
however, as more and more investigators are applying ‘omics’ techniques to tissues and cells from arthritic joints, feature-reduction methods such as this are likely to make a significant contribution to basic and clinical research in this area, especially the stratification of patients based on data from molecular markers of joint inflammation.
despite the growing burden of arthritic diseases, many pharmaceutical companies have abandoned the development of disease modifying oa drugs  because oa is a heterogeneous disease with a variety of phenotypes and pathophysiological drivers.
identification of novel biomarkers and further validation of existing “experimental” markers are likely to facilitate oa drug development.
bioinformatic studies of currently available data from in vitro cartilage models, animal models and oa patients will consolidate existing knowledge of markers of disease progression and reveal new targets that may be invaluable for dmoad development.
the inevitable growth of public and private datasets derived from large cohort studies of joint inflammation will provide further opportunities for applying feature reduction techniques for biomarker identification and validation.
this should provide a paradigm shift in the diagnosis and treatment of arthritis, facilitating new drug discovery and improving the range of effective therapy options for chronic joint diseases.
disclosures
the authors disclose no competing financial interests.
a. mobasheri is the coordinator of the d-board consortium funded by european commission framework  <dig> program .
j. bacardit and c. hodgman are participants in d-board.
author contributions
all of the authors made substantial contribution to the study, the organisation and the conduct of the study, to carrying out the study /including acquisition of study data and to analysis and interpretation of study data.
conceived and designed the study: jb, am.
performed the experiments and the data analysis: als, jb.
drafted the manuscript: als, jb, am, djs, ch.
read, edited and approved the final manuscript: als, djs, ch, da, mha, am, jb.
contributed datasets: am.
authors information
jaume bacardit and ali mobasheri are member of the following european commission consortia:
the d-board european consortium for biomarker discovery, the universities of surrey, nottingham and newcastle, united kingdom; http://cordis.europa.eu/projects/rcn/105314_en.html
the approach consortium  http://ec.europa.eu/research/health/medical-research/severe-chronicdiseases/projects/d-board_en.html
list of abbreviations
atpx: atp synthase subunit b'; cfs: correlation-based feature selection; clus: clusterin; cxcl9: chemokine  ligand 9; ddr2: discoidin domain receptor family, member 2; empai: exponentially modified protein abundance index; fn1: fibronectin 1; fs: feature selection; ibk: k-nearest neighbour; ife: iterative feature elimination; il-1β: interleukin  <dig> beta; ml: machine learning; mmp-3: matrix metalloproteinase- <dig> ; mmp-9: matrix metalloproteinase-9; ms: mass spectrometry; nb: naïve bayes; notch3: notch homolog 3; nsaid: non-steroidal anti-inflammatory drug; oa: osteoarthritis; polg: genome polyprotein; ra: rheumatoid arthritis; rf: random forest; rgife: rule-guided iterative feature elimination; rsad2: radical s-adenosyl methionine domain containing 2; svm: support vector machines; svm rfe: svm-recursive feature elimination; tnr: true negative rate; tpr: true positive rate.
supplementary material
additional file 1
click here for file
acknowledgments
this study received grant support from the biotechnology and biological sciences research council , mars® and waltham®.
the authors are grateful to dr. julia r. smith  for her collaboration and assistance with the generation of the proteomic datasets.
am and mha gratefully acknowledge funding from king abdulaziz city for science and technology  and king abdulaziz university .
declarations
publication charges for this article have been funded by the center of excellence in genomic medicine research , king abdulaziz university, jeddah,  <dig>  kingdom of saudi arabia.
this article has been published as part of bmc genomics volume  <dig> supplement  <dig>  2015: selected articles from the 2nd international genomic medical conference : genomics.
the full contents of the supplement are available online at http://www.biomedcentral.com/bmcgenomics/supplements/16/s1

 the'omics' are making significant contributions to the study of chronic diseases, especially the identification of novel biomarkers. a biomarker is defined as a characteristic that may be objectively measured and evaluated as an indicator of normal biologic processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention. biomarkers are actively investigated in the areas of clinical rheumatology, orthopaedics and sports medicine. osteoarthritis is a degenerative joint disease that affects the entire joint structure. it is characterised by progressive degeneration of cartilage, menisci, ligaments and subchondral bone. synovial inflammation is a major contributor to disease progression and is responsible for the increased production of catabolic and pro-inflammatory mediators that alter the balance of cartilage matrix degradation and repair, leading to excess production of the proteolytic enzymes responsible for cartilage breakdown. oa is currently diagnosed by radiography, once clinical signs of pain and loss of mobility have already appeared, and therefore biomarkers that could identify early signs of oa would significantly aid in diagnosis. current research is aimed at identifying panels of clinically useful biochemical and imaging markers into single diagnostic algorithms that can be used for diagnostic and prognostic applications and for testing the efficacy of new drugs. applying ‘omics’ results in the generation of large datasets that are suitable for bioinformatic analysis using machine learning, to extract important information. bioinformatics tools play an important role in the analysis of data from omics technologies, such as microarrays, next generation sequencing and mass spectrometry, and as a result a wide range of methods have been developed. such methods include supervised machine learning techniques, which are used to build classification models. models are used to automatically label samples of unknown class by using a training set of known labelled samples. there are many different types of ml methods, some of which can be used to identify putative biomarkers from data by observing the attributes used to build the models. rule-based methods are an example of this, as it is possible to read the rules generated to form the model. biohel is a rule based machine learning method which has been used for sample classification in highly dimensional datasets because of its fine-grained embedded feature selection. it has been successfully applied to accurately classify many different types of biological data. rule-based methods construct rule sets that contain at least one rule for each sample group, based on the values associated with the attributes, for example the expression value of the genes. an example of a rule set is shown in figure ml can also be used to identify possible biomarkers in the form of feature selection, a method of data reduction. fs techniques identify a subset of attributes, for example genes or proteins, which could be used to build a more successful model, compared to using the whole dataset. supervised fs methods analyse data with known class labels, with the aim to remove irrelevant or redundant features. using fs techniques can improve model performance, increase classification accuracy, and provide a clearer understanding of attributes, which are useful for generating an effective classification model. their key challenge is to avoid a loss of information by dropping features that contain crucial information for the data being analysed. the application of fs methods is particularly relevant for transcriptomic datasets that often have very large numbers of attributes, in relation to the number of samples. there are three main forms of feature selection: filter, wrapper and embedded methods. filter methods, such as correlation-based feature selection, are used to estimate the effectiveness of a reduction in attributes. wrapper methods, including genetic algorithms, use classifiers to determine if the subset of attributes gives successful classifications. the third method, embedded, includes feature reduction within the process of classification and uses such machine learning techniques as naïve bayes and support vector machines. the aim of this study was to develop an fs heuristic specialized on identifying very reduced sets of variables built on top of the biohel rule-based ml method, in order to increase the accuracy of classification models and to identify putative biomarkers. this heuristic performs an iterative feature elimination process that is guided by an analysis of the rule sets generated by biohel. this guiding process avoids numerous iterations of blind trial-and-error attempts at removing features from the dataset, and quickly finds, in most datasets, very reduced subsets of features. we call this method rgife. other work in this area has involved iterative processes for feature reduction, however these have been based on different methods. one example of such a process for feature reduction is support vector machines recursive feature elimination, which is a form of backward feature elimination. svm rfe has three main steps. firstly it trains the svm classifier, followed by computing a specific ranking criterion for all features. this criterion is based on the weights of a linear svm. the features are ranked based on this value and the feature with the smallest ranking criterion are removed. these steps are then repeated in an iterative process, resulting in a reduced number of features. the rgife method is tested on several proteomics and transcriptomics oa datasets to demonstrate its suitability for multiple data types. its performance is compared to several combinations of feature selection and classification algorithms. the results show that rgife improves biohel’s performance in all datasets, is able to identify very reduced sets of variables in most of them and shows competitive performance to other methods from the literature. methods proteomics and transcriptomics datasets a proteomics dataset was selected, which had been previously analysed by biohel to identify putative biomarkers. the proteomics dataset analysed here consisted of samples analysed by mass spectrometry to identify the proteins present in each sample. the dataset included six samples treated with the pro-inflammatory cytokine interleukin- beta, to simulate oa in culture, and five samples treated with il-1β followed by carprofen, which is a non-steroidal anti-inflammatory drug used to treat oa. also included in the study were six control samples and six samples treated with only carprofen. mascot was applied, using the uniprot database, to determine proteins present in the samples. from this two datasets were generated. the first included an empai quantitation value assigned to each protein present and alternatively, the second provided each protein with a proteinprophet probability. this probability is a measure of how likely it is each protein is present in a sample. to assess the method developed on transcriptomics data, datasets were identified from arrayexpress and ncbi geo by searching for the term ‘osteoarthritis’. of the datasets that this search term returned, those that had more than five samples per group were analysed. this resulted in five datasets that vary in size, but which all contain many more attributes than the proteomics dataset. the sample numbers also vary between samples over classes to samples over classes. these datasets were classified using biohel and other machine learning methods, as the canine proteomic dataset was, reported in a recent paper by swan et al.,. all datasets were partitioned into training and test sets following the leave-one-out cross-validation methodology. the rule-guided iterative feature elimination heuristic with the goal of identifying very reduced and highly accurate sets of variables we propose a feature selection heuristic built on top of the biohel rule-based machine learning method and based on the iterative feature elimination principle. the basic ife method would pick an attribute, remove it from the dataset and train a classifier with the remaining attributes. if the prediction capacity of the new model is equal or better than with the whole set of attributes, the attribute is eliminated. otherwise it is inserted back into the dataset. the basic ife process becomes extremely computationally demanding in high dimensional datasets, such as the case of omics data. therefore, in order to make such ife heuristic feasible it needs to be improved in several directions. the main change is that, rather than picking attributes to be removed at random, we will pick them based on their relevance, estimated from the rule sets generated by biohel from the whole set. then, the first attribute to be picked for elimination are those at the bottom of biohel’s ranking. the second change is remove attributes in blocks, rather than one by one. at the start of the heuristic the block size is set up to a very large number, 25% of the problem’s attributes. if a block cannot be removed because it would decrease the prediction capacity of the classification model then the next block is tested. the block size is reduced to be 25% of the previous block size whenever all attributes have been tested or five consecutive unsuccessful trials have been attempted. as a final change the acceptance criteria of a trial is relaxed in certain scenarios: whenever five consecutive unsuccessful trials have been performed, before reducing the block size the heuristic checks if one of these trials suffered an accuracy drop corresponding to just one sample. if so, this trial is accepted. the iterative process then finishes when the attributes are being removed one at a time and either the removal of all attributes left in the dataset had been tested or five iterations in a row resulted in a drop in the percentage classification accuracy equivalent to more than one sample. the overall workflow of the heuristic is represented in figure experimental design for the experiments of this paper our aim is to show that the rgife heuristic is both able to find feature subsets that are both small and highly accurate. to this aim we have designed a series of experiments that firstly compares the heuristic with other fs methods, and then compares the accuracy obtained by biohel with the reduced feature subsets with other machine learning methods. the feature selection techniques used were correlation-based, svm rfe, random forest, naïve bayes and chi squared, some of which are feature selection methods based on the machine learning methods used. all methods used were those implemented in the machine learning software, weka. for those methods that ranked their selected attributes, rather than identifying a small selection of attributes, the top features were used. the machine leaning methods included in the comparison are naïve bayes, support vector machines, k-nearest neighbour, jrip, j and random forest, also using weka implementations. for all classifications performed, the true positive rate and true negative rate were calculated. the tpr is the measure of the proportion of correctly classified samples and multiplying it by gives the percentage classification accuracy. tpr is calculated by: in contrast, the tnr is a measure of the proportion of negatives that are correctly classified. tnr is calculated by: the combinations of methods were compared in two ways. firstly, the classification accuracies were compared, to determine which ones gave the highest accuracy. then, the attributes selected by the fs techniques, were analysed using david bioinformatics resource to identify the genes or proteins in each reduced dataset, from their microarray identifiers. the genes and proteins included in the biohel reduced datasets were assessed by literature searches with pubmed, to determine any known relevance of the attributes to oa and any other disease or treatment classes included in the datasets.  the rgife heuristic increased the classification accuracy achieved for both proteomics and transcriptomics data by applying the rgife fs heuristic to the canine proteomics dataset, the classification accuracies achieved were considerably higher compared to the accuracy achieved without any feature selection. this result is robust to choice of protein quantification. in previous work we showed that biohel performed favourably as compared with other classification techniques, best performing with proteinprophet probabilities, but out-performed by jrip with empai values. thus our new feature reduction method, with tpr and tnr of 96% and 99% respectively, has considerably outperformed all other methods for classification accuracy on this dataset. all five transcriptomics datasets were analysed using the seven ml methods and biohel+rgife. for three of the five transcriptomics datasets, biohel without feature reduction gave a classification that was higher or equal to the other methods. for the other two datasets, gse and e-geod- the best methods were ibk and svm. the application of rgife increased the classification of all five datasets, except for e-geod- which was already at 100%. after the feature reduction, only one dataset, e-geod- was classified better using an alternative method, svm. rgife compares favourably with other feature reduction methods the comparison shown above could be construed as unfair, as it is comparing a classification using the attributes identified by a feature selection method with classifications using the whole dataset. therefore, it is plausible that application of other feature selection methods could provide improvements comparable to the rgife technique. to test this, we have analysed both the proteomics and the transcriptomics datasets using all combinations of the fs and classification method methods. detailed results are given in supplementary tables 1-. for the proteomics dataset, rgife is the best feature reduction method, both with empai and proteinprophet values. using empai values, the highest accuracy was achieved by biohel for both fs and classification. this is compared to the highest achieved prior to fs, of %, using jrip, a rule-based classifier. using proteinprophet values, the accuracy rose from the highest accuracy of % using biohel to % after application of fs. the two fs and classification method combinations that gave this highest classification are: rgife with ibk and naïve bayes for both fs and classification. the combination of rgife+biohel compares reasonably favourably with these scores. the five transcriptomics datasets were also analysed using the different fs and classification method combinations, with mixed results. for gse, classification accuracies of 100% were achieved using rgife+biohel, as well as with a combination of svm rfe fs with either svm or ibk. for gse, svm rfe fs again gave the highest accuracy of 100%, when used with nb, svm or ibk. for this dataset, rgife misclassified only one of the samples. only one sample was also misclassified when rgife was applied to e-geod-, and again svm rfe combined with nb, svm or ibk gave the highest accuracy of 100%. for dataset e-geod-, a number of fs and classification methods resulted in classifications of 100% accuracy. this included the combination of rgife with either biohel or rf. the fs methods cfs and rf also resulted in classifications of 100% when used with various classification methods. dataset e-geod- was generally less well classified by the various methods, with only two fs and classification method combinations resulting in classifications of 100% accuracy. these were svm rfe with svm and rf for fs with rf. thus for these datasets, no single method stands out as being best. for each combination of classifier and dataset, the best and worse methods were recorded, considering both the tprs and tnrs. tables and compare the six fs methods; they show that the method that most frequently resulted in the highest accuracy was svm rfe, however it also frequently resulted in the lowest tpr and tnr as well, showing it to be very unstable. the comparison showed rgife to be the second best for the highest accuracy, and the second worst for the lowest accuracy, showing it to be the most stable across the datasets tested. feature reduction identifies inflammation-associated proteins, which have been associated previously with cartilage matrix degradation rgife identified ten proteins: six using the proteinprophet quantification and five using empai values, with one protein in common. six of these proteins found have been previously associated with cartilage inflammation and oa: mmp-, il-, thrombospondin-, hyaluronan and proteoglycan link protein, clusterin and fetuin-a. the reduced datasets generated from the transcriptomics datasets vary in size and utility all the transcriptomics datasets were vastly reduced in the number of genes using biohel reduction. table shows the number of genes in each dataset both before and after application of rgife. the numbers of genes the datasets were reduced to varied: four out of five were between five and twenty-four genes. however, e-geod- gave genes, a much larger number. the genes present in all the reduced subsets identified by rgife are listed in supplementary tables –, except for e-geod- due to the large number of genes included. of the genes included in the gse reduced dataset, two genes, rsad and cxcl were found to be associated with ra, one of the disease groups included in the datasets. in the gse reduced dataset, four genes are clearly relevant to the diseases analysed with this dataset. these genes are fn ddr mmp- and notch all of which have been associated with either oa or ra. dataset e-geod- was reduced to only five genes. included in this small subset of genes was cxcl which has previously been suggested as a possible biomarker for ra. none of the genes in the e-geod- reduced dataset were found to be specifically related to oa or ra, the two disease classes included in the dataset. however, there were a number of genes whose functions are currently unknown. discussion comparison of classification accuracies achieved by feature selection methods from the results shown, it can be seen that there is not one fs and classification method combination that is best for all datasets. for the proteomics data, the rgife+biohel combination compared very well with other methods. for all five of the transcriptomics datasets, at least one combination gave a classification accuracy of 100%. however, there was a lot of variation both in the accuracies of the classifications for each dataset, using the different methods, and the classifications performed by each method across the different datasets. rgife gave the equal highest accuracies for two of the five transcriptomics datasets and was close to the highest accuracies for another two datasets. however, for dataset e-geod-, other methods were better. proteins identified from the proteomics dataset by rgife using the two different values for the proteins, empai and proteinprophet, did result in different proteins being selected, however mmp- was included in both. this is because empai is a measure of relative protein quantitation and proteinprophet gives a probability based on how likely it is a protein is present in a sample. therefore, whilst these measures are related, they are not the same. included in the proteomics empai-reduced dataset are two bacterial proteins, genome polyprotein and atp synthase subunit b'. these proteins have fairly low mascot scores, indicating a lack of confidence in this identification and so, based on this, it may be suitable to increase the mascot score threshold to reduce the likelihood of false positives. also, polg was found in only three samples and atpx was present in only one sample. through analysis of the rules generated by biohel it is also clear that these proteins were used to distinguish between the control and carprofen-only treated samples, indicating that carprofen treatment has no detectable direct physiological effect. supplementary tables and show the proteins selected by the other feature selection methods tested. all the proteins identified by rgife for the ms dataset, using proteinprophet probabilities, were also identified by at least one of the other fs techniques tested. for the empai values dataset, two of the five proteins, clus & mmp- were identified by other fs methods. these proteins have previously been identified in oa. for the dataset using empai values, other methods did also select bacterial proteins. biohel reduction method applied to transcriptomics datasets the genes identified by rgife were also compared to the genes selected by the other fs techniques used. supplemental tables 14- list the genes identified by each method, for comparison with the genes included by rgife. for datasets e-geod- and e-geod- no genes selected by rgife were included in the lists of genes identified by the other fs methods. there was some overlap between the genes selected by the other methods, however only very few. dataset gse had one gene, rsad which was included in the reduced dataset of another fs method. rsad has been found to be up-regulated in ra, which was a disease assessed by the gse investigation. five of the genes selected by rgife for this dataset were also identified by other fs methods. when faced with such high dimensional datasets, the results show that the natural bias of each fs method produces very divergent results. perhaps this is not surprising since all of them are heuristic. the transcriptomics reduced-datasets were generally larger than those of the proteomics datasets. quite likely this is due to proteomics datasets having far fewer features than transcriptomics ones, and the features present are less noisy.  the rgife feature-reduction method has been shown to be suitable for the analysis of both transcriptomics and proteomics data. the classification accuracies achieved by this method in combination with the biohel rule-based machine learning method were better than other machine learning methods used for all datasets and better or equal for the majority of datasets with feature selection also applied. the feature reduction method resulted in the selection of a subset of genes for all datasets, some of which had clear links to the diseases related to the datasets. a number of genes were identified that may be suitable as possible biomarkers, however they require further individual analysis to determine their relevance and suitability. given that this is a generic supervised machine-learning technique, this method should also be suitable for analysis of other forms of complex data, including data from other omics areas, such as metabolomics and lipidomics, however testing in this area is still required. some cytokines and chemokines in the joint, such as il, have increased activity during oa, which drive the increased production and secretion of matrix degrading enzymes such as matrix metalloproteinases that mediate the destruction of articular cartilage. therefore, the feature-reduction methods used in this study identified biologically relevant proteins. in summary, to our knowledge bioinformatics feature-reduction tools have never been applied to ‘omics’ data in the area of rheumatology. however, as more and more investigators are applying ‘omics’ techniques to tissues and cells from arthritic joints, feature-reduction methods such as this are likely to make a significant contribution to basic and clinical research in this area, especially the stratification of patients based on data from molecular markers of joint inflammation. despite the growing burden of arthritic diseases, many pharmaceutical companies have abandoned the development of disease modifying oa drugs because oa is a heterogeneous disease with a variety of phenotypes and pathophysiological drivers. identification of novel biomarkers and further validation of existing “experimental” markers are likely to facilitate oa drug development. bioinformatic studies of currently available data from in vitro cartilage models, animal models and oa patients will consolidate existing knowledge of markers of disease progression and reveal new targets that may be invaluable for dmoad development. the inevitable growth of public and private datasets derived from large cohort studies of joint inflammation will provide further opportunities for applying feature reduction techniques for biomarker identification and validation. this should provide a paradigm shift in the diagnosis and treatment of arthritis, facilitating new drug discovery and improving the range of effective therapy options for chronic joint diseases. disclosures the authors disclose no competing financial interests. a. mobasheri is the coordinator of the d-board consortium funded by european commission framework program. j. bacardit and c. hodgman are participants in d-board. author contributions all of the authors made substantial contribution to the study, the organisation and the conduct of the study, to carrying out the study /including acquisition of study data and to analysis and interpretation of study data. conceived and designed the study: jb, am. performed the experiments and the data analysis: als, jb. drafted the manuscript: als, jb, am, djs, ch. read, edited and approved the final manuscript: als, djs, ch, da, mha, am, jb. contributed datasets: am. authors information jaume bacardit and ali mobasheri are member of the following european commission consortia: the d-board european consortium for biomarker discovery, the universities of surrey, nottingham and newcastle, united kingdom; http://cordis.europa.eu/projects/rcn/105314_en.html the approach consortium http://ec.europa.eu/research/health/medical-research/severe-chronicdiseases/projects/d-board_en.html list of abbreviations atpx: atp synthase subunit b'; cfs: correlation-based feature selection; clus: clusterin; cxcl9: chemokine ligand 9; ddr2: discoidin domain receptor family, member 2; empai: exponentially modified protein abundance index; fn1: fibronectin 1; fs: feature selection; ibk: k-nearest neighbour; ife: iterative feature elimination; il-1β: interleukin beta; ml: machine learning; mmp-3: matrix metalloproteinase-; mmp-9: matrix metalloproteinase-9; ms: mass spectrometry; nb: naïve bayes; notch3: notch homolog 3; nsaid: non-steroidal anti-inflammatory drug; oa: osteoarthritis; polg: genome polyprotein; ra: rheumatoid arthritis; rf: random forest; rgife: rule-guided iterative feature elimination; rsad2: radical s-adenosyl methionine domain containing 2; svm: support vector machines; svm rfe: svm-recursive feature elimination; tnr: true negative rate; tpr: true positive rate. supplementary material additional file 1 click here for file acknowledgments this study received grant support from the biotechnology and biological sciences research council, mars® and waltham®. the authors are grateful to dr. julia r. smith for her collaboration and assistance with the generation of the proteomic datasets. am and mha gratefully acknowledge funding from king abdulaziz city for science and technology and king abdulaziz university. declarations publication charges for this article have been funded by the center of excellence in genomic medicine research, king abdulaziz university, jeddah, kingdom of saudi arabia. this article has been published as part of bmc genomics volume supplement 2015: selected articles from the 2nd international genomic medical conference: genomics. the full contents of the supplement are available online at http://www.biomedcentral.com/bmcgenomics/supplements/16/s1
we use this heuristic to identify putative biomarkers of osteoarthritis, articular cartilage degradation and synovial inflammation, using both proteomic and transcriptomic datasets. 24- november 2nd international genomic medicine conference jeddah, kingdom of saudi arabia a key component of typical machine learning pipelines for omics data is feature selection, which is used to reduce the raw high-dimensional data into a tractable number of features. the results have shown the rgife feature reduction method to be suitable for analysing both proteomic and transcriptomics data. feature reduction methods are advantageous for the analysis of omics data in the field of rheumatology, as the applications of such techniques are likely to result in improvements in diagnosis, treatment and drug discovery. in this paper we propose a heuristic for the selection of very small feature subsets, via an iterative feature elimination process that is guided by rule-based machine learning, called rgife. investigations into novel biomarkers using omics techniques generate large amounts of data. our rgife heuristic increased the classification accuracies achieved for all datasets when no feature selection is used, and performed well in a comparison with other feature selection methods. this balance is crucial when the goal of data analysis is the identification of highly accurate but small panels of biomarkers with potential clinical utility. using this method the datasets were reduced to a smaller number of genes or proteins, including those known to be relevant to oa, cartilage degradation and joint inflammation. due to their size and numbers of attributes, these data are suitable for analysis with machine learning methods.
 investigations into novel biomarkers using omics techniques generate large amounts of data. due to their size and numbers of attributes, these data are suitable for analysis with machine learning methods. a key component of typical machine learning pipelines for omics data is feature selection, which is used to reduce the raw high-dimensional data into a tractable number of features. feature selection needs to balance the objective of using as few features as possible, while maintaining high predictive power. this balance is crucial when the goal of data analysis is the identification of highly accurate but small panels of biomarkers with potential clinical utility. in this paper we propose a heuristic for the selection of very small feature subsets, via an iterative feature elimination process that is guided by rule-based machine learning, called rgife. we use this heuristic to identify putative biomarkers of osteoarthritis, articular cartilage degradation and synovial inflammation, using both proteomic and transcriptomic datasets. results and discussion our rgife heuristic increased the classification accuracies achieved for all datasets when no feature selection is used, and performed well in a comparison with other feature selection methods. using this method the datasets were reduced to a smaller number of genes or proteins, including those known to be relevant to oa, cartilage degradation and joint inflammation. the results have shown the rgife feature reduction method to be suitable for analysing both proteomic and transcriptomics data. methods that generate large ‘omics’ datasets are increasingly being used in the area of rheumatology. conclusions feature reduction methods are advantageous for the analysis of omics data in the field of rheumatology, as the applications of such techniques are likely to result in improvements in diagnosis, treatment and drug discovery. 24- november 2nd international genomic medicine conference jeddah, kingdom of saudi arabia  our rgife heuristic increased the classification accuracies achieved for all datasets when no feature selection is used, and performed well in a comparison with other feature selection methods. using this method the datasets were reduced to a smaller number of genes or proteins, including those known to be relevant to oa, cartilage degradation and joint inflammation. the results have shown the rgife feature reduction method to be suitable for analysing both proteomic and transcriptomics data. methods that generate large ‘omics’ datasets are increasingly being used in the area of rheumatology.  feature reduction methods are advantageous for the analysis of omics data in the field of rheumatology, as the applications of such techniques are likely to result in improvements in diagnosis, treatment and drug discovery. 24- november 2nd international genomic medicine conference jeddah, kingdom of saudi arabia 