BACKGROUND
the statistical definition of epistasis was given by fisher in  <dig> as deviations from additive effects in a linear statistical model  <cit> .
the idea of epistasis, or more generally "gene-gene interaction", has reappeared as a popular theme in human genetics over the last ten years.
there is a growing belief that susceptibility to common diseases may be governed by the potentially complex interaction of multiple genetic variants.
this belief is driven largely by the notion that large biochemical networks and gene regulatory processes involving multiple genes have a functional endpoint that may be influenced by the simultaneous presence of multiple variants in those genes  <cit> .
in addition to its theoretical importance, epistasis has been functionally demonstrated to play a role in human disease.
most notably, hirschsprung's disease was found to be influenced by polymorphisms in ret and the erdb <dig> receptor in the old order amish and was confirmed in a mouse model  <cit> .
having both variants simultaneously increases risk of disease far beyond the combined risk of each independent variant.
as epistasis is believed to have important implications for human disease risk, numerous statistical and computational approaches have been developed to examine epistasis in family-based and case-control association studies  <cit> .
multifactor dimensionality reduction  is one such computational method to identify gene-gene interactions in case-control studies where variants may or may not exhibit detectable marginal effects.
mdr has been shown previously to have reasonable power to detect gene-gene interactions in several cases of experimental error and over a variety of simulated genetic models  <cit> .
mdr has also been applied to many disease phenotypes including hypertension  <cit> , multiple sclerosis  <cit> , sporadic breast cancer  <cit> , type ii diabetes  <cit> , coronary artery disease  <cit> , and autism  <cit> .
there have been several notable extensions to the mdr method.
multifactor dimensionality reduction pedigree disequilibrium test  was developed by martin et al.
to examine multi-locus models in extended pedigree data  <cit> .
mdr was extended to use a chi-square statistic as an alternative to prediction error/classification error, and to test other forms of cross-validation and permutation testing  <cit> .
mdr was also modified to include the odds ratio as a quantitative measure of disease risk  <cit>  – as well as extended into a generalized mdr  to include discrete and quantitative covariates as well as dichotomous and continuous phenotypes  <cit> .
velez et al.
evaluated the performance of mdr using balanced accuracy for several examples of class-imbalance  <cit> .
as the balanced accuracy measure provided improved power for cases of class imbalance, we hypothesized that alternate measures of classification performance would improve the performance of mdr.
in this study, we demonstrate through simulated data that altering the scoring measure used in model evaluation and selection can improve the detection and statistical power for complex interaction effects.
for the purposes of this study, simulated epistasis models contain no marginal effect, where all phenotypic variance attributable to genetic variation is due completely to the interaction of genetic loci, as described by culverhouse et al.
<cit>  to detect these genetic effects, the influential loci must be examined jointly.
mdr is a non-parametric statistical method for the analysis of gene-gene and gene-environment interactions <cit> .
figure  <dig> illustrates the mdr algorithm.
additional detail is provided in the "methods" section.
mdr performs an exhaustive search of all n-locus models, collapsing multi-locus genotypes into high-risk and low-risk classes.
these high- or low-risk classes are then compared to the observed status of individuals to produce a score for the classification.
in this manner, all n-locus models are ranked by a scoring measure, and the model with the optimal score is selected as the best or "most fit."
fitness measures
the results of a classification algorithm can be represented as a special type of two-way contingency table .
the true status forms one dimension of the table and the algorithm classification forms the other dimension.
two-way tables can be scored with a variety of measures, many of which have been developed in multiple disciplines using different terminology to describe similar concepts.
there are several basic terms and measures using the four cells of a two-way contingency table .
sensitivity is the classification accuracy of the cases, or the proportion of correctly classified cases among all cases in the data.
in the text classification field, this measure is called recall.
specificity is the classification accuracy of the controls, or the proportion of correctly classified controls among all controls in the data.
positive predictive value or precision is the classification accuracy of the affirmative classification, or the proportion of actual cases among all individuals classified as cases.
negative predictive value is the proportion of controls among all individuals classified as controls.
using these basic values, several composite measures of association have been developed .
the ten measures selected represent a variety of analysis strategies from several fields including text classification, machine learning, diagnostic testing, statistical theory, and information theory.
classification error  and the related quantity, classification accuracy , are two of the most frequently used and contentious measures of classification performance.
it is defined simply as the proportion of examples incorrectly labelled by a classifier.
the technical merits of classification error as a measure of classifier performance have been debated  <cit> .
precision-based and roc-based measures
f-measure   is a measure first used by lewis and gale for assessing text classification effectiveness, and is the inverse of the e-measure  <cit> .
the e-measure is a weighted combination of sensitivity and positive predictive value derived by van rijsbergen to satisfy several conditions of measurement theory  <cit> .
geometric means have been used as performance measures for classification.
kubat et al.
defines two such quantities, here labelled geometric mean  <dig> and geometric mean  <dig>  <cit> .
geometric mean  <dig>   is the geometric mean of sensitivity and precision.
geometric mean  <dig>   is the geometric mean of sensitivity and specificity.
this measure is related to the receiver-operator characteristic  curve and was used in lieu of roc analysis by kubat et al.
as a single measure of classification  <cit> .
euclidean distance from an ideal classification   is a measure also related to roc curves.
this combination of sensitivity and specificity measures the distance from an ideal classification in roc space, where sensitivity and specificity both equal one.
diagnostic testing measures
the youden index   and predictive summary index   are summary measures of certainty for dichotomous diagnostic tests  <cit> .
the youden index is the sum of the sensitivity and specificity minus one.
the predictive summary index is the sum of the positive predictive value and the negative predictive value minus one.
statistical measures
pearson's chi-square goodness-of-fit statistic   is an adjusted sum of the squared differences between observed and expected frequencies  <cit> .
the chi-square is a classic test of association in categorical data analysis.
the likelihood-ratio test   is a related measure that statistically compares the maximum likelihood of an unrestricted model with a restricted model  <cit> .
in this setting, the unrestricted model consists of the observed frequencies in the data and the restricted model consists of the expected frequencies under the null hypothesis of no association.
information theoretic measures
normalized mutual information   was described by wickens as a measure of information transmission, based on shannon's entropy  <cit> .
entropy was developed in communication theory as a measure of dispersion for categorical data.
entropy is often measured in bits, or log base  <dig> units.
given a two-way contingency table, four entropy values can be computed: the row entropy, the column entropy, and two conditional entropies ) not shown):
h=−∑ipilog⁡2pi
h=−∑jpjlog⁡2pj
h=∑ipi
the quantities pi and pj represent the empirical probabilities of the predicted and true class states, respectively, and pij is their joint probability.
using these values, nmi and its transpose   are defined as:
nmi=h−hh
nmit=nmi=h−hh
the nmi value is interpreted as the proportion of information contained in the row variable that is transferred or transmitted to the column variable, or more concisely the amount by which the model reduces our uncertainty about the true state.
RESULTS
detection
all detection results are shown in figure  <dig> and specific detection results are in figure  <dig>  "detection" is the ability of the method to correctly identify all disease loci, but additional non-disease loci may be included in the model also.
"specific detection" is the ability of the method to correctly identify all the disease loci and no additional loci – this could also be phrased as the ability to detect the correct multi-locus model with no false positive loci.
significant differences from classification error  had wilcoxon rank-sum p-values below  <dig> .
over all models, the mean detection using classification error was 62% and the mean for specific detection was  <dig> %.
detection was at or above 80% in all two-locus models, with 100% detection for all two-locus models with ≥  <dig> % heritability.
detection diminishes in the three-locus models, with only three models showing greater than 80%.
detection in four- and five-locus models drops below 80% except for model  <dig>  a five-locus model with 3% heritability, which was detected at 96%.
specific detection using classification error is above 80% for all two-locus models except the  <dig> % heritability models .
all three-, four- and five-locus models are below 80% specific detection except for model  <dig>  with specific detection at 95%.
these models show sporadic detection and specific detection that does not follow trends based on allele frequency or heritability.
the precision-based measures f-measure and geometric mean  <dig> performed poorly.
over all models, f-measure averaged  <dig> % detection, with  <dig> to 23% detection in two-locus models,  <dig> to 15% detection in three-locus models, and less than 5% in four- and five-locus models.
specific detection was near 0% for all models.
allele frequency had no impact on detection.
geometric mean  <dig> performed slightly better than f-measure with 12% overall detection.
detection using this measure was sensitive to the allele frequencies specified in the genetic model.
detection in the  <dig>  maf models ranged between 24–34% in two-locus models, 14–24% in three-locus models, and 4–10% in four- and five-locus models.
detection in the  <dig>  maf models ranged between 15–19% in two-locus models, 10–13% in three-locus models, and 2–5% in four- and five-locus models.
specific detection using geometric mean  <dig> was near 0% for all models.
both f-measure and geometric mean  <dig> showed significantly worse detection and specific detection than classification error in nearly every case.
notably, these two measures outperformed all others for detection of two very difficult models , but in these cases the detection was very low and specific detection was no better than classification error.
the roc-based measures euclidean distance and geometric mean  <dig> also fail to outperform classification error overall, but perform well for some models.
the average detection using euclidean distance over all models was only  <dig> %, but in two-locus models was near 100% except for the lowest heritability models .
across higher order models detection was generally scattered with no discernable trend, ranging from 2–77% in three-locus models, 3–46% in four-locus models, and 0–41% in five-locus models.
similarly, specific detection using euclidean distance was between 90–99% for all two-locus models except the lowest heritability models .
specific detection in higher order models was also scattered, but in general was higher for higher heritability models.
geometric mean  <dig> showed very similar trends for detection and specific detection to euclidean distance.
detection was 85–100% for two-locus models, 13–70% for three-locus models, 6–20% in four-locus models and less than 7% in five-locus models.
specific detection was between  <dig> and 99% in all but the lowest heritability two-locus models.
all others had less than 10% specific detection.
using euclidean distance and geometric mean  <dig>  only a few models show better detection than using classification error.
the most notable of these is model  <dig> where geometric mean  <dig> improved over classification error by  <dig> .
also, both euclidean distance and geometric mean two showed significantly improved specific detection over classification error in all two-locus models except model  <dig>
the diagnostic testing measures youden index and predictive summary index  perform relatively well.
overall, average detection was 64% using the youden index, and specific detection was 57%.
detection of two-locus models was 84–100%,  <dig> –97% in three-locus models, 5–94% in four-locus models, and 0–95% in five-locus models.
specific detection was 90–92% in all two-locus models except the lowest heritability models which were 65–72%.
other models have higher specific detection in higher heritability models that decreases along with heritability.
psi also shows excellent overall average detection at  <dig> % and specific detection at 59%.
detection was 84–100% in two-locus models,  <dig> –97% in three-locus models,  <dig> –94% in four-locus models, and 0–98% in five-locus models.
the detection using psi is lower in two-locus models than the youden index, but psi shows slightly higher detection in a few three-, four- and five-locus models.
compared to classification error, psi shows significantly increased detection in  <dig> of the  <dig> models.
<dig> models show no significant difference in detection, and for  <dig> models, classification error performs significantly better.
most of the improvement in detection is seen in higher order models, and notably, classification error detects as well or better then psi for two-locus models.
the youden index shows better detection for  <dig> models.
for  <dig> models, classification error is significantly better, and there is no significant difference for  <dig> models.
specific detection with psi is improved over classification error in three-, four- and some five-locus models.
specific detection in two-locus models however is not consistently improved.
similar to psi, the youden index does not improve over classification error in two-locus models.
improvement in detection for the youden index is in three- and four-locus models.
detection using the youden index is not improved over classification error in five-locus models.
specific detection with the youden index is significantly improved over classification error in all but five-locus models.
the chi-square and likelihood ratio statistical measures performed well.
using the chi-square, across all models average detection was  <dig> % and average specific detection was  <dig> %, with 78–100% detection for two-locus models, 11–82% detection for three-locus models, and 0–97% detection in four- and five-locus models with little discernable trend.
specific detection patterns also show no trend.
overall detection using the likelihood ratio is  <dig> % and specific detection is  <dig> %.
in two-locus models, detection ranged from 84–100%, 16–96% in three-locus models, and 0–97% in four- and five-locus models.
using chi-square, detection of  <dig> models was significantly better than classification error,  <dig> models showed no significant difference in detection from classification error, and  <dig> models were detected significantly worse than using classification error.
specific detection was significantly better for  <dig> models using chi-square, was not significantly different from classification error for  <dig> models, and was significantly worse than error for  <dig> models.
most of the detection improvement from using the chi-square measure was seen in three- and four-locus models.
detection using likelihood ratio was significantly better or no different than error for all but  <dig> models, and specific detection was significantly improved for all models except five-locus models, where specific detection of  <dig> models was significantly worse using the likelihood ratio and was not statistically different for  <dig> of the models.
the information theoretic measures nmi and nmi-transpose  both perform relatively well.
overall, detection using nmi was  <dig> % across all models, and specific detection was  <dig> %.
with nmi, detection of two-locus models ranged from 84–100%, three-locus models from 16–96% and four- and five-locus models ranged from 0–98% detection.
for specific detection the trends were very similar to those of the likelihood ratio, but specific detection was slightly higher in most cases.
using nmit, overall detection was highest of all measures at  <dig> %, with a specific detection of  <dig> %.
per model detection rates were lower than with other measures, however, with two-locus model detection ranging from 45–98%, three-locus models from 23–98%, and four- and five-locus models ranging from 0–98%.
trends in specific detection were erratic, performing better than nmi in some three- and four-locus models, but much worse power for two-locus models.
nmit did demonstrate higher detection of higher order models.
detection using nmi is significantly better than classification error in  <dig> models, shows no difference in  <dig> models, and performs worse than classification error in  <dig> models.
specific detection is either significantly improved with nmi or shows no significant difference from classification error in all but  <dig> five-locus models.
nmit performs significantly worse than classification error for two-locus models, but shows significantly better detection for three- and four-locus models, with specific detection following the same trend.
in summary, euclidean distance, f-measure, geometric mean  <dig>  and geometric mean  <dig> perform significantly worse than classification error in both detection and specific detection for a majority of simulated models.
two measures, the chi-square and nmit show improvement in detection and specific detection of some models, but are significantly worse than classification error for other models, with especially poor specific detection of two-locus models.
the remaining four measures show either improved or equal detection and/or specific detection across a majority of models and work well for two-locus models.
these measures, youden index, predictive summary index, likelihood ratio, and normalized mutual information were evaluated for statistical power using permutation testing.
power
all power results are shown in figure  <dig> and specific power is shown in figure  <dig>  "power" is detection that is statistically significant , and "specific power" is specific detection of the correct multi-locus model that is statistically significant .
the process used to assign statistical significance to a result is dependent on the assumption that for these simulated data, the permutation distribution of a single dataset is equivalent to the permutation distribution of other datasets simulated using the same genetic model.
after evaluating the variability of α =  <dig>  critical values, this assumption holds well
predictive summary index  and the youden index have power and specific power only in 3% and 2% heritability models – all other models show 0% power and specific power.
psi outperforms the youden index for two-locus models for both power and specific power, but in higher order models, the youden index performs slightly better.
neither of these measures shows improvement over classification error in the context of permutation testing.
compared to classification error, the likelihood ratio shows significantly better power in  <dig> models, significantly worse power in  <dig> models, and no significant difference in  <dig> models.
of the  <dig> models where error was better,  <dig> were five-locus models, one was a  <dig> % heritability three-locus model , and one was a  <dig> % heritability two-locus model .
specific power of the likelihood ratio is significantly better than error for  <dig> models, significantly worse for  <dig> models, and not significantly different for  <dig> models.
these five models are the 2- <dig> % heritability five-locus models with  <dig>  minor allele frequencies.
using nmi, power is significantly better than error in  <dig> models, significantly worse for  <dig> models, and not significantly different in  <dig> models.
similar to the likelihood ratio,  <dig> of the  <dig> models are five-locus models, one was the same  <dig> % heritability three-locus model , and one was a  <dig> % heritability two-locus model .
the same models showing significant differences in specific power from classification error for the likelihood ratio are also significantly different using nmi.
specific power is significantly better for  <dig> models, significantly worse for  <dig> models, and not significantly different for  <dig> models.
comparing likelihood ratio and nmi, both show equal improvement over classification error for specific power.
one model  showed significantly worse power compared to error when using the likelihood ratio, but power was not significantly different when using nmi.
power using likelihood ratio was significantly better than power using nmi for  <dig> three-locus models , and  <dig> four-locus models .
power using nmi was significantly better than power using likelihood ratio for  <dig> five-locus models .
specific power using likelihood ratio was significantly better for  <dig> models –  <dig> three-locus models , and  <dig> four-locus models .
nmi has significantly better specific power for  <dig> five-locus models .
discussion
this study is an exploration of fitness measures in the mdr algorithm.
here, we have evaluated ten alternative measures of fitness for mdr models and compared them to the traditional measure, classification error .
this work is highly motivated by the dispute over the use of classification error, as it is known to be an improper scoring rule.
in addition, mdr has been released in a java software package with a user-friendly graphical user interface version where many of these measures are currently available .
we felt it was important to know which measures are robust for higher order interaction models.
the first series of simulations evaluated the traditional fitness measure for mdr, classification error, for a set of disease models.
from these simulations, we observe some obvious trends in the two locus models as well as some irregular patterns in the higher order models.
with the two-locus models, the detection and power of mdr using classification error decreases as the broad sense heritability of the model decreases.
trends for higher order models are non-linear with respect to broad sense heritability.
most notably, five-locus models with  <dig>  minor allele frequencies had a very erratic pattern, with the  <dig> % and  <dig> % heritability models  having near 0% power while the 1% heritability model has 22% power .
these erratic patterns are likely due to one of two possible scenarios.
one possibility is related to the way in which the  <dig> genetic models were created.
the genetic algorithm-based procedure for creating multi-locus models with no marginal effects is a directed search through the space of all possible penetrance models.
there may be millions of models which satisfy the heritability requirements, and we arrive at a random sample from that model space with each run of the procedure.
thus, there is some random variability between penetrance functions with similar heritabilites selected in this manner.
this variability likely impacts the ability of mdr to detect the effect of that model, as mdr does not measure an effect based on the penetrance table directly, but rather on the derived two-way contingency table.
a second possible scenario is that these penetrance models were generated with the assumption that there are no main effects of any single locus.
however, in a higher-order models, the simulator does not test for all possible lower-order models.
for example, if a three-locus model is being generated, there are no single locus effects for any of the three-loci; however, the software does not test for the presence of two-locus models embedded within the three loci.
thus, it is possible that the lower detection and power results are due to multiple models competing for detection.
this was the case for model  <dig>  for example, where models containing two of the three loci were detected in 70% of evaluations .
for this study, we consider the performance of classification error as a baseline, so while these effects are interesting to note, they are irrelevant for evaluating other classification measures.
mdr using classification error has greater than 80% detection for  <dig> of the  <dig> multi-locus models, and greater than 80% power for  <dig> of the  <dig> models.
while high-order  models with no marginal effects have been speculated to exist, there is little confirmed experimental or statistical evidence to support their role in complex disease.
it is encouraging to see however that mdr has appreciable power for many high-order models in a feasible sample size of  <dig> cases and  <dig> controls.
if susceptibility to common disease truly does involve complex interactions among many variants, tools for detecting these interactions will be critical.
performance of alternate measures
the f-measure and geometric mean  <dig> both show similar detection and specific detection results across all models.
these two measures combine precision and sensitivity in their calculation.
in general these measures perform very poorly compared to classification error, indicating that information about the specificity of an mdr model improves the performance of mdr.
these measures are focused on classifying cases correctly.
an mdr model that classifies cases best is not necessarily based on the most associated genetic factors.
the model would not discern the difference between genetic factors strongly associated to both cases and controls and only those associated to cases.
geometric mean  <dig> and euclidean distance are similar in that they are geometric functions of sensitivity and specificity, functioning similarly to a receiver-operator characteristic analysis.
these measures perform commensurately with classification error for two-locus models, but do not perform especially well for higher order interactions.
euclidean distance in general performs better than geometric mean  <dig> for both detection and specific detection.
the one notable exception to this trend is geometric mean  <dig>  which shows very good specific detection in two-locus models with  <dig>  maf, out-performing all other measures .
the diagnostic measures youden index and psi perform well in detection and specific detection.
the youden index seems to show better detection than psi in two-locus models and high heritability three-locus models.
psi however shows better specific detection over all models.
both measures out-perform classification error in both detection and specific detection, but those measures do not show improved power or specific power over classification error.
one explanation could be that the empirical distribution of these statistics was not as stable , so the standardized distribution for each model failed to properly assign statistical significance.
while the empirical distribution of randomly selected datasets did not differ significantly from the standardized distribution, some subtle variability in the tail regions was noted.
another possibility is that these measures are more susceptible to noise in the data.
the power and specific power are stronger for 3% heritability models than for models with lower heritability.
while these measures still have utility for detection, their usefulness when assigning statistical significance is questionable.
the chi-square shows good detection and specific detection in the four-locus models and a few five-locus models, but fails to out-perform ce in most other cases.
one reason why the chi-square does not perform well in this setting is that theoretically, the chi-square is not a satisfactory measure of association, and may not rank mdr results to produce optimal detection  <cit> .
the chi-square tests deviation from independence, but does not necessarily quantify the strength of an association.
this is an important consideration for using the chi-square test in other studies, particularly whole-genome association studies, as the chi-square may not necessarily rank signals by strength of association
the measures that demonstrate the most consistent improved performance are the likelihood ratio and nmi.
while the improvement is not dramatic, these measures show equal or better detection and power across nearly all models.
the more dramatic improvement is in specific detection and specific power, where the genetic model detected by mdr is the exact model that was simulated.
this is an interesting result because this means that mdr using nmi or lr is less susceptible to over-fitting .
both the likelihood ratio and nmi measures are based on entropy, which is loosely analogous to variance in the two-way table, and both measures show very similar trends for detection and power.
these two measures are ultimately related, as the numerator of nmi is a transformation of the lr  <cit> .
h−h=lr2p••
the likelihood ratio test is a well-established statistical test to examine a sample's deviation from a null hypothesis.
the statistic itself however does not have an intuitive interpretation, and often is transformed to achieve a valid measure of association  <cit> .
nmi of the contingency table treats the "true outcome" and the "model prediction" as a pair of two-state random variables.
nmi quantifies the amount of uncertainty  about the state of the "truth variable" removed by the "model prediction variable".
nmi has a nice interpretability as the amount by which the model reduces our uncertainty about the true state.
while nmit performs well for three-locus models in particular and did outperform classification error in many cases, its poor detection and specific detection of two-locus models makes it an unattractive measure.
in addition, nmit's interpretation makes less sense as the amount the true variable reduces our uncertainty about the model.
nmi includes details of the contingency table not accounted for by the other measures of model predictability.
for example, the numerator of nmi takes into account the power to correctly predict both the cases and controls: a/ and d, respectively.
explicitly, we can rewrite part of the nmi numerator terms as
aln⁡+dln⁡
which is closely related to predictive summary index .
in addition, we can rewrite part of the nmi denominator as
aln⁡+dln⁡
which bears strong resemblance to the youden index in its attempt to balance the model sensitivity and specificity: a/ and d/, respectively.
the detailed form of the nmi measure likely leads to its observed ability to distinguish between closely similar high quality models; and hence, nmi's improved ability to uniquely determine the relevant variables .
also, nmi preferentially selects models that classify either cases or controls perfectly .
these models are more "stable", or less variable, and are thus to be preferred by the measure over models where both cases and controls are misclassified equally.
the dependence of nmi on the contingency table is quite intricate and warrants further investigation to understand its strengths and limitations more fully.
while the power improvement using nmi or the likelihood ratio in most cases is not dramatic, these measures are superior to classification error.
using these measures, there is higher detection and power of high-order interactions and better specific power overall, so an analyst can be more confident that an mdr model does not contain spurious variables.
also, classification error assumes the distribution of the two classes to be equal.
this shortcoming was recently addressed by velez et al.
<cit>  who used an average of sensitivity and specificity to compute a balanced classification error  for mdr model evaluation, and demonstrated its power to detect gene-gene interactions in cases of class imbalance.
both nmi and lr also take into account the sensitivity and specificity of an mdr model, and likewise should not be susceptible to class imbalance.
one clear advantage of classification error is its interpretability.
of the two improved measures, nmi perhaps has the easiest interpretation.
its value ranges from  <dig> to  <dig>  with  <dig> meaning the genotype and status are independent and  <dig> meaning the genotype completely determines the status.
also, as nmi provides a direct information theoretical measure of association, it may be preferred over the likelihood ratio test statistic, which measures deviation from the null hypothesis of independence rather than directly quantifying the degree of association.
for these reasons, we recommend that nmi be used in lieu of classification error for mdr analyses.
for clarity of interpretation, we recommend showing the two-way contingency table along with reporting the nmi of an mdr result.
classification error  is a widely used measure of performance in many areas of research.
the results of this study are specific to classification using the mdr procedure, but this work does provide additional empirical evidence to support general theoretical arguments against the use of classification error  <cit> .
limitations
there are some limitations of this work.
first, this study was conducted using simulated data, so confidence in the results relies on the quality of the simulated data.
all disease loci in a genetic model were simulated with the same minor allele frequency, either  <dig>  or  <dig> , which is a simplification.
the biological relevance of the penetrance functions simulated could be questioned, though we use functions with very small marginal effects as a "worst-case scenario" and expect the results of this study to generalize to situations where marginal effects are detectable.
it is troubling that the trends observed in power and detection do not always follow the expected trends given the broad-sense heritabilities simulated.
however, as this study is focused on the relative performance of classification measures, this point is not critical.
permutation testing for all  <dig>  models evaluated in this study was not computationally feasible, and a single permutation distribution for each of the  <dig> models was generated for the  <dig> measures evaluated for statistical power.
these permutation distributions did not appear to vary significantly by qq-plot from a randomly seeded dataset using the same model.
assigning significance using a permutation distribution uses the tails of the distribution, however, and some variability in the tails was observed in the qq-plots.
but the power results do closely follow detection results, and it seems unlikely that all  <dig> of the permutation distributions used would consistently over estimate or under estimate the tail values.
the simulated data included only ten loci total.
while small-scale studies are still performed, low-cost genotyping solutions have dramatically increased the number of polymorphisms examined in a typical study.
while the computation time required to perform this study prohibited using a large number of snps for this evaluation, we expect that the relative performance of these fitness measures would extend to datasets with larger numbers of snps.
absolute detection and power, however, are influenced by a variety of factors including the number of snps in the dataset.
CONCLUSIONS
over a variety of simulated genetic models, normalized mutual information  and the likelihood ratio demonstrate stellar performance as measures of multifactor dimensionality reduction model fitness; an improvement in comparison to classification error.
the ability of mdr to specifically detect only the simulated disease loci was significantly higher using these two measures for nearly all genetic models.
these measures also show improved statistical power by permutation testing.
of these measures, nmi is perhaps easier to interpret, as it is the amount by which the model reduces our uncertainty about the true disease state.
nmi properly treats imbalanced data and provides superior performance over classification error.
therefore, we recommend using nmi as an alternative to classification error for mdr analyses.

 the statistical definition of epistasis was given by fisher in as deviations from additive effects in a linear statistical model. the idea of epistasis, or more generally "gene-gene interaction", has reappeared as a popular theme in human genetics over the last ten years. there is a growing belief that susceptibility to common diseases may be governed by the potentially complex interaction of multiple genetic variants. this belief is driven largely by the notion that large biochemical networks and gene regulatory processes involving multiple genes have a functional endpoint that may be influenced by the simultaneous presence of multiple variants in those genes. in addition to its theoretical importance, epistasis has been functionally demonstrated to play a role in human disease. most notably, hirschsprung's disease was found to be influenced by polymorphisms in ret and the erdb receptor in the old order amish and was confirmed in a mouse model. having both variants simultaneously increases risk of disease far beyond the combined risk of each independent variant. as epistasis is believed to have important implications for human disease risk, numerous statistical and computational approaches have been developed to examine epistasis in family-based and case-control association studies. multifactor dimensionality reduction is one such computational method to identify gene-gene interactions in case-control studies where variants may or may not exhibit detectable marginal effects. mdr has been shown previously to have reasonable power to detect gene-gene interactions in several cases of experimental error and over a variety of simulated genetic models. mdr has also been applied to many disease phenotypes including hypertension, multiple sclerosis, sporadic breast cancer, type ii diabetes, coronary artery disease, and autism. there have been several notable extensions to the mdr method. multifactor dimensionality reduction pedigree disequilibrium test was developed by martin et al. to examine multi-locus models in extended pedigree data. mdr was extended to use a chi-square statistic as an alternative to prediction error/classification error, and to test other forms of cross-validation and permutation testing. mdr was also modified to include the odds ratio as a quantitative measure of disease risk – as well as extended into a generalized mdr to include discrete and quantitative covariates as well as dichotomous and continuous phenotypes. velez et al. evaluated the performance of mdr using balanced accuracy for several examples of class-imbalance. as the balanced accuracy measure provided improved power for cases of class imbalance, we hypothesized that alternate measures of classification performance would improve the performance of mdr. in this study, we demonstrate through simulated data that altering the scoring measure used in model evaluation and selection can improve the detection and statistical power for complex interaction effects. for the purposes of this study, simulated epistasis models contain no marginal effect, where all phenotypic variance attributable to genetic variation is due completely to the interaction of genetic loci, as described by culverhouse et al. to detect these genetic effects, the influential loci must be examined jointly. mdr is a non-parametric statistical method for the analysis of gene-gene and gene-environment interactions. figure illustrates the mdr algorithm. additional detail is provided in the "methods" section. mdr performs an exhaustive search of all n-locus models, collapsing multi-locus genotypes into high-risk and low-risk classes. these high- or low-risk classes are then compared to the observed status of individuals to produce a score for the classification. in this manner, all n-locus models are ranked by a scoring measure, and the model with the optimal score is selected as the best or "most fit." fitness measures the results of a classification algorithm can be represented as a special type of two-way contingency table. the true status forms one dimension of the table and the algorithm classification forms the other dimension. two-way tables can be scored with a variety of measures, many of which have been developed in multiple disciplines using different terminology to describe similar concepts. there are several basic terms and measures using the four cells of a two-way contingency table. sensitivity is the classification accuracy of the cases, or the proportion of correctly classified cases among all cases in the data. in the text classification field, this measure is called recall. specificity is the classification accuracy of the controls, or the proportion of correctly classified controls among all controls in the data. positive predictive value or precision is the classification accuracy of the affirmative classification, or the proportion of actual cases among all individuals classified as cases. negative predictive value is the proportion of controls among all individuals classified as controls. using these basic values, several composite measures of association have been developed. the ten measures selected represent a variety of analysis strategies from several fields including text classification, machine learning, diagnostic testing, statistical theory, and information theory. classification error and the related quantity, classification accuracy, are two of the most frequently used and contentious measures of classification performance. it is defined simply as the proportion of examples incorrectly labelled by a classifier. the technical merits of classification error as a measure of classifier performance have been debated. precision-based and roc-based measures f-measure is a measure first used by lewis and gale for assessing text classification effectiveness, and is the inverse of the e-measure. the e-measure is a weighted combination of sensitivity and positive predictive value derived by van rijsbergen to satisfy several conditions of measurement theory. geometric means have been used as performance measures for classification. kubat et al. defines two such quantities, here labelled geometric mean and geometric mean. geometric mean is the geometric mean of sensitivity and precision. geometric mean is the geometric mean of sensitivity and specificity. this measure is related to the receiver-operator characteristic curve and was used in lieu of roc analysis by kubat et al. as a single measure of classification. euclidean distance from an ideal classification is a measure also related to roc curves. this combination of sensitivity and specificity measures the distance from an ideal classification in roc space, where sensitivity and specificity both equal one. diagnostic testing measures the youden index and predictive summary index are summary measures of certainty for dichotomous diagnostic tests. the youden index is the sum of the sensitivity and specificity minus one. the predictive summary index is the sum of the positive predictive value and the negative predictive value minus one. statistical measures pearson's chi-square goodness-of-fit statistic is an adjusted sum of the squared differences between observed and expected frequencies. the chi-square is a classic test of association in categorical data analysis. the likelihood-ratio test is a related measure that statistically compares the maximum likelihood of an unrestricted model with a restricted model. in this setting, the unrestricted model consists of the observed frequencies in the data and the restricted model consists of the expected frequencies under the null hypothesis of no association. information theoretic measures normalized mutual information was described by wickens as a measure of information transmission, based on shannon's entropy. entropy was developed in communication theory as a measure of dispersion for categorical data. entropy is often measured in bits, or log base units. given a two-way contingency table, four entropy values can be computed: the row entropy, the column entropy, and two conditional entropies ) not shown): h=−∑ipilog⁡2pi h=−∑jpjlog⁡2pj h=∑ipi the quantities pi and pj represent the empirical probabilities of the predicted and true class states, respectively, and pij is their joint probability. using these values, nmi and its transpose are defined as: nmi=h−hh nmit=nmi=h−hh the nmi value is interpreted as the proportion of information contained in the row variable that is transferred or transmitted to the column variable, or more concisely the amount by which the model reduces our uncertainty about the true state.  detection all detection results are shown in figure and specific detection results are in figure "detection" is the ability of the method to correctly identify all disease loci, but additional non-disease loci may be included in the model also. "specific detection" is the ability of the method to correctly identify all the disease loci and no additional loci – this could also be phrased as the ability to detect the correct multi-locus model with no false positive loci. significant differences from classification error had wilcoxon rank-sum p-values below. over all models, the mean detection using classification error was 62% and the mean for specific detection was %. detection was at or above 80% in all two-locus models, with 100% detection for all two-locus models with ≥ % heritability. detection diminishes in the three-locus models, with only three models showing greater than 80%. detection in four- and five-locus models drops below 80% except for model a five-locus model with 3% heritability, which was detected at 96%. specific detection using classification error is above 80% for all two-locus models except the % heritability models. all three-, four- and five-locus models are below 80% specific detection except for model with specific detection at 95%. these models show sporadic detection and specific detection that does not follow trends based on allele frequency or heritability. the precision-based measures f-measure and geometric mean performed poorly. over all models, f-measure averaged % detection, with to 23% detection in two-locus models, to 15% detection in three-locus models, and less than 5% in four- and five-locus models. specific detection was near 0% for all models. allele frequency had no impact on detection. geometric mean performed slightly better than f-measure with 12% overall detection. detection using this measure was sensitive to the allele frequencies specified in the genetic model. detection in the maf models ranged between 24–34% in two-locus models, 14–24% in three-locus models, and 4–10% in four- and five-locus models. detection in the maf models ranged between 15–19% in two-locus models, 10–13% in three-locus models, and 2–5% in four- and five-locus models. specific detection using geometric mean was near 0% for all models. both f-measure and geometric mean showed significantly worse detection and specific detection than classification error in nearly every case. notably, these two measures outperformed all others for detection of two very difficult models, but in these cases the detection was very low and specific detection was no better than classification error. the roc-based measures euclidean distance and geometric mean also fail to outperform classification error overall, but perform well for some models. the average detection using euclidean distance over all models was only %, but in two-locus models was near 100% except for the lowest heritability models. across higher order models detection was generally scattered with no discernable trend, ranging from 2–77% in three-locus models, 3–46% in four-locus models, and 0–41% in five-locus models. similarly, specific detection using euclidean distance was between 90–99% for all two-locus models except the lowest heritability models. specific detection in higher order models was also scattered, but in general was higher for higher heritability models. geometric mean showed very similar trends for detection and specific detection to euclidean distance. detection was 85–100% for two-locus models, 13–70% for three-locus models, 6–20% in four-locus models and less than 7% in five-locus models. specific detection was between and 99% in all but the lowest heritability two-locus models. all others had less than 10% specific detection. using euclidean distance and geometric mean only a few models show better detection than using classification error. the most notable of these is model where geometric mean improved over classification error by. also, both euclidean distance and geometric mean two showed significantly improved specific detection over classification error in all two-locus models except model the diagnostic testing measures youden index and predictive summary index perform relatively well. overall, average detection was 64% using the youden index, and specific detection was 57%. detection of two-locus models was 84–100%, –97% in three-locus models, 5–94% in four-locus models, and 0–95% in five-locus models. specific detection was 90–92% in all two-locus models except the lowest heritability models which were 65–72%. other models have higher specific detection in higher heritability models that decreases along with heritability. psi also shows excellent overall average detection at % and specific detection at 59%. detection was 84–100% in two-locus models, –97% in three-locus models, –94% in four-locus models, and 0–98% in five-locus models. the detection using psi is lower in two-locus models than the youden index, but psi shows slightly higher detection in a few three-, four- and five-locus models. compared to classification error, psi shows significantly increased detection in of the models. models show no significant difference in detection, and for models, classification error performs significantly better. most of the improvement in detection is seen in higher order models, and notably, classification error detects as well or better then psi for two-locus models. the youden index shows better detection for models. for models, classification error is significantly better, and there is no significant difference for models. specific detection with psi is improved over classification error in three-, four- and some five-locus models. specific detection in two-locus models however is not consistently improved. similar to psi, the youden index does not improve over classification error in two-locus models. improvement in detection for the youden index is in three- and four-locus models. detection using the youden index is not improved over classification error in five-locus models. specific detection with the youden index is significantly improved over classification error in all but five-locus models. the chi-square and likelihood ratio statistical measures performed well. using the chi-square, across all models average detection was % and average specific detection was %, with 78–100% detection for two-locus models, 11–82% detection for three-locus models, and 0–97% detection in four- and five-locus models with little discernable trend. specific detection patterns also show no trend. overall detection using the likelihood ratio is % and specific detection is %. in two-locus models, detection ranged from 84–100%, 16–96% in three-locus models, and 0–97% in four- and five-locus models. using chi-square, detection of models was significantly better than classification error, models showed no significant difference in detection from classification error, and models were detected significantly worse than using classification error. specific detection was significantly better for models using chi-square, was not significantly different from classification error for models, and was significantly worse than error for models. most of the detection improvement from using the chi-square measure was seen in three- and four-locus models. detection using likelihood ratio was significantly better or no different than error for all but models, and specific detection was significantly improved for all models except five-locus models, where specific detection of models was significantly worse using the likelihood ratio and was not statistically different for of the models. the information theoretic measures nmi and nmi-transpose both perform relatively well. overall, detection using nmi was % across all models, and specific detection was %. with nmi, detection of two-locus models ranged from 84–100%, three-locus models from 16–96% and four- and five-locus models ranged from 0–98% detection. for specific detection the trends were very similar to those of the likelihood ratio, but specific detection was slightly higher in most cases. using nmit, overall detection was highest of all measures at %, with a specific detection of %. per model detection rates were lower than with other measures, however, with two-locus model detection ranging from 45–98%, three-locus models from 23–98%, and four- and five-locus models ranging from 0–98%. trends in specific detection were erratic, performing better than nmi in some three- and four-locus models, but much worse power for two-locus models. nmit did demonstrate higher detection of higher order models. detection using nmi is significantly better than classification error in models, shows no difference in models, and performs worse than classification error in models. specific detection is either significantly improved with nmi or shows no significant difference from classification error in all but five-locus models. nmit performs significantly worse than classification error for two-locus models, but shows significantly better detection for three- and four-locus models, with specific detection following the same trend. in summary, euclidean distance, f-measure, geometric mean and geometric mean perform significantly worse than classification error in both detection and specific detection for a majority of simulated models. two measures, the chi-square and nmit show improvement in detection and specific detection of some models, but are significantly worse than classification error for other models, with especially poor specific detection of two-locus models. the remaining four measures show either improved or equal detection and/or specific detection across a majority of models and work well for two-locus models. these measures, youden index, predictive summary index, likelihood ratio, and normalized mutual information were evaluated for statistical power using permutation testing. power all power results are shown in figure and specific power is shown in figure "power" is detection that is statistically significant, and "specific power" is specific detection of the correct multi-locus model that is statistically significant. the process used to assign statistical significance to a result is dependent on the assumption that for these simulated data, the permutation distribution of a single dataset is equivalent to the permutation distribution of other datasets simulated using the same genetic model. after evaluating the variability of α = critical values, this assumption holds well predictive summary index and the youden index have power and specific power only in 3% and 2% heritability models – all other models show 0% power and specific power. psi outperforms the youden index for two-locus models for both power and specific power, but in higher order models, the youden index performs slightly better. neither of these measures shows improvement over classification error in the context of permutation testing. compared to classification error, the likelihood ratio shows significantly better power in models, significantly worse power in models, and no significant difference in models. of the models where error was better, were five-locus models, one was a % heritability three-locus model, and one was a % heritability two-locus model. specific power of the likelihood ratio is significantly better than error for models, significantly worse for models, and not significantly different for models. these five models are the 2- % heritability five-locus models with minor allele frequencies. using nmi, power is significantly better than error in models, significantly worse for models, and not significantly different in models. similar to the likelihood ratio, of the models are five-locus models, one was the same % heritability three-locus model, and one was a % heritability two-locus model. the same models showing significant differences in specific power from classification error for the likelihood ratio are also significantly different using nmi. specific power is significantly better for models, significantly worse for models, and not significantly different for models. comparing likelihood ratio and nmi, both show equal improvement over classification error for specific power. one model showed significantly worse power compared to error when using the likelihood ratio, but power was not significantly different when using nmi. power using likelihood ratio was significantly better than power using nmi for three-locus models, and four-locus models. power using nmi was significantly better than power using likelihood ratio for five-locus models. specific power using likelihood ratio was significantly better for models – three-locus models, and four-locus models. nmi has significantly better specific power for five-locus models. discussion this study is an exploration of fitness measures in the mdr algorithm. here, we have evaluated ten alternative measures of fitness for mdr models and compared them to the traditional measure, classification error. this work is highly motivated by the dispute over the use of classification error, as it is known to be an improper scoring rule. in addition, mdr has been released in a java software package with a user-friendly graphical user interface version where many of these measures are currently available. we felt it was important to know which measures are robust for higher order interaction models. the first series of simulations evaluated the traditional fitness measure for mdr, classification error, for a set of disease models. from these simulations, we observe some obvious trends in the two locus models as well as some irregular patterns in the higher order models. with the two-locus models, the detection and power of mdr using classification error decreases as the broad sense heritability of the model decreases. trends for higher order models are non-linear with respect to broad sense heritability. most notably, five-locus models with minor allele frequencies had a very erratic pattern, with the % and % heritability models having near 0% power while the 1% heritability model has 22% power. these erratic patterns are likely due to one of two possible scenarios. one possibility is related to the way in which the genetic models were created. the genetic algorithm-based procedure for creating multi-locus models with no marginal effects is a directed search through the space of all possible penetrance models. there may be millions of models which satisfy the heritability requirements, and we arrive at a random sample from that model space with each run of the procedure. thus, there is some random variability between penetrance functions with similar heritabilites selected in this manner. this variability likely impacts the ability of mdr to detect the effect of that model, as mdr does not measure an effect based on the penetrance table directly, but rather on the derived two-way contingency table. a second possible scenario is that these penetrance models were generated with the assumption that there are no main effects of any single locus. however, in a higher-order models, the simulator does not test for all possible lower-order models. for example, if a three-locus model is being generated, there are no single locus effects for any of the three-loci; however, the software does not test for the presence of two-locus models embedded within the three loci. thus, it is possible that the lower detection and power results are due to multiple models competing for detection. this was the case for model for example, where models containing two of the three loci were detected in 70% of evaluations. for this study, we consider the performance of classification error as a baseline, so while these effects are interesting to note, they are irrelevant for evaluating other classification measures. mdr using classification error has greater than 80% detection for of the multi-locus models, and greater than 80% power for of the models. while high-order models with no marginal effects have been speculated to exist, there is little confirmed experimental or statistical evidence to support their role in complex disease. it is encouraging to see however that mdr has appreciable power for many high-order models in a feasible sample size of cases and controls. if susceptibility to common disease truly does involve complex interactions among many variants, tools for detecting these interactions will be critical. performance of alternate measures the f-measure and geometric mean both show similar detection and specific detection results across all models. these two measures combine precision and sensitivity in their calculation. in general these measures perform very poorly compared to classification error, indicating that information about the specificity of an mdr model improves the performance of mdr. these measures are focused on classifying cases correctly. an mdr model that classifies cases best is not necessarily based on the most associated genetic factors. the model would not discern the difference between genetic factors strongly associated to both cases and controls and only those associated to cases. geometric mean and euclidean distance are similar in that they are geometric functions of sensitivity and specificity, functioning similarly to a receiver-operator characteristic analysis. these measures perform commensurately with classification error for two-locus models, but do not perform especially well for higher order interactions. euclidean distance in general performs better than geometric mean for both detection and specific detection. the one notable exception to this trend is geometric mean which shows very good specific detection in two-locus models with maf, out-performing all other measures. the diagnostic measures youden index and psi perform well in detection and specific detection. the youden index seems to show better detection than psi in two-locus models and high heritability three-locus models. psi however shows better specific detection over all models. both measures out-perform classification error in both detection and specific detection, but those measures do not show improved power or specific power over classification error. one explanation could be that the empirical distribution of these statistics was not as stable, so the standardized distribution for each model failed to properly assign statistical significance. while the empirical distribution of randomly selected datasets did not differ significantly from the standardized distribution, some subtle variability in the tail regions was noted. another possibility is that these measures are more susceptible to noise in the data. the power and specific power are stronger for 3% heritability models than for models with lower heritability. while these measures still have utility for detection, their usefulness when assigning statistical significance is questionable. the chi-square shows good detection and specific detection in the four-locus models and a few five-locus models, but fails to out-perform ce in most other cases. one reason why the chi-square does not perform well in this setting is that theoretically, the chi-square is not a satisfactory measure of association, and may not rank mdr results to produce optimal detection. the chi-square tests deviation from independence, but does not necessarily quantify the strength of an association. this is an important consideration for using the chi-square test in other studies, particularly whole-genome association studies, as the chi-square may not necessarily rank signals by strength of association the measures that demonstrate the most consistent improved performance are the likelihood ratio and nmi. while the improvement is not dramatic, these measures show equal or better detection and power across nearly all models. the more dramatic improvement is in specific detection and specific power, where the genetic model detected by mdr is the exact model that was simulated. this is an interesting result because this means that mdr using nmi or lr is less susceptible to over-fitting. both the likelihood ratio and nmi measures are based on entropy, which is loosely analogous to variance in the two-way table, and both measures show very similar trends for detection and power. these two measures are ultimately related, as the numerator of nmi is a transformation of the lr. h−h=lr2p•• the likelihood ratio test is a well-established statistical test to examine a sample's deviation from a null hypothesis. the statistic itself however does not have an intuitive interpretation, and often is transformed to achieve a valid measure of association. nmi of the contingency table treats the "true outcome" and the "model prediction" as a pair of two-state random variables. nmi quantifies the amount of uncertainty about the state of the "truth variable" removed by the "model prediction variable". nmi has a nice interpretability as the amount by which the model reduces our uncertainty about the true state. while nmit performs well for three-locus models in particular and did outperform classification error in many cases, its poor detection and specific detection of two-locus models makes it an unattractive measure. in addition, nmit's interpretation makes less sense as the amount the true variable reduces our uncertainty about the model. nmi includes details of the contingency table not accounted for by the other measures of model predictability. for example, the numerator of nmi takes into account the power to correctly predict both the cases and controls: a/ and d, respectively. explicitly, we can rewrite part of the nmi numerator terms as aln⁡+dln⁡ which is closely related to predictive summary index. in addition, we can rewrite part of the nmi denominator as aln⁡+dln⁡ which bears strong resemblance to the youden index in its attempt to balance the model sensitivity and specificity: a/ and d/, respectively. the detailed form of the nmi measure likely leads to its observed ability to distinguish between closely similar high quality models; and hence, nmi's improved ability to uniquely determine the relevant variables. also, nmi preferentially selects models that classify either cases or controls perfectly. these models are more "stable", or less variable, and are thus to be preferred by the measure over models where both cases and controls are misclassified equally. the dependence of nmi on the contingency table is quite intricate and warrants further investigation to understand its strengths and limitations more fully. while the power improvement using nmi or the likelihood ratio in most cases is not dramatic, these measures are superior to classification error. using these measures, there is higher detection and power of high-order interactions and better specific power overall, so an analyst can be more confident that an mdr model does not contain spurious variables. also, classification error assumes the distribution of the two classes to be equal. this shortcoming was recently addressed by velez et al. who used an average of sensitivity and specificity to compute a balanced classification error for mdr model evaluation, and demonstrated its power to detect gene-gene interactions in cases of class imbalance. both nmi and lr also take into account the sensitivity and specificity of an mdr model, and likewise should not be susceptible to class imbalance. one clear advantage of classification error is its interpretability. of the two improved measures, nmi perhaps has the easiest interpretation. its value ranges from to with meaning the genotype and status are independent and meaning the genotype completely determines the status. also, as nmi provides a direct information theoretical measure of association, it may be preferred over the likelihood ratio test statistic, which measures deviation from the null hypothesis of independence rather than directly quantifying the degree of association. for these reasons, we recommend that nmi be used in lieu of classification error for mdr analyses. for clarity of interpretation, we recommend showing the two-way contingency table along with reporting the nmi of an mdr result. classification error is a widely used measure of performance in many areas of research. the results of this study are specific to classification using the mdr procedure, but this work does provide additional empirical evidence to support general theoretical arguments against the use of classification error. limitations there are some limitations of this work. first, this study was conducted using simulated data, so confidence in the results relies on the quality of the simulated data. all disease loci in a genetic model were simulated with the same minor allele frequency, either or, which is a simplification. the biological relevance of the penetrance functions simulated could be questioned, though we use functions with very small marginal effects as a "worst-case scenario" and expect the results of this study to generalize to situations where marginal effects are detectable. it is troubling that the trends observed in power and detection do not always follow the expected trends given the broad-sense heritabilities simulated. however, as this study is focused on the relative performance of classification measures, this point is not critical. permutation testing for all models evaluated in this study was not computationally feasible, and a single permutation distribution for each of the models was generated for the measures evaluated for statistical power. these permutation distributions did not appear to vary significantly by qq-plot from a randomly seeded dataset using the same model. assigning significance using a permutation distribution uses the tails of the distribution, however, and some variability in the tails was observed in the qq-plots. but the power results do closely follow detection results, and it seems unlikely that all of the permutation distributions used would consistently over estimate or under estimate the tail values. the simulated data included only ten loci total. while small-scale studies are still performed, low-cost genotyping solutions have dramatically increased the number of polymorphisms examined in a typical study. while the computation time required to perform this study prohibited using a large number of snps for this evaluation, we expect that the relative performance of these fitness measures would extend to datasets with larger numbers of snps. absolute detection and power, however, are influenced by a variety of factors including the number of snps in the dataset.  over a variety of simulated genetic models, normalized mutual information and the likelihood ratio demonstrate stellar performance as measures of multifactor dimensionality reduction model fitness; an improvement in comparison to classification error. the ability of mdr to specifically detect only the simulated disease loci was significantly higher using these two measures for nearly all genetic models. these measures also show improved statistical power by permutation testing. of these measures, nmi is perhaps easier to interpret, as it is the amount by which the model reduces our uncertainty about the true disease state. nmi properly treats imbalanced data and provides superior performance over classification error. therefore, we recommend using nmi as an alternative to classification error for mdr analyses. 
of the measures evaluated, the likelihood ratio and normalized mutual information are measures that consistently improve the detection and power of mdr in simulated data over using classification error. we simulated genetic models, varying the number of disease loci in the model, allele frequencies of the disease loci and the broad-sense heritability of the model. mdr performs a dimensional reduction by assigning multi-locus genotypes to either high- or low-risk groups and measuring the percentage of cases and controls incorrectly labelled by this classification – the classification error. these measures also reduce the inclusion of spurious variables in a multi-locus model. multifactor dimensionality reduction has been introduced previously as a non-parametric statistical method for detecting gene-gene interactions. overall, detection using nmi was % across all models, and specific detection was % versus detection using classification error at 62% and specific detection was %. the correctly and incorrectly labelled cases and controls can be expressed as a two-way contingency table. in this study, we compare the detection and power of mdr using a variety of measures for two-way contingency table analysis. we sought to improve the ability of mdr to detect gene-gene interactions by replacing classification error with a different measure to score model quality.
 multifactor dimensionality reduction has been introduced previously as a non-parametric statistical method for detecting gene-gene interactions. mdr performs a dimensional reduction by assigning multi-locus genotypes to either high- or low-risk groups and measuring the percentage of cases and controls incorrectly labelled by this classification – the classification error. the combination of variables that produces the lowest classification error is selected as the best or most fit model. the correctly and incorrectly labelled cases and controls can be expressed as a two-way contingency table. we sought to improve the ability of mdr to detect gene-gene interactions by replacing classification error with a different measure to score model quality.  in this study, we compare the detection and power of mdr using a variety of measures for two-way contingency table analysis. we simulated genetic models, varying the number of disease loci in the model, allele frequencies of the disease loci and the broad-sense heritability of the model. overall, detection using nmi was % across all models, and specific detection was % versus detection using classification error at 62% and specific detection was %.  of the measures evaluated, the likelihood ratio and normalized mutual information are measures that consistently improve the detection and power of mdr in simulated data over using classification error. these measures also reduce the inclusion of spurious variables in a multi-locus model. thus, mdr, which has already been demonstrated as a powerful tool for detecting gene-gene interactions, can be improved with the use of alternative fitness functions. 