BACKGROUND
it is crucial to show that a classifier designed using supervised learning performs sufficiently well for the application of interest.
a minimum requirement is that it is performs better than random guessing.
recently gene expression profiling using microarray technology has been widely used for classification of tumors based on supervised learning  <cit> .
various cross-validation and resampling methods aimed at providing reliable and robust estimates of classifier performance have been proposed  <cit> .
a natural measure of the robustness of an algorithm is the variance of the distribution of error rates when the classifiers are designed using the number of training examples available.
recently attempts have been made to obtain confidence intervals based on small sample sizes  <cit> .
these approaches correspond to an idealized case where the bounds on the unknown performance of a classifier designed using nd samples are obtained by repeated designs and tests using new examples.
if this procedure would yield a large set of high quality performance estimates, their distribution could be used to estimate a 95% confidence interval  of the true error rates.
notably, in this approach no point estimate of the performance for the particular classifier of interest is calculated.
the quantity of interest is the 95% ci for the whole distribution of possible true performances.
since this ci covers the true performance of interest with probability 95%, without any additional information available, e.g.
from a conventional holdout test, it represents the current state of uncertainty about the true performance.
since estimation of ci using this method would require access to large amounts of data that are not available in practice, a suggested alternative approach is to estimate the ci using resampling techniques like in the recent work by michiels et al.
<cit> .
in their work, a performance estimation method called repeated random sampling , that was originally described by mukherjee et al.
<cit> , is applied to seven large gene expression data sets.
for almost all the data sets, michiels et al.
demonstrate that the sizes of the cis obtained from the rrs procedure increase with increasing sizes of the design sets.
this is counterintuitive as the variance σd <dig> of the true performances should decrease with increasing size of the design sets.
with more data used for design, the placement of the decision boundary of the classifier will be more stable and as a consequence the resulting σd <dig> will be lower  <cit> .
hence, the variance and confidence interval obtained from rss often have a bias.
in this paper we identify small test-set size as one factor that can lead to the bias in the variance estimate observed using rrs.
we also introduce a first order model of the variance estimate as a function of the number of test examples for a refined, less biased, estimation method called repeated independent design and test .
furthermore, we demonstrate that by modeling the undesirable small sample bias in ridt, it is possible to greatly reduce the bias in the estimates of σd <dig> and therefore in the resulting cis.
for the special case of repeated designs and tests using completely novel samples, we present an exact analytical expression for how the bias in the estimates of σd <dig> decreases with increasing size of the test sets.
RESULTS
the estimated variance in repeated cross-validation depends on the number of test data
fukunaga and hayes  <cit>  pointed out that small test set size nt affects the variance of the performance estimates obtained in repeated hold out experiments.
this variance may be regarded as an estimate of σd2and is denoted rhσdt <dig> to indicate that it depends on nt.
we argued that a similar effect may also affect similar repeated cross-validation methods including rrs.
since the total number of examples is fixed, the number of test examples is automatically decreased when the size of design data is increased in rrs.
to be able to study the effect of test data size on its own, we modified the rss procedure and kept the size of the design data constant while varying the size of the test data.
we used a colon cancer microarray data set containing  <dig> normal and  <dig> colon cancer cases  <cit>  and classified the samples using a modified fisher's linear discriminant classification algorithm .
the size of the design was constant at 30% of the total data set size, while the size of the test set, nt was varied in steps from 70%, , down to 5%.
for each value of nt, the original data set was divided randomly into design and test sets  <dig>  times, while maintaining class proportions.
an almost trivial fact discussed formally by fukunaga and hayes  <cit>  is that the mean md and the variance σd <dig> of the distribution of true error rates is independent of the number of test examples nt.
consequently, the cis for the distribution of true error rates are constant for every choice of the number of design examples nd.
the results in figure  <dig> clearly show that the sizes of the cis obtained using repeated cross-validation are not constant but decreasing with increasing values of nt.
this is similar to what can be observed using rrs in michiels et al.
<cit>  where the number of design examples  is varied.
as shown in figure  <dig> the estimated ci stabilizes as the size of the test set becomes large.
the bias in the ci will be eliminated as the size of the test set becomes very large.
however, for the usually limited set of examples available in most real world problems, the bias is too large to be neglected.
repeated independent design and test
limited testing of each classifier is not expected to be the sole cause of undesirable bias in the rss estimate.
bias may also be attributed to three different statistical dependencies between data sets caused by the repeated design and testing performed using the bag of limited examples available: 1) each pair of design and test sets are dependent.
once the design set has been selected, the remaining examples become the test set deterministically.
2) the design sets are inter-dependent.
given information about the samples in a first design set, a lot of information is gained about the possible samples that may occur in the next design set obtained by means of resampling.
3) the test sets are also inter-dependent.
given information about the samples in a first test set, information has been gained about the possible samples that may occur in the next test set.
in this work we introduce a novel procedure denoted repeated independent design and test , which eliminates the first type of dependence by splitting the original data set of size n into a design bag with nd samples, and a test bag, with nt = n-nd test samples.
thus, for each design a fixed number of examples nd with equal number of samples from each class are drawn with replacement form the bag of nd samples.
this makes the resampling of design examples completely independent of the selection of test set examples.
notably, the design sets remain inter-dependent due to the small design bag and similarly the test sets remain inter-dependent due to the finite size of the test bag.
by repeatedly selecting design sets of size nd from the design bag and testing with data from the test bag, a number of error rate estimates are obtained that subsequently are used to obtain an almost unbiased estimate of the true variance σd <dig>  this variance estimate can in turn be used to construct the desired ci of the distribution of true performances.
a variance model for the ridt procedure
analogous to the variance rhσdt <dig> associated with idealized repeated holdout experiments discussed above, the variance of the error rate estimates obtained with ridt is dependent on the finite value of nt, as well as on nt and is denoted σdt <dig>  to study and reduce estimation biases caused by small sample size, we propose that for a given data set d, the ridt estimate of σdt <dig> may be approximated as
σ^dt2≈α0+α1nt+α2nt.
eq.
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacuwfdpwcgaqcamaadaaaleaacqwgkbazcqwg0badaeaacqaiyagmaagcdaqadaqaamaalaaabagaegymaedabagaemota40aasbaasqaaiabdsfaubqabaaaaogaeiilawyaasaaaeaacqaixaqmaeaacqwgobgtdawgaawcbagaemidaqhabeaaaagccqggsaalcqwgebaraiaawicacaglpaaacqghijyucqwfxoqydawgaawcbagaegimaadabeaakiabcicaoiabdseaejabcmcapiabgucarmaalaaabagae8xsde2aasbaasqaaiabigdaxaqabagccqggoaakcqwgebarcqggpaqkaeaacqwgobgtdawgaawcbagaemivaqfabeaaaagccqghrawkdawcaaqaaiab=f7ahnaabaaaleaacqaiyagmaeqaaogaeiikagiaemiraqkaeiykakcabagaemota40aasbaasqaaiabdsha0bqabaaaaogaeiola4iaaczcaiaaxmaacqqgfbqrcqqgxbqccqqguaglcqqggaaicqqgoaakcqqgxaqmcqqgpaqkaaa@607e@
this equation involves first order linear approximations of the biases introduced by the finite values of nt and nt .
for very large values of nt and nt, the estimate reduces to an unbiased estimate of σd <dig>  hence the first coefficient α <dig> should be an unbiased estimate of σd <dig>  i.e.
<α <dig> >d = σd <dig>  where <>d denotes the expectation operator.
notably, the model treats the size of the test bag nt in a similar way as the size of the test set nt, but ignores effects due to size of the design bag nd.
by evaluating classifications using nb design sets, varying the test bag sizes nt and sizes of tests sets nt for each value of nb, it is possible to estimate the data set dependent coefficients α <dig>  α <dig> and α <dig> in eq.
by multivariate least squares fitting.
in this process one constraint is used that ensures the natural inequality α <dig> ≥  <dig>  with access to the fitted coefficient α <dig> one has obtained an unbiased estimate of the desired quantity σd <dig>
we performed simulations using samples generated from two 2-dimensional normal distributions with mean values and covariance matrices estimated from real micorarray gene expression data  <cit> .
the aim was to validate our model, and to demonstrate its potential for elimination of the bias caused by small sample size .
since the two features  in both distributions are correlated, the simulation takes the dependence that may exist between features  in real data sets into account.
one should also note that we have deliberately chosen to use a classifier that does not contain a feature selection step to avoid additional complexity.
thus, the problems and solution discussed in this paper are equally relevant also for classifier using feature selection.
we have also chosen the strategy to evaluate the performance for each class separately, since it does not require knowledge about the probabilities of observing examples from class  <dig> or class  <dig>
to emphasize that we focused the analysis on class  <dig>  an extra subindex was introduced when denoting the quantities of interest.
thus, the true mean value and variance associated with class  <dig> are denoted md <dig> and σd <dig>  respectively, and corresponding quantities associated with testing using finite data sets are denoted mdt <dig> and σdt <dig>  in the ridt procedure the design bags had equal number of samples from both classes and the number of samples drawn with replacement was the same as the size of the design bag, i.e.
nd = nd.
we also made the assumption that mdt <dig> is an unbiased estimate of md <dig> which was verified .
the mean values of the estimated md <dig> and σd <dig>  and the corresponding two-sided 95% cis for eight different values of nt <dig> , are presented in figure  <dig> for nd = nd =  <dig> .
the true values md <dig> and σd <dig>  obtained by testing  <dig>  independently designed classifiers using  <dig>  test samples each, are also indicated.
apparently, unbiased estimates of md <dig> and σd <dig> are obtained.
we observed that the reduction of the small sample bias yields accurate estimates of σd <dig> on average.
one should note that in general the estimates contain contributions of higher order terms.
however, as the results indicate, the biases caused by these higher order terms may be quite small for commonly used sizes of data sets.
for the data set sizes commonly used it appears that it is possible to obtain practically unbiased estimates of md <dig> and σd <dig> with the ridt method.
variance for independent data
in the special case of truly independent data, i.e.
when each pair of design and test sets are drawn from the underlying true distribution of samples instead of from a finite bag of examples, we derived an exact analytical equation for σdt <dig> :
σdt2=σd2+md−σd2nt.
eq.
mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemizaqmaemidaqhabagaegomaidaaogaeyypa0jae83wdm3aa0baasqaaiabdsgakbqaaggaaiab+jdayaaakiabgucarmaalaaabagaemyba02aasbaasqaaiabdsgakbqabagccqggoaakcqaixaqmcqghsislcqwgtbqbdawgaawcbagaemizaqgabeaakiabcmcapiabgkhitiab=n8aznaadaaaleaacqwgkbazaeaacqaiyagmaaaakeaacqwgobgtdawgaawcbagaemidaqhabeaaaagccqgguaglcawljagaaczcaiabbweafjabbghaxjabb6cauiabbccagiabbicaoiabbkdayiabbmcapaaa@52cc@
this equation shows how the observed variance depends on the number of test samples nt as well as on md and σd <dig>  it can also be noted that for very large values of nt, eq.
reduces to the same form as eq.
.
fukunaga and hayes  <cit>  have previously published an approximation of eq .
one advantage with the exact equation is that it can be used to show that the second term always is larger than zero and that σdt <dig> is always larger than σd <dig>  thus, if σdt <dig> would be an approximation of the variance, the resulting ci would always be conservative.
in order to empirically validate eq.
, simulations were performed using samples drawn from two 8-dimensional normal distributions .
we determined σdt <dig> for different values of nt and nd .
each value of σdt <dig> was obtained using a histogram of  <dig>  independent point estimates .
for comparison,  <dig>  separate and independent high accuracy point estimates of md and σd <dig> were computed, each using  <dig>  test samples for varied design set sizes nd.
the true observed variances in figure  <dig> were then obtained from eq.
.
this demonstrates excellent agreement between the theory and simulations.
discussion
rrs has been proposed as a practical method for estimation of the distribution of error rates obtained when a specified number of data samples are used for design  <cit> .
however, we have demonstrated that the variance estimate of the performance for classifiers designed and tested in a similar way results in a variance estimate that is highly dependent on the number of samples used for test .
a qualitatively analogous effect should occur also in rrs, which is equivalent to using all remaining examples for test in our experiment.
consequently, highly conservative estimates of the variance are obtained with repeated testing methods when the number of examples used for test is small.
in practice the variance estimates have a bias of unknown magnitude, due to the complex statistical dependence between design and test sets.
therefore, it is important to stress that the confidence interval in rrs cannot be used to draw any conclusions about whether it is likely that a classifier performs better than chance.
an example of this inappropriate use of rrs can be found in  <cit>  where the possibilities to predict cancer outcome based on microarray gene expression patterns were investigated in several data sets.
perhaps even more importantly, a large bias in the variance estimate of interest is not a unique feature of the rss procedure but is expected to be found in all other suggested resampling procedures for performance estimation.
for example, estimating the variance of a q-fold cv performance estimate as suggested by mclachlan et al.
<cit>   seems attractive but we are not aware of any theoretical or numerical proofs that those and similar methods result in unbiased estimates of the variance σd2of interest.
on the contrary, the proof of equation  in our manuscript clearly shows that even if it would be possible to draw infinitely many independent design and test sets from the true distribution of samples, the resulting variance estimate of interest is heavily biased when the test sets are small.
there are a number of features of the ridt method that have implications for the use of the method.
first, the ridt performance estimates rely on a split of the data set into two separate parts, one used for repeated design, the other for repeated tests, which is not current practice in cross-validation and bootstrapping and might be interpreted as inefficient use of the few samples available.
we view this as a price that has to be paid in order to provide unbiased estimation of the variance of interest which can not be obtained with other methods.
second, although normal distributions were used in the computer simulations performed to generate the results presented here, the elimination of finite sample effects using eq.
does not assume normally distributed data, but can use data from any type of distribution.
third, even though eq.
does not depend on nd, it is possible to reduce small sample effects and provide unbiased estimates for the specific problem considered here.
the general applicability of this observation awaits further studies but eq.
can easily be extended to include a fourth term that is explicitly dependent on nd, see eq.
.
one possible explanation for the small influence of nd in the ridt method used here is that the design sets are drawn with replacement from the design bag, a procedure that closely reflects what happens in reality.
although not yet explored in detail, there are several explanations for the small bias in σd <dig> observed in figure  <dig> when nt <dig> ≤ 50: 1) we are ignoring higher order terms in the approximations.
2) we do not try to eliminate effects caused by a finite value of nd.
3) we do not take any small sample effects into account at all when estimating md.
4) when using replacement, we employ on average only  <dig> % unique samples in each design  <cit> .
notably, the number of design examples nd remains fixed and, as discussed above, there is no contribution to the bias due to nd being small.
we find that the variance of the inter-design set variance estimate σd <dig> is relatively large and increases with decreasing value of nt <dig>  this means that the estimate of σd <dig> for a particular data set is unbiased, but that it may be associated with a large uncertainty especially if the size of the data set is small.
therefore, it is difficult to directly use the unbiased estimates of md and σd <dig> to construct a ci.
thus it appears that even though we can compensate for biases caused by small sample size, the resampling approach has not provided a method that is practically useful in its present form.
therefore the only rigorous option for estimation of classifier performance that we know of is the classical hold out test combined with a bayesian credibility interval  <cit> , even though this interval is overly conservative and provides very wide intervals.
CONCLUSIONS
one major suggestion from the results of this paper is that previously introduced resampling and cross-validation methods for performance estimation using small sample sets are expected to result in large biases in their estimates of the inter design set variances.
consequently such biased variance estimates lead to inappropriate confidence intervals for the performance of a chosen classifier.
in addition this paper describes a method that is capable of eliminating this bias for a new resampling method  also introduced here.
finally we would like to point out that although this paper provides important experimental and theoretical results, the large variability in the unbiased variance estimate obtained still leaves us one step away from a practically useful solution for small sample based estimation of confidence intervals using resampling.
we therefore also hope that this work will inspire others to consider how to convert the unbiased but highly variable variance estimate of our and similar future procedures into a valid confidence interval.

 it is crucial to show that a classifier designed using supervised learning performs sufficiently well for the application of interest. a minimum requirement is that it is performs better than random guessing. recently gene expression profiling using microarray technology has been widely used for classification of tumors based on supervised learning. various cross-validation and resampling methods aimed at providing reliable and robust estimates of classifier performance have been proposed. a natural measure of the robustness of an algorithm is the variance of the distribution of error rates when the classifiers are designed using the number of training examples available. recently attempts have been made to obtain confidence intervals based on small sample sizes. these approaches correspond to an idealized case where the bounds on the unknown performance of a classifier designed using nd samples are obtained by repeated designs and tests using new examples. if this procedure would yield a large set of high quality performance estimates, their distribution could be used to estimate a 95% confidence interval of the true error rates. notably, in this approach no point estimate of the performance for the particular classifier of interest is calculated. the quantity of interest is the 95% ci for the whole distribution of possible true performances. since this ci covers the true performance of interest with probability 95%, without any additional information available, e.g. from a conventional holdout test, it represents the current state of uncertainty about the true performance. since estimation of ci using this method would require access to large amounts of data that are not available in practice, a suggested alternative approach is to estimate the ci using resampling techniques like in the recent work by michiels et al.. in their work, a performance estimation method called repeated random sampling, that was originally described by mukherjee et al., is applied to seven large gene expression data sets. for almost all the data sets, michiels et al. demonstrate that the sizes of the cis obtained from the rrs procedure increase with increasing sizes of the design sets. this is counterintuitive as the variance σd of the true performances should decrease with increasing size of the design sets. with more data used for design, the placement of the decision boundary of the classifier will be more stable and as a consequence the resulting σd will be lower. hence, the variance and confidence interval obtained from rss often have a bias. in this paper we identify small test-set size as one factor that can lead to the bias in the variance estimate observed using rrs. we also introduce a first order model of the variance estimate as a function of the number of test examples for a refined, less biased, estimation method called repeated independent design and test. furthermore, we demonstrate that by modeling the undesirable small sample bias in ridt, it is possible to greatly reduce the bias in the estimates of σd and therefore in the resulting cis. for the special case of repeated designs and tests using completely novel samples, we present an exact analytical expression for how the bias in the estimates of σd decreases with increasing size of the test sets.  the estimated variance in repeated cross-validation depends on the number of test data fukunaga and hayes pointed out that small test set size nt affects the variance of the performance estimates obtained in repeated hold out experiments. this variance may be regarded as an estimate of σd2and is denoted rhσdt to indicate that it depends on nt. we argued that a similar effect may also affect similar repeated cross-validation methods including rrs. since the total number of examples is fixed, the number of test examples is automatically decreased when the size of design data is increased in rrs. to be able to study the effect of test data size on its own, we modified the rss procedure and kept the size of the design data constant while varying the size of the test data. we used a colon cancer microarray data set containing normal and colon cancer cases and classified the samples using a modified fisher's linear discriminant classification algorithm. the size of the design was constant at 30% of the total data set size, while the size of the test set, nt was varied in steps from 70%,, down to 5%. for each value of nt, the original data set was divided randomly into design and test sets times, while maintaining class proportions. an almost trivial fact discussed formally by fukunaga and hayes is that the mean md and the variance σd of the distribution of true error rates is independent of the number of test examples nt. consequently, the cis for the distribution of true error rates are constant for every choice of the number of design examples nd. the results in figure clearly show that the sizes of the cis obtained using repeated cross-validation are not constant but decreasing with increasing values of nt. this is similar to what can be observed using rrs in michiels et al. where the number of design examples is varied. as shown in figure the estimated ci stabilizes as the size of the test set becomes large. the bias in the ci will be eliminated as the size of the test set becomes very large. however, for the usually limited set of examples available in most real world problems, the bias is too large to be neglected. repeated independent design and test limited testing of each classifier is not expected to be the sole cause of undesirable bias in the rss estimate. bias may also be attributed to three different statistical dependencies between data sets caused by the repeated design and testing performed using the bag of limited examples available: 1) each pair of design and test sets are dependent. once the design set has been selected, the remaining examples become the test set deterministically. 2) the design sets are inter-dependent. given information about the samples in a first design set, a lot of information is gained about the possible samples that may occur in the next design set obtained by means of resampling. 3) the test sets are also inter-dependent. given information about the samples in a first test set, information has been gained about the possible samples that may occur in the next test set. in this work we introduce a novel procedure denoted repeated independent design and test, which eliminates the first type of dependence by splitting the original data set of size n into a design bag with nd samples, and a test bag, with nt = n-nd test samples. thus, for each design a fixed number of examples nd with equal number of samples from each class are drawn with replacement form the bag of nd samples. this makes the resampling of design examples completely independent of the selection of test set examples. notably, the design sets remain inter-dependent due to the small design bag and similarly the test sets remain inter-dependent due to the finite size of the test bag. by repeatedly selecting design sets of size nd from the design bag and testing with data from the test bag, a number of error rate estimates are obtained that subsequently are used to obtain an almost unbiased estimate of the true variance σd this variance estimate can in turn be used to construct the desired ci of the distribution of true performances. a variance model for the ridt procedure analogous to the variance rhσdt associated with idealized repeated holdout experiments discussed above, the variance of the error rate estimates obtained with ridt is dependent on the finite value of nt, as well as on nt and is denoted σdt to study and reduce estimation biases caused by small sample size, we propose that for a given data set d, the ridt estimate of σdt may be approximated as σ^dt2≈α0+α1nt+α2nt.     eq.  mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacuwfdpwcgaqcamaadaaaleaacqwgkbazcqwg0badaeaacqaiyagmaagcdaqadaqaamaalaaabagaegymaedabagaemota40aasbaasqaaiabdsfaubqabaaaaogaeiilawyaasaaaeaacqaixaqmaeaacqwgobgtdawgaawcbagaemidaqhabeaaaagccqggsaalcqwgebaraiaawicacaglpaaacqghijyucqwfxoqydawgaawcbagaegimaadabeaakiabcicaoiabdseaejabcmcapiabgucarmaalaaabagae8xsde2aasbaasqaaiabigdaxaqabagccqggoaakcqwgebarcqggpaqkaeaacqwgobgtdawgaawcbagaemivaqfabeaaaagccqghrawkdawcaaqaaiab=f7ahnaabaaaleaacqaiyagmaeqaaogaeiikagiaemiraqkaeiykakcabagaemota40aasbaasqaaiabdsha0bqabaaaaogaeiola4iaaczcaiaaxmaacqqgfbqrcqqgxbqccqqguaglcqqggaaicqqgoaakcqqgxaqmcqqgpaqkaaa@607e@ this equation involves first order linear approximations of the biases introduced by the finite values of nt and nt. for very large values of nt and nt, the estimate reduces to an unbiased estimate of σd hence the first coefficient α should be an unbiased estimate of σd i.e. <α >d = σd where <>d denotes the expectation operator. notably, the model treats the size of the test bag nt in a similar way as the size of the test set nt, but ignores effects due to size of the design bag nd. by evaluating classifications using nb design sets, varying the test bag sizes nt and sizes of tests sets nt for each value of nb, it is possible to estimate the data set dependent coefficients α α and α in eq. by multivariate least squares fitting. in this process one constraint is used that ensures the natural inequality α ≥ with access to the fitted coefficient α one has obtained an unbiased estimate of the desired quantity σd we performed simulations using samples generated from two 2-dimensional normal distributions with mean values and covariance matrices estimated from real micorarray gene expression data. the aim was to validate our model, and to demonstrate its potential for elimination of the bias caused by small sample size. since the two features in both distributions are correlated, the simulation takes the dependence that may exist between features in real data sets into account. one should also note that we have deliberately chosen to use a classifier that does not contain a feature selection step to avoid additional complexity. thus, the problems and solution discussed in this paper are equally relevant also for classifier using feature selection. we have also chosen the strategy to evaluate the performance for each class separately, since it does not require knowledge about the probabilities of observing examples from class or class to emphasize that we focused the analysis on class an extra subindex was introduced when denoting the quantities of interest. thus, the true mean value and variance associated with class are denoted md and σd respectively, and corresponding quantities associated with testing using finite data sets are denoted mdt and σdt in the ridt procedure the design bags had equal number of samples from both classes and the number of samples drawn with replacement was the same as the size of the design bag, i.e. nd = nd. we also made the assumption that mdt is an unbiased estimate of md which was verified. the mean values of the estimated md and σd and the corresponding two-sided 95% cis for eight different values of nt, are presented in figure for nd = nd =. the true values md and σd obtained by testing independently designed classifiers using test samples each, are also indicated. apparently, unbiased estimates of md and σd are obtained. we observed that the reduction of the small sample bias yields accurate estimates of σd on average. one should note that in general the estimates contain contributions of higher order terms. however, as the results indicate, the biases caused by these higher order terms may be quite small for commonly used sizes of data sets. for the data set sizes commonly used it appears that it is possible to obtain practically unbiased estimates of md and σd with the ridt method. variance for independent data in the special case of truly independent data, i.e. when each pair of design and test sets are drawn from the underlying true distribution of samples instead of from a finite bag of examples, we derived an exact analytical equation for σdt: σdt2=σd2+md−σd2nt.     eq.  mathtype@mtef@5@5@+=feaafiart1ev1aaatcvaufkttlearuwrp9mdh5mbpbiqv92aaexatlxbi9gbaebbnrfifhhdyfgasaach8aky=wiffydh8gipec8eeeu0xxdbba9frfj0=oqffea0dxdd9vqai=hguq8kuc9pgc9s8qqaq=dirpe0xb9q8qilsfr0=vr0=vr0dc8meaabaqaciaacagaaeqabaqabegadaaakeaaiigacqwfdpwcdaqhaawcbagaemizaqmaemidaqhabagaegomaidaaogaeyypa0jae83wdm3aa0baasqaaiabdsgakbqaaggaaiab+jdayaaakiabgucarmaalaaabagaemyba02aasbaasqaaiabdsgakbqabagccqggoaakcqaixaqmcqghsislcqwgtbqbdawgaawcbagaemizaqgabeaakiabcmcapiabgkhitiab=n8aznaadaaaleaacqwgkbazaeaacqaiyagmaaaakeaacqwgobgtdawgaawcbagaemidaqhabeaaaagccqgguaglcawljagaaczcaiabbweafjabbghaxjabb6cauiabbccagiabbicaoiabbkdayiabbmcapaaa@52cc@ this equation shows how the observed variance depends on the number of test samples nt as well as on md and σd it can also be noted that for very large values of nt, eq. reduces to the same form as eq.. fukunaga and hayes have previously published an approximation of eq. one advantage with the exact equation is that it can be used to show that the second term always is larger than zero and that σdt is always larger than σd thus, if σdt would be an approximation of the variance, the resulting ci would always be conservative. in order to empirically validate eq., simulations were performed using samples drawn from two 8-dimensional normal distributions. we determined σdt for different values of nt and nd. each value of σdt was obtained using a histogram of independent point estimates. for comparison, separate and independent high accuracy point estimates of md and σd were computed, each using test samples for varied design set sizes nd. the true observed variances in figure were then obtained from eq.. this demonstrates excellent agreement between the theory and simulations. discussion rrs has been proposed as a practical method for estimation of the distribution of error rates obtained when a specified number of data samples are used for design. however, we have demonstrated that the variance estimate of the performance for classifiers designed and tested in a similar way results in a variance estimate that is highly dependent on the number of samples used for test. a qualitatively analogous effect should occur also in rrs, which is equivalent to using all remaining examples for test in our experiment. consequently, highly conservative estimates of the variance are obtained with repeated testing methods when the number of examples used for test is small. in practice the variance estimates have a bias of unknown magnitude, due to the complex statistical dependence between design and test sets. therefore, it is important to stress that the confidence interval in rrs cannot be used to draw any conclusions about whether it is likely that a classifier performs better than chance. an example of this inappropriate use of rrs can be found in where the possibilities to predict cancer outcome based on microarray gene expression patterns were investigated in several data sets. perhaps even more importantly, a large bias in the variance estimate of interest is not a unique feature of the rss procedure but is expected to be found in all other suggested resampling procedures for performance estimation. for example, estimating the variance of a q-fold cv performance estimate as suggested by mclachlan et al. seems attractive but we are not aware of any theoretical or numerical proofs that those and similar methods result in unbiased estimates of the variance σd2of interest. on the contrary, the proof of equation in our manuscript clearly shows that even if it would be possible to draw infinitely many independent design and test sets from the true distribution of samples, the resulting variance estimate of interest is heavily biased when the test sets are small. there are a number of features of the ridt method that have implications for the use of the method. first, the ridt performance estimates rely on a split of the data set into two separate parts, one used for repeated design, the other for repeated tests, which is not current practice in cross-validation and bootstrapping and might be interpreted as inefficient use of the few samples available. we view this as a price that has to be paid in order to provide unbiased estimation of the variance of interest which can not be obtained with other methods. second, although normal distributions were used in the computer simulations performed to generate the results presented here, the elimination of finite sample effects using eq. does not assume normally distributed data, but can use data from any type of distribution. third, even though eq. does not depend on nd, it is possible to reduce small sample effects and provide unbiased estimates for the specific problem considered here. the general applicability of this observation awaits further studies but eq. can easily be extended to include a fourth term that is explicitly dependent on nd, see eq.. one possible explanation for the small influence of nd in the ridt method used here is that the design sets are drawn with replacement from the design bag, a procedure that closely reflects what happens in reality. although not yet explored in detail, there are several explanations for the small bias in σd observed in figure when nt ≤ 50: 1) we are ignoring higher order terms in the approximations. 2) we do not try to eliminate effects caused by a finite value of nd. 3) we do not take any small sample effects into account at all when estimating md. 4) when using replacement, we employ on average only % unique samples in each design. notably, the number of design examples nd remains fixed and, as discussed above, there is no contribution to the bias due to nd being small. we find that the variance of the inter-design set variance estimate σd is relatively large and increases with decreasing value of nt this means that the estimate of σd for a particular data set is unbiased, but that it may be associated with a large uncertainty especially if the size of the data set is small. therefore, it is difficult to directly use the unbiased estimates of md and σd to construct a ci. thus it appears that even though we can compensate for biases caused by small sample size, the resampling approach has not provided a method that is practically useful in its present form. therefore the only rigorous option for estimation of classifier performance that we know of is the classical hold out test combined with a bayesian credibility interval, even though this interval is overly conservative and provides very wide intervals.  one major suggestion from the results of this paper is that previously introduced resampling and cross-validation methods for performance estimation using small sample sets are expected to result in large biases in their estimates of the inter design set variances. consequently such biased variance estimates lead to inappropriate confidence intervals for the performance of a chosen classifier. in addition this paper describes a method that is capable of eliminating this bias for a new resampling method also introduced here. finally we would like to point out that although this paper provides important experimental and theoretical results, the large variability in the unbiased variance estimate obtained still leaves us one step away from a practically useful solution for small sample based estimation of confidence intervals using resampling. we therefore also hope that this work will inspire others to consider how to convert the unbiased but highly variable variance estimate of our and similar future procedures into a valid confidence interval. 
our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. we also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. however, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets. therefore different methods for small sample performance estimation such as a recently proposed procedure called repeated random sampling is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. in practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. however, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. here we explore such biases and develop a refined algorithm called repeated independent design and test. we show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. a suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. for the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set.
 supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. in practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. a suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. however, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. therefore different methods for small sample performance estimation such as a recently proposed procedure called repeated random sampling is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. here we explore such biases and develop a refined algorithm called repeated independent design and test.  our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. we also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. for the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set.  we show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. however, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets. 