estimating the proportion of true null hypotheses with application to dna microarray data summary we consider the problem of estimating the proportion of true null hypotheses 0 i n a multiple hypothesis set up the tests are based on observed p values we first review published estimators based on the estimator that was suggested by schweder and spjotvoll then we derive new estimators based on nonparametric maximum likelihood estimation of the p value density restricting to decreasing and convex decreasing densities the estimators of 0 are all derived under the assumption of independent test statistics their performance under dependence is investigated in a simulation study we find that the estimators are relatively robust with respect to the assumption of independence and work well also for test statistics with moderate dependence

adapting to unknown sparsity by controlling the false discovery rate we attempt to recover an n dimensional vector observed in white noise where n is large and the vector is known to be sparse but the degree of sparsity is unknown we consider three different ways of defining sparsity of a vector using the fraction of nonzero terms imposing power law decay bounds on the ordered entries and controlling the lp norm for p small we obtain a procedure which is asymptotically minimax for l r loss simultaneously throughout a range of such sparsity classes the optimal procedure is a data adaptive thresholding scheme driven by control of the false discovery rate fdr fdr control is a relatively recent innovation in simultaneous testing ensuring that at most a certain fraction of the rejected null hypotheses will correspond to false rejections in our treatment the fdr control parameter qn also plays a determining role in asymptotic minimaxity if q lim qn 0 1 2 and also qn log n we get sharp asymptotic minimaxity simultaneously over a wide range of sparse parameter spaces and loss functions on the other hand q lim qn 1 2 1 forces the risk to exceed the minimax risk by a factor growing with q to our knowledge this relation between ideas in simultaneous inference and asymptotic decision theory is new our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors these new rules impose complexity penalization of the form 2 log potential model size actual model size we exhibit a close connection with fdr controlling procedures under stringent control of the false discovery rate

size power and false discovery rates modern scientific technology has provided a new class of large scale simultaneous inference problems with thousands of hypothesis tests to consider at the same time microarrays epitomize this type of technology but similar situations arise in proteomics spectroscopy imaging and social science surveys this paper uses false discovery rate methods to carry out both size and power calculations on large scale problems a simple empirical bayes approach allows the false discovery rate fdr analysis to proceed with a minimum of frequentist or bayesian modeling assumptions closed form accuracy formulas are derived for estimated false discovery rates and used to compare different methodologies local or tail area fdr s theoretical permutation or empirical null hypothesis estimates two microarray data sets as well as simulations are used to evaluate the methodology the power diagnostics showing why nonnull cases might easily fail to appear on a list of significant discoveries

false discovery rate control with groups in the context of large scale multiple hypothesis testing the hypotheses often possess certain group structures based on additional information such as gene ontology in gene expression data and phenotypes in genome wide association studies it is hence desirable to incorporate such information when dealing with multiplicity problems to increase statistical power in this article we demonstrate the benefit of considering group structure by presenting a p value weighting procedure which utilizes the relative importance of each group while controlling the false discovery rate under weak conditions the procedure is easy to implement and shown to be more powerful than the classical benjamini hochberg procedure in both theoretical and simulation studies by estimating the proportion of true null hypotheses the data driven procedure controls the false discovery rate asymptotically our analysis on one breast cancer dataset confirms that the procedure performs favorably compared with the classical method

to how many simultaneous hypothesis tests can normal student s t or bootstrap calibration be applied in the analysis of microarray data and in some other contemporary statistical problems it is not uncommon to apply hypothesis tests in a highly simultaneous way the number n say of tests used can be much larger than the sample sizes n to which the tests are applied yet we wish to calibrate the tests so that the overall level of the simultaneous test is accurate often the sampling distribution is quite different for each test so there may not be an opportunity to combine data across samples in this setting how large can n be as a function of n before level accuracy becomes poor here we answer this question in cases where the statistic under test is of student s t type we show that if either the normal or student t distribution is used for calibration then the level of the simultaneous test is accurate provided that log n increases at a strictly slower rate than n 1 3 as n diverges on the other hand if bootstrap methods are used for calibration then we may choose log n almost as large as

multiple testing for neuroimaging via hidden markov random field traditional voxel level multiple testing procedures in neuroimaging mostly p value based often ignore the spatial correlations among neighboring voxels and thus suffer from substantial loss of power we extend the local significance index based procedure originally developed for the hidden markov chain models which aims to minimize the false nondiscovery rate subject to a constraint on the false discovery rate to three dimensional neuroimaging data using a hidden markov random field model a generalized expectation maximization algorithm for maximizing the penalized likelihood is proposed for estimating the model parameters extensive simulations show that the proposed approach is more powerful than conventional false discovery rate procedures we apply the method to the comparison between mild cognitive impairment a disease status with increased risk of developing alzheimer s or another dementia and normal controls in the fdg pet imaging study of the alzheimer s disease neuroimaging initiative

simultaneous testing of grouped hypotheses finding needles in multiple haystacks in large scale multiple testing problems data are often collected from heterogeneous sources and hypotheses form into groups that exhibit different characteristics conventional approaches including the pooled and separate analyses fail to efficiently utilize the external grouping information we develop a compound decision theoretic framework for testing grouped hypotheses and introduce an oracle procedure that minimizes the false nondiscovery rate subject to a constraint on the false discovery rate it is shown that both the pooled and separate analyses can be uniformly improved by the oracle procedure we then propose a data driven procedure that is shown to be asymptotically optimal simulation studies show that our procedures enjoy superior performance and yield the most accurate results in comparison with both the pooled and separate procedures a real data example with grouped hypotheses is studied in detail using different methods both theoretical and numerical results demonstrate that exploit

exact calculations for false discovery proportion with application to least favorable configurations in a context of multiple hypothesis testing we provide several new exact calculations related to the false discovery proportion fdp of step up and step down procedures for step up procedures we show that the number of erroneous rejections conditionally on the rejection number is simply a binomial variable which leads to explicit computations of the c d f the s th moment and the mean of the fdp the latter corresponding to the false discovery rate fdr for step down procedures we derive what is to our knowledge the first explicit formula for the fdr valid for any alternative c d f of the p values we also derive explicit computations of the power for both step up and step down procedures these formulas are explicit in the sense that they only involve the parameters of the model and the c d f of the order statistics of i i d uniform variables the p values are assumed either independent or coming from an equicorrelated multivariate normal model and an additional mixture model for the true false hypotheses is used this new approach is used to investigate new results which are of interest in their own right related to least most favorable configurations for the fdr and the variance of the fdp

on the benjamini hochberg method we investigate the properties of the benjamini hochberg method for multiple testing and of a variant of storey s generalization of it extending and complementing the asymptotic and exact results available in the literature results are obtained under two different sets of assumptions and include asymptotic and exact expressions and bounds for the proportion of rejections the proportion of incorrect rejections out of all rejections and two other proportions used to quantify the efficacy of the method

multiple testing via fdrl for large scale imaging data the multiple testing procedure plays an important role in detecting the presence of spatial signals for large scale imaging data typically the spatial signals are sparse but clustered this paper provides empirical evidence that for a range of commonly used control levels the conventional fdr procedure can lack the ability to detect statistical significance even if the p values under the true null hypotheses are independent and uniformly distributed more generally ignoring the neighboring information of spatially structured data will tend to diminish the detection effectiveness of the fdr procedure this paper first introduces a scalar quantity to characterize the extent to which the lack of identification phenomenon lip of the fdr procedure occurs second we propose a new multiple comparison procedure called fdrl to accommodate the spatial information of neighboring p values via a local aggregation of p values theoretical properties of the fdrl procedure are investigated under weak dependence of p values it is shown that the fdrl procedure alleviates the lip of the fdr procedure thus substantially facilitating the selection of more stringent control levels simulation evaluations indicate that the fdrl procedure improves the detection sensitivity of the fdr procedure with little loss in detection specificity the computational simplicity and detection effectiveness of the fdrl procedure are illustrated through a real brain fmri dataset

parameterized maximum path coloring we study the well known max path coloring problem from a parameterized point of view focusing on trees and low treewidth networks we observe the existence of a variety of reasonable parameters for the problem such as the maximum degree and treewidth of the network graph the number of available colors and the number of requests one seeks to satisfy or reject in an effort to understand the impact of each of these parameters on the problem s complexity we study various parameterized versions of the problem deriving fixed parameter tractability and hardness results both for undirected and bi directed graphs

optimal rates of convergence for estimating the null density and proportion of nonnull effects in large scale multiple testing an important estimation problem that is closely related to large scale multiple testing is that of estimating the null density and the proportion of nonnull effects a few estimators have been introduced in the literature however several important problems including the evaluation of the minimax rate of convergence and the construction of rate optimal estimators remain open in this paper we consider optimal estimation of the null density and the proportion of nonnull effects both minimax lower and upper bounds are derived the lower bound is established by a two point testing argument where at the core is the novel construction of two least favorable marginal densities f 1 and f 2 the density f 1 is heavy tailed both in the spatial and frequency domains and f 2 is a perturbation of f 1 such that the characteristic functions associated with f 1 and f 2 match each other in low frequencies the minimax upper bound is obtained by constructing estimators which rely on the empirical characteristic function and fourier analysis the estimator is shown to be minimax rate optimal compared to existing methods in the literature the proposed procedure not only provides more precise estimates of the null density and the proportion of the nonnull effects but also yields more accurate results when used inside some multiple testing procedures which aim at controlling the false discovery rate fdr the procedure is easy to implement and numerical results are given

false discovery control for random fields this article extends false discovery rates to random fields for which there are uncountably many hypothesis tests we develop a method for finding regions in the field s domain where there is a significant signal while controlling either the proportion of area or the proportion of clusters in which false rejections occur the method produces confidence envelopes for the proportion of false discoveries as a function of the rejection threshold from the confidence envelopes we derive threshold procedures to control either the mean or the specified tail probabilities of the false discovery proportion an essential ingredient of this construnction is a new algorithm to compute a confidence superset for the set of all true null locations we demonstrate our method with applications to scan statistics and functional neuroimaging

stepup procedures controlling generalized fwer and generalized fdr in many applications of multiple hypothesis testing where more than one false rejection can be tolerated procedures controlling error rates measuring at least k false rejections instead of at least one for some fixed k 1 can potentially increase the ability of a procedure to detect false null hypotheses the k fwer a generalized version of the usual familywise error rate fwer is such an error rate that has recently been introduced in the literature and procedures controlling it have been proposed a further generalization of a result on the k fwer is provided in this article in addition an alternative and less conservative notion of error rate the k fdr is introduced in the same spirit as the k fwer by generalizing the usual false discovery rate fdr a k fwer procedure is constructed given any set of increasing constants by utilizing the kth order joint null distributions of the p values without assuming any specific form of dependence among all the p values procedures controlling the k fdr are also developed by using the kth order joint null distributions of the p values first assuming that the sets of null and nonnull p values are mutually independent or they are jointly positively dependent in the sense of being multivariate totally positive of order two mtp2 and then discarding that assumption about the overall dependence among the p values

innovated higher criticism for detecting sparse signals in correlated noise higher criticism is a method for detecting signals that are both sparse and weak although first proposed in cases where the noise variables are independent higher criticism also has reasonable performance in settings where those variables are correlated in this paper we show that by exploiting the nature of the correlation performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer indeed it turns out that the case of independent noise is the most difficult of all from a statistical viewpoint and that more accurate signal detection for a given level of signal sparsity and strength can be obtained when correlation is present we characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary the boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is toeplitz

permutation based simultaneous confidence bounds for the false discovery proportion when multiple hypotheses are tested interest is often in ensuring that the proportion of false discoveries fdp is small with high confidence in this paper confidence upper bounds for the fdp are constructed which are simultaneous over all rejection cut offs in particular this allows the user to select a set of hypotheses post hoc such that the fdp lies below some constant with high confidence our method uses permutations to account for the dependence structure in the data so far only meinshausen provided an exact permutation based and computationally feasible method for simultaneous fdp bounds we provide an exact method which uniformly improves this procedure further we provide a generalization of this method it lets the user select the shape of the simultaneous confidence bounds this gives the user more freedom in determining the power properties of the method interestingly several existing permutation methods such as significance analysis of microarrays sam and westfall and young s maxt method are obtained as special cases

rolling rhythms in front crawl swimming with six beat kick abstract the purpose of this study was to establish the rhythm characteristics of skilled front crawl swimmers using a six beat kick these included the amplitudes of the first three fourier harmonics h1 h2 h3 and their percent contributions to power contained in the angular displacement signals of the shoulders hips knees and ankles with respect to the longitudinal axis in line with the swimming direction three dimensional video data of seven national international level swimmers were collected during simulated 200 m front crawl races in which swimmers maintained six beat kicking patterns swimmers differed in all variables but had small variability across the four 50 m laps modest changes occurred during the 200 m with the exception of shoulder roll which remained constant and was represented almost entirely by a single sinusoid h1 changes across laps reached significance for swimming speed stroke rate hip roll and h3 wave velocity between the knee and ankle a h3 body wave of moderate and increasing velocity travelled caudally from hip to ankle in the light of existing knowledge of aquatic locomotion this was compatible with the goal of generating propulsion in an efficient manner

the effect of correlation in false discovery rate estimation the objective of this paper is to quantify the effect of correlation in false discovery rate analysis specifically we derive approximations for the mean variance distribution and quantiles of the standard false discovery rate estimator for arbitrarily correlated data this is achieved using a negative binomial model for the number of false discoveries where the parameters are found empirically from the data we show that correlation may increase the bias and variance of the estimator substantially with respect to the independent case and that in some cases such as an exchangeable correlation structure the estimator fails to be consistent as the number of tests becomes large

hierarchical false discovery rate controlling methodology we discuss methodology for controlling the false discovery rate fdr in complex large scale studies that involve testing multiple families of hypotheses the tested hypotheses are arranged in a tree of disjoint subfamilies and the subfamilies of hypotheses are hierarchically tested by the benjamini and hochberg fdr controlling bh procedure we derive an approximation for the multiple family fdr for independently distributed test statistics q the level at which the bh procedure is applied times the number of families tested plus the number of discoveries divided by the number of discoveries plus 1 we provide a universal bound for the fdr of the discoveries in the new hierarchical testing approach 2 1 44 q and demonstrate in simulations that when the data has an hierarchical structure the new testing approach can be considerably more powerful than the bh procedure

multiple testing of composite null hypotheses in heteroscedastic models in large scale studies the true effect sizes often range continuously from zero to small to large and are observed with heteroscedastic errors in practical situations where the failure to reject small deviations from the null is inconsequential specifying an indifference region or forming composite null hypotheses can greatly reduce the number of unimportant discoveries in multiple testing the heteroscedasticity issue poses new challenges for multiple testing with composite nulls in particular the conventional framework in multiple testing which involves rescaling or standardization is likely to distort the scientific question we propose the concept of a composite null distribution for heteroscedastic models and develop an optimal testing procedure that minimizes the false nondiscovery rate subject to a constraint on the false discovery rate the proposed approach is different from conventional methods in that the effect size statistical significance and multiplicity issues are addressed in

controlling the familywise error rate with plug in estimator for the proportion of true null hypotheses summary estimation of the number or proportion of true null hypotheses in multiple testing problems has become an interesting area of research the first important work in this field was performed by schweder and spjotvoll among others they proposed to use plug in estimates for the proportion of true null hypotheses in multiple test procedures to improve the power we investigate the problem of controlling the familywise error rate fwer when such estimators are used as plug in estimators in single step or step down multiple test procedures first we investigate the case of independent p values under the null hypotheses and show that a suitable choice of plug in estimates leads to control of fwer in single step procedures we also investigate the power and study the asymptotic behaviour of the number of false rejections although step down procedures are more difficult to handle we briefly consider a possible solution to this problem anyhow plug in step down procedures are not recommended here for dependent p values we derive a condition for asymptotic control of fwer and provide some simulations with respect to fwer and power for various models and hypotheses

a direct approach to sparse discriminant analysis in ultra high dimensions summary sparse discriminant methods based on independence rules such as the nearest shrunken centroids classifier tibshirani et al 2002 and features annealed independence rules fan fan 2008 have been proposed as computationally attractive tools for feature selection and classification with high dimensional data a fundamental drawback of these rules is that they ignore correlations among features and thus could produce misleading feature selection and inferior classification we propose a new procedure for sparse discriminant analysis motivated by the least squares formulation of linear discriminant analysis to demonstrate our proposal we study the numerical and theoretical properties of discriminant analysis constructed via lasso penalized least squares our theory shows that the method proposed can consistently identify the subset of discriminative features contributing to the bayes rule and at the same time consistently estimate the bayes classification direction even when the dimension can grow faster than any polynomial order of the sample size the theory allows for general dependence among features simulated and real data examples show that lassoed discriminant analysis compares favourably with other popular sparse discriminant proposals

variance of the number of false discoveries summary in high throughput genomic work a very large number d of hypotheses are tested based on n d data samples the large number of tests necessitates an adjustment for false discoveries in which a true null hypothesis was rejected the expected number of false discoveries is easy to obtain dependences between the hypothesis tests greatly affect the variance of the number of false discoveries assuming that the tests are independent gives an inadequate variance formula the paper presents a variance formula that takes account of the correlations between test statistics that formula involves o d2 correlations and so a naive implementation has cost o nd2 a method based on sampling pairs of tests allows the variance to be approximated at a cost that is independent of d

plots of p values to evaluate many tests simultaneously summary when a large number of tests are made possibly on the same data it is proposed to base a simultaneous evaluation of all the tests on a plot of cumulative p values using the observed significance probabilities the points corresponding to true null hypotheses should form a straight line while those for false null hypotheses should deviate from this line the line may be used to estimate the number of true hypotheses the properties of the method are studied in some detail for the problems of comparing all pairs of means in a one way layout testing all correlation coefficients in a large correlation matrix and the evaluation of all 2 x 2 subtables in a contingency table the plot is illustrated on real data

a parametric meta analysis in a meta analysis we assemble a sample of independent nonidentically distributed p values the fisher s combination procedure provides a chi squared test of whether the p values were sampled from the null uniform distribution after rejecting the null uniform hypothesis we are faced with the problem of how to combine the assembled p values we first derive a distribution for the p values the distribution is parameterized by the standardized mean difference smd and the sample size it includes the uniform as a special case the maximum likelihood estimate mle of the smd can then be obtained from the independent nonidentically distributed p values the mle can be interpreted as a weighted average of the study specific estimate of the effect size with a shrinkage the method is broadly applicable to p values obtained in the maximum likelihood framework simulation studies show that our method can effectively estimate the effect size with as few as 6 p values in the meta analyses we also present a bayes estimator for smd and a method to account for publication bias we demonstrate our methods on several meta analyses that assess the potential benefits of citicoline for patients with memory disorders or patients recovering from ischemic stroke

empirical bayes analysis of a microarray experiment microarrays are a novel technology that facilitates the simultaneous measurement of thousands of gene expression levels a typical microarray experiment can produce millions of data points raising serious problems of data reduction and simultaneous inference we consider one such experiment in which oligonucleotide arrays were employed to assess the genetic effects of ionizing radiation on seven thousand human genes a simple nonparametric empirical bayes model is introduced which is used to guide the efficient reduction of the data to a single summary statistic per gene and also to make simultaneous inferences concerning which genes were affected by the radiation although our focus is on one specific experiment the proposed methods can be applied quite generally the empirical bayes inferences are closely related to the frequentist false discovery rate fdr criterion

signal classification for the integrative analysis of multiple sequences of large scale multiple tests the integrative analysis of multiple data sets is becoming increasingly important in many fields of research when the same features are studied in several independent experiments it can often be useful to analyse jointly the multiple sequences of multiple tests that result it is frequently necessary to classify each feature into one of several categories depending on the null and non null configuration of its corresponding test statistics the paper studies this signal classification problem motivated by a range of applications in large scale genomics two new types of misclassification rate are introduced and two oracle procedures are developed to control each type while also achieving the largest expected number of correct classifications corresponding data driven procedures are also proposed proved to be asymptotically valid and optimal under certain conditions and shown in numerical experiments to be nearly as powerful as the oracle procedures in an application to psychiatric genetics the procedures proposed are used to discover genetic variants that may affect both bipolar disorder and schizophrenia as well as variants that may help to distinguish between these conditions

on the performance of fdr control constraints and a partial solution the false discovery rate fdr paradigm aims to attain certain control on type i errors with relatively high power for multiple hypothesis testing the benjamini hochberg bh procedure is a wellknown fdr controlling procedure under a random effects model we show that in general unlike the fdr the positive fdr pfdr of the bh procedure cannot be controlled at an arbitrarily low level due to the limited evidence provided by the observations to separate false and true nulls this results in a criticality phenomenon which is characterized by a transition of the procedure s power from being positive to asymptotically 0 without any reduction in the pfdr once the target fdr control level is below a positive critical value to address the constraints on the power and pfdr control imposed by the criticality phenomenon we propose a procedure which applies bh type procedures at multiple locations in the domain of p values both analysis and simulations show that the proposed procedure can attain substantially improved power and pfdr control

proportion of non zero normal means universal oracle equivalences and uniformly consistent estimators d0 we propose a general approach to construct the universal oracle equivalence of the proportion the construction is based on the underlying characteristic function the oracle equivalence reduces the problem of estimating the proportion to the problem of estimating the oracle which is relatively easier to handle in fact the oracle equivalence naturally yields a family of estimators for the proportion which are consistent under mild conditions uniformly across a wide class of parameters the approach compares favourably with recent works by meinshausen and rice and genovese and wasserman in particular the consistency is proved for an unprecedentedly broad class of situations the class is almost the largest that can be hoped for without further constraints on the model we also discuss various extensions of the approach report results on simulation experiments and make connections between the approach and several recent procedures in large scale multiple testing including the false discovery rate approach and the local false discovery rate approach

estimation of false discovery rates for wavelet denoised statistical parametric maps correction for multiple comparisons in neuroimaging data is an important area of research recently wavelet based methods have gained popularity and have been reported to achieve better sensitivity compared to spatial domain methods however these techniques produce smoothed statistical maps which are difficult to interpret the generated maps have to be thresholded again in the spatial domain to delineate active from inactive regions the selection of a proper threshold satisfying the required error rate control is not straightforward in this paper a framework is proposed for thresholding wavelet denoised maps in which a rejection region is fixed and the achieved false discovery rate fdr is estimated this approach provides a meaningful strategy to choose thresholds for wavelet denoised statistical parametric maps spms two fdr estimation algorithms were used to assess the achieved error rate control when thresholding wavelet filtered spms at various rejection regions their performance was evaluated using both simulated and resting fmri data the proposed framework was also applied on in vivo data

the limiting behavior of a modified maximal symmetric 2s spacing with applications this paper presents a solution to an open problem in astrophysics namely that of estimating nonparametrically the strength of the pulsed signal in a series of high energy photon arrival times the newly proposed estimator based on a modified maximal symmetric 2s spacing is shown to be strongly consistent and asymptotically normally distributed and a monte carlo study shows that its small and moderate sample behavior is very satisfactory additionally new results regarding the weak and strong limiting behavior of modified maximal 2 s spacings are derived

on false discovery control under dependence a popular framework for false discovery control is the random effects model in which the null hypotheses are assumed to be independent this paper generalizes the random effects model to a conditional dependence model which allows dependence between null hypotheses the dependence can be useful to characterize the spatial structure of the null hypotheses asymptotic properties of false discovery proportions and numbers of rejected hypotheses are explored and a large sample distributional theory is obtained

multiple hypotheses testing and expected number of type i errors the performance of multiple test procedures with respect to error control is an old issue assuming that all hypotheses are true we investigate the behavior of the expected number of type i errors ene as a characteristic of certain multiple tests controlling the familywise error rate fwer or the false discovery rate fdr at a prespecified level we derive explicit formulas for the distribution of the number of false rejections as well as for the ene for single step step down and step up procedures based on independent p values moreover we determine the corresponding asymptotic distributions of the number of false rejections as well as explicit formulae for the ene if the number of hypotheses tends to infinity in case of fwer control we mostly obtain poisson distributions and in one case a geometric distribution as limiting distributions in case of fdr control we obtain limiting distributions which are apparently not named in the literature surprisingly the ene is bounded by a small number regardless of the number of hypotheses under consideration finally it turns out that in case of dependent test statistics the ene behaves completely differently compared to the case of independent test statistics

simultaneous control of all false discovery proportions in large scale multiple hypothesis testing closed testing procedures are classically used for familywise error rate fwer control but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion fdp in all subsets of the hypotheses in this paper we investigate the special case of closed testing with simes local tests we construct a novel fast and exact shortcut which we use to investigate the power of this method when the number of hypotheses goes to infinity we show that if a minimal amount of signal is present the average power to detect false hypotheses at any desired fdp level does not vanish additionally we show that the confidence bounds for fdp are consistent estimators for the true fdp for every non vanishing subset for the case of a finite number of hypotheses we show connections between simes based closed testing and the procedure of benjamini and hochberg

α investing a procedure for sequential control of expected false discoveries summary investing is an adaptive sequential methodology that encompasses a large family of procedures for testing multiple hypotheses all control mfdr which is the ratio of the expected number of false rejections to the expected number of rejections mfdr is a weaker criterion than the false discovery rate which is the expected value of the ratio we compensate for this weakness by showing that investing controls mfdr at every rejected hypothesis investing resembles spending that is used in sequential trials but it has a key difference when a test rejects a null hypothesis investing earns additional probability towards subsequent tests investing hence allows us to incorporate domain knowledge into the testing procedure and to improve the power of the tests in this way investing enables the statistician to design a testing procedure for a specific problem while guaranteeing control of mfdr

properties of higher criticism under strong dependence the problem of signal detection using sparse faint information is closely related to a variety of contemporary statistical problems including the control of false discovery rate and classification using very high dimensional data each problem can be solved by conducting a large number of simultaneous hypothesis tests the properties of which are readily accessed under the assumption of independence in this paper we address the case of dependent data in the context of higher criticism methods for signal detection short range dependence has no first order impact on performance but the situation changes dramatically under strong dependence there although higher criticism can continue to perform well it can be bettered using methods based on differences of signal values or on the maximum of the data the relatively inferior performance of higher criticism in such cases can be explained in terms of the fact that under strong dependence the higher criticism statistic behaves as though the data were partitioned into very large blocks with all but a single representative of each block being eliminated from the dataset

adaptive linear step up procedures that control the false discovery rate we provide a new two stage procedure in which the linear step up procedure is used in stage one to estimate mo providing a new level q which is used in the linear step up procedure in the second stage we prove that a general form of the two stage procedure controls the false discovery rate at the desired level q this framework enables us to study analytically the properties of other procedures that exist in the literature a simulation study is presented that shows that two stage adaptive procedures improve in power over the original procedure mainly because they provide tighter control of the false discovery rate we further study the performance of the current suggestions some variations of the procedures and previous suggestions in the case where the test statistics are positively dependent a case for which the original procedure controls the false discovery rate in the setting studied here the newly proposed two stage procedure is the only one that controls the false discovery rate the procedures are illustrated with two examples of biological importance

robustness and accuracy of methods for high dimensional data analysis based on student s t statistic summary student s t statistic is finding applications today that were never envisaged when it was introduced more than a century ago many of these applications rely on properties e g robustness against heavy tailed sampling distributions that were not explicitly considered until relatively recently we explore these features of the t statistic in the context of its application to very high dimensional problems including feature selection and ranking the simultaneous testing of many different hypotheses and sparse high dimensional signal detection robustness properties of the t ratio are highlighted and it is established that those properties are preserved under applications of the bootstrap in particular bootstrap methods correct for skewness and therefore lead to second order accuracy even in the extreme tails indeed it is shown that the bootstrap and also the more popular but less accurate t distribution and normal approximations are more effective in the tails than towards the middle of the distribution these properties motivate new methods e g bootstrap based techniques for signal detection that confine attention to the significant tail of a statistic

some results on false discovery rate in stepwise multiple testing procedures the concept of false discovery rate fdr has been receiving increasing attention by researchers in multiple hypotheses testing this paper produces some theoretical results on the fdr in the context of stepwise multiple testing procedures with dependent test statistics it was recently shown by benjamini and yekutieli that the benjamini hochberg step up procedure controls the fdr when the test statistics are positively dependent in a certain sense this paper strengthens their work by showing that the critical values of that procedure can be used in a much more general stepwise procedure under similar positive dependency it is also shown that the fdr controlling benjamini liu step down procedure originally developed for independent test statistics works even when the test statistics are positively dependent in some sense an explicit expression for the fdr of a generalized stepwise procedure and an upper bound to the fdr of a step down procedure are obtained in terms of probability distributions of ordered components of dependent random variables before establishing the main results 1 introduction the main purpose of this paper is to present some formulas and inequalities providing theoretical insight into when and what stepwise procedures control the false discovery rate fdr at a prespecified level in multiple hypotheses testing the fdr first introduced in multiple testing by benjamini and hochberg 1995 is the expected proportion of erroneously rejected null hypotheses with v and r representing respectively the number of true null hypotheses rejected and the total number of null hypotheses rejected in a multiple testing procedure let q v r if r 0a nd 0i fr 0 then e q is the fdr of that procedure as they have argued it is an appropriate error rate to control in many practical situations particularly where a large number of null hypotheses are involved they also put forward an fdr controlling stepup procedure that is more powerful than comparable procedures controlling the traditional familywise error fwe rate when the underlying test statistics are independent benjamini and liu 1999 recently introduced a step down procedure with the same property the concept of fdr is now being used in a wide variety of applications abramovich and benjamini 1996 basford and tukey 1997 drigalenko

a stochastic process approach to false discovery control this paper extends the theory of false discovery rates fdr pioneered by benjamini and hochberg j roy statist soc ser b 57 1995 289 300 we develop a framework in which the false discovery proportion fdp the number of false rejections divided by the number of rejections is treated as a stochastic process after obtaining the limiting distribution of the process we demonstrate the validity of a class of procedures for controlling the false discovery rate the expected fdp we construct a confidence envelope for the whole fdp process from these envelopes we derive confidence thresholds for controlling the quantiles of the distribution of the fdp as well as controlling the number of false discoveries we also investigate methods for estimating the p value distribution

estimation and confidence sets for sparse normal mixtures for high dimensional statistical models researchers have begun to focus on situations which can be described as having relatively few moderately large coefficients such situations lead to some very subtle statistical problems in particular ingster and donoho and jin have considered a sparse normal means testing problem in which they described the precise demarcation or detection boundary meinshausen and rice have shown that it is even possible to estimate consistently the fraction of nonzero coordinates on a subset of the detectable region but leave unanswered the question of exactly in which parts of the detectable region consistent estimation is possible in the present paper we develop a new approach for estimating the fraction of nonzero means for problems where the nonzero means are moderately large we show that the detection region described by ingster and donoho and jin turns out to be the region where it is possible to consistently estimate the expected fraction of nonzero coordinates this theory is developed further and minimax rates of convergence are derived a procedure is constructed which attains the optimal rate of convergence in this setting furthermore the procedure also provides an honest lower bound for confidence intervals while minimizing the expected length of such an interval simulations are used to enable comparison with the work of meinshausen and rice where a procedure is given but where rates of convergence have not been discussed extensions to more general gaussian mixture models are also given

multiscale change point inference summary we introduce a new estimator the simultaneous multiscale change point estimator smuce for the change point problem in exponential family regression an unknown step function is estimated by minimizing the number of change points over the acceptance region of a multiscale test at a level the probability of overestimating the true number of change points k is controlled by the asymptotic null distribution of the multiscale test statistic further we derive exponential bounds for the probability of underestimating k by balancing these quantities will be chosen such that the probability of correctly estimating k is maximized all results are even non asymptotic for the normal case on the basis of these bounds we construct asymptotically honest confidence sets for the unknown step function and its change points at the same time we obtain exponential bounds for estimating the change point locations which for example yield the minimax rate up to a log term finally the simultaneous multiscale change point estimator achieves the optimal detection rate of vanishing signals as n even for an unbounded number of change points we illustrate how dynamic programming techniques can be employed for efficient computation of estimators and confidence regions the performance of the multiscale approach proposed is illustrated by simulations and in two cutting edge applications from genetic engineering and photoemission spectroscopy

the control of the false discovery rate in multiple testing under dependency benjamini and hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems a simple procedure was given there as an fdr controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate we prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses this condition for positive dependency is general enough to cover many problems of practical interest including the comparisons of many treatments with a single control multivariate normal test statistics with positive correlation matrix and multivariate t furthermore the test statistics may be discrete and the tested hypotheses composite without posing special difficulties for all other forms of dependency a simple conservative modification of the procedure controls the false discovery rate thus the range of problems for which a procedure with proven fdr control can be offered is greatly increased 1 1 simultaneous hypotheses testing the control of the increased type i error when testing simultaneously a family of hypotheses is a central issue in the area of multiple comparisons rarely are we interested only in whether all hypotheses are jointly true or not which is the test of the intersection null hypothesis in most applications we infer about the individual hypotheses realizing that some of the tested hypotheses are usually true we hope not all and some are not we wish to decide which ones are not true indicating statistical discoveries an important such problem is that of multiple endpoints in a clinical trial a new treatment is compared with an existing one in terms of a large number of potential benefits endpoints

covariate assisted ranking and screening for large scale two sample inference author s cai t tony sun wenguang wang weinan abstract two sample multiple testing has a wide range of applications the conventionalpractice first reduces the original observations to a vector of p values and then chooses a cutoffto adjust for multiplicity however this data reduction step could cause significant loss ofinformation and thus lead to suboptimal testing procedures we introduce a new framework fortwo sample multiple testing by incorporating a carefully constructed auxiliary variable in inferenceto improve the power a data driven multiple testing procedure is developed by employinga covariate assisted ranking and screening cars approach that optimally combines the informationfrom both the primary and the auxiliary variables the proposed cars procedureis shown to be asymptotically valid and optimal for false discovery rate control the procedureis implemented in the r package cars numerical results confirm the effectiveness of carsin false discovery rate control and show that it achieves substantial power gain over existingmethods cars is also illustrated through an application to the analysis of a satellite imagingdata set for supernova detection

a direct approach to false discovery rates summary multiple hypothesis testing involves guarding against much more complicated errors than single hypothesis testing whereas we typically control the type i error rate for a single hypothesis test a compound error rate is controlled for multiple hypothesis tests for example controlling the false discovery rate fdr traditionally involves intricate sequential p value rejection methods based on the observed data whereas a sequential p value method fixes the error rate and estimates its corresponding rejection region we propose the opposite approach we fix the rejection region and then estimate its corresponding error rate this new approach offers increased applicability accuracy and power we apply the methodology to both the positive false discovery rate pfdr and fdr and provide evidence for its benefits it is shown that pfdr is probably the quantity of interest over fdr also discussed is the calculation of the q value the pfdr analogue of the p value which eliminates the need to set the error rate beforehand as is traditionally done some simple numerical examples are presented that show that this new approach can yield an increase of over eight times in power compared with the benjamini hochberg fdr method

false discovery control with p value weighting we present a method for multiple hypothesis testing that maintains control of the false discovery rate while incorporating prior information about the hypotheses the prior information takes the form of p value weights if the assignment of weights is positively associated with the null hypotheses being false the procedure improves power except in cases where power is already near one even if the assignment of weights is poor power is only reduced slightly as long as the weights are not too large we also provide a similar method to control false discovery exceedance

adaptive false discovery rate control under independence and dependence in the context of multiple hypothesis testing the proportion 0 of true null hypotheses in the pool of hypotheses to test often plays a crucial role although it is generally unknown a priori a testing procedure using an implicit or explicit estimate of this quantity in order to improve its efficency is called adaptive in this paper we focus on the issue of false discovery rate fdr control and we present new adaptive multiple testing procedures with control of the fdr in a first part assuming independence of the p values we present two new procedures and give a unified review of other existing adaptive procedures that have provably controlled fdr we report extensive simulation results comparing these procedures and testing their robustness when the independence assumption is violated the new proposed procedures appear competitive with existing ones the overall best though is reported to be storey s estimator albeit for a specific parameter setting that does not appear to have been considered before in a second part we propose adaptive versions of step up procedures that have provably controlled fdr under positive dependence and unspecified dependence of the p values respectively in the latter case while simulations only show an improvement over non adaptive procedures in limited situations these are to our knowledge among the first theoretically founded adaptive multiple testing procedures that control the fdr when the p values are not independent

shrunken p values for assessing differential expression with applications to genomic data analysis in many scientific problems involving high throughput technology inference must be made involving several hundreds or thousands of hypotheses recent attention has focused on how to address the multiple testing issue much focus has been devoted toward the use of the false discovery rate in this article we consider an alternative estimation procedure titled shrunken p values for assessing differential expression spade the estimators are motivated by risk considerations from decision theory and lead to a completely new method for adjustment in the multiple testing problem in addition the decision theoretic framework can be used to derive a decision rule for controlling the number of false positive results some theoretical results are outlined the proposed methodology is illustrated using simulation studies and with application to data from a prostate cancer gene expression profiling study

false discovery and false nondiscovery rates in single step multiple testing procedures results on the false discovery rate fdr and the false nondiscovery rate fnr are developed for single step multiple testing procedures in addition to verifying desirable properties of fdr and fnr as measures of error rates these results extend previously known results providing further insights particularly under dependence into the notions of fdr and fnr and related measures first considering fixed configurations of true and false null hypotheses inequalities are obtained to explain how an fdr or fnr controlling single step procedure such as a bonferroni or sidak procedure can potentially be improved two families of procedures are then constructed one that modifies the fdr controlling and the other that modifies the fnr controlling sidak procedure these are proved to control fdr or fnr under independence less conservatively than the corresponding families that modify the fdr or fnr controlling bonferroni procedure results of numerical investigations of the performance of the modified sidak fdr procedure over its competitors are presented second considering a mixture model where different configurations of true and false null hypotheses are assumed to have certain probabilities results are also derived that extend some of storey s work to the dependence case

asymptotic results on adaptive false discovery rate controlling procedures based on kernel estimators the false discovery rate fdr is a commonly used type i error rate in multiple testing problems it is defined as the expected false discovery proportion fdp that is the expected fraction of false positives among rejected hypotheses when the hypotheses are independent the benjamini hochberg procedure achieves fdr control at any pre specified level by construction fdr control offers no guarantee in terms of power or type ii error a number of alternative procedures have been developed including plug in procedures that aim at gaining power by incorporating an estimate of the proportion of true null hypotheses in this paper we study the asymptotic behavior of a class of plug in procedures based on kernel estimators of the density of the p values as the number m of tested hypotheses grows to infinity in a setting where the hypotheses tested are independent we prove that these procedures are asymptotically more powerful in two respects i a tighter asymptotic fdr control for any target fdr level and ii a broader range of target levels yielding positive asymptotic power we also show that this increased asymptotic power comes at the price of slower non parametric convergence rates for the fdp these rates are of the form m k 2k 1 where k is determined by the regularity of the density of the p value distribution or equivalently of the test statistics distribution these results are applied to one and two sided tests statistics for gaussian and laplace location models and for the student model

the positive false discovery rate a bayesian interpretation and the q value multiple hypothesis testing is concerned with controlling the rate of false positives when testing several hypotheses simultaneously one multiple hypothesis testing error measure is the false discovery rate fdr which is loosely defined to be the expected proportion of false positives among all significant hypotheses the fdr is especially appropriate for exploratory analyses in which one is interested in finding several significant results among many tests in this work we introduce a modified version of the fdr called the positive false discovery rate pfdr we discuss the advantages and disadvantages of the pfdr and investigate its statistical properties when assuming the test statistics follow a mixture distribution we show that the pfdr can be written as a bayesian posterior probability and can be connected to classification theory these properties remain asymptotically true under fairly general conditions even under certain forms of dependence also a new quantity called the q value is introduced and investigated which is a natural bayesian posterior p value or rather the pfdr analogue of the p value 1 introduction when testing a single hypothesis one is usually concerned with controlling the false positive rate while maximizing the probability of detecting an effect when one really exists in statistical terms we maximize the power conditional on the type i error rate being at or below some level the field of multiple hypothesis testing tries to extend this basic paradigm to the situation where several hypotheses are tested simultaneously one must define an appropriate compound error measure according to the rate of false positives one is willing to encounter then a procedure is developed that allows one to control the error rate at a desired level while maintaining the power of each test as much as possible the most commonly controlled quantity when testing multiple hypotheses is the family wise error rate fwer which is the probability of yielding one or more false positives out of all hypotheses tested the most familiar example of this is the bonferroni method if there are m hypothesis tests each test is controlled so that the probability of a false positive is less than or equal to m for some chosen value of it then follows that the overall fwer is less than or equal to m any

controlling the false discovery rate a practical and powerful approach to multiple testing summary the common approach to the multiplicity problem calls for controlling the familywise error rate fwer this approach though has faults and we point out a few a different approach to problems of multiple significance testing is presented it calls for controlling the expected proportion of falsely rejected hypotheses the false discovery rate this error rate is equivalent to the fwer when all hypotheses are true but is smaller otherwise therefore in problems where the control of the false discovery rate rather than that of the fwer is desired there is potential for a gain in power a simple sequential bonferronitype procedure is proved to control the false discovery rate for independent test statistics and a simulation study shows that the gain in power is substantial the use of the new procedure and the appropriateness of the criterion are illustrated with examples

false discovery rate estimation for large scale homogeneous discrete p values large scale homogeneous discrete p values are encountered frequently in high throughput genomics studies and the related multiple testing problems become challenging because most existing methods for the false discovery rate fdr assume continuous p values in this article we study the estimation of the null proportion and fdr for discrete p values with common support in the finite sample setting we propose a novel class of conservative fdr estimators furthermore we show that a broad class of fdr estimators is simultaneously conservative over all support points under some weak dependence condition in the asymptotic setting we further demonstrate the significant improvement of a newly proposed method over existing methods through simulation studies and a case study

estimation of a two component mixture model with applications to multiple testing we consider a two component mixture model with one known component we develop methods for estimating the mixing proportion and the other unknown distribution nonparametrically given i i d data from the mixture model we use ideas from shape restricted function estimation and develop tuning parameter free estimators that are easily implementable and have good finite sample performance we establish the consistency of our procedures distribution free finite sample lower confidence bounds are developed for the mixing proportion the identifiability of the model and the estimation of the density of the unknown mixing distribution are also addressed we discuss the connection with the problem of multiple testing and compare our procedure with some of the existing methods in that area through simulation studies we also analyse two data sets one arising from an application in astronomy and the other from a microarray experiment

on false discovery rate thresholding for classification under sparsity we study the properties of false discovery rate fdr thresholding viewed as a classification procedure the 0 class null is assumed to have a known density while the 1 class alternative is obtained from the 0 class either by translation or by scaling furthermore the 1 class is assumed to have a small number of elements w r t the 0 class sparsity we focus on densities of the subbotin family including gaussian and laplace models nonasymptotic oracle inequalities are derived for the excess risk of fdr thresholding these inequalities lead to explicit rates of convergence of the excess risk to zero as the number m of items to be classified tends to infinity and in a regime where the power of the bayes rule is away from 0 and 1 moreover these theoretical investigations suggest an explicit choice for the target level alpha m of fdr thresholding as a function of m our oracle inequalities show theoretically that the resulting fdr thresholding adapts to the unknown sparsity regime contained in the data this property is illustrated with numerical experiments