# Robust Semiparametric Optimal Testing Procedure for Multiple Normal Means

Authors: Peng Liu; Chong Wang
Journal: Journal of Probability and Statistics
 (2012)

---

## Abstract

In high-dimensional gene expression experiments such as microarray and RNA-seq experiments,
the number of measured variables is huge while the number of replicates is small. As a
consequence, hypothesis testing is challenging because the power of tests can be very low after
controlling multiple testing error. Optimal testing procedures with high average power while
controlling false discovery rate are preferred. Many methods were constructed to achieve high
power through borrowing information across genes. Some of these methods can be shown to
achieve the optimal average power across genes, but only under a normal assumption of
alternative means. However, the assumption of a normal distribution is likely violated in practice.
In this paper, we propose a novel semiparametric optimal testing (SPOT) procedure for high-dimensional
data with small sample size. Our procedure is more robust because it does not
depend on any parametric assumption for the alternative means. We show that the proposed test
achieves the maximum average power asymptotically as the number of tests goes to infinity.
Both simulation study and the analysis of a real microarray data with spike-in probes show that
the proposed SPOT procedure performs better when compared to other popularly applied
procedures.


---

## Body

## 1. Introduction

The problem of statistically testing mean difference for each of thousands of variables is commonly encountered in genomic studies. For example, the popularly applied microarray technology allows the gene expression study of tens of thousands of genes simultaneously. The recent advance of next-generation sequencing technology allows the measurement of gene expression in an even higher dimension. These high-throughput technologies have revolutionized the way genomic studies progress and provided rich data to explore. However, these experiments are expensive, and as a consequence, such experiments typically involve only a few samples for each treatment group. This results in the “largep, small n” problem for hypothesis testing, and the power of the statistical tests can be very low after controlling the multiple testing error, such as the false discovery rate (FDR).The normalized signal intensities from microarray experiments are generally assumed to follow normal distributions [1–4]. The recently emerging next-generation sequencing data may also be modeled approximately using normal distributions, when the number of reads are large or under certain transformation [5]. Thus multiple testing problem for normal means has wide applications in genetic and genomic studies, and it is also a general statistical question of interest.Several testing procedures have been proposed in the context of microarray study, including the SAM test [6], Efron's t-test [7], the regularized t-test [8], the B-statistic [1] and its multivariate counterpart, the MB-statistic [9], the test of Wright and Simon [10], the moderated t-test [2], the FS test [3] and the test of [11] which is similar to the FS test, the FSS test [4], and the LEMMA test [12]. Although numerous procedures have been proposed, very few can be justified to achieve the optimal power. Among these procedures, Hwang and Liu [4] proposed a framework and showed that an optimal testing procedure can be derived within such a framework. They also proposed a test with maximum average power (the MAP test) and an approximated version, the FSS test. Here the optimality was defined in terms of maximizing the power averaged across all tests for which the null hypotheses are false while controlling FDR. This method provides theoretical guide for developing optimal multiple testing procedures. The popularly applied moderated t-statistic developed by Smyth [2] can also be shown to achieve optimal power asymptotically under different distributional assumptions from the FSS test. Both the moderated t-statistic and the FSS test assume that the mean expression levels (or the mean of interesting contrasts) of all genes follow a normal distribution although the parameters for this distribution vary between the two tests. Yet in practice such distribution depends on the population of genes selected in a particular study and often does not follow the prespecified parametric distribution. This raises concerns about the robustness of the moderated t and the FSS tests.The objective of this paper is to develop an optimal and robust multiple testing procedure without any distributional assumptions on the mean. As in Hwang and Liu [4], the optimality is defined in terms of maximizing the power averaged across all tests for which the null hypotheses are false while controlling FDR. We develop a semiparametric optimal testing procedure which we abbreviate as the SPOT procedure. The distribution of the mean expression across genes is not assumed to follow a parametric model which makes our method robust to violations to normal assumptions. We find that the SPOT procedure works very well in simulation studies and in an analysis of real microarray data with spike-in probes.The remaining of this paper is organized as follows. We first introduce necessary notations in Section2. Then, in Section 3, we describe the general concepts of optimal testing procedures. We propose our semiparametric optimal testing (SPOT) procedure in Section 4 and describe its implementation in Section 5. Section 6 presents simulation studies. Section 7 shows the analysis result of a real microarray dataset. Section 8 provides a summary of this paper.


## 2. Notations

An appropriate linear model is typically fitted for each gene based on the design of a microarray experiment. Section2 of  Smyth [2] provides a nice description of this topic. Given the linear model, suppose that we have an interesting contrast to test for each gene. This contrast may be the difference between the means of two treatment groups or linear combination of means from several treatment groups. For the simplicity of description, we call the genes whose contrast means are not zero as the differentially expressed (DE) genes and the genes whose contrast means equal to zero as equivalently expressed (EE) genes. After fitting the linear model for each gene, we obtain an estimate for the contrast for each gene, Xg. In addition, we get the estimate of the sample residual variance, sg2, for each gene. For each g=1,…,G, Xg and sg2 are related to true parameters, μg and σg2, by Xg∣μg,σg2~N(μg,νgσg2) and sg2∣σg2~(σg2/dg)χdg2, where μg is the contrast mean for gene g, σg2 is the true residual variance for gene g, and the coefficients νg and dg are determined by the design of the experiment. Two examples are given as follows.Example 2.1.
Two-channel microarray experiment to compare two treatments. Assume that each sample from treatment A is paired randomly with a sample from treatment B and each pair of samples is cohybridized onto one slide. After normalization and appropriate transformation, the difference of normalized expression measurements between the two samples on each slide is analyzed for each gene. Hence, this is a paired sample case and the number of data points for each gene isn, the number of slides. We are interested in identifying DE genes. In this case, Xg is the mean difference of the paired samples for gene g. sg2 is the sample variance for gene g. So νg=1/n and dg=n-1.Example 2.2.
Affymetrix microarray experiment with two independent samples. Assume sample sizes aren1 and n2 for treatment A and treatment B, respectively. The statistic Xg is the difference in sample means of normalized expression measurements between two groups for gene g. sg2 is the pooled sample variance. Then νg=1/n1+1/n2 and dg=n1+n2-2.
Given the dataXg and sg2, an ordinary t-test with statistic tg=Xg/νgsg may be used to test the null hypothesis Hg0: μg=0. However, the power of such tests is low after controlling multiple testing error. So statistical methods with higher power are in demand for such high-dimensional testing problem as encountered in gene expression studies.


## 3. Optimal Testing Procedures

In the analysis of high-dimensional gene expression data such as microarray data, we are more interested in the average behavior of the tests across all genes rather than the performance of an individual test. Because the dimension of tests is huge, multiple testing errors should be controlled to avoid too many type I errors. Controlling FDR is an important method for controlling multiple testing errors and is widely used for genomic studies. Although many testing procedures have been developed as reviewed in Section1, the paper by Hwang and Liu [4] provides some theoretical guide on how to derive optimal testing procedures within an empirical Bayes framework. The optimal tests are defined to be the ones that maximize the power averaged across all genes for which the null hypotheses are false while controlling FDR. Such optimal tests have been called MAP tests, where MAP stands for maximum average power [4].In a Bayesian framework, we assume model parameters likeμg and σg2 follow some distributions. The residual variances of genes, σg2, have been modeled by prior distribution like inverse gamma [2, 10] or log-normal [4] distribution independent of whether the null hypothesis is true or false. For EE genes, the mean of contrast Xg, μg, is equal to 0. For DE genes, the mean μg is not 0 almost surely. Denote the alternative distribution of μg by π1(·). Based on the Neyman-Pearson fundamental lemma, for a randomly selected gene g, the most powerful test statistic for testing Hg0:μg=0 versus Hg1:μg~π1(μg) is given by
(3.1)TgNP=∬f(Xg,sg2∣μg,σg2)π1(μg)π(σg2)dμgdσg2∫f(Xg,sg2∣μg=0,σg2)π(σg2)dσg2,
where π(·) denote the prior distributions of σg2. And the test rejects the null hypothesis Hg0 when TgNP is large. The simultaneous testing procedure where all genes are tested using the most powerful statistics TgNP,g=1,2,…,G, achieves the highest average power while controlling FDR, as proved in Hwang and Liu [4].One popular multiple-testing method for microarray data is the moderatedt-test proposed by Smyth [2]. Smyth proposed to model the residual variance σg2 with the prior distribution:
(3.2)1σg2~1d0s02χd02,
where χd02 denotes a chi-square distribution with degrees of freedom d0 and s02 is another hyperparameter. This prior distribution is equivalent to an inverse-gamma distribution and has been shown to fit real data well. Compared to a standard t-test statistic tg=xg/νgsg, Smyth's moderated t-statistic takes the form of
(3.3)t~g=xgνgs~g,
where
(3.4)s~g2=sg2dg+s02d0dg+d0
is a shrinkage estimator of σg2 by shrinking sg2 toward s02.In practice, the unknown hyperparametersd0 and s02 for the distribution of the variance σg2 can be estimated consistently by the method of moments, that is, equating the empirical and expected first two moments of logsg2 [2]. Smyth [2] showed that the moderated t-test is equivalent to the B statistic proposed in Lönnstedt and Speed [1] which was derived as the posterior odds under the assumption that the distribution of μg under the alternative hypothesis follows N(0,ν0σg2), where ν0 is a constant. In fact, we can prove the claim that the moderated t-test achieves the optimal average power asymptotically under their assumptions for μg and σg2. The proof is in the appendix.However, note that the assumption thatμg of DE genes follows a normal distribution with mean zero is restrictive. It is likely that, for example, there are more upregulated genes than downregulated genes for some studies which suggests that the mean of μg should be positive. Hwang and Liu [4] have proposed a more general normal prior distribution of μg for DE genes:
(3.5)π1(μg)~N(θ,τg2),
where the mean of this distribution is not necessarily zero but to be estimated based on data. In addition, the variance for this distribution does not depend on the residual variance. Under this model, they have derived an optimal test and an approximated version of the test statistic (FSS test) that is computationally faster. The FSS statistic shrinks both the estimate of mean μg and the estimate of variance σg2.Both the moderatedt-test and the FSS test have been shown to achieve optimal power asymptotically under the assumption of normal distribution for the alternative means. Simulation studies also confirm that the power of the tests is superior under the model assumptions. However, a single normal distribution assumption on μg for DE genes may not be appropriate for all cases and the distribution of π1(μg) may consist of a mixture of different subgroup distributions, for example, a mixture of two normal distributions with one having a negative mean and the other having a positive mean. If the parametric distributional assumptions of π1(μg) are violated, the power of an optimal test built under those assumptions will suffer.


## 4. Semiparametric Optimal Testing (SPOT) Procedure

To obtain a more robust procedure, we propose to model the distribution of the meanμg nonparametrically while still deriving the optimal procedure. For the variance σg2, the inverse gamma distributional assumption is reasonable and works well in practice, so we still keep this assumption. Hence, we will derive a semiparametric optimal testing procedure that we call the SPOT procedure.Note that the numerator and denominator of the most powerful test statistic (3.1) are the joint marginal distributions of (Xg,sg2), under the alternative and null hypothesis, respectively. By denoting the marginal distributions by
(4.1)m1(Xg,sg2)=∬f(Xg,sg2∣μg,σg2)π1(μg)π(σg2)dμgdσg2,m0(Xg,sg2)=∫f(Xg,sg2∣μg=0,σg2)π(σg2)dσg2,
statistic (3.1) becomes
(4.2)TgNP=m1(Xg,sg2)m0(Xg,sg2).
The null marginal distribution m0(Xg,sg2) only involves integration with respect to variance σg2. With consistent estimators of hyperparameters as proposed in Smyth [2], we can estimate m0(Xg,sg2) consistently. For the alternative marginal distribution m1(Xg,sg2), it is hard to find a consistent estimator without any distributional assumption on μg. If we were to know which genes are DE, then we could estimate m1(Xg,sg2) nonparametrically with observed values of (Xg,sg2) from the DE gene population. Many nonparametric density estimators are consistent, for example, the histogram estimators and the kernel density estimators with proper choices of bandwidths [13]. However, the knowledge of differential expression is the research question of the study and of course is not available for all genes. Considering all genes without separating those that are differentially expressed from those that are not, we have a mixture distribution of differentially expressed and nondifferentially expressed genes. The mixture density of the marginal distributions, denoted by mm(Xg,sg2), can be estimated consistently by nonparametric density estimators with observed (Xg,sg2) for all genes g=1,…,G. Can this consistent estimator of mm(Xg,sg2) help us construct a most powerful test statistic, together with a consistent estimator of m0(Xg,sg2)?Suppose thatp0 and p1 are proportions of EE and DE genes, respectively, with 0≤p0,p1≤1 and p0+p1=1, then the mixture marginal density is
(4.3)mm(Xg,sg2)=p0m0(Xg,sg2)+p1m1(Xg,sg2).The ratio of mixture marginal densitymm(Xg,sg2) and the null marginal density m0(Xg,sg2) is a monotonic function of the statistic TgNP expressed in formula (4.2) because
(4.4)mm(Xg,sg2)m0(Xg,sg2)=p0m0(Xg,sg2)+p1m1(Xg,sg2)m0(Xg,sg2),=p0+p1m1(Xg,sg2)m0(Xg,sg2).
Thus the test that rejects the null hypothesis when mm(Xg,sg2)/m0(Xg,sg2) is large is also a most powerful test. Note that to calculate this statistic, we only need to estimate mm(Xg,sg2) and m0(Xg,sg2) but do not have to estimate the proportions p0 and p1.Letm^m(Xg,sg2) denote any consistent density estimator of mm(Xg,sg2), and let m^0(Xg,sg2) denote any consistent estimator of m0(Xg,sg2), such that
(4.5)m^m(Xg,sg2)⟶Pmm(Xg,sg2)asG↗∞,m^0(Xg,sg2)⟶Pm0(Xg,sg2)asG↗∞,
where →P denotes convergence in probability. Then the statistic m^m(Xg,sg2)/m^0(Xg,sg2) has the optimal testing power asymptotically. Notice the convergence  with respect to G, which is usually huge in the microarray and RNA-seq studies.We have already discussed the availability of a parametric consistent estimator ofm0(Xg,sg2) through estimating the hyperparameters d0 and s02 of σg2 in Section 3. For mm(Xg,sg2), any theoretically consistent density estimator m^m(Xg,sg2) of joint data (Xg,sg2) can be used to construct the test statistic (4.4) with asymptotically optimal average power. For example, nonparametric estimators such as histograms, kernel density estimates, and local polynomial estimators can all be utilized. As our test statistic m^m(Xg,sg2)/m^0(Xg,sg2) involves both parametric and nonparametric parts, we name it the semiparametric optimal test (SPOT).


## 5.  Implementation of SPOT

In this section, we discuss details in implementation of the proposed SPOT procedure.


### 5.1. Estimation ofm0(Xg,sg2)

The null marginal density(5.1)m0(Xg,sg2)=∫f(Xg,sg2∣μg=0,σg2)π(σg2)dσg2=∫e-xg2/(2vgσg2)(2πvgσg2)1/2(dg2σg2)dg/2s2(dg/2-1)e-dgsg2/(2σg2)Γ(dg/2)(d0s022)d0/2σg-2(d0/2+1)e-d0s02/2σg2Γ(d0/2)dσg2=C2⋅sg2(d/2-1)(xg2/vg+d0s02+dgsg22)-(1+d0+dg)/2,
where C2 is a constant. As in Smyth [2] and Hwang and Liu [4], we assume that the distribution of σg2 does not depend on whether a gene is DE or EE. Then, all genes are used to estimate the parameters d0 and s02. We apply the method of moments proposed in Smyth [2] to get estimates of d0 and s02. Replacing unknown parameters d0 and s02 in m0(Xg,sg2) by their consistent method of moments estimates leads to a consistent estimator m^0(Xg,sg2) of m0(Xg,sg2).


### 5.2.  A Hybrid Method for Estimation ofmm(Xg,sg2)

Although any consistent estimatorm^m(Xg,sg2) can be used to construct a SPOT statistic of the form m^m(Xg,sg2)/m^0(Xg,sg2), in practice, a density estimator that converges fast would be always preferred. It is known that the accuracy of the density estimators goes down quickly as the dimension increases [13]. We have tried a few two-dimensional density estimators for m^m(Xg,sg2), including the kernel estimators. Due to the curse of dimensionality, the direct two-dimensional density estimators do not perform as satisfactory as a hybrid estimator that we develop and would suggest to use. This hybrid estimator has a component that is similar to kernel estimators, whereas it also utilizes the prior information on variances σg2 to help improving the accuracy.In constructing this estimator, we first estimate the marginal density ofXg by the typical kernel density estimate:
(5.2)f^(xg)=1G∑i=1G1hK(xg-xih),
where h is a positive value known as bandwidth. We estimate the conditional density of f(sg2∣xg) by using
(5.3)f(sg2∣xg)=∫f(sg2∣σg2,xg)f(σg2∣xg)dσg2=∫f(sg2∣σg2)f(σg2∣xg)dσg2,
where the second equality is a result of the independence between sg2 and xg given the parameter σg2. The distribution of sg2∣σg2 is (σg2/dg)χdg2 for normal-distributed observations. Now we need to estimate f(σg2∣xg). Denote the set of genes that lie within bandwidth distance to gene g as {Ag:i∈Ag if and only if |xi-xg|<h}. We estimate f(σg2∣xg) by the following approximation that is based on the neighborhood of xg, Ag:
(5.4)1#{Ag}∑i∈Agf(si2∣σ2)π(σ2)∫f(si2∣σ2)π(σ2)dσ2.The#{Ag} in formula denotes the number of genes in set Ag. Substituting exact parametric form of f(sg2∣σ2) and π(σ2) into above formulas leads to the explicit form
(5.5)f^(sg2∣xg)=C3⋅1#{Ag}∑i∈Agsg2(d/2-1)(dgsi2+d0s02+dgsg22)-(d0+dg)/2,
where C3 is a constant. The product between the kernel estimate f^(xg) and the conditional estimate f^(sg2∣xg) provides us a joint density estimator of the mixture
(5.6)m^m(Xg,sg2)=f^(xg)⋅f^(sg2∣xg).
With this approximation, we cannot theoretically show that the resulting estimator, m^m(Xg,sg2), is consistent but it works better in practice than the consistent kernel density estimator of the joint density mm(Xg,sg2).


## 5.1. Estimation ofm0(Xg,sg2)

The null marginal density(5.1)m0(Xg,sg2)=∫f(Xg,sg2∣μg=0,σg2)π(σg2)dσg2=∫e-xg2/(2vgσg2)(2πvgσg2)1/2(dg2σg2)dg/2s2(dg/2-1)e-dgsg2/(2σg2)Γ(dg/2)(d0s022)d0/2σg-2(d0/2+1)e-d0s02/2σg2Γ(d0/2)dσg2=C2⋅sg2(d/2-1)(xg2/vg+d0s02+dgsg22)-(1+d0+dg)/2,
where C2 is a constant. As in Smyth [2] and Hwang and Liu [4], we assume that the distribution of σg2 does not depend on whether a gene is DE or EE. Then, all genes are used to estimate the parameters d0 and s02. We apply the method of moments proposed in Smyth [2] to get estimates of d0 and s02. Replacing unknown parameters d0 and s02 in m0(Xg,sg2) by their consistent method of moments estimates leads to a consistent estimator m^0(Xg,sg2) of m0(Xg,sg2).


## 5.2.  A Hybrid Method for Estimation ofmm(Xg,sg2)

Although any consistent estimatorm^m(Xg,sg2) can be used to construct a SPOT statistic of the form m^m(Xg,sg2)/m^0(Xg,sg2), in practice, a density estimator that converges fast would be always preferred. It is known that the accuracy of the density estimators goes down quickly as the dimension increases [13]. We have tried a few two-dimensional density estimators for m^m(Xg,sg2), including the kernel estimators. Due to the curse of dimensionality, the direct two-dimensional density estimators do not perform as satisfactory as a hybrid estimator that we develop and would suggest to use. This hybrid estimator has a component that is similar to kernel estimators, whereas it also utilizes the prior information on variances σg2 to help improving the accuracy.In constructing this estimator, we first estimate the marginal density ofXg by the typical kernel density estimate:
(5.2)f^(xg)=1G∑i=1G1hK(xg-xih),
where h is a positive value known as bandwidth. We estimate the conditional density of f(sg2∣xg) by using
(5.3)f(sg2∣xg)=∫f(sg2∣σg2,xg)f(σg2∣xg)dσg2=∫f(sg2∣σg2)f(σg2∣xg)dσg2,
where the second equality is a result of the independence between sg2 and xg given the parameter σg2. The distribution of sg2∣σg2 is (σg2/dg)χdg2 for normal-distributed observations. Now we need to estimate f(σg2∣xg). Denote the set of genes that lie within bandwidth distance to gene g as {Ag:i∈Ag if and only if |xi-xg|<h}. We estimate f(σg2∣xg) by the following approximation that is based on the neighborhood of xg, Ag:
(5.4)1#{Ag}∑i∈Agf(si2∣σ2)π(σ2)∫f(si2∣σ2)π(σ2)dσ2.The#{Ag} in formula denotes the number of genes in set Ag. Substituting exact parametric form of f(sg2∣σ2) and π(σ2) into above formulas leads to the explicit form
(5.5)f^(sg2∣xg)=C3⋅1#{Ag}∑i∈Agsg2(d/2-1)(dgsi2+d0s02+dgsg22)-(d0+dg)/2,
where C3 is a constant. The product between the kernel estimate f^(xg) and the conditional estimate f^(sg2∣xg) provides us a joint density estimator of the mixture
(5.6)m^m(Xg,sg2)=f^(xg)⋅f^(sg2∣xg).
With this approximation, we cannot theoretically show that the resulting estimator, m^m(Xg,sg2), is consistent but it works better in practice than the consistent kernel density estimator of the joint density mm(Xg,sg2).


## 6. Simulation Study

In order to evaluate the performance of our proposed SPOT procedure, we performed three simulation studies. The gene expression data were simulated from Normal  (μgi,σg2) for observations of gene g in treatment group i. The way to sample μgi and σg2 differs across simulation studies. We assume that there are two treatment groups and 3 replicates per treatment group. For each simulation setting, one hundred sets of gene expression data were independently simulated, and each dataset included 10,000 genes. The performances of the SPOT, moderated t, FSS, and ordinary t-test statistics were evaluated for by comparing their average behavior averaged across the 100 datasets.


### 6.1. Simulation Study I

In the first simulation study, we have two settings that differ in the number of DE genes. For the first setting,G1=2,500 are DE genes whereas the other G0=7,500 are EE. In the second setting, only G1=1,800 are DE while the other G0=8,200 are EE. Gene expression means μgi and variances σg2 were simulated as follows:
(6.1)μg1=0∀g;μg2~Normal(0.5,0.32)forg=1to0.3G1;μg2~Normal(1,0.32)forg=(0.3G1+1)to0.9G1;μg2~t1(0.5)forg=(0.9G1+1)toG1;μg2=0forg=(G1+1)to10000;σg2~Gamma(2,4)∀g.For each simulated data, SPOT, moderatedt, FSS, and ordinary t-test statistics were calculated and evaluated using the number of selected true positives at various FDR levels. The plots of number of true positives versus FDR for SPOT, moderated t, FSS, and ordinary t-test statistics are shown in Figure 1. Simulation settings 1 and 2 generated similar results. The ordinary t-test is the poorest method under comparison. The moderated t-test is considerably better than the ordinary t-test although it is worse than FSS test. Our proposed SPOT test is superior to all other three methods, with the largest number of true positive findings than the other three statistics at the same FDR levels.Simulation study I: plots of number of true positives (# TP) versus false discovery rate (FDR) from analyses using SPOT, moderatedt, and FSS methods.
(a)
Simulation I, setting 1
(b)
Simulation I, setting 2


### 6.2. Simulation Study II

To check how the variance distribution affects the relative ranking of the SPOT procedure, we did another simulation study the same as the setting 1 of simulation study I except that the variances were simulated from a log-normal distribution, which is the assumption under which theFSS test was derived. As Figure 2 shows, the results are similar to those from simulation I. The SPOT procedure still performs much better than all the other three methods.Figure 2
Simulation study II: plots of number of true positives (# TP) versus false discovery rate (FDR) from analyses using SPOT, moderatedt, and FSS methods.


### 6.3. Simulation Study III

Typically, the parametric test achieves higher power than the nonparametric test if the parametric assumption is appropriate. To check the robustness of the SPOT procedure, we simulated data under the parametric assumption for bothμgi and σg2 under which the FSS test was derived. Specifically, for the 2,500 differentially expressed genes, μgi were drawn from a normal distribution with mean 1.2 and standard deviation 0.3, σg2 were sampled from a log-normal distribution with parameters −0.96 and 0.8. Figure 3 shows that the SPOT procedure and the FSS test are comparable to each other when FDR is small (less than 0.05) and they are both much better than the moderated t-test and the ordinary t-test. When FDR is between 0.05 and 0.15, the FSS test is the best while the SPOT procedure is the next best performing procedure, which is still much better than the moderated t-test and the ordinary t-test.Figure 3
Simulation study III: plots of number of true positives (# TP) versus false discovery rate (FDR) from analyses using SPOT, moderatedt, and FSS methods.


## 6.1. Simulation Study I

In the first simulation study, we have two settings that differ in the number of DE genes. For the first setting,G1=2,500 are DE genes whereas the other G0=7,500 are EE. In the second setting, only G1=1,800 are DE while the other G0=8,200 are EE. Gene expression means μgi and variances σg2 were simulated as follows:
(6.1)μg1=0∀g;μg2~Normal(0.5,0.32)forg=1to0.3G1;μg2~Normal(1,0.32)forg=(0.3G1+1)to0.9G1;μg2~t1(0.5)forg=(0.9G1+1)toG1;μg2=0forg=(G1+1)to10000;σg2~Gamma(2,4)∀g.For each simulated data, SPOT, moderatedt, FSS, and ordinary t-test statistics were calculated and evaluated using the number of selected true positives at various FDR levels. The plots of number of true positives versus FDR for SPOT, moderated t, FSS, and ordinary t-test statistics are shown in Figure 1. Simulation settings 1 and 2 generated similar results. The ordinary t-test is the poorest method under comparison. The moderated t-test is considerably better than the ordinary t-test although it is worse than FSS test. Our proposed SPOT test is superior to all other three methods, with the largest number of true positive findings than the other three statistics at the same FDR levels.Simulation study I: plots of number of true positives (# TP) versus false discovery rate (FDR) from analyses using SPOT, moderatedt, and FSS methods.
(a)
Simulation I, setting 1
(b)
Simulation I, setting 2


## 6.2. Simulation Study II

To check how the variance distribution affects the relative ranking of the SPOT procedure, we did another simulation study the same as the setting 1 of simulation study I except that the variances were simulated from a log-normal distribution, which is the assumption under which theFSS test was derived. As Figure 2 shows, the results are similar to those from simulation I. The SPOT procedure still performs much better than all the other three methods.Figure 2
Simulation study II: plots of number of true positives (# TP) versus false discovery rate (FDR) from analyses using SPOT, moderatedt, and FSS methods.


## 6.3. Simulation Study III

Typically, the parametric test achieves higher power than the nonparametric test if the parametric assumption is appropriate. To check the robustness of the SPOT procedure, we simulated data under the parametric assumption for bothμgi and σg2 under which the FSS test was derived. Specifically, for the 2,500 differentially expressed genes, μgi were drawn from a normal distribution with mean 1.2 and standard deviation 0.3, σg2 were sampled from a log-normal distribution with parameters −0.96 and 0.8. Figure 3 shows that the SPOT procedure and the FSS test are comparable to each other when FDR is small (less than 0.05) and they are both much better than the moderated t-test and the ordinary t-test. When FDR is between 0.05 and 0.15, the FSS test is the best while the SPOT procedure is the next best performing procedure, which is still much better than the moderated t-test and the ordinary t-test.Figure 3
Simulation study III: plots of number of true positives (# TP) versus false discovery rate (FDR) from analyses using SPOT, moderatedt, and FSS methods.


## 7. Evaluation Using the Golden Spike Microarray Data

In this section, we compare the performances of different methods using a real microarray dataset from experiments conducted using Affymetrix GeneChip in the Golden Spike Project. The Golden Spike Project generated microarray datasets comparing two replicated groups in which the relative concentrations of a large number of genes are known. The two groups are the spike-in group and the control group, each with three chips. Data and information related to this project are available through the websitehttp://www2.ccr.buffalo.edu/halfon/spike/. More specifically, the Golden Spike dataset included 1309 individual cRNAs “spiked in” at known relative concentrations between the two groups. The fold-changes between the spike-in and control group were assigned at different levels for different cRNAs, and the levels ranged from 1.2 to 4. Hence, these cRNAs were truly “differentially expressed” between groups and we consider them as DE genes. In addition, a background sample of 2551 RNA species was present at identical concentrations in both samples. So these 2551 RNA species were not differentially expressed between the two groups. With the knowledge of the true differential expression status, this real microarray dataset provides an ideal case to evaluate the performances of different methods without imposing any distributional assumption for variances and means as usually is done in simulation studies.With the summary dataset downloaded from the Golden Spike Project website, we calculated the SPOT, the moderatedt, the ordinary t, and the FSS statistics and evaluated their performances using the true statuses of RNA based on the design. Figure 4 shows the plots of number of true positives versus FDR for the ordinary t, moderated t, FSS, and SPOT procedures over a range of FDR ∈[0,0.15] which is of most practical interest. It can be observed that the performance of the SPOT procedure improves over the performances of the other three methods throughout the whole range of FDR in these plots. In addition, the improvement is substantial at lower FDR levels. For example, the SPOT procedure detects 754 true positives at the FDR level of 1% while the moderated t-test only detects 466 and the FSS test only detects 442 true positives (Table 1).Table 1
Golden Spike data: number of true positives selected by three testing procedures at critical FDR levels.
Method
FDR
0.01
0.02
0.05
0.1
0.15
0.2
SPOT
754
847
947
986
1015
1051
Moderatedt
466
588
821
911
969
1018
F
SS
442
563
824
908
975
1016Figure 4
Golden spike data: plots of number of true positive (TP) genes versus false discovery rate (FDR) from analysis using SPOT, moderatedt,  and FSS tests.


## 8. Summary

In this paper, we have derived a semiparametric optimal testing (SPOT) procedure for high-dimensional gene expression data analysis. Although the method is illustrated for analyzing microarray data, it can be applied to any high-dimensional testing problem with normal model. Our test statistic is justified to be asymptotically most powerful, without any assumption on the mean parameter of differential expression. The asymptotic property is derived when the number of genes is large, which is reasonable for high-dimensional gene expression studies. We also provided an approximate version to implement the SPOT procedure in practice and evaluated the performance of our proposed test statistic using both simulation studies and real microarray data analysis. The proposed SPOT method is shown to outperform the popularly applied moderatedt and the FSS statistics, which are optimal only under certain normality conditions of the mean. There is still potential in improving the performance of SPOT procedure if better density estimates can be found for the marginal distributions mm(Xg,sg2) and m0(Xg,sg2).



---

*Source: 913560-2012-07-15.xml*