# Optimal Tests for Combining \(p\)-Values
## 3 Numeric Studies  
In this section, we assess the performances of the proposed tests through a simulation study. In the simulation, we compare the optimal CLRT-based test \(T_{CLRT}\) with the popular and representative gamma distribution-based tests, \(T_{G(0)}\), \(T_{G(1)}\), and \(T_{G(\infty)}\) (i.e., the Min \(p\), Fisher, and z test, respectively). In the simulation study, fifty (\(n=50\)) independent \(p\)-values are simulated and combined. Among these 50 \(p\)-values, \(m\) (\(m=0\), 10, 20, 40, 50) are assumed from the true individual alternative hypotheses, and the rest (\(n-m\)) are from the true individual null hypotheses. When \(m=0\), all 50 individual null hypotheses are true, and the empirical power obtained under this condition is the empirical type I error rate. The \(p\)-values from the true null hypotheses are randomly sampled between 0 and 1 from the uniform distribution. For a true individual alternative hypothesis \(H_{i1}\) (\(i=1,\cdots,m\)), we assume the \(p\)-value \(p_{i}\) is obtained via a random variable \(z_{i}\sim N(\mu_{i},1)\). We randomly set \(k\) of the \(m\)\(u_{i}^{\prime}\)s as positive or negative (those alternative hypotheses with the same direction of the effects are called concordant alternatives) and the rest of \(m-k\) having the other direction. A two-sided \(p\)-value for each true individual alternative hypothesis are obtained via the standard z test by comparing the test statistic with the standard normal distribution \(N(0,1)\). For the true alternative hypotheses, we consider three different scenarios for the effects of \(\mu_{i}\)'s. Scenario 1: \(|\mu_{i}|=\mu v_{i}/\sum_{i=1}^{m}v_{i}\), where \(v_{i}=10^{\prime}\); \(r_{i}\sim N(0.3,1)\); and \(\mu=0.8,0.6,\ 0.4,\ 0.3\) when there are 10, 20, 40, and 50 true individual alternatives, respectively. Scenario 2: \(|\mu_{i}|=\mu v_{i}/\sum_{i=1}^{m}v_{i}\), where \(v_{i}\sim U(1,100)\) and \(\mu=1.2,\ 1.0,\ 0.6,\ 0.5\) when the number of true individual alternatives is \(m=10,\ 20,\ 40,\ \text{and}\ 50\), respectively. Scenario 3: \(|\mu_{i}|=\mu/m\), and \(\mu=1.5,\ 1.2,\ 0.8,\ \text{and}\ 0.6\) when \(m=10,\ 20,\ 40,\ \text{and}\ 50\), respectively. Note that (i) the constants (e.g., \(\mu\), the parameters in the normal distribution for \(r_{i}\) and the uniform distribution for \(v_{i}\)) are chosen in the way so that the empirical powers are appreciable for comparison. (ii) For all the three scenarios, the sum of the absolute effect sizes is equal to \(\mu\); and (iii) for given \(m\), the degree of heterogeneity of the effect sizes among the true individual alternatives decreases from Scenario 1 to Scenario 3. Morespecifically, in scenario 1, the effect sizes are extremely heterogenous when the number of the true individual alternatives is small. In Scenario 3, the effect sizes are more homogenous. The situations in Scenario 2 are between those in Scenarios 1 and 3. By considering those different conditions, we tried to conduct a reasonable and realistic simulation study to fairly compare our proposed tests with others. The empirical power values of the tests are estimated using the rejection proportions based on 1000 replicates at the significance level of 0.05. For the new tests, \(T_{CLRT}\)\(10^{5}\) replicates are used to estimate their \(p\)-values from the resampling method described in Section 2.3.2. Under the overall null hypotheses (i.e., all individual null hypotheses are true), the empirical type I error rates for all methods with different significance levels were obtained. From the simulation study, all methods controlled type I error rate quite well (see Table S4 in the Supplementary File). Figures 1-3 plot the empirical power values of the Min \(p\) (Min), Fisher (Fisher), z test (Z), and the proposed CLRT-based test \(T_{CLRT}\) (LRT_CS) when \(p\)-values are combined under Scenarios 1 to 3, respectively.