Title: Optimal testing using combined test statistics across independent studies
Venue: NeurIPS 2023
Accepted: yes
Keywords: testing

meta-analysis

p-values

e-values

optimal

combining trials
Abstract: Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive a mathematical framework to study standard {meta-analysis} testing approaches in the context of the many normal means model, which serves as the platform to investigate more complex models.

We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us to mathematically quantify the cost of compressing $m$ trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in a sub-optimal {meta-analysis} method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build on recent information theoretic developments in the latter field.

Optimal testing using combined test statistics across independent studies

Abstract

Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive a mathematical framework to study standard meta-analysis testing approaches in the context of the many normal means model, which serves as the platform to investigate more complex models.

We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us to mathematically quantify the cost of compressing m trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in a sub-optimal meta-analysis method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build on recent information theoretic developments in the latter field.

1 Introduction

Given multiple data sets relating to the same hypothesis, one would like to combine the evidence. Sometimes, the full data sets are not available (e.g. due to privacy or proprietary reasons) or difficult to combine directly (e.g. due to the different experimental or observational setups). In such cases, the analysis must be carried out on the basis of the published results for each of the studies. Such 'metaanalysis' can increase the statistical power by combining individually inconclusive or moderately significant tests, while keeping the false positive rate under control. Therefore, meta-analysis has received a lot of attention in various fields, for instance in genetics and system biology, when studying rare variants [4, 16] or in deep learning, for few shot image recognition and neural architecture search, see the review article [24].

∗

The outcomes of the studies concerning hypothesis tests are, typically, summarized as real-valued test statistics and/or associated p-values. One expects the combination of m such p-values to result in an increase in power, but one also expects to pay a price relative to computing a test on the basis of the full, pooled data of the m trials. The question of how to optimally combine independent real-valued test statistics concerning the same hypothesis into a single test has an extensive literature. A multitude of methods for combining independent tests of significance exist. For combining p-values, this starts with Fisher, Tippett and Pearson in the nineteen-thirties, see [42, 18, 32, 36, 21, 28, 45, 14, 31, 50, 48, 12, 47] and references therein. In Section 3, we collect and describe the most popular and frequently used p-value combination techniques.

As noted in [9], there does not exist a general uniformly most powerful p-value combination method for all alternative hypotheses. The distribution of a p-value or its underlying test statistic under the alternative hypothesis should be taken into consideration when selecting a method of combination. The performance of different p-value combination techniques was investigated extensively by empirical experiments in various synthetic and real world scenarios, see for instance [29, 51]. However, a unified, general theoretical description is lacking, especially in non-trivial, multi-dimensional composite testing problems, where the likelihood ratio test is not necessarily uniformly most powerful.

E-values are an increasingly popular and important notion of evidence, see [35, 22, 34]. E-values allow the combination of several tests in a straightforward manner while preserving the prescribed level of the tests (see Section 3.2). Formally, e-values are nonnegative random variables whose expected values under the null hypothesis are bounded by one. In contrast to p-values defined by probabilities, e-values are defined by expectation. This imposes significant differences in their interpretation, application and combination compared to the more standard p-values. However, as for p-values, very little is known about the power of these combination procedures. Theoretical results focus on specific optimality criteria, for instance the worst-case growth-rate (GROW), see [22]. However, these do not directly imply guarantees on the testing power, which is the main focus in practice.

Our focus is on multidimensional models, where a certain loss in power is to be expected, since combining multidimensional data into a real-valued statistic (e.g. p-value or e-value) requires data compression. Typically, summary statistics are combined by some 'reasonable' function C m : R m → R , where 'reasonable' means that C m should not exploit the richness of the real numbers to encode the data in full. We aim to quantify the loss of summarizing, the gain of performing a meta-analysis and the best testing strategies in the individual experiments meta-analysis.

We consider the signal detection problem in the many normal means model, see Section 2 for the detailed description. One possible interpretation of this testing problem is to learn whether a treatment has an effect on any of the dimensions investigated. This model is directly applied in several fields where high-dimensional statistics and machine learning settings are concerned, such as detecting differentially expressed genes [33, 27, 30, 41, 15], bankruptcy prediction for publicly traded companies using Altman's Z-score in finance [5, 6], separation of the background and source in astronomical images [13, 23], and wavelet analysis [1, 26]. Furthermore, the model allows for tractable computations and it typically serves as the platform to investigate more difficult statistical and learning problems, including high- and infinite-dimensional models, see for instance [25, 43, 15, 20]. In each experiment j ∈ { 1 , ..., m } the observations are summarized by an appropriate real-valued summary statistic S ( j ) . These local test statistics (e.g. p- or e-values) are combined into C m ( S (1) , . . . , S ( m ) ) . We consider a general class of combination functions C m , requiring only Hölder type continuity. This introduces only a mild restriction, and includes many standard metaanalysis techniques, for instance the standard p-value combination methods (see Section 3.1); e-value techniques (see Section 3.2); and other ad hoc and natural test statistic combination approaches, see the beginning of Section 3 for additional examples.

Our setting provides a principled and unified framework to study the power of standard meta-analysis testing methods. Within the framework of the many normal means model, we derive a minimax lower bound for the testing (separation) error and provide test statistics with associated combination methods that attain this theoretical limit (up to a logarithmic factor). Our results reveal that there is a certain unavoidable loss associated with compressing the data of each experiment to a real valued test statistic. We see that while it is always possible to obtain better testing rates using m trials instead of the best possible test based on a single trial, there is always a loss incurred when compared to the

full, pooled data and optimal test in moderate- to large dimensional problems. Our theoretical results quantify these gains and losses in terms of the dimension d , sample size n and number of trials m .

Furthermore, we observe an elbow effect, which occurs when the number of trials is large compared to the dimension of the signal. In this regime, combinations of the (locally) optimal test in each individual trial performs sub-optimally as a whole when aggregated and meta-analysis approaches based on directional test statistics are shown to perform better. Finally, we show that the performance of the meta-level tests can substantially improve (in certain regimes, depending on d, m, n ) if a certain amount of coordination between the trials is allowed (e.g. by having access to the same random seed). For the theoretical analysis of meta-analysis techniques we derive connections with the distributed statistical learning literature under communication constraints. Our paper builds on the recent information theoretical developments in distributed testing [2, 39, 40], allowing us to address several fundamental questions for the first time with mathematical rigor.

The paper is organised as follows. In Section 2 we introduce the mathematical framework we consider in our investigation and present the corresponding minimax testing lower bound results. Next in Subsection 2.1 we show that the derived results are sharp by providing several meta-analysis approaches attaining the limits. Then we investigate the benefits of allowing a mild coordination between the trials in Subsection 2.2. We collect and discuss the standard p- and e-value combination methods in Section 3 and demonstrate our theoretical results numerically on synthetic data sets in Section 4. We discuss our results and derive conclusions in Section 5. The proofs of our results are deferred to the Appendix. In Section A.1 we present the proof of our main results while the proofs of the technical lemmas are given in A.2.

Notation: For two positive sequences a n , b n we write a n /lessorsimilar b n if the inequality a n ≤ Cb n holds for some universal positive constant C . Similarly, we write a n /equivasymptotic b n if a n /lessorsimilar b n and b n /lessorsimilar a n hold simultaneously and let a n /lessmuch b n denote that a n /b n = o (1) . Furthermore, we use the notations a ∨ b and a ∧ b for the maximum and minimum, respectively, between a and b . Throughout the paper c and C denote global constants whose value may change from one line to another.

2 Main results

In our analysis, we consider the localized version of the many normal means model, tailored to investigating meta-analysis techniques. We assume that in each local trial or experiment j ∈ { 1 , ..., m } we observe a d -dimensional random variable X ( j ) ∈ R d , subject to

for some unknown f ∈ R d . We denote by P f the joint distribution of the observations and let E f be the corresponding expectation. We note that this framework is equivalent to having n independent N ( f, I d ) observations within each local sample.

Our goal is to test the presence or absence of the 'signal component' f ∈ R d . More formally, we consider the simple null hypothesis H 0 : f = 0 versus composite alternative hypothesis H ρ : ‖ f ‖ 2 ≥ ρ , for some ρ > 0 . This corresponds to testing for joint significance of variables, such as the presence of an effect of a treatment on any of the dimensions investigated. The difficulty in distinguishing the hypotheses depends on the effect size, the sample size and the dimension d . Here, ρ can be seen as the smallest effect size deemed important.

For a { 0 , 1 } -valued test T , define the testing risk R ( T, H ρ ) as the sum of the Type I error probability and worst case Type II error probability, i.e.

In the case of a single trial (i.e. m = 1 ), this testing problem is known to have minimax separation rate or 'detection boundary' ρ 2 /equivasymptotic √ d/n .

This means that if ρ 2 /greatermuch √ d/n , there exist consistent 2 tests T ≡ T d,n in the sense that R ( T, H ρ ) → 0 , whilst no consistent tests exist when ρ 2 /lessmuch √ d/n . That is, for effect sizes of smaller order than

√ d/n , the null hypothesis cannot be consistently distinguished from the alternative hypothesis. Such a testing rate is attainable through a chi-square test based on ‖ √ nX (1) ‖ 2 2 (see e.g. [7]).

In case of m trials, if the full data were pooled (with aggregated sample size nm ), the minimax separation rate would be √ d/ ( mn ) . However, pooling the data might not be possible or allowed in practice and often only real-valued test statistics are available that describe the significance in the local problems (e.g. a p- or an e-value). These m test statistics S ( j ) , j = 1 , ..., m , then can be combined with some combination function C m : R m → R , providing the test statistic in the meta-analysis. We now ask whether the above pooled testing rate is attainable with this meta-analysis procedure.

Without any restrictions on the test statistics S = ( S (1) , . . . , S ( m ) ) or the combination function C m , any of the conventional optimal 'full-data' tests can be reconstructed, since the real numbers and mappings between the real numbers form an overly rich class. We wish to restrict our analysis to S and C m that are reasonable in practice and capture (most of) the relevant meta-analysis methods as listed in Section 3.

Based on each of the local observations X ( j ) , a real-valued test statistic S ( j ) is computed, where each S ( j ) is a function of X ( j ) and possibly a source of randomness U ( j ) independent of X := ( X (1) , . . . , X ( m ) ) .

Assumption 1. For measurable functions f j : R d × R → R and independent random variables U (1) , . . . , U ( m ) which are independent of the data X , the j -th test statistic S ( j ) = f j ( X ( j ) , U ( j ) ) satisfies E 0 | S ( j ) | ≤ M , for some M > 0 , j = 1 , . . . , m .

We consider Hölder continuous combination functions C m : R m → R . Arguably, this is the most important assumption in ruling out bijections between R d and R . This ensures that a small change in the underlying local test statistics cannot result in a large change in the combination of test statistics C m ( S (1) , . . . , S ( m ) ) .

Assumption 2. There exist L, p, q > 0 such that for all s, s ′ ∈ R m

The special case of p = 2 and q = 1 / 2 leads to Lipschitz continuous functions. Assumption 1 and Assumption 2 should be considered in conjunction. By rescaling and centering test statistics S ( j ) , one can typically obtain test statistics satisfying Assumption 1. Rescaling and centering typically does affect how the test statistics need to be combined, which might 'break' Assumption 2.

Finally, following the standard testing approach, we compare the aggregated test statistics C m ( S (1) , . . . , S ( m ) ) to a threshold value. If the combined test statistics result in a large enough value, the null hypothesis of no effect is rejected. We note here that two sided tests can be written as one-sided tests through straightforward transformations (e.g. centering and taking absolute value). More formally, we consider tests T α of level α satisfying the following assumption.

Assumption 3. There exists a strictly decreasing function α ↦→ κ α so that satisfies P 0 T α ≤ α .

The map α ↦→ κ α could be taken as the quantile function of C m ( S (1) , . . . , S ( m ) ) under its null distribution if it is appropriately standardized. If E 0 C m ( S (1) , . . . , S ( m ) ) is bounded in m , we can choose κ α equal to 1 / α times the upper bound, in view of Markov's inequality.

Our first main result, Theorem 1 below, establishes a lower bound for tests of the form (4) and C m and S satisfying the above assumptions. More concretely, under our assumptions, any test T α (of level α ≤ 0 . 1 ) has large Type II-error under alternatives with ρ 2 of smaller order than ( √ m ∧ d log( m ) ) √ d/ ( mn ) . When the number of trials is small compared to the dimension (i.e. m log 2 ( m ) ≤ d 2 ), this means that the separation rate is at least √ d/ ( √ mn ) . Thus even though there is a benefit in terms of separation rate compared to testing based on just a single trial, the

gain is at best the square root of what one would gain based on testing on the pooled data. When m log 2 ( m ) ≥ d 2 , the rate in the lower bound changes to d √ d/ ( mn log( m )) , resulting in an elbow effect.

Theorem 1. Let S (1) , . . . , S ( m ) , C m and T α satisfy Assumptions 1-3 with T α of level α ∈ (0 , 0 . 1] . Then there exists a constant c > 0 depending only on L , p, q and M , such that if

Remark 1 . The ranges of values 0 < α ≤ 0 . 1 and β = 3 / 4 for the Type I and II errors, respectively, are arbitrary. Similar results hold for different choices as well. For instance, one can take arbitrary α ∈ (0 , 1 / 5] and β ∈ (0 , 2 / 3] , see the proof of the theorem for details. The result implies in particular that consistent testing is not possible for signals of a smaller order than the right hand side of (5), where asymptotics can be considered in n, m and d simultaneously.

In the next section we show that the lower bounds in the theorems above are sharp (up to a logarithmic factor).

2.1 Rate optimal combination methods

To attain the lower bound rate derived in Theorem 1, different tests can be considered. The optimal rate displays an elbow effect around m /equivasymptotic d 2 . When the dimension is large compared to the number of trials m (i.e. m /lessorsimilar d 2 ), strategies that combine p-values for the optimal local tests (based on ‖ √ nX ( j ) ‖ 2 2 ∼ H 0 χ 2 d ), turn out to achieve the optimal rate, as exhibited below. Such a test statistic is invariant to the directionality of X ( j ) and invariant under the model in the sense that the resulting power for the alternative P f or P g is the same as long as ‖ f ‖ 2 = ‖ g ‖ 2 .

On the other hand, when the dimension is small compared to the number of trials (i.e. m /greaterorsimilar d 2 ), optimal strategies exhibited below use information on the direction of X ( j ) . In fact, we show in Theorem 4 in the Appendix that if no such information is available (i.e. the events defined by the signs of the ( X ( j ) ) j =1 ,...,m vector are not contained in the sigma algebra generated by the test statistics S ), one cannot obtain a rate better than √ d/ ( √ mn ) . This implies that by combining the locally optimal test statistics S ( j ) = ‖ √ nX ( j ) ‖ 2 2 (or their arbitrary functions, e.g. the corresponding local p-values) would result in information loss and hence sub-optimal rates in the meta-analysis.

Furthermore, it turns out, in accordance with the empirical literature discussed in the introduction, that there does not exist a uniquely best meta-analysis method. In fact, multiple standard meta-analysis techniques provide (up to a logarithmic factor) optimal rates, see below for some standard approaches attaining the lower bounds derived in Theorem 1.

First we consider the scenario when the dimension d of the model is large compared to the number of trials m , i.e. m /lessorsimilar d 2 . Locally the optimal test is based on the test statistic ‖ √ nX ( j ) ‖ 2 2 H 0 ∼ χ 2 d . A natural way to combine these statistics would be to sum these locally optimal test statistics to obtain

which has level α . Alternatively, one could also apply p-value combination methods, such as Fisher's or Edgington's method based on the p-value p ( j ) = 1 -F χ 2 d ( ‖ √ nX ( j ) ‖ 2 2 ) , see Section 3. Lemma 6 in the appendix establishes that these tests are rate optimal.

Second, consider the case that the number of trials is large compared to the dimension, i.e. m /greaterorsimilar d 2 . Rate optimal tests can be constructed based on a variation of Edgington's or Stouffer's method, see Section 3 for their descriptions. Taking a partition of { 1 , . . . , m } = ∪ d i =1 J i where |J i | /equivasymptotic m/d and it holds for all n, m, d ∈ N that

setting S ( j ) = √ nX ( j ) i if j ∈ J i , the meta-level test achieves the lower bounds. The above test is similar to employing Stouffer's method for each of the coordinates and averaging, i.e. computing approximately m/d iid p-values p ( j ) = Φ ( √ nX ( j ) i ) for j ∈ J i and applying the inverse Gaussian CDF Φ -1 ( p ( j ) ) . Alternatively, the following variation of Edgington's method,

is also rate optimal, as proven in Lemma 7 in the appendix. Essentially, these strategies divide the trials accross the d different directions, and combines the evidence for each of the directions. Theorem 4 affirms that the information on the 'direction' of the data is crucial to achieve the optimal rate in the m /greaterorsimilar d 2 case, by showing that strategies that do not contain such information (rotationally invariant strategies such as norm-based test statistics) achieve the rate √ d/ ( √ mn ) at best. We summarize the above testing upper bounds in the theorem below.

Theorem 2. For all α , β ∈ (0 , 1) there exist S , C m : R m → R and tests T α of level α satisfying Assumptions 1-3 such that if we have

for a large enough constant C α , β > 0 depending only on α , β ∈ (0 , 1) , for all n, m, d ∈ N .

2.2 Benefits of coordination between the trials

When the dimension is small relative to the number of trials, as exhibited in the previous section, optimal strategies include information on the directionality of the observation vector. In this section we show that in this regime, there could be an additional benefit from allowing mild coordination between the trials through employing shared randomness, e.g. a shared random seed between the trials. Such a phenomenon has been observed before in the distributed testing literature [3, 2, 39, 40], which forms the basis of our analysis below.

We consider the following variation on Assumption 1, where the key difference is that the source of randomness is allowed to be shared between the m trials.

Assumption 4. For functions f j : R d × R → R and a random variable U which is independent of the data X , the j -th test statistic S ( j ) = f j ( X ( j ) , U ) satisfies E 0 | S ( j ) | ≤ M for some M > 0 and all j = 1 , . . . , m .

Test statistics satisfying this assumption shall be referred to as shared randomness (or public coin) protocols.

The theorem below establishes the optimal rate when coordination through shared randomness is allowed. When the number of trials is small compared to the dimension (i.e. m /lessorsimilar d/ log m ), there is no difference between protocols that coordinate using shared randomness or those without coordination. In fact, the optimal rate ( ρ 2 /equivasymptotic √ d/ ( √ mn ) ) in this case is reached by the test (7) or the ones below it, which do not employ shared randomness. However, when the number of trials is large compared to the dimension (i.e. m /greaterorsimilar d ), the testing rate substantially improves in the shared randomness protocols.

Theorem 3. Let S (1) , . . . , S ( m ) , C m and T α satisfy Assumptions 2-4. Then there exists a constant c > 0 depending only on L , p, q and M , such that if

it holds that sup f ∈ H ρ P f ( T α = 0) > 2 / 3 for all n, m, d ∈ N and any level α ∈ (0 , 0 . 1] .

At the same time, for all α , β ∈ (0 , 1) there exists a constant C α , β > 0 depending only on β , L , p, q , the function α ↦→ κ α and M , such that if

it holds that sup f ∈ H ρ P f ( T α = 0) ≤ β for some test T α of level α satisfying Assumptions 2-4.

Remark 2 . Similarly to Theorem 1 the choice of ranges 0 < α ≤ 0 . 1 and β = 2 / 3 in the lower bound result is arbitrary, other choices are also possible as presented in the proof.

A shared randomness method that attains the rate in (12) is given next. Consider drawing an orthonormal d × d matrix U taking values from the uniform measure on such matrices. As a test statistic, each trial computes ( √ nUX ( j ) ) 1 , which is a N (0 , 1) random variable under the null hypothesis. A level α ∈ (0 , 1) meta-level test is then given by combining the local test statistics as where Φ is the standard Gaussian CDF. The core idea here is that for each trial, the same 1 -dimensional projection of the d -dimensional data is computed, where the projection is taken uniformly at random and the test is conducted along the projected direction. The above method corresponds to Stouffer's method for the p-values p ( j ) = Φ ( √ n ( UX ( j ) ) 1 ) for j = 1 , . . . , m . Lemma 8 in the appendix shows that the above test attains a small Type II error probability whenever ρ 2 /greaterorsimilar d/ ( mn ) .

3 Examples for meta-analysis methods

Combinations of independent test statistics that fall into the framework of Assumptions 1- 4 are subject to the rate optimality theory established by the main theorems in Section 2. In this section, we look into common methods for combining p-values, e-values and other test-statistics, as mentioned in the introduction.

When the distribution under the null hypothesis of the test statistics are known, certain combinations are natural. For example, the sum of normal or chi-square test statistics is again normal or chi-square distributed, respectively. Similarly, voting based mechanisms typically rely on summing Bernoulli random variables. It is easy to see that these and similar combinations methods fall into the framework of Assumptions 1-4.

For more specific test statistics, such as p-values or e-values, many general combination methods have been introduced in the literature. We cover some of the most prominent combination approaches for p-values and e-values in Section 3.1 and Section 3.2, respectively. The list of methods is certainly non-exhaustive and many more combination methods exist, but they serve as context for the range of techniques covered by our general theory. Our main results allow establishing lower bound rates for the ones listed below, whilst in Sections 2.1 and 2.2 attainability of these rates by some of the listed methods was exhibited.

3.1 Combinations of p-values

If p (1) , . . . , p ( m ) are p-values obtained from m independent test statistics concerning the same hypothesis, then under the null p ( j ) ∼ iid U (0 , 1) . One can aim to combine the m p-values to form a test T α ≡ T α ( p (1) , . . . , p ( m ) ) with Type I error probability α , which hopefully has higher power than a test based on one of the individual p-values. Below we list standard methods in the literature.

Fisher's method [18]. Because the variables -2 log p ( j ) 's are iid χ 2 2 -distributed under the null hypothesis, their sum follows a χ 2 2 m -distribution. Therefore the combination method ∑ m j =1 -2 log p ( j ) results in a χ 2 2 m distributed random variable, and the corresponding quantile function provides levelα one-sided tests at the meta-level.

Similar flavour to Fisher's method are the combinations ∑ m j =1 -log(1 -p ( j ) ) (Pearson's method [32]), ∑ m j =1 -log p ( j ) (1 -p ( j ) ) (the logit method / Mudholkar and George method [31]) and m -1 / 2 ∑ m j =1 ( p ( j ) -1 / 2) (Edgington's method [14]).

Order-based methods such as Tippett's method [42] based on min { p (1) , . . . , p ( m ) } H 0 ∼ Beta (1 , m ) .

Methods based on inverse CDF's, such as by Stouffer et. al [36] based on m -1 / 2 ∑ m j =1 Φ -1 ( p ( j ) ) ∼ N (0 , 1) under the null hypothesis.

Generalized averages as considered in [48], T α = { a r,m M r,m ( p (1) , . . . , p ( m ) ) ≤ α } , where M r,m ( p (1) , . . . , p ( m ) ) equals ( m -1 ∑ m j =1 ( p ( j ) ) r ) 1 /r for r ∈ R \ { 0 } , the geometric mean, minimimum (i.e. Tippett's method) and maximum for r = 0 , r →-∞ , and r →∞ , respectively. For r ∈ { -∞ } ∪ [1 / ( m -1) , ∞ ] , a r,m can be taken to obtain precisely level α tests (i.e. P 0 T α = α ). We note that this means that canonical multiple testing methods (see e.g. [19]) such as Bonferroni's correction (which corresponds with taking as M r,m the minimum and a r,m = m ) also fall within our framework.

Lemma 1 below shows that all the methods mentioned above fall into the framework of Assumptions 1-4. This means that the error rate lower bounds of Theorem 1 and Theorem 3 respectively, apply to the p-value combination methods listed above. That is, one cannot attain a better separation rate when considering the worst case Type II error probability for the alternative hypothesis in (2), with any of the p-value combination methods listed above. Whether Assumption 1 or 4 applies depends on whether shared randomness is used in generating the p-values. To confirm that Assumptions 3 and 2 apply to tests based on the combined p-values, some algebra is needed. The proof of the lemma is deferred to the appendix.

Lemma 1. Consider p-values p (1) , . . . , p ( m ) , where each p ( j ) depends on the local data X ( j ) and possibly local randomness that is independent of the data. For each of the combination methods for p-values mentioned above and corresponding test T α of level α ∈ (0 , 1) , the conclusions of Theorem 1 holds.

We remark that the p-values are obtained using shared randomness (i.e. in the sense of Assumption 1), the lower bound rate of Theorem 3 applies. Furthermore, as exhibited in Sections 2.1 and 2.2, for p-values corresponding to well chosen test statistics, these combination methods can achieve the theoretical limits established in Theorems 1 and 3, respectively.

3.2 Combining e-values

An e-value is a nonnegative random variable E such that sup P 0 ∈ H 0 P 0 E ≤ 1 . The threshold test corresponding to E of level α is { E ≥ α -1 } . This test yields a so called strict p-value; for P 0 ∈ H 0 we have P 0 ( E ≥ α -1 ) ≤ α by Markov's inequality.

E-values lend themselves for combining outcomes of independent studies for two main reasons. First, they are easy to combine, see Section 4 in [49] for an indepth discussion of specific combination functions for independent e-values. Second, they are robust to misspecification and offer optional stopping/continuation guarantees [22]. Common examples of e-values are Bayes factors and likelihood ratios, which are nonnegative and have expectation equal to 1 in the case of a simple null hypothesis such as considered in this article.

Several combination methods (e-merging functions) were proposed in the literature. For instance, the product of independent e-values is also again an e-value. This was shown to weakly dominate any other combination of independent e-values in the sense that Π j m =1 E ( j ) ≥ C m ( E ) , for any E = ( E ( j ) ) ∈ [1 , ∞ ) m and E ↦→ C m ( E ) such that C m ( E (1) , . . . , E ( m ) ) is an e-value for any independent e-values E (1) , . . . , E ( m ) , see [49]. Similarly, the average of e-values is again an evalue. The product and the average are admissible in the sense that there is no e-merging function that strictly dominates them on [0 , ∞ ] m . The lemma below shows that these two, arguably most prominent e-value combination methods fulfill Assumptions 1- 4 and hence the lower bounds derived in Theorems 1 and 3 apply.

If C m is the product, assume in addition that E 0 | log E ( j ) | is uniformly bounded. Then, the conclusion of Theorem 1 holds. In case the e-values are generated using shared randomness, then Theorem 3 applies.

4 Simulations

In this section, we investigate the numerical performance of the testing strategies outlined in Section 2.1 on synthetic data sets. We compare the tests based on their receiver operating characteristic (ROC) curve. For a range of significance levels we compute for each tests the 'true positive rate' (TPR) and 'false positive rates' or (FPR), i.e. the fraction of the simulation runs in which the test correctly identifies the underlying signal, falsely rejects the null hypothesis, respectively. Plotting the TPR against the FPR (both given as a function of the significance level) provides us the ROC curve, visualizing the diagnostic ability of the test.

1.00

1.00

True Positive Rate

0.75

True Positive Rate

0.75

0.50

0.50

0.25

0.25

Curve

0.00

0.00

identity

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

一

False Positive Rate

False Positive Rate

directional coordinated

directional uncoordinate

1.00

1.00

single trial

True Positive Rate

0.75

True Positive Rate

0.75

0.50

0.50

0.25

0.25

0.00

0.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

False Positive Rate

False Positive Rate

In our simulations we set m = 20 , n = 30 , let d range from 2 to 20 and take ρ 2 = √ d/ (4 n ) . This value of ρ 2 corresponds to a signal that is almost indistinguishable from noise using just a single trial, whilst consistently detectable if the data were to be pooled with m ≈ 20 (which increases the signal size to noise ratio effectively by a factor √ 20 > 4 . For each level α ∈ { 0 . 01 , 0 . 02 , . . . , 0 . 99 } we compute the power for different combination strategies 100 times, each time drawing a different f ∈ R d with ‖ f ‖ 2 = ρ according to f i = d -1 / 2 ρ R i and R i iid Rademacher random variables for i = 1 , . . . , d . As combination strategies, we compare the strategies (7), (13) and (8) from Section 2,

which are called 'chi-square combined', 'coordinated directional' and 'uncoordinated directional' in the legend of Figure 4. In addition, we display the ROC curves for the chi-square test based on pooled data ('chi-square pooled') and that of a single trial ('single trial').

We make the following observations, in line with our theoretical findings. The meta-analysis methods based on combining the locally optimal chi-squared test statistics (yellow curves) substantially outperformed the chi-squared test statistics based on a single trial (blue curve), but was substantially worse than the chi-square test based on the pooled data (pink curve). Second note that the large dimensional case ( d = 10 and d = 20 ) the best strategy is indeed to combine the local chi-square statistics (yellow curve), while in the low dimensional setting ( d = 2 ) it is more advantageous to combine the directional test statistics X ( j ) i (blue curve). Finally, note that allowing coordination between the trials by a shared randomness protocol can result in improved performance (green curve) compared to the independent experiments (blue curve). In fact this approach provides the best meta-analysis method in the small dimensional setting (e.g. d = 2 and d = 5 for small α , which is the most interesting case).

In the appendix, Section A.6, we explore eight additional simulation settings, where we consider larger values of d and m . Whilst these simulations do not reveal additional phenomena to the ones observed in Figure 4, they do give insight into the relative performance of the testing methods for different values of d and m .

5 Discussion

We briefly summarize our main contributions and discuss possible extensions and research directions. First, by establishing a connection between meta-analysis and distributed learning under communication constraints, we have provided a unified, theoretical framework for evaluating the behaviour of standard meta-analysis techniques. In our analysis, we considered the many normal means model, but these results can be extended to other more complex models as well, building on the connection with distributed computation. For example, minimax estimation rates under communication constraints were derived for other parametric models [53], density estimation [8], signal-in-Gaussian-white-noise [54, 38, 10], nonparametric regression [37] and in abstract settings [52] including binary and Poisson regression, drift estimation, and more. The normal means model allows for a tractable analysis, but results in this model are known to extend to more complicated models, such as discrete density testing (see e.g. [11]). With the due technical work, our results are expected to translate to these settings as well, but we leave this for future endeavor.

In the normal means model we show that by combining the locally optimal chi-square statistics at a meta-level one can gain a factor of √ m compared to using a single trial. Nevertheless, regardless of the choice of the combination method, a factor of √ m is lost compared to the scenario when all data from all trials are at our disposal. This loss is clearly visible even in small sample sizes, dimensions and trial numbers, as demonstrated in our numerical analysis, as can be seen in the corresponding ROC curves. For more complex models, such a numerical study can be a first step to quantify the efficiency of the meta-analysis method. We have also shown that in the small dimension - large number of trials setting combining the locally optimal chi-square statistics (or any rotationally invariant statistics for that matter) results in information loss and sub-optimal accuracy. In this case, better rates can be attained by test statistics based on the direction of the observations combined at the meta-level. In practice, one often cannot choose which test statistics can be obtained from independent trials. In such cases, the √ m -factor loss in the case of e.g. rotationally invariant test statistics is of interest when considering power calculations. Meta-analysis approaches based on directional test statistics are designed for scenarios where individual datasets are not centrally collected, but there is some level of coordination among experimenters.

The assumption throughout the paper of homogeneity between the trials (i.e. each trial consisting of the same number of observations) simplifies the presentation, but the results can be extended to cases where the number of observations in each trial differ by constant factors. Situations where the number of observations differs greatly (e.g. k /lessmuch m trials have as much observations as the other m -k trials combined) are certainly of interest, but beyond the scope of this paper.

Acknowledgements: Co-funded by the European Union (ERC, BigBayesUQ, project number: 101041064). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This research was partially funded by a Spinoza grant of the Dutch Research Council (NWO).

References

ABRAMOVICH, F., BENJAMINI, Y., DONOHO, D. L., AND JOHNSTONE, I. M. Adapting to unknown sparsity by controlling the false discovery rate. The Annals of Statistics (2006), 584-653.

ACHARYA, J., CANONNE, C. L., AND TYAGI, H. Distributed signal detection under communication constraints. In Conference on Learning Theory (2020), PMLR, pp. 41-63.

ACHARYA, J., CANONNE, C. L., AND TYAGI, H. Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction. IEEE Transactions on Information Theory 66 , 12 (Dec. 2020), 7835-7855.

AERTS, S., LAMBRECHTS, D., MAITY, S., VAN LOO, P., COESSENS, B., DE SMET, F., TRANCHEVENT, L.-C., DE MOOR, B., MARYNEN, P., HASSAN, B., ET AL. Gene prioritization through genomic data fusion. Nature biotechnology 24 , 5 (2006), 537-544.

ALTMAN, E. I. Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. The journal of finance 23 , 4 (1968), 589-609.

ALTMAN, E. I., IWANICZ-DROZDOWSKA, M., LAITINEN, E. K., AND SUVAS, A. Distressed firm and bankruptcy prediction in an international context: A review and empirical analysis of altman's z-score model. Available at SSRN 2536340 (2014).

BARAUD, Y. Non-asymptotic minimax rates of testing in signal detection. Bernoulli (2002), 577-606.

BARNES, L. P., HAN, Y., AND ÖZGÜR, A. Lower bounds for learning distributions under communication constraints via fisher information. The Journal of Machine Learning Research 21 , 1 (2020), 9583-9612.

BIRNBAUM, A. Combining independent tests of significance. Journal of the American Statistical Association 49 , 267 (1954), 559-574.

CAI, T. T., AND WEI, H. Distributed nonparametric function estimation: Optimal rate of convergence and cost of adaptation. The Annals of Statistics 50 , 2 (2022), 698-725.

CARTER, A. V. Deficiency distance between multinomial and multivariate normal experiments. The Annals of Statistics 30 , 3 (2002), 708 - 730.

CHEN, Z. Optimal tests for combining p-values. Applied Sciences 12 , 1 (2021), 322.

CLEMENTS, N., SARKAR, S. K., AND GUO, W. Astronomical transient detection controlling the false discovery rate. In Statistical challenges in modern astronomy V (2012), Springer, pp. 383-396.

EDGINGTON, E. S. An additive method for combining probability values from independent experiments. The Journal of Psychology 80 , 2 (1972), 351-363.

EFRON, B. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction , vol. 1. Cambridge University Press, 2012.

EVANGELOU, E., AND IOANNIDIS, J. P. Meta-analysis methods for genome-wide association studies and beyond. Nature Reviews Genetics 14 , 6 (2013), 379-389.

FINNER, H., AND ROTERS, M. Log-concavity and inequalities for chi-square, f and beta distributions with applications in multiple comparisons. Statistica Sinica (1997), 771-787.

FISHER, R. A. Statistical methods for research workers. In Breakthroughs in statistics . Springer, 1992, pp. 66-70.

FROMONT, M., LERASLE, M., AND REYNAUD-BOURET, P. Family-Wise Separation Rates for multiple testing. The Annals of Statistics 44 , 6 (2016), 2533 - 2563.

GINE, E., AND NICKL, R. Mathematical Foundations of Infinite-Dimensional Statistical Models . Cambridge University Press, Cambridge, 2016.

GOOD, I. On the weighted combination of significance tests. Journal of the Royal Statistical Society: Series B (Methodological) 17 , 2 (1955), 264-265.

GRÜNWALD, P., DE HEIDE, R., AND KOOLEN, W. M. Safe testing. In 2020 Information Theory and Applications Workshop (ITA) (2020), pp. 1-54.

GUGLIELMETTI, F., FISCHER, R., AND DOSE, V. Background-source separation in astronomical images with bayesian probability theory-i. the method. Monthly Notices of the Royal Astronomical Society 396 , 1 (2009), 165-190.

HOSPEDALES, T., ANTONIOU, A., MICAELLI, P., AND STORKEY, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence 44 , 9 (2021), 5149-5169.

INGSTER, Y. I. Minimax testing of nonparametric hypotheses on a distribution density in the $l_p$ metrics. 333-337. Number: 2.

JOHNSTONE, I. M., AND SILVERMAN, B. W. Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences. The Annals of Statistics 32 , 4 (2004), 1594 1649.

KRÄMER, A., GREEN, J., POLLARD JR, J., AND TUGENDREICH, S. Causal analysis approaches in ingenuity pathway analysis. Bioinformatics 30 , 4 (2014), 523-530.

LIPTÁK, T. On the combination of independent tests. Magyar Tud Akad Mat Kutato Int Kozl 3 (1958), 171-197.

LOUGHIN, T. M. A systematic comparison of methods for combining p-values from independent tests. Computational statistics & data analysis 47 , 3 (2004), 467-485.

MALONE, B. M., TAN, F., BRIDGES, S. M., AND PENG, Z. Comparison of four chip-seq analytical algorithms using rice endosperm h3k27 trimethylation profiling data. PloS one 6 , 9 (2011), e25260.

MUDHOLKAR, G. S., AND GEORGE, E. The logit statistic for combining probabilities-an overview.

PEARSON, K. On a new method of determining" goodness of fit". Biometrika 26 , 4 (1934), 425-442.

QUACKENBUSH, J. Microarray data normalization and transformation. Nature genetics 32 , 4 (2002), 496-501.

SHAFER, G., ET AL. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society: Series A (Statistics in Society) 184 , 2 (2021), 407-431.

SHAFER, G., SHEN, A., VERESHCHAGIN, N., AND VOVK, V. Test Martingales, Bayes Factors and $p$-Values. Statistical Science 26 , 1 (Feb. 2011), 84-101. arXiv: 0912.4269.

STOUFFER, S. A., SUCHMAN, E. A., DEVINNEY, L. C., STAR, S. A., AND WILLIAMS JR, R. M. The american soldier: Adjustment during army life.(studies in social psychology in world war ii), vol. 1.

SZABÓ, B., AND VAN ZANTEN, H. Adaptive distributed methods under communication constraints. The Annals of Statistics 48 , 4 (2020), 2347 - 2380.

SZABÓ, B., AND VAN ZANTEN, H. Distributed function estimation: Adaptation using minimal communication. Mathematical Statistics and Learning 5 , 3 (2022), 159-199.

SZABÓ, B., VUURSTEEN, L., AND VAN ZANTEN, H. Optimal distributed composite testing in high-dimensional gaussian models with 1-bit communication. IEEE Transactions on Information Theory 68 , 6 (2022), 4070-4084.

SZABÓ, B., VUURSTEEN, L., AND VAN ZANTEN, H. Optimal high-dimensional and nonparametric distributed testing under communication constraints. The Annals of Statistics 51 , 3 (2023), 909-934.

THOMAS, J. G., OLSON, J. M., TAPSCOTT, S. J., AND ZHAO, L. P. An efficient and robust statistical modeling approach to discover differentially expressed genes using genomic expression profiles. Genome Research 11 , 7 (2001), 1227-1236.

TIPPETT, L. H. C., ET AL. The methods of statistics. an introduction mainly for experimentalists. The methods of statistics. An introduction mainly for experimentalists. (1941).

TSYBAKOV, A. B. Introduction to nonparametric estimation . Springer series in statistics. Springer, New York ; London, 2009. OCLC: ocn300399286.

VAART, A. W. V. D. Asymptotic statistics . Cambridge series in statistical and probabilistic mathematics. Cambridge University Press.

VAN ZWET, W., AND OOSTERHOFF, J. On the combination of independent test statistics. The Annals of Mathematical Statistics 38 , 3 (1967), 659-680.

VERSHYNIN, R. High-Dimensional Probability: An Introduction with Applications in Data Science , 1 ed. Cambridge University Press, Sept. 2018.

VOVK, V., WANG, B., AND WANG, R. Admissible ways of merging p-values under arbitrary dependence. The Annals of Statistics 50 , 1 (2022), 351-375.

VOVK, V., AND WANG, R. Combining p-values via averaging. Biometrika 107 , 4 (2020), 791-808.

VOVK, V., AND WANG, R. E-values: Calibration, combination and applications. The Annals of Statistics 49 , 3 (2021), 1736-1754.

WHITLOCK, M. C. Combining probability from independent tests: the weighted z-method is superior to fisher's approach. Journal of evolutionary biology 18 , 5 (2005), 1368-1373.

YOON, S., BAIK, B., PARK, T., AND NAM, D. Powerful p-value combination methods to detect incomplete association. Scientific reports 11 , 1 (2021), 6980.

ZAMAN, A., AND SZABÓ, B. Distributed nonparametric estimation under communication constraints. arXiv preprint arXiv:2204.10373 (2022).

ZHANG, Y., DUCHI, J., JORDAN, M. I., AND WAINWRIGHT, M. J. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. Advances in Neural Information Processing Systems 26 (2013).

ZHU, Y., AND LAFFERTY, J. Distributed nonparametric regression under communication constraints. In International Conference on Machine Learning (2018), PMLR, pp. 6009-6017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n[Update1: During the rebuttal, I updated my score from 5 to 6. The reason is that I want to stronger weigh in that the paper is technically solid. My concern whether NeurIPS is a perfect fit for this paper remains, and I ask the AC to judge that part.]\nThe paper studies aggregation strategies for combining test statistics obtained from independent experiments. It is intuitively clear that the optimal strategy would be to aggregate the whole *data* and then compute one test statistic on the entirety of the data. However, the authors make the assumption that from each experiment one can only use a single real number (aka test statistic) for the final analysis.\n\nThe setting they consider is that they have observations of random variables \n\n$$\nX^{(j)} = f + \\frac{1}{\\sqrt{n}} Z^{(j)},\n$$\nwhere $Z^{(j)}$ are independent $d$ dimensional normals and $n\\in \\mathbb{N}$ and additional parameter. The null hypothesis is $f=0$ which is tested against $\\|f\\|_2 \\geq \\rho$ for some $\\rho >0$. If pooling the data was allowed, the chi-square test is optimal for this setting.\n\nThey derive theoretical results that give two different regimes:\n\n- When $m \\lesssim  d^2$ the optimal rate can be achieved by combining the individually optimal test statistics $\\|\\sqrt(n) X^{(j)}\\|_2^2$, which is undirected.\n\n- When $m \\gtrsim d^2$ then taking the directionality into account leads to better rates. Two examples they discuss are (phrasing in my own words that the authors might adapt):\n  - Split the observations into $d$ groups of size $\\sim m/d$ and test with the $i$-th group whether the null hypothesis holds for the $i$-th dimension. Then combine those.\n  - Choose a 1-dimensional projection of the $d$-dimensional data uniformly at random. Aggregate all observations along this projections (now these are 1-dimensional values, as required in the setting) and combine those directly to test the null hypothesis along the projected direction. (Note that this requires shared randomness between all experiments).\n\nFinally the authors demonstrate their theoretical findings with toy experiments and numerically confirm their findings.\n\nOverall I enjoyed reading the paper and was able to learn something new. Thank you to the authors.\n\nSTRENGTHS:\n- The paper is extremely well written and has \"textbook\" quality. It focuses on a simple toy problem and provides an exhaustive analysis thereof.\n- Combining outcomes from different experiments is certainly an important problem in statistics.\n- The paper seems original, however, admittedly I am unable to judge to what extent similar analyses exist in the stats literature.\n- The theory and experiments are in perfect accordance and complement each other.\n- The paper provides a good overview of existing methods to combine test statistics and discusses both $p$- and $e$-values as examples. They relate existing methods to their contribution.\n\nWEAKNESSES:\n1. My first point concerns the significance of the approach. In the introductions the paper states\n  > Given multiple data sets relating to the same hypothesis, one would like to combine the evidence.\n Sometimes, the full data sets are not available (e.g. due to privacy or proprietary reasons) or difficult\n to combine directly (e.g. due to the different experimental or observational setups). In such cases,\n the analysis must be carried out on the basis of the published results for each of the studies.\n\n    - This motivation reads that each study publishes a test statistic without knowing of the others. I thus think that the second type of approaches the authors provide does not fall in this category.\n    - The second type of approaches require that the \"meta-analyser\" can make some choices of what statistic the individual studies report. Hence *somewhere* all data must still be available. It is unclear why we cannot use that then. I think this needs more motivation.\n    - I presume that the first approach of combining the undirected tests has been studied exhaustively.\n\n2. While the paper is relevant for machine learning in general, it is in itself a very statistical paper. No learning involved etc. Hence, I am posing the question whether NeurIPS is the appropriate venue. I would see a stats journal as a better fit.\n\n3. I think the paper could be a bit more accessible if a bit more intuition about the approaches is provided. I wrote my understanding in the summary, maybe the authors want to correct/modify this and include something like that.\n\nQUESTIONS:\nPlease comment my first two concerns above in the rebuttal. Note that my reservation against acceptance is based on those comments. And I am happy to increase my score after the rebuttal if I am convinced otherwise.\n\n\nMinor:\n- l 124 \"where\" --> \"were\"\n- l. 339 \"form\" --> \"from\"\n\nLIMITATIONS:\nThis paper only studies a simple toy problem.\n\nLimitation of my review: I did not read the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis theory paper provides a minimax lower and upper bounds for the testing risk (sum of Type-I and Type-II errors) for different combination methods in the specific setting of many normal means model. With the testing goal is to detect the presence or absence of the signal component in this normal means model, their results show several combination methods of test statistics (e.g. aggregate p-values and e-values) cannot consitently detect signals below a certain threshold that depend on the number of trials, samples and dimensions of the problem.\n\nSTRENGTHS:\n* The theoretical results are sound and based on several established techniques in distributed testing, assuming several assumptions  hold true. These are proof to be indeed the case in the Appendix for the test statistics combination methods proposed in Section 3 of the main text.\n* The paper is mostly well-written.\n\nWEAKNESSES:\n1. In general I think the phrasing of on the paper's contributions could be make clearer in the last part of the introduction section. \n2. The authors should discuss more on the relevance of the setting -- the many normal means model -- in some more concrete applications. It is arguable that although this is a theoretical paper, the theories inside it is an attempt to quantify a very practical problem of meta learning. I see a lack of evidence for the popularity of many normal means in practice.\n3. Slightly related but not as equally importance, but the authors should have acknowledged that a limitation of their work is that the theoretical results only hold with many normal means model assumption.\n4. Experimental results could include more settings with a variety number of of sizes/dimensions (perhaps in the appendix) to support the theory.\n\nQUESTIONS:\n* I have only skimmed through the proof in the appendix, so this might be my mistake, but I do not see the appearance of the $\\epsilon$ term for binary approximation of the statistics in the main results. Could the authors clarify on this point?\n\nLIMITATIONS:\n* See weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors study a problem of optimal combination of p-values in a meta-analysis context. Specifically, they focused on characterizing the minimax separation rate for a family of \"smooth-ish\" combination methods that aggregate p-values (or e-values). They show that: \n* The family contains a lot of methods that are used in practice.\n* The separation rate for methods in the family is nearly optimal.\n* Optimal combination method depends on the problem setting (sample size, number of p-vals, dimension of the problem); they describe practical consequences. \n\nFurthermore, they explored how the rate can be improved by allowing coordination between the experiments that generate the p-values.\n\nSTRENGTHS:\nThe paper is written in a clear and without excessive statistical jargon. For instance, the concept of separation rate is introduced and explained in a simple and intuitive way, allowing readers to understand the results presented in the paper (in contrast to the mat description found in \"Non-asymptotic minimax rates of testing in signal detection\"). The significance of the results is solidly established on two grounds: derivation of a separation rate and practical guidance for method selection based on n, m, and d (Section 2.2 and 2.3 add extra value).  From my (limited) understanding of the literature, these results are both novel and original.\n\nWEAKNESSES:\nI am genuinely surprised that this problem has not been previously studied. While preparing the review, I came across similar/related results concerning the Family-Wise Error Rate (FWER) in the paper \"Family-Wise Separation Rates for multiple testing.\" However, the combination methods studied are different (Holm–Bonferroni procedure type). \n\nI'd suggest a more comprehensive and thorough discussion of the existing literature on minmax testing for multiple hypothesis testing is added to the paper.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of combining test statistics from multiple independent studies in the context of null-hypothesis significance testing. The authors derive a mathematical framework to quantify the cost of compressing multiple independent trials of a study into one real-valued test statistics, and they derive minimax lower and matching upper bounds for the testing errors. The many normal means models is used as toy example.\n\nSTRENGTHS:\nThe paper addresses the problem of combining the results of multiple empirical studies towards one common hypothesis, an important problem in meta-analysis.\n\nWEAKNESSES:\nThis submission seems to be out of the scope of NeurIPS. While the authors draw a connection between meta-analysis as used in statistics and meta-learning as used in the context of machine learning, this connection is not clarified further. For the rest of the paper, the authors seem to focus on the problem of meta-analysis. \n\nIt was challenging to read to paper as it lacks clarity in the introduction. To improve clarity, I suggest to start the introduction by clearly stating the problem that will be addressed in the paper and clearly introducing the terms used in the text. For example, in line 23, the authors introduce “meta-analysis”, (probably) referring to the technique of combining the results of multiple scientific empirical studies and then set it equal to “meta-learning”, referring to the machine learning technique of improving a learning algorithm to perform well multiple tasks.\n\nI did not go through the technical parts of the paper in all detail. However, there are several statements in the paper that appear to be misleading, e.g., \n\n- line 68: “… includes many standard meta-learning techniques, for instance the standard p-value combination methods […]”. I am not aware of p-value combination being a standard technique in meta-learning.\n- line 313: “Common examples of e-values are Bayes factors and likelihood ratios.” e-values and Bayes factors are closely related, but Bayes factors or likelihood ratios are not e-values, see https://arxiv.org/abs/1912.06116 Appendix A for clarifications.\n\nQUESTIONS:\nCan you clarify to connection between \"meta-analysis\" and \"meta-learning\" (meaning the approach reviewed in your reference number [14]?\n\nLIMITATIONS:\nThe authors do not discuss limitations or potential negative impact of their work. Although being a more technical paper, I think it would have been appropriate to discuss the limitations of the meta-analysis approach as a whole.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper lies in the context of meta-analysis of multidimensional models. Usually, meta-analyses are performed by combining p-values or e-values. In both cases, the statistical power is not well-known. The authors provide a constrained framework of many means normal model. Based on this framework, they derive lower (theorem 1) and upper (theorem 2, for \"rate optimal methods\") bounds for testing methods. Theorem 3 takes advantage of possible shared randomness between trials, especially when the dimension is small relative to the number of trials considered. The authors then compare the performances of rate optimal methods (described in section 2.1), chi-square test on pooled data and single trial approach on simulated datasets.\n\nSTRENGTHS:\nThe paper is well-written and organised. It provides a good overview of state-of-the-art combination techniques for meta-analyses and identifies the lack of knowledge about their relative power. \nThe paper derives bounds for the testing errors by introducing a principled mathematical framework based on multidimensional models, where a loss in power is expected. It also gives insights into rate optimal combination methods and the effects of sharing randomness between trials. The simulation study provides some results on comparing combinations methods for meta-analysis, which is not properly addressed in the current literature.\nAbout the supplementary material: Proofs as well as an R script to reproduce the simulation study are provided.\n\nWEAKNESSES:\nThe main weakness of this paper is the clarity of the mathematical developments. The framework implies several assumptions that could be explained more. The 13-page-long supplementary material provides proof of the theorems but is sometimes hard to follow. \nIn the theorems and their proofs, sometimes arbitrary values are chosen (for $\\alpha$ and $\\beta$ notably) but it seems to make the reasoning more confusing. \n\nAlso, when running the R script, the following error is returned: \nError: object 'dat_long' not found\n\nQUESTIONS:\n- $\\mathbb{E}_0$ is first used in line 141 but not introduced beforehand. Is it the expectation under the null hypothesis? \n- I understand that Assumption 1 aims at restricting the values of S. Is it possible to give a small interpretation of the assumption?\n- Theorem 3 indicates that \"there exists a constant $C_\\beta$\" but the formula indicates \"$C_{\\alpha,\\beta}$\". Is it a typo error? \n- In theorems 1 and 3, arbitrary values are used for $\\alpha$ and $\\beta$, not in theorem 2. Why make the choice of using these values and not giving general results for the corresponding intervals of validity?\n- A similar remark on the proofs, for example in proof A.1. Why use $\\kappa_{1/10}$ and $\\kappa_{1/8}$? \n- It might also be more comprehensible to explicitly add the results taken from the literature and used in the proofs. \n- The provided R script needs to be reviewed. When running it, I get the following error is returned:\n\"Error: object 'dat_long' not found\"\n\nI understand that the chosen mathematical framework provides lower and upper bounds for testing errors. The paper describes some meta-learning techniques that attain these bounds. I am not sure how the simulation study demonstrates the theoretical results. Is it by comparing these meta-learning techniques to the \"chi-squared pooled\" approach and the single trial approach? \nThe indicator of performance for \"optimal testing\" is the ROC curve. Would it be interesting to consider other criteria, such as sensitivity, specificity, precision, or F-score?\n\nOverall, I encourage the authors to add more explanations and interpretations, especially in the mathematical development part. Note that the 9-page limit does not include references. It might also be worth submitting the paper to another journal where the format might be more adequate.\n\nLIMITATIONS:\nThe authors have delimited the framework of their contribution by providing constraints inherent to their model.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers methods to aggregate test statistics from different, independant, sources, in order to construct an aggregated test with hopefully more power. The key contribution of the paper is the study of the minimal treatment effect which can be detected in a standard gaussian noise setting, for which they obtain minimax rates.\nThese rates exhibit an elbow effect when the number of aggregated statistics (m) is close to the square of the dimension of the signal (d^2), which the authors relate to the use of the signal direction in the test (when $m < d^2$, the standard chi2 test would give near optimal power, while for $m > d^2$, the test statistics must encode directional information if optimal power is to be obtained). These rates bring two insights: First, aggregating one dimensional statistics in a multi dimensional setting comes at a price. Second, there is no single optimal aggregating method.\n\nSTRENGTHS:\nOverall, the paper is well presented and obtains conclusive results in the scope considered, in the form of minimax rates. These rates justify previous empirical insights on aggregated testing strategies, notably the need for different aggregating strategies depending on the number of tests and the dimension of the problem. Methods achieving the rates (up to a log factor) are specified. \n\nAs far as I could assess, the mathematical proofs are, up to small typos (see weakness), correct. The presentation of the main results in section 2 can be easily followed (minimax rates in the general case, optimal combination methods then improved minimax rate using coordination between tests).\n\nWEAKNESSES:\nThe proof in the appendix suffers from some small typos. Notably, I believe that in equation (S.1), the $2\\epsilon$ term should be $\\epsilon$ (or $\\epsilon< \\frac{1}{2}\\left(\\kappa_{1/10} - \\kappa_{1/8}\\right)$ in the definition of $\\epsilon$), while in line 538, the conclusion of Markov's inequality is that $D^c$, not $D$, has mass less than $1/64$.\n\nThe methodology used to obtain Figure 1. could be improved. Notably, the Roc Curves for Chi-square combined and Chi-square pooled should not exhibit any randomness, since these two curves can be computed in closed form using the cumulative distribution functions of the chi2 square and non central chi square. If numerical approximations are to be used, it could be possible to obtain curves exhibiting much less noise by increasing the number of repeats and recycling them for all FPR (I could obtain curves exhibiting little to no noise robustly using 10 000 repeats and 100 FDR in less than <2s on my personal computer, so computation time is not an issue).\nMoreover, the way the $f_i$ are drawn, using Rademacher random variables, might have an impact on the directional methods.  While this might or might not be the case, I would suggest recomputing the curve, drawing a random f uniformly on the sphere.\n\nQUESTIONS:\nCould the methodology be extended beyond the current setting? Notably, is there any natural generalisation in the case where the noise level $\\sigma = 1/\\sqrt{n}$ can no longer be assumed to be identical ?\n\nIs there any explanation of the results in terms of the distribution of the p-values under the alternative hypothesis? If so, is there any insight on the best way to aggregate a given set of statistics (instead of considering the best way to aggregate the best statistics for a given m, d)?\n\nLIMITATIONS:\nThe paper derives optimal test aggregation strategies in the context of gaussian noised signals. A first limitation of the paper is that it is assumed that the sample size $n$ considered in each collected test is identical. This can barely be expected in practice, and as such, insight about the impact of uneven tests would be welcome (i.e., does $mn$ translate into $\\sum_{i=1}^m n_i$?, or rather $m \\min(n_i)$?). This issue is not mentionned in the paper.\n\nAnother limitation not mentionned is the fact that, in practice, the test statistics obtained from independent trials are not chosen, but set, and as such, Stouffer's method, which attain the minimax rates when $m>d^2$, is not implementable. In most settings where it could be implemented, the whole $X_i$ information would be known, and therefore the dimension reduction issue would not occur. For this reason, the best rate achievable with realistic one dimensional statistics is of particular interest. Unfortunatly, this is left to the appendix, in Theorem 4.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-31"}