Optimal testing using combined test statistics across independent studies
Authors: Lasse Vuursteen
Botond Szabo
Aad van der Vaart
Harry van Zanten
Venue: NeurIPS 2023 poster (2023)
Keywords: testing
meta-analysis
p-values
e-values
optimal
combining trials
TL;DR: We quantify the cost of compressing real valued test statistics such as p-values from $m$ independent trials concerning the same hypothesis into a single test.

Abstract: Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive a mathematical framework to study standard {meta-analysis} testing approaches in the context of the many normal means model, which serves as the platform to investigate more complex models.

We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us to mathematically quantify the cost of compressing $m$ trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in a sub-optimal {meta-analysis} method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build on recent information theoretic developments in the latter field.

Optimal testing using combined test statistics across independent studies
Lasse Vuursteen"
Botond Szabó
Delft Institute of Applied Mathematics
Department of Decision Sciences and Institute for Data Science and Analytics
Delft University of Technology
1. vwursteendtudellf..cm
Bocconi University
botond, www.bo@@niboccoom....... it
Aad van der Vaart
Harry van Zanten
Delft Institute of Applied Mathematics
Mathematics Department
Delft University of Technology
Vrije Universiteit Amsterdam h. van zanten@vu, nl
wandervaart@tudell.
Abstract
Combining test statistics from independent trials or experiments is a popular method of meta-analysis However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive mathematical framework study standard meta-analysis testing approaches in the context of the many normal means model, which serves the platform to investigate more complex models.
We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us mathematically quantify the of compressing trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e- e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in sub-optimal meta-analysis method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build recent information theoretic developments in the latter field.
Introduction
Given multiple data sets relating the same hypothesis, one would like to combine the evidence. Sometimes, the full data sets are not available (e.g. due to privacy or proprietary reasons) or difficult combine directly (e.g. due to the different experimental observational setups). In such cases, the analysis must be carried out on the basis of the published results for each of the studies. Such "meta- analysis" increase the statistical power by combining individually inconclusive moderately significant tests, while keeping the false positive rate under control. Therefore, meta-analysis has received lot of attention in various fields, for instance in geneties and system biology, when studying rare variants A [6 or in deep learning, for few shot image recognition and neural architecture search,
see the review article 24
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The outcomes of the studies concerning hypothesis tests are, typically, summarized as real-valued test statisties and/or associated p-values. One expects the combination of m such p-values to result in an increase in power, but one also expects to pay a price relative to computing a test on the basis of the full, pooled data of the m trials. The question of how to optimally combine independent real-valued test statistics concerning the same hypothesis into a single test has an extensive literature. A multitude of methods for combining independent tests of significance exist. For combining p- p-values, this starts with Fisher, Tippett and Pearson in the nineteen-thirties, see 28 45
we collect and describe the most popular and frequently used p-value combination techniques.
42 18 32
36 21
47 and references therein. In Section
3
As noted in [2] there does not exist general uniformly most powerful p- combination method for all alternative hypotheses. The distribution of a p-value or its underlying test statistic under the al- ternative hypothesis should be taken into consideration when selecting a method of combination. The performance of different p-value combination techniques was investigated extensively by empirical However, a uni-
experiments in various synthetic and real world scenarios, see for instance 29 51
fied, general theoretical description is lacking, especially in non-trivial, multi-dimensional composite testing problems, where the likelihood ratio test is not necessarily uniformly most powerful.
E-values allow the combination of several tests straightforward manner while preserving the prescribed level of the tests (see Section 3.2). Formally, e-values are nonnegative random variables whose expected values under the null hypothesis are bounded by one. In contrast to p-values defined by probabilities, e-values are defined by expectation. This imposes significant differences in their interpretation, application and combination compared to the more standard p-values. However, as for p-values, very little is known about the power of these combination procedures. Theoretical results focus on specific optimality criteria, for instance the worst-case growth-rate (GROW), see [22 However, these do not directly imply guarantees on the testing power, which is the main focus in practice.
E-values are an increasingly popular and important notion of evidence, see 35
Our focus is on multidimensional models, where a certain loss in power is to be expected, since combining multidimensional data into a real-valued statistic (e.g. p-value or e-value) requires data compression. Typically, summary statistics are combined by some "reasonable" function Gm : R. - R, where "reasonable" means that Cm should not exploit the richness of the real numbers to encode the data in full. We aim to quantify the loss of summarizing, the gain of performing a meta-analysis and the best testing strategies in the individual experiments meta-analysis.
We consider the signal detection problem in the many normal means model, see Section 2 for the detailed description. One possible interpretation of this testing problem is to learn whether a treatment has an effect on any of the dimensions investigated. This model is directly applied in several fields where high- -iheemsiiiinal statistics and machine learning settings are concerned, such as detecting differentially expressed genes [3 27
30
41
15), bankruptcy prediction for
publicly traded companies using Altman's 's Z-score in finance
6 separation of the background
and source in astronomical images [13] 23 and wavelet analysis 11 26l. Furthermore, the model
allows for tractable computations and typically serves as the platform to investigate more difficult statistical and learning problems, including high- and infiniteddimensiona models, see for instance 20) In each experiment f1,...,, m) the observations are summarized by an appropriate real-valued summary statistic 561), These local test statisties (e.g. p- or e-values) are combined into C. (S(1, ........................... ((m)). We consider a general class of combination functions Cm. requiring only Hölder type continuity. This introduces only a mild restriction, and includes many standard meta- analysis techniques, for instance the standard p-vulue combination methods (see Section 3.1 ; e-value techniques (see Section 3.2). ; and other ad hoc and natural test statistic combination approaches, see the beginning of Section for additional examples.
Our setting provides a principled and unified framework to study the power of standard meta-analysis testing methods. Within the framework of the many normal means model, derive minimax lower bound for the testing (separation) error and provide test statistics with associated combination methods that attain this theoretical limit (up to a logarithmic factor). Our results reveal that there is a certain unavoidable loss associated with compressing the data of each experiment to a real valued test statistic. We see that while it is always possible obtain better testing rates using m trials instead of the best possible test based on single trial, there is always a loss incurred when compared to the
2

---

full, pooled data and optimal test in moderate- to large dimensional problems. Our theoretical results quantify these gains and losses in terms of the dimension d, sample size n and number of trials m.
Furthermore, we observe an elbow effect, which occurs when the number of trials is large compared to the dimension of the signal. In this regime, combinations of the (locally) optimal test in each individual trial performs sub-optimally as a whole when aggregated and meta-analysis approaches based on directional test statistics are shown to perform better. Finally, we show that the performance of the meta-level tests can substantially improve (in certain regimes, depending on d, m, n) if a certain amount of coordination between the trials allowed (e.g. by having access to the same random seed). For the theoretical analysis of meta- veeta-anllyiis techniques we derive connections with the distributed statistical learning literature under communication constraints. Our paper builds on the recent information theoretical developments in distributed testing [2] 39 40
allowing us to address several fundamental questions for the first time with mathematical rigor.
The paper is organised as follows. In Section 2 we introduce the mathematical framework we consider in our investigation and present the corresponding minimax testing lower bound results. Next in Subsection 2. we show that the derived results are sharp by providing several meta-analysis approaches attaining the limits. Then we investigate the benefits of allowing a mild coordination
between the trials in Subsection 2.2 We collect and discuss the standard p- and e-value combination
methods in Section 3 and demonstrate our theoretical results numerically on synthetic data sets in g We discuss our results and derive conclusions in Section [ The proofs of our results are deferred to the Appendix. In Section A.T we present the proof of our main results while the proofs of the technical lemmas are given in A.2
Section
Notation: For two positive sequences On bn write On b, if the inequality a, S Cb,, holds for some universal positive constant C. Similarly, we write an x bn if an S b, and b, S an hold simultancously and let an < bn denote that an/bb, = o()). Furthermore, we use the notations V b and a A b for the maximum and minimum, respectively, between a and b. Throughout the paper and C' denote global constants whose value may change from one line to another.
2 Main results
In our analysis, we consider the localized version of the many normal means model, tailored to investigating mell-snalysis techniques. We assume that in each local trial or experiment j E (1,..,m) m) we observe d-dimensional random variable ( Rd. subject to
x-i = f+
1 vn 20)
z Hd N(O, IJ), j j= 1= m,
(1)
for some unknown f E TRd,. We denote by Py the joint distribution of the observations and let E, be the corresponding expectation. We note that this framework is equivalent to having independent N NJ, Ia) observations within each local sample.
Our goal is to test the presence or absence of the "signal component" f E Rd More formally, we consider the simple null hypothesis Ho f = 0 versus composite alternative hypothesis H, +Ifll2 2 p. for some p > 0. This corresponds to testing for joint significance of variables, such as the presence of an effect of a treatment on any of the dimensions investigated. The difficulty in distinguishing the hypotheses depends on the effect size, the sample size and the dimension d. Here, p can be seen as the smallest effect size deemed important.
For {0, 1J-valued test T, define the testing risk R(T, Hp) as the sum of the Type error probability and worst case Type II error probability, i.e.
R(T, Hp) == Po (T 1) + sup Fy (T 0).
SeH
In the case of single trial (i.e. m = 1), this testing problem is known to have minimax separation rate or "detection boundary" p² vd/n.
This means that if p² > vd/n, there exist consisten
tests T Ta,, in the sense that R(T, Hp) -+ 0. whilst no consistent tests exist when p² < vd/n. That is, for effect sizes of smaller order than
For any asymptotics in p, d and n such that p² > vdfn.
3

---

vd/n, the null hypothesis cannot be consistently distinguished from the alternative hypothesis. Such a testing rate is attainable through chi-square test based on (lv (see e.g. (7]).
In case of m trials, if the full data were pooled (with aggregated sample size nm), the minimax separation rate would be vd/(mn). However, pooling the data might not be possible or allowed in practice and often only real-valued test statistics are available that describe the significance in the local problems (e.g. a p or an e-value). These m test statistics 50), j = L,... m. then can be combined with some combination function C. I I. providing the test statistic in the meta-analysis. We now ask whether the above pooled testing rate is attainable with this meta-analysis procedure.
Without any restrictions on the test statistics S = (S()) S(m)) or the combination function Cm- any of the conventional optimal "full-data" tests can be reconstructed, since the real numbers and mappings between the real numbers form an overly rich class. We wish to restrict our analysis to S and C. that are reasonable in practice and capture (most of) the relevant meta-analysis methods as listed in Section g
Based on each of the local observations x real-valued test statistic SG) is computed, where each (() is a function of X() and possibly source of randomness UG) independent of X := (X(1), "il),,...(())) (())..
Assumption 1. For measurable functions fj R.d x TR. - R and independent random variables U()) (mm) which are independent of the data X, the j-th test statistic 5(3) = f satisfies Eo M. for some M >> 0. ....., m.
We consider Hölder continuous combination functions Cm : RR + R. Arguably, this is the most important assumption in ruling bijections between RRd and IR. This ensures that a small change in the underlying local test statistics cannot result in a large change in the combination of test statistics Cm((S)), ((St))....,, S(m))
Assumption 2. There exist L.p.q q O such that for all 8, 8,8' E R
m
IC (s) - Cm (8')l S T
83 s/
The special case p 2 and g = 1/2 leads to Lipschitz continuous functions. Assumption and Assumption 2 should be considered in conjunction. By rescaling and centering test statisties $6) one can typically obtain test statistics satisfying Assumption I Rescaling and centering typically does affect how the test statistics need to be combined, which might "break" Assumption 2
Finally, following the standard testing approach, we compare the aggregated test statistics ..... (S()) ((m)) to a threshold value. If the combined test statistics result in a large enough value, the null hypothesis of no effect is rejected. We note here that two sided tests can be written as one-sided tests through straightforward transformations (e.g. centering and taking absolute value).
More formally, we consider tests To of level satisfying the following assumption.
Assumption 3. There exists strictly decreasing function o + Ka so that
Ta = 1 { (S ....., 5(m)) 2 Ka
(4)
satisfies PoT, S a.
The map a + Ka could be taken as the quantile function of Cm (S under its null distribution it is appropriately standardized. If En (S()) (S.))...,, S(m) is bounded in m, we can choose Ka equal to 1/a times the upper bound, in view of Markov's inequality.
Our first main result, Theorem | below, establishes a lower bound for tests of the form ( and Om and S satisfying the above assumptions. More concretely, under our assumptions, any test To (of level a < 0.1) has large Type II-error under alternatives with p² of smaller order than (/m A log(m)) Vdd(mm). When the number of trials is small compared to the dimension (i.e. m log2 (m) & d²), this means that the separation rate at least vd)(ymm)). Thus even though there is a benefit in terms of separation rate compared to testing based just single trial, the

---

gain is at best the square root of what one would gain based on testing on the pooled data. When m log" (m) 2 dZ, the rate in the lower bound changes to dv dVd/(mm lo)(m)), resulting in an elbow effect.
Theorem 1. Let , Cm and Ta satisfy Assumptions 7 with Ta of level a E (0,
Then there exists constant 0 depending only on L, p. 9 and M, such that if
(vm A
d Vdd
p S
log(m)
(5)
holds for all n, m. d E N that
sup py (Ta = 0) 2 3/4. E
(6)
Remark 1. The ranges of values 0 S 0.1 and | 3/4 for the Type I and II errors, respectively, are arbitrary. Similar results hold for different choices as well. For instance, one can take arbitrary (0, 1/5] and (0,2) 2/3], see the proof of the theorem for details. The result implies in particular that consistent testing is not possible for signals of a smaller order than the right hand side of ( where asymptotics can be considered in n, m and d simultaneously.
In the next section we show that the lower bounds the theorems above are sharp (up to logarithmic factor).
2.1 Rate optimal combination methods
To attain the lower bound rate derived in Theorem D different tests can be considered. The optimal rate displays an elbow effect around m x d². When the dimension is large compared to the number of trials m (i.e. m S d²), strategies that combine p-values for the optimal local tests (based on Ilvxx Ho x2), turn out to achieve the optimal rate, exhibited below. Such test statistic is invariant to the directionality of X and invariant under the model in the sense that the resulting power for the alternative Py or F3 is the same as long as || = ||91|2-
On the other hand, when the dimension is small compared to the number of trials (i.e. m 2 d=), optimal strategies exhibited below use information on the direction of X In fact, we show in Theorem in the Appendix that if no such information is available (i.e. the events defined by the signs of the (X x0))==,..... vector are not contained in the sigma algebra generated by the test statistics S), one cannot obtain rate better than vd/(vmm)) This implies that by combining the locally optimal (/1 |/ (or their arbitrary functions, e.g. the corresponding local p-values) would result in information loss and hence sub-optimal rates in the meta-analysis.
test statistics 5(i) ll v
Furthermore, it turns out, in accordance with the empirical literature discussed in the introduction, that there does not exist a uniquely best meta-analysis method. In fact, multiple standard veta-inalysiis techniques provide (up to a logarithmic factor) optimal rates, see below for some standard approaches attaining the lower bounds derived in Theorem
First we consider the scenario when the dimension d of the model is large compared to the number of trials m, i.e. m S d² Locally the optimal test is based on the test statistic l| ( Ho natural way to combine these statistics would be to sum these locally optimal test statistics to obtain
x2.
m
12= F (1))
Ta =a=1
which has level a. Alternatively, one could also apply v-auuue combination methods, such as Fisher's s or Edgington's method based on the p-value p()) = 1 F (Il /nxx [|3), see Section Lemma in the appendix establishes that these tests are rate optimal.
Second, consider the case that the number of trials is large compared to the dimension, i.e. 2 dZ Rate optimal tests can be constructed based on a variation Edgington's or Stouffer's method, see Section for their descriptions. Taking partition of .....,,m = Ug_ T. where [Ti x m/ m/d and
5

---

setting 81h= -Vūx(") j E Ti, the meta-level test
T. = 1
Σ st)
2
d
1/2
(1
(8)
X2
1=1
JEJ;
achieves the lower bounds. The above test similar to employing Stouffer's method for each of the
coordinates and averaging, i.c. computing approximately m/d iid p-values p = o ( ()) for
j E J. and applying the inverse Gaussian CDF + (p(t)). Alternatively, the following variation of Edgington's 's method,
2
2)
pll)
To =1
Kac
also rate optimal, as proven in Lemma in the appendix. Essentially, these strategies divide the trials accross the d different directions. and combines the evidence for each of the directions. Theorem f Jaffirms that the information on the "direction" of the data is crucial to achieve the optimal rate in the m Z dl case, by showing that strategies that do not contain such information (rotationally invariant strategies such as norm-based test statisties) achieve the rate vd/( at best. We summarize the above testing upper bounds in the theorem below.
Theorem 2. For all a, (0,1) 1) there exist S, Cm R Assumptions 7 such that if
R and tests To of level a satisfying
p² 2 Ca.B
(10)
we have
sup (T= 3
fEHt
for a large enough constant Ca.3 > depending only on a, B E (0, for all n, m, d E N.
2.2 Benefits of coordination between the trials
When the dimension is small relative to the number of trials, as exhibited in the previous section, optimal strategies include information on the directionality of the observation vector. In this section we show that in this regime, there could be an additional benefit from allowing mild coordination between the trials through employing shared randomness, e.g. a shared random seed between the trials. Such phenomenon has been observed before in the distributed testing literature which forms the basis of our analysis below.
t 20
We consider the following variation on Assumption I where the key difference is that the source of randomness is allowed to be shared between the traals.
Assumption 4. For functions ʃj Rd x IR - lR and a random variable which is independent of the data X, the j-th test statistic sti) (X U) satisfies Eg|S(ii < M for some M >> 0 and all j ,,..., m.
Test statistics satisfying this assumption shall be referred to as shared randomness (or public coin) protocols.
The theorem below establishes the optimal rate when coordination through shared randomness is allowed. When the number of trials is small compared to the dimension (i.e. m S d/ m), there is difference between protocols that coordinate using shared randomness those without coordination. In fact, the optimal rate (p² x vd/(ymin) in this case is reached by the test 7 or the ones below it, which do not employ shared randomness. However, when the number of trials is large compared to the dimension (i.e. m. 2 d), the testing rate substantially improves in the shared randomness protocols.
Theorem 3. Let (I.),..., S(m)), Om and Tn satisfy Assumptions > 0 depending only L,, p,q q and M, such that if
Then there exists a constant
(m A
Vd
log(m)
0² <
(11)
mn

---

it holds that sup , (Ta = =0) 2/3 for all n, m, d E N and any level E (0,0.1) feHt,
At the same time, for all a, 3 (0, 1) there exists constant Ca. > 0 depending only on B, L, p. 9.
the function a ++ Ka and M. such that if
(Vm vā) Va
0²
Ca.3
(12)
TN
it holds that sup , (Ta 0) S B for some test Ta of level satisfying Assumptions 2 SEH,
Remark 2. Similarly to Theorem I the choice of ranges < a S 0.1 and 3 = 2/3 in the lower bound result is arbitrary, other choices also possible as presented in the proof.
A shared randomness method that attains the rate in 12 is given next. Consider drawing an orthonormal d x d matrix U taking values from the uniform measure on such matrices. As a test statistic, each trial computes (V)UX(()), which is (N, 1) random variable under the null hypothesis. A level a E (0, meta-level test is then given by combining the local test statistics
m
To
(VnUX()) (1-(11aa1)))11111111111
(13)
where & is the standard Gaussian CDF. The core idea here is that for each trial, the same 1-dimensional projection of the d -dimensional data is computed, where the projection is taken uniformly random and the test is conducted along the projected direction. The above method corresponds to Stouffer's
method for the p-values pli) Φ
for j = 1,..., m. , Lemma 8 in the appendix shows that the above test attains small Type II error probability whenever p² Z (mn))
3 Examples for meta-analysis methods
Combinations of independent test statistics that fall into the framework of Assumptions T are subject to the rate optimality theory established by the main theorems in Section 2 In this section, look into common methods for combining p-values, e-values and other stst-statistics, as mentioned in the introduction.
When the distribution under the null hypothesis of the test statistics are known, certain combinations are natural. For example, the sum of normal or chi-square test statistics is again normal or chi-square distributed, respectively. Similarly, voting based mechanisms typically rely on summing Bernoulli random variables. It easy to see that these and similar combinations methods fall into the framework of Assumptions
For more specific test statistics, such as p-values or e-values, many general combination methods have been introduced in the literature, We cover some the most prominent combination approaches for p-values and e-values in Section 3.I and Section 3.2 respectively. The list of methods is certainly non-exhaustive and many more combination methods exist, but they serve as context for the range of techniques covered by our general theory. Our main results allow establishing lower bound rates for the ones listed below, whilst in Sections and 2.2 attainability of these rates by some of the listed methods was exhibited.
3.1 Combinations of p-values
If ..... ()))....,p(m) p-values obtained from independent test statistics concerning the same hypothesis, then under the null p(i) بd (00 1). One aim to combine the p-values to form test Ta (p)),... ..... (m))) with Type I error probability a. which hopefully has higher power than test based on one of the individual p-values. Below we list standard methods the literature.
(()lgp()))))) pli) are iid x3 disttriri under the null hypothesis, their sum follows x2m. -is--iistributinn. Therefore the combination method Σ 2 log p(i) results in x2.. distributed random variable, and the corresponding quantile function provides level-a one-sided tests at the meta-level.
Fisher's method [8 Because the variables -2 log
7

---

Similar flavour to Fisher's s method are the combinations ΣØ= - -log(1 - (Pearson's s
method
(32) -ø= - log (1-111p1pppp)))111111 (1=)()) (the logit method/ Mudholkar and George method
( and m 1/2 1 (p() 1/2) (Edgington's 's method 14).
Order-based methods such as Tippett's method based on min Beta(1, m).
based on Methods based on inverse CDF's, such as by Stouffer et. 1-=0--10
36
m m-1/22 -y=
N(O, 1) under the null hypothesis.
Generalized averages as considered in 48.. Ta 1 (n mm (p( S a). where (r, ..... (()) (
equals (m " (ob))))) 1/7 for r E R {0), the geometric mean, minimimum (i.e. Tippett's method) and maximum for r 0,
OC, and r oc, respectively. For r E [-x0) U [1/(mm 1mm--1 1), so], armm can be taken to obtain precisely level tests (i.e. PoTa a). We note that this means that canonical multiple testing methods (see e.g. [1 such as Bonferroni's correction (which corresponds with taking as My the minimum and a. m. also fall within our framework.
Lemma below shows that all the methods mentioned above fall into the framework of Assumptions
0 This means that the error rate lower bounds of Theorem D and Theorem B respectively, apply to the p-value combination methods listed above. That is, one cannot attain a better separation rate when considering the worst case Type II error probability for the alternative hypothesis in (2). with any of the p-value combination methods listed above. Whether Assumption Tor Jappiees depends on whether shared randomness is used in generating the p-values. To confirm that Assumptions and 2 apply to tests based on the combined p-values, some algebra is needed. The proof of the lemma is deferred to the appendix.
Lemma 1. Consider p-values ,..,,..., plm), where each pb) depends on the local data X and possibly local randomness that is independent of the data. For each of the combination methods for p-values mentioned above and corresponding test Ta of level a E (0, 1), the conclusions of Theorem 7 holds.
We remark that the p-values obtained using shared randomness (i.e. in the sense of Assumption D. the lower bound rate of Theorem aapplies. Furthermore, as exhibited in Sections 2. and 2.2 for p-values corresponding to well chosen test statistics, these combination methods can achieve the theoretical limits established in Theorems I and respectively.
3.2 Combining e-values
An e-value is nonnegative random variable E such that SuPp.eH, P,E 1. The threshold test corresponding E of level a is 1(E 2 a "1). This test yields so called strict p-value; for Po Ho we have Po(E 2 - ) I a by Markov's inequality.
E-values lend themselves for combining outcomes of independent studies for two main reasons. First, they are easy to combine, see Section 4 in 40 for an indepth discussion of specific combination functions for independent e-values. Second, they robust misspecification and offer optional stopping/continuation guarantees [22). Common examples of e-values are Bayes factors and like- lihood ratios, which are nonnegative and have expectation equal to in the case of a simple null hypothesis such as considered in this article.
Several combination methods (e-merging functions) were proposed in the literature. For instance, the product of independent e-values is also again an e-value. This was shown to weakly dominate any other combination of independent e-values in the sense that 11,= EG) > C. (E), for any E = (E()) E (o and E + C. E) such that (E F.m.) is an e-value for any independent e-values E())... [mm),, see 39 Similarly, the average of e-values is again an value. The product and the average are admissible in the sense that there is no e-merging function that strictly dominates them on [0. xx)" _ The lemma below shows that these two, arguably most prominent e-value combination methods fulfill Assumptions
I 5i and hence the lower bounds derived
in Theorems I and apply.
8

---

...., E(m), where each EU) depends on the local data X and Lemma 2. Consider e-values E.......
possibly local randomness that independent of the data. Let C.. :R R correspond to either the average or the product and let Ta be the corresponding threshold test of level a E (0, 1),
To =a=
F(m)) 20
Cm is the product, assume in addition that Eoo log E is uniformly bounded. Then, the conclusion of Theorem holds. In case the e-values are generated using shared randomness, then Theorem applies.
4 Simulations
In this section, we investigate the numerical performance the testing strategies outlined in Section 2.T synthetic data sets. We compare the tests based their receiver operating characteristic (ROC) curve. For range of significance levels we compute for each tests the "true positive rate" (TPR) and "false positive rates" (FPR), i.e. the fraction of the simulation runs in which the test correctly identifies the underlying signal, falsely rejects the null hypothesis, respectively. Plotting the TPR against the FPR (both given as function of the significance level) provides the ROC curve, visualizing the diagnostic ability of the test.
1.00
1.00
True Positive Rate
0.75
True Positive Rate
0.75
0.50
0.50
0.25
0.25
Curve
0.00
0.00
identity
0.00
0.50
1.00
0.00
0.50
0.75
1.00
Chissuare combined
False Positive Rate
False Positive Rate
directional coordinated
directional uncoordinate
1.00
1.00
single trial
Chi-square pooled
True True Postive Rate
0.75
True True Postive Rate
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
False Positive Rate
False Positive Rate
Figure ROC curves for different values of d, whilst keeping m = 20, 30, p² vd/(4n). From left right, top bottom: 2, d 5, d 10, d 20.
In simulations set 20, 30, let range from 2 to 20 and take 0² = vdd((An) This value of p² corresponds to a signal that is almost indistinguishable from noise using just a single be pooled with m 20 (which increases the
trial, whilst consistently detectable if the data signal size to noise ratio effectively by factor /20 > 4. For each level (0.01, 0.02, ... 0.99) we compute the power for different combination strategies 100 times, each time drawing a different I with || = / according fi d-1/2 and RG iid Rademacher random variables for i 1, i,...,d. d. As combination strategies, compare the strategies 7 and 8 from Section 2

---

which are called "'hi- square combined", "coordinated directional" and "uncoordinated directional" in the legend of Figure | In addition, we display the ROC curves for the chi-square test based on pooled data ("chi-square pooled") and that of single trial ("single trial").
We make the following observations, in line with our theoretical findings. The meta-analysis methods based on combining the locally optimal chi-squared test statistics (yellow curves) substantially out- performed the chi-squared test statisties based on a single trial (blue curve), but was substantially worse than the chi-square test based on the pooled data (pink curve). Second note that the large dimensional case (d = 10 and d 20) the best strategy indeed to combine the local chi-square statistics (yellow curve), while in the low dimensional setting (d = 2) it is more advantageous to combine the directional test statistics x( (blue curve). Finally, note that allowing coordination between the trials by shared randomness protocol can result in improved performance (green curve) compared to the independent experiments (blue curve). In fact this approach provides the best meta-analysis method in the small dimensional setting (e.g. d = 2 and d = 5 for small a, which is the most interesting case).
In the appendix, Section A.6 we explore eight additional simulation settings, where we consider larger values of d and m. Whilst these simulations do not reveal additional phenomena the ones observed in Figure 4 they do give insight into the relative performance of the testing methods for different values of d and
5 Discussion
We briefly summarize our main contributions and discuss possible extensions and research directions. First, by establishing a connection between meta-analysis and distributed learning under communica- tion constraints, we have provided unified, theoretical framework for evaluating the behaviour of standard meta-analysis techniques. In our analysis. we considered the many normal means model, but these results can be extended to other more complex models as well, building on the connection with distributed computation. For example, minimax estimation rates under communication constraints were derived for other parametric models [53], density estimation [ signal-in-Gaussian-- whitun----ttttiieeee
54 38 I0, nonparametric regression [37 and in abstract settings 52 including binary and Poisson regression, drift estimation, and more. The normal means model allows for a tractable analysis, but results in this model are known to extend to more complicated models, such as discrete density testing (see e.g. IT]. With the due technical work, our results are expected to translate to these settings as well, but we leave this for future endeavor.
In the normal means model we show that by combining the locally optimal chi-square statistics at a meta-level one can gain a factor of /m compared to using a single trial. Nevertheless, regardless of the choice of the combination method, factor of /m is lost compared to the scenario when all data from all trials are at our disposal. This loss is clearly visible even in small sample sizes, dimensions and trial numbers, as demonstrated in our numerical analysis, as can be seen in the corresponding ROC curves. For more complex models, such a numerical study can be first step to quantify the efficiency of the meta- meta-analy analysis method. We have also shown that in the small dimension large number of trials setting combining the locally optimal chi-square statistics (or any rotationally invariant statistics for that matter) results in information loss and sub-optimal accuracy. In this case, better rates can be attained by test statistics based on the direction of the observations combined at the meta-level. In practice, one often cannot choose which test statistics can be obtained from independent trials. In such cases, the /m-factor loss in the case of c.g. rotationally invariant test statistics is of interest when considering power calculations. Meta-analysis approaches based on directional test statisties are designed for scenarios where individual datasets are not centrally collected, but there is some level of coordination among experimenters.
The assumption throughout the paper of homogeneity between the trials (i.e. each trial consisting of the same number of observations) simplifies the presentation, but the results can be extended to cases where the number of observations in each trial differ by constant factors. Situations where the number of observations differs greatly (e.g. k & m trials have as much observations as the other m k trials combined) are certainly of interest, but beyond the scope of this paper.
10

---

Acknowledgements: Co-funded by the European Union (ERC, BigBayesUQ, project number: 101041064). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union the European Research Council. Neither the European Union nor the granting authority can held responsible for them. This research was partially funded by Spinoza grant of the Dutch Research Council (NWO).
References
[1] ABRAMOVICH, F., BENJAMINT, Y., DONOHO, D. L.,, AND JOHNSTONE, I. M. Adapting unknown sparsity by controlling the false discovery rate. The Annals of Statistics (2006), 584-653,
[2] ACHARYA, J., CANONNE, C. L.,, AND TYAGI, H. Distributed signal detection under communi- cation constraints. In Conference on Learning Theory (2020), PMLR, pp. 41-63.
[3] ACHARYA, J., CANONNE, C. L.., AND TYAGI, H. Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction. IEEE Transactions on Information Theory 66, 12 (Dec. 2020), 7835-7855.
[4] AERTS, S., LAMBRECHTS, D., MAITY, S., VAN Loo, P., COESSENS, B., DE SMET, F., TRANCHEVENT, L.-C.,, DE MOOR, B., MARYNEN, P., HASSAN, B., ET AL. Gene prioritiza- tion through genomic data fusion. Nature biotechnology 24, (2006), 537-544.
[5] ALTMAN, E. I. Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. The journal of finance 23, (1968), 589-609.
[6] ALTMAN,
SRANICZ-DROZDOWSERR M., LAITINEN, B. K., AND SUVAS. A. Distressed firm and bankruptcy prediction in an international context: A review and empirical analysis of altman's z-score model. Available SSRN 2536340 (2014).
[7] BARAUD, Y. Non-asymptotic minimax rates testing in signal detection. Bernoulli (2002), 577-606.
[8] BARNES, L. P., HAN, Y., AND ÖZGÜR, A. Lower bounds for learning distributions under communication constraints via fisher information. The Journal of Machine Learning Research 21, I (2020), 9583-9612.
[9] BIRNBAUM, A. Combining independent tests significance. Journal of the American Statistical Association 49, 267 (1954), 559-574.
[10] CAL, T. I., AND WEL, H. Distributed nonparametric function estimation: Optimal rate of convergence and cost of adaptation. The Annals of Statistics 50, 2 (2022), 698-725.
[11]
CARTER, A. V. Deficiency distance between multinomial and multivariate normal experiments. The Annals of Statistics 30, 3 (2002), 708 730.
[12] CHEN, Z. Optimal tests for combining p-values. Applied Sciences 12, (2021), 322.
[13] CLEMENTS, N., SARKAR, S. K., AND GUO, W. Astronomical transient detection controlling false discovery rate. In Statistical challenges in modern astronomy V (2012), Springer, pp. 383-396.
[14] EDGINGTON, E. S. An additive method for combining probability values from independent experiments. The Journal of Psychology 80, 2 (1972), 351-363.
[15] EFRON, B. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, vol. 1. Cambridge University Press, 2012.
[16] EVANGELOU, E., AND lOANNIDIS, P. Meta-analysis methods for genome-wide association studies and beyond. Nature Reviews Genetics 14, (2013), 379-389.
[17] FINNER, H., AND ROTERS, M. Log-coneavity and inequalities chi-square, and beta distributions with applications in multiple comparisons. Statistica Sinica (1997), 771-787.

---

[18] FISHER, R. A. Statistical methods for research workers. In Breakthroughs in statistics. Springer, 1992, pp. 66-70.
[19] FROMONT, M.. LERASLE, M., AND REYNAUD REYNAUD-BOURET, P. Family-Wise Separation Rates for multiple testing. The Annals of Statisties 44, 6 (2016), 2533 -256..
[20] GINE, E., AND NICKL. Mathematical Foundations of Infinite-Dimensen Statistical Models. Cambridge University Press, Cambridge, 2016.
[21] GOOD, I. On the weighted combination of significance tests. Journal of the Royal Statistical Society: Series (Methodological) 17, 2 (1955), 264-265.
[22] GRÜNWALD, P.. DE HEIDE, R., AND KOOLEN, W. M. Safe testing. In 2020 Information Theory and Applications Workshop (ITA) (2020), pp. 1-54.
[23] GUGLIELMETTI, F., FISCHER, R., AND DOSE, V. Background source separation in astro- nomical images with bayesian probability theory-i. the method. Monthly Notices of the Royal Astronomical Society 396, (2009), 165-190.
[24] HOSPEDALES, T., ANTONIOU, A., MICAELLI, P., AND STORKEY, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence 44, (2021), 5149-5169.
[25] INGSTER, Y. I. Minimax testing of nonparametric hypotheses distribution density in the SI p$ metrics. 333-337. Number: 2.
[26] JOUNSTONE, I. M., AND SILVERMAN, B. W. Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences. The Annals of Statistics 32, (2004), 1594 1649.
[27] KRÄMER, A., GREEN, J., POLLARD JR, J., AND TUGENDREICI, S. Causal analysis ap- proaches in ingenuity pathway analysis. Bioinformatics 30, (2014), 523-530.
[28] LIPTÁK, Т. On the combination of independent tests. Magyar Tud Akad Mat Kutato Int Kozl 3 (1958), 171-197.
[29] LOUGHIN, T. M. A systematic comparison of methods for combining p-values from indepen- dent tests. Computational statistics & data analysis 47, 3 (2004), 467-485.
[30] MALONE, B. M.. TAN, F., BRIDGES, S. M., AND PENG, Z. Comparison of four chip-seq analytical algorithms using rice endosperm h3k27 trimethylation profiling data. PloS one 6, (2011), e25260.
[31] MUDHOLKAR, G. S., AND GEORGE, E. The logit statistic for combining probabilitics-an overview.
[32] PEARSON, K. On new method of determining" goodness of fit". Biometrika 26, (1934), 425-442.
[33] QUACKENBUSII, J. Microarray data normalization and transformation. Nature genetics 32, 4 (2002), 496-501.
[34] SHAFER, G., ET AL. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society: Series (Statistics in Society) 184, 2 (2021), 407-431,
[35] SHAFER, G., SHEN, A., VERESHCHAGIN, N., AND VOVK, V. Test Martingales, Bayes Factors and ValuVa.ues. Statistical Science 26, (Feb. 2011), 84-101. arXiv: 0912.4269.
[36] STOUFFER, S. A., SUCHMAN, E. A., DEVINNEY, L. C., STAR, S. A., AND WILLIAMS JR, R. M. The american soldier: Adjustment during army (studiss in social psychology in world war ii), vol.
[37] SZABÓ, B., AND VAN ZANTEN, Н. Adaptive distributed methods under communication constraints. The Annals of Statistics 48, 4 (2020), 2347 2380.
12

---

[38] SZABÓ, B., AND VAN ZANTEN, H. Distributed function estimation: Adaptation using minimal communication. Mathematical Statistics and Learning 5, (2022), 199-199.
[39] SZABÓ, B., VUURSTEEN, L., AND VAN ZANTEN, H. Optimal distributed composite testing in high-dimensional gaussian models with 1-bit communication, IEEE Transactions on Information Theory 68, 6 (2022), 4070-4084.
[40] SZABÓ, B., VUURSTEEN, L., AND VAN ZANTEN, H. Optimal high-dimensional and non- parametric distributed testing under communication constraints. The Annals of Statistics 51, (2023), 909-934.
[41] THOMAS, J. G., OLSON, J. M., TAPSCOTT, S. J., AND Zhao, L.. P. An efficient and robust statistical modeling approach to discover differentially expressed genes using genomic expression profiles. Genome Research 11, (2001), 1227-1236.
[42] TIPPETT, H. C., ET AL. The methods of statistics. an introduction mainly for experimentaliits. The methods of statistics. An introduction mainly for experimentalists. (1941).
[43] TSYBAKOV, A. B. Introduction to nonparametrric estimation. Springer series in statistics. Springer, New York London, 2009. OCLC: ocn300399286.
[44] VAART, A. W. D. Asymptotic statistics. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press.
[45] VAN ZWET, W., AND OOSTERHOFF, J. On the combination of independent test statistics. The Annals of Mathematical Statisties 38, 3 (1967), 659-680.
[46] VERSHYNIN, R. High-DDmensional Probability: An Introduction with Applications in Data Science, ed. Cambridge University Press, Sept. 2018.
[47] VOVK, V., WANG, B., AND WANG, R. Admissible ways of merging p-values under arbitrary dependence. The Annals of Statistics 50, (2022), 351-375.
[48] VOVK, V., AND WANG, R. Combining p-values via averaging. Biometrika 107, 4 (2020), 791-808.
[49] VOVK, V., AND WANG, R. E-values: Calibration, combination and applications. The Annals of Statisties 49, 3 (2021), 1736-1754.
[50] WHITLOCK, M. C. Combining probability from independent tests: the weighted z-method is superior to fisher's approach. Journal of evolutionary biology 18, 5 (2005),
[51] YOON, S., BAIK, B., PARK, T., AND NAM, D. Powerful p-value combination methods to detect incomplete association. Scientific reports 11, (2021), 6980.
[52] ZAMAN, A., AND SZABÓ, B. Distributed nonparametric estimation under communication constraints. arXiv preprint arXiv:2204.1037. (2022).
[53] ZHANG, Y., DUCHI, J., JORDAN, M. I., AND WAINWRIGHIT. M. information-theoret lower bounds for distributed statistical estimation with communication constraints. Advances in Neural Information Processing Systems 26 (2013).
[54] ZII, Y., AND LAFFERTY, J. Distributed nonparametric regression under communication constraints. International Conference Machine Learning (2018), PMLR, pp.
13


Reviews:
[{"review_id": "IItryS5yQW", "reviewer": "Reviewer_aj9v", "summary": "The paper considers methods to aggregate test statistics from different, independant, sources, in order to construct an aggregated test with hopefully more power. The key contribution of the paper is the study of the minimal treatment effect which can be detected in a standard gaussian noise setting, for which they obtain minimax rates.\nThese rates exhibit an elbow effect when the number of aggregated statistics (m) is close to the square of the dimension of the signal (d^2), which the authors relate to the use of the signal direction in the test (when $m < d^2$, the standard chi2 test would give near optimal power, while for $m > d^2$, the test statistics must encode directional information if optimal power is to be obtained). These rates bring two insights: First, aggregating one dimensional statistics in a multi dimensional setting comes at a price. Second, there is no single optimal aggregating method.", "questions": "Could the methodology be extended beyond the current setting? Notably, is there any natural generalisation in the case where the noise level $\\sigma = 1/\\sqrt{n}$ can no longer be assumed to be identical ?\n\nIs there any explanation of the results in terms of the distribution of the p-values under the alternative hypothesis? If so, is there any insight on the best way to aggregate a given set of statistics (instead of considering the best way to aggregate the best statistics for a given m, d)?", "limitations": "The paper derives optimal test aggregation strategies in the context of gaussian noised signals. A first limitation of the paper is that it is assumed that the sample size $n$ considered in each collected test is identical. This can barely be expected in practice, and as such, insight about the impact of uneven tests would be welcome (i.e., does $mn$ translate into $\\sum_{i=1}^m n_i$?, or rather $m \\min(n_i)$?). This issue is not mentionned in the paper.\n\nAnother limitation not mentionned is the fact that, in practice, the test statistics obtained from independent trials are not chosen, but set, and as such, Stouffer's method, which attain the minimax rates when $m>d^2$, is not implementable. In most settings where it could be implemented, the whole $X_i$ information would be known, and therefore the dimension reduction issue would not occur. For this reason, the best rate achievable with realistic one dimensional statistics is of particular interest. Unfortunatly, this is left to the appendix, in Theorem 4.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well presented and obtains conclusive results in the scope considered, in the form of minimax rates. These rates justify previous empirical insights on aggregated testing strategies, notably the need for different aggregating strategies depending on the number of tests and the dimension of the problem. Methods achieving the rates (up to a log factor) are specified. \n\nAs far as I could assess, the mathematical proofs are, up to small typos (see weakness), correct. The presentation of the main results in section 2 can be easily followed (minimax rates in the general case, optimal combination methods then improved minimax rate using coordination between tests).\n\n", "weaknesses": "The proof in the appendix suffers from some small typos. Notably, I believe that in equation (S.1), the $2\\epsilon$ term should be $\\epsilon$ (or $\\epsilon< \\frac{1}{2}\\left(\\kappa_{1/10} - \\kappa_{1/8}\\right)$ in the definition of $\\epsilon$), while in line 538, the conclusion of Markov's inequality is that $D^c$, not $D$, has mass less than $1/64$.\n\nThe methodology used to obtain Figure 1. could be improved. Notably, the Roc Curves for Chi-square combined and Chi-square pooled should not exhibit any randomness, since these two curves can be computed in closed form using the cumulative distribution functions of the chi2 square and non central chi square. If numerical approximations are to be used, it could be possible to obtain curves exhibiting much less noise by increasing the number of repeats and recycling them for all FPR (I could obtain curves exhibiting little to no noise robustly using 10 000 repeats and 100 FDR in less than <2s on my personal computer, so computation time is not an issue).\nMoreover, the way the $f_i$ are drawn, using Rademacher random variables, might have an impact on the directional methods.  While this might or might not be the case, I would suggest recomputing the curve, drawing a random f uniformly on the sphere.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere thanks to the Reviewer for the taking the time and effort to thoroughly review, the insightful comments and constructive feedback on our paper. The Reviewer also identifies areas for improvement, which we will address point-by-point below.\n\n*Typos:* Thank you for pointing out these typos, we have corrected them in the revised manuscript.\n\n*Simulation study:* we have rerun the simulations with a much larger number of repeats (i.e. 100 000 repeats) and considered a higher dimensional setting. Furthermore, following the suggestion of the Referee, we have drawn the parameter $f$ uniformly from the sphere. See the attached figures in the general rebuttal. We note that we did not observe a significant difference compared to the Rademacher prior, but we if it is deemed beneficial we can include this in the simulations added to the appendix of the new version of the article. \n\n*Optimal test statistic:* We  agree that in case of independent trials typically the test statistics are set independently from the number of trials. Hence in such cases the direction based approach can not be applied and hence the best attainable rate is $\\frac{\\sqrt{d}}{mn}$ following Theorem 4. We completely agree with the Reviewer, that this result is of particular interest in such settings. In our revised manuscript we have highlighted these more prominently, stressing the importance of theoretically capturing explicitly the loss in terms of error rate (i.e. the $\\sqrt{m}$ worse error rate) when conducting meta-analysis in such scenarios. It is important for practitioners conducting meta-analysis to be aware of this potential loss in statistical power. and provided an extended description of Theorem 4. Nevertheless, one could also consider situations where the number of trials is known in advance or a new round of experiments are executed in addition to some available earlier ones. In such cases a better rate is attainable by choosing the non-standard, directional test statistics is designed for scenarios where individual datasets are not centrally collected, but there's coordination among experimenters (depending on the interplay between m and d). \n\n*Concerning the heterogeneity and different sample sizes:* In the revised paper have included a discussion concerning extensions in this direction. We note that in case the sample sizes are of the same order, i.e.  $n_j \\asymp n_k$ for all $k,j=1,\\dots,m$, the current rates can be proven with minor modifications. If the local sample sizes can be substantially different then the question is more complicated. The rate $m {\\min}_j n_j$ can be of course easily obtained, but it is rather pessimistic, e.g. if $n_1\\gg\\underset{j=2}{\\overset{m}{\\sum}} n_j$ then almost no extra information comes from the experiments $j=2,...,m$ and the rate is determined by the first sample size $n_1$, which is substantially better than $m {\\min}_j n_j$. However, the rigorous analysis of this question goes beyond the scope of our paper. \n\n*Question concerning insight on how to aggregate:* This is a great question, and something that was not highlighted in the old version of the paper. The insight is mainly in the fact that in terms of error rate, many methods perform optimally in principle, as long as the underlying p-value / test statistics have decent power. That is, Fisher's method, Stouffer's or simply averaging all can reach the $\\frac{\\sqrt{d}}{\\sqrt{m}n}$ optimal rate. Beyond that, simulation of the behaviour p-values / test statistics for specific alternatives that are especially deemed important to detect is also an option. We have added a remark reflecting this to the new version of the paper.\n"}, {"review_id": "Z9a8hHfFa8", "reviewer": "Reviewer_qxf1", "summary": "The paper lies in the context of meta-analysis of multidimensional models. Usually, meta-analyses are performed by combining p-values or e-values. In both cases, the statistical power is not well-known. The authors provide a constrained framework of many means normal model. Based on this framework, they derive lower (theorem 1) and upper (theorem 2, for \"rate optimal methods\") bounds for testing methods. Theorem 3 takes advantage of possible shared randomness between trials, especially when the dimension is small relative to the number of trials considered. The authors then compare the performances of rate optimal methods (described in section 2.1), chi-square test on pooled data and single trial approach on simulated datasets.", "questions": "- $\\mathbb{E}_0$ is first used in line 141 but not introduced beforehand. Is it the expectation under the null hypothesis? \n- I understand that Assumption 1 aims at restricting the values of S. Is it possible to give a small interpretation of the assumption?\n- Theorem 3 indicates that \"there exists a constant $C_\\beta$\" but the formula indicates \"$C_{\\alpha,\\beta}$\". Is it a typo error? \n- In theorems 1 and 3, arbitrary values are used for $\\alpha$ and $\\beta$, not in theorem 2. Why make the choice of using these values and not giving general results for the corresponding intervals of validity?\n- A similar remark on the proofs, for example in proof A.1. Why use $\\kappa_{1/10}$ and $\\kappa_{1/8}$? \n- It might also be more comprehensible to explicitly add the results taken from the literature and used in the proofs. \n- The provided R script needs to be reviewed. When running it, I get the following error is returned:\n\"Error: object 'dat_long' not found\"\n\nI understand that the chosen mathematical framework provides lower and upper bounds for testing errors. The paper describes some meta-learning techniques that attain these bounds. I am not sure how the simulation study demonstrates the theoretical results. Is it by comparing these meta-learning techniques to the \"chi-squared pooled\" approach and the single trial approach? \nThe indicator of performance for \"optimal testing\" is the ROC curve. Would it be interesting to consider other criteria, such as sensitivity, specificity, precision, or F-score?\n\nOverall, I encourage the authors to add more explanations and interpretations, especially in the mathematical development part. Note that the 9-page limit does not include references. It might also be worth submitting the paper to another journal where the format might be more adequate.\n", "limitations": "The authors have delimited the framework of their contribution by providing constraints inherent to their model. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and organised. It provides a good overview of state-of-the-art combination techniques for meta-analyses and identifies the lack of knowledge about their relative power. \nThe paper derives bounds for the testing errors by introducing a principled mathematical framework based on multidimensional models, where a loss in power is expected. It also gives insights into rate optimal combination methods and the effects of sharing randomness between trials. The simulation study provides some results on comparing combinations methods for meta-analysis, which is not properly addressed in the current literature.\nAbout the supplementary material: Proofs as well as an R script to reproduce the simulation study are provided.", "weaknesses": "The main weakness of this paper is the clarity of the mathematical developments. The framework implies several assumptions that could be explained more. The 13-page-long supplementary material provides proof of the theorems but is sometimes hard to follow. \nIn the theorems and their proofs, sometimes arbitrary values are chosen (for $\\alpha$ and $\\beta$ notably) but it seems to make the reasoning more confusing. \n\nAlso, when running the R script, the following error is returned: \nError: object 'dat_long' not found", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the thorough analysis and thoughtful feedback, we appreciate the time and effort. We address below the questions and comments of the Referee point-by-point:\n\n* Thank you for pointing this out, we have clarified in the new version of the paper that $\\mathbb{E}_0$ corresponds to the expectation of $\\mathbb{P}_0$, i.e. the probability distribution corresponding to the null hypothesis.\n\n* Assumption 1 requires that the combined test statistics can be \"rescaled'' appropriately. This assumption works in conjunction with Assumption 2, as the degree of Hölder continuity of the combination function depends on the ``scale'' (or rather, the effective support) of the underlying test statistics. We provide a more detailed explanation of the conditions in the new version of the article.\n* The $\\alpha$ subscript is indeed a typo, which we fixed in the new version.\n\t\n* First we have provided an interval of validity for $\\alpha$ and $\\beta$. However, we felt that working with a fixed, reasonable number will increase the readability. Following the comments of the Referee we have changed this back and provide a validity interval.\n* Similarly, for the quantities depending on $\\alpha$ or $\\beta$ (e.g. $\\kappa_{\\alpha}$) we have considered a fixed value $\\alpha$ to improve readability. But as above, they can be given on an interval as well, if requested.\n\t\n* Following the Referee's suggestion, in the new version, to increase the self containedness and improve the readability, we have added the frequently refereed lemmas and theorem to the appendix.\n* We apologise for the error encountered while running the R script. It is caused by a syntax/undefined object related error by a line that is meant to exemplify. The part of the code that runs the simulation (from the \"#SIMULATION\" comment downwards) should run without error.\n\n* We have extended the explanation of the simulation study and described its connection and relevance to the theoretical results in more detail. Indeed, the first message is that meta learning is substantially better than using just one experiment. The second is that combing $\\chi^2$ statistics is substantially worse than computing the $\\chi^2$ test using all the data. Finally, we show that depending on the interplay of $m$ and $d$ either the standard $\\chi^2$ combination or the novel directional test statistics method is better.\n\n* We note, that the ROC curve indicates the trade-off between sensitivity and specificity for each of the tests, hence describe the precision of the test.\n\n * Following the suggestion of the Referee, we have extended the simulation section, see the attached figures in the general rebuttal.\n"}, {"review_id": "FKY21fjw0t", "reviewer": "Reviewer_MHU2", "summary": "The paper addresses the problem of combining test statistics from multiple independent studies in the context of null-hypothesis significance testing. The authors derive a mathematical framework to quantify the cost of compressing multiple independent trials of a study into one real-valued test statistics, and they derive minimax lower and matching upper bounds for the testing errors. The many normal means models is used as toy example. ", "questions": "Can you clarify to connection between \"meta-analysis\" and \"meta-learning\" (meaning the approach reviewed in your reference number [14]? ", "limitations": "The authors do not discuss limitations or potential negative impact of their work. Although being a more technical paper, I think it would have been appropriate to discuss the limitations of the meta-analysis approach as a whole. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the problem of combining the results of multiple empirical studies towards one common hypothesis, an important problem in meta-analysis. ", "weaknesses": "This submission seems to be out of the scope of NeurIPS. While the authors draw a connection between meta-analysis as used in statistics and meta-learning as used in the context of machine learning, this connection is not clarified further. For the rest of the paper, the authors seem to focus on the problem of meta-analysis. \n\nIt was challenging to read to paper as it lacks clarity in the introduction. To improve clarity, I suggest to start the introduction by clearly stating the problem that will be addressed in the paper and clearly introducing the terms used in the text. For example, in line 23, the authors introduce “meta-analysis”, (probably) referring to the technique of combining the results of multiple scientific empirical studies and then set it equal to “meta-learning”, referring to the machine learning technique of improving a learning algorithm to perform well multiple tasks.\n\nI did not go through the technical parts of the paper in all detail. However, there are several statements in the paper that appear to be misleading, e.g., \n\n- line 68: “… includes many standard meta-learning techniques, for instance the standard p-value combination methods […]”. I am not aware of p-value combination being a standard technique in meta-learning.\n- line 313: “Common examples of e-values are Bayes factors and likelihood ratios.” e-values and Bayes factors are closely related, but Bayes factors or likelihood ratios are not e-values, see https://arxiv.org/abs/1912.06116 Appendix A for clarifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the effort of evaluating our paper and we are happy to hear that the Reviewer\n shares the opinion that the problem is important to study. The Reviewer does not specifically comment on the soundness\n of the mathematical framework, but highlights potential misleading use of terminologies and a lack of clarity in the\n introduction, both of which we address below.\n\n*Connection to ML:*\nWe have taken the advice of the Reviewer to heart and improved the clarity of the introduction by providing an\n expanded and more clear explanation of the problem. We have also expanded the introduction to more explicitly describe\n the connection and relevance to more modern machine learning applications. \n\n*Meta learning vs meta analysis:*\nIn our analysis we used meta-learning / meta-analysis techniques as statistical and/or machine learning techniques distilling information across studies and different\n data sets to form a more informed inference. We are sorry that using these terms basically interchangeably in our manuscript caused confusion. In the literature we did not find a standard interpretation of these terms (as is also discussed e.g. in the corresponding wikipedia page) and found that it depends on the specific community how they are interpreted.  However, to avoid confusion, we have decided to abstain from using the term meta-learning and stick to meta-analysis throughout the text.\n\n*Likelihood ratio as e-value:*\nWhilst it is true that in situations where the null hypothesis is composite, a Bayes factor or likelihood ratio is not necessarily an e-value, we are considering a setting in which the null hypothesis is simple. Any Bayes factor given by a likelihood ratio of\n a mixture distribution $P_\\pi := \\int P_f d\\pi(f)$ and the measure under the null hypothesis forms an e-value: $E := \\frac{dP_\\pi}{dP_0}$\n satisifies $E \\geq 0$ and $P_0 E = 1$. We have expanded on the statement: ``Bayes factors and likelihood ratios form common examples of \ne-values.'' to clarify that this should not be read in largest generality, but in the context of hypothesis tests such as considered in this article.\n\nWe have added additional discussion concerning the limitations of the meta-analysis approach as a whole.\n\nWe would like to thank the Reviewer again for their consideration."}, {"review_id": "fsRC3LiMvb", "reviewer": "Reviewer_vLRc", "summary": "Authors study a problem of optimal combination of p-values in a meta-analysis context. Specifically, they focused on characterizing the minimax separation rate for a family of \"smooth-ish\" combination methods that aggregate p-values (or e-values). They show that: \n* The family contains a lot of methods that are used in practice.\n* The separation rate for methods in the family is nearly optimal.\n* Optimal combination method depends on the problem setting (sample size, number of p-vals, dimension of the problem); they describe practical consequences. \n\nFurthermore, they explored how the rate can be improved by allowing coordination between the experiments that generate the p-values.", "questions": "See above", "limitations": "yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is written in a clear and without excessive statistical jargon. For instance, the concept of separation rate is introduced and explained in a simple and intuitive way, allowing readers to understand the results presented in the paper (in contrast to the mat description found in \"Non-asymptotic minimax rates of testing in signal detection\"). The significance of the results is solidly established on two grounds: derivation of a separation rate and practical guidance for method selection based on n, m, and d (Section 2.2 and 2.3 add extra value).  From my (limited) understanding of the literature, these results are both novel and original.", "weaknesses": "I am genuinely surprised that this problem has not been previously studied. While preparing the review, I came across similar/related results concerning the Family-Wise Error Rate (FWER) in the paper \"Family-Wise Separation Rates for multiple testing.\" However, the combination methods studied are different (Holm–Bonferroni procedure type). \n\nI'd suggest a more comprehensive and thorough discussion of the existing literature on minmax testing for multiple hypothesis testing is added to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort spent on evaluating our paper, the positive feedback and the insightful suggestions. Below, we address the specific suggestions raised by the Reviewer.\n \n*Novelty of the results:* Yes, indeed, we were also surprised by the lack of theory for meta-analysis in hypothesis testing with a composite alternative hypothesis. Actually, this was the main motivation of our research, to fill this gap in the literature.\n\n*Connection to multiple testing:* We agree with the reviewer that multiple testing has lot of similarities to the concerned meta analysis problem. The common goal is to aggregate somehow the outcome of multiple (often independently executed) tests. However, there are also major differences. In the meta-analysis  framework (e.g. when combining p-values) one considers a given hypothesis and executes multiple experiments on deciding its validity. Then the outcomes of these tests are combined for a more accurate decision about the common hypothesis of interest. In contrast to this in multiple testing in each experiments (possibly) different hypotheses are considered and the goal is to decide which ones should be rejected. Here the main challenge is to obtain uniform guarantees over all hypotheses (e.g. by controlling the FWER or FDR). Following the suggestion of the Referee we discuss then  connection of these related but different testing problems.\n"}, {"review_id": "ACW6wcrUII", "reviewer": "Reviewer_GjrH", "summary": "This theory paper provides a minimax lower and upper bounds for the testing risk (sum of Type-I and Type-II errors) for different combination methods in the specific setting of many normal means model. With the testing goal is to detect the presence or absence of the signal component in this normal means model, their results show several combination methods of test statistics (e.g. aggregate p-values and e-values) cannot consitently detect signals below a certain threshold that depend on the number of trials, samples and dimensions of the problem.", "questions": "* I have only skimmed through the proof in the appendix, so this might be my mistake, but I do not see the appearance of the $\\epsilon$ term for binary approximation of the statistics in the main results. Could the authors clarify on this point?", "limitations": "* See weaknesses.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The theoretical results are sound and based on several established techniques in distributed testing, assuming several assumptions  hold true. These are proof to be indeed the case in the Appendix for the test statistics combination methods proposed in Section 3 of the main text.\n* The paper is mostly well-written. ", "weaknesses": "1. In general I think the phrasing of on the paper's contributions could be make clearer in the last part of the introduction section. \n2. The authors should discuss more on the relevance of the setting -- the many normal means model -- in some more concrete applications. It is arguable that although this is a theoretical paper, the theories inside it is an attempt to quantify a very practical problem of meta learning. I see a lack of evidence for the popularity of many normal means in practice.\n3. Slightly related but not as equally importance, but the authors should have acknowledged that a limitation of their work is that the theoretical results only hold with many normal means model assumption.\n4. Experimental results could include more settings with a variety number of of sizes/dimensions (perhaps in the appendix) to support the theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for reviewing our manuscript and the constructive feedback. We address below the raised concerns point-by-point.\n\n*1 -- Phrasing of on the paper's contributions:* Based on your feedback, to futher improve our mansucript, we have restructured and emphasised the contributions in the last part of the introduction to clearly outline the scope of our paper and our findings. \n\n*2 -- Relevance of the setting:* The many normal means model is indeed a specific statistical model, but it is foundational to statistical theory, capturing phenomena which readily extend to more complicated statistical models, for instance nonparametric regression, classification, density estimation, just to name a few. In the revised manuscript we provide a more extensive description this model and its connection to other, practically more important statistical models, citing literature that further expand on this connection:\n\n* Large-scale inference: empirical Bayes methods for estimation, testing, and prediction - Efron (2012)\n* Introduction to Nonparametric Estimation - Tsybakov (2003)\n* Mathematical foundations of infinite-dimensional statistical models -  Giné and Nickl (2015)\n* Deficiency distance between multinomial and multivariate normal experiments - Carter (2002)\n\nIt is because of this strong connection to other statistical models that the many-normal-means model serves as a benchmark to derive error rates, even when one has more complicated statistical settings in mind, as often the error rates extend. See for example:\n* Information-theoretic lower bounds for distributed statistical estimation with communication constraints (https://proceedings.neurips.cc/paper_files/paper/2013/file/d6ef5f7fa914c19931a55bb262ec879c-Paper.pdf) - Zhang, Duchi, Jordan and Wainwright (2013)\n* Handling Sparsity via the Horseshoe (https://proceedings.mlr.press/v5/carvalho09a/carvalho09a.pdf) - Carvalho, Polson and Scott\n* Needles and straw in haystacks: empirical bayes estimates of possibly sparse sequences (https://arxiv.org/pdf/math/0410088.pdf) -  Johstone and Silverman (2004)\n\n*3 -- Acknowledgement of limitations:* We have further emphasised in the revised version that our results concern the many-normal means model. Extension to more other, more complex models (see examples above), are left for future work.\n\n*4 -- Experimental:* We have extended the simulation settings, by expanding the appendix. The simulations now include a variety of high-dimensional settings and differing sizes.\n\n*Question concerning $\\epsilon$ dependence:* The $\\epsilon$ term is absorbed in the constant $c > 0$ in the main results to improve the readability. If explicit dependence on $\\epsilon$ is desired in the main results, we will gladly change this in the new version.\n\nIn conclusion, based on your feedback, we have identified areas where we enhanced our manuscript, we hope that the revisions will address your concerns."}, {"review_id": "2OhFCGBmVo", "reviewer": "Reviewer_ThPF", "summary": "[Update1: During the rebuttal, I updated my score from 5 to 6. The reason is that I want to stronger weigh in that the paper is technically solid. My concern whether NeurIPS is a perfect fit for this paper remains, and I ask the AC to judge that part.]\nThe paper studies aggregation strategies for combining test statistics obtained from independent experiments. It is intuitively clear that the optimal strategy would be to aggregate the whole *data* and then compute one test statistic on the entirety of the data. However, the authors make the assumption that from each experiment one can only use a single real number (aka test statistic) for the final analysis.\n\nThe setting they consider is that they have observations of random variables \n\n$$\nX^{(j)} = f + \\frac{1}{\\sqrt{n}} Z^{(j)},\n$$\nwhere $Z^{(j)}$ are independent $d$ dimensional normals and $n\\in \\mathbb{N}$ and additional parameter. The null hypothesis is $f=0$ which is tested against $\\|f\\|_2 \\geq \\rho$ for some $\\rho >0$. If pooling the data was allowed, the chi-square test is optimal for this setting.\n\nThey derive theoretical results that give two different regimes:\n\n- When $m \\lesssim  d^2$ the optimal rate can be achieved by combining the individually optimal test statistics $\\|\\sqrt(n) X^{(j)}\\|_2^2$, which is undirected.\n\n- When $m \\gtrsim d^2$ then taking the directionality into account leads to better rates. Two examples they discuss are (phrasing in my own words that the authors might adapt):\n  - Split the observations into $d$ groups of size $\\sim m/d$ and test with the $i$-th group whether the null hypothesis holds for the $i$-th dimension. Then combine those.\n  - Choose a 1-dimensional projection of the $d$-dimensional data uniformly at random. Aggregate all observations along this projections (now these are 1-dimensional values, as required in the setting) and combine those directly to test the null hypothesis along the projected direction. (Note that this requires shared randomness between all experiments).\n\nFinally the authors demonstrate their theoretical findings with toy experiments and numerically confirm their findings.\n\nOverall I enjoyed reading the paper and was able to learn something new. Thank you to the authors.\n", "questions": "Please comment my first two concerns above in the rebuttal. Note that my reservation against acceptance is based on those comments. And I am happy to increase my score after the rebuttal if I am convinced otherwise.\n\n\nMinor:\n- l 124 \"where\" --> \"were\"\n- l. 339 \"form\" --> \"from\"", "limitations": "This paper only studies a simple toy problem.\n\nLimitation of my review: I did not read the appendix.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The paper is extremely well written and has \"textbook\" quality. It focuses on a simple toy problem and provides an exhaustive analysis thereof.\n- Combining outcomes from different experiments is certainly an important problem in statistics.\n- The paper seems original, however, admittedly I am unable to judge to what extent similar analyses exist in the stats literature.\n- The theory and experiments are in perfect accordance and complement each other.\n- The paper provides a good overview of existing methods to combine test statistics and discusses both $p$- and $e$-values as examples. They relate existing methods to their contribution.", "weaknesses": "1. My first point concerns the significance of the approach. In the introductions the paper states\n  > Given multiple data sets relating to the same hypothesis, one would like to combine the evidence.\n Sometimes, the full data sets are not available (e.g. due to privacy or proprietary reasons) or difficult\n to combine directly (e.g. due to the different experimental or observational setups). In such cases,\n the analysis must be carried out on the basis of the published results for each of the studies.\n\n    - This motivation reads that each study publishes a test statistic without knowing of the others. I thus think that the second type of approaches the authors provide does not fall in this category.\n    - The second type of approaches require that the \"meta-analyser\" can make some choices of what statistic the individual studies report. Hence *somewhere* all data must still be available. It is unclear why we cannot use that then. I think this needs more motivation.\n    - I presume that the first approach of combining the undirected tests has been studied exhaustively.\n\n2. While the paper is relevant for machine learning in general, it is in itself a very statistical paper. No learning involved etc. Hence, I am posing the question whether NeurIPS is the appropriate venue. I would see a stats journal as a better fit.\n\n3. I think the paper could be a bit more accessible if a bit more intuition about the approaches is provided. I wrote my understanding in the summary, maybe the authors want to correct/modify this and include something like that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort the Reviewer has put in evaluating our work. The Reviewer also raises a few concerns to which we respond below.\n\n\n*1 -- Significance of the approach:* \n\nWe agree that in case of independent trials typically the test statistics are set independently from the number of trials. In such cases, the best attainable rate is $\\frac{\\sqrt{d}}{\\sqrt{m}n}$. This is still novel and an important contribution however; theoretically capturing explicitly the loss in terms of error rate (i.e. the $\\sqrt{m}$ worse error rate) when conducting meta-analysis in such scenarios. It is important for practitioners conducting meta-analysis to be aware of this potential loss in statistical power. In our revised manuscript we have highlighted these more prominently and provided an extended description of Theorem 4, emphasising this point. \n\nThe second meta-analysis approach based on directional test statistics is designed for scenarios where individual datasets are not centrally collected, but there's coordination among experimenters. Even if one can make some choices regarding the statistics, it does not mean that all raw data must be accessible. The result shows that (when designing experiments) advance coordination can be beneficial (e.g. one could consider reporting a directional statistic, employ shared randomness with a public seed, etc.). \n\nWe have revised our explanation in the article to better reflect these contributions.\n\n*2 -- The connection to learning theory:* We agree with the reviewer that our paper could be published in a (good) statistics journal as well. However, despite the statistical nature of the paper, we believe that the problem we investigate is relevant to various fields in machine learning, particularly in scenarios where data/evidence aggregation occurs. So far these areas had limited theoretical underpinning and deriving guarantees (from a statistical perspective) helps better understanding these phenomena and improves the explainability of these methods. We believe that NeurIPS provides a suitable multidisciplinary platform to reach an audience interested in these intersections between statistics and machine learning and also appreciate a theoretical oriented work. \n\n*2 -- Accessibility:* \nWe will follow your advice to include a section that translates the technical details into an intuitive understanding. We thank the Reviewer for their take on the regimes and intuition for which testing strategies are optimal, we have adopted this intuition in the introduction and setting sections of our new version of our article. This should help readers unfamiliar with the depth of the statistical methods to grasp the key concepts.\n\nWe thank the Reviewer for pointing out several of the typos, these are corrected in the new version of the paper. "}]

Author rebuttal:
First of all we would like to thank the Reviewers for carefully reading our paper and their interest in our work. We are happy to hear that the majority of the reviewers found our paper "well presented" (aj9v), "well-written and organised" (qxf1),  written in a clear and without excessive statistical jargon (vLRc), "extremely well written" and has "textbook" quality" (ThPF). We also thank that the reviewers found our work "original" (ThPF, vLRc) with "sound theoretical results" (GjrH)
"up to small typos, correct mathematical proofs" (aj9v) and that "The theory and experiments are in perfect accordance and complement each other."

The referees have also raised a few concerns and provided several suggestions which we have addressed point-by-point in the individual rebuttals. Here we collect the main changes in the manuscript.

* We have extended the discussion of our results and numerical analysis following the suggestions of the referees (see the comments below for details). We have also corrected the typos pointed out by the referees.
* We have clarified the use of misinterpretable jargons to further improve the readability.
* We have extended the simulation study, in the new experiment we use substantially more repetitions resulting in a more accurate picture with negligible randomness visible.
* We have expanded on the relevance of our work to the field of machine learning.
* We have added references to existing literature on multiple testing where error rate is of concern.
* We provide a more extensive description the model and its connection to other, practically more important statistical models, whilst also emphasising the limitations of this model.
* We added more intuitive explanation as well as interpretation of which method / regime applies in practice, depending on the experiment design.

We would like to thank all the Reviewers for their consideration.

Meta-review: This meta review is based on the reviews, the authors rebuttal and the discussions with the reviewers, discussions with the SAC, and ultimately my own judgement on the paper. There was a consensus that the paper contributes sound and interesting contributions to the theoretical analysis of decision making. I feel this work deserves to be featured at NeurIPS and will attract interest from the community. I would like to personally invite the authors to carefully revise their manuscript to take into account the remarks and suggestions made by reviewers. Congratulations!
Decision: Accept (poster)