SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lasse Vuursteen*  
Delft Institute of Applied Mathematics  
Delft University of Technology  
1.vuursteen@tudelft.nl
Botond Szabo  
Department of Decision Sciences and Institute for Data Science and Analytics  
Bocconi University  
botond.szabo@unibocconi.it
Aad van der Vaart  
Delft Institute of Applied Mathematics  
Delft University of Technology  
a.w.vandervaart@tudelft.nl
Harry van Zanten  
Mathematics Department  
Vrije Universiteit Amsterdam  
j.h.van.zanten@vu.nl
Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive a mathematical framework to study standard meta-analysis testing approaches in the context of the many normal means model, which serves as the platform to investigate more complex models.
We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us to mathematically quantify the cost of compressing  $m$  trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in a sub-optimal meta-analysis method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build on recent information theoretic developments in the latter field.
Given multiple data sets relating to the same hypothesis, one would like to combine the evidence. Sometimes, the full data sets are not available (e.g. due to privacy or proprietary reasons) or difficult to combine directly (e.g. due to the different experimental or observational setups). In such cases, the analysis must be carried out on the basis of the published results for each of the studies. Such "meta-analysis" can increase the statistical power by combining individually inconclusive or moderately significant tests, while keeping the false positive rate under control. Therefore, meta-analysis has received a lot of attention in various fields, for instance in genetics and system biology, when studying rare variants [4, 16] or in deep learning, for few shot image recognition and neural architecture search, see the review article [24].
The outcomes of the studies concerning hypothesis tests are, typically, summarized as real-valued test statistics and/or associated p-values. One expects the combination of  $m$  such p-values to result in an increase in power, but one also expects to pay a price relative to computing a test on the basis of the full, pooled data of the  $m$  trials. The question of how to optimally combine independent real-valued test statistics concerning the same hypothesis into a single test has an extensive literature. A multitude of methods for combining independent tests of significance exist. For combining p-values, this starts with Fisher, Tippett and Pearson in the nineteen-thirties, see [42, 18, 32, 36, 21, 28, 45, 14, 31, 50, 48, 12, 47] and references therein. In Section 3 we collect and describe the most popular and frequently used p-value combination techniques.
As noted in [9], there does not exist a general uniformly most powerful p-value combination method for all alternative hypotheses. The distribution of a p-value or its underlying test statistic under the alternative hypothesis should be taken into consideration when selecting a method of combination. The performance of different p-value combination techniques was investigated extensively by empirical experiments in various synthetic and real world scenarios, see for instance [29, 51]. However, a unified, general theoretical description is lacking, especially in non-trivial, multi-dimensional composite testing problems, where the likelihood ratio test is not necessarily uniformly most powerful.
E-values are an increasingly popular and important notion of evidence, see [35, 22, 34]. E-values allow the combination of several tests in a straightforward manner while preserving the prescribed level of the tests (see Section 3.2). Formally, e-values are nonnegative random variables whose expected values under the null hypothesis are bounded by one. In contrast to p-values defined by probabilities, e-values are defined by expectation. This imposes significant differences in their interpretation, application and combination compared to the more standard p-values. However, as for p-values, very little is known about the power of these combination procedures. Theoretical results focus on specific optimality criteria, for instance the worst-case growth-rate (GROW), see [22]. However, these do not directly imply guarantees on the testing power, which is the main focus in practice.
Our focus is on multidimensional models, where a certain loss in power is to be expected, since combining multidimensional data into a real-valued statistic (e.g. p-value or e-value) requires data compression. Typically, summary statistics are combined by some "reasonable" function  $C_m: \mathbb{R}^m \to \mathbb{R}$ , where "reasonable" means that  $C_m$  should not exploit the richness of the real numbers to encode the data in full. We aim to quantify the loss of summarizing, the gain of performing a meta-analysis and the best testing strategies in the individual experiments meta-analysis.
We consider the signal detection problem in the many normal means model, see Section 2 for the detailed description. One possible interpretation of this testing problem is to learn whether a treatment has an effect on any of the dimensions investigated. This model is directly applied in several fields where high-dimensional statistics and machine learning settings are concerned, such as detecting differentially expressed genes [33, 27, 30, 41, 15], bankruptcy prediction for publicly traded companies using Altman's Z-score in finance [5, 6], separation of the background and source in astronomical images [13, 23], and wavelet analysis [11, 26]. Furthermore, the model allows for tractable computations and it typically serves as the platform to investigate more difficult statistical and learning problems, including high- and infinite-dimensional models, see for instance [25, 43, 15, 20]. In each experiment  $j \in \{1, \dots, m\}$  the observations are summarized by an appropriate real-valued summary statistic  $S^{(j)}$ . These local test statistics (e.g. p- or e-values) are combined into  $C_m(S^{(1)}, \ldots, S^{(m)})$ . We consider a general class of combination functions  $C_m$ , requiring only Hölder type continuity. This introduces only a mild restriction, and includes many standard meta-analysis techniques, for instance the standard p-value combination methods (see Section 3.1); e-value techniques (see Section 3.2); and other ad hoc and natural test statistic combination approaches, see the beginning of Section 3 for additional examples.
Our setting provides a principled and unified framework to study the power of standard meta-analysis testing methods. Within the framework of the many normal means model, we derive a minimax lower bound for the testing (separation) error and provide test statistics with associated combination methods that attain this theoretical limit (up to a logarithmic factor). Our results reveal that there is a certain unavoidable loss associated with compressing the data of each experiment to a real valued test statistic. We see that while it is always possible to obtain better testing rates using  $m$  trials instead of the best possible test based on a single trial, there is always a loss incurred when compared to the
full, pooled data and optimal test in moderate- to large dimensional problems. Our theoretical results quantify these gains and losses in terms of the dimension  $d$ , sample size  $n$  and number of trials  $m$ .
Furthermore, we observe an elbow effect, which occurs when the number of trials is large compared to the dimension of the signal. In this regime, combinations of the (locally) optimal test in each individual trial performs sub-optimally as a whole when aggregated and meta-analysis approaches based on directional test statistics are shown to perform better. Finally, we show that the performance of the meta-level tests can substantially improve (in certain regimes, depending on  $d, m, n$ ) if a certain amount of coordination between the trials is allowed (e.g. by having access to the same random seed). For the theoretical analysis of meta-analysis techniques we derive connections with the distributed statistical learning literature under communication constraints. Our paper builds on the recent information theoretical developments in distributed testing [2, 39, 40], allowing us to address several fundamental questions for the first time with mathematical rigor.
The paper is organised as follows. In Section 2 we introduce the mathematical framework we consider in our investigation and present the corresponding minimax testing lower bound results. Next in Subsection 2.1 we show that the derived results are sharp by providing several meta-analysis approaches attaining the limits. Then we investigate the benefits of allowing a mild coordination between the trials in Subsection 2.2. We collect and discuss the standard p- and e-value combination methods in Section 3 and demonstrate our theoretical results numerically on synthetic data sets in Section 4. We discuss our results and derive conclusions in Section 5. The proofs of our results are deferred to the Appendix. In Section A.1 we present the proof of our main results while the proofs of the technical lemmas are given in A.2
Notation: For two positive sequences  $a_{n}, b_{n}$  we write  $a_{n} \lesssim b_{n}$  if the inequality  $a_{n} \leq C b_{n}$  holds for some universal positive constant  $C$ . Similarly, we write  $a_{n} \asymp b_{n}$  if  $a_{n} \lesssim b_{n}$  and  $b_{n} \lesssim a_{n}$  hold simultaneously and let  $a_{n} \ll b_{n}$  denote that  $a_{n} / b_{n} = o(1)$ . Furthermore, we use the notations  $a \vee b$  and  $a \wedge b$  for the maximum and minimum, respectively, between  $a$  and  $b$ . Throughout the paper  $c$  and  $C$  denote global constants whose value may change from one line to another.
In our analysis, we consider the localized version of the many normal means model, tailored to investigating meta-analysis techniques. We assume that in each local trial or experiment  $j \in \{1, \dots, m\}$  we observe a  $d$ -dimensional random variable  $X^{(j)} \in \mathbb{R}^d$ , subject to
$$
X ^ {(j)} = f + \frac {1}{\sqrt {n}} Z ^ {(j)}, \quad Z ^ {(j)} \stackrel {i i d} {\sim} N (0, I _ {d}), \quad j = 1, \dots , m, \tag {1}
$$
for some unknown  $f \in \mathbb{R}^d$ . We denote by  $\mathbb{P}_f$  the joint distribution of the observations and let  $\mathbb{E}_f$  be the corresponding expectation. We note that this framework is equivalent to having  $n$  independent  $N(f, I_d)$  observations within each local sample.
Our goal is to test the presence or absence of the "signal component"  $f \in \mathbb{R}^d$ . More formally, we consider the simple null hypothesis  $H_0: f = 0$  versus composite alternative hypothesis  $H_{\rho}: \| f \|_2 \geq \rho$ , for some  $\rho > 0$ . This corresponds to testing for joint significance of variables, such as the presence of an effect of a treatment on any of the dimensions investigated. The difficulty in distinguishing the hypotheses depends on the effect size, the sample size and the dimension  $d$ . Here,  $\rho$  can be seen as the smallest effect size deemed important.
For a  $\{0,1\}$ -valued test  $T$ , define the testing risk  $\mathcal{R}(T,H_{\rho})$  as the sum of the Type I error probability and worst case Type II error probability, i.e.
$$
\mathcal {R} (T, H _ {\rho}) := \mathbb {P} _ {0} (T = 1) + \sup  _ {f \in H _ {\rho}} \mathbb {P} _ {f} (T = 0). \tag {2}
$$
In the case of a single trial (i.e.  $m = 1$ ), this testing problem is known to have minimax separation rate or "detection boundary"  $\rho^2 \asymp \sqrt{d} / n$ .
This means that if  $\rho^2 \gg \sqrt{d} / n$ , there exist consistent tests  $T \equiv T_{d,n}$  in the sense that  $\mathcal{R}(T, H_\rho) \to 0$  whilst no consistent tests exist when  $\rho^2 \ll \sqrt{d} / n$ . That is, for effect sizes of smaller order than
2For any asymptotics in  $\rho ,d$  and  $n$  such that  $\rho^2\gg \sqrt{d} /n$
$\sqrt{d} / n$ , the null hypothesis cannot be consistently distinguished from the alternative hypothesis. Such a testing rate is attainable through a chi-square test based on  $\| \sqrt{n} X^{(1)}\|_2^2$  (see e.g. [7]).
In case of  $m$  trials, if the full data were pooled (with aggregated sample size  $nm$ ), the minimax separation rate would be  $\sqrt{d} / (mn)$ . However, pooling the data might not be possible or allowed in practice and often only real-valued test statistics are available that describe the significance in the local problems (e.g. a p- or an e-value). These  $m$  test statistics  $S^{(j)}$ ,  $j = 1, \dots, m$ , then can be combined with some combination function  $C_m: \mathbb{R}^m \to \mathbb{R}$ , providing the test statistic in the meta-analysis. We now ask whether the above pooled testing rate is attainable with this meta-analysis procedure.
Without any restrictions on the test statistics  $S = (S^{(1)},\ldots ,S^{(m)})$  or the combination function  $C_m$ , any of the conventional optimal "full-data" tests can be reconstructed, since the real numbers and mappings between the real numbers form an overly rich class. We wish to restrict our analysis to  $S$  and  $C_m$  that are reasonable in practice and capture (most of) the relevant meta-analysis methods as listed in Section 3.
Based on each of the local observations  $X^{(j)}$ , a real-valued test statistic  $S^{(j)}$  is computed, where each  $S^{(j)}$  is a function of  $X^{(j)}$  and possibly a source of randomness  $U^{(j)}$  independent of  $X := (X^{(1)}, \ldots, X^{(m)})$ .
Assumption 1. For measurable functions  $f_{j}:\mathbb{R}^{d}\times \mathbb{R}\to \mathbb{R}$  and independent random variables  $U^{(1)},\ldots ,U^{(m)}$  which are independent of the data  $X$ , the  $j$ -th test statistic  $S^{(j)} = f_{j}(X^{(j)},U^{(j)})$  satisfies  $\mathbb{E}_0|S^{(j)}|\leq M$ , for some  $M > 0$ ,  $j = 1,\dots,m$ .
We consider Hölder continuous combination functions  $C_m: \mathbb{R}^m \to \mathbb{R}$ . Arguably, this is the most important assumption in ruling out bijections between  $\mathbb{R}^d$  and  $\mathbb{R}$ . This ensures that a small change in the underlying local test statistics cannot result in a large change in the combination of test statistics  $C_m(S^{(1)}, \ldots, S^{(m)})$ .
Assumption 2. There exist  $L, p, q > 0$  such that for all  $s, s' \in \mathbb{R}^m$
$$
\left| C _ {m} (s) - C _ {m} \left(s ^ {\prime}\right) \right| \leq L \left(\sum_ {j = 1} ^ {m} \left| s _ {j} - s _ {j} ^ {\prime} \right| ^ {p}\right) ^ {q}. \tag {3}
$$
The special case of  $p = 2$  and  $q = 1/2$  leads to Lipschitz continuous functions. Assumption 1 and Assumption 2 should be considered in conjunction. By rescaling and centering test statistics  $S^{(j)}$ , one can typically obtain test statistics satisfying Assumption 1. Rescaling and centering typically does affect how the test statistics need to be combined, which might "break" Assumption 2
Finally, following the standard testing approach, we compare the aggregated test statistics  $C_m(S^{(1)}, \ldots, S^{(m)})$  to a threshold value. If the combined test statistics result in a large enough value, the null hypothesis of no effect is rejected. We note here that two-sided tests can be written as one-sided tests through straightforward transformations (e.g. centering and taking absolute value). More formally, we consider tests  $T_{\alpha}$  of level  $\alpha$  satisfying the following assumption.
Assumption 3. There exists a strictly decreasing function  $\alpha \mapsto \kappa_{\alpha}$  so that
$$
T _ {\alpha} = \mathbb {1} \left\{C _ {m} \left(S ^ {(1)}, \dots , S ^ {(m)}\right) \geq \kappa_ {\alpha} \right\} \tag {4}
$$
satisfies  $\mathbb{P}_0T_\alpha \leq \alpha$
The map  $\alpha \mapsto \kappa_{\alpha}$  could be taken as the quantile function of  $C_m(S^{(1)},\ldots ,S^{(m)})$  under its null distribution if it is appropriately standardized. If  $\mathbb{E}_0C_m(S^{(1)},\dots,S^{(m)})$  is bounded in  $m$ , we can choose  $\kappa_{\alpha}$  equal to  $1 / \alpha$  times the upper bound, in view of Markov's inequality.
Our first main result, Theorem below, establishes a lower bound for tests of the form  $\mathbb{A}$  and  $C_m$  and  $S$  satisfying the above assumptions. More concretely, under our assumptions, any test  $T_{\alpha}$  (of level  $\alpha \leq 0.1$ ) has large Type II-error under alternatives with  $\rho^2$  of smaller order than  $(\sqrt{m} \wedge \frac{d}{\log(m)})\sqrt{d} / (mn)$ . When the number of trials is small compared to the dimension (i.e.  $m\log^2(m) \leq d^2$ ), this means that the separation rate is at least  $\sqrt{d} / (\sqrt{mn})$ . Thus even though there is a benefit in terms of separation rate compared to testing based on just a single trial, the
gain is at best the square root of what one would gain based on testing on the pooled data. When  $m\log^2 (m)\geq d^2$ , the rate in the lower bound changes to  $d\sqrt{d} /(mn\log (m))$ , resulting in an elbow effect.
Theorem 1. Let  $S^{(1)}, \ldots, S^{(m)}$ ,  $C_m$  and  $T_\alpha$  satisfy Assumptions  $\boxed{L-3}$  with  $T_\alpha$  of level  $\alpha \in (0,0.1]$ . Then there exists a constant  $c > 0$  depending only on  $L, p, q$  and  $M$ , such that if
$$
\rho^ {2} \leq c \frac {\left(\sqrt {m} \wedge \frac {d}{\log (m)}\right) \sqrt {d}}{m n}, \tag {5}
$$
it holds for all  $n,m,d\in \mathbb{N}$  that
$$
\sup  _ {f \in H _ {\rho}} \mathbb {P} _ {f} \left(T _ {\alpha} = 0\right) \geq 3 / 4. \tag {6}
$$
Remark 1. The ranges of values  $0 < \alpha \leq 0.1$  and  $\beta = 3/4$  for the Type I and II errors, respectively, are arbitrary. Similar results hold for different choices as well. For instance, one can take arbitrary  $\alpha \in (0,1/5]$  and  $\beta \in (0,2/3]$ , see the proof of the theorem for details. The result implies in particular that consistent testing is not possible for signals of a smaller order than the right hand side of (5), where asymptotics can be considered in  $n,m$  and  $d$  simultaneously.
In the next section we show that the lower bounds in the theorems above are sharp (up to a logarithmic factor).
2.1 Rate optimal combination methods
To attain the lower bound rate derived in Theorem I, different tests can be considered. The optimal rate displays an elbow effect around  $m \asymp d^2$ . When the dimension is large compared to the number of trials  $m$  (i.e.  $m \lesssim d^2$ ), strategies that combine p-values for the optimal local tests (based on  $\| \sqrt{n} X^{(j)}\|_2^2 \sim H_0 \chi_d^2$ ), turn out to achieve the optimal rate, as exhibited below. Such a test statistic is invariant to the directionality of  $X^{(j)}$  and invariant under the model in the sense that the resulting power for the alternative  $\mathbb{P}_f$  or  $\mathbb{P}_g$  is the same as long as  $\| f\|_2 = \| g\|_2$ .
On the other hand, when the dimension is small compared to the number of trials (i.e.  $m \gtrsim d^2$ ), optimal strategies exhibited below use information on the direction of  $X^{(j)}$ . In fact, we show in Theorem 4 in the Appendix that if no such information is available (i.e. the events defined by the signs of the  $(X^{(j)})_{j=1,\dots,m}$  vector are not contained in the sigma algebra generated by the test statistics  $S$ ), one cannot obtain a rate better than  $\sqrt{d} / (\sqrt{mn})$ . This implies that by combining the locally optimal test statistics  $S^{(j)} = \|\sqrt{n} X^{(j)}\|_2^2$  (or their arbitrary functions, e.g. the corresponding local p-values) would result in information loss and hence sub-optimal rates in the meta-analysis.
Furthermore, it turns out, in accordance with the empirical literature discussed in the introduction, that there does not exist a uniquely best meta-analysis method. In fact, multiple standard meta-analysis techniques provide (up to a logarithmic factor) optimal rates, see below for some standard approaches attaining the lower bounds derived in Theorem 1.
First we consider the scenario when the dimension  $d$  of the model is large compared to the number of trials  $m$ , i.e.  $m \lesssim d^2$ . Locally the optimal test is based on the test statistic  $\| \sqrt{n} X^{(j)}\|_2^2 \stackrel{H_0}{\sim} \chi_d^2$ . A natural way to combine these statistics would be to sum these locally optimal test statistics to obtain
$$
T _ {\alpha} = \mathbb {1} \left\{\sum_ {j = 1} ^ {m} \left\| \sqrt {n} X ^ {(j)} \right\| _ {2} ^ {2} \geq F _ {\chi_ {d m} ^ {2}} ^ {- 1} (1 - \alpha) \right\}, \tag {7}
$$
which has level  $\alpha$ . Alternatively, one could also apply p-value combination methods, such as Fisher's or Edginton's method based on the p-value  $p^{(j)} = 1 - F_{\chi_d^2}(\| \sqrt{n} X^{(j)}\|_2^2)$ , see Section 3. Lemma 6 in the appendix establishes that these tests are rate optimal.
Second, consider the case that the number of trials is large compared to the dimension, i.e.  $m \gtrsim d^2$ . Rate optimal tests can be constructed based on a variation of Edgington's or Stouffer's method, see Section 3 for their descriptions. Taking a partition of  $\{1, \ldots, m\} = \cup_{i=1}^{d} \mathcal{I}_i$  where  $|\mathcal{I}_i| \asymp m / d$  and
setting  $S^{(j)} = \sqrt{n} X_{i}^{(j)}$  if  $j\in \mathcal{J}_i$ , the meta-level test
$$
T _ {\alpha} = \mathbb {1} \left\{\frac {\sqrt {d}}{m} \sum_ {i = 1} ^ {d} \left(\sum_ {j \in \mathcal {J} _ {i}} S ^ {(j)}\right) ^ {2} \geq d ^ {- 1 / 2} F _ {\chi_ {d} ^ {2}} ^ {- 1} (1 - \alpha) \right\} \tag {8}
$$
achieves the lower bounds. The above test is similar to employing Stouffer's method for each of the coordinates and averaging, i.e. computing approximately  $m / d$  iid p-values  $p^{(j)} = \Phi (\sqrt{n} X_i^{(j)})$  for  $j\in \mathcal{I}_i$  and applying the inverse Gaussian CDF  $\Phi^{-1}(p^{(j)})$ . Alternatively, the following variation of Edgington's method,
$$
T _ {\alpha} = \mathbb {1} \left\{\frac {\sqrt {d}}{m} \sum_ {i = 1} ^ {d} \left(\sum_ {j \in \mathcal {J} _ {i}} \left(p ^ {(j)} - \frac {1}{2}\right)\right) ^ {2} \geq \kappa_ {\alpha} \right\}, \tag {9}
$$
is also rate optimal, as proven in Lemma 7 in the appendix. Essentially, these strategies divide the trials across the  $d$  different directions, and combines the evidence for each of the directions. Theorem 4 affirms that the information on the "direction" of the data is crucial to achieve the optimal rate in the  $m \gtrsim d^2$  case, by showing that strategies that do not contain such information (rotationally invariant strategies such as norm-based test statistics) achieve the rate  $\sqrt{d} / (\sqrt{mn})$  at best. We summarize the above testing upper bounds in the theorem below.
Theorem 2. For all  $\alpha, \beta \in (0,1)$  there exist  $S, C_m: \mathbb{R}^m \to \mathbb{R}$  and tests  $T_\alpha$  of level  $\alpha$  satisfying Assumptions [7-3] such that if
$$
\rho^ {2} \geq C _ {\alpha , \beta} \frac {(\sqrt {m} \wedge d) \sqrt {d}}{m n}, \tag {10}
$$
$$
\sup  _ {f \in H _ {\rho}} \mathbb {P} _ {f} \left(T _ {\alpha} = 0\right) \leq \beta
$$
for a large enough constant  $C_{\alpha, \beta} > 0$  depending only on  $\alpha, \beta \in (0,1)$ , for all  $n, m, d \in \mathbb{N}$ .
2.2 Benefits of coordination between the trials