SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When the dimension is small relative to the number of trials, as exhibited in the previous section, optimal strategies include information on the directionality of the observation vector. In this section we show that in this regime, there could be an additional benefit from allowing mild coordination between the trials through employing shared randomness, e.g. a shared random seed between the trials. Such a phenomenon has been observed before in the distributed testing literature [3, 2, 39, 40], which forms the basis of our analysis below.
We consider the following variation on Assumption  $\square$ , where the key difference is that the source of randomness is allowed to be shared between the  $m$  trials.
Assumption 4. For functions  $f_{j}:\mathbb{R}^{d}\times \mathbb{R}\to \mathbb{R}$  and a random variable  $U$  which is independent of the data  $X$ , the  $j$ -th test statistic  $S^{(j)} = f_{j}(X^{(j)},U)$  satisfies  $\mathbb{E}_0|S^{(j)}|\leq M$  for some  $M > 0$  and all  $j = 1,\ldots ,m$ .
Test statistics satisfying this assumption shall be referred to as shared randomness (or public coin) protocols.
The theorem below establishes the optimal rate when coordination through shared randomness is allowed. When the number of trials is small compared to the dimension (i.e.  $m \lesssim d / \log m$ ), there is no difference between protocols that coordinate using shared randomness or those without coordination. In fact, the optimal rate  $(\rho^2 \asymp \sqrt{d} / (\sqrt{mn}))$  in this case is reached by the test (7) or the ones below it, which do not employ shared randomness. However, when the number of trials is large compared to the dimension (i.e.  $m \gtrsim d$ ), the testing rate substantially improves in the shared randomness protocols.
Theorem 3. Let  $S^{(1)}, \ldots, S^{(m)}$ ,  $C_m$  and  $T_\alpha$  satisfy Assumptions 2-4. Then there exists a constant  $c > 0$  depending only on  $L$ ,  $p$ ,  $q$  and  $M$ , such that if
$$
\rho^ {2} \leq c \frac {\left(\sqrt {m} \wedge \sqrt {\frac {d}{\log (m)}}\right) \sqrt {d}}{m n}, \tag {11}
$$
it holds that  $\sup_{f\in H_{\rho}}\mathbb{P}_{f}\left(T_{\alpha} = 0\right) > 2 / 3$  for all  $n,m,d\in \mathbb{N}$  and any level  $\alpha \in (0,0.1]$ .
At the same time, for all  $\alpha, \beta \in (0,1)$  there exists a constant  $C_{\alpha,\beta} > 0$  depending only on  $\beta$ ,  $L$ ,  $p$ ,  $q$ , the function  $\alpha \mapsto \kappa_{\alpha}$  and  $M$ , such that if
$$
\rho^ {2} \geq C _ {\alpha , \beta} \frac {\left(\sqrt {m} \wedge \sqrt {d}\right) \sqrt {d}}{m n} \tag {12}
$$
it holds that  $\sup_{f\in H_{\rho}}\mathbb{P}_{f}\left(T_{\alpha} = 0\right)\leq \beta$  for some test  $T_{\alpha}$  of level  $\alpha$  satisfying Assumptions
Remark 2. Similarly to Theorem [1] the choice of ranges  $0 < \alpha \leq 0.1$  and  $\beta = 2/3$  in the lower bound result is arbitrary, other choices are also possible as presented in the proof.
A shared randomness method that attains the rate in (12) is given next. Consider drawing an orthonormal  $d \times d$  matrix  $U$  taking values from the uniform measure on such matrices. As a test statistic, each trial computes  $(\sqrt{n} U X^{(j)})_1$ , which is a  $N(0,1)$  random variable under the null hypothesis. A level  $\alpha \in (0,1)$  meta-level test is then given by combining the local test statistics as
$$
T _ {\alpha} := \mathbb {1} \left\{\left| \frac {1}{\sqrt {m}} \sum_ {j = 1} ^ {m} (\sqrt {n} U X ^ {(j)}) _ {1} \right| \geq \Phi^ {- 1} (1 - \alpha / 2) \right\}, \tag {13}
$$
where  $\Phi$  is the standard Gaussian CDF. The core idea here is that for each trial, the same 1-dimensional projection of the  $d$ -dimensional data is computed, where the projection is taken uniformly at random and the test is conducted along the projected direction. The above method corresponds to Stouffer's method for the p-values  $p^{(j)} = \Phi (\sqrt{n} (UX^{(j)})_1)$  for  $j = 1,\ldots ,m$ . Lemma  $\boxed{8}$  in the appendix shows that the above test attains a small Type II error probability whenever  $\rho^2\gtrsim d / (mn)$ .
3 Examples for meta-analysis methods
Combinations of independent test statistics that fall into the framework of Assumptions [1-4] are subject to the rate optimality theory established by the main theorems in Section 2. In this section, we look into common methods for combining p-values, e-values and other test-statistics, as mentioned in the introduction.
When the distribution under the null hypothesis of the test statistics are known, certain combinations are natural. For example, the sum of normal or chi-square test statistics is again normal or chi-square distributed, respectively. Similarly, voting based mechanisms typically rely on summing Bernoulli random variables. It is easy to see that these and similar combinations methods fall into the framework of Assumptions 1-4
For more specific test statistics, such as p-values or e-values, many general combination methods have been introduced in the literature. We cover some of the most prominent combination approaches for p-values and e-values in Section 3.1 and Section 3.2 respectively. The list of methods is certainly non-exhaustive and many more combination methods exist, but they serve as context for the range of techniques covered by our general theory. Our main results allow establishing lower bound rates for the ones listed below, whilst in Sections 2.1 and 2.2 attainability of these rates by some of the listed methods was exhibited.
3.1 Combinations of p-values
If  $p^{(1)}, \ldots, p^{(m)}$  are p-values obtained from  $m$  independent test statistics concerning the same hypothesis, then under the null  $p^{(j)} \sim {}^{iid} U(0, 1)$ . One can aim to combine the  $m$  p-values to form a test  $T_{\alpha} \equiv T_{\alpha}(p^{(1)}, \ldots, p^{(m)})$  with Type I error probability  $\alpha$ , which hopefully has higher power than a test based on one of the individual p-values. Below we list standard methods in the literature.
- Fisher's method [18]. Because the variables  $-2\log p^{(j)}$  's are iid  $\chi_2^2$ -distributed under the null hypothesis, their sum follows a  $\chi_{2m}^2$ -distribution. Therefore the combination method  $\sum_{j=1}^{m} -2\log p^{(j)}$  results in a  $\chi_{2m}^2$  distributed random variable, and the corresponding quantile function provides level- $\alpha$  one-sided tests at the meta-level.
- Similar flavour to Fisher's method are the combinations  $\sum_{j=1}^{m} -\log(1 - p^{(j)})$  (Pearson's method [32]),  $\sum_{j=1}^{m} -\log p^{(j)}(1 - p^{(j)})$  (the logit method / Mudholkar and George method [31]) and  $m^{-1/2}\sum_{j=1}^{m}(p^{(j)} - 1/2)$  (Edgington's method [14]).
- Order-based methods such as Tippett's method [42] based on  $\min \{p^{(1)},\ldots ,p^{(m)}\} \stackrel {H_0}{\sim}$  Beta(1,m).
- Methods based on inverse CDF's, such as by Stouffer et. al [36] based on  $m^{-1/2} \sum_{j=1}^{m} \Phi^{-1}(p^{(j)}) \sim \mathrm{N}(0, 1)$  under the null hypothesis.
- Generalized averages as considered in [48],  $T_{\alpha} = \mathbb{1}\left\{a_{r,m}M_{r,m}(p^{(1)},\ldots ,p^{(m)})\leq \alpha \right\}$ , where  $M_{r,m}(p^{(1)},\dots,p^{(m)})$  equals  $\left(m^{-1}\sum_{j = 1}^{m}(p^{(j)})^r\right)^{1 / r}$  for  $r\in \mathbb{R}\setminus \{0\}$ , the geometric mean, minimum (i.e. Tippett's method) and maximum for  $r = 0$ ,  $r\to -\infty$ , and  $r\to \infty$ , respectively. For  $r\in \{-\infty \} \cup [1 / (m - 1),\infty ]$ ,  $a_{r,m}$  can be taken to obtain precisely level  $\alpha$  tests (i.e.  $\mathbb{P}_0T_\alpha = \alpha$ ). We note that this means that canonical multiple testing methods (see e.g. [19]) such as Bonferroni's correction (which corresponds with taking as  $M_{r,m}$  the minimum and  $a_{r,m} = m$ ) also fall within our framework.
Lemma below shows that all the methods mentioned above fall into the framework of Assumptions 1-4. This means that the error rate lower bounds of Theorem 1 and Theorem 3 respectively, apply to the p-value combination methods listed above. That is, one cannot attain a better separation rate when considering the worst case Type II error probability for the alternative hypothesis in (2), with any of the p-value combination methods listed above. Whether Assumption 1 or 4 applies depends on whether shared randomness is used in generating the p-values. To confirm that Assumptions 3 and 2 apply to tests based on the combined p-values, some algebra is needed. The proof of the lemma is deferred to the appendix.
Lemma 1. Consider  $p$ -values  $p^{(1)}, \ldots, p^{(m)}$ , where each  $p^{(j)}$  depends on the local data  $X^{(j)}$  and possibly local randomness that is independent of the data. For each of the combination methods for  $p$ -values mentioned above and corresponding test  $T_{\alpha}$  of level  $\alpha \in (0,1)$ , the conclusions of Theorem 7 hold.
We remark that the p-values are obtained using shared randomness (i.e. in the sense of Assumption 1), the lower bound rate of Theorem 3 applies. Furthermore, as exhibited in Sections 2.1 and 2.2 for p-values corresponding to well chosen test statistics, these combination methods can achieve the theoretical limits established in Theorems 1 and 3, respectively.
3.2 Combining e-values
An  $e$ -value is a nonnegative random variable  $E$  such that  $\sup_{\mathbb{P}_0 \in H_0} \mathbb{P}_0 E \leq 1$ . The threshold test corresponding to  $E$  of level  $\alpha$  is  $\mathbb{1}\{E \geq \alpha^{-1}\}$ . This test yields a so-called strict p-value; for  $\mathbb{P}_0 \in H_0$  we have  $\mathbb{P}_0(E \geq \alpha^{-1}) \leq \alpha$  by Markov's inequality.
E-values lend themselves for combining outcomes of independent studies for two main reasons. First, they are easy to combine, see Section 4 in [49] for an indepth discussion of specific combination functions for independent e-values. Second, they are robust to misspecification and offer optional stopping/continuation guarantees [22]. Common examples of e-values are Bayes factors and likelihood ratios, which are nonnegative and have expectation equal to 1 in the case of a simple null hypothesis such as considered in this article.
Several combination methods (e-merging functions) were proposed in the literature. For instance, the product of independent e-values is also again an e-value. This was shown to weakly dominate any other combination of independent e-values in the sense that  $\Pi_{j=1}^{m} E^{(j)} \geq C_m(E)$ , for any  $E = (E^{(j)}) \in [1, \infty)^m$  and  $E \mapsto C_m(E)$  such that  $C_m(E^{(1)}, \ldots, E^{(m)})$  is an e-value for any independent e-values  $E^{(1)}, \ldots, E^{(m)}$ , see [49]. Similarly, the average of e-values is again an e-value. The product and the average are admissible in the sense that there is no e-merging function that strictly dominates them on  $[0, \infty]^m$ . The lemma below shows that these two, arguably most prominent e-value combination methods fulfill Assumptions 1-4 and hence the lower bounds derived in Theorems 1 and 3 apply.
Lemma 2. Consider  $e$ -values  $E^{(1)}, \ldots, E^{(m)}$ , where each  $E^{(j)}$  depends on the local data  $X^{(j)}$  and possibly local randomness that is independent of the data. Let  $C_m: \mathbb{R}^m \to \mathbb{R}$  correspond to either the average or the product and let  $T_{\alpha}$  be the corresponding threshold test of level  $\alpha \in (0,1)$ .
$$
T _ {\alpha} = \mathbb {1} \left\{C _ {m} \left(E ^ {(1)}, \dots , E ^ {(m)}\right) \geq \alpha^ {- 1} \right\}.
$$
If  $C_m$  is the product, assume in addition that  $\mathbb{E}_0|\log E^{(j)}|$  is uniformly bounded. Then, the conclusion of Theorem ① holds. In case the  $e$ -values are generated using shared randomness, then Theorem ③ applies.
In this section, we investigate the numerical performance of the testing strategies outlined in Section 2.1 on synthetic data sets. We compare the tests based on their receiver operating characteristic (ROC) curve. For a range of significance levels we compute for each tests the "true positive rate" (TPR) and "false positive rates" or (FPR), i.e. the fraction of the simulation runs in which the test correctly identifies the underlying signal, falsely rejects the null hypothesis, respectively. Plotting the TPR against the FPR (both given as a function of the significance level) provides us the ROC curve, visualizing the diagnostic ability of the test.
In our simulations we set  $m = 20$ ,  $n = 30$ , let  $d$  range from 2 to 20 and take  $\rho^2 = \sqrt{d} / (4n)$ . This value of  $\rho^2$  corresponds to a signal that is almost indistinguishable from noise using just a single trial, whilst consistently detectable if the data were to be pooled with  $m \approx 20$  (which increases the signal size to noise ratio effectively by a factor  $\sqrt{20} > 4$ ). For each level  $\alpha \in \{0.01, 0.02, \dots, 0.99\}$  we compute the power for different combination strategies 100 times, each time drawing a different  $f \in \mathbb{R}^d$  with  $\|f\|_2 = \rho$  according to  $f_i = d^{-1/2} \rho R_i$  and  $R_i$  iid Rademacher random variables for  $i = 1, \dots, d$ . As combination strategies, we compare the strategies (7), (13) and (8) from Section 2.
which are called "chi-square combined", "coordinated directional" and "uncoordinated directional" in the legend of Figure 4. In addition, we display the ROC curves for the chi-square test based on pooled data ("chi-square pooled") and that of a single trial ("single trial").
We make the following observations, in line with our theoretical findings. The meta-analysis methods based on combining the locally optimal chi-squared test statistics (yellow curves) substantially outperformed the chi-squared test statistics based on a single trial (blue curve), but was substantially worse than the chi-square test based on the pooled data (pink curve). Second note that the large dimensional case ( $d = 10$  and  $d = 20$ ) the best strategy is indeed to combine the local chi-square statistics (yellow curve), while in the low dimensional setting ( $d = 2$ ) it is more advantageous to combine the directional test statistics  $X_{i}^{(j)}$  (blue curve). Finally, note that allowing coordination between the trials by a shared randomness protocol can result in improved performance (green curve) compared to the independent experiments (blue curve). In fact this approach provides the best meta-analysis method in the small dimensional setting (e.g.  $d = 2$  and  $d = 5$  for small  $\alpha$ , which is the most interesting case).
In the appendix, Section A.6, we explore eight additional simulation settings, where we consider larger values of  $d$  and  $m$ . Whilst these simulations do not reveal additional phenomena to the ones observed in Figure 4, they do give insight into the relative performance of the testing methods for different values of  $d$  and  $m$ .
We briefly summarize our main contributions and discuss possible extensions and research directions. First, by establishing a connection between meta-analysis and distributed learning under communication constraints, we have provided a unified, theoretical framework for evaluating the behaviour of standard meta-analysis techniques. In our analysis, we considered the many normal means model, but these results can be extended to other more complex models as well, building on the connection with distributed computation. For example, minimax estimation rates under communication constraints were derived for other parametric models [53], density estimation [8], signal-in-Gaussian-white-noise [54, 38, 10], nonparametric regression [37] and in abstract settings [52] including binary and Poisson regression, drift estimation, and more. The normal means model allows for a tractable analysis, but results in this model are known to extend to more complicated models, such as discrete density testing (see e.g. [11]). With the due technical work, our results are expected to translate to these settings as well, but we leave this for future endeavor.
In the normal means model we show that by combining the locally optimal chi-square statistics at a meta-level one can gain a factor of  $\sqrt{m}$  compared to using a single trial. Nevertheless, regardless of the choice of the combination method, a factor of  $\sqrt{m}$  is lost compared to the scenario when all data from all trials are at our disposal. This loss is clearly visible even in small sample sizes, dimensions and trial numbers, as demonstrated in our numerical analysis, as can be seen in the corresponding ROC curves. For more complex models, such a numerical study can be a first step to quantify the efficiency of the meta-analysis method. We have also shown that in the small dimension - large number of trials setting combining the locally optimal chi-square statistics (or any rotationally invariant statistics for that matter) results in information loss and sub-optimal accuracy. In this case, better rates can be attained by test statistics based on the direction of the observations combined at the meta-level. In practice, one often cannot choose which test statistics can be obtained from independent trials. In such cases, the  $\sqrt{m}$ -factor loss in the case of e.g. rotationally invariant test statistics is of interest when considering power calculations. Meta-analysis approaches based on directional test statistics are designed for scenarios where individual datasets are not centrally collected, but there is some level of coordination among experimenters.
The assumption throughout the paper of homogeneity between the trials (i.e. each trial consisting of the same number of observations) simplifies the presentation, but the results can be extended to cases where the number of observations in each trial differ by constant factors. Situations where the number of observations differs greatly (e.g.  $k \ll m$  trials have as much observations as the other  $m - k$  trials combined) are certainly of interest, but beyond the scope of this paper.
Acknowledgements: Co-funded by the European Union (ERC, BigBayesUQ, project number: 101041064). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This research was partially funded by a Spinoza grant of the Dutch Research Council (NWO).
[1] ABRAMOVICH, F., BENJAMINI, Y., DONOHO, D. L., AND JOHNSTONE, I. M. Adapting to unknown sparsity by controlling the false discovery rate. The Annals of Statistics (2006), 584-653.
[2] ACHARYA, J., CANONNE, C. L., AND TYAGI, H. Distributed signal detection under communication constraints. In Conference on Learning Theory (2020), PMLR, pp. 41-63.
[3] ACHARYA, J., CANONNE, C. L., AND TYAGI, H. Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction. IEEE Transactions on Information Theory 66, 12 (Dec. 2020), 7835-7855.
[4] AERTS, S., LAMBRECHTS, D., MAITY, S., VAN LOO, P., COESSENS, B., DE SMET, F., TRANCHEVENT, L.-C., DE MOOR, B., MARYNEN, P., HASSAN, B., ET AL. Gene prioritization through genomic data fusion. Nature biotechnology 24, 5 (2006), 537-544.
[5] ALTMAN, E. I. Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. The journal of finance 23, 4 (1968), 589-609.
[6] ALTMAN, E. I., IWANICZ-DROZDOWSKA, M., LAITINEN, E. K., AND SUVAS, A. Distressed firm and bankruptcy prediction in an international context: A review and empirical analysis of altman's z-score model. Available at SSRN 2536340 (2014).
[7] BARAUD, Y. Non-asymptotic minimax rates of testing in signal detection. Bernoulli (2002), 577-606.
[8] BARNES, L. P., HAN, Y., AND ÖZGÜR, A. Lower bounds for learning distributions under communication constraints via fisher information. The Journal of Machine Learning Research 21, 1 (2020), 9583-9612.
[9] BIRNBAUM, A. Combining independent tests of significance. Journal of the American Statistical Association 49, 267 (1954), 559-574.
[10] CAI, T. T., AND WEI, H. Distributed nonparametric function estimation: Optimal rate of convergence and cost of adaptation. The Annals of Statistics 50, 2 (2022), 698-725.
[11] CARTER, A. V. Deficiency distance between multinomial and multivariate normal experiments. The Annals of Statistics 30, 3 (2002), 708 - 730.
[12] CHEN, Z. Optimal tests for combining p-values. Applied Sciences 12, 1 (2021), 322.
[13] CLEMENTS, N., SARKAR, S. K., AND GUO, W. Astronomical transient detection controlling the false discovery rate. In Statistical challenges in modern astronomy V (2012), Springer, pp. 383-396.
[14] EDGINGTON, E. S. An additive method for combining probability values from independent experiments. The Journal of Psychology 80, 2 (1972), 351-363.
[15] EFRON, B. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, vol. 1. Cambridge University Press, 2012.
[16] EVANGELOU, E., AND IOANNIDIS, J. P. Meta-analysis methods for genome-wide association studies and beyond. Nature Reviews Genetics 14, 6 (2013), 379-389.
[17] FINNER, H., AND ROTERS, M. Log-concavity and inequalities for chi-square, f and beta distributions with applications in multiple comparisons. Statistica Sinica (1997), 771-787.