SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Permutation test for the interaction measures
test-statistic  $\leftarrow ||\hat{\mathcal{K}}^d ||_{(\tilde{K}^1,\dots,\tilde{K}^d)}^2$    
for  $i\in \{\pi \}$  (here  $\pi = b|b^{\prime})$  do. initialize empty  $P$  -dimensional vector T   
for  $p = 1:P$  do for  $j = 1:d$  do if  $j\in \mathrm{argmin}(|x|)$  then  $x\in \{b,b'\}$ $\tilde{K}^j\gets \tilde{K}_{(s)}^j$  {kernel matrices after random permutations on the observations of  $X^{j}\})$ $T[p]\leftarrow ||\hat{\mathcal{K}}^d ||_{(\tilde{K}^1,\dots,\tilde{K}^d)}^2$  tmp  $\leftarrow \# \{p\in \{1,\ldots ,P\} |\mathbf{T}[p]\geq$  test-statistic} pval  $\leftarrow (\mathrm{tmp} + 1) / (P + 1)\{\mathrm{Monte - Carlo~approximation~[32]}\}$  if pval  $>a$  then reject  $\leftarrow 0$  {'simple correction' [35]}. break   
reject  $\leftarrow 1$    
return reject
By rejecting the null hypothesis of the Streitberg interaction test, it follows that we reject the null hypothesis of the Lancaster interaction test, as well as the null hypothesis of joint independence. The set of sub-hypotheses in joint independence is a subset of the set in the Lancaster interaction test, which is in turn a subset of the set in the Streitberg interaction test. The number of sub-hypotheses for an interaction measure is equal to the number of elements on the second level of the respective lattice.
For Streitberg interaction, this number is  $(2^{d - 1} - 1)$  as follows from the full partition lattice; for Lancaster interaction, this number is  $d$  due to the reduced Lancaster lattice; for joint independence the number of sub-hypotheses stays fixed as the corresponding 2-element lattice always contains precisely one element, i.e.,  $\hat{0}$  corresponding to  $\mathbb{P}_1\cdots \mathbb{P}_d$ .
Remark: Although our focus is on composite interaction tests to unveil high-order interactions by leveraging the vanishing of Streitberg and/or Lancaster interactions, these measures are also valuable to test other lower-order independence hypotheses wherein factorisations can also lead to vanishing interaction measures. For example, both Lancaster and Streitberg statistics can be implemented to test for joint independence (as shown in Section 6), and for marginal independence.
4.4 Generalised Interaction Measure
The measures described so far are only able to handle interactions within one group of variables. But what if we are interested in knowing whether a factorisation (e.g.,  $\mathbb{P}_{12}\mathbb{P}_{34}$ ) factorises further? To answer this, we can simply formulate an interaction measure based on a partition [44]. Because an interval of a lattice  $\mathcal{L}$  is a subset of the form  $[\sigma, \pi] = \{x \mid \sigma \leq x \leq \pi\}$  and is also a lattice [43], the interval lattice can be used to produce generalised interactions for multiple groups of variables:
Definition 3 (General interaction operator).
$$
\Delta_ {S} ^ {\pi_ {s}} \mathbb {P} = \sum_ {\sigma \leq \pi_ {s}} m (\sigma , \pi_ {s}) \mathbb {P} _ {\sigma} = \prod_ {b \in \pi_ {s}} \Delta_ {S} ^ {b} \mathbb {P}, \tag {12}
$$
where  $\pi_s$  are the partitions with no singletons,  $m$  are the Möbius coefficients, and  $b$  are the blocks in the partition  $\pi_s$ .
Here, we only consider blocks of size at least 2, since a factorisation cannot not be defined for singletons. It is then clear that  $\Delta_S^{\pi_s}\mathbb{P} = 0$  if any  $\Delta_S^b\mathbb{P}$  is zero, and the rejection statement on each group will be dependent on their own vanishing conditions. This provides a general framework to detect interactions within blocks of variables. The corresponding kernel-based tests can be constructed by estimating the norm of  $\Delta_{S^s}^{\pi_s}\mathbb{P}$ . In fact, the generalised Streitberg interaction measure can be expressed in terms of ordinary Streitberg interaction measures defined in Section 4.2 [44]:
$$
\Delta_ {S} ^ {\pi_ {s}} \mathbb {P} = \sum_ {\sigma \vee \pi_ {s} = \hat {1}} \prod_ {b \in \sigma} \Delta_ {S} ^ {b} \mathbb {P}. \tag {13}
$$
5 Computational Considerations
The time complexity of computing the Lancaster interaction estimator is  $\mathcal{O}(dn^2)$ , where  $d$  is the order of interaction and  $n$  is the number of samples. This follows immediately after centring the kernel matrices, since  $\hat{1}$  is the only partition with no singletons. In contrast, the Streitberg interaction has a larger number of partitions, given by the Bell number,  $B_d$  [45]. However, by simplifying the computation through centring, we can reduce this to the number of partitions with no singletons, given by  $F_d$  [46]. While this number is still combinatorial, it is significantly smaller compared to  $B_d$  (see Table 1 in Appendix G).
The resulting estimator of the Streitberg interaction criterion then consists of  $F_{d}$  inner products. The summation indices of the inner products come from both partitions and are at most  $d$  (when  $d$  is even) or  $d - 1$  (when  $d$  is odd). Whilst the summations can be contracted, the choice of contraction ordering plays a crucial role in the time complexity [47, 48, 49]. By computing the summations related to one partition in parallel, since the index sets from  $\pi_s$  and  $\pi_s'$  are disjoint, the overall complexity can be reduced to  $\mathcal{O}(n^{min(|\pi_s|,|\pi_s'|) + 1})$  (see Table 2 in Appendix 5). For example, the inner product between the kernel embeddings for  $\mathbb{P}_{1234}$  and  $\mathbb{P}_{12}\mathbb{P}_{34}$  is  $\frac{1}{n^3}\sum_i\sum_j\sum_kK_{ij}^1 K_{ij}^2 K_{ik}^3 K_{ik}^4$ , but an improved contraction ordering is  $\frac{1}{n^3}\sum_i[\sum_jK_{ij}^1 K_{ij}^2][\sum_kK_{ik}^3 K_{ik}^4]$ .
The time complexity of the Streitberg interaction estimator can be further reduced in a recursive setting. Specifically, if  $\pi_s \vee \pi_s' \prec 1$ , then the associative inner product can be expressed as a product of independent sums that can be found in lower order interactions. For example,  $\langle \mu_{\mathbb{P}_{12}\mathbb{P}_{34}}, \mu_{\mathbb{P}_{12}\mathbb{P}_{34}} \rangle$  can be decomposed as  $\langle \mu_{\mathbb{P}_{12}}, \mu_{\mathbb{P}_{12}} \rangle \langle \mu_{\mathbb{P}_{34}}, \mu_{\mathbb{P}_{34}} \rangle$  which would have already been computed for the pairwise interactions of  $\{X^1, X^2\}$  and  $\{X^3, X^4\}$ . This means that if one has already computed
Figure 2: High-order tests on synthetic data. (a) dHSIC is unable to detect the marginal  $\mathbb{P}_1\mathbb{P}_{2345}$  factorisation. (b) Only the Streitberg test is able to capture the factorisation without singletons  $\mathbb{P}_{12}\mathbb{P}_{345}$ . (c) Testing joint independence of a 5-way XOR example, Lancaster and Streitberg tests achieve higher power than dHSIC for the same number of samples ( $n = 80$ ). Only by increasing the number of samples to  $n = 1000$  does dHSIC display comparable power. (d) The modified dHSIC test requires substantially more samples than the Streitberg test to display comparable power when testing all sub-hypotheses of the interaction in the XOR example.
the inner products of  $d - 2$  variables (incremental by 2 since the smallest block is at least size 2), then the only remaining inner products to compute are between those embeddings associated with partitions  $\pi_s$  and  $\pi_s'$  such that  $\pi_s \vee \pi_s' = \hat{1}$ . In other words, the computation of inner products for high-order interactions can be reduced to the computation of lower order interactions, as long as the corresponding partitions can be joined to form a partition of all  $d$  variables. For further discussions on computational complexity, see Appendix G.
We first validate our family of  $d$ -order interaction tests (in particular,  $d = 5$ ) on synthetic datasets with ground truth interactions, and then investigate their application to a neuroimaging dataset. Unless stated otherwise, the significance level is set to  $\alpha = 0.05$ , sample size to  $n = 80$ , and we use Gaussian kernels with the median heuristic as the bandwidth.
Synthetic Experiments: Multivariate Gaussians. We first compare results from applying the dHSIC test (developed solely to detect joint independence) [32], and the Lancaster and Streitberg interaction tests to five-variable multivariate Gaussian distributions  $\mathcal{N}(\mu, \Sigma)$  with mean  $\mu$  and covariance  $\Sigma$ . For the first example,  $\mu = [0, 0, 0, 0, 0]$  and  $\Sigma_1 = [[1, 0, 0, 0, 0], [0, 1, \beta, \beta, \beta], [0, \beta, 1, \beta, \beta], [0, \beta, \beta, 1, \beta], [0, \beta, \beta, \beta, 1]]$  where  $0 \leq \beta \leq 1$  defines the interaction strength between variables and consequently  $\mathbb{P}_{12345} = \mathbb{P}_1\mathbb{P}_{2345}$ . For the second example, we have the same  $\mu$  and  $\Sigma_2 = [[1, \beta, 0, 0, 0], [\beta, 1, 0, 0, 0], [0, 0, 1, \beta, \beta], [0, 0, \beta, 1, \beta], [0, 0, \beta, \beta, 1]]$  and hence  $\mathbb{P}_{12345} = \mathbb{P}_{12}\mathbb{P}_{345}$ .
Figure 2(a-b) shows that dHSIC fails to capture the partial factorisation of the joint distribution in both cases. Whilst the Lancaster interaction test is able to detect the factorisation in the first experiment (Figure 2(a)), it fails on the second experiment where the factorisation contains no singletons (Figure 2(b)). The Streitberg interaction test is the only one that detects both factorisations.
Synthetic Experiments: 5-Way XOR Gate. Next, we investigate the power of the tests using a data set constructed using an XOR gate. We generate  $n$  samples of  $V, W, X, Y, Z \stackrel{i.i.d.}{\sim} \mathcal{U}(0,4), Z_{:i} = (V_{:i} + W_{:i} + X_{:i} + Y_{:i}) \mod 4$  and  $Z_{i+1:n} \sim \mathcal{U}(0,4)$  (where the samples  $[i + 1:n]$  act as noise). We then gradually increase the interaction proportion,  $0 \leq i/n \leq 1$ . By construction, this dataset does not contain pairwise, 3-way or 4-way interactions, and the 5-way interaction becomes increasingly easier to detect as  $i/n$  approaches 1.
Given that the joint distribution in this example cannot be factorised, all tests should reject joint independence. We show in Figure 2(c), that the rejection rates of joint independence for Streitberg and Lancaster approach one as the interaction level increases for a small number of samples  $(n = 80)$  whereas dHSIC displays low power for the same number of samples. Only by increasing the number
Figure 3: Concentration of high-order interactions in brain resting state networks. The percentage of rejected null hypotheses for  $d$ -order ( $d = 2, 3, 4, 5$ ) interaction tests, when the  $d$  regions are selected within each of seven brain resting state networks (RSNs) or drawn at random across the brain. For each order  $d$ , we carry out Fisher-exact tests for over-representation for each RSN vs. random. All Fisher tests are significant ( $p$ -value  $< 0.05$ ) except where shown as (n.s.). Abbreviations for RSNs: DMN: default mode network, SOM: somatomotor, VIS: visual, SAL: salience, DAN: dorsal attention network, FPN: frontoparietal network, LIM: limbic.
of samples to  $n = 1000$  do we observe comparable performance for dHSIC. To test for 5-way interactions in the XOR data set, we omit the Lancaster test which cannot detect factorisations into non-singletons. We thus compare the Streitberg interaction test with a modified dHSIC [34, 35] that tests each individual sub-hypothesis (e.g., when testing  $(X,Y) \perp (Z,W)$  it is sufficient to treat  $(X,Y)$  and  $(Z,W)$  as single variables and perform joint independence tests for two variables). We find again that the modified dHSIC requires a much larger sample size to achieve comparable performance to that of the Streitberg interaction test (Fig. 2(d)).
To investigate how the test power degrades when the sample size is decreased, we performed further experiments in Fig 8 in Appendix F. In all cases, we find that the performance of dHSIC degrades faster than that of Lancaster and Streitberg, i.e., the dHSIC null rejection rate decays substantially faster than that of Lancaster and Streitberg as the sample size decreases.
Neuroimaging Dataset. As a proof of concept of applications to real data, we apply the Streitberg interaction test to detect high-order interactions in brain activity data. The dataset consists of resting-state fMRI data from 50 unrelated subjects part of the Human Connectome Project [50, 51]. For each subject, the data consists of 100 time series capturing the activity of each of the regions in the Schaefer brain atlas [52]. Importantly, these 100 regions can be divided in 7 groups called 'Resting State Networks' (RSN) [53]. Experimental evidence has shown that regions within the same RSN perform similar functions [52], so we hypothesised that high-order interactions would be more probable within RSNs.
To avoid dealing with issues of non-stationarity in the time-series data, we first take temporal averages to obtain the mean activity per region and subject (see Appendix H for preprocessing steps). We then perform the Streitberg interaction test to sets of brain regions that are either taken from within the same RSN or sampled at random from the whole brain. In each case, we sample 500 sets of regions or take all possible combinations, whichever is lowest.
The results in Figure 3 align with our general hypothesis: 2-, 3-, 4- and 5-way interactions are significantly more common among regions belonging to the same RSN as compared to random sets of regions, confirming that our interaction test can successfully detect high-order interactions in real-world data. Notably, our results show a larger percentage of 2-way interactions in the SOM and VIS compared to regions such as the FPN or DAN, suggesting that there is increased redundancy in the system relative to these other regions. The SOM and VIS are structurally coupled, modular sensorimotor processing regions, that benefit from information redundancy to increase robustness. Moreover, the proportion of high-order interactions (3-, 4- and 5-way) to 2-way interactions is larger in SOM and VIS compared to FPN or DAN, potentially reflecting the differing balance of redundancy and synergy in these brain regions [54]. Additional analyses are shown in Appendix H, but the
neuroscience implications of these results, including observed variation across regions, will be the object of future work.
In this work, we have established formal connections between lattice theory and a family of high-order interaction measures based on factorisations of the joint probability distribution. The link to lattice theory facilitates the derivation and interpretation of the measures; highlights the connection to simplicial complexes and other sublattices; and helps formulate the associated permutation tests more efficiently. We have shown empirically that the Lancaster test statistic is a good substitute for dHSIC to test for joint and marginal independence, whilst the Streitberg test statistic is able to better capture all factorisations of the joint distribution. Our work offers a rigorous and systematic approach for the reconstruction of high-order systems, but has some limitations. The computation of the Streitberg interaction estimator can be expensive and although we have proposed strategies to bring down the time complexity, the number of terms remains combinatorial (see Appendix G for discussions on practical limitations). In addition, our theoretical results rely on  $iid$  data, a strong assumption in many real-world applications. This leaves several open directions. In particular, future work will investigate whether the null distribution of  $\Delta_S^d P$  can be jointly approximated without the need for sub-hypotheses, thus reducing computational cost; how it can be accurately approximated when the data has temporal dependence; and what role high-order interactions may play in causal discovery.
MB acknowledges support by the EPSRC under grant EP/N014529/1 funding the EPSRC Centre for Mathematics of Precision Healthcare at Imperial, and by the Nuffield Foundation under the project "The Future of Work and Well-being: The Pissarides Review". RP acknowledges funding from the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project-ID 424778381-TRR 295.
For the experiments in Section 6, data were provided by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centres that support the NIH Blueprint for Neuroscience Research; and by the McDonnell centre for Systems Neuroscience at Washington University.
The authors would like to thank Jianxiong Sun, Xing Liu and Paul Expert for valuable discussions, Asem Alaa for help in maintaining the GitHub repository, and Andrea I. Luppi for assistance with the preprocessing of neuroimaging data.
[1] Federico Battiston, Enrico Amico, Alain Barrat, Ginestra Bianconi, Guilherme Ferraz de Arruda, Benedetta Franceschiello, Iacopo Iacopini, Sonia Kefi, Vito Latora, Yamir Moreno, et al. The physics of higher-order interactions in complex systems. Nature Physics, 17(10):1093-1098, 2021.
[2] Federico Battiston, Giulia Cencetti, Iacopo Iacopini, Vito Latora, Maxime Lucas, Alice Patania, Jean-Gabriel Young, and Giovanni Petri. Networks beyond pairwise interactions: structure and dynamics. Physics Reports, 874:1-92, 2020.
[3] S. Boccaletti, P. De Lellis, C.I. del Genio, K. Alfaro-Bittner, R. Criado, S. Jalan, and M. Romance. The structure and dynamics of networks with higher order interactions. Physics Reports, 1018:1-64, 2023.
[4] Alice Patania, Giovanni Petri, and Francesco Vaccarino. The shape of collaborations. EPJ Data Science, 6:1-16, 2017.
[5] Peter A Abrams. Arguments in favor of higher order interactions. The American Naturalist, 121(6):887-891, 1983.
[6] Margaret M Mayfield and Daniel B Stouffer. Higher-order interactions capture unexplained complexity in diverse communities. Nature ecology & evolution, 1(3):1-7, 2017.
[47] Lam Chi-Chung, P Sadayappan, and Rephael Wenger. On optimizing a class of multidimensional loops with reduction for parallel execution. Parallel Processing Letters, 7(02):157-168, 1997.
[48] Robert NC Pfeifer, Jutho Haegeman, and Frank Verstraete. Faster identification of optimal contraction sequences for tensor networks. Physical Review E, 90(3):033315, 2014.
[49] Frank Schindler and Adam S Jermyn. Algorithms for tensor network contraction ordering. Machine Learning: Science and Technology, 1(3):035001, 2020.
[50] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, WU-Minn HCP Consortium, et al. The WU-Minn Human Connectome Project: An overview. Neuroimage, 80:62-79, 2013.
[51] Matthew F Glasser, Stamatios N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al. The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage, 80:105-124, 2013.
[52] Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cerebral Cortex, 28(9):3095-3114, 2018.
[53] BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuncu, Danial Lashkari, Marisa Hollinshead, Joshua L Roffman, Jordan W Smoller, Lilla Zollei, Jonathan R Polimeni, et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of Neurophysiology, 2011.
[54] Andrea I Luppi, Pedro AM Mediano, Fernando E Rosas, Negin Holland, Tim D Fryer, John T O'Brien, James B Rowe, David K Menon, Daniel Bor, and Emmanuel A Stamatakis. A synergistic core for human brain evolution and cognition. Nature Neuroscience, 25(6):771-782, 2022.
[55] Thomas F Varley, Maria Pope, Maria Grazia Puxeddu, Joshua Faskowitz, and Olaf Sporns. Partial entropy decomposition reveals higher-order structures in human brain activity. arXiv preprint arXiv:2301.05307, 2023.
[56] Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515, 2010.
[57] Robin AA Ince. The partial entropy decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal. arXiv preprint arXiv:1702.01591, 2017.
[58] Lennart Van Hirtum, Patrick De Causmaecker, Jens Goemaere, Tobias Kenter, Heinrich Riebler, Michael Lass, and Christian Plessl. A computation of d (9) using fpga supercomputing. arXiv preprint arXiv:2304.03039, 2023.
Appendix to "Interaction Measures, Partition Lattices and Kernel Tests for High-Order Interactions"
First we prove the following:
Lemma 2. If  $\mathbb{P}_{1\dots d} = \mathbb{P}_i\mathbb{P}_{1\dots i - 1,i + 1\dots d}$  for arbitrary  $i$ , then  $\Delta_L^d\mathbb{P} = 0$
Proof. Without loss of generality let  $i = 1$ . To simplify the notation, we adopt a shorthand convention for expression partitions. For example, a partition  $\{\{X^1,X^3\} ,\{X^2\} ,\{X^4\} \}$  can be written as  $13|2|4$ . We denote the partition  $1|23\dots d$  as  $\pi$ . Notice that the factorisation only consists of an extraction of a singleton, i.e. factorise one variable out of the joint distribution. The Möbius coefficients [41] for the partial partition lattice above are,
$$
\mu (\sigma , \hat {1}) = \left\{ \begin{array}{l l} (- 1) ^ {d - 1} (d - 1) & \text {i f} \sigma = \hat {0} \\ (- 1) ^ {| \sigma | - 1} & \text {o t h e r w i s e} \end{array} \right..
$$
The partial partition lattice is constructed as
$$
\begin{array}{c} 1 2 3 \dots d \\ 1 | 2 3 \dots d \qquad 2 | 1 3 \dots d \qquad 3 | 1 2 \dots d \qquad \ldots \\ 1 | 2 | 3 \dots d \qquad 1 | 3 | 2 \dots d \qquad \ldots \qquad 2 | 3 | 1 \dots d \qquad \ldots \\ \vdots \\ 1 | 2 | 3 | \dots | d \end{array}
$$
Given  $\mathbb{P}_{1\dots d} = \mathbb{P}_1\mathbb{P}_{2\dots d}$ , the partial partition lattice associated with Lancaster interaction collapses into the following:
$$
\begin{array}{c} 1 | 2 3 \dots d \\ 1 | 2 3 \dots d \quad 2 | 1 | 3 \dots d \quad 3 | 1 | 2 \dots d \quad ... \\ 1 | 2 | 3 \dots d \quad 1 | 3 | 2 \dots d \quad ... \quad 2 | 3 | 1 | \dots d \quad ... \\ \vdots \\ 1 | 2 | 3 | \dots | d \end{array}
$$
If a partition  $\sigma$  is not a refinement of  $\pi$ , then it will be transformed to  $\sigma \wedge \pi$ , a partition one level down the lattice (as shown in the lattice above), e.g., let  $\sigma$  be  $2|13\dots d$ , then  $\sigma \wedge \pi$  is be  $2|1|3\dots d$ . When  $\sigma$  is a refinement of  $\pi$ , then  $\sigma \wedge \pi = \sigma$ .
The transformed partitions will be cancelled out with the refinements of  $\pi$  due to the fact that all partitions are counted exactly once and the partitions at two consecutive levels have alternating signs. On the  $(d - 1)$  level of the lattice, exactly  $(n - 1)$  partitions are not refinement of  $\pi$  which are just enough to cancel out  $(n - 1)$  number of  $\hat{0}$  at the bottom level.
When  $\mathbb{P}_{1\dots d} = \mathbb{P}_{\pi_s}$  where  $\pi_{s}$  only contains non-singleton blocks, the Lancaster interaction
$$
\Delta_ {L} ^ {d} \mathbb {P} = \prod_ {b \in \pi_ {s}} \Delta_ {L} ^ {b} \mathbb {P},
$$
which is not necessarily zero unlike the Streitberg interaction. It will be zero, however, if any of the  $\Delta_L^b\mathbb{P}$  is zero. If we factorise one singleton out of an arbitrary block  $b$ ,  $\pi_s$  will be a partition with one singleton block and hence  $\Delta_L^b\mathbb{P} = 0$  by Lemma 2 above and so does  $\Delta_L^d P$ . Obviously if more singletons are factorised out,  $\Delta_L^d P$  will remain zero.