SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhaolu Liu $^{1}$  Robert L. Peach $^{2,3}$  Pedro A.M. Mediano $^{4}$  Mauricio Barahona $^{1*}$
$^{1}$ Department of Mathematics, Imperial College London, United Kingdom
$^{2}$ Department of Neurology, University Hospital Würzburg, Germany
<sup>3</sup>Department of Brain Sciences, Imperial College London, United Kingdom
$^{4}$ Department of Computing, Imperial College London, United Kingdom
Models that rely solely on pairwise relationships often fail to capture the complete statistical structure of the complex multivariate data found in diverse domains, such as socio-economic, ecological, or biomedical systems. Non-trivial dependencies between groups of more than two variables can play a significant role in the analysis and modelling of such systems, yet extracting such high-order interactions from data remains challenging. Here, we introduce a hierarchy of  $d$ -order interaction measures, increasingly inclusive of possible factorisations of the joint probability distribution, and define non-parametric, kernel-based tests to establish systematically the statistical significance of  $d$ -order interactions. We also establish mathematical links with lattice theory, which elucidate the derivation of the interaction measures and their composite permutation tests; clarify the connection of simplicial complexes with kernel matrix centring; and provide a means to enhance computational efficiency. We illustrate our results numerically with validations on synthetic data, and through an application to neuroimaging data.
There is increasing evidence that pairwise relationships are insufficient to model many real world systems [1, 2, 3]. The relevance of high-order interactions has been emphasised in many contexts, as relationships within social [4], ecological [5, 6], and biological systems [7, 8] frequently involve groups of three or more agents, beyond pairwise associations. Such high-order interactions can be neither trivially represented by a linear combination of dyadic relationships, as the presence of high-order interactions can significantly impact the dynamics on networked systems [9, 10, 11, 12, 13, 14, 15], nor can they be simply detected by joint independence tests which inherently ignore other possible factorisations of the joint distribution.
Direct measurements of group interactions are seldom available, leading to their omission, partial representation as projected pairwise interactions [16, 17, 18], or limited recovery through inference methods [19, 20]. Often, only indirect measurements of underlying interactions in real-world complex systems are available, in the form of  $iid$  or time-series data. Methods that learn high-order interactions from time-series data have been developed [11, 21], however, their reliance on heuristic measures makes their interpretation difficult. Alternative methods have been developed with strong foundations in statistics [22] and information theory [23, 24, 25], however, as we show later, these are often based
*Corresponding author: m.barahona@imperial.ac.uk
Figure 1: Factorisations of joint distributions  $\mathbb{P}_{1\dots d}$  for  $d = 2,3,4$ . The black dots indicate the marginal distributions of the singletons. The line, triangular and square shapes represent the joint distribution of two, three and four variables respectively. Different factorisations are presented as partitions of the  $d$  variables ordered by increasing cardinality from top to bottom, so that all the factorisations with the same number of independent blocks appear at the same level. Joint independence considers only the top and bottom levels for each  $d$ , whilst the Lancaster interaction considers all terms except those in the shaded region. The Streitberg interaction considers all partitions. Hence, for  $d = 2$ , we have  $\Delta_I^2\mathbb{P} = \Delta_L^2\mathbb{P} = \Delta_S^2\mathbb{P}$ ; whereas for  $d = 3$ , we have  $\Delta_I^3\mathbb{P} \neq \Delta_L^3\mathbb{P} = \Delta_S^3\mathbb{P}$ , and for  $d \geq 4$ , we have  $\Delta_I^d\mathbb{P} \neq \Delta_L^d\mathbb{P} \neq \Delta_S^d\mathbb{P}$ .
on an incomplete set of interactions and thus fail to capture all possible factorisations of the joint probability distribution [26, 27].
Kernel-based hypothesis tests provide a non-parametric, statistically robust framework for detecting relationships between variables from observational data. Such tests have been implemented for pairwise [28, 29, 30, 31] and  $d$ -variate joint independence [32, 33], and proven to be effective for non-trivial dependencies such as in the 3-way Lancaster test [34, 35]. However, to the best of our knowledge, cases where the number of variables exceeds 3 are still unexplored.
Here, we extend the capability for detection of high-order interactions by introducing a family of tests based on factorisations of the joint probability distribution that generalise systematically to any order  $d$ . At the head of this family, we introduce the Streitberg interaction test, which captures all factorisations of the joint distribution of order  $d$ . We further show that, despite the fact that the naive extension to  $d \geq 4$  of the Lancaster interaction excludes some of the factorisations of the joint distribution, not all is lost, and we detail the conclusions that can be drawn from rejecting the  $d$ -order Lancaster interaction. Furthermore, we show that: (i) lattice theory provides a coherent theoretical foundation for detecting high-order interactions; (ii) interaction measures can be systemically derived from partition lattices; (iii) the corresponding Hilbert-Schmidt norm for kernel embeddings can naturally be extracted from the product lattice; (iv) the composite permutation tests can be performed with regard to the second level of the lattice; and (v) the lattice formulation allows us to propose a generalised interaction measure that can be used to test whether a given factorisation can be factorised further. Despite the inescapable combinatorial nature of testing high-order interactions, we offer approaches to reduce the computational complexity of our  $d$ -order interaction tests informed by our links with lattice theory. Finally, we present numerical validations of our tests on synthetic data before applying them to real-world neuroimaging data.
2 Interaction Measures
The most basic form of interaction between variables occurs between two variables,  $X^1$  and  $X^2$ , and is often characterized by the lack of pairwise independence, i.e., the difference between the joint distribution  $\mathbb{P}_{X^1,X^2}$  and the product of the marginal distributions  $\mathbb{P}_{X^1}$  and  $\mathbb{P}_{X^2}$  (see Fig. 1,  $d = 2$ ). Let us consider  $d$  random variables  $\{X^1,X^2,\ldots ,X^d\}$  with joint distribution  $\mathbb{P}_{X^1,\dots ,X^d} \eqqcolon \mathbb{P}_{1\dots d}$  (for readability, hereafter we use this subscript notation). The  $d$  variables are jointly independent if and only if  $\mathbb{P}_{1\dots d}$  can be factorised as the product of the univariate marginal distributions. Hence we can introduce an interaction measure that vanishes only when the variables are jointly independent:
$$
\Delta_ {I} ^ {d} \mathbb {P} = \mathbb {P} _ {1 \dots d} - \prod_ {i = 1} ^ {d} \mathbb {P} _ {i}. \tag {1}
$$
When  $d > 2$ , however, the criterion for joint independence fails to capture high-order interactions, since additional factorisations must be considered. For instance, if  $\mathbb{P}_{123}$  could be factorised as
$\mathbb{P}_1\mathbb{P}_{23}$  (see Fig. 1,  $d = 3$ ), joint independence would be rejected, yet there is no 3-way interaction. Instead, we seek a way of identifying high-order interactions for  $d$  variables where all lower order independencies can be rejected.
One approach to address the shortcomings of joint independence in identifying high-order interactions for  $d > 2$  variables is to use a signed measure called the Lancaster interaction [34]. The Lancaster interaction for  $d = 3$  is defined as  $\Delta_L^3\mathbb{P} = \mathbb{P}_{123} - \mathbb{P}_1\mathbb{P}_{23} - \mathbb{P}_2\mathbb{P}_{13} - \mathbb{P}_3\mathbb{P}_{12} + 2\mathbb{P}_1\mathbb{P}_2\mathbb{P}_3$ .  $\Delta_L^3\mathbb{P}$  vanishes if  $\mathbb{P}_{123}$  can be factorised in any way, and has been implemented as a kernel-based test statistic to identify non-factorisable joint distributions [34, 35]. Lancaster also generalised this measure to the multivariate case with  $d$  variables:
$$
\Delta_ {L} ^ {d} \mathbb {P} = \prod_ {i = 1} ^ {d} \left(\mathbb {P} _ {i} ^ {*} - \mathbb {P} _ {i}\right), \tag {2}
$$
where  $\mathbb{P}_i^*\mathbb{P}_j^*\dots \mathbb{P}_k^* = \mathbb{P}_{ij\dots k}$ . The Lancaster interaction (2) vanishes when the joint distribution can be factorised into jointly independent subvectors for  $d = 3$ , but it fails when  $d\geq 4$  [26]. Specifically,  $\Delta_L^4 P$  does not vanish if  $\mathbb{P}_{1234}$  factorises into  $\mathbb{P}_{12}\mathbb{P}_{34}$ ,  $\mathbb{P}_{13}\mathbb{P}_{24}$ , or  $\mathbb{P}_{14}\mathbb{P}_{23}$ . Despite this, the Lancaster interaction is not entirely uninformative for  $d\geq 4$ , and in Section 4.1 we examine the necessary conditions for it to vanish.
The desired vanishing property can be achieved by using the Streitberg interaction [26], a more general interaction measure defined using partitions. Let  $D$  be the set of random variables  $\{X^1, X^2, \ldots, X^d\}$ , and let  $\Pi(D)$  denote the set of all partitions of  $D$ , where a partition  $\pi$  is a collection of nonempty, pairwise disjoint subsets (blocks)  $b_j \subseteq D$  that cover  $D$ . Then, the Streitberg interaction measure is defined as
$$
\Delta_ {S} ^ {d} \mathbb {P} = \sum_ {\pi \in \Pi (D)} \left(| \pi | - 1\right)! (- 1) ^ {| \pi | - 1} \mathbb {P} _ {\pi}. \tag {3}
$$
Here,  $|\pi|$  denotes the cardinality of the partition  $\pi$ , and  $\mathbb{P}_{\pi} = \prod_{j=1}^{r} \mathbb{P}_{b_j}$  is the corresponding factorisation with respect to  $\pi = b_1 |b_2| \ldots |b_r$ . It has been proven that  $\Delta_S^d \mathbb{P} = 0$  if the joint distribution can be factorised in any way, although the converse is not true in general [34].
3 Partition Lattices
The expressions of the interaction measures outlined in the previous section can be systematically generated from partition lattices. Interestingly, this formulation also allows us to establish further theoretical links with simplicial complexes.
Notation. A partially ordered set defined on a set  $S$  with a binary relation  $\leq$  is a lattice  $\mathcal{L}$  if for any  $\sigma, \pi \in \mathcal{L}$  there exists a greatest lower bound (meet)  $\sigma \wedge \pi$  and least upper bound (join)  $\sigma \vee \pi$  [36]. We denote the maximum and minimum element of a lattice as  $\hat{1}$  and  $\hat{0}$ . Let  $\Delta(\cdot)$  denote a real-valued function defined on  $S$ , then the sum function  $f(\cdot)$  is analogous to the integration of  $\Delta(\cdot)$  over the interval  $[\hat{0}, \pi]$ , where  $\pi \in \mathcal{L}$ . Then  $\Delta(\cdot)$  can be expressed as the inverse operation of  $f(\cdot)$ :
$$
f (\pi) = \sum \zeta (\sigma , \pi) \Delta (\sigma), \quad \Delta (\pi) = \sum_ {\sigma \leq \pi} \mu (\sigma , \pi) f (\sigma), \tag {4}
$$
where the partial order is encoded by the Zeta matrix, with  $\zeta(\sigma, \pi) = 1$  if  $\sigma \leq \pi$  and 0 otherwise. Its inverse is the Möbius matrix with elements  $\mu(\sigma, \pi)$  (for details see Section D), which can be obtained explicitly [37].
Lattices and Interaction Measures. The construction of the interaction measures in Section 2 is closely related to the partition lattice [26, 37]. The partition lattice is defined on the set  $\Pi(D)$  with ordering given by the notion of partition refinement. A partition  $\sigma$  is said to refine another partition  $\pi$ , denoted as  $\sigma \preceq \pi$ , if every block of  $\sigma$  is fully contained within a block of  $\pi$ . The lattice structure thus allows us to define the least upper bound  $\sigma \vee \pi$  and the greatest lower bound  $\sigma \wedge \pi$  between any two partitions  $\sigma$  and  $\pi$ . Clearly, the maximum element of the partition lattice,  $\hat{1}$ , corresponds to the joint distribution, whereas the minimum element,  $\hat{0}$ , corresponds to the complete factorisation (Fig. 1).
Within this formalism, the interaction measures are obtained when the sum function  $f(.)$  in (4) is the probability distribution function. It then follows that the interaction measures are obtained
from the inverse operation. Indeed, the Streitberg interaction (3) is given by the Möbius inversion defined over the complete partition lattice. In contrast, the Lancaster interaction is obtained when the inversion is defined over the subset of partitions that have at most one non-singleton block, which we denote as the Lancaster sublattice. In other words, the sum function associated with the Lancaster interaction considers fewer interactions than the Streitberg interaction and its Möbius inversion has correspondingly fewer terms, as seen in (2). Note that for  $d = 3$ , the full (Streitberg) lattice and the Lancaster sublattice are the same, hence the interaction measures coincide. For  $d \geq 4$ , however, the Streitberg and Lancaster lattices, and consequently the corresponding interaction measures, are different, as shown by the extra partitions with two non-singleton blocks (shaded region) in Figure 1. Note also that joint independence considers a trivial sublattice with only two elements:  $\hat{0}$  and  $\hat{1}$  (for any  $d$ ). The Möbius inversion on this sublattice leads to (1), which only vanishes for complete factorisations.
Links to Simplicial Complexes. A popular representation of high-order systems in the literature is through simplicial complexes [38]. Importantly, the simplicial complex construction can also be understood in terms of partition lattices. In particular, the elements in a  $(d - 1)$ -simplex have inclusion ordering and thus form a subset lattice, e.g.,  $\{X^1, X^2\}$  is a subset of  $\{X^1, X^2, X^3\}$ . The subset lattice has been utilised in the understanding of information geometry [39] and solving tensor balancing [40]. It can be shown that the deatomised sublattice (with removal of singletons) is isomorphic to the Lancaster lattice [41]. Furthermore, the boundary matrices of a  $(d - 1)$ -simplex appear as block matrices both in the Zeta matrix and the (inverse) Möbius matrix of the  $d$ -order partition lattice. These are explored further in Section E in the SI.
Note that the non-singleton partitions, i.e., those that do not belong to the Lancaster sublattice, are not in the set of simplicial complexes. Hence these blocks and their respective refinements cannot be expressed in terms of boundary matrices. Therefore the full partition lattice and its resulting Streitberg interaction provides more information compared with measures originating from sublattices, and in particular the Lancaster lattice and its related simplicial complex construction [23, 24]. Whilst simplicial complexes, and similarly the Lancaster interaction, can be recursively constructed from lower order elements, this is not possible for the Streitberg interaction due to the partitions without singletons, which has implications on computational efficiency (Section 5).
4 Kernel Interaction Tests
The interaction measures in Section 2 can be utilised as statistics in non-parametric tests when embedded into reproducing kernel Hilbert spaces (RKHS). Given a symmetric, positive definite function  $k^i: \mathcal{X}^i \times \mathcal{X}^i \to \mathbb{R}$ , there is an associative RKHS  $\mathcal{H}^i$  with the reproducing kernel property. For  $X^i \in \mathcal{X}^i$ , we denote  $\phi^i(\cdot)$  as the canonical map of  $k^i(X^i, \cdot)$ . The kernel mean embedding of  $\mathbb{P}_{X^i}, \mu_{\mathbb{P}_{X^i}}$ , satisfies  $\mathbb{E}_{X^i} f(X^i) = \langle f, \mu_{\mathbb{P}_{X^i}} \rangle_{\mathrm{HS}}$ , where HS stands for Hilbert-Schmidt. If the kernel is characteristic, the mean embedding is injective and the norm of the signed measure is zero if and only if the measure is zero itself [28, 34, 42]. These properties enable us to create meaningful non-parametric tests by computing the kernel mean embedding of desired interaction measures.
In this section, we extend interaction measures for random variables to the  $d$ -variate case, and formulate a family of interaction tests including the Streitberg interaction, the Lancaster interaction, joint independence and a generalised interaction.
For proofs of the propositions, please see Appendix A.
4.1 Lancaster Interaction
Let us first consider the Lancaster interaction. Although it does not necessarily vanish when  $d \geq 4$  for all types of factorisations due to the lack of certain partitions, here we find the necessary conditions for the Lancaster interaction to vanish.
Proposition 1. If the joint distribution  $\mathbb{P}_{1\dots d}$  can be factorised into  $\mathbb{P}_{\pi_v}$  where  $\pi_v$  are partitions with at least one singleton, then  $\Delta_L^d\mathbb{P} = 0$ .
Remark: Note that  $\mathbb{P}_{\pi_v}$  is a broader set of partitions compared to the set of partitions in the Lancaster lattice, e.g., for  $d = 5$ ,  $\mathbb{P}_{12}\mathbb{P}_{34}\mathbb{P}_5$  satisfies Proposition 1, yet it is not a constituent of the Lancaster lattice, which only consists of partitions with at most one non-singleton.
We now define the Mixed Central Moment Operator as the kernel embedding of the Lancaster interaction, which follows immediately from the expansion of (2) [27]:
Definition 1 (Mixed Central Moment Operator).
$$
\mathcal {M} _ {d} = (- 1) ^ {n - 1} (n - 1) \mathbb {E} \left[ \prod_ {i = 1} ^ {d} \phi^ {i} \right] + \sum_ {\pi_ {\ell} \neq \hat {0}} (- 1) ^ {| \pi_ {\ell} | - 1} \mathbb {E} \left[ \prod_ {s} \phi^ {s} \right] \prod_ {j} \mathbb {E} [ \phi^ {j} ], \tag {5}
$$
where  $\pi_{\ell}$  are the set of partitions with at most one non-singleton (i.e., those that belong to the Lancaster lattice),  $s$  are the singletons, and  $j$  runs over variables in the non-singleton block.
Proposition 2. By rearranging, the Mixed Central Moment Operator can be simplified to:
$$
\mathcal {M} _ {d} = \mathbb {E} \left\{\prod_ {i = 1} ^ {d} \left[ \phi^ {i} - \mathbb {E} \left[ \phi^ {i} \right] \right] \right\}. \tag {6}
$$
Remark: This simplification transparently re-expresses the operator as a central moment, instead of the complex sum of moments in (5). The simplification eliminates all partitions except  $\hat{1}$ .
We now define the entries in matrix  $K_{ab}^{i} = k^{i}(x_{a}^{i}, x_{b}^{i})$  for  $i$ d samples  $x_{a}^{i}$  and  $x_{b}^{i}$  where  $1 \leq a, b \leq n$  and  $\tilde{K}^{i} = HK^{i}H$  where  $H = I - \frac{1}{n} 11^{\top}$  is the centring matrix. The norm of the embedding above can serve as a test statistic. The estimator of the test statistic is derived as the V-statistic:
Proposition 3 (Lancaster interaction estimator.).
$$
\left| \left| \hat {\mathcal {M}} _ {d} \right| \right| _ {\mathcal {H S}} ^ {2} = \frac {1}{n ^ {2}} \sum_ {a = 1} ^ {n} \sum_ {b = 1} ^ {n} \prod_ {i = 1} ^ {d} \tilde {K} _ {a b} ^ {i}, \tag {7}
$$
4.2 Streitberg Interaction
Similarly we define the Mixed Cumulant operator in terms of kernel embeddings from the Streitberg interaction as:
Definition 2 (Mixed Cumulant operator).
$$
\mathcal {K} _ {d} = \sum_ {\pi \in \Pi (D)} (| \pi | - 1)! (- 1) ^ {| \pi | - 1} \prod_ {b \in \pi} \mathbb {E} \left\{\prod_ {i \in b} \phi^ {i} \right\}, \tag {8}
$$
where  $i$  is an element in block  $b$  of partition  $\pi$  in partition lattice  $\Pi (D)$
This operator can be simplified in a similar way:
Proposition 4.
$$
\mathcal {K} _ {d} = \sum_ {\pi_ {s} \in \Pi (D)} \left(| \pi_ {s} | - 1\right)! (- 1) ^ {| \pi_ {s} | - 1} \prod_ {b \in \pi_ {s}} \mathbb {E} \left\{\prod_ {i \in b} \left[ \phi^ {i} - \mathbb {E} [ \phi^ {i} ] \right] \right\}, \tag {9}
$$
where  $\pi_s$  are the partitions with no singletons in partition lattice  $\Pi(D)$ .
Remark: Note that partitions with singletons are eliminated after centring, and the remaining partitions without singletons form an upper semi-lattice. The relationship between the Mixed Cumulant operator and the Mixed Central Moment operator is given by:
Lemma 1. The Mixed Cumulant operator is equal to the sum of Mixed Central Moment Operator products associated with partitions with no singletons denoted as  $\pi_s$  [27]:
$$
\mathcal {K} _ {d} = \sum_ {\pi_ {s} \in \Pi (D)} \left(| \pi_ {s} | - 1\right)! (- 1) ^ {| \pi_ {s} | - 1} \prod_ {b \in \pi_ {s}} \mathcal {M} _ {b}. \tag {10}
$$
Remark: When  $d = 2,3,\mathcal{K}_d$  and  $\mathcal{M}_d$  are identical, which further confirms the moment-cumulant relationship.
Remark: From the partition lattice,  $\{\hat{0},\hat{1}\} \subseteq \pi_{\ell}\subseteq \pi$  , where  $\pi_{\ell}$  is the set of partitions in the Lancaster lattice, and  $\pi$  is the full set in the Streitberg interaction. When  $d = 2$  , we have  $\{\hat{0},\hat{1}\} = \pi_{\ell} = \pi$  , and when  $d = 3$  , we have  $\{\hat{0},\hat{1}\} \subset \pi_{\ell} = \pi$
The estimation using a V-statistic for the norm of the Streitberg interaction is given by:
Proposition 5 (Streitberg interaction estimator).
$$
\left| \left| \hat {\mathcal {K}} _ {d} \right| \right| _ {\mathcal {H S}} ^ {2} = \sum_ {\pi_ {s}, \pi_ {s} ^ {\prime} \in \Pi (D)} \left(\left| \pi_ {s} \right| - 1\right)! \left(\left| \pi_ {s} ^ {\prime} \right| - 1\right)! (- 1) ^ {\left| \pi_ {s} \right| + \left| \pi_ {s} ^ {\prime} \right|} \frac {1}{n ^ {\left| \pi_ {s} \right| + \left| \pi_ {s} ^ {\prime} \right|}} \sum_ {\left\{b _ {i} \right\}} \sum_ {\left\{b _ {i} ^ {\prime} \right\}} \prod_ {i = 1} ^ {d} \tilde {K} _ {b _ {i}, b _ {i} ^ {\prime}} ^ {i}, \tag {11}
$$
where  $1 \leq b_{i}, b_{i}^{\prime} \leq n$  and  $|\{b_{i}\}| = |\pi_{s}|$ ,  $|\{b_{i}^{\prime}\}| = |\pi_{s}^{\prime}|$ . i.e. indices  $b_{i} = b_{j}$  if  $i, j$  are in the same block in  $\pi_{s}$  similarly for  $b_{i}^{\prime}$ . The sets of indices  $\{b_{i}\}, \{b_{i}^{\prime}\}$  are disjoint. An example for  $d = 4$  is given in the SI.
Norms and the Product Lattice. The test statistics of the interaction measures, i.e., the norm of the interaction operators, can also be directly obtained from the Möbius inversion of the Cartesian product of the upper semilattice, thus avoiding the need to compute the square of the operator. The Cartesian product of two lattices with product ordering is also a lattice [43]. The elements in the Cartesian product are analogous to the inner products of kernel embeddings in the estimators. Therefore, the coefficients can be computed directly from the product lattice, as shown in Appendix B.
4.3 Description of the Statistical Tests
The Streitberg interaction test involves rejecting the null hypothesis that the joint distribution can be factorised in some way, which occurs when  $\Delta_S^d\mathbb{P}\neq 0$ . Similarly, for the Lancaster interaction test, we reject the null hypothesis that the joint distribution can be factorised into partitions with at least one singleton by checking  $\Delta_L^d\mathbb{P}\neq 0$ . Both null hypotheses consist of multiple sub-hypotheses, and an example for  $d = 3$  is discussed in Ref. [34, 35]. Importantly, the number of sub-hypotheses to be tested is related to factorisations with only two blocks, i.e., those that form the second level from the top in the partition lattice (see Figure 1). In the case of rejecting some but not all the sub-hypotheses, please see Appendix C for the discussion.
To test each sub-hypothesis  $\mathbb{P}_{1\dots d} = \mathbb{P}_b\mathbb{P}_{b'}$ , for efficiency we fix the observations of the variables in the largest block, and permute the observations of the remaining variables in the other block to induce the independence structure in the null distribution. We then use the Monte-Carlo approximation to compute the p-value [32], and apply a simple correction[35] generalised to  $d$  variables to correct for multiple hypothesis testing. Finally, we reject the composite null hypothesis if all corresponding sub-hypotheses are rejected. To achieve this, we employ the permutation test outlined in Algorithm 1.