Title: Statistical Insights into HSIC in High Dimensions
Venue: NeurIPS 2023
Accepted: yes
Keywords: High dimensionality; Independence test; Kernel method; Nonlinear dependency.
Abstract: Measuring the nonlinear dependence between random vectors and testing for their statistical independence is a fundamental problem in statistics. One of the most popular dependence measures is the Hilbert-Schmidt independence criterion (HSIC), which has attracted increasing attention in recent years. However, most existing works have focused on either fixed or very high-dimensional covariates. In this work, we bridge the gap between these two scenarios and provide statistical insights into the performance of HSIC when the dimensions grow at different rates. We first show that, under the null hypothesis, the rescaled HSIC converges in distribution to a standard normal distribution. Then we provide a general condition for the HSIC based tests to have nontrivial power in high dimensions. By decomposing this condition, we illustrate how the ability of HSIC to measure nonlinear dependence changes with increasing dimensions. Moreover, we demonstrate that, depending on the sample size, the covariate dimensions and the dependence structures within covariates, the HSIC can capture different types of associations between random vectors. We also conduct extensive numerical studies to validate our theoretical results.

Statistical Insights into HSIC in High Dimensions

Abstract

Measuring the nonlinear dependence between random vectors and testing for their statistical independence is a fundamental problem in statistics. One of the most popular dependence measures is the Hilbert-Schmidt independence criterion (HSIC), which has attracted increasing attention in recent years. However, most existing works have focused on either fixed or very high-dimensional covariates. In this work, we bridge the gap between these two scenarios and provide statistical insights into the performance of HSIC when the dimensions grow at different rates. We first show that, under the null hypothesis, the rescaled HSIC converges in distribution to a standard normal distribution. Then we provide a general condition for the HSIC based tests to have nontrivial power in high dimensions. By decomposing this condition, we illustrate how the ability of HSIC to measure nonlinear dependence changes with increasing dimensions. Moreover, we demonstrate that, depending on the sample size, the covariate dimensions and the dependence structures within covariates, the HSIC can capture different types of associations between random vectors. We also conduct extensive numerical studies to validate our theoretical results.

1 Introduction

Let x = ( X 1 , . . . , X p ) T ∈ R p and y = ( Y 1 , . . . , Y q ) T ∈ R q be two random vectors. The problems of measuring nonlinear dependence between x and y and testing for their independence are fundamental and have a wide range of applications in statistics and machine learning. For example, dependence measures can be applied in feature screening (Fan et al., 2020b), model checking (Clarke et al., 2018), graphical models (Maathuis et al., 2018), and causal inference (Imbens & Rubin, 2015).

Many statistical tools have been developed to measure the dependence between random variables. Classical measures such as Pearson correlation (Pearson, 1920), Spearman's ρ (Spearman, 1904) and Kendall's τ (Kendall, 1938) can only capture linear or monotonic dependence and may fail to detect complex associations. For univariate covariates, several rank-based measures have been proposed to overcome this limitation, such as those by Hoeffding (1948), Blum et al. (1961), Bergsma & Dassios (2014) and Chatterjee (2021). For multivariate covariates, various nonparametric tests have been introduced to quantify arbitrary associations, including the distance correlation (Székely et al., 2007), the projection correlation (Zhu et al., 2017), the mutual information (Berrett & Samworth, 2019), the kernel canonical correlation (Bach & Jordan, 2002), the constrained covariance (Gretton et al., 2005), and the Hilbert-Schmidt independence criterion (HSIC, Gretton et al., 2007). Among these measures, the distance correlation and the HSIC are perhaps the most widely used and studied in both statistics and machine learning. For instance, Li et al. (2012) applied the distance correlation to perform feature screening. Fan et al. (2020a) generalized the distance correlation to a conditional dependence measure under a factor model setting. Albert et al. (2022) developed an adaptive test of independence by aggregating HSIC measures. Pfister et al. (2018) defined a d -variable HSIC to characterize the mutual dependence among d random variables. It is worth noting that the distance correlation is equivalent to HSIC with a specific kernel choice (Sejdinovic et al., 2013).

High-dimensional settings pose new challenges and opportunities for dependence measures. Székely &Rizzo (2013) suggested to estimate the distance correlation with the U -statistic theory to avoid bias accumulations when the dimension is high. Ramdas et al. (2015) found that the power of kernel and distance based independence tests decreases polynomially with increasing dimension against some fair alternatives. Zhu et al. (2020) proved that, when both p and q grow much faster than the sample size n , the sample distance correlation and HSIC can only detect componentwise linear dependences. Gao et al. (2021) established a general condition for the distance correlation based test to have power approaching 1. They also showed that the distance correlation can capture certain pure nonlinear relationship when p = q = o ( n 1 / 2 ) under a specific alternative hypothesis.

In this paper, we aim to provide statistical insights into the HSIC in high dimensions, motivated by the increasing popularity of HSIC and the prevalence of high-dimensional data. We focus on the asymptotic distributions of HSIC under both null and alternative hypotheses when the dimensionality grows with the sample size. Under the null hypothesis, we prove that the rescaled HSIC converges in distribution to a standard normal distribution. We also derive a general condition for the HSIC based tests to have power asymptotically approaching one. These results extend those of Gao et al. (2021) on the distance correlation. Moreover, our analysis is more comprehensive than theirs. By decomposing the general condition, we reveal that HSIC can detect different types of dependences, depending on the dimensionality and sample orders. This sheds light on the performance of HSIC in high dimensions, which has been overlooked in the literature. For illustrative purposes, we let x and y be two random vectors with zero mean and identity covariance matrix. When HSIC is used to test for statistical independence between them, we obtain the following conditions for nontrivial power:

If p →∞ and q is fixed as n →∞ , and there only exists the conditional mean of x 's s -th polynomial given y , then p ( s -1) must grow slower than n .

If both p and q →∞ as n → ∞ , and there only exists the covariance between x 's s 1 -th polynomial and y 's s 2 -th polynomial, then p ( s 1 -1) q ( s 2 -1) must grow slower than n .

These two conditions reveal significant statistical insights and have important implications in practice. For example, when the data dimension q is small and p is larger than the sample size n , according to the first condition, to ensure p ( s -1) smaller than n , s can only be 1. In other words, the HSIC can only measure the conditional mean of x given y . When both p and q are larger than n , according to the second condition, to ensure p ( s 1 -1) q ( s 2 -1) smaller than n , both s 1 and s 2 must be 1. That is, the HSIC can only measure the covariance between x and y . In summary, our main contributions are:

We generalize the results of Gao et al. (2021) for distance correlation to HSIC, which is a more flexible and widely used measure of dependence.

We show that HSIC can capture different types of dependences, depending on the dimensionality and sample orders, which has rarely been realized in the literature before.

In contrast to Zhu et al. (2020), who only focused on the case when both p and q grow much faster than n , our results characterize a full picture of the performance of HSIC based test in high dimensions.

The rest of the paper is organized as follows. We give some preliminaries about the HSIC in Section 2. Then we present the asymptotic null distribution as well as the power analysis for high-dimensional HSIC in Section 3. In Section 4, we provide statistical insights into the HSIC through connecting the association type with the sample size, the high dimensionality, and the dependence structures within covariates. We conduct comprehensive numerical studies in Sections 5 and conclude this paper with a brief discussion in Section 6. All technical details are relegated to the Supplement.

2 Preliminaries

In this section, we give a brief review on HSIC measures proposed by Gretton et al. (2007), which is derived from the notion of cross-covariance operator Baker (1973). We assume that X ∈ X ⊆ R p and Y ∈ Y ⊆ R q are random vectors taking values in Euclidean spaces respectively. Let F and G be the reproducing kernel Hilbert spaces (RKHSs) on X and Y , with associated kernels K ( · , · ) and L ( · , · ) , respectively. Then the cross-covariance operator C xy associated to F and G is defined as the operator mapping from G to F , such that for all f ∈ F and g ∈ G , 〈 f, C xy g 〉 F = cov { f ( x ) , g ( y ) } . Instead of using the largest singular value of this operator, Gretton et al. (2007) suggested to adopt the squared Hilbert-Schmidt norm, which admits a closed form expression

where ( x 1 , y 1 ) and ( x 2 , y 2 ) are independent copies of ( x , y ). When the RKHSs, F and G , are well chosen, (e.g., characteristic, Sriperumbudur et al., 2010), the HSIC can be served as an independence criterion in the sense that it is nonnegative and equals zero if and only if x and y are independent. To normalize the HSIC to range from zero to one, we follow Zhu et al. (2020) to define the squared Hilbert-Schmidt correlation as

In addition, similar to Zhu et al. (2020) and Albert et al. (2022), we focus on the kernels that can be written in the form of K ( x 1 , x 2 ) = k ( ‖ x 1 -x 2 ‖ /γ x ) and L ( y 1 , y 2 ) = l ( ‖ y 1 -y 2 ‖ /γ y ) , where k ( · ) and l ( · ) are some real-valued functions defined in [0 , ∞ ) , γ x and γ y are the bandwidth parameters. This kind of kernel is very commonly used in the literature and includes many positive-definite kernel such as the Gaussian kernel, the Laplacian kernel, the rational quadratic kernel, and the Kernel generating Sobolev spaces, etc. For example, when k ( x ) = exp( -x 2 ) , it corresponds to the Gaussian kernel, and when k ( x ) = exp( -x ) , it corresponds to the Laplacian kernel. One may refer to Genton (2001) for more examples and discussions about isotropic kernels.

3 Asymptotic Properties in High Dimensions

At the sample level, with the random sample { ( x i , y i ) , i = 1 , ..., n } , we estimate the HSIC with U -statistics,

where the indexes in the summands, i.e., ( i 1 , i 2 ) , ( i 1 , i 2 , i 3 ) and ( i 1 , i 2 , i 3 , i 4 ) , are all distinctive from each other. Then the squared Hilbert-Schmidt correlation hCorr 2 ( x , y ) can be estimated as

We now discuss how to use the sample squared Hilbert-Schmidt correlation to distinguish between the null hypothesis H 0 : x is independent of y , and the alternative hypothesis H 1 : x is not independent of y . To implement the HSIC based test, it is required to study the asymptotic distributions for hCorr 2 n ( x , y ) under the null hypothesis. When the dimensions are fixed, Gretton et al. (2007) showed that n hCorr 2 n ( x , y ) converges in distribution to a weighted sum of independent chi-squared random variables as long as x is independent of y . We now study its asymptotic null distribution in high

dimensions. Before that, we define the two quantities H x ( x 1 , x 2 ) and H y ( y 1 , y 2 ) as

We further define another two quantities G x ( x 1 , x 2 ) and G y ( y 1 , y 2 ) as

Then we summarize the asymptotic null distribution for hCorr 2 n ( x , y ) in the following Theorem.

Theorem 1. Assume the kernels are symmetric with finite fourth moment, i.e., K ( x 1 , x 2 ) = K ( x 2 , x 1 ) , L ( y 1 , y 2 ) = L ( y 2 , y 1 ) , E { K 4 ( x 1 , x 2 ) } < ∞ and E { L 4 ( y 1 , y 2 ) } < ∞ . Further assume that p + q →∞ ,

as n →∞ . Then under the null hypothesis, we have 2 -1 / 2 n hCorr 2 n ( x , y ) d → N (0 , 1) .

Different from that when the dimensions are fixed, Theorem 1 reveals that, under certain conditions, n hCorr 2 n ( x , y ) is asymptotically normal in high dimensions. The asymptotic normality makes the limiting null distribution tractable in practice. It greatly expedites the implementation of HSIC based tests because no additional permutations are required to decide critical values. We remark here that the assumptions imposed in Theorem 1 to ensure the asymptotic normality are generally very mild. For the restrictions on the kernels, many commonly used kernels, including Gaussian and Laplacian kernels, are symmetric with finite fourth moment. For the assumption in (4), it is used to restrict the weak dependence within covariates in x and y . It holds true when x and y follow the multivariate normal distributions with bounded eigenvalues. It also holds true when x and y are m -dependent random sequences (Definition 9.1 of DasGupta, 2008). One may refer to Gao et al. (2021) for more discussions about this assumption.

Next, we explore the power performance of the HSIC based test in high dimensions. As discussed in Section 2, we restrict our attentions to the isotropic kernels. Specifically, in the remaining of this paper, we consider K ( x 1 , x 2 ) and L ( y 1 , y 2 ) to be in the form of k ( ‖ x 1 -x 2 ‖ /γ x ) and l ( ‖ y 1 -y 2 ‖ /γ y ) , respectively. Because of the kernel form, we assume without loss of generality that E x = 0 and E y = 0 . We denote by z ∗ def = z /γ z , where z ∈ R d can be either x or y . We say that f ( n ) glyph[equivasymptotic] g ( n ) if f ( n ) = O { g ( n ) } and g ( n ) = O { f ( n ) } . Before summarizing the asymptotic power performance, we make the following assumptions:

(A1) There exists some κ z > 0 such that E {‖ z ∗ ‖ 2 -E ( ‖ z ∗ ‖ 2 ) } 2 k glyph[equivasymptotic] E ( z ∗ 1 T z ∗ 2 ) 2 k glyph[equivasymptotic] d -kκ z for all k ∈ N + .

(A2) Let k 0 ( x ) = k ( x 1 / 2 ) and l 0 ( y ) = l ( y 1 / 2 ) . The first and second derivatives of k 0 ( · ) and l 0 ( · ) are uniformly bounded away from zero to infinity around E ‖ x ∗ 1 -x ∗ 2 ‖ 2 and E ‖ y ∗ 1 -y ∗ 2 ‖ 2 , respectively.

We remark here that Assumption (A1) is used to restrict the dependence structures within the coordinates of z . Because z ∗ = z /γ z and E z = 0 , we have E z ∗ = 0 as well. Hence, both ‖ z ∗ ‖ 2 -E ( ‖ z ∗ ‖ 2 ) and z ∗ 1 T z ∗ 2 are sums of d random variables with mean zero. Their standardized versions may converge in distribution to standard normal distributions under mild conditions (e.g., mixing conditions) by applying the central limit theorem (CLT). For instance, Voln` y (1989) established a CLT for non-stationary mixing processes. Alternatively, there exists some κ z > 0 such that d κ z / 2 {‖ z ∗ ‖ 2 -E ( ‖ z ∗ ‖ 2 ) } and d κ z / 2 z ∗ 1 T z ∗ 2 both converge in distribution to normal random variables. Then Condition (A1) is satisfied. Assumption (A2) imposes some conditions on the kernels, which covers the Gaussian and Laplacian kernels. Moreover, it also makes several requirements on the bandwidth parameters as well. For example, suppose the Gaussian kernel is used, i.e., k 0 ( z ) = exp( -z ) , when γ z is small enough such that E ‖ z ∗ 1 -z ∗ 2 ‖ 2 →∞ , then we have k ′ 0 ( E ‖ z ∗ 1 -z ∗ 2 ‖ 2 ) → 0 , which violates Assumption (A2). Theoretically, the bandwidth parameter γ z can be chosen from a wide range of values, as long as it satisfies condition (A2). In practice, we use the median of

‖ z 1 -z 2 ‖ as a default value for γ z . When it has the same magnitude with E ‖ z 1 -z 2 ‖ , Assumption (A2) can be easily satisfied for many commonly used kernels. Moreover, as demonstrated in the simulated examples in the Supplementary Material, our method exhibits robustness across various selections of γ z .

With these two assumptions, we are ready to provide the asymptotic power performance of the HSIC based test in high dimensions.

Theorem 2. Assume (A1) and (A2) hold true. Then under the alternative hypothesis, if n 1 / 2 hCorr 2 ( x , y ) →∞ as n →∞ , we have n hCorr 2 n ( x , y ) →∞ in probability.

Theorem 2 reveals that, as long as the dependence measured by hCorr 2 ( x , y ) is not too small, n hCorr 2 n ( x , y ) converges to infinity in probability. This, together with Theorem 1, guarantee that the HSIC based test can have nontrivial power in high dimensions. However, as the dimension increases, the signal measured by hCorr 2 ( x , y ) may decay to zero, which makes the test lose power in high dimensions. To gain more insights about how the power is influenced by the dimensions, it is desired to connect hCorr 2 ( x , y ) with the dimensionality, as well as the association types between x and y .

4 Statistical Insights in High Dimensions

We have discussed the asymptotic properties of HSIC based tests in high dimensions, and demonstrated that it can detect the alternative hypothesis as long as the signal strength measured by hCorr 2 ( x , y ) does not decay to zero too fast. However, it is still unclear that how hCorr 2 ( x , y ) is influenced by the dimensions. To gain more insights into the performance in high dimensions, in this section, we study what kind of association types the HSIC can detect under different relationships between the sample size and the covariate dimensions. To this end, we expand hCorr 2 ( x , y ) defined in (1) to gain some insights. Because HSIC( x , y ) characterizes the dependence between x and y , while HSIC( x , x ) and HSIC( y , y ) are the normalizing factors, we expand HSIC( x , y ) and calculate the orders for HSIC( x , x ) and HSIC( y , y ) .

We first calculate the orders for HSIC( x , x ) and HSIC( y , y ) at the population level as the dimensions diverge to infinity. The results are summarized in the following Proposition.

Proposition 1. Under Assumptions (A1) and (A2), HSIC( z , z ) glyph[equivasymptotic] d -κ z if d →∞ .

According to Proposition 1, the condition n 1 / 2 hCorr 2 ( x , y ) → ∞ , which guarantees the HSIC based test would have power approaching 1 in high dimensions, will boil down to n 1 / 2 p κ x / 2 q κ y / 2 HSIC( x , y ) →∞ .

Next, we expand HSIC( x , y ) at the population level. Before that, we introduce some additional notations. Similar to Shao & Zhang (2014), we define MD 2 ( x | y ) as

where ( x 1 , y 1 ) and ( x 2 , y 2 ) are independent copies of ( x , y ) . We show in Lemma 1 that, MD 2 ( x | y ) = 0 if and only if E ( x | y ) = 0 . That is, MD 2 ( x | y ) measures the degree of conditional mean of x given y , which quantifies the difference between E ( x | y ) and E x .

Lemma 1. Assume the kernel L ( y 1 , y 2 ) = l ( ‖ y 1 -y 2 ‖ /γ y ) is a characteristic, bounded continuous, and real-valued positive definite function, then MD 2 ( x | y ) = 0 if and only if E ( x | y ) = 0 almost surely.

We remark here that characteristic condition in Lemma 1 is used to guarantee that the HSIC is nonnegative and equals zero if and only if x and y are independent. One may refer to (Sriperumbudur et al., 2010) for more details about this condition. Furthermore, we let k ( i ) 0 be the i -th derivative of ∗ ∗ 2 ( j ) ∗ ∗ 2

k 0 ( · ) evaluated at E ‖ x 1 -x 2 ‖ , and l 0 is the j -th derivative of l 0 ( · ) evaluated at E ‖ y 1 -y 2 ‖ . Let x ⊗ n be the n -th kronecker power of x , which is defined as x ⊗ n = x ⊗ x ⊗ ( n -1) , x ⊗ 1 = x , and ⊗ denotes the Kronecker product. In addition, we use ‖ · ‖ F to represent the Frobenius norm of a matrix. Then we expand HSIC( x , y ) in the following Theorem.

Theorem 3. Assume (A1) and (A2) hold true. Then under the alternative hypothesis,

when p →∞ and q remains fixed as n → ∞ , if E ( x ⊗ t | y ) = E ( x ⊗ t ) hold true for all t < s for some s ∈ N + , then HSIC( x , y ) = O ( p -sκ x / 2 ) , and

glyph[negationslash]

when p → ∞ and q → ∞ as n → ∞ , if cov ( x ⊗ t 1 , y ⊗ t 2 ) = 0 only when t 1 ≥ s 1 and t 2 ≥ s 2 for some s 1 , s 2 ∈ N + , then HSIC( x , y ) = O ( p -s 1 κ x / 2 q -s 2 κ y / 2 ) , and HSIC( x , y ) = ∑ 2 a 1 + c 1 = s 1 ∑ 2 a 2 + c 2 = s 2 k ( s 1 ) 0 ( -2) c 1 a 1 ! a 1 ! c 1 ! l ( s 2 ) 0 ( -2) c 2 a 2 ! a 2 ! c 2 ! ∥ ∥ ∥ ∥ cov {‖ x ∗ ‖ 2 a 1 x ∗⊗ c 1 , ‖ y ∗ ‖ 2 a 2 y ∗⊗ c 2 T } ∥ ∥ ∥ ∥ 2 F + o ( p -s 1 κ x / 2 q -s 2 κ y / 2 ) .

Theorem 3 enables us to gain some insights into the HSIC in high dimensions. First of all, because when 2 a + c = s , x ∗⊗ c ‖ x ∗ ‖ 2 a is the s -th polynomial for x , and MD 2 ( x ∗⊗ c ‖ x ∗ ‖ 2 a | y ) measures the departure from the conditional mean independence of x given y . Then the first part of Theorem 3 implies that when p → ∞ and q remains fixed as n → ∞ , if E ( x ⊗ t | y ) = E ( x ⊗ t ) hold true for all t < s for some s ∈ N + , the HSIC quantifies the departure from E ( x ⊗ s | y ) = E ( x ⊗ s ) . In other words, when only one dimension of the covariates diverges to infinity, the HSIC first measures the conditional mean of x given y . If there is no such mean dependence, it turns to measure the conditional mean of x ⊗ 2 given y , etc. Similarly, when p →∞ and q →∞ as n →∞ , the second part of Theorem 3 reveals that, the HSIC may first search for the covariance between x given y . If such covariance equals zero, it then turns to search for the covariance between x ⊗ 2 and y , or the covariance between x and y ⊗ 2 , etc. Therefore, Theorem 3 characterizes the changing process of the ability to measure nonlinear dependence in high dimensions for HSIC. As a comparison, Zhu et al. (2020) only showed that, when both dimensions diverge much faster than the sample size, the HSIC can only capture the marginal linear dependence. From this point of view, our result is a generalization of theirs.

When the HSIC is used to test for statistical independence, Theorem 2 and Proposition 1 ensure that, when n 1 / 2 p κ x / 2 q κ y / 2 HSIC( x , y ) → ∞ , the HSIC based test would have asymptotic power 1. To make this condition hold true, the first part of Theorem 3 implies that when p → ∞ and q remains fixed as n → ∞ , if there only exists the conditional mean of x ⊗ s given y , it is required that p ( s -1) κ x = o ( n ) because HSIC( x , y ) is of order O ( p -sκ x / 2 ) in this case. In other words, when n = O { p ( s -1) κ x } , the HSIC based test can have nontrivial power only when there exists the i -th polynomial mean dependence of x given y for some i < s . Similarly, when both p and q diverges as n →∞ , the second part of Theorem 3, when there only exists the covariance between x ⊗ s 1 and y ⊗ s 2 , the test based on HSIC can have nontrivial power only when p ( s 1 -1) κ x q ( s 2 -1) κ y = o ( n ) . Because κ x represents the degree of dependence within covariates, Theorem 3 successfully connects the association type between x and y , with the sample size, the high dimensionality, and the dependence structures within covariates. We remark here that Gao et al. (2021) only justified that the test based on distance correlation can have power approaching 1 when p = q = o ( n 1 / 2 ) if there exist a pure nonlinear dependence between x and y . Therefore, their result is in spirit a special case of ours.

5 Numerical Studies

5.1 Simulations

In this subsection, we conduct some simulation studies to validate our theoretical conclusions on the HSIC based test in high dimensions. In particular, in Example 1, we show that, as long as the dimensions are not very small, the normal approximation can approximate the null distribution quite well, although it requires the dimension to diverge to infinity in theory. In Example 2, we fix the dimension of y and increase the dimension of x , and investigate the finite sample power performance of the HSIC based tests under various mean dependence types. In Example 3, we increase the dimensions of x and y simultaneously, and inspect how the empirical power is influenced by the covariate dimensions under different relationships.

Throughout the simulations, we choose two kinds of commonly used kernels to implement the HSIC based tests, i.e., Gaussian and Laplacian. For the Gaussian kernel, we set K ( x 1 , x 2 ) =

exp {-‖ x 1 -x 2 ‖ 2 / (2 γ 2 x ) } and L ( y 1 , y 2 ) = exp {-‖ y 1 -y 2 ‖ 2 / (2 γ 2 y ) } . While for the Laplacian kernel, we set L ( x 1 , x 2 ) = exp( -‖ x 1 -x 2 ‖ /γ x ) and L ( y 1 , y 2 ) = exp( -‖ y 1 -y 2 ‖ /γ y ) . For both choices of kernels, we set the bandwidth parameters to be γ x = c 0 γ m x and γ y = c 0 γ m y and vary c 0 from 0 . 5 to 2 . Here, γ m z represents the median of {‖ z i -z j ‖} 1 ≤ i<j ≤ n , and z is either x or y . Due to space constraints, we present results specifically for the case when γ x = γ m x and γ y = γ m y . The remaining results are summarized in the Supplementary Material. Notably, our findings remain consistent across different values of γ x and γ y , showing the stability and reliability of our approach.

Example 1 . In this example, we examine the normal approximation accuracy under the null hypothesis. Towards this goal, we simply generate x = ( X 1 , . . . , X p ) T ∈ R p from a multivariate normal distribution with mean zero and covariance matrix Σ x = I p × p . Then we independently generate y = ( Y 1 , . . . , Y q ) T ∈ R q from another a multivariate normal distribution whose coordinates follow iid standard normal distributions. We fix the sample size to be 100 and consider two scenarios for the covariate dimensions. In the first scenario, we fix q = 1 and vary p from { 5 , 25 , 100 } . In the second scenario, we set p = q = d and vary d from { 2 , 5 , 10 } . For each setting, we repeat the experiments 5000 times and compare the empirical null distributions of the test statistics, { 2 -1 / 2 n hCorr 2 n ( x , y ) } , with the standard normal distribution. Specifically, we display the kernel density curves of the test statistics for the two scenarios in Figures 1 and 2, respectively.

-4

-2

0

2

4

6

8

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

-6

-4

-2

0

2

4

6

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

-6

-4

-2

0

2

4

6

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

(A):

p

= 5

(B):

p

= 25

(C):

p

= 100

According to Figures 1 and 2, we can see that, when the dimensions are small, the normal distribution cannot approximate the null distribution well. For example, when p = q = 2 , the empirical null distributions for the tests with both Gaussian and Laplacian kernels are quite distinct from the reference curve in Figure 2 (A). However, as the dimensions increase, the accuracy of the normal approximation for the null distribution becomes quite precise, regardless of the choice of the kernels. For example, in both Figures 1 and 2, the empirical null distributions are almost the same as the reference curve as long as pq are not smaller than 25.

Example 2 . In this example, we investigate how the empirical power is influenced by the dimension when only one of the covariate dimensions increases. For simplicity, we fix the sample size to be n = 100 , as that in Example 1. As for the covariate dimensions, we fix q = 1 and vary p from { 30 , 50 , 100 , 200 , 500 , 1000 } . We consider three models to inspect the power performances. For Models (I) and (II), we generate x from a multivariate normal distribution with mean zero and covariance matrix Σ x = (0 . 5 | i -j | ) p × p . Then we generate an independent error term ε that follows standard normal distribution. The univariate response Y is generated through

Model (I) : Y = X 1 + . . . + X p + ε ; Model (II) : Y = X 2 1 + . . . + X 2 p + ε.

For Model (III), we first generate Y ∼ N (0 , 1) . Conditional on Y , we generate x through

-4

-3

-2

-1

0

1

2

3

4

5

6

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

-4

-3

-2

-1

0

1

2

3

4

5

6

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

-4

-3

-2

-1

0

1

2

3

4

5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

(A):

d

= 2

(B):

d

= 5

(C):

d

= 10

where ρ k,Y = (2 δ k -1)Φ( Y ) , δ k s are iid Bernoulli(0.5) random variables, and Φ( · ) is the cumulative distribution function of a standard normal distribution. In addition, ( X 2 k -1 , X 2 k ) T s are mutually independent conditional on Y . We remark here that in Model (III), X k is marginally independent of Y for all k = 1 , . . . , p . It can be verified that there exist linear, quadratic and quartic conditional means of x given Y in Models (I), (II) and (III), respectively.

As we have demonstrated in Example 1, when q = 1 , the standard normal distribution can approximate the null distribution pretty well as long as p is no less than 25 , we use the standard normal distribution to obtain p -values. We repeat each experiment 500 times and report the empirical powers in Table 1. As a comparison, we also include the results for the distance correlation in Table 1. We can observe that all the empirical powers for Model (I) can still approach one, implying that the linear mean dependence can be easily detected, even when the dimension is much higher than the sample size. For example, when p = 1000 , the power for Model (I) when the Gaussian kernel is used is still as high as 0.954. However, when there only exists some higher order of mean dependence, the empirical powers drop significantly. In addition, the performances deteriorate as the order of mean dependence increase. For example, when p = 100 and we choose the Gaussian kernel to implement the test, the empirical power for Model (III) is only 0.060, which is smaller than 0.484, the empirical power for Model (II).

Example 3 . In this example, we study how the dimensionality can effect the empirical power when both of the covariate dimensions increase simultaneously. We also fix the sample size to be n = 100 . We set p = q = d and vary d from { 6 , 10 , 20 , 50 , 100 , 200 } . Similar to Example 2, we consider three models in this example. For Models (IV) and (V), we generate x the same as that in Models (I) and (II), i.e., x ∼ N { 0 , (0 . 5 | i -j | ) p × p } . The independent error terms ε 1 , . . . , ε d are generated from d independent standard normal distributions. Then we generate y = ( Y 1 , . . . , Y d ) T through

For Model (VI), we first generate all coordinates of y independently from standard normal distributions. Then given y , we generate x through

where ρ k, y = (2 δ k -1)Φ( Y 2 k ) and δ k s are iid Bernoulli(0.5) random variables. Meanwhile, we generate ( X 2 k -1 , X 2 k ) T s independently conditional on y . Similar to Models (I), (II) and (III), there exist covariances between y and linear, quadratic and quartic polynomials of x in Models (IV), (V) and (VI), respectively.

Following that in Example 2, we use the standard normal distribution to obtain p -values and repeat each experiment 500 times to report the empirical powers. The corresponding results are summarized in Table 2, from which we observe a similar phenomenon. That is, the empirical powers decay as the dimensions increase, and the decay rates increase when there only exist higher orders of covariances between x and y . In addition, according to both Tables 1 and 2, the conclusions for the HSIC with Gaussian and Laplacian kernels also apply to the distance correlation. This is in line with our anticipation because the distance correlation is equivalent to the HSIC with some particular kernel.

5.2 Real Data Application

The prices of gasoline and many other fuels may have important impacts on raw materials and related companies. For example, the petroleum is widely used as upstream raw materials by many chemical industries. We are interested in whether there is any dependency between stock prices of the energy sector and the raw material sector in the U.S. stock market. To see this, we extract the monthly mean stock prices of energy companies as well as raw material companies starting from January 2021 to December 2022 from https://finance.yahoo.com/. According to the Global Industry Classification Standard, we get 345 and 294 companies from raw material and energy sectors, respectively. We remove those stocks whose data are not complete in this period, yielding 224 companies from the raw material sector and 214 companies from the energy sector, respectively. At each month t , 1 ≤ t ≤ 24 , we denote the monthly mean stock prices of these companies from the two sectors by x t = ( X t 1 , . . . , X tp ) T and y t = ( Y t 1 , . . . , Y tq ) T , where p = 224 and q = 214 . Let S x ti = log( X ti /X t -1 ,i ) and S y tj = log( Y tj /Y t -1 ,j ) be the stock returns at month t for the i -th company in the raw material sector, and j -th company in the energy sector, respectively. We test whether the stock returns in these two sectors are independent using { ( s x t , s y t ) } 24 t =2 , where s x t = ( S x t 1 , . . . , S x tp ) T and s y t = ( S y t 1 , . . . , S y tq ) T . We implement the test using HSIC with Gaussian and Laplacian kernels. The resulting p -values are 2 . 031 × 10 -10 and 2 . 749 × 10 -9 , respectively. Then we can conclude that there exists dependences between the monthly mean stock prices of the energy sector and the raw material sector.

We also compare the RV coefficient with our proposed method in this dataset. The RV coefficient yields a p -value of 2 . 02 × 10 -4 , which indicates the presence of linear dependences in this dataset. We remark that, in this data set, both covariates dimensions p and q are much larger than sample size n . According to the second part of Theorem 3 and the discussions at the end of Section 4, the HSIC can only have nontrivial power if p ( s 1 -1) κ x q ( s 2 -1) κ y = o ( n ) . In the context of our dataset, this

condition is satisfied exclusively when s 1 = s 2 = 1 , signifying a covariance relationship between x and y . This together with the fact that the test based on HSIC rejected the null hypothesis, we conclude that there exists a linear dependence relationship between x and y . This is consistent with the RV coefficient result. Moreover, we identify that there is a strong linear relationship between stock returns for two pairs of companies: Denison Mines Corp. and Energy Fuels Inc., and Uranium Energy Corp. and Energy Fuels Inc. The R 2 s for the corresponding linear models are 0.8779 and 0.8718, respectively. This means that the models fit the data very well and that the stock returns move together in a predictable way.

6 Conclusion and Discussion

This paper aims to investigate the performance of HSIC in high dimensions from a theoretical perspective. We first prove that the asymptotic null distribution of a rescaled HSIC is a standard normal in the high dimensional setting. Then we derive a general condition for the HSIC based tests to have power asymptotically approaching one. To gain more insights, we decompose this condition and show that it depends on the sample size, the covariate dimensions, the dependence structures within covariates, and the association types between x and y . We also show that the ability of HSIC to detect nonlinear dependence deteriorates as the dimensions increase. Most existing works focused on either fixed dimensions or dimensions that diverge much faster than the sample size. We fill this gap in the literature and provide a comprehensive picture of the performance of HSIC based tests.

Our theory also reveals how the ability of HSIC to capture nonlinear dependence diminishes in high dimensions. To improve the power performance, Zhu et al. (2020) proposed to aggregate marginal sample HSIC as the test statistic instead of using HSIC over the whole features. However, this approach only measures marginal dependences, and it is well known that marginal independence does not imply joint independence. Moreover, because it sums over all pairs of marginal HSIC, it may suffer from a significant power loss when the alternative is sparse. Developing an independence test that has nontrivial power against all kinds of dependences remains an open question. Therefore, it would be interesting to explore new methods that can enhance power performances in high dimensions.

Acknowledgments and Disclosure of Funding

The authors thank five anonymous reviewers for their comments and suggestions that significantly enhanced the quality of the paper. All authors contribute equally to this paper and their names are listed in an alphabetical order. This work is supported by National Natural Science Foundation of China (12271331 and 72192832), and Program for Innovative Research Team of Shanghai University of Finance and Economics.

References

Albert, M., Laurent, B., Marrel, A., and Meynaoui, A. Adaptive test of independence based on hsic measures. The Annals of Statistics , 50(2):858-879, 2022.

Bach, F. R. and Jordan, M. I. Kernel independent component analysis. Journal of Machine Learning Research , 3(Jul):1-48, 2002.

Baker, C. R. Joint measures and cross-covariance operators. Transactions of the American Mathematical Society , 186:273-289, 1973.

Bergsma, W. and Dassios, A. A consistent test of independence based on a sign covariance related to kendall's tau. Bernoulli , 20(2):1006-1028, 2014.

Berrett, T. B. and Samworth, R. J. Nonparametric independence testing via mutual information. Biometrika , 106(3):547-566, 2019.

Blum, J., Kiefer, J., and Rosenblatt, M. Distribution free tests of independence based on the sample distribution function. The Annals of Mathematical Statistics , 32(2):485-498, 1961.

Chatterjee, S. A new coefficient of correlation. Journal of the American Statistical Association , 116 (536):2009-2022, 2021.

Clarke, E. M., Henzinger, T. A., Veith, H., and Bloem, R. Handbook of Model Checking . Springer, 2018.

DasGupta, A. Asymptotic Theory of Statistics and Probability , volume 180. Springer, 2008.

Fan, J., Feng, Y ., and Xia, L. A projection-based conditional dependence measure with applications to high-dimensional undirected graphical models. Journal of Econometrics , 218(1):119-139, 2020a.

Fan, J., Li, R., Zhang, C.-H., and Zou, H. Statistical Foundations of Data Science . Chapman and Hall/CRC, 2020b.

Gao, L., Fan, Y., Lv, J., and Shao, Q.-M. Asymptotic distributions of high-dimensional distance correlation inference. The Annals of Statistics , 49(4):1999-2020, 2021.

Genton, M. G. Classes of kernels for machine learning: a statistics perspective. Journal of Machine Learning Research , 2(Dec):299-312, 2001.

Gretton, A., Herbrich, R., Smola, A., Bousquet, O., and Schölkopf, B. Kernel methods for measuring independence. Journal of Machine Learning Research , 6:2075-2129, 2005.

Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., and Smola, A. A kernel statistical test of independence. Advances in Neural Information Processing Systems , 20, 2007.

Hoeffding, W. A non-parametric test of independence. The Annals of Mathematical Statistics , pp. 546-557, 1948.

Imbens, G. W. and Rubin, D. B. Causal Inference in Statistics, Social, and Biomedical Sciences . Cambridge University Press, 2015.

Kendall, M. G. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938.

Li, R., Zhong, W., and Zhu, L. Feature screening via distance correlation learning. Journal of the American Statistical Association , 107(499):1129-1139, 2012.

Maathuis, M., Drton, M., Lauritzen, S., and Wainwright, M. Handbook of Graphical Models . CRC Press, 2018.

Pearson, K. Notes on the history of correlation. Biometrika , 13(1):25-45, 1920.

Pfister, N., Bühlmann, P., Schölkopf, B., and Peters, J. Kernel-based tests for joint independence. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(1):5-31, 2018.

Ramdas, A., Reddi, S. J., Póczos, B., Singh, A., and Wasserman, L. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Twenty-Ninth AAAI Conference on Artificial Intelligence , 2015.

Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics , pp. 2263-2291, 2013.

Shao, X. and Zhang, J. Martingale difference correlation and its use in high-dimensional variable screening. Journal of the American Statistical Association , 109(507):1302-1318, 2014.

Spearman, C. The proof and measurement of association between two things. American Journal of Psychology , 15:72-101, 1904.

Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Schölkopf, B., and Lanckriet, G. R. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research , 11: 1517-1561, 2010.

Székely, G. J. and Rizzo, M. L. The distance correlation t-test of independence in high dimension. Journal of Multivariate Analysis , 117:193-213, 2013.

Székely, G. J., Rizzo, M. L., and Bakirov, N. K. Measuring and testing dependence by correlation of distances. The Annals of Statistics , 35(6):2769-2794, 2007.

Voln` y, D. A central limit theorem for non stationary mixing processes. Commentationes Mathematicae Universitatis Carolinae , 30(2):405-407, 1989.

Zhu, C., Zhang, X., Yao, S., and Shao, X. Distance-based and rkhs-based dependence metrics in high dimension. The Annals of Statistics , 48(6):3366-3394, 2020.

Zhu, L., Xu, K., Li, R., and Zhong, W. Projection correlation between two random vectors. Biometrika , 104(4):829-843, 2017.

Peer reviews:
comments: SUMMARY:
The paper investigate the performance of HSIC to test the independence of two random vectors. The focus is on high but not ultra high dimensional scenarios where the theory is lacking.  More specifically, the paper presents convergence rates for HSIC as the dimensions grow at different rates, and demonstrates how HSIC’s capacity to measure nonlinear dependence evolves as the dimensions increase.  
The paper also shows that the rescaled HSIC converges in distribution to a standard normal distribution under the null hypothesis and provides the conditions needed to have nontrivial power in high dimensions. The theory is validated by simulations and real-world data involving stock prices from the energy sector and raw material sector in the US stock market.

STRENGTHS:
This is a solid paper with strong theory and convincing numerical support. A key advantage of Theorem 1, which provides the asymptotic distribution (with rates) of the HISC test statistic under the null hypothesis, is that it alleviates the use of permutation test to decide critical values, a requirement commonly observed in other tests of independence.

The phase transition of the convergence rates in Theorem 3 is illuminating.

WEAKNESSES:
None.

QUESTIONS:
How different is the theory between yours and the 2021 paper by Gao et al. in the Annals of Statistics?  What are the differences in technical tools and level of difficulties?

LIMITATIONS:
Some constraints are imposed in the theory but they are fairly mild and reasonable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-02

comments: SUMMARY:
The paper provides insights into the properties of HSIC in high dimensions, more specifically the rate at which sample size must grow in order to detect non-linear correlations if data is high dimensional. The results are categorized based on scenarios where either one or both variables have a "growing" dimension, and they express various types of nonlinearity using conditional expected values of higher orders.

A great summary of the results can be found in lines 61 to 64 of the paper.

STRENGTHS:
The paper effectively presents the main results and is written with clarity. In my opinion, the results are  significant within the sub-area of independence testing. It has been recognized  for some time that the power of HISC diminishes in high dimensions, but as far as I know, no one has provided conditions on the sample size, dimension and degree of non-linearity for the test to detect signal. Formulation of the results in the language of conditional expected values of higher moments yields insightful and concise characterization of the limitations of HSIC in high dimensions.

WEAKNESSES:
I'm happy to see empirical study (section 5.3) in a theoretical paper. Having said that, I would be surprised if a linear test would not reject the null  (pool the returns per sector and run a correlation based test).  

Edit: I read the response, and this still doesn't fully make sense to me. I would have chosen a dataset with a non-linear dependence only  and shown HSIC failing. However, ultimately, this does not detract from the quality of the findings and I stand by 7 (8 makes sense too).

QUESTIONS:
Is there related work for MMD? Seems like similar results should/could hold for MMD.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA paper providing tighter analysis and tests for HSIC statistics for independence in some regimes of interest\n\nNB I have only a nodding acquaintance with this statistic, but have used it and regard it as of high importance. I have done my best to learn the background in the time available.\n\nSTRENGTHS:\nThe model provides an interesting and non-trivial bounds for HSIC to test for dependence of high-dimensional covariates, which is an important and useful setting. If I understand correctly they constrain the complexity of polynomial mean dependence that can be detected given covariate sizes and dimensions.\n\nWEAKNESSES:\nThe conditions are very stringent, and so we are left tow wonder if the results apply to real problems.\n\nthe isotropic kernel choice is a very strong restriction, and one that I would never use in practice - heuristically we expect low discrimination power for kernel-based methods under istoropy for \"most\" kernel methods; it would not be surprsiing if this was true for HSIC in particular. I am not clear how essential the isotropy is but the results start to look like a trivial if that is all they can handle: we might think of these as a \"gaussian thin shell\"-type result in that case.\n\nIndependence is particularly important when it is conditional at which point it gives us Bayesian networks; Do any of these results survive for conditional independence? The authors mention such applications (l34-l37) but I believe thereafter discard them.\n\nI am taking the validity of the proofs here largely on faith. Nothing \"looks\" odd, but I have not stringently checked.\n\nQUESTIONS:\nIs the isotropy essential to these bounds or can we use other kernels? Do the bound still hold if we relax isotropy to some \"better\" distance metric? How about if we use a kernel that incorporates some prior knowledge of the domain?\nHow about other kernels, incl nonstationary ones? Can I improve my bounds by using a polynomial kernel, which is not a characteristic but might be useful for certain types of dependence? Or dot-product-type kernels?\n\nCan we apply these results to conditional independence testing, i.e. inferring graphical models?\n\nIf not, the results are still cool, but they do not, IMO, offer anything practical. They would still be a useful improvement in our understanding, however, and thus I favour publishing this.\n\nLIMITATIONS:\nThe authors are clear about necessary conditions for their theorems to hold, but sufficient conditions are not obvious to me. See above for questions about generality.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors provide the statistical properties of HSIC, which is the measure of independency between two random variables. When the random variables have high-dimensionality and have nontrivial dependency, the authors provide the condition for the number of samples in order to successfully detect the dependency.

STRENGTHS:
For the two cases when only one variable is high-dimensional and when both variables are high-dimensional, the authors provide the number of data conditions that the nontrivial dependency can be reliably detected. In particular, when there is no lower-order dependency, the authors show that HSIC experience a difficulty to measure the dependence appropriately. Because the analysis is asymptotic, it is not guaranteed that the tendency should appear with finite dimensions. However in the experiments shown in this manuscript, the tendency is clearly shown in both synthetic and real data.

WEAKNESSES:
Choice of kernels should be related to the argument that HSIC only measures linear dependences (l336). What is the effect of the change of \gamma on the equations in Theorem 3 and the experiments?

QUESTIONS:
1. Is the derived condition for Gaussians (l58-l59)? The authors should mention those conditions in every theorem explicitly. 
2. In theorem 2, what is the condition when n*hCorr^2 is finite when n^{1/2}hCorr^2 is infinity when (A1) or (A2) does not hold?
3. What are the curves shown in Figure 1 and Figure 2? What are the horizontal and vertical axes?

LIMITATIONS:
The comparison with other methods such as distance correlation could provide more information about the difficulty in each case provided in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-09

comments: SUMMARY:
This article deals with the problems of measuring nonlinear dependence between random vectors living in Euclidean spaces and testing for their independence. The authors provide statistical insights into the performance of one of the two major criteria, the Hilbert-Schmidt independence criterion (HSIC), when the dimensions of the random vectors grow at different rates. Their theoretical contribution is completed with an empirical study involving both artificial and real-world data sets.

STRENGTHS:
The major strong point of the contribution seems to be the real data application, which could be of interest even to the non specialist.

WEAKNESSES:
The paper will not appear as self-contained to the non specialist (like me).

The naive reader will find it contrary to intuition that the computation of a criterion measuring a basic statistical connection between random vectors should involve the choice of two kernel functions. This is all the more strange as the two vectors take their values in Euclidean spaces, but the Euclidean dot product is not an option to be favoured. Could this be motivated in a simple way?

The originality of the contribution is difficult to assess, all the more since the other major criterion, the distance correlation (DC) criterion, has already been the subject of a similar study.

QUESTIONS:
Could the authors provide the definition of technical concepts, like the "degree of conditional mean of x given y", or the "n-th Kronecker power of x"? More generally, could they make the paper technically more self-contained for the non specialist?

Could the comparison between HSIC and DC be developed ?

The typos should be corrected.

LIMITATIONS:
This criterion does not apply here.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-24