Statistical Insights into HSIC in High Dimensions
Authors: Tao Zhang
Yaowu Zhang
Tingyou Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: High dimensionality; Independence test; Kernel method; Nonlinear dependency.

Abstract: Measuring the nonlinear dependence between random vectors and testing for their statistical independence is a fundamental problem in statistics. One of the most popular dependence measures is the Hilbert-Schmidt independence criterion (HSIC), which has attracted increasing attention in recent years. However, most existing works have focused on either fixed or very high-dimensional covariates. In this work, we bridge the gap between these two scenarios and provide statistical insights into the performance of HSIC when the dimensions grow at different rates. We first show that, under the null hypothesis, the rescaled HSIC converges in distribution to a standard normal distribution. Then we provide a general condition for the HSIC based tests to have nontrivial power in high dimensions. By decomposing this condition, we illustrate how the ability of HSIC to measure nonlinear dependence changes with increasing dimensions. Moreover, we demonstrate that, depending on the sample size, the covariate dimensions and the dependence structures within covariates, the HSIC can capture different types of associations between random vectors. We also conduct extensive numerical studies to validate our theoretical results.

Statistical Insights into HSIC in High Dimensions
Tao Zhang
School Information Management and Engineering Shanghai University of Finance and Economics Shanghai 200433, China
dalizhangt@163. com
Yaowu Zhang'
School Information Management and Engineering MoE Key Laboratory of Interdisciplinary Research of Computation and Economics
Shanghai University of Finance and Economics
Shanghai 200433, China
zhang. www.wmaawwllll..a 1. s.ufe.. edu. cn
Tingyou Zhou
School of Data Sciences
Zhejiang University of Finance and Economics
Hangzhou 310018, China
zhoutingyou@zufe www.:ouwuuueeeuu...eeeooooeeeeee cn
Abstract
Measuring the nonlinear dependence between random vectors and testing for their statistical independence is fundamental problem in statistics, One of the most pop- ular dependence measures is the Hilbert-Schmidi independence criterion (HSIC), which has attracted increasing attention in recent years. However, most existing works have focused on either fixed or very nigh-dimensional covariates, In this work, we bridge the gap between these two scenarios and provide statistical insights into the performance of HSIC when the dimensions grow at different rates. We first show that, under the null hypothesis, the rescaled HSIC converges in distribution a standard normal distribution. Then we provide general condition for the HSIC based tests have nontrivial power in high dimensions. By decomposing this condition, we illustrate how the ability of HSIC to measure nonlinear dependence changes with increasing dimensions. Moreover, we demonstrate that, depending the sample size, the covariate dimensions and the dependence structures within covariates, the HSIC can capture different types associations between random vectors. We also conduct extensive numerical studies to validate our theoretical results.
Introduction
Let x "X=,Xx..,,,))" E R y (Yı...., YT)T E R2 two random vectors. The problems of measuring nonlinear dependence between and y and testing for their independence are fundamental and have wide range applications statistics and machine learning. For example, dependence measures be applied in feature screening (Fan et al. 2020b), model checking (Clarke et al., 2018).
graphical models (Maathuis et al. 2018), and causal inference (Imbens & Rubin 2015).
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Many statistical tools have been developed to measure the dependence between random variables. Classical measures such as Pearson correlation (Pearson 1920). Spearman's 's p (Spearman [904] and Kendall's (Kendall, 1938) can only capture linear or monotonic dependence and may fail to detect complex associations. For univariate covariates, several rank-based measures have been proposed to overcome this limitation, such as those by |Hoffdinng (1948). [Blum et al. (1911), ). [Bergsma & Dassios) (2014) and Chatterjee| (202T). For multivariate covariates, various nonparametrie tests have been introduced to quantify arbitrary associations, including the distance correlation (Székely et al. 2007)
the projection correlation (Zhu et al., (017), the mutual information (Berrett & Samworth, 2019).
the kernel canonical correlation (Bach & Jordan 2002), the constrained covariance (Gretton et al.,
2005), and the Hilbert-Schmmde independence criterion (HSIC, Gretton et al. (007). Among these measures, the distance correlation and the HSIC are perhaps the most widely used and studied in both statistics and machine learning. For instance, Li et al. (2012) applied the distance correlation to perform feature screening. |aan et al. (2020a) generalized the distance correlation to a conditional dependence measure under a factor model setting. [Albert et al. (2022) developed an adaptive test of independence by aggregating HSIC measures. |Pister al. (2018) defined a d-variable HSIC to characterize the mutual dependence among d random variables. is worth noting that the distance correlation is equivalent HSIC with a specific kernel choice (Sejdinovic et al.. 2013).
High- figh-dimensiooal settings pose new challenges and opportunities for dependence measures, Székely & Rizzo (2013) suggested to estimate the distance correlation with the U-statistic theory to avoid bias accumulations when the dimension is high. Ramdas et al. (2013) found that the power of kernel and distance based independence tests decreases polynomially with increasing dimension against some fair alternatives. /Zhu et al. (2020) proved that, when both and grow much faster than the sample size n, the sample distance correlation and HSIC can only detect componentwise linear dependences. Gao et al. )021) established a general condition for the distance correlation based test to have power approaching 1. They also showed that the distance correlation can capture certain pure nonlinear relationship when p = q o(n!/2 under specific alternative hypothesis.
In this paper, we aim to provide statistical insights into the HSIC in high dimensions, motivated by the increasing popularity of HSIC and the prevalence of high-dimensional data. We focus on the asymptotic distributions of HSIC under both null and alternative hypotheses when the dimensionality grows with the sample size. Under the null hypothesis, we prove that the rescaled HSIC converges in distribution to a standard normal distribution. We also derive a general condition for the HSIC based tests to have power asymptotically approaching one. These results extend those of Gao et al. (202T) on the distance correlation. Moreover, our analysis is more comprehensive than theirs. By decomposing the general condition, reveal that HSIC can detect different types of dependences, depending on the dimensionality and sample orders. This sheds light on the performance of HSIC in high dimensions, which has been overlooked in the literature. For illustrative purposes, y be two random vectors with zero mean and identity covariance matrix. When HSIC is used to test for statistical independence between them, we obtain the following conditions for nontrivial power:
let x and
If p + do and q is fixed as n - oo, and there only exists the conditional mean of x's s-th polynomial given y, then p(s-1) must grow slower than n.
If both and q + oo as n + oo, and there only exists the covariance between x's $1 s1-th polynomial and y's 82-th polynomial, then p(=:- (=2 must grow slower than n.
These two conditions reveal significant statistical insights and have important implications in practice. For example, when the data dimension g is small and y is larger than the sample size n. according to the first condition, to ensure p(+-1) smaller than n, can only be 1. In other words, the HSIC can only measure the conditional mean of x given y. When both p and g are larger than n, according to the second condition, ensure HSIC can only measure the covariance between x and y. In summary, our main contributions are:
smaller than n, both 8 and 82 must be 1. That is, the
1. We generalize the results of Gao et al. (202T) for distance correlation to HSIC, which is a more flexible and widely used measure of dependence.
2. We show that HSIC can capture different types of dependences, depending on the dimen- sionality and sample orders, which has rarely been realized in the literature before.
3. In contrast to Zhu et al. (202)). who only focused on the case when both p and g grow much faster than n, our results characterize full picture of the performance of HSIC based test in high dimensions.

---

The rest of the paper is organized as follows. We give some preliminaries about the HSIC in Section 2. Then we present the asymptotic null distribution as well the power analysis for high- nigh-dimensional HSIC in Section 3. In Section 4, we provide statistical insights into the HSIC through connecting the association type with the sample size, the high dimensionality, and the dependence structures within covariates. We conduct comprehensive numerical studies in Sections 5 and conclude this paper with a brief discussion in Section 6, All technical details are relegated to the Supplement.
2 Preliminaries
In this section, we give brief review HSIC measures proposed by Gretton et al. (2007). which is derived from the notion of cross-covariance operator Baker (973). We assume that X X C R and Y E y C I8 are random vectors taking values in Euclidaan spaces respectively, Let J and g be the reproducing kernel Hilbert spaces (RKHSs) on X' and ) with associated kernels K(:,) :) and T,, respectively. Then the cross-covariance operator Cxy associated to F and 9 is defined the operator mapping from 9 to F, such that for all f E F and g G. (f. Cxy cov Instead of using the largest singular value of this operator, Gretton et al. (2007) suggested to adopt the squared Hilbert-Schmide norm, which admits a closed form expression
HSIC(x.y) = (X Xy)1 + (E) -26 [E((((()),,) x1 ) yill.
where (x1.y)) and (xg, y2) are independent copies of (x, y). When the RKHSs, F and G, are well chosen, (e.g., characteristic, (Sriperumbudur al. 2010), the HSIC can be served as independence criterion the sense that it is nonnegative and equals zero if and only x and y are independent. To normalize the HSIC to range from zero to one, follow Zhu al. 2020 to define the squared Schheit correlation as
hCCorr (x, del HSIC H)CC 1/2 x HSIC yy.
(1)
In addition, similar to Zhu et al.
2020 and Albert et al. 2022). ). we focus on the kernels that can be
k( x1 X2 122l/7>x and 1.y1,, 2((ly1 Y21l/ny), where k(-) and i(-) are some real-valued functions defined in [0. oo), and Ty are the bandwidth parameters. This kind of kernel is very commonly used in the literature and includes many positive-definite kernel such as the Gaussian kernel, the Laplacian kernel, the rational quadratic kernel, and the Kernel generating Sobolev spaces, etc. For example, when k<x) exp(-x²), corresponds the Gaussian kernel, and when k(z)) exp(-z), it corresponds to the Laplacian kernel. One may refer to Genton (2001) for more examples and discussions about isotropic kernels.
written in the form of [(x,, X2)
3 Asymptotic Properties in High Dimensions
At the sample level, with the random sample {(x, I,,,,,), n], we estimate the HSIC with U-statistics,
(x,c) der E(rrs)) (
2 2 2(innee))) K K(xi, n(n 1)(n 2)
Vis)
n(n - 1)
E(intsrisiss) K (x), X1y ((v...))
n(n (n n 2) (n - 3)
where the indexes in the summands, i.e., (11, 12), (i,, de, is) and (iz, i2, is, i1). are all distinctive from each other. Then the squared Hilbert-Schmide correlation hCorr? y) can be estimated as
hCorr 2 n (x, y) def HSIC,, (x. HSIC, (x.//XxxxxXXXIIIIII x)SSSI", (y..).
We now discuss how to use the sample squared fillber--Schmit correlation to distinguish between the null hypothesis Ho x is independent of y, and the alternative hypothesis H₁ x is not independent of y. To implement the HSIC based test, it is required to study the asymptotic distributions for hoCorr, (x, under the null hypothesis. When the dimensions are fixed. Gretton et al. (007)) showed that n hCrrr (x. y) converges in distribution to weighted sum of independent chi-squared random variables long as x is independent of y. We now study its asymptotic null distribution in high
fixetonGettoon
3

---

E(zi 22) dk= for dimensions. Before that, we define the two quantities Hx (x1, and (V1, V2) as
Hx(X1,,3 X2)
X(x1,x2) -
( x))( x1} - (x1, x2) + (x1,
Y2)
3(y N() y1) F.(T(yy,, y2) + (2)( (3)
We further define another two quantities Gx(x1, and
as
(x), det xx)) (X2, xa) x1,
Gy (21.22)
def E1H vs) y1,32).
Then we summarize the asymptotic null distribution for hCorr in the following Theorem.
Theorem 1. Assume the kernels are symmetric with finite fourth moment, i.e., K xx),
K (xx2, x1), L(y1,y)) Y2) = L1),.y1), E(K (x1,"())x)))) < co and E EXLA(y1,y2)) < co. Further assume that + 9 + 00,
(X x2)) F ((H n (HSIC(,,,)HSIC(YYY x)HSIC(y,y))
0, and x2)
- 00
(4)
{HSIC(X, x) H))IIC(y
as T1 oc. Then under the null hypothesis, we have 2-1/2n lCCorr, yx 4 ().
Different from that when the dimensions are fixed, Theorem D reveals that, under certain conditions, n hCorr, (x, y) is asymptotically normal in high dimensions. The asymptotic normality makes the limiting null distribution tractable in practice. It greatly expedites the implementation of HSIC based tests because no additional permutations are required to decide critical values. We remark here that the assumptions imposed in Theorem 1 to ensure the asymptotic normality are generally very mild. For the restrictions on the kernels, many commonly used kernels, including Gaussian and Laplacian kernels, are symmetric with finite fourth moment. For the assumption in (4). it is used to restrict the weak dependence within covariates in x and y. It holds true when and y follow the multivariate normal distributions with bounded eigenvalues. It also holds true when x and y are m-ddepenenn random sequences (Definition 9.1 of DasGupta 2008). One may refer Gao et al. (202I) for more discussions about this assumption.
Next, we explore the power performance of the HSIC based test in high dimensions. As discussed in Section 2, we restrict our attentions to the isotropic kernels. Specifically, in the remaining of this paper, we consider K (x1, x2) and L(y1.Y2) be in the form of k[[[x, Xg //x) and ((Ily: y2ll/ / respectively. Because of the kernel form, we assume without loss of generality that Ex 0 and
Ey We denote by z def 2/72. where E Rd can be either x y. We say that f(n) x g(n) if f() = [g())} and (()) = ((]))). Before summarizing the asymptotic power performance, we make the following assumptions:
(A1) There exists some Ki such that F|[1z"||2 all & E N+
(A2) Let (o(x) = k(x2/2) and (o(v) = /2)) The first and second derivatives of k- ) and (.)) are uniformly bounded away from zero to infinity around El|x; respectively.
x1|l2 and Ellyy yill..
We remark here that Assumption (A1) is used to restrict the dependence structures within the coordinates of z. Because z = z//% and Ez = O, we have Ez 0 as well. Hence, both 1 E([[z*|]3) and z 22 are sums of d random variables with mean zero. Their standardized versions may converge in distribution to standard normal distributions under mild conditions (e.g., mixing conditions) by applying the central limit theorem (CI.T). For instance, Volný (1989 established CLT for non-stationary mixing processes. Alternatively, there exists some K² O such that " 22; both converge in distribution to normal random variables. Then Condition (A1) satisfied. Assumption (A2) imposes some conditions on the kernels, which covers the Gaussian and Laplacian kernels. Moreover, it also makes several requirements on the bandwidth parameters as well. For example, suppose the Gaussian kernel used, i.e., ko(=) = exp(-z), when is small enough such that E El =2 + co, then we have (Ellzi - 25 [12) - 0. which violates Assumption (A2). Theoretically, the bandwidth parameter 7x can chosen from wide range of values, as long as it satisfies condition (A2). In practice, use the median of
[ 2 [(2) and /"=/

---

|zz - zz]l as default value for "2. When it has the same magnitude with El zzll, Assumption (A2) can be easily satisfied for many commonly used kernels. Moreover, as demonstrated in the simulated examples in the Supplementary Material, our method exhibits robustness across various selections of 7x-
With these two assumptions, we are ready to provide the asymptotic power performance of the HSIC based test in high dimensions.
Theorem 2. Assume (AI) and (A2) hold true. Then under the alternative hypothesis, if n h/ (x,y) + 30 as + 0o, we have n hCorr, (x, (x, y) + 00 in probability.
Theorem reveals that, as long as the dependence measured by hCorr"(x, is not too small, n hCorr n x, y) converges to infinity in probability. This, together with Theorem guarantee that the HSIC based test have nontrivial power in high dimensions. However, the dimension increases, the signal measured by hCorr"(x. y) may decay to zero, which makes the test lose power in high dimensions. To gain more insights about how the power is influenced by the dimensions, is desired to connect hCorr" (x. y) with the dimensionality, as well as the association types between x and y.
4 Statistical Insights in High Dimensions
We have discussed the asymptotic properties of HSIC based tests in high dimensions, and demon- strated that it can detect the alternative hypothesis long the signal strength measured by hCCorr (x,y) y) does not decay to zero too fast. However, it is still unclear that how hCoor y) is influenced by the dimensions. To gain more insights into the performance in high dimensions, in this section, we study what kind of association types the HSIC can detect under different relationships between the sample size and the covariate dimensions. To this end, we expand hCorr? (x, defined in (T) to gain some insights. Because HSIC(x, y) characterizes the dependence between and y, while HSTC(x, x) and HSIC(y. y) are the normalizing factors, we expand HSTC(x, y) and calculate the orders for HSIC(x, x) and HSIC(y,y).
We first calculate the orders for HSIC(x, x) and HSIC(y, y) at the population level as the dimensions diverge to infinity. The results are summarized in the following Proposition.
Proposition 1. Under Assumptions (AI) and (A2), HSIC(z, 2) = d Am if oc.
According to Proposition 1 the condition n n//"hCorr"(x,, - co, which guarantees the HSIC based test would have power approaching in high dimensions, will boil down to n2/2 /
co.
Next, we expand HSIC(x, y) at the population level. Before that, we introduce some additional notations. Similar to Shao & Zhang (2014), we define MD²(x y) as
MD² (x yy)
E((x Fx) (x2 - Ex))(y
where (i) and (X2. Y2) are independent copies of (x, y). We show in Lemma that, MD² (x y) = O if and only if [( y) = 0. That is, MD y) measures the degree of conditional mean of x given y, which quantifies the difference between F y) and Fxx.
1 h/y)) is characteristic. bounded continuous, Lemma 1. Assume the kernel L(y1.y2) = I(lly: y2
and real-valued positive definite function, then MD2 (x y) 0 if and only if E(x y) 0 almost surely.
We remark here that characteristic condition in Lemma is used to guarantee that the HSIC is nonnegative and equals zero if and only if x and y are independent. One may refer (Sriperumbudur et al. 2010) for more details about this condition. Furthermore, we let ((()) be the i-th derivative of
(-) evaluated E||x xiil² and 1(2) is the j-th derivative of (n(-) evaluated at Ellyī y 1|
Let &n be the n-th kronecker power of x, which is defined as x x @ ())-1)) x01 x, and @ denotes the Kronecker product. In addition, we use 11r to represent the Frobenius norm of matrix. Then we expand HSIC(x, y) in the following Theorem.
Theorem 3. Assume (Al) and (A2) hold true. Then under the alternative hypothesis.
5

---

1. when + oc and remains fixed as + 0c, if Exx y) = E(xx*) hold true for all
t < for some E N then HSIC(x, y) O(p snix/2), and
(-2) MD² (x 0 1x11 Iy) + olp skx/2))
HSIC(x,y) = k:)²)
Σ
ala!c!
2a+c=
2. when p 20 and q - 00 50
00, cocvoxx4, y + only when t 2 81 and
l/ 2 82 for some 81, 82 E EN, then H)) 0p-0pppp00 10x-////// ////2 22 and
1 (-2) (600) (-2)
HSIC(x,y) E
E
cove11x
[|1x0] 1 0at lly y 6oz
aglaylc!! a2la2lo2!
2astc 202
+olp "1x//3 -
Theorem 3 enables to gain some insights into the HSIC in high dimensions. First of all, because
when 2a + c = x ||*|| is the s-th polynomial for x, and MD² ID²(xx* (x* ||x* y) measures
the departure from the conditional mean independence of x given y. Then the first part of Theorem
3 implies that when - and remains fixed as co, if E for all ( < for some E N+, the HSIC quantifies the departure from Ex y = E [ @*). In
y) E 01) hold true
other words, when only one dimension of the covariates diverges infinity, the HSIC first measures the conditional mean of x given y. there is no such mean dependence, turns to measure the and
conditional mean of x22 given y, etc. Similarly, when p part of Theorem reveals that, the HSIC may first search for the covariance between x given y. such covariance equals zero, it then turns to search for the covariance between x82 and y, or the covariance between x and y82 etc. Therefore, Theorem 3 characterizes the changing process of the ability to measure nonlinear dependence in high dimensions for HSIC. As comparison, [Zhu] eet al. (2020) only showed that, when both dimensions diverge much faster than the sample size, the HSIC can only capture the marginal linear dependence. From this point of view, our result is generalization of theirs.
- 0O, the second
When the HSIC is used to test for statistical independence, Theorem and Proposition I ensure that, when n
y) - oo, the HSIC based test would have asymptotic power To make this condition hold true, the first part of Theorem implles that when p -> oo and g remains fixed n xo, if there only exists the conditional mean of x8* given y. it is required that p(=-1)(sx 1)=- o(n) because HSIC(x, y) is of order O(p ((pps*x/2) in this case. In other words, when = 0[p(--1)<x J. the HSIC based test can have nontrivial power only when there exists the i-th polynomial mean dependence of x given y for some < 8. Similarly, when both p and q diverges as + co, the second part of Theorem 3 when there only exists the covariance between x 2881 and ( the test based on HSIC can have nontrivial power only when pl= 11)xx g 11)x o(n). Because Nx represents the degree of dependence within covariates. Theorem 3 successfully connects the association type between x and y. with the sample size. the high dimensionality, and the dependence structures within covariates. We remark here that Gao et al. (202T) only justified that the test based on distance correlation have power approaching when p = 9 1/2) if there exist a pure nonlinear dependence between x and Therefore, their result is in spirit a special case of ours.
Numerical Studies
5.1 Simulations
In this subsection, we conduct some simulation studies to validate our theoretical conclusions on the HSIC based test in high dimensions. In particular, in Example we show that, as long as the dimensions are not very small, the normal approximation can approximate the null distribution quite well, although it requires the dimension to diverge to infinity in theory. In Example 2, we fix the dimension of y and increase the dimension of x, and investigate the finite sample power performance of the HSIC based tests under various mean dependence types. In Example 3, increase the dimensions of x and simultaneously, and inspect how the empirical power is influenced by the covariate dimensions under different relationships.
Throughout the simulations, we choose two kinds of commonly used kernels to implement the HSIC based tests, i.c., Gaussian and Laplacian. For the Gaussian kernel, we set K x2)

---

exp[-l|x1 -x2l) and L(y1.y2) exp(-llyi -22l While for the Laplacian kernel, we set X2) = exp(-||x choices of kernels, we set the bandwidth parameters to be 7 co7x and % coty and vary Co from 0.5 2. Here, 7x represents the median of [||z: - and is either x or y. Due to space constraints, we present results specifically for the case when =xx and Jy = =y The remaining results are summarized in the Supplementary Material. Notably, our findings remain consistent across different values of 7 and Ty: showing the stability and reliability of our approach.
- 1 / and (2), = =exp(-llx> 1epo-llyy--y2llll - y2ll/hy). For both
Example 1. In this example, we examine the normal approximation accuracy under the null hypoth- esis. Towards this goal, we simply generate x = =.,,--. x,)) E RR" from a multivariate normal distribution with mean zero and covariance matrix Σx Ipxp. Then we independently generate y y= (Yi,.... Y²)² E _ from another a multivariate normal distribution whose coordinates follow iid standard normal distributions. We fix the sample size be 100 and consider two scenarios for the covariate dimensions. In the first scenario, we fix q = 1 and vary from {5, 25, 100}. In the second scenario, we set p = q and vary d from {2, 5, 10). For each setting, we repeat the experiments 5000 times and compare the empirical null distributions the test statistics, {2 1/2 hCorr (x, y)}, with the standard normal distribution. Specifically, we display the kernel density curves of the test statistics for the two scenarios in Figures[ande respectively.
(A): p
(B): p 25
(C): p 100
Figure 1: The kernel densities of the test statistics under the null hypothesis computed from 5000 simulations. We fix g = 1 and vary p from (5, 25, 100). The horizontal axes represent the observed values of the test statistics, and the vertical axes represent the kernel densities of those values. We choose two different kernels to implement the tests, i.e., Gaussian (dashed line) and Laplacian (dotted line). The solid line is the reference curve, which is the density of the standard normal distribution.
According to Figures aaddd and 2, we can see that, when the dimensions are small, the normal distribution cannot approximate the null distribution well. For example, when = q = 2, the empirical null distributions for the tests with both Gaussian and Laplacian kernels are quite distinct from the reference curve in Figure? 2 (A). However, as the dimensions increase, the accuracy of the normal approximation for the null distribution becomes quite precise, regardless of the choice of the kernels. For example, in both Figures and 2 the empirical null distributions are almost the same as the reference curve long pq not smaller than 25.
Example In this example, we investigate how the empirical power influenced by the dimension when only one of the covariate dimensions increases. For simplicity, we fix the sample size to be As for the covariate dimensions, we fix q 1 and vary p from {30, 50, 100, 200, 500, 1000]. We consider three models to inspect the power performances. For Models (I) and (II), we generate x from multivariate normal distribution with mean zero and Jxxp.
100, as that in Example
covariance matrix Σx= = (0.5li-5l) ) Then we generate an independent error term that follows standard normal distribution. The univariate response Y is generated through
Model (I)
Y X1
+ X. + E:
Model (II)
Y = x + X² +e.
For Model (III), we first generate Y N(0, 1). Conditional on Y, we generate x through
Model (III):
Y} (2x)
Jaxe), NN0, (exe)) k= k=
7

---

10
(A): 2
(B):
(C): d
Figure 2: The kernel densities of the test statistics under the null hypothesis computed from 5000 simulations. We set values of the test statisties, and the vertical axes represent the kernel densities of those values. We choose two different kernels to implement the tests, i.c., Gaussian (dashed line) and Laplacian (dotted line). The solid line the reference curve, which the density of the standard normal distribution.
and vary d from {2, 5, 10}. The horizontal axes represent the observed
Table 1:
The empirical powers of different tests for Models (I)-(III) in Example 2. The significance level 0.05. We fix q and vary p from (30, 50, 100, 200, 500, 1000).
Model
Test
p
30
50
100
200
500
1000
Gaussian
1.000
1.000
1.000
1.000
0.998
0.954
(I)
Laplacian
1.000
1.000
1.000
1.000
0.994
0.916
DC
1.000
1.000
1.000
1.000
1.000
0.998
Gaussian
0.934
0.774
0,484
0.318
0.184
0.132
(II)
Laplacian
0.998
0.978
0.834
0.596
0.314
0.234
DC
0.974
0.894
0.650
0.412
0.226
0.176
Gaussian
0.044
0.050
0.060
0.046
0.060
0.068
(III)
Laplacian
0.050
0.046
0.054
0.048
0.054
0.060
DC
0.050
0.050
0.052
0.034
0.064
0.044
where Pk.y (25g- (Y), dgs are iid Bernoulli(O.5) random variables, and +(.) is the cumulative distribution function of a standard normal distribution. In addition, 1X xxx)'5 are mutually independent conditional of Y for all k = l.....P. It can be verified that there exist linear, quadratic and quartic conditional means of x given Y in Models (I), (II) and (III), respectively.
Y. We remark here that in Model (III), X₁ marginally independent
As we have demonstrated in Example 1, when q = 1, the standard normal distribution can approximate the null distribution pretty well long p no less than 25, to obtain p-values. We repeat each experiment 500 times and report the empirical powers Table As comparison,
the standard normal distribution
also include the results for the distance correlation in Table We observe that all the empirical powers for Model (I) can still approach one, implying that the linear mean dependence can be easily detected, even when the dimension is much higher than the sample size. For example, when = 1000, the power for Model (I) when the Gaussian kernel is used is still as high as 0.954. However, when there only exists some higher order of mean dependence, the empirical powers drop significantly. In addition, the performances deteriorate as the order of mean dependence increase. For example, when p = 100 and we choose the Gaussian kernel to implement the test, the empirical power for Model (III) is only 0.060, which is smaller than 0.484, the empirical power for Model (II).
Example In this example, we study how the dimensionality can effect the empirical power when both of the covariate dimensions increase simultaneously. We also fix the sample size to be n. 100. We set p = q = d and vary d from (6, 10, 20, 50, 100, 200}. Similar to Example 2, we consider three models in this example. For Models (IV) and (V), generate the same as that in Models (I) and (II), i.e., (00 (0.5l pxxp) The independent error terms E1, El..... ... Ed generated from d independent standard normal distributions. Then we generate y = (Y,...., ........ Ya) through
Model (IV)
Y, = Xg + Egg j 1, I,...,;;
8

---

Table 2: The empirical powers of different tests for Models (IV)-VVI in Example 3. The significance level 0.05. We set p = 9 = d and vary d from {6. 10, 20, 50, 100, 200).
d
Model
Test
6
10
20
100
200
Gaussian
1.000
1.000
1.000
1.000
1.000
1.000
(IV)
Laplacian
1.000
1.000
1.000
1.000
1.000
1.000
DC
1.000
1.000
1.000
1.000
1.000
1.000
Gaussian
1.000
1.000
0.944
0.440
0.242
0.140
(V) Laplacian
1.000
1.000
1.000
0.904
0.578
0.282
DC
1.000
0.986
0.844
0.400
0.232
0.136
Gaussian
0.056
0.064
0.046
0.078
0.038
0.072
(VI)
Laplacian
0.058
0.064
0.044
0.074
0.040
0.072
DC
0.060
0.066
0.046
0.078
0.040
0.072
Model (V):
y= + ej, j= j=1,...,dd
For Model (VI), we first generate all coordinates of y independently from standard normal distribu- tions. Then given y, we generate x through
Model (VI): (x yy ~ [ (pkk 12x2), k= =......,, d/2,
where Pkay (26g (Y2x) and dES are iid Bernoulli(0.5) random variables. Meanwhile, we generate (X X2x) )ss independently conditional on y. Similar to Models (I) (II) and (III), there exist covariances between y and linear, quadratic and quartic polynomials of x in Models (IV), (V) and (VI), respectively.
Following that in Example 2, we use the standard normal distribution to obtain p-values and repeat each experiment 500 times to report the empirical powers. The corresponding results are summarized Table 2 from which we observe a similar phenomenon. That is. the empirical powers decay as the dimensions increase, and the decay rates increase when there only exist higher orders of covariances between and y. addition, according to both Tables [ and 2 the conclusions for the HSIC with Gaussian and Laplacian kernels also apply to the distance correlation. This is in line with our anticipation because the distance correlation is equivalent the HSIC with some particular kernel.
5.2 Real Data Application
The prices of gasoline and many other fuels may have important impacts on raw materials and related companies. For example, the petroleum is widely used upstream raw materials by many chemical industries. We are interested in whether there is any dependency between stock prices of the energy sector and the raw material sector in the U.S. stock market. To see this, we extract the monthly mean stock prices of energy companies as well as raw material companies starting from January 2021 to December 2022 from https://fnance, waw.oe...hho.oomm According to the Global Industry Classification Standard, we get 345 and 294 companies from raw material and energy sectors, respectively. We remove those stocks whose data are not complete in this period, yielding 224 companies from the raw material sector and 214 companies from the energy sector, respectively. At each month from the two sectors by x, = (X.1,... X Y and = (Ya...., Yig)", where p 224 and q = 214. Let S% = log (og(Xx: (x//(-1,5) and S log /it//YY ) be the stock returns at month for the i-th company in the raw material sector, and j-th company in the energy sector, respectively. We test whether the stock returns in these two sectors are independent using {(8), s ()) ) where St (S ...... ) and s ( )r We implement the test using HSIC with Gaussian and Laplacian kernels. The resulting p-vulues are 2.031 x 10-10 10 and 2.749 x 10-9, respectively. Then we can conclude that there exists dependences between the monthly mean stock prices of the energy sector and the raw material sector.
1 < 24, we denote the monthly mean stock prices of these companies
We also compare the RV coefficient with our proposed method in this dataset. The RV coefficient yields p-vuuue of 2.02 x 10-4, which indicates the presence of linear dependences in this dataset. We remark that, in this data set, both covariates dimensions p and g are much larger than sample size According to the second part of Theorem 3 and the discussions at the end of Section 4, the HSIC can only have nontrivial power if p(e1-1)8x (12- o(n). In the context of our dataset, this

---

condition is satisfied exclusively when 81 $2 signifying covariance relationship between x and y. This together with the fact that the test based on HSIC rejected the null hypothesis, we conclude that there exists linear dependence relationship between and y. This is consistent with the RV coefficient result. Moreover, we identify that there a strong linear relationship between stock returns for two pairs of companies: Denison Mines Corp. and Energy Fuels Inc., and Uranium Energy Corp. and Energy Fuels Inc. The It's for the corresponding linear models are 0.8779 and 0.87 18, respectively. This means that the models fit the data very well and that the stock returns move together in a predictable way.
Conclusion and Discussion
This paper aims investigate the performance of HSIC in high dimensions from theoretical perspective. We first prove that the asymptotic null distribution of a rescaled HSIC is a standard normal in the high dimensional setting. Then we derive general condition for the HSIC based tests to have power asymptotically approaching one. To gain more insights, we decompose this condition and show that it depends the sample size, the covariate dimensions, the dependence structures within covariates, and the association types between x and We also show that the ability of HSIC detect nonlinear dependence deteriorates the dimensions increase. Most existing works focused on either fixed dimensions or dimensions that diverge much faster than the sample size. We fill this gap in the literature and provide a comprehensive picture of the performance of HSIC based tests.
Our theory also reveals how the ability of HSIC to capture nonlinear dependence diminishes in high dimensions. To improve the power performance, Zhu et al. (2020) proposed aggregate marginal sample HSIC as the test statistic instead of using HSIC over the whole features. However, this approach only measures marginal dependences, and is well known that marginal independence does not imply joint independence. Moreover, because it sums over all pairs of marginal HSIC, it may suffer from significant power loss when the alternative sparse. Developing an independence test that has nontrivial power against all kinds of dependences remains an open question. Therefore, it would be interesting to explore new methods that can enhance power performances in high dimensions.
Acknowledgments and Disclosure of Funding
The authors thank five anonymous reviewers for their comments and suggestions that significantly enhanced the quality of the paper. All authors contribute equally to this paper and their names are listed in alphabetical order. This work is supported by National Natural Science Foundation of China (12271331 and 72192832), and Program for Innovative Research Team of Shanghai University of Finance and Economics.
References
Albert, M., Laurent, B., Marrel, A., and Meynaoui, A. Adaptive test of independence based on hsic measures. The Annals of Statisties, 50(2):858 879, 2022.
Bach, F. R. and Jordan, M. I. Kernel independent component analysis. Journal of Machine Learning Research, 3(Jul):1-48, 2002.
Baker, C. R. Joint measures and cross-covariance operators. Transactions of the American Mathe- matical Society, 186:273-289. 1973.
Bergsma, W. and Dassios, A. A consistent test of independence based on a sign covariance related to kendall's tau. Bernoulli,
2014.
Berrett, T. and Samworth, J. Nonparametric independence testing via mutual information. Biometrika, 106(3):547-56,, 2019.
Blum, J., Kiefer, J., and Rosenblatt, M. Distribution free tests of independence based on the sample distribution function. The Annals of Mathematical Statistics, 32(2):485 498, 1961.
Chatterjee, A new coefficient of correlation. Journal of the American Statistical Association, 116 20022 1.21.
10

---

Clarke, E. M., Henzinger, T. A., Veith, H.,, and Bloem, R. Handbook of Model Checking. Springer, 2018.
DasGupta, A. Asymptotic Theory of Statistics and Probability, volume 180. Springer, 2008.
Fan, J., Feng, Y., and Xia, L. A projection- conditional dependence measure with applications to high-dimensional undirected graphical models. Journal of Econometrics, 218(1)119-139) 2020a.
Fan, J., Li, R., Zhang, C.-H., and Zou, H. Statistical Foundations of Data Science. Chapman and Hall/CRC, 2020b.
Gao, L., Fan, Y., Lv, J., and Shao, Q.-M. Asymptotic distributions sigh-dimensional distance correlation inference. The Annals of Statistics, 4994) 2021.
Genton, M. Classes of kernels for machine learning: statistics perspective. Journal of Machine Learning Research, 2(Dec):299-3122 2001.
Gretton, A., Herbrich, R., Smola, A., Bousquet, O., and Schölkopf, B. Kernel methods for measuring independence. Journal of Machine Learning Research, 6:2075-2129, 2005.
Gretton, A., Fukumizu, K., Teo, C., Song, L.,, Schölkopf, B., and Smola, A. A kernel statistical test of independence. Advances in Neural Information Processing Systems, 20, 2007.
Hoeffding, W. A non non-parametric test of independence. The Annals of Mathematical Statistics, pp. 546- 557, 1948.
Imbens, G. W. and Rubin, D. B. Causal Inference in Statistics, Social, and Biomedical Sciences. Cambridge University Press, 2015.
Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Li, R., Zhong, W., and Zhu, L. Feature screening via distance correlation learning. Journal of the American Statistical Association, (974999::122-1133 2012.
Maathuis, M., Drton, M., Lauritzen, S., and Wainwright, M. Handbook of Graphical Models. CRC Press, 2018.
Pearson, K. Notes on the history of correlation. Biometrika, 13(1):25-455 1920.
Pfister, N., Bühlmann, P., Schölkopf, B., and Peters, J. Kernel-based tests for joint independence. Journal of the Royal Statistical Society: Series (Statisticul Methodology), 80(1):5-31, 2018.
Ramdas, A., Reddi, J., Póczos, B., Singh, A., and Wasserman, L. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Twenty-Nini AAAI Conference on Artificial Intelligence, 2015.
Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. Equivalence of distance-basse and rass-based statistics hypothesis testing. The Annals of Statistics, pp. 2263-2291 2013.
Shao, X. and Zhang, J. Martingale difference correlation and its use in nigh-dimensional variable screening, Journal of the American Statistical Association, 109(507): 2014.
Spearman, C. The proof and measurement association between two things. American Journal of Psychology, 15:72-101, 1904.
Sriperumbudur, K.. Gretton, A., Fukumizu, K., Schölkopf, B., and Lanckriet, G. R. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research, 11: 1517-1561, 2010.
Székely, G. J. and Rizzo, M. L. The distance correlation t-test of independence in high dimension. Journal of Multivariate Analysis, 117:193-233, 2013.
Székely, G. J., Rizzo, M. L.,, and Bakirov, Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35(6):2769-277 2794, 2007.

---

Volný, D. A central limit theorem for non stationary mixing processes. Commentationes Mathemati- cae Universitatis Carolinae, 30(2):005-407, 1989.
Zhu, C., Zhang, X., Yao, S., and Shao, X. Distance-based and rkhs-based dependence metrics high dimension. The Annals of Statistics, 48(6):3366-3394, 2020.
Zhu, L.,, Xu, K., Li, R., and Zhong, W. Projection correlation between two random vectors. Biometrika, 104(4):829-884 823-843333 2017.
12


Reviews:
[{"review_id": "wNOiyo9goi", "reviewer": "Reviewer_WrHD", "summary": "This article deals with the problems of measuring nonlinear dependence between random vectors living in Euclidean spaces and testing for their independence. The authors provide statistical insights into the performance of one of the two major criteria, the Hilbert-Schmidt independence criterion (HSIC), when the dimensions of the random vectors grow at different rates. Their theoretical contribution is completed with an empirical study involving both artificial and real-world data sets.", "questions": "Could the authors provide the definition of technical concepts, like the \"degree of conditional mean of x given y\", or the \"n-th Kronecker power of x\"? More generally, could they make the paper technically more self-contained for the non specialist?\n\nCould the comparison between HSIC and DC be developed ?\n\nThe typos should be corrected.", "limitations": "This criterion does not apply here.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The major strong point of the contribution seems to be the real data application, which could be of interest even to the non specialist.", "weaknesses": "The paper will not appear as self-contained to the non specialist (like me).\n\nThe naive reader will find it contrary to intuition that the computation of a criterion measuring a basic statistical connection between random vectors should involve the choice of two kernel functions. This is all the more strange as the two vectors take their values in Euclidean spaces, but the Euclidean dot product is not an option to be favoured. Could this be motivated in a simple way?\n\nThe originality of the contribution is difficult to assess, all the more since the other major criterion, the distance correlation (DC) criterion, has already been the subject of a similar study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for your valuable feedback and constructive suggestions.\n\nComment 1: The paper will not appear as self-contained to the non specialist (like me). The naive reader will find it contrary to intuition that the computation of a criterion measuring a basic statistical connection between random vectors should involve the choice of two kernel functions. \n\nResponse 1: We agree that the choice of two kernel functions may seem unintuitive and complicated to some readers, especially those who are not familiar with the HSIC test. However, for the sake of generality and completeness, we allow them to be different in our theoretical study. In practice, one can choose the same kernel function for both variables, or use a common kernel function such as the Gaussian kernel, which has been shown to perform well in many previous studies  (see, e.g., Albert et al., 2022). We hope this clarifies our motivation and rationale for choosing two kernel functions in our paper. \n\nComment 2: This is all the more strange as the two vectors take their values in Euclidean spaces, but the Euclidean dot product is not an option to be favored. Could this be motivated in a simple way?\n\nResponse 2: The Euclidean dot product is not an option to be favored because it does not capture the nonlinear dependence between random variables. For example, the covariance uses the Euclidean dot product, which can only measure linear dependences. The kernel functions, on the other hand, can measure the dependence between the variables in a high-dimensional feature space, where the nonlinear dependence can be better detected. This is the essence of the kernel trick, which is widely used in machine learning and statistics. We hope this explains why we do not use the Euclidean dot product in our paper.\n\nComment 3: The originality of the contribution is difficult to assess, all the more since the other major criterion, the distance correlation (DC) criterion, has already been the subject of a similar study.\n\nResponse 3: We appreciate the reviewer’s comment on the originality of our contribution. We agree that the distance correlation (DC) has been studied in the literature, such as in Zhu et al. (2020) and Gao et al. (2021). However, our paper is different from theirs in the following ways.\n- We study the HSIC, which is more general than DC. We prove its asymptotic normality under the null and a general condition for its consistency under the alternative.\n- We provide a much more comprehensive analysis of the HSIC based test in high dimensions than previous works. We show that HSIC can capture different types of dependences, depending on the dimensionality and sample orders, which have not been realized before. Our results characterize a full picture of the HSIC based test in high dimensions, while previous works only focused on some specific cases.\nWe hope this clarifies the originality and contribution of our paper. \n\nComment 4: Could the authors provide the definition of technical concepts, like the \"degree of conditional mean of x given y\", or the \"n-th Kronecker power of x\"? More generally, could they make the paper technically more self-contained for the non specialist?\n\nResponse 4: Thank you for the kind reminder. The degree of conditional mean of x given y quantifies the difference between $E(x\\mid y)$ and $Ex$, which is measured by $MD(x\\mid y)$ in the paper. The n-th Kronecker power of x is defined as  $x^{\\otimes n} = x\\otimes x^{\\otimes(n-1)}$, $x^{\\otimes 1} = x$, and $\\otimes$ denotes the Kronecker product. We sincerely apologize for any lack of clarity in the previous version of the paper. We will make significant efforts to rephrase technical terms and provide more intuitive explanations to aid non-specialist readers.\n\nComment 5: Could the comparison between HSIC and DC be developed?\n\nResponse 5: We appreciate your interest in comparing HSIC and DC. As we mentioned in the paper, DC is a special case of HSIC when the distance-induced kernel is used, and our results also apply for DC. Therefore, the comparison between HSIC and DC depends on the choice of kernels. However, choosing an appropriate kernel for HSIC is not trivial. Therefore, we believe that there is no definitive answer to which method is better in theory, and the performance may vary depending on the data and the application. We hope this clarifies our point of view.\n\nComment 6: The typos should be corrected.\n\nResponse 6: We appreciate the reviewer's comment on correcting the typos in our manuscript. As suggested, we went through the whole manuscript carefully and made every effort to correct typos and grammatical errors. For instance, we corrected \"fiar\" to \"fair\" in line 44 and changed the first parenthesis to braces in line 299. \n\nReferences\n\nAlbert, M., Laurent, B., Marrel, A., and Meynaoui, A. (2022). Adaptive test of inde- pendence based on hsic measures. The Annals of Statistics, 50(2):858–879.\n\nGao, L., Fan, Y., Lv, J., and Shao, Q.-M. (2021). Asymptotic distributions of high- dimensional distance correlation inference. The Annals of Statistics, 49(4):1999–2020.\n\nZhu, C., Zhang, X., Yao, S., and Shao, X. (2020). Distance-based and rkhs-based dependence metrics in high dimension. The Annals of Statistics, 48(6):3366–3394."}, {"review_id": "obd6A2ioPh", "reviewer": "Reviewer_V7Sb", "summary": "The authors provide the statistical properties of HSIC, which is the measure of independency between two random variables. When the random variables have high-dimensionality and have nontrivial dependency, the authors provide the condition for the number of samples in order to successfully detect the dependency. ", "questions": "1. Is the derived condition for Gaussians (l58-l59)? The authors should mention those conditions in every theorem explicitly. \n2. In theorem 2, what is the condition when n*hCorr^2 is finite when n^{1/2}hCorr^2 is infinity when (A1) or (A2) does not hold?\n3. What are the curves shown in Figure 1 and Figure 2? What are the horizontal and vertical axes?\n", "limitations": "The comparison with other methods such as distance correlation could provide more information about the difficulty in each case provided in this paper.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "For the two cases when only one variable is high-dimensional and when both variables are high-dimensional, the authors provide the number of data conditions that the nontrivial dependency can be reliably detected. In particular, when there is no lower-order dependency, the authors show that HSIC experience a difficulty to measure the dependence appropriately. Because the analysis is asymptotic, it is not guaranteed that the tendency should appear with finite dimensions. However in the experiments shown in this manuscript, the tendency is clearly shown in both synthetic and real data.", "weaknesses": "Choice of kernels should be related to the argument that HSIC only measures linear dependences (l336). What is the effect of the change of \\gamma on the equations in Theorem 3 and the experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for your valuable feedback and constructive suggestions.\n\nComment 1: Choice of kernels should be related to the argument that HSIC only measures linear dependences (l336). \n\nResponse 1: Thank you for raising this important point about the choice of kernels. The choice of kernels does not influence the theoretical results provided that the chosen kernels are isotropic and satisfy assumption (A2). It covers many commonly used kernels such as the Gaussian kernel, the Laplacian kernel, the rational quadratic kernel, and the Kernel generating Sobolev spaces, etc. In the empirical study, both dimensions p and q are much larger than n, and this is exactly the case where the conditions of the second part of Theorem 3 hold when $s_1=s_2=1$. In this circumstance, HSIC degenerates to the covariance, which only measures linear dependence. That explains why \"HSIC only measures linear dependences\" in line 336. \n\nComment 2: What is the effect of the change of $\\gamma$ on the equations in Theorem 3 and the experiments?\n\nResponse 2: We appreciate your insightful question about the effect of the change of $\\gamma$ on our results. Theoretically, the bandwidth parameter $\\gamma$ can be chosen from a wide range of values, as long as it satisfies condition (A2). In practice, we use the median of $||z_1-z_2||$ as a default value for $\\gamma$, since it ensures that Assumption (A2) holds for many common kernels. However, our method is robust to different choices of $\\gamma$. We will emphasize this in the revised version of the paper. We vary $\\gamma$ from $0.5\\gamma_m$ to $2\\gamma_m$, where $\\gamma_m$ is the median of $||z_1-z_2||$, and report the performance of our method on Examples 1 to 3. The results are consistent across different values of $\\gamma$, showing the stability and reliability of our approach. For illustration, we present the empirical type-I error rates at the significance level $\\alpha=0.05$ for Example 1 with $p=q=10$ in the table below. We hope this answers your question and clarifies our method.\n\n| | 0.5$\\gamma_m$ | $\\gamma_m$| 1.5$\\gamma_m$| 2$\\gamma_m$|\n|:------|:------|:------|:------|:------|\n|Gaussian| 0.0548 | 0.0580 | 0.0570 | 0.0554|\n|Laplacian| 0.0530 | 0.0590 | 0.0588 | 0.0586|\n\nComment 3: Is the derived condition for Gaussians (l58-l59)? The authors should mention those conditions in every theorem explicitly.\n\nResponse 3: We would like to clarify that we do not assume that the random vectors are Gaussian in our paper. In lines 58 to 59, the zero mean and identity covariance matrix assumptions are only for illustrative purposes, and they are not essential for our theoretical results. In all the theorems in our paper, we do not impose any assumptions on the distributions of the random vectors. We will clarify this point in the revised version of the paper.\n\nComment 4: In theorem 2, what is the condition when $n*hCorr_n^2$ is finite when $n^{1/2} hCorr^2$ is infinity when (A1) or (A2) does not hold?\n\nResponse 4: Thank you for this question. Theorem 2 shows that if  $n^{1/2}hCorr^2\\to\\infty$ and Assumptions (A1) and (A2) hold true, the test based on HSIC would have power approaching 1, i.e., for $n * hCorr_n^2$ to diverge to infinity under the alternative hypothesis. This is because $n*hCorr_n^2$ converges to a normal distribution under the null hypothesis.\n\nAssumption (A1) restricts the dependence structures within the coordinates of $z$, while Assumption (A2) imposes some conditions on the kernels and the bandwidth parameters. For example, Assumption (A1) would be violated if all coordinates of $z$ are identical, and Assumption (A2) would be violated if the Gaussian kernel is used and $\\gamma_z$ is small enough such that  $E||z_1^*-z_2^*||^2\\to\\infty$.  We demonstrate these cases below Assumptions (A1) and (A2) in the paper.\n\nAs for $n^{1/2} hCorr^2$, it goes to infinity as long as the dependence measured by $hCorr^2$ does not decay to zero too fast. We show in Section 4 that it captures different types of dependences, depending on the dimensionality and sample orders.\n\nComment 5: What are the curves shown in Figures 1 and Figure 2? What are the horizontal and vertical axes?\n\nResponse 5: We apologize for not providing enough details in the captions of Figures 1 and 2. The figures show the kernel densities of the test statistics under the null hypothesis, computed from 5000 simulations. The horizontal axes represent the observed values of the test statistics, and the vertical axes represent the kernel densities of those values. We use two different kernels to implement the tests, namely Gaussian (dashed line) and Laplacian (dotted line). The solid line is the reference curve, which is the density of the standard normal distribution. We will revise the captions to include this information.\n"}, {"review_id": "VdmOr0VMkZ", "reviewer": "Reviewer_56Ec", "summary": "A paper providing tighter analysis and tests for HSIC statistics for independence in some regimes of interest\n\nNB I have only a nodding acquaintance with this statistic, but have used it and regard it as of high importance. I have done my best to learn the background in the time available.", "questions": "Is the isotropy essential to these bounds or can we use other kernels? Do the bound still hold if we relax isotropy to some \"better\" distance metric? How about if we use a kernel that incorporates some prior knowledge of the domain?\nHow about other kernels, incl nonstationary ones? Can I improve my bounds by using a polynomial kernel, which is not a characteristic but might be useful for certain types of dependence? Or dot-product-type kernels?\n\nCan we apply these results to conditional independence testing, i.e. inferring graphical models?\n\nIf not, the results are still cool, but they do not, IMO, offer anything practical. They would still be a useful improvement in our understanding, however, and thus I favour publishing this.", "limitations": "The authors are clear about necessary conditions for their theorems to hold, but sufficient conditions are not obvious to me. See above for questions about generality.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model provides an interesting and non-trivial bounds for HSIC to test for dependence of high-dimensional covariates, which is an important and useful setting. If I understand correctly they constrain the complexity of polynomial mean dependence that can be detected given covariate sizes and dimensions.", "weaknesses": "The conditions are very stringent, and so we are left tow wonder if the results apply to real problems.\n\nthe isotropic kernel choice is a very strong restriction, and one that I would never use in practice - heuristically we expect low discrimination power for kernel-based methods under istoropy for \"most\" kernel methods; it would not be surprsiing if this was true for HSIC in particular. I am not clear how essential the isotropy is but the results start to look like a trivial if that is all they can handle: we might think of these as a \"gaussian thin shell\"-type result in that case.\n\nIndependence is particularly important when it is conditional at which point it gives us Bayesian networks; Do any of these results survive for conditional independence? The authors mention such applications (l34-l37) but I believe thereafter discard them.\n\nI am taking the validity of the proofs here largely on faith. Nothing \"looks\" odd, but I have not stringently checked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for your valuable feedback and constructive suggestions.\n\nComment 1: Is the isotropy essential to these bounds or can we use other kernels? Do the bound still hold if we relax isotropy to some \"better\" distance metric? How about if we use a kernel that incorporates some prior knowledge of the domain? How about other kernels, incl nonstationary ones? Can I improve my bounds by using a polynomial kernel, which is not a characteristic but might be useful for certain types of dependence? Or dot-product-type kernels?\n\nResponse 1:  Indeed, the isotropy kernel assumption is essential to the bounds derived in this paper. This kind of kernel is very commonly used in the literature and includes many positive-definite kernels such as the Gaussian kernel, the Laplacian kernel, the rational quadratic kernel, and the kernel generating Sobolev spaces, etc. We believe that our results can be similarly generalized to many other kinds of kernels, including non-stationary ones. However, this would require some additional technical assumptions and modifications of our proofs. We leave this as an open problem for future research.\n\nHowever, there do exist some \"better\" distance metrics that could potentially improve the performance of our method. We are currently working on this direction, but we cannot share the details publicly at this moment. Please understand. To motivate you further, we can provide some references that explore some of these ideas in different settings. For example, Zhu et al. (2020) suggest to aggregate marginal sample HSIC as the test statistic instead of using HSIC over the whole features. Chakraborty and Zhang (2021) propose a new distance for Euclidean space that is capable of detecting marginal nonlinear dependences in high dimensions. \n\nWe hope this answers your question. \n\nComment 2: Independence is particularly important when it is conditional at which point it gives us Bayesian networks; Do any of these results survive for conditional independence? The authors mention such applications (l34-l37) but I believe thereafter discard them. Can we apply these results to conditional independence testing, i.e. inferring graphical models?  \n\nResponse 2: Thank you for inspiring us. We believe that the results summarized in this work can be applied to conditional independence testing methods such as the conditional distance correlation (Wang et al., 2015) in high dimensions. However, this would require some extra effort to prove them rigorously. Therefore, we leave this as an open problem for future research.\n\nReferences\n\nChakraborty, S. and Zhang, X. (2021). A new framework for distance and kernel-based metrics in high dimensions. Electronic Journal of Statistics, 15(2):5455–5522.\n\nWang, X., Pan, W., Hu, W., Tian, Y., and Zhang, H. (2015). Conditional distance correlation. Journal of the American Statistical Association, 110(512):1726–1734.\n\nZhu, C., Zhang, X., Yao, S., and Shao, X. (2020). Distance-based and rkhs-based dependence metrics in high dimension. The Annals of Statistics, 48(6):3366–3394.\n"}, {"review_id": "rSLWZNRcYx", "reviewer": "Reviewer_Uk8i", "summary": "The paper provides insights into the properties of HSIC in high dimensions, more specifically the rate at which sample size must grow in order to detect non-linear correlations if data is high dimensional. The results are categorized based on scenarios where either one or both variables have a \"growing\" dimension, and they express various types of nonlinearity using conditional expected values of higher orders.\n\nA great summary of the results can be found in lines 61 to 64 of the paper.\n", "questions": "Is there related work for MMD? Seems like similar results should/could hold for MMD. ", "limitations": "yes", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper effectively presents the main results and is written with clarity. In my opinion, the results are  significant within the sub-area of independence testing. It has been recognized  for some time that the power of HISC diminishes in high dimensions, but as far as I know, no one has provided conditions on the sample size, dimension and degree of non-linearity for the test to detect signal. Formulation of the results in the language of conditional expected values of higher moments yields insightful and concise characterization of the limitations of HSIC in high dimensions.\n\n\n", "weaknesses": "I'm happy to see empirical study (section 5.3) in a theoretical paper. Having said that, I would be surprised if a linear test would not reject the null  (pool the returns per sector and run a correlation based test).  \n\nEdit: I read the response, and this still doesn't fully make sense to me. I would have chosen a dataset with a non-linear dependence only  and shown HSIC failing. However, ultimately, this does not detract from the quality of the findings and I stand by 7 (8 makes sense too). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for your valuable feedback and constructive suggestions.\n\nComment 1: I'm happy to see empirical study (section 5.3) in a theoretical paper. Having said that, I would be surprised if a linear test would not reject the null (pool the returns per sector and run a correlation based test).\n\nResponse 1: Indeed, in this example, the monthly mean stock prices of energy companies are linearly dependent with the raw material companies without a doubt. We use this example to confirm the assertions we made in Theorem 3. In particular, according to the second part of Theorem 3 and the discussions at the end of Section 4, the HSIC can only have nontrivial power if $p^{(s_1-1)\\kappa_x}q^{(s_2-1)\\kappa_y}=o(n)$. Because in this data set, both p and q are much larger than n, this condition is only satisfied if $s_1=s_2=1$, which corresponds to the covariance between x and y. This together with the fact that the test based on HSIC rejected the null hypothesis, we conclude that there exists a linear dependence relationship between x and y. This type of linear dependence is also confirmed by the RV coefficient as well as the $R^2$s we computed for Denison Mines Corp. and Energy Fuels Inc., and Uranium Energy Corp. and Energy Fuels Inc.. We will emphasize the motivations of this analysis more precisely in the revised version. Thank you for your comment. Please let us know if you have any further questions.\n\nComment 2: Is there related work for MMD? Seems like similar results should/could hold for MMD. \n\nResponse 2: Our results focus on the dependence measures between two random vectors, which can be similarly adapted to the two sample test context using MMD. In fact, our results can shed light on the performance of MMD in high dimensions, as we can translate the two sample problems into independence problems. To illustrate this idea intuitively, we define two new random variables U and V as follows: V is either identical to X or Y, and U is set to be 1 if V=X and 0 if V=Y. Then, $f_X(t)=f_Y(t)$ is equivalent to $f_{V\\mid U=1}(t) = f_{V\\mid U=0}(t)$. This means that U and V are independent, since the conditional distribution of V given U does not depend on the value of U. Therefore, testing whether X and Y have the same distribution is equivalent to testing whether U and V are independent. Note that when X and Y are high-dimensional covariates, V is also high-dimensional and U is univariate. Hence, we can apply the first part of Theorem 3 to understand the behavior of MMD in high dimensions.\n"}, {"review_id": "PKIxXGGxqt", "reviewer": "Reviewer_TY3X", "summary": "The paper investigate the performance of HSIC to test the independence of two random vectors. The focus is on high but not ultra high dimensional scenarios where the theory is lacking.  More specifically, the paper presents convergence rates for HSIC as the dimensions grow at different rates, and demonstrates how HSIC’s capacity to measure nonlinear dependence evolves as the dimensions increase.  \nThe paper also shows that the rescaled HSIC converges in distribution to a standard normal distribution under the null hypothesis and provides the conditions needed to have nontrivial power in high dimensions. The theory is validated by simulations and real-world data involving stock prices from the energy sector and raw material sector in the US stock market.\n\n\n\n", "questions": "How different is the theory between yours and the 2021 paper by Gao et al. in the Annals of Statistics?  What are the differences in technical tools and level of difficulties? ", "limitations": "Some constraints are imposed in the theory but they are fairly mild and reasonable. \n\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a solid paper with strong theory and convincing numerical support. A key advantage of Theorem 1, which provides the asymptotic distribution (with rates) of the HISC test statistic under the null hypothesis, is that it alleviates the use of permutation test to decide critical values, a requirement commonly observed in other tests of independence.\n\nThe phase transition of the convergence rates in Theorem 3 is illuminating. \n", "weaknesses": "None.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for taking the time to read our paper and for your valuable feedback and constructive suggestions.\n\nComment 1: How different is the theory between yours and the 2021 paper by Gao et al. in the Annals of Statistics? What are the differences in technical tools and level of difficulties?\n\nResponse 1: We appreciate your interest in our paper and the work of Gao et al. (2021). Indeed, our first results are parallel to theirs. In particular, we prove that the rescaled HSIC converges in distribution to a standard normal distribution under the null hypothesis. We also derive a general condition for the HSIC based tests to have power asymptotically approaching one. However, the main difference between our theory and theirs is that we focus on the HSIC, which includes the distance correlation as a special case.\n\nIn contrast to their paper, our paper provides a much more extensive analysis of HSIC. We demonstrate that HSIC can detect different kinds of dependences that depend on the dimensionality and sample orders, which is a novel and important insight that has not been explored in the literature before. Furthermore, our proof technique in this part is completely different from theirs, which indicates a higher level of difficulty. This is the main contribution of our paper and sheds light on the performance of HSIC in high dimensions.\n\nReferences\n\nGao, L., Fan, Y., Lv, J., and Shao, Q.-M. (2021). Asymptotic distributions of high- dimensional distance correlation inference. The Annals of Statistics, 49(4):1999–2020."}]

Author rebuttal:

Meta-review: The submission is centered around measuring and testing statistical independence between two random variables taking values in finite-dimensional Euclidean spaces, using the (normalized) Hilbert-Schmidt independence criterion (HSIC), in the high-dimensional setting. The authors establish the asymptotic normality of HSIC  under the null (Theorem 1) and its non-trivial power (Theorem 2) in case of growing dimension, and provide further insights on the influence of dimension on HSIC (Theorem 3). The theoretical results are accompanied with numerical demonstrations on both synthetic and real-world (financial) benchmarks.

Kernel techniques are at the forefront of machine learning and statistics, with leading role in the estimation of statistical independence measures such as HSIC. The reviewers agreed upon the novelty and the clear interest of the submission for the NeurIPS community, both from theoretical and from empirical perspective.
Decision: Accept (poster)