Title: Efficiently Learning Significant Fourier Feature Pairs for Statistical Independence Testing
Venue: NeurIPS 2024
Accepted: yes
Keywords: independence test

learnable Fourier feature
Abstract: We propose a novel method to efficiently learn significant Fourier feature pairs for maximizing the power of Hilbert-Schmidt Independence Criterion~(HSIC) based independence tests. We first reinterpret HSIC in the frequency domain, which reveals its limited discriminative power due to the inability to adapt to specific frequency-domain features under the current inflexible configuration. To remedy this shortcoming, we introduce a module of learnable Fourier features, thereby developing a new criterion. We then derive a finite sample estimate of the test power by modeling the behavior of the criterion, thus formulating an optimization objective for significant Fourier feature pairs learning. We show that this optimization objective can be computed in linear time (with respect to the sample size $n$), which ensures fast independence tests. We also prove the convergence property of the optimization objective and establish the consistency of the independence tests. Extensive empirical evaluation on both synthetic and real datasets validates our method's superiority in effectiveness and efficiency, particularly in handling high-dimensional data and dealing with large-scale scenarios.

Efficiently Learning Significant Fourier Feature Pairs for Statistical Independence Testing

Abstract

We propose a novel method to efficiently learn significant Fourier feature pairs for maximizing the power of Hilbert-Schmidt Independence Criterion (HSIC) based independence tests. We first reinterpret HSIC in the frequency domain, which reveals its limited discriminative power due to the inability to adapt to specific frequency-domain features under the current inflexible configuration. To remedy this shortcoming, we introduce a module of learnable Fourier features, thereby developing a new criterion. We then derive a finite sample estimate of the test power by modeling the behavior of the criterion, thus formulating an optimization objective for significant Fourier feature pairs learning. We show that this optimization objective can be computed in linear time (with respect to the sample size n ), which ensures fast independence tests. We also prove the convergence property of the optimization objective and establish the consistency of the independence tests. Extensive empirical evaluation on both synthetic and real datasets validates our method's superiority in effectiveness and efficiency, particularly in handling high-dimensional data and dealing with large-scale scenarios.

1 Introduction

Testing for independence is a crucial and challenging task in machine learning and statistics, with wide-range applications in causal inference [16, 31], feature selection [6] and deep learning [23, 42]. Its primary objective is to determine whether two random variables, X and Y are independent, based on the observations of the underlying joint distribution P XY . While traditional independence tests, such as Pearson's correlation coefficient [9] and Kendall's τ , can only detect monotonic relationships between low-dimensional variables, more modern tests [26, 43, 7, 25, 27, 35, 19, 20] aim to deal with complex non-linear interactions in much more challenging higher-dimensional space [45, 29].

One class of nonlinear dependence measures [3, 15] aims to capture distributional characteristics using kernel embeddings [13], primarily derived from the cross-covariance operators in the reproducing kernel Hilbert space (RKHS). Among them, Hilbert-Schmidt Independence Criterion (HSIC) [14] is the most popular one. It utilizes the squared Hilbert-Schmidt norm to detect dependence and exhibits outstanding performance across various data contexts by choosing suitable kernels. On the other hand, some other fundamental nonlinear dependence measures employ characteristic functions

to detect the smoothed discrepancy between the joint distribution and the product of marginals. By employing appropriate characteristic functions, the statistic [39, 40] computes the covariance between distances of variable pairs. It has been demonstrated that these distance-based methods are equivalent to HSIC with specific kernels [33]. However, all these measures suffer from the drawback of requiring quadratic time (w.r.t. the sample size n ) to compute the feature covariance and necessitating fixed kernel or distance functions, rendering them impractical on large-scale datasets due to the unaffordable time cost and lacking flexibility in handling complex scenarios.

To address these challenges, a multitude of works grounded on these measures have emerged. Upon HSIC, [44] proposes some linear-time tests including a block-averaged statistic, a statistic with Nyström approximation, and one with finite-dimensional feature mappings using random Fourier features (RFF) [28]. For convenience, these tests are referred to as BHSIC, NyHSIC, and FHSIC, respectively. FHSIC and NyHSIC are observed to have a considerable advantage over BHSIC. However, a remaining drawback of these methods is that the features are not learnable. Therefore, these methods lack enough adaptability to complex settings, thus leading to performance degradation.

In addition to time efficiency, another research direction [1, 30] aims to make independence tests adaptive to better capturing distributional distinctions. These methods either select/combine appropriate kernels from a predefined set or learn parameterized kernels. Nonetheless, their criteria still inherit the quadratic time complexity of HSIC, thus cannot be readily applied to large-scale data.

Furthermore, some approaches [17, 32] try to address both challenges simultaneously. For instance, HSICAgg [32] suggests combining several kernels from a predefined set (e.g. kernels with different preset bandwidths) and aggregating the test results for improving performance. Additionally, an incomplete U -statistic of HSIC is proposed to ensure computational efficiency. Nevertheless, selecting from a predefined set of kernels imposes limitations on flexibility, and in cases where scaling optimization is required on each dimension, the number of kernel pairs escalates exponentially. Also, NFSIC [18] proposes to combine a time-efficient technique called analytic kernel embeddings [8, 17] and learn the important local distributional features. However, its learning objective is merely a lower bound of test power and demands a substantial number of samples to ensure accuracy.

In this paper, we propose a novel test method that flexibly learns distributional features while maintaining high efficiency. We first reinterpret HSIC from a frequency-domain perspective, then we point out its potential shortcomings with an elaborate example and indicate corresponding improvement directions. Finally an central optimization objective is derived by directly modeling test power, which can be computed in linear time while maximizing the test performance. Comparing with [30] that also addresses the kernel learning problem in independence testing with a time/space complexity of O ( n 2 ) , our criteria for learning are designed to have a complexity of O ( n ) for both space and time. Consequently, the whole test framework can efficiently handle large-scale data.

Contributions. In summary, the contributions of the work are as follows: 1) We propose a novel approach that efficiently learns significant Fourier feature pairs for maximizing the power of HSICbased independence tests. 2) We design an optimization objective that can be computed in linear time, which is derived by directly modeling test power. 3) We theoretically establish the non-asymptotic convergence property of the optimization objective and demonstrate the consistency of our method. 4) We conduct extensive experiments on both synthetic and real data, showcasing its superiority in effectiveness and efficiency in handling high-dimensional data (e.g. image data) and addressing large-scale scenarios.

Outline. The rest of the paper is organized as follows: Sec. 2 reviews HSIC-based statistical independence tests. Sec. 3 reinterprets HSIC from a frequency-domain perspective, and explain its potential shortcomings with an elaborate example and indicate corresponding improvement directions. Sec. 4 designs an optimization objective by directly modeling test power, which can be computed in linear time. Sec. 5 presents the theoretical analysis and Sec. 6 evaluates the performance of the proposed method on synthetic and real dataset. We conclude the paper in Sec. 7.

2 Preliminaries and Notations

We begin by introducing notions and reviewing the hypothesis testing framework for independence tests. Let X × Y be separable metric space, typically R d x × R d y . P XY denotes a Borel probability measure defined on X × Y , while P X and P Y denote the respective marginal distributions. Given n

̸

independent and identically distributed (i.i.d) samples Z := ( X,Y ) = { ( x i , y i ) } n i =1 with distribution P XY , we aim to test whether X,Y are independent (i.e., X ⊥ ⊥ Y ). This corresponds to a hypothesis testing problem formulated as H 0 : P XY = P X P Y versus H 1 : P XY = P X P Y .

The testing procedure is as follows: First, define the statistic ρ and calculate its estimated value using the samples. Then, choose a significance level α (typically set to 0 . 05 ), which represents the probability that the sampling of ρ under H 0 is at least as extreme as the observed value. Finally, the null hypothesis H 0 is rejected if the p -value is not greater than α .

Two types of errors may occur in this procedure. Type I error occurs when H 0 is falsely rejected, while Type II error happens when H 0 is incorrect but not rejected. A good test [43] needs to control Type I error within α while maximizing the testing power ( 1 -Type II error rate).

For independence tests, a commonly used statistic is HSIC, defined as follows:

Definition 1. [14]. Let F be an RKHS with kernel k : X × X ↦→ R and G be a second RKHS on Y with kernel l : Y × Y ↦→ R , the HSIC between X and Y , denoted as HSIC ( X,Y ) is defined as

where ( X ′ , Y ′ ) is a independent copy of ( X,Y ) . An estimator of HSIC ( X,Y ) is given by

where k ij := k ( x i , x j ) , l ij := l ( y i , y j ) are the entries of the n × n kernel matrices K , L respectively, H = I -1 n 11 T is the centering matrix and 1 is a vector of ones.

3 Revisiting HSIC from Frequency Domain Perspective

We denote F as the Fourier transform, and F -1 as its inverse. When the kernels k, l are translationinvariant, i.e., there exist functions ψ, ψ k , ψ l such that for all ( x, x ′ ) ∈ X × X and ( y, y ′ ) ∈ Y × Y ,

Then, according to the results of [36, Corollary 4], the HSIC with function ψ can be formulated as

where ω = ( ω x , ω y ) ∈ R d x × R d y , ω x , ω y are the frequencies of X and Y respectively, and

are the characteristic functions of P XY and P X P Y , respectively. Intuitively, Eq. (4) means that HSIC can be understood as the difference between the joint distribution and the product of the marginal distributions in the frequency domain, with different weights ( F -1 ψ )( ω ) being attached to different frequencies, which are determined by the kernel function. When F -1 ψ is almost everywhere nonzero, it can be shown that the kernel is characteristic [36, 10]. The characteristic condition ensures that the criterion is discriminative for discrepancies at almost all frequencies. However, with inappropriate choices of F -1 ψ , the differences may not be significant enough. We explain this with an example:

Example. Consider the Sinusoid model that X × Y := [ -π, π ] 2 and ( X,Y ) ∼ p xy ( x, y ) ∝ 1 + sin( ω 0 x ) sin( ω 0 y ) , where p xy is the probability density function and ω 0 is a positive integer. Combining Eq. (5), we can calculate that ϕ P X P Y ( ω ) = δ ( ω x ) δ ( ω y ) and ϕ P XY ( ω ) = δ ( ω x ) δ ( ω y ) + [ δ ( ω x + ω 0 ) + δ ( ω x -ω 0 )][ δ ( ω y + ω 0 ) + δ ( ω y -ω 0 )] , where δ is the Dirac delta function, thus the difference between them only relies on the frequency ω 0 . Whenthe Gaussian kernels with width √ 2 λ x and √ 2 λ y are used, i.e., k ( x, x ′ ) = exp( -∥ x -x ′ ∥ 2 2 / (4 λ 2 x )) , l ( y, y ′ ) = exp( -∥ y -y ′ ∥ 2 2 / (4 λ 2 y )) , then the inverse Fourier transform of ψ is ( F -1 ψ )( ω x , ω y ) = π -1 λ x λ y exp( -( λ 2 x ω 2 x + λ 2 y ω 2 y )) . Hence HSIC ( X,Y ) = 4 π -1 λ x λ y exp( -( λ 2 x + λ 2 y ) ω 2 0 ) whose maximum is taken at λ ∗ x = λ ∗ y = 1 / ( √ 2 ω 0 ) , indicating that the widths need to be adjusted to focus on some specific frequencies. If the common setting [14, 44] is adapted, which uses mid-widths (i.e., the median distance does

not change with ω 0 since the marginal distributions do not change with ω 0 ), then the criterion will exponentially decline to 0 as ω 0 increases. In contrast, the criterion using the adaptive optimization width (1 /ω 0 , 1 /ω 0 ) decreases at a rate of O ( ω -2 0 ) , which is a considerable improvement.

This example illustrates the loss of the discriminatory power of the criterion when an inappropriate F -1 ψ is chosen. The discriminatory power of the criterion heavily impacts the sample size required for the test to obtain significant results in practice, and existing inflexible configurations may lead to inadequate test power in the presence of reasonably large sample sizes. Consequently, it is important to design learnable F -1 ψ . To this end, we subsequently design a learnable objective and let it be optimized in a data-driven manner. Before this, we provide an approach to make the criterion be computed efficiently. This can be achieved by sampling in the frequency domain. Formally, a finite-dimensional approximation in the frequency domain of the integral in Eq. (4) is given as follows:

where { ω x ; i } D x i =1 , { ω y ; j } D y j =1 are sampled independently with the measure F -1 ψ k , F -1 ψ l , respectively. Note that F -1 ψ is a product measure, i.e., F -1 ψ = ( F -1 ψ k ) ⊗ ( F -1 ψ l ) . This type of approximation is also called random Fourier features (RFF) [28] that had been applied to various kernel algorithms. We will incorporate this technique to efficiently perform computation later.

4 Learning Significant Fourier Feature Pairs

4.1 HSIC with Learnable Fourier Feature Pairs

To design F -1 ψ , we need to make sure that supp ( F -1 ψ ) = R d x × R d y to meet the characteristic condition and that its integral over the full space is 1 to ensure it is a probability measure. Also, for practical utility, F -1 ψ should embody a familiar probability density function, facilitating sampling procedures. Fortunately, a versatile array of options emerges through the judicious selection of kernels 2 with adjustable parameters. Take kernel k as an example, some commonly used kernels are listed in Tab. 1, and their inverse Fourier transforms are listed simultaneously. Additionally, to

be able to apply gradient-based optimization techniques, we invoke a method that disentangle the sampled objects and the learnable parameters. Specifically, we leverage a variable transform T θ k (a bijection function parameterized with θ k ) to convert the probability measure F -1 ψ k into a simple distribution (e.g. a standard Gaussian distribution) p k ( ω ) . Simultaneously, we relocate the learnable component onto X . Consequently, we can focus on learning parameterized transformations T θ k and simplifying the computation by enabling sampling directly from p k ( ω ) .

Remark. The above scheme provides a broader form for designing. The mapping T θ k can be viewed as a feature extractor, which makes it possible to flexibly combine models (e.g., neural network) thus incorporating deep kernel [24] into the framework. Also, it should be noted that the single kernel example can also be extended to multi-kernel setting [11] by executing the procedure for each kernel.

Next, we obtain the learnable independence criterion and utilize the sampling technique as in Eq. (6) to compute efficiently. Note that for simplicity, we take the same value for both D x and D y in Eq. (6) by default. By the definition, the kernel function can be expressed as

By applying the frequency sampling technique, we obtain the approximation as

where { ω k ; j } D/ 2 j =1 are sampled independently with distribution p k ( ω ) and the last equation is because the kernel function is real. To get a more computationally tractable form, we define

called learnable RFF of k then Eq. (8) becomes ψ ( ω ) k ( T θ k x -T θ k x ′ ) = Λ k ( x )Λ k ( x ′ ) T . The expression with a similar form is also given in [44], with the difference that we have added learnable parts. For Y , we define the corresponding symbols by substituting k for l and x for y . Also, for convenience, we default to keeping Y and X the same number of samples D from here on. Then the HSIC with learnable RFF pairs can be obtained by replacing k, l in Eq. (1) to ψ ( ω ) k , ψ ( ω ) l . Also, the corresponding estimator with sample Z can be obtained by replacing K , L in Eq. (2) to the matrices Λ X Λ T X , Λ Y Λ T Y , where Λ X := [Λ k ( x 1 ); ... ; Λ k ( x n )] n × D and so as define for Λ Y . As a result,

where Λ Xc := HΛ X , Λ Y c := HΛ Y . The time complexity is analyzed as follows. Since the computation of the mapping T θ k x depends on the specific design, here we default to analyzing the kernel case shown in Tab. 1. In this case, computing Λ X , Λ Y requires O ( nD ( d x + d y ) ) time. Then calculate Λ Xc , Λ Y c cost O ( nD ) . After that, calculate HSIC ω ( Z ) cost O ( nD 2 ) . Hence, the overall time complexity is O ( nD ( d x + d y + D ) ) , i.e. the running time is linear with n .

4.2 Linear-time Optimization Objective

Next, we model the behavior of HSIC ω ( Z ) to obtain an optimization objective for maximizing the power of the test. By utilizing the property that HSIC ω ( Z ) is a V-statistic, we can extend the results [14, Theorem 1, 2] for HSIC ω ( Z ) , as shown in the following proposition with the proof given in the Appendix. To simplify, we denote ( x i , y i ) as z i to represent the i -th sample and denote ψ ( ω ) k ( T θ k x t -T θ k x u ) as k ( ω ) tu and ψ ( ω ) l ( T θ l y t -T θ l y u ) as l ( ω ) tu .

Proposition 1 (Asymptotics) . Let h ( ω ) ijqr := 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k ( ω ) tu l ( ω ) tu + k ( ω ) tu l ( ω ) vw -2 k ( ω ) uv l ( ω ) tv , where the sum represents all ordered quadruples ( t, u, v, w ) drawn without replacement from ( i, j, q, r ) . Then, Under the null hypothesis H 0 , HSIC ω ( Z ) coverages in distribution to

where χ 2 11 , χ 2 12 , ... are independent χ 2 1 variates and λ l is the solution to the eigenvalue problem as in the right of Eq. (11). Also, under the alternative H 1 , HSIC ω ( Z ) converges in distribution as

According to Proposition 1, the power of the test with HSIC ω can be formulated by

where Φ is the standard normal CDF and r ω is the threshold, i.e. (1 -α ) -quantile of distribution given in Eq. (11) that exactly controls Type I error rate to the nominal level α . Hence, to maximize the power of the test, a natural criterion is [ n E Z HSIC ω ( Z ) -r ω ] / ( √ nσ ω ) . Next, we provide its estimation which can be computed in linear time.

We first consider obtaining the estimator of the numerator part. For the term E Z HSIC ω ( Z ) , we can estimate it with HSIC ω ( Z ) as in Eq. (10). The estimation of the threshold r ω poses a challenge, primarily stemming from the lack of an explicit expression for the distribution of the infinite sum of chi-square variables. One avenue to address this challenge involves employing the permutation method [2, 38] to simulate the distribution under H 0 . However, this method necessitates a significant number of shuffles to accurately approximate the distribution. Furthermore, even with the implementation of parallel schemes, it incurs memory costs proportional to the number of permutations, rendering it impractical for resource-constrained scenarios. Here, we adopt a lightweight approach in practice, leveraging the gamma approximation as proposed by [14]. A gamma distribution is uniquely determined by its first and second-order moments. For these two moments, we present their corresponding linear-time estimators in Theorem 1. As a result, we can obtain the (1 -α ) -quantile of the gamma distribution, denoted as ̂ c α , with estimated parameters γ := E 2 0 / V 0 , β := V 0 / E 0 in linear time. Formally, with the term E 0 and V 0 defined in Theorem 1, ̂ c α is calculated by

where Γ( · ) is the gamma function. By combining the way to estimate the gradients of ̂ c α [30], we enable it for gradient-based optimization with automatic differentiation framework. As a result, we obtain a linear-time differentiable estimator of the numerator part.

Theorem 1 (Linear-Time Estimators) . Under H 0 , the estimators of mean and variance with bias of O ( n -1 ) to E Z [ n HSIC ω ( Z )] and Var Z [ n HSIC ω ( Z )] , denote as E 0 and V 0 , respectively, are given by

where () · 2 is the entry-wise matrix power. Both E 0 and V 0 can be calculated in O ( nD 2 ) time.

For the remain term σ ω , we estimate it with ̂ σ ω that ̂ σ 2 ω := 16 [ 1 n ∑ i ( 1 n 3 ∑ j,q,r h ( ω ) ijqr ) 2 -HSIC 2 ω ( Z ) ] . To calculate ∑ j,q,r h ( ω ) ijqr , the straightforward way is to compute each item h ( ω ) ijqr , which requires total O ( n 4 ) of computation. Here we provide a way to enable it to be calculated in linear time by obtaining a matrix expression. The main result is given by

where the definition of variables A to F with the calculation cost are given in the Fig. 1 and the derivation of Eq. (16) is given in the Appendix. By checking the complexity of the remaining matrix operations in Eq. (16), all the elements with index i can be calculated in O ( nD 2 ) . Combining the results obtained before that HSIC ω ( Z ) can also be calculated in O ( nD 2 ) , thus calculating the term ̂ σ ω cost O ( nD 2 ) time. As a result, we obtain the overall linear-time optimization objective J := [ HSIC ω ( Z ) -̂ c α /n ] / ̂ σ ω , which is a clear contrast to the existing quadratic-time schemes [30].

Time Complexity:

4.3 The Overall Learning Framework

After obtaining the differentiable optimization objective J , we can perform the training process end-to-end. In this process, the overfitting issues may happen especially with insufficient samples, which could influence both Type I and II errors. If we use the same sample for testing, the Type I errors may be uncontrollable [22] when the overfitting issues happen. To address this, we adopt the split scheme as in [24, 18] to allow our tests to maintain validity (controllable Type I errors). The split ratio is set to 0 . 5 to facilitate the balance between the two. Apart from controlling Type I errors, we still want to mitigate overfitting issues as much as possible in order to generalize the optimized

Fourier feature pairs on test data thus improving the power of our tests. To this end, we select smooth function classes to control the model complexity (e.g., as measured by the VC dimension), specifically in this paper, we consider the two classes in Tab. 1 and implement them for experiments later. One is the choice of Gaussian classes that optimize the global scale, and the other we consider Mahalanobis classes and set Σ to be diag ( σ 1 , ..., σ d ) for optimization, which corresponds to optimizing the scale in each dimension (which allows to capture high-frequency signals such as the edge in the image). These smooth choices also bring the advantage of interpretability [30] and it is experimentally proven that this simple choice is already able to handle most of the cases in different settings.

More discussion about split strategy. Currently, there are two major classes of approaches for adaptive independence tests. One involves selecting kernels from a finite/countable set (discrete scenario) and the other involves performing kernel parameter searches in a continuous space (continuous scenario). For the former case, some methods [22, 32] control Type I errors by applying techniques from the selective inference literature without data splitting. However, these methods cannot be directly applied to a continuous scenario due to the uncountable set of kernels involved. To the best of our knowledge, both our scheme and existing methods [24, 18] rely on data splitting for the continuous case. Designing methods to control Type I errors in the continuous case without sample splitting remains a challenging and significant problem for future research.

Algorithm. Our algorithm is outlined in Alg. 1. As a pre-processing step, we split the data into the training data Z tr and the testing data Z te (Line 1). The test contains two phases: 1) We learn the Fourier feature pairs with Adam [21] optimizer using full batches on Z tr (Lines 2 -7 ). 2) With the learned Fourier feature pairs, we calculate the test statistic and threshold (Lines 8 -10 ) to determine the independence (Lines 11 ) on Z te . The overall time complexity is O ( TnD ( d x + d y + D ) ) and the space cost is O ( n ( d x + d y + D ) ) for storing the data as well as the Fourier feature pairs.

5 Theoretical Results

We first give the uniform bound results over a ball in parameter space which guarantees the convergence of our optimizing objective thus ensuring its effectiveness in modeling test power.

Theorem 2 (Uniform Bound) . Let θ k , θ l parameterize T θ k , T θ l in Banach spaces of dimension d k , d l . And T θ k , T θ l are Lipschitz to the parameters θ k , θ l with the non-negative constant L k , L l , respectively. Let Θ c be a set of ( θ k , θ l ) for which σ ω ≥ c > 0 with a positive constant c and ∥ θ k ∥ ≤ R θ k , ∥ θ l ∥ ≤ R θ l . Let r denote the threshold, i.e., (1 -α ) -quantile for the distribution in Eq. (11) and r ( n ) be the threshold with sample size n . Let { ( ω k ; j , ω l ; j ) } D/ 2 j =1 be the samplings of frequency with the sampling number D . Also, we define R ω k := sup j ∥ ω k ; j ∥ , R ω l := sup j ∥ ω l ; j ∥ , d s := max { d k , d l } and ξ ω := HSIC ω ( Z ) . Then with probability at least 1 -δ , we have

Next, we show the consistency of the tests, i.e. the power of the test tends to 1 as the sample size increases. Let the U-statistic of HSIC ω ( Z te ) be HSIC ( u ) ω ( Z te ) , then we have the following results.

Theorem 3 (Consistency) . Let θ ∗ k , θ ∗ l be the parameters after learning, Z te be the test samples of size m , when E Z HSIC ( u ) ω ( Z te ) > 0 , then the probability of the Type II error

Let the mapping functions with learned parameters θ ∗ k , θ ∗ l be T θ ∗ k , T θ ∗ l , and the corresponding range space be compact subsets of R d T x , R d T y , respectively. Also, the diameters of two range spaces are denoted by diam ( T θ ∗ k ) , diam ( T θ ∗ l ) , respectively. Let { ( ω k ; j , ω l ; j ) } D/ 2 j =1 be the frequency samplings with their second moment denoted by σ 2 ω k := E p k ( ω ) [ ω T k ; j ω k ; j ] , σ 2 ω l := E p l ( ω ) [ ω T l ; j ω l ; j ] . Additionally, we denote ξ u := HSIC ( X,Y ) , then under H 1 , we have E Z HSIC ( u ) ω ( Z te ) > 0 with any constant probability when D = Ω ( d T x + d T y ξ 2 u log σ ω k diam ( T θ ∗ k )+ σ ω l diam ( T θ ∗ l ) ξ u ) .

This result can be understood in two parts. The first one is about consistency, i.e., the Type II error rate tends to 0 at the rate of m -1 / 2 when condition E Z HSIC ( u ) ω ( Z te ) > 0 holds. The second part provides the condition when E Z HSIC ( u ) ω ( Z te ) > 0 holds, which requires sufficiently many frequency samplings. The theorem shows that the large value of the criterion HSIC ( X,Y ) helps to reduce the required D . According to the results discussed in Sec. 3, there is an improvement in the criterion by finding the more significant features and thus helps to reduce the required D . To summarize, the significant features further help to guarantee the consistency of the test under the efficient requirements (smaller D ). All proofs as well as additional results are given in the Appendix.

6 Performance Evaluation

We compare the following tests: distance-based statistic dCor [39], the original HSIC QHSIC [14], the copula-based method RDC [26], the three variants of HSIC NyHSIC [44], FHSIC [44], BHSIC [44] and HSICAgg [32], NFSIC [18] as introduced in Sec. 1. Among them, dCor and QHSIC are O ( n 2 ) tests. RDC is calculated in O ( n log n ) time and the rest are O ( n ) tests. A detailed description of the comparing methods is given in the Appendix. For our methods, We provide two variants as mentioned in Sec. 4.3. We name the Gaussian class case as LFHSIC-G , and name the Mahalanobis class case (and set Σ as a diagonal matrix) LFHSIC-M . Additionally, for the comparative methods [30] that are relevant to us, due to their high time overhead and therefore inability to handle some settings of evaluation, we separately provide a comparison with our method under certain feasible experimental settings, the results are given in the Appendix.

Experimental setup. The significance level α is set to 0 . 05 . We use Gaussian kernels for both X and Y in all kernel-based methods. And QHSIC, RDC, NyHSIC, FHSIC, BHSIC are all with the kernel width being set to the Euclidean distance median of the samples. The number of random features D for FHSIC, LFHSIC-G/M, the number of induced variables for NyHSIC, the block size for BHSIC as well as the number of sub-diagonals R for HSICAgg are all kept consistent as recommended in [44, 32] for fair evaluation. Parameter settings for the rest of the methods follow the defaults in the code. More details of the setups are given in the Appendix.

Evaluation protocol. We evaluate on four synthetic datasets [18, 30] and two real datasets [44, 30]. Synthetic datasets consist of Sine Dependency (SD), Sinusoid (Sin), Gaussian Sign (GSign), and independent subspace analysis (ISA) dataset [14]. On real data, we introduce high-dimensional image data and another music dataset to evaluate the capability of all methods in different data scenarios. Unless otherwise specified, we perform 100 repeated randomized experiments and report the average result of test power as default. More details of the generating process of each dataset and the details of the evaluation (including running time) are provided in the Appendix.

6.1 Results on Synthetic Datasets

Settings of SD, Sin, and GSign Dataset . The Sin data corresponds to the example in Sec. 3 that requires the method to focus on differences in specific frequencies. In SD, Y is dependent solely on the first two dimensions of X . In contrast, in GSign, Y is independent of any proper subset of X

SD

Sin

GSign

0.00

0.05

0.10

0.15

0.20

0.25

Type I error rate( ↓ )

Average Results

2000

2500

3000

3500

4000

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Sine Dependency (SD)

1000

1500

2000

2500

3000

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Sinusoid (Sin)

2000

3000

4000

5000

6000

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

SD

Sin

GSign

Datasets

0.00

0.05

0.10

0.15

0.20

0.25

Type I error rate( ↓ )

2000

2500

3000

3500

4000

Sample size n

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

1000

1500

2000

2500

3000

Sample size n

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

dCor

QHSIC

RDC

NyHSIC

FHSIC

BHSIC

HSICAgg

NFSIC

LFHSIC-G

LFHSIC-M

2000

3000

4000

5000

6000

Sample size n

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

0.00

0.25

0.50

0.75

1.00

Angle(×

π

/4)

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Samp:10000, Dim:16, D:300

0.00

0.25

0.50

0.75

1.00

Angle(×

π

/4)

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Samp:60000, Dim:32, D:300

0.00

0.25

0.50

0.75

1.00

Angle(×

π

/4)

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Samp:10000, Dim:16, D:500

RDC

NyHSIC

BHSIC

FHSIC

HSICAgg

NFSIC

LFHSIC-G

LFHSIC-M

0.00

0.25

0.50

0.75

1.00

Angle(×

π

/4)

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Samp:60000, Dim:32, D:500

but dependent on X as a whole. Therefore, it requires the method to learn important local/global features based on the characteristics of the data to improve the test power. For SD and GSign, we set the dimension of X as 4 and 5 , respectively, and the dimension of Y is 1 for both. For Sin, we set the frequency parameter ω = 5 . For calculating the Type I error rate, we evaluate using samples ( n = 2000 ) obtained by permutation for all three datasets.

Performance. The results for D = 100 and D = 500 are shown in Fig. 2. Except for NFSIC and BHSIC, all the other methods succeed in controlling the Type I error rate ≤ 0 . 05 . LFHSIC-M/G, NFHSIC, and HSICAgg perform much better than other methods due to their ability to obtain more appropriate kernels/features for testing. LFHSIC-G/M performs on both settings of D and has a more significant advantage over the others when D is small, implying the optimization objective can still be successfully optimized and the criterion is still powerful under high-speed requirements. In addition, as the sample size increases the test power of LFHSIC-G/M is gradually converging to 1 in both settings, which corroborates the results of Theorem 3.

Settings of ISA Dataset (Large Scale) . We set dimension (of both X,Y ) and sample size as d = 16 , n = 10000 and d = 32 , n = 60000 , then evaluate the average test power with angle parameter θ ∈ [0 , π/ 4] . Note that a larger angle signifies stronger dependency. The quadratic-time methods are not involved in the evaluation due to their inability to handle large-scale settings. For HSICAgg under the challenging setting n = 60000 , d = 32 , the memory space required for parallel implementation leads to memory overflow and hence the results are not given.

Performance. The results for D = 300 and D = 500 are shown in Fig. 3. The results obtained at θ = 0 reflect the Type I error rate. All methods successfully control the Type I error rate ≤ 0 . 05 . LFHSIC-M stably outperforms other methods significantly as the angle increases. Method (LFHSIC-G) that simply optimizes the global bandwidth performs worse as d increases, corroborating the need for more flexible kernel designs for more challenging tasks. Furthermore, comparing the

Type I error

Type II error

0.0

0.2

0.4

0.6

0.8

1.0

Average error rate ( ↓ )

0.05

Results on 3Dshapes

dCor

QHSIC

RDC

NyHSIC

FHSIC

BHSIC

HSICAgg

NFSIC

LFHSIC-G

LFHSIC-M

dCor

QHSIC

RDC

NyHSIC

FHSIC

BHSIC

HSICAgg

NFSIC

LFHSIC-G

LFHSIC-M

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Sample size: 500

dCor

QHSIC

RDC

NyHSIC

FHSIC

BHSIC

HSICAgg

NFSIC

LFHSIC-G

LFHSIC-M

0.0

0.2

0.4

0.6

0.8

1.0

Average test power( ↑ )

Sample size: 1000

results under different settings of D , our method LFHSIC-M performs consistently well and exhibits progressively better performance as D increases.

6.2 Results on Real Data

Settings of Two Real Data . The first real dataset used is a high-dimensional image dataset 3Dshapes as in [30]. In our experiments, we vectorize image X to a vector with dimension 64 × 64 × 3 = 12 , 288 . The sample size is set as 128 . We add standard Gaussian noise N (0 , 1) to the angle Y to make the setting more challenging. The Type I error rate is evaluated by the samples obtained by permutation. Besides, we consider the Million Song Data (MSD) as the second real dataset. The first dimension represents the year of release of each song and is referred to as variable Y . The remaining 90dimensional features (e.g., mean timbre and timbre covariance) constitute variable X . We follow the recommended setting [44], i.e., disturbing each entry of the X with an independent Gaussian noise N (0 , 1000) . For this dataset MSD, in order to fully utilize the data, we randomly select n ∈ { 500 , 1000 } samples as the training set and other n samples from the remaining data 100 times for the evaluation and obtain the average result. The above training and testing processes are repeated 10 times to evaluate the robustness of the optimization scheme.

Performance. The results of two real data with D = 10 are presented in Fig. 4. For the results on 3Dshapes (shown in the left of Fig. 4), all methods except BHSIC and NFSIC control the Type I error well. The linear-time test has relatively lower power compared to the quadratic-time test except for LFHSIC-M, proving that its more significant features obtained in high-dimensional scenarios enable it to achieve outstanding performance even in scenarios with high approximation requirements ( D = 10 ). Similar conclusions can drawn from the MSD dataset (shown in the right of Fig. 4). Additionally, the results for NFSIC and LFHSIC-G/M with different sample sizes indicate increased robustness of the optimization as the sample size increases (reflected in the reduction of variance), and the more flexible design also contributes to this (comparing LFHSIC-M and LFHSIC-G), thus can be more effectively applied to real-world scenarios.

7 Conclusion

In this paper, we propose a novel method to efficiently learn significant Fourier feature pairs for maximizing the power of HSIC-based independence tests. By integrating a learnable Fourier feature module, we improve the flexibility of existing configurations and design a new criterion. The proposed linear-time optimization objective accurately models the power of the test and can be trained end-toend in a data-driven manner, ensuring both effectiveness and efficiency. Both theoretical results and experimental results show the effectiveness of our proposed method. Future work includes further improving the sampling method in the frequency domain.

Acknowledgments and Disclosure of Funding

This work was supported by National Natural Science Foundation (NSFC) (62372116), and National Key Research and Development Program of China (2021YFC3340302 and 2021YFC3300304).

References

Albert, M., Laurent, B., Marrel, A., and Meynaoui, A. (2022). Adaptive test of independence based on hsic measures. The Annals of Statistics , 50(2):858-879.

Arcones, M. A. and Gine, E. (1992). On the bootstrap of u and v statistics. The Annals of Statistics , pages 655-674.

Bach, F. R. and Jordan, M. I. (2002). Kernel independent component analysis. Journal of machine learning research , 3(Jul):1-48.

Bertin-Mahieux, T. (2011). YearPredictionMSD. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C50K61.

Burgess, C. and Kim, H. (2018). 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/.

Camps-Valls, G., Mooij, J., and Scholkopf, B. (2010). Remote sensing feature selection by kernel dependence measures. IEEE Geoscience and Remote Sensing Letters , 7(3):587-591.

Chatterjee, S. (2021). A new coefficient of correlation. Journal of the American Statistical Association , 116(536):2009-2022.

Chwialkowski, K. P., Ramdas, A., Sejdinovic, D., and Gretton, A. (2015). Fast two-sample testing with analytic representations of probability measures. Advances in Neural Information Processing Systems , 28.

Cohen, I., Huang, Y., Chen, J., Benesty, J., Benesty, J., Chen, J., Huang, Y ., and Cohen, I. (2009). Pearson correlation coefficient. Noise reduction in speech processing , pages 1-4.

Fukumizu, K., Gretton, A., Schölkopf, B., and Sriperumbudur, B. K. (2008). Characteristic kernels on groups and semigroups. Advances in neural information processing systems , 21.

Gönen, M. and Alpaydın, E. (2011). Multiple kernel learning algorithms. The Journal of Machine Learning Research , 12:2211-2268.

Gretton, A. (2015). A simpler condition for consistency of a kernel independence test. arXiv preprint arXiv:1501.06103 .

Gretton, A., Borgwardt, K., Rasch, M., Schölkopf, B., and Smola, A. (2006). A kernel method for the two-sample-problem. Advances in neural information processing systems , 19.

Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., and Smola, A. (2007). A kernel statistical test of independence. Advances in neural information processing systems , 20.

Gretton, A., Smola, A., Bousquet, O., Herbrich, R., Belitski, A., Augath, M., Murayama, Y., Pauls, J., Schölkopf, B., and Logothetis, N. (2005). Kernel constrained covariance for dependence measurement. In International Workshop on Artificial Intelligence and Statistics , pages 112-119. PMLR.

Hoyer, P., Janzing, D., Mooij, J. M., Peters, J., and Schölkopf, B. (2008). Nonlinear causal discovery with additive noise models. Advances in neural information processing systems , 21.

Jitkrittum, W., Szabó, Z., Chwialkowski, K. P., and Gretton, A. (2016). Interpretable distribution features with maximum testing power. Advances in Neural Information Processing Systems , 29.

Jitkrittum, W., Szabó, Z., and Gretton, A. (2017). An adaptive test of independence with analytic kernel embeddings. In International Conference on Machine Learning , pages 1742-1751. PMLR.

Kalinke, F. and Szabo, Z. (2024). The minimax rate of hsic estimation for translation-invariant kernels. arXiv preprint arXiv:2403.07735 .

Kim, I. and Schrab, A. (2023). Differentially private permutation tests: Applications to kernel methods. arXiv preprint arXiv:2310.19043 .

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Kübler, J., Jitkrittum, W., Schölkopf, B., and Muandet, K. (2020). Learning kernel tests without data splitting. Advances in Neural Information Processing Systems , 33:6245-6255.

Li, Y., Pogodin, R., Sutherland, D. J., and Gretton, A. (2021). Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems , 34:15543-15556.

Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Sutherland, D. J. (2020). Learning deep kernels for non-parametric two-sample tests. In International conference on machine learning , pages 6316-6326. PMLR.

Liu, L., Pal, S., and Harchaoui, Z. (2022). Entropy regularized optimal transport independence criterion. In International Conference on Artificial Intelligence and Statistics , pages 11247-11279. PMLR.

Lopez-Paz, D., Hennig, P., and Schölkopf, B. (2013). The randomized dependence coefficient. Advances in neural information processing systems , 26.

Podkopaev, A. and Ramdas, A. (2024). Sequential predictive two-sample and independence testing. Advances in Neural Information Processing Systems , 36.

Rahimi, A. and Recht, B. (2007). Random features for large-scale kernel machines. Advances in neural information processing systems , 20.

Ramdas, A., Reddi, S. J., Póczos, B., Singh, A., and Wasserman, L. (2015). On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29.

Ren, Y., Xia, Y., Zhang, H., Guan, J., and Zhou, S. (2024). Learning adaptive kernels for statistical independence tests. In International Conference on Artificial Intelligence and Statistics , pages 2494-2502. PMLR.

Ren, Y., Zhang, H., Xia, Y., Guan, J., and Zhou, S. (2023). Multi-level wavelet mapping correlation for statistical dependence measurement: methodology and performance. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37(5), pages 6499-6506.

Schrab, A., Kim, I., Guedj, B., and Gretton, A. (2022). Efficient aggregated kernel tests using incomplete u -statistics. Advances in Neural Information Processing Systems , 35:18793-18807.

Sejdinovic, D., Sriperumbudur, B., Gretton, A., and Fukumizu, K. (2013). Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The annals of statistics , pages 2263-2291.

Serfling, R. J. (2009). Approximation theorems of mathematical statistics . John Wiley & Sons.

Shekhar, S., Kim, I., and Ramdas, A. (2023). A permutation-free kernel independence test. Journal of Machine Learning Research , 24(369):1-68.

Sriperumbudur, B. K., Gretton, A., Fukumizu, K., Schölkopf, B., and Lanckriet, G. R. (2010). Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research , 11:1517-1561.

Sutherland, D. J. and Schneider, J. (2015). On the error of random fourier features. arXiv preprint arXiv:1506.02785 .

Sutherland, D. J., Tung, H.-Y., Strathmann, H., De, S., Ramdas, A., Smola, A., and Gretton, A. (2016). Generative models and model criticism via optimized maximum mean discrepancy. arXiv preprint arXiv:1611.04488 .

Székely, G., Rizzo, M., and Bakirov, N. (2007). Measuring and testing dependence by correlation of distances. Annals of Statistics , 35(6):2769-2794.

Székely, G. J. and Rizzo, M. L. (2013). The distance correlation t-test of independence in high dimension. Journal of Multivariate Analysis , 117:193-213.

Vershynin, R. (2018). High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press.

Wang, Z., Zhan, Z., Gong, Y., Shao, Y., Ioannidis, S., Wang, Y., and Dy, J. (2023). Dualhsic: Hsicbottleneck and alignment for continual learning. In International Conference on Machine Learning , pages 36578-36592. PMLR.

Zhang, K., Peters, J., Janzing, D., and Schölkopf, B. (2012). Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv:1202.3775 .

Zhang, Q., Filippi, S., Gretton, A., and Sejdinovic, D. (2018). Large-scale kernel methods for independence testing. Statistics and Computing , 28:113-130.

Zhang, T., Zhang, Y., and Zhou, T. (2024). Statistical insights into hsic in high dimensions. Advances in Neural Information Processing Systems , 36.

Appendix Organization

Section A: List of Symbols and Notations.

Section B: Assumptions.

Section C: Some Auxiliary Lemma.

Section D: Proof of Proposition 1.

Section E: Proof of Theorem 1.

Section F: Calculation of Eq. (16).

Section G: Proof of Theorem 2.

Section H: Proof of Theorem 3.

Section I: Smoothness of Optimization Objective.

Section J: Details of Experiment Setup.

Section K: Additional Experiment Results.

Section L: Limitations and Broader Impacts.

A List of Symbols and Notations

B Assumptions

The following are the assumptions required. We denote the parameter spaces of θ k , θ l as Θ k , Θ l .

(a) The mapping functions T θ k , T θ l are Lipschitz to the parameters θ k , θ l , i.e. for all x ∈ X , y ∈ Y and for all θ k , θ ′ k ∈ Θ k , θ l , θ ′ l ∈ Θ l ,

with the nonnegative Lipschitz constant L k , L l .

(b) The range of the mapping functions T θ k , T θ l are bounded.

(c) The parameters θ 0 , θ 1 lie in Banach spaces of dimension d k , d l respectively. Also, the parameters θ k , θ l are bounded by R θ k , R θ l respectively, i.e., ∥ θ k ∥ ≤ R θ k , ∥ θ l ∥ ≤ R θ l .

C Some Auxiliary Lemma

C.1 A Useful Expression

In this part, we give a useful expression for h ( ω ) ijqr for the subsequent proof. By the definition, we have h ( ω ) ijqr = 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k ( ω ) tu l ( ω ) tu + k ( ω ) tu l ( ω ) vw -2 k ( ω ) tu l ( ω ) tv . We simplify it by setting t = i , u = i , v = i and w = i in turn. Then we can show that h ( ω ) ijqr is equal to

By the definition, k ( ω ) tu := ψ ( ω ) k ( T θ k x t -T θ k x u ) = Λ X ( x t )Λ X ( x u ) T is symmetric, i.e. k ( ω ) tu = k ( ω ) ut . And so as l ( ω ) tu . Hence we can merge the identical items (marked with the same color). As a result,

We will use Eq. (20) many times in subsequent proofs.

C.2 Properties of Learnable Random Fourier Feature

Under the assumption (a), RFFs ψ ( ω ) k , ψ ( ω ) l are Lipschitz to the parameters θ k , θ l . Formally,

Lemma 1. (Lipschitz Property of Fourier Feature). Let T θ k , T θ l be the mapping functions of X,Y that are Lipschitz to the parameters θ k , θ l with the non-negative constant L k , L l , respectively. Let { ( ω k ; j , ω l ; j ) } D/ 2 j =1 be the samplings of frequency with the sampling number D . Also, we define R ω k := sup j ∥ ω k ; j ∥ , R ω l := sup j ∥ ω l ; j ∥ , then for the RFFs ψ ( ω ) k , ψ ( ω ) l with mapping functions T θ k , T θ l and frequency samplings { ( ω k ; j , ω l ; j ) } D/ 2 j =1 , for all ( x, x ′ ) ∈ X × X , ( y, y ′ ) ∈ Y × Y and for all θ k , θ ′ k ∈ Θ k , θ l , θ ′ l ∈ Θ l , we have

where ∆ x,x ′ := T θ k x -T θ k x ′ , ∆ ′ x,x ′ := T θ ′ k x -T θ ′ k x ′ and ∆ y,y ′ , ∆ ′ y,y ′ are defined by analogy.

Proof. We prove the result for ψ ( ω ) k only since the proof for ψ ( ω ) l can be obtained in the same way. We start by recall the definition ψ ( ω ) k (∆ x,x ′ ) := 2 D ∑ D/ 2 j =1 cos( ω T k ; j ∆ x,x ′ ) . Then

Since the cosine function is bounded by 1 , by the mean value theorem, for fixed j , we have

Then according to the Cauchy-Schwarz inequality,

By the definition of ∆ x,x ′ and the Lipschitz property of the mapping functions T θ k , T θ l , we have

Combining the above results, we complete the proof.

Under the assumption (b), we can obtain the uniform convergence property as follows.

Lemma 2. (Uniform Convergence of Fourier Features). Let the mapping function of X,Y with parameters θ k , θ l be T θ k , T θ l , and the corresponding range space be a compact subset of R d T x , R d T y , respectively. Also, the diameter of two range spaces is denoted by diam ( T θ k ) , diam ( T θ l ) , respectively. Let { ( ω k ; j , ω l ; j ) } D/ 2 j =1 be the samplings of frequency with the sampling number D , then for the RFFs with mapping functions T θ k , T θ l and frequency samplings { ( ω k ; j , ω l ; j ) } D/ 2 j =1 , we have

where the second moment of frequency samplings σ 2 ω k := E p k ( ω ) [ ω T k ; j ω k ; j ] , σ 2 ω l := E p l ( ω ) [ ω T l ; j ω l ; j ] .

Proof. Based on the derivation of RFFs in Sec. 4.1, We can view it as if the frequency sampling process is performed after the range space is obtained. Since the convergence bounds of the sampling process can be obtained directly through the results of [28, Claim 1], by replacing the input space in [28, Claim 1] to the range space here, then this part of the proof can be completed.

Remark. Combining the technique in [37], the constants in bounds can be further improved.

C.3 Approximation Error Bound

Let HSIC ( u ) ω ( Z ) , also denoted as ξ ( u ) ω , be the U-statistic that corresponding to HSIC ω ( Z ) , i.e., HSIC ( u ) ω ( Z ) := 1 ( n ) 4 ∑ ( i,j,q,r ) ∈ i n 4 h ( ω ) ijqr . The population value of HSIC ( u ) ω ( Z ) is given by E Z ξ ( u ) ω which can be viewed as the result obtained after a frequency sampling approximation on HSIC ( X,Y ) is performed. The bound of approximation error is given by the following Lemma.

Lemma 3. (Approximation Error Bound). For simplify, we denote Λ k ( x ) T Λ k ( x ′ ) , Λ l ( y ) T Λ l ( y ′ ) as k ( ω ) ( x, x ′ ) , l ( ω ) ( y, y ′ ) , respectively. Then we have

Proof. We first represent E Z ξ ( u ) ω in the form corresponding to Eq. (1), i.e.,

Taking one of the items as an example and comparing it to the corresponding item in Eq. (1),

The results can be obtained for the other terms in a similar way, which completes the proof.

D Proof of Proposition 1

In this section, we give a proof of the Proposition 1. We first restate the Proposition 1 here.

Proposition 1 (Asymptotics) . Let h ( ω ) ijqr := 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k ( ω ) tu l ( ω ) tu + k ( ω ) tu l ( ω ) vw -2 k ( ω ) uv l ( ω ) tv , where the sum represents all ordered quadruples ( t, u, v, w ) drawn without replacement from ( i, j, q, r ) . Then, Under the null hypothesis H 0 , HSIC ω ( Z ) coverages in distribution to

where χ 2 11 , χ 2 12 , ... are independent χ 2 1 variates and λ l is the solution to the eigenvalue problem as in the right of Eq. (30). Also, under the alternative H 1 , HSIC ω ( Z ) converges in distribution as

Proof. The proof is mainly based on [34, Chapter 5]. A proof for a similar result has been given in [14, Theorem 1,2]. The difference is that we consider the asymptotic distributions of HSIC ω ( Z ) that used learnable RFF thus is a function of frequency samplings while they consider HSIC b ( Z ) . Thus some steps need to be modified.

Step 1 : we show that HSIC ω ( Z ) is a V-statistic, this can be done since it can be expressed as HSIC ω ( Z ) = 1 n 4 ∑ i,j,q,r h ( ω ) ijqr . To facilitate the conclusions in [34] for the U-statistic, we define the U-statistic HSIC ( u ) ω ( Z ) := 1 ( n ) 4 ∑ ( i,j,q,r ) ∈ i n 4 h ( ω ) ijqr that corresponds to HSIC ω ( Z ) .

Step 2 : Then we prove the result under H 0 . To begin with, we first show that E j,q,r h ( ω ) ijqr = 0 with ( i, j, q, r ) ∈ i n 4 under H 0 . According to Eq. (20), we can calculate E j,q,r h ( ω ) ijqr as

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

̸

where we define simplified notions E i = u = v = w u,v,w whose superscript indicates the restriction. For readability, we will require additional notations: E x k ( ω ) i := E i = u u k ( ω ) iu and E x k ( ω ) := E t = u t,u k ( ω ) tu (the notations for Y is defined by analogy). Under H 0 , X and Y are independence. Hence

̸

̸

Then combining the results [34, Section 5.5.2], we can prove Eq. (30).

Step 3 : Next we prove the asymptotic distribution under H 1 . Weonly need to show that | HSIC ω ( Z ) -HSIC ( u ) ω ( Z ) | ∼ O (1 /n ) . By the definition of k ( ω ) tu , l ( ω ) tu , we can check that | k ( ω ) tu | ≤ 1 , | l ( ω ) tu | ≤ 1 for all t, u , thus | h ( ω ) ijqr | ≤ 4 for all i, j, q, r . Hence we have

Combining the results [34, Section 5.5.1], we can prove Eq. (31).

E Proof of Theorem 1

In this section, we give a proof of the Theorem 1. We first restate the Theorem 1 here.

Theorem 1 (Linear-Time Estimators) . Under H 0 , the estimators of mean and variance with bias of O ( n -1 ) to E Z [ n HSIC ω ( Z )] and Var Z [ n HSIC ω ( Z )] , denote as E 0 and V 0 , respectively, are given by

where () · 2 is the entrywise matrix power. Both E 0 and V 0 can be calculated in O ( nD 2 ) time.

Proof. We first prove the part of the mean. Recall the definition of HSIC ω ( Z ) , we have HSIC ω ( Z ) = 1 n 4 ∑ i,j,q,r h ( ω ) ijqr . Hence E Z [ HSIC ω ( Z )] = 1 n 4 ∑ i,j,q,r E Z h ( ω ) ijqr . When ( i, j, q, r ) ∈ i n 4 , then we can show that under H 0 , E i,j,q,r h ( ω ) ijqr = 0 by performing the same analysis as in Eqs. (32) and (33). Then we consider the case where exactly two elements of i, j, q, r are the same, for a total of 6 n ( n -1)( n -2) terms. By the symmetry of h ( ω ) ijqr , the expectation of these terms all take the same value, and here we take h ( ω ) iiqr as an example. According to Eq. (20), we can represent h ( ω ) iiqr as

Under H 0 , X and Y are independence. Take the expectation on both sides, we have

̸

where we define additional notation E x k ( ω ) := E t = u t,u k ( ω ) tu (the notation for Y is defined by analogy) and use k ω tt = l ω tt = 1 . Hence in this case, the sum of the contributions of all terms to E Z [ n HSIC ω ( Z )] is (1 -E x k ( ω ) -E y l ( ω ) + E x k ( ω ) E y l ( ω ) ) + O (1 /n ) . For the remaining terms, i.e., the case where at least three of i, j, q, r are equal, combined with the boundedness of h ( ω ) ijqr , we can conclude that the sum of their contributions is O (1 /n ) . As a result, we have shown that

The unbiased estimators of E x k ( ω ) , E y l ( ω ) are given by 1 T ( Λ X Λ T X -I n ) 1 , 1 T ( Λ Y Λ T Y -I n ) 1 , respectively. Hence under H 0 , we obtain the estimator of mean with bias of O ( n -1 ) as

Next, we prove the part of the variance. We start by calculating Var Z [ n HSIC ( u ) ω ( Z )] , where the Ustatistic HSIC ( u ) ω ( Z ) := 1 ( n ) 4 ∑ ( i,j,q,r ) ∈ i n 4 h ( ω ) ijqr . According to the results [34, Section 5.2.1, Lemma A], we have

where ζ 1 := E i ( E j,q,r h ( ω ) ijqr ) 2 and ζ 2 := E i,j ( E q,r h ( ω ) ijqr ) 2 . Under H 0 , when ( i, j, q, r ) ∈ i n 4 , we can show that E j,q,r h ( ω ) ijqr = 0 by performing the same analysis as in Eqs. (32) and (33), thus ζ 1 = 0 . For calculating ζ 2 , we mainly focus on the term E q,r h ( ω ) ijqr . We use Eq. (20) again,

Under H 0 , X and Y are independence. Take the expectation E q,r on both sides, we have

̸

where we define additional notation E x k ( ω ) := E t = u t,u k ( ω ) tu , E x k ( ω ) j := E j = u u k ( ω ) ju (the notations for Y are defined by analogy). For simplify, we denote k ( ω ) c ; ij := k ( ω ) ij -E x k ( ω ) i -E x k ( ω ) j + E x k ( ω ) and l ( ω ) c ; ij := l ( ω ) ij -E y l ( ω ) i -E y l ( ω ) j + E y l ( ω ) . Hence combining Eq. (40), we have

̸

Since under H 0 , the bias lead by the difference terms between HSIC ( u ) ω ( Z ) and HSIC ω ( Z ) vanish faster than Eq. (43), hence the variance of HSIC ω ( Z ) is identical. In the following part, we consider the empirical estimate of the leading term in Eq. (43). We estimate k ( ω ) c ; ij with ( Λ Xc Λ T Xc ) ij , then the estimation of E i,j ( k ( ω ) c ; ij ) 2 is given by

where we define notions Λ k := 1 n ∑ n u =1 Λ k ( x u ) . Since computing the value of 1 T ( Λ Xc Λ T Xc ) · 2 1 requires O ( n 2 ) time complexity, we transform it into a more computationally tractable form. We perform the following calculating as

Recall the definition of Λ Xc := [ HΛ X ] n × D that can be calculated in O ( nD ) time, thus the term [ Λ T Xc Λ Xc ] D × D can be calculated in O ( nD + nD 2 ) time. As a result, we obtain the estimator [ 1 T ( Λ T Xc Λ Xc ) · 2 1 ][ 1 T ( Λ T Y c Λ Y c ) · 2 1 ] for the term E i,j ( k ( ω ) c ; ij ) 2 · E i,j ( l ( ω ) c ; ij ) 2 that can be calculated in O ( nD 2 ) time. The only thing left to do is to determine the bias of the estimator. For readable, we define ̂ k ( ω ) ij := k ( ω ) ij , ̂ k ( ω ) i := 1 n ∑ u k ( ω ) iu and ̂ k ( ω ) := 1 n 2 ∑ u,v k ( ω ) uv , then by removing the terms with i = j , a estimate with difference O ( n -1 ) to Eq. (44) is given by

̸

By comparing the difference between the expectation of Eq. (46) and E i,j ( k ( ω ) c ; ij ) 2 , we can show that this error is bound by O (1 /n ) . We illustrate this by taking one of the cross terms as an example and the other terms by analogy, as shown in the following,

̸

Similarly, we can obtain the results for E i,j ( l ( ω ) c ; ij ) 2 . As a result, we have shown that V 0 is a estimator of Var Z [ n HSIC ω ( Z )] with bias O ( n -1 ) thus complete the whole proof.

F Calculation of Eq. (16)

Here, we give the computational details of Eq. (16). We mark colors to indicate correspondences.

According to Eq. (20), we can calculate ∑ j,q,r h ( ω ) ijqr as

We can further represent Eq. (48) in matrices form as

Next, by variable substitution, we obtain the result as

where the definition of variables A to F with the calculation cost are given in the Fig. 1. For convenience, we re-show the diagram here for reference.

Time Complexity:

The computational complexity of each step is illustrated in Fig. 5. We explain some steps here. As a start, recall that the size of Λ X , Λ Y are both n × D . Therefore a time complexity O ( nD 2 ) is required to compute Λ T X Λ Y by matrix multiplication operation. Further multiplying the obtained [ Λ T X Λ Y ] D × D with Λ T Y requires O ( nD 2 ) time complexity. Next, since both Λ X and ( Λ T X Λ Y Λ T Y ) T are of size n × D , the elemental product operation requires a time complexity of O ( nD ) . In a similar way, we can check the time complexity for each remained step. After getting the variables A to F , since 1 T A1 , A1 all can be calculated in O ( nD ) and 1 T B , 1 T C , 1 T D all can be calculated in O ( n ) , we conclude that the results with index i in Eq. (50) can be obtained in O ( nD 2 ) time.

G Proof of Theorem 2

In this section, we give a proof of the Theorem 2. We first restate the Theorem 2 here.

Theorem 2 (Uniform Bound) . Let θ k , θ l parameterize T θ k , T θ l in Banach spaces of dimension d k , d l . And T θ k , T θ l are Lipschitz to the parameters θ k , θ l with the non-negative constant L k , L l , respectively. Let Θ c be a set of ( θ k , θ l ) for which σ ω ≥ c > 0 with a positive constant c and ∥ θ k ∥ ≤ R θ k , ∥ θ l ∥ ≤ R θ l . Let r denote the threshold, i.e., (1 -α ) -quantile for the distribution in Eq. (11) and r ( n ) be the threshold with sample size n . Let { ( ω k ; j , ω l ; j ) } D/ 2 j =1 be the samplings of frequency with the sampling number D . Also, we define R ω k := sup j ∥ ω k ; j ∥ , R ω l := sup j ∥ ω l ; j ∥ , d s := max { d k , d l } and ξ ω := HSIC ω ( Z ) . Then with probability at least 1 -δ , we have

Proof. We take a similar roadmap of proof as [30] and extend it to our optimization objective. The roadmap of the proof is as follows: we first obtain the convergence results (with sample size n ) for each estimator with fixed parameters θ k , θ l , and then extend the results to the entire parameter space via ϵ -net arguments. We begin the proof of the first part, which is based on bounded differences inequality (McDiarmid's inequality) [41, Theorem 2.9.1].

.

Bound of | ξ ω -E Z ξ ω | . Recall the definition of ξ ω := HSIC ω ( Z ) = 1 n 4 ∑ i,j,q,r h ( ω ) ijqr . By the definition of k ( ω ) tu , l ( ω ) tu , we have | k ( ω ) tu | ≤ 1 , | l ( ω ) tu | ≤ 1 for all t, u , thus | h ( ω ) ijqr | ≤ 4 for all i, j, q, r Now we begin by showing the bounded differences property of h ( ω ) ijqr . Concretely, we replace the first sample z 1 = ( x 1 , y 1 ) with z ′ 1 = ( x ′ 1 , y ′ 1 ) and keep the remaining samples the same. The obtained samples are named as Z ′ . Then the difference terms between h ( ω ) ijqr and the new substitution ˘ h ( ω ) ijqr can only happen in the case that at least one of i, j, q, r is equal to 1 . For the case that only one subscript is 1 (here take i = 1 for example), combining Eq. (20), we have

The whole contributes of remaining terms that at least two i, j, q, r are less than O ( n -2 ) , thus

Hence HSIC ω ( Z ) satisfy the bounded differences property with O ( n -1 ) . Using McDiarmid's inequality, for fixed θ 0 , θ 1 , with probability at least 1 -δ , there exist a universal constant C 1 such that

Bound of | r ( n ) ω -r ω | . As r ( n ) ω is the (1 -α ) of the distribution of nξ ω with sample size n under H 0 , according to the Eq. (53), when n is large enough, there exist a universal constant C 2 such that

where the last inequation is because E Z ξ ω ∼ O ( n -1 ) under H 0 (see Theorem 1 for a detailed explanation). Hence | r ( n ) ω | ∼ O (√ n log(1 /α ) ) . Also, by definition r ω is a constant related to α .

Bound of | ̂ σ 2 ω -σ 2 ω | . In this part, We first obtain the bound of | ̂ σ 2 ω -E Z ̂ σ 2 ω | , then obtain the bound of | E Z ̂ σ 2 ω -σ 2 ω | . As before we start by showing the bounded variance property of ̂ σ 2 ω . We replace z 1 = ( x 1 , y 1 ) with z ′ 1 = ( x ′ 1 , y ′ 1 ) and keep the remaining samples the same. The obtained samples are named as Z ′ . For readable, we denote ̂ σ 2 ω with sample Z, Z ′ as ̂ σ 2 ω ( Z ) , ̂ σ 2 ω ( Z ′ ) respectively.

Again, the difference terms between h ( ω ) ijqr and the new substitution ˘ h ( ω ) ijqr can only happen in the case that at least one of i, j, q, r is equal to 1 . Hence, Eqs. (55) and (56) are both O ( n -1 ) . As a result, ̂ σ 2 ω satisfy the bounded differences property with bound O ( n -1 ) . Using McDiarmid's inequality, with probability at least 1 -δ , there exist a universal constant C 3 such that ∣ ∣ ̂ σ 2 ω -E Z ̂ σ 2 ω ∣ ∣ ≤ C 3 √ 1 n log 2 δ . Next we obtain the bound of | E Z ̂ σ 2 ω -σ 2 ω | . We rewrite E Z ̂ σ 2 ω as

By adding further restrictions that i, i ′ , j, q, r, j ′ , q ′ , r ′ are all different, we can obtain the corresponding expression for σ 2 ω . Hence the difference between them can only happen when at least one subscript in i ′ , j, q, r, j ′ , q ′ , r ′ is equal to i . Combining | h ( ω ) ijqr | ≤ 4 , we have | E Z ̂ σ 2 ω -σ 2 ω | ∼ O ( n -1 ) .

ϵ -net arguments. Next, we prove the second part with ϵ -net arguments. Take the parameter space Θ k of θ k as an example. We choose a cover with N (Θ k , r k ) points { p i } N (Θ k ,r k ) i =1 such that for any point p ∈ Θ k , we have min i ∥ p -p i ∥ ≤ r k . According to [41, Proposition 4.2.12], by comparing the volumes, we have N (Θ k , r k ) ≤ (4 R Θ k /r k ) d k . As for the parameter space Θ l of θ l , we can also obtain a cover with N (Θ l , r l ) points that N (Θ k , r k ) ≤ (4 R Θ l /r l ) d l . Here, we set r k = 4 R Θ k / √ n, r l = 4 R Θ l / √ n , thus N (Θ k , r k ) ≤ ( √ n ) d k , N (Θ l , r l ) ≤ ( √ n ) d l . Then combining the Lipschitz property as shown in Lemma 4, we have with probability at least 1 -δ ,

Hence when n is large enough, there exists a positive constant C 3 , with probability at least 1 -δ ,

Similar, when n is large enough, there exists a positive constant C 4 , with probability at least 1 -δ ,

Overall Bound. Now we combine the previously obtained results. We have

Since on Θ c , σ ω ≥ c , according to Eq. (59), we can make ̂ σ ω ≥ c/ 2 happen by assigning probability budget δ/ 2 when n is large enough. Also, combining Eq. (54), | E Z ξ ω | ≤ 1 and r ω is a constant related to α , then with n large enough, there exist positive constants C 5 , C 6 ,

Note that we need to pay for probability budget δ/ 2 for the above conclusion to hold. Then by taking the supremum on both sides in Eq. (60) and assigning the remained probability budget δ/ 2 to Eqs. (58) and (59), we can show that when n is large enough, with probability at least 1 -δ ,

thus complete the proof.

H Proof of Theorem 3

In this section, we give a proof of the Theorem 3. We first restate the Theorem 3 here.

Theorem 3 (Consistency) . Let θ ∗ k , θ ∗ l be the parameters after learning, Z te be the testing samples of size m , when E Z HSIC ( u ) ω ( Z te ) > 0 , then the probability of Type II error

Let the mapping functions with learned parameters θ ∗ k , θ ∗ l be T θ ∗ k , T θ ∗ l , and the corresponding range space be a compact subset of R d T x , R d T y , respectively. Also, the diameter of two range spaces is denoted by diam ( T θ ∗ k ) , diam ( T θ ∗ l ) , respectively. Let { ( ω k ; j , ω l ; j ) } D/ 2 j =1 be the frequency samplings with their second moment denoted by σ 2 ω k := E p k ( ω ) [ ω T k ; j ω k ; j ] , σ 2 ω l := E p l ( ω ) [ ω T l ; j ω l ; j ] . Additionally, we denote ξ u := HSIC ( X,Y ) , then under H 1 , we have E Z HSIC ( u ) ω ( Z te ) > 0 with any constant probability when D = Ω ( d T x + d T y ξ 2 u log σ ω k diam ( T θ ∗ k )+ σ ω l diam ( T θ ∗ l ) ξ u ) .

Proof. The proof consists of two parts, we first give the rate of convergence of Type II error under condition E Z HSIC ( u ) ω ( Z te ) > 0 , and next for condition E Z HSIC ( u ) ω ( Z te ) > 0 we give a lower bound on the number of frequency samplings required for it to hold. To simplify, we denote the U-statistic HSIC ( u ) ω ( Z te ) as ξ ( u ) ω in this proof. We begin to prove the first part. With the learned parameters θ ∗ k , θ ∗ l , the probability of the Type II error is given by

Combing the result of the difference between ξ ω and ξ ( u ) ω as shown in Eq. (34), we have

where C 0 is a positive constant. To apply the rate of convergence of the Central Limit Theorem, we rewrite the right equation in Eq. (64) as

where the standard deviation (defined in Proposition 1) σ ω > 0 under H 1 . Then according to the results in [34, Section 5.5.1 Theorem B], there exist nonnegative constant C 1 such that

̸

where ν h := E i = j = q = r Z | h ( ω ) ijqr | 3 < ∞ . When m is large enough, we further have

̸

̸

where C 2 , C 3 , C 4 are positive constants and using r ( m ) ∼ O ( m 1 / 2 ) we prove in Eq. (54). Hence when E Z ξ ( u ) ω > 0 , the leading term √ m E Z ξ ( u ) ω decrease as m increase. Further, to obtain the decrease rate when m is close to infinity, we consider the asymptotic expansion (when x is close to negative infinity) for the function Φ( x ) as given by

thus Φ ( C 2 -C 3 √ m E Z ξ ( u ) ω + C 4 / √ m ) ∼ O ( m -1 / 2 ) . As a result, the decreasing rate is at least O ( m -1 / 2 ) . We have so far completed the first part of the proof, and we next begin the second part of the proof, i.e. obtain the number of frequency samplings required for the condition E Z ξ ( u ) ω > 0 to

hold. For simplify, we denote Λ k ( x ) T Λ k ( x ′ ) , Λ l ( y ) T Λ l ( y ′ ) as k ( ω ) ( x, x ′ ) , l ( ω ) ( y, y ′ ) , respectively. Then according to Lemma 2, we have

Also, we denote the bounds in Eq. (69) as δ x ( ϵ, D ) , δ y ( ϵ, D ) , respectively. Next we get the bound between E Z ξ ( u ) ω and HSIC ( X,Y ) . According to Lemma 3, the bound is given by

Since by the definition, for all ( x, x ′ ) ∈ X × X , ( y, y ′ ) ∈ Y × Y , we have | k ( ω ) ( x, x ′ ) | ≤ 1 , | l ( ω ) ( y, y ′ ) | ≤ 1 , | k ( x, x ′ ) | ≤ 1 , | l ( y, y ′ ) | ≤ 1 . Hence we have

Combining the results of Eqs. (70) and (71), we obtain

Combining the results as shown in Eq. (69) and allocating the probability budget ϵ , we have

By setting ϵ = ξ u / 2 , and since ξ u > 0 under H 1 , we conclude that E Z ξ ( u ) ω > 0 holds with any constant probability when D = Ω ( d T x + d T y ξ 2 u log σ ω k diam ( T θ ∗ k )+ σ ω l diam ( T θ ∗ l ) ξ u ) .

In the proof of Theorem 3, we obtain the convergence bound of the statistic E Z ξ ( u ) ω . Actually, the convergence result for its estimation can also be obtained, as shown in the following Corollary.

Corollary 1 (Approximation Error Bound of HSIC ω ( Z te ) ) . Maintaining the same conditions and notions as in Theorem 3, we have the uniform convergence bound of HSIC ω ( Z te ) as

Proof. By the definition, we have for all i, j, q, r , | k ( ω ) ij | ≤ 1 , | l ( ω ) ij | ≤ 1 , | k ij | ≤ 1 , | l ij | ≤ 1 , thus

Then according to the results as shown in Eq. (69), we have for all i, j, q, r

Recall the definition of h ( ω ) ijqr = 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k ( ω ) tu l ( ω ) tu + k ( ω ) tu l ( ω ) vw -2 k ( ω ) uv l ( ω ) tv and we further define the corresponding h ijqr = 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k tu l tu + k tu l vw -2 k uv l tv , then for all i, j, q, r ,

After that, using the expressions that we obtained before, i.e., HSIC ω ( Z te ) := 1 n 4 ∑ i,j,q,r h ( ω ) ijqr and HSIC b ( Z te ) := 1 n 4 ∑ i,j,q,r h ijqr , we obtain the final bound that

and thus complete the proof.

I Smoothness of Optimization Objective

We first prove the Lipschitz property for some functions. For ease of reference, we re-list here the definitions of the terms that related to the optimization objective: ξ ω := HSIC ω ( Z ) = 1 n 4 ∑ i,j,q,r h ( ω ) ijqr , ̂ σ 2 ω := 16 [ 1 n ∑ i ( 1 n 3 ∑ j,q,r h ( ω ) ijqr ) 2 -HSIC 2 ω ( Z ) ] and σ 2 ω := 16 [ E i ( E j,q,r h ( ω ) ijqr ) 2 -( E Z h ( ω ) ijqr ) 2 ] . The Lipschitz property of these terms are shown as follows.

Lemma 4 (Lipschitz Property of of ξ ω , E Z ξ ω , ̂ σ 2 ω , σ 2 ω ) . Maintaining the same conditions and notions as in Lemma 1, we have the following Lipschitz property

where we use the symbol ξ ω ( θ k , θ l ) to denote ξ ω with the parameter θ k , θ l and the others by analogy.

Proof. We start by obtaining the result of h ( ω ) ijqr for all i, j, q, r . Since for all i, j, q, r ,

where the property | k ( ω ) ij | ≤ 1 and | l ( ω ) qr | ≤ 1 are used. According the Lemma 1, we have

Combing the definition that h ( ω ) ijqr := 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k ( ω ) tu l ( ω ) tu + k ( ω ) tu l ( ω ) vw -2 k ( ω ) uv l ( ω ) tv , then

Also, combining the definition of ξ ω := HSIC ω ( Z ) = 1 n 4 ∑ i,j,q,r h ( ω ) ijqr , we obtain

By using | E Z [ ξ ω ( θ k , θ l )] -E Z [ ξ ω ( θ ′ k , θ ′ l )] | ≤ E Z [ | ξ ω ( θ k , θ l ) -ξ ω ( θ ′ k , θ ′ l ) | ] , we have

For the results of ̂ σ 2 ω , σ 2 ω , we first proof the following results. For all i, j, q, r, i ′ , j ′ , q ′ , r ′ , we have

where the first inequality holds due to property | h ( ω ) ijqr | ≤ 4 . Then we use the expression

and combine the results in Eq. (85). As a result, we obtain that

In a similar way, we can obtain the corresponding expression of σ 2 ω as

Then we obtain a similar result as before, i.e.,

which completes the proof.

Also for the term associated with the estimated threshold (recall that it is computed from the first two moments), we obtain the following properties of E 0 , V 0 as defined in Theorem 1.

Lemma 5 (Lipschitz Property of of E 0 , V 0 ) . Maintaining the same conditions and notions as in Lemma 4, we have the following Lipschitz property

Proof. The expression in Theorem 1, while easy to compute, is not suitable for this part of our proof. We begin by obtaining equivalent expressions for E 0 , V 0 . According to Eq. (39), we have

And for V 0 , we use Eq. (45) and obtain

Also, we define h ( k ) ijqr := 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) k ( ω ) tu k ( ω ) tu + k ( ω ) tu k ( ω ) vw -2 k ( ω ) uv k ( ω ) tv that corresponding to h ( ω ) ijqr and also define h ( l ) ijqr := 1 4! ∑ ( i,j,q,r ) ( t,u,v,w ) l ( ω ) tu l ( ω ) tu + l ( ω ) tu l ( ω ) vw -2 l ( ω ) uv l ( ω ) tv . Then we can further rewrite the term for X as 1 n 2 Tr ( Λ X Λ T X HΛ X Λ T X H ) = 1 n 4 ∑ i,j,q,r h ( k ) ijqr and for Y by analogy. Then the properties of h ( ω ) ijqr can also be obtained for h ( k ) ijqr and h ( l ) ijqr , e.g., | h ( k ) ijqr | ≤ 4 , | h ( l ) ijqr | ≤ 4 for all i, j, q, r . Next we start to prove the Lipschitz property of E 0 , V 0 . For E 0 , according to Eq. (81) and combining the results 0 ≤ 1 T ( Λ X Λ T X ) 1 ≤ n 2 and that for Y , we can show that

where the last inequation is obtained similar to Eq. (85), thus completes the proof.

The following results extend the results in [30] to the more general case (we only restrict the mapping functions to satisfy the Lipschitz property, and thus include the Gaussian kernel case of their proof).

Theorem 4 (Smoothness of Optimization Objective) . Let the sample of size n be Z , and with a small positive constant c , let the set of the parameters be Θ c := { ( θ k , θ l ) | ̂ σ ω ≥ c, V 0 ≥ c, E 0 ≥ c } , then there exist a nonnegative constant L such that ∥∇ ( θ k ,θ l ) J ∥ ≤ L on Θ c , where the optimization objective J := [ HSIC ω ( Z ) -̂ c α /n ] / ̂ σ ω is that we used in practice.

Proof. According to Lemma 4, we have shown that HSIC ω ( Z ) , ̂ σ ω both fits the Lipschitz condition. Also, according to Lemma 5, we have shown that E 0 , V 0 are also Lipschitz with respect to θ k , θ l . Since the threshold ̂ c α is completely determined by these two moments, combining the smoothness property of the mapping from two moments to thresholds as analyzed in [30, Theorem 2], we obtain that ̂ c α is also Lipschitz with respect to θ k , θ l on Θ c . As a result, we complete the entire proof based on the Lipschitz property of composite mappings.

Remark. E 0 and V 0 are positive is almost satisfied in practice since according to the definition only [ 1 T Λ · 2 Xc 1 ][ 1 T Λ · 2 Y c 1 ] and [ 1 T ( Λ T Xc Λ Xc ) · 2 1 ][ 1 T ( Λ T Y c Λ Y c ) · 2 1 ] need to be greater than 0 .

J Details of Experiment Setup

In this section, we give an introduction to the comparison methods in our experiments and provide the implementation details of each method.

J.1 Details of Comparison Methods

The methods of comparison used in the experiment are described below.

dCor [39]: An independence test that is based on the distance covariance.

QHSIC [14]: The original quadratic-time HSIC independence test.

RDC [26]: The randomized dependence coefficient that measures the independence using the canonical correlation between a finite set of random features of the copula.

NyHSIC [44]: A variant of HSIC that uses the Nyström method to approximate kernels.

FHSIC [44]: A variant of HSIC that uses the random Fourier feature to approximate kernels.

BHSIC [44]: A variant of HSIC with the block-based statistic.

HSICAgg [32]: An aggregated kernel test with the incomplete statistic of HSIC.

NFSIC [18]: A test uses the normalized version of the finite set independence criterion and chooses features on a hold-out validation set to optimize a lower bound on the test power.

Below are the GitHub URLs for each comparison method.

dCor: https://pypi.org/project/dcor .

QHSIC: https://github.com/amber0309/HSIC/blob/master/HSIC.py .

RDC: https://github.com/lopezpaz/randomized_dependence_coefficient .

NyHSIC: https://github.com/oxcsml/kerpy/blob/master/independence_testing .

FHSIC: https://github.com/oxcsml/kerpy/blob/master/independence_testing .

BHSIC: https://github.com/oxcsml/kerpy/blob/master/independence_testing .

HSICAgg: https://github.com/antoninschrab/mmdagg/tree/master/mmdagg .

NFSIC: https://github.com/wittawatj/fsic-test/blob/master/fsic .

Time Complexity. Among them, dCor and QHSIC are the tests of quadratic complexity with sample size n , i.e., O ( n 2 ) . RDC is calculated in O ( n log n ) and the rest are linear-time tests, i.e., O ( n ) .

Threshold. For dCor, QHSIC, RDC, NyHSIC, FHSIC, and BHSIC, we permute the samples 100 times to simulate the null distribution and compute the threshold. The thresholds for the remaining methods are obtained by asymptotic null distribution, i.e., we set the test threshold to the ( 1 -α )-quantile of χ 2 ( J ) for NFSIC and obtain the test threshold of LFHSIC-G/M by gamma approximation.

Details of Setup. The number of random features for FHSIC, LFHSIC-G/M, the number of induced variables for NyHSIC, the block size for BHSIC and the number of sub-diagonals R for HSICAgg are all kept consistent as recommended in [44] for fair evaluation. Specifically, we set the number of random mappings in RDC to 20 to ensure compatibility with large-scale datasets. The test location parameter J of NFHSIC is set as default as 10 , since it differs from other approximation methods that blindly increasing J may lead to a loss of power as shown in [18] and can significantly escalate time costs due to its cubic time complexity O ( J 3 ) . In the optimization step, for stabilizing the training, in the implementation of NFSIC we determine the initial bandwidth by searching the best from 25 bandwidth combinations (including the median bandwidth combination). For LFHSIC-G/M, to be fair, we perform the same grid search on SD, Sin, and GSign datasets. In other experiments, we still use the median bandwidth as initialization for LFHSIC-G/M. Also, the maximum number of iterations for the optimization is set to 100 for NFSIC and LFHSIC-G/M. The default learning rate of the optimization of LFHSIC-G/M is set as 0 . 05 in all the experiments. As for HSICAgg, the default implementation of the predefined 25 pairs of bandwidths in its code is used. For synthetic data, we set the split ratio to 0 . 5 for NFSIC and LFHSIC-G/M, i.e., we randomly sample half of the data for training and use the remaining for independence testing, while the other methods use all data for testing. For real MSD data, we divide a small portion of the data for training and then extract 100 random subsets of the remaining data (disjoint from the training set) for evaluation.

J.2 Details of Datasets

The details of the four synthetic datasets and two real datasets are described below.

Sine Dependency (SD): In this model, X follows a d -dimension multivariate normal distribution N d (0 , I d ) , and Y is defined as 20 sin ( 4 π ( X 2 1 + X 2 2 ) ) + Z , where X i is the i -th dimension of X , and Z ∼ N (0 , 1) represents independent noise. Notably, when d > 2 , Y exhibits a nonlinear relationship solely with the first two dimensions of X .

Sinusoid (Sin): This model introduces a localized alteration in the probability density function p xy over X × Y := [ -π, π ] 2 , specified as ( X,Y ) ∼ p xy ( x, y ) ∝ 1 + sin( ωx ) sin( ωy ) , where ω denotes the frequency. Increasing the frequency enhances the similarity between the sampled data and that drawn from Uniform( [ -π, π ] 2 ), thereby augmenting the challenge of detecting dependency with limited sample sizes. An example visualization is shown on the left of Fig. 6.

Gaussian Sign (GSign): In this model, X follows a d -dimension multivariate normal distribution N d (0 , I d ) , and Y is expressed as | Z | ∏ d i =1 sgn ( X i ) , where sgn ( · ) represents the sign function, X i denotes the i -th dimension of X , and Z ∼ N (0 , 1) is independent of X . The challenge lies in Y being independent of any proper subset of X but dependent on X as a whole, underscoring the importance of considering all dimensions of X simultaneously in independence testing.

ISA Dataset. We construct the data through the following steps: First, we generate n i.i.d samples of two univariate random variables with a mixture of Gaussian model, i.e. 1 2 N ( -1 , 0 . 01) + 1 2 N (1 , 0 . 01) . Second, we mix these random variables using a rotation matrix parameterized by an angle θ , which varies from 0 to π/ 4 . A zero angle implies independence between the data, while a larger angle signifies stronger dependency. Third, we append noise with a distribution of N d -1 (0 , I d -1 ) to each of the mixtures. Finally, we multiply an independent random d -dimensional orthogonal matrix to obtain vectors dependent across all observed dimensions. The resulting random variables X and Y are dependent but uncorrelated. When d is greater than 1 , the problem is associated with the independent subspace analysis (ISA) problem [14]. For the case d = 1 , θ = π/ 10 , an example visualization is shown in the middle of Fig. 6.

3DShapes Dataset. This dataset [5] comprises images depicting 3D scenes, complete with additional features like shadows and backgrounds. It encompasses six fundamental latent factors: floor hue, wall hue, object hue, object scale, object shape, and orientation, all adjustable to generate corresponding images. Orientation is treated as a dependency factor for independence testing, where we test the dependency between the image X and its orientation Y . To heighten the challenge, we maintain the object shape as a ball, minimizing the apparent orientation feature compared to other shapes like squares, while randomizing the remaining factors. An example visualization is given on the right side of Fig. 6.

Million Song Dataset. The dataset, a subset of the Million Song Data 3 [4], comprises 515 , 345 songs with 91 -dimensional features. The first dimension represents the release year of each song, designated as variable Y , while the remaining features (e.g., timbre average and timbre covariance) form variable X . Our objective is to identify the dependency between X and Y .

-2.5

0.0

2.5

X

-2

0

2

Y

n

=5000,

ω

=2

0.012

0.012

0.012

0.012

0.012

0.012

0.012

0.012

0.018

0.018

0.018

0.018

0.018

0.018

0.018

0.024

0.024

0.024

0.030

0.030

0.030

0.030

0.030

0.030

0.030

0.030

0.036

0.036

0.036

0.036

0.036

0.036

0.036

0.042

-2

0

2

X

-2

-1

0

1

2

Y

n

=1000,

θ

=

π

/10

0.04

0.04

0.04

0.04

0.08

0.08

0.08

0.08

0.12

0.12

0.12

0.12

0.16

0.16

0.16

0.16

0.20

0.20

0.20

0.20

0.24

0.24

Y(Orientation)

X(Image)

Floor hue

Wall hue

Object hue

Object shape

(ball)

Object scale

K Additional Experiment Results

In this section, we provide additional experimental results, mainly including the visualization results on the Sin synthetic dataset, the results with more comparing methods (as explained in the main paper, due to the high time overhead therefore do not participate in the evaluation of the main paper) as well as the running time of each method.

K.1 The visualization results on the Sin model

We provide the visualization results on the Sin synthetic dataset to illustrate the performance of our optimization objective for the example mentioned in the main paper. The results are shown in Fig. 7, where the setup follows our experiments ( n = 2000 , ω = 5 , D = 100 ). For visualization, the negative of our optimization objective J is shown. As can be seen, our optimization objective guides to letting the bandwidth adapt to improve the test power, and here we can see that regions with bandwidths around 0 . 2 (corresponding to the theoretical optimal solution in our main paper) indicate better power, thus corroborating the validity of our optimization objective. Also, notice that the landscape is smooth over a wide range as demonstrated in Theorem 4, and thus contributes to non-convex optimization.

-2

0

2

X

-3

-2

-1

0

1

2

3

Y

n

=2000,

ω

=5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Optimization objective: -

J

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

Mid-widths

-0.210

-0.175

-0.140

-0.105

-0.070

-0.035

0.000

0.035

0.070

0.105

K.2 Additional experimental comparisons

As mentioned in the main paper, the method 4 [30] is not involved in the comparison because it takes too much time to run in some settings. Here, we compare it with ours under some feasible settings to illustrate the improvement of our method on power-runtime trade-offs. The methods using Gaussian kernel with global width and Gaussian kernel with widths of each dimension (corresponding with ours) are employed, referred to QHSIC-O and QHSIC-W, respectively. For fairness, the same grid search procedure is employed as the initialization of the optimization. We perform the evaluation on the SD data and plot the results of the test power over time as shown in Fig. 8. Also, for our methods, we provide the results under the setting D = 100 and D = 500 as in the main paper. The experiments are conducted with the same equipment, specifically a 6-core CPU with a 3080 GPU.

Results. Our test consistently results in a better power-runtime tradeoff at different D settings. At D = 100 , one test can be completed in less than a second when the sample size n = 6000 . As D increases ( D = 500 ), the number of samples required to achieve the same power decreases, but the increase in D leads to an overhead in runtime, and overall our test is still completed in a few seconds. In contrast, even though QHSIC-O/W requires fewer samples to reach the same power than our tests, the runtime rises rapidly as the sample size increases. When n = 1000 , it already takes more than 10 s to perform a test. When n = 3000 , it needs nearly a minute to perform a test, which may greatly limit the practical application.

0.7

1.0

2.0

3.0

5.0

10.0

20.0

40.0

60.0

Average running time (s)

0.2

0.4

0.6

0.8

1.0

Test power ( ↑ )

n=1000

n=3000

n=1000

n=3000

n=4000

n=4000

n=6000

n=6000

n=2000

n=2000

n=4000

n=4000

QHSIC-O

QHSIC-W

LFHSIC-G D=100

LFHSIC-M D=100

LFHSIC-G D=500

LFHSIC-M D=500

K.3 Running Time

In this part, we evaluate the running time of each method on ISA datasets d = 10 . We set D = 100 and plot the results of the running time versus sample size n as shown in Fig. 9. Shown on the left are the results of tests with O ( n ) and O ( n log n ) time complexity. Since the quadratic time complexity test cannot handle large-scale inputs of 100,000 sample size (excessive runtime and large memory overhead to store the kernel matrix), we evaluate them separately on the right. The experiments are all conducted on the same equipment, specifically a 14-core CPU with a 4090 GPU.

0

20000

40000

60000

80000

100000

Sample size

0

10

20

30

40

50

60

Running time (s)

RDC

NyHSIC

BHSIC

FHSIC

HSICAgg

NFSIC

LFHSIC-G

LFHSIC-M

0

1000

2000

3000

4000

5000

Sample size

0

50

100

150

200

Running time (s)

dcor

QHSIC

QHSIC-O

QHSIC-W

LFHSIC-G

LFHSIC-M

Results. The experimental results on the left show that our method is faster than other methods with optimizable options (HSICAgg and NFSIC), and can complete a test within 10 seconds even with 100 , 000 samples. Even though HSICAgg uses parallelism to optimize the computational efficiency of the scheme, the actual implementation of the parallelism is time-consuming, and hence leads to a high time overhead for a single practical test. For the results on the right, it can be seen that the quadratic complexity methods face a dramatic increase in time overhead as the sample size rises, and this is especially severe for the methods (QHSIC-O/W) that need to be optimized since the optimizing objective needs to perform multiple squared complexity operations. In contrast, our linear-time learning objective allows us to handle huge data samples very efficiently.

L Limitations and Broader Impacts

Limitations. According to the experimental results in the main paper as well as in the Appendix, no one method is better than the others in all settings, so it is important to choose several appropriate tests for real scenarios and summarize their results in order to obtain a more reliable conclusion.

Broader Impacts. This work proposes a novel framework for independence testing. The proposed linear-time optimization objective can be trained end-to-end in a data-driven manner, ensuring both effectiveness and efficiency in high-dimensional and large-scale scenarios. This could be beneficial for developing more reliable downstream algorithms in a variety of areas, including causal discovery, feature selection, and deep learning.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: I've clearly stated the contribution.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See the Sec. L in the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Sec. 5 in the main paper. Also, the summarized assumptions and the proofs are provided in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the details to reproduce the main experimental results, please See Sec. J in Appendix, and we also provide the experimental data/code in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the experimental data/code in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we provide the details. See Sec. J in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experimental results are accompanied by statistical significance tests. See Sec. 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computer resources. See Sec. K in Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: I read it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See the Sec. L in the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the methods used and list the URLs of the comparison methods.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to extend RFF with parametric transforms of the Fourier features of shift invariant kernels. A path similar to the seminal work of Gretton et al 2007 is followed, where the proposed $\\text{HSIC}_\\omega$ statistic is shown to converge in distribution to a Gaussian with the true mean (under the null). Then, linear time estimates of the first two moments are provided, which are needed in vanilla HSIC to analytically compute an approximation to the critical region.\n\nSTRENGTHS:\nThis is a work that has the potential to be a seminal paper in the field of conditional independent testing for large scale data.\n\nThe major improvement is the ability to incorporate modern learning methods in the well-established HSIC framework, while controlling the typically quadratic complexity.\n\nThe claims come with a concentration bound of the proposed statistic, linear time 1st and 2nd moment estimators, a uniform (upper) bound for the convergence of the optimisation and a probabilistic upper bound of Type II error given a lower bound on the dimension, that depends on the dataset.\n\nThis is a simple idea but the supporting theory is far from trivial.\n\nWEAKNESSES:\nThe main weakness of this method is its limitation to translation invariant kernels, which is, however, quite a usual case.\n\nAnother weakness is that the dataset must be split in two parts to avoid the selective inference problem.\n\nFinally, as an extension of HSIC, this method most likely also suffers from some hindrances in the latter, for instance that the Null distribution (or, in specific, the necessary quantile) has to be empirically approximated, say using bootstrapping, or a possibly looser bound needs to be used.\n\nSome comments:\n* Table 1 needs some more space around it to avoid confusion.\n* ln 51: statistics\n* $\\xi_\\omega$ needs to be defined in  Theorem 2, and for completeness also $\\sigma_\\omega$, $\\hat\\sigma_\\omega$.\nPerhaps these could be added in the Appendix as a per-lemma symbol table?\n\nQUESTIONS:\nI had difficulty following the implications of Theorem 2, given its (reasonably) over-packed formulation and I believe that a couple more sentences explaining the involved quantities would make it easier to read.\n\nLIMITATIONS:\nThis work is limited to translation invariant kernels which can be expressed as an integral using the spectral measure, as well as the dataset splitting issue.\n\nA brief discussion could improve the proposed work by making these issues explicit.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method of estimating HSIC to determine the independence of two random variables. The original formulation HSIC_b is reformulated and approximated using Monte Carlo integration, HSIC_w, with frequency samples. This reformulation was not derived by the authors but was borrowed from previous literature. The authors’ contribution is the sampling of frequencies proportionally to the inverse Fourier transform of kernel functions.\n\nSTRENGTHS:\nThe formulations derived in the paper are mostly reasonable. When the inverse transform of the kernel resembles the equation of well-known density functions, HSIC can be approximated using frequency samples. For that kind of kernels, Gaussian (or Mahalanobis) and Laplace kernels are introduced. For other kernels, it is not clear whether their inverse Fourier transformations can result in the form of well-defined probability density functions. Even for Laplace kernel, domain of w_d is not provided, and its density function for frequency sampling is not well-defined.\n\nOriginally, the calculation of a pair of samples could be calculated using a single equation, but it is designed to be calculated using samples. It looks like an unnecessary additional cost, but the calculation of nxn matrix multiplications finally have been converted into the calculation of DxD matrix multiplications as in Eq. 10, and I enjoyed reading the conversion of the equation. The matrix multiplication for the calculation of original HSIC costs O(n^3) with the number of data n, but the proposed method provided an algorithm O(nD^3) with the number of frequency samples D. The calculation should significantly reduce the cost at the expense of the accuracy.\n\nWEAKNESSES:\nThe proposed estimation is novel, but there are several downsides of the paper. \n\nMany explanations are simply provided without proper self-contained information. First, the purpose of constructing criterion J is unclear. The introduction of $c_\\alpha$ is unclear. Second, the definition of Type I error for evaluation is not provided. Type I error is for the false positives. What is the definition of positive decisions in the experiment? What is the test power in the experiment? Those concepts are used without definition.\n\nThe authors used neural networks for T_\\theta x. Conventional HSIC can also use similar transformation of x. For example, k(T_\\theta x, T_\\theta x’) can be calculated directly with a learned transformation T_\\theta x. The results do not necessarily support the superiority of the calculation in the frequency domain.\n\nIn the experiments, I expected to see the consistency of the proposed estimator with large D, which is not provided.\n\nIn theory, the convergence is derived for E[HSIC_w], not for HSIC_b. \n\nMinor comments:\nIn the last line of Algorithm 1, HSIC_b should be HSIC_w.\n\nQUESTIONS:\nCould you provide a detailed description of the objective J, Type I error, and test power used in the paper?\n\nPlease explain if we can use k(Tx_1, Tx_2) instead of using the proposed method.\n\nLIMITATIONS:\nThe authors presented an interesting algorithm, but there are several notions that are unclear.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel consistent estimator for HSIC, which is computationally efficient, and tries to maximize the testing power under controlled type-1 error. The idea is to begin with a known relation between HSIC and Fourier-based distance between the relevant characteristic functions. The estimator can be understood as a sample based approximation of this. For standard kernels, it is noted that the distribution from which the samples need to be generated is from a Gaussian. In general, it is assumed that a transformation of variables converts this distribution as a standard Gaussian. It is shown that this estimator has linear computational complexity, for given transformation parameters.\n\nFurther, it is proposed to learn the transformation's parameters (which in turn is same as learning the kernel parameters). And the objective for learning is proposed to be maximizing test power under controlled t1error. To this end, using asymptotic convergence arguments, an expression for this objective is analytically arrived at (prop1). It is discussed how to estimate this objective again in linear time (theorem1). Consistency of this objective and the overall estimator are are proved in theorems 2,3.\n\nSimulations on synthetic and benchmarks datasets show the improved test power and computational complexity when compared with existing baselines.\n\nSTRENGTHS:\n1. The idea of sample based estimation of the Fourier-characteristic function based HSIC equivalent seems novel. This is also interesting because it leads to linear time complexity.\n\nWEAKNESSES:\n1. It would have been nice if the introduction section had more description of the methodology at an intuitive level. Currently, most of the introduction is motivation, related work and the actual methodology is restricted to one small para. This would give the reader a intuitive understanding and help to understand the later sections easily. Without this it took me some effort to understand what is happening in the overall methodology. (I may be still missing some important points?)\n\n2. The derivations and the main algorithm in section 4.2,4.3 have a very close resemblance with [30] https://proceedings.mlr.press/v238/ren24a/ren24a.pdf . This significantly weakens the contribution and novelty. More importantly, main details in this section could have been cited from [30] or postponed to Appendix. The current presentation seems to give an impression that the sections are entirely new.\n\n3. Since [30] is a close methodology, empirical comparison with it seems to be crucial. However this seems missing.\n\nQUESTIONS:\nfew minor comments:\n\n1. notations at places is a bit confusing: e.g., line 121, 146 w_x means different etc.\n\n\n--- after reading the author responses and other reviews I increase my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}