Absolute Neighbour Difference based Correlation Test for Detecting Heteroscedastic Relationships

1 Introduction . Detecting relationships between variables is a fundamental problem of data-driven discovery . The well-known Pearson ’ s correlation coefficient ( Cor ) , Spearman ’ s rank correlation , and Kendall ’ s tau are concise , computationally efficient , and theoretically well understood , however can capture only linear or monotonic relationships . Developing more powerful association detection methods has been a challenging research and attracted extensive attention in the past decades . A number of techniques have been developed to estimate the score of mutual information ( MI ) as a measure of the dependence between variables , such as bining ( partitioning ) [ 19 , 20 , 11 ] , kernel density estimation ( KDE ) [ 17 , 25 ] , and k-nearest neighbor distances ( kNN ) [ 4 , 13 ] . Distance correlation ( dCor ) is a kind of more general statistics defined on reproducing kernel Hilbert spaces ( RKHS ) [ 9 , 10 ] . It has a compact representation analogous to Cor , but can capture a wider range of associations [ 24 , 23 ] . Randomized dependence coefficient ( RDC ) is a measure of nonlinear dependence between random variables of arbitrary dimension based on the Hirschfeld-GebeleinRenyi maximum correlation coefficient [ 16 ] . Other approaches include maximal correlation [ 21 , 3 ] , kernel canonical correlation analysis ( KCCA ) [ 1 ] , principal curve based methods [ 5 , 6 ] , and nonlinear spectral correlation [ 15 ] . Recently , [ 28 ] introduced an order statistics based association measure called the neighbour correlation coefficient ( nCor ) . With sufficient sample size , nCor is able to detect any type of functional relationships and approximate the R2 of the underlying association . The associations between variables are sometimes too complicated to be quickly and thoroughly identified . Heteroscedasticity is a kind of association pattern that often occurs in real world data and always indicates that an interaction may exist between the independent variable under test and one or more other known or unknown variables . To address this problem , a number of statistical tests have 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . been proposed such as the Park test , the Goldfeld–Quandt test , the Glejser test , the White test , and the Lin-Qu ’ s test in the context of linear , nonlinear , parametric or nonparametric regression analysis [ 18 , 8 , 26 , 12 , 14 ] , and generalized autoregressive conditional heteroscedasticity ( GARCH ) model for time series data [ 2 , 7 ] . Nevertheless , these tests are calculated based on the residuals of regression modelling , so that they all rely on explicitly identifying the functional dependency between variables and sometimes can only capture certain types of heteroscedastic residuals depending on the residual models used . All the aforementioned association detection methods are regression free , but fail to diagnose the heteroscedasticity of data explicitly . Even though some approaches such as MI estimation are able to measure the total dependence between variables , they still can not distinguish heteroscedasticity ( nonfunctional relationships ) from functional relationships . Here , we propose a new association measure called absolute neighbour difference based neighbour correlation coefficient ( nCor|∆| ) to examine whether the statistical dispersion of a variable is unequal across the range of values of a second variable that predicts it . It can be used together with other association measures to separately detect the impacts of the independent variable on the expect value and variance of the dependent variable , that will obviously lead to a better understanding of the data . The rest of this study is organized as follows . Sections 2 and 3 briefly review the background of data association and heteroscedasticity , and introduce a new correlation test for examining whether a data relationship is heteroscedastic or not . In Section 4 , empirical studies are performed to evaluate the new statistics and make comparisons with previous approaches . Finally , in Section 5 , conclusions are drawn to summarize the study . To facilitate the readability of the study , all the relevant theoretical proofs are enclosed in appendices in supplementary material . 2 The proposed method . 2.1 Heteroscedasticity and interaction . Aside from the central tendency , independent variables may sometimes influence the dispersion of dependent variable , which indicates the presence of heteroscedasticity . Consider the general form of a relationship between ( X , Y ) expressed as follows . Y = E [ Y |X ] + E = E [ Y |X ] + √ Var ( E|X ) Θ = f ( X ) + g ( X ) Θ ( 1 ) where Y ∈R , X = ( Xi|1≤i≤M ) ∈RM , and E∈R respectively denote dependent variable , independent variables , and additive noise . Θ∈R denotes the normalized noise that is defined as Θ = E/ √ Var ( E|X ) so that E [ Θ ] = 0 and Var ( Θ ) = 1 . E [ ·|· ] and Var ( ·|· ) denote conditional expectation and conditional variance operators respectively . f ( · ) and g ( · ) are the two functions respectively characterize how the expectation of Y and the standard deviation of E are expressed conditional on X . For the special case that E is homoscedastic , Var ( E|X ) = Var ( E ) , and g ( X ) becomes a constant . Here , we assume that all the Xi are independent to each other , and all the functional relationships either are continuous at all points of their domains or have discontinuities of the first kind . Heteroscedasticity is a kind of unpredictable dependence which is also referred to as nonfunctional relationship . As defined in ( 1 ) , it is the nature and manifestation of the interaction of ( X , Θ ) in which the simultaneous influence of the two variables is not additive . In such a situation , X may distinctly influences the statistical dispersion of E ( which equals to Y with its predictable part removed ) , but nevertheless E can not be predicted from X alone by using any function , that is the reason why it is so-called nonfunctional . Interactions can be diagnosed by using multivariate tests , such as nCor , dCor , and MI , but only when performed on the exact set of interactive variables . Hence , they are always computational costly to be detected exactly in real applications especially when dealing with large datasets in which there are too many variable subsets that need to be examined one by one . When an interaction occurs , the pairwise relationship between the dependent variable and each interacting independent variable often appear to be heteroscedastic and at least partially nonfunctional . That is to say , a concise heteroscedasticity test may assist in detecting the interactivity of variables which can be described as whether and how much one variable impacts on anther variable through interaction effect . Such a pairwise examination could quickly find out the interacting variables , and does not require any prior knowledge about the subset partitioning and inter-dependencies of the variables . The detected variables , then , can be used preferentially to perform further multivariate tests for measuring the exact interaction effects . It will obviously reduce the computational cost and make the following detection and analysis more directional and efficient . 2.2 The population nCor|∆| . If a functional relationship exists between ( X , Y ) , two observations in the domain of f ( · ) should have similar values of f ( X ) when the difference between their values of X are sufficiently small . Suppose there is a set of variables X′ = ( X ′1 , . . . , X ′M ) that X ′i = Xi + ∆Xi , ∀1≤i≤M ( 2 ) where each ∆Xi denotes a random variable representing an increment of Xi with extremely small absolute value . Let Λ = ‖∆X‖ = ‖X′ −X‖ be the Euclidean norm . It is obviously that E [ Λ ] can also be made arbitrarily small when each E [ |∆Xi| ] is sufficiently small , and vice versa . In this sense , X′ can be considered as a near neighbour of X . Correspondingly , the neighbouring dependent variable can be derived as Y ′ = f ( X′ ) + E′ = f ( X ) + ∆f + E′ ( 3 ) where E′ is an independent copy of E , that is , E and E′ are independent and identically distributed ( i.i.d. ) . ∆f denotes the difference between the two noise free dependent variables which can be computed as ∆f = f ( X′ ) − f ( X ) . Then , the neighbour difference ∆Y = Y ′ − Y should have the following property . Lemma 1 . Consider a relationship between ( X , Y ) as given in ( 1 ) . Let random neighbouring variables ( X′ , Y ′ ) be defined as above . Then , it holds that almost surely lim E [ Λ ] →0 E [ ∆Y ] = 0 lim E [ Λ ] →0 Var ( ∆Y ) = 2Var ( E ) = 2E [ g ( X ) 2 ] ( 4 ) Figure 1 shows an example where the heteroscedasticity of one variable arises from the interaction of two other variables . When we investigate the bivariate relationship between Y and either X1 or X2 , the other one can be naturally viewed as an unknown noise . It is clearly that when ∆Xi approaches 0 , the expectation of |∆Y | implies the conditional dispersion of Y given Xi with E [ Y |Xi ] removed . That is to say , the heteroscedasticity of E can be detected by examining whether the expectation of |∆Y | is a function of Xi or a constant . Based on this concept , we propose a new correlation test to detect whether a g ( · ) exists between ( X , Y −f ( X ) ) or not , without modelling either f ( · ) or g ( · ) explicitly . Let X′′ and X′′′ be other two neighbouring variables defined as below . { X′′ = X′ + ∆X′ X′′′ = X′′ + ∆X′′ ( 5 ) where ∆X′ and ∆X′′ are random increments defined the same as ∆X . The corresponding dependent variables are defined as { Y ′′ = f ( X′′ ) + E′′ = f ( X′ ) + ∆f ′ + E′′ Y ′′′ = f ( X′′′ ) + E′′′ = f ( X′′ ) + ∆f ′′ + E′′′ ( 6 ) where E′′ and E′′′ are independent copies of E. Consider the difference ∆Y ′′ = Y ′′′ − Y ′′ . By Lemma 1 , we have that almost surely when E [ Λ′′ ] → 0 , if E is heteroscedastic , then Var ( ∆Y ′′ ) → 2E [ g ( X′′ ) 2 ] , otherwise Var ( ∆Y ′′ ) → 2Var ( E′′ ) . Then , by taking advantages of the mechanism of nCor , a new association measure is defined as below . Definition 1 . The absolute neighbour difference based neighbour correlation coefficient ( nCor|∆| ) when applied to a population is defined as nCor|∆| ( X , Y ) = Cov ( |∆Y | , |∆Y ′′| ) √ Var ( |∆Y | ) Var ( |∆Y ′′| ) ( 7 ) Theorem 1 . Consider a relationship between ( X , Y ) as given in ( 1 ) . Let ( X′ , Y ′ ) , ( X′′ , Y ′′ ) , and ( X′′′ , Y ′′′ ) be defined as above . Let Λ denotes the vector valued Euclidean norms { ‖∆X‖ , ‖∆X′‖ , ‖∆X′′‖ } . The population nCor|∆| ( X , Y ) has following properties . First , |nCor|∆| ( X , Y ) | ≤ 1 . Second , if E is homoscedastic , then nCor|∆| ( X , Y ) = 0 . Third , if E is heteroscedastic , then almost surely lim E [ Λ ] →0 nCor|∆| ( X , Y ) = Var ( g ( X ) ) E [ |Θ ′ −Θ | ] 2 2E [ g ( X ) 2 ] − E [ g ( X ) ] 2 E [ |Θ ′ −Θ | ] 2 > 0 ( 8 ) See Appendices C and B in supplementary material for the proofs . Remark 1 . ( 8 ) clearly suggests that when Λ → 0 , the value of nCor|∆| is determined only by g ( X ) , regardless of whether and what type of f ( X ) exists . This is contrary to the nCor since limE [ Λ ] →0 nCor ( X , Y ) = Var ( f ( X ) ) /Var ( Y ) = R 2 [ 28 ] , that is , the value of nCor is dependent only on f ( X ) but completely insensitive to g ( X ) . The two tests have distinct but complementary capabilities , and therefore can be used to respectively examine to what extent X influences the central tendency and the statistical dispersion of Y . In orther words , one is for measuring functional relationship , and the other one is for detecting nonfunctional relationship .

Summary: This paper proposed a new statistical measure, called absolute neighbour difference based correlation test, to detect the existence of heteroscedastic noise in the residual of a functional relationship.   Theorem 1 shows the proposed measure nCor_Delta is only affected by the dispersion function of the residual g(x) and not affected by the functional relationship f(x). As a result, compared to those existing statistical test of heteroscedastic noise, the major advantage of the proposed measured is to avoid fitting the regression functions. The estimator of the measure on real-world datasets is also presented in Definition 2. The statistical test of heteroscedastic noise vs. homoscedastic noise is presented in Theorem 2.   In the experimental results, the proposed nCor_Delta is firstly compared against two existing tests of heteroscedastic noise, including Parker test and White test. It shows the nCor_Delta can correctly test for heteroscedastic noise while its competing alternatives either fail on some datasets or uses over-complicated regression functions.  The proposed nCor_Delta is also evaluated against a few existing non-linear dependency measures that are not designed to test heteroscedastic noise, including dCor, RDS, MIC, MI and nCor. These measures cannot detect heteroscedastic noise as expected.