Absolute Neighbour Difference based Correlation Test for Detecting Heteroscedastic Relationships
Authors: Lifeng Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: association measure
heteroscedasticity
nonfunctional dependence

Abstract: It is a challenge to detect complicated data relationships thoroughly. Here, we propose a new statistical measure, named the absolute neighbour difference based neighbour correlation coefficient, to detect the associations between variables through examining the heteroscedasticity of the unpredictable variation of dependent variables. Different from previous studies, the new method concentrates on measuring nonfunctional relationships rather than functional or mixed associations. Either used alone or in combination with other measures, it enables not only a convenient test of heteroscedasticity, but also measuring functional and nonfunctional relationships separately that obviously leads to a deeper insight into the data associations. The method is concise and easy to implement that does not rely on explicitly estimating the regression residuals or the dependencies between variables so that it is not restrict to any kind of model assumption. The mechanisms of the correlation test are proved in theory and demonstrated with numerical analyses.

Absolute Neighbour Difference based Correlation Test for Detecting Heteroscedastic Relationships
Lifeng Zhang
School of Information
Renmin University of China
59, Zhongguancun street, Haidian, Beijing, P.R.China 100872 zhang@ruc.eduu.co
Abstract
It is a challenge detect complicated data relationships thoroughly. Here, we propose new statistical measure, named the absolute neighbour difference based neighbour correlation coefficient, to detect the associations between variables through examining the heteroscedasticity of the unpredictable variation of depen- dent variables. Different from previous studies, the new method concentrates on measuring nonfunctional relationships rather than functional or mixed associations. Either used alone in combination with other measures, enables not only a convenient test of heteroscedasticity. but also measuring functional and nonfune- tional relationships separately that obviously leads to a deeper insight into the data associations. The method is concise and easy to implement that does not rely on explicitly estimating the regression residuals or the dependencies between variables that it is not restrict to any kind of model assumption. The mechanisms of the correlation test are proved theory and demonstrated with numerical analyses.
1 Introduction
Detecting relationships between variables is fundamental problem of data-driven discovery, The well-known Pearson's 's correlation coefficient (Cor), Spearman's 's rank correlation, and Kendall's tau are concise, computationally efficient, and theoretically well understood, however can capture only powerful association detection methods has been
linear monotonic relationships. Developing a challenging research and attracted extensive attention the past decades.
A number of techniques have been developed to estimate the score of mutual information (MI) as
a measure of the dependence between variables, such as bining (partitioning) [19
20
11] kernel
density estimation (KDE) [17 25), and k-nearest neighbor distances (kNN) 4
13
Distance
correlation (dCor) is a kind of more general statistics defined on reproducing kernel Hilbert spaces
[0 It has compact representation analogous to Cor. but can capture wider range Randomized dependence coefficient (RDC) is a measure nonlinear dependence between random variables of arbitrary dimension based the Hirschfeld Gebelli-n--eee----- Renyi maximum correlation coefficient [66]. Other approaches include maximal correlation [1 3), kernel canonical correlation analysis (KCCA) M. principal curve based methods 6. and nonlinear spectral correlation [15]. Recently, (28 introduced an order statistics based association measure called the neighbour correlation coefficient (nCor). With sufficient sample size, nCor is able to detect any type of functional relationships and approximate the R² the underlying association.
(RKHS) 9
of associations 24 2X
The associations between variables are sometimes too complicated to be quickly and thoroughly identified. Heteroscedasticity is a kind of association pattern that often occurs in real world data and always indicates that an interaction may exist between the independent variable under test and one or more other known or unknown variables. To address this problem, a number of statistical tests have
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

been proposed such as the Park test, the Goldfeld- Quanntt test, the Glejser test, the White test, and the Lin- Qu's 's test in the context linear, nonlinear, parametric or nonparametric regression analysis 18
g 26 12 [4 and generalized autoregressive conditional heteroscedasticity (GARCH) model
for time series data 2
Nevertheless, these tests are calculated based on the residuals of regression modelling, so that they all rely on explicitly identifying the functional dependency between variables and sometimes can only capture certain types of heteroscedastic residuals depending on the residual models used. All the aforementioned association detection methods are regression free, but fail to diagnose the heteroscedasticity of data explicitly. Even though some approaches such as MI estimation are able to measure the total dependence between variables, they still cannot distinguish heteroscedasticity (nonfunctional relationships) from functional relationships.
Here, we propose new association measure called absolute neighbour difference based neighbour correlation coefficient (nConra) A)) to examine whether the statistical dispersion of a variable is unequal across the range of values of a second variable that predicts it. It can be used together with other association measures to separately detect the impacts of the independent variable on the expect value and variance of the dependent variable, that will obviously lead a better understanding of the data.
The rest of this study is organized as follows. Sections and 3 briefly review the background of data association and heteroscedasticity. and introduce new correlation test for examining whether data relationship is heteroscedastic or not. In Section 4, empirical studies are performed to evaluate the new statistics and make comparisons with previous approaches. Finally, in Section 5, conclusions drawn to summarize the study. To facilitate the readability of the study, all the relevant theoretical proofs are enclosed in appendices in supplementary material.
2 The proposed method
2.1 Heteroscedasticity and interaction
Aside from the central tendency, independent variables may sometimes influence the dispersion of dependent variable, which indicates the presence of heteroscedasticity. Consider the general form of relationship between (X, Y) expressed as follows.
Y || + E = [[Y|X] VV [()) g(x)
(1)
where YER, X = (X. and EER respectively denote dependent variable, inde- pendent variables, and additive noise. OER denotes the normalized noise that is defined as Va)(EX) so that E[Ø]= = and Var(@) = E[||-] and Var(-I-) denote conditional expectation and conditional variance operators respectively. f(-) and g(-) are the two functions respectively characterize how the expectation of Y and the standard deviation of F are expressed Var(E), and gXX)
conditional on X. For the special case that E is homoscedastic, Var(E|X) becomes a constant. Here, we assume that all the X, are independent to each other, and all the functional relationships either are continuous at all points of their domains have discontinuities of the first kind.
Heteroscedasticity is a kind of unpredictable dependence which is also referred to as nonfunctional relationship. As defined in (1), it is the nature and manifestation of the interaction of (X, Q) in which the simultaneous influence of the two variables is not additive. In such a situation, X may distinctly influences the statistical dispersion of E (which equals to Y with its predictable part removed), but nevertheless F. cannot be predicted from X alone by using any function, that is the reason why it is so-called nonfunctional.
Interactions can be diagnosed by using multivariate tests, such as nCor, dC'or, and MI, but only when performed on the exact set of interactive variables. Hence, they are always computational costly to be detected exactly in real applications especially when dealing with large datasets in which there are too many variable subsets that need to be examined one by one. When an interaction occurs, the pairwise relationship between the dependent variable and each interacting independent variable often appear to be heteroscedastic and at least partially nonfunctional. That is to say, a concise heteroscedasticity test may assist in detecting the interactivity of variables which can be described as whether and how much one variable impacts on anther variable through interaction effect. Such pairwise examination could quickly find out the interacting variables, and does not require any

---

prior knowledge about the subset partitioning and anter-epennenncier of the variables. The detected variables, then, can be used preferentially perform further multivariate tests for measuring the exact interaction effects. It will obviously reduce the computational cost and make the following detection and analysis more directional and efficient.
The population nCCoral
If functional relationship exists between (x. Y). two observations in the domain of f(-) should have similar values of f(X) when the difference between their values of X are sufficiently small. Suppose there of variables X'
(.,,..., ..................... MM) that
X X; AX, VISISM
where each A.; denotes random variable representing increment of X; with extremely small absolute value. Let = || = X x|| || be the Euclidean norm. obviously that [E[A] can also be made arbitrarily small when each E|A., [X|]]] is sufficiently small, and vice versa. In this sense, X' can be considered as near neighbour of X. Correspondingly, the neighbouring dependent variable can be derived as
Y' = +'= +f
(3)
where E' is an independent copy of E, that is, E and E' are independent and identically distributed (i.i.d... Aʃ denotes the difference between the two noise free dependent variables which can be computed as = f(X') f(X). Then, the neighbour difference AY - should have the following property.
Lemma 1. Consider relationship between (X,Y) given in (1). Let random neighbouring variables (X', be defined as above. Then, it holds that almost surely
lim E| =
B[A
lim Var( = 2Var(E) = 2X{} B[A|-0
Figure shows an example where the heteroscedasticity one variable arises from the interaction of two other variables. When investigate the bivariate relationship between Y and either X₃ or X2, the other one can be naturally viewed as an unknown noise. It is clearly that when AX; approaches 0, the expectation of |AY| implies the conditional dispersion of Y given X, with E[Y]X; removed. That is to say, the heteroscedasticity of E can be detected by examining whether the expectation of |A| is function of X; constant.
X,
x
(a) The surface versus Xi and X₂
(b) The scatterplots of Y versus Xi and Xa
X
(c) The scatterplots of AY versus X1 and X²
(d) The scatterplots of |AY| versus X1 and X₂
The surface and scatterplots of = (1 X2) where X, E [-1, 1]. The grey shaded areas and red lines respectively indicate the dispersion of the corresponding variables and their conditional expected values.
Figure 1:
3

---

Based on this concept, we propose new correlation test to detect whether 9(-) exists between (X, Y not, without modelling either f(-) or 9(-) explicitly. Let X and X" be other two neighbouring variables defined as below.
X AX'
(5)
X =X + AX"
where AX' and AX" are random increments defined the same AX. The corresponding dependent variables are defined as
X-- = = ( ==" f(X) A + E" f(X") A E
(6)
y= I(x") E = =
where E" and E" are independent copies of E. Consider the difference AY" Y"" YY" By Lemma 1, we have that almost surely when E[A" + 0, if E is heteroscedastic, then Var(AY") 4 ())",, otherwise Var( (") + Var E""). Then, by taking advantages of the mechanism of nCoo, new association measure is defined as below.
Definition 1. The absolute neighbour difference based neighbour correlation coefficient (nCor }
when applied population is defined
(X, Cov(|AY),
Cor(IAY],AAY)) |AY"))
(7)
Theorem 1. Consider a relationship between (X, as given (I). Let (X,Y"), (X", Y"). and (X" y")) be defined as above. Let A denotes the vector valued Euclidean norms [|aAX|] II, AX [|]" The population nCCona((XXY) has following properties. First, InCor A [X, Y) S Second, if E is homoscedastic, then (X, = Third, if E is heteroscedastii, then almost surely
Var (g( B 01)²
lim (CCona(X,Y)) B[A]-O
x - E (x) 0112 >0
(8)
See Appendices C and B in supplementary material for the proofs.
Remark 1. (8) clearly suggests that when A + 0, the value of nCorja) is determined only by g(X), regardless of whether and what type of f(X) exists. This is contrary to the nCor since limgja]-oo nCoroX,
R2 [28), that is, the value of nCor is dependent only on f(X) but completely insensitive g gXX). The two tests have distinct but complementary capabilities, and therefore can be used to respectively examine to what extent X influences the central tendency and the statistical dispersion Y. In orther words, one is for measuring functional relationship, and the other one is for detecting nonfunctional relationship.
Var (f(X)) /Var)/rr(()
3 The nCorjai test for a sample
Let M()), (X) B(N)) be the paired sequences of N independent observations from the joint distribution of (X, Y). For computing nCor]ai |A) the realizations of the dummy variables (X', Y"), (X", y"), and (X" y"") can be obtained by resampling from the original dataset, based on the criterion of minimizing the vector valued norms X.
3.1 Data reordering
In this study, we use the same data reordering methods as used for computing nCor in 28 to generate neighbouring data sequences. The data reordering based resampling process ensures each data point has an equal opportunity be used in computing the correlation values, and thereby avoids the appearance of a few super data points whose values dominate the resampled sequences due to being copied many times. this process, each data point copied four times, respectively, as the observations of the four sets of neighbouring variables. Therefore, the four new samples should obey an identical joint distribution as the original one, which perfectly comply with the concept of neighbouring variables given in Section 3.

---

In the bivariate case where M = 1, the concepts of order statistics and concomitants are presented as follows. First, sorting (z) } in an increasing order of its values to yield a new sequence denoted by 211:N) (N:NN)) where I(K:N) is known the k-th order statistic of { In addition, let fng I=kINj denotes the reordering permutation, that is, if n& = t then I(t:N) =(())- Second, rearranging dependent variable data in accordance with {n₂} to obtain У[:N] M/::M): ... N[ where V|]:N] is known as the k-th concomitant, and can be defined as 9[k:N] У(t) when 72 t.
sequence
For multivariate data (M > 1), the optimal reordering permutation can be obtained by minimizing the total distance between each pair of neighbouring data points in X space derived as follows.
fnill k SkS
arg min "16(1,2 N), VIVEN Peginn Mipy
( X(m))l
The nearest neighbor (NN) algorithm, then, can be used to obtain the optimal or most likely sub-optimal permutation } for generating (x(::) and }. The NN algorithm is a kind of constructive search heuristic that is easy to implement and executes quickly. It starts one data point, then visits the data point that is nearest to the starting point. Afterwards, it visits the nearest unvisited data point, and repeats this process until all data points have been visited. Finally, it outputs the visiting path as the permutation }n for data reordering.
3.2 Data resampling and correlation test
Let rearranged data sequences ((te,)) (((+++1NN) V)e+1:N]) ((x(1+22)) 31++2:N)) and {(x(++3)N) }[]+3:]])} be considered as the observations of Y), Y"), (x", Y"), and
(x", Y") respectively. Then, the neighbour difference can be calculated as AyjaN V[++1:N] {[::)] where S k N 3.
Figure 2 shows four examples that the absolute difference, especially conditional expectation, provides a more explicit indication of the heteroscedasticity of the data than the direct investigation of the relationship between (I., y)
% (a) Example (g(x) 0.3)
(b) Example 2 (g(z) 0.22 + 0.2z)
Ua parrs
(c) Example (g(x) 0.3 cos(mz))
(d) Example (g(x) (.2sig)(cos(Tz2))
Figure 2: The scatterplots of the dependent variable and absolute difference for four simulated examples with sample size 200. The examples can be generally represented as y = B (fx)))))))))))) g(z)0, where U(-1,1) and 0 NN.,,) 1). The red lines indicate the conditional Expectations.
Let (xx() ((tss be observed from a relationship as defined in (1), and Ay be computed with the optimal reordering permutation as defined (9). It holds that almost surely,
Lemma 2.
lim Ay = 0 N
(10)
lim var( (gy 2var(e) 2g(x)² +20
where the overbar and var(-) respectively denote the mean value and sample variance of the sequence.
5

---

Definition 2. Let { and { {Bin}t denote the absolute neighbour difference sequences that is defined as that for any 1 I k I N
Am Ny 11+++:N1 3[k:N]]
(11)
BB AWKKNNss 181K+3:N 3[++2:N]
The nCorpa |) when applied to a sample is defined as
B (N 33AB
nCooja(Xx
(12)
N-3 Σ Alw (N F)
N-3 Σ
B Bly
(N 3)
V
Theorem 2.
Let {Ap, and {B be the absolute neighbour differences computed as given in (11).
A hypothesis test rejects the null hypothesis of homoscedastic if
- nCor(x, y) > taah (5-1(1-a)/v o//1-//V(NN)) 6)
(13)
where denotes the significance level, and Φ(-) is the standard normal cumulative distribution function. If is heteroscedastic that a g(.) exists between x and Var(e), then almost surely
var(g(x))
lim nCorja((X,)) 2g(x)² g(x) N
0
(14)
8
See Appendices and D in supplementary material for the proofs. In summary, the proposed method a very concise association measure and only requires three computational steps, that is, reordering the sample points by order statistics or NN algorithm, generating the absolute neighbour difference sequences by (11), and then computing nCorja) [A] by (12).
Remark 2. (i) nCoorAy ||] is designed to detect the heteroscedasticity of the underlying noise approxi- mated by the absolute neighbour difference, rather than the residuals of any regression model. Hence, it does not require any kind of parametric nonparametric modelling process. (ii) In most cases, pairwise nCoraa | test may be sufficient for detecting nonfunctional relationships between variables. be used to quickly examine the heteroscedasticrry arising from multiple x; in the context of multivariate regression. When dealling with complex g(-) and high dimensional may exhibit less detection power (as demonstrated in section 4. 1. (ii) The proposed test requires comparatively large amounts of samples to ensure the overwhelming majority of data points being close to each other in x space, otherwise it may display less detection power. In general. several hundreds of data points might be sufficient for conducting the test (as illustrated in section 4.1 and Appendix E in supplementary material).
Multivariate nCorja
Remark 3. When dealing with high-dimens datasets. finding interacting variables is one of the most important challenges in feature selection and has received wide attention (27)22). Interaction effect kind of multivariate functional dependence which cannot be directly identified by any pairwise association measure. Since heteroscedasticin always arises from interaction, the bivariate nCor(s) test can also be viewed as convenient method for detecting interacting variables in the multivariate case. As demonstrated in Appendix G in supplementary material, we can use mCor | (I;, v) to examine the heteroscedasticity of each pair of (I,, y) for screening the potential interacting {2z}, and subsequently conduct the multivariate nCor and the COI tests as introduced in (28) to identify the exact interactions on the reduced independent variable subset that only contains the potential interacting {sa}- This procedure will be obviously much more efficient than identifying interactions through detecting the full dataset.
4 Empirical studies
4.1 Effectiveness, consistency, and robustness
In this section, we conducted set of numerical experiments performed on artificial data to evaluate the performance of nCooraa as well as to make comparisons with previous approaches. The four examples given in Figure 2 were employed as the trial functions. Three data distributions (uniform,

---

x-U
X-V
E-E
x-U
x-VV
Average P vallue
White:
E.
B.4
White: E.
2.2
E.
Avenge
10
16ng
2000
20g Neg 1000
2000
20p
900
Hoog
2000
200
1000
200g
20g
Hoop
2000
200
R00
1006)
(a) The average p-values and the nonlinearities obtained by using the Park test and the White test
R²-0.0
0.4
R²-0.8
R20000
R²-0.4
R-0.8
E.2
012
200
300
10n)
2000
200
300
1000
2000
200
300
1000
2060
200
300
1000
2000
20g
500
1000
2000
200
300
2000
2500
04
E.4
X-U
N
X-E
200
500
1000
2000
200
30g
100
2000
200
1000
2000
20g
500
100g
2000
200
-00
1000
2000
200
300
1000
(b) The box plots obtained from using nC Cororaa
8² =²=0.0
R=404
R²=0.8
R²=0.0
R²=044
=²=0.8
0.5
0.5
RDC
200
500
(00M)
2000
20g
50g)
1000
2000
20g
50g
100g
2188)
500)
1000
2000
200
500
1000
2000
20g
50g
(00)
0.8
0.5
MI
0.4
04e
200
S0g
10x)
2000
20g Stn
(ong
300
200
Stn
100g
20g
20x)
Styy
100g)
2000
200
Stn
2000
300
Ste
100
U-U
OFF
nCoor
100g 200
20g Sty
200
S00
20g
Siy
10ng
2016
(c) The average scores obtained from using dCor, RDC, MIC, MI, nCor, and nCor
Figure 3: The detection results obtained from using the previous heteroscedasticity tests, association measures, and the proposed method for the four examples given in Figure (E. 1 to E. 4). Three data distributions (U(-1, 1). (0, 1). and E()) and four sample sizes (from 200 to 2000) were implemented in these experiments. In (b), the grey shaded areas indicate the 95% confidence intervals.
normal, and exponential), four sample sizes (200, 500, 1000, and 2000), and three R² (0, 0.4, and 0.8) were considered. For each case, 50 random datasets were generated for conducting the detection.
First, two widely used heteroscedasticity tests, the Park test and the White test (by using polynomials of degree from to 15, and then selecting the lowest p-value as the final result), were performed to verify the homogeneity of residuals obtained by assuming the best fitted regression model (ıi = 3 sin(Tz)) was unknown. Figure 3(a) shows that due to the fixed model structure, the Park test failed to reject the hypothesis of homoscedasticity in examples 3 and 4. The nonlinear polynomial based White test successfully detect all the heteroscedastic residuals, however, rather high degree
7

---

of nonlinearity is needed. Especially in example 3, when using normally distributed r only by the polynomials with degree of k >> 10 can the heteroscedasticity be detected as significant at p S 5%. The disadvantage of the two tests and many other approaches of the same kind is that, they all require properly estimated regression residuals and well-specified model assumption so that they are obviously not suitable for quick and convenient association detection.
Second, as shown in Figure 3(b), nCoMAl successfully detected the heteroscedasticity of the data whatever type of underlying g(x) applied. The vast majority of nCorjaa |a] values are inside the 95% confidence interval in Example 1, and contrarily, the correlation scores are significant in the other three examples. The figure clearly suggests that nCoonal is consistent measure that has statistical power increasing as the sample size increases in every case. Moreover, nCooss is quite robust to both data distribution and 1t² as it exhibited very similar performance with respect to the statistical power under different situations.
To further assess the performance of the proposed method in the extreme situations of small sample size (N 30, 50, 100, 150) and extremely small noise (R² 0.9, 0.99, 0.999), more experiments were conducted and the results can be found in Appendix E in supplementary material. The experiment results confirm that nCorjJ may exhibit less detection power under smaller sample sizes, in which case the data points are not guaranteed to be within close distance of each other in x. Moreover, the occurrence of outliers (which also lead to large distances between reordered data points) accompanied by extremely large Ri² may make the proposed test to be prone to inflated type I error. The experiment results suggest that this problem can be addressed by removing the isolated data points that have extreme values of x from the datasets for nCorjal computation.
Third, nCor and four typical and widely accepted association measures including dCor, RDC, MIC, and KNN based MI estimator were also implemented for comparison purpose. Here, the MI scores were re-scaled the range of [0, 1] 1 exp( 22M y)). (i) As shown in Figure 3(c), despite being able to detect g(z) when R² = O. dCor and RDC displayed less and less detection power as R² increases until becoming heteroscedasticity insensitive when R² 0.8. (ii) The scores of MIC and MI in the last three examples are perceptibly greater than that in Example I even though R2 0.8, which should reflect both the dependencies of f(x) and g(x). Despite measuring the total dependence between (x,y). y). they still cannot distinguish between the impacts of x on the expectation and variance of y. in other words, to what extent the dependence is functional (predictable) or not. For example, the scores of MI for E.2 with R² = O are very similar to that for E. with R = 0.4. Actually, the first dependency is completely unpredictable, and the second one is just the opposite. Both MI and MIC failed to diagnose these conditions. (iii) nor properly measured the R² of all the functional relationships, but was completely insensitive to the heteroscedastic relationships. In contrast, nCorja) showed stable performance on detecting heteroscedasticity whatever R² is. Clearly, the two tests are ideally complementary to each other, and outperforms the other methods.
Finally, we extended the four examples to multivariate case as given in Figure 4. The figure shows that the detection power of nCorjai decreases as the dimension of x increases, and varies with different g(-) and R² especially when M is large and f(-) is highly nonlinear. This situation can be gradually improved by increasing the sample size, however, it still cannot perform as well as nCor (see Appendix E). This because, g()) is more difficult to detected than f()) since g(-) blended with 0 through interaction. In these experiments, moreover, each x; has exactly the same impact on y, which leads to another difficulty that there is no individual heteroscedasticity that dominates the entire relationship. As shown in Appendix E, with large M and nonlinear g(-), the heteroscedastic relationships become too blurred to be diagnosed even by a visual inspection the scatterplots.
nCory (N=200)
0.4
nCorg (N- 1000))
0.4
nCorg (500000)
0.4
E. (000 - E. (R==0.0)
E. (0
- E. -R
0.2
0.
0.2
E. (R -0-8)
E (R
E. (00-0)
- E. (R8-0.0)
E. ()(-4)
F. (PF-
10
2
10
1
10
2 (12-88)
R. (1F-18)
Figure 4: The average nCor ( ... MM. 3) scores for y = 3 EM
f(x)) + fxx))
-M g(xz) where
f() and g()) were defined the same as in Figure 2, and #1 to FM were randomly generated with uniform distribution and data length of 200, 1000, 5000.
8

---

4.2 Detecting complex nonfunctional relationships
In this section, two functional and six nonfunctional associations were employed to further evaluate the proposed method. These data associations were originally introduced by Reshef et al. [19], and most of them were inspired by the patterns found in real-world data, such as line & parabola, ellipse. and non-coexistence. All the data sequences were generated with length of 10000, and corrupted by uniform noises. the first two experiments, both horizontal and vertical noises with different variances were applied, that is, not only y but also z contain noises. In the last two experiments, noises were separately added into x and y.. For more details see Appendix in supplementary material. Figure 5 depicts the simulated data, and the nCor and nCor A test results.
For D3-, D4-, D6-, D7-, and D8-, whatever type of noise was applied, the nCor A test results suggest that they are heteroscedastic so that there exist strong nonfunctional relationships in these datasets. The nCor test also yielded considerable scores, that is to say, functional relationships coexist so that there is merely part of the variation of each that can be predicted from x. For D5-, all the nCor || scores were significant, and meanwhile, the nCor values felled below the 95% confidence limit. This means that is completely unpredictable even though an obvious dependence exists between (x. v). this case, the central tendency of y should be a straight line parallel to x axis. From another perspective, the detection results can also be explained as that in these datasets x influences y through interaction to some extent with other unknown variable(s). As shown in Appendix F, actually, there is another interacting variable in each case that is invisible but indeed also impacts on y. Whenever known, the relationship will become fully functional and predictable. In brief, the two tests detected all the relationships in the datasets, and properly revealed the composition of the data associations.
Di-xys
D2-xyS
D3-xyS
D4-xSS
D5-xyS
D6-xyS
D7-xyS
D8-xyS
El-xyS
E2-xyS
n
L
6
D1-xyL
D2-xyL
D3-xyL
D4-xyL
D5-xyL
D6-xyL
D7-xyL
D8-xyL
el-xyL
e2-xyL
L
X
O
D1-yl.
D2-yl.
D3-yl.
D4-yt.
D5-yl.
D6-yl.
1D7-yl.
D8-yl.
el-yl.
22-yl.
W
X
MM
D1-xI.
D2-xL.
D3-xL.
D4-xL.
D5-xL
D6-xI.
D7
D8-xI.
82-x1.
L
X
(a) The simulated data (horizontal and vertical correspond to and y)
(b) Residuals
0.4
xyS
XyL
yL
xL
nCor
nCor
0.5-
0.2
D1
D2
D3
D4
D5
D6
D7
D8
D1
D2
D3
D4
D5
D6
D77
D8
(c) The detection results obtained from using nCor, and nCCor
Figure 5: The scatterplots of the types of bivariate associations corrupted by types of uniform noises. The noise free relationships of DI- and D2- are functional, and the rest are nonfunctional. In D -xxyS and -xyI., both horizontal and vertical noises were added to the data with different noise levels (S and L denotes small and large respectively), by contrast, only vertical noise was added in D D-LL, and only horizontal noise was added in D -xL. The red lines in (a) represent the prediction results for the functional relationships obtained by using the best fitted ANN models (3-layers, hidden neurons for DI- and 7 hidden neurons for D2-), and the regression residuals (c) are given in (b). (c) shows the detection results, and the grey shaded areas represent the 95% confidence intervals.
For the functional datasets, DI- and D2-, the nCor values far exceeded the confidence limit. is noted that the nCoriai tests indicate that the underlying noises in four of the eight datasets

---

heteroscedastic. It may seem a little counterintuitive that the scatterplots of D2-xyS, D2-xyL, and D2-xL all look more even than that of D2-yL. but nevertheless, by nCorja only D2-yL was judged to be homoscedastic and purely functional. This is because in the three datasets the observed was corrupted by horizontal noise Ez, which may lead to an extra interaction term as y = f(x) + M(x,,, 6z) + Cy. Since ez is unknown, h(-)) becomes a heteroscedastic component of the underlying noise and will be remained in regression residuals. Similarly, the test results shows that DI-xL is heteroscedastic, and the heteroscedasticity of DI-xyS and DI-xyL are too weak to be detected due to the corruption by Ey. To confirm these findings, feedforward artificial neural network (ANN) was employed to identify the associations and then check on the assumption of homoscedasticity by visual examination of residual plots. Figure 5(a) and 5(b) show the predictive curves and residuals obtained from using ANN models, which are entirely consistent with the results of the nCor | test. In brief, nCorja) successfully prejudged the heteroscedasticity of the residuals without regression modelling.
5
Conclusions
Here, we have proposed a new association measure, nCona to detect the heteroscedastic relation- ships between variables. When used alone. it straightforwardly examines the heteroscedasticity of the underlying noise contained in the data rather than the residuals of a regression analysis, and therefore it is much more efficient and easier to conduct in comparison with the traditional heteroscedasticity tests. When used together with nCor, the two tests form an enhanced toolkit that can not only detect a wider range of complex associations, but also distinguish between functional and nonfunctional relationships. Compare previous association measures, they can obviously get deeper insight into the nnter-structure of the data associations, and may provide promising direction for further analysis. The new method can be widely used to provide solutions for feature selection, model validation, and many other issues in data analysis, interpretation, and modelling.
Acknowledgement
This work was partially supported by CNPC Northwest Marketing Company (MIS Project No. XBYS-2020-JS-9).
References
[1] Francis R. Bach and Michael I. Jordan. Kernel independent component analysis. Journal of Machine Learning Research, 3(1):1-48, 2003.
[2] Tim Bollerslev. Generalized autoregressive conditional heteroskedasticity. Journal of Econo- metrics, 31:307-327, 1986.
[3] L. Breiman and J. H. Friedman. Estimating optimal transformations for multiple regression and correlation. Journal of the American Statistical Association, 80:580-598, 1985.
[4] G. A. Darbellay and I. Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45:1315-1321, 1999.
[5] Pedro Delicado. Another look at principal curves and surfaces. Journal of Multivariate Analysis, 77(1):84-116, 2001.
[6] Pedro Delicado and Marcelo Smrekar. Measuring non-linear dependence for two random variables distributed along a curve. Statistics and Computing, 19(3):255-22) 269,, 2009.
[7] Robert F Engle. Autoregressive conditional heteroscedasticcty with estimates of the variance of united kingdom inflation. Econometrica, 50:987-1007, 1982.
[8] H. Glejser. A new test for heteroskedasticity. Journal of the American Statistical Association, 64:315-323, 1969.
[9] A. Gretton, K. Fukumizu, CH. Teo, L. Song, B. Scholkopf, and AJ. Smola. A kernel statistical test of independence. In Advances in neural information processing systems (NIPS) 20 pages 585-592, September 2008
10

---

[10] Arthur Gretton and L. Gyorf. Consistent nonparametric tests of independence. Journal of Machine Learning Research, 11:1391-1423, 2010.
[11] Ruth Heller, Yair Heller, Shachar Kaufman, Barak Brill, and Malka Gorfine. Consistent distribution-free k-sample and independence tests for univariate random variables. Journal of Machine Learning Research, 17:1-54, 2016.
[12] Peter Kennedy. A Guide to Econometrics. MIT Press,, 2008.
[13] Alexander Kraskov, Harald Stoghauer, and Peter Grassherger. Estimating mutual information. Physical Review 69:066138 2004.
[14] Jin Guan Lin and Xiao Yi Qu. A consistent for heteroscedasticity semi-parametre regression with nonparametric variance function based on the kernel method. Stats, 46(4-6):565- 576, 2012.
[15] Peipei Liu, Hoon Sohn, and Ikgeun Jeon. Nonlinear spectral correlation for fatigue crack detection under noisy environments. Journal of Sound and Vibration, 400:305-316, 2017.
[16] David Lopez-Paz, Philipp Hennig, and Bernhard Scholkopf. The randomized dependence coefficient. In Advances in neural information processing systems (NIPS) 27, Vancouver, Canada, 2013.
[17] Young Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Phys Rev E, 52(3):2318-2322 1995.
[18] R. E. Park. Estimation with heteroscedastic error terms. Econometrica, 34:888, 1966.
[19] D. N. Reshef, Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. Mc J. Turnbaugh, E. S. Lander, M. Mitzenmacher, and C. Sabeti. Detecting novel associations in large data sets. Science, 334(6662))1518-15222 2011.
[20] David Reshef, Yakir Reshef, Pardis Sabeti, and Michael Mitzenmacher. An empirical study of the maximal and total information coefficients and leading measures of dependence. The Annals of Applied Statistics, 12:123-155, 03 2018.
[21] A. Rényi. On measures of dependence. Acta Mathematica Hungarica, 10:441-457 1959.
[22] Alexander Shishkin, Anastasia Bezzubtseva, Alexey Drutsa, Ilia Shishkov, Ekaterina Gladkikh, Gleb Gusev, and Pavel Serdyukov. Efficient high-order interaction-aware feature selection based on conditional mutual information. Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4652-4652. Citeseer, 2016.
[23] Gabor Szćkely and Maria Rizzo. Brownian distance covariance. The Annals of Applied Statistics, 3:1236-1265, 10 2009.
[24] Gabor Székely, Maria Rizzo, and Nail Bakirov. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35:2769-2794, 2007.
[25] Qiang Wang, Yi Shen, and Jian Qiu Zhang. nonlinear correlation measure for multivariable data set. Physica D, 200:287-295 2005.
[26] H. White. A heteroskedassicityyccosssss covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48:817-838, 1980.
[27] Zilin Zeng, Hongjun Zhang, Rui Zhang, and Chengxiang Yin. A novel feature selection method considering feature interaction. Pattern Recognition, 2868):6656-266 2015.
[28] Lifeng Zhang. Systematically exploring associations among multivariate data. In AAA/-20, New York, USA, 2020.


Reviews:
[{"review_id": "bww6NKiLHOy", "reviewer": "Reviewer_cGGJ", "summary": "This paper proposed a statistical measure named the absolute neighbour difference based neighbour correlation coefficient, to detect the associations between variables through examining the heteroscedasticity of the unpredictable variation of dependent variables. The method can somehow measure nonfunctional relationships. The method is simple and easy to implement that does not rely on explicitly estimating the regression residuals or the dependencies between variables so that it is not restrict to any kind of model assumption. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Heteroscedasticity is a complicated scenario in which the noise level of measurement is controled by (or dependent on) a function g() of the independent variable, and therefore difficult to estimate. The authors examined carefully the variations of the  variable difference and proposed a new statistical test method todetect the heteroscedasticity relation. Overall this is an interesting work and the proposed method looks reasonable. I have a few detailed questions/suggestions\n\n1. The authors claimed that heteroscedasticity can also arise from interacting variables, besides the multiplicative relation between the normalized nosie term Theta and the independent variable Xi. Then is it necessary to discriminate between these two scenarios or is it mathematically infeasible to do so? From practical point of view they are of different values.\n\n2. The proposed method is still based on pairs of variable. In case of multiple (more than 2) interacting independent variables, can you apply it to obtain the high-order interaction?\n\n3. I would suggest the authors clearly describe the algorithm by listing each step (with involved variables) one by one, in order to facilitate the understanding of the readers. Currently the procedures and formula appears throughout the paper and appears a bit distributed. An ``algorithm'' figure summarizing the procedures would be helpful. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "V_qHeZ9vqvO", "reviewer": "Reviewer_v9y1", "summary": "This paper proposes a correlation test to detect if heteroscedasticity exists between X and Y.   The method makes use of the neighboring variables and take the pearson correlation between $\\Delta$ Y and $\\Delta$ Y\".   This test does not require the computation of the regression residuals that requires well-specified model assumptions.  The authors claim that this test was able to successfully detected the heteroscedasticity of the data.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting idea to detect any heteroscedastic relationships between variables.  The principle behind is very simple but it is believed to be novel.  This work may have significant impact if there are practical examples that require the detection of heteroscedastic relationships.  The key idea is that the neighboring data point assumption enables us to focus on the residuals directly without the need of finding the regression residuals,.  The test is developed base on the relationship described in equation 1 which is a general model form.   One key point is that all data points have to be close to each other, otherwise the assumption does not hold.  Therefore, it is not surprising that it works well in single dimension case when data are relatively close to each other.  However, as the dimension increases, due to the curse of dimensionality, we need a much larger dataset in order to guarantee the closest property.   Besides, in high dimensional cases, it is said that the optimal reordering permutation is obtained by minimizing the total distance between each pair of neighboring data points and nearest neighbor algorithm is used to obtain the optimal or sub-optimal permutation.  On one hand, it is quite demanding to compute the permutation for large datasets.  On the other hand, the sub-optimal results may affect the closest assumption as well.   \n\nThe paper is nicely written and well organized.  The experiments are clearly presented and evaluated.   It is expected that the results can easily be reproduced.   In the experiment, $f(x) = \\beta \\sin(\\pi  x) $ and $g(x)$ is related to $ \\cos(\\pi x) $ in examples 3 and 4.  f(x) and g(x) have the same frequency.  I wonder if it is more difficult to detect the heteroscedastic relationships if  f(x) and g(x) have different frequencies, say $g(x)$ is related to $ \\cos(n \\pi x) $ instead.  \n  \nIn line 118, it says Lemma 2.1.  Should it be Lemma 1 ?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be good if the authors can include some discussions about the potential applications of the detection heteroscedastic relationships. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "QTjjemhjRar", "reviewer": "Reviewer_sHbg", "summary": "This paper proposes a new test for heteroscedasticity.  Heteroscedasticity is an important issue in real-world applications that in my opinion is under-considered and under-addressed.  The paper presents theoretical results on type I and type II error and provides significant empirical results for the benefits of the method over others such as the Park test and the White test.  It discusses limitations such as decreased relative power in high-dimensional settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To my knowledge the proposed test is novel.  The empirical results and theoretical results show it is promising, especially in the motivating setting where we wish to know if one variable affects the spread of values for another variable.  The paper is intuitive and enjoyable to read, with a nice mix of theory and empirical results.  The paper also assesses its limitations.\n\nOne concern is that most real-world applications now are high-dimensional, and the methods seems to have weaknesses in this setting relative to other methods, as the paper accurately acknowledges and discusses.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, very good discussion of limitations.  More accurately understanding the relationships among variables can be important for social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "one hour"}, "extra_text": {}}, {"review_id": "1179-p8kfGr", "reviewer": "Reviewer_53UA", "summary": "The work studies the detection of \"heteroscedastic relationships\" in data samples. Based on the previous model of \"neighbour correlation coefficien (nCor)\", the work further proposes an \"absolute neighbour difference based neighbour correlation coefficient (nCor_|△|)\" and shows some properties of the coefficient, supported with preliminary experimental results on artificial data samples.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n--1. I am not familiar with the problem of \"heteroscedasticity\" that is studied in the paper, while it seems a meaningful problem to me. \n\n--2. The writing of the paper is not difficult to follow, although there is much room to improve.\n\nConcerns:\n\n--1. This work seems a marginal improvement to Zhang's paper at AAAI'2020 (ie reference [28]). The key result of \"nCor\" test has been established there. The new proposal of \"nCor_|△|\" seems not that significant to me. Correct me if I ignored important issues or further justifications on this point are expected in the rebuttal phase.\n\n--2. It seems to me that the proposed method might need a lot of data samples. Further justification or illustration on this point will be beneficial to readers to fully understand the proposed method.\n\n--3. In addition to artificial datasets, evaluation results on high-dimensional real datasets are strongly encouraged. Furthermore, it would be desirable to see if the author(s) could give some examples that the proposed model succeed but the previous testing methods fail to detect.\n\nIn line 73, the citation/reference to the key definition of \"heteroscedasticity\" seems to be problemastic.\n\n==================\nThe author(s) addressed my concerns in the rebuttal and I tend to accept the work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See \"Main Review\"", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~5"}, "extra_text": {}}, {"review_id": "-N0LD2nwOTE", "reviewer": "Reviewer_Xjiq", "summary": "This paper proposed a new statistical measure, called absolute neighbour difference based correlation test, to detect the existence of heteroscedastic noise in the residual of a functional relationship. \n\nTheorem 1 shows the proposed measure nCor_Delta is only affected by the dispersion function of the residual g(x) and not affected by the functional relationship f(x). As a result, compared to those existing statistical test of heteroscedastic noise, the major advantage of the proposed measured is to avoid fitting the regression functions. The estimator of the measure on real-world datasets is also presented in Definition 2. The statistical test of heteroscedastic noise vs. homoscedastic noise is presented in Theorem 2. \n\nIn the experimental results, the proposed nCor_Delta is firstly compared against two existing tests of heteroscedastic noise, including Parker test and White test. It shows the nCor_Delta can correctly test for heteroscedastic noise while its competing alternatives either fail on some datasets or uses over-complicated regression functions.\n\nThe proposed nCor_Delta is also evaluated against a few existing non-linear dependency measures that are not designed to test heteroscedastic noise, including dCor, RDS, MIC, MI and nCor. These measures cannot detect heteroscedastic noise as expected.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key contribution of this work, i.e. a new statistical test for heteroscedastic noise, seems novel. The writings of the major contributions of related works are also quite clear.\n\nCould the author elaborate on how to combine nCor_Delta and nCor to better discover relationships between variables? For example, if nCor_Delta can detect the heteroscedastic noise between input X and output Y. Given this discovery, how to apply nCor to find a better functional relationship compared to directly applying nCor?\n\nIn section 4.1, 50 datasets are created for each functional relationship. However, in real-world data, only a single realization is available. Can the authors show if the statistical test in Theorem 2 leads to the same conclusion as the bootstrapped 95% confidence interval regarding the existence of heteroscedastic noise?\n\nWhy should nCor_Delta be insensitive to the value of R^2? Is this a desirable property? Suppose R^2 becomes extremely close to 1, i.e. the noise level is much smaller than Var(f(x)), how would nCor_Delta behave?\n\nOne explanation for the worse performance of nCor_Delta when X becomes multivariate could be the curse of dimensionality makes it more difficult to identify the nearest neighbors with extremely close Euclidean distance. Since the number of samples is also finite in real-world applications, would it be possible to establish a bound of sample complexity, i.e., the minimal number of samples required to reliably test the existence of the heteroscedastic noise, even in the one-dimensional case?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author did clarify the limitations of this work in terms of less power in high-dimensional datasets. Since this is a general statistical test, I didn't see any negative societal impact of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a measure called absolute neighbour difference-based neighbour correlation coefficient, to detect the "heteroscedasticity" in the nonfunctional relationship between variables. All reviewers agree that the detection of "heteroscedastic relations" is an important issue in learning and that the proposed method, although simple and intuitive, is novel and has practical implications.
Decision: Accept (Poster)