Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new test for heteroscedasticity.  Heteroscedasticity is an important issue in real-world applications that in my opinion is under-considered and under-addressed.  The paper presents theoretical results on type I and type II error and provides significant empirical results for the benefits of the method over others such as the Park test and the White test.  It discusses limitations such as decreased relative power in high-dimensional settings.
main_review: To my knowledge the proposed test is novel.  The empirical results and theoretical results show it is promising, especially in the motivating setting where we wish to know if one variable affects the spread of values for another variable.  The paper is intuitive and enjoyable to read, with a nice mix of theory and empirical results.  The paper also assesses its limitations.

One concern is that most real-world applications now are high-dimensional, and the methods seems to have weaknesses in this setting relative to other methods, as the paper accurately acknowledges and discusses.
limitations_and_societal_impact: Yes, very good discussion of limitations.  More accurately understanding the relationships among variables can be important for social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The work studies the detection of "heteroscedastic relationships" in data samples. Based on the previous model of "neighbour correlation coefficien (nCor)", the work further proposes an "absolute neighbour difference based neighbour correlation coefficient (nCor_|△|)" and shows some properties of the coefficient, supported with preliminary experimental results on artificial data samples.
main_review: Pros:

--1. I am not familiar with the problem of "heteroscedasticity" that is studied in the paper, while it seems a meaningful problem to me. 

--2. The writing of the paper is not difficult to follow, although there is much room to improve.

Concerns:

--1. This work seems a marginal improvement to Zhang's paper at AAAI'2020 (ie reference [28]). The key result of "nCor" test has been established there. The new proposal of "nCor_|△|" seems not that significant to me. Correct me if I ignored important issues or further justifications on this point are expected in the rebuttal phase.

--2. It seems to me that the proposed method might need a lot of data samples. Further justification or illustration on this point will be beneficial to readers to fully understand the proposed method.

--3. In addition to artificial datasets, evaluation results on high-dimensional real datasets are strongly encouraged. Furthermore, it would be desirable to see if the author(s) could give some examples that the proposed model succeed but the previous testing methods fail to detect.

In line 73, the citation/reference to the key definition of "heteroscedasticity" seems to be problemastic.

==================
The author(s) addressed my concerns in the rebuttal and I tend to accept the work.
limitations_and_societal_impact: See "Main Review"
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a correlation test to detect if heteroscedasticity exists between X and Y.   The method makes use of the neighboring variables and take the pearson correlation between $\Delta$ Y and $\Delta$ Y".   This test does not require the computation of the regression residuals that requires well-specified model assumptions.  The authors claim that this test was able to successfully detected the heteroscedasticity of the data.  
main_review: This paper presents an interesting idea to detect any heteroscedastic relationships between variables.  The principle behind is very simple but it is believed to be novel.  This work may have significant impact if there are practical examples that require the detection of heteroscedastic relationships.  The key idea is that the neighboring data point assumption enables us to focus on the residuals directly without the need of finding the regression residuals,.  The test is developed base on the relationship described in equation 1 which is a general model form.   One key point is that all data points have to be close to each other, otherwise the assumption does not hold.  Therefore, it is not surprising that it works well in single dimension case when data are relatively close to each other.  However, as the dimension increases, due to the curse of dimensionality, we need a much larger dataset in order to guarantee the closest property.   Besides, in high dimensional cases, it is said that the optimal reordering permutation is obtained by minimizing the total distance between each pair of neighboring data points and nearest neighbor algorithm is used to obtain the optimal or sub-optimal permutation.  On one hand, it is quite demanding to compute the permutation for large datasets.  On the other hand, the sub-optimal results may affect the closest assumption as well.   

The paper is nicely written and well organized.  The experiments are clearly presented and evaluated.   It is expected that the results can easily be reproduced.   In the experiment, $f(x) = \beta \sin(\pi  x) $ and $g(x)$ is related to $ \cos(\pi x) $ in examples 3 and 4.  f(x) and g(x) have the same frequency.  I wonder if it is more difficult to detect the heteroscedastic relationships if  f(x) and g(x) have different frequencies, say $g(x)$ is related to $ \cos(n \pi x) $ instead.  
  
In line 118, it says Lemma 2.1.  Should it be Lemma 1 ?
limitations_and_societal_impact: It would be good if the authors can include some discussions about the potential applications of the detection heteroscedastic relationships.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a statistical measure named the absolute neighbour difference based neighbour correlation coefficient, to detect the associations between variables through examining the heteroscedasticity of the unpredictable variation of dependent variables. The method can somehow measure nonfunctional relationships. The method is simple and easy to implement that does not rely on explicitly estimating the regression residuals or the dependencies between variables so that it is not restrict to any kind of model assumption. 
main_review: Heteroscedasticity is a complicated scenario in which the noise level of measurement is controled by (or dependent on) a function g() of the independent variable, and therefore difficult to estimate. The authors examined carefully the variations of the  variable difference and proposed a new statistical test method todetect the heteroscedasticity relation. Overall this is an interesting work and the proposed method looks reasonable. I have a few detailed questions/suggestions

1. The authors claimed that heteroscedasticity can also arise from interacting variables, besides the multiplicative relation between the normalized nosie term Theta and the independent variable Xi. Then is it necessary to discriminate between these two scenarios or is it mathematically infeasible to do so? From practical point of view they are of different values.

2. The proposed method is still based on pairs of variable. In case of multiple (more than 2) interacting independent variables, can you apply it to obtain the high-order interaction?

3. I would suggest the authors clearly describe the algorithm by listing each step (with involved variables) one by one, in order to facilitate the understanding of the readers. Currently the procedures and formula appears throughout the paper and appears a bit distributed. An ``algorithm'' figure summarizing the procedures would be helpful. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a new statistical measure, called absolute neighbour difference based correlation test, to detect the existence of heteroscedastic noise in the residual of a functional relationship. 

Theorem 1 shows the proposed measure nCor_Delta is only affected by the dispersion function of the residual g(x) and not affected by the functional relationship f(x). As a result, compared to those existing statistical test of heteroscedastic noise, the major advantage of the proposed measured is to avoid fitting the regression functions. The estimator of the measure on real-world datasets is also presented in Definition 2. The statistical test of heteroscedastic noise vs. homoscedastic noise is presented in Theorem 2. 

In the experimental results, the proposed nCor_Delta is firstly compared against two existing tests of heteroscedastic noise, including Parker test and White test. It shows the nCor_Delta can correctly test for heteroscedastic noise while its competing alternatives either fail on some datasets or uses over-complicated regression functions.

The proposed nCor_Delta is also evaluated against a few existing non-linear dependency measures that are not designed to test heteroscedastic noise, including dCor, RDS, MIC, MI and nCor. These measures cannot detect heteroscedastic noise as expected.
main_review: The key contribution of this work, i.e. a new statistical test for heteroscedastic noise, seems novel. The writings of the major contributions of related works are also quite clear.

Could the author elaborate on how to combine nCor_Delta and nCor to better discover relationships between variables? For example, if nCor_Delta can detect the heteroscedastic noise between input X and output Y. Given this discovery, how to apply nCor to find a better functional relationship compared to directly applying nCor?

In section 4.1, 50 datasets are created for each functional relationship. However, in real-world data, only a single realization is available. Can the authors show if the statistical test in Theorem 2 leads to the same conclusion as the bootstrapped 95% confidence interval regarding the existence of heteroscedastic noise?

Why should nCor_Delta be insensitive to the value of R^2? Is this a desirable property? Suppose R^2 becomes extremely close to 1, i.e. the noise level is much smaller than Var(f(x)), how would nCor_Delta behave?

One explanation for the worse performance of nCor_Delta when X becomes multivariate could be the curse of dimensionality makes it more difficult to identify the nearest neighbors with extremely close Euclidean distance. Since the number of samples is also finite in real-world applications, would it be possible to establish a bound of sample complexity, i.e., the minimal number of samples required to reliably test the existence of the heteroscedastic noise, even in the one-dimensional case?
limitations_and_societal_impact: The author did clarify the limitations of this work in terms of less power in high-dimensional datasets. Since this is a general statistical test, I didn't see any negative societal impact of this work.
review_title: null

Meta-review: This paper proposes a measure called absolute neighbour difference-based neighbour correlation coefficient, to detect the "heteroscedasticity" in the nonfunctional relationship between variables. All reviewers agree that the detection of "heteroscedastic relations" is an important issue in learning and that the proposed method, although simple and intuitive, is novel and has practical implications.
Decision: Accept (Poster)