Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a whiteness hypothesis test for spatio-temporal graphs. The proposed AZ-test is an extension of traditional tests designed for system identification within graph signals, to detect dependencies among temporal observations as well as spatial dependencies among graph neighborhoods. Evaluation is performed on both synthetic as well as real datasets on problems of forecasting in spatio-temporal graphs by analyzing the prediction of residuals within the graph stream.
strengths_and_weaknesses: STRENGTHS:

The paper is easy to follow and the presentation. The technical details and supporting theoretical analysis are clearly presented. The theoretical setup of extending the A-Z whiteness hypothesis test to (spatio-temporal) graphs is interesting and novel in formulation.

WEAKNESSES:

The applicability of the proposed framework beyond the considered forcasting setup is a bit unclear. If my understanding is correct, the null hypothesis test is constructed as a test for a zero mean and as well as spatially and temporally uncorrelated generating process. As such the test statistic C(G) does not distinguish between spatial and  temporal correlation patterns. Given that this is a rather strong assumption, I am unsure how such a test would be generally useful beyond examining residuals in the forecasting setting. 
 
questions: 1. The abstract and the first paragraph of section 4 mentions that the proposed work is applicable in the case of dynamic graphs ( allowing for changing connectivity patterns over time). This is handled by converting to a larger graph (Fig. 1) with nodes as a disjoint union of all the temporal nodes, (weighted) edges as a union of spatial and temporal edges. In the case that a new node v (and corresponding signal Xv[t]) is not present at time t, but added at at time point t+1, what is the corresponding value of Xv[t] used for forming the larger graph?

2. In the results in Table 2, the authors mention that "From the Temporal and Spatial variations of the AZ-test, we see that the temporal correlation appears more prominent than the spatial one for FCRNN". Is this based on the absolute values of C(G*) [without normalisation] in the two columns? If so, is this a fair comparison given that the number of spatial vs temporal edges can be vastly different in number?
limitations: There isn't a dedicated discussion on the limitations of the proposed framework. In this context, it would be good to consider and address the point mentioned in weaknesses about applicability and potential impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a novel model assessment tool by testing the whiteness (of the residuals) for dynamic graphs. Asymptotic normal null distribution (theorem 1) is developed, which enables easy calibration of the test. 
strengths_and_weaknesses: Strength:

The author proposed a statically solid testing method. Most importantly, the asymptotic null distribution is developed. Extensive numerical results help support the effectiveness of the method. Implementation is also provided to help recover the results.

Weakness:
1. (minor) notations: What is $\gamma$ in eq. (2)? I can see it is the testing threshold on line 122, but I think it should be clarified right after the equation; In theorem 1, I would prefer using A1 for Assumption 1 instead of Ass1.

2. What is the power of this test? I would expect the relation between power and (sample size, dimensionality). (PS: I do see the numerical assessment, but since you have the asymptotic null distribution, it would be better if you also have the (asymptotic) power. Or what is the technical difficulty is preventing you to give the power?)

3. (major) Although this method is novel in terms of the setting, I would expect numerical comparison to existing approaches for static graphs with the mentioned work in line 85.
questions: See above in weakness
limitations: No. Could we extend this test to graphs with negative weights? I am asking this since in practice the interactions between nodes could be both triggering effect or inhibiting effect.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a whiteness test to examine whether a given graph is "white" or not. The designed test extends classical methods so that it can deal with general case of spatio-temporal correlation. This method applies for uni or multivariate node signals, statistic or dynamic grah, (possibly) weighted edges. The author also show that this test can be used to access the optimality of a given forecating model, which is an interesting and important application. 
strengths_and_weaknesses: **Originality**: This is the first whiteness test that applies for spatio-temporal signals on graph. The design of this test share some idea from Geary' test, but make some additional advancements. Moreover, the designed test is general enough to deal with weighted and dynamic graphs with multivariate node signals.

**Quality/Significance**: From theoretical perspective, the test procedure together with the limiting distribution of testing statistic is derived. I have checked the proof of their theoretical result in detail, and I believe it is clear and correct. From practical perspective, the author examine the ability of AZ-test in various settings and present an interesting application that it can examine the optimality of forecating model. Both parts are excellent contributions.

**Clarity**: The overall presentation is good. However, minor points can be updated to make it more reader-friendly. See some comments in the Limitation section.


**Weakness**: 
1. The author pointed out this test share some insights from Geary' test but made some extra innovations. The detailed comparison is not clear to me. I would expect the author to clarify this point in the related work section. 
2. Some presentations can be improved (See Limitations section).
questions: 1. The distribution of testing statistic is only characterized when the number of edges $|E|\to\infty$. Does it mean this test may not be powerful for small graph? Is there any possible idea to address this issue?
2. In section 4, to perform whiteness test for dynamic and weighted graph, the author argued that we should pre-process it to make it static and then previous approach in Section 3 applies. Would this pre-processing procedure bring increase of type-I/type-II error for AZ-test? Is there any pre-processing procedure and which one is optimal? From my view point, this procedure seems too ad-hoc, and I hope to see some theoretical results behind this procedure.
limitations: 1. Some notations and terminologies are introduced right after they first appeared, e.g., definition of $\text{sng}, E_{\leftrightarrow}, |E|, \rightarrow$. At least the author should create a notation subsection part in Section 1 to introduce all special notations that will be frequently used. If possible, the author may create a preliminary section in main content or supplementary to talk about basics of static/dynamic graph in detail, instead of briefly touching the basics in Section 3/4.
2. Assumptions (Ass1)-(Ass3) are important and repeatedly discussed in later section. The author are suggested to create a special "Assumption" environment to include them instead of putting them in Theorem 1. Also, comments of the assumption can be moved from supplementary to the main content if the paper gets accepted.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes an asymptotic test of whiteness for graph signals that can be used to assess graph signal forecasting models. The main idea is that a whiteness test for graph signals is useful for evaluating forecasting models. Since the model is optimal (in a second order sense) if the residuals are white such that there is no bias and there is no more correlation in the error that the model could account for.  The test is general under the key assumptions including that the signals across space and time can be considered one large graph that is growing in nodes and edges. This generality allows for dynamic graph topology. The test is demonstrated on synthetic and real world datasets. 



strengths_and_weaknesses: Strengths:
The utility of a robust whiteness measure as the signature of correct modeling of graph signals has practical utility and significance. The paper introduces a general test that is scalable and has an asymptotic characterization. The use of only the sign of the inner product and weights is straightforward and yields a simple asymptotic distribution. The papers coverage of the development and application is thorough. Many more results are mentioned in the supplemental material. The language is easy to read and relations to prior work are stated. 

Weaknesses: A key critique is the occasional lack of precise definitions  (especially at the onset) and implicit assumptions that are only revealed after a careful reading. Thus the paper is not clear what is required and how the asymptotic nature is practically realized. 
 
Another critique is that the practical application to real world data and models seems to have mixed results in terms of the model performance (MAE), median of residuals, and the proposed test. 

Not until the conclusion do we learn the alphabetic nature of the name. It wasn't apparent to me throughout...



questions: Suggestions:

The idea of appending the signal to the graph and defining G as including the signal can lead to subsequent confusion.

Description of whiteness on line 36 is not precise. The quantities involved are vectors or matrices (assuming an outer product) but zero is a scalar. (From the figures row vectors are denoted which means that the second expectation in line 36 is a inner product, but this doesn't make much sense to me. Later on line 110 uses an inner product, so the reader can now be assured that a correlation matrix is intended on line 36.) Furthermore the precise nature of the stochastic time series is not made clear. The stated assumption in the footnote is that the signal is stationary. However, in line 36 it is not clear if the expectations are functions of time or only the time lag like in the footnote. In practice, it is unlikely to have assess to multiple realizations for estimation so assumptions on ergodicity/stationarity should be made, but this point isn't made until around line 52. Also in the footnote it is not clear that $\mathbf{x}$ is multivariate or univariate. 

Line 102: Is does null equal zero for the weights? I'm trying to see if this is subtle variation or if a null is somehow distinct from zero. 

On line 59, it is not clear what is meant by "; indeed, authors can consider different statistics."

"Supplemental material accompanies the manuscript." It would be useful to detail what this is rather than the blanket statement as provided. 

Line 109: It would be better for continuity to use ", where" or equivalent instead of ending the sentence "Theorem 1".

Line 156: I think the G should have an arrow on top in equation 6. 


The number of trials used to compute the power is a bit too small. The numbers for the null hypothesis vary. A safer choice of 1000 may enable the reader to better assess whether the false positive rate. 

On lines 290 and 291 the comment about p-values being significantly different from zero should be altered to simply say that the null hypothesis cannot be rejected.

Reference list has some problems with format namely book titles should be in title case. [1] appears to be missing punctuation.  The publication type for [12] Geary's test is unclear. 

Questions:

In Theorem 1, what is the relationship of the edge cardinality and the node cardinality? 
Part of my confusion is that Theorem 1 seems to rely on a distribution over random graphs. For the number of edges to go to infinity, the number of nodes would also have to. But this initially seems to involve a distribution over graph topologies not really a distribution over graph signals. Assumption 3 is important but it described initially as simply being 'technical'. (Line 139 mentions milder assumptions, but it is not clear if that is directly relating to Ass3 or something else.  ) Later on I understand this to be formed from linking nodes through time to have a growing graph. In Section 4, the reader can infer that the growth in edges/nodes required for Ass3 practically corresponds to product graphs that grow through time. 

In Table 2 it is not clear why the optimal predictor has a higher MAE than the models.  Additionally the DCRNN model has a significantly non-zero median but shows uncorrelated residuals. Should a user always use both tests to assess modeling performance?  Finally, the description states that the test statistic is not valid since Ass2 is not held. Overall, these mixed results are not very encouraging on the practical usefulness of the method. 






limitations: It is not clear if there is performance thresholds for large number of vertices and a few temporal snapshots.

The paper is perhaps missing baselines for tests of whiteness for signals on regular grids (time series or spatial-temporal forecasting in geospatial settings) where other tests can be readily applied. In these cases, higher resolution (more nodes) can be readily be obtained.
review_title: null

Meta-review: metareview: The paper studies the whitness hypothesis test for spatio-temporal graph, which is a fundamental problem and can be relevant to many machine learning tasks. The authors have done a great job in the rebuttal phase in addressing reviewers’ comments. I believe it is a worthwhile paper to be published in NeurIPS. 
confidence: Certain
Decision: Accept