Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the author first makes an observation that the filters of GNNs show similarities with principal component analysis (PCA), in which data is projected on the eigenspace of the covariance matrix. Then they proposed the covariance neural network (VNN) that operates on sample covariance matrices as graphs are motivated by this observation. They theoretically demonstrate VNN stability to perturbations in the covariance matrix, indicating a qualitative advantage over traditional PCA-based data analysis approaches that are prone to instability due to close eigenvalues and principal components. The author also makes several real work experiments to empirically prove their statement.
strengths_and_weaknesses: 
Strengths:
In this paper, the author links the coVariance filter with the graph convolution filter in GNN and proposes a DL architecture based on the coVariance filter. The computing covariance matrix for a large graph is always computation expensive, by alternatively computing graph Fourier transformation show potential for further GNN development.

The author also theoretically analyzes the stability of the covariance filter and covariance graph neural network and empirically evaluates VNNs for transferability and stability.


Weakness:

There is no related work section in this paper. Although those who study GNN are familiar with GNN, PCA, and graph Fourier transformation, it would be better to give an overview of different kinds of GNN and summaries them.


questions: How about the scalability of VNN? The experience of this paper is mainly focused on 'small' graphs. When it comes to large graphs, is it still computationally feasible? When VNN is used in large graphs, what will be the biggest challenge?
limitations: There is no limitation and social impact in this paper. According to Neurips instructions, hope these two parts will appear in a future version.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Motivated by similarities between principal components analysis (PCA) and graph convolutional filters in GNN, the authors introduce a new GNN architecture that uses sample covariance matrices as the graph representation of the data. They develop “coVariance” filters analogous to graph convolutional filters in GNNs, show that PCA is a special case of applying such filters, and propose a deep learning architecture based on “coVariance” filters they call “coVariance Neural Networks” (VNNs). Using perturbation theory for sample covariance matrices they theoretically establish the stability of VNNs to perturbations in the sample covariance matrix (in terms of number of samples). They then empirically evaluate VNN stability relative to PCA on synthetic and real world data, and show that VNNs can be used transferably on one multiresolution dataset. 
strengths_and_weaknesses: Strengths:
- The work is original, making an interesting connection between GNNs and PCA and rigorously following this through the development, theoretical stability, and empirical validation of VNNs.
- The results are likely to be of interest to a broad audience, given the widespread use of PCA and prevalence of correlation matrices across disciplines. 
- The theoretical stability analysis is sound and backed up by validation on synthetic and interesting real world neuroimaging data.
- The demonstration of multi-scale transferability is very cool, and goes beyond what PCA is capable of.
- The manuscript is clearly written and code is provided for replicability.

Weaknesses: 
- I find the paper to have few weaknesses. 

questions: I would like to know more about model training time and computational cost in terms of covariance matrix size and number of samples. 
limitations: I think the authors adequately address limitations in their work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose a coVariance Neural Network (VNN) in analogy to graph neural networks based on covariance matrices of datasets, similar to what is used in PCA. The authors also introduce a “covariance transform” which projects a new datapoint into the PCA space by projecting to the eigenvectors of the covariance operator $UX$ and a covariance filter which learns coefficients of polynomial of the covariance matrix.  They show that using covariance filters one can recover the covariance transform. They also show that this polynomial filter is stable with respect to perturbations in the sampled data from which the sampled covariance matrix is taken. 
strengths_and_weaknesses: Strengths:

The authors correctly note the similarity between the covariance matrix and an adjacency matrix of a graph, i.e., positive semidefinite matrices whose eigenvectors and values have specific properties in terms of describing the heterogeneity in the data. In fact it can be argued the other way that graph fourier transforms and graph signal processing are the result of a “kernel trick” based on PCA which is far older.  So in this sense GNNs/GCNs are a more general class of networks that subsumes VNNs. 

The note that this results in a stable PCA is interesting although over the years there have been robust forms of PCA that have been developed including variations of Principal component pursuit.

Weakness:

However,  I do not believe their proposal constitutes a new neural network of any sort. I believe this is just an application of GCNs to the graph consisting of data features as nodes and data points as signals on the nodes (rather than vice versa). While this may be desirable for some types of analysis (see Tong et al. IEEE ICCASP 2022 for analysis of cells as signals over gene graphs), I do not believe it has advantages over robust PCA for this specific application. 

The paper should potentially be reconfigured to simply talk about an application of GCNs/GNNs to feature covariance matrices and situations where that could be useful. 

Further a key weakness here is regarding relationships between data features as defined by covariances. This is a strictly linear relationship, if this was changed to mutual information or some other relationship type then indeed a more complex relational graph would be necessary and this is precisely where GNNs/GCNs have contributed. 

Other uses of the covariance filter seem to amount to low-rank approximations done via SVD and again I don't see much advantage in using a neural network for this kind of linear operation. 


questions: What are other applications of the covariance filter that would not be subsumed by SVD low rank approximations? 
limitations: I do not see a discussion of limitations in this manuscript.
review_title: null

Meta-review: metareview: This paper proposes coVariance neural networks (VNN), which is a new architecture of graph neural network that is more robust to perturbations in covariance matrix. Most reviewers liked the new architecture as the intuition is clearly presented and the experiment results are interesting (in particular the results demonstrating multi-scale transferability). There are some concerns that this new architecture can be viewed as a more direct modification of GNN, I recommend the authors to clarify this relationship more clearly and emphasize the motivation.
confidence: Certain
Decision: Accept