coVariance Neural Networks
Authors: Saurabh Sihag
Gonzalo Mateos
Corey McMillan
Alejandro Ribeiro
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Convolutional Networks
Covariance
Principal Component Analysis
Stability
Transferability

Abstract: Graph neural networks (GNN) are an effective framework that exploit inter-relationships within graph-structured data for learning. Principal component analysis (PCA) involves the projection of data on the eigenspace of the covariance matrix and draws similarities with the graph convolutional filters in GNNs. Motivated by this observation, we study a GNN architecture, called coVariance neural network (VNN), that operates on sample covariance matrices as graphs. We theoretically establish the stability of VNNs to perturbations in the covariance matrix, thus, implying an advantage over standard PCA-based data analysis approaches that are prone to instability due to principal components associated with close eigenvalues. Our experiments on real-world datasets validate our theoretical results and show that VNN performance is indeed more stable than PCA-based statistical approaches. Moreover, our experiments on multi-resolution datasets also demonstrate that VNNs are amenable to transferability of performance over covariance matrices of different dimensions; a feature that is infeasible for PCA-based approaches.

CO Variance Neural Networks
Saurabh Sihag
Gonzalo Mateos University of Rochester rrw.ossbbeceerrchhtteeete .dd
University of Pennsylvania
sihags@pennmedicine cine ede
Corey McMillan University Pennsylvania cucmilla@pennmedccine upenn. edu
Alejandro Ribeiro University Pennsylvan vania aribeirodseas. upenn edu
Abstract
Graph neural networks (GNN) are an effective framework that exploit inter- relationships within graph-structured data for learning. Principal component analy- sis (PCA) involves the projection of data on the eigenspace of the covariance matrix and draws similarities with the graph convolutional filters in GNNs. Motivated by this observation, we study GNN architecture, called co Variance neural network (VNN), that operates on sample covariance matrices as graphs. We theoretically establish the stability of VNNs to perturbations in the covariance matrix, thus, implying an advantage over standard PCA-based data analysis approaches that are prone to instability due to principal components associated with close eigenval- ues. Our experiments on real-world datasets validate our theoretical results and show that VNN performance is indeed more stable than PCA-based statistical ap- proaches. Moreover, our experiments on multi resluitisoo datasets also demonstrate that VNNs amenable to transferability of performance covariance matrices different dimensions; a feature that is infeasible for ICAA-assed approaches.
Introduction
Convolutional neural networks (CNN) are among the most popular deep learning (DL) architec- tures due to their demonstrated ability to learn latent features from Euclidean data adaptively and automatically (T] Motivated by the fact that graph models can naturally represent data in many practical applications [2), the generalizations of CNNs to adapt to graph data using GNN architectures have rapidly emerged in recent years (3. A practical challenge to DL architectures is scalability to high-ding nigh-dimensional data [4]. Specifically, DL models can typically have millions of parameters that require long training times and high storage capacity (5). GNN architectures have two attractive features towards learning from high-dimensional data: 1. the number of parameters associated with graph convolution operations in GNNs is independent of the graph size; and 2. under certain assumptions, GNN performance can be transferred across graphs of different dimensions [6].
Principal component analysis (PCA) is traditional, non-parametric approach for linear dimension- ality reduction and increase of interpretability of high dimensional datasets 7 Accordingly, is common to embed the PCA transform within DL pipeline 4
g 9 Such approaches usually use
PCA to optimize the DL model parameters by removing redundancy in the features in intermediate layers. The scope of our work is different. In this paper, leverage the connections between PCA and the mathematical concepts used in GNNs to propose a learning framework that can recover PCA and also, inherit additional desirable features associated with GNNs.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1.1 PCA and Graph Fourier Transform
PCA is an orthonormal transform which identifies the modes of maximum variance in the dataset. The transformed dimensions of the data (known as principal components) are uncorrelated. Com- puting the PCA transform reduces to finding the eigenbasis of the covariance matrix or equivalentyy the singular value decomposition of the data matrix [10] We note that PCA computation using eigendecomposition of covariance matrix has similarities with the graph Fourier transform (GFT) in the graph signal processing literature [ This observation is discussed more formally next.
Consider an m dimemniinionll random vector X E Rmxx with the true or ensemble covariance matrix of X defined C f [ [X - T. In practice, may not observe the data model for X and its statistics, such as the covariance matrix C directly. Alternatively, we have access to a dataset consisting of n random, independent and identically distributed (i.i.d) samples of X, given by x; E [ Vi E (1..... n...
where the data matrix is &n, _ X.]- Using the samples X,, Vi E {1, (1,...,, n), we can form estimate of the ensemble covariance matrix, conventionally referred to as the sample covariance matrix
On 71
X xx x)²
where x is the sample mean across n samples and . refers to the transpose operator. Given the eigendecomposition of Čn
C = -WW
(2)
where = [u um! is the matrix with its columns as the eigenvectors and W diag( Wi,..., Wm) is the diagonal matrix of eigenvalues, such that, wi u2.... 2 Wms the imple-
mentation PCA is given by [10
yn = X,.
(3)
The eigenvectors of Č, form the principal components and the transformation in (3 is equivalent to the projection of the sample x; in the eigenvector space of C,, A detailed overview of PCA is provided in Appendix B We note that in the context of graph signal processing, (3) is equivalent to
the graph Fourier transform (GFT) if the covariance matrix C, is considered the graph representation
or graph shift operator for the data x, 12
1.2 Contributions
Motivated by the observation in Section I.] we introduce the notion of co Variance Fourier transform and define co Variance filters similar graph convolutional filters in GNNs. Our analysis shows that standard PCA can be recovered using co Variance filters. Furthermore, we propose a DL architecture based on co Variance filters, referred to as coVariance neural networks (VNN)) a permutation- invariant architecture derived from the GNN framework that uses convolutional filters defined on the covariance matrix. Understanding VNNs is relevant given the ubiquity of so-termed correlation networks [13 Chapter 7.3.1] ] and covariance matrices as graph models for neuroimaging [14], user preference [5), financial [16], and gene expression data [3 Chapter 7.3.4], to name a few timely domains. Summary of our contributions is as follows:
Stability of VNN: For finite the eigenvectors of the sample covariance matrix Č₁ are perturbed from the eigenvectors of the ensemble covariance matrix C. Moreover, the principal components corresponding to eigenvalues that are close are unstable, i.e., small changes in the dataset may cause large changes in such principal components [77 and subsequently, irreproducible statistical results [18] Statistical learning approaches such as regression, classification, clustering that use PCA as the first step for dimension reduction inherit this instability. Our main theoretical contribution is to establish the stability of VNNs to perturbations in the sample covariance matrix in terms of sample size n. For this purpose, we leverage the perturbation theory for sample covariance matrices and show that for given Č,,, the perturbation in the VNN output Tb ((CC.) (C.)) with respect to the VNN output
We emphasize on V in co Variance for abbreviations to avoid confusions with any existing machine learning frameworks.

---

for C, ((C) satisfies |((C,) - (1/ná== for some < 1/2 with high probability. Therefore, our analysis ties the stability VNNs the quality of sample covariance matrices, which is distinct from existing studies that study stability of GNNs under abstract, data-independent discrete or geometric perturbations to graphs [9 20
Our experiments on real-world and synthetic datasets show that VNNs are indeed stable to perturbations in the sample covariance matrix. Moreover, on multi-resolution dataset, we also illustrate the transferabiliiy of VNN performance to datasets of different dimensions without re- training. These observations provide convincing evidence for the advantages offered by VNNs over standard PCA-based approaches for data analysis.
Empirical Validations:
1.3 Related Work
GNNs: GNNs broadly encapsulate all DL models adapted to graph data E 21. Recent survey articles in and [I categorize the variants of GNNs according diverse criteria that include mathematical formulations, algorithms, and hardware/softwar implementations. Convolutional GNNs (22), graph autoencoders [23] recurrent GNNs, and gated GNNs 24 are among few prominently studied and applied categories of GNNs. The taxonomy pertinent to this paper is that of graph convolutional network (GCN) that generalizes the concept of convolution to graphs. Specifically, VNNs are based on GCNs that model the convolution operation via graph convolutional filters (25 The graph convolutional filters are defined as polynomials over the graph representation where the order of the polynomial controls the aggregation of information over nodes in the graph [12]. We extend the
definition of graph convolutional filters to filters over sample covariance matrices to define VNNs. The ubiquity of PCA and prevalence of covariance matrices across disciplines for data analysis motivate us to study VNNs independently of aforementioned GCNs. Morcover, the unique structure inherent covariance matrices (relative to arbitrary graphs) and their sample-based perturbations facilitates sharper stability analyses, which offer novel insights.
Robust PCA: Robust PCA is a variant of PCA that aims to recover the low rank representation of a dataset arbitrarily corrupted due to outliers [26] 27 (8). The outliers considered in robust PCA approaches could manifest in the dataset due to missing data, adversarial behavior, defects or contamination data collection. In contrast to such settings, here we consider the inherent statistical uncertainty in the sample covariance matrix, where ill-defined eigenvalues and eigenvectors of the sample covariance matrix can be the source of instability in statistical inference. Therefore, the notion of stability of VNNs in this paper is fundamentally distinct from the notion of stability or robustness in robust PCA approaches.
co Variance Filters
If m dimensions of data X., can be represented as the nodes of an m-node, undirected graph, the sample covariance matrix C, is equivalent its adjacency matrix. In GNNs, graph Fourier transform projects graph signals in the eigenspace of the graph and is leveraged to analyze graph convolutional filters [12]. Therefore, we start by formalizing the notion of Variance Fourier transform (abbreviated as VFT). For this purpose, we leverage the eigendecomposition of C, in 2
Definition Varianee Fourier Transform). Consider a sample covariance matrix Cn as defined in 1. The coVariance Fourier transform (VFT) of a random sample is defined its projection on the eigenspace of Cn and is given by
x U
(4)
The i-th entry of x, i.e., [x]; represents the i-th Fourier coefficient and is associated with the eigenvalue wi. Note that the similarity between PCA and VFT implies that eigenvalue w; encodes the variability of dataset x,, in the direction of the principal component u, In this context, the eigenvalues of the covariance matrix are mathematical equivalent of the notion of graph frequencies in graph signal processing [II. Next, we define the notion of co Variance filters (VF) that are polynomials in the covariance matrix.
3

---

Definition 2 (coVariance Filter). Given set of real valued parameters [ha]k=o the co Variance filter for the covariance matrix Cn is defined
H(C,,)
hg
C
(5)
k=0
The output of the covariance filter H(C,) for input is given by
hy C H(C,)).
(6)
The Variancee filter H(C,)) follows similar analytic concepts of combining information in different neighborhoods as in the well-studied graph convolutional filters [12]. Moreover, the filter is defined by the parameters (he k== and therefore, for the ensemble covariance matrix C, the co Variance filter is given by H(C). On taking the VFT of the output in (6) and leveraging (2)
we have
m
& U -UZ=UU> Σ lu [UWUT]*x
hi W
k=
k=0
where 7 holds from the orthonormality of eigenvectors and definition of VFT in (4) Using 7 we
can further define the frequency response of the co Variance filter over the covariance matrix C₁ in the domain of its principal components as
hww)) - haw
(8)
k=0
such that, from 7 and (). the i-th element of 2, has the following relationship
h((::) [x]]
Equation 9 reveals that performing the co Variance filtering operation boils down to processing (c.g.. amplifying attenuating) the principal components of the data. This observation draws analogy with the linear-time invariant systems in signal processing where the different frequency modes (in this case, principal components) can be processed separately using Variance filters, in way determined by the frequency response values h(w;). For instance, using a narrowband co Variance filter whose frequency response is h(A) = 1, if X w; and h(A) O, otherwise, recover the score corresponding to the projection of on u, i.c, the i-th principal component of C,. Therefore, there exist filterbanks of narrowband co Variance filters that enable the recovery of the PCA transformation. This observation is formalized Theorem 0
Theorem 1 (co Variance Filter Implementation of PCA). Given covariance matrix Cn with eigen- decomposition in ( if the PCA transformation of input x is given by y = -Ux, there exists filterbank of co Variance filters (H,(C,)) : E m....,,,]., such that, the score of the projection of input x on eigenvector u, can be recovered by the application of a co Variance filter H; H((Cn) as:
[y]; = H(C,)
(10)
where the frequency response h₂(A) the filter H, is given by
[Tis if A=
h₂(A)
(1I)
10, otherwise
Theorem establishes equivalence between processing data samples with PCA and processing data samples with a specific polynomial on the covariance matrix. As we shall see in subsequent sections, the processing on a polynomial of covariance matrix has advantages in terms of stability with respect to the perturbations in the sample covariance matrix. The design of frequency response of different co Variance filters sufficient to implement PCA transformation according to Theorem [ iss shown figuratively in Appendix D If it is desired to have PCA- data transformation be followed by dimensionality reduction or statistical learning tasks such as regression or classification, the co Variance bister-baeed PCA can be coupled with the post-hoc analysis model to have an end-to- end framework that enables the optimization of the parameters 7) in the frequency response of the co Variance filters.

---

3 coVariance Neural Networks (VNN)
In this section, we propose co Variance Neural Network (VNN), which provides an end-to-end, non-linear, parametric mapping from the input data x to any generic objective r and is defined as
r (x; C., H)
(12)
for sample covariance matrix C, where H is the of filter coefficients that characterize the representation space defined by the mapping (-)). The VNN (x;; C., H) may be formed by multiple layers, where each layer consists of two main components: i) a filter bank made of VFs similar to that in ( and ii) pointwise non-lineat activation function (such as ReLU, tanh). Therefore, in principle, the architecture of VNNs is similar to that of graph neural networks with the covariance matrix C, as the graph shift operator [12]. We next define the co Variance perceptron, which forms the building block of a VNN and is equivalent to I-layer VNN.
Definition 3 (co Varvinnnce Perceptron). Consider a dataset with the sample covariance matrix O.. For a given non non-linear activation function o-)). input x. a coVariance filter H(C,) = -x= hy C and its corresponding coefficient set H. the co Variance perceptron is defined as
(x: C,,, H) - (H((C.))).
(13)
The VNN can be constructed by cascading multiple layers of co Variance perceptrons (shown in Appendix Diin the Supplementary file). Note that the non-linear activation functions across different layers allow for non-linear transformations, thus, increasing the expressiveness of VNNs beyond linear mappings such co Variance filters. Furthermore, similar to GNNs, we can further increase the representation power of VNNs by incorporating multiple parallel inputs and outputs per layer enabled by filter banks every layer 112 29] In this context, we remark that the definition of one layer perceptron in (13) can be expanded to the following.
Remark Varianee Perceptron with Filter Banks). Consider a co Variancc perceptron with Fin m-dimensional inputs and Fout m-dimensional outputs. Denote the input at the perceptron by Xin = Xin [xi[[1)..... Xin [Fn[Fin]] and the output at the perceptron by Xout = +xout] [1],.... Xout [out]] Each inpul Xin (g), Vy {I,..., Fin & is processed by Fowr coVariance filters in parallel. For f Fouth J. the f-th output in Nout is given by
Fin
Xout [f]= =
D(Xin: Cn.y H.y)
(14)
where Hf is the set of all filter coefficients in coVariance filters [H ( Cn in 14p
Therefore, the VNN with filter bank implementation deploys Fin X Fout number of VFs in layer defined by the covariance perceptron in Remark [ Next, we note that the definitions of Varianece filter in Section 2 and VNN are with respect to the sample covariance matrix C, However, due to finite sample size effects, the sample covariance matrix will be a perturbed version of the ensemble covariance matrix C. PCA-based approaches that rely on eigendecomposition of the sample covari- matrix can potentially be highly sensitive to such perturbations. Specifically, if small changes are made to the dataset X,, certain ill-defined eigenvalues and eigenvectors can induce instability in the outcomes of PCA-based statistical learning models [17]. Therefore, it is desirable that the designs of co Vorainacc filters and VNNs are not sensitive to random perturbations in the sample covariance matrix as compared to the ensemble covariance matrix C. In this context, we study the stability of co Variance filters and VNNs in the next section.
4 Stability Analysis
We start with the stability analysis of co Variance filters, which will also be instrumental in establishing the stability of VNNs. Our results will leverage the eigendecomposstiin of C, given by
C= VAVT
(15)
where V E RR is the matrix of eigenvectors such that, A diag Ai.... Am) is the diagonal matrix of eigenvalues, such that, A > 2 Am
[v....., vml and
5

---

4.1 Stability of co Variance Filters
To study the stability of co Variance filters, we analyze the effect of statistical uncertainty induced in the sample covariance matrix with respect to the ensemble covariance matrix on the co Variance filter output. To this end, we compare the outputs H(C)x and H H(C,,) ) for any random instance of X. Without loss of generality, we present our results over instances of X where ||]] S 1. We also consider the following assumption on the frequency response.
Assumption: The frequency response of filter H(C) satisfies:
A
Ay1
(h(A.) h(A,)) &
(16)
ki
where K; - v X for some constant M > O and for all non-zero eigenvalues A + ij. (1....,m) m] of C. Here, k; is a measure kurtosis of the distribution of X. The
Next, we provide the result that establishes the stability of coVariance filters in Theorem 2 impact of the assumption in (T6) the filter stability is discussed subsequently. To present the results in Theorem 2 we also use the following definitions:
kiin mE(L.....m M.
min
and K
max L.
(17)
DA
Ag
Theorem 2 (Stability of co Variance Filter). Consider random vector X RTx1 such that, For a sample its corresponding covariance matrix is given by C = F - -X -
covariance matrix Cn formed using ii.d instances of X and a random instance x of X, such that, ||x|| S and under assumption (16), the following holds with probability at least 1 for any E (0, 1/2]:
2xm/
H H(C.)) H(C)
-H(C)
M O
@Omm++
Vm+
Cllvllg mn
(18)
n
kmin
Proof. See Appendix
The right-hand side term in (18) and the conditions in the assumption in (66) are obtained by the analysis of the finite sample size effect-driven perturbations in C₂ and its eigenvectors with respect to that in C. From Theorem 2 we note that [[,(C,)) H decays with the number of samples n at least at the rate of 1/n Thus, we conclude that the stability of the co Variance filter improves as the number of samples n increases. This observation is along the expected lines as the estimate Č, becomes closer to the ensemble covariance matrix by the virtue of the law of large numbers. Next, we briefly discuss two aspects of the assumption in (66). Note that the upper bound in (T6) controls the variability of the frequency response h(() with respect to A. For any pair of eigenvalues A and A of C, this variability is tied to the eigengap A Ayy and the factor k; We discuss this next.
Discrininability between close eigenvalues: Firstly, for given i; and eigenvalue A₁. the response of the filter for any eigenvalue Xy.J j # becomes closer to h(x:) if A Xy1 decreases. From the perturbation theory of eigenvectors and eigenvalues of sample covariance matrices, we know that the sample-based estimates of the eigenspaces corresponding eigenvalues A; and Xj become harder to distinguish as |λ; XyI decreases [30 Therefore, the co Variance filter that satisfies (16) sacrifices discrininability between close eigenvalues to preserve its stability with respect to the statistical uncertainty inherent in the sample covariance matrix.
Stability with respect to kurtosis and estimation quality: Since we have E|||Xx v;,l [[]]]]* the factor k; is tied the measure of kurtosis of the underlying distribution of X in the direction of v;. Distributions with high kurtosis tend to have heavier tails and more outliers. Therefore, smaller i;; indicates that the distribution of X has fast decaying tail in the direction of v; which allows for a more accurate estimation of X; and v,. We refer the reader to [30] Section 4. 1.33 for additional details. In the context of co Variance filters, we note that the upper bound in (16) is more liberal if X; is associated with smaller k; or equivalently, the distribution of X has a smaller kurtosis in the direction of v,. This observation implies that if X, and v, are "easier" to estimate, the frequency response for xy in the vicinity of X, is less constrained.

---

4.2 Stability of co Variance Neural Networks
The stability of VNNs is analyzed by comparing (xx: C., H) and (xx; C, H... Note that stable graph convolutional filters imply the stability of GNNs for different perturbation models [9]. In VNNs, the perturbations are derived from the finite sample effect in sample covariance matrix, which distinct from the lata-independent perturbation models considered in the existing literature on GNNs, and allow us to relate sample size n with VNN stability. We formalize the stability of VNNs under the assumption of stable co Variance filters in Theorem 3 For this purpose, we consider a VNN (: H) with number of m-dimenniooal inputs and outputs per layer as F and L layers, with the filter bank given by H. {Hi Vf.g E FI....,FF.4 E {l,....
L}. {L....,L}.
Theorem 3 (Stability of VNN). Consider a sample covariance matrix Ö. and the ensemble covari- ance matrix C. Given bank of variannce filters {H ). such that [ (A)] S 1 and a pointwise non-linearity a ()), such that, (a(a) ob)) S a b). if the covariance filters satisfy
H ig (C,) (C)I S On
(19)
for some an 0, then, we have
[x([x C.., H) - (x:: C, 70ll S LF an
(20)
The proof of Theorem 3 follows from [2 Appendix E] for any generic an. The parameter Og represents the finite sample effect on the perturbations in C, with respect to C. From Theorem 2
we note that an scales as 1/nå-c with respect to the number of samples n for the co voraiance filters whose frequency response depends the eigengap and kurtosis of the underlying distribution of the data in (I6). Furthermore, the stability of a VNN decreases with increase in number of m- mn-imennsionl inputs and outputs per layer F and number of layers I., which is consistent with the stability properties of GNNs. Therefore, our results present a more holistic perspective to the stability of VNNs than that possible for generic GNNs.
Remark (Computational Complexity of VNN). For a co cavaanacee percepiron defined in ( (14), the computational cost is given by O(m2 Fin Fout), where T S m is the maximum number of filter taps in its associated filter bank.
From Remark 2 we note that the computational complexity of VNNs can be prohibitive for large m. However, oftentimes sparsity is imposed as a regularization to estimate high-dimensional correlation matrices; see c.g. 33. As result, the computational complexity becomes O(EETFFi
where [E] is the number of non-zero correlations (edges in the covariance graph) and can be markedly smaller than m Since VNN architecture is analogous to that of GNN, the property of GNN transferability (see (6) across different sized graphs can also establish scalability of VNNs to high-dimensional datasets for settings where multi-resolution datasets are available. In the next section, we empirically validate our theoretical results on the stability of VNNs. Moreover, on a set of multi-resolution datasets, we also empirically evaluate VNNs for transferability.
5 Experiments
In this section, we discuss our experiments on different datasets. Primarily, we evaluate VNNs on regression problem different neuroimaging datasets curated at University of Pennsylvania, where we regress human chronological age (time since birth) against cortical thickness data. Cortical thickness is measure of the gray matter width and it is well established that cortical thickness varies with healthy aging 32.. Additional details on the neurological utility of this experiment are included in Appendix Brief descriptions of these datasets are provided next.
ABC Dataset: ABC dataset consists of the cortical thickness data collected from a heterogencous population of n 341 subjects (mean age 68.1 years, standard deviation = 13) that consists of healthy adults, and subjects with mild cognitive impairment or Alzheimer's disease. For each individual, joint-label fusion B3} used to quantify mean cortical thickness in m 104 anatom- ical parcellations. Therefore, for every subject, we have a 104 dimensional vector whose entries correspond to the cortical thickness in distinct brain regions.
Multi-resolution FTDC Datasets: FTDC Datasets consist of the cortical thickness data from 170 healthy subjects (mean age 64.26 years, standard deviation 8.26). For each subject,
7

---

the cortical thickness data is extracted according to multiresolution Schaefer parcellation [34, at 100 parcel. 300 parcel, and 500 parcel resolutions. Therefore. for each subject. we have the cortical thickness data consisting of m 100 features, m = 300 features or higher number of features providing the cortical thickness data of a brain at a finer resolution. We leverage the different resolutions of data available to form three datasets: FTDC100, FTDC300, and FTDC500, which form the cortical thickness datasets corresponding to 100, 300, and 500 features resolutions, respectively.
500 features, with the
Our primary objective is to illustrate the stability and transferability of VNNS that also imply advantages over traditional PCA- approaches. Hence, we use PCA-based regression as the primary baseline for comparison against VNNs. We use the ABC dataset to demonstrate the higher stability of VNNs over PCA-basee regression models in Section In Section 5.2 we use the FTDC datasets to demonstrate transferability of VNN performance across the multi-resolution datasets without re-training. The experiments in Section 5.2 clearly lay beyond the scope of PCA- statistical learning models. Details on data and code availability for the experiments are included in Appendix A
5.1 Stability against Perturbations in Sample Covariance Matrix
In this section, we evaluate the stability or robustness of the trained VNN and PCA reg-reerssoonn models against perturbations in the sample covariance matrix used in training. To this end, we first train nominal VNN and PCA-regression models. The effects of perturbations in the sample covariance matrix the performance of nominal models are evaluated subsequently. We first describe the experiment designs for nominal models based on VNN and PCA-regression.
VNN Experiments: We randomly split ABC dataset into 90/10 train/test split. The sample covariance matrix is formed from 307 samples in the training set, i.c., we have C307 of size 104 x 104. The VNN consists of 2 layers with 2 filter taps each, a filter bank of 13 m-dimensiooal outputs per layer for m 104 dimensions of the input data, and readout layer that calculates the unweighted mean of the outputs at the last layer to form an estimate for age. The hyperparameters for the VNN architecture and learning rate of the optimizer in this experiments and all subsequent VNN experiments in this section are chosen by hyperparameter optimization framework called Optuna (35). The training set is randomly subdivided into subsets of 273 and 34 samples, such that, the VNN is trained with respect to the mean squared error loss between the predicted age and the true age in 273 samples. The loss is optimized using batch stochastic gradient descent with Adam optimizer available in PyTorch library [36] for up to 100 epochs. The learning rate for the optimizer set to 0.0151. The VNN model with the best minimum mean squared error performance on the remaining 34 samples (which acts as a validation set) is included in the set of nominal models for this permutation of the training set.
PCA-regression: The PCA-regression pipeline consists of two steps: i) we first identify the principle components using the eigendecomposition of the sample covariance matrix Č3or: and then, ii) to maintain consistency with VNN, transform the 273 samples from the training set used for VNN training to fit to the corresponding age data using regression model. Regression is implemented using sklearn package in python (37 with 'linear' and radial basis function ('rbf") kernels. PCA regreession with 'rbf' kernel enables us to accommodate non-linear relationships between cortical thickness and age. PCA-regression with linear kernel in the regression model is referred to as PCA-LR and that with 'rbf' kernel in the regression model is referred to as PCA-rbf. The optimal number of principal components in the PCA-regression pipeline are selected through a 10-fold -oos-vaaliattion procedure on the training set, repeated 5 times.
For 100 random permutations of the training set in VNN and PCA-regression experiments, we form a set of 100 nominal models and evaluate their stability. To evaluate the stability of VNN, we replace the sample covariance matrix Č307 with C,,, for n' [5, 341), n' * 307. For PCA-regression models, we re-evaluate the principal components corresponding to C, to transform the training data while keeping the regression model learnt for PCA transformation from Cy07 fixed. Clearly, C,, will be perturbed with respect to C307 due to finite sample size effect.
For each nominal model, we evaluate the model performances in terms of mean absolute error (MAE) and correlation between predicted age and true age for the training set and the test set.
8

---

training set
test set
PCA rot
KCA-1r
0.30
10
0.75
MAXE
9.70
0.00
100 150
350
250
number of sampaap 4o.
number sumples,
(e)
POALE PCAIE
FCkr
7.30
0.70
FCAUB
1.00
0.90
6.0
250
300
Hurmber samppla.
Hummber samplles
(f)
Figure I: Stability of regression performance for VNN and PCA-regression models trained on Ca07 formed from ABC dataset. Panels (a) and (b) illustrate the performance variation on training set for the VNN and PCA-regression models when sample covariance matrix C307 is perturbed by addition or removal of samples (n = 307 marked by blue arrow in (a)). Panels (c)-(f) correspond to the variation in performances in terms of MAE and correlation between predicted age and true age only 307 is marked with a blue arrow in panel (c). Panels (c) and (d) illustrate the variation in mean MAE performance the VNN and PCA-regression models trained using C307 on the test set (n = 307 marked by blue arrow in (c)) with panel (d) zooming in on the range 290 341 with error bars included. Panel (e) illustrates the variation in correlation between true age and predicted age by the VNN and PCA-regression models with panel (f) zooming in on the range 290 341 with error bars included.
on the test set. The point
Figures I b)-b) illustrate the variation in MAE performance with perturbations Č307 on the training set. Figures cf- correspond to similar results on the test set for MAE and correlation. The mean performance in terms of MAE over 100 nominal models is marked by a blue arrow in Fig. a) for training set and in Fig. [ c) for the test set. For PCA-regression models, we observe that both training and test performance terms of MAE degrades significantly when the sample covariance matrix perturbed from C307 by removing or adding even a small number of samples (also seen Fig. () and d), which are the plots focused only the range of 290 341 samples from Fig, a and c), respectively, and include error bars.). In contrast, the VNN performances on both training and test sets are stable with respect to perturbations to C307. as suggested by our theoretical results. However, the number of samples decrease to n'' < 50. we observe significant decrease in VNN stability in Fig. a and c). We also observe that the correlation between the predicted age and true age in the test set for VNN is consistently
the entire range of samples evaluated (Fig. ()). Moreover, Fig. e) also demonstrates that in the range of n' 50 samples, there is sharper decline in the correlation for PCA-regression models as compared to VNN despite the MAE for PCA-regression models having smaller MAE than VNN in this range. Thus, our experiments demonstrate the stability of VNNs while also illustrating that PCA-regression models may be overfit on the principal components of the sample covariance matrix used in training. Additional experiments on synthetic data and FTDC datasets also illustrate the stability of VNNs (see Appendix E)
stable than that for PCA-regression models
5.2 Transferability
We evaluate the transferability for VNN models trained on FTDC datasets across different resolutions. For this purpose, the VNN is trained specifie resolution and its sample covariance matrix is replaced by the sample covariance matrix at different resolution for testing. The training of VNNs

---

MAE
Correlation
Test
Test
Train
Train
FTDC100
FTDC300
FTDC500
FTDC100
FTDC300
FTDC500
FTDC100
5.38 2006
5.47 20007
FTDC100
0.59 20000
0.59 20:05
FTDC300
5.33 +023
5.37
FTDC300
0.62 1030
0.61 1000
FTDC500
5.35 40.5
5.38 2004
FTDC500
0.59 #3005
0.58 #0.06
(a)
(b)
Figure 2: Transferability for VNNs across FTDC datasets.
this context follows similar procedure as described Section 3.1 and results in 100 random VNN models for different permutations of the training set. For FTDC500, the VNN model consists of 1-layer with a filter bank of 27 m-dimennional outputs per layer for m = 500 dimensions of the input data, and 2 filter taps in each layer. The learning rate for the Adam optimizer is set 0.008. For FTDC300, the VNN model consists of 2-layer architecture, with 44 m-dimensional outputs per layer for = 300 dimensions in both layers and 2 filter taps in each layer. The learning rate for the Adam optimizer is set to 0.0033. For FTDC100, 00, the VNN model consists of 2-layer architecture, with 93 m-dimensional outputs per layer for layer. The learning rate for the optimizer is set to 0.0047. The readout layer in each model evaluates the unweighted mean of the outputs of the final layer form an estimate for age.
100 dimensions both layers and 2 filter taps in each
We tabulate the MAE in the matrix in Fig. 2 a) and correlation between true age and predicted age in the matrix in Fig. 2b), where the row ID indicates the dataset for which VNN trained and the column ID indicates the dataset on which the VNN was tested to evaluate transferability of VNN performance across datasets with different resolutions. For instance, the element coordinates (1, 2) in Fig. 2a) represents the MAE evaluated on complete FTDC300 dataset (m 300) for VNNs trained FTDC100 dataset (m 100). The results in Fig. show that the performance of VNNs in terms of MAE and correlation between predicted age and true age can be transferred across different resolutions in the FTDC datasets. Note that this experiment
feasible for PCA-regression models, where the principal components and the regression model would need to be re-evaluated for data from different resolution.
6 Conclusions
In this paper, we have introduced co Variance neural networks (VNN) that operate on covariance matrices and whose architecture derived from graph neural networks. We have studied the stability properties of VNNs for sample covariance matrices and shown that the stability of VNNs improves with the number of data samples n. Our experiments on real datasets have demonstrated that VNNs are significantly more stable than PCA-based approaches with respect to perturbations in the sample covariance matrix. Also, unlike PCA-based approaches, VNNs do not require the cigendecomposition of the sample covariance matrix. Furthermore, set of multiresolution datasets, we have observed that VNN performance is also transferable across cortical thickness data collected at multiple resolutions without re-training.
Acknowledgements
This research was supported by National Science Foundation under the grants CCF-1750428 and CCF-1934962. The ABC dataset was provided by the Penn Alzheimer's Disease Research Center (ADRC; NIH AG072979) at University of Pennsylvania. The MRI data for FTDC datasets were provided by the Penn Frontotemporal Degeneration Center (NIH AG066597). Cortical thickness data were made available by Penn Image Computing and Science Lab at University of Pennsylvania.
10

---

References
[1] Yann LeCun, Yoshua Bengio, and Geoffre Hinton. Deep learning. Nature, 521((553)
2015.
[2] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20( 20(1))61-80, 2008,
[3] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and
Learning Systems, 32(1):4-24, 2020.
[4] Isha Garg, Priyadarshini Panda, and Kaushik Roy. A low effort approach to structured CNN design using PCA. IEEE Access, 8:1347-1360, 2019.
[5] Mohammad- Hosseini, Senbao Lu, Kavin Kamaraj, Alexander Slowikowski, and Hay- greev C Venkatesh. Deep learning architectures. In Deep learning: concepts and architectures, pages 1-24. Springer, 2020.
[6] Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the trans- ferability of graph neural networks. Advances in Neural Information Processing Systems, 33: 1702-1712, 2020.
[7] Ian Jolliffe and Jorge Cadima. Principal component analysis: A review and recent devel- opments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):0010020) 2016.
[8] Tsung-Hat Chan, Kui Jía, Shenghua Gao, Jiwen Lu,, Zinan Zeng, and Yi Ma. Pcanet: A simple deep learning baseline for image classification? IEEE transactions on image processing, 24 (12):5017-500 103221 2015.
[9] Muhammad Umer, Zainab Imtiaz, Saleem Ullah, Arif Mehmood, Gyu Sang Choi, and Byung- Won On. Fake news stance detection using deep learning architecture (CNN-LSTM), IEEE Access, 8:156695 11156695-156766, 2020.
[10] Jonathon Shlens. tutorial on principal component analysis. arXiv preprint 2014.
[11] Antonio Ortega, Pascal Frossard, Jelena Kovačević, José MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106 (5):008-88 -28808888, 2018.
[12] Fernando Gama, Elvin Isufi, Geert Leus, and Alejandro Ribeiro. Graphs, convolutions, and neural networks: From graph filters to graph neural networks. IEEE Signal Processing Magazine, 37(6):128-1) 138, 2020.
[13] E. D. Kolaczyk. Statistical Analysis of Network Data: Methods and Models. Springer, New York, NY, 2009.
[14] Alaa Bessadok, Mohamed Ali Mahjoub, and Islem Rekik. Graph neural networks in network neuroscience. arXiv preprint arXiv:2106.0000 0335355003355.. 2021.
[15] Weiyu Huang, Antonio G. Marques, and Alejandro R. Ribeiro. Rating prediction via graph signal processing. IEEE Transactions on Signal Processing, 66(19):5066-500 2018. doi: 10.1 1L110099TSP2208828644452
[16] José Vinícius de Miranda Cardoso, Jiaxi Ying, and Daniel Perez Palomar. Algorithms for learning graphs in financial markets. arXiv preprint arXiv:2012. 15410. 2020.
[17] Ian Joliffe and BJT Morgan. Principal component analysis and exploratory factor analysis. Statistical Methods in Medical Research, 1(1):69-95, 1992.
[18] Eran Elhaik. Principal component analyses (PCA)-based findings in population genetic studies are highly biased and must be reevaluated. Scientific Reports, 12(1): 2022.
[19] Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability properties of graph neural networks. IEEE Transactions on Signal P Processing, 68:5680-5695, 2020.
[20] Nicolas Keriven, Alberto Bietti, and Samuel Vaiter. Convergence and stability of graph convo- lutional networks on large random graphs. Advances in Neural Information Processing Systems, 33:21512-21523, 2020.
11

---

[21] Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alonso, and Eduard Alarcón. Comput- ing graph neural networks: survey from algorithms to accelerators. ACM Computing Surveys (CSUR), 54(9):1-38, 2021.
[22] Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: comprehensive review. Computational Social Networks, 6(1):1-23, 2019.
[23] Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07300 2016.
[24] Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow, Gated graph sequence neural networks. In Proceedings of ICLR' 16, 2016.
[25] Luana Ruiz, Fernando Gama, and Alejandro Ribeiro. Graph neural networks: architectures, stability, and transferability. Proceedings of the IEEE, 109(5):660-633 682, 2021.
[26] Emmanuel Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
[27] Zihan Zhou, Xiaodong Li, John Wright, Emmanuel Candès, and Yi Ma. Stable principal component pursuit. In JEEE International Symposium on Information Theory, pages 1518- 1522, 2010. doi: 10. 10011001211220005555555
[28] Huan Xu, Constantine Caramanis, and Sujay Sanghavi. Robust PCA via outlier pursuit. Advances in Neural Information Processing Systems, 23, 2010.
[29] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[30] Andreas Loukas. How close are the eigenvectors of the sample and actual covariance matrices? In International Conference on Machine Learning, pages 2228-2237. PMLR, 2017.
[31] Jacob Bien and Robert J Tibshirani. Sparse estimation of a covariance matrix. Biometrika, 98 (4):807-82) 00077-20,220 2011.
[32] Madhav Thambisetty, Jing Wan, Aaron Carass, Yang An, Jerry L Prince, and Susan M Resnick. Longitudinal changes in cortical thickness associated with normal aging. Neuroimage, 52(4): 1215-1223, 2010.
[33] Hongzhi Wang. Jung W Suh, Sandhitsu R Das, John B Pluta, Caryne Craige, and Paul Yushkevich. Multi-atlas segmentation with joint label fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(3):611-633 2012.
[34] Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy Laumann, Xi-Nian Zuo, Avram Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity MRI. Cerebral Cortex, 28(9):0095-311 2018.
[35] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2623-2631 2019.
[36] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[37] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. The Journal of Machine Learning Research, 12:2825-288 2830, 2011.
[38] Gene H Golub and Charles F Van Loan. Matrix computations. JHU Press, 2013.
[39] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge University Press, 2018.
[40] Dane Taylor, Juan G Restrepo, and François G Meyer. Ensemble-based estimates of eigenvector error for empirical covariance matrices. Information and Inference: A Journal of the IMA, 8(2): 289-312, 2019.
12

---

[41] Xavier Mestre. On the asymptotic behavior of the sample estimates of eigenvalues and eigen- vectors of covariance matrices. IEEE Transactions on Signal Processing. 56(11))5353-55 2008.
[42] Hedieh Sajedi and Nastaran Pardakhti. Age prediction based on brain MRI image: A survey. Journal of Medical Systems, 43(8):1-30, 2019.
[43] SA Valizadeh, Jürgen Hänggi, Susan Mérillat, and Lutz Jäncke. Age prediction the basis of brain anatomical measures. Human Brain Mapping, 38(2):997-1088, 2017.
[44] Habtamu M Aycheh, Koun-K Seong, Jeong-Hyeon Shin, Duk L Na, Byungkon Kang, Sang W Seo, and Kyung-Ah Sohn. Biological brain age prediction using cortical thickness data: large scale cohort study. Frontiers Aging Neuroscience, 10:252, 2018.
[45] Jeffrey S Phillips, Fulvio Da Re, David J Irwin, Corey T McMillan, Sanjeev N Vaishnavi, Sharon X Xie, Edward B Lee, Philip A Cook, James C Gee, Leslie M Shaw, et al. Longitudinal progression of grey matter atrophy in non-amnestic Alzheimer's disease. Brain, 142(6):1701-
1722, 2019.
[46] Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discussed limitations of our
work in Remark 2 and the discussion thereafter.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Data and code
availability statement is available in Appendix [A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The hyperparameters in the VNN architecture and learning rate of the optimizer are chosen by hyperparameter optimization framework called Optuna [55
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
The experiments were run on 12GB Nvidia GeForce RTX 3060 GPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [Yes] All participants in the ABC and FTDC datasets took part in an informed consent procedure approved by an Institutional Review Board convened at University of Pennsylvania.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] The data contains no personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
participants in the ABC and FTDC datasets received monetary honorarium (of $25 $40 per participant for participation).
14


Reviews:
[{"review_id": "O5GeMELKAA", "reviewer": "Reviewer_TzhN", "summary": "In this paper, the author first makes an observation that the filters of GNNs show similarities with principal component analysis (PCA), in which data is projected on the eigenspace of the covariance matrix. Then they proposed the covariance neural network (VNN) that operates on sample covariance matrices as graphs are motivated by this observation. They theoretically demonstrate VNN stability to perturbations in the covariance matrix, indicating a qualitative advantage over traditional PCA-based data analysis approaches that are prone to instability due to close eigenvalues and principal components. The author also makes several real work experiments to empirically prove their statement.", "questions": "How about the scalability of VNN? The experience of this paper is mainly focused on 'small' graphs. When it comes to large graphs, is it still computationally feasible? When VNN is used in large graphs, what will be the biggest challenge?", "limitations": "There is no limitation and social impact in this paper. According to Neurips instructions, hope these two parts will appear in a future version.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\nIn this paper, the author links the coVariance filter with the graph convolution filter in GNN and proposes a DL architecture based on the coVariance filter. The computing covariance matrix for a large graph is always computation expensive, by alternatively computing graph Fourier transformation show potential for further GNN development.\n\nThe author also theoretically analyzes the stability of the covariance filter and covariance graph neural network and empirically evaluates VNNs for transferability and stability.\n\n\nWeakness:\n\nThere is no related work section in this paper. Although those who study GNN are familiar with GNN, PCA, and graph Fourier transformation, it would be better to give an overview of different kinds of GNN and summaries them.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "F1acbaHS4EW", "reviewer": "Reviewer_NdkQ", "summary": "The authors propose a coVariance Neural Network (VNN) in analogy to graph neural networks based on covariance matrices of datasets, similar to what is used in PCA. The authors also introduce a “covariance transform” which projects a new datapoint into the PCA space by projecting to the eigenvectors of the covariance operator $UX$ and a covariance filter which learns coefficients of polynomial of the covariance matrix.  They show that using covariance filters one can recover the covariance transform. They also show that this polynomial filter is stable with respect to perturbations in the sampled data from which the sampled covariance matrix is taken. ", "questions": "What are other applications of the covariance filter that would not be subsumed by SVD low rank approximations? ", "limitations": "I do not see a discussion of limitations in this manuscript. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors correctly note the similarity between the covariance matrix and an adjacency matrix of a graph, i.e., positive semidefinite matrices whose eigenvectors and values have specific properties in terms of describing the heterogeneity in the data. In fact it can be argued the other way that graph fourier transforms and graph signal processing are the result of a “kernel trick” based on PCA which is far older.  So in this sense GNNs/GCNs are a more general class of networks that subsumes VNNs. \n\nThe note that this results in a stable PCA is interesting although over the years there have been robust forms of PCA that have been developed including variations of Principal component pursuit.\n\nWeakness:\n\nHowever,  I do not believe their proposal constitutes a new neural network of any sort. I believe this is just an application of GCNs to the graph consisting of data features as nodes and data points as signals on the nodes (rather than vice versa). While this may be desirable for some types of analysis (see Tong et al. IEEE ICCASP 2022 for analysis of cells as signals over gene graphs), I do not believe it has advantages over robust PCA for this specific application. \n\nThe paper should potentially be reconfigured to simply talk about an application of GCNs/GNNs to feature covariance matrices and situations where that could be useful. \n\nFurther a key weakness here is regarding relationships between data features as defined by covariances. This is a strictly linear relationship, if this was changed to mutual information or some other relationship type then indeed a more complex relational graph would be necessary and this is precisely where GNNs/GCNs have contributed. \n\nOther uses of the covariance filter seem to amount to low-rank approximations done via SVD and again I don't see much advantage in using a neural network for this kind of linear operation. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Byfa96xBZVs", "reviewer": "Reviewer_EVHZ", "summary": "Motivated by similarities between principal components analysis (PCA) and graph convolutional filters in GNN, the authors introduce a new GNN architecture that uses sample covariance matrices as the graph representation of the data. They develop “coVariance” filters analogous to graph convolutional filters in GNNs, show that PCA is a special case of applying such filters, and propose a deep learning architecture based on “coVariance” filters they call “coVariance Neural Networks” (VNNs). Using perturbation theory for sample covariance matrices they theoretically establish the stability of VNNs to perturbations in the sample covariance matrix (in terms of number of samples). They then empirically evaluate VNN stability relative to PCA on synthetic and real world data, and show that VNNs can be used transferably on one multiresolution dataset. ", "questions": "I would like to know more about model training time and computational cost in terms of covariance matrix size and number of samples. ", "limitations": "I think the authors adequately address limitations in their work.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The work is original, making an interesting connection between GNNs and PCA and rigorously following this through the development, theoretical stability, and empirical validation of VNNs.\n- The results are likely to be of interest to a broad audience, given the widespread use of PCA and prevalence of correlation matrices across disciplines. \n- The theoretical stability analysis is sound and backed up by validation on synthetic and interesting real world neuroimaging data.\n- The demonstration of multi-scale transferability is very cool, and goes beyond what PCA is capable of.\n- The manuscript is clearly written and code is provided for replicability.\n\nWeaknesses: \n- I find the paper to have few weaknesses. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept