Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors establishes a connection between GPCA and graph convolution (graph convolution as a first order approximation of GPCA), and uses that connection to 1. propose a GPCANet model and 2. propose a new initialization strategy for GNNs. 
main_review: The biggest strength of this paper is the strong experimental results on their new initializations scheme on deep GNNs. I think that result alone could be the basis of a decent paper, if other major concerns are addressed. 

Below is a major concern that the authors need to address: 

1. the first order approximation statement (in the sense of the first order Taylor series approximation) only holds if the learned parameters W in the graph NN are the eigenvectors of X^T \phi_\alpha^{-1}X. If the learned parameters W in the NN are not the appropriate eigenvectors, then there is no meaningful first order approximation going on. The authors did not provide any justification for why the weights of the NN will be the eigenvectors of said matrix (though this is an interesting conjecture given that GPCA provides good initialization). I find the authors first-order approximation claim, at least as stated on a theoretical level, quite inaccurate. Perhaps a more accurate description be "there exist some parameter W for the GCN under which the GCN convolutional operator becomes a first order approximation of GPCA". This would be a much weaker (though more accurate) statement of the authors' claims. 


summary_of_the_review: Nice experimental results. Theoretical claim misleading/inaccurate as stated. If theoretical claim is fixed, then this could be a good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper relates GCN to PCA from the perspective of optimization. The authors propose Graph PCA that is a general form of GCN. They further introduce a regularization term that enforces nodes with same labels close to each other. 
main_review: The proposed method is technically sound.  The relation between PCA and GCN is well established in this paper.  My concerns are as follows,

1. The experimental results of GPCANET-INIT is insignificant. GPCANET-INIT performs very close to XAIVER-INIT when L=2 and L=3. Although GPCANET-INIT performs better than  XAIVER-INIT as the number of layers increase, it is not meaningful to compare them since the performance does not get better when the model gets deeper.
2. The paper lacks a unified ablation study. It is unclear to me which part of the proposed model contribute most to the model performance.
3. The computation cost of the proposed method is expensive due to eigenvalue decomposition.
 
summary_of_the_review: The biggest contribution of this paper is that it derives a more general GCN from the perspective of PCA.  The paper is well written and easy to follow. However, due to computation bottleneck and insignificant empirical results, I would rate this paper with a weak accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to establish the relationship between Graph PCA and Graph Convolutional Layer, so as to define a new Graph Neural Network based on graph PCA.
main_review: ### Reasons to accept
1. The paper shows us that regularization of weights may be necessary for GNNs.
2. A new mixhop network architecture has been proposed that seems to be more promising than mixup performance.
3. This paper is well-written and easy to follow.

### Reason to reject
In general, I think the novelty of this article is insufficient mainly in the following points.
1. GPCA is only GC Operation with an orthogonal constraint. Eq (5) ||X-ZW^\top||_F^2+\alpha\Tr{Z^\top L Z} is able to reformulated as:
||XW-ZW^\topW||_F^2+\alpha\Tr{Z^\top L Z}. Because of W^\topW=I, we can have a GC operator defined in (1). So all things in this paper are based on the GC operator with the orthogonal constraint. 
2. The definition of supervised formulation can hardly be considered as an original definition, and such methods are widely used in semi-supervised manifold learning (so much relevant literature), but this paper does not seem to be a suitable reference for the relevant methods.
3. The performance of the algorithm, as I analyzed earlier, is not very different and has significant advantages.

### Recommendation
1. Unless authors can theoretically analyze what benefits the orthogonal constraint actually brings, it is really hard to evaluate the contribution of this paper.
2. cite papers about semi-supervised (or supervised) manifold learning.

(1) Zhu and et, al Interpreting and Unifying Graph Neural Networks with An Optimization Framework, WWW 2021.

summary_of_the_review: Basically, the whole paper is based on posing an orthogonal constraint on GCN. However, they do not provide any convincing theoretical justification for that. Although it is a good paper to read, I tend to reject this paper because of a lack of novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript looks at the classic graph-regularized PCA (GPCA) and try to build the connection with GPCA and the state-of-the-art GCN, and finally proposes a new deep graph network GPCANet.  The authors make a number of clear contributions as listed in the paper: 1) they build the connection between GPCA and GCN, 2) Based on this connection, they propose novel way of using such GPCA as a graph layer or as an initialization process for training GCN etc. and 3) thus present a new architecture of GPCANet that performs well on their tests. 
main_review: The contributions presented in this manuscript are quite obvious to my knowledge, but not very novel. Basically the paper is well-presented and written with great readability.  I cannot see any obvious flaws in the theory, analysis and experiments presented in this paper. In addition, the paper provides sufficient experimental evidence to back the main claims introduced by the authors. 
The paper builds around the careful revisit to the classic graph-regularized PCA (GPCA). This observation is very interesting indeed. GPCA combines the PCA and the graph-structure information when building embedding (or latent representation) of the data, i.e., the feature linear transformation is restricted to the orthogonal projection. In fact, the PPNP is a similar way without restricting to the orthogonal feature transformation. In theory, for both GCN and PPNP, it is naturally to consider orthogonal transformation (parameter W) to make them as a constrained network, thus GPCANet becomes this special case. From this aspect, the novelty of the paper is limited. Similarly the idea of the semi-supervised version is not absolutely new, this has been long in application in supervised dimensionality reduction framework.    
summary_of_the_review: In general, there is no theoretical flaw in the paper, and the paper is well presented. However the novelty is limited as the proposed model is a parameter constrained NNPN or GCN.
review_title: null

Meta-review: The paper presents several related results. The initial main result consists in relating GPCA to GCN, showing that GPCA can be understood as a first order approximation of some specific instance of GCN where the W matrix is directly defined on data. This result is then exploited to define a supervised version of GPCA. As a follow-up the authors propose a novel GPCA-based network (GPCANet) and a GPCANet initialisation for GNNs. The paper is well written and easy to read. Empirical results are reported to verify the above mentioned connection between  GPCA and GCN, as well as the performances of  GPCANet  and the proposed initialisation for GNNs. Overall, while the mentioned connection was never explicitly reported in the literature, its existence is not surprising and thus its significance seems to be limited. Also the performances of GPCANet do not seem to be significant from a statistical point of view. The novel initialisation procedure for GNNs seems to be interesting and promising, although the used datasets may not make evident its full power. Authors rebuttal and discussion did not change the reviewers' initial assessment.
Decision: Reject