Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
- In addition to encouraging the node embeddings to be smooth over the graph space, the paper further regularize the embeddings to be smooth among different features.
- The method outperforms comparison methods when data is polluted with three types of random noise.
- The paper is well-organized and clearly written. To my best knowledge, the method is technically sound.

Cons:
- Compared with previous methods, the proposed method only achieves comparable accuracy performance on the real-world datasets.
- The Gaussian noise is too weak to evaluate the robustness of the method.
- It is better to give an analysis of the computational cost of Equ.(11) and provide an empirical speed comparison.
review_title: Regularizing GNN on feature graph is interesting, but more careful evaluation is needed.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a new graph convolutional network. It considers not only the original graph structure information but also the latent correlations between features, resulting in a graph neural network as a bi-directional low-pass filter. The new filter is derived using the alternating direction method of multipliers (ADMM) algorithm. Experiments show the new model's denoising performance is better than previous models.

Pros:

- The idea of using feature correlations in graph neural networks is interesting. This technique seems to improve the model’s denoising performance.

- The new model outperforms previous graph neural networks on most of the benchmark datasets in different noisy settings.

Cons:

- The ADMM algorithm is applied to solve the optimization problem (4). However, the convergence of this algorithm with Taylor approximations is not provided in this paper. In fact, there are simpler algorithms with convergence guarantees that can be applied to solve the Sylvester equation (5), e.g., the gradient based iterative algorithm in "Gradient based iterative algorithms for solving a class of matrix equations" by Feng Ding and Tongwen Chen.

- All graph neural networks are compared when they have two-layers. However, in the new model, $L_2$ is a learnable symmetric matrix, which makes the new model more complex compared with other models. This comparison might be unfair to other models.

Because of the above reasons, I am leaning towards rejection. Below are some additional comments.

1. On page 3, "the smoothness of a graph signal $x$ can be measure through..." should be "... measured ...".

2. The notations in Figure 1 are not consistent with the notations in the paragraphs. It is better to make them consistent to avoid confusion.

3. On page 4, "$L$ is the (normalized) Laplacian matrix." It is better to give the definition of the normalized Laplacian matrix for completeness.

4. On page 4, section 4.2, $L_2$ and $L'$ are used interchangeably. It is better to make them consistent.

5. On page 5, "all of them require Schur decomposition which including Householder transforms..." should be "... includes..."

6. On page 5, equation (6), why is the L2 term evenly split into $f(Y_1)$ and $g(Y_2)$? What will happen if we set $f(Y_1) = ||Y_1-F||_F^2 + \lambda_1 trace(Y_1^TL_1Y_1)$ and $g(Y_2) = \lambda_2 trace(Y_2L_2Y_2^T)$?

7. On page 5, "... by choosing appropriate hyper-parameters $p$..." The matrix $L_2$ is learnable in the model, which makes it difficult to guarantee that the eigenvalues of $2\lambda_2L_2/(1+p)$ all fall into $[-1,1]$.

8. Please include the initializations of the ADMM algorithm.

9. On page 6, "we update $Y_1$ by appling the... then update $Y_2$ by appling the..." should be "... applying... applying..."

10. On page 6, "It also explains that BiGCN is more expressive that single-direction los-pass filtering GCNs" should be "... than..."
review_title: Official Blind Review #4

final_score: rating: 6: Marginally above acceptance threshold
review_content: The manuscript introduces a graph convolutional layer based on the optimal solution of the minimization problem of recovering the true graph signal given a noisy observation. Moreover, the authors propose to consider the original graph structure information together from the latent correlation between nodes features. The proposed solution is interesting and the paper is well written. Only section 4.2 turns out to be a bit difficult to read. My suggestion is to explain a bit more deeply the concept of latent features connection graph already in the introduction section.  Another point that it is not clear to me, is related to the fact that the authors highlight several times the connection between the proposed convolution and the low pass filter concept. In (NT & Maehara, 2019) the authors state that many papers on graph neural networks iteratively multiply the adjacency matrix and this operation corresponds to a low-pass filter. Therefore it is not clear to me the reason why the proposed convolution differs from the convolutions in the literature on this aspect.

For what concerns the experimental results it is interesting to notice that the BiGCN is significantly more robust than the models considered in the comparison it is not clear how the proposed model performs on clean data. The results reported in appendix D.2 show comparable performance with the other considered models. The problem is that methodology used in the comparison in my opinion is not completely fair. Indeed the authors in Appendix D state that they used the same hyperparameters (lr,  weight decay, and dropout) for all benchmarks and baselines, while the epoch was chosen based on the validation set. Honestly, I think that in this way the authors introduce a bias on the comparison that could affect the results. In this regards in appendix E2 the authors state “We tune our hyperparameters for each model using validation data” (to me it is not clear what the authors meaning with “our hyperparameters”) but the information about how the validation process is performed is missing (grid-search/random search?). It is also not clear if this procedure is used also for the baseline models (for instance, how the hidden dimension had been chosen?).

In the last part of section 5, the authors discuss the “Structure mistake case”. Since the authors consider also the semi-supervised node classification task, where dataset like Cora, Pubmed, and Citeser basically use a very small sub-graph as a training set, it is interesting to know how much the introduced incorrect interaction relationships among the nodes, interest the training/test/validation sets. In the paragraph about the “Noise rate case” a more deep discussion about the results in the Cora dataset should be inserted. In particular to me, it is not clear why on Cora the GCN outperforms the BiGCN (in Citeseer the performance is very similar), while in Pubmed GCN archives one of the lower accuracy. Is there any difference in the data that justify this behavior?

In my opinion, in the set of models considered in the comparison, the authors should also consider ARMA, which has the advantage to be more robust than the other models in the literature (as reported in section 2.2). I suggest also considering in the comparison some more novel models proposed in the literature e.g. the ones defined in “Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks” by Luan et al. (2019),  that obtained very interesting results in the considered datasets on semi-supervised nodes classification.

Minor comments:
-In section 3 the symbol \lambda_i is used without been introduced it in advance;
-in sections 3/4 the authors use the same symbol (L) for the  Laplacian Matrix and for the normalized Laplacian matrix.
-please insert the complete appendix name when it is referred. For instance, in the last part of section 4, the authors state “ More technical details are added in Appendix”, without specifying which is the appendix referred to in this context.
review_title: A good paper that introduces a GCN that is more robust to noisy data than some models in the literature

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 

The authors proposed a to apply “low pass filtering” on both node and feature domain.

Pros:
1.	Interesting idea on trying to define and apply low pass filter on feature domain
2.	The experiment results sort of validates the proposed idea.

Cons: 
1.	The definition of “feature graph” seems questionable. Moreover, the authors propose to learn the feature graph $L_2$ in the actual implementation. Compare to node domain where the graph topology is given by data, this asymmetric is very weird to me.
2.	The motivation of applying additional filtering on feature domain seems redundant. Doesn’t the weight matrix $W^{(l)}$ in each layer work as the feature transformation already?

Detailed comments:

The main weakness of the paper would be its motivation. Indeed, as the author claim, there might be some correlations among features. However, the weight matrices $W^{(l)}$ already serve as feature transformation for the $l^{\text{th}}$ GCN layer. Why do we need additional “low pass filtering” in feature domain? If the methodology of this paper is correct, then why we still need the additional weighted matrix $W^{(l)}$ as in equation (11)? Also, even if we are given with some feature graph, why the low pass filtering defined with respect to this feature graph is reasonable? Note that the low pass filtering on graph is strongly related to the “Homophily principle”[1] which has been verified through extensive studies in network science community. However, it is not clear that the same conclusion will hold for the newly defined feature graph. Lastly, even in node domain the homophily principle is known to be invalid for some real graphs [2]. I think the authors should explain and discuss in more detail on this.

Consider the case were we only have $1$ node with $2$ features $Y = [y_1,y_2]$. Furthermore, consider the node label function $f(y_1,y_2) = sign(y_2-y_1-1)$. Also, let $y_2 = 11y_1$ and $y_1$ is generated uniformly in range $[0,1]$. In this case, we can see that graph topology in node domain is irrelevant and we can focus on the feature domain. Also, the label function is positive with high probability (0.9) and the correlation coefficient between $y_1,y_2$ is $1$. According to the definition of feature graph (Section 4.2, page 4), the node $y_1$ and $y_2$ are linked and the corresponding low pass filter is just averaging $\frac{y_1+y_2}{2}$ (up to a constant factor). Hence, the output feature matrix becomes closer to $[\frac{y_1+y_2}{2},\frac{y_1+y_2}{2}]$ as $\lambda_2$ larger (equation (4)). Note that even if we know the optimal label function $f$, now we are more likely to predict negative label (with probability 1 when $\lambda_2\rightarrow\infty$). In this example, it seems like applying low pass filtering in feature domain actually hurt the label prediction performance. What I try to convey here is that merely considering the correlation in node feature is insufficient. We have to also take the label correlation into account. Maybe I misunderstand something here but I hope the authors can elaborate on this.

Even if we accept the idea that we really need some low pass filtering in feature domain, the definition of “low pass filtering” given by authors seems problematic to me. In node domain, since we are given the graph topology directly from data, we are able to define “frequency” based on the spectrum of graph Laplacian. In contrast, we are not given “feature graph” that characterizes the correlation between features. The author proposed to learn the graph Laplacian of the underlying feature graph $L_2$ but I doubt the correct $L_2$ can be learnt. Furthermore, the authors use different $L_2$ for each layer which is a bit counterintuitive to me. It is weird that the underlying “feature graph” topology will change across layers. Note that the graph topology remains the same for all layers in GNNs. Finally, instead of using this convoluted design, why not just apply multilayer perceptron (MLP) to do the job? It is known that MLP can approximate arbitrary function and thus should be able to approximate the underlying “low pass filter” for feature graph. I think the authors need to explain better on why the proposed design is necessary under their motivation.

As a final comment, I think the experimental results seems interesting. Indeed, the noise is injected artificially but it kind of verifies the idea of low pass filtering in feature domain for Gaussian noise case. The paper would be greatly improved if the concerns above can be addressed.

Reference:

[1] “Birds of a feather: Homophily in social networks.,” McPherson et al., Annual review of sociology, 2001. 

[2] “Geom-GCN: Geometric Graph Convolutional Networks,” Pei et al., ICLR 2020.


review_title: The experiment seems to show some evidence for the proposed idea but the methodology itself is doubtful

Meta-review: From the positive side the problem addressed by the paper could be of potential interest in the case there is noise in the features associated to each node of the graph. The paper is mostly well written and clear. The proposed approach is based on solid mathematical grounds.

On the other hand there are concerns about:

i) motivation: it is not clear how significant the proposed approach is since the authors were not able to clearly highlight the advantages with respect to the standard approach where already the weight matrix (via learning) can play the role of a low-pass filter for node’s features. Maybe the main advantage is given by the fact that the network does not have to learn a low-pass filter, however this needs a better clarification;

ii) suggested approach: the authors are using an approach that seems to be more complex with respect to simpler ones already proposed in literature and not mentioned in the paper. In addition to that, the simpler approaches have convergence guarantees that have not been proved for the proposed approach;

iii) significance of the experimental results: the obtained experimental results are obtained by using a model with more parameters with respect to the baselines. Comparisons versus baselines with a similar number of parameters are necessary to have a fair assessment of the merits of the proposed approach.

Decision: Reject