Defending Graph Neural Networks via Tensor-Based Robust Graph Aggregation

Abstract: Graph Neural Networks (GNNs) have achieved outstanding success in a wide variety of domains and applications. However, they are still vulnerable to unnoticeable perturbations of graphs specially designed by attackers, causing significant performance drops. Developing algorithms to defend GNNs with robust graphs vaccinating from adversarial attacks still remains a challenging issue. Existing methods treat every edges individually and regularize them by specific robust properties, which ignores the structural relationships among edges and correlations among different properties. In this paper, we propose a tensor-based framework for GNNs to learn robust graphs from adversarial graphs by aggregating predefined robust graphs to enhance the robustness of GNNs via tensor approximation. All the predefined robust graphs are linearly compressed into and recovered from a low-rank space, which aggregates the robust graphs and the structural information in a balanced manner. Extensive experiments on real-world graph datasets show that the proposed framework effectively mitigates the adverse effects of adversarial attacks and outperforms state-of-the-art defense methods.

Reviews and author responses:

The paper presents a tensor-based framework to improve robustness of GNNs by denoising the adversarial graphs. By leveraging the low-rank and feature similarity properties of graph data, the authors use three predefined robust graphs with the adversarial graph to form a graph tensor, and then apply tensor decomposition methods to learn a robust graph. The experimental results show that the framework outperforms baselines under different adversarial attacks.  We summarize the Strengths and Weakness of this paper as following:

**Strengths**:
1. Using tensor decomposition methods and different predefined graphs with different properties to learn the low-rank robust graph is reasonable.
2.The paper is well organized and easy to understand.
3.The experiment is comprehensive, and the results demonstrate that the proposed method outperforms baseline methods by a large margin.

**Weaknesses**:
1. Using low-rank and feature smoothness to learn clean graphs does not have significant novelty beyond currently existenting methods. For example, Pro-GNN also proposed a similar idea. Moreover, the three predefined graphs (PRUNE, SVM and kNN) are also not new.
2.The tensor decomposition methods already used in some graph-based tasks. Please refer to these papers [1], [2], which also implement similar approaches. 
3. In the experimental section, the authors compare the proposed method with baselines, which are most coupled GNNs, such as GCN and GAT. How about the decoupled GNNs which achieve state-of-the-art performance in node classification task, such as APPNP [3]?
4. The proposed method is highly dependent on node features, so it might not be suitable for feature attack. For example, if there are adversaries perturbing the node features as well, the perturbed information can also affect the generation of the robust graphs. This will jeopardize the robustness. 

**[1]** Balažević, Ivana, Carl Allen, and Timothy M. Hospedales. "Tucker: Tensor factorization for knowledge graph completion." arXiv preprint arXiv:1901.09590 (2019).
**[2]** Kazemi, Seyed Mehran, and David Poole. "Simple embedding for link prediction in knowledge graphs." arXiv preprint arXiv:1802.04868 (2018)
**[3]** Klicpera, Johannes, Aleksandar Bojchevski, and Stephan Günnemann. "Predict then propagate: Graph neural networks meet personalized pagerank." arXiv preprint arXiv:1810.05997 (2018).
 Based on the discussion above, the methods in the paper may not have great novelty. However, the experimental performance improvement is significant and the experiment is comprehensive. We would vote for “weak accept” to this paper. 

The paper proposes a tensor-based defense strategy against adversarial perturbations to graph neural networks. Specifically, the authors propose to aggregate several "robust" adjacency matrices obtained from previously proposed methods into a tensor, which is then decomposed. Based on this decomposition, we can then obtain a new robust graph, with the idea of combining the strengths of the different robust approaches. Adversarial training and perturbations of GNNs is clearly an interesting topic which fits well into the landscape of ICLR. The authors propose a technique based on tensor decomposition to improve the robustness of GNNs in practice.  The numerical experiments are indeed relatively promising. I have some major concerns with this work however.

There is basically no theoretical analysis provided as to why this approach should work. The intuition is explained by the authors, but not much more. In particular since the tensor decomposition is effectively a low-rank approximation approach, I do not quite see how certain graph features would be maintained if the "robust" adjacency matrices included in the tensor would not all pick them out?

The computational complexity of their approach is not discussed at all as well. If I understood correctly we essentially would have to perform a large suite of robust methods to obtain the robust adjacency matrices to start with, before we could use this approach. This is a very large additional cost and there seems to be no way around this? 

In addition, computing a tensor decomposition is itself a computational difficult problem. How do the authors guarantee that they have found the right decomposition? How to chose which decomposition to use, with which number of components etc? All this seems not very clear.

Given that there is no real theory here, I would also have expected more in-depth comparisons to other methods (more GNN architectures beyond GCNs; more datasets; not just poisoning attacks), to support their claims. 


Minor points:
- The abstract claims that current methods for vaccinating GNNs treat edges independently and individually; however this is not true for the low-rank approximation methods (SVD based). In summary, I think the paper contains a somewhat interesting idea and some promising numerical experiments.

What is a however lacking is
a) a solid theoretical foundation for why this method should work in practice
b) an analysis of the computational costs
c) a more thorough discussion on the computations and the choices made for the tensor decomposition
d) a stronger empirical validation (given in particular the apparent lack of theoretical results).

This paper studies the problem of defending against adversarial attacks for graph neural networks. It first defines some robust adversarial graphs and aggregates the predefined graphs to form a powerful regularization. Then low-rank tensor approximation is applied to generate a more robust graph structure while jointly learning the GNN parameters.
graph structure. Extensive experiments demonstrated the effectiveness of the proposed method.
 The defense again adversarial attack is an important problem in the GNN community. It is interesting to see that the author formulates the graph learning process as a 3D tensor decomposition problem. As shown in the empirical results, the improvement is impressive under various types of attacks and the learned graphs are shown to preserve the common patterns of robust structures.

However, there are still some concerns that need to be addressed.
1. Analysis of time/model complexity should be included.  The predefined graphs involve the construction of kNN graphs and singular value decomposition, which can be time-consuming for large graphs. It is desired to provide some comparison on the time/model complexity between the proposed method and existing defense methods.
2. It would be more convincing to include larger datasets to demonstrate the effectiveness of the proposed model. As the proposed method assumes the adjacency matrix to be low-rank, it remains a question whether proposed model can be applied to other types of graph datasets beyond citation networks (i.e., cora, citeseer and pubmed).
3. The method seems to rely on the unperturbed features. I am curious whether the proposed defense will be easily fooled under adversarial attacks perturbing node features. In my view, the feature perturbation may mislead the construction of kNN graph and result in terrible performance. 
 Overall, I feel this is a boarderline paper. The empirical improvement seems significant but there are still some aspects that can be improved.

In this paper, the authors propose a tensor-based framework for GNNs to learn robust graphs from adversarial graphs by aggregating predefined robust graphs to enhance the robustness of GNNs via tensor approximation.
 ### reasons to accept
1. this paper is well-written and easy to follow.
2. the proposed method is efficient and effective. 
3. The idea seems can be used in more than the adversary attacks.


### reasons to reject
1. For me, I believe the real reason the proposed method works well is ensemble learning. This article lacks the necessary ablation experiment to demonstrate the necessity of tensor approximation. For example, maybe just using PCA to find low-rank representation for graphs is good enough because it is able to ignore the noise and keep the invariant components for aggregation.


### recommendation
Honestly, I find this method quite interesting. It is absolutely able to employ far more than the adversary scenario. For example, in most cases without noise, it is able to outperform different GNNs. Thus I guess there is theoretical justification to prove something. I really encourage the authors to think about the properties of this framework, not just throw it out and let readers think about it.
 The idea is interesting. And the result is good enough. But I really hope the authors can dig something more rather than just propose a method for the adversary scenario. I tend to give a 6 to encourage the authors to show us something more convincing. If it is not, I will turn my score down.

Meta-review: This paper has been reviewed by four reviewers with three borderline scores leaning towards an accept and one clear reject. Reviewers have raised a number of issues. They feel that *the paper is borderline* as *the paper may not have great novelty* due to the use of low-rankness even though it is used for the low-rank tensor approximation and that *larger datasets* should be used to demonstrate the effectiveness of the proposed approach (even though there are no papers doing it on the large scale graphs to be fair).

Also, reviewers note that they would like to see more theoretical justifications rather than just to see authors *propose a method for the adversary scenario* without full theoretical analysis. For instance, reviewers xxhm and WHUo were seeking the novel theoretical analysis in the context of adversarial robustness rather than a statement that *the problem of recovering the data under gross error has gained much attention* followed by the list of prior papers and an outline of their findings.

While all reviewers agree that the empirical results look very promising, they also agree that the theoretical analysis needs an improvement. For the above reasons, however tempting, even if overlooking the reject score from the reviewer xxhm, it is difficult for AC to advocate for a clean accept.

Decision: Reject