Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper brings fresh ideas to adversarial graph learning. The proposed methods provide new schemes to train GNN models by predictive coding instead of backpropagation. In particular, Graph Predictive Coding Network modifies predictive coding from general machine learning to message-passing neural networks (GNNs). The design of the experimental section provides a new perspective (using confidence to compare robustness) to show GCPN's better effectiveness on attacked graph samples than other baselines.
strength_and_weaknesses: Strength:

1. This paper has a clear motivation and advancing idea. In the introduction section, the authors point out that the robustness of graph neural networks remains a lot of issues to address. Inspired by prior works in other domains, the work presents a new training mechanism on graph neural networks. It finds that the energy-based model and the predictive coding are worth replacing the commonly used backpropagation, then introduces both ideas into message-passing neural networks. 

2. The proposed methods establish a non-back propagation approach to train GNNs, which is a technically novel style and different from prior GNNs training works. In detail, Graph Predictive Coding Networks include an energy-based model and predictive coding, which not only minimize the same energy function with two different message-passing mechanisms but also filter the message being passed down from direct neighbors of a particular node.

3. Experimental section is plentiful. Unlike previous work that evaluates the robustness of graph neural networks, this paper measures the test results of the proposed model from a more fundamental perspective (e.g., using confidence to compare robustness). Table 1 also shows that the proposed GCPN achieves the best robustness against perturbations compared with GCN and RGCN.

Weakness:

1. The technical contributions in methods are heavily similar to prior works, making the novelty limited. An idea of predictive coding has been already presented in [1, 2]. Although using the energy-based model and the predictive coding to replace backpropagation is novel to graph learning tasks, the detailed techniques seem like simply adoption on message-passing networks from general machine learning. In addition, the technical description cannot provide new insights for graph neural networks.

[1] An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity, Neural computation.

[2] Learning on arbitrary graph topologies via predictive coding, arXiv.

2. In the experimental comparisons, there are insufficient baselines for comparison. Table 1 compares the proposed GCPN only with the GCN and RGCN. There are, however, numerous other defensive approaches to graph neural networks that may be equally or more effective than RGCN. Since readers may be curious about the efficacy of GCPN in the context of the entire domain, additional baselines need to be added.

3. The number of attack methods used for evaluation is insufficient. Specifically, in Table 4 Poisoning Attack and Section 4.3 Evasion Attack, the authors create adversarial graph samples using only Mettack and Nettack, respectively. Mettack and Nettack are not the only methods for attacking sample generation; many others may be equally or more effective. If the authors provide additional attack methods during evaluation, the defense results of GCPN will demonstrate its superiority more convincingly.

clarity,_quality,_novelty_and_reproducibility: The motivation for this study is clear, i.e., to use techniques (such as energy-based design and predictive coding) that have been demonstrated to be effective in enhancing the robustness of deep learning in other more general domains. Graph Predictive Coding is a straightforward method that is easy to follow. However, its novelty is limited because the proposed techniques have been developed in prior work that has not been mined for graph-specific insights.
summary_of_the_review: The paper is easy-to-follow and the methodology is plainly stated. The proposed techniques are an early attempt at graph tasks. To evaluate GPCN's robustness, the experiments are pretty comprehensive from various dimensions: poisoning and evasion, global and targeted, direct and indirect. However, because both the energy-based model and predictive coding that makeup GPCN are existing technologies, the originality, and novelty of the paper are limited. Overall, I tend to reject this submission.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a predictive coding approach to graph representation learning. The authors show that this approach yields performance that is often worse than GCN and other variants like GAT, but that robustness to structural attacks are better. 
strength_and_weaknesses: Overall, I have seen the use of predictive coding as an alternative to backpropagation since PC is more biologically plausible and since it’s easier to parallelize at scale.  The motivation for using PC here doesn’t really come through as they don’t seem to value the biological plausibility or provide benchmarking for efficiency in training. 

It does appear to be more robust than the chosen comparators (especially GCNs) on some of the tasks. However, the other formulations (robust GCN and graph attention network) seem to outperform it, which makes me wonder what the real benefit to this model is.

The authors state that GAT and RGCN, which consistently outperform the present method, do so because they are trained with batch normalization and dropout and both use attention mechanisms. Why not try those mechanisms with GPCN? Is there some reason those cannot be used, and if so, isn’t that a limitation of the present model?
clarity,_quality,_novelty_and_reproducibility: I think the use of predictive coding on graphs is novel, it has been tried more often with image processing using convolutional layers. The results overall seem reproducible. However, the attempt at predictive coding does not seem very successful here. 

The predictive coding model proposed here could use a schematic.

Table 4: Is GPCN-GCN supposed to be the present study’s architecture? Why are references to the model inconsistent?
What are the +/-
Argument that GPCN outperforms GCN across the board but the confidence intervals (or standard deviation, or whatever is being reported, which should be made clear) are overlapping in over 50% of comparisons.

Figure 3: what do the points represent? What do the different colors of the points represent? Also they are illegible  plotted over the boxplot as presented.

There are typos and awkward phrasings throughout. 

summary_of_the_review: This seems to be an interesting idea but not yet mature enough for publication given its performance on chosen tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper address the problem of GNNs being vulnerable to imperceptible adversarial attacks and also has some issues in generalizing out-of-distribution data. The proposed solution, graph predictive coding network (GPCN), uses a novel message-passing scheme developed based on the theory of predictive coding. The authors argue that their approach enhances the robustness of the learned representations. Their experiments show that the representations learned are more robust to adversarial attacks as well as deliver improved results over the GCN counterpart. 
strength_and_weaknesses: Pros:
1. This work attempts to expand Predictive coding techniques for multilayer graph neural networks. There has been limited success for PC graphs and this work identifies and extends the predictive coding approach. 
2. The choice of experiments and related methods are well chosen.

Comments:
1. Line 3 of Preliminaries section, should it be $X\in \mathbb{R}^{|V|\times d}$ ?
2. In eq.1, what is (t) ? Please define it properly.
3. “Here, the <aggregation> function is a weighted combination of neighbour characteristics with predetermined fixed weights, and aggregate function is a linear transformation.” You meant the <update> function?
4. In eq. 4, what is $x_{u, t}$ ? Shouldn’t it be $h_{u, t}$ ? It seems to me that the notation got mixed up with the referred PC graph paper. 
5. (page 3.) typo `PG graphs’? Should be PC graphs. 

[PC graph paper]:  “Learning on Arbitrary Graph Topologies via Predictive Coding”

There are lots of typos and errors in the write-up. I have pointed out a few of them. The paper is slightly difficult to follow. I think the contribution of this paper is very incremental in terms of architecture novelty. It is a direct extension of the PC graph paper. The results on adversarial attacks presented are however good and seems promising. I would request the authors to revisit their draft to polish their presentation as IMO the current draft is not ready for the conference. 
clarity,_quality,_novelty_and_reproducibility: <mentioned in comments above.>
summary_of_the_review: There are lots of typos and errors in the write-up. I have pointed out a few of them. The paper is slightly difficult to follow. I think the contribution of this paper is very incremental in terms of architecture novelty. The results on adversarial attacks presented are however good and seems promising. I would request the authors to revisit their draft to polish their presentation as IMO the current draft is not ready for the conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper applies predictive coding (PC)—an alternative to backpropagation—to graph neural networks, and attempts to show an improvement in robustness to adversarial attacks. The authors describe the predictive-coding scheme for graph convolutional networks (GCNs), and evaluate robustness against several forms of adversarial attacks compared to other GNNs.
strength_and_weaknesses: This paper applies PC onto GCNs. It is an interesting idea and is a natural next step (after previous work), and there are analyses which quantify robustness. The authors have done comparisons across several tasks and datasets.

Below are some weaknesses which I feel detract from the paper’s main points:
### There is little intuition or exploration on why PC might improve robustness
Although existing works on PC and on classifier energy are cited, it is not well explained why applying PC to models like GCNs might improve robustness against adversarial attacks. The authors offer some minor amount of justification based on the concept of “low-pass filtering” in the abstract, but this is not really expanded upon significantly. It is unclear why one might expect PC to be a reasonable technique for improving robustness (relative to backpropagation).
### The analyses on predictive performance and ECE/MCE are only against severely weakened GCNs
The analyses on predictive performance (Tables 1 and 2) and the reliability diagrams only compare GPCNs (GCNs trained with PC) against vanilla GCNs which are not trained with things like normalization. It is rather well-known that normalization is very important for GCNs, and so these comparisons are rather unfair—the GCNs are severely weakened and so these results are not particularly compelling. It would be more reasonable to compare with a standard GCN trained with dropout/normalization.
### Comparisons in robustness are essentially only against a single method, where other more recent methods exist
Although many methods for improving robustness in GNNs exist (the authors cite several in Section 5), the comparisons on robustness really only compare GPCNs with GCNs, RGCNs, or GATs, and not consistently. In these comparisons, RGCN is the only model I would expect would be a fair baseline for robustness. GCNs and GATs are less meant for robustness, although Table 4 shows that GATs oftentimes outperform GPCNs in terms of robustness anyways.
clarity,_quality,_novelty_and_reproducibility: The paper demonstrates limited technical novelty, as it largely reapplies existing work on predictive coding on graphs, onto GCNs. The authors do attempt to show that this may improve robustness against adversarial attacks.

The paper suffers from several issues in clarity and in writing. Below is an incomplete list:
- BP is never defined as “backpropagation”
- In Section 2, the update function is not defined
- In Eq. 4, $x$ should be $h$, and in Eq. 5, $\theta$ should be $w$
- “We follow the formulation of PG graphs” should be “PC”, I believe
- I think the citation next to “GCN” in Table 3 is a mistake
- The model architecture is not entirely clear from the supplement (more in-depth explanation would be nice), although the table of hyperparameters is much appreciated
- There is a broken figure link in Appendix D
- There are many grammatical errors in the writing, which should be fixed
summary_of_the_review: The paper offers an interesting application of predictive coding (PC) for GNNs, and attempts to show that GCNs trained with predictive coding rather than traditional backpropagation are more robust against adversarial attacks. Unfortunately, the comparisons in robustness are only done against a single method (i.e. RGCN), even though several other methods exist. Since the models trained with PC are not confidently outperforming existing methods in terms of predictive performance or robustness, this detracts a lot from the main point of the paper. If the paper were to be rewritten more as an exploration of PC on GCNs rather than a novel way to improve robustness, it would be more compelling as a piece of scientific work. Unfortunately, the paper also suffers from numerous issues in terms of clarity and writing, which make it difficult to support its passage.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: There was a clear consensus that the paper addresses an interesting problem setting. However, the reviewers also agreed on various negative points including incremental technical contribution, lack of presentation, and insufficient comparisons. The author feedback did not clarify these points sufficiently. The authors are encouraged to revise the paper.
justification_for_why_not_higher_score: As mentioned, there are several weaknesses that need to be addressed in a revised version.
Decision: Reject