Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces GRAND++, GRAph Neural Diffusion with a source term. As its predecessor GRAND, this method is focused on the development of a new continuous-depth GNN to tackle the over-smoothing and the bottleneck issues which are typical of deep GNNs.
The authors introduce a random-walk interpretation of GRAND, which shows that when the network is very deep it is still prone to over-smoothing. To address this problem they add a source term to the problem formulation, which is shown to provide better results in very deep models and in classification tasks which are characterised by a low amount of labeled data.
main_review: 


Pros:
- The problem is, in my opinion, both interesting and relevant for the research community
- The model seems to perform particularly well when there are very, very few labeled data points

Cons:
- A few parts are introduced without much context or references in the text, e.g. Laplace and Poisson learning are cited as related work but the connection to the present one is not clear enough IMO (except for the problem of inference inconsistency at low labeling rate)

- Formulation is not always clear, e.g.:
  - in Eq (3), $\tilde L$ is introduced as a shorthand for $(I-\delta_{t}L)$. Does this matrix have any particular interpretation? 
  - in page 4, between (6) and (7), $h$ is used both as an index for layers and as the max value (something like $\sum_{l=1..h} A^{l}(X)$ or even just using $l$ would make it more readable)
  - the random walk definition in (8) did not convince me (perhaps because it was introduced with no further explanation). How is the step size $\delta_t$ defined? To me it should not be possible to use it as in $1 - \delta_{t}$ to define a probability value, as there was no definition of $T$ thus it can take any value greater than zero. 
  - in Eq (10.5), $\pi_{j}=: \tilde{x}$ shouldn't this be $\bar{x}$ instead?

- experimental results are, in my opinion, not convincing:
  - related to the claim of performing better with fewer labeled data, while there are some datasets in which the method actually does well there also many cases (CoauthorCS, most of Computer, and Photo) in which it is worse or comparable to simpler approaches (learning the reasons why this happens, though, would be interesting per se)
  - still related to numbers, the noise of low-labels dataset is such (and the variance is so high) that even results out of 2000 experiments are not statistically very significant: some of them (e.g. CORA-20, PubMed-2, Photo-20) have a p-value > 0.1.
  - it is not clear why depth should be such a useful treat when both GRAND and GRAND++ have deteriorating performances when it grows (e.g. in Figure 2, 1-3 from left). In these cases GRAND seems to perform better or comparably most of the times (with the advantage of being simpler). Plots with small amount of labeled samples, instead, look more convincing in my opinion (i.e. they clearly show a use case in which GRAND++ would be preferable)

summary_of_the_review: I think the paper tackles an interesting problem and provides evidence that, in some extreme cases such as incredibly low amount of labels per class, it can outperform other approaches.

However, of its two main claims (avoiding over-smoothing and working well with few labels), I think the former is a bit harder to defend (GRAND++ is always better than GRAND when depth is very large but not always otherwise, in addition it is does not always exhibit the same behavior and it not clear why that happens). For the low-label case, I think there are more chances to convince about the improvement brought by the model but I think it would be important to better understand the reasons for it. Intuitively, a strong bias is introduced by the insertion of the source term that definitely helps by providing useful signal for the classification. This should be, in my opinion, discussed more in detail and verified with more examples.

For the above reasons, I consider the paper still not mature enough for publication but I think a more in-depth analysis could bring valuable results.

----

The authors' replies addressed my concerns and after reading their answers to other reviewers and the updates to their manuscript I decided to increase my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies neural-based diffusion models for graph data. It considers deep learning on graphs as a continuous diffusion process and treats GNNs as discretizations of an underlying PDE. The authors build upon an existing work (GRAND) by adding a source term to the objective function. The source term is designed to alleviate the over-smoothing problem of GNN, a well-known phenomenon that deeper GNN learns similar representation for each node. They theoretically show the effectiveness of the source term in mitigation over-smoothing problem. Furthermore, they empirically assess their method's effectiveness in learning with limited labeled data and using deep architectures on several benchmark datasets.
main_review: Pros:

The authors provide comprehensive background knowledge for the different parts of their algorithem.

They theoretically show that the source term helps alleviate the over-smoothing problem.

The experiment regarding limited label data is interesting.

Cons and concerns:

One of my main concerns is the empirical results reported in the paper. Specifically, I would be grateful if the authors could elaborate on why there is a mismatch between the results reported for the "GRAND" method (when for each label, 20 samples were used) in the original paper and the numbers reported in the current submission. For instance, on CORA dataset, the GRAND paper reported 83.6 ± 1.0 while the submission reported GRAND test accuracy as 82.86 ± 2.39. This is important because the test accuracy of GRAN++ is 82.95 ± 1.37.




To actually show that GRAND++ is empirically better than other methods, the authors need to do a t-test. That is because std. of the reported test accuracy is too high (16 percent for CORA) when the number of observed samples for each label is less than 20. Note, this is different from having a small improvement; the t-test has to be done to show whether there is actually an improvement or not.



Lack of experiments on datasets that needs larger depth NN. Authors motivate their method by mentioning that they avoid over-smoothing; however, there must be an empirical need to develop such methods.  I suggest authors to experiment on a dataset that needs large depth to show the importance of their contribution.



summary_of_the_review: Please look at the comments provided above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a modification of the continuous-time graph neural network architecture GRAND where a source term is added to the differential equation in order to (a) avoid convergence of the dynamics towards constant vectors and thus mitigate the over-smoothing effect which affects the long time behaviour of the original dynamical system and graph neural networks with many layers in general and (b) increase classification performance under limited number of training points. 
main_review: The paper addresses an important issue which affects neural ODEs and deep NN on graphs. In particular, concerning the GRAND differential equation, I think that adding a source term is an important idea essential to obtain a non-trivial long term behaviour of the dynamical system. This is clear when looking at linear dynamics (as done in the paper) as it is well known that the system converges to the dominant eigenmodes in that case. In fact, the addition of a source term is typical of most diffusion models on graphs, including standard Label Spreading/Label Propagation and their variants. 

Weaknesses:
The paper is sometimes poorly written with several typos affecting the mathematics. I list below some of the major points that require improvement to my opinion:  
- In the notation, the Hadamard product should be explicitly defined (or maybe just say that this is entrywise multiplication)
- Eq (1) uses notations that are not mentioned in the main paper such as $\nabla$ or $\mathrm{div}$ operators (they are in the supp material). I believe it would be very helpful to recall what this operators are in the main text. 
- Right after eq (1): you consider the simplest case where $G$ is only dependent on the initial $X=X(0)$. However, this is not clear. This is a particular choice of $G$ that does not change in time, it is not the only one. Also, your choice seems to have no dependence on $X$ but rather on $W$ (the graph structure). So this seems to be the case of a $G$ that does not depend on $t$ NOR on $X$. In fact, soon afterwards at the beginning of $\S 3$ you consider another case where there is no dependence on $t$ but there is dependence on $X$ via the attention matrix $A(X)$. Also, what do you mean by "so we pick $G_{ij} = 1/d_i$? When do you pick this? Overall, I find this paragraph very confusing 
- Why does the step size in Euler depend on $t$? It seems to me this is just a constant in your subsequent derivations
- Before eq (6) you quote eq (2) while I think you should refer to eq (1)
- You define GRAND-l after eq (7) but this definition is not clear to me. Right before you say that GRAND is based on the solution of (5) via ODE solvers. But then, when you define GRAND-l you refer to the attention matrix $A(X)$, which may not appear in (5). As you will use GRAND-l often in the rest of the paper, I believe this deserves a more clear definition here
- In the definition of the random walk (8), is $x^{(i)}=x^{(i)}(0)$? Also, this is quite unusual definition: typically the state space of a random walk on a graph are the nodes not on the their feature embedding. What is the advantage of defining the RW on the features?
- Again, the notation in Proposition 3 is not clear: what is $x^{(j)}$? 
- Equation (1): what does "the source at feature vector of node $j$" mean?
- Starting from section 4 you move from $X(t)$ to $Z(t)$ in a way that is to my opinion inconsistent and not clear.
- In the centred equation after (10) there are several typos: I think $x$ should be $z$ or maybe you should use $x$ in (10) rather than $z$? $\tilde x$ was $\bar x$ in prop 3; I think your approximate dynamics should be $-x^{(i)}(t) + A\bar x$ rather than $-x^{(i)}(t) + \tilde x$, or could you clarify why this is not the case?
- Right after that equation you state: "for $i\in I$ we choose $c$ so that $z'(t)\approx 0$. This is not clear to me. What is $c$? What does $z'(t)\approx 0$ mean?
- The statement of Proposition 5 is quite sloppy. First, it is not clear what $z(k\delta_t)$ is? Is this the one from (14)? Second, mathematically, "it does not converge to a constant vector" makes not much sense? For example, if constant means that the limit does not depend on $t_0$, then this is probably not true. If it means that it has all constant entries, than still this might no be true as it would depend on the initial feature space. A much more precise statement to my opinion would say what the sequence converges to, rather than what it does not. 
- Concerning this convergence questions, overall I do not really understand why you are considering the convergence of the sequence obtained via Euler integration, rather than the real dynamical system. Since you are anyway considering the simplest case of a linear dynamical system, why don't you consider the exact solution and study its behaviour? The fact that Euler converges to something might not mean that any numerical integrator would
- As stated above, in the experiments you mention GRAND-l and GRAND++-l but it is not clear to me what exactly are those


summary_of_the_review: I like very much the idea of modifying the Laplacian diffusion equation div-grad used in GRAND adding a source term. The theoretical analysis proposed in the paper deals mostly with the simplest case of a linear diffusion operator, which is certainly useful to gain intuition. However, no attempt to transfer the results to more general nonlinear settings are discussed. Moreover, the mathematics contains many typos and a number of statements that are vague or not clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a graph deep learning method called GRAND++. This framework based on GRAND and can work with a limited number of labeled nodes. Experiments are also conducted to demonstrate the effectiveness of the method.
main_review: Strengths: The problem faced by the paper is interesting and timely and the proposed approach seems reasonable. The article is well written, the method is clearly described, and the overall quality is good. The authors also provide the source code to facilitate experimental replication.

Weakness: 
1. This paper is more like an external version of GRAND. The author spends a lot of time on background knowledge and GRAND's work, which may undermine the contribution of this paper.
2. The assumption of Proposition 2 and 3 seems a little bit strong. We cannot guarantee how the graph data will look like, nor can we guarantee that the graph is connected. 
3. I have some doubts about the 5.2 experiment, why use a setting where each class has only one label. In the above experiment, the authors used 20 labels per class, while in this experiment just 1 label was used. I would like to have some intermediate stage of experiments, like 5/10 labels per class.

summary_of_the_review: The overall quality of the paper is good, with a clear narrative. The problems addressed are also much needed. The methodology is straightforward and clear, and there is also theoretical guarantee. So I recommend it for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel graph diffusion model with a source term in order to solve the over-smoothing issue of GRAND model and GNNs with diffusive property when the depth of the GNNs is large. This design also leads to a class of more accurate GNN based classification models when the labeling rate in the graph is low. 
main_review: Strengths:
1) Overall, I feel this is a well-organized paper and I enjoy reading it. This paper describes clearly the motivations of their model design. It also provides complete theoretical results and examples to explain the claimed issue (over-smoothing) and how the proposed method could solve it. 
2) The proposed model is simple and neat. By simply adding the constant source term to the original diffusion equation and adjusting the initial conditions, this model is able to prevent the over-smoothing issue of diffusive GNN models. Moreover, this model could reuse the design and implementations of its source-term free version (GRAND). 
3) The ablation study plots suggest that this model is able to effectively alleviate the important problem of over-smoothing issue of GNN models and benefits from adding more depth to the network. 

Weakness:
1) The accuracy gains are not stable. The proposed method has worse classification accuracy than GNN and GCN significantly in the CoauthorCS dataset. The causes remain unexplained. Also, on CORA and CiteSeer the added source-term are worse than the baseline GRAND model. 
2) This paper uses the classification with low-labeling rate as the prediction task to validate the effectiveness of preventing over-smoothing. However, over-smoothing could happen even without low-labelling rate. It is not clear whether the proposed method could practically solves the over-smoothing problems. 
summary_of_the_review: In general, this is a solid paper which proposes a novel idea to solve the over-smoothing problem the GNN models using a special form of graph diffusion equation. Solid theoretical and experimental results are provided in the paper.
review_title: null

Meta-review: The paper presents a continuous framework for GNNs based on neural diffusion PDE and is an evolution of a previous method (GRAND). The main novelty appears to be the additional source term, which the author show to be beneficial in reducing the oversmoothing effect typical in deep GNNs. While novelty is somewhat limited, the paper provided detailed theoretical and experimental assessment of the idea. Overall, the reviewers liked the approach and expressed some questions/concerns that were satisfactorily addressed in the rebuttal. We recommend acceptance.
Decision: Accept (Poster)