Dissecting the Diffusion Process in Linear Graph Convolutional Networks

Abstract: Graph Convolutional Networks (GCNs) have attracted more and more attentions in recent years. A typical GCN layer consists of a linear feature propagation step and a nonlinear transformation step. Recent works show that a linear GCN can achieve comparable performance to the original non-linear GCN while being much more computationally efficient. In this paper, we dissect the feature propagation steps of linear GCNs from a perspective of continuous graph diffusion, and analyze why linear GCNs fail to benefit from more propagation steps. Following that, we propose Decoupled Graph Convolution (DGC) that decouples the terminal time and the feature propagation steps, making it more flexible and capable of exploiting a very large number of feature propagation steps. Experiments demonstrate that our proposed DGC improves linear GCNs by a large margin and makes them competitive with many modern variants of non-linear GCNs.


Reviews and author responses:

 Thanks for appreciating our explanations. We will add the discussions in the revision as you suggested. Also, thanks for your interesting paper list and we will have a look.

Along the line of linear GCNs, this paper dissects the feature propagation steps via continuous graph diffusion. With theoretical analysis on numerical methods to approximate Graph Heat Equation, the paper proposes a linear GCN, named as DGC, that decouples terminal time $T$ and propagation steps $K$. As shown in real-world experiments, DGC enjoys competitive results compared with linear and non-linear GNNs.  Linear GCN is a rising topic in the domain of Graph Neural Networks, since it enjoys much faster speed during training and scalability on large graphs. This paper decouples terminal time $T$ and propagation steps $K$ to design a Linear GCN with optimal tradeoff between oversmoothing and undersmoothing, while most typical models do not pay attention to such decoupling. I like the setting of Theorem 3 as a motivation to find the optimal terminal time $T^*$. However, there are some concerns to address.

Concerns:

1. mistakes in the bound in Theorem 3: to derive the last inequality in line 46 of appendix, (1) a coefficient of 2 is missing as $||a+b||^2\le 2||a||^2+2||b||^2$, and (2) $M$ is a bound for the norm of $X_c$ but the paper is seemingly using $M$ as a bound for norm of $X_0$ where $X_0=e^{T^*L}X_c$.
2. possible experiments to extend: (1) there is an earlier related work [1] that concatnates different orders of $A^kX,k=1,\dots$. It would be interesting to compare SIGN with DGC in a fair style, such as with the same terminal time $T$. Note that data split for Reddit dataset in [1] looks different from DGC. (2) it would be more promising to try DGC on OGB node classification benchmark.
3. For computation time in Section 5.4, it is necessary to explicitly report the pre-processing time of DGC. I suppose it uses approximately $50\times$ pre-processing time for DGC(K=100) compared with SGC(K=2), which makes the total training time $3\times$ considering about 5% of training time of SGC is for pre-processing. Then, as I mentioned in Concern 2, if SIGN [1] works with similar performance with DGC, since SIGN is with a closer pre-processing time to SGC, it might raise questions on design details of DGC.
4. Theorem 1 is not too novel compared with results (around Proposition 1) in [2] that GCN asymptotically converges to non-informative equilibrium.
5. although the paper mentions a variant DGC-RK based on 4-order Runge-Kutta method, its performance is not shown. It would be interesting to conduct ablation study with it to explore whether it is necessary to utilize higher-order finite difference method and strengthen the argument of Theorem 2 on numerical errors.

Typos:

1. in lines 30-31 of appendix, both $n$ should be $k$.
2. in line 46 of appendix, the left square bracket is not at the appropriate position in the first inequality.
3. re-statement of Theorem 3 in appendix is not consistent with that in the main text.

[1] Sign: Scalable inception graph neural networks. F Frasca, E Rossi, D Eynard, B Chamberlain, M Bronstein, F Monti.

[2] Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. K Oono, T Suzuki.

Therefore, I suggest it is below the acceptance threshold for novelty compared with [1,2]. If my concerns about SIGN are addressed, I am willing to raise my score.

======================== post rebuttal =======================

I would like to increase my score from 5 to 6 based on authors' responses. No.

 Thanks for your reply. For now my main concerns are properly addressed, so I would like to increase my score by 1 to 6 (above threshold). Please add related discussions to the revision. 

Besides, from the community, there is a recent list of efficient and scalable gnns: https://github.com/chaitjo/awesome-efficient-gnn, which might share research interests with this paper.

 
Thanks for your detailed feedback. We address your concerns as follows.

---
**Q1.** Why there is a large gap of preprocessing time between SIGN and DGC with $K=r=100$? I thought they should be very close, due to the same power of normalized adjacency matrices.

**A1.** We believe the difference in preprocessing time between SIGN and DGC is mainly due to the time for expanding the working GPU memory as $K$ (or $r$) increases to 100.

**$O(r)$ v.s. $O(1)$ memory complexity.** As discussed above, SIGN is a multi-scale method, and **it stores every intermediate scale of $A$**, i.e., $[A^1,A^2,\dots,A^r]$ in the preprocessing stage with a memory complexity $O(r)$. On the contrary, DGC is a single-scale method and only needs to store **one single propagation matrix** $S^{(K)}$, so its memory complexity is $O(1)$. 

| $K$ / $r$ | 2 | 10 | 20 | 50 | 100 |
|---|  ---| ---| ---| ---| ---|
| DGC | 3.8 ms | 17.5 ms | 34.9 ms | 86.5 ms | 169.2 ms |
| SIGN | 5.9 ms | 20.5 ms | 51.5 ms | 162.3 ms | 2.4 s|

In modern deep learning frameworks (PyTorch in our case), it takes time to keep **expanding the working GPU memory due to the copy operation**. As a result, when $r$ or $K$ is very large (like 100), SIGN can be much more memory-intensive than SGC (100x). This results in a large difference in the preprocessing time between SIGN and DGC, as shown above. There might be some tricky ways to optimize the pipeline, but here we stick to the vanilla (also official) implementation for a fair comparison.


---
**Q2.** I would like to know why the last layer is linear instead of nonlinear, as [1] differs from SGC with a final MLP. A discussion or experiments, especially on Reddit and OGB-arxiv, will make this work more complete.

**A2.** A linear GCN like SGC is composed of two parts, **a linear diffusion process**, and **a linear classification head**. In this paper, our analysis focuses on dissecting the linear diffusion process from a continuous perspective. For a fair comparison with SGC, we leave the linear classification head unchanged, and it is not the main focus of this work. 

For completeness, we also try MLP classification heads after DGC's linear diffusion following your suggestions. The results on a range of benchmark datasets are listed as follows.

| Model | Cora | CiteSeer | Pubmed | Reddit | OGB-arxiv |
| --- | --- | --- | --- | --- | --- |
| DGC-linear | 83.5 | 74.5 | 80.2 | 95.8 | 70.1 |
| DGC-MLP | **85.2** | **74.9** | **80.8** | **95.8** | **71.6** |

We can see the applying an MLP head can bring a significant improvement on most datasets. The improvement is more clear on medium-sized graph data, like Cora, CiteSeer, and Pubmed. Remarkably, it achieves SOTA results on these datasets and surpasses all nonlinear GCNs listed in Table 2. On OGB-arxiv, it also improves 1.5 points over DGC-linear and achieves comparable performance to the multi-scale model SIGN (71.9) while being more parameter-efficient. 


| Model | Cora | CiteSeer | Pubmed | Reddit | OGB-arxiv |
| --- | --- | --- | --- | --- | --- |
| SGC | 81.0 | 71.9 | 78.9 | 94.9 | 68.9 | 
| gfNN [1] (SGC-MLP) | 82.6 | 72.4 | 79.9 | 95.4 | 70.8 |
| DGC-MLP | **85.2** | **74.9** | **80.8** | **95.8** | **71.6** |


Besides, we also compare DGC-MLP with gfNN [1] as you suggested. gfNN is a very interesting work with fruitful observations. Nevertheless, we can see that gfNN achieves limited performance by only replacing SGC's linear head with an MLP. In comparison, combining our DGC with an MLP head yields much better results, which indicates that a properly designed diffusion process (or low-pass filter) is also important for cultivating the expressiveness of the classification model. 

We will add this discussion in the revision.

[1] Revisiting Graph Neural Networks: All We Have is Low-Pass Filters. Hoang NT and Takanori Maehara.


---
**Q3.** To stress the advantage of DGC, my intuition is to 1) run a vanilla GCN on OGB-arxiv (seemingly there is a GCN result on the leaderboard), and to 2) change the discrete process of feature propagation in each GCN layer to that in DGC. (This suggestion might be inconsistent with authors' motivation.)

**A3.** As you have noticed, we are afraid this part is beyond the scope of this paper. As mentioned above, the main focus of this paper is to **dissect the linear diffusion process** of linear GCNs, and this topic worths attention because linear diffusion is preferable in many ways, such as computation efficiency (by preprocessing) and memory efficiency (as a single-scale method). Accordingly, we have shown that a properly designed linear diffusion can yield significant improvement over SGC on a wide range of benchmark datasets. As shown above, we further demonstrate that our DGC diffusion can achieve SOTA results with an MLP head. In this way, we believe that we have investigated the linear diffusion process thoroughly and fulfilled our plans. 

As for your suggestions on applying DGC to the **nonlinear diffusion process** of nonlinear GCNs where the diffusion is applied after nonlinear transformations, it might be **inconsistent** with our analysis for linear diffusion due to the intermediate nonlinearities. Nevertheless, we believe that it is an interesting direction and we will explore it in the future.

---
Thanks again for your detailed review. Please let us know if you have further concerns.

 Thanks for your reply, I have some questions as below:

1. Could you please comment on why there is a large gap of preprocessing time between SIGN and DGC with $K=r=100$? I thought they should be very close, due to the same power of normalized adjacency matrices.

2. For comparison with SIGN, authors classifies SIGN as nonlinear. But in the authors' reponse to Reviewer 1oqx, a nonlinear GNN suffers memory inefficiency, which might include SIGN? On the other hand, while this paper proposes an interesting definition of feature propagation, I would like to know why the last layer is linear instead of nonlinear, as [1] differs from SGC with a final MLP. A discussion or experiments, especially on Reddit and OGB-arxiv, will make this work more complete. The key difference between DGC and classical GNNs is DGC can pre-compute features augmented by graph structure. I suppose linear vs nonlinear can be substituted with more precise terms.

3. It would be much better if authors can finish the experiments on OGB-arxiv. The current result will decrease this work's impact. Cora, citeseer and pubmed are too easy to some extent. Additionally, to stress the advantage of DGC, my intuition is to 1) run a vanilla GCN on OGB-arxiv (seemingly there is a GCN result on the leaderboard), and to 2) change the discrete process of feature propagation in each GCN layer to that in DGC. (This suggestion might be inconsistent with authors' motivation.)

[1] Revisiting Graph Neural Networks: All We Have is Low-Pass Filters. Hoang NT and Takanori Maehara.

 
Thanks sincerely for your detailed and insightful review. We address your main concerns as follows.

---
**Q1.** Mistakes in Theorem 3: 1)  a coefficient of 2 is missing; 2) $M$ is a bound for the norm of $X_c$ but the paper is seemingly using $M$ as a bound for norm of $X_0$ where $X_0=e^{T^*L}X_c$.

**A1.** 1) Thanks for pointing it out. We forgot the coefficient $2$ here, and will fix it in the revision. 2) There are some misunderstandings here. $X_c$ denotes the ground-truth features in Eq. 19 (Appendix B.3). In Eq. 20, the initial state $\tilde X_0$ is the ground-truth $X_c$, so $\tilde X_0=X_c$. Thus, our proof is correct. Nevertheless, there are some typos. In Eq. 22, $\hat X$ should be $X$; in the 4th line of Eq. 24, $X_0$ should be $\tilde X_0$. We will fix them in the revision.

---
**Q2.** Comparison with SIGN in a fair style. 

**A2.** SIGN is indeed an interesting related work that bears many similarities to our DGC. The SIGN model is 
$$Y=\xi(Z\Omega),\quad  Z=\sigma([X\Theta_0,A_1X\Theta_1,\dots,A_rX\Theta_r]),$$
where $\sigma,\xi$ are nonlinearities, $A_k=A^k$ is the $k$-hop propagation matrix, and $\Theta_k,\Omega$ are weight matrices. Our DGC-Euler model takes the form 
$$
 Y=\xi(X^{(K)}\Omega),\quad X^{(k)}=(1-T/K)X^{(k-1)}+(T/K)\cdot AX^{(k-1)},\quad k=2,\dots,K.
$$

a) Discussions

The two models 1) both apply all feature propagation before the classification model and so that can 2) both pre-process the propagation matrix and save it for later training.  Nevertheless, there are several critical differences between SIGN and DGC:
- **Linear v.s. Nonlinear.** DGC is a linear model, while SIGN is nonlinear.
- **Multi-scale SGC (SIGN) v.s. single-scale continuous diffusion (DGC).** SIGN is a multi-scale method that extracts every possible scale ($A^rX, r=0,1,\dots$) for feature propagation. Thus, SIGN resembles a multi-scale SGC, but still inherits some of the limitations of SGC, e.g. a fixed step size $\Delta t=1$. On the contrary, the goal of DGC is to find the optimal tradeoff between under-smoothing and over-smoothing with a flexible choice of $T$ (real-numbered) and fine-grained integration ($K$), so it only uses a single-scale propagation kernel (Eq. 12). Thus, we believe the two works explore two **orthogonal directions** to improve SGC.
- **Model size.** As a result, the model size of  (linear) SIGN is proportional to the number of scales $r$, while the model size of DGC is independent of $T$ and $K$.

Overall, we can see that the two are closely related, and we will add this discussion in the revision. 

b) Empirical Comparison

Besides, we also conduct additional experiments comparing the two methods with the same terminal time $T$ as you suggested. 


**Setup.** We conduct the experiments on the Cora dataset (semi-supervised). We re-produce SIGN  as it has not reported results on these datasets. For comparison, we follow the same protocol of SGC, using the same optimizer, learning rate, training epochs; and (automatically) tune the weight decay and propagation steps ($K$ or $r$) at each terminal time $T$. 


| T |  1 |  2 |  3 |  4 |  5 | 5.5 |  6 |
|---| ---| ---| ---| ---| ---| ---| ---|
| SGC | 72.4 | 80.5 | 79.2 | 81.0 | 78.8 | N/A | 80.5 |
| SIGN | 72.4 | 77.3 | 78.9 | 80.6 | 81.3 | N/A | **81.7** |
| DGC | **78.0** | **81.9** | **82.5** | **83.0** | **83.1** | **83.5** | 81.5 |


These findings suggest that 
- DGC still outperforms SIGN for $T\leq 5$, while being slightly worse at $T=6$ due to over-smoothing;
- DGC can flexibly choose a **real-numbered** terminal time, e.g., $T=5.5$, to find the best tradeoff between under-smoothing and over-smoothing (83.5 acc), while the terminal time of SIGN and SGC has to be an **integer**;
- Single-stage methods (SGC & DGC) have bigger advantages at earlier time, while SIGN can surpass SGC at later stages by aggregating multi-scale information.

The empirical results show that although useful, multi-scale techniques cannot fully solve the limitations of SGC, while DGC can overcome these limitations by decoupling $T$ and $K$.

c) Computation Time

As you suggested, we report the explicit pre-processing time, training time, and total time as follows. We also include SIGN with a different number of scales ($r$) as a baseline. The experiments are conducted on the same platform with an NVIDIA GeForce RTX 3090 GPU.

We note that the comparison of DGC (K=100) v.s. SGC (K=2) is merely designed to show how the extra propagation steps do not contribute much to the total time. Remarkably, it does not mean that DGC takes 100 steps at all settings. Sometimes, a few steps (K<10) are enough to attain the best performance.

|Method | Preprocessing Time | Training Time | Total Time|
|------ |---------|---------| -------- |
|SGC / DGC (K=2) | 3.8 ms | 61.5 ms | 65.3 ms|
|SIGN (r=2) | 5.9 ms | 78.7 ms | 84.6 ms |
|DGC (K=100) | 169.2 ms | 55.8 ms | 225.0 ms|
|SIGN (r=100) | 2.4 s | 106.9 ms | 2.6 s |
|GCN | 0 | 17.0 s | 17.0 s|

From the table above, we have the following findings:
- Both SIGN and DGC can be much faster than GCN by pre-processing the propagation kernels.
- DGC is still faster than SIGN with the same propagation steps by being a single-scale method.

We will make this discussion more clear following your advice.

---
**Q3.** Try DGC on the OGB node classification benchmark.

**A3.** We conduct experiments on the OGB benchmark following your suggestions. Due to the limit of time, we only try on the OGB-arxiv dataset and have not exhaustively tuned the hyperparameters. Here are our preliminary results compared to SGC:

| Method | Acc (%) |
| ----| ---|
| SGC | 68.9 | 
| DGC | 70.1|

We can see that DGC still outperforms SGC by a large margin, demonstrating the effectiveness of decoupling $T$ and $K$. We also notice there is still some gap between linear and nonlinear models on OGB-arxiv, possibly due to the large scale of the dataset. Nevertheless, linear models still have their advantages by being much more computationally efficient. 

---
**Q4.** Theorem 1 is not too novel compared with results (around Proposition 1) in [1] that GCN asymptotically converges to non-informative equilibrium.

**A4.** Indeed, the two are similar in spirit and both describe the exponentially vanishing behavior of the eigenvalues. Similar results also appear earlier, such as in one of our references, Li et al. [2]. The main technical difference to [1,2] is that we take a continuous perspective and directly analyze the behavior of the underlying diffusion.

Besides, the main reason why we put it here is that it serves as one of the three limitations of SGC. Specifically, we use this asymptotic property to show how SGC over-smooths (Line 123) and how DGC will not over-smooth with a fixed terminal time (Line 180).

We will add more discussions on its relationship with previous work in the revision.

[1] Oono K, Suzuki T. Graph neural networks exponentially lose expressive power for node classification. In ICLR. 2020.

[2] Li Q, Han Z, Wu XM. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI. 2018.


---
**Q5.** Show the performance of the 4th-order Runge-Kutta (RK) method.


**A5.** We have compared the two numerical schemes, Euler and RK, in Figure 3 (middle) and Lines 304-307. We find that RK performs better than Euler when the propagation steps $K$ are relatively small. This is because RK has smaller numerical errors, as analyzed in Theorem 2 & 3. Nevertheless, the advantage disappears with a sufficiently large $K$. As a result, we adopt the Euler scheme for its simplicity and efficiency.


---
Thank you again for reviewing our paper and hope you find our explanations helpful. We are willing to take your further questions before the discussion session ends.

 
Thanks sincerely for your positive feedback and for appreciating the novelty and solidness of our work. We address your concerns as follows.

---
**Q1.**  Why the training time DGC is only 3 times slower than SGC in Figure 2? Is it because SGC converges slower than DGC?

**A1.** We have discussed this phenomenon in Section 5.4 (Lines 295-296). Here we additionally list the explicit time consumption of each part.


|Method | Preprocessing Time | Training Time | Total Time|
|---- |---------|---------| -------- |
|SGC (K=2) | 3.8 ms | 61.5 ms | 65.3 ms|
|DGC (K=2) | 3.8 ms | 61.5 ms | 65.3 ms|
|DGC (K=100) | 169.2 ms | 55.8 ms | 225.0 ms|
|GCN | 0 | 17.0 s | 17.0 s|

We can see that in SGC, the preprocessing time is very small compared to the training time  (<5% in the total time). As SGC and DGC have similar training time, using $K=100$ propagation steps in DGC is only 3 times slower than SGC in total. 

---
**Q2.** When linear models like DGC should be used in real applications, and when non-linear models should be used?

**A2.** Our paper shows that a properly designed linear GCN can achieve comparable performance to nonlinear models, while being
- much more **memory-efficient**  with much fewer parameters; 
- much more **computationally efficient** in training and inference as linear models.

Therefore, for application scenarios that care about the above properties, e.g., online advertisement, terminal device computing, motor control, linear GCNs could be a better option than nonlinear GCNs.

Instead, if there are enough response time and computation resources, nonlinear GCNs could be tuned to yield a little better results, while linear GCNs could still have advantages on relatively small and noisy datasets.

We will add this discussion in the revision.

---
**Q3.** Why DGC is more robust to feature noise?

**A3.** As discussed in Theorem 3, the diffusion process in DGC can be seen as a denoising procedure applied before the classification. From Figure 1, we can see that our DGC diffusion indeed effectively denoises the node features and make them more distinguishable. Consequently, node features of different classes have larger margins after diffusion, thus they could be more resistant to feature noise.

Therefore, DGC's robustness to feature noise can also be contributed to the optimal tradeoff between over-smoothing and under-smoothing with a flexible choice of $T$ and $K$. On the contrary, SGC is not as good as DGC because it cannot find such a sweet spot accurately (analyzed in Section 3.3). 

---
Thank you again for reviewing our paper. Hope you find your explanations helpful!

 
Thanks sincerely for your encouraging feedback and highlighting the potential impact of our work. We address your concerns as follows.

---
**Q1.** Readability should be improved. 

**A1.** Thanks for your suggestions. We will add a preliminary section to introduce the background on GCN and the feature propagation process. And we will also add captions to Figure 1 about the meaning of $T$ and $K$ to make it more clear.

---
**Q2.** Some subfigures in Figure 1 are less informative (e.g., second line, the second one (left to right)). 

**A2.** As we have discussed in Section 4.2 (Lines 190-192), this figure delivers an important message here, that is, a too large step size (because $T=5.3, K=1$, we have $\Delta t=5.3$) might lead to **feature collapse**. As we analyzed in Section 3, SGC is also a finite difference with a large step size ($\Delta t=1$). According to Theorem 2 & 3, this will lead to a **large numerical error** and affects the final performance. Therefore, Figure 1 (2nd row, 2nd col) indeed reveals one of the limitations of SGC and highlights the necessity of more propagation steps. 

We will elaborate on the figures better in the revision.

---
**Q3.** In some datasets, some non-linear GCNs still outperform linear ones. What properties contained in these datasets make non-linear ones better? And vice versa.

**A3.** This is an interesting question. Here we report some preliminary insights from our side. 

We notice that some datasets, like Cora and Reddit, tend to benefit more from deep layers, while some datasets, like Citeseer and Pubmed, are more favorable of linear GCNs. 

One noticeable difference between the two kinds of datasets is that the former tends to have higher accuracies, while the latter has lower accuracies. For example, in the semi-supervised setting, the SOTA results of Cora are around 84-85, and 72-74 for Citeseer, 79-80 for Pubmed, 95-96 for Reddit. Thus we can say that either the labels of Citeseer and Pubmed are relatively noisy, or these datasets are harder to learn. For such datasets, nonlinear GCNs tend to overfit the training set, while a properly designed linear GCN (i.e., DGC) can yield better results with a strong regularization effect due to a low model capacity.

In this work, one of our key messages is that linear GCNs can achieve comparable results to SOTA nonlinear GCNs across a range of benchmark datasets, and the nonlinearity can only bring a slight advantage, if any. Therefore, we highlight the necessity of re-evaluating the real expressiveness of these nonlinear models. 

---
Thank you again for reviewing our paper and please let us know if you have additional concerns.

 
Thank you sincerely for appreciating the novelty, simplicity, and solidness of our work. We will address your concerns as follows.

---
**Q1.** As most of the current models are non-linear, why do we need to study linear GCN? 

**A1.** Linear GCN is a rising topic in the GNN literature. In this paper, we show that a properly designed linear GCN (DGC) can be on-par with state-of-the-art non-linear GCNs while possessing many desirable properties. We have mentioned some points in the Introduction (Lines 20-21), and here we give a more detailed analysis.
- **Memory-Efficient**. As a linear model, it requires much fewer parameters than non-linear GCNs, which makes it very memory efficient. Thus, linear GCNs could have big advantages for both 1) large scale applications, such as web service, recommendation; 2) deploying on terminals, like mobile phones. 
- **Computation-efficient (both training and inference).** As a linear model, its training and inference are both much faster than non-linear models. As shown in Figure 2(right), our models can be more than 100x faster than non-linear counterparts, making it more preferable for computation-demanding applications.

In conclusion, with comparable performance, linear GCNs can be more preferable than nonlinear ones and thus worth more attention. 

We will add this motivation in the revision.

---
**Q2.** The difference between with self-loop and without self-loop?

**A2.** In Section 4.1 (Lines 166-175), we have discussed the relationship between DGC propagation (Eq. 12) and the augmented self-loop in GCN. 

We show that DGC is a natural way to incorporate self-loop into the propagation, thus we might not need to directly add self-loop in the adjacency matrix $S$. Motivated by this, we study the difference between the two Laplacians:
- $L_{\rm sym}=I-S_{\rm sym}$, where the adjacency matrix $S_{\rm sym}$ has no self-loop; and 
- $L=I-S$, where adjacency matrix $S=S_{\rm sym} + I$ adds self-loop as in GCN.

Comparing the two Laplacians, $L$ has a smaller spectral range, thus, according to Theorem 2, it will have a faster convergence rate. Figure 3 (left) empirically verifies this point, and it also shows that DGC propagation with and without self-loop can **both** converge to the optimum with increasing propagation steps $K$.

---
**Q3.** The authors used the step size to be $T/K$. Why exactly $T/K$? Can we use larger or smaller than $T/K$?

**A3.** As shown in Eq. 11, DGC is a general linear GCN framework utilizing continuous diffusion, and it can work with different ODE solvers. Given the terminal time $T$ and propagation number $K$, the forward Euler scheme will take a constant step size $T/K$. Besides, as you mention, we can also adopt those adaptive ODE solvers, where the step size does not always have to be $~T/K$.

In this work, we mainly consider the simplest setting by using the forward Euler scheme, and find that it is enough to yield good results. More rigorously, we also study the 4th-order Runge-Kutta (RK) method. As shown in **Figure 3 (middle)**, RK performs better than DGC with a small number of propagation steps $K$, and the advantage disappears with a sufficiently large $K$.

---
Thank you again for reviewing your paper and hope you find our explanations helpful. We are also willing to address your further concerns before the discussion session ends.

The paper investigated the feature propagation steps of linear GCNs from a perspective of continuous graph diffusion, and analyzed why linear GCNs fail to benefit from more propagation steps. From the perspective of diffusion equation, the authors decoupled the terminal time and the feature propagation steps, and proposed DGC to make it more flexible and capable of exploiting a very large number of feature propagation steps. Theoretical insights and extensive experiments on benchmark and large graph datasets have verified the superiority of DGC.   Strength:
1.	The paper is very insightful and inspiring, especially on the analysis of the linear part of GCN (SGC), which provide a clear explanation on why SGC will oversmooth, i.e., cannot benefit from more propagation steps. The view from diffusion process is novel and has solid theoretical foundations.  
2.	The proposed strategy of decoupling the terminal time T and the propagation steps K is simple but effective. It enables a flexible tradeoff between under-smoothing and over-smoothing and a more fine-grained numerical integration scheme. Besides, the authors provided a solid theoretical analysis for their method with asymptotic, numerical and risk properties from a continuous perspective. 
3.	Lots of experiments are provided to understand and evaluate the proposed methods. They show that the proposed method can outperform canonical GCNs significantly and obtain SOTA results among linear GCNs. Its performance is even close to that of SOTA non-linear GCNs, thus can serve as a strong baseline for future studies on GCNs. The experiments covered lots of aspects, including linear/non-linear models and extensive ablation studies, which definitely convinced me of the effectiveness of the proposed method. 

Weakness:
1.	The background on linear GCNs may not be clear. Why do we need to study linear GCN? Most of current models are non-linear. 
2.	The difference between with self loop and without self loop?
3.	The authors used the step-size to be T/K. Why exactly T/K? can we use larger or smaller than T/K? No significant social risk

Non-linear GCNs have attracted a lot of attention recently, and many methods are proposed around them. However, this paper finds and investigates that current GCN variants still have not shown significant advantages over a properly designed linear GCN (their proposal). This is significant to the whole field and can help shape a new direction to propose real powerful non-linear GCNs.
  Pros:

1. Although recent works have shown that a linear GCN can achieve comparable performance to the original non-linear GCNs, researchers still do not investigate this situation deeply. This paper fills this gap and, correspondingly, proposes a powerful linear GCN. This strong linear-case solution can help researchers rethink the most important part of the non-linear GCN. 

2. The paper is technically sound. It is novel to dissect the feature propagation steps of linear GCNs from a perspective of continuous graph diffusion. Experimental results demonstrate the effectiveness of the proposed method.

3. It is novel to link oversmoothing issue to the terminal time. The proposed dissection of linear GCNs suggests a different understanding of the oversmoothing problem. In Theorem 1, oversmoothing is an inevitable phenomenon of (canonical) GCNs, while this paper finds a terminal time to achieve an optimal tradeoff between under-smoothing and oversmoothing, which is novel and significant.

Cons:

1. Readability should be improved. For a reader who does not know GCN, it is hard to understand the feature propagation steps of GCNs and the meaning of continuous graph diffusion. Some basic intros regarding them should be added. For example, the preliminary section can be added.

2. In addition, T and K should be introduced in the caption as well to improve the readability of Figure 1. Some subfigures in Figure 1 are less informative (e.g., second line, the second one (left to right)). It is hard to get the main information from them.

3. The difference between linear and non-linear GCNs should be analyzed deeply. In some datasets, some non-linear GCNs still outperform linear ones. What properties contained in these datasets make non-linear ones better? In some datasets, linear GCNs are better than non-linear ones. What properties contained in these datasets make linear ones better? It would be better to discuss the results by visiting datasets in detail. The authors have not discussed limitations and potential social impacts. I suggest discussing the potential social impact from the trained networks that have low accuracy (in the node classification tasks).

This work studies linear GCNs and tries to provide a solution to find the optimal tradeoff between under-smoothing and over-smoothing. It first pointed out the limitations of SGC, then propose the Decoupled Graph Convolution (DGC) model to address the limitations. The key idea is to view the feature propagation step in linear GCN from a perspective of continuous graph diffusion, and dissect the terminal time and the propagation steps to enable choosing a flexible continuous terminal time (and therefore can benefit from more propagation steps).  Strength：
- Different from the discrete linear GCN model, this paper utilizes the continuous version of GCN so that the termination time T can be controlled independantly from the number of propagation setps K. Decoupling terminal time and number of steps is a very good point.
- Theoretical part is clear and seems solid. The authors first pointed out the weakness of SGC from the theoretical perspective and then show the proposed DGC address these weakness successfully. 
- Experiments from different aspects are conducted to verify the proposed DGC. The comparisons and the ablation studies are comprehensive and make it clear to know how and why DGC works. 

Weakness:
- In Figure 2, the authors set the DGC to K=100, i.e., propagating 100 times. And SGC only propagates 2 times, so why the training time DGC is only 3 times slower than SGC, is it because SGC converges slower than DGC?
- Could the authors give some insights on when linear models like DGC should be used in real applications, and when non-linear models should be used?
- The authors showed the robustness to feature noise in Sec. 5.4, but I am curious to know why DGC is better here? see the above weakness.

Meta-review: This is a half understanding and half methodology paper, studying why existing linear GCNs cannot benefit from more feature propagation steps, and proposing *decoupled graph convolution* to make linear GCNs possible to benefit from a very large number of feature propagation steps. The writing is clear, the motivation is strong, the idea is novel, and the results are significant. Thus, I think it should be accepted for publication.

Decision: accept