DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment
Authors: Gongpei Zhao
Tao Wang
Congyan Lang
Yi Jin
Yidong Li
Haibin Ling
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph neural network
direct feedback alignment
graph learning
non-BP training approach
Primary area: graph_neural_networks

Abstract: Graph neural networks (GNNs) are recognized for their strong performance across various applications, with the backpropagation (BP) algorithm playing a central role in the development of most GNN models. However, despite its effectiveness, BP has limitations that challenge its biological plausibility and affect the efficiency, scalability and parallelism of training neural networks for graph-based tasks. While several non-backpropagation (non-BP) training algorithms, such as the direct feedback alignment (DFA), have been successfully applied to fully-connected and convolutional network components for handling Euclidean data, directly adapting these non-BP frameworks to manage non-Euclidean graph data in GNN models presents significant challenges. These challenges primarily arise from the violation of the independent and identically distributed (i.i.d.) assumption in graph data and the difficulty in accessing prediction errors for all samples (nodes) within the graph. To overcome these obstacles, in this paper we propose DFA-GNN, a novel forward learning framework tailored for GNNs with a case study of semi-supervised learning. The proposed method breaks the limitations of BP by using a dedicated forward training mechanism. Specifically, DFA-GNN extends the principles of DFA to adapt to graph data and unique architecture of GNNs, which incorporates the information of graph topology into the feedback links to accommodate the non-Euclidean characteristics of graph data. Additionally, for semi-supervised graph learning tasks, we developed a pseudo error generator that spreads residual errors from training data to create a pseudo error for each unlabeled node. These pseudo errors are then utilized to train GNNs using DFA. Extensive experiments on 10 public benchmarks reveal that our learning framework outperforms not only previous non-BP methods but also the standard BP methods, and it exhibits excellent robustness against various types of noise and attacks.

DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment
Gongpei Zhao 1.2 Tao Wang Congyan Langl? Yi Jin 1.2, Yidong Li Haibin Ling²
Keyy Laboratory of Big Data & Artificial Intelligence in Transportation, Ministry of Education, China Schooo of Computer Science & Technology. Beijing Jiaotong University, China Department of Computer Science, Stony Brook University, USA {csgpzhao, twang, cylang, yjin, ydli}@bjte (.. edu.e..ucc.. hling@es stonybrook.
Abstract
Graph neural networks (GNNs) are recognized for their strong performance across various applications, with the backpropagation (BP) algorithm playing central role in the development of most GNN models. However, despite effectiveness, BP has limitations that challenge its biological plausibility and affect the efficiency, scalability and parallelism of training neural networks for graph-based tasks. While several canpbbccprropattionn (non-BP) training algorithms, such the direct feedback alignment (DFA), have been successfully applied to fully-connected and convolutional network components for handling Euclidean data, directly adapting these non-BP frameworks to manage non- -un-Euclidean graph data in GNN models presents significant challenges. These challenges primarily arise from the violation the independent and identically distributed (i.i..) assumption in graph data and the difficulty in accessing prediction errors for all samples (nodes) within the graph. To overcome these obstacles, this paper we propose DFA-GNN, a novel forward learning framework tailored for GNNs with case study of semi-supervised learning. The proposed method breaks the limitations of BP by using a dedicated forward training mechanism. Specifically, DFA-GNN extends the principles of DFA to adapt to graph data and unique architecture of GNNs, which incorporates the information of graph topology into the feedback links to accommodate the non- Euclidean characteristics of graph data. Additionally, for semi-supen supervised graph learning tasks, we developed a pseudo error generator that spreads residual eтrors from training data to create pseudo error for each unlabeled node. These pseudo errors are then utilized train GNNs using DFA. Extensive experiments on 10 public benchmarks reveal that our learning framework outperforms not only previous non-BP methods but also the standard BP methods, and it exhibits excellent robustness against various types of noise and attacks.
Introduction
As class of neural networks (NNs) specifically designed to process and learn from graph data, graph neural networks (GNNs) [Zhou et al. 2020 Wu et al. 2020 have gained significant popularity addressing graph analytical challenges. They have demonstrated outstanding success in various
applications, including recommendation systems (Wu et al. 2022). drug discovery [Xiong et al. 2021
and question answering (Yasunaga et al., 202T]. The impressive accomplishments GNNs, well as other neural network models, are largely attributed to the backpropagation (BP) algorithm (Hecht Nielsen 1992]. which has emerged as the standard technique for training deep neural networks.
"Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(x) HP
(b)
(c) FORWARGNN-SP
(1) DEA-GNN
Loss
Goodness
Goodace
I Local lons
Los
layer
layer
layer?
GNN layer?
Gooduess
Goodness
Local loss
GNN
GNN layer
GNN layer
GNN Iayer
Forward pass
Backpropagution
Liser traliinng
Graph structure
X: Node fentures
(2+, Grt:; Graph stracture plottieloogativv extending
Xes, Xww; Nodr fezatures by postireenngaiive extending
G: Augmented graph stracture
Figure Illustrations of BP, FF, FORWARDGNN and proposed DFA-GNN.
The backpropagation algorithm adjusts neural network weights based on the loss between the prediction and the ground truth, and allows the network learn and improve over time. However, despite its effectiveness, BP draws concerns on its biological plausibility for two main reasons [Hinton 2022 Lillicrap et al.] 2016]: (1) it uses the same weights in reverse order for both feedforward and feedback paths, creating the weight symmetry problem [Lillicrap et al.) (016);; and (2) its parameter updating relies on the activity of all downstream layers, leading to the update locking problem [Dellafereera and Kreiman 2022). These limitations may well impact the efficiency, scalability and parallel processing capabilities of neural network training.
To address these limitations, direct feedback aligmment (DFA) [ookland] 2016) offers an effective alternative to BP by training neural networks through a single forward pass. DFA uses fixed random feedback connections to project output errors directly onto hidden neurons, allowing for parallel gradient computation and eliminating the need for sequential backward error propagation. While demonstrated a limited accuracy penalty compared with BP, DFA aligns with brain-like learning mechanisms through its use of global error modulation and local synaptic activity, making it a notable non-BP method applicable in areas such as image classification (Zhao et al. 2023] and privacy
protection OOhna et al., 2021].
Directly applying DFA to GNNs, however, faces two challenges: (1) graph data often violates the
independent and identically distributed (i.i.d) assumption and thus ties the supervision gradients with the graph structure, making the straightforward error projection of DFA inadequate; and (2) DFA requires the prediction errors of all the input samples, while for graph data, especially under the semi-supervised setting, samples (nodes) without ground truth meet problems for the eтror calculation, complicating the deployment of DFA to GNNs.
To tackle these challenges, in this paper we propose DFA-GNN, a non-BP learning framework tailored for graph neural networks. Our primary contribution is to improve and extend DFA to graph neural networks. Specifically, we redesign the random feedback strategy for graph data to make the DFA portable to GNNs. The information from the graph topology, in the form of an adjacency matrix, is incorporated into the feedback links to accommodate the non- oon-uulideenn characteristics of graph data. We take graph convolutional network (GCN) (ipff and Welling 2016) as a case study, and derive the specific formula for updating parameters in each GCN layer. Furthermore, for the semi-supervised graph learning task, we develop a novel pseudo error generator that spreads residual errors from training data to generate pseudo error for each unlabeled node. Such pseudo eтTors are then used for the training of graph neural networks by DFA.
In summary, our proposed learning procedure for GNNs contributes in three significant folds:
We introduce DFA-GNN, a non-BP training algorithm that extends DFA to GNN architec- tures. It offers a more biologically plausible alternative to traditional BP methods.
For semi-supervised graph learning tasks, we develop a novel pseudo error generator that propagates residual errors from the training data to create pseudo errors for unlabeled nodes.

---

We prove the convergence of our DFA-GNN, and validate its effectiveness on 10 bench- marks. The experimental results demonstrate the superiority of our DFA-GNN against both traditional BP and the state-of-the-ar non-BP approaches.
2
Related Work
The biological implausibility of BP mainly lies in weight transport and update locking issues. feedback alignment (FA) [Lillicrap et al.) 2016) addresses the weight transport by using fixed random weights to convey error gradients. Building
this, direct feedback alignment (DFA) [Nokland 2016), direct random target projection (DRTP) [Frenkel et al.) 2021] and PEPITA (Dellaferrera and Kreiman 2022] further tackle the update locking problem with non-BP update methods. Prompted by the recent critiques of [Hinton| [222]] the
(FF) algorithm emerges as a more neurophysiologicall aligned alternative, using dual forward passes with positive and negative data to simplify the training process and accommodate non-differentiable elements. The recently proposed cascaded forward algorithm (CaFo) (Zhao et al., 20233 attaches class predictor each layer. where only the layer-wise predictors are locally trained, with each neural block being randomly initialized and remaining fixed throughout.
Our work aims to push the frontier of the non-BP training algorithm for GNNs, which is a field still in its infancy. A remarkable recent work along the line is FORWARDGNN (Park et al. 2023) inspired by the forward-forward algorithm. FORWARDGNN avoids the constraints imposed by BP via an effective layer-wise local forward training. It trains GNNs using single forward pass with the assistant of a data augmentation strategy. The augmented graph structure integrates virtual nodes linked only labeled nodes, leaving the local topology of unlabeled nodes unchanged. The augmentation strategy makes it possible to operate without generating negative inputs. Despite of its advantages, FORWARDGNN still suffers from the greed-based training strategy, and thus results in inferiority in prediction performance in comparison with traditional BP algorithm.
In contrast, method does not necessitate data augmentation for graph data; instead, it directly utilizes the discrepancy between predictions and actual ground truth to update each layer. Our approach directly outputs the prediction of the multi-class distribution, eliminating the need to calculate the goodness between unlabeled nodes and virtual nodes. As a result, our method offers convenience for multi-class prediction tasks and gains improvements in prediction performance.
3 Preliminaries
3.1 Problem Definition
An attributed relational graph of n nodes be represented by G (V,E,X), where V {11, U2 ... {n} represents the set of n nodes, and È = {eiy } signifies the set of edges. X ... x E Rnxd is the attribute set for all nodes, with x; being the d-dimensional attribute vector for node u; The adjacency matrix A {aaj} RRxn denotes the topological structure of graph G, where a:j > O if there exists an edge eij between nodes i2 and ny and a:j = O otherwise.
For semi-supervised node classification, the node set V can be split into labeled node set V C V with attributes X,, C X and an unlabeled one Vi = V/V, with attributes X = X/Xr, We assume that each node belongs to exactly one class, and denote yı {vi} as the ground-truth labels of node set V2 where y denotes the class label of node U; E Vz.
The objective of semi cemi-supervised node classification is to train a classifier using the graph and the known labels YL. and then apply this classifier predict the labels for the unlabeled nodes Wu. Define classifier fo No) +- Yo(X,A, where 0 is the parameters of model. Yz and Yu are the predicted labels of nodes Vz and vy respectively. Generally, the goal to make the predicted labels y, align as closely as possible with the actual ground-truth labels yr in favor of:0" arg ming d( u(5L.yz Ny) = arg ming d(fo(X. A, A.).))).. where d(.,.) represents a measure of some type of distance between two sets of labels.
For notation conciseness, we abuse the set notation and matrix notation interchangeably whenever appropri- ate. For example, X represents both set of attributes and matrix.
3

---

3.2 Direct Feedback Alignment
While BP relies on symmetric weights for error propagation to hidden layers, there is evidence suggesting that symmetrical weight distribution may not be crucial for learning. For example, the study of feedback alignment (FA) shows that learning can still occur when errors are back propagated using randomly fixed weights. Direct feedback alignment (DFA) advances in this direction by directly transmitting output errors to each hidden layer through fixed linear feedback links. Specifically, for an L layer network, feedback matrices B E RR) are employed to replace the derivatives 0x(L) /0x(1) of output neurons with respect to hidden neurons in the 1-th layer. The approximate gradient dw()) for the weights of the I-th hidden layer is then computed as: ow() Oc B [1+1) 0x (+1)
aWO)> where x()) represents the latent representation of sample at the 1-th layer, and L the loss value. In DFA, the feedback matrices assigned to hidden layers are randomly selected and remain unchanged throughout the training process. The effectiveness of DFA hinges on the alignment between forward weights and feedback matrices, leading congruence between the estimated and the actual gradient. When the angle between these gradients remains below 90 degrees, the update direction points to downward trajectory. DFA has been successfully implemented in popular deep learning architectures such as fully-connected neural network (FC) (Zhang et al.) [017] and convolutional neural network (CNN) eLi al. 2021a However, extending DFA graph neural networks remains unexplored.
4 Proposed DFA-GNN
As depicted in Fig- D our DFA-GNN aims train GNN models in non-BP framework by extending the DFA algorithm. Different from the original DFA algorithm designed on FC for Euclidean data, two key issues should be solved when extending it to GNN for graph data: (1) the original random feedback operations need to be reformulated to handle the dependence between samples (nodes); and (2) high-quality pseudo errors for test samples are required as they are not isolated from the training procedure. To this end, in Sec. 1. we redesign the random feedback strategy specified for graph data, and in Sec. [4.2 we develop novel pseudo error generator for sumi-supervised graph learning tasks. Finally, we provide deep insight of our DFA-GCN about its convergence and optimization in Sec. 4.3
4.1 Generalizing DFA to GNN
The training process of traditional BP for GNN is listed in Algo. of Appx. A. and it uses both a forward propagation and a backward one in each epoch. Generally, different GNN models may differ in the operations of aggregation and combination, and we provide typical implementation in Algo. M We take graph convolutional network (GCN) [Kipf and Welling 2016]. one of the most classic and successful GNN models,
study to integrate DFA for graph learning.
For illustrative purpose, we consider a three-layer GCN model with ReLU for hidden activation and sigmoid for output activation. The forward propagation process could be written
Layer 1 H (x)
x relu H (()))
Layer 2 (!) S
(2) relu H ( (w)),
Output layer: H (2) (x
X H (2) w(2)
= sigmodi((X(())
where = D A + I is the adjacency matrix of graph G after adding self loop. = (A) the diagonal matrix of A, ((1-1) a trainable weight matrix the /-th layer and a a non-linear activation function. x(!) pand denotes the latent representation matrix of the I-th layer and ()) = X. If we choose sigmoid activation function in the output layer and binary cross-entropy (BCF) loss function, the loss J for a graph with n nodes and the gradient E at the output layer are calculated as:
=
N
1
ΣD> Y
m. log Ym + (1 Ym,k) log(1 - Ym,,))
(2)
m,k
JJ
E = 8X (3)
Y - Y
(3)
(3
(3)

---

where Y and Y E Raxe are respectively prediction and one-hot ground truth; m and l are respectively sample index and output unit. important to highlight that represents the exact error between the prediction and the ground truth. For GCN, the gradients for hidden layers are calculated according to Algo. as: ox = S قH!
S'EW(a)T 6x(1) JH S w(1)T
As demonstrated by the work of FF (Lillicrap al. 2016) and DFA [Nokland 2016], learning can
be effective when eπrоrs are back propagated using randomly fixed weights. Similarly, we establish parallel direct feedback links for each layer. We approximate the update directions for the hidden layers as follows:
ox EBB
(1 (H) 5 (()
where B (()) is fixed random weight matrix with appropriate dimension. (x can be then written as: ox ST x ( S" EB (2) ()
) "
(5)
and the weight updates for all layers are calculated as:
vw()) H HOT ox(1,
sw() H HUT (x
5w H (2:TE
(6)
The above derivations can be easily extended to GCNs with more layers, as done our experiments.
4.2 Pseudo Error Generator by Spreading Residual Errors
In the computation of gradient (Eq. 6) the labels of all nodes required, but not all nodes labeled in the semi-supervised learning task. To address this issue, we introduce a mechanism to generate errors, assigning a pscudo error to each unlabeled node. The underlying principle is the expectation that errors in initial predictions are likely to propagate along the graph edges. That is, an error at given node suggests higher likelihood of similar errors to its the neighbor nodes. This concept of error propagation across the graph is supported by previous studies (Jia and Benson] 2202). Our approach draws inspiration from the strategy of residual propagation used in node regression tasks, and more broadly, from the frameworks of generalized least squares and correlated error models (Shalizi 2013].
In semi-supervised graph learning, the error matrix E = (e]: eh: ; } E Raxe as described in Eq. Bis modified as the residual on the training nodes, while being set zero for all other nodes. This adjustment entails initializing e; as a zero vector for all nodes U; E Vy.
The residuals in the rows of E for the training nodes are zero only when the forward process achieves perfect predictions. We utilize the label spreading technique [Zhou et al., 2003] to smooth the error with the goal of optimizing the following objective:
Z arg min tr(zz (1-S))) + p E ZERnxe
where tr(-) denotes the trace of matrix. The first term enhances the smoothness of the error estimation throughout the graph, while the second term ensures that the final solution stays consistent with the initial error estimate E. Following the optimization methodology in Zhou al. [2003 and
Huang et al. [222], the solution to Eq. can be obtained through iterative processing
(++1) = (1-a) + asz
a
1 1+ μ
O = E.
(8)
This process represents the diffusion of error, and such propagation is demonstrably appropriate within the context of regression problems under a Gaussian assumption. Nevertheless, for classification tasks like ours, the smoothed error Z* might not align with the correct scale. Typically,
(t <
(1-a) S||S||| (()) 22- (1 ax)||El|2 + a azlz || Starting with (0) E, find that | < ||E||2 indicating a need to adjust the scale of residuals adaptively. The aim is to match the magnitude of error in Z" to that in E as closely possible. Given that we only have accurate error information for labeled nodes, we use the average error across these nodes to estimate the appropriate scale. Specifically, with e,, z R representing the i-th row of E and Z" respectively, the adjusted enror for an unlabeled node j is calculated as: é, = 1 ///1|2/] | zJ, in which 1) T E,,v, ||e:]]]-
5

---

Give the rescaled pseudo enror for each unlabeled node, we define E E R where the i-th row is set to e for nodes U; E Vn. and to è for other nodes. The matrix E can then be directly utilized in Eqs. and GCN training. Nevertheless, not all rescaled error of unlabeled nodes are accurate and useful. To address this, a mask is implemented to filter out these nodes. Define Y = Y E as the corrected prediction. For labeled nodes, this corrected prediction equals to the one-hot ground truth. We introduce a mask vector E R" to facilitate this process. Formally, the setup is as follows:
1, { P
if count(;; > c) = 1,
0,
otherwise.
where p; is the i-th element (1 I S ni) of the vector p. c a manually set threshold for controlling filtering and c-)nt(-) a counting function. 9 focuses on retaining only those well predicted nodes,
characterized by a single category being identified as positive and the rest as negative. With the mask, the filtering operations could be performed on E and S by row through the mask. The weight updates in Eq. 6 are modified as:
sw(0) (0)
H (O)T (2)T EB",, (w()) = H (1)T ( EBB Sw(2) = H -H E
(2
(10)
where s/A) Ey. H represent the row filtering of sh E and H respectively, according to the mask p. As ( exactly denotes the accumulation of errors related to the k-hop neighbors of each node, the difference between Eq. and Eq. [T]lies in the partial sampling of neighbors approximate the update directions, as compared with using all neighbors. Given that only poorly predicted nodes excluded and they constitute a small fraction, the update directions in both approaches generally remain aligned. The formal description of our algorithm is provided in Appx. A.2
4.3 Insights of DFA-GNN
DFA-GNN provides non-BP training procedure for GNN. For GCN we provide detailed analysis on how such an asymmetric feedback path introduced in Sec. 4.1 can provide learning by aligning the gradients of backward propagation and forward propagation with its own. Nokland [2016] has originally proved the conclusion for fully-connected layer architectures. We can show that this conclusion is equally valid for the GCN architecture, and provide the detailed proof in Appx. A.3
Experiments in Fig. aaa validate the dynamic process of alignment between B and W during training, and the weight alignment leads to gradient alignment because for weight alignment of DFA:
((0c
B (UT (((1), w(D)
B (T)T.
(11)
where the symbol xx represents a positive scalar multiple relationship. As gradient alignment ST 5"xx(++ WW(T the weight alignment directly implies gradient alignment if the feedback matrices are assumed right-orthogonal, i.e.. BB" 1. This assumption holds if the feedback matrices elements are sampled i.i.d. from a Gaussian distribution since E B & I, hence Eq. 11 induces the weights, by the orthogonality condition, to cancel out by pairs of two:
requires 5x(() (I)
5x11) i.e., (
ST OX (1) x (BB)
(L) B(JT. BB..IIB ( (ST) (EB(
(12)
The alignments of weights and gradients make our method trained with DFA tend to converge to a specific region within the landscape, guided by the structure of the feedback matrices, while the optimization paths trained with BP according to stochastic gradient descent often exhibit divergent within the loss landscape, as shown in Fig. (d).
For deeper understanding of the training mechanism in our method, we divide the entire training process into three stages: Stage (train layers I and while freezing layer 3), Stage (freeze layers I and 2 while training layer 3) and Stage 3 (train layers 1 and 2 while freezing layer 3). The results in Figs. bb,, show a strong correlation between weight alignment and the fitting degree of the model, as indicated by the loss. Notably, even though our method updates parameters of each layer in parallel, the effective update follows a wookkwwwd-t--oowwaeee the parameters of layer are not effectively learned, updating the preceding layers does not enhance fitting ability of the model. This behavior contrasts with the characteristies of traditional BP, which indicates that the alignment of weights and gradients also adhere to backward-te -orrwwrrdd sequence.
manner. As shown Fig. 2 (b), when

---

Saag
SA4E 2
sage1
towingg boc.
AMG 8b)
2MC 2M000000)
60-
6:25
62
accunsty
6.50-
18
DEXgatt
1poon
420ch
6ponn
(b)
(c)
Figure 2: For three-layer GCN model trained by DEA-GNN on Cora, (a) the accuracy and angle between W and B; (b) the change in loss across different stages; (c) the change in angle between W and B across different stages; (d) difference optimization direction between BP and our method.
Table 1:
Results on datasets: mean accuracy (%) + 95% confidence interval. The best result on each dataset is indicated with bold.
BP
PEPITA
CaFo+MSE
CaFo+CE
FF+LA
FF+VN
SF
Ours
86.04
25.78
71.79
71.78
84.20
74.50
84.54
87.72
Cora
10.62
16.24
11.76
#1.71
10.85
#1.54
+0.77
=1.63
78.20
21.24
65.43
63.12
75.25
69.97
73.84
80.49
CiteSeer
+0.57
+2.63
+0.99
#1.15
#1.09
+1.08
+1.02
=0.41
85.24
36.13
77.66
78.29
83.68
79.60
84.68
86.28
PubMeb
+0.28
10.88
10.82
10.64
10.38
10.62
#0.61
0067
93.03
70.63
89.48
90.59
86.39
15.56
92.48
93.04
Photo
+0.59
+7.13
+0.33
+0.26
#3.46
+9.38
+0.33
=031
89.48
63.25
83.02
82.94
75.87
12.27
84.04
86.72
Computer
+0.37
+88.8
+0.59
+0.73
+4.55
+160
+0.65
=0688
54.26
39.67
56.39
31.14
59.67
19.67
37.71
79.51
Texas
+3.44
+18002
+3.44
#3.44
+3.28
112.0
+2.5
1.97
71.31
50.49
28.85
36.72
45.90
18.52
60.66
75.24
Cornell
+4.43
20066
+2.95
+5..3
+8.04
122.6
+5.2
=4.92
31.94
21.55
23.83
23.83
33.58
15,05
28.33
34.07
Actor
+0.88
+3.86
10.76
10.64
#1.54
+7.26
+1.38
=075
41.28
36.97
37.36
36.48
33.21
24.76
42.35
41.19
Chameleon
+2.29
+2.45
+1.91
+1.44
+1.99
+2.61
+2.27
=1.56
37.81
33.99
31.00
31.00
33.66
18.91
36,00
38.17
Squirrel
+-0.1
+1.24
+1.18
+00.2
+-0.87
+-4.28
+1,50
-2.21
Experiments
5.1
Comparison with Baseline Training Algorithms
We evaluate our method on 10 benchmark datasets across various domains and compare it with the
BP |Rumelhart et al. 1986), PEPITA (Dellaferrera and Kreiman) 2022), two versions of the FF (abbr:
FF+LA, FF+VN) [Hinton] 2022, Park al.) 2023). two versions of CaFo (abbr. CaFo+MSE,
2023 Detailed CaFo+CE) (Zao et al.) [2023] and the FORWARDGNN-SRT (abbr. SF) PPark et al.
datasets and experimental setup information can found in Appx. A.4 and Appx. A.5 respectively.
The comparative analysis of various algorithms benchmark datasets summarized in Tab. While the non-BP methods such as PEPITA, CaFo and FF have proven effective for architectures involving fully-connected and convolutional layers with Euclidean data, they exhibit weaker performance with non- non-Euclidean graph data. This is primarily due to the unique challenges posed by graph data.
Firstly, in typical fully-connected and convolutional layers, shallow layers capture coarse- coarse-grained features while deep layers handle fine-grained features, with these two types of features usually being highly correlated. However, in GNNs, different layers aggregate information from varying neighborhood ranges, resulting layers that often contain uncorrelated information. Particularly in heterophilic graphs, the information extracted by deep and shallow layers may be entirely unrelated

---

Table 2: Ablation study results on different datasets with proposed designs.
EG
NF
Cora
CiteSeer
PubMed
Actor
Chameleon
Squirrel
x
83.02
08.17:0.77
82.
39.09=1.17 30.72::..22
34.61#1.01
/
86.70=1.000
39.62=1.31 31.9711.46 84.0110.23 79.26:1.90
34.87:1.78
/
87.72++1.63
80.49-0.41
86.28
34.07=40.75
1.56
38.17+2.21
even have opposing effects on predictions. This lack of correlation complicates the application of layer-wise optimization strategies, which rely on greedy strategies and local loss calculations common in traditional networks. This is a key reason for the underperformance of methods like PEPITA, CaFo and FF in GNNs, as evidenced Tab. ! particularly on datasets characterized by low homophily. Secondly, since graph data do not adhere to the i.i.d. assumption, sampling positive and negative samples based on features (FF+LA) and topology (FF+VN) for the FF algorithm can be unreliable, potentially leading to inconsistent results. Notably, FF+VN modifies the original graph topology by introducing virtual nodes into both positive and negative graphs, which results in overall unsatisfactory performance in benchmarks. For CaFo, the rigidity fixing the parameters of each block, with only the predictors being learnable, further constrains its adaptability, As the most recently proposed non-BP GNN training approach, SF shows performance that is superior to PEPITA, CaFo and FF but still lags behind the traditional BP method on most datasets. The reason lies in that SF also introduces virtual nodes that disrupt graph topology and employs a layer- wise training strategy. By contrast, our method well adapts to graph data and gains significant improvement in testing accuracy in comparison with the baseline algorithms, achieving the best or second-best results across all datasets.
The training times for each method are shown in Appx. Our approach demonstrates general time advantage over CaFo, FF and SF. Our method includes a forward propagation and a parameter update where all layers execute in parallel during each iteration, which offers greater parallelism compared with layer-wise update methods like CaFo, FF and SF. Our method has higher training time consumption compared with BP, primarily due to the additional time needed for generating pseudo errors and filtering masks, as discussed in Sec.
5.2 Ablation Study
We ablate the proposed method to study the importance of designs in DFA-GNN. Two designs are studied including the pseudo error generator (abbr: EG) and the node filter (abbr. NF). For trials with EG, the pseudo error generator is applied according to Eq, to assign a pseudo error for each unlabeled node, worth noting that when ER is removed from the method, only the errors of labeled nodes are used for the updating of parameters according to Eq. 6 For trials with NF, the mask calculated as Eq. is introduced in training process and the parameters are updated according to Eq. TO The ablation results are included in Tab. We note that even the most naive version of DFA-GNN elaborated in Sec. 4.1 achieves comparable results in comparison with BP. Furthermore, both the two designs introduced in Sec. 4.2 contribute
training framework, making significant enhancement to DFA-GNN to outperform BP method.
5.3 Visualization of Convergence
To better illustrate the training convergence of DFA-GNN, we plot the training and validation accuracy of the proposed DFA-GNN and BP over training epochs on three datasets, as shown in Fig. 3 In general, our method shows similar convergence to BP. For both BP and our method, the convergence of validation accuracy occurs much earlier than that of training accuracy due to overfitting. The validation convergent epoch of DFA-GNN is nearly the same as BP on Cora and CiteSeer (around 100 epochs), while it 100 epochs later than BP on PubMed (around 200 epochs). Our method achieves better validation accuracy on all these datasets and suffers less from overfitting compared with BP. In terms of training accuracy, the convergence of our method is slightly slower than BP because the update direction of our method is not exactly opposite to the gradient direction but maintains a small angle. Since our method considers both the errors labeled nodes and pseudo errors of unlabeled nodes as supervision information, which is different from BP that only uses the loss of labeled nodes
8

---

Oum train
Accurary
Accurary
0Lm V20
Epoch
Epoch
Epoch
(a) Cora
(b) CiteSeer
(c)
PubMed
Figure 3: Visualization of the convergence of BP and our method on Cora, CiteSeer, and PubMed.
CloserSt Pabliet Pdddlie
Prantct
Core Cadderrouu
Caderrun
Chicler car
444
66
06
06
04
0.6
6.6
the number of layors
parturbation rate
perturbation rate
parturbation rate
(a)
Figure 4: (a) Test accuracy with the model layers increasing. (b-d) Test accuracy with the perturbation rate (b: add, c: remove, flip) increasing.
for supervision, the convergent value of training accuracy for our method is slightly lower than BP. However, this does not affect our method achieving better validation results.
5.4 Robustness Analysis
We focus on over-smoothing and random structural attack, two common sources of perturbation that reduce GNN performance. Over-smoothing [Keriven] 2022] is problematic issue in GNNs, stemming from the aggregation mechanism within GNNs, which hinders the expansion of GNN models to large number of layers. We test the robustness our method against over-smoothing in Fig, (a). Our method demonstrates greater robustness compared with BP, particularly when dealing with architectures that have large number of layers. This enhanced robustness is due the fact that the global loss directly contributes to the optimization of each individual layer. SF is less effected by over-smoothing due to its layer-wise optimization with local loss. However, its performance largely depends on shallow layers and the best performance on each dataset is inferior to ours.
For random structural attack [Li al. 2021b), three random attack types implemented on the original graph topology with a perturbation rate À from 0.2 to 0.8. To better compare the robustness of different methods, we employ more challenging experimental setup with a sparse supervision pattern, where each class has only 20 labeled nodes. The node split follows Kipf and Welling (2016). The detailed operating description for attacking type is summarized as follows: (1) add: randomly add A|E| edges to the original graph for denser topology: (2) remove: randomly remove X|E| edges from the original graph for sparser topology; and (3) flip: randomly choose X | node pairs, and remove the edge if there exists one between the pair, and otherwise add an edge to connect the pair.
Our method is less sensitive all types of perturbations shown in Figs. (b-d), consistently outperforms two approaches on each trial, and exhibits exciting robustness even under high perturbation rate. As the pseudo error generator derives pseudo error for each unlabeled node to update each layer, this supervision generation mechanism helps enhance robustness of the model against noise and attacks. Interestingly, comparison of results across three different types of attacks shows that removing edges has the least adverse effect, suggesting that injecting incorrect topological information could be more detrimental GNN performance than losing valuable original topology.
5.5 Scalability on Large Datasets
Our method is well-suited for large datasets. When the graph scale is large, we can use edge indices instead of adjacency matrix to store the graph. For forward propagation (Eq IT. the complexity

---

Table 3: Results on three large datasets. OOM denotes out of memory.
FF+LA
FF+VN
PEPITA
CaFo+MSE
CaFo+CE SF
BP
Ours
Flickr
6.09
42,40
49.28
50,02
49.69
46.47
50.79
49.80
Reddit
12.44
00M
OOM
88.15
91.55
94.38
94.34
94.49
ogbn-arxiv
56.38
19.84
35.16
53.51
60.57
66.54
68.78
67.83
Table 4: Performance of our method integrated with different GNN models.
Cora
CiteSeer
PubMeb
Photo
Computer
SGC+BP
85.30++0.79
79.45=00.87
79.72+t0.35
91.71=0.44
85.07::0.27
SGC+ours
88.60:10.55
81.02:10.78
80.40:10.18
92.01=0.29
85.94::0.31
GAT+BP
85.35:+0.62
1..0
85.78+00.3
93.16=#0.43
88.91+0.78
GAT+ours
86.96
79.77=1.15
86.18
93.24=00.51
87.38+00666
GraphSage+BP
87.04++0.84
19.65+1.111
88.32++0.28
92.89-t0.51
88.00++0.29
GraphSage+ours
1.00
7..69=1.33
87.65=+0.28
93.95=0033
88.14:10.39
APPNP+BP
85.75:10.89
80.46110.37
85.81=10.28
91.55=10.81
85.50:0.557
APPNP+ours
8.181+1.11
79.47+0.80
85.75++0.32
91.66=00.99
85.68+00.43
ChehNet+BP
83.45++1.07
76.93++0.71
87.09--0.31
90.89-0.74
86.51+0.78
ChebNettours
1..5
77.85
86.43++0.77
92.65=0.49
87.02:+0.65 6.62
of neighbor aggregation can be reduced from (n d) to ((EE)), where |8] denotes the number of edges. For direct feedback alignment (Eq. 10p. y is exactly the aggregation of errors for k times, the time and space complexity can be reduced to (kc) without the need to calculate the k--th power of the adjacency matrix. Similarly, complexity reduction can also be achieved in the
node filtering process. The experimental results on the Flickr [Zeng et al. 2020). Reddit [Hamilton)
al. 2217), and ogbn-arxiv (Hu et 2020] datasets, presented in Tab. 3 demonstrate that our method is effective on large-scale datasets, delivering strong performance. Our method achieves results comparable
BP while surpassing other non- -P methods, all with small memory footprint (2043 MiB for Flickr, 11675 MiB for Reddit, and 2277 MiB for ogbn-arxiv). This indicates that our method not only scales well large graphs but also maintains efficiency in terms of space usage.
5.6 Portability Analysis
We apply our training algorithm to five popular GNN models [Wu 2019 Veličković et al. 2018
Hamilton et al. 2017 Gasteiger et al. 2018 Defferrard et al. 2016 and report the mean accuracy across ten random splits in Tab. 4 Each of the testing models is modified to fit our framework. Specifically, for SGC, which only has a single learnable linear output layer, the training of our framework involves direct feedback but only incorporates the pseudo error generator and node filter. For GraphSage, we utilize a aggan-aggregator for message aggregation. Our observations indicate that our method can be effectively ported to mainstream GNN models. All test models integrated with our algorithm work well and surpass the performance of traditional BP in most scenarios. It demonstrates the effectiveness of our method across various GNN models and underscores its excellent portability and potential generalization ability to other innovative GNN models.
6 Conclusion
In this paper, we investigate the potential of non-baccproppggation training methods within the context of graph learning. We adapt the direct feedback alignment algorithm for training graph neural networks on graph data and introduce DFA-GNN. This new approach incorporates a meticulously designed random feedback strategy, pseudo error generator, and node filter to effectively spread residual errors. Through mathematical formulations, we demonstrate that our method can align with backpropagation in terms of parameter update gradients and perform effective training. Extensive experiments on real-world datasets confirm the effectiveness, efficiency, robustness and versatility of proposed forward graph learning framework.
10

---

Acknowledgments
This work is supported by the National Nature Science Foundation of China (Nos. 62076021 and 62376020). Haibin Ling was not supported by any fund for this work.
References
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey graph neural networks, IEEE Transactions Neural Networks and Learning Systems, 32(1):4-24, 2020.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys, 55(5):1 2022.
Jiacheng Xiong, Zhaoping Xiong, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng. Graph neural networks for automated de novo drug design. Drug Discovery Today, 2686):138 2021.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint arXiv:2/00.06377, 2021.
Robert Hecht-Nielsen. Theory of the backpropagation neural network. In Neural Networks for Perception, pages 65-93. 1992.
Geoffrey Hinton. The forwardffrwardd algorithm: Some preliminary investigations. arXiv preprint arXiv:2212..33355 2022.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1): 13276, 2016.
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit assignment problem without a backward pass. In International Conference on Machine Learning, pages 4937-4955, 2022.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in Neural Information Processing Systems, 29, 2016.
Gongpei Zhao, Tao Wang, Yidong Li, Jin, Congyan Lang, and Haibin Ling. The cascaded forward algorithm for neural network training, arXiv preprint XXXx:2303.0972. 2023,
Ruben Ohana, Hamlet Medina, Julien Launay, Alessandro Cappelli. Iacopo Poli, Liva Ralaivola, and Alain Rakotomamoniy. Photonic differential privacy with direct feedback alignment. Advances in Neural Information Processing Systems, 34:22010-22020 2021.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2016.
Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers in Neuroscience, 15:629892, 2021.
Namyong Park, Xing Wang, Antoine Simoulin, Shuai Yang, Grey Yang. Ryan A Rossi, Puja Trivedi, and Nesreen K Ahmed. Forward learning graph neural networks. In International Conference Learning Representations, 2023.
Yuchen Zhang, Jason Lee, Martin Wainwright, and Michael Jordan. the learnability of fully- connected neural networks. In Artificial Intelligence and Statistics, pages 83-91, 2017.

---

Zewen Li, Fan Liu, Wenjie Yang, Shoubeng Peng, and Jun Zhou. A survey of convolutional neural networks: analysis, applications. and prospects. IEEE Transactions on Neural Networks and Learning Systems, 33(12):6999-7019 2021a.
Junteng Jia and Austion R Benson. Residual correlation in graph neural network regression. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pages 588-598, 2020.
Cosma Shalizi. Advanced data analysis from an elementary point of view. 2013.
Dengyong Zhou, Olivier Bousquet. Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. Advances Neural Information Processing Systems, 16, 2003.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. Combining label propagation and simple models out-performs graph neural networks. In International Conference on Learning Representutions, 2021.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by Dack-propagating errors. Nature, 323(6088):533
1986.
Nicolas Keriven. Not too little, not too much: theoretical analysis of graph (over) smoothing. Advances in Neural Information Processing Systems, 35:2268 -522218-2281, 2022.
Yaxin Li. Wei Jin, Han Xu, and Jiliang Tang. Deeprobust: a platform for adversarial attacks and defenses. In AAAI Conference on Artificial Intelligence, volume 35, pages -6078-16080, 2021b.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. International Conference on Learning Representations, 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, 30, 2017.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning graphs. Advances in Neural Information Processing Systems, 33:221 18-22133, 2020.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simpli- fying graph convolutional networks. In International Conference Muchine Learning, pages 6861-6871, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2018
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in Neural Information Processing Systems, 29, 2016.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad -Rad. Collective classification in network data. Al Magazine, 29(3):93-93, 2008.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learming, pages 40-48, 2016.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. image-based recommendations styles and substitutes. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 43-52, 2015.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2)ccnab014, 2021.
12

---

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuar Chang, Yu Lei, and Bo Yang. Geom gen: Geometric graph convolutional networks. In International Conference on Learning Representations, 2019.
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? In International Conference on Learning Representations. 2022.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J. Smola, and Zheng Zhang. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR Workshop Representation Learning on Graphs and Manifolds, 2019.
13

---

end for A APPENDIX
Here we provide some implementation details our methods help readers further understand the algorithms and experiments in this paper.
A.1 GNN Training Process of BP
Algorithm 1 GNN Training Process of BP
Input: G. X. W. YL max_epoch.
Output: y. W.
1: for epoch in [0, I,... mux moux_epoch-1] do
2: for l in [0. I,.., L 1] do (Forward propagation)
3: for v in do
h EvE eva Xu ( (Aggregation)
4:
5:
6:
X (H Wi));; (Combination)
7:
end for
loss Loss_ cotssslomplnngg (() .NL)
8:
9:
for l [T. L 2, 0] do (Backward propagation]
(w -H (1)T ox 11+1), Combination)
10:
(H (++ W(I)T [Combination)
11:
for v in do
13:
6x = = Suv _h
(1)
( [Aggregation]
14:
end for end for
15:
Weighl_ Biight_upddainngeppdd w()), sw()) w(L-1), ow(l-1)))
16:
end for
18:
arg arg max Y arg max x (1))
A.2 GCN Training Process of Our Method
Algorithm 2 GCN Training Process
Input: G, X, W. Fi. max_epoch.
Output: y. W.
1: Initialize (B (1) B(2) B(L-1)):
2: for epoch [0, 1, .... max_ cerrxx_ppoc-]]]]_ do
3: Compute the prediction Y given by Eq. 1 (Forward propagation
4: Compute the error E. using Y,y for labeled nodes according to Eq.
5: Compute pseudo error Z' for unlabeled nodes by Eqs. g and rescale to get rescaled error Æ;
6: Compute mask vector p given by Eq. D
7:
for in [0. L do
8:
if = L then Compute (w using É. p according to Eq.
9:
10 (Direct feedback]
10:
else
11:
Compute (w() using É, p and B (I+1) according to Eq.
(Direct feedback)
12: end if
13: end for
14: W Weight_uppatiigg
w sw()) w(L-1) ow((-1));
15:
end for
-arg max Y.
14

---

A.3 Proof of Theorem in Sec. 4.3
Theorem. For GCN model with two hidden layers &o and where l connects to k + 1 in sequence, we have x(++1) (a((+1)) and a (k+1) g (Wx(k)), where is the activation function and g()) the aggregation operation in Algo. Let the layers be updated according to the non-zero 6x and
update directions (x( and (x(++1) where [|x((
(x [xx(++1] are constant for each data point. The
negative update directions will minimize the following layer-wise criterion:
5x ( (8)
(x((+1/xx(++))
P ple) + plk+1)
(13)
0 [9
[xx(k+1)||
Minimizing will lead an increase in the gradient, thereby enhancing the alignment criterion:
(x(
(x(++15Tc(b+))
= (k) (+++)
(14)
|xx(H)]
[,x(k+11]
where
(1)) 0x
0x(k) 8x (++1) ('x'(0x((+t))
(++1) @ (((aa(()))),
(15)
((+1)) (x(++1) (x (x(1) g'(W8x(() (k)) @ o' (a(aat))))
g'()) is the aggregation of gradients in Algo. (line 13). If O()) 0, then -8x(2) serves direction of descent to minimize p(k+1)
Proof. Let be the any of the layers & or k + 1, and preseribed update - (xx(1) is the steepest descent
direction to minimize p()) Since any partial derivative of ||x
(x
zero, have:
O i
(x (0)
Ox(0)
(x())
(1)
(0) 6x(0,
(16)
(x(1)
(x(()
|ox(00]
Ox(1)
[xx())
[8x(0|| Ox(1)
where (1)
As
(a
(x
6x(1)
(x)
C o' ()), the gradients maximizing Q())
[|
Ca
and Q (++1)
are:
aQ (
a
5x(iTT
(())
(x(1)
|c(()
ael() 5x(1)
ox(())
Oc(1)
deli)
|8x(11|
dc(i]
|xx(1)
Deli)
[xx(1|
aQ (k+1)
aQ
(k+1)
(c(++1)
de(k+l) oW = (1-11) ('x(1))((xx((+t) (xx(x++1) 2 (00(2 = =a (k+1) g' (x+ (1+ (xx
OW
(k(
delk) (Q(k)
aw de(KYT = y (fx(k+1) )
(k) g' (1+
aW
(17)
When ignoring the magnitude of the gradients we have onto (x we have (6)
80
80 ++)
ow If x is projected
ow
Aw
5x
1 2 (x + Kres. The prescribed update for W is:
ow (k+11) (k+1+
OW (0x о o'
g (8+(11)))) (K)T (0) (x()) +
(k)
(k) yr Xyes)
(18)
(k) 00 00(0) (8)
(()())((k)) (o((k)))) (A) (6a
g' (k+(1) (b)T
oW
is obvious that Q and (k+1) can be maximized when the component of (Q maximized by minimizing p(A) The gradient to minimize p(() is the prescribed update -8x(t The
in 8W is
aw
15

---

angle between (x(()) and the gradient of BP (A)) is within 90º if Q 0 because the cosine of the two vector Q(4)
| > O, and it also indicates that c(*) is nonzero and therefore a descending trend.
Consequently, 8x(2) will be oriented towards a descending direction since any vector that lies within 90 of the steepest descent direction will similarly point downwards, in other words, for GCN, broad spectrum of asymmetric feedback paths can offer a descending gradient direction for a hidden
layer as long as Q (1) > 0.
From Eq. 5 is obvious one advantage of our method that 8x(1) is non-zero for any non-zero error e, as randomly generated matrix B()) is highly likely to be of full rank. Ensuring 5x(() is non-zero crucial for achieving Q > O. Maintaining static feedback across training helps preserve the characteristic, and also simplifies the process of maximizing (6) due to the more consistent direction to 8x(*), Our method shows better biological plausibility in GNN training, which introduces asymmetric feedback paths to take place of BP, not only solving the weight transport problem and partially solving the update locking problem. but also releasing the requirement to store neural activations and accumulated gradients for backward propagation.
A.4 Datasets Statistics
We evaluate our method on 10 benchmark datasets across domains: citation networks (Cora, Cite-
Seer, PubMed) [Sen et al.) 2008 Yang al., 2016), Amazon co-purchase graph (Photo, Com- puter) [McAuley et al. 20155. Wikipedia graphs (Chameleon, Squirrel) [Rozemberczki et al. 2021 actor co-occurrence graph (Actor) PPi et al.) 2019 and webpage graphs from WebKB (Texas, Cornell) (Pei et al. 2019]. The datasets adopted are representative which describe diverse real- life scenarios. Some of them are highly homophilic while others are heterophilic. Note that for Chameleon and Squirrel, we use the filtered version from Platonovv et al.] [2022). as the original version from [Rozemberezki et al. 202T] may contain duplicated nodes. The detailed statistics of the datasets are summarized in Tab. We compute the node homophily for each dataset using the method proposed by Pei al. 2019 referred to as the homophily value.
Table 5: Datasets statistics.
Cora CiteSeer PubMed Computer
Photo
Chameleon Squirrel Actor
Texas Cornell
Nodes
2708
3327
19717
13752
7650
890
2223
7600
183
183
Edges
5278
4552
44324
245861
119081
17708
93996
26659
279
277
Features
1433
3703
500
767
745
2325
2089
932
1703
1703
Classes
7
6
5
10
8
5
5
5
5
Homophily 0.825
0.706
0.792
0.785
0.836
0.244
0.190
0.220
0.057
0.301
Cora, CiteSeer and PubMed [Sen et al. 2008 are three classic homophilic citation net- works. In these networks, nodes correspond to academic papers, and edges signify the citation links between papers. The node features are derived from bag-of-word representa- tions of the papers, and the labels categorize each paper into specific research topics.
Computer and Photo [McAuley et al. 2015 are segments of the Amazon co-purchase graph, where nodes represent goods, edges indicate that two goods are frequently bought together, node features are bag-of-words encoded product reviews, and class labels are given by the product category.
Chameleon and Squirrel [Rozemberezki al., 2021 are two heterophilic networks derived from Wikipedia. In these networks, nodes represent Wikipedia web pages, and edges correspond hyperlinks between these pages. The features are comprised of informative nouns extracted from the Wikipedia content, while the labels reflect the average traffic of each web page.
Actor (Pei al., 2019 is a heterophilic actor co-occurrence network where nodes represent actors, and edges signify that two actors have appeared together in the same movie. The features are derived from keywords found on the actors' Wikipedia pages, while the labels consist of significant words associated with each actor.
16

---

Cornell and Texas [ei et al., 2019 are heterophilic networks from the WebKBI project representing computer science departments at three universities. Nodes are departmental web pages, edges represent hyperlinks, features are derived using hag-of-words, and labels categorize page types. These networks illustrate heterophilic connections where linked pages often differ in type.
A.5
Experimental Settings
We conduct the sumi-uupee node classification task using a basic GCN model, where the node set is randomly divided into the train/validation/test set with 60%/20%/20%. For faimesss, we generate 10 random splits using different seeds and evaluate all approaches on these identical splits, reporting the average performance for each method. Our method is compared with five baseline training strategies including the traditional backpropagation (BP) [Rumelhart al., 1986]. PEPITA (Dellaterrera and Kreiman 2022]. two versions of the forward-forwand algorithm (abbr: FF+LA, FF+VN) [Hinton] 2022 Park et al. 2023 two versions the cascaded forward algorithm (abbr: CaFo+MSE, CaFo+CE) Zhho et al. 2023 and the FORWARDGNN Single-Forward algorithm (abbr: SF) [Park et al., 2023 specifically designed for GNNs. To ensure fairness, we train approaches using the same GCN architecture, which includes 3 graph convolutional layers and 64 hidden units sufficiently representative for all datasets. We employ Adam as the optimization algorithm, refraining from using any regularization techniques other than appropriate L2 penalty specific to Adam. The evaluation metric used is accuracy (acc), presented with a 95% confidence interval.
Table 6: Hyper-parameters of proposed method real-world datasets.
Learning rate Hidden unit
a
Iteration epoch
Weight decay Training epoch
Cora
0.01
64
0.1
50
0.5
0.0005
1000
CiteSeer
0.01
64
0.01
200
0.5
0.0005
1000
PubMeb
0.01
64
0.1
200
0.5
0.0005
1000
Photo
0.001
64
0.01
50
0.5
0.0005
1000
Computer
0.001
64
0.01
50
0.5
0.0005
1000
Texas
0.01
64
0.9
50
0.5
0.0
1000
Cornell
0.01
64
0.5
50
0.5
0.0
1000
Actor
0.01
64
0.5
50
0.5
0.0
1000
Chameleon
0.01
64
0.5
200
0.5
0.0
1000
Squirrel
0.01
64
0.5
200
0.5
0.0
1000
The codes of DFA-GNN are based on the GNNs in the PyTorch version by Deep Graph Library
(DGL) [Wng et al. 2019). To generate pseudo errors, we search the optimal a in Eq. 8 within (0.001, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9), the iteration epoch for Eq. α within (5, 10, 30, 50, 100, 150, 200), in Eq. within (0.3. 0.5, 0.7, 0.9]. J. For the training of our method, we search the learning
rate within (0.001, 0.01, 0.1) and weight decay within (0.0005, OJ. All the experiments are run on AMD EPYC 754225222 32-Core Processor with Nvidia GeForce RTX 3090. We list the
values used in our model Tab.
17

---

A.6 Time Comparison
Table 7: Average running time per epoch (s). For layer-wise training methods like PEPITA, CaFo, FF and SF, the total time taken by each layer per epoch is reported.
BP
PEPETA
CaFo+CE
FF+LA
FF+VN
SF
ours
Cora
7.56e
8.73e 3
7.6le 1
3.14e
2.83e
5.49e
5.66c
CiteSeer
1.06e
1.11e-2
7.68e
2.59e-1
2.61e-1
6.88e
5.68e
PubMed
1.07e
1.07e
8.24e
6.94e-1
7.61e
5.34e
6.76e
Photo
8.74e
1.03e
7.98e
2.11
1.91
4.87e
5.81e -2
Computer
1.08e
1.05e
7.80e
4.82
4.14
7.61e
6.29e
Texas
6.13e
1.07e
8.05e
1.47e
1.50e
6.88e
5.60e
Cornell
5.42e
1.06e
7.46e
1.51e
1.24e
3.59e
5.53e
Actor
9.45e 3
1.03e 2
7.83e
6.84e
6.71e
2.80e
5.80e"
Chameleon
6.24e-3
1.13e-2
7.97e
2.28e-1
2.09e
6.88e
5.61e
Squirrel
7.77e
1.20e
7.78e
5.82e
5.05e
1.21e-1
5.79e
A.7
Comparison of BP and DFA with Pseudo-Error Generation
Although the pseudo-error generation process is essential for our method, is also an optional choice for backpropagation. We integrate this component into BP, and the experimental results from that although this component contributes to DFA in our method, it does not positively enhance BP overall. Even with pseudo-error generation, BP cannot outperform our method. This observation indicates that direct feedback of eπrors may benefit more from pseudo-error, -erorrs rather than the ba-yyr backward pass.
Table 8: Results of BP and DFA with pscudo-error generation spreading: mean accuracy (%)=95% confidence interval.
Cora
CiteSeer
PubMed
Photo
Computer
Actor
Chameleon
Squirrel
86.04
78.20
85.24
93.03
89.48
31.94
41.28
37.81
#0.62
+0.57
+0.28
#0.59
=0.37
+0.88
=2.29
+0.71
87.41
80.24
84.74
91.95
87.78
31.53
38.71
35.78
BP+EG
+0.80
+1.11
+0.41
+0.41
+0.51
+1.91
2223
+1.52
(1.37f)
(2.041)
(0.50L)
(1.081)
(1.704)
(0.41
(2.574)
(2.03J)
83.02
78.17
82.92
91.75
84.02
30.72
39.09
34.61
DFA
+1.36
+0.71
+0.42
+0048
11.54
+1.72
11.1
+1.01
87.72
80.49
86.28
93.04
86.72
34.07
41.19
38.17
DFA+EG (ours)
+1.63
#0.41
+0.67
#0.31
-0668
+0.75
=1.5
+2,11
(4.701)
(2.321)
(3.361)
(1.291)
(2.701)
(3.351)
(2.101)
(3.56t)
A.8 DFA-GNN with Alternative Activation Functions
We conduct experiments for our method with four different activation functions (i.e., Sigmoid, Tanh, ELU and LeakyRel.U) shown in Tab. g The results demonstrate method is well integrated with different activation functions and derives consistently good results.
18

---

Table 9: Results of our method with different activation functions.
ourstLeakyReLU
ours+Sigmoid
ours+Tanh
ours+Rel.U
ours+ELU
(slope=0.2)
Cora
1..6
88.17=1.66
87.72++1.63
8.75
87.64:1..65
CiteSeer
79.94++0.74
80.01=+0.75
80.49++041
81.11+0.86
80.43++0.75
PubMed
84.57:10.32
85.57:10.40
86.28:0.67
85.53:10.48
84.99-10.39
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Secs. and 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
19

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please Sec. 4.3 and Appx. A.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
The detailed experimental settings are included in Appx. A.5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
20

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In Appx. A.4 we provide the sources of the datasets. We provide the key parts of the algorithm in the supplementary. The complete version will be made publicly available once it fully organized.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guides httpl///uiios//eeeuuuiisssss SoodeSubbmsssionoll115
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https 7/nips. wwwpc/ppwppppp......e.eee..opeee htaliiigggiis///eeee
sosoooooiisiiiiioiiicioccc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please Sec. 5 and Appx.
A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
21

---

Answer: [Yes]
Justification:
We report mean accuracy (%) + 95% confidence interval in Tabs
2 and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is discussed in Appx. A.5 and A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
22

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed and we cannot foresee any possible negative impacts, due to the highly technical nature of the task under consideration.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., dep.lymeln of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package dataset.
Guidelines:
23

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:
[NA]
Justification:
No crowdsourcing or research with human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing research with human subjects involved.

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "G6hA6jPsSZ", "reviewer": "Reviewer_k6XA", "summary": "Recently, some studies have been exploring new optimization methods that can replace backpropagation, and one of the popular methods is direct feedback alignment (DFA). This paper adapts DFA to graph neural networks (GNNs). It replaces the gradient with a randomly initialized matrix multiplying the prediction error, so that we do not need to use backpropagation to update the parameters. We can directly update the model parameters based on the prediction error. Besides, for semi-supervised graphs where the label is scarce, we only have prediction errors for a few nodes, so the parameter update is not that effective. To overcome this drawback, the paper assumes that nearby nodes have similar prediction errors, so it uses a smoothing technique to generate pseudo prediction errors for unlabeled nodes.", "questions": "(1) What is $W$ in Formula (7)?\n\n(2) In Section 4.3, the authors point out that W will converge to be similar to B. Since B is randomly initialized, if B is at a high point in the loss landscape, then will the optimization fail? Is the randomness of B important to the performance?\n\n(3) I might have not fully understood the intuition behind DFA. I am confused about why DFA performs better than backpropagation. I look forward to intuitive explanations.\n\n(4) Table 6 shows that all the non-backpropagation methods require more time than backpropagation. If I have correctly understood, DFA can directly calculate the parameter update for every layer from the loss, so it does not need to calculate the parameter update layer by layer as in backpropagation, then why does it need more time than backpropagation?", "limitations": "The authors have discussed the limitations. I encourage the authors to further explain what kinds of datasets and what kinds of GNN models DFA is most suitable for.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The studied problem is very interesting. The non-backpropagation optimization method is a very new field.\n\n(2) The method design is good. Though it is an adaptation of direct feedback alignment (DFA) which has been used for Euclidean data, since this area has been rarely studied, I think the adaptation is also a contribution.\n\n(3) There are some theories supporting the method.\n\n(4) Extensive experiments verify its effectiveness, and it can outperform backpropagation.", "weaknesses": "(1) In Formula (4), it ignores the activation function when calculating the parameter update. Since the activation function is ReLU, ignoring it might not cause a big problem. However, when we apply other activation functions, or when we use a complex GNN architecture, I am afraid this method might not work as well.\n\n(2) From Table 1, we can see that DFA achieves the best performance, while backpropagation is also very good on some datasets. I encourage the authors to add some discussions about when the proposed method is more useful than backpropagation, on what kinds of datasets, and what kinds of GNN models. Besides, the authors explained why DFA is better than previous non-backpropagation methods in Lines 254-274, but it would be better if the authors could explain why DFA is better than backpropagation.\n\n(3) The authors can consider introducing a little more about the limitations of backpropagation and the non-backpropagation optimization methods in the introduction or related work. Since most readers are not familiar with non-backpropagation optimization methods, it deserves more introduction.\n\n(4) The authors can improve some notations and make them clearer. For example, in Formula (5), the multiplication of $B^{(2)}$ and $B^{(1)}$ is written as $B^{(1)}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and positive assessment of our work. Below, we provide individual responses to your questions.\n\n$\\textbf{\\large Response to W1:} $\n\nWe take the GCN with ReLU activation function as an example to elaborate on our method since GCN is one of the most classic GNN models and ReLU is the most popular activation function. \n\nConsidering another activation function $\\sigma(\\cdot)$ with a deterministic derivative function $\\sigma^{'}(\\cdot)$, Eq.4 can be modified as: $\\delta\\textbf{X}^{(2)}=\\textbf{S}^{\\text{T}}\\textbf{EB}^{(2)}\\odot\\sigma'(\\textbf{H}^{(1)}\\textbf{W}^{(1)})$ and $\\delta\\textbf{X}^{(1)}$ is approximated as $(\\textbf{S}^{\\text{T}})^{2}\\textbf{EB}^{(1)}\\odot\\sigma'(\\textbf{H}^{(0)}\\textbf{W}^{(0)})$, which is similar to DFA in MLPs. The symbol $\\odot$ denotes element-wise multiplication, and $\\sigma'(\\textbf{H}^{(k)}\\textbf{W}^{(k)})$ is computed during forward propagation. As shown in **Tab.VIII** in the supplementary PDF, our method is well integrated with different activation functions (e.g., Sigmoid, Tanh, ELU and LeakyReLU) and derives consistently excellent performance. \n\nRegarding different GNN architectures, our method is portable to complex GNNs. Once $\\delta\\textbf{X}^{(k)}$ is derived in parallel, the parameters of each GNN layer can be computed by gradient descent. This adaptability ensures that our method remains effective while still eliminating the backpropagation of gradients among different GNN layers. The results in Tab.3 of Sec.5.4 demonstrate the excellent portability of our method.\n\n$\\textbf{\\large Response to W2 and Limitations:} $\n\nThanks for your suggestion, and we will add more discussion in the revised version. \n\nIt is observed from Tab.1 that our method has overall superior performance than BP. In particular, our method shows a much more significant accuracy improvement on relatively small datasets, such as Texas and Cornell. The reason mainly lies in that GNNs trained by BP often suffer from heavy overfitting due to the scarcity of supervision on small datasets, especially for semi-supervised tasks. By contrast, the pseudo error generated in our method not only is for gradient computation when adapting DFA to GNNs, but also serves as a data augmentation strategy, enhancing the robustness of GNNs (shown in Fig.3) and improving the ability for handling overfitting with scarce supervision. The gains derived from the pseudo-error strategy is not so significant on larger datasets (e.g., PubMed, Photo), but it still brings overall positive effects to our methods. Interestingly, BP does not benefit from the same pseudo-error generation strategy (please see **Tab.IX** in the supplementary PDF, and refer to $\\textbf{the response to reviewer buRu}$ for detailed discussion).\n\nAdditionally, from Tab.3, it is observed that our method shows greater superiority when the aggregation strategy of the GNN models is isotropic or the receptive field of aggregation is larger (e.g., GCN, SGC and ChebyNet). We attribute this phenomenon to the fact that isotropic and larger receptive field GNN layers can better utilize the direct feedback of errors. This is an interesting finding that deserves further exploration.\n\n$\\textbf{\\large Response to W3:}  $\n\nThanks for the suggestion. We will introduce more about the limitations of BP and the non-BP optimization methods in the introduction and related work in the revised version.\n\n$\\textbf{\\large Response to W4:} $\n\nThanks for your suggestion, we will provide notations for this variable in revision. \n\n$\\textbf{\\large Response to Q1:}  $\n\nSorry for the typo, the $\\textbf{W}$ in Eq.7 should be $\\textbf{Z}$, which is the variable to be optimized.\n\n$\\textbf{\\large Response to Q2}  $\n\nSorry for the confusion caused. The alignment of $\\textbf{W}$ and $\\textbf{B}$ means the directions of the two variables tend to converge to within 90 degrees of each other (as shown in Fig.2 (a)) rather than being numerically similar. In fact, the initialization of  $\\textbf{B}$ is not completely random, but follows a normal distribution that satisfies $\\mathbb{E}[\\textbf{B}\\textbf{B}^{T}]\\propto \\textbf{I}$. Under such constraints, as discussed in Sec. 4.3, $\\textbf{W}$ will converge to be alignment to $\\textbf{B}$ even starting from a high point.  We will add related description in the revised version for clarity.\n\n$\\textbf{\\large Response to Q3:}  $\n\nDFA is BP-free training method that has not been fully explored yet. Intuitively, DFA updates model parameters via gradient computation that is similar to BP. But differently, DFA obtains approximate gradients directly from the loss rather than backpropagation layer-by-layer. As discussed in Sections 1 and 2, this difference makes DFA more biologically plausible and promising in terms of generalization and parallelism. In previous studies, DFA usually exhibits similar or inferior performance in comparison with BP. The performance improvement of our method mainly derives from the pseudo-error generation strategy, which has been discussed in the response to W2.\n\n$\\textbf{\\large Response to Q4:}  $\n\nYes, DFA can directly calculate the parameter update for every layer from the loss, and it thus can improve training efficiency by parallelization.  However, to generalize DFA to graph data we introduce the pseudo-error generator as elaborated in Sec.4.2, and the main time cost is from the pseudo-error generation (Eqs.7, 8) and node filtering (Eq.9). According to Eqs.7 and 8, the time complexity of the two parts is $\\mathcal{O}(tcn^{2})$ and $\\mathcal{O}(cn)$ respectively, where $t, c, n$ denote the iteration epochs for error spreading, the number of categories and the number of nodes, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BePiI7cAal", "reviewer": "Reviewer_buRu", "summary": "The authors propose to apply the Direct Feedback Alignment (DFA) algorithm for backpropagation-free training to graph neural networks. The DFA algorithm is combined with a pseudo-error generation mechanism to provide additional error signals for missing targets in the setting of semi-supervised node classification. The experimental results with a 3-layer GCN model over 10 commonly adopted benchmarks show the proposed model performing generally better than backpropagation and other training algorithms. The ablation analysis confirms that improvements by the proposed DFA-GNN approach can be ascribed to the pseudo-error generation mechanism.", "questions": "1. Why is pseudo-error generation spreading, which must be done at teach gradient update iteration, instead of augmenting training labels via label spreading (plus the masking of eq. (9)), which would be only required once?\n2. How would the other training methods such as backpropagation perform with pseudo-error generation spreading?\n3. How many epochs are required for training convergence by DFA-GNN? How do they compare with the number of epochs required by the other training algorithms?", "limitations": "The limitations of this submission have not been properly discussed. For example, in lines 278-280, it is stated that the \"method has a higher training time consumption compared with BP, primarily due to the additional time needed for generating pseudo errors and filtering masks\". However, in the conclusions, it is claimed that *efficiency* is one of the advantages of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality**\nThe application of DFA to GNNs was already introduced in *J. Launay, I. Poli, F. Boniface, and F. Krzakala, “Direct feedback alignment scales to modern deep learning tasks and architectures,” NeurIPS 2020*. However, the addition of pseudo-error generation and the extensive experiments are entirely novel contributions.\n\n**Quality**\nThe submission is technically sound and its claims are well supported.\n\n**Clarity**\nThe paper is clearly written and well organized.\n\n**Significance**\nThe results of this paper can contribute to the research on addressing the training challenges of deep graph neural networks.", "weaknesses": "1. The computational cost per epoch of the proposed algorithm is 5 to 10 times larger than backpropagation.\n2. No comparison of the overall training costs, including the different number of epochs required for training convergence among the different methods.\n3. The ablation indicates that the accuracy improvements are ascribed to the pseudo-error propagation mechanism. As the latter could be applied also to e.g. backpropagation, such experimental evaluation would have provided further interesting insights.\n4. In the experiments the model is limited to 3 layers; deeper models would have provided more interesting insights, as depth increases the issues connected with gradient backpropagation. (Long-range graph benchmarks such as those from *V. P. Dwivedi et al., “Long Range Graph Benchmark,” in NeurIPS 2022 Track on Datasets and Benchmarks* could be adopted in this case.)\n5. Only node classification tasks are considered, no graph-level or edge-level tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments and positive assessment of our work. After carefully reviewing your feedback, below we provide answers to the comments you raised.\n\n$\\textbf{\\large Response to W1:}  $\n\nThe standalone direct feedback alignment formulated in Eqs.5 and 6 could be faster than BP. However, to generalize DFA to graph data we introduce the pseudo-error generator as elaborated in Sec.4.2, and the main time cost comes from the pseudo-error generation (Eqs.7, 8) and node filtering (Eq.9). According to Eqs.7 and 8, the time complexity of the two parts is $\\mathcal{O}(tcn^{2})$ and $\\mathcal{O}(cn)$, respectively, where $t, c, n$ denote the iteration epochs for error spreading, the number of categories and the number of nodes, respectively. Although the two introduced parts result in increased time cost, they are essential for generalizing DFA to GNNs. The ablation study in Tab.2 demonstrates the significant contribution of these two components. \n\n$\\textbf{\\large Response to W2 and Q3:}  $\n\nThanks for your comments. For our semi-supervised node classification task, we train all the methods for 1000 epochs, calculating the prediction accuracy for validation nodes after each epoch and saving the model that achieves the highest validation accuracy. We use the saved model to make predictions for the test nodes. This is a common experimental setting in semi-supervised graph learning. Thus, the overall training cost is simply one thousand times the numbers shown in Tab.6. \n\nTo better illustrate the training convergence of our DFA-GNN, we plot the training and validation accuracy of the proposed DFA-GNN and BP over training epochs on three datasets, as shown in **Fig. 4** in the supplementary PDF. In general, our method shows similar convergence as BP. For both BP and our method, the convergence of validation accuracy occurs much earlier than that of training accuracy due to overfitting. The validation convergent epoch of DFA-GNN is nearly the same as BP on Cora and CiteSeer (around 100 epochs), while it is 100 epochs later than BP on PubMed (around 200 epochs). Our method achieves better validation accuracy on all these datasets and suffers less from overfitting compared with BP.\n\nIn terms of training accuracy, our method exhibits slightly lower convergent value and lower convergence speed than BP.  The reason is that our method considers both the errors of labeled nodes and pseudo errors of unlabeled nodes as supervision signals. \nHowever, this does not prevent our method from achieving better validation results.\n\nWe do not include other non-BP methods in Fig. 4 because these methods (i.e., SF, FF, CaFo, etc.) follow a layer-by-layer update according to local loss. Therefore, the convergence behaviors varies across different layers, and it is hard to directly visualize them in a global view. \n\n$\\textbf{\\large Response to W3 and Q2:}  $\n\nThanks for your interesting suggestion. Although the pseudo-error generation process is essential for our method, it is also an optional choice for BP. We try integrating this component into BP, and the experimental results in **Tab.IX** in the supplementary PDF show that this component does not positively enhance BP in general. It only contributes to BP on Cora and CiteSeer while degrading the performance on the other six datasets. By contrast, it benefits DFA remarkably across all datasets and makes our method finally outperform BP in most scenarios. The different suitabilities of the pseudo-error generation to BP and DFA mainly lie in that DFA is more noise-tolerant than BP where the generated pseudo error always contains different degrees of noise, which has been discussed in some previous studies [a,b].\n\n[a]. R. Ohana et. al., Photonic Differential Privacy with Direct Feedback Alignment. NeurIPS 2021.\n\n[b]. J. Lee and D. Kifer. Differentially Private Deep Learning with Direct Feedback Alignment. arXiv 2020.\n\n$\\textbf{\\large Response to W4:}  $\n\nThanks for the suggestion. We plot the prediction accuracy with the number of layers increasing from 2 to 10, as shown in **Fig. 5** in the supplementary PDF. Both the BP and our method show a decrease in accuracy, but our method is less negatively affected and consistently outperforms BP. This superiority is derived from the fact that our method adopts direct feedback of errors for the update of each layer, reducing the problem of oversmoothing and other gradient issues caused by backward gradient accumulation. This observation indicates the potential of our method for deep GNN training.\n\n$\\textbf{\\large Response to W5:}  $\n\nThanks for your suggestion, we will explore the potential of DFA-GNN for graph-level and edge-level tasks in future work.\n\n$\\textbf{\\large Response to Q1:}  $\n\nThe pseudo error is essential for our method because the parameter update formula (Eqs.5 and 6) for each layer requires the error of all nodes. Spreading labels may not be well-suited to our update mechanism. Additionally, since the pseudo error depends on the prediction of each epoch, updating pseudo errors in real-time based on the results of forward propagation in each epoch is a more reasonable choice. Augmenting only once in an early epoch may lead to performance degradation of model.\n\n$\\textbf{\\large Response to Limitations:} $\n\nWe apologize for the confusing expression. The efficiency advantage of our method is mainly reflected in the comparison with other non-BP algorithms, showing significant time superiority over CaFo, FF, and SF, as presented in Tab.6. As BP has been extensively researched for decades and benefits from robust software and hardware support, our method has not yet achieved comparable time efficiency of BP. However, the parallel update strategy for each layer in our method makes it more conducive to parallel computing, which has the potential to further improve time efficiency.  We will revise the conclusion section of the article to make it more rigorous.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OLRudGcFJC", "reviewer": "Reviewer_mUbj", "summary": "The paper introduces DFA-GNN, a new framework designed to train Graph Neural Networks (GNNs) using Direct Feedback Alignment (DFA). Traditional methods like backpropagation (BP), though effective, have several limitations, including inefficiency, scalability issues, and a lack of biological plausibility. DFA-GNN aims to address these problems by utilizing a forward training mechanism specifically tailored for the unique challenges posed by graph data.", "questions": "Why the proposed method doesnot outperform on the computer and chameleon dataset.\n\nHow does DFA-GNN scale when applied to larger datasets e.g. flicker, Reddit, OBGN ?\n\nWhat is the time complexity of the proposed method?", "limitations": "No societal impacts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to follow.\n\nPerformance: DFA-GNN achieves better performance compared to other non-BP methods and even standard BP methods.\n\nRobustness: The framework exhibits resilience to noise and different types of attacks, ensuring its reliability in various scenarios.\n\nEfficiency: By enabling parallel gradient computation, DFA-GNN improves the efficiency of the training process.", "weaknesses": "The novelty of this paper is somewhat limited as it primarily extends the principles of DFA to GNNs. The core ideas are based on the foundations laid out in the Shalizi et al. (2013) paper.\n\nThe paper lacks a discussion on the computational complexity of the proposed algorithm.\n\nThere is a notable variation in the performance improvements observed across different datasets. For instance, DFA-GNN shows a significant accuracy improvement on the Texas and Cornell datasets, while the improvements on other datasets are relatively modest. The paper does not adequately explain the reasons behind this variation in performance.\n\nThe dataset used for experiments is quite small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the time taken to assess our work and for the valuable feedback. We address each point individually. “W/Q” numbers the weakness or question followed by our response.\n\n$\\textbf{\\large Response to W1:} $\n\nThanks for your comments. Our method builds on some ingenious previous studies, especially $\\textit{Nøkland A. Direct feedback alignment provides learning in deep neural networks, NeurIPS 2016}$. However, our work is the first to systematically study DFA's applicability to GNNs with extensive ablation studies and to provide update formulas and convergence proof for DFA in GNNs. The addition of pseudo-error generation is another highlight that is entirely novel for DFA. Our work demonstrates the exciting potential of non-BP training in graph deep learning, offering potential directions for addressing challenging issues such as oversquashing, topology distortion, and label scarcity, which warrant further exploration in the future.\n\n$\\textbf{\\large Response to W2 and Q3:} $\n\nThe time complexity of our method depends on the architecture of the GNN models. Given specific GNN layers with a defined forward propagation formula, the computational time cost of our method includes four parts: forward propagation (Eq.1), pseudo-error generating (Eqs.7, 8), node filtering (Eq.8) and direct feedback alignment (Eq.10). The complexity of forward propagation is the same as BP because our method makes no change in this process. Supposing the dimension of node features and hidden units is $d$, for the GCN model formulated in Eq.1 whose forward propagation formula for $i$-th layer is written as $\\textbf{X}^{(i)}=relu(\\textbf{S}\\textbf{X}^{(i-1)}\\textbf{W}^{(i-1)})$, the complexity is $\\mathcal{O}(n^{2}d+nd^{2})$, where $n$ denotes the number of nodes. Since $n\\gg d$ holds in most datasets, the complexity can be approximated as $\\mathcal{O}(n^{2}d)$. For direct feedback alignment, according to Eq.10, the time complexity for each layer is $\\mathcal{O}(n_{f}^{2}c+dn_{f}c+d^{2}c)$ and is further approximated as $\\mathcal{O}(n_{f}^{2}c)$ due to $n_{f}\\gg d \\gg c$, where $n_{f}$ is the number of nodes after filtering and $n_{f}<n$, $c$ the number of category. In comparison, BP for GCN model has a complexity of $\\mathcal{O}(n^{2}d+nd^{2}+d^{3})$ for each layer, which could be approximated as $\\mathcal{O}(n^{2}d)$. Although both the methods have a quadratic complexity related to the number of nodes, our method makes parallel update for each layer while BP has to run update layer by layer.\n\nThe other two components pseudo-error generator and node filter bring additional time cost to our method compared with BP. According to Eqs.7 and 8, the time complexity of the two parts is $\\mathcal{O}(tcn^{2})$ and $\\mathcal{O}(cn)$, respectively, where $t$ denotes the iteration epochs for error spreading. Although the two introduced parts result in increased time cost, they are essential for generalizing DFA to GNNs. The ablation study in Tab.2 demonstrates the significant contribution of these two components. Overall, our methods has the same quadratic time complexity as BP on GCN, which explains why our method has a consistent five to ten times consumption per epoch compared with BP, rather than an exponential difference, regardless of the dataset size.\n\n$\\textbf{\\large Response to W3 and Q1:} $\n\nThanks for the suggestion, and we will add related discussion in the revised version.\n\nOur method shows a much more significant accuracy improvement on relatively small datasets, such as Texas and Cornell. The reason mainly lies in that GNNs trained by BP often suffer from heavy overfitting due to the scarcity of supervision on small datasets, especially for semi-supervised tasks. By contrast, the pseudo error generated in our method not only is for gradient computation when adapting DFA to GNNs, but also serves as an data augmentation strategy, enhancing the robustness of GNNs (shown in Fig.3) and improving the ability for handling overfitting with scarce supervision. The gains derived from the pseudo-error strategy is not so significant on larger datasets (e.g., PubMed, Photo), but it still brings overall positive effects to our methods. \n\nFor Computer and Chameleon, our method does not outperform BP, however, the accuracy gap on Chameleon is quite small (only 0.09\\%). The testing accuracy depends on both the training algorithm and GNN model. Although our method underperforms BP for GCN on Computer, however, our method outperforms BP for SGC, GraphSage, APPNP and ChebyNet on this dataset as shown in Tab.3. Generally, our non-BP method has comparable results compared with BP and even show better performance on most experimental trails, which indicates our non-BP training method has the inspiring potential to become a substitute for BP in the field of graph deep learning.\n\n$\\textbf{\\large Response to W4 and Q2:} $\n\nOur method is well-suited for large datasets. When the graph scale is large, for memory saving we can use edge indices instead of an adjacency matrix to store the graph. For forward propagation (Eq.1), the space requirement for neighbor aggregation can be reduced from $\\mathcal{O}(n^{2}+nd)$ to $\\mathcal{O}(|\\mathcal{E}|+nd)$, where $|\\mathcal{E}|$ denotes the number of edges. For direct feedback alignment (Eq.10), as $(\\textbf{S}^{\\text{T}})^{k}\\hat{\\textbf{E}}$ is exactly the aggregation of errors for $k$ times, the space requirement can be reduced to $\\mathcal{O}(|\\mathcal{E}|+nc)$, without the need to store the $k$-th power of the adjacency matrix. Similarly, space requirement reduction can also be achieved in the node filtering process. \n\nAs for prediction accuracy, the experimental results on Flickr, Reddit, and ogbn-arxiv (shown in **Tab.X** in the supplementary PDF) demonstrate that our method obtains comparable results to BP (within 1% accuracy) and outperforms other non-BP methods with only a small memory cost (2049 MiB, 11675 MiB, 2277 MiB for the three datasets, respectively).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
$\definecolor{mybgcolor}{RGB}{249,242,244}$
$\definecolor{Flickr}{RGB}{140,27,19}$
$\definecolor{Reddit}{RGB}{140,27,19}$
$\definecolor{ogbn-arxiv}{RGB}{140,27,19}$
$\newcommand{\highlight}[2]{\colorbox{#1}{\textcolor{#2}{#2}}}$
 
Dear ACs, PCs and all reviewers,

We would like to express our gratitude to all the reviewers for their valuable comments and feedback on our work. All reviewers recognized the soundness, presentation and quality of our manuscript. We have responded to all the questions posed by each reviewer. In this summary, we aim to provide you with a clear understanding of the changes made during the rebuttal process.

$\textbf{\large 1. Inclusion of large-scale graph datasets for experiments}$

In response to the insightful suggestions from reviewer **mUbj**, we discuss the portability of our method to large graph datasets and conduct additional experiments on three new large-scale graph datasets the reviewer suggested. $\textbf{Table X}$ in the supplementary PDF displays the results on $\highlight{mybgcolor}{Flickr}$, $\highlight{mybgcolor}{Reddit}$ and $\highlight{mybgcolor}{ogbn-arxiv}$ in comparison with seven baseline algorithms. The results demonstrate our method outperforms all the non-BP methods and has comparable performance to BP (within 1\% accuracy), with only a little memory cost.

$\textbf{\large 2. More extensive ablation studies}$

- As requested by reviewer **buRu**,  we plot the training and validation accuracy of the proposed method and BP over training epochs on three datasets to demonstrate the convergence of our method as shown in $\textbf{Fig.4}$ in the supplementary PDF. The results show our method has similar convergence compared with BP. Additionally, we provide a detailed discussion of the figures to highlight the training effectiveness of our method.

- As requested by reviewer **k6XA**, to demonstrate that our method is feasible with different activation functions other than ReLU, we conduct experiments for our method with four different activation functions (i.e., Sigmoid, Tanh, ELU and LeakyReLU) as shown in $\textbf{Tab.VIII}$ in the supplementary PDF. The results demonstrate our method is well integrated with different activation functions and derives consistently excellent performance.

- As requested by reviewer **buRu**, we plot the prediction accuracy with the number of layers increasing from 2 to 10, as shown in $\textbf{Fig. 5}$ in the supplementary PDF. Both the BP and our method show a decrease in accuracy with the number of layers increasing due to oversmoothing. However, our method is less negatively affected and consistently outperforms BP. This superiority is derived from the fact that our method adopts direct feedback of errors for the update of each layer, reducing the problem of oversmoothing and other gradient issues caused by backward gradient accumulation. This observation indicates the potential of our method for deep GNN training.

- As requested by reviewer **buRu**, we integrate the proposed pseudo-error spreading component into BP. The experimental results from $\textbf{Tab.IX}$ in the supplementary PDF show that although this component contributes to DFA in our method, it does not positively enhance BP overall. It indicates that DFA is more robust against noise in generated pseudo error than BP, which is consistent with the analysis in some previous studies.

$\textbf{\large 3. Algorithm complexity analysis}$

In response to the insightful suggestions from reviewer **mUbj**, we provide a detailed analysis of the time complexity of our method. We present the time complexity of each component and demonstrate that our method has the same order of time complexity as BP. To apply our method to large datasets, we explain how to reduce the space requirement of graph information aggregation from $\mathcal{O}(n^{2}+nd)$ to $\mathcal{O}(|\mathcal{E}|+nd)$ through algorithm optimization.

$\textbf{\large 4. Highlighting the superiority compared with BP and non-BP methods}$

- As requested by reviewer **k6XA**, we provide a discussion about when our method is more useful than BP. We point out that the superiority of our method is more significant than BP when the dataset scale is relatively small,  the GNN model is isotropic, or the receptive field of aggregation is large.

- As requested by reviewer **k6XA**, we provide an intuitive explanation of why our method outperforms BP. We attribute the superior performance of our method to the data augmentation from our well-designed pseudo-error generator and the robustness of the DFA mechanism to noisy augmentation.

- As requested by reviewer **mUbj**, we provide an explanation of why our method shows a notable variation in the performance improvements observed across different datasets.

We want to express our gratitude to the reviewers for their valuable suggestions and would greatly appreciate any further questions they may have regarding our paper. If our responses have well-addressed your questions, we would kindly ask for your reconsideration of the scores.

Meta-review: This paper receives overall positive reviews and the authors have actively engaged in rebuttals. One of the reviewers was concerned/confused with the computational complexity, and the authors have been trying to actively address them during rebuttals. However, this reviewer did not provide a final confirmation as to if the concerns have been addressed. I did a check and believe the concern has been properly addressed. Thus an accept is recommended.
Decision: Accept (poster)