DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment

Authors: gongpei  zhao
tao  wang
congyan  lang
yi  jin
yidong  li
haibin  ling
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 03:41:51

Abstract: Graph neural networks (GNNs) are recognized for their strong performance across various applications, with the backpropagation (BP) algorithm playing a central role in the development of most GNN models. However, despite its effectiveness, BP has limitations that challenge its biological plausibility and affect the efficiency, scalability and parallelism of training neural networks for graph-based tasks. While several non-backpropagation (non-BP) training algorithms, such as the direct feedback alignment (DFA), have been successfully applied to fully-connected and convolutional network components for handling Euclidean data, directly adapting these non-BP frameworks to manage non-Euclidean graph data in GNN models presents significant challenges. These challenges primarily arise from the violation of the independent and identically distributed (i.i.d.) assumption in graph data and the difficulty in accessing prediction errors for all samples (nodes) within the graph. To overcome these obstacles, in this paper we propose DFA-GNN, a novel forward learning framework tailored for GNNs with a case study of semi-supervised learning. The proposed method breaks the limitations of BP by using a dedicated forward training mechanism. Specifically, DFA-GNN extends the principles of DFA to adapt to graph data and unique architecture of GNNs, which incorporates the information of graph topology into the feedback links to accommodate the non-Euclidean characteristics of graph data. Additionally, for semi-supervised graph learning tasks, we developed a pseudo error generator that spreads residual errors from training data to create a pseudo error for each unlabeled node. These pseudo errors are then utilized to train GNNs using DFA. Extensive experiments on 10 public benchmarks reveal that our learning framework outperforms not only previous non-BP methods but also the standard BP methods, and it exhibits excellent robustness against various types of noise and attacks.

Main paper: 
 introduction: 
 As a class of neural networks (NNs) specifically designed to process and learn from graph data, graph neural networks (GNNs) [Zhou et al., 2020 , Wu et al., 2020] have gained significant popularity in addressing graph analytical challenges. They have demonstrated outstanding success in various applications, including recommendation systems [Wu et al., 2022] , drug discovery [Xiong et al., 2021] and question answering [Yasunaga et al., 2021] . The impressive accomplishments of GNNs, as well as other neural network models, are largely attributed to the backpropagation (BP) algorithm [Hecht-Nielsen, 1992] , which has emerged as the standard technique for training deep neural networks. The backpropagation algorithm adjusts neural network weights based on the loss between the prediction and the ground truth, and allows the network to learn and improve over time. However, despite its effectiveness, BP draws concerns on its biological plausibility for two main reasons [Hinton, 2022 , Lillicrap et al., 2016] : (1) it uses the same weights in reverse order for both feedforward and feedback paths, creating the weight symmetry problem [Lillicrap et al., 2016] ; and (2) its parameter updating relies on the activity of all downstream layers, leading to the update locking problem [Dellaferrera and Kreiman, 2022] . These limitations may as well impact the efficiency, scalability and parallel processing capabilities of neural network training.To address these limitations, direct feedback alignment (DFA) [Nøkland, 2016] offers an effective alternative to BP by training neural networks through a single forward pass. DFA uses fixed random feedback connections to project output errors directly onto hidden neurons, allowing for parallel gradient computation and eliminating the need for sequential backward error propagation. While demonstrated a limited accuracy penalty compared with BP, DFA aligns with brain-like learning mechanisms through its use of global error modulation and local synaptic activity, making it a notable non-BP method applicable in areas such as image classification [Zhao et al., 2023] and privacy protection [Ohana et al., 2021] .Directly applying DFA to GNNs, however, faces two challenges: (1) graph data often violates the independent and identically distributed (i.i.d.) assumption and thus ties the supervision gradients with the graph structure, making the straightforward error projection of DFA inadequate; and (2) DFA requires the prediction errors of all the input samples, while for graph data, especially under the semi-supervised setting, samples (nodes) without ground truth meet problems for the error calculation, complicating the deployment of DFA to GNNs.To tackle these challenges, in this paper we propose DFA-GNN, a non-BP learning framework tailored for graph neural networks. Our primary contribution is to improve and extend DFA to graph neural networks. Specifically, we redesign the random feedback strategy for graph data to make the DFA portable to GNNs. The information from the graph topology, in the form of an adjacency matrix, is incorporated into the feedback links to accommodate the non-Euclidean characteristics of graph data. We take graph convolutional network (GCN) [Kipf and Welling, 2016] as a case study, and derive the specific formula for updating parameters in each GCN layer. Furthermore, for the semi-supervised graph learning task, we develop a novel pseudo error generator that spreads residual errors from training data to generate a pseudo error for each unlabeled node. Such pseudo errors are then used for the training of graph neural networks by DFA.In summary, our proposed learning procedure for GNNs contributes in three significant folds:• We introduce DFA-GNN, a non-BP training algorithm that extends DFA to GNN architectures. It offers a more biologically plausible alternative to traditional BP methods. • For semi-supervised graph learning tasks, we develop a novel pseudo error generator that propagates residual errors from the training data to create pseudo errors for unlabeled nodes.• We prove the convergence of our DFA-GNN, and validate its effectiveness on 10 benchmarks. The experimental results demonstrate the superiority of our DFA-GNN against both traditional BP and the state-of-the-art non-BP approaches. 
related work: 
 The biological implausibility of BP mainly lies in weight transport and update locking issues. feedback alignment (FA) [Lillicrap et al., 2016] addresses the weight transport by using fixed random weights to convey error gradients. Building on this, direct feedback alignment (DFA) [Nøkland, 2016] , direct random target projection (DRTP) [Frenkel et al., 2021] and PEPITA [Dellaferrera and Kreiman, 2022] further tackle the update locking problem with non-BP update methods. Prompted by the recent critiques of Hinton [2022] , the forward-forward (FF) algorithm emerges as a more neurophysiologically aligned alternative, using dual forward passes with positive and negative data to simplify the training process and accommodate non-differentiable elements. The recently proposed cascaded forward algorithm (CaFo) [Zhao et al., 2023] attaches a class predictor to each layer, where only the layer-wise predictors are locally trained, with each neural block being randomly initialized and remaining fixed throughout.Our work aims to push the frontier of the non-BP training algorithm for GNNs, which is a field still in its infancy. A remarkable recent work along the line is FORWARDGNN [Park et al., 2023] inspired by the forward-forward algorithm. FORWARDGNN avoids the constraints imposed by BP via an effective layer-wise local forward training. It trains GNNs using a single forward pass with the assistant of a data augmentation strategy. The augmented graph structure integrates virtual nodes linked only to labeled nodes, leaving the local topology of unlabeled nodes unchanged. The augmentation strategy makes it possible to operate without generating negative inputs. Despite of its advantages, FORWARDGNN still suffers from the greed-based training strategy, and thus results in inferiority in prediction performance in comparison with traditional BP algorithm.In contrast, our method does not necessitate data augmentation for graph data; instead, it directly utilizes the discrepancy between predictions and actual ground truth to update each layer. Our approach directly outputs the prediction of the multi-class distribution, eliminating the need to calculate the goodness between unlabeled nodes and virtual nodes. As a result, our method offers convenience for multi-class prediction tasks and gains improvements in prediction performance.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please see Secs. 4 and 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 An attributed relational graph of n nodes can be represented by G = (V, E, X), whereV = {v 1 , v 2 , • • • , v n } represents the set of n nodes, and E = {e ij } signifies the set of edges. X = {x T 1 ; x T 2 ; • • • ; x T n } ∈ R n×dis the attribute set for all nodes, with x i being the d-dimensional attribute vector for node v i . The adjacency matrix A = {a ij } ∈ R n×n denotes the topological structure of graph G, where a ij > 0 if there exists an edge e ij between nodes v i and v j and a ij = 0 otherwise.For semi-supervised node classification, the node set V can be split into a labeled node set V L ⊂ V with attributes X L ⊂ X and an unlabeled one V U = V/V L with attributes X U = X/X L . 2 We assume that each node belongs to exactly one class, and denote y L = {y i } as the ground-truth labels of node set V L where y i denotes the class label of node v i ∈ V L .The objective of semi-supervised node classification is to train a classifier using the graph and the known labels y L , and then apply this classifier to predict the labels for the unlabeled nodes v U . Define a classifier f θ : (ỹ L , ỹU ) ← f θ (X, A, y L ), where θ is the parameters of model. ỹL and ỹU are the predicted labels of nodes v L and v U respectively. Generally, the goal is to make the predicted labels ỹL align as closely as possible with the actual ground-truth labels y L in favor of:θ * = arg min θ d(ỹ L , y L ) = arg min θ d(f θ (X, A, y L ), y L ), where d(•, •) represents a measure of some type of distance between two sets of labels.While BP relies on symmetric weights for error propagation to hidden layers, there is evidence suggesting that symmetrical weight distribution may not be crucial for learning. For example, the study of feedback alignment (FA) shows that learning can still occur when errors are back propagated using randomly fixed weights. Direct feedback alignment (DFA) advances in this direction by directly transmitting output errors to each hidden layer through fixed linear feedback links. Specifically, for an L layer network, feedback matrices B (l) ∈ R n L ×n l are employed to replace the derivatives ∂x (L) /∂x (l) of output neurons with respect to hidden neurons in the l-th layer. The approximate gradient δW (l) for the weights of the l-th hidden layer is then computed as:δW (l) = ∂L ∂x (L) B (l+1) ∂x (l+1) ∂W (l), where x (l) represents the latent representation of a sample at the l-th layer, and L the loss value. In DFA, the feedback matrices assigned to hidden layers are randomly selected and remain unchanged throughout the training process. The effectiveness of DFA hinges on the alignment between forward weights and feedback matrices, leading to a congruence between the estimated and the actual gradient. When the angle between these gradients remains below 90 degrees, the update direction points to a downward trajectory. DFA has been successfully implemented in popular deep learning architectures such as fully-connected neural network (FC) [Zhang et al., 2017] and convolutional neural network (CNN) [Li et al., 2021a] . However, extending DFA to graph neural networks remains unexplored.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Please see Sec. 4.3 and Appx. A.3.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
proposed dfa-gnn: 
 As depicted in Fig. 1 , our DFA-GNN aims to train GNN models in non-BP framework by extending the DFA algorithm. Different from the original DFA algorithm designed on FC for Euclidean data, two key issues should be solved when extending it to GNN for graph data: (1) the original random feedback operations need to be reformulated to handle the dependence between samples (nodes); and(2) high-quality pseudo errors for test samples are required as they are not isolated from the training procedure. To this end, in Sec. 4.1 we redesign the random feedback strategy specified for graph data, and in Sec. 4.2 we develop a novel pseudo error generator for semi-supervised graph learning tasks. Finally, we provide deep insight of our DFA-GCN about its convergence and optimization in Sec. 4.3.The training process of traditional BP for GNN is listed in Algo. 1 of Appx. A.1, and it uses both a forward propagation and a backward one in each epoch. Generally, different GNN models may differ in the operations of aggregation and combination, and we provide a typical implementation in Algo. 1. We take graph convolutional network (GCN) [Kipf and Welling, 2016] , one of the most classic and successful GNN models, as a case study to integrate DFA for graph learning.For illustrative purpose, we consider a three-layer GCN model with ReLU for hidden activation and sigmoid for output activation. The forward propagation process could be written as:EQUATIONwhere S = D-1 2 Ã D-1 2 , Ã = A + I is the adjacency matrix of graph G after adding self loop. D = diag( Ã) the diagonal matrix of Ã, W (l-1) a trainable weight matrix of the l-th layer and σ a non-linear activation function. X (l) ∈ R n×d denotes the latent representation matrix of the l-th layer and X (0) = X. If we choose sigmoid activation function in the output layer and a binary cross-entropy (BCE) loss function, the loss J for a graph with n nodes and the gradient E at the output layer are calculated as:EQUATIONEQUATIONwhere Ỹ and Y ∈ R n×c are respectively prediction and one-hot ground truth; m and k are respectively sample index and output unit. It is important to highlight that E represents the exact error between the prediction and the ground truth. For GCN, the gradients for hidden layers are calculated according to Algo. 1 as:δX (2) = S T δH (2) = S T EW (2)T , δX (1) = S T δH (1) = S T δX (2) W (1)T .As demonstrated by the work of FF [Lillicrap et al., 2016] and DFA [Nøkland, 2016] , learning can be effective when errors are back propagated using randomly fixed weights. Similarly, we establish parallel direct feedback links for each layer. We approximate the update directions for the hidden layers as follows:EQUATIONwhere B (i) is a fixed random weight matrix with appropriate dimension. δX (1) can be then written as:EQUATIONand the weight updates for all layers are calculated as:EQUATIONThe above derivations can be easily extended to GCNs with more layers, as done in our experiments.In the computation of gradient (Eq. 6), the labels of all nodes are required, but not all nodes are labeled in the semi-supervised learning task. To address this issue, we introduce a mechanism to generate errors, assigning a pseudo error to each unlabeled node. The underlying principle is the expectation that errors in initial predictions are likely to propagate along the graph edges. That is, an error at a given node v suggests a higher likelihood of similar errors to its the neighbor nodes. This concept of error propagation across the graph is supported by previous studies [Jia and Benson, 2020] .Our approach draws inspiration from the strategy of residual propagation used in node regression tasks, and more broadly, from the frameworks of generalized least squares and correlated error models [Shalizi, 2013] .In semi-supervised graph learning, the error matrix E = {e T 1 ; e T 2 ; • • • ; e T n } ∈ R n×c as described in Eq. 3 is modified as the residual on the training nodes, while being set to zero for all other nodes. This adjustment entails initializing e i as a zero vector for all nodes v i ∈ V U .The residuals in the rows of E for the training nodes are zero only when the forward process achieves perfect predictions. We utilize the label spreading technique [Zhou et al., 2003] to smooth the error with the goal of optimizing the following objective:EQUATIONwhere tr(•) denotes the trace of a matrix. The first term enhances the smoothness of the error estimation throughout the graph, while the second term ensures that the final solution stays consistent with the initial error estimate E. Following the optimization methodology in Zhou et al. [2003] and Huang et al. [2021] , the solution to Eq. 7 can be obtained through iterative processingEQUATION)This process represents the diffusion of error, and such propagation is demonstrably appropriate within the context of regression problems under a Gaussian assumption. Nevertheless, for classification tasks like ours, the smoothed error Z * might not align with the correct scale. Typically,∥Z (t+1) ∥ 2 ≤ (1 -α)∥E∥ 2 + α∥S∥ 2 ∥E (t) ∥ 2 = (1 -α)∥E∥ 2 + α∥Z (t) ∥ 2 .Starting with Z (0) = E, we find that ∥Z (t) ∥ 2 ≤ ∥E∥ 2 , indicating a need to adjust the scale of residuals adaptively. The aim is to match the magnitude of error in Z * to that in E as closely as possible. Given that we only have accurate error information for labeled nodes, we use the average error across these nodes to estimate the appropriate scale. Specifically, with e i , z * i ∈ R c representing the i-th row of E and Z * respectively, the adjusted error for an unlabeled node j is calculated as:êj = η/∥z * j ∥ 1 • z * j , in which η = 1 |V L | vi∈V L ∥e i ∥ 1 .Give the rescaled pseudo error ê for each unlabeled node, we define Ê ∈ R n×c , where the i-th row is set to e T i for nodes v i ∈ V L and to êT i for other nodes. The matrix Ê can then be directly utilized in Eqs. 5 and 6 for GCN training. Nevertheless, not all rescaled error of unlabeled nodes are accurate and useful. To address this, a mask is implemented to filter out these nodes. Define Ŷ = Ỹ -Ê as the corrected prediction. For labeled nodes, this corrected prediction equals to the one-hot ground truth. We introduce a mask vector p ∈ R n to facilitate this process. Formally, the setup is as follows:EQUATIONwhere p i is the i-th element (1 ≤ i ≤ n) of the vector p, ϵ a manually set threshold for controlling filtering and count(•) a counting function. Eq. 9 focuses on retaining only those well predicted nodes, characterized by a single category being identified as positive and the rest as negative. With the mask, the filtering operations could be performed on Ê and S by row through the mask. The weight updates in Eq. 6 are modified as:EQUATIONwhereS (k) f , Êf , H (k) frepresent the row filtering of S k , Ê and H (k) respectively, according to the mask p. As (S T ) k Ê exactly denotes the accumulation of errors related to the k-hop neighbors of each node, the difference between Eq. 6 and Eq. 10 lies in the partial sampling of neighbors to approximate the update directions, as compared with using all neighbors. Given that only poorly predicted nodes are excluded and they constitute a small fraction, the update directions in both approaches generally remain aligned. The formal description of our algorithm is provided in Appx. A.2.DFA-GNN provides a non-BP training procedure for GNN. For GCN we provide a detailed analysis on how such an asymmetric feedback path introduced in Sec. 4.1 can provide learning by aligning the gradients of backward propagation and forward propagation with its own. Nøkland [2016] has originally proved the conclusion for fully-connected layer architectures. We can show that this conclusion is equally valid for the GCN architecture, and provide the detailed proof in Appx. A.3.Experiments in Fig. 2 (a) validate the dynamic process of alignment between B and W during training, and the weight alignment leads to gradient alignment because for weight alignment of DFA:EQUATIONwhere the symbol ∝ represents a positive scalar multiple relationship. As gradient alignment requires δX(l) DFA ∝ δX (l)BP , i.e., (S T ) L-l EB (l) ∝ S T δX (l+1) W (l)T , the weight alignment directly implies gradient alignment if the feedback matrices are assumed right-orthogonal, i.e., BB T = I. This assumption holds if the feedback matrices elements are sampled i.i.d. from a Gaussian distribution since E[BB T ] ∝ I, hence Eq. 11 induces the weights, by the orthogonality condition, to cancel out by pairs of two:EQUATION)The alignments of weights and gradients make our method trained with DFA tend to converge to a specific region within the landscape, guided by the structure of the feedback matrices, while the optimization paths trained with BP according to stochastic gradient descent often exhibit divergent within the loss landscape, as shown in Fig. 2 (d) .For deeper understanding of the training mechanism in our method, we divide the entire training process into three stages: Stage 1 (train layers 1 and 2 while freezing layer 3), Stage 2 (freeze layers 1 and 2 while training layer 3) and Stage 3 (train layers 1 and 2 while freezing layer 3). The results in Figs. 2(b, c ) show a strong correlation between weight alignment and the fitting degree of the model, as indicated by the loss. Notably, even though our method updates parameters of each layer in parallel, the effective update follows a backward-to-forward manner. As shown in Fig. 2 (b), when the parameters of layer l are not effectively learned, updating the preceding layers does not enhance fitting ability of the model. This behavior contrasts with the characteristics of traditional BP, which indicates that the alignment of weights and gradients also adhere to a backward-to-forward sequence. 5 ExperimentsQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The detailed experimental settings are included in Appx. A.5. 
comparison with baseline training algorithms: 
 We evaluate our method on 10 benchmark datasets across various domains and compare it with the BP [Rumelhart et al., 1986] , PEPITA [Dellaferrera and Kreiman, 2022] , two versions of the FF (abbr. FF+LA, FF+VN) [Hinton, 2022 , Park et al., 2023] , two versions of the CaFo (abbr. CaFo+MSE, CaFo+CE) [Zhao et al., 2023] and the FORWARDGNN-SF (abbr. SF) [Park et al., 2023] . Detailed datasets and experimental setup information can be found in Appx. A.4 and Appx. A.5, respectively. The comparative analysis of various algorithms on benchmark datasets is summarized in Tab. 1. While the non-BP methods such as PEPITA, CaFo and FF have proven effective for architectures involving fully-connected and convolutional layers with Euclidean data, they exhibit weaker performance with non-Euclidean graph data. This is primarily due to the unique challenges posed by graph data.Firstly, in typical fully-connected and convolutional layers, shallow layers capture coarse-grained features while deep layers handle fine-grained features, with these two types of features usually being highly correlated. However, in GNNs, different layers aggregate information from varying neighborhood ranges, resulting in layers that often contain uncorrelated information. Particularly in heterophilic graphs, the information extracted by deep and shallow layers may be entirely unrelated The training times for each method are shown in Appx. A.6. Our approach demonstrates a general time advantage over CaFo, FF and SF. Our method includes a forward propagation and a parameter update where all layers execute in parallel during each iteration, which offers greater parallelism compared with layer-wise update methods like CaFo, FF and SF. Our method has a higher training time consumption compared with BP, primarily due to the additional time needed for generating pseudo errors and filtering masks, as discussed in Sec. 4.2.We ablate the proposed method to study the importance of designs in DFA-GNN. Two designs are studied including the pseudo error generator (abbr. EG) and the node filter (abbr. NF). For trials with EG, the pseudo error generator is applied according to Eq. 7 to assign a pseudo error for each unlabeled node. It worth noting that when ER is removed from the method, only the errors of labeled nodes are used for the updating of parameters according to Eq. 6. For trials with NF, the mask calculated as Eq. 9 is introduced in training process and the parameters are updated according to Eq. 10. The ablation results are included in Tab. 2. We note that even the most naive version of DFA-GNN elaborated in Sec. 4.1 achieves comparable results in comparison with BP. Furthermore, both the two designs introduced in Sec. 4.2 contribute to our training framework, making significant enhancement to DFA-GNN to outperform BP method.To better illustrate the training convergence of DFA-GNN, we plot the training and validation accuracy of the proposed DFA-GNN and BP over training epochs on three datasets, as shown in Fig. 3 . In general, our method shows similar convergence to BP. For both BP and our method, the convergence of validation accuracy occurs much earlier than that of training accuracy due to overfitting. The validation convergent epoch of DFA-GNN is nearly the same as BP on Cora and CiteSeer (around 100 epochs), while it is 100 epochs later than BP on PubMed (around 200 epochs). Our method achieves better validation accuracy on all these datasets and suffers less from overfitting compared with BP.In terms of training accuracy, the convergence of our method is slightly slower than BP because the update direction of our method is not exactly opposite to the gradient direction but maintains a small angle. Since our method considers both the errors of labeled nodes and pseudo errors of unlabeled nodes as supervision information, which is different from BP that only uses the loss of labeled nodes for supervision, the convergent value of training accuracy for our method is slightly lower than BP. However, this does not affect our method achieving better validation results.We focus on over-smoothing and random structural attack, two common sources of perturbation that reduce GNN performance. Over-smoothing [Keriven, 2022] is a problematic issue in GNNs, stemming from the aggregation mechanism within GNNs, which hinders the expansion of GNN models to a large number of layers. We test the robustness of our method against over-smoothing in Fig. 4 (a). Our method demonstrates greater robustness compared with BP, particularly when dealing with architectures that have a large number of layers. This enhanced robustness is due to the fact that the global loss directly contributes to the optimization of each individual layer. SF is less effected by over-smoothing due to its layer-wise optimization with local loss. However, its performance largely depends on shallow layers and the best performance on each dataset is inferior to ours.For random structural attack [Li et al., 2021b] , three random attack types are implemented on the original graph topology with a perturbation rate λ from 0.2 to 0.8. To better compare the robustness of different methods, we employ a more challenging experimental setup with a sparse supervision pattern, where each class has only 20 labeled nodes. The node split follows Kipf and Welling [2016] .The detailed operating description for attacking type is summarized as follows: (1) add: randomly add λ|E| edges to the original graph for a denser topology;(2) remove: randomly remove λ|E| edges from the original graph for a sparser topology; and (3) flip: randomly choose λ|E| node pairs, and remove the edge if there exists one between the pair, and otherwise add an edge to connect the pair.Our method is less sensitive to all types of perturbations as shown in Figs. 4 (b-d ), consistently outperforms two approaches on each trial, and exhibits exciting robustness even under a high perturbation rate. As the pseudo error generator derives pseudo error for each unlabeled node to update each layer, this supervision generation mechanism helps enhance robustness of the model against noise and attacks. Interestingly, a comparison of results across three different types of attacks shows that removing edges has the least adverse effect, suggesting that injecting incorrect topological information could be more detrimental to GNN performance than losing valuable original topology.Our method is well-suited for large datasets. When the graph scale is large, we can use edge indices instead of an adjacency matrix to store the graph. For forward propagation (Eq. 1), the complexity For direct feedback alignment (Eq. 10), as (S T ) k Ê is exactly the aggregation of errors for k times, the time and space complexity can be reduced to O(kc|E|), without the need to calculate the k-th power of the adjacency matrix. Similarly, complexity reduction can also be achieved in the node filtering process. The experimental results on the Flickr [Zeng et al., 2020] , Reddit [Hamilton et al., 2017] , and ogbn-arxiv [Hu et al., 2020] datasets, as presented in Tab. 3, demonstrate that our method is effective on large-scale datasets, delivering strong performance. Our method achieves results comparable to BP while surpassing other non-BP methods, all with a small memory footprint (2043 MiB for Flickr, 11675 MiB for Reddit, and 2277 MiB for ogbn-arxiv). This indicates that our method not only scales well to large graphs but also maintains efficiency in terms of space usage.We apply our training algorithm to five popular GNN models [Wu et al., 2019 , Veličković et al., 2018 , Hamilton et al., 2017 , Gasteiger et al., 2018 , Defferrard et al., 2016] and report the mean accuracy across ten random splits in Tab. 4. Each of the testing models is modified to fit our framework. Specifically, for SGC, which only has a single learnable linear output layer, the training of our framework involves no direct feedback but only incorporates the pseudo error generator and node filter.For GraphSage, we utilize a mean-aggregator for message aggregation. Our observations indicate that our method can be effectively ported to mainstream GNN models. All test models integrated with our algorithm work well and surpass the performance of traditional BP in most scenarios. It demonstrates the effectiveness of our method across various GNN models and underscores its excellent portability and potential generalization ability to other innovative GNN models. 
conclusion: 
 In this paper, we investigate the potential of non-backpropagation training methods within the context of graph learning. We adapt the direct feedback alignment algorithm for training graph neural networks on graph data and introduce DFA-GNN. This new approach incorporates a meticulously designed random feedback strategy, a pseudo error generator, and a node filter to effectively spread residual errors. Through mathematical formulations, we demonstrate that our method can align with backpropagation in terms of parameter update gradients and perform effective training. Extensive experiments on real-world datasets confirm the effectiveness, efficiency, robustness and versatility of our proposed forward graph learning framework. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The information on the computer resources is discussed in Appx. A.5 and A.6.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes, we do.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have cited the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Here we provide some implementation details of our methods to help readers further understand the algorithms and experiments in this paper.A for l in [0, 1, ..., L -1] do {Forward propagation} 3:for v in v do 4:h (l) v = u∈ Ñ (v) e vu • x (l) u ; {Aggregation} 5:end for 6:X (l+1) = σ(H (l) W (l) ); {Combination} 7:end for8: loss = Loss_computing(X (L) , y L ) 9:for l in [L -1, L -2, ..., 0] do {Backward propagation} 10: l+1) ;{Combination} 11:δW (l) = H (l)T δX (δH (l) = δX (l+1) W (l)T ;{Combination} 12: for v in v do 13: δx (l) v = u∈ Ñ (v) e uv • δh (l) u ; {Aggregation} 14:end for 15:end for 16:W eight_updating(epoch, W (0) , δW (0) , ..., W (L-1) , δW (L-1) ); 17: end for 18: ỹ = arg max Ỹ = arg max X (L) .Algorithm 2 GCN Training Process Input: G, X, W, y L , max_epoch. Output: ỹ, W.1: Initialize {B (1) , B (2) , ..., B (L-1) }; 2: for epoch in [0, 1, ..., max_epoch-1] do 3:Compute the prediction Ỹ given by Eq. 1; {Forward propagation} 4:Compute the error E using Ỹ, y L for labeled nodes according to Eq. 3; 5:Compute pseudo error Z * for unlabeled nodes by Eqs. 8, and rescale it to get rescaled error Ê; 6:Compute mask vector p given by Eq. 9; 7:for l in [0, 1, ..., L -1] do 8: if l == L -1 then 9:Compute δW (l) using Ê, p according to Eq. 10; {Direct feedback} 10: else 11:Compute δW (l) using Ê, p and B (l+1) according to Eq. 10; {Direct feedback} W eight_updating(epoch, W (0) , δW (0) , ..., W (L-1) , δW (L-1) ); 15: end for 16: ỹ = arg max Ỹ. Sec. 4.3 Theorem. For a GCN model with two hidden layers k and k + 1 where k connects to k + 1 in sequence, we have x (k+1) = σ(a (k+1) ) and a (k+1) = g(Wx (k) ), where σ is the activation function and g(•) the aggregation operation in Algo. 1. Let the layers be updated according to the non-zero update directions δx (k) and δx (k+1) where δx (k) ∥δx (k) ∥ and δx (k+1) ∥δx (k+1) ∥ are constant for each data point. The negative update directions will minimize the following layer-wise criterion:EQUATION)Minimizing P will lead to an increase in the gradient, thereby enhancing the alignment criterion:EQUATION)where k+1) ).EQUATION(15)g ′ (•) is the aggregation of gradients in Algo. 1, (line 13). If Q (k) > 0, then -δx (k) serves as a direction of descent to minimize P (k+1) .Proof. Let i be the any of the layers k or k + 1, and prescribed update -δx (i) is the steepest descent direction to minimize P (i) . Since any partial derivative of δx (i) ∥δx (i) ∥ is zero, we have:-EQUATION)whereα (i) = 1 ∥δx (i) ∥ > 0. As δa (i) = ∂x (i) ∂a (i) δx (i) = δx (i) ⊙ σ ′ (a (i)), the gradients maximizing Q (k) and Q (k+1) are:EQUATION)When ignoring the magnitude of the gradients we have∂Q ∂W ≈ ∂Q (k) ∂W ≈ ∂Q (k+1) ∂W . If x (i) is projected onto δx (i) we have x (i) = x (i)T δx (i) ∥δx (i) ∥ 2 δx (i) + x (i)res . The prescribed update for W is:δW = -δx (k+1) ∂x (k+1) ∂W = -g ′ (δx (k+1) ⊙ σ ′ (a (k+1) ))x (k)T = -g ′ (δa (k+1) )x (k)T = -g ′ (δa (k+1) )(α (k) P (k) δx (k) + x (k) res ) T = -α (k) P (k) g ′ (δa (k+1) )δx (k)T -g ′ (δa (k+1) )x (k)T res = -P (k) ∂Q (k) ∂W -g ′ (δa (k+1) )x (k)T res (18)It is obvious that Q (k) and Q (k+1) can be maximized when the component of ∂Q (k) ∂W in δW is maximized by minimizing P (k) . The gradient to minimize P (k) is the prescribed update -δx (k) . The angle between δx (k) and the gradient of BP c (k) is within 90• if Q (k) > 0 because the cosine of the two vector is Q (k)∥c (k) ∥ > 0, and it also indicates that c (k) is nonzero and therefore in a descending trend. Consequently, δx (k) will be oriented towards a descending direction since any vector that lies within 90 • of the steepest descent direction will similarly point downwards, in other words, for GCN, a broad spectrum of asymmetric feedback paths can offer a descending gradient direction for a hidden layer as long as Q (i) > 0.From Eq. 5, it is obvious one advantage of our method is that δx (i) is non-zero for any non-zero error e, as a randomly generated matrix B (i) is highly likely to be of full rank. Ensuring δx (i) is non-zero is crucial for achieving Q (i) > 0. Maintaining static feedback across training helps preserve the characteristic, and also simplifies the process of maximizing Q (i) due to the more consistent direction to δx (i) . Our method shows better biological plausibility in GNN training, which introduces asymmetric feedback paths to take place of BP, not only solving the weight transport problem and partially solving the update locking problem, but also releasing the requirement to store neural activations and accumulated gradients for backward propagation.We evaluate our method on 10 benchmark datasets across domains: citation networks (Cora, Cite-Seer, PubMed) [Sen et al., 2008 , Yang et al., 2016] , Amazon co-purchase graph (Photo, Computer) [McAuley et al., 2015] , Wikipedia graphs (Chameleon, Squirrel) [Rozemberczki et al., 2021] , actor co-occurrence graph (Actor) [Pei et al., 2019] and webpage graphs from WebKB (Texas, Cornell) [Pei et al., 2019] . The datasets adopted are representative which describe diverse reallife scenarios. Some of them are highly homophilic while others are heterophilic. Note that for Chameleon and Squirrel, we use the filtered version from Platonov et al. [2022] , as the original version from [Rozemberczki et al., 2021] may contain duplicated nodes. The detailed statistics of the datasets are summarized in Tab. 5. We compute the node homophily for each dataset using the method proposed by Pei et al. [2019] , referred to as the homophily value. Sen et al., 2008] are three classic homophilic citation networks. In these networks, nodes correspond to academic papers, and edges signify the citation links between papers. The node features are derived from bag-of-word representations of the papers, and the labels categorize each paper into specific research topics.• Computer and Photo [McAuley et al., 2015] are segments of the Amazon co-purchase graph, where nodes represent goods, edges indicate that two goods are frequently bought together, node features are bag-of-words encoded product reviews, and class labels are given by the product category.• Chameleon and Squirrel [Rozemberczki et al., 2021] are two heterophilic networks derived from Wikipedia. In these networks, nodes represent Wikipedia web pages, and edges correspond to hyperlinks between these pages. The features are comprised of informative nouns extracted from the Wikipedia content, while the labels reflect the average traffic of each web page.• Actor [Pei et al., 2019] is a heterophilic actor co-occurrence network where nodes represent actors, and edges signify that two actors have appeared together in the same movie. The features are derived from keywords found on the actors' Wikipedia pages, while the labels consist of significant words associated with each actor. 7.56e -3 8.73e -3 7.61e -1 3.14e -1 2.83e -1 5.49e -2 5.66e -2 CiteSeer 1.06e -2 1.11e -2 7.68e -1 2.59e -1 2.61e -1 6.88e -2 5.68e -2 PubMed 1.07e -2 1.07e -2 8.24e -1 6.94e -1 7.61e -1 5.34e -1 6.76e -2 Photo 8.74e -3 1.03e -2 7.98e -1 2.11 1.91 4.87e -1 5.81e -2 Computer 1.08e -2 1.05e -2 7.80e -1 4.82 4.14 7.61e -1 6.29e -2 Texas 6.13e -3 1.07e -2 8.05e -1 1.47e -1 1.56e -1 6.88e -2 5.60e -2 Cornell 5.42e -3 1.06e -2 7.46e -1 1.51e -1 1.24e -1 3.59e -2 5.53e -2 Actor 9.45e -3 1.03e -2 7.83e -1 6.84e -1 6.71e -1 2.80e -1 5.80e -2 Chameleon 6.24e -3 1.13e -2 7.97e -1 2.28e -1 2.09e -1 6.88e -2 5.61e -2 Squirrel 7.77e -3 1.20e -2 7.78e -1 5.82e -1 5.05e -1 1.21e -1 5.79e -2Although the pseudo-error generation process is essential for our method, it is also an optional choice for backpropagation. We integrate this component into BP, and the experimental results from Tab. 8 show that although this component contributes to DFA in our method, it does not positively enhance BP overall. Even with pseudo-error generation, BP cannot outperform our method. This observation indicates that direct feedback of errors may benefit more from pseudo-errors rather than the layer-by-layer backward pass. We conduct experiments for our method with four different activation functions (i.e., Sigmoid, Tanh, ELU and LeakyReLU) as shown in Tab. 9. The results demonstrate our method is well integrated with different activation functions and derives consistently good results. Justification: The abstract and introduction clearly state the claims made.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: We report mean accuracy (%) ± 95% confidence interval in Tabs.1, 2 and 4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: There is no societal impact of the work performed and we cannot foresee any possible negative impacts, due to the highly technical nature of the task under consideration.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).For notation conciseness, we abuse the set notation and matrix notation interchangeably whenever appropriate. For example, X represents both a set of n attributes and a matrix. 


Decision: This paper receives overall positive reviews and the authors have actively engaged in rebuttals. One of the reviewers was concerned/confused with the computational complexity, and the authors have been trying to actively address them during rebuttals. However, this reviewer did not provide a final confirmation as to if the concerns have been addressed. I did a check and believe the concern has been properly addressed. Thus an accept is recommended.

Reviews: review: paper_summary: paper_summary: Recently, some studies have been exploring new optimization methods that can replace backpropagation, and one of the popular methods is direct feedback alignment (DFA). This paper adapts DFA to graph neural networks (GNNs). It replaces the gradient with a randomly initialized matrix multiplying the prediction error, so that we do not need to use backpropagation to update the parameters. We can directly update the model parameters based on the prediction error. Besides, for semi-supervised graphs where the label is scarce, we only have prediction errors for a few nodes, so the parameter update is not that effective. To overcome this drawback, the paper assumes that nearby nodes have similar prediction errors, so it uses a smoothing technique to generate pseudo prediction errors for unlabeled nodes.
strength_weakness: strength_weakness: (1) In Formula (4), it ignores the activation function when calculating the parameter update. Since the activation function is ReLU, ignoring it might not cause a big problem. However, when we apply other activation functions, or when we use a complex GNN architecture, I am afraid this method might not work as well.

(2) From Table 1, we can see that DFA achieves the best performance, while backpropagation is also very good on some datasets. I encourage the authors to add some discussions about when the proposed method is more useful than backpropagation, on what kinds of datasets, and what kinds of GNN models. Besides, the authors explained why DFA is better than previous non-backpropagation methods in Lines 254-274, but it would be better if the authors could explain why DFA is better than backpropagation.

(3) The authors can consider introducing a little more about the limitations of backpropagation and the non-backpropagation optimization methods in the introduction or related work. Since most readers are not familiar with non-backpropagation optimization methods, it deserves more introduction.

(4) The authors can improve some notations and make them clearer. For example, in Formula (5), the multiplication of $B^{(2)}$ and $B^{(1)}$ is written as $B^{(1)}$.
questions: questions: (1) What is $W$ in Formula (7)?

(2) In Section 4.3, the authors point out that W will converge to be similar to B. Since B is randomly initialized, if B is at a high point in the loss landscape, then will the optimization fail? Is the randomness of B important to the performance?

(3) I might have not fully understood the intuition behind DFA. I am confused about why DFA performs better than backpropagation. I look forward to intuitive explanations.

(4) Table 6 shows that all the non-backpropagation methods require more time than backpropagation. If I have correctly understood, DFA can directly calculate the parameter update for every layer from the loss, so it does not need to calculate the parameter update layer by layer as in backpropagation, then why does it need more time than backpropagation?
limitations: limitations: The authors have discussed the limitations. I encourage the authors to further explain what kinds of datasets and what kinds of GNN models DFA is most suitable for.
paper_summary: paper_summary: The authors propose to apply the Direct Feedback Alignment (DFA) algorithm for backpropagation-free training to graph neural networks. The DFA algorithm is combined with a pseudo-error generation mechanism to provide additional error signals for missing targets in the setting of semi-supervised node classification. The experimental results with a 3-layer GCN model over 10 commonly adopted benchmarks show the proposed model performing generally better than backpropagation and other training algorithms. The ablation analysis confirms that improvements by the proposed DFA-GNN approach can be ascribed to the pseudo-error generation mechanism.
strength_weakness: strength_weakness: 1. The computational cost per epoch of the proposed algorithm is 5 to 10 times larger than backpropagation.
2. No comparison of the overall training costs, including the different number of epochs required for training convergence among the different methods.
3. The ablation indicates that the accuracy improvements are ascribed to the pseudo-error propagation mechanism. As the latter could be applied also to e.g. backpropagation, such experimental evaluation would have provided further interesting insights.
4. In the experiments the model is limited to 3 layers; deeper models would have provided more interesting insights, as depth increases the issues connected with gradient backpropagation. (Long-range graph benchmarks such as those from *V. P. Dwivedi et al., “Long Range Graph Benchmark,” in NeurIPS 2022 Track on Datasets and Benchmarks* could be adopted in this case.)
5. Only node classification tasks are considered, no graph-level or edge-level tasks.
questions: questions: 1. Why is pseudo-error generation spreading, which must be done at teach gradient update iteration, instead of augmenting training labels via label spreading (plus the masking of eq. (9)), which would be only required once?
2. How would the other training methods such as backpropagation perform with pseudo-error generation spreading?
3. How many epochs are required for training convergence by DFA-GNN? How do they compare with the number of epochs required by the other training algorithms?
limitations: limitations: The limitations of this submission have not been properly discussed. For example, in lines 278-280, it is stated that the "method has a higher training time consumption compared with BP, primarily due to the additional time needed for generating pseudo errors and filtering masks". However, in the conclusions, it is claimed that *efficiency* is one of the advantages of the proposed method.
paper_summary: paper_summary: The paper introduces DFA-GNN, a new framework designed to train Graph Neural Networks (GNNs) using Direct Feedback Alignment (DFA). Traditional methods like backpropagation (BP), though effective, have several limitations, including inefficiency, scalability issues, and a lack of biological plausibility. DFA-GNN aims to address these problems by utilizing a forward training mechanism specifically tailored for the unique challenges posed by graph data.
strength_weakness: strength_weakness: The novelty of this paper is somewhat limited as it primarily extends the principles of DFA to GNNs. The core ideas are based on the foundations laid out in the Shalizi et al. (2013) paper.

The paper lacks a discussion on the computational complexity of the proposed algorithm.

There is a notable variation in the performance improvements observed across different datasets. For instance, DFA-GNN shows a significant accuracy improvement on the Texas and Cornell datasets, while the improvements on other datasets are relatively modest. The paper does not adequately explain the reasons behind this variation in performance.

The dataset used for experiments is quite small.
questions: questions: Why the proposed method doesnot outperform on the computer and chameleon dataset.

How does DFA-GNN scale when applied to larger datasets e.g. flicker, Reddit, OBGN ?

What is the time complexity of the proposed method?
limitations: limitations: No societal impacts.

Comments: title: Response to Author's Rebuttal
Response to Author's Rebuttal
Reply to the rebuttal
comment: Dear Reviewer,

Thank you for your valuable feedback on our paper. We have carefully addressed your comments in our response and submitted it for your review. As the deadline for responses is approaching, we wanted to kindly inquire if there are any additional questions or points of clarification you would like us to address. If any questions are not answered or our response is unclear, we would appreciate the opportunity to communicate further with you.

We sincerely appreciate your time and effort in reviewing this manuscript.

Best regards,

Authors
We apologize for the confusing expression. The standalone direct feedback alignment (DFA, $\mathcal{O}(n^{2})$) as formulated in Eqs. 5 and 6 of the manuscript can potentially be faster than backpropagation (BP, $\mathcal{O}(n^{2})$) since DFA directly calculates the parameter updates for each layer from the loss, allowing for improved training efficiency through parallelization. However, to generalize DFA for graph data, we introduced the pseudo-error generator as detailed in Sec. 4.2. Consequently, the main time cost of our method arises not only from DFA but also from the pseudo-error generation (Eqs. 7, 8) and node filtering (Eq. 9).

According to Eqs. 7 and 8, the time complexity of these two components is $\mathcal{O}(tcn^{2})$ and $\mathcal{O}(cn)$, respectively, where $t$, $c$, and $n$ denote the iteration epochs for error spreading, the number of categories, and the number of nodes, respectively (In our earlier analysis, we approximated the complexity of these components as $\mathcal{O}(n^{2})$ and $\mathcal{O}(n)$ to highlight that our method maintains the same quadratic complexity with respect to $n$, the most significant factor). When these two components are added, our method consistently incurs a time cost that is 5 to 10 times larger than that of BP even though they have the same order of complexity. Although these additional components increase the time cost, they are crucial for adapting DFA to GNNs and handling graph-structured data. The ablation study in Tab. 2 highlights the substantial contribution of these components. 

Although our method has achieved significant practical time efficiency advantages over existing non-BP methods, we will continue to explore ways to further improve the time efficiency of our training algorithm in future work to bring it closer to that of BP, and we will revise the related sections of the article to reflect these nuances more accurately.


Thank you again for your attention and support.

Best regards,

Authors
According to this statement  although both the method have quadratic complexity, "our method make parallel update for each layer while BP has to update layer by layer". Therefore, the proposed method should theoretically be faster than BP. However, this is not evident from the results in Table 6 of the appendix. Could you please clarify this discrepancy?
Thank you once again for your valuable feedback on our paper.

1. The time comparison of our method is presented in Table 6 of Appendix A.6 in the original manuscript (for your convenience, we have displayed the table below). Here, we compare the training time per epoch of our method with BP and other state-of-the-art non-BP methods. For the semi-supervised node classification task, all methods are trained for 1000 epochs. During training, we calculate the prediction accuracy on the validation nodes after each epoch and save the model that achieves the highest validation accuracy. This saved model is then used to make predictions on the test nodes. This approach is standard in semi-supervised graph learning. Therefore, the overall training cost is simply one thousand times the numbers shown in Table 6.


2. As noted in our response to W2 and Q3, both BP and the DFA in our method have a complexity of $\mathcal{O}(n^{2})$. Besides, our method introduces an additional complexity of $\mathcal{O}(n^{2})$ due to the pseudo-error generator. As a result, the overall complexity for both our method and BP remains the same $\mathcal{O}(n^{2})$. This explains why our method consistently takes five to ten times longer per epoch compared with BP as shown in Table 6, rather than showing an exponential difference, regardless of dataset size. While BP benefits from decades of research and strong software and hardware support, our method has not yet reached comparable time efficiency of BP. However, our method demonstrates superior time efficiency compared with most non-BP state-of-the-art methods, and the current time cost difference compared with BP is not substantial. Additionally, the parallel update strategy employed by our method for each layer offers considerable potential for parallel computing, which could be further explored to enhance time efficiency.

3. We theoretically derive formulas to integrate the direct feedback alignment mechanism into GNNs, as DFA for fully connected layers is not directly applicable to graph data. Additionally, we provide theoretical proof of the convergence of our proposed method (as shown in Section 4.3 and Appendix A.3). Our work highlights the promising potential of non-BP training in graph deep learning and opens up avenues for tackling some challenges within graphs, which merit further exploration in future research.


If our rebuttal has satisfactorily addressed your concerns, we would greatly appreciate it if you could consider reevaluating the score of our paper. Regardless of your decision, we are truly grateful for your guidance and the time you have invested in reviewing our work.

Thank you again for your attention and support.

Best regards,

Authors


$\newline$

**Table 6: Average running time per epoch (s). For layer-wise training methods like PEPITA, CaFo, FF, and SF, the total time taken by each layer per epoch is reported.**

| Datasets  | BP      | PEPITA  | CaFo+CE | FF+LA  | FF+VN  | SF      | ours    |
|----------|---------|---------|---------|--------|--------|---------|---------|
| Cora     | 7.56e-3 | 8.73e-3 | 7.61e-1 | 3.14e-1 | 2.83e-1 | 5.49e-2 | 5.66e-2 |
| CiteSeer | 1.06e-2 | 1.11e-2 | 7.68e-1 | 2.59e-1 | 2.61e-1 | 6.88e-2 | 5.68e-2 |
| PubMed   | 1.07e-2 | 1.07e-2 | 8.24e-1 | 6.94e-1 | 7.61e-1 | 5.34e-1 | 6.76e-2 |
| Photo    | 8.74e-3 | 1.03e-2 | 7.98e-1 | 2.11   | 1.91   | 4.87e-1 | 5.81e-2 |
| Computer | 1.08e-2 | 1.05e-2 | 7.80e-1 | 4.82 | 4.14   | 7.61e-1 | 6.29e-2 |
| Texas    | 6.13e-3 | 1.07e-2 | 8.05e-1 | 1.47e-1 | 1.56e-1 | 6.88e-2 | 5.60e-2 |
| Cornell  | 5.42e-3 | 1.06e-2 | 7.46e-1 | 1.51e-1 | 1.24e-1 | 3.59e-2 | 5.53e-2 |
| Actor    | 9.45e-3 | 1.03e-2 | 7.83e-1 | 6.84e-1 | 6.71e-1 | 2.80e-1 | 5.80e-2 |
| Chameleon| 6.24e-3 | 1.13e-2 | 7.97e-1 | 2.28e-1 | 2.09e-1 | 6.88e-2 | 5.61e-2 |
| Squirrel | 7.77e-3 | 1.20e-2 | 7.78e-1 | 5.82e-1 | 5.05e-1 | 1.21e-1 | 5.79e-2 |
I would like to thank the authors for your response.  However, I still have concerns in these points:
1.I could not find a table comparing the time efficiency of your method with state of the art method.
2. I have concerns about the time complexity. The additional step also contribute to an O(n²) complexity? How much it isfaster than standard BP methiod?
3. The novelty of your approach seems limited, as it primarily involves integrating DFA into GNNs.
Thank you for your valuable feedback and for acknowledging our efforts. We respect your decision and sincerely appreciate your constructive critique. Your insights have been instrumental in helping us refine our work, and we are grateful for your thorough and thoughtful review.
Thanks to the authors for their response. I leave my score unchanged.
Thank you for your insightful feedback and for recognizing our efforts. We appreciate your suggestion to further discuss the method's limitations, particularly regarding its suitable scenarios and training efficiency. 

During the rebuttal period, we conducted additional experiments to address these concerns, focusing on the method's suitability for various activation functions, its performance on large-scale graphs, training time, and convergence behavior. These insights have provided a more comprehensive understanding of the method's capabilities and constraints. In the final version of the paper, we will include a dedicated section discussing these aspects, offering a balanced view of the method's strengths and areas for future improvement.

We respect your decision and are grateful for your constructive critique. Your feedback has been instrumental in refining our work, and we thank you once again for your thorough and thoughtful review.
Thank you for your reply! Your responses have addressed most of my concerns. The authors conducted a lot of experiments to verify the proposed method from various perspectives. Nonetheless, it seems that this method has some limitations regarding its suitable scenario and training efficiency that are worth further discussion in the paper. It would be better if the authors could include more discussions about the insights and limitations in the paper. I would like to maintain my score.