Forward Learning of Graph Neural Networks
Authors: Namyong Park
Xing Wang
Antoine Simoulin
Shuai Yang
Grey Yang
Ryan A. Rossi
Puja Trivedi
Nesreen K. Ahmed
Venue: ICLR 2024 poster (2024)
Keywords: graph neural networks
forward learning
forward-forward algorithm
Primary area: learning on graphs and other geometries & topologies
TL;DR: This paper develops a new forward learning procedure for graph neural networks.

Abstract: Graph neural networks (GNNs) have achieved remarkable success across a wide range of applications, such as recommendation, drug discovery, and question answering. Behind the success of GNNs lies the backpropagation (BP) algorithm, which is the de facto standard for training deep neural networks (NNs). However, despite its effectiveness, BP imposes several constraints, which are not only biologically implausible, but also limit the scalability, parallelism, and flexibility in learning NNs. Examples of such constraints include storage of neural activities computed in the forward pass for use in the subsequent backward pass, and the dependence of parameter updates on non-local signals. To address these limitations, the forward-forward algorithm (FF) was recently proposed as an alternative to BP in the image classification domain, which trains NNs by performing two forward passes over positive and negative data. Inspired by this advance, we propose ForwardGNN in this work, a new forward learning procedure for GNNs, which avoids the constraints imposed by BP via an effective layer-wise local forward training. ForwardGNN extends the original FF to deal with graph data and GNNs, and makes it possible to operate without generating negative inputs (hence no longer forward-forward). Further, ForwardGNN enables each layer to learn from both the bottom-up and top-down signals without relying on the backpropagation of errors. Extensive experiments on real-world datasets show the effectiveness and generality of the proposed forward graph learning framework. We release our code at https://github.com/facebookresearch/forwardgnn.

Published as a conference paper at ICLR 2024
FORWARD LEARNING OF GRAPH NEURAL NETWORKS
Namyong Park', Xing Wang', Antoine Simoulin', Shuai Yang', Grey Yang', Ryan Rossi?, Puja Trivedi', Nesreen Ahmed Meta AI, Adobe Research, UUniversity of Michigan, 'Intel Labs
ABSTRACT
Graph neural networks (GNNs) have achieved remarkable success across wide range of applications, such as recommendation, drug discovery, and question an- swering. Behind the success of GNNs lies the backpropagation (BP) algorithm, which is the de facto standard for training deep neural networks (NNs). However, despite its effectiveness, BP imposes several constraints, which are not only bio- logically implausible, but also limit the scalability, parallelism, and flexibility in learning NNs. Examples of such constraints include storage of neural activities computed in the forward pass for use in the subsequent backward pass, and the dependence of parameter updates on non-local signals. To address these limita- tions, the forward-forard algorithm (FF) was recently proposed as an alternative to BP in the image classification domain, which trains NNs by performing two forward passes over positive and negative data. Inspired by this advance, we pro- pose FORWARDGNN in this work, a new forward learning procedure for GNNs, which avoids the constraints imposed by BP via effective layer-wise local for- ward training. FORWARDGNN extends the original FF to deal with graph data and GNNs, and makes it possible to operate without generating negative inputs (hence no longer forward-forward,. Further, FORWARDGNN enables each layer to learn from both the bottom-up and top-down signals without relying on the backpropa- gation of errors. Extensive experiments on real-world datasets show the effective- ness and generality of the proposed forward graph learning framework. We release our code at https::
INTRODUCTION
Graph neural networks (GNNs) (Zhou et al.) 2020 Wu et al.) 2021) are a family of deep learning methods designed to operate on graph-structured data, such as social networks, knowledge graphs, and citation networks. In recent years, GNNs have achieved state-of-the-arr results across wide range of applications that involve graphs, such as recommendation (Fan et al.] 2019), drug discov- cry (Zhang et al.) 2022), traffic forecasting (Jiang & Luo 2022), question answering (Park et al. 2022a 2020), and program analysis (Allamanis, 2022). Behind the huge success of GNNs, as well as other types of neural networks, lies the backpropagation (BP) algorithm (Rumelhart et al. 1986.
Plaut et al. 1986), which is the de facto standard for training deep neural networks.
BP adjusts the parameters of a neural network in a way that reduces the discrepancy between the net- work's output and the target value (e.g., ground-truth class labels). Specifically, these updates by BP are performed in two passes. First, in the forward pass, the input is fed forward through the network to produce an output, while storing intermediate activations of the neurons. Then, in the backward pass, an error between the output and target gets propagated backward through the network using the chain rule, which uses the stored activations to calculate the gradients of the loss with respect the network parameters. Then using gradient descent (Kiefer & Wolfowitz 1951). parameters are adjusted in the negative direction of the gradients, thereby reducing the loss.
1952 Robbins & Monro,
Spurred by the success of BP, several prior works investigated whether BP could be a model of how the brain learns (Lillicrap et al., 2020; (Scellier & Bengio 2017, Crick 1989). However, there is no evidence that the brain follows the learning constraints imposed by BP, e.g., (i) neural activations computed in the forward pass need to be stored for use in the backward pass, which increases memory overhead, (ii) parameter updates depend on the signals from all downstream neurons, while biological synapses learn locally, i.e., from the signals of locally connected neurons (Whittington & Bogacz 2019), and (iii parameter updates occur only after the forward pass, and in its reverse order (i.e., from the last down to the first layer). These constraints not only render BP biologically implausible, but also limit the scalability, parallelism, and flexibility in learning neural networks.

---

Published as a conference paper at ICLR 2024
The forward-forwart algorithm (FF) (Hinton 2022) is a representative recent approach designed to be a biologically plausible alternative to BP. FF avoids the above constraints of BP by replacing the forward and backward passes in BP with two forward passes, which operate and negative data, respectively, with opposite objectives. By adopting a layer-wise forward-only training scheme, FF avoids the restriction of BP that a perfect knowledge of the computations in the forward pass needs to be retained to compute the derivatives for backpropagation. In preliminary experiments on image classification, FF showed comparable classification performance to BP.
the positive
The training of GNNs is limited by the same constraints of BP as discussed above, which restrict the scalability and flexibility of model training, as well as rendering the learning process biologically implausible. While FF and its recent variants (Hinton 2022, Zhao et al. 2023: Lee & Song 2023] Paliotta et al., 2023) saw promising results, they fail to provide an effective and efficient alternative for learning GNNs for the following reasons. First, the FF algorithm involves generating negative data, which is a task-specific process (e.g. handcrafting masks for images with varying long range correlations, and manipulating images via overlaying class labels), and needs to be defined to be used for new tasks. Existing FF methods are mainly designed for image classification, and thus are difficult or ineffective for handling graph data. Second, the current way FF generates negative data is not scalable as it takes an increasing amount of memory and computation time as more negative samples are used. To date, the potential of FF for the fundamental graph learning (GL) tasks, i.e., node classification and link prediction, has not been studied, and it remains to be explored how effectively FF or alternative forward learning procedures can train GNNs for fundamental GL tasks.
To bridge this gap, we develop FORWARDGNN in this paper, a forward learning framework for GNNs, which builds upon and improves the FF algorithm (Hinton) 2022) in learning GNNs for graph learning tasks. First, FORWARDGNN proposes two orthogonal graph-specific approaches for extending the FF algorithm for GNNs. Importantly, FORWARDGNN removes the need to explicitly construct negative inputs needed for FF to work (i.e., FORWARDGNN requires just a single for- ward pass, and hence no longer forward-forward) This greatly improves the efficiency of both training and inference processes. Further, in the local layer-wise training of FORWARDGNN, each layer not only receives input from the lower layer, but also incorporates what the upper layers have learned without relying the backpropagation of errors, which further improves performance the bottom-up and top-down signals jointly inform the forward training. We show the effectiveness and generality of the proposed forward graph learning framework in an extensive evaluation using three representative GNNs on five real-world graphs. In summary, we make the following contributions.
Forward Graph Learning. We systematically investigate the potential of forward graph learn- ing, i.e., biologically plausible forward-only learning of graph neural networks for fundamental graph learning tasks, namely, node classification and link prediction.
a novel forward learning framework for GNNs. FORWARDGNN agnostic to the message passing schemes of GNNs, and is capable of learning GNNs via a single forward pass, while incorporating top-down signals.
Novel Learning Framework. We develop FORWARD
Effectiveness. Extensive experiments show that (1) FORWARDGNN outperforms, or performs on par with BP on link prediction and node classification tasks, while being more scalable in memory usage; and (2) the proposed single-forward approach improves upon the FF-based methods.
2
BACKGROUND AND RELATED WORK
2.1 GRAPH NEURAL NETWORKS
Given a graph G (V, with nodes V and edges E, graph neural networks (GNNs) learn the representation of a node in graph G by repeatedly performing a neighborhood aggregation or mes- sage passing over its local neighborhood across multiple layers. Let x; E F be the input features of node i, h E H be the representations or embeddings of node i learned by layer E (f 2 1), and e(j.)) E IR F2 be optional features of the edge (j. i) from node j to node i. The message passing pro- cedure that produces the embeddings of node i via the f-th GNN layer can be described as follows:
(8) = (() h
h
(E-1)
(0 (n! (e-1) nn (C-1) , )
JEA(())
Here, (0) is set to x; for neighborhood aggregation, which summarizes the information of the nodes i and j. as well as the
N; (i) is the neighbors of node i, and (--) is a function that extracts a message
2

---

Published as a conference paper at ICLR 2024
optional edge features ((j.) if available. ①(-) denotes permutation permutation-invariaat function (e.g., mean or max) to aggregate incoming messages, and is function that produces updated embeddings of node by combining node i's embeddings with aggregated messages. With multi-layer GNNs, the embeddings of node learned via local message passing capture information from its k-hop neigh- bors. This message passing scheme can be considered as generalizing the convolution operator in CNNs to rregularl--structuree graph data (Zhou et al. 2020 Daigavane al. 2021]. where con- nections among nodes are often highly skewed and irregular in contrast to grid sridd-stuctured images.
Various GNN architectures be described by this framework, including graph convolutional net- works (GCN) (Kipf & Welling, 2017), GraphSAGE (Hamilton et al.) 2017), and graph attention networks (GAT) Velickovie al.. (018)) They differ in terms of how they define the functions in Eq. [ i.e. ((0)) (-) (-), and vf (.). For instance, to define the aggregator (-),, GraphSAGE examines a few options, such as the mean operator, and a max pooling on top of trainable neighbor transformations, while GAT uses attention mechanism to perform a learnable weighted averaging.
As GNN training is done by BP, it inherits the same constraints of BP discussed in Section] | work, we develop alternative learning procedure for GNNs without those constraints.
In this
2.2 THE SERRWARD-FOWAARR ALGORITHM
The forward-fowaat algorithm (FF) (Hinton 2022 a greedy layer-wise learning procedure for optimizing multi-layer neural networks (NNs). That is, FF trains layers one at a time, going from the bottom to the top, and trained layers are no longer further optimized when upper layers are trained. The main idea of FF is to replace the forward and backward passes of BP with two forward passes, which operate on the positive (real) and negative (incorrect) data. Given images and their class labels, FF constructs positive samples by overlaying each image with its one-hot encoded labels. Negative samples are constructed in the same way, but using a randomly chosen incorrect label.
Goodness. In FR, each layer has its own objective, which is to make the forward pass at the layer assign high goodness to positive data, and low goodness to negative data. As positive and negative data differ only in the included labels, the layer is then trained to ignore the features that do not cor- relate with the label. Specifically, FF defines the goodness of a layer to be the sum of the squares of the ReLU activations. an input vector for object i. The goodness g( of object i by the E-th layer is computed as
Let a (4) denote the ReLU activation that the ê-th layer produces by processing
cf ||a (2) 2
(2)
To train each layer to be able to distinguish between positives and negatives using goodness, FF defines the probability that object i is considered to be positive by layer E as follows:
p(positive) = a (g!"
(3)
where (a) 1/(1+e=)) is the logistic function, and 0 is a threshold, which is to make the goodness be well above the given value for positive data, and well below it for negative data.
Normalization. Note that Eq. (3) uses activation's 's length in L2 norm to compute the goodness. To prevent the next layer from distinguishing positive from negative data by just using the length of the input vector (in 1.2 norm), FF normalizes the length of the activation before passing it down the next layer. This way, the next layer is guided to use the orientation of the input vector, not its length.
Applications and Extensions. FF was recently used for several applications, e.g., biomedical and hyperspectral image classification (Angulo & Paheding 2023 Paheding & Angulo 2023) on-device training (De Vita et al. (2023), imitation learning (Chung et al. 2023), and optical NN training (Oguz) et al. 2023). A few studies also presented extensions of FF. Instead of training the layers to predict goodness (Eq. 2 CaFo (Zhao al. (023) attaches a class predictor to each layer. In CaFo, the given NNs are randomly initialized and fixed the whole time, and only the layer-wise predictors are locally trained. SymBa (Lee Song 2023) adopts an alternative loss to maximize the goodness gap between positive and negative items via direct comparison. The predictive FF (Ororbia & Mali] 2023) combines predictive coding with FF, which jointly learns representation and generative cir- cuits in biologically plausible manner. In contrast to FF and follow-up works designed for image classification, GFF (Paliotta et al., 2023) adapts FF to graph classification by defining the goodness of graph. However, GFF is designed specifically for graph classification, and cannot be applied to the two fundamental GL tasks, i.e., node classification and link prediction. Improving upon GFF and earlier works, FORWARDGNN enables forward learning for these GL. tasks for the first time.
3

---

Published as a conference paper at ICLR 2024
(a) Backpropagation
(b) Forward-Forward (Sec.3.)
(c) Single Forward (Sec. 3.2)
(d) Single Forward w/ Top-Down Signals (Sec. 3.3)
Loss
Good ness
Local
Signal Path: Top To-Los
Loss
Layer 2
Layer 2
Layer 2
Local Losss
Good
Good
Local Loss
Layer 2
Layer
Layer
Layer
Tavall Loss
graph
featss X
neg graph
neg feats. x
pos graph &pes
graph G
Layer
feats, Xroes
Guegr
graph & feats.
graph G
Signal Path:
Top-To-Input
graph
leats.
graph G fests. X
Forward pass
Backpropagation of errors
Layer-wise local training
Learning objective
Figure I: Backpropagation (BP) and proposed forward learning methods for GNNs. (a) BP involves
one forward pass, followed by one backward pass through the network. (b) Forward-forwat (Sec.
3.1 involves two forward passes on positive and negative inputs. (c) Single forward (Sec.3.2 learns via just one forward pass. (d) Single forward is extended to incorporate top-down signals (Sec. 3.3
3 FORWARD LEARNING OF GRAPH NEURAL NETWORKS
We develop new forward learning procedures for GNNs in this section. We start by extending the FF algorithm for GNNs, and design a forward graph learning procedure with just a single forward pass. We then extend them so that both the bottom-up and top-down signals can inform the training. Figure Soureeprovve provides an overview of the proposed forward graph learning approaches and BP.
3.1 ADAPTING THE FHRWARD-FFRWARE ALGORITHM FOR GRAPH NEURAL NETWORKS
Positive and negative samples play essential role in FF, layers are trained to tell them apart in terms of the goodness (Eq. 2) Designed for image classification, FF creates them by overlaying cor- rect or incorrect labels over images. To adapt this idea to deal with graph data and GNNs, we propose two orthogonal approaches for extending node features and graph structure using labels (Fig. [p).
Extending Node Features With Node Labels. In this approach, we transform node labels into aux- iliary features and use them to augment node features. Formally, the extended features xi of node i is
(4) where (abel-To-Feaa(1) is a function that maps node i's label to additional features, and [x]|y]] de- notes concatenation of vectors x and y. Positive and negative features are created by using variants of Label-To-Featti) which return the one-hot encoding of the correct, and randomly selected incor- rect labels for node respectively. Note that not all nodes have labels, some nodes are held out for testing, or nodes may have been partially labeled from the beginning. This differs from the im- age classification setup, where all training images are labeled. For nodes without labels, we define Label-To- -eat())) to be a uniform distribution over the node classes such that all classes contribute equally for these nodes. This approach affects only the node features, while retaining the graph structure. Thus the positive and negative embeddings of training nodes are obtained by running GNNs (Eq. lb on the same graph G using differently altered features 2.00 and (-+ (x=)))"") as n,)).
x = [a:; || Label-To-Feat
Extending Graph Structure With Virtual Nodes. This approach introduces virtual nodes corre- sponding to the node classes (i.e., there are as many virtual nodes as the classes), and use the links be- tween real and virtual nodes to make positive and negative samples. For positive samples, each real node linked to the correct virtual node, that is, the one corresponding to the true label of the node. For negative samples, real nodes are linked with randomly chosen incorrect virtual nodes. Formally, given a set C = (k of K virtual nodes, the augmented graph G' = (V', E') is defined to be
(5) where Viateled S V is the set of labeled nodes, and Label-RE(1) and Label-VR(c. are functions that map real node and virtual node to the class labels, respectively. Specifically, in order to create positive and negative samples, Label-RE(i) returns correct, and randomly chosen incorrect labels, respectively, for real node while Label-VRI (C) consistently returns the true label for virtual node
G' (V= UC, EU ((,,e) | i Visbeled C CEC, Label-RE(t) = Label-
4

---

Published as a conference paper at ICLR 2024
With this augmented graph structure, real nodes can aggregate information from their real neighbors and a virtual node. Similarly, virtual nodes aggregate information from the real nodes they are linked with, which makes them representative of those connected real nodes. E.g., with positive connec- tions, virtual nodes can be considered as class representatives, as virtual nodes are only connected to the real nodes that belong to the virtual node's class. From the perspective of real nodes, the only difference between positive and negative cases is the link to the virtual node. All in all, GNN layers are trained to utilize this difference to assign the goodness (Eq. 2 appropriately, egg. by ignoring or downplaying some of the aggregated node features that do not correlate well with the node label. The positive and negative embeddings of training nodes are generated by running GNNs (Eq. 1) on differently augmented graphs Grost and {Gmees''), Jy with the same input features æ; = h, (") Alg. in Appendix Snveendixprreeeem presents the forward learning algorithm discussed in this subsection.
Inference. After training GNNs with the above methods, GNN layers can produce the goodness of node with respect to label I, i.e., by running GNNs with node features extended with label (Eq. 4 or an extended graph where test nodes are linked to the virtual node corresponding to label (Eq. 5
To predict for a given node, we compute the goodness for all classes using each GNN layer, and accu- mulate the goodness of all layers. Then the label with the highest accumulated goodness is selected.
3.2 FORWARD GRAPHI LEARNING VIA A SINGLE FORWARD PASS
Based on FF, the approaches in Section 3.1 operate by learning to distinguish between positive and negative inputs. Specifically, this process involves (1) creating positive and negative inputs in the form of differently augmented graphs or node features, and (2) performing multiple forward passes of GNNs with those augmented graphs and features, which overall has high computational and memory requirements. This limitation is because they incorporate the label information at the imput level, i... by perturbing the inputs to create positive (real) and negative (incorrect) samples, which require separate processing. We tackle this limitation by designing a learning framework that can utilize the label information for the forward GNN training with just a single forward pass (Fig. [T)).
Framework. Our main idea is to enable the network to generate learning signals based on the avail- able node labels, instead of requiring positive and negative samples to start the training, which are created by perturbing the input data using labels. In other words, we want the GNN layer to avoid the need for positive and negative samples by generating its own training targets in one forward pass, which can then be used to learn node embeddings that reflect the class structure among the nodes.
To this end, we empower GNN layers to generate class representatives as their learning targets via graph convolution (Eq. D Specifically, we turn to our proposed idea of graph augmentation (Eq. (5) and use the positively augmented graph Grost for the forward pass, in place of the original graph G. As discussed in Sec. 3.1 a virtual node in Grost can be considered to be representative of the nodes in the corresponding class due to its label- -ased connections to real nodes and the message passing process. This way, GNNs can produce the embeddings of real nodes and class representatives simul- taneously in just one forward pass, without having to design separate architecture to obtain class representatives. As a result, compared the forward-forward algorithm that involves two forward passes, this framework is no longer
and hence a significant departure from FF.
Learning Signals. Given the embeddings and labels of real nodes as well as class representatives, we would want the learning signals to train GNNs in such a way that nodes that are similar and in the same class would be close to each other in the embedding space learned by GNNs, while nodes that are dissimilar and in different classes would be away from each other. To train the 2-th GNN layer, we generate such learning signals with the following contrastive learning objective C(6)
(0) L(h ,0, O), i( = - logg exp(C(h, (Labe(())/T)
((0)
Σ
(6)
Viabetee
iE Vissaked
(8) ExEpLK (xp(C(h, cx )/T)
where 10 and c are embeddings of node and class learned by the 8-th GNN layer; K is the number of node classes; Label(i) maps node i into its class; C(-,-) is critic function that measures the similarity of two embeddings (e.g.. dot product); and 7 is the temperature parameter. This objec- tive produces the learning targets by contrasting real nodes with class representatives, and maximizes the probability of discriminating the correct (node, class) pair among negative unmatched pairs. Alg. in Appendix presents the algorithm for the proposed single forward approach.
5

---

Published as a conference paper at ICLR 2024
Inference. Once GNNs are trained, GNN layers can produce the probability of node belonging to each class via the softmax inside the logarithm of Eq. (6). To predict for the given node, we compute the class distribution across all GNN layers, and select the label with the highest average probability.
3.3 INCORPORATING TOP-DowN SIGNALS
In the above forward learning frameworks (Secs. 3.1 3.2). GNN layers are trained progressively from the bottom to the top, i.e., once a layer is trained, it remains fixed while upper layers are trained. As a result, when layer E [1, I.] trained, it can only use the bottom-up signals from lower layers 1 to i - 1, but not the top-down signals from upper layers e + 1 to L. By contrast, with BP, layers can also learn from the top-down signals via the backpropagation of errors throughout the network. Inspired by recent works the part-whole hicrarchy representation in NNs (Hinton 2021 2022) and collaborative FF (Lorberbom et al., 2023). we extend the proposed single forward pass approach (Sec. (.2) to incorporate the top-down signals via two paths (Fig. I
Signal Path: Top-To-Input. This path allows to incorporate the output of upper layers into the input of a GNN layer. We now consider the forward pass training to be done over time, such that the training of GNN layers at time / incorporates the top-down signals from the previous time step / - 1.
(2-1) Let it
be the embeddings of node produced by layer e- 1 time t. The output of upper layers
at the previous time t - 1, i... i,t-1 ................................ h (L)
11111 constitutes tow- signals for node i. Then (e- (e-1y Merge(h (t-1) (h 1,1-1
(f-1)
(E+1)
(L) (1-1))
is given to layer l as input instead of ( (f-1), Several options exist for Merge(-), e.g., all upper layers, or an immediate upper layer alone may provide top-down signals, which are averaged and concate- nated with n/, (E-1) This way, each layer can learn from the representations learned by upper layers.
Signal Path: Top-To-Loss. This path allows to incorporate the output from upper layers into the output of a GNN layer to be used in the loss computation and the inference step. Specifically, the following loss replaces Eq. (6) so that both bottom-up and top-down signals can inform the training,
cl0 EE
C(h 1,(0) Viatetaal where h (E)" Merge(h (h (n(--1)(=4))) (8)
which uses augmented embeddings h A (L-1)" that incorporate top-down signals, instead of (6-1) Via this signal path, each layer can learn to generate outputs in light of what the upper layers learned.
Importantly, even with these signal paths, no gradient flows through GNN layers. Thus, the learning steps for top-down signal incorporation.
procedure remains to forward only. Alg. 3
3.4 APPLICATION TO LINK PREDICTION
The ideas proposed in Secs. 3.1 and 3.2 use the label information (e.g., label-based virtual nodes), with a focus on the node classification task. Here we discuss that the proposed framework can be naturally used for the forward learning of the link prediction (LP) task. We note that the standard procedure for training and evaluating GNNs for I.P shares a lot of similarities with the FF algorithm, and is thus readily adaptable for use in the forward learning framework. Given embeddings h, and hj of nodes i and j. the link probability Pt.j between nodes and j is modeled as follows:
Pi.j LinkProb (EdgeEmb(h;; h2)))
where Edge h2;) is a function that returns the embedding of edge (i,j) (e.g., element-wise product), and LinkProb(-) transforms the edge embedding into a probability (e.g., summation fol- lowed by sigmoid function). The training goal is to learn to distinguish between the existing (positive) and nonexistent (negative) edges in terms of the estimated link probability. This process is similar to FF in the sense that both aim to tell positive and negative samples apart using the learned scores (i.e., link probability and goodness). Differently from the FF applied for node classification (Fig. (p). positive and negative edge embeddings
be generated via single forward pass using the standard training process for LP. Notably, this forward learning approach differs from the BP- based LP training in that each layer is locally trained without backpropagation of errors. The idea in Sec. 3.3 Sec.3.3 can also be used to incorporate top-down signals in learning the embeddings of nodes, and hence the embeddings of edges. The forward learning algorithm for LP is given in Alg. in App. C
Inference, Once GNNs are trained, each GNN layer produces the probability of given edge. We compute the link probability to be the average of the link probabilities estimated by all layers.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 EVALUATION SETUP
Our goal is systematically evaluate the effectiveness and generality of the proposed forward graph learning approach on fundamental graph learning tasks, namely, node classification and link predic- tion, using representative GNN models and real-world graphs.
Node Classification. Given a graph where each node is assigned to a single class, the task is to predict the class of the given node. We randomly generate the train-validation-test node splits with a ratio of 64%-16%--00% and evaluate the performance in terms of classification accuracy.
Link Prediction. The task to predict whether the given edge is positive (existing) or not. We split the edges randomly into train-validation-tes sets, with a ratio of 64%-16%--20% which form pos- itive edge sets. We randomly select the same amount of nonexistent edges as the positive edges, which form negative edge sets. We obtain the link probability (Eq. 9 via dot product of the two nodes' embeddings, followed by sigmoid. We measure the performance using the ROC-AUC score.
For both tasks, we perform evaluation using five randomly generated splits, and measure the average and standard deviation of the corresponding performance metric.
Datasets. We use five real-world graphs drawn from three domains: PUBMED, CITESEER, and CORAML are citation networks; AMAZON a co-purchase network; GITHUB followership graph. Table 2 in Appendix A provides the summary statistics of these graphs. Appendix A also presents more detailed description of these datasets.
Graph Neural Networks. We evaluate FORWARDGNN and BP using three representative GNNs, namely, graph convolutional network (GCN) (Kipf & Welling 2017), GraphSAGE (SAGE) (Hamil- ton al. 2017). and graph attention network (GAT) Velickovic et al. 2018).
Experimental settings, e.g., details of GNNs and software used in this work, are given in Appendix B
4.2 COMPARISON WITH THE BACKPROPAGATION ALGORITHM
We compare the proposed forward learning method with backpropagation (BP) using two criteria, namely, (1) task-specific performance (e.g., accuracy) and (2) GPÚ memory usage increase as more layers are used. Among several forward graph learning approaches we develop in Sec. 3 we report the results obtained with the single-forward learning approach (SF) (Sec. 3.2h in this subsection. Also, for link prediction, we apply the cross entropy loss the SF method. We later present comparison among different forward learning approaches in Sec. 4.3
Result Visualization. In Figure 2 we show the task-specific performance (y-axis) and the memory usage of multi-layer GNN compared to the I-layer GNN (x-axis), obtained with training GNNs with the proposed SF and BP. In Figure 2] the shape and color of a symbol indicate the type of GNNs and dataset, respectively, and symbols that are filled and empty, respectively, denote that the corresponding GNN was trained with BP and SF. For ease of comparison, the filled and empty symbols corresponding to the same GNN on the same dataset are linked by dotted line.
Node Classification (Figure 2a Using this visualization, we make the following observations.
The proposed forward learning method (SF) performs similarly BP many cases (which correspond to the near horizontal lines), or even outperforms BP in several cases (corresponding to the lines that go towards the lower right corner), across different GNNs and graphs.
As we use GNNs with a larger number layers, BP requires more memory for training (the hor- izontal distance between the filled and empty symbols gets longer), while the memory required by SF remains nearly the same (empty symbols are located near the left end of the figures). The memory usage increase when using BP goes up to 18x as we use four layers.
Link Prediction (Figure 2b). With the same visualization, we have the following observations.
The proposed SF mostly achieves a higher link prediction performance than BP (most lines go towards the lower right corner). This is mainly because the link prediction performance of the GNNs trained with BP declines as more GNN layers are used, while the performance of GNNs trained with SF improves to varying degrees (or remains nearly the same) as more layers are used. As in the node classification result, BP incurs more memory usage as the number of layers in- creases, while the amount of memory required by SF remains the same.
7

---

Published as a conference paper at ICLR 2024
A SF-GGCN
OSF-GAT
SAGE
Amazon
CiteSeer
ML
BP-GCN
BP-GAT
+ BP BA-SAGE
GitHub
PubMed
100
100
100
Accuracy (%)
Accuracy (%)
Accuracy (%)
80-
60
60
60
Better
Better
Better
40-
40
1x
4x
8x
12x
16x
1x
4x
8x
12x
16x
1x
4x
8x
12x
16x
Mem usage relative Ilaaeer model Mem usage relative Ilaaer model Mem usage relative 1-layer model
(1) 2-lyee models
(2) 3-layer models
(3) 4-layer models
(a) Node classification accuracy (y-axis) vs. memory usage increase (x-axis). The proposed songle-forward graph learning (SF, empty symbols) performs similarly to hackpropagation (BP, filled symbols) in many cases (iee, near horizontal lines), even better than BP in several cases (ie., lines going towards the lower right comne). with no increase in memory usage as more layers are used. Figure|ishows these results per GNN.
100
100
100
Better
Better
Better
ROC ROCAUC AUCAUCCV (%)
Better
ROC AUC (%)
Better
ROC AUC (%)
Better
80
70-
70
70
1.0x
1.5x
200x
2.5x
3.0x
1.0x
1,5x
2.0%
2.5x
3,0x
1.0%
1,5x
2.0%
2.5x
3,0x
Mem usage relative I-layer model Mem usage relative 1llayer model Mem usage relative 1-layer model
(I) 2-lyyrrmodeek models
(2) 3-layer models
(3) 4-layer models
(b) Link prediction performance (y-axis) vs. memory usage increase (x-axis). The proposed SF outperforms backpropagation (BP, filled symbols) (corresponding to most lines going towards the lower right corner), with no increase in memory usage as more layers are used. Figure shows these results per GNN.
The single-forward approach (SF, empty symbols) vs. backpropagation (BP, filled sym- bols) on (a) node classification and (b) link prediction. Lines towards the lower right corner (SF out- performs BP), and upper right corner (BP outperforms SF); horizontal lines (both perform the same).
Figure 2:
Additional Results. Figures 5 and ssow the results of Figures 2a and 2b 2blin detail, per GNN model.
4.3 COMPARISON AMONG FORWARD LEARNING APPROACHE
In this section, we evaluate the proposed forward learning approaches (Sec. and existing forward learning methods, terms of the task performance and memory usage.
Node Classification. We evaluate the following forward learning methods: (1) FF-VN and (2) FF- LA, the forward-forwand (FF) graph learning approaches (See. [.11 which augment graph data with virtual nodes, and label-appending technique, respectively; (3) SF. the single-forward graph learning method (Sec, 3.2) (4) So-TopDown, the SF method with the top-to-input signal path (Sec. 3.3); (5) FF-SymBa (Lee & Songl (203)) and (6) CaFo (Zhao et al.] (023). which are two extensions of the FF algorithm (Hinton 2022); and (7) PEPITA DDelafeererra & Kreiman 2022), a prior state-of-the- art forward learning method preceding FF. Note that FF-SymBa is not directly applicable to node classification, so we adapted it to operate on top of the proposed virtual node method (Sec. 3.1
Figure 3 shows classification ac- curacy (y-axis) versus memory usage (x-axis), GNNs (denoted by the symbol shape) are trained with the above learning methods (denoted by the symbol color). In this section, we report results ob- tained with two-layer GNN models (Figures 3 and 4
Table 1:
Top-down signals improve classification results. Aver- age classification accuracy obtained with SF and SooDon.
Method CITESEER CORAML PUBMED AMAZON GITHUB
SF So-Top
91.14
84.95
$1.90
89.69
83.16
93.58
86.01
82.21
92.09
83.68
and Table Tb.
The best classification accuracy is achieved by either the proposed single-forward approaches (SF and Doono)))wn) or forward-forward (FF) methods (FF-VN, FF-LA, and FF-SymBa). However, FF methods (FF-VN, FF-LA, and FF-SymBa) icantly outperformed by the single-forward (SF) approaches. Also, they require much larger memory than the SF methods, as they need construct multiple negative inputs to train GNNs.
often unstable, and are signif-
8

---

Published as a conference paper at ICLR 2024
AGCN OGAT #SAGE
ST-TopDown (Sec 3.3)
SF (Sec 3.2)
FFF-AA (Sec 3.1)
PF-VN (Sec 3.1)
FF-SymBa
CaFo
PKPITA
100-
Agpo
Accuracy (%)
75.
Accuracy (%)
Accuracy (%%
Accuracy (%)
Accuracy (%)
#0-
80
A4
60-
60
50-
04
Better
Better
50-
Better
40
86
Better
Better
48
Better
Better
Better
1000
1000
100
200
200
Mem usage (MB)
Mem usage (MB)
Mem usage (MB)
Mem usage (MB)
Mem usage (MB)
Amazon
GitHub
CiteSeer
PubMed
CoraML
Figure 3: Node classification accuracy (y-axis) and memory usage (x-axis) as three GNNs (denoted
by symbol shape) are trained with different forward learning approaches (denoted by symbol color).
CaFo uses the least amount of memory, but performs significantly worse than the proposed meth- ods. This because CaFo optimizes only the layer-wise predictors, not the GNN model itself.
Also, PEPITA fails to train GNNs effectively for node classification.
Incorporating top-down signals enables SF to perform node classification more accurately as shown in T which reports the classification accuracy on five graphs, averaged over different
GNN models and varying number of layers.
Link Prediction. We evaluate the following forward learning methods, which are applicable for link prediction: (1) ForwardGNN-CE, (2) ForwardGNN-FF, and (3) ForwardGNN-Synin
which are all based on the proposed single-forward framework (Sec. (.4). while using different learning
objectives (i.e., binary cross entropy (BCE), forward-forward objective (Eq. [3). and the objective in (Lee & Song. ).23), respectively); and (4) CaFo (Zhao et al. (023), which uses layer-wise link predictors trained with BCE. Other forward learning methods used above (e.g., PEPITA) are not applicable for link prediction as they are designed specifically for node classification.
Fig. 4 shows the link prediction result (y-axis) and memory usage (x-axis) of different GNNs.
The proposed forward learning methods, especially ForwardGNN-CE and ForwardGNN-- SymBa, consistently outperform CaFo by a large margin. Freezing the GNN layers is too strin- gent a restriction for CaFo to effectively learn to do link prediction.
ForwardGNN-CE shows the overall best results considering both the accuracy and stability of predictions. ForwardGNN-Symbl often suffers from high variability in the prediction accuracy.
AGCN OGAT SAGE
ForwardGNN-CE FoNNFrdGNN-FF
ForwardCNN-SymB.
CaFo
(%)
ROC ROCAAUCS (%)
66
ROCAN (U))
ROC ROCAUC AUC AUC(%) (%)
ROC (5))
90
90-
ROC AUC
Befter
Better
Better
80
#0-
Better
30
70-
70-
70
Better
Better
Better
Better
200
400
500
1000
20
100
150
25
75
Mem usage (MB)
Mem usage
Mem usage (MB)
Mem usage (MB)
Mem usage (MB)
Amazon
GitHub
CiteSeer
PubMed
CoraML
Figure 4: Link prediction performance (y-axis) and memory usage (x-axis) as three GNNs (denoted
by symbol shape) are trained with different forward learning approaches (denoted by symbol color).
Further Results. We present the performance and memory usage for all learning methods, GNNs, and graphs in Tables and 4 for node classification, and in Tables 5 and for link prediction.
5 CONCLUSION
Backpropagation (BP) has been the standard algorithm for training graph neural networks (GNNs). However, despite its effectiveness, BP introduces several constraints in learning GNNs. In this work, we investigate the potential of forward-one training in the context of graph learning, and propose FORWARDGNN, a forward learning framework for GNNs. Overall, our contributions are as follows.
Forward Graph Learning. We systematically investigate the potential of biologically plausible forward learning of GNNs for fundamental GL tasks, ie., node classification and link prediction.
Novel Learning Framework. We develop FORWARDGNN, a novel forward learning framework for GNNs. FORWARDGNN can be used with GNNs with different message passing schemes.
Effectiveness.. Extensive experiments show that (1) FORWARDGNN outperforms, or performs on par with BP on link prediction and node classification tasks, while being more scalable in memory usage; and (2) the proposed single-forwand approach improves upon the FF-based methods.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper develops new learning framework for graph neural networks. To evaluate the proposed framework, we use public datasets, which do not contain any personally identifiable information or offensive materials to the best of our knowledge. Also, experiments in this work focus evaluating the prediction accuracy and memory usage, and do not involve human subjects. We do not foresee any ethical concerns with this work.
REPRODUCIBILITY STATEMENT
For reproducibility, we describe our experimental settings in Section 4.1 and Appendices A and B
including details of data splitting, evaluation tasks and metries (Sec. 4.1p; dataset description and statistics (App. [ and Table 2 and hyperparameter settings, software, and hardware specifications
(App. B All datasets used in this work are publicly accessible (Sec. 4.1). We release our code at
https://gt httpg//tthu...ooo. htt.:/ccc/.cccoooooo.oooo
REFERENCES
Miltiadis Allamanis. Graph neural networks in program analysis. Graph neural networks: founda- tions, frontiers, and applications, pp. 483-497, 2022.
Abel A. Reyes Angulo and Sidike Paheding. The forward-forward algorithm as a feature extractor for skin lesion classification: preliminary study. CoRR, 0lss23377.00611 2023.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In ICML, volume 97 of Proceedingg of Machine Learning Research, pp. 583-593.
PMLR, 2019.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. CoRR, abs/1407.7906, 2014.
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsuper- vised inductive learning via ranking. In ICLR (Poster). OpenReview.ne., 2018.
Miguel Á. Carreira-Perpiñán and Weiran Wang. Distributed optimization of deeply nested systems. In AISTATS, volume 33 of JMLR Workshop and Conference Proceedings, pp. 10-19. JMLR. 2014.
Insik Chung, Isaac Han, and Kyung-Joong Kim. Imitation learning using the harward-forwand algo- rithm, 2023. URL httpthtts////ppreeeeeeeeeeeeeeee
www.tee..oorur d= 9F 9F
Francis Crick. The recent excitement about neural networks. Nature, 337(6203):129 132, 1989.
Wojeiech Marian Czamecki, Grzegorz Swirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 904 912. PMLR, 2017.
Hanjun Dai. Zornitsa Kozareva, Bo Dai, Alexander J. Smola, and Le Song. Learning steady-states of iterative algorithms over graphs. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 1114-1122. PMLR, 2018.
Ameya Daigavane, Balaraman Ravindran, and Gaurav Aggarwal. Understanding convolutions graphs. Distill, 2021. doi: 00.23315/disttll..o
gnns.
Fabrizio De Vita, Rawan M. A. Nawaisch, Dario Bruneo, Valeria Tomaselli, Marco Lattuada, and Mirko Falchetto. p-ff: On-device bowward-frward training algorithm for microcontrollers. In 2023 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 49-56, 2023.
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit as- signment problem without backward pass. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 4937-4955. PMLR, 2022.
Wengi Fan, Yao Ma, Qing Li. Yuan He, Yihong Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In wWW. pp. 417-426. ACM, 2019.
10

---

Published as a conference paper at ICLR 2024
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR,
2019.
Manas Gupta, Arulmurugan Ambikapathi, and Savitha Ramasamy. Hebbnet: A simplified hebbian learning framework to do biologically plausible learning. In /CASSP, pp. 3115-3119. IEEE, 2021.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pp. 1024-1034, 2017.
Alois P. Heinz. Pipelined neural tree learning by error forward propaaatio.....n.... In ICNN, pp. 394-397.
IEEE, 1995.
Geoffrey E. Hinton. How to represent part-whole hierarchies in a neural network. CoRR, Jos/2102.11267., 2021.
Geoffrey E. Hinton. The forward-forward algorithm: Some preliminary investigations. CoRR,
abs/2212.133455, 2022.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In ICML., volume 70 of Proceedings of Machine Learning Research, pp. 1627-1635. PMLR, 2017.
Weiwei Jiang and Jiayun Luo. Graph neural network for traffic forecasting: A survey. Expert Syst. Appl. 207: 17921, 2022.
Adrien Journé, Hector Gareia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without feedback. In ICLR. OpenReview.net, 2023.
Jack Kiefer and Jacob Wolfowitz. Stochastic estimation of the maximum of regression function. The Annals of Mathematical Statistics, pp. 462-466, 1952.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR (Poster). OpenReview.net, 2017.
Adam Kohan, Edward A Rietman, and Hava T Siegelmann. Signal propagation: The framework for learning and inference in a forward pass. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Adam Kohan, Edward A. Rietman, and Hava T. Siegelmann. Error forward prapaaarddpopaaaiiio.. Reusing feedforward connections to propagate errors in deep learning. CoRR, abs/ 1808.08883 03357, 2018.
Eukasz Kuśmierz, Takuya Isomura, and Taro Toyoizumi. Learning with three factors: modulating hebbian plasticity with errors. Current opinion in neurobiology. 46: 77-1177 2017.
Julien Launay, Iacopo Poli, and Florent Krzakala. Principled training of neural networks with direct feedback alignment. CoRR, abs/1906.0455 2019,
Dong- Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In ECML/PKDD (I), volume 9284 of Lecture Notes in Computer Science, pp. 498-515. Springer, 2015.
Heung-Chang Lee and Jeonggeun Song. Symba: Symmetric backpropagationn-ire contrastive learn- ing with forward-forward algorithm for optimizing convergence. CoRR, abs/2303.08418, 2023.
WB Levy and O Steward. Temporal contiguity requirements for long-term associative potentia- tion/depression in the hippocampus. Neuroscience, 8(4):791-797, 1983.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 13276, 2016.
Timothy Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, 21(6):335-346 2020.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.

---

Published as a conference paper at ICLR 2024
Chuang Liu, Yibing Zhan, Jia Wu, Chang Li, Bo Du, Wenbin Hu, Tongliang Liu, and Dacheng Tao. Graph pooling for graph neural networks: Progress, challenges, and opportunities. In IJCA,, pp. ijcai.org, 2023.
Yue Liu, Jun Xia, Sihang Zhou, Siwei Wang, Xifeng Guo, Xihong Yang. Ke Liang, Wenxuan Tu, Stan Z. Li, and Xinwang Liu. A survey of deep graph clustering: Taxonomy, challenge, and application. CoRR, abs/2211.1225
2022.
Guy Lorberbom, Itai Gat, Yossi Adi, Alexander G. Schwing, and Tamir Hazan. Layer collaboration in the forward-forwart algorithm. CoRR, abs/2305. 12393, 2023.
Chenxiang Ma, Rui Yan, Zhaofei Yu, and Qiang Yu. Deep spike learning with local classifiers. IEEE Trans. Cybern,, 53(5):3333-3375 2023.
Alexander Meulemans, Francesco S. Carzaniga, Johan A. K. Suykens, João Sacramento, and Ben- jamin F. Grewe. theoretical framework for target propagation. In NewrlPS, 2020.
Hesham Mostafa, Vishwajith Ramesh, and Gert Cauwenberghs. Deep supervised learning using local errors. Frontiers in neuroscience, 12:371677, 2018.
Arild Nokland. Direct feedback alignment provides learning in deep neural networks. In NIPS, pp- 1037-1045, 2016.
Ilker Oguz, Junjie Ke, Qifei Wang, Feng Yang. Mustafa Yildirim, Niyazi Ulas Dinç, Jih-Liang Hsieh, Christophe Moser, and Demetri Psaltis. Forward-forward training of optical neural network. CoRR, abs/2305.191700 2023.
Alexander Ororbia. Brain-inspired machine intelligence: survey of neurobiooogically-laassbbb credit assignment. CoRR, abs/2312.09255 2023.
Alexander Ororbia and Ankur Arjun Mali. Biologically motivated algorithms for propagating local target representations. AAAI, pp. 4651-4658. AAAI Press, 2019.
Alexander Ororbia and Ankur Arjun Mali. The predictive forward-forward algorithm. CoRR, hts/2301.014522 2023.
Alexander Ororbia, Ankur Arjun Mali, Daniel Kifer, and C. Lee Giles. Conducting credit assignment by aligning local representations. CoRR, Abs//800.015 2018.
Sidike Paheding and Abel A. Reyes Angulo. forward-forwald algorithm for hyperspectral image classification: A preliminary study. CoRR, abs/2307.0022) 2023.
Daniele Paliotta, Mathieu Alain, Bálint Máté, and François Fleuret. Graph neural networks go forward-forward. CoRR, abs/2302.05282, 2023.
Namyong Park, Andrey Kan, Xin Luna Dong, Tong Zhao, and Christos Faloutsos. Multilmport: Inferring node importance in knowledge graph from multiple input signals. In KDD, pp. 503- 512. ACM, 2020.
Namyong Park, Fuchen Liu, Purvanshi Mehta, Dana Cristofor, Christos Faloutsos, and Yuxiao Dong. EvoKG; Jointly modeling event time and network structure for reasoning over tempo- ral knowledge graphs. WSDM, pp. 794-803. ACM, 2022a.
Namyong Park, Ryan A. Rossi, Eunyee Koh, Iftikhar Ahamath Burhanuddin, Sungchul Kim, Fan Du, Nesreen K. Ahmed, and Christos Faloutsos. CGC: contrastive graph clustering for community detection and tracking. WWW, pp. 1115-1126. ACM, 2022b.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein. Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang. Zachary Veti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In NeurlPS, pp. 8024-8035, 2019.
David C Plaut, Steven Nowlan, and Geoffrey E Hinton. Experiments on learning by back propa- gation. 1986.
12

---

Published as a conference paper at ICLR 2024
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400 1000-407,100 1951.
Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. J. Complex Networks, 9(),, 2021.
David E Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back- propagating errors. nature, (288))8) 1986.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Trans. Neural Networks, 20(1):61-80, 2009.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Fronliers in computational neuroscience, 11:24, 2017.
Jürgen Schmidhuber. Networks adjusting networks. Proceedings of Distributed Adaptive Neural Information Processing", " pp. 197-208, 1990.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/ 1.05668, 2018
Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit B. Patel, and Tom Goldstein. Train- ing neural networks without gradients: A scalable ADMM approach. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, 2722-2731. JMLR.org, 2016.
Matteo Tiezzi, Giuseppe Marra, Stefano Melacci, Marco Maggini, and Marco Gori. A lagrangian approach information propagation in graph neural networks. ECAI, volume 325 of Frontiers in Artificial Intelligence and Applications, pp. 1539-1546. IOS Press, 2020.
Matteo Tiezzi, Giuseppe Marra, Stefano Melacei, and Marco Maggini. Deep constraint-based prop- agation in graph neural networks. IEEE Trans. Pattern Anal. Mach. Intell., 44(2):727-739, 2022.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR (Poster). OpenReview.net, 2018.
Yewen Wang, Jian Tang, Yizhou Sun, and Guy Wolf. Decoupled greedy learning of graph neural networks. 2020.
James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in cognitive sciences, 23(3):235-250, 2019.
Zonghan WU, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey graph neural networks. IEEE Trans. Neural Networks Learn. Syst., (1):4-24, 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-GCN: layer-wise and learned efficient training graph convolutional networks. CVPR, pp. 2124-2132. Computer Vision Foundation IEEF, 2020.
Zehong Zhang, Lifan Chen, Feisheng Zhong, Dingyan Wang, Jiaxin Jiang, Sulin Zhang, Hualiang Jiang. Mingyue Zheng, and Xutong Li. Graph neural network approaches for drug-targer interac- tions. Current Opinion in Structural Biology, 73: 73:102327, 2022.
Gongpei Zhao, Tao Wang, Yidong Li, Yi Jin, Congyan Lang, and Haibin Ling. The cascaded forward algorithm for neural network training. CoRR, abs/2303.09728. 2023.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. AI Open, 1:57-81, 2020.
13

---

Published as a conference paper at ICLR 2024
In the appendix. we provide the dataset description (Appendix A experimental settings (Ap-
pendix algorithms of the proposed forward graph learning framework FORWARDGNN (Ap-
pendix discussion the extensibility of FORWARDGNN to other graph learning tasks (Ap-
pendix Dj. additional related work (A (Appendix Eb, and additional experimental results (Appendix
A DATASET DESCRIPTION
In experiments, we use the following five real- world graphs drawn from three different domains. PUBMED, CITESEER, and CORAML are citation networks from Bojchevski & Günnemann (2018), where nodes represent documents, and edges de- note citations between the corresponding papers. Node labels denote the topic of the paper, and node features correspond the bag-of- woods representa-
Table 2: Summary of the graph datasets.
Dataset
Nodes
# Edges # Features # Classes
PURMED
19,717
88,648
500
3
CITESEER
4,230
10,674
602
6
CORAML
2,995
16,316
2,879
AMAZON
7.650
238,162
745
GITHUB
37,700
578,006
128
tion of the paper. AMAZON is co-purchase network from [Shhhur et al. (2018), which corresponds to the Photo segment, where nodes represent goods, and edges denote that two goods are frequently bought together. Node labels represent item categories, and node features denote the bag-of-words representation of the product review. GITHUB is a followership graph from Rozembercckii et al. (2021), where nodes are developers, and edges denote mutual follower relationships. Nodes classified into two groups of web and MI developers, and node features denote the developer's 's in- formation, such employer name and email address. Table provides the statistics of these graphs.
B EXPERIMENTAL SETTINGS
Hyperparameter Settings, Model Configurations, and Evaluation Details. Following the default configurations, we ran graph convolutional network (GCN) (Kipf & Welling 2017) with a normal- ized adjacency matrix, Le.. self-loops were added and symmetric normalization was applied. We used the mean aggregator with GraphSAGE (SAGE) (Hamilton et al., 2017). For graph attention network (GAT) (Velickovic et al. 2018), we used four attention heads, and concatenated the multi- head attention. Also, based on the default settings of GAT, we added self-loops to the graph, and used a negative slope of 0.2 for LeakyReLU nonlinearity. For all GNNs and across all graph learning tasks, we set the size of hidden units to 128. We used the Adam optimizer with a learning rate of 0.001, and a weight decay of 0.0005. For both tasks, we split the data (i.e., nodes and edges) ran- domly into train- vrain-aliddttion-eer epochs to 1000, and applied validation-basee early stopping with a patience of 100, for both tasks. In the node classification tasks, some learning algorithms, such as the Forward-Forwart algorithm (FF), explicitly construct negative samples for training. For those algorithms, we used all available classes to create negative samples for each node, i.e., given K node classes, K 1 negative samples were used for the training of each node. For memory usage, we measured the memory used by the GPU for loading the model parameters, activations, gradients, and the first and second moments of the gradients. For FF and its variants, we set the threshold parameter 0 to 2.0, and for FF-SymBa, we set its a parameter to 4.0, following the settings in the original papers. We the temperature pa- rameter r to 1.0 in Eq. (6. We define the Merge function in Eq. 7 to incorporate the signal from the immediate upper layer, i.e., h and concatenate it with the input h (.!(") For Eq. (8). we define the Merge function to incorporate signals from upper layers, which affect only the loss, but are not fed into the input for the next layer. To apply FF-SymBa (Lee & Song 2023) for node classification, we used FF-SymBa on top of the virtual node technique (Sec. 3.1 SO it can be used for GNNs.
sets, with a ratio of 14%%-66%-20% Also, we set the max training
Software. We implemented FORWARDGNN in Python 3.8, using PyTorch (Paszke al. 2019) v1.13.1. We build upon the implementation of the forward-forwant algorithm for image classi- fication provided by Nebuly We used PyTorch Geometric (Fey & Lenssen 2019) v.2.2.0 for the implementations of GCN, SAGE, and GAT, and for the module that performs edge splitting and sam- pling used in link prediction. For PEPITA (Dellaferrera & Kreiman (022), we adapted the original implementatiorf[ for GNNs, i.e., after performing forward passes through GNN layers, we compute the error with respect to node labels, and use to construct modulated inputs for the training nodes.
Hardware. Experiments were performed on a Linux server running CentOS 9, with an NVIDIA H100 GPU, AMD EPYC 9654 96-Core Processors, and 2.2TB RAM.
httesss//iiibbbb.ccm///eelyyyy/
httass//githbbb..ccm///ooo/////p
14

---

Published as a conference paper at ICLR 2024
C ALGORITHMS
We present the algorithms of the proposed forward graph learning approach for node classification, Le., the horward-forward learning approach (Alg. [. the single-forward learning approach (Alg. 2), and the single-forward learning approach with top-down signal incorporation (Alg. We then present the forward learning algorithm for link prediction (Alg.
Algorithm 1: Forward- Learning of GNNs for Node Classification (Section
Input: GNN G, graph C, input features X, node labels y Output: trained GNN g
1 Construct positive and negative inputs extending graph with virtual nodes then
Gron Greg! & positive and negative graphs built using virtual nodes and labels (Sec. 3.1
X X <- X
else if extending input features X by appending labels then Gront G +- G,G
X10o X positive and negative features built by appending labels using y (Sec.
3.1
end
1 Train layer by layer progressively from the bottom the top
HPsst Hase X negr
for layer in GNN layers do
10
while not converged do
Fud (tryer.forwarr(G*MM H.Oos)
11
Ho
+ (ayer.forwardd(""ee H
Compute the local loss (e.g., the FF loss (Eq, 3 for layer using H H and H
13
14
Update layer's parameters using an optimizer of choice, e.g., SGD and Adam
15
end
16
best layer +- load the layer with the best params. found during the local training (lines 10 to 15 if extending graph with virtual nodes then
17
H Hh +- bebettloo layer for HPwM), best lass.aay..forardc... ,HP else if extending input features X by appending labels then
18
19
Hoo H best layer. forward Grost Hoos),, payer end 21
20
layeealorwaaareeeee. forward(
HOsS H
Hrost .detach() H detach(
22
23
end
Algorithm 2: Single-Forward Learning of GNNs for Node Classification (Section 3.2)
Input: GNN g. graph G, input features X, node labels Output: trained GNN G
G <a positively augmented graph using virtual nodes and node labels y (Section 3.2
1 Train layer by layer progressively from the bottom to the top
H X
for layer GNN g layers do while not converged do Hind + layer forward(G' H)
Compute the local loss given by Eq. 66) for layer using Hind
Update layer's parameters using an optimizer of choice, e.g., SGD and Adam
end
best layer + load the layer with the best parameters found during the local training (lines to 10
H +-
H))
11
H + dedetach(
end
Note that in Alg. 3 with the top-to-input signal path, we set the top-down signal for the topmost hidden layer come from the context vector (we used one hot encoding corresponding to the node
15

---

Published as a conference paper at ICLR 2024
label for training nodes, and uniform distribution vector of the same size for other nodes), following Hinton 2022). Also, in updating the GNN layers with the incorporation of top-down signals, we support both synchronous and asynchronous updates. With asynchronous (i.e., alternating) updates, even-numbered layers are updated based on the activities of odd-numbered layers, and then odd- numbered layers updated based on the activities of even-numbered layers. With synchronous updates, updates to all layers occur simultancously.
Algorithm 3: Single-Forward Learning of GNNs with Top-Down Signal Paths for Node Classi- fication (Section 3.3)
Input: GNN G. graph C, input features X, node labels Output: trained GNN G
G' + positively augmented graph using virtual nodes and node labels y (Section 3.2
(8) be // to store embeddings at the previous and current time sleps 1 Forward-only training with top-down signal incorporation while not converged do
Initialize (H and {H
for E, layer in enumerate(GNN G 's layers) do
// layer index & starts from 1 if signal-path "top-to-input" then
Construct H!"-1 by applying Eq. with (H 1- and (HI!")e
else
H 1 + H!!-1)
end
H (2) + forward(r (((-IV),detach()
11
if signal-path "top-to-loss" then
Compute the local loss given by Eq. for layer using {H and {H be
else
Compute the local loss given by Eq. 6 for layer using H
15
end
16
Update layer's parameters using an optimizer of choice, e.g., SGD and Adam
end
18
foreach E [] do H + H
29
end
21
end
Algorithm 4: Single-Forward Learning of GNNs for Link Prediction (Section 3.4
Input: GNN G. graph G, input features X
Output: trained GNN g
Erust Eney positive and negative edges sampled from graph G (Section 3.4
1* Train layer by layer progressively from the bottom to the top
H <- X
for layer in GNN layers do while not converged do Hjwd +- layer forward H)
Compute the local loss for layer using the link probability (Eq. with node embeddings Htwd, and positive and negative edge sets 1.pes/ and Bey
Update layer's parameters using an optimizer of choice, e.g., SGD and Adam
end
best layer + load the layer with the best parameters found during the local training (lines to H best wor.er.forwarlCC.. H)
10
11
H H. H.detach()
end
16

---

Published as a conference paper at ICLR 2024
D EXTENSIBILITY OF FORWARDGNN TO OTHER GRAPH LEARNING TASKS
In this work, we demonstrate how FORWARDGNN can be applied to the two fundamental graph learning tasks, namely, node classification and link prediction. Here we discuss the extensibility of FORWARDGNN to two other important graph learning tasks.
Graph Classification. Given a set of graphs, the goal of graph classification is to assign a label to each graph, i.e., the prediction is done the graph level as opposed to the node- and edge-level predictions made for node classification and link prediction, respectively. In the case of graph clas- sification, GFF (Paliotta et al.) 2023) presented the first FF-based approach. FORWARDGNN can be extended to improve upon GFF for graph classification by replacing the multiple forward passes required by GFF with a single forward pass. Similar how FORWARDGNN learns generate rep- resentatives of node classes to address node classification via a single forward pass, this extension requires effective mechanisms to learn class representatives at the graph level. To that end, we can investigate different approaches, e.g., application of graph pooling methods (Liu et al. (023),, ) possi- bly in combination with virtual nodes, which can be used to identify and connect related nodes and graphs. Once we can generate class representatives of graph instances, the local contrastive learning objective used for node classification (Equation (6)) can be used similarly for graph classification.
Graph Clustering. Given graph, the goal of graph clustering is to assign the nodes in the graph into (potentially overlapping) clusters. The major difference from node classification is that we do not have ground truth node labels for model training in graph clustering problems, and the number of clusters in the graph is usually not available. Due to this difference, the techniques presented in Sec- tion 3.1, including the use of virtual nodes, cannot be used for graph clustering. Thus, the learning signals (Equation (6)) designed for node classification cannot be directly utilized. Instead, the con- trastive forward learning objective given by Equation (6) can be extended to incorporate the learning signals of deep graph clustering approaches (Liu et al., 2022), which can operate when node labels are unavailable and the number of classes are not known in advance. For instance, they construct self-supervised learning signals by utilizing input node features, and the characteristics of worl-world graphs such as network homophily and hierarchical community structure (Park et al. (022b). With the extended local learning objective, the proposed single-forward learning framework (Section 3.2) can be applied in nearly the same way for clustering the input graph.
E ADDITIONAL. RELATED WORK
In Sec. 2
we discuss alternatives to backpropagation (BP) for learning GNNs based on the forward- forward algorithm (Hinton 2022). Here we present a review of other alternative approaches and re- lated works for learning GNNs, as well as overview of aatoooicalll-iispieee learning algorithms.
E. ALTERNATIVE APPROACHES AND RELATED WORK FOR LEARNING GNNS
Carreira-Perpinán Perpiñán & Wang (014) and Taylor et al. (206)) proposed alternative approaches to BP for learning neural networks (NNs), where the learning of NNs is cast as a constrained optimization problem in the Lagrangian framework, with neural computations expressed constraints. These approaches are inherently local, in the sense that the gradient computation relies on neighboring neurons, not on the BP over the entire network. However, they are not scalable due to the high mem- ory requirement, and thus are inapplicable to large-scale problems. Based on this Lagrangian-base approach, Tiezzi et al. (2220 2022) presented an alternative method for learning GNNs. Specifi- cally, they simplified the learning process of the GNN model (Scarselli et al., 2009), representa- tive recurrent GNN model, which performs graph diffusion repeatedly until it converges
stable
fixed point. The I.P-NNN proposed in Tiezzi et al. (2020) 2022) adopts a mixed strategy to enable a more efficient training of GNN*, which employs constraint- avoiding the explicit computation of the fixed point of the diffusion process in GNN*, while using BP in updating the NNs that model the state transition and output functions. Accordingly, although LP-GNN departs from the usual Ba-ba GNN training, it still relies on BP for learning GNNs. By contrast, N is free from BP as a forward-only learning scheme for GNNs.
propagation scheme, thereby
Stochastic steady- state embedding (SSE) (Dai et al. 2018)) presents another learning scheme, in the context of learning steady-state solutions of iterative algorithms over graphs, e.g., PageRank scores and connected components. To obtain steady-state solutions efficiently, SSE proposes a stochas-
17

---

Published as a conference paper at ICLR 2024
tic fixed-point gradient descent process, inspired by the policy iteration in reinforcement learning. While SSE is an alternative learning approach, SSE is not, per se, a general training scheme for GNNs like FORWARDGNN, as its goal is to learn an algorithm achieving steady-state solutions of iterative graph algorithms, which differs from the task of GNN training FORWARDGNN addresses.
Another line of works investigated waie approaches for learning GNNs. A layer- wise learning idea closely related to our work first appeared in the vision domain (Belilovsky et al. 2019). which
proposed a greedy layer-wise learning technique for deep convolutional neural networks (CNNs), and obtained promising results on large-scale datasets such as ImageNet. In In[Belilovsky et al. 2019),
a CNN model gets trained progressively and greedily, i.e., layer by layer from the bottom to the top. To enable layer-wise training, they formulate and solve an auxiliary problem for each layer, where an auxiliary classifier is attached to the layer, forming a block, and the original CNN layer and the auxiliary classifier in the block are jointly optimized in terms of the training accuracy of the auxiliary classifier. You et al. (2020) extended this approach for a layer-wise training of GNNs, in which the CNN layers are replaced with the GNN layers. Furthermore, You et al. (2020) streamlined the graph convolution operation by decoupling the feature aggregation and transformation and introduced a learnalle RNN controller, which automatically decides when to stop the training of each layer via reinforcement learning. A follow-up work by Wang et al. (2020) further improved the efficiency of layer-wise GNN training of You et al. (2020) via the proposed parallel training and lazy update scheme. While these methods train GNNs layer by layer, they still rely on BP, since each block multi-layer architecture in general, and as result, the error derivatives are propagated across the layers in the block for optimization. By contrast, FORWARDGNN operates in the forward-only learning regime, which is the focus of this work. Also, their optimization framework allows only bottom-up up signal propagation, and does not provide way to incorporate the top-down signals. Thus, using the above layer- wise methods, the training of each layer cannot be informed by what the upper layers have learned as in FORWARDGNN. Further, they only investigate the node classification task, and miss out the other fundamental GL task of link prediction. In this work, we systematically explore the potential of forward-only learning for both node classification and link prediction tasks.
E.2 BIOLOGICALLYYINSPIREE LEARNING ALGORITHMS
For the past several decades, many Mologgcalll-inspieee algorithms have been developed to train neural networks without relying on the biologically implausible backpropagation of errors. A recent survey (Ororbial 2023) organizes them based on how they address the central question of learning algorithm, namely. "Where do the signals for learning the elements of a neurul network come from, and how they are produced?" Following the taxonomy of OOorbia (203)), biologically-inspreed al- gorithms can be grouped follows.
Implicit Signal Algorithms. No explicit targets or signals are involved in these algorithms. Instead, they utilize implicit signals, which are produced by the feedforward process of the neural network. The computations for inference in these methods are the same as those for model training, and maaming-based approaches (Jouréé et al. 20233] Gupta et al.) 2021 Levy & Steward 1983), only the pre-synaptic and post-synaptic activations are used to adjust the weights connecting the adjacent layers (i.e., learning.
parameter updates are done using purely local information. For example, in Hebbian
two-factor Hebbian), rendering the parameter adaptation to be a correlation-
Global Explicit Signal Algorithms. In algorithms in this and the following categories, the learning process is driven by explicit signals. Among them, approaches in this category adopt a global approach, in which a global feedback pathway is employed carry signals for adjusting synaptic weights. Random feedback alignment and its variants (Lillicrap et al., 2016 Nokland| 2016: Launay al. 2019) construct teaching signals using random feedback weights, and establish asymmetric forward and backward pathways, thereby addressing the weight transport problem of BP, in which the same parameter matrices are used for both the inference and model training. Neuromodulatory approaches, such as three-factor Hebbian plasticity (Kušmierz et al. 2017). are another family of algorithms in this category, which produce and broadcast modulatory signals that drive synaptic adjustments, e.g., binary gating variables to either accept reject synaptic adjustment.
Non-Synergistic Local Explicit Signal Algorithms. Algorithms in this category adopt exclusively local machinery, such local classifier, to produce signals based on locally available information (e.g., information from a pair of adjacent layers), which drive the learning of a network. One group of algorithms in this category can be referred to as synthetic local updates (SLU) (Jaderberg et al.
18

---

Published as a conference paper at ICLR 2024
2017
Czarnecki et al. 2017 Schmidhuber 1990). The basic idea of SLU is to extend each layer with a learnable parametric model (e.g., an additional weight matrix) to approximate and predict the weight updates gradients based on the local information from adjacent layers. Local signalers (predictors) (Mostafa et al. 2018 Ma et al., (023) present a different yet related approach based on non-synergistic signals, where each layer is augmented with local predictor (e.g.. classifier), which projects the layer's s activation into the class score distribution via a fixed random weight matrix, and gets adjusted according to the local cost.
Synergistic Local Explicit Signal Algorithms. In contrast to the non-synergistic schemes discussed above, algorithms in this category produce local learning signals via varying degrees of (indirect) coordination among the layers of network. Note that this coordination across layers is not through a single global pathway as in the glohal explicit signal algorithms or BP. One group of algorithms in this class are based on discrepancy reduction, which performs synaptic updates by reducing the mismatch between the model's current representations and the the target representations produced by complementary neural processes, which may span several layers, e.g., an approximate inversion pathway of the network the case of target propagation approaches (Bengio 2014] Lee et al. 2015]
Meulemans al. 2020), and synaptic message passing structure in the case of local representation alignment schemes (Ororbia & Mali, 2019 Ororbia et al. 2018).
Forward-only learning approaches (Hinton 2022
2018 Linsker 1988 ZZao et al.) 2023 Dellaferrera & Kreiman, 2022 are another family of algorithms in this
Heinz 1995 Kohan et al. 2023
category, which performs synaptic updates only using the inference process of neural system, i.e., without involving (recurrent) feedback mechanisms as in some of the aforementioned algorithm families. Forward-only approaches can be divided into two groups, one based on supervised context and the other based on self-supervised context. The first group of methods (Kohan et al.) [0233] 20188) Zhao et al., 2023] Dellaferrera & Kreiman 2022) presents a forward-only scheme that performs synaptic adjustments driven by signals arising in supervised context, e.g., they learn the model such that the representations of an input and the corresponding context are closely located via a set of local loss functions. We notice that this idea is related to FORWARDGNN as both generate local learning signals for the inputs and contexts, although the way they are produced as well as the inference process greatly differs. Importantly, prior works are not applicable to graphs and GNNs, and provide no or limited mechanisms for incorporating top-down signals. The second group consists of self- supervised context-driven approaches (Hinton 2022 Lee & Song 2023 Ororbia & Mali 2023) Paliotta et al., 2023). such as the forward- algorithm (FF) (Hinton (022), which aim to distinguish between positive samples (i.e., items taken from the real data distribution) and negative samples (i.e, incorrect, adversarially generated items). Training with these algorithms is performed so as to maximize the goodness for positive samples, while minimizing for negative samples. With this objective, the loss of a neural network is defined to be the sum of local layer-wise loss functionals based on the goodness. Thus, the design of goodness and how negative samples are generated are key to the effectiveness of these approaches. These algorithms perform two forward passes on positive and negative samples, where the second forward pass replaces the usual backward pass of BP. Building upon FF and follow-up works, FORWARDGNN makes further improvements for an effective forward learning of GNNs by removing the need to design and generate negative inputs via its fonweeforward learning framework, and letting each layer learn from both bottom-up and top-down signals without relying on BP. Section 2.2 provides a more detailed discussion on FF (Hinton 2022) as well as its extensions and applications.
For a comprehensive review and discussion of biologically-inspiie learning algorithms, we refer the reader to Ororbia 22033)
19

---

Published as a conference paper at ICLR 2024
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 NODE CLASSIFICATION RESULTS
Figure presents the node classification accuracy (y-axis) vs. memory usage increase (x-axis) per graph neural networks.
Table 3 presents the node classification performance of GCN, SAGE. and GAT models averaged over five runs on five real-world graphs, obtained with different number of layers, and using different learning approaches.
presents the GPU memory usage (in MB) during the GNN training for node classification, observed with different number of layers, and using different learning approaches.
A SF--GCN OSE-GAT {oSF-SAGE BP-GCN
Amazon GitHub
CiteSeer
CoraML
BP-GAT + BP-SAGE
Pub
100
100
100
Accuracy (%)
GCN
80
80
80-
Better
Better
Better
Better
Better
Better
1x
16x
16x
16x
100
100
100
Accurncy (%)
80-
GAT
Better
Better
Better
Better
Better
Better
40
40-
1x
4%
8x
12x
16x
1x
4x
8x
12x
16x
1x
4x
8xx
12x
16x
100
100
100
Accuracy (%)
SAGE
Better
Better
Better
80
70
Better
70
Better
70-
Better
1x
2x
4x
6
8x
1x
2x
4x
6k
8x
1x
2x
4x
6x
8x
Mem usage relative 1-layer model Mem usage relative 1-layer model
Mem usage relative 1-layer model
(1) 2-layer models
(2) 3-layer models
(3) 4-layer models
Figure 5: Node classification accuracy (y-axis) vs. memory usage increase (x-axis). The proposed single forward graph learning (SF, empty symbols) performs similarly backpropagation (BP, filled symbols) in many cases (i.e., near horizontal lines), or even better than BP in several cases (i.e. lines going towards the lower right corner), with no increase memory usage as more layers are used. Lines towards the lower right corner (SF outperforms BP), and upper right corner (BP outperforms SF); horizontal lines (both perform the same).
20

---

Published as a conference paper at ICLR 2024
Table 3: Node classification performance of GCN, SAGE, and GAT' averaged over five runs on five real-world graphs, obtained with different number of layers, and using different learning methods. The best results are in bold font, and the best results among the forward learning methods are red.
(a) AMAZON
Accuracy (t)
Method
# Layers==
Layers=2
# LLyers=3
# Layers=4
Backpropagation-GCN
45.83=0.4
64.88=16.1
91.03=133
90.21
SymBa-GCN
60.80 1.4
6..86.1.3
60.95-:11
61.06:1:22
CaFo-GCN
42.52=1.2
61.02=8.9
60.82-8.9
60.69
PEPITA-GCN
13.87=688
21.8244.9
19.80=3.5
15.57:4.00
FF-LA-GCN (Sec
3.1
91.20=1.0
18.54=10.4
18.08=10.5
49.74=10.5
FF-VN-GCN Sec
3.1
22.69=166
26.43:44.5
22.92-3.8
20.84+3.3
SI-GGC (Sec
93.67-0.6
93.73-40.4
93.48-022
93.48=0.3
To-Top-To-Loss-deee (Sec
3.3
93.76=0.5
83.18#3.8
60.43=6.9
47.40=2.9
To-pTpppTooIpuu----e (Sec
93.19-0.6
92.88.0.7
92.16-0.7
92.30.0.6
Backpropagation-SAG
33.79-6.9
91.58:1099
04.
93.27
SymBa-SAGE
91.87=11
9187=009
91.75=0.9
91.71
CaRo-SAGE
35.86=5.6
36.07
36.16-5.8
33.
PEPITA-SAGE
14.54=7.3
23.56=14
24.59-04
17.49=4.6
FF-LA-SAGE (Seo
88.95=1.1
88.80
88.80=13
88.75
FF-VN-SAGE (See
76.44=2.3
80.93=1.9
8244=220
82.80
SF-SAGE (See 3.2
92.90=0.3
93.63=004
93.70=044
93.
S1-Top--To-
87.62-1.9
84.22
ST-ToopTTo-nput-SSae (Sed
92.18=099
92.99=0.9
93.28=0.8
92.92
Backpropagation-GAT
37.63-5.4
16.54=16.1
90.33-1.7
89.33--0.7
SymBa-GAT
92.12-0.5
92.30:0.5
92.30=0.5
92.
CaFo-GAT
44.88=1.6
45.33=1.5
10.86=11.8
52.10=108 10.8
PEPITA-GAT
17.19-6.1
16.27
22.80=7.4
18.94
PE-LA-GAT (Sec
90.93-122
39.28-16.0
19.05-16.00
39.74=16.0
FF-VN-GAT (Se
47.65+28.5
49.40=27.9
50.20=28.3
50.16=28.6
SF-GAT (See 32
60.07=144
87.02=0.7
90.18=0.6
90.68=0.8
SI-Top-To- To-p--TT-sss---- Los- GAT (Sec 3.3
63.14=3.1
90.26:1.11
90.73-1.1
91.46
ST-Top-To--Iput (Sec
89.96=099
90.55=1.6
91.531111
91.10=1.0
(b) GITHUB
Accuracy
Method
Layers=1
Layens=2
Layers=3
# Layers=4
Backpropagation-GCN
74.16=0.5
74.17=0.5
78.76-6.0
86.34:0.2
SymBa-GCN GCN
7..16:0.55
74.19:10.
1..46-0.3
74.47.104
CaFo-GCN GCN
74.17:10.5
PEPITA-GCN
55.111:22.4
74.14:0.55
0.5
74.16:0.5
FF-LA- GCN (Sec))
3.1
80.90
77.1911.7
1.1
74.91
FF-VN-GCN
74.16=0.5
63.87=13.4
19.29-19.11
61.67155.8
SF-GCN (See 3.2
85.74=0.2
85.90.1.22
85.86=0.2
8..
To-Top--T--Los--G- (Sec
3.3
85.91=0.2
76.56=115
74.17=0.5
74.17=0.5
TE-Too-TT-llppt--CC (Sec 5.3
80.66=0.4
8..69:100
83.94=0.6
83.92.+0.5
Backpropggation
74.17=0.5
86.06.10.3
86.57
16.71.0.44
SymBa-Se SymBa-SAGE SAGE
86.44-0.4
16.5
86.37-0.4
86.4
CaFo-SAGE
74.16=0.5
74.17-0.5
74.17=0.5
74.17=0.5
PEPITA-SAGE
26.53.122.1
74.1610.5
74.17-0.5
74.40.0.9
FF-LA-SAGE (Sec
3.1
76.21=3.0
75.58=2255
15.11=1.4
7..67+0.8
FF-VN- SAGE (Sec
85.66=0.1
85.88.10.3
86.02=0.4
86.09-0.4
SF-SAGE (See (Sec
32
80.27=14
81.96=0.9
82.92-0.7
83.54=0.6
ST-Top-To-Los--AAAe (Soo
79.80=0.8
12.15
1.
83.03
Sollpp--T--ppu-SAAAAC (Sec 3
81,98=04
84.96=0.33
84.70=0.7
80-00.
Backpropagation
7..17-0.5
74.17.10.5
74.17
86.24.1.2
SymBa-GAT
86.76=0.4
86.67=0.4
86.66=0.4
8..32:10.6
CaFo-GAT
7..16=0055
74.17:1.55
0..17
74.17.10.5
PEPITA-GAT
63.50+18.4
75.26:22.1
74.20-0.5
74.22+0.6
FF-LA-GAT (Sec
82.14=0.3
79.59
79.7=2.11
80.18:2..6
FF-VN-GAT (Sed
3.T
85.97=0.3
86.29+0.3
86.70=0.3
86.64+0.3
SF-GAT (See
74.29=0.7
83.01=0.88
83.83-0.6
84.71
SI-Top-To- Los (Sec 3.3
74.27-0.6
79.8821.8
82.87-122
11.45+1.7
Sg-Top-To-Input--G (See
79.82=100
85.44
85.57=0.4
85.69
21

---

Published as a conference paper at ICLR 2024
Table 3: (Continued from the previous table) Node classification performance of GCN, SAGE, and GAT averaged over five runs on five real-world graphs, obtained with a different number of layers, and using different learning methods. The best results are in bold font, and the best results among the forward learning methods are in red.
(c) CITESEER
Accuracy
Method
Layers=1
Layers=2
# #Lyerrs=
# Layers=4
Backpropagation-GG
84.28=3.0
94.28
94.89=0.8
94.87=0.5
SymBa-GCN
87.
88.061:.
88.01=1.4
88.01
CaRo-GCN
68.25=2.2
69.79=0.5
69.31=0.7
1.
PEPITA-GCN
19.22=4.5
19.53#34
17.09=177
19.31
FI-LA-GCN (Sec
95.39=044
95.86=0077
98.89=0.6
90.
FF-VN-GCN
43.22+14.22
53.31=10.7
12.81=10.4
54.28-11.2
SF-GCN (See 3.2
92.60=0.7
94.18=0.7
94.33=0.7
94.6610.6
To-Top-To-Los--GGG (Sec
3.3
92.62=0.4
94.11=0.5
94,56=0.5
94.02=0.7
SF-Top-To-nnput-GCN (Sec
5.5
94.30=0.4
94.78:0.22
94.87=0.5
94.96=0.7
Backpropagation-AAGE
87.28=0.9
94.56.0.8
95.20=0.7
9..20.1..7
SymBa-SAGiB
88.53.0.9
91.99
93.85-0.4
93.76.10.5
CaFo-SAGE
61.70=2.0
61.682.0
60.99-2.2
61.02+3.2
PEPITA-SAGE
20.454.0
21.8014.
19.03-3.1
18.13117
FF-LA-SAGE (Sec
3.1
95.41=0.7
95.53=0.6
95.58=0.8
95.63=0.7
FF-VN-SAGE (Sec
89.95=100
93.85=0.6
95.39=0.6
95.56.0.7
SF-SAGE (See
79.88-211
89.602188
92.27=1.1
92.74+1.0
To-Top-To-Los--AAeee
79.39-1.7
92.51=0.7
94.02=0.8
93.43=0.9
To-TTop-To-Inpu--SAAA (Sec
90.57=0.5
93.90=0055
93.40=0.7
93.22=0.77
Backpropagation
1..17
94.1810.77
1.0
94.49
SymBa-GAT
91.51=0.8
95.01=1.1
95.25=0.9
95.
CaRo-GAT
73.29=007
72.60
0.
7..1
PEPITA-GAT
15.65-2.8
16.93=15
17.94=32
18.79=3.3
FF-LA-GAT (..c
95.34=0.4
95.48=0.6
95.27=0.5
95.37=0.6
FF-VN-GAT (See
90.99-1.2
94,04=1.0
95.48=0.6
95.
SF-GAT (See 32
82.06=244
93.33=00.9
93.78-1.1
94.21+100
S1-Top-To-LLor To-TTo-TT-LLos--O (Sec
81.21-2.6
91.9620.8
92.0-0077
91.89-0.8
ST-TTop-To-iput-GAA (See
91.16=0.8
94.02=0.7
93.95=0.8
93.83=0.6
(d) PUBMED
Accuracy (t)
Method
Layers=1
Layers=2
Layers=3
Layers=4
Backpropugation-GGGC
63.821
86.74.0.3
86.9820.7
86.01 =0.5
SymBa-GCN
82.1310.8
82.10.0.77
82.0610.7
82.04.1.77
CaFo-GCN
69.9011.4 1.4
69.1612.1
69.78122
68.89
PEPITA-GCN
33.
39.66:1..7
42.17424
41.91
BF-LA-GCN (Sec
0..
83.57.004
83.88
83.97.1..3
FF-YN-GCN (Sec
51.69
$2.89
1.196=1.1
54.07=44.0
SF-GCN (Sec 3.2
86.98
87.61
8..66:0.4
87.61:104
SF-Top-Toe Te-Top-Too-Lsseee To--os---oGG-- (Sec
3
86.82+0.5
86.59=0.66
86.17=0.3
85.69=008
SF-Top- To-Input- To-Input-GCN (Sec
3.5
77.91
83.36=00.7
84.01
04.55
Backpeopagation-AAG
70.66.1..3
88.42.10.5
89.11:0.4
88.691104
SymBa-SAGF:
88.27-10.6
88.33
88.4
88.30.10.3
CaFo-SAGE
62.66+2.8
63.69.3.11
63.01
63.34:13.2
DEPITA-SAGI
33.24.15.3
40.19
1.7
40.62.12.0
FF-LA-SAGE (Sec
3.1
83.54=0.5
83.48:20.6
53.5410.6
83.50-00.6
FE-VN-SAGE (Sec
3.T
88.83
88.93-10.5
08.
88.8
SF-SAGE (Sec
78.49=
81.33
82.87=0.7
83.54
SF-Top-To-Losse To-Loss-SAGE (Sec
3.3
78.23=
80.84=0.7
81.
82.07=0.4
SF-Top To- Teapp-To---tutSS-i Input-SAGE (Sec
3.3
79.53-0.5
83.00=20.6
13.5
83,32-10.5
Backpropagation GGT
63.68.10.5
80.4
85.37.10.6 10.6
SymBa-GAT
88.93-0.4
89.47-0.22
89.60-0.4
89,53-10.5
CaFo-GAT
0.9
73.03:10.8
1.4
71.61.1.6
PEPITA-GAT
32.23=8..4
35.69
41.99=2.2
39.28 1.7
FF-LA-GAT (Soc
3.1
8..62=0.4
83.55
83.40-0..4
83.43-10.4
FF-VN-GAN See
88.22.100
88.77.10.6
88.86.00.6
88.86110.6
SF-GAT (See 3.2
64.41=7.4
78.41=2..6
12.57=1111 81.341155
SF-Top GAT (See
63.641665
79.11:15
81.7110.9
82.15
Te-TTo-TTo-Inut--ee (Sec
77.24=4.1
83.34=0.7
83.53=0.5
83,61
22

---

Published as a conference paper at ICLR 2024
Table (Continued from the previous table) Node classification performance of GCN, SAGE, and GAT averaged over five runs on five real-world graphs, obtained with a different number of layers, and using different learning methods. The best results are in bold font, and the best results among the forward learning methods are in red.
(e) CORAML
Accuracy (1)
Method
Layers=1
# Layers=2
# Layers=3
Luyers=4
Backpropagation
31.72=4.8
86.84=1.0
88.65=14
SymBa-GCN
82.94
82.97
83.37
83.34 1.9
CaRo-GCN
31.45
30.28
29.65 29.65
28.95
PEPITA-GCN
13.19=1.9
16.76=5.8
15.63+2..0
18.30=8.0
FF-LA-GCN (Sec
81.74
63.77:12.7
63.71
64.54.1..0
FF-VN-GCN (Sec
18.61=199
28.11=3.2
27.25
25.81=4.6
SF-GCN (Sec 3.2
87.75 1.4
87.95=144
18.15
88.48
Te-Top-To-Loss-OOee (Sec 3.3
87.55-14
84.07=0.9
85.64+1.6
84.67=2.6
To-To--To-n (Sec
87.05 1.4
1.
87.05
87.68=1.3
Backpropagation-S..
4.087=5.0
88.75
88.21
85.98
SymBa SAGE
8..45-0.8
CaFo-SAGE
28.98=2.1
28.71=2.1
1..
28.95 1.6
PEPITA-SAGE
13.39
24.31
17.13
19.13 8.9
FF-LA- (Sec 3.1
85.41
85.34-0.88
85.08 1.3
85.18 1.3
FF-VN-SAGE (See
86.01=0.7
1.
87.85
87.71 1.3
SF-SAGE (See
3.2
85.14
87.35
17.61
87.55
SE-Top-To-Loss--S (Sec 3.3
84.77=14
85.94=1.3
85 1.
85.31= 1.8
SF-Top-To- To-Input-SAAGG (Sec
86.08=24
86.58:1122
86.84-1.2 1.2
86.84 1.1
Backpropagation GAT
33.82
80.97
87.58 1.6
86.04 1.7
SymBa-GAT
86.68 1.3
87.51=1.2
80.6
0.6
CaFo-GAT
3.7
32.95-14.9
35.9
36.29:11.99
PEPITA-GAR
15.13
6..
15.13
16.66-1..7
FF-LA-GAT (Sec 3.1
82.67=122
61.0=664
63.57-4..9
63.24=4.1
EF-VN-GAT (Sec
84.61 1.5
85.64.1..5
86.4812.1
86.74
SF-GAT (Sec
66.81=2.0
83.17=2.0
84.34=19
85.14=2.2 83.57=1.5
ST-TopToo-Loss Loss-GAT (Sec 3.3)
2.9
79.50=2.4
84
SF-Top-TooInput-GAA (Sec 3.3
83.27=2.0
84.51=2.1
85.28=1.7
84.84=2.8
Table 4: GPU memory usage (in MB) for node classification. The best results are in bold font, and the best results among the forward learning methods are red.
(a) CITESEER
GPU Memory Usage (3)
Method
Layers=l
Layers=2
Layers=3
Layers=d
Backpropagationn-GGr
0.52
4.01
6.61
9.21
SymBa-GCN
30.62
30.62
30.62
CaFo-GCN
3.40
3,40
3.40
3,40
PEPITA-GCK
16.56
22.04
25.28
29.70
FF-LA- GCN (Sec
24.50
25.20
25.20
25.20
FF- GCN (Sec
25.29
SF-GCN (Sec 3.2
15.51
15.51
15.51
15.51
SF-Top Topp-T--Looe GCN--os--GC GCN (Sec
33
11.10
13.17
15.24
17.11
SF-Top- To-Input- (S.
18.79
19.12
Backpropugation-SAAG. SAGE
10.01
16.42
21.07
25.72
SymBa-SAGE:
89.40
89.40
89.40
89.40
CaFo-SAGE
4.58
4.58
DEPITA-SAGE
25.99
34,26
40.29
46.54
EF-LA-SAGE
FF-LA SAGE (Sec
3.1
84.54
84,54
84,54
84.54
FF-VN-SAGE (Sec
3.T
83.45
83.45
83.45
83.45
SF-SAGE (See
26.03
26.03
26.03
26.03
SF-Top-To-LL Toopp-To-LLs--AACA- SAGE (Sec
3.3
23.34
25.40
26.93
29.00
To-oo--TT-ppu----e (Soc
3.3
28.30
33.05
33.79
33,52
Backpropagation-coe
3.81
19.14
32.07
44.30
SymBa-GiA
111.56
111.87
111.87
111.87
CaFo-GAT
3.41
3.41
3.41
3.41
PEPITA GAT
32.33
63.22
FF-LA-GAT (Soc
3.1
88.15
88.18
88.60
88,60
EF-VN-GAT See
106.06
106.54
106.54
106.54
SF-GAT (See
28.13
28.13
28.13
28.13
SF-Top-To- Tooop-To-Lssss GAT (See
3.3
24.39
26.27
28.53
30.60
SF-Top To-Input- (Sec (.3
30.16
32.56
32.46
32.56
23

---

Published as a conference paper at ICLR 2024
Table 4: (Continued from the previous table) GPU memory usage (in MB) for node classification.
The best results are in bold font, and the best results among the forward learning methods are red.
(b) PUBMED
GPU Memory Usage (U)
Method
Layers=1
Layers=2
# Layers=3
Layers=4
BackprougationnCGC
2.56
15.22
27.16
39.11
SymBa-GCh
62.58
63.22
63.93
63.96
CaFo-GCN
PEPITA-GCN
64.82
84.82
102.92
121.62
FF-LA ACN (Sec
5.1
$5.26
$5.26
$5.26
55.26
FF-YN-GCN (Sec
56.37
56.37
57.72
57.75
SF-GCN (Sec
3.2
$1.72
57.72
Te-TTo-TT-LLs----
(Sec
3.3
41.14
50.77
60.39
69.59
SF-Top-To-Iopure To-Ioput-GCN (Sec
3.3
58.71
69.43
69.72
69.95
Backpopagation-SAAEE
38.12
59.38
79.21
99.04
SymBa-SAGiP
168.62
168.62
168.62
168.62
CaFo-SAGE
11.98
11.98
11.98
11.98
PEPITA-SACE SAGE
126.34
152.01
177.67
FE-LA-SAGE (Sec
163.08
163.08
163.08
163,08
HE-VN-SAGE (Sec
3.
162.41
162.41
162.41
162.41
SF-SAGE (See
93.77
93.77
93.77
93.77
T--TT-TT--Los--- (Sec
3.3
76.98
86.61
105.87
SF-Top-To Le-Inppt--SAAA (Sec
95.97
116.35
116.58
116.87
Backpropagation GAT
261.51
SymBa-GAT
316.04
316.04
316.04
316.04
CaFo-GAT
11.00
11.75
11.75
11.75
PEPITA GAT
68.37
156.77
243.17
333.78
EF-LA-GAT (See3.1
265.23
265.23
265.25
265.25
FF-VN-GAY (Sec
309.59
309.62
309.62
309.62
SF-GAT (See 3.2
141.61
141.61
141.61
141.61
SF-Top- TosLoss GAT (Sec
125.81
135.07
145.07
154.15
SF-Top-To-Inpee To-Top-TT-Ipp--GAA
143.29
154.68
154.22
154.38
(c) AMAZON
GPU Memory Usage (3)
Method
Layers=l
Layers=2
Layers=3
Layerss4
Buckpropugation-GGNN
5.20
15.01
23.69
32,36
SymBa-GCN
106.78
106.78
106.78
106.78
CaFo-GCN
5.55
5,55
5.56
5,56
PEPITA-GCN
41.12
53.31
64,39
75,46
FF-LA GCN (Sec
88.53
89.43
89.43
89.43
HE-YN-GCN
SF-GCN (Sec 3.2
34.81
34.81
34.81
34.81
SF-Top To Loss GCN (Sec
1.4
21.79
26.08
29.27
34.11
SF-Top- To-Input- (Sec
36.30
40.89
40.77
42.11
Buckpropagation-SAGE
22.35
32.61
40.61
48.61
SymBa-SAGH
243.25
243:25
243.25
243.25
CaFo-SAGE
7.00
7.00
PEPITA SAGE
$8.43
69.47
79.58
89.69
FE-LA S-AGE (Sec
3.1
230.49
230.49
230.49
230.49
FF- VN-SAGE (Sec
S.T
228.5
228.54
228.54
228.54
SF-SAGE
$3.15
SF-Top-To-La Po-To--TT-LLs-S... SAGL (Sec
3.3
41.54
45.28
49.02
53,68
SF-Top To- fo-Input InputtSAGE SAGE (Sec
5.3
$6.36
64.81
65.04
64.81
Backpropagation GAT
$1.66
199.31
346.07
492.84
SymBa-GAT
1255.72
1255.72
1255.72
1255.72
CaFo-GAT
5.55
5.55
5.55
PEPITA GAT
DE-LA-GAT (Soc
3.1
1193.36
1193.36
1193.36
1193.36
FF-VN GAT (Sec
1241
1241.26
1241.26
1241.26
SF-GAT (See
3.2
178.39
178.39
178.39
178.39
SF-Top Loss GAT (See
3
174.34
178.02
SF-Top-To-Inge Inputt--T-T (Seo
3.5
181.03
185.83
185.01
185.83
24

---

Published as a conference paper at ICLR 2024
Table 4: (Continued from the previous table) GPU memory usage (in MB) for node classification.
The best results are in bold font, and the best results among the forward learning methods are red.
(d) GITHUB
GPU Memory Usage (U)
Method
# Layens=1
Layers=2
# Layers=3
Layers=4
BackprougationnCGCN
12.41
42.81
73.22
103.62
SymBa-GCh
87.01
87.01
87.01
CaFo-GCN
19.22
PEPITA-GCN
42.78
84.96
127.15
169.33
FF-LA MCN (Sec
5.1
84.49
84.49
84.49
84.49
FF-YN-GCN (Sec
87.01
87.01
87.01
SF-GCN (Sec
3.2
62.64
62.64
62.64
62.64
SF-Top-To-Lose To-p-TT--L---GOO (See
3.
81.30
99.71
118.12
136,00
SF-Top-To-Ippure To-Ioput-GCN (Sec
3.3
63.19
81.49
81.80
82.38
Backpopagation-SAAE
19.08
$6.54
94,00
131.46
SymBa-SAGiP
98.07
98.07
98.07
98.17
CaFo-SAGE
19.47
19.47
19.47
19.47
PEPITA-SACE SAGE
98.29
147.14
195.98
FE-LA-SAGE (Sec
3.1
98.65
98.65
98.65
98.65
FF-VN-S SAGE (Sec
TE-VN-SAGE
3.T
98.07
98.07
98.07
98.07
SF-SAGE (See
68.10
68.10
68.10
68.10
To-TT--T--Los--- (Sec
3.3
87.01
105.42
123.83
142.24
SF-Top-To Le-IIppt--SAAA (Sec
69.19
106.42
106.71
107.57
Backpropagation GAT
842.55
1228.98
SymBa-GAT
850.82
850.82
850.82
850.82
CaFo-GAT
19.22
19.22
19.22
19.22
PEPITA GAT
$5.10
453.16
851.37
1250.04
EF-LA-GAT (See (See3.1
796.80
797.53
797.53
797.53
FF-VN-GAY (Sec
850.83
850.83
850.83
850.83
SF-GAT (See 3.2
444.83
444.83
444.83
444.83
SF-Top- To-Loss GAT (Sec
463.50
481.91
500.32
$19.08
SF-Top-To-Inpee To-Top-TT-Ipp--GAA
445.10
463.59
463.59
463.59
(e) CORAML
GPU Memory Usage (3)
Method
Layers=l
Layers=2
Layers=3
Layersed
Buckpropugation-GGNN
0.83
7.99
10.07
12.58
SymBa-GCN
30.71
30.71
30.71
30.71
CaFo-GCN
7.22
7.61
7.61
7.61
PEPITA-GCN
55.80
63.90
66.92
69.93
FF-LA GCN (Sec
25.73
25.74
25.74
25.74
HE-YN-GCN
SF-GCN (Sec 3.2
41.56
41.56
41.56
41.56
SF-Top To Loss GCN (Sec
1.
19.66
19.25
20.56
22.24
SF-Top- To-Input- (Sec
3.3
49.47
49.16
52.56
Buckpropagation-SAGE
33.66
47.25
$1.49
54.93
SymBa-SAGH
263.63
263.63
263.63
263.63
CaFo-SAGE
12.84
PEPITA SAGE
87.37
96.56
100.67
105.23
FE-LA SAGGE (Sec
3.1
259.62
259.62
259.67
259.62
FF- VN-SAGE (Sec
S.T
258.92
258.92
258.92
SF-SAGE 3.2
79.71
SF-Top-To-Losse SAGE (Sec
3.3
61.16
62.87
64.64
66.04
SF-Top To- To-Input InputtSAGE SAGE (Sec
5.3
92.35
98.58
98.57
94,90
Backpropagation GAT
5.65
24.03
38.63
$2.80
SymBa-GAT
134,08
134.08
134,08
134,08
CaFo-GAT GAT
7.23
7.62
7.62
7.62
PEPITA GAT
76.84
DE-LA-GAT (Soc
3.1
110.94
111.28
111.28
111.28
PF-VN GAT (Sec
129.36
129.36
129.36
129.36
SF-GAT (See
3.2
55.75
55.75
55.75
SF-Top Loss GAT (See
3
33.53
38.37
SF-Top-To-Inng In--ttutGGAT (Seo
62.67
63.76
67.17
64.20
25

---

Published as a conference paper at ICLR 2024
F.2 LINK PREDICTION RESULTS
Figure 6 presents the link prediction accuracy (y-axis) vs. memory usage increase (x-axis) per graph neural networks.
pl the link prediction performance of GCN, SAGE, and GAT models averaged over five runs on five real-world graphs, obtained with a different number of layers, and using different learning approaches.
Table 6 presents the GPU memory usage (in MB) during the GNN training for link prediction, observed with different number of layers, and using different learning approaches.
A SF-GCN
OSF-GAT (SSE-SAGE
CiteSeer
CoraML
BP-GCN
BP-GAT BB--AGE
PubMed
100
100-
100-
Better
Better
Better
AUC (%)
Belfer
Better
Beller
ROCAUC
90
90
90
1.00X
1.25x
1.50x
1.75x
1.00x
1.25x
1,50x
1.75x
1.00x
1.25x
1,50x
1.75x
Better
Better
Better
ROCAUC AUC(()) (%)
90
Betfer
90
Better
90-
Better
GAT
80
80
80
1.0%
1.5x
2.0x
2.5x
3.0
1.0x
1.5x
2.0x
2.5x
3.0x
1.0x
1.5x
2.0
2.5x
3.0x
Better
Better
Better
ROC ROC AUC (%)
(%))
Better
Better
Befler
SAGE
80
80
70-
70
70-
1.0x
1.2x
1.4x
1.6x
1.0x
1.2x
1.4x
166
1.0x
1.2x
1.4x
1.6x
Mem usage relative to I-layer model
Mem usage relative to 1-layer model
Mem relative to I-layer model
(1) 2-layer models
(2) 3-layer models
(3) 4-layer models
Figure 6: Link prediction performance (y-axis) vs. memory usage increase (x-axis). The proposed forwle-fowward approach (SF, empty symbols) outperforms backpropagation (BP, filled symbols) (corresponding to most lines going towards the lower right corner), with no increase in memory usage more layers are used. Lines towards the lower right corner (SF outperforms BP), and upper right corner (BP outperforms SF); horizontal lines (both perform the same).
26

---

Published as a conference paper at ICLR 2024
Table 5: Link prediction performance of GCN, SAGE, and GAT averaged over five runs on five real-world graphs, obtained with a different number of layers, and using different learning methods. The best results are in bold font, and the best results among the forward learning methods are in red.
(a) CITESFER
ROC AUC (1)
Method
Layers=1
# Layers=2
# Iayers=
# Laaerrad
Backpropagation-GG
90.00:10.6
84.31:10.6
80,9311100
80.05:10.9
CaFo-GCN
81.35:+0.9
86,89+0077
87.60+0.7
88.17=0.7
BawardGNN-SSymB--GG
89.83-10.3
90.65:t1.1
90.09:1.99
90.65+1.11
ForvarGGNX-FF-GCN
85.11+0.7
91.9444.5
91.95+0.5
91,94+0.5
FowwarGNN--EE-GNN
89.32442.4
93.61+100
93.49:0.99
93.61=10.
ForwarGGNN-CI-TTopTT
92.15:40.6
88.3941.11
88.47-00.7
88.06=20166
Backpropagation
86.46.10.5
70.63
68.35.11.4
67.71111.5
CaFo-SAGE
73.42#1.1
65.64:14.0
60.20+2..0
1.
DawardGNN-SSymB-SS..
73.68:10.9
71.3113.00
71.34:1..00
71.31
ForvarGGNN-FF-SAGEE
85.31+0.4
85.01400.4
85.13+0.4
85.01+0.4
ForvarGONN-CE-SAGE
87.16:00.6
88.02:10.5
87,94110.5
88.02:1..5
75,51
79.31+0.5
80.15#0.99
79.28+0.3
Backpropagation GAT
88.42-10.5
84.83.0088
81.02.3..0
77,27-1099
CaFo-GAT
69.67:11.5
84.61::1.2
85.15:1.0
8..05
ForwardGNN- --T
85.16.10.4
86.4910..
86.4910.4
86,49
ForvardGNN-FF-GAN
78.14:55.5
89.64440.7
89.62:0.77
89,6440.7
ForvarGNNN-CE-GAT
87.68:10.4
90.22:10.6
90.12-10.5
90.22:10.6
88.21
84,98+00.7
84,20+11.0
85.06+0.44
(b) PURMED
ROC AUC (t)
Method
Layers=
Layers=2
Layers=3
#Layers=4
Backpropagation-GGN
91.83-0.33
88.90400.2
89.201+0.2
89.4140.2
CaFo-GCN
86.35
89,92+1.2
90.70::1.2
91.08+100
ForvardGiNN-Symee SCCN
96.20:10.2
95.56-10.5
95.74-10.3
95.56-111.5
ForvarKGNN-FF-GGNN
96.53-20.1
96.86:20.1
96.8720.11
96.86:20.1
CiwarGGNN-CF-COG
96.31.10.1
96.23.10.1
96.3110.11
ForwardKiNN-CF-TTpp-
96.8411.1
94.15:10.3
94.03:10.2
93.96.10.2
BaekpropagationSAAGE
59,3840.8
81.4140.5
80.81+1.0
77.06+2.4
CaFo-SAGE
73.47#1
68.174:11
66.224:1.5
6..95+110
BiraadGGNN-SymtBa--S
77.61
77.56-10.2
77.56+40.2
77.56-10.2
ForwarGGNN-FF-SAGE
82.92:40.5
82.6940.55
82.81
82,69-0.55
ForvardGNN-CFF- SAGE
87.07-40.4
87.07-10.4
86.46:104
85.51=1111
84.05::0.7
84.71+00.6
Backpropagation-GAT
81.36+00.6
78.85+40.4
78.5940.4
78.31+0.4
CaFo-GAT
77.25+10.7
81.92-+3.1
81.7843.22
82.06-13.0
CrrvarGNNN-SymBnee
91.96:20.3
92.01+-0.3
91.97-+0.3
92.01=0033
ForwarGGNN-FF-GAT
92.33-10.3
92.36:10.3
92.36:10.3
92,36:10.3
ForvardGGN--CE.GAT
89.22-40.3
90.36:10.2
90.53-40.2
90.36-10.2
Yavaar/GNN--CE--ppp---tppp.oo
92.38:40.2
89.14410.2
89.71
89.30-10.4
27

---

Published as a conference paper at ICLR 2024
Table 5: (Continued from the previous table) Link prediction performance of GCN, SAGE, and GAT averaged over five rums on five real-world graphs, obtained with a different number of layers, and using different learning methods. The best results in bold font, and the best results among the forward learning methods are red.
AMAZON
ROC AUC (t)
Method
Layers=1
# Layers=2
# #Lyers=3
# Layers=4
BackpropagatonnGGN
82,41
81.38.1-0.3
88.6910.1
89.47:10.2
CaFo-GCN
81.11#1.1
94.02:40.6
94.05400.6
94.00=00.6
82.26:77.4
83,95:6.2
84.0646.4
83.9546.2
ForvardGNN-FF--GG
96.18.10.2
96.16.10.2
96.17.0022
96.16::0.2
ForvarGGNN-CE-GGN,
93.84+0.2
95.49440.2
95,38+0.2
95,49+0.22
97.13:10.1
96.81:10.3
96.32:10.6
95.84:10.7
Backpropagation-SGGE
70.73.10.4
83.43.10.1
86.10.0.2
87.07.1..5
CaFo-SAGE
70.93-11.0
75.30-1199
77.2811.66
78,20-11.7
ForwardGNN- SyY.BGGA-
96.25:10.1
96.46:10.1
96.4610.1
96.46::..1
ForwardGNNN FF--EFF-EEE
94.07.10.5
94.94.10.6
94,86.0.55
94,9410.6
ForwardGNN-CE-S..
93,28-10.1
94.96:10.1
94.88+0.1
94,96+t0.1
ForwardGNN-CE-Tog To-Inpur-SAGE
97.13:10.1
95.11110.6
95.15:10.6
15.12:00.6
Backpnpagatton GAT GAT
71.67.10.8
68.54111.2
67.02.L1.8
66.50.1
CaFo-GAT
73.73-11.1
88.0943.99
90.58442.1
91,38-2.33
ForwardGNN SYNNNN---SBBGGTTTT
97.09110.1
94,86.14.4
97.0110.1
94.86.14.4
ForwardGNN GAT
94.10:1..11
94.1011..1
DowardGNN-CE-GG.
92.311100.5
94.34:1:0.2
94.36110.2
94.3410.2
ForwardGNN-CE: -op---p----------- To-Input GAT
96.90.10.11
96.97.10.6
93.02.10.99
93.41110.6
(d) CORAMI.
ROC AUC (1)
Method
# Layers=1
Layers=2
# Layers=3
Layerse4
BackpopagationGGNN
82.20:10.3
79.86:0.77
79.71 79.
CaRo-GCN
73.23++0.6
83.88=3.2
85.26=2.7
85 85.41=288
FowaadGNN-SymB--GGY
87.68+55.3
87.5 87.52=5..6
88.02=5.8
87.52=5.6
ForwandGNN-HF-GOON
91.91=1.3
92.46-20.9
92.39-1.1
92.46-0.9
ForwadGNN-CE-GGNN
91.77=1033
93.30-20.3
93.26=0.3
93.30=0.33
93.64--0.22
91.03-10.6
91.57-0.44
91.53-0099
Backpooaaation-SAGE
65.48:10.6
76.75::2.0
74.65:10.5
74.2111.88
CaFo-SAGE
63.28::0.9
67.46=2.3
69.46=0.7
70.67
ForvadONN.SymBe--...
71.12+8.7
78.22+
7.
18
ForvandGNN--FFSAGEE
89.06-0055
38.96-20.5
89.00=0.5
ForwadGNN-CE-SAGE
90.36::0.6
91.96=0055
91.93=0.4
91,9620.5
FooxintGNNN-CC---p--To--pppp
89.39-10.3
89.72-00.5
89.32-0..5
89.18204
Backpopagation-GATT
76.751008
1.0
69.61.10.3
66.041107
CaFo-GAT
7.8
83.29--2.3
84.71=
1.106=1.1
ForvadGNN-SymB.e.
80.11: 10.8
85.03-33.6
85.041:..6
85.03
ForkadGNNN-FFGGAT
89 09.42.0..8
90.45-00.9
90.45-009
90
ForvadGNN-CE-GATT
88.96:10.4
90.98=0.4
90.96:0055
90.98:0.4
91.07=00.4
88.87-03
89 09.90=0..5
89.70=0.6
28

---

Published as a conference paper at ICLR 2024
Table 5: (Continued from the previous table) Link prediction performance of GCN, SAGE, and GAT averaged over five runs on five real-world graphs, obtained with a different number of layers, and using different learning methods. The best results are in bold font, and the best results among the forward learning methods are red.
(e) GITHUB
ROC AUC
Method
Layers=1
Layers=2
Layers=
Layersed
Backpopugation-CNN
89.60=0.1
91.36:10.1
91.71=0.1
91.7940.11
CaFo-GCN
91.10::0.1
91.4240.1
91.32:102
91.24-+0.2
SowardGNN-SymBu--Co
11.7
81.51+1588
81.51
81.51
ForwanttiNN-Fi-Ge
93.27=00.1
93.4840000
93.48:00.0
93.4840.00
JerwanGGNN-CE-GGN
91.87=0.1
91.30-20.1
91 15.1100.11
91.30
ForwardGNNN -o-T----------------------------
To-Input Input- GCN
94.0910.1
91.82.10.4
91.44.0.4
91.64.10.5
Backpopagation-AAGE
83.85:0.11
84.45:10,2
85.03:10..8
86.90+1..2
CaFo-SAGE
76.63++0.9
77.21+1.0
77.29-10.9
77.37++0.9
93.88:0.11
92.55+2.3
93.69+00.0
92.55422.3
ForvandGNN-FF-SAGGEE
89.64:00.3
89.62-20.3
89.63-20.3
89.62400.3
ForvandGNN-CE-SAGE
87.17=00.3
88,35-+0.3
08.29:00.4
88,354+0.3
88.93-10.6
14.45+1.3
84.43-10.9
83.88+0.77
Backpoopagation-GAT
78.75:10.4
73.41
73.94:11.4
68.26:10.5
CaFo-GAT
81.88:+0.9
82.66+00.6
83.09:10.7
83.12-40.7
ForwardGNN- F
SyMBa-GAT
92.99:10.2
92.53-10.2
92.49:10.2
92.53+10.2
ForwardGNN-Fi-GAT
88, 18:
88.63-20.8
88.63-00.8
88.63+00.8
ForwarGGNN-CE-GAT
82.72:+0.4
10:10.5
82.13:+0.6
82.10
ForwarGGNN-CE. Tor
89.59:10.2
80.91
81.39:10.3
81.81+0.44
Table GPU memory usage (in MB) for link prediction. The best results are in bold font, and the best results among the forward learning methods are in red.
(a) CITESEER
GPU Memory Usage (1)
Method
Layerss 1
# Layers=2
Layers=3
Layerse4
Backpropagation GCN
10.16
13.48
16.41
18.14
CaFo-GCN
13.19
13.19
13.24
13.24
BowardGNN-SyyBBa-.CO
17.82
18.04
18.04
18.04
ForvarGGNN-FF-GEY
17.84
18.06
18.06
18.06
ForvarGGNN-CE-GCN
14.47
14.47
14.47
14.47
21.64
24.79
24.61
25.01
Backpropagation SAGE
22.38
26.28
30.13
34.78
CaFo-SAGE
13.44
13.44
13.62
13.62
FurvardGNN-SymmBB-SAAA SymBB-AAGBBBB
29.07
29.07
29.07
29.07
ForwardGNN-FF-SACC
29.09
29.09
29.09
29.09
ForwardGNN-CE.
24.17
24.17
24.17
24.17
32.93
39.41
39.78
39.03
Backpmipagation GAT
18.39
30.70
41.60
53.16
CaFo-GAT
13.20
13.20
13.20
13.20
ForwardGNNN SymBa-Gaa GAT
25.66
26.23
26.23
26.23
ForwardGNN-FF-GAN
25.68
26.25
26.25
26.25
ForwardGNN
21.91
21.91
21.91
21.91
TarvarOGNN--CE--oo-----tooo
31.61
33.78
33.03
32.89
29

---

Published as a conference paper at ICLR 2024
Table 6: (Continued from the previous table) GPU memory usage (in MB) for link prediction. The best results are in bold font, and the best results among the forward learning methods are in red.
(b) PUBMED
GPU Memory Usage
Method
Layers=1
# Layers=2
LLayer=3
*Layer=44
Backpropagation-GGN
68.44
79.70
90.37
103.04
CaFo-GCN
76.86
76.86
76.86
76.86
BawaadGGNN-SSyBBa.cc
123.38
123.38
ForwardGNN-FF
123.59
123.59
123.59
123.59
SowarGGNN-CF-COO
95.39
95.39
95,39
95.39
ForwardGNN-CE- --A.-C---To-----n----------- To-Inpur-GCN
106.89
118.13
118.13
118.05
Backpropogaton--AGE
104.82
124.65
144.48
164.31
CaFo-SAGE:
77.76
77.76
77.76
77.76
BorvarGGNN-Syymu-SS.
161.00
161.00
ForwardGNN FF SAGE
161.22
161.22
161.22
161.22
DorvarGGNNCCF-SAM
133.19
133.19
133.
133.19
ForwardGNN-CE Top To-Inpur-SAGE
144.99
166.28
166.36
165.61
Backpropagation-GAT
121.26
184.54
249.24
313.18
CaFo-GAT
76.11
76.11
76.86
76.86
TorvarGGNN-SymBa-GGG
176.70
176.70
176.70
ForwardGNNN FF-GAA
176.92
176.92
176.92
176.92
ForvardGNN-CE-GG
148.60
148.60
148.60
148.60
ForwardGNN Co- To-Inpur GAT
160.54
170.59
171.04
170.29
(c) AMAZON
GPU Memory Usage (1)
Method
Layers=1
# Layers=2
Layers=3
# Layers=4
BackpropagatonnGGNN
157.68
164.72
171.76
178.80
CaFo-GCN
158.62
158.62
158.62
158.62
ForvardGNN-SymBBeee
307.27
307.27
307.27
307.27
ForwardGNN-FF-GGO
307.85
307.85
307.85
307.85
ForvarGGNN-CE-GON
232.11
232.55
232.55
232.55
TawardGNN--CE-TooTTo--n
182.17
186.42
186.42
186.47
Backpropagation
177.82
185.83
193.83
201.83
CaFo-SAGE
160.07
160.07
160.07
160.07
326.97
326,97
326.97
326.97
ForwardGNN FF-S
327.55
327.55
327.55
327.55
ForvarGGNN-CE-SAGE
252.25
252.25
252.25
252.25
ForwarGNNN-CE-Top SAGE
203.32
212.05
212.05
212.05
Backpropggttio GAT GAT
249.88
348.22
446.55
$44.89
CaFo-GAT
158.62
158.62
158.62
158.79
ForwardGNN SyBAAT
400.60
400.60
400.60
400.60
ForwardGNN CEA-GGNN------AAAA GAT
ForvardGNN-CE-GAN
324.31
324.31
324.31
324.31
ForwardGNN-CE -aE-NN---E--------- TosoopTTo-ngeeeee Gn-T
273.61
277.97
277.97
277.86
30

---

Published as a conference paper at ICLR 2024
Table 6: (Continued from the previous table) GPU memory usage (in MB) for link prediction. The best results are in bold font, and the best results among the forward learning methods are in red.
(d) CORAML
GPU Memory Usage
Method
Layers=1 1
# Layers=2
Layers=3
*Layer=44
Backpropegation-GGN
17.58
19.55
21.65
23.62
CaFo-GCN
19.92
19.92
19.92
19.92
BawardGGNN-SSyBB..cc
ForwardGNN-FF
28.46
28.46
28.46
28.46
CorwardGNN-CF-oo CE-GCN
22.68
22.68
22.68
22.68
ForwardGNN-CE- CIA---CE-Too-----p---GG---- To-Inpur-GCN
61.34
61.34
61.36
62.08
Backpropogatin--AGE
55.84
59,28
63.43
66.87
CaFo-SAGE:
24.66
25.00
25.00
25.00
JowvarGGNN-Symmu-SS.
66.06
66.06
ForwardGNN FF SAGE
66.10
66. 10
66.10
66. 10
DorvarGGNNCCF-SAM
60.94
60.94
60.94
60.94
ForwardGNN-CE Top To-Inpur-SAGE
100.92
107.69
107.72
104.29
Backpropagation-GAT
26.47
37.33
48,19
59.05
CaFo-GAT
19.05
19.05
19.05
19.05
CorvarGGNN-SymBa-GGG
37.14
37.14
37.68
ForwardGNN- FF-GAA
37.18
37.18
37.72
38.13 13
ForvarGGNN-CE-GG
32.18
32.18
32,18
32.18
ForwardGNN CE: Co- To-Inpur- Inpur-Gal GAT
65.54
68.17
68.17
67.25
(e) GITHUB
GPU Memory Usage (1)
Method
Layers=1
# Layers=2
Layers=3
# Layers=4
BackpropagatonnGGNN
390.25
416.72
442.69
469.09
CaFo-GCN
400.17
400.17
400.17
400.17
ForvardGNN-SymBBoee
751.07
751.07
751.07
751.07
ForwardGNN-FF-GGO
752.48
752.48
752.48
752.48
ForvarGGNN-CE-GON
569.88
569.88
569.88
569.88
FawardGNN--CE-TooTTo--n GCN
409.54
428.44
428.48
428.48
Backpropagation-SAGG
400.17
437.63
475.09
$12.55
CaFo-SAGE
401.16
401.16
401.16
401.16
SAGG
762.69
762.69
762.69
762.69
ForwardGNN FF-SAE
764.10
764.10
764.10
764.10
ForvarGGNN-CE-SAGE
581.54
581.54
581,54
581.54
ForwarGNN--CE-Top SA-E
420.49
459.05
459.05
459,23
Backpropggttio GAT GAT
633,29
901.83
1171.35
1441.82
CaFo-GAT
400.17
400.17
400.17
400.17
ForwardGNN SanBAT
994.10
994.10
994.10
994.10
ForwardGNN CFA-GGNN--FF-GAAAA GAT
995.52
995.52
ForvardGNN-CE-GAN
812.95
812.95
812.95
812.95
ForwardGNN-CE. -o--T-------------- To- lo-pnput-GAT
652.58
671.37
671.37
671.37
31

---

Published as a conference paper at ICLR 2024
F.3 TRAINING TIME
To evaluate the training efficiency of different learning approaches, we report in Table 7 the time taken to train GNNs for 100 epochs for node classification on GITHUB and CITESEER, with varying number of GNN layers. From T we make the following observations.
The proposed forward learning approaches Sections 3.1 and 3.2 train GCN and SAGE models much faster than backpropagation (up to 9 times when using four layers). With these greedy forward learning approaches, such as those presented Sections 3.1 and 3.2 GNN training can be made much more efficient than with backpropagation by exploiting the fact that the neighborhood aggregation procedure can be performed just once while the parameters of each GNN layer are optimized over multiple training epochs. This because, similar to You et al. (2020), the input to each layer (i.e., the output from the lower layer) remains the same throughout the training of the layer, and thus the neighborhood aggregation results also do change due to the way the greedy forward training scheme and GNNs work. This way, only the feature update step after the aggregation needs to performed multiple times during the training, while the aggregation is done only once, which can lead to significant speedup in training GNNs, especially for large-scale graphs.
While the training with forward learning approaches takes longer than BP in some cases, the increase in training time in such cases is still modest in comparison to BP (e.g., up 1.6 times longer time to train four-layer GCNs on GITHUB). With forward learning approaches with top- down signal paths, or with models like GAT, the aforementioned speedup cannot be used, as the output from the neighborhood aggregation procedure does no longer remain the same throughout the training of the layer, due to the changes the input resulting from the incorporation of top- down signals, or due to the changes in the parameters used for attentive neighborhood aggregation. Still, the training time increase such cases is not that significant, and the forward learning methods can scale up in much more memory-efficient manner than BP.
The sorgle-forward (SF) approach improves the training efficiency of the forward-forward
based approaches as SF avoids the computational overhead to perform multiple forward passes.
32

---

Published as a conference paper at ICLR 2024
Table 7: Training time (in seconds) for node classification. The best results are in bold font, and the best results among the forward learning methods are in red.
(a) GITHUB
Training time (secunds) (U)
# Layers=l
Layers=2
# Layers=3
Layers=4
Backpropagation-GGNN
1.77
10.37 1.54
13.04:1.7
SymBa-GCN
1.411.00
2.09
2.78.11.0
3.48.100
PEPITA-GCN
1.40=100
7.15=0.8
13.09
18.95
FF-LA-GON (Sec
14711.
2.061:12
2.65113
3.24
FF-VN-GCN (Sec
1.47+10
2.19=100
2.91
3.64=1.1
SF-GCN (Sec
1.10+0.9
1.57=0.9
2.04:0.9
2,52:0.99
SF- To-Loss GCN (Sec
5.6610.8
11.23
16.061111
21111111
Te-TTo-TTo-nnut--CG (Sec
5.67=0.8
10.80=0.9
15.91
2112+111
Backpropagation-SAAN
4.54122
7.99
1.
14.91
SymBa-SAGIE
1.28114
1.55144
1.2114
2.09114
PEPITA-SAGE
6.62
11.740.9
17,58=15
22.50=0.7
FF-LA-SAGE (Sec
3.1
1.3
1.8
2.04144
1.4
FF-VN SAGE (Sec
0.8310077
1.11007
1.81077
1.6610.77
SF-SAGE (See
3.2
0.82+0.7
108
13420.7
161=077
SF- To---T--LLsss-s-- S--LLS--AGEE SAGE (Sec
3.3
4.60:1
7.5
10.99
14.61
SF-Top-To To-Inpur- IoGEt-SAGE (Sec
5.3
4.08
10.44=0.8
16.76=0.8
23.11=0088
BackpropugationnGGN
1.891244
1.79
3.453.4
2.64
SymBa-Gaa
1.72.0.88
2.85
3.98
5.12.10.8
PEPITA GAT
0.83-0.8
1.13:0.88
166
1.8
EF-LA-GAT (See 1.ee
3.1
1.70:0.9
2.78100.9
3.85
4.93-1.99
DR-VN-GAT (Sec
1.7510.8
2.9010.8
4.06.00.
5.2410.8
SF-GAT (See 3.2
1.210088
1.89
2.5620.8
3.24
SF-Top- TosLoss GAT (Sec
1.59
2.36
2.70
3.3
Te--To--To-Inptt-CO (Seo 5.5
1.25=0.8
2.1115
3.45+200
3.24
(b) CITESEER
Training time (seconds)
Method
Layers=l
Layers=2
Layers=3
#layers:4
Backpropugation-GCNN
2.1722.5
2.33
188118
1.88117 7
SymBa-GCN
1.96=1.5
2.81=15
3.67
4.54=1.5
PEPITA-GGN
1.02+1.0
1.33210
1.841.3
2.1621.5
PF-LA-GON (Sec
3.1
1.91
2.71=1.5
3.52+1.5
4.34=1.5
FF-VN GCN (Sec
2.09
3.00
3.92.117
4.83
SE-GON
1.4
1.6721.
1.90+1.7
2.11177
Te-Top-To-LLoss JCN GCN (Sec
3.3
0.93.0.8
1.27
1.8
1,97
SF-Top- To Input GCN (Sec
1.3
1.0
1.55.112
2.14
2.47.1.5
BackpropugaaionnSAAGi
2.0322.3
2.51
2.6222.7
2.244200
SymBa-SAGE
2.21=2.1
2.90=2.1
3.59+2.1
4.27+2.1
PEPITA-SAGHA
1.27-0.8
1.991099
1.66-0.99
2.7-1177
FE-LA-SAGE
1.90
2.57.1.7
3.19-18
3.81199
VN-SAGE (Sec
S.T.
1.3.1.0
1.921100
2.491100
3.07
SE-SAGE (Sec 3.2
1.24+144
1.43114
166214
1
SF-Top To-Loss- SAGE (See 3.3
1.151..7
2.03:1155
1.781088
2.04
SF-Top Input SAGE
2.442..0
2.28.10.7
Backpropugation-GGAT
1.1418
2.30
1.69199
1.6411.7
SymBa-GAT
1.52-0.7
2.33-0.7
3.31
4.21=0.7
PÉPITA-GLE GAT
0.76.0.8
0.88
1,50114
1.16-0.8
BE-LA-GAT (Sec
3.1
1.71+1.0
2.67+10
3.62-1.11
4.61=1.11
DR-VN-GAT GAT (SeeSTT
1.610.8
2.29
3.1310.9
3.97
SEGAAT (See
1.32-112
1.55
1.77412
SF-Top-To-LL To-Los GAT (See 3.3
0.82:0.8
1.05
1.5514
LS5
SF-Top To Input GAT (Sec
5.51
0.81.1..7
1.03-0.88
1.20.0.77
1.1
33

---

Published as a conference paper at ICLR 2024
OFF-GAT SFF-GCN
SE-SEGE
Plametoid-Pubbie
Planefoid-Citesee
BP-GCN
BP-GAT
BP-SAGE
Planetoid-Cora
80-
80-
Ageuracy (%)
Accuracy (%%
Ageuracy (%)
70
770
770
Better
Better
Better
60-
60-
Better
Better
Better
50
50
50
1x
6x
12x
18x
24x
1x
6x
12x
18x
24x
1x
6x
12x
18x
24x
Mem usage relative 1-layer model Mem usage relative I-layer model
Mem usage relative 1-layer model
(1) 2-layer models
(2) 3-layer models
(3) 4-yyyer models
Figure 7: The single-forward approach (SF, empty symbols) vs. backpropagation (BP, filled sym- bols) on node classification in terms of classification accuracy (y-axis) vs. memory usage increase (x-axis). Lines towards the lower right corner (SF outperforms BP), and upper right comner (BP outperforms SF); horizontal lines (both perform the same).
AGON SSAGE
SF-TopDown (See 3.3)
FF-LA (Sec3.1)
FF-SymBa
PEPITA
OGAT
SF (Sec 3.2)
FF-VN (See 3.1)
CaFo
80
g
Accuracy (%)
Accuracy (%)
Accuracy (%)
60-
8
60
60-
+
Better
40-
4
Better
40-
Better
4
40-
Better
Better
20-
Better
160
200
50
160
200
Mem usage (MB)
Mem usage (MB)
Mem usage (MB)
Planetoid-PubMed
Planetoid-Cora
Planetoid-Citeseer
Figure 8: Node classification accuracy (y-axis) and memory usage (x-axis) on Planetoid datasets (Table as three GNNs (denoted by symbol shape) are trained with different forward learning approaches (denoted by symbol color).
Table 8: Summary statisties of the graph datasets with limited training data.
Dataset
# Nodes
Edges
Training Nodes
Validation Nodes
Testing Nodes
# Features Classes
PLANETOID-CORE -ORA
2.708
10,556
140
500
1.000
1,433
PLANETOID- CITEESEERRR
3.327
9,104
120
500
1.000
3,703
MAANTTID-PUBMEES
19,717
88,648
60
500
1,000
500
F.4 NODE CLASSIFICATION RESULTS WITH LIMITED TRAINING DATA
In this section, we evaluate how well the proposed forward learning approach performs in compari- son to BP when given small amount of training data. To that end, we use three citation networks, namely, CLANETOID-CORAA
LLANETOID-PUBMER and CAANETOID-CITESTI which provide much less amount of training data than the graphs used the main text ().
listed in Table
Figure 7 shows the classification accuracy (y-axis) versus memory usage increase (x-axis) obtained with the single forward (SF) and backpropagation (BP) algorithms on the three graphs with limited training data. From Figure 7 make the following observations.
In these datasets with sparse training data, the proposed SF (empty symbols) perform similarly to BP (filled symbols), or even outperforms BP few cases, e.g., when four GNN layers are used. These results are similar to the results obtained in the setup with richer training data (Figure
With two layers (Figure 11). BP mostly outperforms SF, which is different from the results in the counterpart in Figure 2a 1), where BP and SF performed similarly. The contrastive single forward learning mechanism tends require more training data than BP arrive similar level of predictive accuracy, but the gap closes as more layers are used.
SF shows near ceartconstant memory usage, while the relative memory usage of BP keeps increasing (up to around 21 x). as the number of layers increases.
34

---

Published as a conference paper at ICLR 2024
Figure shows the node classification accuracy (y-axis) memory usage (x-axis), GNNs (denoted by the symbol shape) are trained with different forward learning approaches (denoted by the symbol color). Overall. Figure 8 shows a similar pattern to Figure 3 which shows the perfor- mance of forward learning approaches on datasets with richer training labels. Thus our discussion on Figure 3 in Section similarly applies to Figure 8
F.5 IMPACT OF THE DIRECTIONALITY OF EDGES BETWEEN REAL AND VIRTUAL NODES
In the main text, the results of the proposed approaches that employ virtual nodes were obtained using the augmented graph where real nodes and virtual nodes are connected via bidirectional edges (i.e., in both ways between the real and virtual nodes). Since virtual nodes are processed in the same way as the real nodes, they aggregate information from a large number of real nodes, and then propagate aggregated information back to those real nodes, which can potentially adversely impact the classification performance. Here we evaluate a different approach to utilize virtual nodes, which to connect the real and virtual nodes via unidirectional edges (i.e., in way from the real to the virtual nodes), such that virtual nodes only receive information from the real nodes, without sending the aggregated message back them. In Figure
g report the node classification performance when the edges between the real and virtual nodes are bidirectional or unidirectional, obtained with (a) the forward-forward (FF) approach (Section (.I). (b) the single forward (SF) approach (Sec- tion (.2). and (c-d) the SF approaches with the top-down signal path (Section 3.3 Results show that the impact of using unidirectional edges, comparison to when bidirectional edges are used, varies a lot among forward learning approaches. With the FF approach (Figure [() and the SF approach with the top-to-loss signal path (Figure(id), using bidirectional edges led to better results than using unidirectional edges in most cases. In particular, for the FF approach, the information from the virtual nodes to the real nodes, which lacks when unidirectional edges are used, tends to play important role for the model to Iearn to distinguish between correct and incorrect classes. By contrast, with the SF approach (Figure Ib)), using unidirectional edges led to better performance in several cases, especially top-to-input signal path (Figure c))) both types of edges performed similarly to each other.
MED and GITHUB. With the SF approach that employs the
35

---

Published as a conference paper at ICLR 2024
GCN SAAGE OGAT
CiteSeer
PubMed
CoraML
Amazon
GitHub
One way: Real Virtaal
One way: Real - Virtual
100
One way: Real - Virtual
75
75-
50
50
50
25
25
25-
2
25
25
50
160
Both ways: Real e- Virtual
Both ways: Real 4- Virtual
Both ways: Real c- Virtual
(1) Accuracy (%) w/ 2-layers
(2) Accuracy (%) w/ 3-layers
(3) Accuracy (%) w/ 4-layers
(a) Performance with the forward-forwand approach (Section
3.1
ACCN SAGE OGAT
CiteSeer
PubMed
CoraML
Amazon
GirHub
One way: Real Virtual
One way: Real - Virtual
95
One way: Real -, Virtual
95
4
90
90
85
00
80-
95
Both ways: Real + Virtual
Both ways: Real +- Virtual
Both ways: Real +- Virtual
(1) Accuracy (%) w/ 2-layers
(2) Accuracy (%) w/ 3-layers
(3) Accuracy (%) 4-layers
(b) Performance with the single forward approach (Section
3.2
AGNN SSAGE OGAT
CiteSeer
Pub
CoraMI.
Amazon
GitHub
One way: Real - Virtual
95
One way: Real - Virtual
95
One way: Real - Virtual
95
90-
90
90
20
85
85-
85.
85
90
95
85
90
95
85
90
95
Both ways: Real - Virtual
Both ways: Real - Virtual
Both ways: Real + Virtual
Accuracy (%) w/ 2-layers
(2) Accuracy (%) w/ 3-layers
(3) Accuracy (%) w/ 4-layers
(c) Performance with the single forward approach with the top-to-input signal path (Section
AGCN ESAGE OGAT
CiteSeer
PubMed
CoraML.
Amazon
GitHib
One way: Real - Virtual
One way: Real- Virtual
One way: Real- Virtual
80-
80
80-
40
60-
60
60-
40
40
40
60
80
40
60
80
40
60
80
Both ways: Real Virtual (1) Accuracy (%) w/ 2-layers
Both ways: Real Virtual
Both ways: Real Virtual
(2) Accuracy (%) w/ 3- layee
(3) Accuracy (%) w/ 4ayerss
(d) Performance with the single forward approach with the top-to-loss signal path (Section 3.3
Figure 9: Node classification accuracy using virtual node-based forward learning approaches when the edges between the real and virtual nodes are bidirectional (i.e., in both ways between real and virtual nodes), and unidirectional (i.e., in one way from the real to the virtual nodes), obtained with (a) the haward-forward (FF) approach (Section 3. (b) the single forward (SF) approach (Sec- tion 3.2), and (c-d) the SF approaches with the top-down signal path (Section 3.3
36


Reviews:
[{"review_id": "cXwfaVs6BM", "reviewer": "Reviewer_yhLp", "summary": "This paper is an application of forward forward learning on graph neural networks. Authors modify the forward forward learning algorithm to train GNNs with several novel designs including node labels, virtual nodes, single forward pass, and the top-down signals. In summary, this paper is a good practice by applying forward forward algorithm to GNN learning and provides sufficient technical contribution with sound experiments. I raise a marginally accept for the lack of important experiments.", "questions": "1. From my point of view, the motivation of this paper is to apply the forward forward algorithm to GNN training. They do not mention what the problem is with the current BP training method on GNN. The three points (scalability, parallelism, and flexibility) they mentioned do not seem to be graph-related. I suggest finding a stronger motivation in the introduction, such as what the problem FF wants to solve on graph learning.\n2. By training layer by layer, FF can be seen as a time-for-space algorithm. In the experiment, authors only show the limited space utilization without mentioning the overhead training time. I think it is inappropriate.\n3. In the node classification task, authors randomly selected 64% for training, which is not the common practice in graph learning. In most cases, we only select a small percentage for training, as in the GCN, GAT, and SAGE paper author mentioned. I think more experiments on limited training data are needed as their proposed algorithm seems sensitive to that.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Authors systematically investigate the forward learning algorithm on GNNs.\n2. This paper provides several technical contributions to forward forward algorithm, which is inspiring and important.\n3. Extensive experiments have been carried out to prove their effectiveness.", "weaknesses": "1. Motivation is not well illustrated\n2. Time efficiency is not analyzed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H9nYynfzu8", "reviewer": "Reviewer_ztUM", "summary": "Traditional training of GNNs relies on the backpropagation (BP) algorithm, which imposes certain constraints that limit scalability, parallelism, and flexibility in learning. To overcome these limitations, the authors propose FORWARDGNN, inspired by the forward-forward (FF) algorithm used in image classification. FORWARDGNN extends FF to work with graph data, eliminating the need for generating negative inputs and allowing layer-wise local forward training. The new method enables each layer to learn from both bottom-up and top-down signals without relying on error backpropagation. The paper demonstrates the effectiveness and generality of FORWARDGNN through experiments on five real-world datasets and three GNNs, showing that it performs on par or better than BP on link prediction and node classification tasks while being more memory efficient.", "questions": "see weakness.\n\nHow is this paper different from: \"Decoupled Greedy Learning of Graph Neural Networks\"", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Forward-Forward learning proposed by Hinton is a very new and interesting research topic. This paper adopts that in GNN setup, and propose an alternative ( Single forward ) which only runs a single forward pass.\n\n\nExperiments are very comprehensive, including both effectiveness and training efficiency.", "weaknesses": "Overall I think this is an interesting paper. Given the Forward-Forward Learning is a very new concept, this work is an interesting trial on the direction.\n\nOne question is can the proposed forward-only method only work for graph learning? Or it can be generalized to other tasks? If yes, better to show such results to make this work more solid; if not, better to explain clearly the assumption and some unique properties of graph to make this method work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rdKEJF6J9I", "reviewer": "Reviewer_BuBQ", "summary": "The de-facto standard algorithm for training Graph Neural Networks (GNNs) is *Backpropagation*. Despite several advantages, the need to backpropagate gradients through the neural architecture hinders its scalability whenever the architecture depth increases. The recent Forward-Forward (FF) approach by Hinton et al. has inspired several works that aim at local/forward-only learning procedures. In this work, the authors propose ForwardGNN, which investigates FF in the context of GNNs for node classification and link prediction tasks, as well as and proposing a novel approach that requires only a single forward pass to learn GNNs.", "questions": "Please also refer to the **Weaknesses** section.\n\n1) It is not clear to me wheter virtual nodes are processed as standard nodes by the GNNs -- e.g. they need initial nodal features $h_i^{(0)}$ and both receive and send message towards neighbors -- or they are solely used as *receiver* nodes, e.g. they only have incoming edges-- in order to compute their representation $c_k^{(\\ell)}$.  \n\n2) The virtual node approach seem to be very prone to the issue of over-squashing [1,2]. What happens when varying the number of classes with respect to the graph scale (with millions of nodes)? An analysis on this could improve the paper contribution. \n\n3) The experimental setup analyzes some competitor algorithms that were not devised for GNNs, in the tasks of node classification and link prediction. Given that the **GFF** model by Paliotta et al. [3] is explicitly devised for GNNs, why did the authors not compare with **GFF**? Is the proposed approach compatible with the Graph classification task?\n\n4) The authors analyzed the memory impact of the proposed method. What about the time complexity/execution timings?  And what are the model ability to scale to bigger graphs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well written and structured. The proposed approach is original, given that it extends the **FF** approach to GNNs by proposing a forward-only mechanism that avoids multiple forward passes for the positive and negative samples,  that would be required by standard FF.\nMoreover, the interesting incorporation of top-down signals from upper layers is a clever intuition.", "weaknesses": "There are some details of the approach that have not been clearly described. \nIn the case of the approaches that leverage virtual nodes (Sections 3.1-bottom and 3.2): the authors specify that the graph topology is enriched by such virtual nodes. It is not clear to me wheter such virtual nodes are processed as standard nodes by the GNNs -- e.g. they need initial nodal features $h_i^{(0)}$ and both receive and send message towards neighbors -- or they are solely used as *receiver* nodes, e.g. they only have incoming edges-- in order to compute their representation $c_k^{(\\ell)}$.  \n\nThe role of the virtual nodes in the graph topology raises another question: if I did get it correctly, there are as many virtual nodes as classes. Hence, many nodes (depending on the graph scale) will be connected to the same virtual node forming a bottleneck. This approach seem to be very prone to the issue of over-squashing [1,2]. What happens when the number of classes is very low with respect to the graph scale (with millions of nodes)? An analysis on this could improve the paper contribution. \n\nThe experimental setup analyzes some competitors that were not devised for GNN in the tasks of node classification and link prediction. Given that the **GFF** model by Paliotta et al. [3] is explicitly devised for GNNs, why did the authors not compare with **GFF**? Is the proposed approach compatible with the Graph classification task?\n\nRegarding related work, there are some works that proposed alternative local rules for learning in GNNs that depart from the BackProp approach [4, 5]. Describing differences and advantages could help the reader in understanding the paper contributions.  \n\n\n*Minors*\nThe authors refer to Alg. 1, 2, 3 that are not in the main paper (it should be clarified). \n\n[1] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications.\narXiv preprint arXiv:2006.05205, 2020.\n\n[2] Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio, and Michael M\nBronstein. On over-squashing in message passing neural networks: The impact of width, depth,\nand topology. In International Conference on Machine Learning, pp. 7865–7885. PMLR, 2023.\n\n[3] Daniele Paliotta, Mathieu Alain, Balint Me, and Francois Fleuret. Graph neural networks go forward-forward. CoRR, abs/2302.05282, 2023\n\n[4] Tiezzi, Matteo, et al. \"Deep constraint-based propagation in graph neural networks.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.2 (2021): 727-739.\n\n[5] H. Dai, Z. Kozareva, B. Dai, A. J. Smola, and L. Song, “Learning steady-states of iterative algorithms over graphs,” in Proc. Int.\nConf. Mach. Learn., 2018, pp. 1114–1122", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoHRVtUkIF", "reviewer": "Reviewer_TBYv", "summary": "The paper proposes ForwardGNN, a novel forward learning framework for Graph Neural Networks (GNNs) that addresses the limitations of the backpropagation (BP) algorithm, such as memory overhead and biological implausibility. By building upon and improving the forward-forward algorithm (FF), ForwardGNN is designed to work with graph data and GNNs without generating negative inputs. This results in a more efficient training and inference process. The framework also enables each layer to learn from both bottom-up and top-down signals without relying on backpropagation. Experiments conducted on five real-world datasets and three representative GNNs demonstrate the effectiveness and generality of the forward graph learning framework, showing that it outperforms or performs on par with BP while using less memory for training.", "questions": "1. The proposed method involves layer-by-layer training of GNNs. In the literature, layer-wise training of deep neural networks has been a long-standing topic [1,2]. A more extensive literature review is necessary to differentiate the present paper. Additionally, the paper's contribution could be strengthened by including experiments for comparison with [2].\n\n2. The data splitting differs significantly from those in previous papers. I recommend that the authors conduct experiments using standard splitting for semi-supervised node classification tasks.\n\n3. In Figure 2, the upper-left corner is crowded with too many methods, making it difficult to read. Presenting the results in a table might be a better approach (the table should be much simpler compared to Table 2 and Table 3).\n\n4. In Algorithms 1-4, the authors state \"optimize layer using the computed loss\" but the details of the optimizer are not clear, e.g., what optimizer is used.\n\n5. In addition to memory, training time is an important metric. Including the training time of the proposed method and baselines will complete the picture.\n\n6. In Table 2, why does SF-Top-To-Loss achieve state-of-the-art performance with one layer, but the performance degrades with more layers? An explanation is needed.\n\n7. In Table 2, the best-performing method varies significantly across datasets. Providing an intuition for this variation would be beneficial.\n\n8. In Table 3, the memory of some proposed methods is higher than BP (even 20 times higher in Table 3 (c) GAT). Would this be a significant bottleneck?\n\n[1] Belilovsky E, Eickenberg M, Oyallon E. Greedy layerwise learning can scale to imagenet, ICML 2019. \n[2] You Y, Chen T, Wang Z, et al. L2-gcn: Layer-wise and learned efficient training of graph convolutional networks, CVPR 2020.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. This paper systematically explores the potential of forward graph learning, paving the way for biologically plausible optimization techniques in GNNs.\n\n2. The performance is impressive; this marks the first instance where FF algorithms outperform BP-trained deep neural networks in real-world applications.\n\n3. The study proposes numerous algorithms that are model-agnostic, potentially inspiring further research on forward-forward algorithms in various applications.", "weaknesses": "There are a few areas for improvement:\n\n1. The proposed methods share a close relationship with layer-wise training of neural networks, which could potentially diminish the significance of this paper. (Refer to Question 1)\n\n2. The data splitting does not adhere to standard practices. (Please see Question 2)\n\n3. The presentation is difficult to follow and should be improved. (Please refer to Question 3,4,5)\n\n4. Table 2, and 3 contains many interesting results but lacks explanations and intuitions. (Please refer to Questions 6, 7, 8)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)