Gradient Rewiring for Editable Graph Neural Network Training
Authors: Zhimeng Jiang
Zirui Liu
Xiaotian Han
Qizhang Feng
Hongye Jin
Qiaoyu Tan
Kaixiong Zhou
Na Zou
Xia Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: Gradient
Graph Neural Networks
Primary area: optimization_for_deep_networks

Abstract: Deep neural networks are ubiquitously adopted in many applications, such as computer vision, natural language processing, and graph analytics. However, well-trained neural networks can make prediction errors after deployment as the world changes. \textit{Model editing} involves updating the base model to correct prediction errors with less accessible training data and computational resources.
Despite recent advances in model editors in computer vision and natural language processing, editable training in graph neural networks (GNNs) is rarely explored. The challenge with editable GNN training lies in the inherent information aggregation across neighbors, which can lead model editors to affect the predictions of other nodes unintentionally. In this paper, we first observe the gradient of cross-entropy loss for the target node and training nodes with significant inconsistency, which indicates that directly fine-tuning the base model using the loss on the target node deteriorates the performance on training nodes. Motivated by the gradient inconsistency observation, we propose a simple yet effective \underline{G}radient \underline{R}ewiring method for \underline{E}ditable graph neural network training, named \textbf{GRE}. Specifically, we first store the anchor gradient of the loss on training nodes to preserve the locality. Subsequently, we rewire the gradient of the loss on the target node to preserve performance on the training node using anchor gradient. Experiments demonstrate the effectiveness of GRE on various model architectures and graph datasets in terms of multiple editing situations. The source code is available at \url{https://github.com/zhimengj0326/Gradient_rewiring_editing}.

Gradient Rewiring for Editable Graph Neural Network Training
Zhimeng Jiang', Zirui Liu?, Xiaotian Han Qizhang Feng', Hongye Jin', Qiaoyu Tan', Kaixiong Zhou², Na Zou", Xia Hu"
Texas A&M University, University of Minnesota, "Case Western Reserve University, NYU Shanghai, 'North Carolina State University, University of Houston, Rice University
Abstract
Deep neural networks are ubiquitously adopted in many applications, such as computer vision, natural language processing, and graph analytics. However, well- trained neural networks can make prediction errors after deployment as the world changes. Model editing involves updating the base model to correct prediction errors with less accessible training data and computational resources. Despite recent advances in model editors in computer vision and natural language process- ing, editable training in graph neural networks (GNNs) is rarely explored. The challenge with editable GNN training lies in the inherent information aggrega- tion across neighbors, which can lead model editors to affect the predictions other nodes unintentionally. In this paper, first observe the gradient of cross- entropy Ioss for the target node and training nodes with significant inconsistency, which indicates that directly fine-tuning the base model using the loss the target node deteriorates the performance on training nodes. Motivated by the gradient inconsistency observation, we propose simple yet effective Gradient Rewiring method for Editable graph neural network training, named GRE. Specifically, we first store the anchor gradient of the loss on training nodes preserve the locality. Subsequently, we rewire the gradient of the loss on the target node to preserve performance on the training node using anchor gradient. Experiments demonstrate the effectiveness of GRE on various model architectures and graph datasets terms of multiple editing situations. The source code available http::/gghhhb..oommmmmmmmng2/2gf hetprrddieeeeeeiinneeeeddiiiieei
1 Introduction
Graph Neural Networks (GNNs) have demonstrated exemplary performance for graph learning tasks, such as recommendation. link prediction, molecule property analysis [1] 2 3
5
6 77. With
message passing, GNNs learn node representations by recursively aggregating the neighboring nodes' representations. Once trained, GNN models deployed to handle various high-stake tasks, such as credit risk assessment in financial networks [8] and fake news detection in social networks 19]. However, the impact of erroneous decisions such influential applications can be substantial. For instance, misplaced credit trust in undetected fake news can lead to severe financial loss.
An ideal approach to tackle such errors should possess the following properties: 1) the ability to rectify severe errors in the model's predictions, 2) the capacity to generalize these corrections to other similar instances of misclassified samples, and 3) the ability to preserve the model's prediction accuracy for all other unrelated inputs. To achieve these goals, various model editing frameworks have been developed to rectify emrors by dynamically adjusting the model's behavior when errors detected 110 The core principle is to implement minimal changes to the model to correct the error while keeping the rest of the model's behavior intact. However, model editing is not simple
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

plug-and-pp and- solution. These frameworks often require an additional training phase to prepare for editing before they can be used effectively for editing [TO] T2 [3]. Although model editing techniques have shown significant utility in computer vision and language models, there is rare work focused on rectifying critical errors in graph data. The unique challenge arises from the inherent message-passing mechanism in GNNs when edits involve densely interconnected nodes [14, [5]. Specifically, editing the behavior of a single node can unintentionally induce a ripple effect, causing changes that propagate throughout the entire graph. [14] theoretically and empirically demonstrate the complexity of editing GNNs through the lens of the loss landscape of the Kullback-Liebe divergence between the pre-traines ire-trained node features and edited final node embeddings. Moreover, a simple yet effective model structure, named EGNN, is proposed with stitched peer multi-layer perception (MLP), where only the stitched MLP is trained during model editing.
In this work, we investigate the model editing problem for GNNs from a brand-new gradient perspective, which is compatible with existing work [14). Specifically, we first found a considerable inconsistency between the gradients of the cross-entropy loss for the target node and the training nodes for GNNs. Such inconsistency implies that direct fine-tuning of the base model using the loss of the target node can lead to a deterioration in the performance on the training nodes. Motivated by the above observation, we propose simple yet effective Gradient Rewiring method for Editable graph neural network training, named GRE. Specifically, we first calculate and store the anchor gradient of the loss on the training nodes. This anchor gradient represents the original learning direction that we wish to preserve. Then, during the editing process, we adjust the gradient of the loss on the target node based on the stored anchor gradient. This adjustment, "rewiring" ensures that the changes made to the target node do not adversely affect the performance on the training nodes. Experiments demonstrate the effectiveness of our proposed method for various model structures and graph datasets. Moreover, the proposed method is compatible with the existing EGNN baseline and further improves the performance.
Cora
Flickr
Arxiv
A-photo
400
GCN
100
750
Gradese
200
MLP
SAGE
Gradess
Grodes
200
Gradees
500
200
250
2
Epochs
4
0
Epochs
Epochs
0
0
2
Epochs
6
0.9
40
60
0.6
Loyan
140
-2
20
0
0
2
6
6
Epochs
Epochs
Epochs
Epochs
10
Coopet
5
0
6
0
2
4
6
Epochs
Epochs
Epochs
Epochs
Figure 1: (a) Top: RMSE distance hetween the gradients of cross-entropy loss over training datasets and over the targeted sample for different architectures. (b) Middle: Cross-entropy loss over training datasets when the model is updated using target loss. (c) Bottom: Cross-entropy loss over the targeted sample when the model is updated using target loss.
2 Preliminary and Related Work
We first introduce the notations used throughout this paper. A graph is given by G (V,E) .hhee
V = (V), uN) is the set of nodes indexed from 1 to n, and = (e), em) CV x V is the set of edges. n = [V] and m |g| are the numbers of nodes and edges, respectively. Let X E Raxd be the node feature matrix, where d is the dimension of node features. A E Raxn is the graph adjacency
matrix, where A.j- 1 if (v,, v,) E E clse Arj = 0. À= = D (A + nD is the normalized

---

adjacency matrix, where D is the degree matrix of A + I. The node label is defined as for node 2z- We consider node classification tasks with C' classes in this paper.
2.1 Graph Neural Networks
Graph Neural Networks have been successfully applied across various domains/tasks, including knowledge graphs [16] graph condensation [18] [0]. event extraction [21], and entity relation tasks [22]. Most graph neural networks follow a neighborhood aggregation procedure to learn node representation via propagating representations of neighbors and then follow up with feature transformation (23] The 1-th layer of graph neural networks is given by:
a (1)
PROPAGATION (fx4
a- .x.-)))))))))
(1)
TRANSFORMATION (a (
((a))
where x is the representation node v2 at 1-th layer and x; initialized as node feature x,, i..e, the i-th row node feature matrix X. Many GNNs, such GCN (24), GraphSAGE (25). and GAT [26), can be defined under this computation paradigm via adopting the different propagation and transformation operations. For example, the 1-th layer in GCN can be defined as:
( o (Ax"-))
where X Raxd and (1-1) R nxd are the node representation matrix containing the h, for each node at the layer and layer I 1. respectively. w Raxd is layer-specific trainable weight matrix, and or (-) a non-lineat activation function (c.g., ReLU).
2.2 Model Editing
Model editing aims to modify base model's responses for misclassified sample Zig and its analogs. This is typically achieved by fine-tuning the model using only a single pair of input Tig and the desired output Vtg. while preserving the model's responses to unrelated inputs [10 Our contribution lies in the novel application of model editing to graph data, domain where misclassifications on a few pivotal nodes can trigger substantial financial losses, fairness issues, or even the propagation of adversarial attacks. Consider the scenario of node classification where well-trained GNN incorrectly predicts particular node. Model editing can be employed to rectify this erroneous prediction. By leveraging the node's 's characteristies and the desired label, the model can updated to correct such behavior. The ideal outcome of model editing twofold: first, the updated model should correctly predict the specific node and its similar instances; second, the model should maintain its original behavior on unrelated inputs. It is important to note that some model editors require a preparatory training phase before they can be applied effectively [10 [3] 12 II]. This crucial step ensures that the model editing process is both precise and effective in its application.
14.
3 Methodology
In this section, we first provide the preliminary experimental results as the motivation to rewire gradients for model editing. Subsequently, propose our gradient rewiring method for editable graph neural networks training (GRE) and an advanced version (GRE+) to improve the effectiveness of model editing, respectively.
3.1 Motivation
In the preliminary experiments, we first pre-train GCN, GraphSAGE, and MLP on the training dataset Virain (e.g., Cora, Flickr, ogbn-arxiv, and Amazon Photo datasets) using cross-entropy loss. Subsequently, we find the misclassified samples in the validation dataset and randomly select one sample as the target sample (Xtg. Btg)) During the model editing, we update the pre-trained model using cross-entropy loss over the target sample using gradient descent, i.e., the models are truined we perform 50 independent edits and report the
inductively. Following previous work [10 averaged metrics.
3

---

It is well-known that model editing incurs training performance degradation [1 T0 12 for many model architectures. To deeply delve into the underlying reason, we investigate performance degradation from a model gradient perspective. We further define the training loss as Lerain = [irain] Eievirat.. C' (Xx), 1s), where fo(.) E RCC a prediction model parameterized with O E TR. C. I. are the number of classes and model parameters, CF((.-) is the cross-entropy loss, the target loss is given by Ltg ( (Xtg) ) B/tg). For example, model fa(-) can be instantiated by simple MIP. For model editing, the gradient for training and target loss is given by Strain -Larain E R and Itg = HLiz
GNNs with the number of layers defined in Eq. (
09 E R respectively. To investigate why the model editing leads to training performance degradation, we use gradient RMSE Moott-Mean-Squared-FFe
0e
i.e., Grad = /llgtrain 9tg 2 to measure the model editing discrepancy for training datasets and target sample.
The model editing curves for gradient RMSE 2 training loss, and target loss across various model architectures (GCN, GraphSAGE, and MLP) are shown in Figure Although the gradient RMSE for training datasets and target sample is close to O, the model parameters demonstrate significant inconsistent behavior in terms of training loss due large gradient discrepancy in the initial editing stage. We observe that: 1) Even though the target loss decreases during model editing, the training loss increases significantly. 2) The increasing rates of training loss for GCN and GraphSAGE are significantly higher than that of MLP. The above observations imply that editing training for graph neural networks is more challenging due to higher gradient discrepancy between the training dataset and the target sample.
3.2 Gradient Rewiring Approach
Preliminary results show high discrepancy in training loss and target loss for GNNs, which implies that the vanilla model editing hampers the performance on the overall training dataset and thus results in high accuracy drop for node classification tasks. Therefore, we aim tackle the training dataset performance degradation from the gradient rewiring approach.
We propose simple yet effective gradient rewiring approach for editable graph neural network training, named GRE. We first formulate constrained optimization problem regulate model editing and then solve the constrained optimization problem via gradient rewiring.
GRE
Model editing aims correct the prediction for the target sample while maintaining the prediction accuracy on the training nodes. The objective function focuses on minimizing the loss at the target node. To preserve the predictions on the training nodes, introduce two constraints: (1) the training loss should not exceed its value prior to model editing (see Eq. B ); and (2) the differences in model
predictions after editing should remain within a predefined range (see Eq. (4). Define Oo and O' as the model parameters before and after model editing. Then we have the following constrained optimization problem:
min s.t.
Lig (x(g)) Ves)
(2)
Lirain (for, Verain) S Lirain (Fog Verain)
(3)
(x; foo (x:))1 $ 8,
Derain
i iEVeram
where 0 and 6' represent the model parameters before and after model editing, respectively, the hyperparameter 8' represents the maximum average prediction difference on training nodes. Notice that the model parameters update adopts gradient descent using target loss without any constraints, i.e., 0' 0n agttg where a is step size model editing. The key idea of our proposed solution is to rewire gradient 9tg g, which is obtained by satisfying the involved constraints. Note that the model editing usually corrects the model prediction on the target sample within few steps, i.e. there are no significant model parameter differences, thus we adopt Taylor expansion to tackle such constrained
'The reason for focusing on the training is that during model editing, can only use the training set and not the test set.
"There is no variance for gradient estimation since gradient calculation is based on backpropagation. The large variance model performance and gradient discrepancy derives from the randomly selected target node.

---

optimization problem. For target loss Lig, we can approximate it as:
Lig (for (Xeg) Mig) se Lig (X1g), (ro)
Meo) + 96g 10' - 00)
Leg (Xxng) Vra) 09:.9-
(5)
To optimize the objective function Eq. (2). it is easy to conclude that the gradient cosine similarity
Bight should be maximized. Given the gradient before/after model editing is fixed, the maximization
of gradient cosine similarity 9ig9 is equivalent to the minimization of 119tg gl|². To satisfy Eq. (3) we also adopt Taylor expansion on Ltrasn and it is easy to obtain that the gradient cosine similarity should be positive, i.e., 9tg9 O. As for the constraint in Eq. (4). similarly, Taylor expansion is used to express the relationship between the model predictions before and after the model editing, as follows:
05a (x:)
(x,) T
(x:) = (x;)+
+
(6' 0) (x,)
agg
(6)
00
00
Therefore, obtain the following approximation Eq.
|| Vereinl iEVerain
Σ
fx, - (x:)]l 1 ( (-ag)] 58,
where gradient for a model prediction is defined as îtrain Veraan
EieViruin
fng (x,)
84
RIXC Therefore, the model prediction difference constraint can be transformed into 1 llåtrain a Appeeelll/ I 8, where || llapect represents matrix spectrum norm and l1 spect is fixed in model editing, and 8 - S In a nutshell, our goal is to correct the target sample (i.e., minimize l1gtg - g) and minimize gradient discrepancy for model prediction among training dataset and target sample (i.e., ||gll3), while guaranteeing non-increased training loss (i.e., Itrain9 2 0). The original constraint optimization problem simplified as gradient rewiring, i.e.,
S
1 min 2 [g - l + 2 ||r|| = min 2 9 9ig9 + 29ghtg
s.t. Iraaigg 2 0, (8)
where A 2 0 is the hyperparameter to control the balance between target sample correction and gradient discrepancy for model prediction. is casy to obtain that Eq. is a quadratic program (QP) in L-variables (the number of model parameters is usually high in neural networks). Fortunately, we can effectively solve this problem in the dual space via transforming as smaller QP problem with only one variable v (27). where the relation between primal and dual variable is Strain U - (1+ A)g = Itg. Then we have the following problem:
(1 + A)
min
(Strain U 9tg) (gtroin V + 9tg)
s.t. V 2 0.
(9)
2
It is easy to obtain the optimal dual variable v² = minf Oeeiin Otg
0} and the optimal rewired gradient g" = (1 +1+ A) (gtg v" Strain). In other words, the gradient rewiring procedure is quite
gieain geewin
simple: for the gradient of the target loss Utg. reduce its projection component on Itrain and then scale it by (1 + A)-1
highlight that the gradient for training loss Itrain must be stored before model editing. In this way, gradient rewiring can be conducted to remove the harmful gradient component on target loss that increases training loss. Since shallow GNNs model performs well in practice (8),, the model size of GNNs is small and the memory cost (L) for storing anchor gradient is negligible.
Additionally,
GRE+ In GRE, the training loss after model editing is required not to be larger than that before model editing. However, it is still possible that the training loss on specific sub-training sets performs worse after model editing. At the same time, the training loss for the whole training dataset, after model editing, is on par with or even lower than that of before editing. To tackle this issue, proposed advanced gradient rewiring approach, named GRE+, via applying loss constraint on multiple disjoint sub-training sets. Specifically, we split training dataset Verain into K
sub-training sets (Vtrain: V2 Verain: ... yk ). Similarly, we define grrain Eieys
ack actmm
R.. where
09
C VE Perain
(E(ff(x,) y.).
train
5

---

Following the derivative clue in GRE, we can replace the training loss constraint on the whole training dataset with multiple training loss constraints on training subsets, and obtain the advanced gradient rewiring approach as follows:
min 2 E11g- Sty 12 X =0loll min 1 1+ 9
1 T
9tg + 2
2
s.t. (gtrain)
g 2 0, for any S S K.
(10)
Notice that 14) is quadratic program (QP) in L-variables (the number of model parameters are usually high in neural networks), and we can effectively solve this problem in the dual space via transforming as smaller QP problem with only K variables v E R (27. Define gradient matrix as
G Inrainn Prriin ... Ihrainl E Rxx,, then the relation between primal and dual variable is given by Gv (1+ A)g following dual problem:
Itg- The original optimization problem can be transformed into the
(1+ V GG v v+
min
+L+ + (1+ 1 9tg9tg
2 V& 2. for any k
S.t.
(11)
The dual problem is a QP with K < L variables, and we usually consider the value of K to be smaller than in practice. Once we tackle dual QP problem (IT) for v*, we can recover the rewired gradient as g (1 + x) + 9tg). Similarly, the gradient for training loss girain; where k < K, is required to be stored before model editing. The corresponding memory cost is given by ((L)).
Table The results on four small-scale datasets after applying one single edit. The reported number averaged over 50 independent edits. SR is the edit success rate, Ace is the test accuracy after editing, and DD are the test drawdown, respectively. "OOM" is the out-of-memon error. The best/second besst results are highlighted in boldface/underlined respectively.
Editor
Azc?
DDJ
SEI
Aoc1
Ascompatess
DDL
SR1
AseT
A-plunoo
DDJ
Aeet
Couthee:
DDJ
15
3385-033
03%
6.788048
6.81-0:91
1.00
0:31-0015
1.00
MIP
ENN
37.16s3.80
44
52.24e4.76
2.50
096
1.00
15.51=10.9
61.21a1.26
72.36x1088
18.79e1.26 26
16.71:1481
73.56e1
77.07:15.20
16.44=1.41
1.00
1.00
93.27401.09
4.94-33.78
80.43+3.34
0.73a0000
1.00
1.00
GRB-
71. 71.1940.2
0.81-0.28
0.96
11.2241-55
18.23+1.15
1.00
78.25e1 15
11.71e1.1 .15
1.00
73.0330000
0.27.0:07
1.00
04.334584
503-6.40
1000
41
23.09922.92
20-21
25.08-2013
1.00
9122243338
3.93-3.31
1.00
GCN
ENN
GRC
3a 47
37.1613.30
52.24
4.0250447
0.6
100
15.51
38.
72.36210.87
51.7243.47
0.98
1,00
16.71=14.81
35, 03.88-2222 26
71.07115.20
36. 12-23.26
0.99
1.00
4,9413.78
29.466002
39.4323.34
4.54.0.2 29
1.00
1.00
GRE-
88
8440.000 38
0.660.35
0.6
7540044 45
40 2540.45
1,00
50.13e1..3
36
49.87=1.36
1.00
0060000000 10
2.01+0.30
1.00
GD
32.0123133
4.3
100
01 1112012
1,00
35
35
55.0222035
1.00
90.153335 3N
5.0123.32
1.00
Gagh
ENN
45
53.4422.23
1000
16.9911698
65 141116675
1,00
15.06=11.92
1.92
19.
1.00
13.71=273
$1.4522.11
1.00
SAGE
GRE
50.6440.20
20
1.00
20.1142.300
62.9922.30
0.96
41.9611.7
12
0.98
91.07
91.0760.44
3,9320.44
1.00
GRD-
86
50
4.400.077
100
22 2361 600
60. 77&1.60
0.97
44.05-0.83
5033-0.83
1.00
91. 45.464
3,25+0.43
1.00
GD
31
100
0.75
05.7
BGNN-
GRE
41
1.3
1.00
13.3811.20
4.221.20
0.87
5.9921.20
0.86
93.9221107
0.0820.11
0L94
GCN
GRE-
68 0110
0.05-0.21
100
88. 10+1.1
0.51+1.21
1,00
94. 84.2240000 08
1.00
94 5240.00
-0.52+0.06
1.00
GD)
1101
0.30
0.81
3550010011
0.90
BGNN- SAGE
GRC
14.7900.19
121 19
1.00
81 11.9911.71
3.06+1.71
0.96
$3.55e1.10 19
3.6561119
0.95
94.85+0008
0.15.0.05
1.00
ORE+
86 23
-02441143 143
1.00
85. 9750.3
-O. 16:083
1.00
94.07
-0 -0.0000000
0.98
93. 05.07750.00
-0.0750.00
1.00
4 Experiments
In this section, conduct experiments to evaluate the effectiveness of our proposed GRE and GRE+, with the goal of answering the following three research questions. RQI: Can the proposed solution correct the wrong model prediction with lower accuracy drop after model editing in the independent and sequential editing setting? RQ2: What's 's the tradeoff performance between accuracy drop and success rate in the independent editing setting? RQ3: How sensitive are the proposed GRE and GRE+ methods to the key hyperparameter X?
4.1 Experimental Setting
We follow the standard experimental setting for GNNs 14. Specifically, we first randomly split the train/validation/tes dataset. Then, we ensure that each class has 20 samples in the training and 30 samples in the validation sets. The remaining samples are used for the test set. The target node is randomly selected 50 times from the validation set and the well-trained model makes the wrong prediction. The average model editing performance (e.g., success rate, drawdown) is reported for evaluation.

---

Datasets and Models. In our experiments, we utilize a selection of eight graph datasets from diverse domains, split evenly between scall-scale and large-scale datasets. The small-scale datasets include Cora, A-computers [29] A-photo [29] and Coauthor-CS CS [29 On the other hand, the large-scale datasets encompass Reddit 25 Flickr 2], ogbn-arxiv B and agbn-products [.. Note that our approach is based on gradient rewiring, which is orthogonal to model architectures. We adopt two prevalent models GCN [24] and GraphSAGE (25), where both of them are trained with the entire graph at each step. We evaluate our method under the inductive setting, which means the model is trained on a subgraph containing only the training node, and evaluated on the whole graph.
Baselines. Our methods are evaluated against three notable baselines: the traditional gradient descent editor (GD), the Editable Neural Network editor (ENN) [IO], and editable training for GNNs[14]. The GD editor is a straightforward application using gradient descent on the target loss with respect to the GNNs model parameters until the desired prediction outcome is achieved. ENN adopts a different approach by initially training the GNN parameters for a few steps to prime the model for subsequent edits. After this preparatory phase, ENN, like GD, applies the gradient descent on the parameters of GNN until the correct prediction is attained. EGNN [44] stitches peer MLP and only trains MLP during model editing. Note that our method is compatible with EGNN, and different GNN architectures integrated with EGNN (e.g. EGNN-GCN, BGNN-GraphSAGGN are treated as distinct architectures.
Independent, sequential, and batch editing. All independent, sequential, and batch editing processes involve well-trained GNN models using training datasets, with target samples randomly selected multiple times from misclassified instances in the validation dataset. The key differences lie in the base model that needs to be edited. For independent editing, the same well-trained model using the training datasets is edited multiple times. In contrast, for sequential editing, the model is edited iteratively, with each editing step using the previously edited model from the last target sample, incorporating both the training datasets and partial samples from the validation dataset. For batch editing, all batched samples are edited simultaneously in one editing process.
4
Evaluation Metries. Consistent with preceding studies 100 12 II assess the effectiveness of the various methods using two primary metrics: (1) Accuracy (Acc): We use accuracy for the test dataset to evaluate the effectiveness after model editing. (2) DrawDown (DD): This metric measures the mean absolute difference in test accuracy before and after model editing. A lower drawdown value signifies a superior editor locality. (3) Success Rate (SR): This metric evaluates the proportion of edits in which the editor successfully amends the model's prediction. Both metrics offer a different perspective on the effectiveness of the editing process.
Table 2: The results on four large-scale datasets after applying one single edit. "OOM" the out-of-memory error. The best/secondd-ber results are highlighted in boldfacc/underlined, respectively. The results for more backbones (e.g.. MLP, EGNN-GCN EGNN-SAGER are in Appendix D.I
Editor
Fliskr
Reldi
ogha
products
og5n
Acc
D0
SRT
Ape
D03
SR.
SKT
Acc
DD1
SRT
G13
13
1300
37.2511130
1.00
25.2022..30
20.32211.30
1.00
23.11
1.00
33. 33.3
20.1220
T300
GUN
GRF
RNN
25. 64 96
17.
17.36+155 50
25. 18.38816000
$9.64-1.50
0:98
1.00
24.744 34.74+1.90
11 (1.1665.0 10
45.26-1.92
56 16
1.00
1.00
27 77.51+1.16
16.
16.5967.00 20
18. 1.11
55.226 70
1.00
1.00
53.9000.0
00M
60
2001+0000
O0M
00M
1.00
GRF+
22.9-0.57
28.17
0:07
15+15
1..3
1.00
80.61-1.10
15.59-1.10
1.00
57.43-1.0
330
16.50+1.8 30
1.00
GD
17 7116:12.20 2:20
3138=1220
1.00
55 85122.30
01.T4220.50 30
1.00
19
1.00
02 22
10
1.322110
100
Gapph SAGE
EXN
26, 71:5500
20.91-5.40
1.00
5.8863.00
30.6654.30
1.00
8.1428.60
47.6127.00
1.00
DOM
00M
00M
GRE
20.691162
26.3121.22
0.59
21 21.9540.94
47.0720094
1.00
47.16+1 1.22
48 48.8451.22
1.00
63.9611
4.58a1.02
1.00
GRE
38. 48.41.1.17
10.59s1.17
0.52
29 29.2622.00 10
39. 39.7462.10
10
1.00
58 58.2942.35
37.7142.35
1.00
63. 63.2662.25
3.2942.25
1,00
4.2 Experimental Results in the Independent and Sequential Editing Setting
In many
scenarios. well-trained models often produce inaccurate predictions on unseen data. To evaluate the practical effectiveness of editors for independent editing (RQ1), we randomly choose nodes from the validation set that were misclassified during the training. The editor is then
'MEND M and SERAC [12 are tailed for NLP application and are hard to extend to the graph area. MEND requires caching the input each weight. Unfortunately, for graph data, the model edits cannot be done in a mini-batch way since the inference still runs whole-batch, i... MEND requires caching the whole graph embedding at each layer.
"The experimental results batch editing are in Appendix D.4
7

---

applied to rectify the model's predictions for these misclassified nodes, and we evaluate the drawdown and edit success rate on the test set.
We edit one random single node 50 times and report the mean and standard deviation results in Tables and 2 for small-scale and large-scale graph datasets, respectively. Our observations made below:
GCN
SAGE
80
GD GRE
drseetoen CS)
60
GRE
40
1400
20
Cora
A.computers
A-photo
Coauthon:C
Arcompuners
Appooo
Coauthon-CS
80
test cro.down MM3
GRE GRE+
GRE
test, Sr..errdennnn
60
GRE
20-
Flickr
Reddit
Anxiv
Flickr
Reddlt
Andv
Figure 2: The test accuracy drawdown in sequential editing setting for GCN and GraphSAGE on various datasets. The units for y-axis percentages (%).
Contrasting model editing on textual data (77 72 30y, all editors can effectively rectify model predictions in the graph domain. As shown in Table all editors achieve high success rate (typically from 96% ~ 100%) after editing GNNs, which is highly different from transformers with below 30% SR. This finding indicates that GNNs, unlike transformers, can be more easily adjusted to produce correct predictions. However, this improvement comes at the expense of substantial drawdown on other unrelated nodes, underscoring the key challenge of maintaining prediction locality for unrelated nodes before and after editing.
@ Our proposed GRE and GRE+ notably surpass both GD and ENN in terms of test drawdown. This advantage stems mainly from the rewired gradient based on the pre-stored training loss gradient, which facilitates target sample correction while preserving the training loss. GD and ENN attempt to rectify model predictions by updating the parameters of GNNs without incorporating training loss information. In contrast, GRE and GRE+ maintain much better test accuracy after model editing. For example, for amazon-photon the accuracy drop dwindles from roughly 65.08% to around 43.87 43.87%, a 43.9% improvement the baseline. This is due the gradient rewiring approach that facilitates target sample correction while preserving the training loss. Interestingly, when applied to GNNs, ENN performs markedly worse than the basic editor GD. Moreover, GD) performs well in MIP, which is consistent with the low gradient discrepancy of MLP.
Our proposed GRE and GRE+ compatible with EGNN and further improve the performance. We observe that while GRE occasionally underperforms, GRE+ consistently shows better performance than GD in reducing accuracy. For instance, when the A-computers dataset is evaluated with EGNN- GCN, GRE, and GRE+ exhibit an average accuracy drop of 4.62% and 0.51% respectively, whereas find that for 7 out of 8 datasets, GRE+ with EGNN-
GD shows decrease of 0.73%. Notably, SAGE shows a negative drop in accuracy, meaning that the test accuracy actually increases after model editing. This points towards the superior performance of the EGNN-SAGE model architecture.
In the sequential editing setting, we select a sequence of nodes from the validation set that were misclassified during the training phase. The editor is then used iteratively correct the model's predictions for these sequentially misclassified nodes, and we measure the resulting drawdown and success rate of edits the test set.
In Figure 2 we report the test accuracy drawdown in the sequential setting, more challenging scenario that warrants further investigation. In particular, we plot the test accuracy drawdown compared to GD across various GNN architectures and graph datasets. Our observations are as follows: t The proposed GRE and GRE+ consistently oulperform GD in the sequential setting. However, the drawdown significantly higher than the single edit setting. For instance, GRE+ exhibits 43.87% drawdown for GCN on the A-photo dataset in the single edit setting, which

---

escalates up to a 65% drawdown in the sequential edit setting. These results also highlight the challenge of maintaining the locality of GNN prediction sequential editing. @ The improvement of GRE+ over GRE is quite limited in the sequential setting. For example, GRE+ exhibits a 24.52% drawdown over GRE for GCN on the A-photo dataset in the single edit setting while is on par with GRE in the sequential edit setting. These results further verify the difficulty of sequential editing and indicate more comprehensive training subset selection may be promising.
Cora
A-photos
Flickr
ogbn-arxiv
GCN
100.00
09.00
89
BB
GBE+2
G
100.0
44.5
91.3
90
99
104
100
98:32
09.2
99.2
63
08:00
Test
Test
304
100.00
102
Graph
302
102
SAGE
100
23.37
42
100
25.
80
0 57
27
Test D0
Test 00
Test DO
Figure 3: The success rate and test accuracy drawdown tradeoff in independent editing setting for GCN and Gra on various datasets. The trade-off curve close to the top left corner means better trade-off performance. The units for x- and y-axis are percentages (%).
4.3 Trade-off Performance Comparison
We further compare the trade-off between the accuracy drawdown and the success rate of our method various GNN architectures and graph datasets. As shown in Figure 3 we plot Pareto front curves by assigning different hyperparameters for the proposed methods. The upper-left corner point represents the ideal performance, i.e., the highest SR and lowest accuracy drawdown. The results show that GRE+ achieves better trade-off results compared to GRE, and all methods consistently maintain high success rate on various GNN architectures and graph datasets.
Cora
Flickr
ogbn-anxiv
Test DD
GBE
45
GCN
08213
28
0.0
011
1.0
10.0
50.0
0.0
0.1
1.0
10.0
50.0
Hyperpameter
Hyperpameter A
Hyperpameter A
50.0
Graph -SAGE
Test DD
25
47.5
42.5
10.
10.0
10.0
0.0
1.0
10.0
50.0
Hyperpameter. Figure 4: The hyperparameter study on test accuracy drawdown in independent editing setting w.r.t. A.
Hyperpameter
Hyperpameter
4.4 Hyperparameter Study
In this experiment, we investigate the sensitivity of our proposed method w.r.t across a va- riety of GNN architectures and graph datasets. Specifically, we search for X from the set of {0.0, 0.1, 1.0, 10.0, 50.03. As shown in Figure 4 the test accuracy drop remains relatively sta- ble despite variations in X, suggesting that meticulous tuning of this parameter may not be crucial. For the ogbn-arxiv dataset, an uptick in accuracy drop corresponds with an increase X, reflecting the inherent difficulty of this dataset. Intriguingly, in the case of GRE+5 with 5 training subsets, the test accuracy drop exceeds that of GRE+2 and GRE+3, a pattern that diverges from the trend observed in other datascts.

---

Conclusion
In this paper, we explore the editing of graph neural networks from new gradient perspective. Through empirical observations, we discover that conventional model editing techniques often underperform due the gradient discrepancy between the training loss and target loss in GNNs. To address this issue, we propose a gradient rewiring approach. Specifically, we formulate constrained optimization problem to regulate the model performance during model editing and identify simple yet effective gradient rewiring approach to explicitly satisfy the constraints. In this way, the proposed approach can correct the target sample while preventing an increase training loss. Experiments demonstrate the effectiveness of our approach, and our proposed method is also compatible with the existing bascline EGNN and can further improve performance. Future work includes more comprehensive training subset selection in GRE+ and a tailed approach for editable graph neural networks training in the sequential editing setting.
Acknowledgements
The authors thank the anonymous reviewers for their helpful comments. This work is in part supported by NSF grants NSF IIS-23 12260, IIS-2224843, IIS-245062,, IIS-2431515 and IIS-2239257. The views and conclusions contained this paper are those of the authors and should not be interpreted as representing any funding agencies.
10

---

References
[1] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 983,9983244 2018.
[2] Hanqing Zeng. Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method. In International Conference on Learning Representations, 2020.
[3] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00667 2020.
[4] Zhimeng Jiang, Xiaotian Han, Chao Fan, Zirui Liu, Na Zou, Ali Mostafavi, and Xia Hu. Fmp: Toward fair graph message passing against topology bias. arXiv preprint 2202::2200.041888. 2022.
[5] Keyu Duan, Zirui Liu, Peihao Wang, Wenqing Zheng, Kaixiong Zhou, Tianlong Chen, Xia Hu, and Zhangyang Wang. A comprehensive study on large-scale graph training: Benchmarking and rethinking. arXiv preprint arXiv:22 10.07494, 2022.
[6] Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. In International Conference on Machine Learning, pages -248- PMLR, 2022.
[7] Hongyi Ling. Zhimeng Jiang, Meng Liu, Shuiwang Ji, and Na Zou. Graph mixup with soft alignments. In International Conference on Machine Learning. PMLR, 2023.
[8] Daniele Petrone and Vito Latora. A dynamic approach merging network theory and credit risk techniques to assess systemic risk in financial networks. Scientific Reports, 8(1):5561, 2018.
[9] Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22-36, 2017.
[10] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. arXiv preprint vrXiv:2004.00345 2020.
[11] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:21 11309, 2021.
[12] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing scale. In International Conference on Machine Learning, pages 15817-15831. PMLR, 2022.
[13] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint 1eXix:200400866 2021.
[14] Zirui Liu, Zhimeng Jiang, Shaochen Zhong, Kaixiong Zhou, Li Li, Rui Chen, Soo-Hyun Choi, and Xia Hu. Editable graph neural network for node classifications. arXiv preprint arXiv:2305. 15529, 2023.
[15] Shaochen Zhong, Duy Le, Zirui Liu, Zhimeng Jiang, Andrew Ye, Jiamu Zhang, Jiayi Yuan, Kaixiong Zhou, Zhaozhuo Xu, Jing Ma, et al. Gnns also deserve editing, and they need it more than once. In International Conference on Machine Learning, 2024.
[16] Yezi Liu, Qinggang Zhang, Mengnan Du, Xiao Huang, and Xia Hu. Error detection on knowledge graphs with triple embedding. In 2023 31st European Signal Processing Conference (EUSIPCO), pages 1604-1608, IEEE, 2023.
[17] Junnan Dong, Qinggang Zhang, Xiao Huang, Keyu Duan, Qiaoyu Tan, and Zhimeng Jiang. aaear multi-hop question answering over knowledge graphs. In Proceedings of the ACM Web Conference 2023, pages 2519-2527, 2023.

---

[18] Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang. and Neil Shah. Graph condensation for graph neural networks. arXiv preprint 1rX1v:2110.07580, 2021.
[19] Yezi Liu and Yanning Shen. Tinygraph: Joint feature and node condensation for graph neural networks. arXiv preprint arXiv:2407.08064 2024.
[20] Qizhang Feng, Zhimeng Stephen Jiang, Ruiquan Li, Yicheng Wang, Na Zou, Jiang Bian, and Xia Hu. Fair graph distillation. Advances in Neural Information Processing Systems, 36:80644-80660, 2023.
[21] Pei Chen, Hang Yang, Kang Liu, Ruihong Huang. Yubo Chen, Taifeng Wang, and Jun Zhao. Reconstructing event regions for event extraction via graph attention networks. In Kam-Fai Wong, Kevin Knight, and Hua Wu, editors, Proceedings of the 1st Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 81 1-1820, Suzhou, China, December 2020. Association for Computational Linguistics.
[22] Pei Chen, Haibo Ding, Jun Araki, and Ruihong Huang. Explicitly capturing relations between entity mentions via graph neural networks for domain-specific named entity recognition. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pupers), pages 735-742, Online, August 2021. Association for Computational Linguistics.
[23] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263- 1272. PMLR, 2017.
[24] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
[25] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 1025-1035, 2017.
[26] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representutions, 2018.
[27] Jorge Nocedal and Stephen J Wright. Quadratic programming. Numerical optimization, pages 448-492, 2006.
(28) Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861-6871. PMLR, 2019.
[29] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05888. 2018.
[30] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-pptcher:: One mistake worth one neuron. In The Eleventh International Con- ference on Learning Representations, 2023.
[31] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008
[32] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 3762-3773. PMLR, 2020.
[33] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
12

---

[34] Cheng Chen, Ji Zhang, Jingkuan Song. and Lianli Gao. Class gradient projection for continual learning. In Proceedings of the 30th ACM International Conference on Multimedia, pages 5575-5583, 2022.
[35] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. On modulating the gradient for meta-learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings. Part VIII 16, pages 556-577 Springer, 2020.
[36] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
[37] Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive gradient-basee meta-learning methods. Advances in Neural Information Processing Systems, 32, 2019.
[38] Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu, Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3937-3946, 2021.
[39] Zhiqiang Gao, Shufei Zhang, Kaizhu Huang, Qiufeng Wang, and Chaoliang Zhong. Gradient distribution alignment certificates better adversarial domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8937-8946, 2021.
13

---

A Experimental Setting
A.1 More details on EGNN
We provide more details on editable graph neural networks (EGNN), method free of neighborhood propagation, designed specifically for correcting misclassified node predictions [14. EGNN uniquely integrates a peer MLP (matching in the number of hidden units and layers) with GNNs (such as GCN and GraphSAGE), and trains solely the MLP model using the target loss during model editing. This strategy enables EGNN to leverage the propagation-fres free advantages of MLPs for model editing. However, it's important to note that EGNN incompatible with EGNN, as the model editing in EGNN refines stitched MLP, which is not employed during model training. Our proposed methodologies, GRE and GRE+, are founded on gradient rewiring, which is orthogonal to the EGNN approach. In the appendix, we illustrate how our proposed methodologies can further augment the effectiveness of the EGNN approach and MLP models.
Table Statistics information for datasets used for node classification.
Datasets
Cora
A-computers
A-photo
Coauthor-CS
Flickr
Reddit
ogbn-arxiv
ogbn-products
# Nodes
2,485
13,381
7,487
18,333
89,250 899,756
232,965 23,213,838
169,343
2,449,022
# GREes
5,069
245,778
119,
81,894
1,166,243
61,859,140
Classes
7
10
41
47
Feat
1433
767
745
6805
500
602
128
218
A.2 Datasets
The statistical information of all datasets is summarized in Table The details of the datasets utilized for node classification are described as follows:
Cora This citation network comprises 2,708 publications interconnected by 5,429 links. Each publication is characterized by 1.433-dimensional binary vector that signifies the presence or absence of specific words from a predetermined vocabulary.
A-computers [29]: This dataset is segment of the Amazon co-purchase graph. In this network. nodes denote goods, and GREes represent frequent co-purchases of two goods. Node features are encoded as rag-of-wordd product reviews.
A-photo [29]: Similar to A-computers, this is another segment of the Amazon co-purchase graph. Node features are also bag-of-words encoded product reviews.
Coauthor-CS [29]: Derived from the Microsoft Academic Graph from the KDD Cup 2016 challenge this co-authorship graph has nodes representing authors who are linked they have co-authored a paper. Node features denote paper keywords for each author's publications, while class labels indicate an author's most active fields of study.
Reddit (55):
This dataset is formulated from Reddit posts, with each node representing post. associated with different communities.
This dataset represents the citation network among arXiv papers. Each node denotes a paper, and each GREe signifies a citation between two papers. Node features are generated from the average 128ddimensional word vector of each paper's title and abstract.
ogbn-arxiv [3]
ogbn-products 3]:
This is an Amazon product co-purchasing network, where nodes represent Amazon products and GREes denote co-purchases of two products. Node features are created from low-dimensional representations of product description text.
A.3 Implementation Details
The hyperparameters for model architecture, learning rate, dropout rate, and training epochs are shown in Table 4 For EGNN, we also adopt GNNs and MLPs with hyperparameters in Table For GRE, we use the hyperparameters 7 {0.0, 0.1, 10.0, 50.03. For GRE+, we also select hyperparameters 7 = (0.0, 0.1, 1 1.0, 10.0, 50.0} and K (1,2,3, 5). As for QP problem Eq. (11). use standard package qpsolvers with version 3.4.0 to tackle this QP problem with ecos solver.
14

---

Table 4: Training hyperparameters configurations in the experiments
Model
Configuration Cora A-computers A-photo Coauthor-CS. Flickr Reddit oghn-arxiv oghn-products
#Layers
2
2
2
3
#Hidden
32
32
32
32
256
256
128
256
Graph- SAGE
Ir
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.002
Dropout
0.1
0.1
0.1
0.1
0.3
0.5
0.5
0.5
Epoch
200
400
400
400
400
400
500
500
#Laaers
#Hidden
32
32
32
32
256
256
128
256
GCN
Ir
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.002
Dropout
0.1
0.1
0.1
0.1
0.3
0.5
0.5
0.5
Epoch
200
400
400
400
400
400
500
500
#Layers
2
2
2
4
2
3
3
#Hidden
32
32
32
32
256
256
128
256
MLP
Ir
0.01
0.01
0.01
0.01
0.01
0.01
0.01
0.002
Dropout Epoch
0.1
0.1
0.1
0.3
0.5
0.5
200
400
400
400
400
400
500
500
A.4 Running Environment
For hardware configuration, all experiments are exccuted on a server with 251GB main memory, 24 AMD EPYC 7282 16-core processor CPUs, and single NVIDIA GeForce-RTX 3090 (24GB). For software configuration, we use CUDA=11.311 python=3.8.0, pytorch=1.1211, higher=0.2.1. 1,,. barch-geometric-11.222 torch-sparse=006.166 in the software environment. Additionally, package of higher in https://gihbbbooomm/rrmmmichchhh for ENN implementation.
the
B Limitations and Discussions
While our proposed GRE and GRE+ methods effectively mitigate the accuracy dropdown compared conventional gradient descent algorithms, the success of approaches largely contingent the precision of the pre-stored gradient for training loss. Despite the relatively few required model edit steps for single node editing, the accuracy the pre-stored gradient may not sustain over long-term sere-term model editing, as the pre-stored gradient for training loss could exhibit significant discrepancy from the gradient of training loss for the edited model. address such discrepancy, straightforward strategy could involve leveraging critical training samples to estimate the true gradient of training loss for the edited model. Another possible direction is to identify critical samples instead of random samples for GRE+ with the aim of further constraining the model's 's behavior before and after model editing.
Notice that the proposed gradient rewiring method is not inherently specific to graphs, the gradient rewiring method is particularly suitable in the graph domain due the small model size. Specifically, graph models are typically a few layers and thus are smaller in model size compared to models (e.g.. Transformers) used in NLP and CV tasks. This results lower computational and storage costs for gradients, making our strategy particularly suitable for the graph domain. Additionally, it is more challenging edit nodes in a graph due the inherent propagation process within neighborhoods. Such propagation may lead to significant gradient discrepancies within the graph domain.
C Algorithms
We show the algorithms of GRE and GRE+ during model editing in Algorithm and respectively.
D More Experimental Results
In this section, we present experimental results showcase the improved efficacy of our proposed methods, GRE and GRE+. These techniques enhance the performance of EGNN, a specifically designed editable graph neural network, across both independent and sequential editing settings.
15

---

Algorithm Gradient Rewiring Editable (GRE) Graph Neural Networks Training
1: Input: Target samples Kig), hyperparameter X, well-trained GNN model fo(-), and its
corresponding gradient for the training subgraph.
2: Output: Updated GNN model (-)
3: while (X # Yu do
4: Compute the model gradient 9rg for the target loss Lgg
5: Rewire the target loss gradient Ohg by reducing the projection component on Grain then scale
with (1+
6:
(1 A) (Ukg v" (mmiin))
7: Replace 9ig with g" and update the model parameters using the optimizer to obtain 6'.
8: end while
Algorithm 2 Gradient Rewiring Editable Plus (GRE+) Graph Neural Networks Training
1: Input: Target samples (Xtg.Ytg), hyperparameters A, well-trained GNNs model fo(-), and its
corresponding model gradient for training subgraph.
2: Output: Editable GNNs model før (+).
3: while {x = Yig do
4: Compute model gradient Utg for target loss Lig-
5: Solve QP problem Eq. (11 via standard QP solver package and obtain the optimal dual
variable v*.
6: Calculate the rewired gradient using g" (1 + A) (Gv" + gtg).
7: Replace Sig with 9" and then adopt optimizer to update model parameters as 0'.
8:
end while
D.1 More results on Model Architectures for Independent Editing
In this section, conduct sequence of 50 single-node edits and present the and standard de- viation results in for large-scale graph datasets. Similarly, GRE occasionally underperforms, and GRE+ consistently shows better performance than GD with respect the reduction in accuracy. For instance, when the Reddit dataset is evaluated with EGNN-GCN, GRE and GRE+ exhibit an average accuracy drop of 1.48% and -..21 %, respectively, whereas GD shows a decrease of 1. 1.28%. Moreover, GRE+ with EGNN-SAGE shows negative drop in accuracy among 6 out of 8 datasets, i.c., the test accuracy actually increases after model editing.
Table The results on four large scale datasets after applying one single edit. "OOM" the out- of-memory emror.
B
Flickr
Rabdit
oglo- arkiy
produces
Apc
DD.
SR
Acc!
DDJ
SRT
Ace
DDL
SKI
ApsT
DDJ
SRT
035
35 3434 1.3
11.1691
0.3
45.27 15.27+00:77
7.13-0017
[139]
70.31.0044
2.004 4B1
1.00
74 13x3335444
O 0.20F:140
T.B0
MLP
ENN
CRE
25.
26.1240.57
54:52214900 90
25. 35.58116.90
10.5340.57
0.81
1,00
11.1645.0 10
35.85+2.00
64.36e3 10
17.1582.0
1.00
1.00
62. 20.2050.00 94
16.99e7. 70
10,060 94
$3.6226 70
1.00
1,00
53.9940 60
O0M
20.0110.60
GOM
1.00
C
GRE+
43-2340.1 17
3.7F40.17
0.84
11.3340.87
11.020087
0.99
64.1140 95
5.4040 95
1.00
57-434 30
10.5241
1.00
4.3030.91
08.72660.55
1.250.55
56,096003 83
0:0829.17
73.5040112
0.2750.12
EGNN
GRE
45,7010997
3.300099
68.52a01
1.48,0.51
3.2221.34
73.6560 16
0.3520.16
1.00
OCN
GRE+
50.6050.11 15
0.900..15
0.99
0740093
0.2110.38
(3.5),50.55
2. 2.316045
0669.4
-0.8050.45
1.00
1.2221.13
U.11.9006
EGNN-
12.25:1.64
3.75x1.64
67.3461135
1.64e0035
94..9911.25
1.911129
75.9040005
0,1060.115
1.00
SAGE
05x1.42
0.11-0.78
-0.06w0.10
28.0.21
-0.12+0.31
D.2 Experimental Results other Model Architectures for Sequential Editing Setting
In the sequential editing setting, we take a sequence of 50 misclassified nodes and use the editor to iteratively correct the model's predictions for EGNN across different GNN architectures. The test accuracy drop associated with various model editing methods for different graph datasets is reported in Figure 7 Our observations indicate that the proposed GRE and GRE+ methods consistently outshine GD in this sequential setting. For instance, with the Coauthor- CS dataset and EGNN-GCN, our proposed methods achieve virtually no decrease accuracy, while GD exhibits a drop of over 7%. Another compelling observation is that the improvement demonstrated by our methods over GD for EGNN is markedly larger than for GNNs. This suggests potential synergies berween optimizer selection and model architecture design.
16

---

Cora
A A-photos
Flickr
ogbn-
58-
EGNN- GCN
0 100
102-
GR6-+2
96
51 100
96-
03
0.2
6
104
120
100.0
EGNN- SAGE
100
93.5
00.0
20-
82.3
Test DD
Test
DD
Figure 5: The success and accuracy dropdown tradcoff in independent editing setting for EGNN-GCN and EGNN-SAGE various datasets. The trade-off curve close to the top left corner means better trade-off' performance. The units for the x- and y-axis are percentages (%).
D.3 Trade-off Performance Comparison on other Model Architectures
We extend our evaluation by comparing the trade-off between the accuracy drop and success rate of method EGNN across various graph datasets. By adjusting different hyperparameters for the proposed methods, we construct Pareto front curves as shown in Figure 5 The results underscore that both GRE+ and GRE outperform GD in achieving superior trade-off outcomes. Importantly, our proposed methods exhibit robust preservation of the success rate across various GNN architectures and graph datasets.
D.4 More Experimental results on Batch Editing
In this section, we present the experimental results of applying batch editing on four small-scale datasets (Ta and four large-scale datasets (Table
For the small-scale datasets, our proposed methods, GRE and GRE+, consistently outperform the baseline methods. For example, on the Cora dataset, GRE+ achieves the highest accuracy with minimal drawdown and a high success rate. Specifically, GRE+ can reduce 56.9% and 30.3% drawdown compared with 2nd best baseline in GCN and GraphSAGE architectures, respectively. For datasets
the large-scale datasets, GRE+ again demonstrates superior performance. GRE+ can reduce 2.5% and 0.5% drawdown compared with 2nd best baseline GRE in GCN and GraphSAGE architectures, respectively, while maintaining high success rate.
Table The results on four small-scale scale datasets after applying batch edit. is success rale, Acc the test accuracy after editing, and DD are the test drawdown, respectively. The best/second results are highlighted in boldfaceeumderlined respectively.
Edior
Apc
DDD:
D01
Acc
D00
Acc
DDC
31 72355333
7.2531.44
0098522
0.53
44.3211.35
19.E113
67 678555
06.62705
100
GCN
CIRB
ENN
82 52.5642.47
32.1642
66 46.12e1
60643.29
0.95
0.93
25.91213.01
64.543477
23.351135
21.9914511
0.61
0.25
54.6526 26
9.99:031
-99960.9
90:4428.82
0.34
0.05
45.59&13.21
762265.25
-81 53.5511555
17.045877
0.62
1.00
GRF+
66 86-4040 36
280-0.55
0.97
65 36-354433
22 72-00665
0.63
57 87.83-1.9
36 13-92-9
0.50
76.80+9.0
17.204995
1.00
76.8551.3
K3251.70
1,00
2535511
330823335
38.8215521
47.02217332
0T/REE7.5
232117799
Graph
ENN
32.16:2.21
.3.8861.68
0.99
0.9940000
0.08:0.01
0.08
14.8157922
4.81118.23
17.5582.12
1.00
SAGB
CIRE
50.284117
B3224.17
0.99
16:3242.25
25.12410:43
0.54
16.00-11.7
32.05=20.55
0.46
75.6EE8.4 64
19.3248.99
1.00
GRE
82 87
4.40e1.07
1.00
51.24:12877
82.51415.77
0.62
02.00e11.8
41.40.22.99
0.52
76.116621
18.0946.30
1.00
Table 7: The results on four large-scale datasets after applying batch edit. "OOM" is the out-of of of-memory error The best/sccond-berd results highlighted in boldface/underiined, respectively.
ogto- undi
ogln- prodaces
51.21=12:55
32.7754.00
65.70+3.11
24.47s3.77
20.10s5.31
24.74s1192
41.96a7.30
28.0447.51
0.62
20.1666
25.0026.47
52.614425
NC1=11220
273067777
4033864188
Grsph-
-15.10e5.0
1..9222.46
23.0863.06
38.65a7.22
30.35s8:44
16.31s4.02
27.97s1.17
21.0167.99
38.0147.32
26.2443.47
17

---

D.5 The Edit Time and Memory Comparison for Editing Methods
In this section, we present the experimental results of the edit time and memory required for editing across four large-scale datasets (Table 8
We observe that GRE+ takes 1.5 2.5x wall-clock editing time than the GD/GRE editor in terms of the wall-clock edit time. This is because GRE+ requires QP solver to obtain the rewired gradient. In terms of memory consumption, the overall memory overhead insignificant. For example, GRE+ (5) requires 17.9% GPU memory than GD editor in obgn-products dataset and GCN architecture. The reason that the anchor gradient is required to store in memory and QP solver computation in memory.
Table 8: The edit time and memory required for editing. ET (ms) and PM (MB) represent the edit time in milliseconds and peak memory megabytes, respectively.
Flickr
Reddit
ogbn-arxiv
ogbn-products
Editor
(ms)
PM (MB)
ET (ms)
PM (MB)
(ms)
PM (MB)
(ms)
(M
GD
67.46
707.0
345.23
3244.8
94.38
786.2
2374.15
14701.7
ENN
109.82
666.8
405.24
3244.8
242.85
786.2
OOM
GRE
63.93
695.8
391.54
3491.3
84.74
956.9
2400.78
17336.6
GCN
GRE+ (2)
100.45
696.0
457.08
3493.2
121.11
957.8
2413.69
17338.7
GRE+ (3)
115.29
697.9
509.44
3493.9
131.06
957.9
2471.23
17338.9
GRE+ (5)
155.05
698.6
603.85
3495.6
162.24
958.3
2591.06
17339.2
GD
117.74
843.0
1024.12
4416.53
107.63
891.3
2125.07
13832.2
ENN
134.50
843.0
2597.21
4416.5
277.29
891.3
OOM
Graph- SAGE
GRE
116.03
952.4
1089.29
4955.4
100.09
1072.5
2132.02
16254.1
GRE+ (2)
167.17
954.5
1267.13
4959.0
136.28
1073.7
2135.88
16255.9
GRE+ (3)
176.66
955.5
1363.53
4960.7
154.29
1074.0
2211.63
16256.0
GRE+ (5)
219.81
957.5
1603.03
4964.2
180.73
1075.5
2275.72
16256.3
D.6 More Test Accuracy Results on Sequential Editing
In this subsection, present the after-editing test accuracy results of applying sequential editing various datasets for both GCN and GraphSAGE models Figure The test accuracy is reported as percentage for each dataset.
Overall, our proposed methods, GRE and GRE+, consistently outperform the baseline methods GD and ENN across all datasets in terms of test accuracy. For example, on the Reddit dataset, the proposed methods can achieve more than 100% improvement over GD and ENN in terms of accuracy. Besides, compared with GRE and GRE+, the improvement is marginal most occasions except A-computer dataset in GraphSAGE, which indicates the limited effectiveness of the fine-grained gradient rewiring in GRE+.
E
More Related Work
Gradient-based method for other tasks. The existing literature on gradient modification mainly incorporates continual learning and meta learning. In continual learning, work 32) proposes gradient projection methods to update the model with gradients in the orthogonal directions of old tasks, without access to old task data. GPM (33 identifies the bases of these subspaces by examining network representations after learning each task using Singular Value Decomposition (SVD) in single-shot manner and stores them memory as gradient projection memory. Class gradient projection is proposed in (3 to address the class deviation in gradient projection. In meta-learming work [35) proposes meta-learning algorithm to learn to modulate the gradient in the absence of abundant data. The implicit model-agnostic meta-learning (iMAML) algorithm is developed (36] for optimization-base meta-learning with deep neural networks that remove the need for differentiating through the optimization path. 37 provides theoretical framework for designing and understanding practical meta-learning methods that integrate sophisticated formalizations of task-similarity.
18

---

GCN
SAGE
80
80
ENN
test 136)
60
GRE
test acc 1%)
60
GRE+
40
40
20
Cora
A-computers
A-photo
Coauthor-CS
Cora
A-computers
A-photo
Coauthor-CS
40
60
test. acc (%)
test acc (%9
20
20
Flickr
Reddit
Anxiv
Flickr
Reddit
Andiv
Figure 6: test accuracy in sequential editing setting for GCN and GraphSAGE on various datasets. units for y-axis are percentages (%).
The
F
More discussion
Comparison with Curriculum Learning. Curriculum learning and model editing two distinct approaches in the field of machine learning. Curriculum learning is an approach where the network is trained in structured manner, starting with simpler tasks and gradually introducing ones. This method aims to improve the learning process by mimicking how humans learn. Model editing is fast and efficient approach to patch the well-trained model prediction for several failed test cases. Although both are multi-stage training stages, there are several key differences: (1) Goals: Curriculum Learning aims improve the overall learning process by structuring the training data in a way that mimics human learning. In contrast, model editing aims to make targeted adjustments pre-trained model correct undesirable behaviors. (2) Approach: curriculum learning mainly focuses on the sequence and complexity of the training data. Model editing typically modifies the model's parameters architecture correct undesirable behavior goals. (3) Additional information in the multi-stage process. Model editing requires failure feedback for well-trained models as the target samples patch, e.g., test failure cases after production is launched. other words, such feedback can only be obtained after model pertaining. In curriculum learning, all information is given in multi-stage training. In summary, curriculum learning focuses on structuring the training process to improve overall learning, while model editing focuses making targeted adjustments to pre-trained model to correct specific behaviors. Both approaches can be complementary and used together to achieve better model performance.
complex
the best our knowledge, many existing methods integrate source and target gradients in the loss function. For example, (38) aims to minimize the gradient discrepancy for unsupervised DA, and (39 aligns gradient distribution for better adversarial DA. However, these methods can not be applied in graph model editing since (1) gradient discrepancy is required to successfully edit model prediction; (2) model editing collapses (i.e., no gradient discrepancy) at the initial stage; (3) regulating gradient behavior is insufficient for model editing task since the main problem is how to get an edited model instead of cross cross-domain generalization. To this end, rewire the gradient before the model parameters update, and derive a closed-form form, instead of learning based, gradient rewiring method accelerate model editing.
Comparison with Domain Adaptation. in domain adaptation (DA) 38 39
19

---

EGNN-GCN
EGNN-SAGE
GD
GD
test drswdown (MG
GRE
test. drawdown drawdown
GRE+
GRE+
10
Cora
Accomputers
A-pheto
Coauthor CS
Cara
A.computers
A-phota
Coauthar
GD
GRE
test drawdown (S))
GRE+
15
GRE
10
Flickr
Reddit
Andi
Flickr
Reddit
Figure The accuracy dropdown in sequential editing setting for EGNN-GCN and EGNN-SAGE, various datasets. units the y-axis are percentages (%).
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We list main claims and contributions as the bullet items at the end of the introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Please check Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
20

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: In this paper, we don't include theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide a clear algorithm in the method section. Besides, we provide implemental details in the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same datasct, provide access to the model. In general. releasing code and data is often
21

---

reproducible, they one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We plan to clean up and release the code during the camera-ready stage.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://ipss.c//pbbiic/ quiddes/CndeSubnisiionpllll
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https) httpubbiicgguidee//oeeeeiiiiiiin
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include all implemented details in the appendix.
Guidelines:
22

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the experimental results, we include the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide compute resources in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://weuiip..ccpuuiicchhhssus
Answer: [Yes]
Justification: We confirm the paper meets the NeurlPS Code of Ethics in every respect.
23

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper has no positive societal impacts or negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no safeguard risk
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
24

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license the existing paper and assets
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode wtwpriihoood.cco/attttttseeeeeee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
25

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjec
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "CwgDfCo1Uk", "reviewer": "Reviewer_VRYA", "summary": "This paper tackles the challenge of editing GNNs. The authors highlight a key issue in GNN editing: the gradient inconsistency between target and training nodes, which can degrade performance when the model is fine-tuned using only the target node’s loss. To address this, they introduce the Gradient Rewiring (GRE) method, which preserves the performance of training nodes by storing an anchor gradient and rewiring the target node’s loss gradient accordingly. The efficacy of GRE is validated through experiments across various GNN architectures and graph datasets, demonstrating its potential to enhance model adaptability without compromising existing accuracies.", "questions": "- In what scenarios would people be willing to accept a substantial overall performance drop in order to modify predictions for individual samples?\n- Are there any methods developed for editing neural networks on non-graph data based on gradient modification? Or are there any works on gradient modification that are relevant to the method in this paper?", "limitations": "As the authors discussed in Appendix B.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper is well-motivated and well-organized, addressing a research question that remains to be further investigated. The provided visualization also aids in understanding the motivations behind the study.\n- The authors offer a detailed derivation process, making the methodology easy to follow.\n- The method was tested on large-scale datasets, demonstrating the model's scalability.", "weaknesses": "- The significance of editing graph neural networks remains unclear. Editing seems to lead to significant performance degradation, especially on large-scale datasets. The authors should discuss why such significant efforts are warranted to modify predictions on individual samples.\n- I recognize that considering graph neural network editing from the perspective of gradient rewriting is novel, but the authors have only discussed model editing in related work. The lack of gradient modification literature makes it hard to position this paper.\n- Some experimental details are missing, such as dataset splitting. These details are crucial for evaluating the proposed method.\n- The time complexity is not provided.\n- Some minor issues (e.g., a superscript linking to nothing in the caption of Figure 1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf. \n\n**[W1: The significance of editing graph neural networks remains unclear]**\n\nAns: Model editing is crucial for ensuring that machine learning models remain reliable and effective post-deployment. This practice addresses high-profile failure cases and misbehaviors that emerge after a model's initial training, often brought to attention through user feedback.\n\nIn general, there are **editing-worthy scenarios, such as high-profile failures and prioritizing critical mistakes**. For example, users may try to prompt large language models (LLMs) to misbehave (e.g., giving criminal advice, system prompt, etc.) to induce \"high-profile failures.\" In the context of self-driving cars, misclassifying a child as a cat poses far greater risks than misclassifying a cat as a dog. Although graph applications typically don't have this kind of direct interaction with users and are not as intuitive, they certainly have high-stake scenarios, such as **patient readmission [1] and flood prediction [2]**, which warrant the study of model editing.\n\nIn short, while model editing has been widely explored in computer vision (CV) and natural language processing (NLP) tasks, it has rarely captured attention in the graph learning community. It is indispensable to investigate graph model editing for high-stake graph applications.\n\n[1] Predicting patient readmission risk from medical text via knowledge graph enhanced multiview graph convolution. SIGIR 2021\n[2] Kazadi, Arnold N., et al. \"Flood prediction with graph neural networks.\" Climate Change AI. Climate Change AI (2022).\n\n**[W2: Please discuss gradient modification literature in related work]**\n\nAns: We have added a discussion on the literature regarding gradient modification in continual learning and meta-learning. Please see more details in Appendix~E.\n\n\n**[W3: More experimental details such as data spliting]**\n\nAns: For all datasets, we first randomly split the data into train, validation, and test sets. Specifically, we ensure that each class has 20 samples in the training set and 30 samples in the validation set. The remaining samples are used for the test set. The target node is randomly selected multiple times from the validation set where the well-trained model makes incorrect predictions.\n\n\n**[W4: The time complexity is not provided.]**\n\nAns: For comparing time complexity and memory consumption, we measured the wall-clock editing time in milliseconds and GPU peak memory in megabytes, reporting average values over 50 independent edits in **Appendix D.5**. Due to the limited rebuttal length, we only show time complexity and memory consumption for GCN.\n\nOur results show that the proposed method is scalable in terms of memory consumption and has manageable editing time overhead. For example, in the GraphSAGE architecture on the Flickr dataset, GRE+ (5) results in only a 13.5% increase in peak memory compared to GD. In terms of wall-clock editing time, the most time-consuming version, GRE+ (5), shows an insignificant overhead, with only a 6.31% increase on the ogbn-products dataset in the GraphSAGE architecture. These observations demonstrate the scalability of GRE+ for large datasets. It is noteworthy that model editing is usually efficient and fast, making slightly slower editing affordable while improving editing effectiveness.\n\n|                |       Editor        | Flickr ET (ms) | Flickr PM (MB) | Reddit ET (ms) | Reddit PM (MB) | ogbn-arxiv ET (ms) | ogbn-arxiv PM (MB) | ogbn-products ET (ms) | ogbn-products PM (MB) |\n|----------------|---------------------|----------------|----------------|----------------|----------------|--------------------|--------------------|-----------------------|-----------------------|\n| **GCN**        | **GD**              | 67.46          | 707.0          | 345.23         | 3244.8         | 94.58              | 786.2              | 2374.15               | 14701.7               |\n|                | **ENN**             | 109.82         | 666.8          | 405.24         | 3244.8         | 242.85             | 786.2              | --                    | OOM                   |\n|                | **GRE**             | 63.93          | 695.8          | 391.54         | 3491.3         | 84.74              | 956.9              | 2400.78               | 17336.6               |\n|                | **GRE+ (2)**        | 100.45         | 696.0          | 457.08         | 3493.2         | 121.11             | 957.8              | 2413.69               | 17338.7               |\n|                | **GRE+ (3)**        | 115.29         | 697.9          | 509.44         | 3493.9         | 131.06             | 957.9              | 2471.23               | 17338.9               |\n|                | **GRE+ (5)**        | 155.05         | 698.6          | 603.85         | 3495.6         | 162.24             | 958.3              | 2591.06               | 17339.2               |\n\n\n\n**[W5: Some minor issues (e.g., a superscript linking to nothing in the caption of Figure 1).]**\n\nAns: We have carefully revised the manuscript. Please check the revised manuscript for the details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1Vu1dk0QX", "reviewer": "Reviewer_LHi3", "summary": "The work introduces a novel method called Gradient Rewiring (GRE) to address the challenge of editable training in graph neural networks. Traditional fine-tuning approaches often struggle with maintaining performance for both target and training nodes. GRE aims to overcome this limitation by rewiring gradients in a way that preserves locality, leading to improved performance for both types of nodes. The method is designed to enhance the training process in graph neural networks by effectively updating node representations while maintaining the network's overall structure.", "questions": "1. It would be beneficial to provide a more detailed explanation of the experimental process and results in Figure 1. In the first row of Figure 1, the images indicate that the gradients for testing and training become very similar, suggesting that the test loss and train loss should decrease simultaneously. However, in the second and third rows, the test loss decreases while the training loss increases. Please further explain this observation.\n\n2. The training + rewriting pipeline is similar to curriculum learning, where the network is trained in two phases: first on simple samples, then on more difficult ones. I recommend the authors conceptually compare their method with curriculum learning. Additionally, I suggest experimenting with batch data rewriting, which might be more practical compared to rewriting individual samples or sample sequences.", "limitations": "The limitations has been discussed in the Appendix.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method of the paper is clear, intuitive, and easy to implement.\n\n2. The writing is clear, making it easy to understand and read.", "weaknesses": "1. Although the experiments were conducted on graph datasets, the proposed method is not specifically designed for graphs. This general approach can be tested on various tasks.\n\n2. There are noticeable formatting errors and typos in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf. \n\n**[W1: This general approach can be tested on various tasks.]**\n\nAns: Thank you for your insightful observation. We agree that while our experiments were conducted on graph datasets, the proposed gradient rewiring method is not inherently specific to graphs, the gradient rewiring method is particularly **suitable in the graph domain due to the small model size**. Specifically, graph models are typically a few layers and thus are smaller in model size compared to models (e.g., Transformers) used in NLP and CV tasks. This results in lower computational and storage costs for gradients, making our strategy particularly suitable for the graph domain. Additionally, it is more challenging to edit nodes in a graph due to the **inherent propagation process within neighborhoods**. Such propagation may lead to significant gradient discrepancies within the graph domain. \n\nIn this work, we primarily focus on **gradient rewiring within the graph domain**, exploring multiple editing manners (i.e., independent editing, sequential editing, and batch editing). **We acknowledge the potential applicability of our gradient rewiring strategy to other domains and leave its exploration for future work in Appendix B**.\n\n**[W2: There are noticeable formatting errors and typos in the paper.]**\n\nAns: We have carefully revised the manuscript. Please check the revised manuscript for the details.\n\n**[Q1: Provide a more detailed explanation of the experimental process and results in Figure 1.]**\n\nAns: Thank you for the insightful comment. Gradient similarity does not necessarily imply consistent test and training loss. Loss consistency primarily depends on the model parameters rather than the first-order gradient. Therefore, there are cumulative effects of gradient inconsistency; significant initial gradient discrepancies can lead to substantial differences in model parameters, resulting in inconsistent training and test loss. We have added a more detailed explanation in Figure 1.\n\n**[Q2a: Conceptually compare their method with curriculum learning]**\n\nAns: Thanks for the insightful comment. We have added the discussion with curriculum learning in **Appendix~F**. Here is the discussion:\n\nCurriculum learning and model editing are two distinct approaches in the field of machine learning. Curriculum learning is an approach where the network is trained in a structured manner, starting with simpler tasks and gradually introducing more complex ones. This method aims to improve the learning process by mimicking how humans learn. Model editing is a fast and efficient approach to patch the well-trained model prediction for several failed test cases. Although both are multi-stage training stages, there are several key differences: \n(1) **Goals**: Curriculum Learning aims to improve the overall learning process by structuring the training data in a way that mimics human learning. In contrast, model editing aims to make targeted adjustments to a pre-trained model to correct undesirable behaviors. (2) **Approach**: curriculum learning mainly focuses on the sequence and complexity of the training data. Model editing typically modifies the model's parameters or architecture to correct undesirable behavior goals.\n(3) **Additional information in the multi-stage process**. Model editing requires failure feedback for well-trained models as the target samples to patch, e.g., test failure cases after production is launched. In other words, such feedback can only be obtained after model pertaining. In curriculum learning, all information is given in multi-stage training. In summary, curriculum learning focuses on structuring the training process to improve overall learning, while model editing focuses on making targeted adjustments to a pre-trained model to correct specific behaviors. Both approaches can be complementary and used together to achieve better model performance.\n\n**[Q2b: More experimental results on batch editing.]**\n\nAns: We have added experimental results on batch editing in **Appendix D.4 (Tables 6 and 7)**. We observe that, compared to independent editing, batch editing is more challenging as all batch samples need to be patched simultaneously. Additionally, GRE and GRE+ both demonstrate significant performance improvements compared to GD and ENN on various model architectures and datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ig8IBIDVkS", "reviewer": "Reviewer_KXu3", "summary": "The authors propose a model editing technique motivated by an observed inconsistency between the gradient of target and training nodes' cross-entropy losses. Their proposed method, Gradient Rewiring for Editable Graph Neural Networks (GRE), stores the anchor of the gradients of the training nodes and uses the anchors during editing to preserve performance. Finally, the authors empirically evaluate their proposed method on a collection of real-world datasets.", "questions": "Please see the weaknesses above.", "limitations": "The authors are discussing limitations in the appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I believe the problem to be an important practical problem and that the paper is well-motivated.\n- The main observation of the paper, i.e., that the gradients of the target and the training nodes are inconsistent, is interesting and novel in the context of Graph Neural Networks.\n- The proposed method seems to perform well consistently. The authors have performed a comprehensive empirical evaluation, and I believe the results are solid.\n- The authors are overcoming the quadratic problem in the number of model parameters by solving a dual problem with significantly fewer variables.", "weaknesses": "- The method requires storing the gradients of the training dataset. This would raise the memory consumption. It would be great if the authors could include a table with the memory requirements for different models. \n- There are a lot of writing/grammar issues. In the following, I will give a (non-exhaustive) list of examples of missing articles and text where the writing could generally be improved. This would generally not be a big deal, but in this case, the writing makes the text very hard to follow and comprehend. I would encourage the authors to do a few revision rounds before updating the manuscript. In my personal experience, free tools such as Grammarly also help with identifying weird formulations or grammar mistakes:\n\t- 16-17: the gradient of *the* loss\n\t- Line 21: \"interpreting the features and topology of graph data\" - *interpreting* is a somewhat unusual way to describe learning representations on graphs; please consider rephrasing. \n\t- Line 42: \"(...) intricate editing for GNNs through the lens of landscape.\" - this phrasing is very confusing; what landscape? I suggest the authors rephrase to something more akin to \"through the lens of the loss landscape of the Kullback-Lieber divergence between the initial node features and the final node embeddings\" since this seems to be the technique that [[1]] is proposing.\n\t- Line 46: \"(...) perspective, and is compatible with existing work\" - please consider rephrasing to something similar to \"(...) perspective, which is compatible with existing work\".\n\t- Line 49: \"(...) can lead to a deterioration in the performance *of* the training nodes\" -> \"(...) can lead to a deterioration in the performance *on* the training nodes\" - the performance is of the model on the training nodes\n\t- Line 55: similar, \"performance *of* the training nodes\" -> \"performance *on* the training nodes\"\n\t- The caption of Figure 1 is somewhat confusing; please consider rephrasing.\n\t- Line 93-94: Somewhat confusing: \"motivation to rewire gradients *of* model editing\" or \"motivation to rewire gradients *for* model editing\"?\n\t- Line 95: \"and advanced version (GRE+)\" -> \"and *an* advanced version (GRE+)\"\n\t- Lines 98-99: \"we pre-train (...) on (...) datasets\" -> \"we pre-train (...) on *the* (...) datasets\"\n\t- Lines 101-102: please consider rephrasing to something more akin to \"we fine-tune the well-trained model using the cross-entropy loss of the target sample via gradient descent\"\n\t- Lines 106-108: \"we investigate performance degradation from model gradient perspective\" -> \"we investigate *the* performance degradation from *a* model gradient perspective\"; \"we further define training loss\" -> \"we further define *the* training loss\"; \"where ... is prediction model ... CE is cross-entropy loss\" -> \"where ... is *a* prediction model ... CE is *the* cross-entropy loss\".\n\t- Footnote 3 is very confusing; please consider rephrasing the entire footnote.\n\t- Lines 131-147 are also somewhat confusing and hard to comprehend due to language. Please consider rephrasing all of the paragraphs.\n\t- Line 148: \"where gradient for model prediction is defines as\" -> \"where *the* gradient for *the* model prediction is *defined* as\". Also, the inline *\\frac* is not aesthetically pleasing, but that is a very minor complaint.\n\t- Line 161: Please consider either removing \"*that*\": \"it is easy to obtain *that* the optimal dual variable v=...\" -> \"it is easy to obtain the optimal dual variable $v*$=...\" or rephrasing to something more akin to \"it is easy to see that the optimal dual variable is $v*$=...\"\n\t- Please also consider rephrasing the next lines, from 162 to 164.\n\t- Missing commas on line 172: \"the training loss for the whole training dataset, after model editing, is on par with (...)\"\n\t- Please consider re-phrasing lines 199-200 to something more akin to \"We randomly select a node from the validation set on which the well-trained model makes a wrong prediction.\"\n- Both Tables 1 and 2 have results with different decimal precision for the standard deviation. For instance, GraphSAGE/GRE/CORA/DD has a single decimal ($3.36 \\pm 0.2$) while GraphSAGE/GRE+/CORA/DD contains two decimals ($0.41\\pm 0.07$). Please consider using the same amount of decimals for all of the results. It makes the table more aesthetically pleasing and easier to read.\n- \"OOM\" is mentioned in the caption of Table 1 as being an out-of-memory error, but there are no OOMs in Table 1. However, Table 2 contains OOMs but the caption does not explain what OOM means.\n- Please consider increasing the font of the text for both Figures 1 and 2.\n- Figure 4, first row, first column: the figure covers half the \"N\" of \"GCN\". The plots are also not vertically aligned on the two columns.\n\nOverall, I believe that the proposed method is solid due to the very good empirical results. However, the writing is lacking to a degree that makes the text very hard to follow and comprehend. I would encourage the authors to revise the text significantly, with multiple rounds of proofreading.\n\nDue to concerns about the clarity and writing, I currently recommend rejection. However, I will revise my score if the authors significantly improve the writing during the rebuttal. I think that the overall work could potentially be impactful and useful from a practitioner's perspective.\n\n\n[1]: https://arxiv.org/pdf/2305.15529", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf. \n\n**[W1: Add comparison in terms of editing time and memory.]**\n\nAns: For comparing time complexity and memory consumption, we measured the wall-clock editing time in milliseconds and GPU peak memory in megabytes, reporting average values over 50 independent edits in **Appendix D.5**. \n\nOur results show that the proposed method is scalable in terms of memory consumption and has manageable editing time overhead. For example, in the GraphSAGE architecture on the Flickr dataset, GRE+ (5) results in only a 13.5% increase in peak memory compared to GD. In terms of wall-clock editing time, the most time-consuming version, GRE+ (5), shows an insignificant overhead, with only a 6.31% increase on the ogbn-products dataset in the GraphSAGE architecture. These observations demonstrate the scalability of GRE+ for large datasets. It is noteworthy that model editing is usually efficient and fast, making slightly slower editing affordable while improving editing effectiveness.\n\n|                |       Editor        | Flickr ET (ms) | Flickr PM (MB) | Reddit ET (ms) | Reddit PM (MB) | ogbn-arxiv ET (ms) | ogbn-arxiv PM (MB) | ogbn-products ET (ms) | ogbn-products PM (MB) |\n|----------------|---------------------|----------------|----------------|----------------|----------------|--------------------|--------------------|-----------------------|-----------------------|\n| **GCN**        | **GD**              | 67.46          | 707.0          | 345.23         | 3244.8         | 94.58              | 786.2              | 2374.15               | 14701.7               |\n|                | **ENN**             | 109.82         | 666.8          | 405.24         | 3244.8         | 242.85             | 786.2              | --                    | OOM                   |\n|                | **GRE**             | 63.93          | 695.8          | 391.54         | 3491.3         | 84.74              | 956.9              | 2400.78               | 17336.6               |\n|                | **GRE+ (2)**        | 100.45         | 696.0          | 457.08         | 3493.2         | 121.11             | 957.8              | 2413.69               | 17338.7               |\n|                | **GRE+ (3)**        | 115.29         | 697.9          | 509.44         | 3493.9         | 131.06             | 957.9              | 2471.23               | 17338.9               |\n|                | **GRE+ (5)**        | 155.05         | 698.6          | 603.85         | 3495.6         | 162.24             | 958.3              | 2591.06               | 17339.2               |\n| **Graph-SAGE** | **GD**              | 117.74         | 843.0          | 1024.12        | 4416.53        | 107.63             | 891.3              | 2125.07               | 13832.2               |\n|                | **ENN**             | 134.50         | 843.0          | 2597.21        | 4416.5         | 277.29             | 891.3              | --                    | OOM                   |\n|                | **GRE**             | 116.03         | 952.4          | 1089.29        | 4955.4         | 100.09             | 1072.5             | 2132.02               | 16254.1               |\n|                | **GRE+ (2)**        | 167.17         | 954.5          | 1267.13        | 4959.0         | 136.28             | 1073.7             | 2135.88               | 16255.9               |\n|                | **GRE+ (3)**        | 176.66         | 955.5          | 1363.53        | 4960.7         | 154.29             | 1074.0             | 2211.63               | 16256.0               |\n|                | **GRE+ (5)**        | 219.81         | 957.5          | 1603.03        | 4964.2         | 180.73             | 1075.5             | 2275.72               | 16256.3               |\n\n**[W2: Tackle lots of writing/grammar issues]**\n\nAns: Thanks for the careful review. We have revised the manuscripts significantly according to your comments. \n\n**[W3: Format issues on Table and Figures]**\n\nAns: We have revised **the caption and uniform decimal precision in Tables 1 and 2, increased the font of the text for both Figures 1 and 2, and revised Figure 4**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYhLKNu8qq", "reviewer": "Reviewer_X56x", "summary": "Model editing involved fine-tuning a pre-trained model specifically on training examples where it cannot predict the correct output in order to correct these errors. While model training has been explored for CV and NLP models, GNNs pose a unique challenge due to the unordered data type and node-level classification problem. This paper attempts to rectify this error by proposing a Gradient Rewiring+ (GRE+) method for Editable graph neural network training while preserving the original train/test set performance. GRE+ is evaluated on several small and large classification datasets.", "questions": "- Fig 3 where are results on ENN?", "limitations": "A limitations section is provided in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method aims to rectify errors that still persist after training, without compromising the original performance. To quantify this, the authors report three metrics: The accuracy after model editing has been performed, the DropDown metric measuring how this impacts the original performance and the success rate of the method at actually rectifying the error. The three of these provide a clear, multi-faceted view of the effect of the proposed method against the baselines. \n- In most situations, the proposed GRE+ proves to be the best method.\n- The introduction and motivation of the paper is generally well-written and easy to follow.", "weaknesses": "- Biggest difference is the presentation and decision to treat GRE and GRE+ as different methods to compare to baseline approaches. This is because the experimental results in Tables 1 and 2 show that GRE on its own is not really noteworthy as it fails to consistently rank as the 2nd best behind GRE+ and is outperformed by GD and ENN on numerous occassions. \n- This hampers the writing in the method which focuses on what GRE does first, before expanding to talk about GRE+ with more specifics. In contrast to this writing decision, it would be better if GRE and GRE+ were not treated separately, GRE meant GRE+ and instead of having 2 rows for your method in Tabs 1/2, you just had GRE(+), then later did an ablation study (where one option is the GRE config as mentioned in the submitted paper) to show the efficacy/necessity of the whole method.\n- Line 243 This is the first time the word \"Transformers\" is mentioned in the paper and its jarringly brought up out of the blue. The introduction broadly mentioned that model editing exists for CNNs and NLP tasks but not specifically the Transformer architecture. This prose should be deeply revised before the paper is publication ready.\n- There is no analysis/study of the cost/complexity of the model editing methods which is a major weakness. Looking at Fig 3 while it is true that GRE+ improves little/none over GRE, in Tab 1/2 the story is much different. There does not seem to be a situation where GRE is better so I must question why it is presented as a distinct approach itself compared to GRE+. Is it because it is computationally less expensive?\n- Some statements are not properly substantiated with citations:\n\t- L105 \"It is well-known that model editing incurs training performance degredation\"\n\t- L167 \"Since shallow GNNs model performs well in practice\" seems like a handwave. Also, spelling/grammar check.\n- Other nitpicks:\n\t- Tab 1 caption mentions \"OOM\" for some methods but that is only used in entries for Tab - Fig 2 should have larger fonts it is very hard to read.\n\t- L242, should read \", respectively.\" when enumerating like that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf. \n\n**[W1: Presentation on GRE and GRE+]**\n\nAns: hank you for the constructive comment. We respectfully disagree with the assertion that GRE isn't consistently ranked as the second-best method behind GRE+. We believe that treating GRE and GRE+ as different methods in our writing is reasonable.\n\nGRE achieves second-best performance in **both sequential editing and batch editing**, while demonstrating competitive performance in independent editing. In the independent editing task, all methods can achieve a high success rate since it is relatively simple to correct one target sample. However, GRE still secures second-best performance in several instances, such as on the ogbn-arxiv and ogbn-products datasets in GCN. More importantly, GRE excels in more complex editing scenarios, such as sequential editing and batch editing. For sequential editing, GRE achieves second-best performance across various datasets, as shown in **Figures 3 and 6**. Similarly, for batch editing, GRE also ranks second-best, as evidenced by the results in **Tables 6 and 7**.\n\nIn summary, GRE shows significant improvement over baselines in sequential and batch editing. From a technical perspective, GRE provides a closed-form solution for gradient rewiring, while GRE+ requires a numerical QP solver. Therefore, we treat GRE and GRE+ separately due to the significant performance improvements offered by GRE and its distinct closed-form solution.\n\n\n\n**[W2: Model editing exists for CNNs and NLP tasks but not specifically the Transformer architecture]**\n\nAns: To avoid confusion, we have revised the statement of Observation 1 in Section 4.2. Specifically, we removed the part about transformers and generally mentioned that all editors can effectively rectify model predictions for independent editing in the graph domain.\n\n**[W3: No analysis/study of the cost/complexity of the model editing methods]**\n\nAns: For comparing time complexity and memory consumption, we measured the wall-clock editing time in milliseconds and GPU peak memory in megabytes, reporting average values over 50 independent edits in **Appendix D.5**. Due to the limited rebuttal length, we only show time complexity and memory consumption for GCN.\n \nOur results show that the proposed method is scalable in terms of memory consumption and has manageable editing time overhead. For example, in the GraphSAGE architecture on the Flickr dataset, GRE+ (5) results in only a 13.5% increase in peak memory compared to GD. In terms of wall-clock editing time, the most time-consuming version, GRE+ (5), shows an insignificant overhead, with only a 6.31% increase on the ogbn-products dataset in the GraphSAGE architecture. These observations demonstrate the scalability of GRE+ for large datasets. It is noteworthy that model editing is usually efficient and fast, making slightly slower editing affordable while improving editing effectiveness.\n\n|                |       Editor        | Flickr ET (ms) | Flickr PM (MB) | Reddit ET (ms) | Reddit PM (MB) | ogbn-arxiv ET (ms) | ogbn-arxiv PM (MB) | ogbn-products ET (ms) | ogbn-products PM (MB) |\n|----------------|---------------------|----------------|----------------|----------------|----------------|--------------------|--------------------|-----------------------|-----------------------|\n| **GCN**        | **GD**              | 67.46          | 707.0          | 345.23         | 3244.8         | 94.58              | 786.2              | 2374.15               | 14701.7               |\n|                | **ENN**             | 109.82         | 666.8          | 405.24         | 3244.8         | 242.85             | 786.2              | --                    | OOM                   |\n|                | **GRE**             | 63.93          | 695.8          | 391.54         | 3491.3         | 84.74              | 956.9              | 2400.78               | 17336.6               |\n|                | **GRE+ (2)**        | 100.45         | 696.0          | 457.08         | 3493.2         | 121.11             | 957.8              | 2413.69               | 17338.7               |\n|                | **GRE+ (3)**        | 115.29         | 697.9          | 509.44         | 3493.9         | 131.06             | 957.9              | 2471.23               | 17338.9               |\n|                | **GRE+ (5)**        | 155.05         | 698.6          | 603.85         | 3495.6         | 162.24             | 958.3              | 2591.06               | 17339.2               |\n\n\n**[W4: Add citations and other nitpicks.]**\n\nAns: We have added citations for several statements and revised the format and writing issues accordingly.\n\n**[Q1: Fig 3 where are results on ENN?]**\n\nAns: Thank you for your constructive comment. Model editing consists of two stages: the pre-training stage, where a well-trained model is obtained, and the editing stage, where undesirable behavior is corrected. The GD, GRE, and GRE+ methods perform editing during the editing stage using the **same pre-trained model**. For these methods, a higher test accuracy drawdown implies a lower test accuracy after editing. In contrast, the key goal of the ENN baseline is to improve the model during the pre-training stage. The objective at this stage is to obtain a model that can be easily edited using gradient descent while maintaining good task performance. Since **the test accuracy of the pre-edited models varies**, it is infeasible to compare these methods in terms of test drawdown. To fairly compare all methods, we have **added a comparison of the test accuracy performance after editing for sequential editing in Appendix D.6**. It is observed that the accuracy of GRE+ is higher than all baselines.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# A summary of our rebuttal.
We thank all reviewers for your valuable time and comments. We are glad that many reviewers found the following:
* **Our paper is well-motivated and well-organized**
     * R `VRYA`: This paper is well-motivated and well-organized, addressing a research question that remains to be further investigated.
     * R `KXu3`: I believe the problem to be an important practical problem and that the paper is well-motivated. The main observation of the paper, i.e., that the gradients of the target and the training nodes are inconsistent, is interesting and novel in the context of Graph Neural Networks.
     * R `X56x`: The introduction and motivation of the paper are generally well-written and easy to follow.
- **Our method is clear, intuitive, and easy to follow.**
     * R `VRYA`: The authors offer a detailed derivation process, making the methodology easy to follow.
     * R `LHi3`: The method of the paper is clear, intuitive, and easy to implement.
- **The experimental results are solid and strong.**
     * R `VRYA`: The method was tested on large-scale datasets, demonstrating the model's scalability.
     * R`KXu3`: The proposed method seems to perform well consistently...., and I believe the results are solid.
     * R`X56x`: In most situations, the proposed GRE+ proves to be the best method.

On the other hand, aside from some cosmetic suggestions, the reviewers have brought up the following suggestions:
- **Editing time and peak memory comparison** (R `VRYA`, `KXu3`, `X56x`): We conduct editing time and peak memory comparison in ** Appendix D.5**.
- The **significance of editing graph neural networks** remains unclear (R `VRYA`): there are multiple editing-worthy scenarios, such as high-profile failures and prioritizing critical mistakes. Two specific applications such as patient readmission and flood prediction.
- **Batch editing experimental results** (R `X56x`): The batch editing results are shown in **Appendix D.4** (Tables 6 and 7).
- ENN results in sequential editing (R `X56x`): The test accuracy comparison results are shown in **Appendix D.6** (Figure 6).
- This general approach can be tested on various tasks (R `LHi3`):  (1) the gradient rewiring method is particularly suitable in the graph domain due to the small model size. (2) it is more challenging to edit nodes in a graph due to the inherent propagation process within neighborhoods. (3) acknowledge the potential applicability of our gradient rewiring strategy to other domains in **Appendix B**.
- Related work on gradient modification (R `VRYA`): We add Related work on gradient modification in **Appendix E**.
- Detailed explanation for Figure 1 (R `LHi3`): We add a more detailed discussion in Section 3.1.
- Discussion on the differences with curriculum learning (R `LHi3`): We have added the discussion with curriculum learning in **Appendix~F**.

Based on all reviewers' comments, we carefully revised the manuscript. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf.

Meta-review: This paper proposes a method to simply rewire (modify) the gradient for the target loss to maintain the overall performance during the model editing of GNNs. In specific, it proposes a novel constrained optimization problem to regulate the change of the training loss after the model editing, and solves it efficiently by the gradient rewiring using the anchor gradient of the overall training loss. Experimental results on various graph datasets show that the proposed method, called GRE(+), improves performances over baselines in reducing the overall accuracy dropdown(drawdown?) while maintaining the high target success rate.

Overall, the motivation is clear, and the proposed method seems to be technically sound, supported by extensive empirical validation. The authors also thoroughly address many concerns and issues raised by the reviewers, and the reviewers’ scores are one weak accept and three borderline accepts.

However, the description of the proposed method and its writing still need to be improved. Especially, the derivation of the proposed constrained optimization is somewhat unclear. Why do we have such two constraints as Eq. (3),(4)? What if we just take one of them? What if we take an average of prediction differences in Eq. (4)? Even, there is no ablation study on this. In addition, what is \theta in Eq. (2) different from \theta’?, and how can we derive the approximation in Eq. (6)? Also, figures and tables should be placed close to their descriptions. Especially, Table 1 and 2 in the middle of Method Section disrupt clearly understanding the explanation of the method.

In conclusion, this paper is a borderline paper and has been improved significantly through the rebuttal. Based on the consensus among the reviewers, I would recommend the paper to be accepted. However, the authors should revise the description of the proposed method more clearly, and more explanation and ablation studies regarding the proposed optimization problem should be conducted.
Decision: Accept (poster)