If You Want to Be Robust, Be Wary of Initialization
Authors: Sofiane ENNADIR
Johannes F. Lutzeyer
Michalis Vazirgiannis
El houcine Bergou
Venue: NeurIPS 2024 poster (2024)
Keywords: Adversarial Robustness
Graph Neural Networks
Primary area: graph_neural_networks
TL;DR: We theoretically study the direct relationship between initial weights, number of training epochs and the model’s adversarial vulnerability, offering new insights into adversarial robustness beyond conventional defense mechanisms.

Abstract: Graph Neural Networks (GNNs) have demonstrated remarkable performance across a spectrum of graph-related tasks, however concerns persist regarding their vulnerability to adversarial perturbations. While prevailing defense strategies focus primarily on pre-processing techniques and adaptive message-passing schemes, this study delves into an under-explored dimension: the impact of weight initialization and associated hyper-parameters, such as training epochs, on a model’s robustness.
We introduce a theoretical framework bridging the connection between initialization strategies and a network's resilience to adversarial perturbations. Our analysis reveals a direct relationship between initial weights, number of training epochs and the model’s vulnerability, offering new insights into adversarial robustness beyond conventional defense mechanisms. While our primary focus is on GNNs, we extend our theoretical framework, providing a general upper-bound applicable to Deep Neural Networks.
Extensive experiments, spanning diverse models and real-world datasets subjected to various adversarial attacks, validate our findings. We illustrate that selecting appropriate initialization not only ensures performance on clean datasets but also enhances model robustness against adversarial perturbations, with observed gaps of up to 50\% compared to alternative initialization approaches.

If You Want to Be Robust, Be Wary of Initialization
Sofiane Ennadir' KTH Stockholm, Sweden
Johannes F. Lutzeyer LIX, Bcole Polytechnique IP Paris, France
Michalis Vazirgiannis KTH & Ecole Polytechnique Stockholm, Sweden
El Houcine Bergou UM6P Benguerir, Morocco
Abstract
Graph Neural Networks (GNNs) have demonstrated remarkable performance across spectrum of graph-related tasks, however concerns persist regarding their vul- nerability to adversarial perturbations. While prevailing defense strategies focus primarily on pre-processing techniques and adaptive pessaag-passing schemes, this study delves into an under-explored dimension: the impact of weight initialization such as training epochs, on a model's robustness. We introduce a theoretical framework bridging the connection between initializa- tion strategies and network's 's resilience adversarial perturbations. Our analysis reveals direct relationship between initial weights, number of training epochs and the model's vulnerability, offering new insights into adversarial robustness beyond conventional defense mechanisms. While our primary focus is on GNNs, we extend our theoretical framework, providing general upper-bound applicable Deep Neural Networks. Extensive experiments, spanning diverse models and real-world datasets subjected to various adversarial attacks, validate our findings. We illustrate that selecting appropriate initialization not only ensures performance on clean datasets but also enhances model robustness against adversarial perturbations, with observed gaps of up to 50% compared to alternative initialization approaches.
and associated hyper-parameee
1 Introduction
Neural networks have demonstrated remarkable prowess across various domains, ranging from computer vision [8] to natural language processing (29), proving their ability to model and extract complex insights from real-world datasets. Recently, Graph Neural Networks (GNNs) [3]]]]]] [3] have emerged powerful extension of neural networks specifically tailored to tackle graph- structured data. These models have led to rapid progress in solving tasks such as node and graph classification where their application have spanned from drug design
(20), protein resistance analy- sis (24), session-basee recommendations [33 to tabular data 22. Concurrently with their success, deep learning architectures have been shown to be unstable when subject to adversarial perturbations 15]. resulting in unreliable predictions, consequently questioning these models' applicability in critical domains. While most adversarial robustness studies focus the domain of computer vision, recent work [16] studying the robustness of GNNs has emerged. Given their rich nature, graphs allow different attack schemes, where the attacker can either choose to edit the graph structure (by adding/deleting edges) or edit the node/edge features. In parallel, recent studies have been devoted studying approaches defend against these attacks and enhance GNN robustness, such input pre- meeprocesssigg techniques (32. low-rank approximation [1], edge-pruning [38 or adapting the
message- -psssing schemes (I].
*Corresponding Author: ennadir@kch.se
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The majority of available defense studies focus on understanding the inner dynamics of GNNs to pinpoint and mitigate adversarial vulnerabilities. While analyzing the pessagg-passinggg mechanism and implementing input peepproesssin techniques remains viable direction, comprehensive under- standing necessitates exploration beyond traditional avenues. In this sense, investigating factors such as weight initialization strategies and the impact of other hyperparameters, notably those associated with optimization mechanisms, can offer new insights and perspectives on achieving GNN global robustness. Hyperparameter choices and tuning play critical role in striking a balance between learning the underlying signals in the data and preventing overfitting to ensure the model's gener- alization. Hence, existing studies initialization mainly evolve around understanding its effect on the model's s convergence, stability and performance [34 23]. In contrast, our work primarily focuses on examining the effect of initialization on model's 's underlying adversarial robustness, representing to the best of our knowledge the first exploration of its kind. Our main objective is to provide theoretical understanding of the link between weight initialization and other dynamics such as the number of training steps and the resulting model's robustness. With this perspective in mind, start by formalizing robustness in the context of GNNs when subjected to structural and node feature-based adversarial attacks. Subsequently, we derive an upper bound that connects the model's robustness to the weight initialization strategies. Specifically, we illustrate that this bound depends on the initial weight norms and the number of training epochs. Finally, we validate our theoretical findings by demonstrating the effects of employing various initialization strategies on the model's 's robustness using benchmark adversarial attacks on real-world datasets. Note that while our analysis primarily focuses on the widely used Graph Convolutional Networks (GCNs) [21] and Graph Isomorphism Networks (GINs) [6], we highlight the versatility of our approach by providing a general upper bound applicable to any Deep Neural Networks in Section 5 This underlines the potential for extending our analysis to a wide range of architectures, showcasing its broad applicability in understanding and enhancing adversarial robustness in neural networks. We summarize our contributions as follows:
We provide a theoretical analysis that links weight initialization strategies with adversarial robustness in GNNs. We specifically derive an upper bound connecting a model's robust- ness to weight initialization and the number of training epochs, demonstrating that the initialization strategy can significantly influence the network's adversarial robustness.
We validate our theoretical findings by conducting extensive experiments across various models using different benchmark adversarial attacks on real-world datasets. These exper- iments demonstrate that certain weight initialization strategies can enhance the model's defense against adversarial attacks, without degrading its performance on clean datasets.
While our primary focus is on GNNs, we extend our analysis to Deep Neural Networks, illustrating the broader applicability of our theoretical analysis and its corresponding insights.
2
Related Work
Graph Adversarial Attacks. Multiple studies focus on designing adversarial attacks capable of fooling graph-based classifier [6
35 10 The majority of these methods 4 37 approach the
adversarial aim as an optimization problem and employ different methods to solve it such as meta- learning [41. Furthermore, Nettack 40) constrained the problem by preserving degree distribution and imposing constraints on feature co-occurrence to generate unnoticeable perturbations. Finally, reinforcement learning was proposed recently as a means to generate graph adversarial attacks 7
Graph Adversarial Defenses. Recent efforts have emerged to defend against the aforementioned adversarial attacks. In particular, methods such as low-rank matrix approximation coupled with graph anomaly detection (22) have been used. For example, GNN-Jaccard (32) proposed to pre-process the graph's adjacency matrix detect potential manipulation of edges. Other methods such as edge pruning [38] and transfer learning (28) have been leveraged to limit the effect of poisoning scheme, such as employing orthogonal
attacks. Additionally, adaptations of the message-passine weights [1] or introducing noise during training [9], have been shown to perform well in terms of defense. Furthermore, there a growing interest in exploring robustness certificates 42 as a means of ensuring model robustness. For instance, [5) used randomized smoothing to provide a highly scalable agnnsiig certificate for graphs. Additionally, other robustness certificates for GCN-based graph classification under topologiaal perturbations have been proposed [19].

---

Weight Initialization. The impact of weight initialization has been extensively studied both theoreti- cally and empirically where the main line of study consists of understanding the interplay between initialization techniques and the implicit regularization they induce, thereby elucidating their influence on a model's generalization capabilities [34 23 For instance, has been showcased that sampling initial weights from the orthogonal group can speed up convergence [18]. Similarly, alternative initialization approaches such as the Glorot Initialization [13] and Kaiming Initialization [17] have
been proposed in efforts to improve the model's performance.
Our work stands apart from existing research on adversarial robustness as it represents, to the best of our knowledge, the first attempt to theoretically investigate the impact of initialization on model's robustness. Moreover, our approach diverges fundamentally from existing literature on weight initialization as our focus lies in theoretically understanding the effect of initialization on model's robustness rather than its implications for generalization or convergence.
3 Graph Adversarial Robustness
In this section, we start by introducing the notation and some fundamental concepts related to GNNs. We afterwards establish the problem setup together with the set of considered assumptions. We finally lay out a GNN's robustness formalization on which we will build our theoretical analysis.
3.1 Preliminaries
Let G - =VV,) be a graph where (V) = ni) is its set of vertices and its set of edges. We denote A E A {0, 1}"X" its adjacency matrix. The graph nodes are annotated with feature vectors X X C Rnd (the i-th row of X corresponds to the feature of node i). We denote by (i) the neighbors of node i E V and ||2 the Euclidean (resp., spectral) norm for vectors (resp., matrices).
In this work, we consider the task of node classification. In this task, every node assigned exactly one class from C = f1. C} C y and we consider dy as a distance within the output space y. The learning objective is to find function fw. parameterized by W. that assigns each node i V class c E C while minimizing some classification loss (c. cross- cross-entropy loss), denoted as L.
GNNs. A GNN model consists of series of neighborhood aggregation layers that use the graph structure and the node features from the previous layers to generate new node representations. Specifically, GNNs update node feature vectors by aggregating local neighborhood information. In the particular case of GCNs, this process is described by the following iterative propagation:
h0 = 0010 (An(C-WW())
with (t)) 12 being the weight matrix in the é-th layer, p and q embedding dimensions and (i(() is non-linear activation function. Moreover, Ā E R= denotes the normalized adjacency matrix Ā D-1/2 AD-1/2 where D diag((W(1)),, (N2))...,, is the degree matrix.
Problem Setup. For our theoretical analysis, we assume that the model is based on 1-Lipschitz activation functions (which is characteristic of commonly used activation functions such tanh). Additionally, we consider the training loss function L to be L-smooth and that it is minimized using gradient descent. We denote by W. the local optimum towards which gradient descent iteratively converges. Specifically, for a learning rate S E. the update at time step t for layer i is:
wht W VL (w(())
It is worth emphasizing that although we focus on the node classification task, which is prevalent and well-studied in the literature of adversarial robustness, our analysis is equally applicable to other tasks such as graph classification. Moreover, while our theoretical analysis predominantly centers around using gradient descent as the optimizer, this choice does not limit the generality of our findings. One can employ a different optimizer and still yield the same insights and results by following a similar approach as the one outlined in this paper. Consequently, this specific setup should not be perceived as limitation but rather as analytical choice.
3

---

3.2 Adversarial Robustness for Graph Neural Networks
Let f (A, + y be a GNN-classifier following the framework outlined in Section 3.1 An adversarial attacks consists of generating an alternative graph (A, X) that perturbs the original prediction fAA, X) while not being far (semantically) from the original graph. Typically, this generated graph must adhere to a number constraints related to its similarity to the original graph, defined by perturbation budget controlling the number of edited edges or features. The set of these
graphs is written as B((A,x];e) = (A, min (IIA- - PAPT ll2 + ||X - PX se).
where II represents the set of permutations of the adjacency matrix. While the previous formulation relies on the f2 norm, other norms may be used depending on the domain of application and the specific use case. Building on previous work [9]. the adversarial risk of a GNN can be defined as the expected error of adjacent graphs within the considered graph's 's neighborhood defined by written as:
dy (f (A.X)) :(Ax))
Reffi = E (A,X)~D [A..)EB((AXX)))
sup
(2)
In the current analysis, we focus on the l2 norm as our output distance dy (which can be substituted by any norm - given the equivalence of norms). We theoretically approach the introduced adversarial risk by deriving an upper-bound, which reflects the model's expected error under input perturbation. Intuitively, a smaller upper bound reflects a smaller adversarial risk which in turn suggests a robust behavior locally. In this perspective, Definition | draws the link between the considered risk quantity and model's robustness.
Definition 1. (Adversarial Robustness). The graph-based function : (A, - y is said to be (c,7) - robust if its adversarial risk is upper-bounded by y, i.e., R.[f] s
The current definition addresses adversarial risk from worst-case scenario perspective, which is the most prevalent approach in the literature. This means we aim to identify the neighbor graph that maximizes the harm e.. causes the greatest deviation from the original prediction). By upper- bounding the risk associated with this "worst-case" graph, we inherently account for all other potential adversaries within the same neighborhood, as their risk will be less than or equal to that of the worst- case scenario. We note that the nuances between the "average" and "worst-case" approaches have been thoroughly examined and justified previous research 25
On the Effect of Initialization
We start by considering the Graph Convolutional Networks (GCNs) within the broader context of Message Passing Neural Networks for node classification. This study investigates how initialization and other hyperparameters impact the final model's robustness. In this context, aim to establish connection between the introduced adversarial risk (Equation (2) and the initial weight distribution and its evolution during training. Specifically, we seek to demonstrate that different choices in the initialization distribution and other relevant parameters lead to varying levels of model robustness, offering new insights into the potential trade-offs between initialization strategies and robustness. In this sense, we derive an soper-bound (denoted as y in Definition I on the stability of a GCN-based classifier when the input graph's node features are subject to ad adversarial attacks.
Theorem 2. Let J (A, + y denote graph-based function composed of GCN layers, where the initial weight matrix of the i-th layer is denoted by ((()) For adversarial attacks only targeting node features of the inpul graph, with budget e, we have (in respect Desinition 7
7= II 2 [:
2t+1
w ())
Σ
NEV
with being the number of training epochs and u. denoting the sum of normalized walks of length (T - 1) starting from node u.
The proof of Theorem 2 is provided in Appendix Al Theorem 2 provides a formal connection between the robustness of a GCN-based classifier and its initial weights, offering valuable insights into their

---

effects. From first perspective, the derived upper-bound depends on the initial weight's norm. Specifically, a lower norm corresponds a smaller upper-bound, indicative of more robust model. However, while setting all initial weights to zero theoretically yields the smallest upper-bound and consequently the optimum robustness, this direction can detrimentally affect the model's 8 performance on the learning task. Empirical evidence suggests that initializing weights to zero (or a constant) often leads to poor learning outcomes, as it constrains weight behavior during propagation, limiting subsequent paok-propagation operations and resulting in convergence to unsatisfactory local minima (e. g., see Page 301 in (4). From a second perspective, it appears that higher number of training epochs leads to the looseness of the upper-bounde resulting in increased adversarial vulnerability. This latter observation provides proof and highlights the existence of the usually discussed trade- off between clean and attacked accuracy. Achieving a balance between increasing the number of epochs to achieve satisfactory clean accuracy and limiting them to attain a robust model is hence essential. While theoretically challenging identify this equilibrium point, our experimental results demonstrate its existence. We note that the dependence of y on & can be sharpened by having (1 + nL) instead of 2 With small 77 (which is usually the case in practice), (1 + T, ~ + In resulting in bound which depends linearly on f. The same remark applies to the remaining bounds derived in the paper. These insights, in the case of node-feature-bait based adversarial attacks, also extend to structural perturbations where Theorem 3 provides the exact bound for this case.
Theorem 3. Let f: (A, + Y denote a graph-based fanction composed of GCN layers, where the initial weight matrix of the i-th layer is denoted by 5 Let f be the number of used training epochs. When f is subject to structural attacks, with a budget e, we have (in respect to Definition 7):
T
T
2 = =
II 2
[(())
+ +2+1 1) |XX||
[++TII
2 [/
+2+1 | 1))
The computed upper-bound suggests that the effect of initialization is greater in the case of structural perturbations. This emphasis is resulting from the distinct dynamics within the message passing mechanism, where the influence of the adjacency matrix and node features varies during each propagation step. Precisely, for structural perturbations, the effect of the attack is considered at each propagation step through the perturbed adjacency matrix (in the aggregation step). Moreover, the impact is also amplified by the affected residual layers from previous iterations, resulting in a more significant attack result. This is different in the case of node-feature based adversarial attacks, since the node features are only directly taken into account in the first propagation. Overall, the robustness enhancements can be derived from choosing the right intiall weight's 's distribution and the right number of training epochs. We illustrate this specific point by analyzing the effeet of the initial distributions choices on the model's ss robustness. Specifically, we consider the case of the Gaussian distribution, where Lemma studies how the parameters of this distribution namely, the mean and variance exert an influence on the expected (in respect to the initial distribution) value of the adversarial risk.
main takeaway of the provided analysis in Theorems 2 and 3 that "approximatell-
Lemma 4. Let I XA. - y denote a graph-based based function composed T GCN layers for which the initial weight are drawn from the Gaussian distribution N(, E). When subject to node features based adversarial attacks, we have the following:
Ewo-N (R
S
II 2 22
+ 21+1
Σ wu VEEV
The proof of Lemma 4 is provided in Appendix C Given that a tighter upper bound inherently results in a higher level of robustness, the results derived in Lemma 4 illlstratee the clear effect of initialization in the case of the Gaussian distribution. The derived bound shows that increasing the distribution parameters, both the mean and variance values, leads to a decrease in the victim model's underlying robustness. While one might intuitively aim to set these parameters as low possible to achieve optimal robustness, doing so could potentially compromise the model's performance on clean datasets. Therefore, as previously mentioned, striking the right balance between clean accuracy and adversarial robustness is crucial.
Extending the Results to the GIN. The same previously applied analysis for the GCN-based models can be extended to take into account GIN-based classifiers. We consider the same set of assumptions and the same problem setup considered during the previously studied GCN case. We additionally
5

---

assume that the imput node feature space to be bounded, i. |X|| B. We note that this boundedness a realistic assumption and that the value B can be easily computed for any real-workh dataset.
Theorem 5. Let (A, + Y denote graph-based function composed of GIN layers, where the initial weight matrix of the i-th layer is denoted by ( For adversarial attacks only targeting node features of the input graph, with budget we have:
T
y= II 22
[/
+ 2
|w=""l)
BT max deg(u) (u) + WEV
with being the number of training epochs and deg(u) is the degree of node
The proof of Theorem 3 is provided in Appendix D Theorem establishes an upper bound on the robustness of a GIN-based classifier against adversarial attacks targeting node features. We observe analogous insights, to the ones derived for a GCN-based classifier, regarding the influence of the initialization distribution and number of training epoch on the model's underlying robustness.
5 Generalization to Other Models
While our primary research focus lies within the domain of graph representation learning, a sub-field of the broader landscape of Deep Learning models, the fundamental principles of our theoretical analysis are applicable across various model architectures. Notably, and to our knowledge, the absence of comparable study in current adversarial literature motivates our endeavor to bridge this gap. In this section, we aim to fill this gap by presenting a comprehensive anal framework that provides the connection between weight initialization and the robustness of neural networks.
Let Te denote an input vector where no is the input dimension. Let W R=i-1-9 be the weight matrix and by E R" the bias of the 1hh layer with ny being its dimensionality. We focus on the general family of neural networks for which the computation during layer using an activation function ( can be written as
h h = 200) (W(0)(-1) + b(9)
We consider the same of assumptions (stated in Section 3.1 as the one from previous section. We consider the &g norm as our input and output distances within the metric space 1n and we consider an input attack budget c. The introduced adversarial risk in Equation 2 can be easily extended and tailored to the family of considered neural networks discussed in this section. Further clarification on this extension is provided in the Appendix (Section G. From this standpoint, by adapting the Definition] D analogous effects of the weight initialization, provided in Theorem O can be observed.
Theorem 6. Let f X C R + y Rot be a T-layers neural network with ( denoting the initial weight matrix of the i-th layer. When subject adversarial attacks, f is (E, robust with:
II 2t
w/(0
+ 2 |w!
7
The proof of Theorem 6 can be found in Section E of the Appendix. Similar to previous findings, the upper bound relies key elements of the initialization process, specifically the initial weight norm and the number of training epochs. These results validate and extend the established link between initialization and model's robustness in neural networks, highlighting the importance of selecting appropriate parameters. From the derived upper bound, which is also applicable to GCN and GIN cases, we observe that the number of training epochs exerts an effect on the bound. Specifically, while increasing the number of epochs can improve the model's performance on a clean dataset, simultaneously leads deterioration in the model's adversarial robustness. Ideally, adversarial defense strategies aim to avoid this trade-off between clean and attacked accuracy, striving for robust models that do not compromise the initial performance. In this context, considering the strong- convexity of the loss function L, in addition to the previously made assumptions, we observe that the effect of the number of training epochs becomes less pronounced. Lemma specifically provides the computed bound under these assumptions.

---

Lemma 7. Let & + be a T-layers neural network trained with a p-strongly convex and L-smooth loss function. Let w denote the initial weight matrix of the i-th layer. When subject to adversarial attacks, with a budget e, we have that f is (),7)) - robust with:
== (L- m/L)* [/00 + | i=1
The proof of Lemma provided in Section[Fjof the Appendix. Since I L, increasing the number of training epochs results in the diminishing influence of the initialization weights. In this scenario, the bound depends solely on the final weights, phenomenon previously explored in works such as Parseval networks [6] for neural networks and GCORN I for GNNs. This observation highlights the necessity of convexity in the loss function when training neural network, as it plays a crucial role enhancing the model's s robustness, beyond the traditional considerations classical training optimization perspectives.
6 Experimental Results
This section aims to empirically validate our theoretical findings using real-world benchmark datasets. We start by laying out our experimental setting. then we study the impact of various initialization strategies on GCN's robustness. Next, we analyze the influence of training epochs on adversarial robustness. Finally, we extend our experimentation to considered family DNNs in Section 5
6.1 Experimental Setting
Consistent with our theoretical analysis, this section focuses the node classification task. We leverage the citation networks Cora and CiteSeer [27), with additional results on other datasets provided in the Appendix G To mitigate the impact of randomness during training, each experiment was repeated 10 times, using the train/validation/tesl splits provided with the datasets. A 2-layers GCN classifier with identical hyperparameters and activation functions was employed across all the experiments. The models were trained using the cross-entropy loss function, and consistent values for the number of epochs and learning rate maintained across all analysis. Further implementation details can be found in Appendix H The necessary code to reproduce all our experiments is available on github https://githubb.com wwwp///ennnn//iiiaaazattiioooooi
Experimental Setup.
Adversarial Attacks. We consider two main gradient-bace structural adversarial attacks: (i) 'Mettack' (with the 'Meta-Self" training strategy) [41] that formulates the problem bi-level problem solved using meta-gradients (ii) and the Proximal Gradient Descent (PGD) [35] which consists of iteratively adding small crafted perturbations using the gradient of the classifier's 's loss. We additionally provide results for the 'Dice' attack [4] in Appendix G For our experiments, we considered perturbation rates ranging from 10% (i. e., E)1 to 40% (i.e., 0.4(El).
Evaluation Metrics. We report the experimental findings in terms the 'Attacked Accuracy', which is the model's test accuracy when subject to the attacks. Additionally, given that initialization have an impact on the model's generalization and performance, solely reporting the attacked accuracy fails in some specific cases to provide comprehensive perspective. Thus, adopt for some experiments the "Success Rate" metric, also commonly employed in adversarial literature, which encompasses the number of successfully attacked nodes while taking into account the model's initial clean accuracy.
6.2 Effect Of Training Epochs
The theoretical analysis presented in Section 4 established a connection between the number of training epochs and the model's resulting robustness. The derived bound suggests that increasing the number of epochs results in the model becoming more vulnerable to adversarial attacks. The objective of this experimental section is to empirically validate this assertion using real-world datasets. To this end, at each training epoch, we assess the model's performance on the test set, considering both its clean accuracy and its accuracy under adversarial attacks.
Figure T illluttrates the results of this analysis. The initial two subplots (a.b) display the findings on the Cora dataset, while the subsequent (c.d) subplots present results from the CiteSeer dataset. For
7

---

085
677
404
DAn
04
20.0
30.3
26.0
20.0
380
320
100
Expoare
Figure 1: Effect of training epochs on the model's 's robustness on Cora (a,b) and CiteSeer (c,d).
Ppd- Douavier Intlsliatinn Dore
-p. Ceussirr Intastation Dianet
Molach
Metlisk Ceurwenn Intutaalee.. Cleaer
20
Figure 2: Effect of the variance parameter on the model's robustness in the case of Gaussian Initialization on PGD [on Cora (a) and Citeseer (b)] and Mettack [on Cora (a) and Citeseer (b)].
each dataset, the first plot showcases the clean and attacked accuracy. while the second plot shows the Success Rate (the discrepancy between the clean and attacked accuracy for each budget). The experimental results demonstrate the existence of the previously discussed trade-off between clean and robust accuracies. Specifically, as anticipated, the clean accuracy exhibits a continual increase until reaching a plateau, corresponding to the convergence of the loss function to minimum. Conversely, the attacked accuracy demonstrates rising trend until reaching an inflection point, beyond which begins to decline. These findings confirms the observations from the derived upper-bound, indicating that a higher number of epochs leads to increased vulnerability in the model. Ideally, users would aim to stop training at the inflection point, where the attacked accuracy is maximized while the clean accuracy remains proximal to its convergence point.
6.3 Effect Of Initial Weight Distribution
We aim to validate the impact of the initial weight norms on the model's adversarial robustness. As previously discussed in Section 4. larger weight norm leads to the relaxation of the upper-bound, potentially resulting in the model being more susceptible to adversarial attacks.
In this perspective, we start by investigating the effect of sampling from a Gaussian distribution, as studied in Lemma [ We hence consider this latter by setting the mean value μ to constant, and analyzing the impact of the variance parameter Intuitively, based on the upper-bound analysis. higher variance value anticipated to result in reduced model robustness. Figure illustrates the resulting Success Rate across various variance values for both the "PGD" and "Mettack" methods, applied to the Cora and Citeseer datasets. The findings uncquivocally validate the thcoretical insights, demonstrating direct correlation between increasing the variance (a) and a higher Success Rates, indicating heightened vulnerability and reduced robustness of the model. Moreover, the impact of initialization becomes more pronounced when considering larger attack budgets, outlined in the computed upper- bound. Notably, for certain budgets (e.g.. 30% and 40%). the observed gap ranges between 5% and 15%, underscoring the initial weights significant implications on the robustness.
Within the same context, we explore alternative initialization strategies, focusing on two primary cases. First, we investigate sampling initial weights from a uniform distribution (1(-3,3), where can be seen as a scaling parameter for weight norms. Second, we consider employing scaled orthogonal weight initialization strategy. While this our aim can be approached by sampling weights
8

---

Pge- Luliune Incissaionn
Pgg- LIFDEE sEutAaion Cheee
Ustaa Lettan isaratoo
Pgr- Ortoggon www.taatoooono Core
Pgr- Orioggn Inwitaaioonn Chaveer
Motaet- Ornoot--rroreeeee Wittraaoon Core
Melted Otrgggaa Iww.tattiooooooo Chesee
[g]
(h3
Dren 2p1
Orale Maice
Ocee Value
Figure 3: Effect of the scaling parameter 3 on the model's robustness in the case of Uniform (a-d) and Orthogonal (e-h) Initialization when subject to PGD and Mettack using Cora and CiteSeer.
from a scaled random Gaussian distribution, we adopt the orthogonal initialization strategy proposed in prior work (26), which we further rescale by factor examine the impact on weight norms. In both cases, higher scaling parameter values of 3 are anticipated to theoretically yield higher upper-bounds and consequently render the model more vulnerable, as indicated by our computed bounds. We conduct numerical computations resulting adversarial robustness of GCN across various 3 values, as provided in Figure B] The experimental results exactly aligned with our theoretical findings showcasing the effect of the weight norm in the adversarial robustness. To summarize, while traditionally overlooked in prior studies adversarial robustness, our experimentation underscores the critical importance of selecting appropriate initialization distributions and strategies for enhancing model robustness.
both the Cora and Citeseer datasets to assess the
6.4 Experimental Generalization
We extend our experimenta- tion to empirically validate the thcoretical generalizations provided in both Section for the GINs and Section 5 for DNNs. To this end, we con- sider these two models with various initialization schemes, including the previously used Orthogonal 26 and Uniform initialization in addition to the
Figure 4: Effect of initialization on the GIN (a) and DNN (b) for different attack budgets.
Dr04 18477
Kaiming 117 and Xavier Ini-
tialization [3]. Our analysis primarily focuses on the PGD adversarial attack, using iden- tical attack budgets as the previous sections. Figure presents the results on the GIN (a) using the Cora dataset and (b) on the DNN using the MNIST dataset. Notably, we observe that the different initialization methods yield similar clean accuracy (e = 0), yet as the attack budget increases, the discrepancy in attacked accuracy between them also grows. For instance, in the case of DNNs, the accuracy gap between the best and worst initialization methods for = 0.1 ranges around 60% proving our main assumption related to the impact of initialization the model's robustness.
400
3.0 0.43 008 035
8.10
6.20
0255 3.00

---

7 Conclusion & Limitations
The current study shows that the dynamics of learning in GNNs and DNNs have an important effect on the model's final robustness. Specifically, we theoretically showed that the model's robustness is connected to the weight initialization and the number of training epochs. We empirically validate our findings, where we can see that choosing the right initialization can yield huge "almost-free" robustness improvement. We additionally showed the existence the right number of epochs to have the best clean accuracy and the most robust model. While the current work did not propose an alternative or a solution, it has introduced a new perspective. which research
trade-off between choosing
to knowledge, direction either by proposing new initialization schemes to improve robustness while guaranteeing good generalization or new gradient-based weight updates enforce the robustness of the model yet again by tracking robustness metrics alongside the loss function throughout training.
absent from the adversarial literature, opening the door
Acknowledgements
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computation (on GPUs) was enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at Alvis partially funded by the Swedish Research Council through grant agreement no. "2024/22-309". We furthermore want to thank Dr. Yassir Jedra for revising the manuscript and for very helpful discussion its different elements.
References
[1] Yassine Abbahaddou, Sofiane Ennadir, Johannes F. Lutzeyer, Michalis Vazirgiannis, and Henrik Boström. Bounding the expected robustness of graph neural networks subject to node feature attacks. In The Twelfth International Conference Learning Representations, 2024.
[2] Amr Alkhatib, Sofiane Ennadir, Henrik Boström, and Michalis Vazirgiannis. Interpretable graph neural networks for tabular data. In Proceedings of the 27th European Conference Artificial Intelligence, volume 392 of Frontiers Artificial Intelligence and Applications, pages 1848-1855, 2024.
[3] Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning, pages 291-301. PMLR, 2019.
[4] Aleksandar Bojehevski and Stephan Günnemann. Certifiable robustness graph perturbations, 2019.
[5] Aleksandar Bojchevski, Johannes Klicpera, and Stephan Günnnemann. Efficient robustness certificates for discrete data: Sparsity-aware randomized smoothing for graphs, images and more, 2020.
[6] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning, pages $54-863. PMIR, 2017.
[7] Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial Attack Graph Structured Data. In Proceedings of the 35th International Conference on Machine Learning, pages 11 1115-1124, 2018.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[9] Sofiane Ennadir, Yassine Abbahaddou, Johannes F Lutzeyer, Michalis Vazirgiannis, and Henrik Boström, simple and yet fairly effective defense for graph neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 21063-21071, 2024.
10

---

[10] Sofiane Ennadir, Amr Alkhatib, Giannis Nikolentzos, Michalis Vazirgiannis, and Henrik Boström. Unboundatack:: Generating unbounded adversarial attacks to graph neural net- works. In International Conference on Complex Networks and Their Applications, pages 100-111. Springer, 2023.
[11] Negin Entezari, Saba A Al-Sayouri, Amirali Darvishzadeh, and Evangelos E Papalexakis. All you need is low (rank) defending against adversarial attacks graphs. In Proceedings of the 13th international conference on web search and data mining, pages 169-177, 2020.
[12] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with Py PyTorc Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
[13] Xavier Glorot and Yoshua Bengio. Understanding the diffieulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
[14] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[15] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamesssing adver- sarial examples. In International Conference on Learning Representations (ICLR), 2015.
[16] Stephan Günnemann. Graph neural networks: Adversarial robustness. In Graph Neural Networks: Foundations, Frontiers, and Applications, pages 149-176. Springer, 2022.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
[18) Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In International Conference Learning Representations, 2020.
[19] Hongwei Jin, Zhan Shi, Venkata Jaya Shankar Ashish Peruri, and Xinhua Zhang. Certified robustness of graph convolution networks for graph classification under topological attacks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8463-8474. Curran Associates, Inc., 2020.
[20] Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of Computer-Aided Molecular Design, 30(8):595 608, 2016.
[21] Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations (ICLR), 2017.
[22] Xiaoxiao Ma, Jia Wu, Shan Xue, Jian Yang, Chuan Zhou, Quan Z. Sheng, Hui Xiong, and Leman Akoglu. A comprehensive survey on graph anomaly detection with deep learning. IEEE Transactions Knowledge and Data Engineering, pages 1-1, 2021.
[23] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Arlificial Intelligence and Statistics, pages 1922-1932. PMLR, 2018
[24] Aymen Qabel, Sofiane Ennadir, Giannis Nikolentzos, Johannes F. Lutzeyer, Michail Chatzianas- tasis, Henrik Boström, and Michalis Vazirgiannis. Advancing antibiotic resistance classification with deep learning using protein sequence and structure. bioRxiv, 2023.
[25] Lestie Rice, Anna Bair, Huan Zhang, and J Zico Kolter. Robustness between the worst and average case. Advances in Neural Information Processing Systems, 144:77400-7788 2021.
[26] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning deep linear neural networks. International Conference on Learning Representations (ICLR), 2014.
11

---

[27] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93 93, 2008.
[28] Xianfeng Tang, Yandong Li, Yiwei Sun, Huaxiu Yao, Prasenjit Mitra, and Suhang Wang. Transferring robustness for graph neural network against poisoning attacks. In Proceedings of the 13th International Conference on Web Search and Data Mining. ACM, jan 2020.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[30] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In ICLR, 2018.
[31] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Het- erogeneous graph attention network. In The world wide web conference, pages 2022-2032 2019.
[32] Huijun Wu, Chen Wang. Yuriy Tyshetskiy, Andrew Docherty, Kai Lu, and Liming Zhu. Ad- versarial examples for graph data: Deep insights into attack and defense. In Proceedings of the eventy-Eightle International Joint Conference on Artificial Intelligence, IJCAI-19, pages 4816 4823. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
[33] Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based Recommendation with Graph Neural Networks. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 346-353, 2019.
[34] Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein. Samuel Schoenholz, and Jeffrey Pen- nington. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pages 1393-5402. PMLR, 2018.
[35] Kaidi Xu, Hongge Chen, Sijia Liu, Pin- Yu Chen, Tsui- Wei Weng, Mingyi Hong, and Xue Lin. Topology attack and defense for graph neural networks: An optimization perspective. In International Joint Conference Artificial Intelligence (IICAI), 2019.
[36] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In 7th International Conference on Learning Representations, 2019.
[37] Haoxi Zhan and Xiaobing Pei. Black-box Gradient Attack on Graph Neural Networks: Deeper Insights in Graph-based Attack and Defense. arXiv preprint arXiv:2104. /5061, 2021.
[38] Xiang Zhang and Marinka Zitnik. Gnnguard: Defending graph neural networks against adversarial attacks. In NewrIPS, 2020.
[39] Dingyuan Zhu, Ziwei Zhang, Peng Cui, and Wenwu Zhu. Robust graph convolutional networks against adversarial attacks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1399-1407,2011
[40] Daniel Züggerr, Amir Akbarnejad, and Stephan Günnemann. Adversarial Attacks on Neural Networks for Graph Data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2847-2856, 2018
[41] Daniel Zügner and Stephan Günnemann. Adversarial attacks on graph neural networks via meta learning. In 7th International Conference on Learning Representations, 2019.
[42] Daniel Zügner and Stephan Günnemann. Certifiable robustness and robust training for graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, jul 2019.
12

---

Supplementary Material: If You Want to Be Robust, Be Wary of Initialization
A Proof of Theorem 2
Theorem. Let f: (A, - denote a graph-based function composed of GCN layers, where the initial weight matrix of the i-th layer is denoted by w For adversarial attacks only targeting node features of the input graph, with budget we have (in respect to Definition
T
II 22'
(2
+ + ||w
Σ
iu
with being the number of training epochs and w. denoting the sum of normalized walks of length (T' - 1) starting from node u.
Proof. Let's consider graph-function ff that is based T GCN-layers. The gradient descent update at epoch for a layer is written as:
/ w.( LWW().
Since consider that our loss function C to be L-smooth, we have the following result:
V s -||w w!
Consequently, after training epochs, can write:
w/ (0)
= ( - v v/
+ +
| (3)
2-1 w
< (1 + nb)
+ TtL W
In addition, we have that 17 < +. Hence, by recursion, find that:
(/ 5 (1 + nL)*
+ Σ 2d |
(3)
h=0
s (1 + nI)
[
2
[w
+
Giving that considering feature-based adversarial attacks, let X denote the original node features and X' denote the perturbed adversarial features. With an attack budget e, from the work ① have the following result:
T
V|A, B -
<
Σ Wi
(5)
AEEV
with wh denoting the sum of normalized walks of length (T - 1) starting from node Consequently:
T
sup (AA.)])
[|(AA)) - -(A, <
w
Σ vua
(6)
LuEV

---

From Equations 3 and 6 we conclude that:
T II
sup [M..))((4A
II(AA - f(A,x))
2
W
+
2
|
i
NeE
We conclude that is (e:7))robust with:
T
II
2 (
+
+
2
|
Σ
MEE
B Proof of Theorem 3
Theorem. Let f: (A, y denote graph- based function composed I GCN layers, where the initial weight matrix of the i-th layer is denoted by W Let f be the number of used training epochs. When f is subject to structural attacks, with a budget e, we have (in respect to Definition)
T
T
= II
2
W6
10 + 2
[w:0l) |X|||
[
2
W
(9)
2
Proof. Similar to the previous proof, let's consider graph-function f that is based on T GCN-layers and trained using gradient descent for epochs. We have the following result from Equation
w < 2"
2 w
For this proof, we are considering the model f to be subject to structural perturbations. In this perspective, let Á denote the input non-attacked adjacency and A' denote the attacked/perturbed adjacency, with h denoting its corresponding hidden representation. From the work [.] we have:
T
T - T IIIII
v| e ((AA..)),) - f(A,,
II
(
| e
By combining the two previous results, we get the following inequality and hence the desired result:
sup JA (x..))((AAA))).) IS(Á,)) -(())--(AAAA-) ((
2
w
+
2
|w!" |X
T 1 + T
2
(/
+ 2
i=1
C Proof of Lemma 4
Lemma. Let f: (A, -y denote graph-based function composed of GCN layers for which the initial weight are drawn from the Gaussian distribution ((,, E). When subject to node features based adversarial attacks, we have the following:
EwgrNNa. [R.
<
II
2 2
(r(2)
2
Σ @u LEe

---

Proof. Let consider f to be graph classifier based on 7-GCN layers for which the initial weight drawn from the Gaussian distribution. Specifically, Vi L, w/ (N,, 2). We have that:
| S vllal1++r(()).
E
From Theorem we have the following:
T
II
2
W
(6)
+ +
|
Σ
wr
Hence, combining the two elements results in the following:
II 2 V2 +r(() 2
Σ
LEE
D Proof of Theorem 5
Theorem. Let XA, - y denote graph-based function composed of T GIN layers, where the initial weight matrix of the i-th layer denoted by ((()) For adversarial attacks only targeting node features of the input graph, with budget have:
2" w/00
2
[w
[BT max deg(u)
7
wEV
with being the number training epochs and deg(u) is the degree of node
Proof. Let's consider a graph-function f that is based on GIN-layers and trained using gradient descent for t epochs. We have the following result from Equation 3
w s (1+ n1) |
21+1
S 2t
|
2
|
(8)
Let X denote the original node features and X' the perturbed adversarial features. For attack budget e, from the work [I] we have the following:
v[A', B(AA,),,,,, -
II
|
BT max deg(u) +e]
(9)
wEV
Consequently, we can merge the two inequalities resulting in the following:
7
2 w
2
BT max dug(u) + wEV
3

---

E Proof of Theorem 6
Theorem. Let f : X Rd + y S
Roat be a Tlaayers neural network with (( denoting the initial weight matrix of the i-th layer. When subject adversarial attacks, f is (c, robust with:
II
2
+ 2 |
Proof. Let f be T-lyyers neural network. We additionally assume that its corresponding activation functions are I-Lipschitz. Let æ (with h its hidden representation) be input vector and x' (corresp. h') its corresponding crafted adversarial input (corresp. hidden representation). For an adversarial attack with budget e, we have the following:
Va' E x: || x'll S se, ll)))) - [(x'1
1,00 1000
10500 (win + =b00) - (W(O)/--1) + (0)
| //- - 101111)
Recurrently, we find the final result as:
T [lf(z) - f(z))l II
sup
(10)
-Ex:|===>11
Note that similar results and analysis have been provided in previous work 6 3 By using the result derived in Equation we have:
w: s 2' 1 + 2++1 |
(11)
By merging these two inequalities, and applying the Markov Inequality, we find the following upper-bound:
7 II 2 | + 2 | |).
On the Case of Strong-Convexity Proof of Lemma 7
Lemma. Let : + out be a Tlayyess neural network trained with a u-strongly convex and L-smooth loss function. Let w denote the initial weight matrix of the i-th layer. When subject adversarial attacks, with a budget e, we have that f is (6,7) robust with:
= II =
(1-w/D)
M/L)²
[/(') +2 [
Proof. We consider f to be Tlyyess neural network (following the same propagation as equation the one presented in Section 5) From Section we have the following:
[I((x) s(>')ls w

---

In addition to the previous assumption of L-smoothness of the loss function, we consider that its je-strongly convex. Hence, for the layer (I), we have the following result:
Wx (9
- (1 m/
W w
w
(12)
$ (1 L/
+
w
(13)
When subject to adversarial attacks, we can use the previous result from tion (10):
specifically from Equa-
sup ot -|
[1(()) f(''|l II
(14)
1=1
Hence, by merging the two previous results, we deduce that:
II (n LT | +
I)
(15)
40%
3.5
(c)
2.5
Suczass
Succees
2.0
1.5
2
0.0
0.1 0.2 0.4 0.5 06 0.7 0.8 0.9 Value of
0.1 0.2 0.3
0.4 0.5
0.8 0.9
0.1 0.2 0.3 0.4 0.5 0.8 0.7 0.8 0.9
0.1
0.2 0.3
0.5 0.500.00 0889
Value
Value of
Value
Figure 5: Effect of the variance the model's robustness in the case of Gaussian Initialization when subject to DICE (a.b) and Random Attacks (c,d) for both Cora and CiteSeer.
G Additional Results
G.1 Adversarial Robustness Deep Neural Networks
We consider the general family of neural networks for which the computation during layer I, using activation function (!), can be written as
h h
S (L-1 60)).
with W(O) E 1. being the weight matrix and by E R the bias of the 1hh layer.
In this perspective, let f Ra + R be neural network no being the input dimension. The adversarial task in this case consists finding perturbed input ã for which the prediction differs from the original prediction fzx). The perturbed input ã should hence adhere to the similarity constraints defined by perturbation budget e. Let's consider the & norm within both the input space Inde and the output space IR, hence define the set of valid adversarial perturbation as:
5

---

((x; = { || و e).
Similar to Section 3 we can introduce the adversarial risk of DNN within the input's neighborhood defined by the budget the following:
R. = E sup [I((()) [(x)]]
(16)
[EB((;))
From this adapted adversarial risk, we can introduce the notion of DNN's adversarial robustness Definition (DNN Adversarial Robustness). The neural network f R - R is said to be (6,7) robust if its adversarial risk is upper-bounded by y, [ 7-
G.2 Additional Adversarial Attacks
addition to the previously reported Mettack and PGD adversarial attack, we consider two additional adversarial attacks. Notably, we first consider "DICE" which involves iteratively perturbing graph's s structure by adding or removing edges while ensuring connectivity, and then adjusting the perturbation based on the gradient of the graph neural network's loss function to generate an adversarial example. The process aims to find a minimal perturbation that misleads the network's predictions while keeping the perturbation size small. We additionally consider a "Random" attack which consists of randomly perturbing the adjacency matrix by dropping or adding edges. Figure5]shows shows the adversarial accuracy results the Cora and CiteSeer dataset when subject to DICE and Random attacks for different values of a of the Gaussian initialization. Similarly, Figure 66 shows the effect scaling both a uniform initialization and Orthogonal one previously explained in Section 6
Dice Uniform Cora
Dice Uhifom- Cileseer
Random Orthogonal Cora
Rendom Orthogonal Cileseer
1.4
6
1.2
Success Rate
10
10
Success Rate
1.0
0.8
0.6
0.4
0.2
0.00.51
1....02.33.....22 3.54.0 Sca Value
0.0.0.5 1.0 2.5 3.0.4.
1.0
3.0
4.0
1.0
1.5
2.0
2.5
3.0
3.5
4,0
Scale Value
Scale Value
Scale Value
Figure 6: Effect of Uniform and Orthogonal Initialization on the model's robustness in the case of DICE Attack on Cora (a.c) and CiteSeer (b,d).
G.3
Additional Datasets
We additionally extend the results to the ACM Dataset 31 preseets the results using the Mettack, PGD and DICE for the ACM dataset for the Gaussian initialization (effect of o), the Uniform and Orthogonal initialization.
within the node classification setting.
G.4 Additional Models
As previously explained in Section 5 while our theoretical analysis primarily focuses on GCN, GIN, and DNN models, the derived insights extend to other models as well. To illustrate this point, we examine the effect of initialization distribution on the performance of defense methodologies. Specifically. we first consider RGCN [39, which employs Gaussian distributions in its hidden layers mitigate the effects of adversarial attacks. We additionally consider GCN-Jaccard [32 which preprocesses the network by eliminating edges that connect nodes with jaccard similarity of features smaller than certain level. We use various initialization schemes, similar to those in our previous experiments, and evaluate against the same adversarial attacks (PGD, Mettack, and DICE). Figure & (resp. Figure 9 presents the adversarial accuracy and defense performance of RGCN (resp. (CN-Jaccard) the Cora, CiteSeer, and ACM datasets. Although the performance gap is not very

---

45%
1N1
20
Sucerm
Vilacce
Pgy. Oatlager bolotoriie
14
13
10
25
2,5
10
25
28
3.0
3.9
08
06
20
30
Scal
Value
Figure 7: Effect the Gaussian (a: b; c), Orthogonal (d; e; f) and Uniform (g:h;i) Initialization on the ACM dataset.
pronounced for Cora, it is clearly observed for CiteSeer and ACM. This demonstrates the broader applicability of our insights across different models but also defense methods.
H Datasets and Implementation details
Datasets Characteristics and information about the node classification datasets used in our experi- mental study are presented in T As outlined in the main paper, we conduct experiments on of citation networks, including Cora, CiteSeer (in the main paper), and ACM dataset (Appendix For all these datasets, we adhere to the train/valid/test splits provided by with the dataset.
31
About the architectures. In all of the experiments, the models employed 2-layer convolutional architecture (consisting of two iterations of message passing and updating) stacked with Multi-Layer Perception (MLP) readout. The intent was compare the models in iso- iso-architectural setting, to ensure fair evaluation their robustness. We maintained the same hyperparameters, including a learning rate of le-2, 300 epochs, and hidden feature dimension of 16 have been. To account for the impact of random initialization, each experiment was repeated 10 times.
Reproducibility of the experiments. We emphasize that all experiments should be easily repro- ducible by directly using the provided code. The archive contains ReadMe file containing a small documentation on how to run the experiments.
Table 1: Statistics of the node classification datasets used our experiments.
DATASET
#FEATURES
#NoDES
#EDGES
#CLASSES
CORA
1433
2708
5208
CITESEER
3703
3327
4552
6
7

---

Ottogand
Uniorr
Tpl. Oala
Cieseer
Pal.
0.70
542
0.20
082
0.00
3.79
0.44
845
8.00
22bc3 bu
ABloc bod.e-r
1batack- Dura
Mertick Auce
0.5
070
0.55
0.00
382
500 5. 610
815 422 045 330 5.5 5.6
000 305
015 029 345 1.30 0,5 3/0
0.00
005
0.10
0.15
620
035 020 520 0.00
Figure 8: Effect of the initial distribution on RGCN's robustness and performance when subject to structural adversarial attacks.
On the adversarial attacks. For the PGD attack on the MNIST dataset, we used step-size of 0.1 and we set the number of iterations to 100 (which was observed to be enough for the attack convergence). Note that we set these parameters for all the considered initializations in Figure as our aim is to compare the effect of the different distribution on the final robustness.
Implementation details. Our implementation is available in the supplementary materials (and will be publicly available afterwards). It is built using the open-source library PyTorch Geometric (PyG) under the MIT license [2]. We used the publicly available implementation of the adversarial attacks provided in the DeepRobust package httpss//gitub..ommDSS wwwp/D/SE-SSUDDeeRRooeeeo used the implementation from the same package. The experiments have been run on both NVIDIA A100 GPU where training a GCN takes around 1.2(2) s.
For RGCN, we
8

---

Otbogand
Uniom
Tpl. Ocla
Clesee
Pal.
3.78
672
6.07
8.00
22bch bu
Alloc bod.e-----
1Matack Dura
Mertick Aace
0.70
0.05
872
389
0.0
0.70
0.00
500 5.6 6.0
815 622
330 5.5 5.60
000 305
3.1
0.5 929
945 8:00 0.5 340
0.00 005
0.10 0.5 620 035
020
0.00
Figure 9: Effect of the initial distribution on GCN-Jaccard's robustness and performance when subject to structural adversarial attacks.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In addition to stating the novelty of our proposed approach, we used our abstract and introduction summarize our main findings and contributions related the effect of initialization on the adversarial robustness (as theoretically justified and empirically tested in the following sections).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Together with our conclusion, we presented the set limitations of work. Specifically, we stated that while work is innovative, we didn't 't provide solution to

---

the initialization problem from an adversarial defense perspective. We also discussed in the "problem setup" section our different theoretical choices (the smoothness of the loss function) and how realistic they are.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For each Theorem, Lemma and theoretical claim, we provide the proof in the Appendix and point out to the corresponding section in the main paper. We also stated all the assumptions and analytical choices in the Preliminaries (Section 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
10

---

Justification: In addition to providing the code as supplementary materials, we have provided all the implementations details that are sufficient to reproduce the results. These details include the used syper-parameters (the architecture, learning rate ... and also for the used adversarial attacks we provide the different parameters used. We also point out the dataset The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
that we used (which are public) and that we used the same public folds as the one provided with the datasets.
Guidelines:
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the anonymized code following the Neurips guidelines. Specifi- cally, we submitted the code with the supplementary material section and we clearly state the steps to run it using a ReadMe file. Please note that for this question, we consider "open source" as providing the code to the reviewers and making it public afterwards for the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
11

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ht.ps..cc/ubblcc/uuddeeee Ssw.aeessssoooSeessssssaaasaaase wwwpeeSuuuiiissiiiiiii ssiiioliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provided the details about the architecture, the used for the considered models (Section H of the Appendix) and all the typer-parameter used for our adversarial attacks. Note that our work's goal is to provide comprehensive overview of the effect of initialization on the robustness, hence making sure that the same choice of hyper-parametent enough to ensure the fairness of the experiments. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We reproduce each experiment 10 times to take into account the factor of randomization and we report the mean value. Note that since we use mainly figures (which are appropriate for our setting - given the different attack budgets we are using). this seemed as the perfect approach. For the train/test folds, we use the public folds provided with each dataset and hence reducing the effect of randomization.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
12

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resourees
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We reported the details of implementation in Section H of the Appendix, where we specified the GPU that was used and the average time to do the experiments. Note that while we have chosen to use a GPU, our experiments can be easily done using a CPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow the guidelines of the Neurips Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provided overview the harm that adversarial attacks can have the applications of Deep Learning models. The main goal of our paper is to identify new potential factors related adversarial attacks and hence should rather have positive impact on the society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
13

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In this work, study the theoretical effect of initialization the adversarial robustness. We don't 't provide any new pre-trained model nor new datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We made sure to cite the papers that are relevant to our work and that were used justify some theoretical empirical insights. For the different code implementations, cited clearly the license and the owner the used function/code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
14

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: have provided the implementation code together with all the experimental details to reproduce work. We also clearly justiff the use of the packages and their license. Note that the code have been anonymized and provided as supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
There is no crowdsourcing nor research with human subjects our case.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no crowdsourcing nor research with human subjects in our case.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
15

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16


Reviews:
[{"review_id": "O86594TjIH", "reviewer": "Reviewer_bCNj", "summary": "The paper investigates the under-explored impact of weight initialization on the robustness of Graph Neural Networks (GNNs) against adversarial perturbations. The authors present a theoretical framework linking weight initialization strategies and training epochs to the model's resilience to adversarial attacks. They derive an upper bound that connects the model's robustness to initial weight norms and training epochs, showing that appropriate initialization strategies enhance robustness without degrading performance on clean datasets. The findings are validated through extensive experiments on various GNN models and real-world datasets subjected to different adversarial attacks, extending the theoretical insights to Deep Neural Networks (DNNs).", "questions": "1. Interaction with Other Defense Mechanisms: how do the proposed initialization strategies interact with existing adversarial defense mechanisms, such as adversarial training or regularization techniques? For example, does combining your initialization methods with adversarial training lead to improved robustness, or are there any diminishing returns?", "limitations": "The theoretical framework is primarily validated on GNNs, leaving uncertainty about its applicability to other types of neural networks, such as recurrent neural networks (RNNs) or transformer models. This limitation reduces the perceived impact and versatility of the findings. Extending the theoretical analysis and experimental validation to include these other neural network architectures would enhance the generalizability of the results.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality:\n    * The paper addresses a novel dimension of adversarial robustness in GNNs by focusing on weight initialization strategies, an area largely unexplored in existing literature.\n    * Extends the theoretical framework to apply broadly to DNNs, showcasing its versatility and broader applicability.\n\n2. Quality:\n\n    * The theoretical analysis is rigorous, providing a clear connection between initialization strategies, training epochs, and model robustness.\n    * Extensive experiments with diverse models and real-world datasets validate the theoretical findings, demonstrating the practical significance of the proposed framework.\n\n3. Clarity:\n\n    * The paper is well-structured, with a logical flow from theoretical analysis to experimental validation.\n\n4. Significance:\n\n    * The insights on the impact of weight initialization on adversarial robustness can influence future research directions and the development of more robust GNNs and DNNs.", "weaknesses": "1. Hyperparameter Tuning:\n    * The paper does not discuss the sensitivity of the proposed initialization strategies to other hyperparameters, such as learning rate, batch size, or the number of layers. This omission could lead to challenges in replicating and generalizing the findings, as the effectiveness of the initialization strategies might vary with different hyperparameter settings.\n\n2. Practical Implementation:\n    * The practical implementation details of the proposed initialization strategies in real-world scenarios are not deeply explored. Practitioners might struggle to adopt these strategies without clear guidance on integrating them into existing workflows. Including a discussion on how to balance the trade-off between choosing the right number of epochs to achieve optimal clean accuracy and the most robust model would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for their feedback and comments. \n\nAs detailed in our “General Rebuttal”, the main focus of our research is related to GNNs, hence why the experimental setting was rather focused on this side. The extension to other models (such as DNNs in the paper or the additional ResNet on Cifar-10 that we provide in the general rebuttal - Table 1) was rather to show the extensibility of the theoretical results which we have seen to be missing from the corresponding literature. We will point out this further in our limitation section. In what follows we aim to address the raised concerns and questions one-by-one.  \n\n**[W1 - Regarding the hyper-parameters]** Note that the provided theoretical results (Theorem 2 & 3) does indeed show the possible effect of the “number of layers” and “learning rate” (Equation 3 Section A of the Appendix). Hence, the reviewer is right on their intuition regarding the possible effect of hyper-parameters. Since the goal of our experimental setting (described in Appendix H) was aimed to validate the theoretical results, we followed the same hyperparameters that are “usually” used in the literature to ensure the classifier and the attacker’s convergence. Typically, we focused on a 2-layer GCN (since it can reach state-of the art accuracy and also because too many more layers result in the downgrade of the accuracy – which is known as over-smoothing in the GNN literature).  \n\n\n**[W2 - Regarding the practical usage of the results]** This is indeed a fair and important point. Our theoretical analysis main result is that the dynamics of the training (Number of epochs, learning rate and the number of layers) also affect the final model’s adversarial robustness. Consequently, one possible future direction that could be investigated is a robustness metric to track the advancement of a model’s robustness (similar to the validation accuracy which is used to select the top model). Typically, one possible metric or measure would be to approach the proposed adversarial risk quantity (Equation2) using an estimator (such as randomised or stratified sampling from the input’s neighbourhood and evaluating the distance to the input’s prediction). \nTo summarise, in addition to taking into account our proposed results (such as finding the right trade-off of epochs to reach convergence and adversarial robustness and choosing the right initialization), we need to investigate a model’s validation approach to keep track of its adversarial aspect. We will incorporate these practical guidelines  in our revised manuscript, serving to make our theoretical and experimental insights more useful in practice. \n\n**[Q1 - On the interaction with the adversarial defenses]** We have indeed studied the possible combination of our proposed insights with some “benchmark” graph adversarial defenses in Appendix G.4 where we studied RGCN (Figure 8) and GNN-Jaccard (Figure 9). For the sake of generalization, and we thank the reviewer for pointing out the idea, we have computed the results on adversarial training (based on PGD) and using TRADES [1] on the ResNet model using the Cifar-10 dataset. Table 2 reports the clean and attacked accuracy, showcasing therefore the application of our theoretical insights both to adversarial defenses but also to other architectures (besides GNNs and DNNs). \n\nTable 2: Effect of Initialization on ResNet on the clean and attacked accuracy when subject to PGD adversarial training (AT-PGD) and TRADES with $\\epsilon=8/255$.\n| Initialization | Clean Accuracy (AT-PGD) | Attacked Accuracy (AT-PGD) | Clean Accuracy (TRADES) | Attacked Accuracy (TRADES) |\n|----------------|:-----------------------:|:------------------------:|:-----------------------:|:------------------------:|\n| Orthogonal     |     83.8 $\\pm$ 0.31     |      51.0 $\\pm$ 0.37     |     82.6 $\\pm$ 0.47     |      55.6 $\\pm$ 0.32     |\n| Uniform        |     82.9 $\\pm$ 0.17     |      54.1 $\\pm$ 0.20     |     82.1 $\\pm$ 0.23     |      57.9 $\\pm$ 0.28     |\n| Kaiming        |     83.8 $\\pm$ 0.22     |      46.5 $\\pm$ 0.27     |     82.8 $\\pm$ 0.27     |      52.4 $\\pm$ 0.35     |\n| Xavier         |     83.2 $\\pm$ 0.15     |      46.9 $\\pm$ 0.19     |     82.7 $\\pm$ 0.19     |      52.2 $\\pm$ 0.23     |\n\n\n\n—\n\n[1] Zhang, Hongyang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. \"Theoretically principled trade-off between robustness and accuracy.\" In International conference on machine learning, pp. 7472-7482. PMLR, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5ojRDRl9ss", "reviewer": "Reviewer_7mx1", "summary": "This work investigates the relationship between weight initialization and adversarial robustness, specifically in Graph Neural Networks (GNNs). In this setting, a defender wants  to train a GNN for which, given an input graph X, and adversary cannot find a similar graph which induces a different output from the GNN than X. The authors prove an upper bound on the adversarial risk of a given GNN that corresponds to the norm of the initial weights of the model. Bounds of this type are proved for both node feature robustness and structural robustness, with initialization shown to be more important in the structural case. Similar bounds are derived for other GNN architectures, like the GIN, and for DNNs more generally. Through experiments, it is shown that, as indicated by the theory, training models for more epochs can lead to higher robustness and lower variance parameters in weight initialization leads to higher robustness. These trends are shown to hold for different architectures, datasets, and weight initialization techniques.", "questions": "- How tight are the bounds presented in sections 4 and 5? Do you think future work might be able to tighten these bounds?\n- In section G.1, it looks like the definition of adversarial risk is average-case (taking the expectation over all perturbations in a neighborhood), rather than worst-case. Was this intentional? It seems in conflict with the definition offered in equation 2.\n- It seems like these results may also suggest that regularization (to target final weights with a smaller norm) is good for robustness. There have been recent papers that study the relationship between regularization and robustness [1,2], do you think your paper relates to this line of research?\n- What are the implications of this for fine-tuning? Is this framework still useful when starting from a pretrained model?\n\n\n[1] Nern, Laura F., et al. \"On transfer of adversarial robustness from pretraining to downstream tasks.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Dai, Sihui, Saeed Mahloujifar, and Prateek Mittal. \"Formulating robustness against unforeseen attacks.\" Advances in Neural Information Processing Systems 35 (2022): 8647-8661.", "limitations": "Limitations are addressed in the problem setup and conclusion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper seemingly has high originality, being the first to study the impact of weight initialization on model robustness. It may have significant implications for the field of adversarial robustness, and may lead to improved adversarial training techniques. Clarity is one of this paper's main strengths. It is clear how the claims made in the introduction relate to the theoretical results, and the proofs provided in the appendix are easy to follow. Overall, I think this is a well-written paper with interesting results.", "weaknesses": "While the experimental section does validate many of the main results of the paper, there are other experiments that I would have liked to see to back up some claims made in earlier sections. Specifically, it would have been nice to see empirical results mirroring figure 2 but showing how different initializations impacted clean accuracy to show what the trade-off between accuracy and robustness looks like in a realistic setting.\n\nAlso, I think the framing and ordering of this paper may limit its reach. Based on the abstract, this paper seems to be targeted towards those who are interested in graph neural networks, despite the fact that the generalized results would be compelling even absent the discussion of GNNs. I feel this paper would be more effective if it put the generalized results (section 5) first, and then followed that up with a case study in GNNs to give an example of how this theory can be applied in specific settings.\n\nFigures 1-3 are difficult to parse without closely reading their descriptions in the text. I would appreciate if the legends were made more descriptive and the captions were more detailed as to what is being shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank the reviewer for their feedback and their comments. \n\nWe are grateful for your comment on the novelty of our theoretical analysis and direction. Regarding the generalization of the results, we have drafted a complete response in the “General Rebuttal” as all the reviewers evoked this point. While this interest is certainly encouraging, we note that the framing of the paper was in direction with our main interest which revolves around GNNs in which we have enough knowledge to make assumptions about the architectures and the internal dynamics. For instance, in the GCN’s case (Theorem 2), the upper-bound is dependent on the “normalized walks” within the input graph (meaning that denser graphs shall result in more adversarial effect). We believe that similar insights can be derived for other architectures such as those in the image domain. Hence, this part of our manuscript can be seen as pointers to more advanced specific studies in each domain that can eventually result in interesting upper-bounds. Finally, we would like to refer the reviewer to the additional results on ResNet using the CIFAR 10 dataset in our general rebuttal (Table 1) and also the effect of adversarial training and other defenses such as TRACES on the same architecture (Table 2 - Response to Reviewer 1). \n\nIn what follows, we try to address the additional questions/concerns raised by the reviewer: \n\n**[W1 - In respect to the clean accuracy of Figure 2]** In our experimental results, we have chosen to report the success of the attack since we thought it’s a better experimental representation of the theoretical results (our adversarial risk definition consists of the gap between the clean/attacked model). Nonetheless, ensuring a reasonable initial/clean accuracy (close to the state-of-the art by a 2% order) was important, hence the choice of the distribution’s range of parameters (value of $\\sigma$ of the gaussian distribution and the scale value in the Orthonormal and Uniform distributions). Table 2 reports the clean accuracy in the case of the extreme cases of the initial distributions used in the analysis in Figure 2. We will add the complete figure of clean accuracies in the Appendix.\n\nTable 3: GCN’s clean accuracy when subject to different initialization distribution using different parameters.\n\n|          | Gaussian ($\\sigma=0.1$) | Gaussian ($\\sigma=0.9$) | Uniform ($\\text{scale}=1$) | Uniform ($\\text{scale}=4$) | Orthogonal ($\\text{scale}=1$) | Orthogonal ($\\text{scale}=4$) |\n|----------|:-----------------------:|:-----------------------:|:--------------------------:|:--------------------------:|:-----------------------------:|:-----------------------------:|\n| Cora     |      83.5 $\\pm$ 0.4     |      82.8 $\\pm$ 0.9     |       83.8 $\\pm$ 0.6       |       82.5 $\\pm$ 0.8       |        84.1 $\\pm$ 0.31        |         83.2 $\\pm$ 0.5        |\n| CiteSeer |      73.1 $\\pm$ 0.3     |      71.6 $\\pm$ 0.7     |       72.8 $\\pm$ 0.7       |       70.5 $\\pm$ 0.5       |         72.4 $\\pm$ 0.4        |         71.6 $\\pm$ 0.6        |\n\n\n**[Q1 - Regarding the tightness of the upper-bounds]** Indeed, these bounds can be tightened depending on the used optimization algorithm and the considered assumptions. In the case of Gradient Descent (GD) that we considered in the paper, we did not find better bounds for the iterates than those presented in the appendix (Eqs. (3) and (4) under non convex assumption and Eqs. (12) and (13) under strong convexity assumption). Regarding the final bounds we gave in sections 4, they can be tightened as stated after Theorem 2: \" the dependence of $\\gamma$ on $t$ can be sharpened by having $(1+\\eta L)^t$ instead of $2^t$. With small $\\eta$ (which is the case usually in practice), $(1+\\eta L)^t \\approx 1+ t\\eta L$ resulting in a bound which depends linearly in $t$”.\n\n**[Q2 - On the definition of adversarial risk]** This is a small typo in writing, note that we have used the correct formulation in the Theorem’s proof (Appendix E - Eq. 10). We thank the reviewer for spotting this and we will correct it in the revised manuscript. \n\n\n**[Q3 - Linking to regularization techniques]** The idea of adversarial defense through regularization techniques is indeed very related to our work. The main difference is that in addition to the final weight’s norm, we also consider the initial weights, the learning rate and the number of epochs. Hence, our method can be seen as a generalization of studying the regularization techniques, in which we also focus on the training dynamics. \n\n**[Q4 - Extending to fine-tuning]** This is indeed a very interesting direction that could be seen as a sub-result of our theoretical analysis. For now, there isn’t much work on pre-trained GNNs, but we expect this field to expand in the coming years, which makes this question all the more relevant. In the case of other domains (such as NLP), there are two directions that could be investigated in this perspective. The first direction is when we consider that we have no control over the pre-trained model, hence the only control is over the classification/regression head that would be trained. In this case, our theoretical result is applied in the choice of the initial distribution of the weights and the number of fine-tuning epochs where a balance between accuracy and robustness should be found. In the second direction, where we have control over the pre-trained model, we don’t think our theoretical result can be applied since the pre-training consists of a number of “self-supervised tasks” (such as token masking or contrastive learning), and we believe that our adversarial risk definition doesn’t necessary extend from the pre-training embedding space to the downstream tasks. This latter point can be indeed an excellent research direction in which the goal would be to investigate the best pre-training dynamics capable of ensuring the downstream adversarial robustness of a pre-trained model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J99XrEjGDh", "reviewer": "Reviewer_ZfjT", "summary": "This paper studies the impact of weight initialization (and training epochs) on adversarial robustness of a GNN model, both theoretically and empirically. The analysis is also extended to DNNs in general, although this is not the focus of this paper. The theoretical and empirical analysis both suggest that increase to norm of weights of initialization and increase in training epoch both have negative influence on adversarial robustness. The paper later compares different initialization strategies and show they lead to difference in robustness in their experiments.", "questions": "1. Regarding to Fig.2 , do they  all trained to convergence and how their test accuracy look like?\n2. Regarding to Fig.4, is it a single run of experiment or average of multiple runs?", "limitations": "Presentation of limitations of this paper is decent.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper studies the influence of model initialization and training epochs to the adversarial robustness of the model, with both theoretical and empirical analysis. This paper also studies both GNN and DNN in general, with a great balance between staying focused, and comprehensiveness of the paper. The presentation of the paper is great and easy to follow.", "weaknesses": "Although there are many theories and claims presented in the first half of the paper, they can all simply be interpreted as \"smaller norm of weights lead to more tightly bounded distance in output space given a same perturbation and thus better robustness\", which is not that interesting. Especially, it seems that it did not consider the softmax function, or other type of normalization that may be applied.\n\nGiven these theories give very loose bound anyways, I am more interested to see how different initialization make a difference in practice. One big question of Fig.2 is that if they all trained to convergence and how their test accuracy look like. I hope to see more details about the experiment setting as well.\n\nSection 6.4 is the most interesting and important section in my view, where it compares different initialization strategies and their influence on robustness. It is unfortunately too short and it could be made great if authors could expand the experiments, draw some conclusions, or reveal more deep insights on the choice of initialization. To prove that different initialization do lead to different robustness,  I think it need to cover more models and attacks. And importantly, as there is randomness in the initialization and training (if using dropout for example), I don't know if the results shown in Fig.4 is a single run of experiment or average among multiple runs. It should be average of multiple runs to avoid randomness which can lead to different conclusions.\n\nI am happy to be convinced to the value of this paper if these major concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. We would like to answer their main concerns and questions as follows:\n\n**[W1 - General comment]** We feel slightly misunderstood by this comment and therefore want to apologize for any confusion that may have arisen from a possible lack of clarity in our explanations. In what follows, we try to address some of the perspectives that we feel weren’t fully clear:\n\n- Starting from our risk definition, we can fairly consider that if for an input point $x$, the output representation of the clean classifier and the attack classifier is close in the output space, then we can expect that the attack has failed (since the softmax function is just a monotone transformation based on the final output presentation). The motivation for not taking into account the softmax function in our setting was due to the desire to be general which is important in the case of GNNs since different downstream tasks are considered (node classification, node regression, graph classification/regression and link prediction).\n- “Smaller norm weights” was actually already investigated by other papers (such as Parseval regularization[1]). In our case, we rather consider how different training dynamics (such as number of epochs, the considered initialization and the learning rate) can affect reaching robustness. Hence, the novelty of our approach is focused on the training dynamics instead of making the statement that  “smaller weight norm” will help in robustness (which is already well investigated and well known in the adversarial literature).\n- Regarding the tightness of the bounds, the idea was to show the link between the previously discussed dynamics and the adversarial robustness. The provided bounds were actually the best we could come up without additional assumptions (making the study not relevant in practice). For instance, the bound can be tightened as stated after Theorem 2: \" the dependence of $\\gamma$ on $t$ can be sharpened by having $(1+\\eta L)^t$ instead of $2^t$. With small $\\eta$ (which is the case usually in practice), $(1+\\eta L)^t \\approx 1+ t\\eta L$ resulting in a bound which depends linearly in $t$”.\n\n**[W2 - Regarding the generalization aspect]** We are grateful that this sub-result is of interest to you and the other reviewers. We have drafted a more detailed clarification to this point in our “General Rebuttal”. The main idea is that our interest revolves around GNNs in which we have enough knowledge to make assumptions about the architectures and the internal dynamics of the Message-passing framework. For instance, in the GCN’s case (Theorem 2), the upper-bound is dependent on the “normalized walks” within the input graph (meaning that denser graphs shall result in more adversarial effect). We believe that similar insights can be derived for other architectures such as those in the image domain. Consequently, the proposed generalization serves as a theoretical pointer to other researchers from the images or NLP domain to extend these results and eventually find more relevant and concise upper-bounds. We finally would like to refer the reviewer to the additional results on ResNet using the CIFAR 10 dataset in our general rebuttal (Table 1) and also the effect of adversarial training and other defenses such as TRACES on the same architecture (Table 2 - Response to Reviewer 1) which will be added to our revised manuscript. \n\n**[Q1 - On the convergence of the models]** All the considered models in Figure 2 were trained until convergence. Of course as pointed out, the initialization can have an effect of the clean accuracy. We have therefore chosen the distribution’s range of parameters (value of $\\sigma$ of the gaussian distribution and the scale value in the Orthonormal and Uniform distributions) such as to be close to the state-of-the art performance of a GCN (with a maximum 2% margin). In the now following Table 4, we present the clean accuracy in the case of the extreme cases of the initial distributions:\n\nTable 4: GCN’s clean accuracy when subject to different initialization distribution using different parameters.\n\n|          | Gaussian ($\\sigma=0.1$) | Gaussian ($\\sigma=0.9$) | Uniform ($\\text{scale}=1$) | Uniform ($\\text{scale}=4$) | Orthogonal ($\\text{scale}=1$) | Orthogonal ($\\text{scale}=4$) |\n|----------|:-----------------------:|:-----------------------:|:--------------------------:|:--------------------------:|:-----------------------------:|:-----------------------------:|\n| Cora     |      83.5 $\\pm$ 0.4     |      82.8 $\\pm$ 0.9     |       83.8 $\\pm$ 0.6       |       82.5 $\\pm$ 0.8       |        84.1 $\\pm$ 0.31        |         83.2 $\\pm$ 0.5        |\n| CiteSeer |      73.1 $\\pm$ 0.3     |      71.6 $\\pm$ 0.7     |       72.8 $\\pm$ 0.7       |       70.5 $\\pm$ 0.5       |         72.4 $\\pm$ 0.4        |         71.6 $\\pm$ 0.6        |\n\n**[Q2 - On the number of trials]** We apologize if this wasn’t highlighted enough in our manuscript. As explained in our experimental setting (Section 6.1 and in details in Appendix H): “To mitigate the impact of randomness during training, each experiment was repeated 10 times, using the train/validation/test splits provided with the datasets.” We will add this also to the figure caption as well in the revised manuscript.\n\n—\n\n[1] Cisse, Moustapha, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. \"Parseval networks: Improving robustness to adversarial examples.\" In International conference on machine learning, pp. 854-863. PMLR, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mUAAQhrIMz", "reviewer": "Reviewer_A9tB", "summary": "This paper provides a theoretical study on the impact of number of epoch and initialisation to adversarial attack for GNN, potentially generalise to DNN. The theoretical evidence is supported by some empirical results.", "questions": "Please see the weakness.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The study of how number of training epoch and initialisation affect GNN robustness is novel and interesting. \n\nThe presentation is clear.\n\nBoth theoretical and empirical evidence are provided.", "weaknesses": "The study presents novel insights into Graph Neural Networks (GNNs), however, it is somewhat discussed in Deep Neural Networks (DNNs) regarding the number of training epochs [1, 2, 5] and initialisation methods [3, 4].\n\nWhile the study is interesting, its contribution is somewhat limited due to the lack of proposed methods based on the findings.\n\nFrom my understanding, the empirical validation is based on a single architecture, which may introduce limitations and bias. Given that the findings focus on initialisation, the empirical validation should cover a wider range of architectures.\n\nThe additional study on DNN and GIN is appreciated. However, the empirical evidence primarily focuses on outdated adversarial setups. The evidence would be more convincing if the authors provided empirical results using current and more practical adversarial setups, such as large-scale datasets (e.g., ImageNet-1K), modern architectures (e.g., ResNet, ConvNeXt, ViT), and recent adversarial attacks (e.g., AutoAttack, APGD) [5].\n\nI believe that without the proposed method, validating the theoretical findings on a wider range of adversarial setups, as suggested, would help improve the contribution of this paper.\n\n[1] Mo, Yichuan, et al. \"When adversarial training meets vision transformers: Recipes from training to architecture.\" NeurIPS 2022.\n\n[2] Pang, Tianyu, et al. \"Bag of tricks for adversarial training.\" ICLR 2021.\n\n[3] Hua, Andong, et al. \"Initialization matters for adversarial transfer learning.\"  CVPR 2024.\n\n[4] Vaishnavi, Pratik, Kevin Eykholt, and Amir Rahmati. \"A Study of the Effects of Transfer Learning on Adversarial Robustness.\" Transactions on Machine Learning Research.\n\n[5] Singh, Naman Deep, Francesco Croce, and Matthias Hein. \"Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models.\" NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and we would like to answer their main concerns and questions as follows:\n\n**[W1 - Regarding the provided references]** We thank the reviewer for these references that we weren’t aware of and we will include a complete analysis and discussion in our “Related work” section. It seems that the majority of these works are rather interested in the specific effect of initialization and other dynamics in the case of adversarial training. Our framework is rather more focused on the classical training and how different parameters (such as number of layers, number of epochs and learning rate) can affect the final underlying robustness. The majority of these proposed papers are also focusing on the empirical side hence our theoretical analysis can be of relevant to close the gap. In fact, we believe that our theoretical analysis can be adapted for the case of adversarial training (where we can consider that the “real” inputs have a gradient towards the downstream task and the “generated adversarial” inputs have a rather negative one).  \n\n**[W2 - On the considered benchmarks]** We would like to thank the reviewer for seeing the value of the generalization section of our theoretical framework (which we note is not our main focus). We have drafted a more complete answer in our “General Rebuttal” as it was a common point between the reviewers. Typically, we wanted to underline that our main interest revolves around GNNs in which we have enough knowledge to make assumptions about the architectures and the internal dynamics of the Message-passing framework. For instance, in the specific case of GCN (Theorem 2), the upper-bound is dependent on the “normalized walks” within the input graph (meaning that denser graphs shall result in more adversarial effect). With a higher knowledge of the domain, similar theoretical results can be derived in the case of other domains such as NLP or Images resulting in more useful insights. Consequently, we believe that the introduced generalization part can be seen as pointers aimed to help close a gap which we have seen to be missing in the literature and which can be a first direction for other researchers to expand (such as the previously discussed adversarial training setting). Upon your suggestion (for which we are deeply grateful), we have initiated some new experiments to show the validity of our results on the ResNet family as well and to extend from MNIST (which was used for the DNN) to the Cifar-10 dataset. Table 1 reports the effect of the initialization on the ResNet model with both PGD and FGSM attacks. We will add these results (and expand them to other attacks) in our revised manuscript. \n\n\nTable 5: Effect of Initialization on ResNet when subject to FGSM and PGD attacks using CIFAR-10 Dataset.\n| Initialization | Clean Accuracy | FGSM($\\epsilon=0.03$) | FGSM($\\epsilon=0.07$) | PGD($\\epsilon=0.03$) | PGD($\\epsilon=0.07$) |\n|----------------|:--------------:|:---------------------:|:---------------------:|:--------------------:|:--------------------:|\n| Orthogonal     | 91.6 $\\pm$ 0.2 |     44.1 $\\pm$ 0.6    |     39.2 $\\pm$ 0.5    |    21.9 $\\pm$ 0.6    |    11.8 $\\pm$ 0.5    |\n| Uniform        | 91.2 $\\pm$ 0.3 |     46.8 $\\pm$ 0.4    |     41.8 $\\pm$ 0.3    |    24.3 $\\pm$ 0.3    |    13.6 $\\pm$ 0.4    |\n| Kaiming        | 92.3 $\\pm$ 0.1 |     42.3 $\\pm$ 0.2    |     36.9 $\\pm$ 0.2    |    20.7 $\\pm$ 0.4    |    10.1 $\\pm$ 0.3    |\n| Xavier         | 92.1 $\\pm$ 0.2 |     42.9 $\\pm$ 0.3    |     37.6 $\\pm$ 0.4    |    21.2 $\\pm$ 0.5    |    10.6 $\\pm$ 0.1    |\n\n\nWe have also conducted a number of experiments on the interaction of our theoretical results with possible other defense methods (such as Adversarial training and TRADES [1]) which are in the same direction as the references you have provided. \n\nTable 6: Effect of Initialization on ResNet on the clean and attacked accuracy when subject to PGD adversarial training (AT-PGD) and TRADES with $\\epsilon=8/255$.\n| Initialization | Clean Accuracy (AT-PGD) | Attacked Accuracy (AT-PGD) | Clean Accuracy (TRADES) | Attacked Accuracy (TRADES) |\n|----------------|:-----------------------:|:------------------------:|:-----------------------:|:------------------------:|\n| Orthogonal     |     83.8 $\\pm$ 0.31     |      51.0 $\\pm$ 0.37     |     82.6 $\\pm$ 0.47     |      55.6 $\\pm$ 0.32     |\n| Uniform        |     82.9 $\\pm$ 0.17     |      54.1 $\\pm$ 0.20     |     82.1 $\\pm$ 0.23     |      57.9 $\\pm$ 0.28     |\n| Kaiming        |     83.8 $\\pm$ 0.22     |      46.5 $\\pm$ 0.27     |     82.8 $\\pm$ 0.27     |      52.4 $\\pm$ 0.35     |\n| Xavier         |     83.2 $\\pm$ 0.15     |      46.9 $\\pm$ 0.19     |     82.7 $\\pm$ 0.19     |      52.2 $\\pm$ 0.23     |\n\n\n—\n\n[1] Zhang, Hongyang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. \"Theoretically principled trade-off between robustness and accuracy.\" In International conference on machine learning, pp. 7472-7482. PMLR, 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Comment to all the Reviewers:**

We are grateful to all the reviewers for their comments on the potential novelty of our theoretical insights and direction and we are also happy that our “generalization to other model” section has caught their attention. We would like to point out that our main research area revolves around GNNs (which in itself represents a vast research area from which different applications have emerged). In this perspective, our deep knowledge on the different architectures and adversarial attacks/defenses (which are different from the Image domain for instance, given the discrete aspect of the graphs) is within this area. The generalization part (Section 5) has been added as we have seen that our theoretical framework can easily be adapted to different architectures and that similar theoretical insights are missing from other domains (such as Images and NLP). In this direction, the adaptation to DNN (Theorem 3) aimed to give pointers and allow other researchers interested in these domains, which are more knowledgeable in architectures such as CNNs or RNNs, to unlock a new dimension in adversarial robustness. 

Nonetheless,  upon specific request of some reviewers, we have provided additional results using ResNets in Table 1 on the Cifar-10 Dataset with both PGD and FGSM attacks in order to show the extension of the DNN’s previously provided results to this type of architecture. We will add the corresponding figures (exploring a range of $\epsilon$) to our manuscript. 

We will additionally try to expand our results to the ImageNet dataset (which was a bit of struggle to manage for the rebuttal deadline given that each experiment is based on training 10 models of a specific initialization and evaluating the attacks). 

Table 1: Effect of Initialization on ResNet when subject to FGSM and PGD attacks using Cifar-10 Dataset.
| Initialization | Clean Accuracy | FGSM($\epsilon=0.03$) | FGSM($\epsilon=0.07$) | PGD($\epsilon=0.03$) | PGD($\epsilon=0.07$) |
|----------------|:--------------:|:---------------------:|:---------------------:|:--------------------:|:--------------------:|
| Orthogonal     | 91.6 $\pm$ 0.2 |     44.1 $\pm$ 0.6    |     39.2 $\pm$ 0.5    |    21.9 $\pm$ 0.6    |    11.8 $\pm$ 0.5    |
| Uniform        | 91.2 $\pm$ 0.3 |     46.8 $\pm$ 0.4    |     41.8 $\pm$ 0.3    |    24.3 $\pm$ 0.3    |    13.6 $\pm$ 0.4    |
| Kaiming        | 92.3 $\pm$ 0.1 |     42.3 $\pm$ 0.2    |     36.9 $\pm$ 0.2    |    20.7 $\pm$ 0.4    |    10.1 $\pm$ 0.3    |
| Xavier         | 92.1 $\pm$ 0.2 |     42.9 $\pm$ 0.3    |     37.6 $\pm$ 0.4    |    21.2 $\pm$ 0.5    |    10.6 $\pm$ 0.1    |

Meta-review: The paper explores the relationship of weight initialization and adversarial robustness. Many of the proposed insights extend beyond GNNs, which are the primary focus of this work. During the discussion period, the reviewers shared concerns over the experimental protocol and whether the attacks are on par with the more recent attacks used in the literature. I share those concerns and believe that the experimental protocol should be stronger. On the theoretical front, I encourage the authors to do a deeper search, since there are previous works that discuss the relationship between robustness and initialization, e.g., "Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)" (NeurIPS'22).
Decision: Accept (poster)