SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(ii) Spectral perspective. We solve for  $g_{\theta}(\Lambda)$  in the polynomial approximation  $Vg_{\theta}(\Lambda)V^{\top}H \approx \sum_{k=0}^{K}\gamma_{k}\mathring{L}^{k}H$  to obtain a possible graph filter  $g_{\theta}(\Lambda) = V^{\top}(\sum_{k=0}^{K}\gamma_{k}\mathring{L}^{k})V$ . Following Balcilar et al. [15], we study the spectral characteristics w.r.t.  $I - D^{-1/2}\mathbf{A}\mathbf{D}^{-1/2}$ . Then, the diagonal entries of  $g_{\theta}(\Lambda)$  correspond to the (relative) magnitude of how signals of frequency  $\lambda$  are present in the filtered signal. Vice versa, a low value corresponds to suppression of this frequency. Recall, low frequencies correspond to the small eigenvalues and high to the large eigenvalues. In Figure 7, we show how adversarial training and the permissible perturbations affect the spectra of the learned graph filters.
(iii) Spatial perspective. Robust diffusion (monomial basis) can be summarized as softmax  $(\pmb{T}\pmb{H})$  where  $\pmb{T} = \sum_{k=0}^{K} \gamma_k \mathring{\pmb{L}}^k$  is the total diffusion matrix. The coefficient  $T_{uv}$  indicates the diffusion strength between node  $u$  and node  $v$ . For example, we visualize the total diffusion matrix  $\pmb{T}$  in Figure 4 on Karate Club [16] with different training strategies. Depending on the learning signal we give, GPRGNN is able to adjust its diffusion.
4 LR-BCD: Adversarial Attack with Local Constraints
Motivation for local constraints. The just-discussed interpretability of robust diffusion provides empirical evidence for the importance of local constraints. Specifically, from Figure 4c, we see that GPRGNN adversarially trained without local constraints learns a diffusion that almost ignores the graph structure. While this model is certainly very robust w.r.t. structure perturbations, it is not a very useful GNN since it cannot leverage the structure information anymore. In contrast, we show in Figure 4d that GPRGNN trained adversarially with local constraints results in a robust model that can still incorporate the structure information.
The local predictions in node classification yield an alternative motivation. In the absence of local constraints, an adversary typically has the power to rewire the entire neighborhood of many nodes. When attacking GNNs, we empirically observe that a majority of successfully attacked nodes are perturbed beyond their degree even for moderate global attack budgets (see Figure 5). That such perturbations are not reasonable is evident in the fact that studies on local attacks [7, 17], where the adversary targets a single node's prediction, do not consider perturbations (far) beyond the node degree. However, a  $5\%$  attack budget on Cora-ML allows changing 798 edges while the majority of nodes have a degree less than three.
Figure 5: Number of successfully attacked nodes by node degree and number of connected adversarial edges. We attack self-trained GCN with PR-BCD and global budgets  $10\%$  (left) and  $25\%$  (right) on Cora-ML and aggregate the results over three different data splits. We observe that a notable amount of nodes is perturbed beyond their degree.
Traditionally, adversarial changes are judged by no
ticeability [18] since unnoticeable changes do not alter the semantics. However, for graphs, manual inspection of its content is often not feasible, and the concept of (un-)noticeability is unclear. However, using generative graph models Gosch et al. [6] revealed that perturbations beyond the node degree most often do alter the semantics. Perturbations that alter semantics can pose a problem for adversarial training. Similar observations have been done in graph contrastive learning where perturbations that do not preserve graph semantic preservation have a direct effect on the achievable error bounds [19].
Locally constrained Randomized Block Coordinate Descent (LR-BCD). We next introduce our LR-BCD attack; the first attack targeting multiple nodes at once while maintaining a local constraint for each node next to a global one. For this, we extend the so-called PR-BCD attack framework.
Projected Randomized Block Coordinate Descent (PR-BCD) [17] is a gradient-based attack framework applicable to a wide range of models. Its goals are to generate a perturbation matrix  $P \in \{0, 1\}^{n \times n}$  that is applied to the original adjacency matrix  $\mathbf{A}$  to construct a perturbed matrix  $\tilde{\mathbf{A}} = \mathbf{A} + (\mathbb{I} - 2\mathbf{A}) \odot P$ , where  $\mathbb{I}$  is an all-one matrix and  $\odot$  the element-wise product. For undirected graphs, only the upper-triangular parts of all matrices are considered. To construct  $P$  in an efficient manner, PR-BCD relaxes  $P$  during the attack from  $\{0, 1\}^{n \times n}$  to  $[0, 1]^{n \times n}$  and employs an iterative process for  $T$  iterations. It consists of three repeating steps. (1) A random block of size  $b$  is sampled. That is, only  $b$  (non-contiguous) elements in the perturbation matrix  $P_{t-1}$  are considered and all other elements set to zero. Thus,  $P_{t-1}$  is sparse. (2) A gradient update w.r.t. the loss to compute relaxed perturbations
$S_{t}$  is performed, i.e.,  $S_{t} \gets P_{t-1} + \alpha_{t-1} \nabla_{P_{t-1}} \ell(P_{t-1})$ , where  $P_{t-1}$  are the previous perturbations,  $\alpha_{t-1}$  is the learning rate, and  $\nabla_{P_{t-1}} \ell(P_{t-1})$  are the perturbation gradients through the GNN. Finally, and most crucially (3) a projection  $\Pi_{\mathcal{B}(\mathcal{G})}$  ensures that the perturbations are permissible given the set of allowed perturbations  $\mathcal{B}(\mathcal{G})$ , i.e.  $P_{t} = \Pi_{\mathcal{B}(\mathcal{G})}(S_{t})$ . Now, the process starts again with (1), but all zero-elements in the block  $b$  (or at least  $1/2$  of the lowest-value block elements) are resampled. After  $T$  iterations, the perturbations  $P_{T}$  are discretized from  $[0, 1]^{n \times n}$  to  $\{0, 1\}^{n \times n}$  to obtain the discrete  $\tilde{\pmb{A}}$ . Specifically,  $P_{T}$  is discretized via sampling, where the elements in  $P_{T}$  are used to define Bernoulli distributions.
Global projection. As mentioned above, the projection  $\Pi_{\mathcal{B}(\mathcal{G})}$  of PR-BCD is the crucial step that accounts for the attack budget. Geisler et al. [17] develop an efficient projection for an adversary implementing a global perturbation constraint, i.e., for  $\mathcal{B}(\mathcal{G}) = \{\tilde{\pmb{A}}\in \{0,1\}^{n\times n}\mid \| \tilde{\pmb{A}} -\pmb {A}\| _0\leq \Delta \}$  with  $\Delta \in \mathbb{N}_{\geq 0}$ . This is achieved by formulating the projection as the following optimization problem
$$
\Pi_ {\Delta} (\boldsymbol {S}) = \arg \min  _ {\boldsymbol {P}} \| \boldsymbol {P} - \boldsymbol {S} \| _ {F} ^ {2} \quad \text {s u b j e c t} \quad \sum_ {i, j} P _ {i, j} \leq \Delta \tag {5}
$$
$$
\boldsymbol {P} \in [ 0, 1 ] ^ {n \times n}
$$
with Frobenius norm  $\| \cdot \| _F^2$  and the sum aggregating all elements of the matrix.  $\Pi_{\Delta}(S)$  can be solved with the bisection method [17]. However, the projection  $\Pi_{\Delta}$  does not support limiting the number of perturbations per node, i.e.,  $\sum_{j = 1}^{n}P_{i,j}\leq \Delta_i^{(l)}$  where  $\pmb{\Delta}^{(l)}\in \mathbb{N}_{\geq 0}^{n}$  is the vector of local budgets for each node. Indeed, extending the optimization problem above to include local constraints leads to the notoriously hard problem of Euclidean projection to polyhedra [20].
A locally constrained global projection (LR-BCD). With the goal of an efficient attack including local constraints, we have to develop an alternative and scalable projection strategy for  $\mathcal{B}(\mathcal{G}) = \{\tilde{\boldsymbol{A}}\in$ $\{0,1\}^{n\times n}\mid \| \tilde{\boldsymbol{A}} -\boldsymbol {A}\| _0 <   \Delta \wedge \| \sum_j(\tilde{\boldsymbol{A}}_{ij} - \boldsymbol {A}_{ij})\| _0\leq \Delta_i^{(l)}\forall i\in [n]\}$  . Our novel projection  $P =$ $\Pi_{\Delta}^{(l)}(\pmb {S}) = \Pi_{[0,1]}(\pmb {S})\odot C^{*}$  chooses the largest entries from the perturbation matrix  $S\in \mathbb{R}^{n\times n}$  using  $C^* \in [0,1]^{n\times n}$  and clipping operator  $\Pi_{[0,1]}(\pmb {s})\in [0,1]$  , s.t. we obey global and local constraints. The optimal choices  $C^*$  can be calculated by solving a relaxed multi-dimensional knapsack problem:
$$
C ^ {*} = \underset {C} {\arg \max } \sum_ {i, j} S \odot C \quad \text {s u b j e c t} \quad \sum_ {i, j} P _ {i, j} ^ {\prime} \leq \Delta \tag {6}
$$
$$
\sum_ {j} P _ {i, j} ^ {\prime} \leq \Delta_ {i} ^ {(l)} \quad \forall i \in [ n ]
$$
$$
\boldsymbol {C} \in [ 0, 1 ] ^ {n \times n}
$$
where  $P' = \Pi_{[0,1]}(S) \odot C$ , i.e.,  $\Pi_{[0,1]}(S)$  represents the weights of the individual choices, while  $S$  captures their value.  $\sum_{i,j} P_{i,j}'$  aggregate all elements in the matrix. The local constraint  $\sum_{j} P_{i,j}' \leq \Delta_i^{(l)}$ , constrain the perturbations in each neighborhood, with node-specific budgets  $\Delta_i^{(l)}$ . Even though this optimization problem has no closed-form solution, it can be readily approximated with greedy approaches [21]. The key idea is to iterate the non-zero entries in  $S$  in descending order and construct the resulting  $C^*$  (or directly  $P$ ) as follows: For each perturbation  $(u,v)$  related to  $S_{uv}$ , we check if the global  $\Delta$  or local budgets  $\Delta_u^{(l)}$  and  $\Delta_v^{(l)}$  are not yet exhausted. Then,  $C_{uv}^* = 1$  (i.e.,  $P_{u,v} = \Pi_{[0,1]}(S_{uv})$ ). Otherwise,  $C_{uv}^* = \min\{\Delta, \Delta_u^{(l)}, \Delta_v^{(l)}\}$ . The final discretization is given by  $C^*$  corresponding to the solution of Equation (6) for  $S_T$ , but changing the weight of each choice to 1 (i.e.,  $P' = C$ ), guaranteeing a binary solution due to the budgets being integer. We give the pseudo-code and more details in Appendix B.4.
Our projection yields sparse solutions as it greedily selects the largest entries in  $S$  until the budget is exhausted. Moreover, it is efficient. Its time complexity is  $\mathcal{O}(b\log (b))$  due to the sorting of the up to  $b$  non-zero entries in the sparse matrix  $S$ . Its space complexity is  $\mathcal{O}(b)$ , assuming we choose the block size  $b \geq \Delta$  as well as  $b \geq n$ . Thus, LR-BCD has the same asymptotic complexities as the purely global projection by Geisler et al. [17] (which we will from now on denote as PR-BCD).
Summary. We will now summarize our LR-BCD along the four dimensions proposed by Biggio and Roli [22]. (1) Attacker's goal: Although our focus is to increase the misclassification for the target nodes in node classification, we can apply LR-BCD to different node- and graph-level tasks depending on the loss choice. (2) Attacker's knowledge: We assume perfect knowledge about the model and dataset, which is reasonable for adversarial training. (3) Attacker's capability: We propose
Table 1: Comparison on Citeseer of regularly trained models, the state-of-the-art SoftMedian GDC defense, and adversarially trained models (train  $\epsilon = 20\%$ ). The first line shows the robust accuracy [%] of a standard GCN and all other numbers represent the difference in robust accuracy achieved by various models in percentage points from this standard GCN. The best model is highlighted in bold and grey background marks our robust diffusion.
Model	Adv. trn.	A. eval. → A.trn.↓	Clean	LR-BCD ε=0.1	PR-BCD	LR-BCD ε=0.25	PR-BCD	Certifiable accuracy / sparse smoothing
Clean	3 add.	5 del.
GCN	X	-	72.0 ± 2.5	54.7 ± 2.8	51.7 ± 2.8	45.3 ± 3.4	38.0 ± 3.8	38.3 ± 11.5	1.7 ± 0.7	4.8 ± 1.5
GAT	X	-	-3.6 ± 2.7	-3.9 ± 3.4	+0.5 ± 3.5	-15.9 ± 5.3	-2.3 ± 7.3	-14.0 ± 12.0	-1.7 ± 0.7	-4.8 ± 1.5
APPNP	X	-	+0.2 ± 1.1	+1.7 ± 0.7	+1.9 ± 1.4	+3.0 ± 1.2	+2.2 ± 2.5	+8.9 ± 9.1	+31.2 ± 6.4	+31.6 ± 6.4
GPRGNN	X	-	+2.2 ± 4.3	+4.2 ± 2.7	+3.6 ± 4.9	+5.5 ± 3.9	+7.9 ± 4.6	+17.9 ± 6.9	+42.4 ± 4.4	+41.3 ± 3.7
ChebNetII	X	-	+1.1 ± 2.2	+5.8 ± 2.5	+5.0 ± 2.4	+10.4 ± 2.6	+7.6 ± 3.4	+24.6 ± 9.9	+55.6 ± 1.2	+54.0 ± 0.9
SoftMedian	X	-	+0.9 ± 1.7	+9.5 ± 2.2	+9.3 ± 1.9	+16.2 ± 2.4	+14.6 ± 2.9	+25.2 ± 10.5	+60.3 ± 1.4	+57.5 ± 0.8
GCN	✓	LR-BCD	-0.2 ± 1.2	+7.8 ± 1.6	+5.9 ± 1.5	+10.9 ± 2.1	+8.1 ± 2.3	-3.0 ± 9.4	+10.7 ± 4.6	+13.1 ± 5.1
PR-BCD	+0.0 ± 1.9	+6.9 ± 0.9	+5.3 ± 1.6	+8.6 ± 2.3	+5.8 ± 2.4	+4.2 ± 14.6	+10.1 ± 5.5	+11.4 ± 4.5
GAT	✓	LR-BCD	+0.8 ± 1.6	+5.9 ± 3.7	+9.0 ± 3.0	+8.4 ± 5.1	+13.1 ± 3.5	-2.0 ± 23.0	+1.4 ± 1.7	+4.0 ± 2.1
PR-BCD	+1.1 ± 2.2	+8.9 ± 2.8	+13.2 ± 3.7	+10.3 ± 2.3	+21.8 ± 4.8	-10.1 ± 13.7	+1.2 ± 2.0	+2.8 ± 1.2
APPNP	✓	LR-BCD	-0.8 ± 1.3	+7.6 ± 1.6	+5.6 ± 1.9	+11.1 ± 1.9	+8.3 ± 3.6	+21.7 ± 9.5	+41.3 ± 2.9	+44.1 ± 1.2
PR-BCD	-0.2 ± 2.2	+6.2 ± 2.3	+5.5 ± 3.1	+9.0 ± 2.9	+6.5 ± 3.8	+19.3 ± 7.2	+41.0 ± 3.8	+41.9 ± 3.0
GPRGNN	✓	LR-BCD	+1.7 ± 3.0	+15.7 ± 3.6	+13.6 ± 3.5	+24.8 ± 4.2	+22.9 ± 4.3	+34.0 ± 11.5	+67.4 ± 1.5	+65.0 ± 2.5
PR-BCD	+0.6 ± 3.6	+15.0 ± 3.6	+15.9 ± 4.2	+23.2 ± 4.3	+26.3 ± 5.4	+32.9 ± 11.8	+69.0 ± 1.5	+65.9 ± 2.3
ChebNetII	✓	LR-BCD	+3.7 ± 1.4	+11.5 ± 1.6	+11.5 ± 1.1	+16.4 ± 1.9	+16.0 ± 2.9	+31.5 ± 12.7	+62.3 ± 1.9	+59.8 ± 2.5
PR-BCD	+3.4 ± 1.7	+13.2 ± 2.2	+14.3 ± 1.2	+19.5 ± 1.6	+19.8 ± 2.3	+30.7 ± 13.2	+65.4 ± 2.6	+62.9 ± 3.7
to use a threat model that constrains the number of edge perturbations globally as well as locally for each node. (4) Attacker's strategy: LR-BCD extends PR-BCD by a projection incorporating local constraints. Most notably, we relax the unweighted graph during the attack to continuous values and leverage randomization for an efficient attack while optimizing over all possible  $n^2$  edges.
5 Empirical Results
Inductive learning. All results in this section follow a fully inductive semi-supervised setting. That is, the training graph  $\mathcal{G}$  does not include validation and test nodes. For evaluation, we then successively add the respective evaluation set. This way, we avoid the evaluation flaw of prior work since the model cannot achieve robustness via "memorizing" the predictions on the clean graph. We obtain inductive splits randomly, except for OGB arXiv [11], which comes with a split. In addition to the commonly sampled 20 nodes per class for both (labeled) training and validation, we sample a stratified test set consisting of  $10\%$  of all nodes. The remaining nodes are used as (unlabeled) training nodes.
Setup. We study the citation networks Cora, Cora-ML, CiteSeer [23], Pubmed [24], and OGB arXiv [11]. Furthermore, WikiCS [25, 26], which has significantly more heterogeneous neighborhoods as well the heterophilic dataset Squirrel [27] and (contextual) Stochastic Block Models (SBMs) [28] with a heterophilic parametrization (see Appendix B.1). As models we choose GPRGNN, ChebNetII, GCN, APPNP, and GAT [29]. Further, we compare to the state-of-the-art evasion defenses Soft Median GDC [17] in the main part and GRAND [30] in Table D.6. We apply adversarial training (see Section 2 and Appendix B.5) using both PR-BCD that only constraints perturbations globally and our  $LR-BCD$  that also allows for local constraints. Moreover, we use adversarial training in conjunction with self-training. Due to the inductive split, this does not bias results. We use the tanh margin attack loss of [17] and do not generate adversarial examples for the first 10 epochs (warm-up). We evaluate robust generalization on the test set using L/PR-BCD, which corresponds to adaptive attacks. Adaptive attacks are the gold standard in evaluating empirical robustness because they craft model-specific perturbations [31]. We use  $\epsilon$  to parametrize the global budget  $\Delta = \lfloor \epsilon \cdot \sum_{u\in \mathcal{A}}d_u / 2\rfloor$  relative to the degree  $d_{u}$  for the set of targeted nodes  $\mathcal{A}$ . We find that  $\Delta_u^{(l)} = \lfloor d_u / 2\rfloor$  is a reasonable local budget for all datasets but arXiv where we use  $\Delta_u^{(l)} = \lfloor d_u / 4\rfloor$ . We report averaged results with the standard error of the mean over three random splits. We use GTX 1080Ti (11 Gb) GPUs for all experiments but arXiv, for which we use a V100 (40 GB). For details see Appendix B. We discuss limitations in Appendix F and provide code at https://www.cs.cit.tum.de/daml/adversarial-training/.
Certifiable robustness. We use the model-agnostic randomized/sparse smoothing certificate of Bojchevski et al. [32] to also quantify certifiable robustness. Sparse smoothing creates a randomized ensemble given a base model  $f_{\theta}$  s.t. the majority prediction of the ensemble comes with guarantees. For the randomization, we follow Bojchevski et al. [32] and uniformly drop edges with  $p_{-} = 0.6$  as well as add edges with  $p_{+} = 0.01$ . Sparse smoothing then determines if a node-level prediction is
certified (does not change) for the desired deletion radius  $r_{-}$  or addition radius  $r_{+}$ . The guarantee holds in a probabilistic sense with significance level  $\alpha$ . We report the "certified accuracy"  $\gamma(r_{-}, r_{+})$  that is the average of correct and certifiable predictions over all nodes. We choose  $\alpha = 5\%$  and obtain 100,000 random samples. For simplicity, we report in the main part the certified accuracies  $\gamma(r_{-} = 0, r_{+} = 0)$ ,  $\gamma(5, 0)$ , and  $\gamma(0, 3)$ . See Appendix D.4 for more details and results.
Finding I: Adversarial training is an effective defense against structure perturbations. This is apparent from the results in Table 1, where we compare the empirical and certifiable robustness between the aforementioned models on CiteSeer. Our adversially trained robust diffusion models GPRGNN and ChebNetII outperform the other baselines both in empirical and certifiable robustness. This includes the state-of-the-art defense Soft Median, which we outperform with a comfortable margin. Thus, we close the gap in terms of the efficacy of adversarial training in the image domain vs. structure perturbations. Notably, the increased robustness does not imply a lower clean accuracy. For example, our LR-BCD adversarily trained GPRGNN achieves a 1.7 percentage points higher clean accuracy than a GCN, while with an adaptive LR-BCD attack and  $\epsilon = 25\%$  perturbed edges, we outperform a GCN by 24.8 percentage points. This amounts to a clean accuracy of  $73.7\%$  and a perturbed accuracy of  $70.1\%$ . We show the empirical robustness gains of adversarial training on on the other datasets in Table 2, compared to a GCN and regularly trained GPRGNN. We show that robust diffusion consistently and substantially improves the robustness – not only on homophilic datasets but also
Table 2: Robust accuracy  $[\% ]$  on further datasets. Attack types match for training  $(\epsilon = 0.2)$  and evaluation. Grey shading highlights our robust diffusion.
Model	Adv. trn.	Attack eval. & trn.	Clean	ε=0.1	ε=0.25
Cora ML	GCN	X	LR-BCD	82.5 ± 1.9	59.2 ± 2.8	48.5 ± 1.6
PR-BCD	82.5 ± 1.9	57.4 ± 2.3	38.0 ± 2.3
GPRGNN	X	LR-BCD	83.5 ± 2.6	64.8 ± 2.3	57.0 ± 1.5
PR-BCD	83.5 ± 2.6	61.9 ± 1.8	46.6 ± 1.4
✓	LR-BCD	83.3 ± 1.2	76.8 ± 1.1	74.4 ± 1.8
PR-BCD	82.5 ± 0.8	73.5 ± 1.5	69.0 ± 2.6
Cora	GCN	X	LR-BCD	79.9 ± 0.9	60.8 ± 0.2	47.9 ± 1.8
PR-BCD	79.9 ± 0.9	59.1 ± 0.4	44.1 ± 0.7
GPRGNN	X	LR-BCD	81.8 ± 1.0	69.4 ± 0.2	61.7 ± 0.6
PR-BCD	81.8 ± 1.0	64.3 ± 0.6	52.0 ± 0.6
✓	LR-BCD	82.4 ± 0.9	72.6 ± 1.1	70.0 ± 1.5
PR-BCD	79.9 ± 1.3	71.9 ± 1.0	66.3 ± 0.9
Pubmed	GCN	X	LR-BCD	77.4 ± 0.4	60.4 ± 0.7	52.8 ± 1.4
PR-BCD	77.4 ± 0.3	54.0 ± 0.5	39.3 ± 1.2
GPRGNN	X	LR-BCD	78.8 ± 1.0	61.1 ± 1.3	54.0 ± 0.9
PR-BCD	78.8 ± 1.0	57.0 ± 1.2	42.7 ± 1.4
✓	LR-BCD	80.5 ± 0.8	75.8 ± 1.0	74.7 ± 1.1
PR-BCD	80.4 ± 0.5	73.0 ± 1.1	68.8 ± 1.6
SBM (hetero)	GCN	X	LR-BCD	62.0 ± 4.5	51.5 ± 4.3	45.1 ± 4.5
PR-BCD	62.0 ± 4.5	49.2 ± 3.4	47.1 ± 4.8
GPRGNN	X	LR-BCD	86.5 ± 2.1	67.3 ± 3.3	59.6 ± 3.3
PR-BCD	85.9 ± 4.9	68.4 ± 2.9	55.6 ± 4.4
✓	LR-BCD	85.5 ± 0.5	72.1 ± 1.7	67.0 ± 3.3
PR-BCD	85.5 ± 2.9	69.0 ± 0.5	60.9 ± 2.9
Squirrel	GCN	X	LR-BCD	42.0 ± 0.3	20.1 ± 1.3	15.8 ± 1.4
PR-BCD	41.9 ± 0.3	12.1 ± 0.3	6.9 ± 0.6
GPRGNN	X	LR-BCD	40.0 ± 0.6	29.8 ± 4.8	28.4 ± 5.1
PR-BCD	40.4 ± 0.9	20.6 ± 0.6	15.3 ± 1.4
✓	LR-BCD	37.8 ± 1.4	34.1 ± 3.4	33.8 ± 3.7
PR-BCD	38.4 ± 1.6	32.3 ± 4.7	28.9 ± 9.1
WikiCS	GCN	X	LR-BCD	74.6 ± 2.8	42.5 ± 2.3	35.4 ± 2.6
PR-BCD	75.1 ± 1.5	38.6 ± 2.9	30.2 ± 2.5
GPRGNN	X	LR-BCD	72.8 ± 1.1	52.8 ± 3.2	49.7 ± 4.9
PR-BCD	72.8 ± 0.4	52.7 ± 1.0	50.0 ± 1.2
✓	LR-BCD	73.3 ± 2.7	64.4 ± 2.2	62.8 ± 2.1
PR-BCD	73.2 ± 0.2	60.0 ± 1.9	54.1 ± 0.8
under heterophily (SBM, Squirrel). We present more results and an ablation study in Appendix D.
Finding II: Choose the set of permissible perturbations wisely! As argued already in Section 4, the right set of admissible permutations can guide the learned robust diffusion. Importantly, the set of admissible perturbations is application specific. That is, depending on the choice, the resulting model might have different robustness characteristics w.r.t. different threat models. For example, in Table 1, a LR-BCD adversarially trained GPRGNN is more robust to LR-BCD attacks than to a model trained with PR-BCD, and vice versa. Importantly, the learned message passing is very different as the spectral analysis (Figure 7) reveals a striking pattern. Adversarial training with PR-BCD flipped the behavior from low-pass to high-pass if compared to a regularly trained model. However, adversarial training with LR-BCD seems to preserve the general characteristic while using a larger fraction of the spectrum to enhance robustness. Note that such a filter (less integral Lipschitz) opposes the theoretical stability result of Gama et al. [33]. Moreover, the polynomial coefficients (Figure 6) resulting from adversarial training without local constraints (Figure 6c) are very similar to coefficients that Chien et al. [13] associated with a non-informative graph structure. See Appendix D.2 for other datasets.
Finding III: Adversarial training with local constraints is efficient and scalable. We show in Figure 8 the results of an adversially trained GPRGNN using LR-BCD with different relative budgets  $\epsilon$ . Adversarial training on arXiv requires less than  $20~\mathrm{GB}$  of RAM and an epoch takes approx. 10 seconds. In contrast, the globally constrained adversarial training of Xu et al. [2] would require multiple terabytes of RAM due to  $n^2$  possible edge perturbations [17]. We not only show that robust diffusion with LR-BCD is scalable, but also that it improves the robustness on arXiv. That is, twice as many edge perturbations are required to push the accuracy below that of an MLP, if comparing  $\epsilon = 10\%$  to standard training.
Figure 7: Respective spectral filters of Figure 6. Low filter magnitude denotes suppression of the respective frequencies associated with eigenvalues  $\lambda$ .
(a) Regul. (b) w/ loc. (c) w/o 1.
Figure 6: Learned coefficients  $\gamma$  for GPRGNN on Citeseer. We use  $\epsilon = 20\%$  in (b) and (c) for adv.trn.
Figure 8: Acc. on arXiv under adv. attack for GPRGNN with different training pert. strengths  $\epsilon$  using LR-BCD.
We next relate to prior work and refer to Appendix E for more detail: (1) Learning setting. Conceptual problems by remembering training data are not unique to adversarial training and have been mentioned in Zheng et al. [40] for graph injection attacks and by Scholten et al.