SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given a perturbation set  $\mathbb{B}$ , interval bound propagation proves the robustness of a prediction  $y_{n} = \arg \max_{k} H_{n,k}^{(L)} = \arg \max_{k} Z_{n,k}^{(L-1)}$  by computing elementwise lower and upper bounds  $\pmb{R}^{(l)}$  and  $\pmb{S}^{(l)}$  on the pre-activation values  $\pmb{Z}^{(l)}$  and elementwise lower and upper bounds  $\hat{\pmb{R}}^{(l)}$  and  $\hat{\pmb{S}}^{(l)}$  on the post-activation values  $\pmb{H}^{(l)}$  via interval arithmetic. If  $\forall k \neq y_{n}: R_{n,y_{n}}^{(L-1)} > S_{n,k}^{(L-1)}$ , then the prediction  $y_{n}$  for node  $n$  is provably robust.
In our case, the perturbation set is
$$
\mathbb {B} = \left\{\left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \mid \boldsymbol {X} \in \{0, 1 \} ^ {N \times D}, \boldsymbol {A} \in \{0, 1 \} ^ {N \times N}, \right.
$$
$$
\boldsymbol {A} ^ {\prime} = \boldsymbol {A}, \tag {8}
$$
$$
c _ {\boldsymbol {X}} ^ {+} \cdot | | \left(\boldsymbol {X} ^ {\prime} - \boldsymbol {X}\right) _ {+} | | _ {0} + c _ {\boldsymbol {X}} ^ {-} \cdot | | \left(\boldsymbol {X} ^ {\prime} - \boldsymbol {X}\right) _ {-} | | _ {0} \leq \epsilon , \tag {8}
$$
$$
\forall n: c _ {\boldsymbol {X}} ^ {+} \cdot | | (\boldsymbol {X} _ {n} ^ {\prime} - \boldsymbol {X} _ {n}) _ {+} | | _ {0} + c _ {\boldsymbol {X}} ^ {-} \cdot | | (\boldsymbol {X} _ {n} ^ {\prime} - \boldsymbol {X} _ {n}) _ {-} | | _ {0} \leq \rho_ {n} \},
$$
with global budget  $\epsilon$  and local budgets  $\rho_{n}$ .
We propose to compute the lower and upper bounds  $\pmb{R}^{(0)}$  and  $\pmb{S}^{(0)}$  for the first pre-activation values by solving a knapsack problem with local constraints (see Definition 2) via the methods discussed in Appendix F.1. We define the cost matrix  $C \in \mathbb{R}_+^{N \times D}$  to be
$$
C _ {n, d} = \left\{ \begin{array}{l l} c _ {\boldsymbol {X}} ^ {+} & \text {i f X _ {n , d} = 0} \\ c _ {\boldsymbol {X}} ^ {-} & \text {i f X _ {n , d} = 1}. \end{array} \right.
$$
To obtain an upper bound  $S_{i,j}^{(0)}$ , we define the value matrix  $\mathbf{V}\in \mathbb{R}_{+}^{N\times D}$  to be
$$
V _ {n, d} = \left\{ \begin{array}{l l} \tilde {A} _ {i, n} \cdot \max (W _ {d, j} ^ {(0)}, 0) & \mathrm {i f} X _ {n, d} = 0 \\ \tilde {A} _ {i, n} \cdot \max (- W _ {d, j} ^ {(0)}, 0) & \mathrm {i f} X _ {n, d} = 1. \end{array} \right.
$$
That is, setting an attribute  $X_{n,d}$  from 0 to 1 makes a positive contribution to upper bound  $S_{i,j}^{(0)}$  if the corresponding entry of the weight matrix is positive and there is an edge between nodes  $n$  and  $i$ . Setting an attribute  $X_{n,d}$  from 1 to 0 makes a positive contribution, if the corresponding entry of the weight matrix is negative and there is an edge between nodes  $n$  and  $i$ . We then solve the knapsack problem and add its optimal value to the original, unperturbed pre-activation value  $Z_{i,j}^{(0)}$ .
To obtain a lower bound  $R_{i,j}^{(0)}$ , we analogously define the value matrix  $V \in \mathbb{R}_+^{N \times D}$  to be
$$
V _ {n, d} = \left\{ \begin{array}{l l} \tilde {A} _ {i, n} \cdot \max (- W _ {d, j} ^ {(0)}, 0) & \text {i f X _ {n , d} = 0} \\ \tilde {A} _ {i, n} \cdot \max (W _ {d, j} ^ {(0)}, 0) & \text {i f X _ {n , d} = 1}, \end{array} \right.
$$
multiply the optimal value of the knapsack problem with  $-1$  and then add it to the original, unperturbed pre-activation value  $Z_{i,j}^{(0)}$ .
After we have obtained the lower and upper bounds for the first layer's pre-activation values, we can use the same procedure as Liu et al. [12] to bound the subsequent post- and pre-activation values:
$$
\hat {\boldsymbol {R}} ^ {(l)} = \operatorname {R e L U} (\boldsymbol {R} ^ {(l - 1)}) \qquad \qquad \text {f o r} l = 1, \dots , L - 1,
$$
$$
\hat {\boldsymbol {S}} ^ {(l)} = \operatorname {R e L U} (\boldsymbol {S} ^ {(l - 1)}) s \quad \text {f o r} l = 1, \dots , L - 1,
$$
$$
\boldsymbol {R} ^ {(l)} = (\tilde {\boldsymbol {A}} \hat {\boldsymbol {R}} ^ {(l)}) \boldsymbol {W} _ {+} ^ {(l)} + (\tilde {\boldsymbol {A}} \hat {\boldsymbol {S}} ^ {(l)}) \boldsymbol {W} _ {-} ^ {(l)} + \mathbf {1} (\boldsymbol {b} ^ {(l)}) ^ {T} \qquad \text {f o r} l = 1, \dots , L - 1,
$$
$$
\boldsymbol {S} ^ {(l)} = (\tilde {\boldsymbol {A}} \hat {\boldsymbol {R}} ^ {(l)}) \boldsymbol {W} _ {-} ^ {(l)} + (\tilde {\boldsymbol {A}} \hat {\boldsymbol {S}} ^ {(l)}) \boldsymbol {W} _ {+} ^ {(l)} + \mathbf {1} (\boldsymbol {b} ^ {(l)}) ^ {T} \qquad \text {f o r} l = 1, \dots , L - 1.
$$
with  $W_{+} = \max(\mathbf{0}, \mathbf{W})$  and  $W_{-} = \min(\mathbf{0}, \mathbf{W})$ . If  $\forall k \neq y_{n}: R_{n, y_{n}}^{(L - 1)} > S_{n, k}^{(L - 1)}$ , then the prediction  $y_{n}$  for node  $n$  is provably robust. In our experiments, we use linear programming and not the exact dynamic programming solver for the knapsack problems (see Appendix F.1).
F.3 Convex outer adversarial polytopes for attribute perturbations
Zügner and Gümnmann [10] also propose robustness guarantees for  $L$ -layer graph convolutional networks (see Eq. (7) under attribute perturbations with uniform cost ( $c_{\mathbf{X}}^{+} = c_{\mathbf{X}}^{-} = 1$ ,  $c_{\mathbf{A}}^{+} = c_{\mathbf{A}}^{-} = \infty$ ). However, their method solves a convex optimization problem and yields robustness guarantees that are always at least as strong as interval bound propagation. Again, we want to generalize their guarantees to arbitrary attribute perturbation costs  $c_{\mathbf{X}}^{+}, c_{\mathbf{X}}^{-} \in \mathbb{R}_{+}$ .
Let  $\mathbf{Z}^{(L-1)}(\mathbf{X},\mathbf{A})$  be the pre-activation values in the last layer, given attribute matrix  $\mathbf{X} \in \{0,1\}^{N\times D}$  and adjacency matrix  $\mathbf{A} \in \{0,1\}^{N\times N}$ .
A prediction  $y_{n} = \arg \max_{k}\pmb{Z}^{(L - 1)}(\pmb {X},\pmb {A})_{n,k}$  is robust to all perturbed  $\pmb{X}^{\prime},\pmb{A}^{\prime}$  from a perturbation set  $\mathbb{B}$  if  $\forall k\neq y_{n},\forall (\pmb{X}^{\prime},\pmb{A}^{\prime})\in \mathbb{B}:Z^{(L - 1)}(\pmb{X}^{\prime},\pmb{A}^{\prime})_{n,y_{n}} > Z^{(L - 1)}(\pmb{X}^{\prime},\pmb{A}^{\prime})_{n,k}$ . This is equivalent to showing that, for all  $k\neq y_{n}$ :
$$
\min  _ {\left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \in \mathbb {B}} \boldsymbol {Z} ^ {(L - 1)} \left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) _ {n, y _ {n}} - \boldsymbol {Z} ^ {(L - 1)} \left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) _ {n, k} > 0. \tag {9}
$$
Solving these  $k$  optimization problems is generally intractable due to the ReLU nonlinearities and the discrete set of perturbed graphs  $\mathbb{B}$ . Zügner and Gunnemann [10] thus propose to make two relaxations. The first relaxation is to relax the integrality constraints on the perturbation set  $\mathbb{B}$ . In our case, this leads to the relaxed set
$$
\mathbb {B} = \left\{\left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \mid \boldsymbol {X} \in [ 0, 1 ] ^ {N \times D}, \boldsymbol {A} \in [ 0, 1 ] ^ {N \times N}, \right.
$$
$$
\boldsymbol {A} ^ {\prime} = \boldsymbol {A}, \tag {10}
$$
$$
c _ {\boldsymbol {X}} ^ {+} \cdot | | (\boldsymbol {X} ^ {\prime} - \boldsymbol {X}) _ {+} | | _ {1} + c _ {\boldsymbol {X}} ^ {-} \cdot | | (\boldsymbol {X} ^ {\prime} - \boldsymbol {X}) _ {-} | | _ {1} \leq \epsilon ,
$$
$$
\forall n: c _ {\pmb {X}} ^ {+} \cdot | | (\pmb {X} _ {n} ^ {\prime} - \pmb {X} _ {n}) _ {+} | | _ {1} + c _ {\pmb {X}} ^ {-} \cdot | | (\pmb {X} _ {n} ^ {\prime} - \pmb {X} _ {n}) _ {-} | | _ {1} \leq \rho_ {n} \}.
$$
Note that we now use  $\ell_1$  instead of  $\ell_0$  norms.
The second relaxation is to relax the strict functional dependency between pre- and post-activation values  $\mathbf{H}^{(l)} = \mathrm{ReLU}(\mathbf{Z}^{(l)})$  to its convex hull:
$$
\left(Z _ {n, d} ^ {(l)}, H _ {n, d} ^ {(l)}\right) \in \operatorname {h u l l} \left(\left\{a, \operatorname {R e L U} (a) \in \mathbb {R} \mid R _ {n, d} ^ {(l)} \leq a \leq S _ {n, d} ^ {(l)} \right\}\right),
$$
where  $\pmb{R}^{(l)}$  and  $S^{(l)}$  are elementwise lower and upper bounds on the pre-activation values  $Z^{(l)}$ . We propose to use the modified interval bound propagation method introduced in Appendix F.2 to obtain these bounds.
These two relaxations leave us with an optimization problem that is almost identical to the one discussed in [10]. We just weight insertions and deletions with constants  $c_{\mathbf{X}}^{+}$  and  $c_{\mathbf{X}}^{+}$  instead of uniform weight 1 and have different values for the elementwise pre-activation bounds. We can thus go through the same derivations as in the proof of Theorem 4.3 in [10] while tracking the constants  $c_{\mathbf{X}}^{+}$  and  $c_{\mathbf{X}}^{+}$  to derive the dual of our relaxed optimization problem, which lower-bounds Eq. (9). We highlight the difference to the result for uniform costs in red. The dual problem for local budgets  $\rho$
and global budget  $\epsilon$  is
$$
\max  _ {\boldsymbol {\Omega} ^ {(1)}, \dots , \boldsymbol {\Omega} ^ {(L - 1)}, \boldsymbol {\kappa}, \lambda} g _ {\boldsymbol {\rho}, \epsilon} (\boldsymbol {X}, \boldsymbol {\Omega}, \boldsymbol {\kappa}, \lambda)
$$
s.t.  $\pmb{\Omega}^{(l)}\in [0,1]^{N\times h^{(l)}}$  for  $l = L - 1,\ldots ,1$  (11)
$$
\boldsymbol {\kappa} \in \mathbb {R} _ {+} ^ {N}, \lambda \in \mathbb {R} _ {+},
$$
where  $h^{(l)}$  is the number of neurons in layer  $l$  and
$$
\begin{array}{l} g _ {\boldsymbol {\rho}, \epsilon} (\boldsymbol {X}, \boldsymbol {\Omega}, \kappa , \lambda) = \sum_ {l = 1} ^ {L - 1} \sum_ {(n, j) \in \mathcal {I} ^ {(l)}} \frac {S _ {n , j} ^ {(l)} R _ {n , j} ^ {(l)}}{S _ {n , j} ^ {(l)} - R _ {n , j} ^ {(l)}} \left[ \hat {\Phi} _ {n, j} ^ {(l)} \right] _ {+} - \sum_ {l = 0} ^ {L - 1} \mathbf {1} ^ {T} \boldsymbol {\Phi} ^ {(l + 1)} \boldsymbol {b} ^ {(l)} \\ - \operatorname {T r} \left[ \boldsymbol {X} ^ {T} \hat {\boldsymbol {\Phi}} ^ {(0)} \right] - | | \boldsymbol {\Psi} | | _ {1} - \sum_ {n = 1} ^ {N} \rho_ {n} \cdot \kappa_ {n} - \epsilon \cdot \lambda , \\ \end{array}
$$
where  $\mathcal{I}^{(l)}$  is the set of unstable neurons in layer  $l$ , i.e.  $\mathcal{I}^{(l)} = \{(n,j) \mid R_{n,j}^{(l)} \leq 0 < S_{n,j}^{(l)}\}$ , and
$$
\Phi_ {j} ^ {(L)} = \left\{ \begin{array}{l l} - 1 & \text {i f} j = y ^ {*} \\ 1 & \text {i f} j = k \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
$\hat{\Phi}^{(l)} = \tilde{A}^T\Phi^{(l + 1)}(W^{(l)})^T$  for  $l = L - 1,\ldots ,0$
$$
\Phi_ {n, j} ^ {(l)} = \left\{ \begin{array}{l l} 0 & \text {i f} S _ {n, j} ^ {(l)} \leq 0 \\ \hat {\Phi} _ {n, j} & \text {i f} R _ {n, j} ^ {(l)} > 0 \\ \frac {S _ {n , j} ^ {(l)}}{S _ {n , j} ^ {(l)} - R _ {n , j} ^ {(l)}} \left[ \hat {\Phi} _ {n, j} ^ {(l)} \right] _ {+} + \Omega_ {n, j} ^ {(l)} \left[ \hat {\Phi} _ {n, j} ^ {(l)} \right] _ {-} & \text {i f} (n, j) \in \mathcal {I} ^ {(l)} \end{array} \right. \quad \text {f o r} l = L - 1, \ldots , 1
$$
$$
\Psi_ {n, d} = \left\{ \begin{array}{l l} \max  \left(\Delta_ {n, d} - c _ {\boldsymbol {X}} ^ {+} \cdot (\kappa_ {n} + \lambda), 0\right) & \text {i f} X _ {n, d} = 0 \\ \max  \left(\Delta_ {n, d} - c _ {\boldsymbol {X}} ^ {-} \cdot (\kappa_ {n} + \lambda), 0\right) & \text {i f} X _ {n, d} = 1 \end{array} \right.
$$
$$
\Delta_ {n, d} = \left[ \hat {\Phi} ^ {(0)} \right] _ {+} \cdot (1 - X _ {n, d}) - \left[ \hat {\Phi} ^ {(0)} \right] _ {-} \cdot X _ {n, d}.
$$
The indexing changes are necessary because we consider  $L$  and not  $(L - 1)$ -layer networks. The sign changes are necessary because we define the clipping of negative values differently, i.e.  $X_{-} = \min(X,0)$  and not  $X_{-} = -\min(X,0)$ . Aside from that, the only difference are the costs  $c_{X}^{+}$  and  $c_{X}^{-}$  in the definition of  $\Psi$ .
Because this is a dual problem, any choice of  $\Omega$ ,  $\kappa$  and  $\lambda$  yields a lower bound on Eq. (9). But using optimal parameters yields a tighter bound.
Proposition 13. Define value matrix  $V = \Delta$  and cost matrix  $C = c_{X}^{+}\cdot [1 - X]_{+} + c_{X}^{-}\cdot [X]_{+}$ . Then,
- The optimal value of  $\lambda$  is  $\lambda^{*} = \min_{n,m} V_{n,m} / C_{n,m}$  s.t.  $Q_{n,m}^{*} > 0$ , with  $Q^{*}$  computed as in Algorithm 3. That is,  $\lambda^{*}$  is the smallest value-to-cost ratio for which some global budget is allocated.
- Define  $o_n = \min_m V_{n,m} / C_{n,m}$  s.t.  $L_{n,m} > 0$  with  $\mathbf{L}$  computed as in Algorithm 3. That is,  $o_n$  is the smallest value-to-cost ratio in row  $n$  for which some local budget is allocated. The optimal value of  $\kappa_n$  is  $\kappa_n^* = \max(0, o_n - \lambda^*)$ .
Proof. For any fixed  $\Omega^{(1)},\ldots ,\Omega^{(L - 1)}$ , we can go through the same derivations as in the proofs of Theorems 4.3 and 4.4 in [10] – while keeping track of constants  $c_{\pmb{X}}^{+},c_{\pmb{X}}^{-}$  to show that the above optimization problem is up to an additive constant equivalent to
$$
\begin{array}{l} \min  _ {\lambda \in \mathbb {R} _ {+}, \boldsymbol {\kappa} \in \mathbb {R} _ {+} ^ {N}, \boldsymbol {U} \in \mathbb {R} _ {+} ^ {N \times D}} \sum_ {n, d} U _ {n, d} + \sum_ {n = 1} ^ {N} \rho_ {n} \cdot \kappa_ {n} + \epsilon \cdot \lambda \\ \mathrm {s . t .} U _ {n, d} \geq V _ {n, d} - C _ {n, d} \cdot \kappa_ {n} - C _ {n, d} \cdot \lambda \\ \end{array}
$$
with  $V = \Delta$  and  $C = c_{\pmb{X}}^{+}\cdot [1 - \pmb {X}]_{+} + c_{\pmb{X}}^{-}\cdot [\pmb {X}]_{+}$
By standard construction, the dual of the above linear program is
$$
\begin{array}{l} \max  _ {Q \in \mathbb {R} _ {N \times M}} \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M} V _ {n, m} \cdot Q _ {n, m} \\ \text {s . t .} \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M} C _ {n, m} \cdot Q _ {n, m} \leq \epsilon , \quad \forall n: \sum_ {m = 1} ^ {M} C _ {n, m} \cdot Q _ {n, m} \leq \rho_ {n}, \\ \forall n, m: 0 \leq Q _ {n, m}, \quad \forall n, m: Q _ {n, m} \leq 1. \\ \end{array}
$$
This is exactly the linearly relaxed knapsack problem with local constraints from Eq. (6). During our proof of Proposition 12 via Karush-Kuhn-Tucket conditions, we have already shown that  $\lambda^{*}$  and  $\kappa^{*}$  are the optimal values of the dual (here: primal) variables.
Since we have optimal values for variables  $\lambda$  and  $\kappa$  of the optimization problem in Eq. (11), we only need to choose values for  $\Omega^{(1)},\ldots ,\Omega^{(L - 1)}$ . They can be either be optimized via gradient ascent or set to some constant value. In our experiments, we choose
$$
\Omega_ {n, j} ^ {(l)} = \frac {S _ {n , j} ^ {(l)}}{S _ {n , j} ^ {(l)} - S _ {n , j} ^ {(l)}},
$$
as suggested in [10]. As discussed in [10], the efficiency of the certification procedure can be further improved by slicing the attribute and adjacency matrix to only contain nodes that influence the hidden representation of node  $n$  in each layer  $l$ .
F.4 Bilinear programming for adjacency perturbations
Zügner and Gümnmann [13] derive guarantees for the robustness of graph convolutional networks to edge deletions ( $c_{\mathbf{X}}^{+} = c_{\mathbf{X}}^{-} = c_{\mathbf{A}}^{+} = \infty, c_{\mathbf{A}}^{-} = 1$ ). Generalizing these guarantees to arbitrary costs  $c_{\mathbf{A}}^{-} \in \mathbb{R}_{+}$  does not require any additional derivations. Multiplying the cost by a factor  $k$  is equivalent to dividing global budget  $\epsilon$  and local budgets  $\rho_1, \ldots, \rho_n$  by factor  $k$ .
F.5 Linearization and dualization for adjacency perturbations
Jin et al. [15] derive robustness guarantees for graph classifiers consisting of a 1-layer graph convolutional network followed by a linear layer and mean pooling under adjacency perturbations with uniform cost ( $c_{\mathbf{X}}^{+} = c_{\mathbf{X}}^{-} = \infty, c_{\mathbf{A}}^{+} = c_{\mathbf{A}}^{-} = 1$ ). In the following, we want to generalize their guarantees to arbitrary  $c_{\mathbf{A}}^{+}$  and  $c_{\mathbf{A}}^{-}$ .
In fact, the authors propose two different approaches. The first one involves linearization of the neural network and the formulation of a Lagrange dual problem to enforce symmetry of the perturbed adjacency matrix. The second one combines Fenchel biconjugation with convex outer adversarial polytopes to derive a lower bound that is then optimized via conjugate gradients. While the second objective is sound, its optimization via conjugate gradient may be numerically unstable, which is why an entirely different solver is used in practice.<sup>6</sup> In order to not mispresent the contents of [15], we focus on the first approach, which offers similarly strong guarantees (see Fig.4 in [15]).
The considered graph classification architecture is
$$
F (\boldsymbol {X}, \boldsymbol {A}) = \sum_ {n = 1} ^ {N} \operatorname {R e L U} \left(\boldsymbol {D} ^ {- 1} \tilde {\boldsymbol {A}} \boldsymbol {X} \boldsymbol {W}\right) \boldsymbol {U} / N,
$$
where  $\tilde{\boldsymbol{A}} = \boldsymbol{A}$  is the adjacency matrix after introducing self-loops, i.e. setting all diagonal entries to  $1$ ,  $\pmb{D} \in \mathbb{R}^{N \times N}$  with  $\pmb{D}_{i,i} = \pmb{1}^T \tilde{\pmb{A}}_i$  is the diagonal degree matrix of  $\tilde{\boldsymbol{A}}$ ,  $\pmb{W} \in \mathbb{R}^{D \times H}$  are the graph convolution weights and  $\pmb{U} \in \mathbb{R}^{H \times K}$  are the linear layer weights for  $K$  classes.
See https://github.com/RobustGraph/RoboGraph/blob/master/robograph/attack/cvx_envSolver.py
As before, robustness for a specific prediction  $y = \arg \max_k F(\pmb {X},\pmb {A})_k$  under a perturbation set  $\mathbb{B}$  can be proven by showing that the classification margin is positive, i.e.
$$
\forall k \neq y: \min  _ {(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) \in \mathbb {B}} F (\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) _ {y} - F (\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) _ {k} > 0.
$$
For this specific architecture, this is equivalent to showing that, for all  $k \neq y$
$$
\min  _ {(\boldsymbol {X}, \boldsymbol {A}) \in \mathbb {B}} \sum_ {n = 1} ^ {N} f _ {n, k} (\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) > 0
$$
$$
\text {w i t h} f _ {n, k} \left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) = \left(\mathbf {1} ^ {T} \tilde {\boldsymbol {A}} _ {n} ^ {\prime}\right) ^ {- 1} \operatorname {R e L U} \left(\left(\tilde {\boldsymbol {A}} _ {n} ^ {\prime}\right) ^ {T} \boldsymbol {X} ^ {\prime} \boldsymbol {W}\right) \left(\boldsymbol {U} _ {:, y} - \boldsymbol {U} _ {:, k}\right) / N.
$$
Note that we could move the degree of node  $n$  out of the nonlinearity because it is only a scalar factor. Our perturbation set is
$$
\begin{array}{l} \mathbb {B} = \left\{\left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \mid \boldsymbol {X} \in \{0, 1 \} ^ {N \times D}, \boldsymbol {A} \in \{0, 1 \} ^ {N \times N}, \right. \\ X ^ {\prime} = X, \\ c _ {\boldsymbol {A}} ^ {+} \cdot | | (\boldsymbol {X} ^ {\prime} - \boldsymbol {X}) _ {+} | | _ {0} + c _ {\boldsymbol {A}} ^ {-} \cdot | | (\boldsymbol {X} ^ {\prime} - \boldsymbol {X}) _ {-} | | _ {0} \leq \epsilon , \tag {12} \\ \forall n: c _ {\pmb {A}} ^ {+} \cdot | | (\pmb {X} _ {n} ^ {\prime} - \pmb {X} _ {n}) _ {+} | | _ {0} + c _ {\pmb {A}} ^ {-} \cdot | | (\pmb {X} _ {n} ^ {\prime} - \pmb {X} _ {n}) _ {-} | | _ {0} \leq \rho_ {n} \}. \\ \end{array}
$$
**Linearization.** Unless one wants to find the worst-case perturbation for each  $f_{n,k}$  via brute-forcing (which is a viable approach for very small budgets and discussed in [15]), the first step of the certification procedure is to find  $N$  linear models that lower-bound the nodewise functions  $f_{n,k}$  for all possible perturbed inputs. That is, we want to find vectors  $\pmb{q}^{(1)}, \dots, \pmb{q}^{(N)} \in \mathbb{R}^N$  and scalars  $b^{(1)}, \dots, b^{(N)} \in \mathbb{R}^K$  such that
$$
\forall_ {\left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \in \mathbb {B}}: f _ {n, k} \left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \geq \underline {{f}} _ {n, k} \left(\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}\right) \tag {13}
$$
$$
\text {w i t h} \underline {{f _ {n , k}}} (\boldsymbol {X} ^ {\prime}, \boldsymbol {A} ^ {\prime}) = \left(\mathbf {1} ^ {T} \tilde {\boldsymbol {A}} _ {n} ^ {\prime}\right) ^ {- 1} \left((\tilde {\boldsymbol {A}} _ {n} ^ {\prime}) ^ {T} \boldsymbol {q} ^ {(n)} + b ^ {(n)}\right). \tag {14}
$$
Like in [15], we use the linear bounds from CROWN [133]. Computing these bounds requires elementwise lower and upper bounds  $\boldsymbol{r}^{(1)}, \ldots, \boldsymbol{r}^{(N)}$  and  $\boldsymbol{s}^{(1)}, \ldots, \boldsymbol{s}^{(N)}$  on the pre-activation values  $\boldsymbol{z}^{(n)} = ((\tilde{\boldsymbol{A}}_{n}')^T \boldsymbol{X}' \boldsymbol{W})$  with  $\boldsymbol{z}^{(n)} \in \mathbb{R}^H$ .
For our new perturbation set, we propose to compute these elementwise lower and upper bounds by solving knapsack problems with local constraints via the algorithms discussed in Appendix F.1. To bound the pre-activation values  $\pmb{z}^{(n)}$  of the  $n$ th node, we define the cost matrix  $\pmb{C} \in \mathbb{R}_{+}^{1 \times N}$  as
$$
C _ {1, m} = \left\{ \begin{array}{l l} \infty & \text {i f} m = n \\ c _ {\boldsymbol {A}} ^ {+} & \text {i f} m \neq n \wedge A _ {n, m} = 0 \\ c _ {\boldsymbol {A}} ^ {-} & \text {i f} m \neq n \wedge A _ {n, m} = 1. \end{array} \right.
$$
The infinite cost for diagonal elements ensures that they are not adversarially perturbed. Afterall, there is no benefit to attacking elements that are anyway overwritten by self-loops. The cost matrix has shape  $1 \times N$ , because in every row  $n$ , there are only  $N$  edges that can be perturbed.
To obtain an upper bound  $s_h^{(n)}$ , we define the value matrix  $\mathbf{V} \in \mathbb{R}_+^{1 \times N}$  to be
$$
V _ {1, m} = \left\{ \begin{array}{l l} \max  \left(\boldsymbol {X} _ {m} ^ {T} \boldsymbol {W} _ {:, h}, 0\right) & \text {i f} A _ {n, m} = 0 \\ \max  \left(- \boldsymbol {X} _ {m} ^ {T} \boldsymbol {W} _ {:, h}, 0\right) & \text {i f} A _ {n, m} = 1. \end{array} \right.
$$
We then solve the knapsack problem, which tells us how much the pre-activation value can change under  $\mathbb{B}$ , and then add this optimal value to the original, unperturbed pre-activation value  $z_h^{(n)}$ .
To obtain a lower bound  $r_h^{(N)}$ , we analogously define the value matrix  $\mathbf{V} \in \mathbb{R}_+^{1 \times N}$  to be
$$
V _ {1, m} = \left\{ \begin{array}{l l} \max  \left(- \boldsymbol {X} _ {m} ^ {T} \boldsymbol {W} _ {:, h}, 0\right) & \text {i f} A _ {n, m} = 0 \\ \max  \left(\boldsymbol {X} _ {m} ^ {T} \boldsymbol {W} _ {:, h}, 0\right) & \text {i f} A _ {n, m} = 1. \end{array} \right.
$$
These linear bounds are referred to as doubleL in [15].
multiply the optimal value of the knapsack problem with  $-1$  and then add it to the original, unperturbed pre-activation value  $z_{h}^{(N)}$ .
Nodewise guarantees. The next step is to use the constructed linear lower bounds  $f_{1,k}, \ldots, f_{N,k}$  to lower-bound the value of functions  $f_{1,k}, \ldots, f_{N,k}$  for each possible number of perturbations admitted by local budgets  $\rho_1, \ldots, \rho_N$ . Note that we now need to take the degree normalization into account.
For our new perturbation set, we propose to use Algorithm 1 to perform these precomputations. We set the cost parameters to  $c^{+} = c_{\mathbf{A}}^{+}$  and  $c^{-} = c_{\mathbf{A}}^{-}$ . To prevent perturbations of the diagonal entries that are overwritten with self-loops, we set the index set parameters  $\mathbb{I}$  and  $\bar{\mathbb{I}}$  to  $\mathbb{I} = \{(1,m) \mid A_{n,m} = 0\} \cup \{1,n\}$  and  $\bar{\mathbb{I}} = \{(1,m) \mid A_{n,m} = 1\} \cup \{1,n\}$ . We set the global and local budget parameter of the algorithm to  $\epsilon$  and  $\rho_{n}$ , respectively. Because Algorithm 1 is defined to solve a maximization and not a minimization problem, we define the value matrix  $V \in \mathbb{R}^{1\times N}$  to be
$$
V _ {1, m} = \left\{ \begin{array}{l l} - q _ {m} ^ {(n)} & \text {i f} A _ {n, m} = 0 \\ q _ {m} ^ {(n)} & \text {i f} A _ {n, m} = 1. \end{array} \right.
$$