SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[52] J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra, “Beyond homophily in graph neural networks: Current limitations and effective designs,” Advances in Neural Information Processing Systems, vol. 33, pp. 7793–7804, 2020.
[53] A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov, "Opacus: User-friendly differential privacy library in PyTorch," arXiv preprint arXiv:2109.12298, 2021.
[54] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, "Self-normalizing neural networks," Advances in neural information processing systems, vol. 30, 2017.
Limitations. While our experimental results demonstrate several benefits of using DPDGC designs over standard convolutions, we do not believe that the current DPDGC model is the ultimate solution for GDP-aware graph learning methods. To support this claim, we note that the nonprivate state-of-the-art performance for learning on large-scale homophilic graphs is achieved by standard graph convolution models [48, 49]. Our experiments for edge GDP settings also reveal that standard graph convolution designs (such as GAP) appear to work well when the parameter  $\epsilon$  is small. We therefore conjecture that further improvements for decoupled graph convolution designs are possible.
Broader Impacts. We are not aware of any negative social impacts of our work. In fact, our work establishes a formal DP framework for graph learning termed GDP. This is beneficial to many applications that require rigorous protection of the privacy of users when their sensitive information is stored as a graph dataset to be leveraged by graph learning methods. Note that DP is the gold standard for quantifying the privacy of user data and it has found widespread applications. Given the popularity of graph learning methods and the need for user privacy protection, we believe that our work will actually have a positive social impact.
B Generalized adaptive composition theorem
In order to ensure that a graph learning method satisfies GDP, we need to follow a commonly employed approach that involves the following steps: (1) introducing appropriate perturbations to the learned graph representation (e.g.,  $\mathbf{H}^{(0)}$  for GAP or  $\mathbf{Z}$  for DP-DGC as shown in Figure 2), and (2) training the downstream DP-MLP using DPSGD. Subsequently, employing composition theorems allows for establishing an overall privacy budget. However, as we argued in Section 4, it is important to establish "partial DP" guarantees for intermediate node embeddings  $\mathbf{H}^{(k)}$  or  $\mathbf{Z}$  (i.e., ensure that only  $\mathbf{Z}_{\backslash r}$  is DP and exclude  $\mathbf{Z}_r$  from latter mechanisms, as in Theorem 5.3). When feeding the entire  $\mathbf{Z}$  to the subsequent mechanism  $\mathcal{B}(\mathbf{Z}_{\backslash r},\mathbf{Z}_r)$ , part of the input may not be DP and may exhibit correlations with other (processed) data components. This renders the standard composition theorem for DP [40, 41, 50] inapplicable. To address this issue, we developed a novel generalized composition theorem that can handle potential dependencies.
Theorem B.1. Let  $\mathcal{A}:\mathcal{D}\to (\mathcal{A}_1(\mathcal{D}),\mathcal{A}_2(\mathcal{D}))\in \mathcal{Z}_1\times \mathcal{Z}_2$  satisfy the  $(\alpha ,\varepsilon_{1})$  -RDP constraint in its first output argument, i.e.,  $D_{\alpha}\left(\mathcal{A}_{1}(\mathcal{D})\| \mathcal{A}_{1}(\mathcal{D}^{\prime})\right)\leq \varepsilon_{1}$  . Let  $\mathcal{B}:\mathcal{Z}_1\times \mathcal{Z}_2\rightarrow \mathcal{W}$  satisfy the  $(\alpha ,\varepsilon_2)$  RDP constraint in its second input argument, i.e.,  $\max_{\mathbf{z}_1,\mathbf{z}_2,\mathbf{z}_2'}D_\alpha (\mathcal{B}(\mathbf{z}_1,\mathbf{z}_2)\| \mathcal{B}(\mathbf{z}_1,\mathbf{z}_2'))\leq \varepsilon_2$  . Let the random noise used in  $\mathcal{A}$  and  $\mathcal{B}$  be independent. Then  $(\mathcal{A}_1(\mathcal{D}),\mathcal{B}(\mathcal{A}(\mathcal{D})))$  jointly meets the  $(\alpha ,\varepsilon_{1} + \varepsilon_{2})$  -RDP guarantee.
Proof. For ease of explanation, we assume that the density (and the conditional density) of  $\mathcal{A}$  and  $\mathcal{B}$  with respect to the Lebesgue measure exist. Let
$\cdot (\mathcal{A}_1(\mathcal{D}),\mathcal{A}_2(\mathcal{D}))\coloneqq (\mathbf{Z}_1,\mathbf{Z}_2)$  and  $(\mathcal{A}_1(\mathcal{D}'),\mathcal{A}_2(\mathcal{D}'))\coloneqq (\mathbf{Z}_1',\mathbf{Z}_2')$
$\mathcal{B}(\mathcal{A}(\mathcal{D}))\coloneqq \mathbf{W}$  and  $\mathcal{B}(\mathcal{A}(\mathcal{D}^{\prime}))\coloneqq \mathbf{W}^{\prime}$
In addition, let  $f_{\mathbf{W}}(\mathbf{w})$ ,  $f_{\mathbf{Z}_1}(\mathbf{z}_1)$ , and  $f_{\mathbf{Z}_2}(\mathbf{z}_2)$  be the density of  $\mathbf{W}$ ,  $\mathbf{Z}_1$  and  $\mathbf{Z}_2$ , respectively. Observe that
where (a) holds by the tower rule of conditional expectations; (b) holds due to the joint quasi-convexity of the Rényi divergence [51]; (c) holds since  $\mathcal{B}$  is  $\varepsilon_{2}$  Rényi DP; and (d) holds because  $\mathcal{A}_{1}$  is  $\varepsilon_{2}$  Rényi DP.
Remark B.2. Note that if  $\mathcal{A}_2(\mathcal{D})$  is independent of  $\mathcal{A}_1(\mathcal{D})$ , then the above composition theorem reduces to the standard sequential composition [40], which follows directly from the chain rule for the Rényi divergence. However, in the analysis of GDP, we need a stronger result capable of handling the dependence between  $\mathcal{A}_1(\mathcal{D})$  and  $\mathcal{A}_2(\mathcal{D})$ .
In the privacy analysis of our proposed DPDGC, we set  $\mathcal{A}_1(\mathcal{D})$  and  $\mathcal{A}_2(\mathcal{D})$  in Theorem B.1 to  $\mathbf{Z}_{\backslash r}$  and  $\mathbf{Z}_r$ , respectively, and let  $\mathcal{B}(\mathcal{D})$  be the weights of the DP-MLP $_W$  trained via DP-SGD (see Appendix H). Nevertheless, we note that since the Gaussian noise samples added to  $\mathbf{Z}_{\backslash r}$  and  $\mathbf{Z}_r$  are indeed independent in DPDGC, by grouping  $(\mathcal{A}_2(\mathcal{D}),\mathcal{B}(\mathcal{D}))$ , one can directly apply the classical composition theorem [40] to  $\mathcal{A}_1(\mathcal{D})$  and  $(\mathcal{A}_2(\mathcal{D}),\mathcal{B}(\mathcal{D}))$  and arrive at the same conclusion. Similarly, in the one-hop GAP illustrated in Figure 2, we applied the same privacy analysis, with  $\mathcal{A}_1(\mathcal{D})$ ,  $\mathcal{A}_2(\mathcal{D})$ , and  $\mathcal{B}(\mathcal{D})$  set to  $\mathbf{H}_{\backslash r}^{(0)},\mathbf{H}_r^{(0)}$ , and the weights of DP-MLP $_W$ , respectively, which allowed us to obtain the GDP guarantee.
However, it is worth noting that for a  $L$ -hop GAP (i.e., when aggregation is performed  $L$  times), the DP noise introduced in  $\mathbf{H}_{\backslash r}^{(L)}$  and that introduced in  $\mathbf{H}_r^{(L)}$  are no longer independent due to the aggregation step, and hence the standard composition theorem may not be applicable. In this case, our generalized composition theorem rigorously leads to the desired GDP guarantees.
In summary, our generalized composition theorem (Theorem B.1) provides a "cleaner argument" for the case that merely part of intermediate node embedding  $\mathbf{Z}$  and  $\mathbf{H}^{(k)}$  are DP (i.e.,  $\mathbf{Z}_{\backslash r}$  and  $\mathbf{H}_{\backslash r}^{(k)}$  are DP but not the entire  $\mathbf{Z}$  and  $\mathbf{H}^{(k)})$ ). Furthermore, our Theorem B.1 also allows for handling more complicated cases (i.e., when the noise in  $\mathcal{A}_1(\mathcal{D})$  and the one in  $\mathcal{A}_2(\mathcal{D})$  are not independent).
C Proof of Theorem 5.3
Theorem. For any  $\alpha > 1$  and  $\mathcal{D} \stackrel{N}{\sim} \mathcal{D}'$ , assume that  $D_{\alpha}((\mathbf{W}^{(A)}, \mathbf{b})||(\mathbf{W}^{(A)}, \mathbf{b})') \leq \gamma_1$  and that both  $\mathbf{A}$ ,  $\mathbf{A}'$  has bounded out-degree  $D$ . Let the replaced node index be  $r$  and  $\mathbf{Z}_{\backslash r}$  be the matrix  $\mathbf{Z}$  excluding its  $r^{th}$  row. Then the embedding  $\mathbf{Z}$  in DPDGC satisfies  $D_{\alpha}(\mathbf{Z}_{\backslash r}||\mathbf{Z}_{\backslash r}') \leq \gamma_1 + \frac{2D\alpha}{2s^2}$ .
Proof. We start by examining  $\| [\mathbf{A}'\mathbf{W}^{(A)}]_{\backslash r} - [\mathbf{A}\mathbf{W}^{(A)}]_{\backslash r}\| _F$ . Note that
$$
\| [ \mathbf {A} ^ {\prime} \mathbf {W} ^ {(A)} ] _ {\backslash r} - [ \mathbf {A} \mathbf {W} ^ {(A)} ] _ {\backslash r} \| _ {F} ^ {2} = \sum_ {i \in [ n ] \backslash \{r \}} \| (\mathbf {A} _ {i} ^ {\prime} - \mathbf {A} _ {i}) \mathbf {W} ^ {(A)} \| ^ {2},
$$
which is the sum of row-norms of all nodes except for node  $r$ . Let us denote the out-neighborhood of node  $r$  with respect to  $\mathbf{A}$  as  $N(r) = \{i : \mathbf{A}_{ir} = 1\}$ . Similarly, we also have  $N'(r) = \{i : \mathbf{A}_{ir}' = 1\}$ . There are four possible cases for  $i \in [n] \setminus \{r\}$ : (1)  $i \in N(r) \setminus N'(r)$ , (2)  $i \in N'(r) \setminus N(r)$ , (3)  $i \in N(r) \cap N'(r)$ , and (4)  $i \notin N(r) \cup N'(r)$ . Clearly, for  $i$  covered by cases (3) and (4), the corresponding row norm is 0 since  $\mathbf{A}_i' = \mathbf{A}_i$  for  $i \in N(r) \cap N'(r)$  and  $i \notin N(r) \cup N'(r)$ . For cases (1) and (2), we have
$$
\left\| \left(\mathbf {A} _ {i} ^ {\prime} - \mathbf {A} _ {i}\right) \mathbf {W} ^ {(A)} \right\| ^ {2} = \left\| \mathbf {e} _ {r} ^ {T} \mathbf {W} ^ {(A)} \right\| ^ {2} \stackrel {(a)} {=} 1,
$$
where  $(a)$  is due to the fact that  $\mathbf{W}^{(A)}$  is row-normalized. By the bounded out-degree assumption, we know that  $\max (|N(r)|, |N'(r)|) \leq D$ . Thus, the worst case upper bound of  $\|[\mathbf{A}'\mathbf{W}^{(A)}]_{\backslash r} - [\mathbf{A}\mathbf{W}^{(A)}]_{\backslash r}\|_F^2$  arises when  $N(r)$  and  $N'(r)$  are disjoint, which results in a contribution of  $2D$  for cases (1) and (2). As a result, we have
$$
\| [ \mathbf {A} ^ {\prime} \mathbf {W} ^ {(A)} ] _ {\backslash r} - [ \mathbf {A} \mathbf {W} ^ {(A)} ] _ {\backslash r} \| _ {F} ^ {2} = \sum_ {i \in [ n ] \backslash \{r \}} \| (\mathbf {A} _ {i} ^ {\prime} - \mathbf {A} _ {i}) \mathbf {W} ^ {(A)} \| ^ {2} \leq 2 D.
$$
This implies a sensitivity bound  $\sqrt{2D}$ . As a result, using the Gaussian noise mechanism with standard deviation  $s$  leads to the term  $\frac{2D\alpha}{2s^2}$  in the divergence bound. Finally, since the entire  $\mathbf{W}^{(A)}$  is DP by the assumption, applying the standard DP composition theorem [40] completes the proof.
Regarding the assumption used, it can met by applying a standard DP-optimizer with an group size  $D + 1$ , where  $\gamma_{1}$  depends on the noise multiplier of the DP-optimizer. This is due to the fact that the out-degree is bounded by  $D$ , so replacing one node can affect at most  $D$  neighbors.
D Proof of Theorem 5.4
Theorem. For any  $\mathcal{D} \stackrel{N_k}{\sim} \mathcal{D}'$ , let the index of the replaced node be  $r$ ; also, let  $\mathbf{Z}_{\backslash r}$  be the matrix  $\mathbf{Z}$  with its  $r^{th}$  row excluded, and assume that  $D_{\alpha}((\mathbf{W}^{(A)}, \mathbf{b})||(\mathbf{W}^{(A)}, \mathbf{b})') \leq \gamma_1$ . For any  $\alpha > 1$ , the embedding  $\mathbf{Z}$  in DPDGC satisfies  $D_{\alpha}(\mathbf{Z}_{\backslash r}|| \mathbf{Z}_{\backslash r}') \leq \gamma_1 + \frac{k\alpha}{2s^2}$ .
Proof. The proof is nearly identical to the proof for the node-GDP case (Theorem 5.3). The only difference arises when for the case  $i \in [n] \setminus \{r\}$ . Recall that our goal is to analyze
$$
\| [ \mathbf {A} ^ {\prime} \mathbf {W} ^ {(A)} ] _ {\backslash r} - [ \mathbf {A} \mathbf {W} ^ {(A)} ] _ {\backslash r} \| _ {F} ^ {2} = \sum_ {i \in [ n ] \backslash \{r \}} \| (\mathbf {A} _ {i} ^ {\prime} - \mathbf {A} _ {i}) \mathbf {W} ^ {(A)} \| ^ {2}.
$$
Again, there are four possible cases for  $i \in [n] \setminus \{r\}$ : (1)  $i \in N(r) \setminus N'(r)$ ; (2)  $i \in N'(r) \setminus N(r)$ ; (3)  $i \in N(r) \cap N'(r)$ ; and (4)  $i \notin N(r) \cup N'(r)$ . Clearly, for  $i$  covered by cases (3) and (4), the corresponding row norm is 0 as in this case,  $\mathbf{A}_i' = \mathbf{A}_i$ . For cases (1) and (2), we have
where  $(a)$  is due to the fact that  $\mathbf{W}^{(A)}$  is row-normalized. By the definition of  $\mathcal{D}\stackrel {N_k}{\sim}\mathcal{D}'$ , we know that there are at most  $k$  rows corresponding to cases (1) and (2):
$$
\left| \left\{i \in N (r) \backslash N ^ {\prime} (r) \right\} \right| + \left| \left\{i \in N (r) ^ {\prime} \backslash N (r) \right\} \right| \leq k.
$$
$$
\| [ \mathbf {A} ^ {\prime} \mathbf {W} ^ {(A)} ] _ {\backslash r} - [ \mathbf {A} \mathbf {W} ^ {(A)} ] _ {\backslash r} \| _ {F} ^ {2} = \sum_ {i \in [ n ] \backslash \{r \}} \| (\mathbf {A} _ {i} ^ {\prime} - \mathbf {A} _ {i}) \mathbf {W} ^ {(A)} \| ^ {2} \leq k.
$$
This implies a sensitivity bound of  $\sqrt{k}$ . As a result, applying the Gaussian noise mechanism with standard deviation  $s$  leads to the term  $\frac{k\alpha}{2s^2}$  in the divergence bound. Finally, since the entire  $\mathbf{W}^{(A)}$  is DP by assumption, applying the standard DP composition theorem [40] completes the proof.
E Proof of Theorem 5.1
Theorem. For any  $\alpha > 1$  and  $\mathcal{D} \stackrel{N}{\sim} \mathcal{D}'$  or  $\mathcal{D} \stackrel{N_k}{\sim} \mathcal{D}'$ , assume that  $D_{\alpha}(\mathbf{W}^{(X)}||\mathbf{W}^{(X)'}') \leq \gamma_1$  and that both  $\mathbf{A}$ ,  $\mathbf{A}'$  have bounded out-degree  $D$ . Let the replaced node index be  $r$  and let  $\mathbf{Z}_{\backslash r}$  be the matrix  $\mathbf{Z}$  excluding its  $r^{th}$  row. Then the embedding  $\mathbf{Z}$  used in GAP satisfies  $D_{\alpha}(\mathbf{Z}_{\backslash r}||\mathbf{Z}_{\backslash r}'') \leq \gamma_1 + \frac{4DL\alpha}{2s^2}$ .
Proof. We start by showing that for any  $\mathcal{D} \stackrel{N}{\sim} \mathcal{D}'$ ,  $D_{\alpha}(\mathbf{H}_{\backslash r}^{(1)}||\mathbf{H}_{\backslash r}^{(1)'}') \leq \gamma_1 + \frac{4D_{\alpha}}{2s^2}$ . By examining  $\|[\mathbf{A}\mathbf{H}]_i - [\mathbf{A}'\mathbf{H}']_i\|$  for all  $i \in [n] \setminus \{r\}$ , we find that there are three cases that contribute nonzero norms. Let  $N(r)$  and  $N'(r)$  denote the neighbor node set of  $r$  with respect to  $\mathbf{A}$  and  $\mathbf{A}'$ , respectively. The three cases are: (1)  $i \in N(r) \setminus N'(r)$ , (2)  $i \in N'(r) \setminus N(r)$ , and (3)  $i \in N(r) \cap N'(r)$ . For cases (1) and (2),  $\|[\mathbf{A}\mathbf{H}]_i - [\mathbf{A}'\mathbf{H}']_i\| \leq 1$  due to  $\mathbf{H}^{(0)}$  and  $\mathbf{H}^{(0)'}$  being row-normalized. For case (c), we have  $\|[\mathbf{A}\mathbf{H}]_i - [\mathbf{A}'\mathbf{H}']_i\| = \| \mathbf{H}_r - \mathbf{H}_r'\| \leq 2$ . Since the out-degree is upper bounded by  $D$ , we know that  $\max(|N(r)|, |N'(r)|) \leq D$ . The worst-case arises for  $|N(r) \cap N'(r)| = D$ . Hence we have the following worst-case upper bound
$$
\parallel [ \mathbf {A H} ] _ {\backslash r} - [ \mathbf {A ^ {\prime} H ^ {\prime}} ] _ {\backslash r} \parallel_ {F} ^ {2} = \sum_ {i \in [ n ] \backslash \{r \}} \parallel [ \mathbf {A H} ] _ {i} - [ \mathbf {A ^ {\prime} H ^ {\prime}} ] _ {i} \parallel^ {2} \leq 4 D ^ {2}.
$$
This leads to the term  $\frac{4D\alpha}{2s^2}$  in the divergence bound. By applying Theorem B.1 and the assumption on  $\mathbf{W}^{(X)}$ , we can show that  $D_{\alpha}(\mathbf{H}_{\backslash r}^{(1)}||\mathbf{H}_{\backslash r}^{(1)'}|\leq \gamma_1 + \frac{4D\alpha}{2s^2}$ .
For the  $L$ -hop result  $\mathbf{Z}_{\backslash r}$ , one can apply Theorem B.1 and induction. The base case  $L = 1$  is already established above. For the induction step, we have  $D_{\alpha}(\mathbf{H}_{r}^{(L - 1)}||\mathbf{H}_{r}^{(L - 1)'}|\leq \gamma_1 + \frac{4D(L - 1)\alpha}{2s^2}$ . By applying Theorem B.1 and repeating the previous analysis, we have
$$
D _ {\alpha} \left(\mathbf {H} _ {\backslash r} ^ {(L)} \right\lvert   \left\| \mathbf {H} _ {\backslash r} ^ {(L) ^ {\prime}} \right. \leq \gamma_ {1} + \frac {4 D (L - 1) \alpha}{2 s ^ {2}} + \frac {4 D \alpha}{2 s ^ {2}} = \gamma_ {1} + \frac {4 D L \alpha}{2 s ^ {2}}. \tag {3}
$$
Here, the key idea is that although  $\mathbf{H}_r^{(L - 1)}$  is not private, it is still row-normalized so that the above sensitivity analysis still applies. For the case of  $\mathcal{D}\stackrel {N_k}{\sim}\mathcal{D}'$ , the worst case scenario still arises for  $|N(r)\cap N'(r)| = D$ , for all  $k\geq 0$ . This is in fact the case for  $N_0$  (i.e.,  $\mathbf{A} = \mathbf{A}'$ ). Thus, the same result holds for the case  $\mathcal{D}\stackrel {N_k}{\sim}\mathcal{D}'$ , for all  $k\geq 0$ . This completes the proof.
F RDP to DP conversion
For a given  $(\alpha, \gamma)$  Rényi DP guarantee, the following conversion lemma [34-36] allows us to convert it back to a  $(\varepsilon, \delta)$ -DP guarantee:
Lemma F.1. If  $\mathcal{A}$  satisfies  $(\alpha, \gamma(\alpha))$ -RDP for all  $\alpha > 1$ , then, for any  $\delta > 0$ ,  $\mathcal{A}$  satisfies  $(\varepsilon_{\mathrm{DP}}(\delta), \delta)$ -DP, where
$$
\varepsilon_ {\mathsf {D P}} (\delta) = \inf _ {\alpha > 1} \gamma (\alpha) + \frac {\log (1 / \alpha \delta)}{\alpha - 1} + \log (1 - 1 / \alpha).
$$
G Edge GDP analysis for DPDGC and GAP
DPDGC. In this setting, ensuring that the embedding  $\mathbf{Z}$  is DP is sufficient to guarantee the overall model being GDP. We can replace the remaining DP-MLP modules in Figure 2 with standard MLP (i.e., training with a standard optimizer).
Theorem G.1. For any  $\alpha > 1$  and  $\mathcal{D} \stackrel{E}{\sim} \mathcal{D}'$ , assume  $D_{\alpha}((\mathbf{W}^{(A)}, \mathbf{b})||(\mathbf{W}^{(A)}, \mathbf{b})') \leq \gamma_1$ . Then the embedding  $\mathbf{Z}$  of DPDGC satisfies  $D_{\alpha}(\mathbf{Z}||\mathbf{Z}') \leq \gamma_1 + \frac{\alpha}{2s^2}$ .
Proof. Since there is only one replaced entry of the adjacency matrix for any  $\mathcal{D} \stackrel{E}{\sim} \mathcal{D}'$ ,  $\|(\mathbf{A}' - \mathbf{A}) \mathbf{W}^{(A)}\|_F \leq \max_{i \in [n]} \| \mathbf{W}_i^{(A)}\|_2 = 1$  (i.e., the maximum row-norm of  $\mathbf{W}^{(A)})$ ). This implies that the sensitivity of  $\mathbf{A} \mathbf{W}^{(A)}$  is 1. Thus, applying the Gaussian noise mechanism with standard deviation  $s$  and standard DP composition rule [40] results in  $D_{\alpha}(\mathbf{Z}||\mathbf{Z}') \leq \gamma_1 + \frac{\alpha}{2s^2}$ .
Regarding the assumption, it can be met by applying a standard DP-optimizer with group size 1, where  $\gamma_{1}$  depends on the noise multiplier of the DP-optimizer. This is due to the fact that at most one row of  $\mathbf{A}$  is different for any  $\mathcal{D} \stackrel{E}{\sim} \mathcal{D}'$ . By further applying Theorem B.1, we can establish the edge GDP guarantees for DGDGC (Corollary H.1).
GAP. In this setting, all DP-MLP modules in Figure 2 can be replaced with a standard MLP for GAP. Thus, we only need to ensure that the non-trainable module in GAP (i.e., PMA) is DP.
Theorem G.2. For any  $\alpha > 1$  and  $\mathcal{D} \stackrel{E}{\sim} \mathcal{D}'$ , the embedding  $\mathbf{Z}$  of GAP satisfies  $D_{\alpha}(\mathbf{Z}||\mathbf{Z}') \leq \frac{L\alpha}{2s^2}$ .
Proof. The analysis is similar to that of Theorem G.1, except that we have  $\mathbf{AH}$  instead of  $\mathbf{AW}_A$ . Following the same argument, we know that  $D_{\alpha}(\mathbf{H}^{(1)}||\mathbf{H}^{(1)'}|)\leq \frac{\alpha}{2s^2}$ . Then, by the standard DP composition theorem, we arrive at the claimed result.
H Formal GDP guarantees for the complete DPDGC and GAP models
Corollary H.1. For any  $\alpha > 1$ , the DPDGC model is edge  $(\alpha, \gamma_1 + \frac{\alpha}{2s^2})$ -GDP.
Proof. This is a direct consequence of applying Theorem G.1 (for guarantees pertaining to  $\mathbf{Z}$ ) and the DP composition theorem.
Corollary H.2. Assume that the graph has bounded out-degree  $D$ . For any  $\alpha > 1$ , the DPDGC model is node  $(\alpha, \gamma_1 + \gamma_2 + \frac{2D\alpha}{2s^2})$ -GDP, where the remaining weights satisfy  $(\alpha, \gamma_2)$ -RDP.
Proof. This is a direct consequence of Theorem 5.3 (for guarantees pertaining to  $\mathbf{Z}_{(r)}$ ), the standard DP-optimizer result [14] and Theorem B.1 (the generalized adaptive composition theorem).
Corollary H.3. For any  $\alpha > 1$ , the DPDGC model is  $k$ -neighbor  $(\alpha, \gamma_1 + \gamma_2 + \frac{k\alpha}{2s^2})$ -GDP, with the remaining weights are  $(\alpha, \gamma_2)$ -RDP.
Proof. This follows by applying Theorem 5.4 (with guarantees for  $\mathbf{Z}_{\backslash r}$ ), the standard DP-optimizer result [14], and Theorem B.1 (generalized adaptive composition theorem).
Corollary H.4. For any  $\alpha > 1$ , the GAP model is edge  $(\alpha, \frac{L\alpha}{2s^2})$ -GDP.
Proof. This follows by applying Theorem G.2 (guarantees of  $\mathbf{Z}$ ) and the DP composition theorem.
Corollary H.5. Assume that the underlying graph has bounded out-degree  $D$ . For any  $\alpha > 1$ , the GAP model is node or  $k$ -neighbor  $(\alpha, \gamma_1 + \gamma_2 + \frac{4DL\alpha}{2s^2})$ -GDP, while the remaining weights are  $(\alpha, \gamma_2)$ -RDP.
Proof. This follows by applying Theorem 5.1 (with guarantees for  $\mathbf{Z}_{\backslash r}$ ), the standard DP-optimizer result [14], and Theorem B.1 (generalized adaptive composition theorem).
I Discussion on inductive settings
While we focus on the main text was on the transductive setting, we note that the inductive setting is actually significantly easier from the perspective of insuring privacy. There are two different settings for inductive graph learning. We call the first scenario the "fully inductive setting". What this means is that the information pertaining to test nodes is completely unavailable during the training phase and not subject to privacy protection [3]. One can think of this as a case where the training graph and the test graph are disjoint. In this case, model DP is sufficient to guarantee prediction DP, as one can only access training data information through the model weights. As a result, the extension of DP-SGD for GNNs [15] suffices to ensure rigorous user data privacy.
We refer to the second setting as the "incrementally inductive setting". What this means is that we will use all training and test node information during inference, despite the test nodes not being used during the training phase. In this case, DP of model weight does not ensure the DP of the model
prediction. Hence, we will need our GDP guarantees to ensure user data privacy protection. Since we only need to ensure the privacy of training data, the RDP condition in Definition 4.2 no longer needs to specify which test node is to be predicted. This is due to the fact that any test node  $v$  that could be the target for prediction cannot be a training node. Thus, we will not replace it in the adjacent dataset. The remainder of the GDP analysis is similar to the previous one.
J Practical privacy meaning of  $k$  in  $k$ -neigbor GDP