SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sketch of the proof: We start by showing that for any  $\mathcal{D} \stackrel{N}{\sim} \mathcal{D}'$ ,  $D_{\alpha}(\mathbf{H}_{r}^{(1)}||\mathbf{H}_{r}^{(1)'}') \leq \gamma_1 + \frac{4D\alpha}{2s^2}$ , which is done by examining the sensitivity of  $[\mathbf{A}\mathbf{H}^{(0)}]_{\backslash r}$ . For simplicity, we abbreviate  $\mathbf{H}^{(0)}$  to  $\mathbf{H}$ . Note that  $\left\| [\mathbf{A}\mathbf{H}]_{\backslash r} - [\mathbf{A}'\mathbf{H}']_{\backslash r}\right\|_F^2 = \sum_{i \in [n] \setminus \{r\}} \left\| [\mathbf{A}\mathbf{H}]_i - [\mathbf{A}'\mathbf{H}'']_i\right\|^2$ . We find that there are three cases of  $i$  that contribute to a nonzero norm in the summation. Let  $N(r)$  and  $N'(r)$  denote the out-neighbor node sets of  $r$  with respect to  $\mathbf{A}$  and  $\mathbf{A}'$ , respectively (i.e.,  $N(r) = \{i : \mathbf{A}_{ir} = 1\}$ ). The three cases are: (1)  $i \in N(r) \setminus N'(r)$ , (2)  $i \in N'(r) \setminus N(r)$ , and (3)  $i \in N(r) \cap N'(r)$ . For case (1) and (2),  $\| [\mathbf{A}\mathbf{H}]_i - [\mathbf{A}'\mathbf{H}'']_i\| = 1$  due to  $\mathbf{H}$  and  $\mathbf{H}'$  being row-normalized. For case (3), we have  $\| [\mathbf{A}\mathbf{H}]_i - [\mathbf{A}'\mathbf{H}'']_i\| = \| \mathbf{H}_r - \mathbf{H}_r'\| \leq 2$ . Since the out-degree is upper bounded by  $D$ , we know that  $\max(|N(r)|, |N'(r)|) \leq D$ . The worst case arises for  $|N(r) \cap N'(r)| = D$  (i.e., shares common neighbors). Thus the sensitivity equals  $2\sqrt{D}$  (i.e.,  $D$  of case (3)), which leads to the term  $\frac{4D\alpha}{2s^2}$  in the divergence bound. By applying DP composition theorem [40] and the assumption on  $\mathbf{W}^{(X)}$ , we can prove that  $D_{\alpha}(\mathbf{H}_{r}^{(1)}||\mathbf{H}_{r}^{(1)'}') \leq \gamma_1 + \frac{4D\alpha}{2s^2}$ . For the  $L$ -hop result  $\mathbf{Z}_{\backslash r}$ , one can apply DP composition theorem as part of an induction. For the case of  $k$ -neighbor-level adjacency, the worst case scenario still arises for  $|N(r) \cap N'(r)| = D$  for any  $k \geq 0$ . Thus, the same result holds for the case  $k$ -neighbor-level adjacency for all  $k \geq 0$ . Note that the above analysis/result is tight, with the worst case arising when  $\mathbf{H}_r = -\mathbf{H}_r'$  and  $\max(|N(r)|, |N'(r)|) = D$ .
Regarding the assumption on  $\mathbf{W}^{(X)}$ , it can be met by invoking a standard DP-optimizer result [14], where  $\gamma_{1}$  depends on the noise multiplier of the DP-optimizer. By using further the DP composition theorem and standard DP-optimizer results, we can conclude that the weights of the DP- $\mathrm{MLP}_f$  module are also DP. At the inference stage, since our GDP definition requires that  $r \neq v$  (i.e., nodes to be predicted are not subject to replacement in adjacent graph dataset pairs), one only needs to ensure that  $\mathbf{Z}_{\backslash r}$  - instead of the entire  $\mathbf{Z}$  - to be DP. This establishes the node and  $k$ -neighbor GDP guarantees for GAP (Corollary H.5).
Surprisingly, the resulting divergence bound does not depend on the parameter  $k$  for the  $k$ -neighbor GDP setting. It implies that the privacy noise scale  $s$  required by GAP is the same even when  $k$  is much smaller than  $D$ , or even equal to zero. At first, this result seems counter-intuitive but can be understood as follows. By replacing  $\mathbf{H}_r$  with an all-zeros row, it becomes impossible to get information about the  $r^{th}$  row and column of  $\mathbf{A}$  through  $[\mathbf{AH}]_{\backslash r}$ . Therefore, DP-GNNs based on standard graph convolution designs such as GAP cannot exploit the intrinsic privacy-utility trade-offs induced by  $k$ -neighbor-level adjacency constraints. Furthermore, the resulting divergence bound grows linearly to the value of the maximum degree  $D$ . Consequently, one has to preprocess the graph so that the maximum degree is upper-bounded by a pre-defined value  $D$ , which inevitably causes graph information distortion.
Based on our analysis, we observe that the issue arises from the graph convolution operation  $\mathbf{A}\mathbf{H}$ , where both the graph structure (topology)  $\mathbf{A}$  and transformed node feature  $\mathbf{H}$  change simultaneously to  $\mathbf{A}'$  and  $\mathbf{H}'$  on  $\mathcal{D}'$ . As a result, rows corresponding to case (3) contribute 2 to the norm bound, which indicates greater privacy leakage (sensitivity). This motivates us to decouple the graph convolution
so that there are no products of  $\mathbf{A}$  and  $\mathbf{H}$  to work with, which motivates introducing our DPDGC model discussed in the next section.
Remark 5.2. While the proof of Theorem 5.1 is mainly inspired by the proof in [17], it has several technical differences. First, the analysis in [17] asserts that the entire  $\mathbf{Z}$  is  $DP$  (see Lemma 3 in [17]). In contrast, we only ensure that  $\mathbf{Z}_{\backslash r}$  is  $DP$ . This is crucial as  $\|[\mathbf{A}\mathbf{H}]_r - [\mathbf{A}'\mathbf{H}']_r\| = 2D$  in the worst-case sensitivity analysis, which leads to  $O(D^2)$  in the divergence bound. Also, while we leverage standard  $DP$  composition theorems [40] in the sketch of proof similar to [17] for the sake of simplicity, we argue in Appendix B that it is more appropriate to use our novel generalized adaptive composition theorem (Theorem B.1) for a rigorous GDP analysis.
5.2 GDP guarantees of DPDGC and benefits of decoupled graph convolution
DPDGC training and inference. We first describe the training process of DPDGC depicted in Figure 2. We first pretrain the DP-Emb module separately and freeze its weights  $(\mathbf{W}^{(A)}, \mathbf{b})$ . Then we generate the intermediate embedding  $\mathbf{Z}$  and cache it. Finally, we use  $(\mathbf{X}, \mathbf{Z}, \mathbf{Y})$  to train the remaining modules in an end-to-end fashion. At the inference stage, the node prediction is obtained by  $(\mathbf{X}, \mathbf{Z})$  and the trained weights. See the pseudocode in Appendix L for further details.
Node  $k$ -neighbor GDP guarantees. For DPDGC, we only need the bounded out-degree assumption for node GDP but not  $k$ -neighbor GDP. The key idea of the GDP guarantee proof is to ensure the cached embedding  $\mathbf{Z}$  to be DP, except for the replaced node  $r$ . We start by introducing the DP-Emb module in DPDGC, which guarantees both the model weight  $(\mathbf{W}^{(A)}, \mathbf{b})$  and  $\mathbf{Z}_{\backslash r}$  to be DP:
$$
\text {T r a i n i n g :} \widehat {\mathbf {Y}} ^ {(A)} = \sigma (\mathbf {A W} ^ {(A)} + \mathbf {b}) \mathbf {R} \quad \text {G e n e r a t e} \mathbf {Z}: \mathbf {Z} = \mathbf {A W} ^ {(A)} + \mathbf {b} + \mathbf {N}, \tag {2}
$$
where  $\mathbf{W}^{(A)}\in \mathbb{R}^{n\times h}$  and  $\mathbf{b}\in \mathbb{R}^h$  are learnable weights. Here,  $\mathbf{R}\in \mathbb{R}^{h\times C}$  is a random but fixed projection head of hidden dimension  $h$ ,  $\sigma (\cdot)$  is some nonlinear activation function, and  $\mathbf{N}\in \mathbb{R}^{n\times h}$  is a Gaussian noise matrix whose entries are i.i.d. zero mean Gaussian random variables with standard deviation  $s$ . Importantly, we constraint  $\mathbf{W}^{(A)}$  to be row-normalized, which is critical in proving  $\mathbf{Z}_{\backslash r}$  DP. In what follows, we focus on privatizing  $\mathbf{Z}_{\backslash r}$ , and the GDP guarantee for the overall model then follows by applying DP composition theorem [40].
Theorem 5.3. For any  $\alpha >1$  and  $\mathcal{D}\stackrel {N}{\sim}\mathcal{D}^{\prime}$ , assume that  $D_{\alpha}((\mathbf{W}^{(A)},\mathbf{b})||(\mathbf{W}^{(A)},\mathbf{b})^{\prime})\leq \gamma_{1}$  and both A,  $\mathbf{A}'$  have out-degree bounded by  $D$ . Let the replaced node index be  $r$  and let  $\mathbf{Z}_{\backslash r}$  be the matrix  $\mathbf{Z}$  with the  $r^{th}$  row excluded. Then the embedding  $\mathbf{Z}$  in DPDGC satisfies  $D_{\alpha}(\mathbf{Z}_{\backslash r}||\mathbf{Z}_{\backslash r}^{\prime})\leq \gamma_{1} + \frac{2D\alpha}{2s^{2}}$ .
Theorem 5.4. For any  $\mathcal{D} \stackrel{N_k}{\sim} \mathcal{D}'$ , assume that  $D_{\alpha}((\mathbf{W}^{(A)}, \mathbf{b})||(\mathbf{W}^{(A)}, \mathbf{b})') \leq \gamma_1$  and both A. Let the replaced node index be  $r$  and let  $\mathbf{Z}_{\backslash r}$  be the matrix  $\mathbf{Z}$  with the  $r^{th}$  row excluded. For any  $\alpha > 1$ , the embedding  $\mathbf{Z}$  in DPDGC satisfies  $D_{\alpha}(\mathbf{Z}_{\backslash r}||\mathbf{Z}_{\backslash r}') \leq \gamma_1 + \frac{k\alpha}{2s^2}$ .
Sketch of the proof: For any  $\mathcal{D} \stackrel{N}{\sim} \mathcal{D}'$  ( $\mathcal{D} \stackrel{N_k}{\sim} \mathcal{D}'$ ), we examine  $\|[\mathbf{A}\mathbf{W}^{(A)}]_i - [\mathbf{A}'\mathbf{W}^{(A)}]_i\|$ ,  $i \in [n] \setminus \{r\}$ . The worst case row norm equals 1 for cases (1) and (2) (defined in the proof of Theorem 5.1) since  $\mathbf{W}^{(A)}$  is row-normalized. The main difference to the proof in Section 5.1 is the case (3), where  $\|[\mathbf{A}\mathbf{W}^{(A)}]_i - [\mathbf{A}'\mathbf{W}^{(A)}]_i\| = 0$ , since  $\mathbf{A}_i = \mathbf{A}_i'$  for  $i \in N(r) \cap N'(r)$ . As a result, the worst case arises for  $|N(r) \cap N'(r)| = \emptyset$  and there are at most  $2D(k)$  rows of cases (1) and (2). As a result, the sensitivity of  $\mathbf{Z}_{\backslash r}$  is  $\sqrt{2D}(\sqrt{k})$  which leads to the term  $\frac{2D\alpha}{2s^2}\left(\frac{k\alpha}{2s^2}\right)$  in the divergence bound. Once more, our sensitivity bound can be shown to be tight, with the worst case as described above.
The key improvement, when compared to GAP, arises from using  $\mathbf{A}\mathbf{W}^{(A)}$  instead of  $\mathbf{A}\mathbf{H}$ . In the case of  $\mathbf{W}^{(A)}$  being DP, the sensitivity analysis only requires one to consider the difference between  $\mathbf{A}\mathbf{W}^{(A)}$  and  $\mathbf{A}'\mathbf{W}^{(A)}$  of two adjacent graph datasets,  $\mathcal{D}$  and  $\mathcal{D}'$ . In contrast, for GAP, the sensitivity analysis of  $\mathbf{A}\mathbf{H}$  requires taking both  $\mathbf{A}$  and  $\mathbf{H}$  into account since both can vary when the underlying dataset changes. Note that even if DP- $\mathrm{MLP}_X$  is trained via a DP-optimizer,  $\mathbf{H}$  is not DP since it is dependent on the weights of DP- $\mathrm{MLP}_X$  and the node feature  $\mathbf{X}$ .
The assumption on  $(\mathbf{W}^{(A)},\mathbf{b})$  can be met by applying a standard DP-optimizer with a group size [41] of  $D + 1(k + 1)$ , where  $\gamma_{1}$  depends on the noise multiplier of the DP-optimizer. This follows from the fact that the out-degree is bounded by  $D$  for node GDP or from the definition of  $k$ -neighbor-level adjacency, since replacing one node can affect at most  $D(k)$  neighbors. By further applying the DP
Table 1: Dataset statistics. Datasets are sorted by homophily.
Squirrel	Chameleon	Facebook	Pubmed	Computers	Cora	Photo
nodes	5,201	2,277	26,406	19,717	13,471	2,708	7,535
edges	216,933	36,051	2,117,924	88,648	491,722	10,556	238,162
features	2,089	2,325	501	500	767	1,433	745
classes	5	5	6	3	10	7	8
edge density	0.0160	0.0139	0.0061	0.0005	0.0054	0.0029	0.0084
homophily	0.0254	0.0620	0.3687	0.6641	0.7002	0.7657	0.7722
composition theorem and the standard DP-optimizer result, we arrive at the node ( $k$ -neighbor) GDP guarantees for DPDGC (see Corollary H.2, H.3).
Compared to GAP, DPDGC requires significantly lower DP noise whenever  $k < D$ . It can thus ensure a graph topology privacy-utility trade-off that is not possible with GAP. Furthermore, the divergence bound for DPDGC is independent of the maximum degree within the  $k$ -neighbor-level adjacency setting. Hence, DPDGC does not require preprocessing the graph, which alleviates the issue of added graph distortion.
We test graph learning models that can achieve GDP guarantees under various settings, including nonprivate, edge-level,  $k$ -neighbor-level ( $N_{k}$ ) for  $k \in \{1, 5, 25\}$ , and node-level. Note that all node GDP methods require the bounded out-degree constraint. We follow [15, 17] to subsample the graph so as to satisfy this constraint.
Methods. In addition to DPDGC and GAP introduced in Section 5, we also test (DP-)MLP and several DP-GNN baselines that can achieve GDP guarantees, including RandEdge+SAGE [29] and DP-SAGE [15] for edge and node GDP, respectively. The RandEdge+SAGE approach privatizes the adjacency matrix directly via randomized response [29] and feeds it to GraphSAGE [3], a standard GNN backbone. The DP-SAGE approach trains the GraphSAGE model with the strategy proposed in [15] so that the weights are DP. To further ensure that the predictions are DP as well, we follow the strategy of [17] to add perturbations directly at the output layer during inference. For each GDP setting, we specify  $\epsilon$  to indicate that all methods satisfy GDP with privacy budget  $(\epsilon, \delta)$  according to Lemma F.1. Note that  $\delta$  is set to be smaller than either  $\frac{1}{\#edges}$  or  $\frac{1}{\#nodes}$ , depending on the GDP setting. Addition experimental details are deferred to Appendix K.
Datasets. We test 7 benchmark datasets available from either Pytorch Geometric library [42] or prior works. These datasets include the social network Facebook [43], citation networks Cora and Pubmed [44,45], Amazon co-purchase networks Photo and Computers [46], and Wikipedia networks Squirrel and Chameleon [47]. The edge density equals  $2|\mathcal{E}| / (n \times (n - 1))$ , while the formal definition of the homophily measure is relegated to Appendix K.
Results. We examine the performance of each model under different GDP settings, including the nonprivate case. The results are summarized in Table 2. For the edge GDP setting, we observe that DPDGC achieves the best performance on heterophilic datasets, while on par with GAP on most homophilic datasets. Surprisingly, for the node GDP setting, we find that DP-MLP achieves the best performance on four datasets. This indicates that for some datasets, the benefits of graph information cannot compensate for the utility loss induced by privacy noise that protects the graph information (for all tested private graph learning methods). As a result, achieving node GDP effectively is highly challenging and highlights the importance of investigating the  $k$ -neighbor GDP setting. For the  $k$ -neighbor GDP setting, we can see that indeed DPDGC has a much better utility for  $k = 1$  and the performance gradually decreases as  $k$  grows. The required privacy noise scale is the same for GAP and DP-MLP for any  $k$ : this phenomenon is discussed in more depth in Section 5. DPDGC starts to outperform DP-MLP on Photo and Computers for sufficiently small  $k$ , which demonstrates the unique utility-topology privacy trade-offs of our method that cannot be achieved via models with standard graph convolution design such as GAP. However, DPDGC still underperforms compared to DP-MLP on Cora and Pubmed.
Experiments on synthetic contextual stochastic block models (cSBMs). In order to test our conjecture that DP-MLP can only outperform DPDGC when the graph information is "too weak," we conduct an experiment involving cSBMs [18], following a setup similar to that reported in [19].
Table 2: Test accuracy (\%) with  $95\%$  confidence interval. A bold font indicates the best performance one can achieve under the same GDP guarantee. Underlined entries indicate a result within the confidence interval when compared to the best possible. Note that for the  $k$ -neighbor  $(N_k)$  GDP setting, the results of GAP and DP-MLP are identical to those of node GDP.
Squirrel	Chameleon	Facebook	Pubmed	Computers	Cora	Photo
none private	DPDGC	79.92 ± 0.26	79.24 ± 0.54	86.06 ± 0.24	88.34 ± 0.46	92.27 ± 0.15	82.44 ± 0.83	94.98 ± 0.29
GAP	36.86 ± 1.35	50.35 ± 1.37	79.52 ± 0.24	89.75 ± 0.12	91.05 ± 0.16	86.53 ± 0.46	95.13 ± 0.16
SAGE	35.47 ± 0.58	41.61 ± 0.86	84.62 ± 0.13	88.17 ± 0.98	91.76 ± 0.23	84.19 ± 0.76	94.05 ± 0.38
MLP	34.14 ± 0.77	46.78 ± 1.39	51.16 ± 0.16	87.25 ± 0.19	85.27 ± 0.28	76.48 ± 0.91	91.35 ± 0.22
edge ε = 1	DPDGC	38.18 ± 1.48	53.83 ± 1.11	62.04 ± 0.33	88.59 ± 0.16	87.74 ± 0.26	77.71 ± 0.95	92.59 ± 0.41
GAP	35.15 ± 0.47	49.47 ± 0.88	69.75 ± 0.44	87.79 ± 0.22	87.74 ± 0.20	76.95 ± 0.90	92.94 ± 0.36
RandEdge+SAGE	19.79 ± 0.69	21.70 ± 1.23	25.27 ± 2.00	87.88 ± 0.18	48.44 ± 1.48	59.95 ± 1.98	46.42 ± 0.55
DP-MLP	34.14 ± 0.77	46.78 ± 1.39	51.16 ± 0.16	87.25 ± 0.19	85.27 ± 0.28	76.48 ± 0.91	91.35 ± 0.22
N1ε = 16	DPDGC	42.71 ± 1.43	48.63 ± 1.78	80.94 ± 0.27	84.33 ± 0.40	83.49 ± 0.29	59.98 ± 0.81	88.38 ± 0.44
GAP	33.82 ± 0.60	38.68 ± 0.59	51.57 ± 0.28	85.28 ± 0.14	77.50 ± 0.20	54.36 ± 1.14	81.27 ± 0.31
DP-MLP	34.46 ± 1.09	38.19 ± 1.97	50.12 ± 0.22	85.72 ± 0.11	80.01 ± 0.37	64.29 ± 0.80	85.61 ± 0.42
N5ε = 16	DPDGC	41.00 ± 1.19	47.22 ± 1.90	76.84 ± 0.36	84.31 ± 0.46	80.60 ± 0.44	59.56 ± 0.97	87.02 ± 0.49
GAP	33.82 ± 0.60	38.68 ± 0.59	51.57 ± 0.28	85.28 ± 0.14	77.50 ± 0.20	54.36 ± 1.14	81.27 ± 0.31
DP-MLP	43.46 ± 1.09	38.19 ± 1.97	50.12 ± 0.22	85.72 ± 0.11	80.01 ± 0.37	64.29 ± 0.80	85.61 ± 0.42
N25ε = 16	DPDGC	40.51 ± 0.85	46.32 ± 1.87	68.66 ± 0.32	84.27 ± 0.38	78.25 ± 0.31	59.26 ± 0.87	84.82 ± 0.54
GAP	33.82 ± 0.60	38.68 ± 0.59	51.57 ± 0.28	85.28 ± 0.14	77.50 ± 0.20	54.36 ± 1.14	81.27 ± 0.31
DP-MLP DP-MLP	34.46 ± 1.09	38.19 ± 1.97	50.12 ± 0.22	85.72 ± 0.11	80.01 ± 0.37	64.29 ± 0.80	85.61 ± 0.42
node ε = 16	DPDGC	36.17 ± 0.62	46.43 ± 1.21	56.65 ± 0.64	84.55 ± 0.32	76.62 ± 0.47	58.97 ± 1.05	82.15 ± 0.54
GAP	33.82 ± 0.60	38.68 ± 0.59	51.57 ± 0.28	85.28 ± 0.14	77.50 ± 0.20	54.36 ± 1.14	81.27 ± 0.31
19.81 ± 0.99	20.96 ± 1.27	32.15 ± 0.78	39.68 ± 0.70	39.13 ± 0.52	39.13 ± 0.52	15.86 ± 1.55	31.41 ± 0.93
34.46 ± 1.09	38.19 ± 1.97	50.12 ± 0.22	85.72 ± 0.11	80.01 ± 0.37	80.01 ± 0.37	64.29 ± 0.80	85.61 ± 0.42
Figure 3: (a): Trade-off between utility and the parameter  $\epsilon$  for edge-GDP on the Facebook dataset. (b): Trade-off between utility and the parameter  $\epsilon$  for node and  $k$ -neighbor  $(N_{k})$  GDP on the Facebook dataset. (c): Trade-off between utility and the parameter  $k$  for  $k$ -neighbor GDP. (d): The cSBM experiment, which shows that when the graph information is strong enough (i.e.,  $|\phi|$  is large enough), DPDGC can indeed outperform DP-MLP.
The parameter  $\phi \in [-1, 1]$  indicates the "strength of information" in the graph topology  $(|\phi| \to 1)$  and the node features  $(|\phi| \to 0)$ . The sign of  $\phi$  indicates whether the graph topology is homophilic (+) or heterophilic (-). See Appendix K for full details about the experiment setting. The result is depicted in Figure 3 (d). When  $|\phi|$  is small, DP-MLP outperforms DPDGC. In contrast, when  $|\phi|$  is sufficiently large (i.e., the information in the graph topology is strong enough), DPDGC outperforms DP-MLP. This experiment suggests that one should not leverage DP-GNNs when the graph topology information is too weak, as the cost of privatizing the graph topology information is too high.
Utility-privacy trade-off. Lastly, we examine the utility-privacy trade-off for different GDP settings in greater detail. In Figure 3 (left) we show that DPDGC starts to outperform GAP for larger privacy budgets  $-\epsilon \geq 4-$  in the edge GDP setting. This figure also shows the advantage of GAP when the privacy budget  $\epsilon$  is small for edge GDP. Whether a more advanced decoupled graph convolution design can outperform GAP in this regime is left for future studies. Figure 3 (middle) reports the trade-offs between utility and the privacy budget  $\epsilon$  of all methods under different GDP settings (node,  $N_{1}, N_{5}, N_{25}$ ) for the Facebook dataset. We report the trade-off between utility and  $k$  under  $k$ -neighbor GDP with  $\epsilon = 16$  for the Computer dataset in Figure 3 (right). Adopting the  $k$ -neighbor GDP definition allows us to control the privacy of the graph structure while maintaining the same privacy guarantee on node features and labels. This is in contrast with changing the parameters  $(\epsilon, \delta)$  directly as they control the privacy of the entire dataset. DPDGC along with the  $k$ -neighbor GDP definitions hence allows a more fine-grained privacy control of the graph topology as needed in different practical applications.
We performed an analysis of a novel notion of Graph Differential Privacy (GDP), specifically tailored to graph learning settings. Our analysis established theoretical privacy guarantees for both model weights and predictions. In addition, to offer multigranular protection for the graph topology, we introduced the concept of  $k$ -neighbor-level adjacency, which is a relaxation of standard node-level adjacency. This allows for controlling the strength of privacy protection for node neighborhood information via a parameter  $k$ . The supporting GDP approach ensured a flexible trade-off between utility and topology privacy for graph learning methods. The GDP analysis also revealed that standard graph convolution designs failed to offer this trade-off. To provably mitigate the problem associated with standard convolutions, we introduced Differentially Private Decoupled Graph Convolution (DPDGC), a model that comes with GDP guarantees. Extensive experiments conducted on seven node classification benchmark datasets and synthetic cSBM datasets demonstrated the superior privacy-utility trade-offs offered by DPDGC when compared to existing DP-GNNs that rely on standard graph convolution designs.
EC, CP and OM were funded by NSF grants CCF-1816913 and CCF-1956384. PL was supported by JPMC AI Research award. WC and AÖ were supported by NSF grant CCF-2213223. The authors would like to thank Sina Sajadmanesh for answering questions regarding the GAP method. The authors would like to thank the anonymous reviewers and area chair for their feedback and effort, which helped to significantly improve the manuscript.
[1] J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun, "Spectral networks and locally connected networks on graphs," in International Conference on Learning Representations (ICLR2014), CBLS, April 2014, 2014, pp. http-openreview.
[2] T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks," in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/forum?id=SJU4ayYgl
[3] W. Hamilton, Z. Ying, and J. Leskovec, "Inductive representation learning on large graphs," Advances in neural information processing systems, vol. 30, 2017.
[4] P. Velicković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=rJXMpikCZ
[5] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in International conference on machine learning. PMLR, 2017, pp. 1263–1272.
[6] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 974–983.
[7] X.-M. Zhang, L. Liang, L. Liu, and M.-J. Tang, "Graph neural networks and their current applications in bioinformatics," Frontiers in Genetics, vol. 12, 2021.
[8] Z. Liu, Y. Dou, P. S. Yu, Y. Deng, and H. Peng, "Alleviating the inconsistency problem of applying graph neural network to fraud detection," in Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, 2020, pp. 1569-1572.
[9] Z. Liu, C. Chen, X. Yang, J. Zhou, X. Li, and L. Song, "Heterogeneous graph neural networks for malicious account detection," in Proceedings of the 27th ACM international conference on information and knowledge management, 2018, pp. 2077-2085.
[10] Q. Zhong, Y. Liu, X. Ao, B. Hu, J. Feng, J. Tang, and Q. He, "Financial defaulter detection on online credit payment via multi-view attributed heterogeneous information network," in Proceedings of The Web Conference 2020, 2020, pp. 785-795.