SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The notion of  $k$ -neighbor GDP with  $(\epsilon, \delta) = (0, 0)$  implies that an adversary cannot simultaneously infer information about any  $k$  in-edges and  $k$  out-edges in  $\mathbf{A}$ . As discussed in the main text, even the case  $k = 1$  already provides a similar but stronger topology privacy protection than edge GDP. Although an adversary cannot infer the existence of any edges for the  $k = 1$  case, they might be confident that there is one edge among two node pairs even though they cannot be certain which one it is. For instance, even though an adversary cannot individually infer whether  $\mathbf{A}_{12}$  and  $\mathbf{A}_{13}$  are 1 or 0, they may be confident that the event  $\{\mathbf{A}_{12} = 1 \vee \mathbf{A}_{13} = 1\}$  is true. The  $k = 2$  setting provides additional protection so that the adversary cannot simultaneously infer information about any  $k = 2$  edges. Nevertheless, the adversary may still be confident that the event  $\{\mathbf{A}_{12} = 1 \vee \mathbf{A}_{13} = 1 \vee \mathbf{A}_{14} = 1\}$  is true (in the worst case). A similar reasoning holds for general values of  $k$ . As a result, selecting an intermediate value of  $k$  may reveal some edge information, but it allows for a trade-off between the potentially sensitive edge information and model utility. It is worth noting that in many practical scenarios, edge information is less sensitive than that of node features, making this notion of privacy particularly useful. For instance, if we are satisfied with the graph structure protection from edge-level GDP, we can simply use  $k = 1$  for additional protection of sensitive node features and labels, with similar graph topology privacy.
K Additional experimental details
Datasets. We test seven benchmark datasets available from either the Pytorch Geometric library [42] or prior works. These datasets include the social network Facebook [17, 43], citation networks Cora and Pubmed [44, 45], the Amazon co-purchase networks Photo and Computers [46], and Wikipedia networks Squirrel and Chameleon [47]. Pertinent dataset statistics can be found in Table 1. We also report the class insensitive edge homophily measure defined in [39], with value in  $[0, 1]$ ; the value 1 indicates the strongest possible homophily. Its definition is as follows:
$$
\text {h o m o p h i l y} = \frac {1}{C - 1} \sum_ {c = 1} ^ {C} \max  (0, h _ {c} - \frac {| \{i : \mathbf {Y} _ {i c} = 1 \} |}{n}),
$$
$$
h _ {c} = \frac {| \{(v , w) : (v , w) \in \mathcal {E} \wedge \mathbf {Y} _ {v c} = \mathbf {Y} _ {w c} = 1 \} |}{| \mathcal {E} |}.
$$
Note that  $h_c$  represents the edge homophily ratio of nodes of class  $c$ , where the general edge homophily is defined in [52].
The cSBM model. We mainly follow the cSBM model as described in [19]. The cSBM model includes Gaussian random vector node features in addition to the classical SBM graph topology. For simplicity, we assume that there are  $C = 2$  equally sized communities with node labels  $v_{i} \in \{+1, -1\}$ . Each node  $i$  is associated with a  $f$  dimensional Gaussian vector  $b_{i} = \sqrt{\frac{\mu}{n}} v_{i}u + \frac{Z_{i}}{\sqrt{f}}$ , where  $n$  is the number of nodes,  $u \sim N(0, I / f)$ , and  $Z_{i} \in \mathbf{R}^{f}$  has independent standard normal entries. The graph in cSBM is described by an adjacency matrix  $\mathbf{A}$  defined as
$$
\mathbf {P} \left(\mathbf {A} _ {i j} = 1\right) = \left\{ \begin{array}{l l} \frac {d + \lambda \sqrt {d}}{n}, & \text {i f v _ {i} v _ {j} > 0} \\ \frac {d - \lambda \sqrt {d}}{n}, & \text {o t h e r w i s e} \end{array} \right..
$$
Similar to the classical SBM, given the node labels, the edges are independent. The symbol  $d$  stands for the average degree of the graph. Also, recall that  $\mu$  and  $\lambda$  control the strength of the information content conveyed by the node features and the graph structure, respectively.
One reason for using the cSBM to generate synthetic data is that the information-theoretic limit of the model has already been characterized in [18]. This result is summarized below.
Theorem K.1 (Informal main result from [18]). Assume that  $n, f \to \infty$ ,  $\frac{n}{f} \to \xi$  and  $d \to \infty$ . Then there exists an estimator  $\hat{v}$  such that  $\lim \inf_{n \to \infty} \frac{|\langle \hat{v}, v \rangle|}{n}$  is bounded away from 0 if and only if  $\lambda^2 + \frac{\mu^2}{\xi} > 1$ .
In our experiment, we set  $n = 10,000, f = 200$ , so that  $\xi = 50$ . We vary  $\mu$  and  $\lambda$  along the arc  $\lambda^2 + \mu^2 / \xi = 1 + \epsilon$ , for some  $\epsilon > 0$ , to ensure that we are within the allowed parameter regime. We also set  $\epsilon = 3.25$  in all our experiments.
Experiment environments. All experiments are performed on a Linux Machine with 48 cores, 376GB of RAM, and an NVIDIA Tesla P100 GPU with 12GB of GPU memory. We use PyTorch Geometric $^{3}$  [42] for graph-related operations and models, autodp $^{4}$  for privacy accounting and Opacus $^{5}$  [53] for the DP-optimizer. Our code is developed based on the GAP repository $^{6}$  [17] and follows a similar experimental pipeline.
Hyperparameters. For all methods, we set the hidden dimension to 64, and use SeLU [54] as the nonlinear activation function. The learning rate is set to  $10^{-3}$ , and do not decay the weights. Training involves 100 epochs for both pretraining and classifier modules. We use a dropout 0.5 for nonprivate and edge GDP experiments and no dropout for the node GDP and  $k$ -neighbor GDP experiments. For DPDGC, we find that row-normalizing  $\mathbf{W}^{(A)}$  to  $10^{-8}$  and reducing  $s$  accordingly gives better utility in practice (see Algorithm 2, where we set  $c = 10^{-8}$ ). Following the analysis of DPDGC, we know that this changes the sensitivity of  $\mathbf{A}\mathbf{W}^{(A)}$  to  $\sqrt{2Dc}$  for node GDP and  $\sqrt{kc}$  for  $k$ -neighbor GDP, respectively. Note that when  $c = 1$ , we recover the results of Theorem 5.3 and 5.4. For GAP, we tune the number of hops  $L \in \{1,2,3\}$ . For both DPDGC and GAP, the MLP modules have 2 layers in general, except for the  $\mathrm{MLP}_A$  and  $\mathrm{MLP}_X$  modules in DPDGC which have 1 layer. For MLP, we use 3 layers following the choice of [17]. We upper bound the out-degree by  $D = 100$  for all datasets, following the default choice stated in [17]. The batch size depends on the dataset size, where we choose 256 for Facebook and PubMed, and 64 for the rest. Note that we train without mini-batching whenever the training does not involve a DP-optimizer and the method has better performance. For the cSBM experiments, we adopted the settings used for the Facebook dataset. We observe that DPDGC (nopriv) can have severe overfitting issues for  $\phi = 0$ , which causes the results to be unstable. For this particular case, we change the learning rate of the embedding module to  $10^{-5}$  in order to mitigate the issue.
DP-optimizer. We use DP-Adam by leveraging the Opacus library. We retain the default setting for all methods.
L Pseudocode for GAP and DPDGC
<sup>3</sup>https://github.com/pyg-team/pytorch.Geometry
<sup>4</sup>https://github.com/yuxiangw/autodp
$^{5}$ https://github.com/pytorch/opacus
$^{6}$ https://github.com/sisaman/GAP
Algorithm 1 Training process of GAP
1: Model input: node feature  $\mathbf{X}$ , adjacency matrix  $\mathbf{A}$ , training labels  $\mathbf{Y}$ .  
2: Parameters: noise std  $s$ , hops  $L$ , and max degree  $D$ .  
3: if edge GDP then  
4: Train DP- $\mathrm{MLP}_X$  using a standard optimizer.  
5: else  
6: Train DP- $\mathrm{MLP}_X$  using a DP-optimizer with group size of 1.  
7: end if  
8:  $\mathbf{H} \leftarrow \mathrm{DP - MLP}_X(\mathbf{X})$ .  
9:  $\mathbf{H}^{(0)} \leftarrow$  row-normalized  $(\mathbf{H})$  and cache  $\mathbf{H}^{(0)}$ .  
10: if node or  $k$ -neighbor GDP then  
11: Subsample  $\mathbf{A}$  so that the out-degree (column-sum) is bounded by  $D$ .  
12: end if  
13: for  $1 \leq i \leq L$  do  
14:  $\mathbf{H}^{(i)} \leftarrow$  row-normalized  $(\mathbf{H}^{(i-1)} + N(0, s^2))$ .  
15: end for  
16:  $\mathbf{Z} \leftarrow \|_{i=0}^{L} \mathbf{H}^{(i)}$  and cache  $\mathbf{Z}$ .  
17: if edge GDP then  
18: Train DP- $\mathrm{MLP}_f$  using a standard optimizer.  
19: else  
20: Train DP- $\mathrm{MLP}_f$  using a DP-optimizer with a group size 1.  
21: end if
Algorithm 2 Training process of DPDGC
1: Model input: node feature  $\mathbf{X}$ , adjacency matrix  $\mathbf{A}$ , training labels  $\mathbf{Y}$ .  
2: Parameters: noise std  $s$ , maximum degree  $D$  (for node GDP) or  $k$  for  $k$ -level GDP, row-normalized constant  $c$ .  
3: if node GDP then  
4: Subsample  $\mathbf{A}$  so that the out-degree (column-sum) is bounded by  $D$ .  
5: end if  
6: if node GDP then  
7:  $x = D + 1$   
8: else if  $k$ -neighbor GDP then  
9:  $x = k + 1$   
10: else  
11:  $x = 1$   
12: end if  
13:  $(\mathbf{W}^{(A)}, \mathbf{b}) \gets$  Train DP-Emb using a DP-optimizer with group size  $x$  and constrain  $\mathbf{W}^{(A)}$  to be row-normalized to  $c$ .  
14:  $s \gets c \times s$ .  
15:  $\mathbf{Z} \gets$  row-normalized  $(\mathbf{AW}^{(A)} + N(0, s^2) + \mathbf{b})$  and cache  $\mathbf{Z}$ .  
16: if edge GDP then  
17: Train the remaining modules using a standard optimizer.  
18: else  
19: Train the remaining modules using a DP-optimizer with group size 1.  
20: end if