SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DC-CSBM With 2 Classes. Let us define the class assignments  $(\epsilon_{i})_{i\in [n]}$  as independent and identically distributed (i.i.d.) Bernoulli random variables coming from  $\mathrm{Ber}(\frac{1}{2})$ , where  $n = |\mathcal{V}|$  is the number of nodes in the graph  $\mathcal{G}$ . These class assignments divide  $n$  nodes into 2 classes:  $C_0 = \{i\in [n]:\epsilon_i = 0\}$  and  $C_1 = \{i\in [n]:\epsilon_i = 1\}$ . Assume that inter-class edge probability is  $q$  and intra-class edge probability is  $p$ , and no self-loops are allowed. For each node  $i$ , we additionally introduce a degree-correction parameter  $\theta_{i}\in (0,n]$ , which can be interpreted as the propensity of node  $i$  to connect with others. Note that to keep the DC-SBM identifiable and easier to analyze, we adopt a normalization rule to enforce the following constraint:  $\sum_{i\in C_0}\theta_i = |C_0|$ ,  $\sum_{i\in C_1}\theta_i = |C_1|$  and thus  $\sum_{i\in V}\theta_i = n$ .
Assumptions on Adjacency Matrix and Feature Matrix. Conditioning on  $(\epsilon_{i})_{i\in [n]}$ , each entry of the adjacency matrix  $\mathbf{A}$  is a Poisson random variable with  $\mathbf{A}_{ij} \sim \mathrm{Poi}(\theta_i\theta_jp)$  if  $i, j$  are in the same class and  $\mathbf{A}_{ij} \sim \mathrm{Poi}(\theta_i\theta_jq)$  if  $i, j$  are in different classes. On top of this, let  $\mathbf{X} \in \mathbb{R}^{n\times d}$  be the feature matrix where each row  $\mathbf{x}_i$  represents the node feature of node  $i$ . Assume each  $\mathbf{x}_i$  is an independent  $d$ -dimensional Gaussian random vector with  $\mathbf{x}_i \sim \mathcal{N}(\boldsymbol{\mu}, \frac{1}{d}\mathbf{I})$  if  $i \in C_0$  and  $\mathbf{x}_i \sim \mathcal{N}(\boldsymbol{\nu}, \frac{1}{d}\mathbf{I})$  if  $i \in C_1$ . We let  $\boldsymbol{\mu}, \boldsymbol{\nu} \in \mathbb{R}^d$  to be fixed  $d$ -dimensional vectors with  $\| \boldsymbol{\mu} \|_2, \| \boldsymbol{\nu} \|_2 \leq 1$ , which serve as the Gaussian mean for the two classes.
Given a particular choice of  $n, \mu, \nu, p, q$  and  $\theta = (\theta_i)_{i \in [n]}$ , we can define a class of random graphs generated by these parameters and sample a graph from such DC-CSBM as  $\mathcal{G} = (\mathbf{A}, \mathbf{X}) \sim \mathrm{DCCSBM}(n, \mu, \nu, p, q, \theta)$ .
4.3 Linear Separability After Graph Convolution
Linear Separability. Linear separability refers to the ability to linearly differentiate nodes in the two classes based on their feature vectors. Formally, for any  $\mathcal{V}_s\subseteq \mathcal{V}$ , we say that  $\{\tilde{\mathbf{x}}_i:i\in \mathcal{V}_s\}$  is linearly separable if there exists some unit vector  $\mathbf{v}\in \mathbb{R}^d$  and a scalar  $b$  such that  $\mathbf{v}^{\top}\tilde{\mathbf{x}}_i + b < 0$ ,  $\forall i\in C_0\cap \mathcal{V}_s$  and  $\mathbf{v}^{\top}\tilde{\mathbf{x}}_i + b > 0$ ,  $\forall i\in C_1\cap \mathcal{V}_s$ . Note that linear separability is closely related to GNN performance. Intuitively, more nodes being linearly separable will lead to better GNN performance.
Degree-Thresholded Subgroups of  $C_0$  and  $C_1$ . To better control the behavior of graph convolution operation, we will focus on particular subgroups of  $C_0$  and  $C_1$  where the member nodes having degree-corrected factor larger or equal to a pre-defined threshold  $\alpha > 0$ . Slightly abusing the notations, we denote these subgroups as  $C_0(\alpha)$  and  $C_1(\alpha)$ , which are formally defined below.
Definition 4.1 ( $\alpha$ -Subgroups). Given any  $\alpha \in (0, n]$ , define  $\alpha$ -subgroups of  $C_0$  and  $C_1$  as follows:
$$
C _ {0} (\alpha) = \{j \in [ n ]: \theta_ {j} \geq \alpha a n d j \in C _ {0} \},
$$
$$
C _ {1} (\alpha) = \{j \in [ n ]: \theta_ {j} \geq \alpha a n d j \in C _ {1} \}.
$$
Let  $\mathcal{V}_{\alpha} := C_0(\alpha) \cup C_1(\alpha)$ , we are interested in analyzing the linear separability of the node representations after the graph convolution operation, namely  $\{\tilde{\mathbf{x}}_i : i \in \mathcal{V}_{\alpha}\}$ . Recall that for each node  $i$ ,  $\tilde{\mathbf{x}}_i = \frac{1}{\deg(i)} \sum_{j \in \mathcal{N}(i)} \mathbf{x}_j$ , where  $\mathcal{N}(i)$  is the set of neighbors of node  $i$ .
Relationship Between  $\alpha$  and Linear Separability. We first make the following assumptions about the DC-CSBM, closely following the assumptions made by Baranwal et al. [3].
Assumption 4.2 (Graph Size). Assume the relationship between the graph size  $n$  and the feature dimension  $d$  follows  $\omega(d \log d) \leq n \leq O(poly(d))$ .
Assumption 4.3 (Edge Probabilities). Define  $\Gamma(p, q) \coloneqq \frac{p - q}{p + q}$ . Assume the edge probabilities  $p, q$  satisfy  $p, q = \omega (\log^2(n)/n)$  and  $\Gamma(p, q) = \Omega(1)$ .
Theorem 4.4 asserts that if the threshold  $\alpha$  is not too small, then the set  $\mathcal{V}_{\alpha} = C_0(\alpha) \cup C_1(\alpha)$  can be linear separated with high probability. The proof of Theorem 4.4 can be found in Appendix C.
Theorem 4.4 (Linear Separability of  $\alpha$ -Subgroups). Suppose that Assumption 4.2 and 4.3 hold. For any  $(\mathbf{X},\mathbf{A})\sim DC-CSBM(n,\pmb {\mu},\pmb {\nu},p,q,\theta)$ , if  $\alpha = \omega$ $\left(\max \left(\frac{1}{\log n},\frac{\log n}{dn(p + q)\|\pmb{\mu} - \pmb{\nu}\|_2^2}\right)\right)$ , then
$$
\mathbb {P} \left(\left\{\tilde {\mathbf {x}} _ {i}: i \in \mathcal {V} _ {\alpha} \right\} \text {i s l i n e a r l y s e p a r a b l e}\right) = 1 - o _ {d} (1),
$$
where  $o_d(1)$  is a quantity that converges to 0 as  $d$  approaches infinity.
Note that Theorem 4.4 suggests that, when the heterogeneity of node degrees is taken into consideration, the nodes with degrees exceeding a threshold  $\alpha$  are more likely to be linearly separable. And the requirement for the threshold  $\alpha$  depends on the DC-CSBM parameters:  $n,p,q,\mu ,\nu$
Remark 4.5. If we let  $p, q \in \Theta\left(\frac{\log^3 n}{n}\right)$  and  $\| \boldsymbol{\mu} - \boldsymbol{\nu} \|_2$  be fixed constant, then the requirement can be reduced to  $\alpha \in \omega\left(\frac{1}{\log n}\right)$ , which is not too large. Given this particular setting and reasonable selection of  $p, q$ , the regime of acceptable  $\alpha$  is broad and thus demonstrates the generalizability of Theorem 4.4.
4.4 Implications on Gini-Degree
Finally, we qualitatively discuss the relationship between Gini-Degree and GNNs' performance using the results from Theorem 4.4. For any  $\alpha >0$  that meets the criteria in the statement, we can consider,
1. Negative correlation between Gini-Degree and the size of  $\mathcal{V}_{\alpha}$ : If the number of nodes and edges is fixed, a higher Gini-Degree implies more high-degree nodes in the network and thus the majority of nodes are receiving lower degrees. Clearly, if most of the nodes have lower degrees, then there will be fewer nodes having degrees exceeding a certain threshold proportional to  $\alpha^1$  and being placed in  $\mathcal{V}_{\alpha}$ . Hence, a dataset with a higher (or lower) Gini-Degree will lead to a smaller (or larger) size of  $\mathcal{V}_{\alpha}$ .
2. Positive correlation between the size of  $\mathcal{V}_{\alpha}$  and model performance: Intuitively, the GNN performance tends to be better if there are more nodes that can be linearly separable after graph convolution. Consequently, the GNN performance is positively relevant to the size of  $\mathcal{V}_{\alpha}$  corresponding to the minimum possible  $\alpha$ .
Combining the two factors above, our analysis suggests that Gini-Degree tends to have a negative correlation with GNNs' performance.
5 Controlled Experiment on Gini-Degree
To further verify whether there is a causal relationship between the degree distribution of graph data (in particular, measured by Gini-Degree) and the GNN performance, we conduct a controlled experiment using synthetic graph datasets.
Experiment Setup. We first generate a series of synthetic graph datasets using the GraphWorld library [31]. To investigate the causal effect of Gini-Degree, we manipulate the data generation parameters to obtain datasets with varying Gini-Degree while keeping a bunch of other properties fixed. Specifically, we use the SBM generator in GraphWorld library and set the number of nodes  $n = 5000$ , the average degree as 30, the number of clusters as 4, cluster size slope as 0.5, feature center distance as 0.5, the edge probability ratio  $p / q = 4.0$ , feature dimension as 16, feature cluster variance as 0.05. The parameters above are fixed throughout our experiments, and their complete definition can be found in the Appendix. By manipulating the power law exponent parameter of the generator, we obtain five synthetic datasets with Gini-Degree as 0.906, 0.761, 0.526, 0.354, and 0.075, respectively.
Then we train the same set of GNN models and MLP model as mentioned in Table 1 on each dataset. We randomly split the nodes into training, validation, and test sets with a ratio of 3:1:1. We closely follow the hyperparameters and the training protocol in the GLI library [27], which is where we obtain the metadata in Section 3. We run five independent trials with different random seeds.
Experiment Results. The experiment results are shown in Table 2. We observe an evident monotonically decreasing trend for the performance of the graph-based models, GCN, GAT, GraphSAGE, MoNet, MixHop, and LINKX, as Gini-Degree increases. However, there is no clear pattern for the non-graph model, MLP. This result suggests that these widely-used GNN models are indeed sensitive to Gini-Degree, which validates our result of sparse regression analysis. Note that MLP does not take the graph structure into consideration, and hence the degree distribution has less influence
<sup>1</sup>Note that the expected value of the degree of node  $i$  is proportional to  $\theta_{i}$  when we ignore self-loops. (See Appendix C for more information.) Thus, the lower bound  $\alpha$  on degree-corrected factors can be translated to the lower bound  $n(p + q)\alpha$  on degrees.
Table 2: Controlled experiment results for varying Gini-Degree. Standard deviations are derived from 5 independent runs. The performances of all models except for MLP have an evident negative correlation with Gini-Degree.
Gini-Degree	GCN	GAT	GraphSAGE	MoNet	MixHop	LINKX	MLP
0.906	0.798±0.004	0.659±0.01	0.76±0.005	0.672±0.002	0.804±0.005	0.832±0.002	0.595±0.006
0.761	0.817±0.001	0.732±0.005	0.818±0.004	0.696±0.015	0.817±0.004	0.849±0.002	0.756±0.002
0.526	0.874±0.004	0.742±0.006	0.825±0.013	0.8±0.028	0.826±0.003	0.853±0.002	0.655±0.005
0.354	0.906±0.002	0.737±0.008	0.857±0.008	0.83±0.013	0.837±0.002	0.867±0.002	0.66±0.07
0.075	0.948±0.002	0.746±0.005	0.878±0.002	0.92±0.002	0.84±0.002	0.893±0.001	0.705±0.002
on the performance of MLP. The result on MLP also indicates that we have done a reasonably well-controlled experiment.
In this work, we propose a novel metadata-driven approach that can efficiently identify critical graph data properties influencing the performance of GNNs. This is a significant contribution given the diverse nature of graph-structured data and the sensitivity of GNN performance to these specific properties. We also verify the effectiveness of the proposed approach through an in-depth case study around one identified salient graph data property.
As a side product, this paper also highlights the considerable impact of the degree distribution, a salient data property identified through our metadata-driven regression analysis, on the GNN performance. We present a novel theoretical analysis and a carefully controlled experiment to demonstrate this impact.
The authors would like to thank Pingbang Hu for the feedback on the draft.
[1] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning, pages 21–29. PMLR, 2019.
[2] Robert J Adler, Jonathan E Taylor, et al. Random fields and geometry, volume 80. Springer, 2007.
[3] Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Graph convolution for semi-supervised classification: Improved linear separability and out-of-distribution generalization. arXiv preprint arXiv:2102.06966, 2021.
[4] Norbert Binkiewicz, Joshua T Vogelstein, and Karl Rohe. Covariate-assisted spectral clustering. Biometrika, 104(2):361-377, 2017.
[5] Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021.
[6] Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318, 2020.
[7] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pages 1725-1735. PMLR, 2020.
[8] Yash Deshpande, Subhabrata Sen, Andrea Montanari, and Elchanan Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems, 31, 2018.
[44] Sukwon Yun, Kibum Kim, Kanghoon Yoon, and Chanyoung Park. Lte4g: Long-tail experts for graph neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 2434-2443, 2022.
[45] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-7804, 2020.
[46] Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11168-11176, 2021.
A Definitions of Dataset Properties
We introduce the formal definitions of the dataset properties mentioned in Section 3.2. Following the definitions in Section 4.1, we further define  $n = |\mathcal{V}|$  and  $m = |\mathcal{E}|$  to denote the number of nodes and edges of graph  $\mathcal{G}$ . Also, in the context of the node classification task, we define  $\mathcal{Y} \in \mathbb{R}^n$  as the vector of node labels and  $C$  as the number of classes.
A.1 Basic
Edge Density: The edge density for an undirected graph is calculated as  $\frac{2m}{n(n - 1)}$ , while for a directed graph, it is computed as  $\frac{m}{n(n - 1)}$ .
Average Degree: The average degree for an undirected graph is defined as  $\frac{2m}{n}$ , while for a directed graph, it is defined as  $\frac{m}{n}$ .
Degree Assortativity: The degree assortativity is the average Pearson correlation coefficient of all pairs of connected nodes. It quantifies the tendency of nodes in a network to be connected to nodes with similar or dissimilar degrees and ranges between -1 and 1.
A.2 Distance
Pseudo Diameter: The pseudo diameter is an approximation of the diameter of a graph and provides a lower bound estimation of its exact value.
A.3 Connectivity
Relative Size of Largest Connected Component (RSLCC): The relative size of the largest connected component is determined by calculating the ratio between the size of the largest connected component and  $n$ .
A.4 Clustering
Average Clustering Coefficient (ACC): First define  $T(u)$  as the number of triangles including node  $u$ , then the local clustering coefficient for node  $u$  is calculated as  $\frac{2}{\deg(u)(\deg(u) - 1)} T(u)$  for undirected graph, where  $\deg(u)$  is the degree of node  $u$ ; and is calculated as  $\frac{2}{\deg^{tot}(u)(\deg^{tot}(u) - 1) - 2\deg^{\leftrightarrow}(u)} T(u)$  for directed graph, where  $\deg^{tot}(u)$  is the sum of in-degree and out-degree of node  $u$  and  $\deg^{\leftrightarrow}(u)$  is the reciprocal degree of  $u$ . The average clustering coefficient is then defined as the average local clustering coefficient of all the nodes in the graph.
Transitivity: The transitivity is defined as the fraction of all possible triangles present in the graph. Formally, it can be written as  $3^{\# \text{triangles}}$ , where a triad is a pair of two edges with a shared vertex.
Degeneracy: The degeneracy is determined as the least integer  $k$  such that every induced subgraph of the graph contains a vertex with its degree smaller or equal to  $k$ .
A.5 Degree Distribution
Gini Coefficient of Degree Distribution (Gini-Degree): The Gini coefficient of the node degrees of the graph.
A.6 Attribute
Edge Homogeneity [31]: The edge homogeneity is defined as the ratio of edges whose endpoints have the same node labels.
In-Feature Similarity [31]: First define within-class angular feature similarity as 1—angular_distance( $\mathbf{x}_i$ ,  $\mathbf{x}_j$ ) for an edge  $(i, j)$  with its endpoints  $i$  and  $j$  have the same node labels. In-Feature Similarity is the average within-class angular feature similarity of all such edges in the graph.
Out-Feature Similarity [31]: First define between-class angular feature similarity as 1- angular_distance  $(\mathbf{x}_i,\mathbf{x}_j)$  for an edge  $(i,j)$  with its endpoints  $i$  and  $j$  have different node labels. Out-Feature Similarity is the average between-class angular feature similarity of all such edges in the graph.
Feature Angular SNR [31]: The feature angular SNR is computed as the ratio between in-feature similarity and out-feature similarity.
Homophily Measure [23]: The homophily measure is defined as
$$
\hat {h} = \frac {1}{C - 1} \sum_ {k = 1} ^ {C} \left[ h _ {k} - \frac {\left| C _ {k} \right|}{n} \right] _ {+},
$$
where  $[a]_{+} = \max(a, 0)$ ,  $|C_k|$  is the total number of nodes having their label  $k$  and  $h_k$  is the class-wise homophily metric defined below,
$$
h _ {k} = \frac {\sum_ {u : \mathcal {Y} _ {u} = k} d _ {u} ^ {\left(\mathcal {Y} _ {u}\right)}}{\sum_ {u : \mathcal {Y} _ {u} = k} d _ {u}},
$$
where  $d_u$  is the number of neighbors of node  $u$  and  $d_u^{(y_u)}$  is the number of neighbors of node  $u$  having the same node label.
Attribute Assortativity: The attribute assortativity is the average Pearson correlation coefficient of all pairs of connected nodes. It quantifies the tendency of nodes in a network to be connected to nodes with the same or different attributes (here node label) and ranges between -1 and 1.
B Experiment Setup for Obtaining Metadata
In this section, we describe more details of the experimental setup to obtain GNNs' performance that we use in Section 3.2, mostly following Ma et al. [27]. For completeness, we list down the model setting used by them in the following paragraphs.
GCN [19], GAT [37], GraphSAGE [13], MoNet [29], MLP, and MixHop [1] are set to have two layers with hidden dimension equals to 8. For LINKX [23],  $MLP_{A}$ ,  $MLP_{X}$  are set to be a one-layer network and  $MLP_{f}$  to be a two-layers network, following the setting in Lim et al. [23].
For the rest of the training settings, we adopt the same configuration for all experiments. Specifically, we set learning rate  $= 0.01$ , weight decay  $= 0.001$ , dropout rate  $= 0.6$ , max epoch  $= 10000$ , and batch size  $= 256$ . We use Adam [18] as an optimizer for all models except LINKX. AdamW [26] is used with LINKX in order to comply with Lim et al. [23]. For datasets with binary labels (i.e., penn94, pokec, genius, and twitch-gamers), we choose the ROC AUC score as the evaluation metric; while for other datasets, we use test accuracy instead.
We also let all the detailed model settings remain consistent with the same with Ma et al. [27]. Namely,
- GAT: Number of heads in multi-head attention  $= 8$ . LeakyReLU angle of negative slope  $= 0.2$ . No residual is applied. The dropout rate on attention weight is the same as the overall dropout.
GraphSAGE: Aggregator type is GCN. No norm is applied.
- MoNet: Number of kernels = 3. Dimension of pseudo-coordinte = 2. Aggregator type = sum.
- MixHop: List of powers of adjacency matrix  $= [1,2,3]$ . No norm is applied. Layer Dropout rate  $= 0.9$
LINKX: No inner activation.
C Proof of Theorem 4.4
Proof Sketch. To prove Theorem 4.4, we first show that degree and the neighborhood distribution of each node concentrate with high probability. Then we claim that the node features after the
convolution operation will be centered around specific mean values, depending on the node classes. Finally, we demonstrate that the nodes in different classes can be linearly separated by the hyperplane passing through the mid-point of the two mean values  $\mu, \nu$  with high probability.
We prove the intermediate results in Lemma C.4 (degree and neighborhood distribution concentration inequalities) by utilizing Lemma C.5 (Chernoff bound for Poisson random variable) and in Lemma C.6 (convoluted feature concentration) by making use of Lemma C.7 (Borell's inequality). Finally, given the requirement of  $\alpha$  stated in Theorem 4.4, we argue that the convoluted node features in two classes can be linearly separated with a high level of confidence.
Novelty of our Proof. The general structure of our proof follows that of Baranwal et al. [3]. However, our analysis requires non-trivial adaptation of the proof by Baranwal et al. [3]. This is because we have a more general data model, DC-CSBM, where the CSBM assumed by Baranwal et al. [3] is a restricted special case of ours. In particular, we assume each edge is generated by the Poisson random variable following DC-SBM, instead of the Bernoulli random variable assumed by CSBM; we also incorporate the degree-corrected factor in our analysis to model node degree heterogeneity within communities, which gives us the flexibility to discuss linear separability for subgraphs with different levels of sparsity.
Before we state and prove Lemma C.4, let us first define the following events that we will work on.
Definition C.1 (Class Size Concentration). For any  $\delta > 0$ , define
$$
\mathbf {I} _ {1} (\delta) = \left\{\frac {n}{2} (1 - \delta) \leq | C _ {0} |, | C _ {1} | \leq \frac {n}{2} (1 + \delta) \right\}.
$$
Definition C.2 (Degree Concentration). For any  $\delta' > 0$  and for each node  $i \in [n]$ , define
$$
\mathbf {I} _ {2, i} (\delta^ {\prime}) = \left\{\frac {1}{2} (p + q) (1 - \delta^ {\prime}) \theta_ {i} \leq \frac {D _ {i i}}{n} \leq \frac {1}{2} (p + q) (1 + \delta^ {\prime}) \theta_ {i} \right\}.
$$
Definition C.3 (Neighborhood Distribution Concentration). For any  $\delta' > 0$  and for each node  $i \in [n]$ , define
$$
\begin{array}{l} \mathbf {I} _ {3, i} \left(\delta^ {\prime}\right) = \left\{\frac {(1 - \epsilon_ {i}) p + \epsilon_ {i} q}{p + q} \left(1 - \delta^ {\prime}\right) \leq \frac {\left| C _ {0} \cap \mathcal {N} _ {i} \right|}{D _ {i i}} \leq \frac {(1 - \epsilon_ {i}) p + \epsilon_ {i} q}{p + q} \left(1 + \delta^ {\prime}\right) \right\} \\ \bigcap \left\{\frac {(1 - \epsilon_ {i}) q + \epsilon_ {i} p}{p + q} (1 - \delta^ {\prime}) \leq \frac {| C _ {1} \cap \mathcal {N} _ {i} |}{D _ {i i}} \leq \frac {(1 - \epsilon_ {i}) q + \epsilon_ {i} p}{p + q} (1 + \delta^ {\prime}) \right\}, \\ \end{array}
$$
where  $\mathcal{N}_i$  denotes the set of nodes connected to node  $i$ .
Then in Lemma C.4, we argue that for nodes in the  $\alpha$ -subgroup defined in 4.1 for some appropriately chosen  $\alpha > 0$ , the above events will happen simultaneously with high probability.
Lemma C.4 (Concentration Inequalities). Given  $\alpha \in (\frac{1}{\log n}, n]$ ,  $C_0(\alpha), C_1(\alpha)$  defined by Definition 4.1, and  $\mathcal{V}_{\alpha} = C_0(\alpha) \cup C_1(\alpha)$ . Let  $\delta = n^{-1/2 + \epsilon}$  and  $\delta' = (\alpha \log n)^{-1/2 + \epsilon}$ , then for  $\epsilon > 0$  small enough, we have for any  $c > 0$ , there is some  $C > 0$  such that
$$
\mathbb {P} \left(\mathbf {I} _ {1} (\delta) \bigcap_ {i \in \mathcal {V} _ {\alpha}} \mathbf {I} _ {2, i} \left(\delta^ {\prime}\right) \bigcap_ {i \in \mathcal {V} _ {\alpha}} \mathbf {I} _ {3, i} \left(\delta^ {\prime}\right)\right) \geq 1 - \frac {C}{n ^ {c}}.
$$
Proof. Firstly, we consider the event  $\mathbf{I}_1(\delta)$ . Since  $(\epsilon_i)_{i\in [n]}\sim \mathrm{Ber}(\frac{1}{2})$ , by the Hoeffding's inequality for independent Bernoulli random variables [38, Theorem 2.2.6], we have for any  $\delta >0$  that