SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qizhang Feng $^{1}$ , Zhimeng Jiang $^{1}$ , Ruiquan Li $^{2}$ , Yicheng Wang $^{1}$ , Na Zou $^{1}$ , Jiang Bian $^{3}$ , Xia Hu $^{4}$ $^{1}$ Texas A&M University,  $^{2}$ University of Science and Technology of China,  $^{3}$ University of Florida,  $^{3}$ Rice University
As graph neural networks (GNNs) struggle with large-scale graphs due to high computational demands, graph data distillation promises to alleviate this issue by distilling a large real graph into a smaller distilled graph while maintaining comparable prediction performance for GNNs trained on both graphs. However, we observe that GNNs trained on distilled graphs may exhibit more severe group fairness issues than GNNs trained on real graphs for vanilla and fair GNNs training. Motivated by these observations, we propose fair graph distillation (FGD), an advanced graph distillation approach to generate fair distilled graphs. The challenge lies in the deficiency of sensitive attributes for nodes in the distilled graph, making most debiasing methods (e.g., regularization and adversarial debiasing) intractable for distilled graphs. We develop a simple yet effective bias metric, named coherence, for distilled graphs. Based on the proposed coherence metric, we introduce a framework for fair graph distillation using a bi-level optimization algorithm. Extensive experiments demonstrate that the proposed algorithm can achieve better prediction performance-fairness trade-offs across various datasets and GNN architectures.
Real-world data, like chemical molecules, social networks, and transportation networks, can be represented as graphs [Han et al., 2022a, Ling et al., 2023a, Jiang et al., 2022a, Ying et al., 2018, Ling et al., 2023b, Tong et al., 2020, Han et al., 2022b]. Graph neural networks (GNNs) excel at capturing structural information but struggle with large-scale graphs due to memory consumption and computational expense caused by the neighborhood explosion problem[Hamilton et al., 2017, Liu et al., 2023c]. This cost becomes unaffordable in situations requiring repeated GNN training, such as neural architecture search and continual learning [Liu et al., 2018, Zhou et al., 2019, Li and Hoiem, 2017, Liu et al., 2023b]. Dataset distillation is a promising solution to address computation challenges by generating small, informative distilled data for neural network training in downstream tasks [Jin et al., 2021, 2022, Zhao et al., 2021a,b, Nguyen et al., 2021]. Techniques like dataset condensation [Zhao et al., 2021b, Jin et al., 2021] can significantly reduce training data size without major performance degradation in the image and graph domains. However, focusing solely on prediction performance may introduce fairness issues, as sensitive information can be condensed into distilled data for prediction. A natural question is raised: Is the model trained on the distilled graph fair, and if not, how can we achieve fair graph distillation?
In this work, we focus on the group fairness<sup>1</sup> for node classification tasks under binary sensitive attribute setting. We discover that GNNs trained on distilled small graphs exhibit more severe group fairness issues than those on real graphs. In other words, graph distillation can even amplify graph
<sup>1</sup>Group fairness ensures equitable treatment of diverse demographic groups by algorithms Mehrabi et al. [2021]. Such as in mortality prediction, issues arise when true positive rates significantly differ between sensitive groups. Group fairness metrics will be introduced in Section 5.1.
data bias, which challenges the applicability of graph distillation in high-stake applications [Mehrabi et al., 2021, Suresh and Guttag, 2019]. To this end, we propose a fair graph distillation framework to offer a significantly reduced graph size and also better utility-fairness trade-off while maintaining predictive performance.
Many debias methods explicitly use sensitive attributes, but these are inherently missing in distilled graphs because they are excluded from the data attributes and the meaning of the attributes may change during the optimization process. In this paper, we point out the relationship between the space of real graphs and the space of distilled graphs and develop a simple estimator of sensitive attributes and introduce a bias measurement called consistency. We then propose a bi-level optimization algorithm for fair graph distillation: the outer loop generates a fair and informative distilled graph using gradient matching and coherence loss, while GNNs train on distilled graphs in the inner loop. In a nutshell, the contributions can be summarized as follows:
- To our knowledge, this is the first paper to identify group fairness issues in conventional graph distillation methods with binary sensitive attributes, motivating the formulation of a fair graph distillation problem in node classification tasks.
- We discover the relationship between the space of real graphs and the space of distilled graphs. We develop a bias metric called coherence for distilled graphs and propose a bi-level optimization framework using this metric to achieve fair graph distillation.
- We perform extensive experiments on various real-world datasets and GNN architectures to validate the effectiveness of the proposed FGD algorithm. Results demonstrate that FGD achieves a better accuracy-fairness trade-off compared to vanilla graph distillation methods and numerous baselines.
We consider node classification tasks given a graph dataset  $\mathcal{G} = \{\pmb {A},\pmb {X},\pmb {Y},\pmb {S}\}$  with  $N$  nodes. Here,  $\pmb {A}\in \{0,1\}^{N\times N}$  is the adjacency matrix, and  $A_{ij} = 1$  represents there exists an edge between node  $i$  and  $j$ .  $\pmb {X}\in \mathbb{R}^{N\times D}$  is the node feature matrix, where  $D$  is non-sensitive feature dimension for each node.  $\pmb {Y}\in \{0,1,\dots ,C - 1\} ^N$  denotes the node labels over  $C$  classes. For simplicity, we consider a binary sensitive attribute2  $\pmb {S}\in \{0,1\} ^N$  .  $\Pi^s$  is the sensitive membership diagonal matrix.  $\Pi_{ii}^{s} = 1$  if and only if  $i$  th node belongs to sensitive group  $s$  . The distilled small graph dataset is marked as  $\mathcal{G}' = \{\pmb {A}',\pmb {X}',\pmb {Y}'\}$  which contains  $N^{\prime}$  nodes and  $N^{\prime}\ll N$  . Note that elements of the distilled adjacency matrix satisfy  $A_{ij}^{\prime}\in [0,1]$  and no sensitive attributes exist in  $\mathcal{G}'$  . The latent node representation of real graph is  $\pmb{Z}$  , and the span space of it is  $\operatorname {span}(\pmb {Z})\coloneqq \left\{\sum_{i = 1}^{N}w_{i}\pmb{z}_{i}|1\leq i\leq N,w_{i}\in \mathbf{R}\right\}$  . Similar definition of  $\pmb{Z}'$  and  $\operatorname {span}(Z^{\prime})$  for distilled graph.
Figure 1: AUC and  $\Delta_{DP}$  of the GNN trained on real graph data and distilled graph data. Both utility and fairness performance deteriorates after vanilla graph distillation.
2.2 Graph Distillation via Gradient Matching
The purpose of graph distillation is to generate a distilled graph  $\mathcal{G}'$  such that the GNN model, denoted as  $\mathrm{GNN}_{\theta}$  with parameters  $\theta$ , trained on distilled graph performs comparably to the model trained on the real graph  $\mathcal{G}$ . The objective can be formulated as the following bi-level optimization problem:
$$
\min  _ {\mathcal {G} ^ {\prime}} \mathcal {L} \left(\mathrm {G N N} _ {\theta^ {\mathcal {G} ^ {\prime}}} \left(\boldsymbol {A}, \boldsymbol {X}\right), \boldsymbol {Y}\right) \text {s . t} \theta^ {\mathcal {G} ^ {\prime}} = \underset {\theta} {\arg \min } \mathcal {L} \left(\mathrm {G N N} _ {\theta} \left(\boldsymbol {A} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right), \boldsymbol {Y} ^ {\prime}\right) \tag {1}
$$
where  $\theta^{\mathcal{G}'}$  denotes the optimal parameter trained on distilled small graph  $\mathcal{G}'$ , and  $\mathcal{L}(\cdot, \cdot)$  denotes the loss function. To tackle the above optimization problem, the gradient matching method Zhao et al.
2The sensitive attribute  $S$  represents the attribute that the respondents do not want to be disclosed, such as gender or age. Sensitive attribute  $S$  is not included in the normal features  $X$ .
[2021a] is proposed. The intuition is to let the GNN parameters  $\theta^{\mathcal{G}'}$  trained on distilled graph follow a similar path to the GNN parameters  $\theta^{\mathcal{G}}$  trained on the real graph during model optimization. The gradient of the GNN parameters is forced to be the same over the real and distilled graphs:
$$
\min  _ {\mathcal {G} ^ {\prime}} \left[ \sum_ {t = 0} ^ {T - 1} D \left(\nabla_ {\theta} \mathcal {L} (\mathcal {G}), \nabla_ {\theta} \mathcal {L} \left(\mathcal {G} ^ {\prime}\right)\right) \right], \tag {2}
$$
where  $D(\cdot, \cdot)$  is a distance function,  $T$  is the number of steps of model parameters trajectory, and  $\theta_t^G$ ,  $\theta_t^{G'}$  denotes the model parameters trained on  $\mathcal{G}$  and  $\mathcal{G}'$  at time step  $t$ , respectively. The gradient calculated on  $\mathcal{G}$  and  $\mathcal{G}'$  is denoted as  $\nabla_{\theta} \mathcal{L}(\mathcal{G}) \coloneqq \nabla_{\theta} \mathcal{L}(\mathrm{GNN}_{\theta_t}(A, X), Y)$  and  $\nabla_{\theta} \mathcal{L}(\mathcal{G}') \coloneqq \nabla_{\theta} \mathcal{L}(\mathrm{GNN}_{\theta_t}(A', X'), Y')$ , respectively.
3 Bias Measurement for Distilled Graph
In this section, we empirically demonstrate the fairness issue in the distilled graph. Motivated by this, we pursue fair graph distillation. Although distilled graphs lack sensitive attributes  $S'$ , we observe that between the node representations of the real and distilled graphs: their barycenter remain consistent, and their spaces are also consistent. We leverage this phenomenon to develop a simple, effective bias measurement for distilled graphs.
3.1 Is Graph Distillation Really Fair?
Our empirical investigation assesses the fairness of graph distillation across various datasets and architectures. We compare the utility (AUC) and fairness (demographic parity  $(\Delta_{DP})$  [Beutel et al., 2017]) of GNNs trained on real graphs and those trained on distilled graphs created by the vanilla graph distillation method. The utility and fairness performance are shown in Figure 1. We can find that: For datasets like Pokec-n, German, and Credit, distilled graph-based GNNs have higher  $\Delta_{DP}$  and lower AUC performance, suggesting a compromise in fairness and prediction performance. We also notice that, for Pokec-z and Recidivism datasets, these GNNs exhibit lower  $\Delta_{DP}$  and significantly lower AUC performance (shown in Table 1), indicating a trade-off between improved fairness and reduced prediction performance. We observe similar results when using other fair GNN models. More details can be found in Appendix E. Motivated by these observations, we aim to find a better prediction performance and fairness trade-off via chasing the fair graph distillation method.
3.2 Geometric Connections in Data Distillation
The distilled data can be generated via minimizing gradient distance in Equation 2. To simplify the analysis, we consider  $D(\cdot ,\cdot)$  as Euclidean distance and those model parameters during optimization trajectory satisfying  $\theta \sim \mathcal{P}_{\theta}$ , where  $\mathcal{P}_{\theta}$  is certain but unknown parameters' distribution. Therefore, the objective can be transformed as
$$
\min  _ {\mathcal {G} ^ {\prime}} \mathbb {E} _ {\theta \sim \mathcal {P} _ {\theta}} \left[ \left| \left| \nabla_ {\theta} \mathcal {L} (\mathcal {G}) - \nabla_ {\theta} \mathcal {L} \left(\mathcal {G} ^ {\prime}\right) \right| \right| ^ {2} \right]. \tag {3}
$$
We consider three assumptions of the model parameters' distribution and the convergence for loss minimization.
Assumption 3.1 (Model parameters' distribution). We assume that each model parameter in the last softmax layer satisfies the same distribution.
Assumption 3.2 (Loss minimization). We assume that exists at least one distilled dataset that minimizes Equation 3.
Figure 2: Geometric intuition of sensitive attribute estimation. The projection distance indicates the extent to which the node belongs to the sensitive group. (a) Unfair node representations have a large coherence bias. (b) Fair node representations have a small coherence bias.
Theorem 3.3 (Consistent Span Space). We empirically show that  $\operatorname{span}(\mathbf{Z}') \approx \operatorname{span}(\mathbf{Z})$  via calculating the principle angles between them. We also provide the rigorous proof of  $z' \in \operatorname{span}(\mathbf{Z})$  under distribution matching in Appendix D.
Theorem 3.4 (Consistent Geometric Barycenters). Under Assumptions 3.1 and 3.2, the barycenter of the last representation for the optimal distilled graph and the real graphs are consistent, i.e.  $\frac{1}{N}\sum_{i=1}^{N}z_i = \frac{1}{N'}\sum_{i=1}^{N'}z_i'$ . Please see proof in Appendix B.
3.3 Sensitive Attribute Estimation
The consistent span space and geometric barycenter suggest that we can estimate sensitive attributes from the representations of both distilled and real graphs. We frame sensitive attribute estimation as a classification problem: Given a data representation  $z' \in Z'$ , what is the probability that  $z'$  belongs to the sensitive group?
Ridge regression for distance measurement. Notice that the representation of each sensitive group for the real graph is known, we define  $Z_0$  and  $Z_1$  as the representation matrix for sensitive group  $s = 0$  and  $s = 1$ . To measure the probability that  $z'$  belongs to these two sensitive groups, we first find the closest vector  $z_{proj}' = Z_s^\top q \in \operatorname{Span}(Z_s)$  to approximate the representation  $z'$ , and then use the norm of  $z' - z_{proj}'$  to measure the distance between  $z'$  and sensitive group  $Z_s$ . Specifically, we adopt ridge regression to find the optimal coefficient vector  $q*$ , which can be formulated as
$$
\operatorname {D i s t} \left(\boldsymbol {z} ^ {\prime}, \boldsymbol {Z} _ {s}\right) = \left\| \boldsymbol {z} ^ {\prime} - \boldsymbol {Z} _ {s} ^ {\top} \boldsymbol {q} ^ {*} \right\| _ {2} \tag {4}
$$
$$
\operatorname {s. t} \boldsymbol {q} ^ {*} = \underset {\boldsymbol {q}} {\arg \min } \gamma \| \boldsymbol {z} ^ {\prime} - \boldsymbol {Z} _ {s} ^ {\top} \boldsymbol {q} \| _ {2} ^ {2} + \| \boldsymbol {q} \| _ {2} ^ {2}, \tag {5}
$$
where  $\gamma$  is the hyperparameter for ridge regression. For the optimal  $q^{*}$ , we have
$$
\boldsymbol {p} ^ {s} = \boldsymbol {z} ^ {\prime} - \boldsymbol {Z} _ {s} ^ {\top} \boldsymbol {q} ^ {*} = \boldsymbol {z} ^ {\prime} - \gamma \boldsymbol {Z} _ {s} ^ {\top} \left(\boldsymbol {I} + \gamma \boldsymbol {Z} _ {s} \boldsymbol {Z} _ {s} ^ {\top}\right) ^ {- 1} \boldsymbol {Z} _ {s} \boldsymbol {z} ^ {\prime}, \tag {6}
$$
where  $\top$  represents matrix transpose,  $p^s$  is approximately the projection onto the orthogonal complement of the subspace  $\operatorname{span}(Z_s)$ . The proof is in Appendix C.
Sensitive attribute soft estimation. Since  $\pmb{p}^s = \pmb{z}' - \gamma \pmb{Z}_s^\top (\pmb{I} + \gamma \pmb{Z}_s \pmb{Z}_s^\top)^{-1} \pmb{Z}_s \pmb{z}'$  can be viewed as approximately the projection of  $\pmb{z}$  onto the orthogonal complement of sensitive group  $\pmb{Z}_s$ ,  $\| \pmb{p}^s \|_2$  is small if  $\pmb{z}$  is in sensitive group  $\pmb{Z}_s$  and large otherwise. The probability of the given data representation  $\pmb{z}$  belongs to the sensitive group  $\pmb{Z}_s$  can further be inferred via a softmax function:
$$
\pi^ {s} \left(\boldsymbol {z} ^ {\prime}\right) = \frac {\exp (- \lambda \| \boldsymbol {p} ^ {s} \| _ {2})}{\sum_ {s = 0} ^ {1} \exp (- \lambda \| \boldsymbol {p} ^ {s} \| _ {2})} \in [ 0, 1 ], \tag {7}
$$
where  $\lambda$  is the temperature hyperparameter. The sensitive attribute probability of  $z^{\prime}$  for distilled graph can be estimated as probability distribution  $[\pi^{s = 0}(\pmb {z}^{\prime}),\pi^{s = 1}(\pmb {z}^{\prime})]$ , where  $\pi^{s = 0}(\pmb {z}^{\prime}) + \pi^{s = 1}(\pmb {z}^{\prime}) = 1$
3.4 Bias Measurement
Given the estimated sensitive attribute probability for  $z'$  of each distilled node, how can we measure the bias for them? For a fair representation, we can not distinguish which representation is more likely to be a specific sensitive group. Therefore, we adopt a simple surrogate bias measurement, named coherence, the variance of the estimated sensitive group membership. Given the whole distilled data representation  $Z' = [z_1, \dots, z_{N'}]^\top$ , the bias can be defined as:
$$
C o h ^ {s} \left(\boldsymbol {Z} ^ {\prime}\right) = \widehat {V a r} \left(\boldsymbol {\pi} ^ {s} \left(\boldsymbol {Z} ^ {\prime}\right)\right) = \frac {1}{N ^ {\prime}} \sum_ {n = 1} ^ {N ^ {\prime}} \left(\boldsymbol {\pi} ^ {s} \left(\boldsymbol {z} _ {n} ^ {\prime}\right) - \frac {1}{N ^ {\prime}} \sum_ {n = 1} ^ {N ^ {\prime}} \boldsymbol {\pi} ^ {s} \left(\boldsymbol {z} _ {n} ^ {\prime}\right)\right)
$$
Note that  $Coh^{s = 0}(\mathbf{Z}') = Coh^{s = 1}(\mathbf{Z}')$ , and we adopt abbreviation  $Coh(\mathbf{Z}')^3$ .
Geometric intuition. The intuition of sensitive attribute estimation, as illustrated in Figure 2, can be grasped from a geometric standpoint. In a toy example with a two-dimensional data representation,  $z' \in \mathbb{R}^2$ , we consider two demographic groups for a binary sensitive attribute. The subspace spanned by the data representations from these groups is denoted by  $S^0$  and  $S^1$ . Data representations to be estimated are  $z_0'$  and  $z_1'$ .  $p_0^0$ ,  $p_1^0$  and  $p_0^1$ ,  $p_1^1$  is the projection of  $z_0'$  and  $z_1'$  onto the orthogonal complement of  $S^0$  and  $S^1$ . As for fair data representation, zero coherent encourages all representations aligned with a "line" so that all representations are with the same normalized similarity with sensitive groups. Figure 2 (a) shows the case in which the data representation is biased where  $z_0'$  and  $z_1'$  can be easily distinguished. Figure 2 (b) shows that fairer data representation as they are less separable.
For multiple-value sensitive attribute, we can use average coherence  $Coh(Z^{\prime})$  across all sensitive groups.
Figure 3: An overview of the proposed framework. The synthesizer generates the attribute matrix and adjacency matrix of the distilled small graph  $\mathcal{G}'$ . The Cross-Entropy loss  $\mathcal{L}_{CE}$  guides the update of GNNs model during the inner optimization loop. Gradient matching loss  $\mathcal{L}_{GM}$  and coherence loss  $\mathcal{L}_{Coh}$  guide the update of the synthesizer during the outer optimization loop for utility and fairness.
Based on the proposed coherence metric, we argue that if  $Coh(\mathbf{Z}')$  is reduced, bias in the distilled graph can be mitigated. As a result, if GNNs are trained on such distilled graphs, the bias issues in downstream tasks could also be alleviated. The problem is formally defined as: Given an undirected attributed network  $\mathcal{G} = \{\mathbf{A}, \mathbf{X}, \mathbf{Y}, \mathbf{S}\}$ , our goal is to obtain an debiased distilled graph  $\mathcal{G}' = \{\mathbf{A}', \mathbf{X}', \mathbf{Y}'\}$  via reducing  $Coh$ , so that the fairness issues of GNNs trained on  $\mathcal{G}'$  is mitigated. Hence the overall objective goal for generating a fair and condensed graph is:
$$
\min  _ {\mathcal {G} ^ {\prime}} \mathcal {L} _ {G M} + \alpha \mathcal {L} _ {C o h} \left(\mathrm {G N N} _ {\theta \mathcal {G} ^ {\prime}} \left(\boldsymbol {A} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right), \mathrm {G N N} _ {\theta \mathcal {G} ^ {\prime}} \left(\boldsymbol {A}, \boldsymbol {X}\right)\right)
$$
$$
\operatorname {s. t} \theta^ {\mathcal {G} ^ {\prime}} = \underset {\theta} {\arg \min } \mathcal {L} _ {C E} \left(\mathrm {G N N} _ {\theta} \left(\boldsymbol {A} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right), \boldsymbol {Y} ^ {\prime}\right) \tag {8}
$$
4.2 Fair Graph Distillation Loss
Gradient Matching Loss. We adopt gradient matching, as shown in equation (2), for graph distillation to distill useful information for node classification tasks. However, treating both  $\mathbf{X}'$  and  $\mathbf{A}'$  as learnable parameter  $^4$  and directly optimizing  $\mathbf{A}'$  is unaffordable due to  $O(N^2)$  computation complexity. Following previous work Jin et al. [2021], we parameterize  $\mathbf{A}'$  as a function of  $\mathbf{X}'$ :
$$
\boldsymbol {A} _ {i, j} ^ {\prime} = \text {S i g m o i d} \left(\frac {\operatorname {M L P} _ {\phi} \left([ \boldsymbol {x} _ {i} ^ {\prime} ; \boldsymbol {x} _ {j} ^ {\prime} ]\right) + \operatorname {M L P} _ {\phi} \left([ \boldsymbol {x} _ {j} ^ {\prime} ; \boldsymbol {x} _ {i} ^ {\prime} ]\right)}{2}\right), \tag {9}
$$
where  $A_{i,j}^{\prime}$  is  $i$ -th row,  $j$ -th column of  $A^{\prime}$ ,  $\mathrm{MLP}_{\phi}$  is a multi-layer neural network parameterized with  $\phi$  and  $[\cdot; \cdot]$  denotes concatenation. Note that  $A^{\prime}$  is controlled to be symmetric since  $A_{i,j}^{\prime} = A_{j,i}^{\prime}$ . Sigmoid function pushes  $A^{\prime}$  close to 0 or 1 to encourage its sparsity. For simplicity, we denote the parameterized adjacency matrix as  $A_{\phi}^{\prime}$ . In this way, we can reduce the complexity to  $O(N)$ .
The distance metric  $D$  measures the similarity of gradients over the real graph and distilled graph. We adopt the summation of the gradient distance over all layers as the final gradient distance:
$$
D \left(\nabla_ {\theta} \mathcal {L} (\mathcal {G}), \nabla_ {\theta} \mathcal {L} (\mathcal {G}) ^ {\prime}\right) = \sum_ {i} \left(1 - \frac {\nabla_ {\theta} \mathcal {L} (\mathcal {G}) _ {i} \cdot \nabla_ {\theta} \mathcal {L} (\mathcal {G}) _ {i} ^ {\prime}}{\| \nabla_ {\theta} \mathcal {L} (\mathcal {G}) _ {i} \| \| \nabla_ {\theta} \mathcal {L} (\mathcal {G}) _ {i} ^ {\prime} \|}\right) \tag {10}
$$
4The distilled label  $\mathbf{Y}^{\prime}$  is sampled from real label  $\mathbf{Y}$  with the same class probability, and it is fixed.
where  $\nabla_{\theta}\mathcal{L}(\mathcal{G})_i$  is the i-th column vectors of the gradient matrices. Hence the loss objective for the graph distillation module is given by:
$$
\mathcal {L} _ {G M} = \mathbf {E} _ {\theta \sim P _ {\theta}} \left[ \sum_ {t = 0} ^ {T - 1} D \left(\nabla_ {\theta} \mathcal {L} (\mathcal {G}), \nabla_ {\theta} \mathcal {L} \left(\mathcal {G} ^ {\prime}\right)\right) \right] \tag {11}
$$
where  $\mathcal{G}' = \{\pmb{X}',\pmb{A}',\pmb{Y}'\}$ ,  $t$  is the training epoch, and  $\theta_{t}$  is well-trained GNNs model parameters. To reduce the impact of parameter initialization, the initial model parameters  $\theta_0$  are sampled from a distribution of random initialization.
Coherence loss. In Section 3.4, we introduce coherence as a bias metric for distilled data. To mitigate bias, we use coherence bias as a regularization for fair synthesis. This calculation employs real graph node attribute  $X$  and distilled node attribute  $X'$  to estimate sensitive group memberships but overlooks structural bias in graph data. Given the GNN propagation mechanism, bias can exist in both node attributes and graph structure Dong et al. [2022]. Even without attribute bias, node representation may still be biased if structural bias is present.