Title: Fair Graph Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Distillation

Algorithmic Fairness
Abstract: As graph neural networks (GNNs) struggle with large-scale graphs due to high computational demands, data distillation for graph data promises to alleviate this issue by distilling a large real graph into a smaller distilled graph while maintaining comparable prediction performance for GNNs trained on both graphs. However, we observe that GNNs trained on distilled graphs may exhibit more severe group fairness problems than those trained on real graphs. Motivated by this observation, we propose \textit{fair graph distillation} (\Algnameabbr), an approach for generating small distilled \textit{fair and informative} graphs based on the graph distillation method. The challenge lies in the deficiency of sensitive attributes for nodes in the distilled graph, making most debiasing methods (e.g., regularization and adversarial debiasing) intractable for distilled graphs. We develop a simple yet effective bias metric, called coherence, for distilled graphs. Based on the proposed coherence metric, we introduce a framework for fair graph distillation using a bi-level optimization algorithm. Extensive experiments demonstrate that the proposed algorithm can achieve better prediction performance-fairness trade-offs across various datasets and GNN architectures.

Fair Graph Distillation

Abstract

As graph neural networks (GNNs) struggle with large-scale graphs due to high computational demands, graph data distillation promises to alleviate this issue by distilling a large real graph into a smaller distilled graph while maintaining comparable prediction performance for GNNs trained on both graphs. However, we observe that GNNs trained on distilled graphs may exhibit more severe group fairness issues than GNNs trained on real graphs for vanilla and fair GNNs training. Motivated by these observations, we propose fair graph distillation (FGD), an advanced graph distillation approach to generate fair distilled graphs. The challenge lies in the deficiency of sensitive attributes for nodes in the distilled graph, making most debiasing methods (e.g., regularization and adversarial debiasing) intractable for distilled graphs. We develop a simple yet effective bias metric, named coherence, for distilled graphs. Based on the proposed coherence metric, we introduce a framework for fair graph distillation using a bi-level optimization algorithm. Extensive experiments demonstrate that the proposed algorithm can achieve better prediction performance-fairness trade-offs across various datasets and GNN architectures.

1 Introduction

Real-world data, like chemical molecules, social networks, and transportation networks, can be represented as graphs [Han et al., 2022a, Ling et al., 2023a, Jiang et al., 2022a, Ying et al., 2018, Ling et al., 2023b, Tong et al., 2020, Han et al., 2022b]. Graph neural networks (GNNs) excel at capturing structural information but struggle with large-scale graphs due to memory consumption and computational expense caused by the neighborhood explosion problem[Hamilton et al., 2017, Liu et al., 2023c]. This cost becomes unaffordable in situations requiring repeated GNN training, such as neural architecture search and continual learning [Liu et al., 2018, Zhou et al., 2019, Li and Hoiem, 2017, Liu et al., 2023b]. Dataset distillation is a promising solution to address computation challenges by generating small, informative distilled data for neural network training in downstream tasks [Jin et al., 2021, 2022, Zhao et al., 2021a,b, Nguyen et al., 2021]. Techniques like dataset condensation [Zhao et al., 2021b, Jin et al., 2021] can significantly reduce training data size without major performance degradation in the image and graph domains. However, focusing solely on prediction performance may introduce fairness issues, as sensitive information can be condensed into distilled data for prediction. A natural question is raised: Is the model trained on the distilled graph fair, and if not, how can we achieve fair graph distillation?

In this work, we focus on the group fairness 1 for node classification tasks under binary sensitive attribute setting. We discover that GNNs trained on distilled small graphs exhibit more severe group fairness issues than those on real graphs. In other words, graph distillation can even amplify graph

data bias, which challenges the applicability of graph distillation in high-stake applications [Mehrabi et al., 2021, Suresh and Guttag, 2019]. To this end, we propose a fair graph distillation framework to offer a significantly reduced graph size and also better utility-fairness trade-off while maintaining predictive performance.

Many debias methods explicitly use sensitive attributes, but these are inherently missing in distilled graphs because they are excluded from the data attributes and the meaning of the attributes may change during the optimization process. In this paper, we point out the relationship between the space of real graphs and the space of distilled graphs and develop a simple estimator of sensitive attributes and introduce a bias measurement called consistency. We then propose a bi-level optimization algorithm for fair graph distillation: the outer loop generates a fair and informative distilled graph using gradient matching and coherence loss, while GNNs train on distilled graphs in the inner loop. In a nutshell, the contributions can be summarized as follows:

To our knowledge, this is the first paper to identify group fairness issues in conventional graph distillation methods with binary sensitive attributes, motivating the formulation of a fair graph distillation problem in node classification tasks.

We discover the relationship between the space of real graphs and the space of distilled graphs. We develop a bias metric called coherence for distilled graphs and propose a bi-level optimization framework using this metric to achieve fair graph distillation.

We perform extensive experiments on various real-world datasets and GNN architectures to validate the effectiveness of the proposed FGD algorithm. Results demonstrate that FGD achieves a better accuracy-fairness trade-off compared to vanilla graph distillation methods and numerous baselines.

2 Preliminaries

2.1 Notations

We consider node classification tasks given a graph dataset G = { A , X , Y , S } with N nodes. Here, A ∈ { 0 , 1 } N × N is the adjacency matrix, and A ij = 1 represents there exists an edge between node i and j . X ∈ R N × D is the node feature matrix, where D is non-sensitive feature dimension for each node. Y ∈ { 0 , 1 , · · · , C -1 } N denotes the node labels over C classes. For simplicity, we consider a binary sensitive attribute 2 S ∈ { 0 , 1 } N . Π s is the sensitive membership diagonal matrix. Π s ii = 1 if and only if i -th node belongs to sensitive group s . The distilled small graph dataset is marked as G ′ = { A ′ , X ′ , Y ′ } which contains N ′ nodes and N ′ ≪ N . Note that elements of the distilled adjacency matrix satisfy A ′ ij ∈ [0 , 1] and no sensitive attributes exist in G ′ . The latent node representation of real graph is Z , and the span space of it is span ( Z ) := {∑ N i =1 w i z i | 1 ≤ i ≤ N,w i ∈ R } . Similar definition of Z ′ and span ( Z ′ ) for distilled graph.

2.2 Graph Distillation via Gradient Matching

The purpose of graph distillation is to generate a distilled graph G ′ such that the GNN model, denoted as GNN θ with parameters θ , trained on distilled graph performs comparably to the model trained on the real graph G . The objective can be formulated as the following bi-level optimization problem:

where θ G ′ denotes the optimal parameter trained on distilled small graph G ′ , and L ( · , · ) denotes the loss function. To tackle the above optimization problem, the gradient matching method Zhao et al.

0.08

0.06

0.04

△DP

0.02

0.010

Credit_Vanilla

German_Vanilla

0.005

Pokecn_Vanilla

Credit_Real

German_Real

0.000

Pokecn_Real

0.64

0.66

0.68

0.70

0.72

0.74

0.76

0.78

AUC

[2021a] is proposed. The intuition is to let the GNN parameters θ G ′ trained on distilled graph follow a similar path to the GNN parameters θ G trained on the real graph during model optimization. The gradient of the GNN parameters is forced to be the same over the real and distilled graphs:

where D ( · , · ) is a distance function, T is the number of steps of model parameters trajectory, and θ G t , θ G ′ t denotes the model parameters trained on G and G ′ at time step t , respectively. The gradient calculated on G and G ′ is denoted as ∇ θ L ( G ) := ∇ θ L ( GNN θ t ( A , X ) , Y ) and ∇ θ L ( G ′ ) := ∇ θ L ( GNN θ t ( A ′ , X ′ ) , Y ′ ) , respectively.

3 Bias Measurement for Distilled Graph

In this section, we empirically demonstrate the fairness issue in the distilled graph. Motivated by this, we pursue fair graph distillation. Although distilled graphs lack sensitive attributes S ′ , we observe that between the node representations of the real and distilled graphs: their barycenter remain consistent, and their spaces are also consistent. We leverage this phenomenon to develop a simple, effective bias measurement for distilled graphs.

3.1 Is Graph Distillation Really Fair?

Our empirical investigation assesses the fairness of graph distillation across various datasets and architectures. We compare the utility (AUC) and fairness (demographic parity ( ∆ DP ) [Beutel et al., 2017]) of GNNs trained on real graphs and those trained on distilled graphs created by the vanilla graph distillation method. The utility and fairness performance are shown in Figure 1. We can find that: For datasets like Pokec-n, German, and Credit, distilled graph-based GNNs have higher ∆ DP and lower AUC performance, suggesting a compromise in fairness and prediction performance. We also notice that, for Pokec-z and Recidivism datasets, these GNNs exhibit lower ∆ DP and significantly lower AUC performance (shown in Table 1), indicating a trade-off between improved fairness and reduced prediction performance. We observe similar results when using other fair GNN models. More details can be found in Appendix E. Motivated by these observations, we aim to find a better prediction performance and fairness trade-off via chasing the fair graph distillation method.

3.2 Geometric Connections in Data Distillation

The distilled data can be generated via minimizing gradient distance in Equation 2. To simplify the analysis, we consider D ( · , · ) as Euclidean distance and those model parameters during optimization trajectory satisfying θ ∼ P θ , where P θ is certain but unknown parameters' distribution. Therefore, the objective can be transformed as

We consider three assumptions of the model parameters' distribution and the convergence for loss minimization.

Assumption 3.1 (Model parameters' distribution) . We assume that each model parameter in the last softmax layer satisfies the same distribution.

Assumption 3.2 (Loss minimization) . We assume that exists at least one distilled dataset that minimizes Equation 3.

Theorem 3.3 (Consistent Span Space) . We empirically show that span ( Z ′ ) ≈ span ( Z ) via calculating the principle angles between them. We also provides the rigorous proof of z ′ ∈ span ( Z ) under distribution matching in Appendix D.

Theorem 3.4 (Consistent Geometric Barycenters) . Under Assumptions 3.1 and 3.2, the barycenter of the last representation for the optimal distilled graph and the real graphs are consistent, i.e. 1 N ∑ N i =1 z i = 1 N ′ ∑ N ′ i =1 z ′ i . Please see proof in Appendix B.

(a)

(b)

3.3 Sensitive Attribute Estimation

The consistent span space and geometric barycenter suggest that we can estimate sensitive attributes from the representations of both distilled and real graphs. We frame sensitive attribute estimation as a classification problem: Given a data representation z ′ ∈ Z ′ , what is the probability that z ′ belongs to the sensitive group?

Ridge regression for distance measurement. Notice that the representation of each sensitive group for the real graph is known, we define Z 0 and Z 1 as the representation matrix for sensitive group s = 0 and s = 1 . To measure the probability that z ′ belongs to these two sensitive groups, we first find the closest vector z ′ proj = Z ⊤ s q ∈ Span ( Z s ) to approximate the representation z ′ , and then use the norm of z ′ -z ′ proj to measure the distance between z ′ and sensitive group Z s . Specifically, we adopt ridge regression to find the optimal coefficient vector q ∗ , which can be formulated as

where γ is the hyperparameter for ridge regression. For the optimal q ∗ , we have

where ⊤ represents matrix transpose, p s is approximately the projection onto the orthogonal complement of the subspace span ( Z s ) . The proof is in Appendix C.

Sensitive attribute soft estimation. Since p s = z ′ -γ Z ⊤ s ( I + γ Z s Z ⊤ s ) -1 Z s z ′ can be viewed as approximately the projection of z onto the orthogonal complement of sensitive group Z s , ∥ p s ∥ 2 is small if z is in sensitive group Z s and large otherwise. The probability of the given data representation z belongs to the sensitive group Z s can further be inferred via a softmax function:

where λ is the temperature hyperparameter. The sensitive attribute probability of z ′ for distilled graph can be estimated as probability distribution [ π s =0 ( z ′ ) , π s =1 ( z ′ )] , where π s =0 ( z ′ ) + π s =1 ( z ′ ) = 1 .

3.4 Bias Measurement

Given the estimated sensitive attribute probability for z ′ of each distilled node, how can we measure the bias for them? For a fair representation, we can not distinguish which representation is more likely to be a specific sensitive group. Therefore, we adopt a simple surrogate bias measurement, named coherence, the variance of the estimated sensitive group membership. Given the whole distilled data representation Z ′ = [ z 1 ..., z N ′ ] ⊤ , the bias can be defined as:

Note that Coh s =0 ( Z ′ ) = Coh s =1 ( Z ′ ) , and we adopt abbreviation Coh ( Z ′ ) 3 .

Geometric intuition. The intuition of sensitive attribute estimation, as illustrated in Figure 2, can be grasped from a geometric standpoint. In a toy example with a two-dimensional data representation, z ′ ∈ R 2 , we consider two demographic groups for a binary sensitive attribute. The subspace spanned by the data representations from these groups is denoted by S 0 and S 1 . Data representations to be estimated are z ′ 0 and z ′ 1 . p 0 0 , p 0 1 and p 1 0 , p 1 1 is the projection of z ′ 0 and z ′ 1 onto the orthogonal complement of S 0 and S 1 . As for fair data representation, zero coherent encourages all representations aligned with a 'line" so that all representations are with the same normalized similarity with sensitive groups. Figure 2 (a) shows the case in which the data representation is biased where z ′ 0 and z ′ 1 can be easily distinguished. Figure 2 (b) shows that fairer data representation as they are less separable.

GNN Layer

GNN Layer

Coherence

Loss

Gradient Matching Loss

Inner Loop Optimization Update

S=0

Forward Pass

S=1

BP Gradient of Real Graph

Synthesizer

Synthetic node

Outer Loop Optimization Update

BP Gradient of Distilled Graph

CE

CGM（，）

X'

MLP

A'

Lcoh

4 Methodology

4.1 Problem Statement

Based on the proposed coherence metric, we argue that if Coh ( Z ′ ) is reduced, bias in the distilled graph can be mitigated. As a result, if GNNs are trained on such distilled graphs, the bias issues in downstream tasks could also be alleviated. The problem is formally defined as: Given an undirected attributed network G = { A , X , Y , S } , our goal is to obtain an debiased distilled graph G ′ = { A ′ , X ′ , Y ′ } via reducing Coh , so that the fairness issues of GNNs trained on G ′ is mitigated. Hence the overall objective goal for generating a fair and condensed graph is:

4.2 Fair Graph Distillation Loss

Gradient Matching Loss. We adopt gradient matching, as shown in equation (2), for graph distillation to distill useful information for node classification tasks. However, treating both X ′ and A ′ as learnable parameter 4 and directly optimizing A ′ is unaffordable due to O ( N 2 ) computation complexity. Following previous work Jin et al. [2021], we parameterize A ′ as a function of X ′ :

where A ′ i,j is i -th row, j -th column of A ′ , MLP ϕ is a multi-layer neural network parameterized with ϕ and [ · ; · ] denotes concatenation. Note that A ′ is controlled to be symmetric since A ′ i,j = A ′ j,i . Sigmoid function pushes A ′ close to 0 or 1 to encourage its sparsity. For simplicity, we denote the parameterized adjacency matrix as A ′ ϕ . In this way, we can reduce the complexity to O ( N ) .

The distance metric D measures the similarity of gradients over the real graph and distilled graph. We adopt the summation of the gradient distance over all layers as the final gradient distance:

where ∇ θ L ( G ) i is the i-th column vectors of the gradient matrices. Hence the loss objective for the graph distillation module is given by:

where G ′ = { X ′ , A ′ , Y ′ } , t is the training epoch, and θ t is well-trained GNNs model parameters. To reduce the impact of parameter initialization, the initial model parameters θ 0 are sampled from a distribution of random initialization.

Coherence loss. In Section 3.4, we introduce coherence as a bias metric for distilled data. To mitigate bias, we use coherence bias as a regularization for fair synthesis. This calculation employs real graph node attribute X and distilled node attribute X ′ to estimate sensitive group memberships but overlooks structural bias in graph data. Given the GNN propagation mechanism, bias can exist in both node attributes and graph structure Dong et al. [2022]. Even without attribute bias, node representation may still be biased if structural bias is present.

Work by Dong et al. [2022] suggests that structural bias can be measured through graph representation bias. Leveraging this, we aim for low coherence in node attributes and representations to fully remove bias from our distilled graph. Specifically, we introduce attribute and structural coherence to decrease attribute and structural bias, respectively, by minimizing the variance in sensitive group membership estimation for node attributes and representations. Given a real graph data G = { A , X , Y , S } and a distilled graph G ′ = { A ′ , X ′ , Y ′ } , we feed them into a L -layer GNN, where the l -th layer latent representation in the GNN is denoted as Z l . The latent representation for node attribute after l -hop propagation contains both attribute bias as well as structural bias. Note the node attribute X and X ′ before propagation as Z 0 and Z ′ 0 , we get a set of latent presentation { Z 0 , Z 1 , ..., Z L } for G and { Z ′ 0 , Z ′ 1 , ..., Z ′ L } for G ′ . The objective to measure bias of Z ′ l is:

where C j = γ j ( I + γ j Z l Π j Z T l ) -1 . Π j is introduced in Sec 4.1. Since we consider the binary sensitive attribute, j is set as 0 without losing generality and is omitted in the notations as π j ( · ) := π ( · ) . After considering all the latent representations, the coherence loss objective is defined as the summation of all coherence over all layers, i.e.,

Prediction loss for GNN training. The GNNs model is trained on distilled graph G ′ = { A ′ , X ′ , Y ′ } with prediction loss. We adopt L -layer GNNs model, where θ is the parameter of the GNN. We also adopt cross-entropy loss by default:

4.3 Final Objective and Training Algorithm

Outer loop optimization. In the outer loop, we optimize the fair graph synthesizer with gradient matching loss and coherence loss:

where α is a hyperparameter to regularize the debiasing intensity. The distilled node attribute X ′ and the distilled node label Y ′ are initialized with the nodes uniformly sampling from real graph data G .

Inner loop optimization. The GNN parameter θ is optimized in the inner loop:

Instead of using the real graph data G to calculate the loss, we use the distilled graph G ′ . It empirically shows good performance and better efficiency. But the adversarial training baseline uses G as it needs the sensitive attribute for discriminator training.

5 Experiments

We design experiments to validate the effectiveness of the proposed framework FGDand answer the following research questions: RQ.1 How well can FGD mitigate the bias in the distilled graph and alleviate the fairness issue of the GNNs trained on the distilled small graph? RQ.2 How well can FGD balance the trade-off between accuracy and bias mitigation compared with other debiasing baselines? RQ.3 Can FGD further improve the utility or bias mitigation as an add-on module to other bias mitigation methods?

5.1 Experimental setting

Datasets. We use five real-world datasets, including Pokec-z, Pokec-n [Dai and Wang, 2021, Takac and Zabovsky, 2012], German, Credit, and Recidivism [Agarwal et al., 2021]. The detailed setting of the datasets is in Appendix F

GNN Models. We adopt three popular GNN variants in our experiments, including GCN [Kipf and Welling, 2016], SGC [Wu et al., 2019], GraphSAGE [Hamilton et al., 2017].

Baselines. Given the absence of fair graph distillation work, we compare our approach with four baselines. (1) Real uses real graph data to train a GNN model. (2) Vanilla applies a vanilla graph distillation algorithm [Jin et al., 2022] for distilled graph data learning and GNN model training. (3) FairGNN [Dai and Wang, 2021] trains a fair GNN model adversarially on real graph data, aiming to achieve good prediction while fooling a discriminator. (4) EDITS [Dong et al., 2022] is a modelagnostic debiasing method that rewires graph data for fair GNNs.

Evaluation Metrics. We evaluate model performance from model utility and bias measurements. Good performance represents low bias and high model utility. For model utility metrics, we adopt accuracy (ACC), the area under the receiver operating characteristic curve (AUC), and F1 score to measure prediction performance. For bias measurement, we adopt two commonly used fairness metrics, i.e., demographic parity ( ∆ DP ) and equal opportunity ( ∆ EO ) [Beutel et al., 2017, Louizos et al., 2015]. Denote the binary label as y ∈ { 0 , 1 } , and sensitive attribute as s ∈ { 0 , 1 } . ˆ y ∈ { 0 , 1 } denotes the model prediction. The violation of DP and EO are given by ∆ DP = | P (ˆ y = 1 | s = 0) -P (ˆ y = 1 | s = 1) | , and ∆ EO = | P (ˆ y = 1 | y = 1 , s = 0) -P (ˆ y = 1 | y = 1 , s = 1) | .

Pokec-Z

Pokec-n

German

Credit

Recidivism

0.035

0.030

0.025

★

FGD

Vanilla

FairGNN

DP

0.04

0.02

0.03

Vanilla

FGD

0.02

0.05

0.04

0'0

FGD

Vanilla

FairGNN

EDITS

0.12

0.08

0.10

0.06

★

FGD

EDITS

Vanilla

FairGNN

dQ

0.04

E00

0.01

200

FGD

FairGNN

EDITS

Vanilla

0.020

0.01

FairGNN

100

0.04

0.700

0.705

0.7100.715

AUC

0.500 0.525 0.550 0.575 0.600 0.625 0.650

AUC

0.52

0.54

AUC

0.56

0.58

0.60

0.64

0.65

0.660.670.680.690.70

AUC

0.00

0.60

0.65

AUC

0.70

0.75

0.80

Pokec-Z

Pokec-n

German

Credit

Recidivism

O

0.040

0.035

0.030

0.020

0.025

0.015

★

FGD

Vanilla

FairGNN

0.05

0.04

0.03

0.02

FGD

Vanilla

FairGNN

O

0.010

0.015

0.005

一

FairGNN

FGD

Vanilla

EDITS

0.04

0.08

0.06

★

FGD

Vanilla

EDITS

FairGNN

0.04

0.03

0.02

0.01

FGD

Vanilla

FairGNN

0.700

AUC

0.7100.715

0

0.500 0.525 0.550 0.575 0.600 0.625 0.650

AUC

0.52

0.54

AUC

0.56

0.58

0.60

0.02

0.64

0.650.66

AUC

0.670.680.690.70

0.00

0.60

0.65

AUC

0.70

0.75

0.80

5.2 Debiasing distilled Graph

In response to RQ.1 , we assess FGD's bias mitigation and prediction performance across various GNN architectures and datasets, as shown in Table 1. We compare the ∆ DP and ∆ EO values of GNNs trained on real graphs ( Real ), vanilla distilled graphs ( Vanilla ), and debiased distilled graphs via FGD ( FGD ). Key findings include: (1) Models trained on Real graphs consistently outperform Vanilla and FGD in utility, though FGD 's utility matches or surpasses Vanilla . (2) FGD consistently yields lower bias than Vanilla , and outperforms Real on 4 out of 5 datasets, excluding Poken-n.

We compare the coherence bias of distilled graphs generated by Vanilla and FGD methods across five real-world datasets with the GCN architecture (Table 2). Our analysis reveals that FGD reduces unfairness, reflected in lower coherence bias in the distilled graphs. This consistency confirms the effectiveness of coherence bias as a measure of distilled graph bias.

5.3 Trade-Off Comparison

In response to RQ.2 , we compare the tradeoff between model utility and bias mitigation against other baselines using the GCN architecture. We utilize the Pareto frontier Ishizaka and Nemery [2013] to evaluate our approach's utility-fairness trade-off, using different hyperparameters. The Pareto frontier graphically represents optimal trade-offs in multi-objective optimization. We use AUC as the utility metric and ∆ DP and ∆ EO as fairness metrics. Higher AUC and lower ∆ DP / ∆ EO are preferred, so models with Pareto frontier curves closer to the bottom right corner (AUC on the horizontal axis and ∆ DP / ∆ EO on the vertical) have better trade-off performance.

Table 2: Coherence bias comparison between vanilla distilled graph (denoted as Vanilla) and fair distilled graph (denoted as FGD). The lower, the better. The best ones are marked in bold. The architecture model is GCN.

Figure 4 shows the results for models trained on the real graph, the distilled graph debiased by baseline methods (vanilla graph distillation, FairGNN, and EDITS 5 ) and the distilled graph debiased by FGD. We can observe: (1) From a model utility perspective, FGD performs comparably to other baselines, like vanilla graph distillation, FairGNN, and EDITS 6 , suggesting it preserves sufficient information for node classification. (2) In terms of bias mitigation, all baselines show effectiveness, with FGD exhibiting the best results. (3) When considering the utility-fairness trade-off, FGD's Pareto front curve lies at the bottom right corner of all baselines, signifying it offers the best balance. Thus, FGD outperforms other baselines in balancing model utility and bias mitigation.

5.4 Add-on Module

In addition to its superior trade-off performance, our method, FGD, can enhance other debiasing baselines like FairGNN and EDITS by acting as an add-on debias module. This compatibility is due to the fact that these baselines can replace the cross-entropy loss in the GNN training module. To answer RQ.3 , we conducted experiments on the Credit dataset comparing FairGNN/EDITS performance with and without FGD. As shown in Figure 5, FairGNN/EDITS coupled with FGD delivers better utility-fairness

trade-off, demonstrating FGD's potential to boost other debias methods.

6 Related Work

Dataset Distillation & Knowledge Distillation. Dataset Distillation (DD) and Knowledge Distillation (KD) are methods to improve the efficiency of training deep neural networks. DD synthesizes a small dataset encapsulating the knowledge of a larger one, achieving comparable model performance [Wang et al., 2018, Kim et al., 2022, Lee et al., 2022, Zhao et al., 2021a, Yang et al., 2022]. It employs a bi-level optimization approach, with dataset condensation (DC) speeding up the process via gradient matching of model parameters. DD also helps with repeated training or privacy applications like continual learning, neural architecture search, and privacy-preserving scenarios. Meanwhile, graph data condensation methods have been developed for node and graph classification tasks [Jin et al., 2022]. KD, on the other hand, enhances computational efficiency through model compression and acceleration. It trains a compact student model using the knowledge from a larger teacher model Gou et al. [2021]To address the scarcity and high complexity of labeled data in GNNs, knowledge distillation (KD) was introduced to enhance existing GNNs Liu et al. [2023a], Wang et al. [2023], also for fairness problem Dong et al. [2023]. While KD focuses on model compression, DD targets data compression, each improving efficiency from model-centric and data-centric perspectives.

Fair Graph Learning. Fairness in machine learning has attracted many research efforts[Chuang and Mroueh, 2020, Zhang et al., 2018, Du et al., 2021, Jiang et al., 2022b, Han et al., 2023, Jiang et al., 2023]. Many technologies are introduced in graph neural networks to achieve fair graph learning in node classification tasks, including optimization with regularization [Jiang et al., 2022a], rebalancing [Zeng et al., 2021], adversarial learning [Dai and Wang, 2021, Bose and Hamilton, 2019, Fisher et al., 2020] and graph rewiring [Köse and Shen, 2021, Dong et al., 2022]. For link prediction, dyadic fairness and corresponding graph rewiring solutions are also developed in [Li et al., 2021]. Another line of work focuses on solving the individual fairness problem on the graph data Song et al. [2022], Dong et al. [2021], Kang et al. [2020].

7 Conclusion

Despite the ability of graph distillation to condense valuable graph data, this study finds that the vanilla method can worsen fairness issues. Therefore, we introduce a fair graph distillation process to generate fair distilled graph data. As the distilled graph lacks the nodes' sensitive attributes, conventional fair methods cannot be directly applied. However, we identify a consistent geometric phenomenon in graph distillation to estimate these sensitive attributes. We also introduce a new bias metric, coherence, and propose a bi-level optimization framework, FGD, for fair graph distillation. Experimental results validate FGD's effectiveness in mitigating bias while maintaining model utility across various GNN architectures and datasets. Future work will focus on addressing individual fairness issues and non-binary sensitive attribute conditions, among other aspects, as discussed in Appendix H.

References

Agarwal, H. Lakkaraju, and M. Zitnik. Towards a unified framework for fair and stable graph representation learning. In Uncertainty in Artificial Intelligence , pages 2114-2124. PMLR, 2021.

Beutel, J. Chen, Z. Zhao, and E. H. Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00075 , 2017.

Bose and W. Hamilton. Compositional fairness constraints for graph embeddings. In International Conference on Machine Learning , pages 715-724. PMLR, 2019.

C.-Y. Chuang and Y. Mroueh. Fair mixup: Fairness via interpolation. In International Conference on Learning Representations , 2020.

Dai and S. Wang. Say no to the discrimination: Learning fair graph neural networks with limited sensitive attribute information. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining , pages 680-688, 2021.

Dong, J. Kang, H. Tong, and J. Li. Individual fairness for graph neural networks: A ranking based approach. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 300-310, 2021.

Dong, N. Liu, B. Jalaian, and J. Li. Edits: Modeling and mitigating data bias for graph neural networks. In Proceedings of the ACM Web Conference 2022 , pages 1259-1269, 2022.

Dong, B. Zhang, Y. Yuan, N. Zou, Q. Wang, and J. Li. Reliant: Fair knowledge distillation for graph neural networks. In Proceedings of the 2023 SIAM International Conference on Data Mining (SDM) , pages 154-162. SIAM, 2023.

Du, S. Mukherjee, G. Wang, R. Tang, A. H. Awadallah, and X. Hu. Fairness via representation neutralization. arXiv preprint arXiv:2106.12674 , 2021.

Fisher, A. Mittal, D. Palfrey, and C. Christodoulopoulos. Debiasing knowledge graph embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7332-7345, 2020.

Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision , 129:1789-1819, 2021.

Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Han, Z. Jiang, N. Liu, and X. Hu. G-mixup: Graph data augmentation for graph classification. arXiv preprint arXiv:2202.07179 , 2022a.

Han, Z. Jiang, N. Liu, Q. Song, J. Li, and X. Hu. Geometric graph representation learning via maximizing rate reduction. In Proceedings of the ACM Web Conference 2022 , pages 1226-1237, 2022b.

Han, Z. Jiang, H. Jin, Z. Liu, N. Zou, Q. Wang, and X. Hu. Retiring $\delta \text{DP}$: New distribution-level metrics for demographic parity. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=LjDFIWWVVa .

Ishizaka and P. Nemery. Multi-criteria decision analysis: methods and software . John Wiley & Sons, 2013.

Jiang, X. Han, C. Fan, Z. Liu, N. Zou, A. Mostafavi, and X. Hu. Fmp: Toward fair graph message passing against topology bias. arXiv preprint arXiv:2202.04187 , 2022a.

Jiang, X. Han, C. Fan, F. Yang, A. Mostafavi, and X. Hu. Generalized demographic parity for group fairness. In International Conference on Learning Representations , 2022b.

Jiang, X. Han, H. Jin, G. Wang, R. Chen, N. Zou, and X. Hu. Chasing fairness under distribution shift: a model weight perturbation approach. 2023.

Jin, L. Zhao, S. Zhang, Y. Liu, J. Tang, and N. Shah. Graph condensation for graph neural networks. arXiv preprint arXiv:2110.07580 , 2021.

Jin, X. Tang, H. Jiang, Z. Li, D. Zhang, J. Tang, and B. Yin. Condensing graphs via one-step gradient matching. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 720-730, 2022.

Kang, J. He, R. Maciejewski, and H. Tong. Inform: Individual fairness on graph mining. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 379-389, 2020.

J.-H. Kim, J. Kim, S. J. Oh, S. Yun, H. Song, J. Jeong, J.-W. Ha, and H. O. Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning , pages 11102-11118. PMLR, 2022.

N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Ö. D. Köse and Y. Shen. Fairness-aware node representation learning. arXiv preprint arXiv:2106.05391 , 2021.

Lee, S. Chun, S. Jung, S. Yun, and S. Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning , pages 12352-12364. PMLR, 2022.

Li, Y . Wang, H. Zhao, P. Hong, and H. Liu. On dyadic fairness: Exploring and mitigating bias in graph connections. In International Conference on Learning Representations , 2021.

Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

Ling, Z. Jiang, M. Liu, S. Ji, and N. Zou. Graph mixup with soft alignments. In International Conference on Machine Learning . PMLR, 2023a.

Ling, Z. Jiang, Y. Luo, S. Ji, and N. Zou. Learning fair graph representations via automated data augmentations. In International Conference on Learning Representations , 2023b.

Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 , 2018.

Liu, T. Zheng, G. Zhang, and Q. Hao. Graph-based knowledge distillation: A survey and experimental evaluation. arXiv preprint arXiv:2302.14643 , 2023a.

Liu, Z. Jiang, S. Zhong, K. Zhou, L. Li, R. Chen, S.-H. Choi, and X. Hu. Editable graph neural network for node classifications. arXiv preprint arXiv:2305.15529 , 2023b.

Liu, K. Zhou, Z. Jiang, L. Li, R. Chen, S.-H. Choi, and X. Hu. DSpar: An embarrassingly simple strategy for efficient GNN training and inference via degree-based sparsification. Transactions on Machine Learning Research , 2023c. ISSN 2835-8856. URL https://openreview.net/ forum?id=SaVEXFuozg .

Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel. The variational fair autoencoder. arXiv preprint arXiv:1511.00830 , 2015.

Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) , 54(6):1-35, 2021.

Nguyen, R. Novak, L. Xiao, and J. Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems , 34:5186-5198, 2021.

Song, Y. Dong, N. Liu, and J. Li. Guide: Group equality informed individual fairness in graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1625-1634, 2022.

Suresh and J. V. Guttag. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 , 2, 2019.

Takac and M. Zabovsky. Data analysis in public social networks. In International scientific conference and international workshop present day trends of innovations , volume 1. Present Day Trends of Innovations Lamza Poland, 2012.

Tong, J. Huang, G. Wolf, D. Van Dijk, and S. Krishnaswamy. Trajectorynet: A dynamic optimal transport network for modeling cellular dynamics. In International conference on machine learning , pages 9526-9536. PMLR, 2020.

Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Wang, B. Hooi, Y. Liu, and N. Shah. Graph explicit neural networks: Explicitly encoding graphs for efficient and accurate inference. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining , pages 348-356, 2023.

Wu, A. Souza, T. Zhang, C. Fifty, T. Yu, and K. Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Yang, Z. Xie, H. Peng, M. Xu, M. Sun, and P. Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329 , 2022.

Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 974-983, 2018.

Zeng, R. Islam, K. N. Keya, J. Foulds, Y. Song, and S. Pan. Fair representation learning for heterogeneous information networks. In Proceedings of the International AAAI Conference on Web and Social Media , volume 15, pages 877-887, 2021.

H. Zhang, B. Lemoine, and M. Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 335-340, 2018.

Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. ICLR , 1(2):3, 2021a.

Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. ICLR , 1(2):3, 2021b.

Zhou, Q. Song, X. Huang, and X. Hu. Auto-gnn: Neural architecture search of graph neural networks. arXiv preprint arXiv:1909.03184 , 2019.

A Training Algorithm

The training algorithm for fair graph distillation is shown in Algorithm 1.

Algorithm 1 Fair Graph Distillation

Input: Training graph data G = { X , A , S , Y } , hyperparameters α , temperature γ , number of alternative optimization step T alt , distilled label Y ′ .

Initialize X ′ based on real attributes, synthesizer model ϕ , and GNNs model θ .

for t = 1 to T alt do

Given GNNs model GNN θ t , calculate the gradient distance D ( ∇ θ L ( G ) , ∇ θ L ( G ′ t ) ) over the real graph G and distilled graph G ′ t .

Train GNNs model using distilled graph G ′ t and Equation (15) to obtain GNN θ t .

Calculate coherence loss based on GNNs model GNN θ t , real graph G and distilled graph G ′ t .

Train synthesizer model using prediction loss as Equation (16).

end for

Output: The fair distilled graph G ′ = { A ′ , X ′ , Y ′ } .

B Proof of Theorem 3.4

We consider GNNs model to learn node presentation z i in the real graph G and then followed a linear classifier W = [ w 0 , · · · , w C -1 ] and softmax layer, where w j is the weight vector connected to the j -tj output neuron. We first focus on the relation between the latent representation and the gradient of the linear classification layer. It is easy to obtain the cross-entropy loss J i ( J ′ i ) for i -th node with label y i in real graph G (distilled graph G ′ ) as follows:

Then we define gradient over weight vector as g i,j = ∂J i ∂ w j and g ′ i,j = ∂J ′ i ∂ w j in the real and distilled graph. If j = y i , we can obtain

̸

Similarly, for j = y , we have

In other words, the gradient of the loss for i -th node with label y i with respect to the weight vector connected to the j -th output neuron is given by

Based on Assumption 3.1, each model parameter in the last softmax layer satisfies the same distribution. In other words, the expectation of all predictions are the same, i.e.,

Note that the gradient calculation is based on backpropagation, the gradient for the last linear classification layer is quite critical for the gradient of other layers. Hence we consider the gradient of the last linear classification layer in the real graph, shown by

Similarly, we have the gradient of the last linear classification layer in the distilled graph as follows:

Under assumption 3.2, it is easy to know that the optimal solution to minimizing the objective min G ′ E θ ∼P W [ ||∇ W L ( G ) -∇ W L ( G ′ ) || 2 ] satisfy ∇ W L ( G ) = ∇ W L ( G ′ ) . Since the distilled label is sampling to keep class label probability, we have |{ i : y ′ i = j }| N ′ = |{ i : y i = j }| N for any class index i . Therefore, based on Equations (22) and (23), we have the optimal distilled graph satisfy

C Proof of Ridge Regression

Define objective function J = γ ∥ z ′ -Z ⊤ s q ∥ 2 2 + ∥ q ∥ 2 2 , it is easy to obtain

Therefore, the optimal q ∗ = γ ( I + γ Z s Z ⊤ s ) -1 Z s z ′ . Therefore, the projection of representation z ′ in the complement space of sensitive group Z s is given by

D More Results on Consistent Span Space

We conduct experiments to measure the distance between span ( Z ) and span ( Z ′ ) using principle angles between subspaces and emperically shows that span ( Z ) ≈ span ( Z ′ ) in the real dataset.

The concept of principal angle is used in linear algebra to measure the similarity between two subspaces of a vector space. It helps quantify how close or far apart these subspaces are. Given subspace, L , M ⊆ R n , with dim L = l ≥ dim M = m , there are m principal angles between L and Mdenoted as 0 ≤ θ 1 ≤ θ 2 ≤ · · · ≤ θ m ≤ π 2 between L and M are recursively defined, where

Notably, when the two subspaces are aligned, the principal angels are close to 0. We report the average principal angles of span ( Z ) and span ( Z ′ ) on all datasets as following:

Pokec-z: 1 . 08 × 10 - 6

Pokec-n: 1 . 03 × 10 - 6

•

German: 4 . 84 × 10 - 7

Credit: 2 . 57 × 10 - 7

Recidivism: 3 . 87 × 10 - 7

In the experiments, the principal angles of span ( Z ) and span ( Z ′ ) on all dataset are nearly 0. This indicates that the distance between space span ( Z ′ ) and space span ( Z ) are quite small in practice.

Additionally, we would like to mention that [1] provides the rigorous proof of z ′ ∈ span ( Z ) for distribution matching under several assumptions (although we can not prove it under gradient matching setting). According to formulas 21 from [1], it is assumed that (1) the **linear extractor** iid

ψ θ : R d → R k such that k < d, θ = [ θ i,j ] ∈ R k × d , θ i,j ∼ N (0 , 1) and for an input z , ψ θ ( x ) = θ z . When using distribution match method for data condensation, we have:

where d := θ ( 1 N ∑ N j =1 z j -1 N ′ ∑ N ′ j =1 z ′ j ) , E [ θ ⊤ θ ] = k I d by definition of θ , and I d is the identity matrix of R d . the projection components of span( Z ) ⊥ remain zero throughout the optimization process of DM. And we use z i to initialize z ′ i , thus z ′ ∈ span ( Z ) . However, in the implementation we use gradient matching instead of distribution matching.

E Preliminary Motivation

We have added experiments comparing the fairness performance of various fair GNNs trained on synthetic and real graph data. Specifically, we report the results (using demographic parity (DP), equal opportunity (EO), and individual unfairness (IND) Song et al. [2022] as metrics) with EDITS Dong et al. [2022], FairGNN Dai and Wang [2021], InFoRM Kang et al. [2020], and REDRESS Dong et al. [2021] on five datasets in our paper. EDITS is a pre-processing debiasing method, FairGNN is an in-processing debiasing method, and InFoRM and REDRESS focus on individual fairness. We encountered out-of-memory (OOM) issues when implementing GUIDE and REDRESS on an NVIDIA GeForce RTX A5000 (24GB GPU memory), so we used InFoRM as the baseline. Due to the extensive training time required for REDRESS, we only report results on the German dataset for REDRESS. We use demographic parity (DP), equal opportunity (EO), and individual unfairness (IND) as metrics.Table 4 demonstrates the result. From Table 4, we can see that in terms of the group fairness metrics (DP, EO), the fairness problem becomes uniformly worse on the Credit, German, and Pokecn datasets for all debiasing methods. For the Recidivism dataset, the distilled graph shows fewer fairness issues (lower DP or EO), especially for the EDITS method. This may result from the drop in utility of the model trained on the distilled graph (AUC is too low). As shown in Figure 4 of our paper, FGD can achieve a better performance-fairness trade-off compared to the baselines.

F Dataset Statistics

Pokec. The Pokec dataset consists of millions of anonymized user profiles from Slovakia's most popular social network in 2012, with information such as gender, age, hobbies, interests, education, and working field. The dataset was sampled into Pokec-z and Pokec-n based on user province, with region as the sensitive attribute. The task is to predict user working field.

German. The German Graph credit dataset has 1,000 client records with attributes like Gender and LoanAmount, used to classify individuals as good or bad credit risks. The similarity between node attributes is calculated using Minkowski distance and nodes are connected if the similarity is 80% of the maximum similarity.

Credit. Credit dataset, consisting of 30,000 individuals with features such as education, credit history, age, and derived spending and payment patterns. The similarity between two node attributes is calculated using Minkowski distance as the similarity measure and the credit defaulter graph network is constructed by connecting nodes with a similarity of 70% of the maximum similarity between all nodes.

Recidivism. The US state court bail outcome dataset (1990-2009) contains 18,876 defendant records with past criminal records, demographic attributes, etc. The similarity between node attributes is calculated using Minkowski distance and nodes are connected if the similarity is 60% of the maximum similarity.

G More Experimental Details

G.1 Parameter Study

Here we aim to study the sensitivity of FGD w.r.t. hyper-parameters. Specifically, we show the parameter study of α on Recidivism dataset. Here α controls the intensity to regularize the coherence bias of the distilled small graph. The results in Table 5 indicate that α can control the debiasing and utility performance of the distilled small graph.

G.2 Implementation Details

Synthesizer training. We adopt Adam optimizer for synthesizer training with 0 . 0002 learning rate. MLP ϕ consists of 3 linear layer with 128 hidden dimension. The outer loop number is 16 while the inner loop is 4 for each epoch. For each experiment, we train with a maximum of 1200 epochs and 3 independent runs. The temperature parameter γ is set to 10. X ′ and ϕ are optimized alternatively.

GNN training. We adopt Adam optimizer for GNN training with 0 . 005 learning rate. All GNN models are 2 layers with 256 hidden dimensions. For Pokec-z, Pokec-n, German, Credit, and Recidivism the training epochs are 1500 , 1500 , 4000 , 1000 , and 1000 respectively.

（

a

）

（

b

）

0.20

Real s=0

1.00

Reals=0

Real s=1

Real s=1

国

0.15

Syn

0.75

Vanilla

Real_center

FGD

0.50

★

Syn_center

0.10

0.25

0.05

0.00

0.00

0.25

0.50

0.05

0.75

0.10

1.00

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

G.3 More Visualization

We also visualize the node representation using PCA. We could observe that the barycenter of node from real graph and distilled graph is very close. And The distribution of node representation after being normalized to the circumference is consistent with the geometric intuition shown in Figure 6.

H Limitations and Future Work

H.1 Non-binary Sensitive Attribute

For categorical sensitive attributes, if only one sensitive membership group's embeddings are far away from others, then the mean embeddings will still be close to the majority embeddings, especially for many categories, resulting in low variance (coherence). We argue that only this group with distant embedding (a small portion of samples) can have their sensitive attribute detected using embedding distributions. From a metric perspective, if we adopt the maximized ∆ DP over any sensitive attribute group pair, the bias should be large due to considering the worst case. The proposed coherence may not work well in this scenario, and an advanced coherence can be developed for this case, e.g., the maximized variance over any sensitive group pair. We leave the advanced coherence development for categorical, multiple, or even continuous sensitive attributes in future work.

H.2 Individual Fairness

From Table 4, we find that all datasets suffer from a surprisingly more severe individual fairness problem (much higher IND score) when the model is trained on the distilled graph, even if we use InFoRM or REDRESS. This could be an interesting direction for future work, and we will add discussion with references in the related work section.

H.3 Other Tasks

Our paper mainly focuses node classification tasks and it is possible to extend our method to other tasks or other group fairness problems. For instance, FGD may alleviate group fairness issues in link prediction tasks by reducing the coherence bias among different link groups. Exploring other tasks (e.g., recommendation, graph classification) or other fairness metrics (e.g., individual fairness, rank fairness) could be interesting for future work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes fair graph distillation (FGD), as an advanced graph distillation approach to generate fair distilled graphs. FGD focuses on the group fairness issue in graph distillation methods and aims to generate fair distilled graphs with respect to sensitive attributes for nodes. This paper proposes a simple yet effective metric for measuring the bias in representation space, namely coherence, for distilled graphs, and a bi-level optimization framework to generate fair graph distillations iteratively. Experimental results illustrate that the proposed methods can achieve  performance-fairness trade-offs across various datasets and architectures.\n\nSTRENGTHS:\n- This paper considers group fairness issue, which is novel in graph distillation.\n- The proposed metric for bias measurement is simple yet convincing.\n- The theorems and algorithms introduced in paper is well presented.\n- Experimental results show the effectiveness of FGD in debiasing distillation.\n\nWEAKNESSES:\n- More baselines in graph distillation are needed. For example, *FairGNN* and *EDITS* are introduced as baselines in debiasing, while no graph distillation methods (such as [1]) are compared. \n- Results in Table 1 shows that distilled graphs could achieve some improvements in debiasing, while the AUC performance drops significantly (e.g. from 94% to 70%), which is unacceptable for node classification.\n\n\n[1] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset Condensation with Gradient Matching. In ICLR 2021\n\nQUESTIONS:\n- Is FGD capable of handling muti-class cases? In the paper authors mentioned to use average coherence to optimize the distilled graph, yet there are no experiments on this.\n- What is the computational complexity of the overall algorithm?\n\nLIMITATIONS:\n- The distilled graphs can't achieve comparable performance compared with original graphs.\n- The presented method is only tested in binary classification scenario.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aim to address the issue of fairness in graph data distillation, a process that condenses large real graphs into smaller distilled versions for more manageable computation with GNNs. They proposed FGD by introducing a new bias metric called coherence and using a bi-level optimization algorithm, which has shown to provide improved performance-fairness trade-offs in numerous experiments.\n\nSTRENGTHS:\n1.\tThis paper studies an interesting issue of fairness within distilled graphs, which arises due to the absence of sensitive features.\n2.\tThis paper devised a bias measurement named coherence specifically for distilled graphs, and suggests a framework utilizing this metric to facilitate the realization of fair graph distillation. The theoretical analysis in this paper is well founded.\n3.\tThe authors have conducted a thorough experimental analysis, and the presented results indicate that the proposed framework is adaptable to numerous renowned GNNs. This framework improves the trade-off between prediction performance and fairness across a range of datasets, signifying the framework's effectiveness and wide-ranging applicability.\n\nWEAKNESSES:\n1.\tAre there any fairness studies in dataset distillation in other fields such as computer vision and natural language processing? If so, there is a lack of discussion comparing the proposed framework with other similar works.\n2.\tThe time and space complexity are not mentioned.\n3.\tIn terms of methodology, this paper primarily uses strategies that were already developed before, and the unique contribution is the incorporation of a new loss term, Overall, the technical contribution of this work seems incremental.\n4.\tThe experimental part is inadequate, lacking integration of the proposed framework with large-scale and advanced GNNs, experiments in large datasets and the extra overhead of proposed framework.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nWhether the proposed framework can be applied to graph-level tasks, not just node-level ones.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper discovered the fairness problem in the distilled GNN methods and then introduce a fair graph distillation process to generate fair distilled graph data. To propose the algorithm, they also introduce a new bias metric, coherence, and propose a bi-level optimization framework, FGD, for fair graph distillation. Theoretical analysis are provided. Experimental results validate FGD’s effectiveness in mitigating bias while maintaining model utility across various GNN architectures and datasets.

STRENGTHS:
1.	Good presentation. This paper is easy and comfortable to read and follow.
2.	Solid theoretical analysis and experimental validation.
3.	This paper is the first paper to solve the fairness problem in distilled GNN lines, which have good novelty and contribution.

WEAKNESSES:
1. Some terminology needs simple descirption. It is better to include some preliminary knowledge or terminology description to make paper more self-included.

QUESTIONS:
1. What is the full name of FGD？ The full needs to be provided when this word first appears.
2. Why the sensitive attribute S is a diagonal matrix, not a vector with the dimension of number of node? Is it for computation convenience?
3. Some terminology needs simple description, such as span space and barycenter.
4. Is this methods only applicable to node classification task? Can it be extended to link prediction or graph classification?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the task of graph distillation (GD) from a fairness perspective. The authors found that current GD method amplifies bias in GNN training compared to training on original graphs. Since the distilled graphs do not contain node attributes, it's intractable to directly apply previous debiasing methods. To address this issue, the authors first made assumptions on the representation space of the distilled graph. Then they propose to measure the bias in the distilled graph representations using the least square distance between the distilled representations and the subgroup representations in the original graph. Technically a variance-based regularization is utilized to punish the model w.r.t. this measure. Extensive experiments demonstrate that the proposed method can benefit GNN training on distilled graphs with improved fairness.\n\nSTRENGTHS:\n1. This paper studies a novel topic and has its applicability in real-world scenarios. And it might bring in broader impact and more discussions on the characteristics and caveats of data distillation in other domains.\n\n2. The experiments are thorough.\n\nWEAKNESSES:\n1. The authors' design of the coherence loss in section 3.4 is confusing to me. Figure 2 shows that the intuition is to minimize the distance between z0 and z1 (as depicted by the red arcs in the figure), and this makes sense because fair representations should coincide in the attribute dimension. However, the definition in section 3.4 actually is the variance within group 0 or 1, and reducing this variance would only lead to more compact representations in each group, instead of bringing the two groups closer. Since the main contributions of this paper are based on this design, the authors should clarify this in a more rigorous manner.\n\n2. The empirical results on Pokec-z, German and Credit all show that the vanilla GD method can achieve comparable utilities with real graphs (within 5% accuracy gap) and worse fairness. Meanwhile, the results on Pokec-n and Recidivism show significantly worse utilities (up to 20% or 30% gap in accuracy), but better fairness. The improved fairness on these two datasets questions the motivation of this paper, that is the finding that the current GD method worsens GNN fairness. Also, I may wonder whether the authors have tuned the vanilla method right on these two datsets given such a large utility gap.\n\n3. There is not much logical connection between the theorems in sections 3.2 and the following part of the paper. Even if the span space does not match, or the barycenters are not consistent, it doesn't matter with the proposed method. The authors may need to justify the necessity of these theorems.\n\n4. Some typos. In the introduction part, the authors term the bias measurement as *consistency* (line 43), while it's *coherence* in other places. Line 143, it should be projection of z' instead of z. The defition of coherence loss in section 3.4 misses a superscript 2, since without this the formula is constant 0.\n\nQUESTIONS:\nSee comments above.\n\nLIMITATIONS:\nSee comments above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}