Reproducibility Study Of Learning Fair Graph Representations Via Automated Data Augmentations
Authors: Thijmen Nijdam
Juell Sprott
Taiki Papandreou-Lazos
Jurgen de Heus
Venue: Accepted by TMLR ()

Abstract: In this study, we undertake a reproducibility analysis of "Learning Fair Graph Representations Via Automated Data Augmentations" by Ling et al. (2022). We assess the validity of the original claims focused on node classification tasks and explore the performance of the Graphair framework in link prediction tasks. Our investigation reveals that we can partially reproduce one of the original three claims and fully substantiate the other two. Additionally, we broaden the application of Graphair from node classification to link prediction across various datasets. Our findings indicate that, while Graphair demonstrates a comparable fairness-accuracy trade-off to baseline models for mixed dyadic-level fairness, it has a superior trade-off for subgroup dyadic-level fairness. These findings underscore Graphair’s potential for wider adoption in graph-based learning. Our code base can be found on GitHub at https://github.com/juellsprott/graphair-reproducibility.

Reproducibility Study Of Learning Fair Graph Representa- tions Via Automated Data Augmentations
Thijmen Nijdam
thijmen.niydamm@sttdennnnn
University of Amsteriam
Juell Sprott University of Amsteniam
juell.sprott@studeen..uuuu uvo.
Taiki Papandreou-Lazze
taiki. www.nnrreeo.aazosssttteeeeoee
University of Amsterdam
Jurgen de Heus Universily of Amsterdam
jurgen.
Reviewed on OpenReview: https: // /pwpweeeeeeee.e..oooe/ net/ forum? id- 4WigHopXQX
Abstract
In this study, we undertake a reproducibility analysis of 'Learning Fair Graph Representa- tions Via Automated Data Augmentations" by Ling et al. (2022). We assess the validity of the original claims focused on node classification lasks and explore the performance of the Graphair framework in link prediction tasks. Our investigation reveals that we can partially reproduce of the original three claims and fully substantiate the other two. Additionally, we broaden the application of Graphair from node classification to link pre- dict ion across various datasets. Our findings indicate that, while Graphair demonstrates a comparable fairracy-accuracy trade-off to baseline models for mixed dyadic-level fairness, has a superior tradc-off for subgroup dyadic-level fairness. These findings underscore Graphair's potential for wider adoption in graph-based learning. Our code base can be found on GitHub at
1 Introduction
Graph Neural Networks (GNNs) have become increasingly popular for their exceptional performance in various applications (Hamaguchi et al. 2017 Liu et al. 2022 Han et al. 2022a A key application area is graph representation learning (Grover & Leskovec 2016 Hamilton 2020 Han et al. 2022b), where a significant concern is the potential for GNNs to inherit or amplify biases present in the graph representation training data. This can lead to discriminatory model behavior (Dai & Wang 2022). To address this issue, Ling al. (2022) introduced Graphair, an automated data augmentation tochnique aimed at learning fair graph representations without maintaining biases from the training data.
This work evaluates the main claims made by Ling et al. 2022). which were based solely the performance of the framework in node classification tasks. We expand our evaluation by conducting additional experiments to assess the adaptability and generalizability of Graphair through its application to a different downstream task, namely, link prediction. This approach allows us to further test. the performance and fairness of the embeddings.
For this purpose, we apply Graphair to new real-world datasets, adapting certain aspects of the framework and fairness metrics to suit. link prediction. Our contributions summarized as follows:

---

We replicate the original experiments to assess the reproducibility of the primary claims. We find that one of the three claims made by the original authors was partially reproducible, while the remaining claims are fully verified.
We adapt the Graphair framework for link prediction on various real-world datasets, which required modifications to both the framework and the fairness metrics. These adjustments provided valuable insights into Graphair's adaptability and generalizabiity to another downstream task. Our findings suggest that Graphair achieves superior trade-off in one of the fairness metrics used for this task.
2 Scope of reproducibility
The original paper Learning Fair Graph Representations Via Automated Data Augmentations by Ling et al. (0022) introduces Graphair, an innovalive aulomated graph augmentation method for fair graph represen- tation learning. This approach stands out from prior methods (Tuo et al. 2022 Zhao et al. 2022 Agarwal et al. 2021) by utilizing a dynamic and automated model to generate new, fairer version of the original graph, aiming to balance fairness and informativeness (Ling et al.) 2022).
In this work, we study the reproducibility of this paper. Besides examining the three main claims, we also assess the adaptability and effectiveness of Graphair in different context, namely link prediction. This extension tests the framework's ability to maintain fairness and informativeness in different downstream tasks. We will test this on variety of datasets. The claims and our extension are as follows:
Claim 1: Graphair consistently outperforms state-of-the-art baselines in node classification tasks for real-world graph datasets. Our extension evaluates whether this superior performance extends to link prediction lasks as well.
Claim 2: Both fair node features and graph topology structures contribute to mitigating prediction bias.
Claim 3: Graphair can automatically generate new graphs with fair node topology and features.
3 Methodology
3.1
Graphair
The Graphair framework introduces a novel approach to graph augmentation, focusing on the dual objec- tives of maintaining information richness and ensuring fairness. incorporates a model which adeptly transforms an input graph G {A,X, S}, where A represents the adjacency matrix, X the node features, and the sensitive attributes, into an augmented counterpart G' {A', .S. This transformation uti- lizes two main operations: Ta for adjusting the adjacency matrix A and Tx for masking node features X. A GNN-based encoder Jene precedes these transformations, tasked with extracting deep embeddings to inform these operations. The resulting augmented graph G" is designed to capture the core structural and feature-basee elements of the original graph G, while simultaneously keeping fairness principles to prevent the propagation of sensitive attribute information. This objective is achieved by training an adversarial model and using contrastive loss to optimize G'. An overview of the Graphair framework presented in Figure in Appendix A.1
3.1.1 Adversarial Training for Fairness
To ensure the fairness of the augmented graph, the model employs an adversarial training strategy. The adversarial model & learns to predict sensitive altributes S from the graph's features. Simultaneously, the representation encoder f and augmentation model g are optimized to minimize the predictive capability of the adversarial model, effectively removing biases from the augmented graph. The primary goal of adversarial training is to guide the encoder in generating representations that are free from sensitive attribute information. This optimization can be formally defined as follows:
2

---

n &
min max Ladv
min max 9.f
St log S; + (1 St) log(1 5.)
(1)
9.f
T1
Here, S, represents the predicted sensitive attributes, and n is the number of nodes.
3.1.2 Contrastive Learning for Informativeness
The model employs contrastive learning to enhance the informativeness of the augmented graphs. This method focuses on ensuring that the node representations between the original graph h2 and the augmented graph h4 maintain a high degree of similarity, thereby preserving key information. The positive pair this context defined as any pair (hę, kt). where h2 the node representation in the original graph and hę the corresponding representation in the augmented graph.
The contrastive learning function used to compute the loss for these positive pairs, is specified as follows:
exp(sim(h4, h/)/T)
Uh,, h() = --log log
(2)
E exp(sim(h;,, Ej Li exp(sim(hi, h15/T))
where 7 is the Lemperature scaling parameter, and lij is an indicator function that equals 1 if j o i and otherwise. This loss function aims to minimize the distance between similar node pairs while maximizing the distance between dissimilar pairs in the embedding space.
The overall contrastive loss Leon given by:
1 n
Loon
[1h,, h() + ((44 h.)
(3)
2n
3.1.3 Reconstruction Based Regularization to Ensure Graph Consistency
To ensure that the augmentation model produces graphs that do not deviate significantly from the input graphs, the model includes reconstruction-base regularization term in its overall training objective. Specif- ically, let LECF and LMSE represent the binary cross- entss-yytropy and mean squared error losses, respectively. The regularization term is mathematically expressed as:
Trrecnnst LBCE(A (A, A') XLMSE(X,X') X')
(4)
Aij log (A'i;) + (1 Aij) log (1 - A';;)] + X
Here, À is hyperparameter, and ||- denotes the Frobenius norm of a matrix.
3.1.4 Integrated Training Objective
By emphasizing informativeness and fairness properties Graphair seeks to produce augmented graphs that are less susceptible to bias. This approach contributes to fairer graph representation learning while preserving the valuable information contained in the data. The overall training process is described as the following min-max optimization procedure, where a, 3, and y are hyperparamcters that balance the different loss components:
min max L = min max aLady + BLoon + > Precanst
(5)
f.9
1.9
3

---

3.2 Datasets
To replicate the main claims, we use the same datasets the original paper by Ling el al. (2022) which include specific dataset splits and sensitive and target attributes. We employ three real-world graph datasets: N containing player statistics, and two subsets of the Pokec social network from Slovakia, namely Pokec- n and Pokec-z (Dai & Wang 2021). The specifics of these datasets are summarized in Table 1
For the link prediction task, we utilize well-establishhen benchmark datasets in this domain: Cora, Citeseer, and Pubmed (Spinelli et al. 2021 Chen al., 2022 Current et al. 2022 Li et al., 2021). These datasets feature scientific publications as nodes with oag-of-words vectors of abstracts
node features. Edges represent citation links between publications. Notably, these datasets possess a broader range of features compared to those used by Ling et al. (2022) and have a larger set of possible sensitive attributes |S]. Full details are provided in Table
Table 1: Dataset Statistics.
Dataset
S
[S]
Features
Nodes
Edges
NBA
nationality
2
39
403
16,570
Pokec-z
region
2
59
67,797
882,765
Pokec-n
region
2
59
66,569
729,129
Citeseer
paper class
6
3,703
3,327
9,104
Cora
paper class
7
1,433
2,708
10,556
PubMed
paper class
3
500
19,717
88,648
3.3 Experimental Setup
We obtain the model's codebase from the DIG library, more specifically, from the FairGraph module To enhance reproducibility, we employ complete seeding across all operations, which was missing in some operations of the original code. A key difference in the experimental sctup between the one reported by the original authors and ours is that we conducted 10,000-epoch grid search for the Pokec dataset, instead of the 500-epoch grid search initially reported by (Ling et al. 2022 This modification was recommended by the original authors to enhance reprodueibility. We refer to the subsection A.4 for more details, where we show that a 500-epoch search does not yield optimal results for the Pokec datasets, but higher epochs improve performance in accuracy and fairness. To verify Claims and 2, we follow the procedure described by Ling et al. 2002).. For Claim 3, due memory constraints, we compute the homophily and Spearman correlation values for the Pokec datasets on mini-batch instead of the entire graphs Aside from this, we adhere to the same procedures for this claim well.
3.4 Link Prediction
In our study, we extend the scope of the original work by performing different downstream task, namely link prediction. We implement several modifications to adjust the Graphair network for the downstream task of link prediction. First of all, we modifly the output size of the adversarial network from two, which corresponded to the binary sensitive feature in the Pokec and NBA datasets, to the respective number of distinct values of the sensitive feature in the corresponding dataset.
To create the link emboddings for the input of the classifier, we compute the Hadamard product of the node embeddings (Horn& & Johnson, 2012), which pairs node embeddings effectively. For nodes and u, we define the link embedding hyu as:
h = h o h.
(6)
These new link embeddings require labels that reflect the sensitive attributes of the connected nodes. To this end, we integrate dyadic-level fairness criteria into our fairness assessment for link prediction. We form

---

dyadic groups that relate sensitive node attributes to link attributes, following the mixed and subgroup dyadic-level fairness principles suggested by Masrour et al. (2020). The mixed dyadic-level groups classify links as either inter- or intra-group based on whether they connect nodes from the same or different sensitive groups. The subgroup dyadic-level approach assigns each link to a subgroup based on the sensitive groups of the nodes it connects. A subgroup formed for each possible combination of sensitive groups. This method facilitates the measurement of fairness two distinct aspects. Firstly, assesses how well each protected subgroup is represented in link formation at the subgroup dyadic-level. Secondly, it evaluates node homogeneity for links at the mixed dyadic-level.
For asscssing fairness in link prediction, we aim to optimize for Equality of Opportunity (EO) and Demo- graphic Parity (DP) using these dyadic groups. Following the original paper's definition, DP is defined as:
P( = D 0) P(V -
and EO as:
|P( = O,Y = = -1
where Y is a ground-truth label, Y is prediction, and in the case of link prediction, D denotes the dyadic group to which the link belongs. These definitions extend to multiple dyadic groups (ID) > 2), which is the case for subgroup dyadic analysis. As our final metrics, we define the Demographic Parity difference (ADP) as the selection rate gap and the Equal Opportunity difference (AEO) as the largest discrepancy true positive rates (TPR) and false positive rates (FPR) across groups identified by D:
ADP max P( D d) min d), d
ATPR = max PP& = =1|D d, =) min p(y = 1|D = d,Y Y = 1),
AFPR = max IP8 = 1ID = d,y = Pin = 1|D = =d 0),
d
AEO = max ATPR, AFPR).
Link prediction performance is measured using accuracy and the Area Under the ROC Curve (AUC) metrics, representing the trade off between true and false positives.
3.4.1 Baselines
We adopt FairAdj (Li et al. 2021 and FairDrop (pinelli et al. 2021 as our benchmarks. FairAdj learns a lair adjacency matrix during an end-to-end link prediction lask. It utilizes a graph variational autoencoder and employs two distinct optimization processes: one for learning a fair version of the adjacency matrix and the other for link prodiction. We used two versions of FairAdj: one with epochs (r = 5) and the other with 20 epochs (r 20). FairDrop, on the other hand, proposes a biased edge dropout algorithm counteract homophily and improve fairness in graph representation learning. We used two different versions of FairDrop: one with Graph Convolutional Network (GCN) (Kipf & Welling 2016) and the other with Graph Attention Networks (GAT) (Velickovic et al. 2017).
3.5 Hyperparameters
Node Classification To align our experiments closely with the original study, adopt the hyperparam- eters specified by the authors. This includes conducting grid search on the hyperparancters a, 7, and X with the values {0.1, 1.0, 10.0), as performed in the original work <Ling et al. 2022). We use the default settings from the original code where specific hyperparameters are not disclosed, a choice validated by the original authors. complete list of all hyperparameters provided Table subsection A.2
Link Prediction We replicate the grid search from the node classification experiments for link prediction the Citeseer, Cora, and PubMed datasets. Initially, conduct grid search the model parameters,
5

---

including varying the number of epochs, the learning rates for both the Graphair module and the classifier, and the sizes of the hidden layers for both components. We select the most notable model setup based on performance metrics (accuracy and ROC) and fairness values, and then perform a subsequent grid search on the loss hyperparameters a, X, and y to fine-tune the model further.
We compare the results of Graphair with baseline results from Spinelli et al. (2021 and Li et al. (2021 which also underwent grid searches. For FairAdj, we conducted a grid search focusing on model parameters, involving variations in learning rates, hidden layer sizes, number of outer epochs, and the specific configu- ration parameter. We evaluated two versions of FairAdj: one with 5 epochs and the other with 20 epochs. For FairDrop, we also performed grid search on the model parameters, testing different learning rates, epoch counts, and hidden layer sizes. We evaluated two configurations of FairDrop: one using a Graph Convolutional Network (GCN) and the other using Graph Attention Networks (GAT). More detailed infor- mation on the hyperparameters fine-tuned during the grid search for each model presented in Table subsection A.2
3.6 Computational requirements
All of our experiments conducted on high-performance computing (HPC) cluster, that features NVIDIA A100 GPUs, divided into four partitions with combined memory of 40 GB. For detailed overview of the GPU hours required for each experiment, see Table 9 in subsecttin A.3 A rough estimate suggests that a total of 80 GPU hours are necessary to complete all experiments.
4 Results
This section presents the outcomes of our experimental results aimed at reproducing and extending the findings the original work on Graphair. We discuss the reproducibility of specific claims made in the original paper in |subsoction 4.1 and explore the performance of Graphair in a link prediction downstream task in ionsecttion 4.2
4.1 Results reproducing original paper
To verify Claim we performed node classification on the NBA, Pokec-n, and Pokec-z datasets. Table 2 shows a comparison of the results reported by the original authors and the results of bascline models given by the original authors with those obtained by us through replicating the experimental setup described in subsection 3.3 Consistent with the original study, our results are derived by selecting the best outcome from the grid search procedure. We observe that the results for the NBA dataset are similar to those reported by the authors. However, for the Pokec datasets, our Graphair model gets better fairness scores at the cost of worse accuracies.
Claim 1:
When examining the fairness-accuracy trade-off in Figure I which uses the ADP fairness metric, we see that for the NBA dataset reflected by similar trend, but with lower accuracy scores. The Pokec-n dataset also shows similar trend but fails to reach the higher accuracies of the original model. Considering that the code we used from the DIG library differs from what the original authors used, combined with the fact that a different number of epochs, namely 10,000 was used for the Pokec dataset instead of the originally reported 500, we think there might still be some differences in the experimental setups. Even though these discropancies are probably minor, they do not allow us achieve better performance in terms of the accuracy accuracy-fairness trade-off for all datasets compared to baseline models, which makes only partially able to reproduce Claim 1.
achieve similar trade-off. For the Pokec-z data, small discrepancy is

---

Table 2: Comparison of the results reported by the original authors with those obtained by us.
NBA
Pakeo-n
Polle--
Methous
Aro 4
ACC
Anp
Apo
WallWalk
64.54
2.35
3.67
1.28
9.12
7.06
67.07
0.24
1.12
0.74
8.24
0.75
65.23
0.78
4.45
1,25
4.59
0.96
Pair W
60.74
1.71
14.61
+ 4.98
12.01
5:38
69:01
+ 0.38
T.59
0.56
9.69
+ 0.09
67.65 + 0.00
4.46
+ 0.38
6.11 + 0.54
CRACE
70.14
1.40
7.49
3.78
7.67
3.78
68.25
0.90
6.41
0.71
7.38
0.84
67.81
0.41
10.77
0.68
10.69
0.69
GCA
70.48 + 1.19
18.08
+ 4.80
20.04
4.34
69.34 + 0.20
6.07
0.56
7.39
+ 0.82
67
+ 0.14
7:90
1.30
8.03 + 1.07
FairDrop
69.01 1.11
1.06
7.61
2:21
67.78
0.60
5.77
1.83
5.48
1.32
67.32
0.61
1.05 4.05
1,00
NIFTY
60.33 0.09
3.31
1.02
4.70
1.04
67.15 + 0.43
4.40
0.59
3.75 + 1.04
65.52
+ 0.31
6.51
0.51
0.14 0.68
AngrAng
66.38 0.85
4.90
1.95
0.49
6.77
+ 0.19
5.10
0.60
0.84
Graphair
69.36
0.45
2.56
0.41
4.64
0.17
67.43 2: 0.25
2.02
0.40
1,82
A 0.47
68.17 20.08
2.10
4 0.17
2.76 0.19
Grapheir (ours)
68.54
0.32
0.19
0.32
65.76
0.72
0.34
0.41
0.40
65.22
0.01
0.29
2.24
0.31
NBA
Pokec
Pokec N
66.5
Drighta
060
Targag
Crepher lousi
Craphar
ACC
26.5
Orighar buss
Gushhar
03
Terkug
ADP
ADP
ADP
Figure 1: ACC and DP trade-off for baselines, Graphair and our results for Graphair. Upper-left corner (high accuracy, low demographic parity) is preferable.
Claim 2: Table 3 confirms that Claim 2 is substantiated for both the NBA and Pokee-z datasets. In com- parisons of Graphair with and without feature masking (FM) or edge perturbation (EP), notable incrcases in fairness metrics are observed. This supports the claim that each model component contributes to mitigating prediction bias. Interestingly, we notice an increase in accuracy when removing feature masking across all datasots, a result that deviates from findings the original work, which showcased similar accuracy scores when EP removed. We attribute this to the use of more training epochs in our experimental setup. It seems logical that performance would improve when edge perturbation is removed, as the encoder model Jene utilize the original adjacency matrix. This allows the classifier exploit increased homophily in the network, thereby incrcasing accuracy and worsening fairness.
Table 3: Comparisons among different components in the augmentation model.
Polne-n
Methods
Apo
Agr
ACC
Agr I
Ago
Graphalr (ours)
68.54 0:40
0.27
=
66.76 0.02
0.72
0.42
1.32
2.24 0.35
Graphair w/o EP (ours)
72.68
0.40
2.95
0/05
2.53
67.26 +:
2.11 0.00
1.37 0.38
60.25
0.10
6.56 0.74
6.73 0.65
Graphair w/o FM (ouns]
67.70
0.32
10.73 + 1.12
26.70 2.53
64.61 0.36
3.8 + 0.47
3.18 = 0.27
57.91 0.13
6.19 + 0.74
6.34 0:33
Claim 3: The plots in Figure 2 clearly illustrate that the homophily values for the augmented graph are lower than Uhose for the original graph. These results support the authors' clain that Graphair aulomatically generates new graphs with a fairer node topology. The plots in Figure 3 reveal that, for all three datasets, the features with the highest Spearman correlation to the sensitive feature generally exhibit lower values the fair view. These findings lend support to the authors' claim that Graphair produces features that are fairer. We therefore conclude that Claim 3 is fully reproducible.
7

---

NBA
Pokec
N
FST vied Origie
Farv Oriynal
Fervel Oiiina
1363
Denstyy
0.03
0.03
Volue
Voluee
Vo.uue
Figure 2: Node sensitive homophily distributions in the original and the fair graph data.
NBA
Pokec
Pokec N
Original
Org
corviation
172
Sgwea mean cormelation
8:38
corvelation
0.2
0.2
0.35
Feature Index
Feature Index
Figure 3: Spearman correlation between node features and the sensitive attribute in the original and the fair graph data.
4.2 Results beyond original paper
The performance of Graphair compared to baseline models is presented in [Tbble 4 Table 5] and Table 6 For lairness metrics, subscripts m and denote application to mixed groups and subgroups, respectively. Our findings indicate that Graphair is outperformed by both FairDrop variants in terms of accuracy and AUC across all datascts. However, compared to FairDrop, Graphair excels all fairness metrics on all datasets, particularly in the subgroup variants the Citeseer and Cora datasets. FairAdj demonstrates comparable performance in accuracy and AUC relative to Graphair but performs worse in subgroup fairness metrics, only excelling in the ADP,, metric across all datasets.
We further investigate the trade-off between accuracy and ADP for both mixed and subgroup variants across the three datasets for each model, as shown in Figure 4 The trade-off for Graphair is comparable to the baseline models for the mixed variant, while achieving a superior trade-off for the subgroup variant. While Spinelli al. (2021) suggest that high scores for subgroup fairness metrics are due dataset characteristics, find that Graphair improves this metric through its augmentative approach, which generates augmenta- tions with similar predictive power across different subgroups. This enables Graphair to consistently predict links with the same probability across various subgroups, resulting in lower subgroup dladic-level fairness while maintaining predictive power.
Table 4: Link Prediction on Citesocr
Method
Accuracy
AUC f
ADP
AEO..
ADP,
AEO,
FairAdje-a
77.1 1: 2.5
83.4 1: 2.3
33.8
8.2
4.1
65.6 : 6.2
80.0 9.0
FairAdje=z0
73.5 1 2.8
81.2 1 3.0
26.0
1 3.4
5.3
1
3.5
56.5 1 7.5
70.1 8.00
GCN FrooDop
87.3 + 1.7
97.1 - 1.6
46.7 = 3.0
8.8
4.5
08.9 = 6.0
41.1 + 10.0
Gar+FairDrop
86.3 + 13
96.6 + 1.2
45.0 = 2.7
9.6
+
4.8
08.4 - 5.9
39.3 9.8
Graphair (ours)
79.2 + 0.7
86.9 +: 0.6
40.5 = 0.6
1.1
+
1.5
2.9 4 3.0
11.3 = 3.8
8

---

Table 5: Link Prediction on Cora
Method
Accuracy
AUC
ADP 4
ABO..
ADP,
AEO,
FairAdje=5
76.6 i
83.8
38.9
4.7
78.5 ::
100.0 8.0
Adjee00
73.0 =
78.9 +
29.3 3.1
4.7 = 4.7
83.3 =
97.5 7.8
GCN PairDrop
90.4 1 1.1
97.0 1 0.8
58.2 1 2.7
17.3
5.2
93.6 1 3.7
98.5 1 0.5
C.T Drop
85.4 + 144
96.2 +: 1.2
53.2
1 3.0
17.8
1 4.6
84.7 = 2.3
98.2 1 0.5
Graphair (ours)
75.2 + 0.8
83.3 + 0.9
38.7
+ 0.8
12.7 +
1.7
37.9 + 3.2
13.3 + 2.1
Table 6: Link Prediction on PubMed
Method
Accuracy
AUC
ADP
AEO J
ADP, 4
ΔEO,
FairAdje=5
83.7 + 3.2
90.6 + 4.7
40.76
+ 3.7
35.5
+ 2.5
40.7 = 3.2
16.7 + 1.9
76.1 = 1.8
83.7
2.1
37.2
= 2.5
22.6
2.0
53.4
5.5
39.9 - 5.4
GCN + FairDrop
92.3 1 0.4
97.4 1 0.2
44.2
1 0.5
7.6
0.7
54.1
1 1.5
15.3 1 266
GATT+FairDro
92.2 = 0.8
97.3 + 0.7
43.7
0.9
7
1.1
54.5
= 2.1
14.3 - 4.1
Graphair (ours)
82.3 0.2
89.8 + 2.9
37.6 + 0.4
2.6 + 0.7
36.4
+ 2.4
8.0
2.4
Citeseer
Cora
PubMed
Cupter
Srrpp eCR KiCOp
PSIDRE GCN-ITeEErop Taridj-s farid)-x
GAT-Tertinp GoHe/axtinp
Patibatt
1ind
Faniq
ACC
70
20
23
90
25
25
ADPm
ADPm
ADPm
PubMed
Goguar
025
Tark--x
ACC
aeppoo
GRT har22pp
05
farAd
ADP:
Figure 4: ACC and DP trade-off for the baselines and our Graphair for link prediction. The top row shows the ADP,, metric, and the bottom shows the ADP, metric. Points in the upper-left corner are desired.
5
Discussion
Upon revisiting the three claims in our study, we find that Claim is partially reproducible, whereas Claims and are fully reproducible. In the case of Claim 1, while we were able to replicate the performance of the NBA dataset consistent with the original paper, discrepancies emerged with the Pokec datasets. Specifically, results showed improved fairness scores the expense of lower accuracies compared to the original findings. This could be attributed to differences in experimental setup, particularly the number of training epochs used, which deviated from the original study's methods. We used 10,000 epochs for the Pokec datasets as opposed to the 500 reported in the original paper, a change recommended by the original authors.
Further analysis of Graphair's performance link prediction indicates that, while demonstrates a compa- rable fairncss- trade-off baseline models for mixed dyadic-level fairness, Graphair has superior trade-off for subgroup dyadic-level fairness.

---

5.1 What was easy and what was difficult
The clarity of the code within the DIG library significantly facilitated reproducibility. The original paper provided a clear outline of the experiments, enabling a straightforward process to identify the necessary components for reproducing the study's 's claims and implementing our link prediction extensions.
We encountered initial challenges with the reproducibility of Claim 1, which necessitated seeking clarification from the authors. Correspondence with the original authors resolved issues related unspecified hyperpa- rameter settings and a bug in the code. Reproducing Claim for the Pokec datasets proved non-trivial due to the large memory requirements for processing the full graph, necessitating solutions to acquire experimental results.
5.2 Communication with original authors
We initiated contact with one of the authors, Hongyi Ling, via email to seek clarification our initial results that did not match those of the original paper. These discrepancies were resolved, and the authors responded promptly to our emails, providing valuable feedback. Most notably, they recommended change in our experimental setup for the Pokec datasets, specifically increasing the number epochs from 500 to 10,000.
10

---

References
Chirag Agarwal, Himabindu Lakkaraju, and Marinka Zitnik. Towards unified framework for fair and stable graph representation learning. In Uncertainty in Artificial Intelligence, pp. 2114-2124. PMLR, 2021.
April Chen, Ryan Rossi, Nedim Lipka, Jane Hoffswell, Gromit Chan, Shunan Guo, Eunyce Koh, Sungchul Kim, and Nesreen K Ahmed. Graph learning with localized neighborhood fairness. arXiv preprint arXiv:2212. 12040, 2022.
Sean Current, Yuntian He, Saket Gurukar, and Srinivasan Parthasarathy. Fairegm: Fair link prediction and recommendation via emulated graph modification. In Proceedings of the 2nd ACM Conference on Equity and Access in Algorithmss, Mechanisms, and Oplimization, pp. 1 14, 2022.
Enyan Dai and Suhang Wang. Say no to the discrimination: Learning fair graph neural networks with limited sensitive attribute information. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pp. 680-688, 2021.
Enyan Dai and Suhang Wang. Learning fair graph neural networks with limited and private sensitive attribute information. IEEE Transactions Knowledge and Data Engineering, 2022.
Aditya Grover and Jure Leskovec. node2voc: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855 864, 2016.
Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. Knowledge transfer for out-of- knowledge-bar entities: A graph neural network approach. arXiv preprinl. arXiv: 10l::7706.0555 706.05674, 2017.
William L Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image is worthh graph of nodes. Aduances in Neural Information Processing Systems, 35:8291 1882991-333,, 2022a.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, Qingquan Song, Jundong Li, and Xia Hu. Geometric graph representation learning via maximizing rate reduction. In Proceedings of the ACM Web Conference 2022, pp. 1226-1237, 2022b.
Roger A Horn and Charles R Johnson. Matriz analysis. Cambridge university press, 2012.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint ar wxXt:1609.02900 2016.
Peizhao Li, Yifei Wang, Han Zhao, Pengyu Hong, and Hongfu Liu. On dyadic fairness: Exploring and mitigating bias graph connections. In International Conference on Learning Representations, 2021.
Hongvi Ling, Zhimeng Jiang, Youzhi Luo, Shuiwang Ji, and Na Zou. Learning fair graph representations via automated data augmentations. In The Fleventh International Conference on Learning Representations, 2022.
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztckin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Represenlations (ICLR), 2022.
Youzhi Luo, Michael McThrow, Wing Yee Au, Tao Komikado, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Automated data augmentations for graph classification. arXiv preprint arXiv:2202.182 2022.
Farzan Masrour, Tyler Wilson, Heng Yan, Pang-Ning Tan, and Abdol Esfahanian. Bursting the filter bubble: Fairness-aware network link prediction. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 841-848, 2020.
Indro Spinelli, Simone Scardapane, Amir Hussain, and Aurelio Uncini. Fairdrop: Biasod edge dropout for enhancing fairness in graph representation learning. IEEE Transactions on Artificial Intelligence, 3(3): 344-30 154, 2021.
11

---

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, Yoshua Bengio, et al. Graph attention networks. stal, 100020))00-48550, 2017.
Tong Zhao, Wei Jin, Yozen Liu, Yingheng Wang, Gang Liu, Stephan Günnemann, Neil Shah, and Meng Jiang. Graph data augmentation for graph machine learning: A survey. arXiu preprint arXin:2202.088 088711 2022.
12

---

A Appendix
A.1
Overview of the Graphair model
Input Graphy
Augmentation Moduleg
Fair Viewg"
MLPAC-)
Samplng
GNN
A'
Bene
M
Sampling
M
M
X'
X
Adversary k(
GNN f(
Shared Weights
GNN f())
Sensitive Attribute
Node Feature
Add Delete Edge
Maximize Agreement
Fairness
Mask Node Feature
con
Informativeness
Figure Overview of the Graphair framework (Ling et al. 2022
A.2
List of all hyperparameters
Table 7: Overview of hyperparameters for the Graphair model m and the evaluation classifier on all datasots.
Hyperparameter
NBA
Pokec-n
Pokec-z
Citeseer
Cora
PubMed
1.0
10.0
0.1
10.0
10.0
0.1
1.0
10.0
0.1
10.0
10.0
7
0.1
0.1
0.1
0.1
0.1
0.1
À
1.0
10.0
10.0
1.0
10.0
0.1
Chidden
128
128
128
128
128
128
Qlarningg_ate
le-3
le-3
le-3
5e-3
5e-3
5e-3
Cweight__ doeay
le-5
le-5
le-5
1c-5
le-5
le-5
Meaarinng
le-4
le-4
le-4
le-4
le-4
le-4
Mweight decuy
1c-5
1c-5
1c-5
1c-5
1o-5
1c-5
Table 8
presents comprehensive overview of the hyperparameters adjusted during the grid search. The initial seven rows correspond to the Graphair model, while the subsequent models used for link prediction tasks. Regarding the epoch count for node classification with Graphair, 500 epochs were used for the NBA dataset and 10,000 for the Pokec datasets. For grid searches with the Graphair model for link prediction, the number of epochs was set to 200.
correspond to the baseline
13

---

Table 8: Overview hyperparameters tuned the grid search.
Hyperparameter
Node Classification
Link Prediction
a
0.1, 1, 10}
0.1, 1, 10}
0.1, 10]
{0.1, 10]
A
(0.1, 10]
10,1, 10)
Classifier Ir
le-3
{le-2, le-3, le-4)
Model Ir
le-4
{le-2, 1e-3, le-4]
Classifier Hidden Dimension
128
(64, 128, 256)
Model Hidden Dimension
128
{64, 128, 256]
Model Ir (PairAdj)*
{0.1, le-2, le-3)
hidden1 (FairAdj)
{16, 32, 64)
hidden2 (FairAdj)
{16, 32, 64)
outer epochs (FairAdj)
{4, 10, 20}
{100, 200, 500, 1000)
Model Ir (FairDrop)
{5e-2, 2, le-2, 5e-3, le-3, 5e-4, le-4
Hidden dim. (FairDrop)
{64, 128, 256, 512)
A.3 GPU Run hours
Table 9: Computational Requirement Overview
Name of the experiment
GPU Hour (Hour)
Max GPU Memory Usage (GB)
Claim (grid search)
20
3.70
Claim
2
3.67
Claim
0.5
3.70
Link Prediction (grid search)
60
3.70
GPU hour is measured by the amount of time each experiment script noeded from the start the end. Maximum GPU memory usage determined by max memory allocated method from the Pytorch library.
A.4 Impact of the number of epochs on the accuracy and fairness results
Pokec Z
Pokec N
0.05
Accurack
Faieness LT0
Accuracy
Fairness DP Feirness
Farmess EO
0.61
0.63
Accuracy
Faimess
accuracy
Fairness
0.82
0.62
0.62
0.61
0.00
0.60
0.59
8000
3000
2000
4000
7000
Epochs
Epachs
Figure 6: Impact of the number of epochs on the accuracy and fairness results.
Following the original authors' rocommendation increase the number of epochs order to replicate their results, we performed an ablation study examine the impact of the number of epochs on accuracy and fairness metrics. As illustrated in Figure there is a positive correlation between the number of epochs and
14

---

both accuracy and fairness metries. Based on these findings, conducted experiments using 10,000 epochs for the Pokec datasets as this led to notable increase in performance.
15


Reviews:
[{"review_id": "upyt3NI3JQ", "reviewer": "Reviewer_QVW1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper are as follows.\n\n- The reproducibility analysis is welcome for TMLR, which expects diverse and high-practicality submissions.\n- The authors have established a more reasonable experimental setup compared with the original paper of ICLR 2023.\n- The authors have released their codes.\n\nThe weaknesses of this paper are as follows.\n\n- The authors have ignored a careful grid search of the other compared baselines. If the proposed method does not use grid search, it is questionable for the compared methods.\n- The Graphair has poor accuracy while achieving high accuracy. As shown in Table 4, Table 5, and Table 6, Graphair is far worse on accuracy-based metrics: Accuracy and AUC. As we know, for graph tasks such as link prediction, etc., the drop in 0.01-level on AUC could be a significant drop. Therefore, Graphair is still not so powerful even if the authors have conducted a careful grid search.\n- Although reproducibility analysis is a good topic for TMLR, we would like to see more insightful observations in addition to the original paper, since TMLR has very high standards.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts a reproducibility analysis of Graphair (ICLR 2023) with an extensive grid search, which results in new findings different from the existing papers. The paper also explores wider applications of Graphair.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my concerns above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WhmCny3bj3", "reviewer": "Reviewer_mrv3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Not only attempts to replicate prior work, but also extends its scope to link prediction.\n- Addresses a topic that should be of high interest to TMLR readers.\n\nWeaknesses:\n- Findings are of questionable value to the community due to the limited scope of the study. It is unclear why the results obtained by the authors for Claims 1 and 2 differ so much from Ling et al. (2022). I elaborate further in Requested Changes.\n- Several presentation issues. See both the major and minor issues listed in Requested Changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors attempt to reproduce the results of the paper \"Learning Fair Graph Representations Via Automated Data Augmentations\" by Ling et al. (2022), where the Graphair framework was proposed. Furthermore, the authors extend the Graphair framework from the original node classification setting in Ling et al. (2022) to link prediction. The main findings are as follows:\n- One of the claims in Ling et al. (2022) can be fully verified, while the other two can only be partially verified.\n- The Graphair framework also performs quite well on the link prediction, resulting in smaller deviations from fairness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major issues:\n- Table 2 only shows a comparison between your implementation of Graphair and the results reported by the authors. The other methods that Ling et al. (2022) compare against are missing, so it is difficult to interpret the results. After examining Table 1 in Ling et al. (2022), I see that the accuracy values you are obtaining on the Pokec data sets are far lower than the other baselines, which is a large difference that needs to be accounted for. To further investigate the discrepancy you observe,  consider expanding the scope of your replication study to include additional methods. Perhaps this will help you to identify what is causing the discrepancy with Graphair.\n- Description of results for Claim 2 in Section 4.1 are misleading: \"we observe clear increases in both fairness metrics and accuracy\". Indeed, fairness metrics are improving, but accuracy is not. This discussion should be revised.\n\nMinor issues:\n- What is the FairAdj method in Section 4.2? It is not discussed at all. Add at least a citation and brief description.\n- Hyperparameter $\\lambda$ in Section 3.3 has not been mentioned previously in the paper, unlike the other 3 hyperparameters.\n- Fairness metrics reported in Tables 4 and 5 have $m$ and $s$ subscripts, which are not described. I assume these correspond to the mixed and subgroup fairness principles discussed in Section 3.4.1.", "broader_impact_concerns": "None that I observe.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p9Uie7YuL8", "reviewer": "Reviewer_yRgK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The reproduced paper GraphAir, as of March 2024 has 29 citations so it has reasonable impact. Therefore, a reproduction of it will be of interest to some of the audience of TMLR. \n2. The paper is generally well written and easy to follow. \n\n## Weaknesses\nBefore I state the weaknesses of this paper, I would admit that I am not an expert in fair graph learning, so my comments are from a generally high level. \n1. Not sure about the 'training instability of GraphAir'. This paper states in Section 4.1 that 'this issue might also stems from the observed training instability', but no evidence regarding the training instability is given or discussed, which makes it hard to understand the detailed cause of this paper.\n2. I am not so sure about the setting of citation datasets. According to the authors, the sensitive attributes are paper classes. I am wondering why the setting is done as such, as paper classes are generally not considered sensitive. Moreover, the paper classes are just the node labels and may be very directly related to the citation relations. Therefore, I am a bit confused about why the setting is done as such. Of course, the authors say that this follows another paper (Spinelli et al. 2021), which is fine. However I do like to know some rationale underlying it. \n3. I am not so convinced by the results in Table 4-6. As can be seen in Table 4-6, there is a very clear tradeoff between fairness and accuracy, and GraphAir commonly excels on fairness but not so on Accuracy. I think it is difficult to draw a conclusion about which method is definitely better, or whether the performance gap is large or small, as comparison on two dimensions cannot be solely determined by one dimension. Are there any ways to make the comparison more thorough by involving both dimensions (accuracy, fairness)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the reproducibility of the paper \"Learning fair graph representations via automated data augmentations\" as well as extending the scope of the paper (method of which is called GraphAir) to link prediction. The reproduction experiments of GraphAir shows that two of the three claims are partially verified while the third one is fully verified. In addition, the authors experiment GraphAir on link prediction tasks, with results showing the good performances of GraphAir.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please show the training stability issue of GraphAir, and discuss whether some common ways to mitigate instability (e.g. gradient clipping, regularization, larger batches, etc.) will work to mitigate it. \n2. Please provide some insights about why the paper classes are used as sensitive attributes. I have a slight concern that this attribute may not act similarly as commonly considered sensitive attributes (e.g. gender, age, etc.). \n3. Please consider some better evaluation methods to jointly evaluate both the fairness and the accuracy for Tables 4-6.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the reviewers recommended to accept the paper while one recommended rejection. The points raised by the reviewer recommending rejection were (1) a lack of grid search for the baselines, which have been addressed in the revision and (2) a concern regarding the relevance of reproducibility analysis for TMLR. Overall, and without further details from the reviewer, I believe the paper should be accepted. 

When preparing the final version of the paper, please address the following two points raised by reviewer mrv3:

- Section 3.1.3: lambda is missing in front of Frobenius norm in the second line of the equation.
- Inappropriate use of text citations without parentheses in several locations, such as Section 3.4.1: "FairAdj Li et al. (2021) and FairDrop Spinelli et al. (2021)" should both be replaced with the parenthesis-based citations. This also appears in some other locations, such as Section 3.4: "the node embeddings Horn & Johnson (2012)"
Decision: Accept (poster)