Reproducibility study of FairAC
Authors: Gijs de Jong
Macha J. Meijer
Derck W. E. Prinzhorn
Harold Ruiter
Venue: Accepted by TMLR ()

Abstract: This work aims to reproduce the findings of the paper "Fair Attribute Completion on Graph with Missing Attributes" written by Guo et al. (2023) by investigating the claims made in the paper. This paper suggests that the results of the original paper are reproducible and thus, the claims hold. However, the claim that FairAC is a generic framework for many downstream tasks is very broad and could therefore only be partially tested. Moreover, we show that FairAC is generalizable to various datasets and sensitive attributes and show evidence that the improvement in group fairness of the FairAC framework does not come at the expense of individual fairness. Lastly, the codebase of FairAC has been refactored and is now easily applicable for various datasets and models.

Published in Transactions on Machine Learning Research (06/2024)
Reproducibility study of FairAC
Gijs de Jong' University of Amstendam
gijs.de. http.dd..ong@sstudedd.....vv
Macha J. Meijer University of Amsterdam
macha..acijrrB@studeed........
Derck W. E. Prinzhorn'
derdkppriichoom@@ssdddee......
University of Amstendam
Harold Ruiter"
harold. www.errttttrrrrrrta.e uvo.nd
University of Amsterdam
Reviewed on OpenReview:
Abstract
This work aims to reproduce the findings of the paper "Fair Attribute Completion on Graph with Missing Attributes' written by Guo, Chu, and Li by investigating the claims made in the paper. This paper suggests that the results of the original paper reproducible and thus, the claims hold. However, the claim that FairAC is generic framework for many downstream tasks is very broad and could therefore only be partially tested. Moreover, we show that FairAC generalizable to various datasets and sensitive attributes and show evidence that the improvement in group fairness of the FairAC framework does not come at the expense of individual fairness. Lastly, the codebase of FairAC has been refactored and is now casily applicable for various datasets and models.
1 Introduction
In recent years, graphs are an increasingly common data utructure used in real-world applications 2 Graph neural networks (GNNs) have shown promising performance a large range of tasks, such as community detection 3 node classification 4 link prediction 5 and graph classification 6]. However, there is a possibilityy that the graph data may be biased, for example due to under-representation of specific group 7. If the model is trained this biased graph, the model may be unfair with respect to certain sensitive attributes such demographic or gender 8 g Aside from this, the node information of graph datasets often is incomplete, for example when a node was recently added to network 10
To address the problem of attribute completion and fairness in graph data, the FairAC framework was proposed by Guo, Chu, and Li [. FairAC is a fair attribute completion framework which can be applied a dataset, after which the result can be used as input for many downstream tasks, such as node classification or link prediction.
The aim of this paper is to reproduce the original paper of FairAC and extend on the original pa- per. In summary, the following contributions are made:
The original FairAC paper is reproduced and the original claims are analysed.
2. The codebase of FairAC is improved and more casily applicable to other datascts and models.
'Equal contribution.

---

Published in Transactions on Machine Learning Research (06/2024)
3. The original paper is analyzed further by using new datasets, testing the framework various sensitive attributes and evaluating the performance individual fairness.
2 Scope of reproducibility
This study describes the reproducibility of 'Fair Attribute Completion on Graph with Missing Attributes' by Guo, Chu, and Li [ In earlier research, various fair graph methods have been developed, such as Fairwalk [1 and fair GraphSAGE
12 While previous research addressed fairness in graphs with partially incomplete attributes 7. FairAC uniquely targets nodes with entirely missing attributes. This approach also diverges from methods like those in 7 by addressing both node feature unfairness and topological unfairness. In the conlext of fairness, feature unfairness refers to bias within single node's embedding, which can be viewed as unfair when it contains sensitive information. Topological unfairness arises when
aggregating neighbouring nodes introduces sensitive information in the final embedding proposes FairAC, a fair attribute completion model for graphs. For more details on this method, see Section 3
13 The paper
The claims that the paper made are as follows:
Claim 1 A new framework, namely FairAC, is proposed, which can be used for fair graph attribute comple- tion and addresses both feature and topological unfairness in the resulting graph embeddings.
Claim FairAC is generic and can be used in many graph-based downstream tasks.
Claim FairAC effective in climinating unfairness while maintaining an accuracy comparable to other methods.
Claim 4
FairAC effective even a large amount of the attributes are missing.
Claim 5 Adversarial learning is necessary to obtain better performance on group fairness.
Beyond reproducing the original study, we extended our work assess FairAC's generalizability. This involved evaluating the framework across different datasets and varying sensitive attributes. Additionally, conducted more in-thpth analysis of FairAC's fairness capabilities, using metric for individual fairness.
3 Methodology
The FairAC implementation is openly accessible, but the baseline code for GCN and FairGNN is not included in this repository. To address this, we integrated these separate codebases, which are also publicly available, into unified framework. In the restructuring, we enhanced the codebase to support the use of various datasets and different. sensitive attributes in combination with FairAC. Furthermore, we expanded FairAC's evaluation criteria by incorporating a measure of individual fairness, offering contrast to the original paper's focus on group fairness.
3.1 Model description
FairAC is novel framework designed to ensure fairness in attribute completion for graph nodes, as depicted This framework is composed of roughly three components.
in
figure
The auto-encoder is utilized to generate node embeddings of complete nodes. This involves adversarial training with sensitive classifier tasked with detecting the presence of sensitive information in embeddings, guiding the auto-encoder towards more fair feature representation.
Auto-encoder.
Attention-based attribute completion. FairAC employs an attention mechanism to determine the importance of neighbouring nodes when completing the target node's attributes. The final embedding of the target node is created through weighted aggregation, with the weights assigned by the attention mechanism.
2

---

Published in Transactions on Machine Learning Research (06/2024)
3. Topological fairness. The sensitive classifier reappears in this component to evaluate topological lairness of the larget node's embedding, which is completely based on neighbouring nodes. Tf sensitive information is present, it indicates that it originated from the aggregation of neighboring nodes, highlighting potential topological bias.
Topological emto)
Adtention matric
Ts
Advercatial leariing
Classtfist
Feature cmbodcing(H)
Adversarial leaming
Reczesttiction
Compaitin
The FairAC framework consists of mainly three components: An auto-encoder to generale embed- dings, an satention-based mechanism for attribute completion and a sensitive classifier to apply adversarial learning
Figure 1:
FairAC's loss function integrates these three components (detailed in Appendix Ap. A hyperparameter B is used to adjust the influence of the sensitive classifier on the overall loss. B is multiplied with the loss of the sensitive classifier in both the topological and the node embedding prediction parts. The loss function is defined as L = L + Lo + BLr, where Ep represents the feature fairness component loss, defined as CF fae 3Lc.
3.2 Datasets
The authors have made available csv files of the three real-world datasets used for their experiments. We follow the paper and reproduce their evaluations on multiple datasets for each method. We present the relevant datascts in detail Table
NBA dataset. This an extended version of Kaggle datase containing performance statistics of approximately 400 NBA basketball players from the 2016-2017 season. It includes personal infor- mation such age, nationality, gender and salary with player interactions Twitter defining the network relationships. The node label indicates whether player's salary is above the median and nationality serves as the sensitive attribute, intended to be excluded from the embeddings.
Pokec datasets. Derived from Pokec, Slovakian social network 14 the Pokec-z and Pokec-n datasets are adapted from Dai and Wang 7. While the original FairAC paper used region as the sensitive attribute for both Pokec datasets, our study also evaluates age and gender. Age is converted to a binary variable by indicating if a person is younger than 21 years or not, similar to the approach in the credit datasot 15
Additional datascts, German Crodit and Recidivism, were also used to evaluate FairAC 15 These datascts were selected for their different themes and small size, making attribute completion particularly useful due to the higher importance of each data point. The code to preprocess the dataset was adapted from PyGDebias]] For AairAC training, the first 25% of each dataset was used, a limitation due to memory constraints of the machines used for the original paper. The subsequent GNN training utilized 50% of the data, with the remaining 25% reserved for evaluation. All data splits follow those used in the original paper.
httte///wwwwkaagge.mmm//aaasaats /och poweer--
3

---

Published in Transactions on Machine Learning Research (06/2024)
3.3 Hyperparameters
In order to reproduce the experiments of the original author closely as possible, the hyperparameters used in the original paper were used when available. This means that the feature drop rate (a) is 0.3 unless mentioned otherwise. 3 is 1.0 for all datascts, except for pokec-n, where 3 is equal to 0.5. In case of missing hyperparameters, the parameters in the provided shell scripts were adapted. For training, 3000 epochs were used including 200 epochs used for autoencoder pretraining. An Adam optimizer was used with a learning rate of 0.001 and a weight decay of 1-10-5 The details of all hyperparameters can be found in Appendix G.
In the original paper, all experiments are run three times. We assumed that this was done three different seeds. Since, in the available scripts, only one seed is stated, we' we've decided to use the seeds 40, 41 and 42, per advice of the authors.
3.4 Evaluation metrics
Model performance is evaluated in terms of accuracy, AUC and fairness. When evaluating fairness, important distiction can be made between two fairness types: group fairness and individual fairness. Group fairness measures equality between different protected groups, while individual fairness measures the similarity in classification outcome for similar individuals 16]. The original paper uses two important metrics to evaluate in lerms of (group) fairness: statistical parily and equal opportunity. Additionally, we add an individual fairness metric, consistency. The label y denotes the ground-truth node label and the sensitive attribute indicates scnsitive group. For example, we consider binary node classification task and two sensitive groups E 10.
Statistical Parity. Statistical Parity Difference (ASP) 9 is used to capture the difference in model prediction outcomes for different sensitive groups:
ASP = P(jls = 0) - P(yls = 1)
(1)
Equal Opportunity. Equal Opportunity Difference (AEO) 17 is used to capture the difference in true
positive rates for different scnsitive groups:
AEO Plğ 1|s y.y 1) P(g = 1 1)
(2)
Consistency. Consistency is used to capture the individual fairness by measuring the consistency of out- comes between individuals who are similar to each other 18 The formal definition regarding fairness similarity matrix Sf is:
consistency= = ΣxE; 1u
SF.
Vi Vi
Σ. Σ3 S
3.5
Experimental setup and code
All specific experiments conducted are described below, together with description of the training details and the code.
3.5.1 Model training
The original paper provided brief overview of the training algorithm; for more details refer to [ Our analysis found that certain critical details were missing in the original description. In the first part, the data split, it was found that the AC model is trained on only 25% of the nodes in the datasct, and during the test on the rownstream task, the full 100% of the graph used to complete any missing attributes and create new embeddings for the downstream task. After the data is split, the training process started. An interesting detail in this process is that the auto-cncoder the AC model is pro-trained for 200 itcrations. This iteration count remains fixed across all datasets, meaning that for smaller datasets the auto-encoder is pre-trained using comparatively less data, which is the case in for example the NBA German credit
4

---

Published in Transactions on Machine Learning Research (06/2024)
dataset, as described in (Section 3.2 Furthermore, the auto-encoder and attribute completion model are optimized differently during pre-training than during the training iterations. During pre-training they are optimized using Lue + Lc. This to be expected as the sensitive classifier is not being trained yet.
Evaluation of the model is done for the first time after 1000 epochs. Every 200 epochs, a GCN is trained on the embeddings generated by FairAC. This GCN is evaluated and provides results for the performance of FairAC. The final results reported are the results of the model with the best fairness performance that scores above the accuracy and AUC threshold that are set at the start of training. For all datasets, the thresholds were set to 0.65 and 0.69, respectively. The complete model training procedure for the FairAC model is given in Appendix
3.5.2 Code
The original paper published their code, including shell scripts to run specific experiments, based on the FairGNN codebase However, the code was difficult to work with and contained several peculiarities that are not mentioned in the original work (as mentioned in Section 3.5.1). In order to run the code on additional datasets and downstream tasks, the FairAC framework has been rewritten into an ergonomic library that be used for any GNN without any significant changes to the code. The source code for this reproduc- tion has been made available (See [Appendxx D for more details). In addition to creating the library, the implementation has been made faster, which makes the runtime about 33% faster.
3.5.3 Experiment reproducibility study
To accurately reproduce the original study, various experiments were conducted. Firstly, to verify claim 1, 2 and 3, the FairAC framework was applied on the default GCN on the datasets NBA, pokec-z and pokec-n. The results of this were compared with the results of using only a GCN and using the FairGNN method. Conducting experiments on different datasets provides evidence for claim 2 specifically. To verify claim 4, experiments on all three models and the FairAC model without adversarial training were conducted with different attribute missing rates, and the results are compared. Similarly, for claim 5, the FairAC model trained with different 3 values, to see the effect of adversarial lcarning.
3.5.4 Experiments beyond original paper
To further analyze the FairAC framework, additional experiments were conducted. To provide additional ev- idence for claim 2, the framework was tested on two extra datasets, as described in Seetioo 3.2 Additionally, the framework was lested on different sensitive attributes, to provide further evidence for generalizability. Specifically, two additional experiments were run on the pokec-z dataset, with the sensitive attributes age and gender. Lastly, to test whether there is trade-off between the group fairness provided by FairAC and individual fairness, all experiments were tested the individual fairness. The consistency between the default GCN and FairAC was compared to see the influence of FairAC on individual fairness.
additional metrie, consistency, which measures
3.6 Computational requirements
All experiments were performed on NVIDIA Titan RTX GPU. Training one FairAC model takes about 30 minutes on this GPU, while training a (Fair)GNN model takes about 10 minutes. Therefore, reproducing all original experiments costs about 31 GPU hours. Roughly 15 GPU hours were used for performing the additional experiments. Thus, 46 GPU hours were used in total. More details about the GPU usage can be found in Appendix F
5

---

Published in Transactions on Machine Learning Research (06/2024)
4 Results
In this section, the results of the reproducibility study described. This includes all experiments ran to verify the claims described in [Section 2 In addition to this, the results of the work done beyond the original study are described.
4.1 Results reproducibility study
In Table 1 the reproduced results of the original study are shown. Together with this, the performance individual fairness is shown, which analyzed in [Section 4.2 In the original study, the main trends observed are that fairAC performs similarly to other methods on accuracy and AUC and out performs all other alternatives statistical parity and equal opportunity. The same trends can be observed in the results from the reproducibility study. The accuracy and AUC scores are similar, although not as good as FairGNN. On statistical parity and equal opportunity, FairAC outperforms all other methods almost every dataset. The actual numbers of all methods are slightly different from the original paper. This could be due to the optimization method not being deterministic. These results verify claim 1 and 3, as listed in [Section Namely, FairAC addresses unfairness better than other methods while it maintains a comparable accuracy.
Dataset
Method
M
Ace f
AUC f
ASP
AEO 1
ASP+AEO
Consistency
GCN
66.98 1.18
76.15 1 1.40
0.14 0.13
0.57 0.06
0.71 + 0.18
2.64 1 0.00
ALFR
X
64.3 1: 1.3
0.3
2.3 0.9
3.2
5,5 2.4
ALFR-c
66.0
1.0
4.7
4.7
9.4 1441344
NBA
Debias
X
63.1
7.7
2.5
1.1
3.4
Debias-c
X
6..6
1.2
5.3
1.1
844
FOGE
X
66.0 1.5
1.5
2.9
3.0 1.2
5.9 3.9-222
FairGNN
68.39+3.11
74.29 1.19
2.81 4.01
3.00 + 4.07
5.81 = 8.08
2.64 4.4400000
AairCC (Ours)
66.51 1.09
1.31 75.69
0.09 + 0.08
0.10 0.00
0.08
2.64 0.00
GCN
65.10 0.24
0.12
1.72 1.17
1.37 0.51
3.08 = 1.68
41.35 0.01
ALFR
x
65.4 1:.4-000 0.4
0.3
2.8
1.1
3.9 = 0.9
ALFR-e
68.0 0.6
74.0
5.8+0.4
2.8
8.6 1.2
Debias
0.7
0.6
1.9
1.4
3,8 1.0
Pokec-z
Debias-c
x
67.5
7.7
4.7100
3.0
2.7
FCGE
X
63.9
7..0
3.1
1.6
4.1111
FairGNN
68.16 0.59
75.67
0.52
1.56 0.45
3.17 11174100 1.07
4.73 1.47
41.35 - 0.01
FairAC (Ours)
65.33 0.30
1.74 71.20
0.55 + 0.10
0.13 +: 0.15
0.68 0.09
41.33 +: 0.00
GCN
67.88 1.46
72.86
1.44
3.22 + 1.29
5.93 2.76
9.15 4.05
45.93 + 0.00
ALFR
x
63.1 0.6
67.7 + 0.5
3.05 0.5
3.9 +.6
3.95 1.1
ALFR-e
X
66.2 -
71.9 1.0
1.1888
4.6 1.7
8.7 3.5
Debias
62.6
0.7
2.4
2.6 2.6110
5.0 =
Pokec-n
Debias-e
65.6 2.4
71.7
0.9
4.4
2.2
FCGE
1.5 67 17.06-20 + 0.37
69.5
4.1
5.5 1.2
9.6
FairGNN
71.58 2.58
0.55 = 0.50
0.30 + 0.20
0.85 = 0.31
45.93 0.00
FairAC (Ours)
67.00 = 1.93
72.57 + 1.68
0.11
0.47
0.81
0.58 0.76
45.94 0.02
Table 1: Comparison of FairAC with FairGNN the nba, pokce-z and pokec-n dataset. The methods are applied on the GCN classifier. The best results denoted in bold. The ALFR, ALFR-e, Debias, Debias-e and FCGE baselines are adapted from Guo, Chu, and Li
In addition to the main comparison of different baselines, ablation study different attribute missing rates (a) was done in the original paper to verify claim 4. The results shown in Table 2 The trends in the reproduced results are very similar to the original trends, as FairAC performs best on fairness for different a. However, for a = 0.1, it is observed that in the reproduced results, BaseAC, which is the same architecture as FairAC but without adversarial training, performs slightly better than FairAC. addition to this, it observed that adversarial training is more useful with large attribute missing ratc, an observation that wasn't made in the original paper.

---

Published in Transactions on Machine Learning Research (06/2024)
Although the original results reproduced, the reproduction would not have been possible without access to the codebase. A lot of the training details, such as hyperparameters and pretraining of the auto-encoder were not mentioned in the paper, only in the codebase. In addition to this, the random seeds used the original study were only retricved after contact with the authors.
Method
Acc 1
AUC
ASP 4
ΔEO 4
ASP+AEO
Consistency
GCN
65.93 = 0.33
69.86 + 1.73
3.52 + 2.94
2.73 3.18
6.26 6.12
41.33 = 0.00
FairGNN
68.67 +: 2.52
76.95 0.53
1.97 + 0.52
2.23 + 0.31 0.31
4.20 0.59
41.33 = 0.00
0.1
BaseAC
66.16 + 0.11
69.22 0.07
0.08 0.01
0.40 + 0.20
0.48 + 0.21
41.33 = 0.00
FairAC
66.04 + 0.69
70.72
0.26 0.28
0.50 + 0.62
0.76 0.90
41.33 = 0.00
GCN
65.10 + 0.24
68.42 + 0.12
1.17
1.37 0.51
3.08 + 1.68
41.35 = 0.01
FairGNN
68.16 + 0.59
75.67 + 0.52
1.45
1.171107
4.73 = 1.47
41.35 0.01
0.3
BaseAC
66.26 + 0.32
71.11 + 1.73
0.35 0.15
1.18
1.08 = 1.32
41.33 = 0.00
FairAC
65.33 0.30
71.20 1.74
0.55 0.10
0.13 + 0.15
0.68 0.09
41.33 0.00
GCN
65.65 + 0.97
69.72 2.15
2.75 2.09
3.50 + 3.32
6.25 5.40
41.38 = 0.02
FairGNN
65.97 + 0.56
72.99 + 0.44
2.40 1.44
3.07 2.41
5.47
41.37 0.01
0.5
BaseAC
65.45 + 0.40
70.64 + 1.10
0.13 - 0.20
0.33 + 0.58
0.47 0.77
41.33 = 0.00
FairAC
65.62 + 0.02
71.11 1.02
0.05 0.05
0.30 + 0.52
0.35 0.50
41.33 0.00
GCN
65.37 1.30
71.62 2.33
5.10 3.12
5.53 3.86
10.64 + 6.84
41.47 0.04
FairGNN
63.81 + 0.50
67.57 + 0.30
2.96 0.28
1.77 + 1.33
4.73 = 1.10
41.44 0.02
0.8
BaseAC
66.03 + 0.69
71.06 + 1.21
0.32 + 0.39
0.67 + 0.65
0.99 + 0.86
41.33 = 0.00
FairAC
65.38 0.14
71.51 0.68
0.23 1 0.37
0.03 + 0.06
0.27 0.43
41.33 0.00
Table 2: Comparison of FairAC with FairGNN on the pokec-z dataset with different attribute missing rates (ax). The methods are applied on the GCN classifier. The best results are denoted in bold.
Furthermore, hyperparameter study of B was done in the original paper. 3 balances fairness and accu- racy (details in Section 3.1). The original study intro- duced adversarial training to improve group fairness and claimed that adversarial training is necessary for good performance on group fairness, based on | hy- perparameter study. Tt reported that increasing 3 im- proves fairness but slightly reduces accuracy by about 0.5%. However, the lack of standard deviation in the original figure makes unclear whether the experi- ment was done three times or onc. In this study, the experiment done three times. The mean and stan- dard deviation are shown in Figure 2 In general, the same trends observed. Notably, ß = 0.8, ob- served marginal increase in fairness, contrary to the original study's cont inuous decline. From this, it can be concluded that adversarial learning helps fairness performance the FairAC framowork.
Arruney and Difieett Seeds
CP 4503
67
Figure 2: Hyperparameter value study for 8, which influences the trade-off between accuracy and fairness.
4.2 Results beyond original paper
In the original study, FairAC was presented as generic method applicable to a wide range of downstream tasks. This claim primarily supported by tests different datasets. Therefore, we performed additional experiments to test this claim. Firstly, our study focused on examining FairAC's performance across various sensitive attributes. Unlike the original experiment. that only considered region the sensitive variable, in

---

Published in Transactions on Machine Learning Research (06/2024)
this study age and gender were used sensitive variables. The findings, detailed in Table 3 show that gender as sensitive attribute produces results comparable to those obtained with region. However, for age, the fairness performance decreases drastically. In this scenario, the standard GCN model outperforms both fair models. This outcome is due to age being an important feature for the final prediction tasks, which challenges the achievement of set accuracy and AUC thresholds. If the thresholds are set lower, the accuracy decreases but the fairness increases. Experiments on different thresholds values can be found in Appendix E
Sensitive attribute
Method
Acc
AUC 1
ASP
AEO 4
ASP+ARO
Consistency ?
GCN
65.10 0.24
68.42 + 0.12
1.72 + 1.17
1.37 + 0.51
3.08 + 1.68
41.35 + 0.01
Region
FairGNN
68.16 0.59
75.67 0.52
1.56 + 0.45
3.17
4.73 + 1.47
41.35 + 0.01
FairAC
65.33 1 0.30
71.20 +: 1.74
0.55 0.100
0.13 1: 0.15
0.68 0.09
41.33 4 0.00
GCN
63.40 0.20
68.36
3.28 0.71
0.61
6.24 1.13
41.35 + 0.01
Gender
Fai
64.25 0.41
72.25 2.49
2.27 0.95
2.63 0.31
4.90 1: 0.77
41.35 1 0.01
FairAC
66.44 0.47
73.39 0.20
0.66 + 0.56
0.30 0.10
0.96 0.52
41.33 + 0.00
GCN
64.9 1.11
71.33 = 1.94
24.69 + 3.21
20.57 3.84
45.26 + 6.96
41.35 + 0.01
Age
FairGNN
65.79 1 0.20
72.53 L 1.42
39.83 1 4.80
37 23 1 2.20
77.07 1 6.70
41.35 1 0.01
FairAC
65.82 = 0.69
74.26 + 0.42
27.46 = 1.94
19.90 + 2.52
47.36 + 4.38
41.33 + 0.00
Table 3: Comparison of FairAC with FairGNN and GCN on the pokec-z dataset with different sensitive altribules. The methods are applied on the GCN classifier. The best results are denoted in bold.
The FairAC framework's performance was evaluated on two additional datasets beyond those used in the original study, with results detailed in Table 4 In the recidivism datasct, the fairness results are almost per- fect across all methods. While FairGNN marginally outperforms FairAC in group fairness, FairAC performs better in individual fairness. The credit dataset shows similar pattern, with FairAC achieving the best fairness performance among the methods tested, while maintaining comparable overall performance. These outcomes lend further support to the claim that FairAC provides better fairness results than comparable methods and thus support the claim that FairAC is a generic method. Moreover, we exlended our analysis by incorporating an additional metric, consistency, as explained in Section 3.4.
Contrary the common trade-off between group and individual fairness 16 our findings (see Table 1 indicate similar consistency levels across GCN, FairGNN and FairAC. This suggests that improving group fairness does not necessarily compromise individual fairness.
Dataset
Method
Acc 1
AUC t
ASP 1
AEO J
ASP+AEO $
Consistency
GCN
72.16 + 4.00
61.80 + 0.45
7.43 + 0.92
5.97 + 0.95
13.40 + 1.46
26.16 + 0.27
Credit
FairGNN
77.19 = 0.85
64.66 + 0.46
2.02 1.84
0.83 : 1.10
2.85 2.91
28.02 1.41
FairAC
69.78 + 2.94
65.13 + 0.07
0.68 + 0.51
0.50 - 0.61
1.18 0.29
27.24 0.81
GCN
62.37 + 0.02
63.01 + 2.43
0.01 + 0.02
0.07 0.12
0.08 4 0.14
3.93 + 0.00
Recidivism
FairGNN
70.00 = 0.00
56.66 1.46
0.00 0.00
0.00 0.00
0.00 0.00
6.60 + 0.00
FairAC
63.03 + 1.17
70.32 + 13.02
0.04 + 0.08
0.00 0.00
0.04 + 0.08
3.93 + 0.01
Table 4: Comparison of FairAC with FairGNN and GCN on the pokec-z dataset with different sensitive attributes. The methods are applied on the GCN classifier. The best results are denoted in bold.
5 Discussion
In this study, various experiments on reproducing the study Fair Attribute Completion on Graph with Missing Attribute [ were presented. The results of these experiment support the original claims of the authors. Specifically, FairAC has a better fairness performance than other graph attribute completion methods, while maintaining comparable accuracy, even when a large part of the attributes missing. The method uses adversarial learning, which indeed improves the performance. To support the claim that FairAC generic method, various additional experiments were performed. From these experiments, it can be concluded that
8

---

Published in Transactions on Machine Learning Research (06/2024)
FairAC is applicable across multiple datasets in different fields. Also, FairAC gives accurate results for various sensitive attributes, while the performance might drop if the attribute is important for the downstream task. To examine the impact of FairAC on individual fairness, an additional metric was implemented, from which can be concluded that there is almost no tradc-off between group fairness and individual fairness in this study. Since this contrary to most findings in literature 16 for future research would be interesting to look into the causes of trade-off individual fairness and group fairness.
The code implementation of this study, together with the refactored code from the FairAC paper, can be found on Hit
5.1 What was easy and what was difficult
The original paper provides complete codebase of FairAC. This was very helpful, since the codebase also includes the hyperparameters utilized in the experiments of the original paper. This made running the experiments easier. However, understanding the code turned out to be a non-trivial task. A lot of training and implementation details were not mentioned in the paper and the code was not clearly structured or documented. This made very difficult to adjust the codebase for additional experiments, therefore we refactored all original code. In addition to this, the GCN and FairGNN basclines used the original paper were not part of the codebase, so these were adapled from the FairGNN codebase and changed meet the requirements of the FairAC paper.
5.2 Communication with original authors
We originally reached out the authors to ask for clarification the setup on some of their experiments, namely the random seeds used and the exact setup for the BaseAC experiment, since the description of this experiment was inconsistent with the codebase. We received very quick clear response to these questions. Later in the project, we reached out again to ask some questions regarding ambiguities in the data handling and splitting that was taking place the code. We received short response with some clarification regarding the askod questions.
References
[1] D. Guo, Z. Chu, and S. Li, "Fair attribute completion graph with missing attributes," " in The Eleventh International Conference on Learning Regresentations, 2023. [Online]. Available: https:// openreview net/ htte//oorummd-ssvccctpppy
[2] J. Zhou, G. Cui, S. Hu, el ol., "Graph neural networks: A review of methods and applications," AI open, vol. 1, pp. 57 17-81, 2020.
[3] D. Jin, Z. Liu, W. Li, D. He, and W. Zhang, "Graph convolutional networks meet markov random fields: Semi-supervised community detection in attribute networks," " vol. 33, pp. 152 159, Jul. 2019. DOL: 100.169//aall 3301152 [Online]. Available: https://ojs wwt. www.aaaaaa.aaaaaaa.....aaaaaaaaa htd///nddxx..// Bwtpeex..ppp/AAA.. php/AAAI/
article/vies/3780
[4] N. Kipf and M. Welling, Semi-superoised classification with graph convolutional networks, 2017.
arXiv: 1609. 02907 LG] T. N. Kipf and M. Welling, Variational graph auto-encoders, 2016. Xiv: 1611.07308 [stat. .ML]
[5]
[6] M. Zhang, Z. Cui, M. Neumann, and Y. Chen, "An end-to-end deep learning architecture for graph classification," " in Proceedings of the Thirty-Second AAAT Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Louisiana, USA: AAAI Press, 2018, ISBN: 978-1-577777 978-1-87735-800-8.
7 E. Dai and S. Wang, Say no to the discrimination: Learning fair graph neural networks with limited sensitive altribule information, 2021. arXiv: 2009.01454 [c]
8 A. Beutel, J. Chen, Z. Zhao, and E. H. Chi, Data decisions and thcoretical implications when adver- sarially learning fair representations, 2017. arXiv: 7707.0007 [cs. LG]

---

Published in Transactions on Machine Learning Research (06/2024)
[9] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, Fairness through awarencss, 2011. arXiv:
1104. 3913 [cs.CC]
[10]
Chen, S. Chen, J. Yao, H. Zheng, Zhang, and W. Tsang, "Learning on attribule-missingg graphs," " IEEE Tronsactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 2, pp. 740-
757, Feb. 2022, ISSN: 1939-3539. DOE: 10 1109/ /ttami 2020 3032189 [Online]. Available: hhtpp:
//dx. org/10. 1109/TPAMI 2020.00322
[11] T. Rahman, B. Surma, M. Backes, and Y. Zhang, Fairwalk: Towards fair graph embedding, 2019.
[2] S. Garg, R. Mujumdar, and R. Gajawada, Eaploring fairness in heterogenous gruph embeddings, 2021. [Online]. Available: https://roh: jmujumdar
umdar gitgug....... http//oojecs///ST__PPPA... Final_Repord
[3]] Z. Jiang, X. Han, Fan, ol., Topology malters in fair graph learning: A theoreticul pilot study, 2023.
[Online]. Available: https://openrevieee wwt. teeeffoorrm
[14] L. Takac and M. Zábovský, "Data analysis in public social networks," " International Scientific Confer- ence and International Workshop Present Day Trends of Innovations, pp. 1-6, Jan. 2012.
[15] C. Agarwal, H. Lakkaraju, and M. Zitnik, "Towards unified framework for fair and stable graph representation learning," " in Uncertointy in Artificial Intelligence, PMLR, 2021, pp. 2114 2124.
16] Binns, "On the apparent conflict between individual and group fairness," " in Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 514-524.
[17] M. Hardt, E. Price, and N. Srebro, Equality of opportunity in supervised learning, 2016. arxiv 1610.
02413 [CS.LG]]
188 P. Xu, Y. Zhou, B. An, W. Ai, and F. Huang, Gfairhint: Improving individual fairness for graph neural networks via fairness hint, 2023. arXiv: 2305 15622 [cs LG]
[19] Y. Dong, J. Ma, S. Wang, C. Chen, and J. Li, "Fairness in graph mining: A survey," " IEEE Transactions
on Knowledge and Data Engineering, 2023.
[20] B. Perozzi, R. Al-Rfou, and S. Skiena, "Deepwalk: Online learning of social representations" CoRR.
vol. 1403.6652 2014. arXiv: 1403.6652 [Online]. Available:
6652
10

---

Published in Transactions on Machine Learning Research (06/2024)
Appendices
A
Model details
this section the components of the loss function are described more detail. As explained in Section 3 the loss fuucctio used by FairAC consists of three components: feature fairness loss (Cr), completion loss (Cc) and topological fairness loss (Cr), with parameter 8 to control the influence of the sensitive classifier:
== Lr + Lc + BLT
(4)
Feature fairness. The feature fairness loss consist two components, an auto-encoder loss and sensitive classifier loss. The feature fairness loss leverages the sensitive classifier C, adversarially train the auto- encoder, such that the encoder is able to generate fair feature embeddings that can fool the sensitive classifier.
Cr Lac BBcc
Auto-Encoder. The auto-encoder encodes original attributes X; to feature embeddings Hl and reconstructs the attributes &; from latent space. The reconstructed attributes should be close the original attributes. The loss optimizes for this:
Σ (x)-2))²
Lae
(6)
Vkeepl
iEVacop
Sensitive classifier. The sensitive classifier takes feature embeddings Hi as input and predicts the sensit.ve attribute 3. Since the sensitive attribute is binary, binary cross entropy loss is used:
Lc, = [Kkeep
81 sclog 8 + (1 8;) log(1 - 8:)
LEVenap
Attribute completion. The attribute completion loss measures the difference between the true feature embeddings and the predicted feature embeddings across nodes whose attributes are intentionally dropped (Varop). This loss avcraged over all such nodes:
Lc
Σ [dropl iEVerop
V H.)
Topological fairness. The attribute completion may introduce topological unfairness by assuming topo- logical information is similar to attributes. To address this issue, FairAC leverages the sensitive classifier (C.) to help mitigate this unfairness. The loss function aims to learn feature embeddings that can fool the sensitive classifier (C2) to predict uniformly over the sensitive category:
LT
Σ VEVarop
8; l, 8 + (1 s;) log(1 - 82).
(9)
Verool
11

---

Published in Transactions on Machine Learning Research (06/2024)
B
Datasets
Dataset
NBA
Pokec-z
Pokec-n
Credit
Recidivism
# of nodes
403
67,797
66,569
30,000
18,876
# of edges
16,570
882,765
729,120
200,526 0.00045
321,300 ,3088
Density Sensitive attributes
0.20456
0.00038
0.00032
0.001804
country salary
region working field
region
age
race
Predicted labels
working field
no default next month
bail vs. no bail
Table 5: Statistics of five graph datasets, namely nba, pokec-z, pokec-n, german credit and recidivism.
In Table 5| the statisties for all five datasets used in this study are given. There is a big difference in size between all datasets. Pokec-z and Pokec-n are the largest dataset, followed by Credit, Recidivism and NBA, respectively. NBA is dataset with only small number of nodes. However, the density of NBA is a lot larger than the density of all other dataset. Lastly, the default sensitive attributes are displayed, together with the node labels that are predicted in the downstream node prediction task.
C Model training details
Algorithm 1 Model training
Input: 9 = S
Output: auto-encoder fAe, Sensitive classifier C., Attribute completion fAC
1: Obtain topological embedding T with WeepWalk
2: repeat Pre-train fAE
Obtain the feature embeddings H with sAe Optimize fAE prevent unstable embeddings by Lae + Lo until 200 iterations
6: repeat Train SAE, C. and SAC using downstream task
7:
Obtain the feature embeddings H with fAE Optimize the C. by Lc,
8:
9:
Optimize fAE to mitigate feature unfairness by loss CF
10:
Divide V+ into Visep and Verop based on a Obtain the feature embeddings of nodes with missing attributes Varop by SAC Optimize SAc to achieve altribute completion by loss Lo
11:
12:
13:
Optimize fAC to mitigate topological unfairness by loss LT until convergence
14:
In Algorithm D. the model training process is shown. First of all, the auto-cncoder is pre-trained for 200 epochs. After this, the auto-encoder is optimized, logether with the sensitive classifier and the altribute completion mechanism. More details can be found in Section 3.5.1
D Codebase
The source is published Inttpa://githbb www/ookitsune/feetec.cc A complete refactor of the original codebase has been completed in order to make the framework significantly easier to apply various downstream tasks. The published code can be used a library, with any GNN. The GNN only requires
hhtpe///gghhub.com/////gg/////
12

---

Published in Transactions on Machine Learning Research (06/2024)
slight modification, the final layer needs to be excluded in order to train the FairAC model. For more details see the README. md file in the repository.
E
Fairness performance on age sensitive attribute with different thresholds
Threshold
Method
Acc 1
AUC1 1
ASP 1
AEO
ASP+AFO
Consistency 1
GCN
64.94 1.11
71.33 # 1.94
24.69 3.21
20.57 3.84
45.26 6.96
41.35 +
65% threshold
GNi
65.79
72.53 1 1.42
39.83 4.80
37.23 2.20
77.07 6.70
41.35 0.01
69% threshold
FairAC
65.82 0.69
74.26 0.42
27.46 1.94
2.52 19.90
47.36 4.38
41.33 0.00
GCN
61.85 1 2.57
70.52 1: 0.80
18.55 1 1.91
14.37 1 1.25
33. 33.122 1: 2.29
41.35 0.01
60% ace threshold 64% aue threshold
FairGNN
59.77 0.29
09.36 + 3.51
19.66 + 8.48
17.37 + 5.34
37.03 + 13.82
41.35 + 0.01
FairAC
61.27 0.84
73.59 = 0.85
17.14 +: 2.77
12.07 = 2.44
29.21 +: 5.13
41.33 = 0.00
GCN
53,60 + 0.06
55.34 = 1.86
0,06 1: 0.06
0.03 = 0.06
0.09 + 0.08
41.35 0.01
50% are threshold 50% aue threshold
FairGNN
53.59 0.00
54.29 + 2.23
0.00 0.00
0.00 + 0.00
0.00 0.00
41 35 0.01
FairAC
53.59 0.00
54.32 1.96
0.00
0.00 0.00
0.00 0.00
41.33 0.00
Table 6: Comparison of FairAC with FairGNN and GCN on the pokec-z dataset with age as sensitive attribute for different accuracy and auc thresholds. The methods are applied the GCN classifier. The best results are denoted in bold.
In Table 6 the experiments with different thresholds for the sensitive altribule age are displayed. In the top row, the results of the original threshold are shown. As one can see and as was analysed in [Section 4.2) the model embeddings are a lot more unfair than for other sensitive attributes, respectively 47.36 versus 0.96. As the thresholds decrease, the embeddings get more fair, until the point where they reach 0.0. This means that the model is able to produce fair embeddings, but for important attributes, it cost lot of performance.
F
GPU usage
Experiments
FairAC models
# PairGNN models
#GNN models
#ddatasets
#ecods
GPU usage (hours)
Main
Table
3
3
7.5
Alpha
Table 2
8
4
1
3
16
Beta Figure 2
5
0
0
1
3
7.5
Sensitive attributes
Table 3
Table
4
4
1
3
10
Other datasets
Table
2
3
Total:
Table 7: GPU usage for all experiments publishod in this report. Every FairAC model takes roughly 30 minutes train, every GNN and FairGNN model takes about 10 minutes train.
In Table 7 the specifics of the GPU hours used per experiment are displayed. The alpha experiments cost the most GPU hours, since all models, including FairAC without adverserial training, had to be trained for every alpha.
G Hyperparameters
For training the models, various hyperparameters were set. All hyperparameters used in this study were adaptod from the original study. Per default, all models were trained for 3000 epochs, which includes 200 epochs pretraining of the auto-encoder. An initial learning rate of 0.001 was used. On this learning rate, weight decay of 1 10-5 is applied. While training, dropout of 0.5 used. The main auto-encoder has
13

---

Published in Transactions on Machine Learning Research (06/2024)
hidden dimension of 128 and uses attention head. The default feature drop rate (a) was set to 0.3, unless mentioned otherwise. was set to 1.0 for all datasets, expect for the pokec-n dataset, where it was set to 0.5. As an accuracy threshold, 65.0 was used and for the auc threshold, 69.0 was used. The hyperparameters used for all dataset are shown in Table 8 For the dataset that were not used in the original study, the hyperparameters were adapted from Dong, Ma, Wang, al.
Dataset
Minimum number of datapoints used for AC training
Minimum numher of datapoints used for sensitive classifier training
Pokec-n
200
Pokee-x
500
200
NBA
100
50
Credit
6000
500
Recidivism
||
200
100
Table 8: Hyperparameters specific to the data loading for all datascts used in this study.
In addition to these hyperparameters, the GNN and FairGNN model had to be pretrained. The pretraining was done for 800 epochs. For the pretraining, all other hyperparameters are equal to the normal training hyperparameters.
FairAC uses topological input embeddings which are created using Deepwalk 20 was created using 10 epochs with a walk length of 100 and window size of 5. workers were used, and the dimension adapted was 64. Lastly, learning rate of 0.05 was adapted.
This embedding
14


Reviews:
[{"review_id": "a2EQBKi4Jy", "reviewer": "Reviewer_Hp7d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The submission is a comprehensive reproducibility study, covering various aspects of the result reproduction, including both the theoretical front and engineering front. For practitioners, the submission would be of high value.\n\n- The writing quality is high and it is easy to follow.\n\nWeaknesses:\n- As a reproducibility study, the submission may be adding limited value to the existing literature. A large portion of the submission verifies the existing claims and research findings instead of proposing or presenting novel things. Though the study extends the original empirical results to more dataset, more sensitive attributes, and more fairness criteria, the novel findings and discussions are relatively few (mostly in Section 4.2).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission is a reproducibility report of an existing paper FairAC: \"Fair Attribute Completion on Graph with Missing Attributes\". The submission reports the main technical contribution of that paper, reproduces the main empirical results, and extends the study to more datasets, more sensitive attributes, and more fairness criteria. The submission verifies that the results of the original paper are reproducible and the claims hold.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "There are no specific requested changes from my side. However, I think adding more discussion and connecting the work to more existing literature would further improve the submission quality. Moreover, I would recommend the work submitting to the ML Reproducibility Challenge at https://reproml.org/.", "broader_impact_concerns": "There is no major broader impact concern. Adding a statement or discussion about the fairness definitions and their limitations would be great.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0L1ULLsdB3", "reviewer": "Reviewer_mddg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It studies the trade-off between group fairness and individual fairness.\n2. It revise the codebase which accelerate the training process.\n\nWeakness:\n1.  It has limited novelty. Other than reproduction, it does not propose new method or improvement.\n2. The presentation is redundant. The paper looks like a report rather than a technical paper and includes many details that are not particularly informative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper reproduces FairAC method and conduct some new experiments on additional settings. The reproduction shows the effectiveness of the FairAC method and also reveals additional findings beyond that. Specifically, the author finds out that in some situation, the trade-off between group and individual fairness is marginal.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is better to investigate the reason behind the experiment results, such as propose new method to evaluate that or based on that to further revise it.\n2. The writing could be improved by making the presentation concise and accurate.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BzVVNcDpmc", "reviewer": "Reviewer_ovzk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n   - Consistent results with the original paper\n   - Refactored code released with baselines\n\nWeaknesses: \n   - I am not sure that such reproducibility paper is well fitted with journal venues as TMLR. Some workshops are specifically dedicated for this\n   - More importantly, I am not sure that the paper brings much beyond what we already knew from the original paper, except that results can be reproduced. New experiments are not so different. In what really differ the new datasets considered ? \n   - The paper is really not self-contained and really lacks of formalization of the task / presentation  of the method / discussion of the field \n   - Not any discussion about what should be done to go further, beyond some limits of the approach", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"Reproducibility study of FairAC\" reproduces experiments of a previously published fairness paper. The reproduced paper is focused on fairness for graph attributes completion. After code refactor, It observes same results as original ones on the same datasets, and similar tendencies on others.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- A large amount of formalization / more detailled presentation / discussion is needed before the paper could be published in TMLR from my point of view. Reading this paper alone is clearly not enough for the reader, it does not allow to understand the differences of the considered approach with its competitors in the field. \n- FairGCN is mentioned without any reference nor any explanation about what this method is\n- I feel authors should rather focus maybe on an innovation to improve individual fairness in the context of graph conception, to better perform on that task they introduce in the paper, rather than only limiting on observing performances from others", "broader_impact_concerns": ".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a reproducibility study that verifies whether the experiments in the FairAC paper is reproducible. The submission conducts a comprehensive reproducibility study, covering various aspects of the result reproduction, including both the theoretical front and engineering front. Also, the experiments were done on more datasets than in the original paper, showing the generalizability and robustness of the original method. The results presented in the paper are useful for practitioners. Though the reviewers are negative about this paper due to lack of novelty, I believe it meets the acceptance criteria for reproducibility papers in TMLR. I would recommend acceptance of this submission.
Decision: Accept (poster)