Package: jurity
Jurity: Fairness & Evaluation LibraryJurity (LION'23,ICMLA'21) is a research library
that provides fairness metrics, recommender system evaluations, classification metrics and bias mitigation techniques.
The library adheres to PEP-8 standards and is tested heavily.Jurity is developed by the Artificial Intelligence Center of Excellence at Fidelity Investments.
Documentation is available atfidelity.github.io/jurity.Fairness MetricsAverage OddsDisparate ImpactEqual OpportunityFalse Negative Rate (FNR) DifferenceFalse Omission Rate (FOR) DifferenceGeneralized Entropy IndexPredictive EqualityStatistical ParityTheil IndexBinary Bias Mitigation TechniquesEqualized OddsRecommenders MetricsAUC: Area Under the CurveCTR: Click-through rateDR: Doubly robust estimationIPS: Inverse propensity scoringMAP@K: Mean Average PrecisionNDCG: Normalized discounted cumulative gainPrecision@KRecall@KInter-List Diversity@KIntra-List Diversity@KClassification MetricsAccuracyAUCF1 ScorePrecisionRecallQuick Start: Fairness Evaluation# Import binary and multi-class fairness metricsfromjurity.fairnessimportBinaryFairnessMetrics,MultiClassFairnessMetrics# Databinary_predictions=[1,1,0,1,0,0]multi_class_predictions=["a","b","c","b","a","a"]multi_class_multi_label_predictions=[["a","b"],["b","c"],["b"],["a","b"],["c","a"],["c"]]memberships=[0,0,0,1,1,1]classes=["a","b","c"]# Metrics (see also other available metrics)metric=BinaryFairnessMetrics.StatisticalParity()multi_metric=MultiClassFairnessMetrics.StatisticalParity(classes)# Scoresprint("Metric:",metric.description)print("Lower Bound: ",metric.lower_bound)print("Upper Bound: ",metric.upper_bound)print("Ideal Value: ",metric.ideal_value)print("Binary Fairness score: ",metric.get_score(binary_predictions,memberships))print("Multi-class Fairness scores: ",multi_metric.get_scores(multi_class_predictions,memberships))print("Multi-class multi-label Fairness scores: ",multi_metric.get_scores(multi_class_multi_label_predictions,memberships))Quick Start: Probabilistic Fairness EvaluationWhat if we do not know the protected membership attribute of each sample? This is a practical scenario that we refer to asprobabilisticfairness evaluation.At a high-level, instead of strict 0/1 deterministic membership at individual level, consider the probability of membership to protected classes for each sample.An easy baseline is to convert these probabilities back to the deterministic setting by taking the maximum likelihood as the protected membership. This is problematic as the goal is not to predict membership but to evaluate fairness.Taking this a step further, while we do not have membership information at the individual level, consider access tosurrogate membershipatgroup level. We can then infer the fairness metrics directly.Jurity offers both options to address the case where membership data is missing. We provide an in-depth study and formal treatment inSurrogate Membership for Inferred Metrics in Fairness Evaluation (LION 2023).fromjurity.fairnessimportBinaryFairnessMetrics# Instead of 0/1 deterministic membership at individual level# consider likelihoods of membership to protected classes for each samplebinary_predictions=[1,1,0,1]memberships=[[0.2,0.8],[0.4,0.6],[0.2,0.8],[0.9,0.1]]# Metricmetric=BinaryFairnessMetrics.StatisticalParity()print("Binary Fairness score: ",metric.get_score(binary_predictions,memberships))# Surrogate membership: consider access to surrogate membership at the group level.surrogates=[0,2,0,1]print("Binary Fairness score: ",metric.get_score(binary_predictions,memberships,surrogates))Quick Start: Bias Mitigation# Import binary fairness and binary bias mitigationfromjurity.mitigationimportBinaryMitigationfromjurity.fairnessimportBinaryFairnessMetrics# Datalabels=[1,1,0,1,0,0,1,0]predictions=[0,0,0,1,1,1,1,0]likelihoods=[0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.1]is_member=[0,0,0,0,1,1,1,1]# Bias Mitigationmitigation=BinaryMitigation.EqualizedOdds()# Training: Learn mixing rates from the labeled datamitigation.fit(labels,predictions,likelihoods,is_member)# Testing: Mitigate bias in predictionsfair_predictions,fair_likelihoods=mitigation.transform(predictions,likelihoods,is_member)# Scores: Fairness before and afterprint("Fairness Metrics Before:",BinaryFairnessMetrics().get_all_scores(labels,predictions,is_member),'\n'+30*'-')print("Fairness Metrics After:",BinaryFairnessMetrics().get_all_scores(labels,fair_predictions,is_member))Quick Start: Recommenders Evaluation# Import recommenders metricsfromjurity.recommendersimportBinaryRecoMetrics,RankingRecoMetrics,DiversityRecoMetricsimportpandasaspd# Dataactual=pd.DataFrame({"user_id":[1,2,3,4],"item_id":[1,2,0,3],"clicks":[0,1,0,0]})predicted=pd.DataFrame({"user_id":[1,2,3,4],"item_id":[1,2,2,3],"clicks":[0.8,0.7,0.8,0.7]})item_features=pd.DataFrame({"item_id":[0,1,2,3],"feature1":[1,2,2,1],"feature2":[0.8,0.7,0.8,0.7]})# Metricsauc=BinaryRecoMetrics.AUC(click_column="clicks")ctr=BinaryRecoMetrics.CTR(click_column="clicks")dr=BinaryRecoMetrics.CTR(click_column="clicks",estimation='dr')ips=BinaryRecoMetrics.CTR(click_column="clicks",estimation='ips')map_k=RankingRecoMetrics.MAP(click_column="clicks",k=2)ncdg_k=RankingRecoMetrics.NDCG(click_column="clicks",k=3)precision_k=RankingRecoMetrics.Precision(click_column="clicks",k=2)recall_k=RankingRecoMetrics.Recall(click_column="clicks",k=2)interlist_diversity_k=DiversityRecoMetrics.InterListDiversity(click_column="clicks",k=2)intralist_diversity_k=DiversityRecoMetrics.IntraListDiversity(item_features,click_column="clicks",k=2)# Scoresprint("AUC:",auc.get_score(actual,predicted))print("CTR:",ctr.get_score(actual,predicted))print("Doubly Robust:",dr.get_score(actual,predicted))print("IPS:",ips.get_score(actual,predicted))print("MAP@K:",map_k.get_score(actual,predicted))print("NCDG:",ncdg_k.get_score(actual,predicted))print("Precision@K:",precision_k.get_score(actual,predicted))print("Recall@K:",recall_k.get_score(actual,predicted))print("Inter-List Diversity@K:",interlist_diversity_k.get_score(actual,predicted))print("Intra-List Diversity@K:",intralist_diversity_k.get_score(actual,predicted))Quick Start: Classification Evaluation# Import classification metricsfromjurity.classificationimportBinaryClassificationMetrics# Datalabels=[1,1,0,1,0,0,1,0]predictions=[0,0,0,1,1,1,1,0]# Available: Accuracy, F1, Precision, Recall, and AUCf1_score=BinaryClassificationMetrics.F1()print('F1 score is',f1_score.get_score(predictions,labels))InstallationJurity requiresPython 3.7+and can be installed from PyPI usingpip install jurityor by building from source as shown ininstallation instructions.CitationIf you use MABWiser in a publication, please cite it as:@article{DBLP:conf/lion/Melinda23,author={Melinda Thielbar, Serdar Kadioglu, Chenhui Zhang, Rick Pack, and Lukas Dannull},title={Surrogate Membership for Inferred Metrics in Fairness Evaluation},booktitle={The 17th Learning and Intelligent Optimization Conference (LION)},publisher={{LION}},year={2023}}@inproceedings{DBLP:conf/icmla/MichalskyK21,author={Filip Michalsk{\'{y}} and Serdar Kadioglu},title={Surrogate Ground Truth Generation to Enhance Binary Fairness Evaluation in Uplift Modeling},booktitle={20th {IEEE} International Conference on Machine Learning and Applications,{ICMLA} 2021, Pasadena, CA, USA, December 13-16, 2021},pages={1654--1659},publisher={{IEEE}},year={2021},url={https://doi.org/10.1109/ICMLA52953.2021.00264},doi={10.1109/ICMLA52953.2021.00264},}SupportPlease submit bug reports and feature requests asIssues.LicenseJurity is licensed under theApache License 2.0.