Package: fairness-indicators
Fairness IndicatorsFairness Indicators is designed to support teams in evaluating, improving, and comparing models for fairness concerns in partnership with the broader Tensorflow toolkit.The tool is currently actively used internally by many of our products. We would love to partner with you to understand where Fairness Indicators is most useful, and where added functionality would be valuable. Please reach out attfx@tensorflow.org. You can provide feedback and feature requestshere.Key linksIntroductory VideoFairness Indicators Case StudyFairness Indicators Example ColabPandas DataFrame to Fairness Indicators Case StudyFairness Indicators: Thinking about Fairness EvaluationWhat is Fairness Indicators?Fairness Indicators enables easy computation of commonly-identified fairness metrics forbinaryandmulticlassclassifiers.Many existing tools for evaluating fairness concerns don’t work well on large-scale datasets and models. At Google, it is important for us to have tools that can work on billion-user systems. Fairness Indicators will allow you to evaluate fairenss metrics across any size of use case.In particular, Fairness Indicators includes the ability to:Evaluate the distribution of datasetsEvaluate model performance, sliced across defined groups of usersFeel confident about your results with confidence intervals and evals at multiple thresholdsDive deep into individual slices to explore root causes and opportunities for improvementThiscase study, complete withvideosand programming exercises, demonstrates how Fairness Indicators can be used on one of your own products to evaluate fairness concerns over time.Installationpip install fairness-indicatorsThe pip package includes:Tensorflow Data Validation (TFDV)- analyze the distribution of your datasetTensorflow Model Analysis (TFMA)- analyze model performanceFairness Indicators- an addition to TFMA that adds fairness metrics and easy performance comparison across slicesThe What-If Tool (WIT)](https://github.com/PAIR-code/what-if-tool- an interactive visual interface designed to probe your models betterNightly PackagesFairness Indicators also hosts nightly packages athttps://pypi-nightly.tensorflow.orgon Google Cloud. To install the latest
nightly package, please use the following command:pipinstall--extra-index-urlhttps://pypi-nightly.tensorflow.org/simplefairness-indicatorsThis will install the nightly packages for the major dependencies of Fairness
Indicators such as TensorFlow Data Validation (TFDV), TensorFlow Model Analysis
(TFMA).How can I use Fairness Indicators?Tensorflow ModelsAccess Fairness Indicators as part of the Evaluator component in Tensorflow Extended [docs]Access Fairness Indicators in Tensorboard when evaluating other real-time metrics [docs]Not using existing Tensorflow tools? No worries!Download the Fairness Indicators pip package, and use Tensorflow Model Analysis as a standalone tool [docs]Model Agnostic TFMA enables you to compute Fairness Indicators based on the output of any model [docs]Examplesdirectory contains several examples.Fairness_Indicators_Example_Colab.ipynbgives an overview of Fairness Indicators inTensorFlow Model Analysisand how to use it with a real dataset. This notebook also goes overTensorFlow Data ValidationandWhat-If Tool, two tools for analyzing TensorFlow models that are packaged with Fairness Indicators.Fairness_Indicators_on_TF_Hub.ipynbdemonstrates how to use Fairness Indicators to compare models trained on differenttext embeddings. This notebook uses text embeddings fromTensorFlow Hub, TensorFlow's library to publish, discover, and reuse model components.Fairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynbdemonstrates how to visualize Fairness Indicators in TensorBoard.More questions?For more information on how to think about fairness evaluation in the context of your use case, seethis link.If you have found a bug in Fairness Indicators, please file aGitHub issuewith as much supporting information as you can provide.Compatible versionsThe following table shows the  package versions that are
compatible with each other. This is determined by our testing framework, but
otheruntestedcombinations may also work.fairness-indicatorstensorflowtensorflow-data-validationtensorflow-model-analysisGitHub masternightly (1.x/2.x)1.13.00.44.0v0.44.02.121.13.00.44.0v0.43.02.111.12.00.43.0v0.42.01.15.5 / 2.101.11.00.42.0v0.41.01.15.5 / 2.91.10.00.41.0v0.40.01.15.5 / 2.91.9.00.40.0v0.39.01.15.5 / 2.81.8.00.39.0v0.38.01.15.5 / 2.81.7.00.38.0v0.37.01.15.5 / 2.71.6.00.37.0v0.36.01.15.2 / 2.71.5.00.36.0v0.35.01.15.2 / 2.61.4.00.35.0v0.34.01.15.2 / 2.61.3.00.34.0v0.33.01.15.2 / 2.51.2.00.33.0v0.30.01.15.2 / 2.40.30.00.30.0v0.29.01.15.2 / 2.40.29.00.29.0v0.28.01.15.2 / 2.40.28.00.28.0v0.27.01.15.2 / 2.40.27.00.27.0v0.26.01.15.2 / 2.30.26.00.26.0v0.25.01.15.2 / 2.30.25.00.25.0v0.24.01.15.2 / 2.30.24.00.24.0v0.23.01.15.2 / 2.30.23.00.23.0