Package: tensorboard-plugin-fairness-indicators
Evaluating Models with the Fairness Indicators Dashboard [Beta]Fairness Indicators for TensorBoard enables easy computation of
commonly-identified fairness metrics forbinaryandmulticlassclassifiers.
With the plugin, you can visualize fairness evaluations for your runs and easily
compare performance across groups.In particular, Fairness Indicators for TensorBoard allows you to evaluate and
visualize model performance, sliced across defined groups of users. Feel
confident about your results with confidence intervals and evaluations at
multiple thresholds.Many existing tools for evaluating fairness concerns don’t work well on large
scale datasets and models. At Google, it is important for us to have tools that
can work on billion-user systems. Fairness Indicators will allow you to evaluate
across any size of use case, in the TensorBoard environment or inColab.RequirementsTo install Fairness Indicators for TensorBoard, run:python3 -m virtualenv ~/tensorboard_demo
source ~/tensorboard_demo/bin/activate
pip install --upgrade pip
pip install fairness_indicators
pip install tensorboard-plugin-fairness-indicatorsNightly PackagesTensorboard Plugin also hosts nightly packages athttps://pypi-nightly.tensorflow.orgon Google Cloud. To install the latest
nightly package, please use the following command:pipinstall--extra-index-urlhttps://pypi-nightly.tensorflow.org/simpletensorboard-plugin-fairness-indicatorsThis will install the nightly packages for the major dependencies of Tensorboard
Plugin such as TensorFlow Model Analysis (TFMA).Demo ColabFairness_Indicators_TensorBoard_Plugin_Example_Colab.ipynbcontains an end-to-end demo to train and evaluate a model and visualize fairness evaluation
results in TensorBoard.UsageTo use the Fairness Indicators with your own data and evaluations:Train a new model and evaluate usingtensorflow_model_analysis.run_model_analysisortensorflow_model_analysis.ExtractEvaluateAndWriteResultAPI inmodel_eval_lib.
For code snippets on how to do this, see the Fairness Indicators colabhere.Write a summary data file usingdemo.py, which will be read
by TensorBoard to render the Fairness Indicators dashboard (See theTensorBoard tutorialfor more information on summary data files).Flags to be used with thedemo.pyutility:--logdir: Directory where TensorBoard will write the summary--eval_result_output_dir: Directory containing evaluation results
evaluated by TFMApython demo.py --logdir=<logdir> --eval_result_output_dir=<eval_result_dir>`Or you can also usetensorboard_plugin_fairness_indicators.summary_v2API to write the summary file.writer = tf.summary.create_file_writer(<logdir>)
with writer.as_default():
    summary_v2.FairnessIndicators(<eval_result_dir>, step=1)
writer.close()Run TensorBoardNote: This will start a local instance. After the local instance is started, a link
will be displayed to the terminal. Open the link in your browser to view the
Fairness Indicators dashboard.tensorboard --logdir=<logdir>Select the new evaluation run using the drop-down on the left side of
the dashboard to visualize results.Compatible versionsThe following table shows the  package versions that are
compatible with each other. This is determined by our testing framework, but
otheruntestedcombinations may also work.tensorboard-pluginntensorflowtensorflow-model-analysisGitHub masternightly (2.x)0.44.0v0.44.02.12.00.44.0v0.43.02.11.00.43.0v0.42.02.10.00.42.0v0.41.02.9.00.41.0v0.40.02.9.00.40.0v0.39.02.8.00.39.0v0.38.02.8.00.38.0v0.37.02.7.00.37.0v0.36.02.7.00.36.0v0.35.02.6.00.35.0v0.34.02.6.00.34.0v0.33.02.5.00.33.0v0.30.02.4.00.30.0v0.29.02.4.00.29.0v0.28.02.4.00.28.0v0.27.02.4.00.27.0v0.26.02.3.00.26.0v0.25.02.3.00.25.0v0.24.02.3.00.24.0v0.23.02.3.00.23.0