NIST AI RMF Playbook
AI RMF Playbook > Measure > MEASURE 2.11 (part 3)

Transparency and Documentation:
### Organizations can document the following

- To what extent are the established procedures effective in mitigating bias, inequity, and other concerns resulting from the system?
- If it relates to people, does it unfairly advantage or disadvantage a particular social group? In what ways? How was this mitigated?
- Given the purpose of this AI, what is an appropriate interval for checking whether it is still accurate, unbiased, explainable, etc.? What are the checks for this model?
- How has the entity identified and mitigated potential impacts of bias in the data, including inequitable or discriminatory outcomes?
- To what extent has the entity identified and mitigated potential bias-statistical, contextual, and historical-in the data?
- Were adversarial machine learning approaches considered or used for measuring bias (e.g.: prompt engineering, adversarial models)

### AI Transparency Resources

- GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities. [URL](https://www.gao.gov/products/gao-21-519sp)
- Artificial Intelligence Ethics Framework For The Intelligence Community. [URL](https://www.intelligence.gov/artificial-intelligence-ethics-framework-for-the-intelligence-community)
- WEF Companion to the Model AI Governance Framework- WEF - Companion to the Model AI Governance Framework, 2020. [URL](https://www.pdpc.gov.sg/-/media/Files/PDPC/PDF-Files/Resource-for-Organisation/AI/SGIsago.pdf)
- Datasheets for Datasets. [URL](https://arxiv.org/abs/1803.09010)

References:
Ali Hasan, Shea Brown, Jovana Davidovic, Benjamin Lange, and Mitt Regan. "Algorithmic Bias and Risk Assessments: Lessons from Practice." Digital Society 1 (2022). [URL](https://doi.org/10.1007/s44206-022-00017-z)

Richard N. Landers and Tara S. Behrend. "Auditing the AI Auditors: A Framework for Evaluating Fairness and Bias in High Stakes AI Predictive Models." American Psychologist 78, no. 1 (2023): 36-49. [URL](https://doi.org/10.1037/amp0000972)

Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. "A Survey on Bias and Fairness in Machine Learning." ACM Computing Surveys 54, no. 6 (July 2021): 1-35. [URL](https://doi.org/10.1145/3457607)

Michele Loi and Christoph Heitz. "Is Calibration a Fairness Requirement?" FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, June 2022, 2026-34. [URL](https://doi.org/10.1145/3531146.3533245)

Shea Brown, Ryan Carrier, Merve Hickok, and Adam Leon Smith. "Bias Mitigation in Data Sets." SocArXiv, July 8, 2021. [URL](https://doi.org/10.31235/osf.io/z8qrb)

Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall. "NIST Special Publication 1270 Towards a Standard for Identifying and Managing Bias in Artificial Intelligence." National Institute of Standards and Technology (NIST), 2022. [URL](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf)

Microsoft Research. "AI Fairness Checklist." Microsoft, February 7, 2022. [URL](https://www.microsoft.com/en-us/research/project/ai-fairness-checklist/)

Samir Passi and Solon Barocas. "Problem Formulation and Fairness." FAT* '19: Proceedings of the Conference on Fairness, Accountability, and Transparency, January 2019, 39-48. [URL](https://doi.org/10.1145/3287560.3287567)

Jade S. Franklin, Karan Bhanot, Mohamed Ghalwash, Kristin P. Bennett, Jamie McCusker, and Deborah L. McGuinness. "An Ontology for Fairness Metrics." AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, July 2022, 265-75. [URL](https://doi.org/10.1145/3514094.3534137)

Zhang, B., Lemoine, B., & Mitchell, M. (2018). Mitigating Unwanted Biases with Adversarial Learning. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. https://arxiv.org/pdf/1801.07593.pdf

Ganguli, D., et al. (2023). The Capacity for Moral Self-Correction in Large Language Models. arXiv. https://arxiv.org/abs/2302.07459

Arvind Narayanan. "Tl;DS - 21 Fairness Definition and Their Politics by Arvind Narayanan." Dora's world, July 19, 2019. [URL](https://shubhamjain0594.github.io/post/tlds-arvind-fairness-definitions/)

Ben Green. "Escaping the Impossibility of Fairness: From Formal to Substantive Algorithmic Fairness." Philosophy and Technology 35, no. 90 (October 8, 2022). [URL](https://doi.org/10.1007/s13347-022-00584-6)

Alexandra Chouldechova. "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments." Big Data 5, no. 2 (June 1, 2017): 153-63. [URL](https://doi.org/10.1089/big.2016.0047)

Sina Fazelpour and Zachary C. Lipton. "Algorithmic Fairness from a Non-Ideal Perspective." AIES '20: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, February 7, 2020, 57-63. [URL](https://doi.org/10.1145/3375627.3375828)

Hemank Lamba, Kit T. Rodolfa, and Rayid Ghani. "An Empirical Comparison of Bias Reduction Methods on Real-World Problems in High-Stakes Policy Settings." ACM SIGKDD Explorations Newsletter 23, no. 1 (May 29, 2021): 69-85. [URL](https://doi.org/10.1145/3468507.3468518)

ISO. "ISO/IEC TR 24027:2021 Information technology - Artificial intelligence (AI) - Bias in AI systems and AI aided decision making." ISO Standards, November 2021. [URL](https://www.iso.org/standard/77607.html)

Shari Trewin. "AI Fairness for People with Disabilities: Point of View." arXiv preprint, submitted November 26, 2018. [URL](https://arxiv.org/abs/1811.10670)

MathWorks. "Explore Fairness Metrics for Credit Scoring Model." MATLAB & Simulink, 2023. [URL](https://www.mathworks.com/help/risk/explore-fairness-metrics-for-credit-scoring-model.html)

Abigail Z. Jacobs and Hanna Wallach. "Measurement and Fairness." FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, March 2021, 375-85. [URL](https://doi.org/10.1145/3442188.3445901)