Building the NIST AI Risk Management Framework: Workshop #2 

1

Day 1: Tuesday March 29, 2022

Theme/Goals: All Aspects of the AI RMF

Topic Speaker(s) Panel Member’s Suggested Resources

Welcome, Workshop Goals and 

Logistics

Elham Tabassi, National Institute of Standards and Technology

Opening Remarks Alondra Nelson, Office of Science and Technology Policy

Panel 1: AI Risk Framing and AI 

RMF Audience

Discussion on how to provide a 

structured, yet flexible approach to 

framing AI risk--both enterprise and 

societal, and the intended audience 

for the AI RMF.

Moderator: 

Hodan Omaar, Center for Data Innovation, Information 

Technology & Innovation Foundation

Panelists: 

Tara Hairston, Alliance for Automotive Innovation

Richard Mallah, Future of Life Institute

Ufuk Topcu, University of Texas at Austin

Christian Troncoso, BSA: The Software Alliance

Confronting Bias: BSA’s Framework to Build Trust in AI

Panel 2: AI RMF Function Map

Discussion about the AI RMF Map 

Function (establish the context and 

apply the AI RMF taxonomy to 

frame risks related to an AI system).

Moderator: 

Catherine Aiken, Center for Security and Emerging Technology 

Panelists: 

Christine Custis, Partnership on AI

Rayid Ghani, Carnegie Mellon University

Doug Johnson, Consumer Technology Association

Marilyn Zigmund Luke, America’s Health Insurance Plans

Classifying AI Systems

Classifying AI Systems and accompanying data visualization

ML Resource Library

ML Reference Document

Panel 3: AI RMF Function Measure

Discussion about the AI RMF 

Measure Function (analysis, 

quantitative or qualitative 

assessment, and track risks related to 

an AI system and its impact).

Moderator: 

Navrina Singh, Credo AI

Panelists:

Jack Clark, Anthropic

David Danks, University of California, San Diego

Jane Pinelis, The Joint Artificial Intelligence Center, Johns 

Hopkins University Applied Physics Lab

Why and How Governments Should Monitor AI Development
Building the NIST AI Risk Management Framework: Workshop #2 

2

 

Panel 4: AI RMF Function Manage

Discussion about the AI RMF 

Manage Function (addressing the 

risks that have been mapped and 

measured to maximize benefits and 

minimize adverse impacts).

Moderator: 

Brittany Smith, Data & Society

Panelists:

Jiahao Chen, Parity AI

Vincent Southerland, New York University School of Law

Grace Yee, Adobe

The Ethics of AI Ethics: An Evaluation of Guidelines

Managing Risks: A New Framework

Day 2: Wednesday March 30, 2022

Theme: All Aspects of the AI RMF

Topic Speaker(s) Panel Member’s Suggested Resources

Welcome Elham Tabassi, National Institute of Standards and Technology

Panel 5: AI RMF Function 

Govern

Discussion about AI RMF 

Govern Function (cultivate 

and implement a culture of 

risk management within 

organizations developing, 

deploying, or acquiring AI 

systems).

Moderator: 

Patrick Hall, bnh.ai

Panelists:

Natasha Crampton, Microsoft

Agus Sudjianto, Wells Fargo

Teresa Tung, Accenture

5 things lawyers should know about artificial intelligence

How to Fight Discrimination in AI

A United States Fair Lending Perspective on Machine Learning

Responsible AI: from principles to practice

How brain-inspired technologies can support ethical AI

Run Our Business Responsibly | COBE

Panel 6: Alignment of AI 

RMF with Standards and 

Frameworks

Discussion about 

international standards and 

framework development 

Moderator: 

Stephanie Ifayemi, Department for Digital, Culture, Media and Sport, 

United Kingdom

Panelists:

Heather Benko, American National Standards Institute

Sebastian Hallensleben, VDE e. V.

Jeanna Matthews, Clarkson University

AIST: National Institute of Advanced Industrial Science and Technology

JISC: Japanese Industrial Standards Committee

OECD AI classification

Data Ethics Commission (Germany)
Building the NIST AI Risk Management Framework: Workshop #2 

3

activities and how they can be 

leveraged for the AI RMF.

Roy Sugimura, National Institute of Advanced Industrial Science and 

Technology, Japan 

Resources from Jeanna Matthews

Statement on Algorithmic Transparency and Accountability

Privacy, Equity, and Justice in Artificial Intelligence

Democratic Use of Artificial Intelligence

Panel 7: International 

Perspectives

Discussion about general 

trends and different 

approaches to AI governance 

and risk management in 

international contexts.

Moderator:

Mark Latonero, National Institute of Standards and Technology

Panelists:

Olufemi Adeluyi, Ministery of Communications, Nigeria

Gry Hasselbalch, InTouchAI.EU

Aurelie Jaquet, Standards Australia

Karine Perset,, Organisation for Economic Cooperation and 

Development

Proposal for a Regulation of the European Pariliament and of the 

Council Laying Down Harmonised Rules on Artificial Intelligence 

(ARTIFICIAL INTELLIGENCE ACT) and Amending Certain Union 

Legislative Acts

Culture by design: A data interest analysis of the European AI policy 

agenda

Data Ethics of Power: A Human Approach in the Big Data and AI Era

White Paper on Data Ethics in Public Procurement of AI-based 

Services and Solutions

A framework for a data interest analysis of artificial intelligence

ALTAI - The Assessment List on Trustworthy Artificial Intelligence

OECD Framework for the Classification of AI systems

The OECD Framework for the Classification of AI systems 2-page 

summary

The OECD Framework for Classifying AI Systems to assess policy 

challenges and ensure international standards in AI. 

From principles to practice: tools for implementing trustworthy AI,

What are the tools for implementing trustworthy AI? A comparative 

framework and database.

Scoping the OECD AI principles
Building the NIST AI Risk Management Framework: Workshop #2 

4

Day 3: Thursday March 31, 2022

Theme: Contextual requirements for identifying and mitigating AI bias in socio-technical systems

Topic Speaker(s) Panel Member’s Suggested Resources

Introduction Reva Schwartz, National Institute of Standards and Technology

Panel 9: Field report – How 

does AI Bias really impact 

healthcare?

Discussion about the human 

costs of AI bias in 

healthcare.

Moderator: 

Miriam Vogel, Equal AI 

Panelists: 

Aneesh Chopra, Care Journey 

David Vawdrey, Geisinger

The Latest from Miriam Vogel

Board Responsibility for Artificial Intelligence Oversight

Panel 10: Who and what gets 

counted? Contextual 

requirements for datasets

Discussion about contextual 

requirements for datasets 

used in AI.

Moderator: 

Katie Shilton, The University of Maryland

Panelists:

Susan Aaronson, George Washington University

Razvan Amironesei, Google

Amandalynne Paullada, University of Washington

Data and its (dis)contents: A survey of dataset development and use in 

machine learning research

Excavating awareness and power in data science: A manifesto for 

trustworthy pervasive data research. Big Data & Society

Listening to Users and Other Ideas for Building Trust in Digital Trade

Could Trade Agreements Help Address the Wicked Problem of CrossBorder Disinformation?

Data Is Dangerous: Comparing the Risks That the United States, 

Panel 8: Closing and Next 

Steps 

Workshop Synthesis: 

Bringing it all together.

Elham Tabassi, National Institute of Standards and Technology

Panelists:

Catherine Aiken, Center for Security and Emerging Technology

Patrick Hall, bnh.ai

Mark Latonero, National Institute of Standards and Technology

Hodan Omaar, Center for Data Innovation, Information Technology & 

Innovation Foundation

Navrina Singh, Credo AI

Classifying AI Systems

Classifying AI Systems and accompanying data visualization

5 things lawyers should know about artificial intelligence

How to Fight Discrimination in AI

A United States Fair Lending Perspective on Machine Learning
Building the NIST AI Risk Management Framework: Workshop #2 

5

Canada and Germany See in Data Troves

America’s Uneven Approach to AI and its Consequences, in Great 

Decisions: 2020, Foreign Policy Association

Data Minefield? How AI is Prodding Governments to Rethink Trade in 

Data

Panel 11: Context is 

everything: Requirements for 

Test, Evaluation, Validation 

and Verification (TEVV)

Discussion about contextual 

requirements for TEVV in AI

Moderator: 

Apostol Vassilev, National Institute of Standards and Technology

Panelists:

Arvind Narayanan, Princeton University

Cathy O'Neil, ORCAA

Kush Varshney, IBM

NIST SP 1270: Towards a Standard for Identifying and Managing Bias 

in Artificial Intelligence, March 2022

Can You Tell? SSNet - A Biologically-Inspired Neural Network 

Framework for Sentiment Classifiers

Trustworthy Machine Learning

IBM Research AI FactSheets 360

IBM Research Uncertainty Quantification 360

AI Fairness 360

AI Privacy 360

AI Explainability 360

The Shame Machine Reviewed in the New York Times

Fairness and Machine Learning: Limitations and Opportunities

Irreproducibility in Machine Learning

Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 

Papers

Semantics derived automatically from language corpora contain 

human-like biases
Building the NIST AI Risk Management Framework: Workshop #2 

6

Panel 12: Design 

Approaches for AI: Keeping 

Human Values and Ethics at 

the Core of AI Design

Discussion about how design 

techniques can assist the 

development of AI systems 

that are more connected to 

human values and less likely 

to have harmful impacts.

Moderator: 

Kristen Greene, National Institute of Standards and Technology

Panelists: 

Kathy Baxter, Salesforce

Batya Friedman, University of Washington 

Value Sensitive Design: Shaping Technology with Moral Imagination

Envisioning Cards Toolkit

Value Sensitive Design Lab

Panel 13: Is a human in the 

loop the solution? 

Discussion about the 

challenges and evaluating 

expert-driven systems.

Moderator: 

Jenn Wortman Vaughan, Microsoft Research 

Panelists: 

Krzysztof Gajos, Harvard University

Marzyeh Ghassemi, Massachusetts Institute of Technology

Ben Green, University of Michigan

Proxy tasks and subjective measures can be misleading in evaluating 

explainable AI systems

To trust or to think: Cognitive forcing functions can reduce 

overreliance on ai in ai-assisted decision-making

Do People Engage Cognitively with AI? Impact of AI Assistance on 

Incidental Learning

Designing AI for trust and collaboration in time-constrained medical 

decisions: A sociotechnical lens

How machine-learning recommendations influence clinician treatment 

selections: the example of antidepressant selection

A Human in the Loop is Not Enough: The Need for Human-Subject 

Experiments in Facial Recognition

Designing Disaggregated Evaluations of AI Systems: Choices, 

Considerations, and Tradeoffs

A Human-Centered Agenda for Intelligible Machine Learning

Manipulating and Measuring Model Interpretability

Interpreting Interpretability: Understanding Data Scientists' Use of 

Interpretability Tools for Machine Learning

Understanding the Effect of Accuracy on Trust in Machine Learning 

Models

The Flaws of Policies Requiring Human Oversight of Government 

Algorithms