AI Ethics Maturity Model
By Kathy Baxter, Principal Architect, Salesforce Ethical AI Practicekbaxter@salesforce | @baxterkb

Introduction
Consumers don’t trust AI systems but they expect
companies to use them responsibly
Research shows that 90% of consumers believe thatcompanies have a responsibility to improve the stateof the world. There is guidance for how companies can
responsibly create and use technology but many consumers
are still concerned about how companies are implementing
technology. For example, a global survey in March 2021
found that citizens have low trust in AI systems but expectorganizations to uphold the principles of trustworthy AI. If yourcompany is creating and/or implementing AI and want to earn
your customers’ trust in AI while avoiding both brand and legalrisk, you need to implement an ethical AI practice in orderto develop and operationalize principles like Transparency,
Fairness, Responsibility, Accountability, and Reliability. Thismaturity model lays out a roadmap for how you might dothat based on our own experience, as well as that of othertech companies.
For the last few years, Yoav Schlesinger and I have thought a
lot about how to grow and mature our AI ethics practice at
Salesforce. We’ve spent time in self-reflection and talking toour peers at other large, U.S. enterprise tech companies thathave built their own teams and practices. From this, we’ve
identified a maturity model for building an ethical (or “trusted”
or “responsible,” choose your own word) AI practice.

Salesforce Ethical Business & Leadership Survey

Ethical AI Practice Maturity Model

Ad HocOrganized & Repeatable

Managed & Sustainable

Optimized & Innovative

Someone raises their hand

Executive buy-in established

Ethical considerations are

End-to-end inclusive

baked into the beginning ofdesign practices thatand starts asking not just“Can we do this?” but

Ethical principles andproduct development andcombine ethical AI product“Should we do this?”guidelines are agreed upon

reviews happen throughout

and engineering dev withthe lifecycle

privacy, accessibility, andInformal advocacy

Build a team of diverse

builds a groundswellexpertsof awarenessCompany-wide education

legal partners
Build or buy bias
assessment and mitigation

Ethical features andtooling

resolving ethical debt area formal part of roadmap

Ad hoc reviews and risk
assessments take place

Ethics reviews are added onto

Metrics are identified toamong “woke” teams

existing reviews, often at the

track progress and impact

end of the dev process

post-market for regular

Poor ethics metrics

auditsblock launchand resourcing

Ethical AI Practice Maturity Model: Ad hoc, Organized & Repeatable, Managed & Sustainable, Optimized & InnovativeAd Hoc
Many of the ethical AI teams created in the last 3-5 years
were the result of employee advocacy or interrogation of the
AI models or applications their companies or teams werebuilding. Some asked about potential bias in data sets being
used to do model training, while others saw the output
of biased systems like Microsoft Tay or facial recognition
systems and asked how that could happen. They also foundother like-minded individuals in the AI community via
conferences, social media, and meetups to learn from eachother. Today more executives and companies are recognizingthat unethical AI can result in legal, brand, and financial risk.
As a result, more executives are initiating the creation of
ethical AI teams.

Salesforce Ethical Business & Leadership Survey

In the ad hoc stage of the maturity model, individuals

The process of having this formal role created and filled canbegin identifying unintended consequences and informallytake a year or more of building trust among leaders andadvocating for the need to consider bias, fairness,demonstrating the importance of developing AI responsibly.

accountability, and transparency in their companies’ AI. And

However, as more executives see the importance of a

it is this advocacy that creates a groundswell of awarenessresponsible AI practice, companies without an internalamong other individual contributors and managers to pause

advocate are now looking to hire from outside.

and ask not just “can we do this?” but “should we do this?”Creating a discussion group on the company’s internal

Entire teams and dedicated budgets do not emerge

social media channel is a great way to share knowledge,

overnight, so ethics reviews by the lone ethics expert are

excitement, and identify advocates for the work. Informal

often ad-hoc and limited to individuals or small teams

tech talks are another way of spreading knowledge andthat have bought into the importance of a responsible AI

creating dialog within the company. Historically, early

practice. These small successes are critical in building up a

advocates for this approach have taken on full-time roles

portfolio of “wins” and earning more advocates across thewithin their companies to build an ethical AI practice.

company.

Organized and Repeatable
At this stage, executive buy-in has been established and the company isdeveloping a culture where responsible AI practices are rewarded. Part ofthis culture creation is the development of a set of ethical principles andguidelines. Virtually every company with an ethical AI team - including
Salesforce (einstein.ai/ethics) - has published a set of guiding principles.
There is significant overlap between the principles published by eachcompany and yet it may require significant time to create alignment
amongst key internal stakeholders, to solicit commitment to theseprinciples, and then articulate or publish them publicly. This is an important
exercise to create dialog across the company, raise awareness of potential
harm from AI systems, situate the conversation in the context of thecompany’s values, and gain true investment. Simply taking a generic set ofprinciples and publishing them on your company website will likely be little

FICO: State of Responsible AI: 2021 Report

more than “ethics washing” and result in minimal change.

Salesforce Trusted AI Principles

Responsible

Accountable

Transparent

Empowering

Inclusive

Safeguard humanrights and protect
the data we are

Seek and leverage
feedback forcontinuous
improvement.
Adhere toDevelop a transparent
user experience toguide users throughmachine-driven
recommendations.
Communicate how AI
was developed and its

Promote economic
growth andemployment for ourcustomers, theiremployees, and society
as a whole.
Give customers tools to

Respect thesocietal valuesof all those
impacted, notjust those of thecreators.

limitations.

use AI responsibly.

entrusted with.

regulations.

Salesforce Trusted AI Principles: From Principles to Practice

Responsible

Accountable

Transparent

Empowering- Work with humanrights experts- Invite customer
feedback- Strive for model
explainability- Build AI apps withclicks not code

- Test models withdiverse data sets

-Educate andempower customers
and partners- Engage Ethical Use
Advisory Council- Customers control of
their data and models- Free AI education
with Trailhead- External ethics
review of high risk AI
research papers- Clear disclosure ofterms of use- Deliver AI researchbreakthroughs- Conduct
Consequence
Scanning
Workshops- Open development
and sharing ofresearch

Inclusive- Build inclusive
teams

It is also during the Organized and Repeatable stage thatIn this stage, it’s likely that questions of scale are emerging.

an actual team is formed. This may be through current

You may have a public, company-wide commitment but

employees changing their roles to focus on AI ethics

you probably don’t have the resources to ensure that every

full-time or through hiring external experts. Ideally, thisteam building or implementing AI is doing so responsibly.

team is composed of many skill sets, backgrounds, andFormal employee education is needed because, likeintersectional diversity of race, age, gender identity, sexual

security, ethics is every employee’s responsibility, regardlessorientation, backgrounds, and more. A mix of professionalof their job title. Informal tech talks have likely been

experience in human rights, ethics and philosophy, user

happening already but now you must think about what the

research, AI, policy and regulations, as well as data science,foundational information is that every employee working

product and program management, will also yield betteron AI should know and how to contextualize it acrossoutcomes. Diversity is your superpower because different

different roles (e.g., engineering, product management,

value systems require different mechanisms for fairUX) and product teams. You want to ensure that teams aredecision-making.

asking the right questions and looping in the AI Ethics team(if you have a centralized team) or the ethics expert on yourGartner predicts that through 2022, 85% of AI
projects will deliver erroneous outcomes due tobias in data, algorithms, or the teams responsible formanaging them.
One other thing to note: The individuals who inhabit these
Responsible AI roles should not be evaluated on KPIs like
product launches or revenue generation. They should be
empowered as neutral evaluators who are not penalized
when they identify ethical risks or attempt to prevent thelaunch of a model/AI application because of those risks.
Independence is required for honesty and integrity in yourethical AI practice.

team (if you do not have a centralized team) for moderate
to high-risk use cases but aren’t inundating you with low orno risk requests.

43% of survey respondents believe they haveno responsibilities beyond meeting regulatorycompliance to ethically manage AI systems whose
decisions may indirectly affect people’s livelihoods

Through education and outreach, you will likely encounter

development resources. PM and engineering teams hate

employees that are passionate about ensuring thehaving to spend resources resolving debt, in this caseresponsible creation and use of AI. These employees are“ethical debt,” accumulated in previous releases.

likely embedded throughout the company and with deeper
training can become your eyes and ears, providing some

“Ethical debt” is the formal logging of ethics issues

minimal guidance to others in day-to-day discussions or

identified in a prior release. You may already be familiar

design decisions.

with “technical debt;” the cost of additional rework causedby choosing a cheaper/faster/easier solution instead ofAnother way to scale is by adding ethics reviews to existing

using an optimal approach that would take longer/cost

AI product reviews. If an ethics review is tacked onto themore. “Ethical debt” is accrued when you launch featuresend of the product development process, right before

that violate your ethical AI principles because, for example,

launch, it leaves teams little time to make significant

you didn’t do a bias assessment or you didn’t mitigatechanges. As a result, this review should happen early. For

the bias that was found. When ethical AI debt is found,

example, if the training data used are biased and/or not

it can be far more costly than your standard technicalrepresentative of all the users that will be impacted bydebt because you may have to identify new training dataan AI system, there is little mitigation that can be done

and retrain your model or remove features that you later

to address the potential harm once the model is built

identified cause harm. Regrettably, it may take a few painful

and ready to deploy. Although tacking a review on at thecases of blocking a launch of a potentially serious violation

end of the development cycle is seemingly the easiestof the company’s AI ethics principles in order for the ethics

approach and causes the least friction, it is not the mostreviews to be added much earlier and throughout theefficient either in terms of harms remediation or deploying

development lifecycle.

Managed and Sustainable
Although it is not perfect, you now have a manageable

At this point, your company has introduced ethics

practice that can scale as the company grows. Depending

checkpoints throughout the product lifecycle. Formal

on the size of your company and success at educatingprocesses like consequence scanning workshops, ethics

existing employees, you may be able to shift your focus

canvas, harms modeling, community juries, and creation

to ensuring new employees know what their role is inof documentation like model cards (like nutrition labels

ensuring responsible AI. Employees at many companies

for models) or FactSheets are implemented and required

have a lot of mandatory training to attend so it is worth

by management. The addition of new processes andconsidering how much training should be mandatory. Every

documentation will likely grow as your practice matures.

employee working on AI should at least know your ethicalAI principles and any customer restrictions on how your AI
can be used (for example, at Salesforce, we do not allow
our vision AI to be used for facial recognition). Beyond
that, deeper training can be limited to a smaller subset of
employees. For example, it might be the case that only
engineers or data scientists building AI need to know how
to quantitatively assess bias and how to mitigate it.

Responsible AI Development Lifecycle

Scope

Review

Test

Mitigate

Launch & Monitor

Should this exist?

Who will be impacted?

Bias Assessment in
Dataset & Model

Bias Mitigation

Publish model cards

What are yourassumptions?

Who is excluded?

Retrain Model

Post-LaunchCompare to Threshold
Set for Launch

Assessments

Ethical Red Teaming
Consequence Scanning

Who are you
designing for?

Community Feedback

Logging

User Research
In-App Support

Known risks?

Community Feedback

Responsible AI Development Lifecycle following the Agile Development Lifecycle stages of Story, Sprint, and Release

The use of FATE (fair, accountable, transparent, explainable)

If you began your ethical AI practice in one country and you

tooling and engineering practices by AI engineering andhave customers outside of that country, you must expand

data scientists is typically introduced at this stage in order toyour work to be inclusive of multiple languages, culturalconsistently identify potential bias and mitigate it in training

values, and contexts of use. You can’t simply overlay what

data and models, as well as to increase the explainability ofyou were doing in the US, for example, onto your customers

the models. The output from these tools and practices willin Japan or Mexico and expect that you will identify all of

inform your model cards or FactSheets.

the ethical risks or know how to mitigate them. It bears
repeating that different value systems require different

The other new practice that is introduced at this stage ismechanisms for fair decision-making. The historical or

the establishment of metrics to track the progress of thesocietal bias that exists in training data will differ by region

ethics work and the impact of bias on customers. You canand language and therefore must be examined within thenever claim that a data set or model is 100% bias-free orcontext of those regions and languages. If you haven’t

completely fair. Any model that appears to be 100% bias-

already, you need to start hiring in other countries.

free will suffer from overfitting and, in all likelihood, poor
performance. Instead, you can only share what type of bias
you looked for, how you measured it, what you found, andwhat you did to mitigate the bias and potential harms you
anticipate. In order to know if you are making progress andOptimized andInnovative

what impact your AI systems are having on your customers,
you need to establish metrics to track, a reporting

This is the end state you are striving for. But we intentionallymechanism to publish those metrics, and an incentive

refer to our work as a “practice” because the goal isstructure that rewards continuous improvement.

continuous improvement -- there is no such thing as“perfection” in this work. As new AI applications andApplying metrics to your internal processes and models is

methodologies are developed, new ethical risks are

just the beginning. You also need to conduct post-market

identified and new ways of mitigating them may be needed.

monitoring and auditing to understand the real-world
impact of our AI on customers and society. Bias and fairness

In order to create end-to-end ethics-by-design, mature AI

metrics in the lab are only an approximation of what will

ethics practices combine ethical AI product development

be measured in the wild. For example, even if your facialand engineering with privacy, legal, user research, design,

recognition technology (FRT) had the same error rate forand accessibility partners to create a holistic approach to

all genders and skin tones before deployment, the impacts

the development, marketing, sale, and implementation ofwhen the system “gets it wrong” can be quite different

AI. You may also have moved from a large centralized AI

for some populations. The experiences faced by innocent

ethics team to a hybrid or hub-and-spoke model. In theblack men misidentified by FRT will likely be different fromhybrid model, a centralized ethics team owns standards

innocent white women that are misidentified. You needand the creation of new processes while individual ethicists

some way

are embedded in AI product teams to provide dedicated,to monitor for harm and for individuals to ask for redresscontext-specific, and timely expertise. There is no one

and remediation.

“right” model; it depends on the size of your company, thenumber of product teams building AI applications, howdiverse those offerings are, your company’s culture,and more.

At this stage of the maturity model, product roadmaps andresources explicitly require that ethical debt is addressedand new features to help customers use your AI responsibly
are highlighted. Since metrics were established in theprevious stage, it is now possible to set minimum thresholds
for launch in order to block the launch of any new productor feature that does not meet that threshold. Of course, weknow that metrics can be manipulated so you don’t want
to depend on a single metric, or even a few, as the sole go/
no-go factor in the decision-making process around launch.
However, these metrics should be discussed and deeply
understood by those deciding what is ready to launch.

Conclusion
The Ethical AI field is relatively new and we are all learning
together as we understand risks and harms associated withcertain AI technologies or applications of them to different
populations. The proposed maturity model will change as ourunderstanding and practice develops and it is our hope that we
can co-create this field together.

Additional Resources
AI Ethics Blog Posts

einstein.ai/ethics

Online Learning

Salesforce Model
Cards

sfdc.co/EthicalandIncl
usiveProductsTrailmix

bit.ly/3oNlBMs

Ethical & Inclusive Products
Trailmix

Kathy Baxter, Principal Architect, Salesforce Ethical AI Practicekbaxter@salesforce | @baxterkb