Designing trustworthy machine learning systems
By: Georgina Ibarra
Tags: Ethics, UX, Machine Learning, AI, Trust

The journey towards understanding calibrated trust

Co-authored by Meena Tharmarajah and David Douglas

Many methods have been developed to promote fairness, transparency, and accountability in the predictions made by artificial intelligence (AI) and machine learning (ML) systems. A technical approach is often the focal point of these methodologies, however, to develop truly ethical machine learning systems that can be trusted by their users, it is important to supplement this with a human-centred approach.

A deep understanding of the people, contexts and environments connected to these systems can provide the foundations for designing and presenting information in such a way that the people that use the systems can make balanced decisions on whether or not they should trust them.

We’ve established some of the key considerations for taking a human-centred approach to building machine learning systems, breaking down this complex and very technical world to understand the components of trust and best practice for designing trustworthy machine learning systems. Here we take you through some of our process and learnings using the example of a recent project involving Australian law enforcement agencies.

Connecting technology with the people that will use it

As a product and design team working for a national research organisation, we take a human centred design approach to designing technology.

This means researching the users of the technology we are creating to ensure we understand their context — their problems, needs and jobs to be done. We collaborate with social scientists to further examine the social identities and dynamics associated with these environments, contexts and people, together establishing the key insights needed to create an inclusive system considerate of a diversity of needs.

This process lifts our focus from software deliverables, enabling us to consider the social and ethical consequences of the technology we are creating.

We consider it our responsibility to create responsible technology.

Project context

Partnered with Australian law enforcement agencies, our brief was to extend agencies’ graph data capabilities and the maturity of their graph data infrastructure by developing a graph analytics software system driven by machine learning.

High security restrictions meant that working with and talking to experts in law enforcement had many limitations, restricting our ability to piece together a full picture of what we needed to solve with complete context.

What we could establish is that domain and data experts building criminal investigation cases use data to create intelligence insights with the goal of identifying connections and patterns within the data that can form the basis of further action by investigators.

We also learned that many of their data-assisted decisions have high consequences.

A human centred approach to user trust

Important frameworks have been established in recent years that outline the risks and weaknesses of artificial intelligence and machine learning, which has contributed to a dialogue around the ethical implications of this technology.

We take a domain-focused and human-centred design approach to facilitating human interaction with a specific machine learning system. This enables the dissection of critical components that a particular user needs to build their trust in the model outcomes and ultimately the trustworthiness of a particular system.

For this project we worked closely with stakeholders, users and beneficiaries of the system to design each component so they can be interpreted, evaluated and trusted by the people that use them.

This lens is what led us to our key research question: For experts making high-risk and high-consequence decisions, what are the conditions required for them to accept and use predicted data from a machine learning model to assist decision-making in their work?

Exploring the social identities and dynamics of our users

Credit: istockphoto.com/portfolio/serefkontek

To build on our understanding of the social context of the people that needed to interact with our machine learning system for this project, we collaborated with the Responsible Innovation Future Science Platform to explore concepts in machine learning, trust in automation and criminal investigation. This culminated in the recently published report“Machine Learning and Responsibility in Criminal Investigations”.

The report explains the responsibilities of criminal investigators and how these responsibilities could be affected when using ML. It identifies the factors that influence the level of trust users place in automated systems and how this trust can be influenced by different social and environmental factors.

Through a combination of user experience and social science approaches, we examined how the level of trust investigators place in the findings of ML systems can be calibrated to reflect the trustworthiness of those systems.

Three interwoven concepts of trust

The concept of trust, trustworthiness, and calibrated trust is crucial in the human use of ML systems for high-risk decision making. These concepts of trust are interdependent, making it vital to clarify their terms:

User Trust — ‘I trust the system to perform this goal.’

This trust is embodied by the user and can be triggered by multiple factors, such as their general willingness to trust automation, training and experience, situational factors such as their mood workload, and the complexity of the task. System Trustworthiness — ‘The system is trustworthy enough to perform this function.’

The onus is on the system to demonstrate how worthy it is of a user’s trust, usually for a particular function in a specific context. Calibrated Trust — ’The user’s trust is calibrated to the trustworthiness of the system.’

Ideally, the user’s trust in an ML system corresponds with the system’s capabilities. Regardless of how trustworthy the system is, the user is able to make a judgement on the best use of its predictions.

Like any good design challenge, the issue of trust in machine learning is much easier to comprehend when it is in context. Who needs to trust the outcomes from the machine learning system and why do they need to trust them? What do they need to do with the information, how well can they evaluate it, and what goal are they seeking to achieve by using it?

The data users

There are two primary users that employ data to understand networks as part of criminal investigations:

Intelligence analysts are experts in understanding business requirements of the data, but are non-technical and rely on data scientists to use big data. Data scientists are experts in creating data solutions but rely on domain experts for business requirements.

We established that data scientists and intelligence analysts were the primary users of data to build intelligence insights as part of criminal investigations.

The main reason these people use data is to build intelligence that can influence investigations and ultimately lead to prosecution. Many data assisted decisions have high consequences; trustworthiness of the system is critical for user trust.

Another more secondary user who we call the “intermediary” — a project manager or business analyst — manages data projects and processes, often championing new tools and methodologies.

The data ecosystem

The following diagram is a visual representation of where data commonly originates in criminal investigations, the transformation it underwent when applied to our machine learning system, and some of the pathways that may result from the data insights produced by the system.

Focus in the data ecosystem — the ‘mid-stream’ actors. Our contextual studies did not include the ‘up-stream’ and ‘down-stream’ actors in this ecosystem so we don’t know as much about who they are and the role(s) they may play in these parts of the ecosystem .

We then overlay the data ecosystem with some of the issues that surfaced during our research (eg, the information needed to accurately communicate the prediction data, the need for machine learning literacy in the industry’s workforce, etc).

The text highlighted in purple most closely concerns us as a technology team; the text highlighted in blue concerns stakeholders in the broader ecosystem.

These are quite high level ecosystem mappings and are just an indication of the considerable context we surfaced during our research, but they provided a way to promote conversations with the teams and stakeholders we were working with around developing user trust, system trustworthiness and calibrated trust. They also provided a starting point for understanding who is responsible for each part of the system and the role they play in mitigating challenges that arise.

Existing literature

To understand more about trust in machine learning, a literature review was undertaken to explore the methods and practices currently in use to build trust in machine learning algorithms.



Written by The Partnership on AI, “Report on Algorithmic Risk Assessment Tools in the U.S. Criminal Justice System”, provided a robust assessment of the ‘paradigmatic example of the potential social and ethical consequences of automated AI decision-making.’ It highlights three key challenges:

The ability to evaluate the accuracy, bias and validity of ML tools Accepted standards in governance, transparency and accountability The interface between the tools and the humans that interact with them.

These three key challenge areas reflect many of the important issues surfaced in our review of other literature, which are detailed in the report mentioned earlier in this article. As user researchers and designers focused on the interaction component of ML systems, the third challenge area was our focal point. It is however important to note that considerations must be made across all these areas for technology teams building ML systems that are safe for production deployment.

Guiding best practice

After completing our research on visualising predicted data and our users’ needs, context, and social dynamics, the team intended to dive straight into designing data scenarios and visualisations to test how well users understood and trusted the properties of predicted data in different situations.

But our research uncovered a gap: best practice did not yet appear to have been established. As a result, the team designed a set of practical guidelines* (see footnote) to enable informed decisions at each stage of the design and development process, and ensure a human centred approach was always front of mind.

Unfortunately, high security restrictions prevented the application of our guidelines to designing realistic scenarios with real data used for criminal investigations.

What next?

During this project, we progressed our knowledge of this complex area significantly, developing our understanding of key issues and considerations in the emerging space of designing for trust.

It’s now time for us to put this knowledge to the test with our next research project, and apply it in practice. We’ll be constructing realistic scenarios using data related to a specific domain, understanding the critical components needed for users of a machine learning system and learning from the ways that they interact with different characteristics of predicted data.

Stay tuned for our next blog post where we will share the guidelines we developed: “Guidelines for the Creation of AI Tools for Humans”, followed by a case study for how we have applied the guidelines in practice.