The Role of Artificial Intelligence in Usability Testing and its Potential Impact on Equity and Medical Solution Design:
A Narrative Literature Review (Preprint)
Authors: Lombardi Selena , Montague Enid 
Published: October 2024

BACKGROUND

Usability testing is a critical component in the development of medical devices and services to minimize potential use errors that result in injury and death. Despite its importance, usability testing is often resource- and time-intensive, and traditional approaches have demonstrated inconsistencies in producing reliable results across evaluators. These limitations have raised concerns regarding whether usability evaluations adequately address the needs of diverse user groups. With the growing integration of artificial intelligence (AI) to augment usability testing practices, there is increasing uncertainty surrounding the extent to which these advancements will improve the quality of usability testing results and ensure equitable outcomes, particularly for marginalized communities.

OBJECTIVE

The purpose of this study is to understand how AI will be implemented into usability testing processes and its potential impact on the inclusivity and equity of usability testing practices.

METHODS

A narrative literature review was conducted to summarize the current state of AI in usability evaluation practices. An adapted version of the Society of Automotive Engineers (SAE) five levels of automation was used to assess the extent to which AI automates usability testing processes. The literature was also evaluated on a five-level scale for their contribution to equitable testing practices, ranging from no explicit consideration of equity (Level 0) to comprehensive equity integration (Level 5).

RESULTS

46 studies were reviewed, of which 35 focused on the development of AI tools designed to evaluate digital products, including mobile applications. These AI tools primarily supported data analysis for various usability testing methods, with particular emphasis on think-aloud protocols. The three most frequently developed tools were those aimed at identifying usability issues, performing sentiment analysis, and evaluating interface usability, leveraging innovative AI techniques. Most tools were designed to assist UX experts by automating specific stages of usability testing (Levels 1 and 2 automation). Six studies utilized AI to simulate human participants or substitute evaluators under specific conditions (Levels 3 and 4 automation). Despite these advancements, 23 studies did not address the inclusion of diverse user groups (Level 0 equity consideration), while 22 studies acknowledged equity considerations without incorporating them into their methodologies (Levels 1-3 equity consideration). Only three studies specifically explored the application of AI-supported usability tools with marginalized communities.

CONCLUSIONS

The literature review reveals a significant trend toward integrating novel AI models into usability testing, particularly over the past few years. However, despite these advancements, questions about the equity and inclusivity of these tools remain insufficiently explored. Future research efforts should develop standardized and equitable usability testing guidelines for traditional and AI-informed methods. Additionally, comparing these findings with evaluations of current AI-informed usability tools could provide insights into the market’s direction concerning equitable medical device and service development.