Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: An Iterative Convergent Mixed Methods Study (Preprint)
Authors: Lowe Cabella , Browne Mitchell , Marsh William , Morrissey Dylan 
Published: April 2022

BACKGROUND

Musculoskeletal disorders negatively impact millions of patients globally placing significant demand on healthcare systems. Digital technologies improving clinical outcomes and efficiency across the care pathway are development priorities, however rigorous evaluation must be completed to ensure efficacy. We have developed a musculoskeletal Digital Assessment Routing Tool (DART) enabling users to self-assess and be directed to right care, first time that requires usability evaluation prior to clinical trials.

OBJECTIVE

This study aimed to assess and resolve all serious DART usability issues, optimizing system performance to create a positive user experience and system adoption. In turn, providing justifiable confidence to progress to full scale randomized controlled trials when DART is integrated into musculoskeletal management pathways.

METHODS

An iterative-convergent mixed-methods design was used, with 22 adult participants using 50 different clinical presentations over 5 testing rounds across 4 DART iterations. Participants were recruited using purposive sampling with quotas for age, habitual internet use and English language ability. Quantitative data collection was defined by the constructs within the ISO 9241-210-2019 standard, with user satisfaction measured by the system usability scale. Study endpoints were resolution of all Grade 1 and 2 usability problems and a mean system usability scale score of 80 or over across a minimum of 3 user group sessions.

RESULTS

All participants (mean age 51.3 years, range 20-77 years) completed the study. Every assessment resulted in a recommendation with no DART system errors and a mean completion time of 5.2 minutes (SD 4.44, range 1-18). Usability problems reduced from 12 to zero, with trust and intention to act improving during the study. The relationship between electronic health literacy and age as explored with a scatterplot and calculation of a Pearson correlation coefficient was performed for all participants r=-0.2 (n=20) and repeated with a potential outlier removed r= -0.23, with no meaningful relationship observed or found for either. The mean satisfaction in daily internet users was highest (n, mean, 90% CL: 19, 86.5, ±1.78), compared with non-native English speakers (6, 78.1, ±9.17) and infrequent internet users scoring the lowest (3, 70.8, ±3.79), albeit the confidence intervals do overlap. The mean score across all groups was 84.3 (±4.67), corresponding to an “excellent” system, with qualitative data from all participants confirming DART was simple to use.

CONCLUSIONS

All serious DART usability issues were resolved and a good level of satisfaction, trust, and willingness to act on the DART recommendation was demonstrated, thus allowing progression to randomized controlled trials assessing safety and effectiveness against usual care comparators. The iterative convergent mixed methods design proved highly effective in fully evaluating DART from a user perspective and could provide a blueprint for other researchers of mHealth systems.