# Integrate Usability Evaluation into Clinical
Investigation: Why, When and How?

Laura Douze, Jessica Schiro, Sylvia Pelayo

► To cite this version:

Laura Douze, Jessica Schiro, Sylvia Pelayo. Integrate Usability Evaluation into Clinical Investiga-
tion: Why, When and How?. Studies in Health Technology and Informatics, 2025, Studies in Health
Technology and Informatics, 326, pp.54-58. 10.3233/SHTI250236. hal-05153390

HAL Id: hal-05153390

https://lilloa.hal.science/hal-05153390v1

Submitted on 20 Aug 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
This article is published online with Open Access by IOS Press and distributed under the terms
of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0).
doi:10.3233/SHTI250236

Integrate Usability Evaluation into Clinical
Investigation: Why, When and How?

Laura DOUZE<sup>a,b</sup> Jessica SCHIRO<sup>a,b</sup> and Sylvia PELAYO<sup>a,b,1</sup>

ORCiD ID: Laura Douze https://orcid.org/0000-0003-1759-0273

**Abstract.** There is an increased interest in real-world usability data, especially various global regulatory bodies for Medical Devices emphasize the importance of clinical evaluation of the usability of the devices. As promising as the combination of clinical investigations in a real-world setting and usability evaluation can be, it is challenging in practice to combine these two well-established methodologies. This paper presents an exploratory study which asked why, when and how to conduct this type of combined study.

**Keywords.** Medical Device, Usability Study, Clinical investigation, Regulation

# 1. Introduction

In recent years, the landscape of medical device development and marketing has undergone significant transformation due to an increase in regulatory oversight. Governments and regulatory bodies worldwide have introduced stricter compliance requirements to ensure the safety, efficacy, and quality of these devices [1]. A growing recognition of the critical role that a human factors approach plays in ensuring device safety and effectiveness, allowed the introduction of stringent requirements emphasizing usability engineering and human factors considerations. These regulations aim to minimize risks associated with user error by ensuring that devices are designed with the end user—whether a patient or healthcare professional—in mind.

Usability studies aim to inform the design during the device development process and then to validate the final design, establishing safe and effective use in the intended use environment and by the intended users. Usability tests are typically simulated use studies, often occurring in a lab environment rather than actual use environments and often using simulated rather than actual delivery of the medical device therapy. However, simulated environments may have inherent limitations. One significant drawback is that the controlled nature of simulated environments often fails to replicate the complexities and unpredictability of real-world use. Additionally, simulated studies are conducted within a limited time window, restricting opportunities to observe how users interact with devices over extended periods. Finally, some simulations are difficult to render close enough to real-life situations, e.g. in the operating room, the fitting of implants.

<footnote><sup>a</sup> Univ. Lille, CHU Lille, ULR 2694 - METRICS : Évaluation des technologies de santé et des pratiques médicales, Lille, FR 59000</footnote>

<footnote><sup>b</sup>Inserm, CIC-IT 1403, Lille, FR 59000</footnote>

<footnote n="1">¹ Corresponding Author: Sylvia Pelayo, sylvia.pelayo@univ-lille.fr.</footnote>
Clinical studies provide a unique opportunity to gather real-world usability data for medical devices, offering insights that go beyond what can be observed in preclinical or simulated environments [2]. These studies bridge the gap between controlled usability testing and the complexities of real-world application, while maintaining a framework that may ensure patient safety [3]. But conducting this type of combined study requires merging well-established, highly regulated, and scientific test cultures. Merging these two different cultures is not a simple task, especially given the complexity of the typical clinical investigation framework [4]. Our paper presents an exploratory study which asked why, when and how to conduct this type of combined study.

## 2. Methods

Ten semi-structured interviews were conducted between April and May 2021. Participants were clinical research experts with knowledge on usability, or usability experts with clinical evaluation experience. The interviews were conducted based on a three-part interview grid:

* participants' characteristics: profile (academic or industrial, main activity) and number of years of experience.

* Presentation of combined clinical and usability studies in which they participated: the evaluated MD, the objectives, the collected data and method(s). Each participant could present 1 to 3 study(ies).

* Potential encountered difficulties, necessary adaptations, recommendations and caution points to conduct combined studies.

Interviews were recorded and subsequently analyzed. A content analysis was carried out to answer the following questions: why is it worth to integrate usability into clinical investigations? When it is possible to combine the two? And how (method, resources, points of caution, etc.) integrate usability in clinical investigation?

## 3. Results

Interviewees had diverse profiles as described in **Table 1**. Each one presented one to three examples of combined study.

### 3.1. Why should usability evaluation be incorporated into clinical investigation?

Two main reasons were underscored to combine usability studies with clinical investigations. First, considering highly innovative devices, the analysis of the contexts of use based on existing similar devices is not possible or even irrelevant (P10). For example, a risk related to inject in hairy zones for an innovative autoinjector without needle was not identified with the analysis of the context of use of an autoinjector pen with a needle (P5); the risk only emerged during a clinical investigation. In addition, groundbreaking technologies may operate in environments where practices and workflows may vary significantly across institutions or user groups. In such cases, combining usability studies with multicenter clinical investigations becomes essential to identify these heterogeneous practices and work organizations (P4).
<table><caption>**Table 1.** Participants characteristics and examples of studies combining usability and clinical outcomes</caption><thead><tr><th>N°</th><th>Academic/industrial</th><th>Main activity</th><th>Years of experience</th><th>Example of DM</th><th>Type / purpose study</th><th>DM dev stage</th><th>Data collected</th><th>Method</th></tr></thead><tbody><tr><td rowspan="2">P1</td><td rowspan="2">Academic</td><td rowspan="2">Clinical</td><td rowspan="2">15</td><td>Lumbar belt</td><td>Efficiency</td><td>Prototype</td><td>Satisfaction</td><td>Questionnaire</td></tr><tr><td>Drop mat</td><td>Technical validation</td><td>Prototype</td><td>Feedback to improve design</td><td>Informal feedback from investigators</td></tr><tr><td rowspan="4">P2</td><td rowspan="4">Academic</td><td rowspan="4">Clinical</td><td rowspan="4">20</td><td rowspan="2">Immersion cube for autistic children</td><td rowspan="2">Performance</td><td rowspan="2">Prototype</td><td>Satisfaction</td><td>Questionnaire</td></tr><tr><td>Feedback to improve design</td><td>Observations</td></tr><tr><td>Drop mat</td><td>Performance</td><td>Final DM</td><td>Satisfaction</td><td>Questionnaire</td></tr><tr><td>Exoskeleton</td><td>Performance</td><td>Final DM</td><td>Patient autonomy<br>DM use</td><td>Interviews<br>Questionnaire</td></tr><tr><td>P3</td><td>Academic</td><td>Clinical</td><td>11</td><td>Connected T-Shirt</td><td>Post market surveillance</td><td>Final DM</td><td>DM use</td><td>Interviews</td></tr><tr><td>P4</td><td>Industrial</td><td>Clinical</td><td>10</td><td>Endoscope</td><td>Feasibility</td><td>Final DM</td><td>Usability problems</td><td>Observations</td></tr><tr><td>P5</td><td>Industrial</td><td>Usability</td><td>--</td><td>Auto-injector pen</td><td>Efficiency</td><td>Final DM</td><td>Satisfaction<br>Use errors (close call)</td><td>Observation</td></tr><tr><td rowspan="3">P6</td><td rowspan="3">Academic</td><td rowspan="3">Clinical</td><td rowspan="3">13</td><td>Saliva diagnostic tool</td><td>Needs analysis</td><td>Prototype</td><td>Acceptance</td><td>Questionnaire</td></tr><tr><td>Surgical robot</td><td>Preclinical cadaver</td><td>Prototype</td><td>Usability problems</td><td>Telephone interviews</td></tr><tr><td>Dose calculation software</td><td>Comparison with/without tool</td><td>Prototype</td><td>Use errors</td><td>Comments</td></tr><tr><td>P7</td><td>Industrial</td><td>Usability</td><td>0</td><td>Pediatric drug dispensing</td><td>Phase 3 clinical trial (drug)</td><td>Final DM</td><td>Patient adherence to treatment</td><td>Abandonment rate survey</td></tr><tr><td>P8</td><td>Academic</td><td>Usability</td><td>3</td><td>Fetal biosensor</td><td>Feasibility</td><td>Prototype</td><td>Perceived usability<br>Usability problems<br>Use of the device</td><td>Questionnaire<br>Filmed observations</td></tr><tr><td rowspan="2">P9</td><td rowspan="2">Industrial</td><td rowspan="2">Clinical</td><td rowspan="2">17</td><td rowspan="2">Auto-transfuser</td><td rowspan="2">Safety and Performance</td><td rowspan="2">Final DM</td><td>Satisfaction<br>Opinion on the device<br>Usability problems</td><td>Questionnaire<br>Observations</td></tr><tr><td>Post market surveillance</td><td>Final DM</td><td>Satisfaction<br>Opinion on the device</td><td>Questionnaire</td></tr><tr><td>P10</td><td>Academic</td><td>Usability</td><td>--</td><td>Intra-oricular injection</td><td>Phase 3 clinical trial (drug)</td><td>Final DM</td><td>Usability problems<br>Injecting the correct dose</td><td>Observations<br>Interviews</td></tr></tbody></table>
Secondly, during the final stages of usability validation, integrating clinical investigations provides a more accurate assessment of the device's performance and interaction in authentic settings (P1, P7, P9, P10). Observing device use in clinical environments ensures that critical factors—such as multitasking, time constraints, or environmental variables—are properly accounted for, reducing the risk of unforeseen usability issues post-launch; “as it's not the same context (simulated VS real-life conditions), it allows us to see if user reactions are the same” (P9).

## 3.2. When should usability evaluation be incorporated into clinical investigation?

Usability data collection during clinical investigation is of interest in every phase of device evaluation: in early evaluation phases to collect data for design, in performance evaluations to collect validation data, or in post-market clinical studies to collect data not anticipated prior to real-world use.

But the objectives of the clinical investigation can lead to deviate from real-world use of the device; “those involved in clinical research do not necessarily realize that this is not a true reflection of real life” (P8). Practices or operating modes may be highly protocolized, leaving no room for errors or deviations (and fortunately). “In a clinical investigation everything is limited to meet the primary outcome which is always clinical” (P5). For example, users may not be representative end-users, e.g. an highly trained clinician in charge of injection instead of the patient (P5). For example, a unique investigator may use the device, leading to usage data from a single user (P1). For example, a device dedicated to be used at home which is tested in a hospital (P7, P10). It is therefore mandatory to assess the gap between the clinical investigation design and the real-world use of the device.

## 3.3. How should usability evaluation be incorporated into clinical investigation?

Classical usability methods are usually used to be combined with clinical investigations: mostly questionnaires (P1, P2, P3, P6, P8, P9), observations (P2, P4, P5, P8, P9, P10) and/or interviews (P3, P6, P10). Analyzing data initially collected for the clinical investigation could also be of interest for usability purposes (e.g. Adverse events or Quality of life) (P7, P8).

Difficulties with different stakeholders were often mentioned (P2, P5, P6, P7, P8, P10), such as ethical boards members unfamiliar with studies involving qualitative data. For this reason, ancillary or opportunistic studies are to be proposed. P2 for example tried to integrate a usability outcome in a clinical investigation protocol to evaluate the use of an immersion cube for autistic children. The methodologist refused this outcome, only an ancillary qualitative study was accepted.

Finally, interviewees stressed the importance of having as little impact as possible on the clinical investigation to ensure the success of the combined study (P1, P3, P4): not modify the clinical protocol, have as little impact as possible on the time required to collect data, not lengthen the human time needed for patient inclusions to be accepted by the actors who will conduct the study. Collaboration between the clinical team and the usability team is considered a key factor (P3, P4), as well as clinical staff awareness and training (P3, P6, P8, P10) and as far as possible a dedicated person in charge of usability data collection (P4, P9).
## 4. Discussion

Sometimes it may be required to assess usability of a device use within a real-world environment, requiring the integration of methods from the human factor and clinical worlds. Such studies may provide a broad view of interactions with a device, in the context of a real-world setting and over an extended duration. Such studies seek to collect data towards both usability and clinical endpoints, and to understand how such endpoints are related. For example, it is interesting to fund clinical usability claims such as those to feasibility of long-term treatment supported by a certain level of engagement and satisfaction with a device. As promising as the combination of clinical investigations in real-world setting and usability evaluations can be, it is challenging in practice to combine these two well-established methodologies.

Based on combined studies experience, this exploratory study provides some insights on why, when and how to combine such studies. The value of integrating the two types of study will depend on the comparability of the uses during the clinical study with the uses in real-world, i.e. the end-users or the environment concerned. The feasibility will depend on the impact that the usability study objectives have on the clinical investigation. Relying on the resources already in place for the clinical investigation is a major advantage whenever possible. In this type of study, the clinical investigation is at the forefront, with usability secondary endpoints. A close collaboration between both clinical and usability teams is needed. But sometimes, “clinical usability studies” are needed with the combination of medical claims that are dependent on the use and the usability of the device [3]. Connected devices and Artificial Intelligence technologies are pushing the classical simulated-use studies to their limits. Clinical usability studies enable the assessment of compliance and/or adoption in a naturalistic environment. They allow for the investigation of device usability over time, in the context of disease progression or health beliefs. Finally, they facilitate the investigation of root causes associated with product use. But the greatest tension in such a format is the need to collect “pure”, unmanipulated, real-world data from actual patients and the necessary periodic interactions of the study staff with the participants.

## References

[1] Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices. 32017R0745, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32017R0745 (2017, accessed 15 January 2024).

[2] Douze L, Schiro J, Heyndels L, et al. Evaluations of medical device usability during clinical investigations: a scoping review of clinical study protocols. *Expert Review of Medical Devices* 2024; 21: 781–788.

[3] Jalil S, Myers T, Atkinson I, Soden M. Complementing a Clinical Trial With Human-Computer Interaction: Patients' User Experience With Telehealth. JMIR Hum Factors. 2019 Jun 6;6(2):e9481. doi: 10.2196/humanfactors.9481. PMID: 31172958; PMCID: PMC6592491.

[4] Irving C, Culverhouse I. Human factors integration with clinical investigations. J Med Eng Technol. 2023 Nov;47(8):396-402. doi: 10.1080/03091902.2024.2355322. Epub 2024 Jun 10. PMID: 38856984.

[5] Stüdeli, T., Hochberg, L.. Clinical Usability Studies – Clash of Cultures? Study Design Proposal from Lessons Learned. In: Black, N.L., Neumann, W.P., Noy, I. (eds) Proceedings of the 21st Congress of the International Ergonomics Association (IEA 2021). IEA 2021. Lecture Notes in Networks and Systems, vol 223. Springer, Cham. https://doi.org/10.1007/978-3-030-74614-8_89