Clinical Turing tests with user certainty analysis to create and validate synthetic electrocardiogram images for artificial intelligence-enhanced algorithm development
Authors: Bodagh N , Tun K S , Barton A , Javidi M , Kotadia I , Klis M , Gharaviri A , Vigneswaran V , Niederer S , O'neill M , Bernabeu M O , Williams S E 
Published: May 2024

Abstract

Background
Artificial intelligence-enhanced electrocardiogram (AI-ECG) algorithms have primarily been created using digitised signal data, owing to a relative absence of publicly available image-based datasets. ECGs are often scanned or photographed into electronic health records. For maximum clinical utility, AI-ECG algorithms should be applicable to these data. Synthetic data could expedite the creation of extensive, fully anonymised image-based ECG datasets to permit training image-based AI algorithms, but it is essential that such datasets contain the artefacts encountered in clinical practice. We investigated whether iterative clinical Turing tests with user certainty analysis could be used to develop and validate synthetic ECG data.

Purpose
To create synthetic ECG images containing the artefacts typically encountered in clinical practice, and to validate the images through iterative Turing testing and user certainty analysis.

Methods
Synthetic ECG images containing artefacts were created using the PTB-XL dataset (a publicly available signal-based dataset comprising 21799 ECGs) as source data. Iterative clinical Turing tests were conducted where healthcare professionals completed an online survey comprising 60 real and synthetic ECGs. Participants were asked to select whether they thought ECGs were real or synthetic. For user certainty analysis, participants were asked to rate their confidence in their answers using a five-point Likert scale (Figure 1). Likert scale responses were converted into a signed ordinal scale representing user certainty in the identification of real or synthetic data. This scale was used to perform Receiver Operating Characteristic (ROC) analysis. Following quantitative survey completion, qualitative feedback was sought and used to iteratively improve the realism of the synthetic images.

Results
A total of 26 healthcare professionals completed the clinical Turing tests over three rounds. Qualitative feedback was used to improve the fidelity of the synthetic ECG images between rounds (Table 1). During iterative testing, the proportion of synthetic ECGs correctly identified fell from 61.5% to 53.7%, and the proportion of real-world ECGs correctly identified fell from 66.3% to 53.0% (Figure 1). Following the final Turing test, ROC analysis revealed no discriminative ability for identifying synthetic data (C-statistic 0.480, 95% confidence interval 0.432-0.529).

Conclusion
Iterative Turing testing with user certainty analysis and qualitative user feedback may be used to create synthetic ECG images containing the artefacts typically encountered in clinical practice. Iterative Turing testing improved the images’ realism confirming their potential to augment image-based AI algorithm development. The presented methodology establishes a framework to develop high fidelity, synthetic patient datasets presenting a significant opportunity to enhance the uptake of AI within electrophysiology, cardiology, and medicine.