Expert panel as reference standard procedure in diagnostic accuracy studies: a systematic scoping review and methodological guidance
Authors: Kellerhuis Bas , Jenniskens Kevin , Kusters Mike P.T. , Schuit Ewoud , Hooft Lotty , Moons Karel GM , Reitsma Johannes B. 
Published: November 2024

Background

In diagnostic accuracy studies, when no reference standard test is available, a group of experts, combined in an expert panel, is often used to assess the presence of the target condition using multiple relevant pieces of patient information. Based on the expert panel’s judgment, the accuracy of a test or model can be determined. Methodological choices in design and analysis of the expert panel procedure have been shown to vary considerably between studies as well as the quality of reporting.

Objectives

To map the current landscape of expert panels used as reference standard in diagnostic accuracy or model studies.

Design

PubMed was systematically searched for eligible studies published between June 1, 2012, and October 1, 2022. Data extraction was performed by one author and, in cases of doubt, checked by another author. Study characteristics, expert panel characteristics, and expert panel methodology were extracted.

Eligibility criteria

Articles were included if the diagnostic accuracy of an index test or diagnostic model was assessed using an expert panel as reference standard and the study was reported in English, Dutch, or German.

Results

After initial identification of 4,078 studies, 318 were included for data extraction. Expert panels were used across numerous medical domains, of which oncology was the most common (20%). The number of experts judging the presence of the target condition in each patient was 2 or less in 29%, 3 or 4 in 55%, and 5 or more in 16% of the 318 studies. Expert panel types used were an independent panel (i.e., each expert returns a judgement without conferring with other experts in the panel) in 33% of studies, a panel using a consensus method (i.e., each case was discussed by the expert panel) in 27%, a staged (i.e., each expert independently returns a judgement and discordant cases were discussed in a consensus meeting) target condition assessment approach in 11%, and a tiebreaker (i.e., each expert independently returns a judgement and discordant cases were assessed by another expert) in 8%. The exact expert panel decision approach was unclear or not reported in 21% of studies. In 5% of studies, information about remaining uncertainty in experts about the target condition presence or absence was collected for each participant.

Conclusions

There is large heterogeneity in the composition of expert panels and the way that expert panels are used as reference standard in diagnostic research. Key methodological characteristics of expert panels are frequently not reported, making it difficult to replicate or reproduce results, and potentially masking biasing factors. There is a clear need for more guidance on how to perform an expert panel procedure and specific extensions of the STARD and TRIPOD reporting guidelines when using an expert panel.

Strengths and limitations of this study

This review provides an overview of trends in the use of expert panels as reference standard in diagnostic accuracy studies.

This review touches on several aspects of expert panel use that have previously not been considered, including incorporation, differential verification, uncertainty in expert judgements and diagnosis using AI.

Though this review has systematically searched PubMed, other electronic databases have not been searched, so it is possible not all diagnostic accuracy studies using an expert panel as reference standard are included.