# Different latent class models were used and evaluated for assessing the accuracy of campylobacter diagnostic tests: overcoming imperfect reference standards?

J. Asselineau, A. Paye, E. Bessede, P. Perez, Cecile Proust-Lima

► To cite this version:

J. Asselineau, A. Paye, E. Bessede, P. Perez, Cecile Proust-Lima. Different latent class models were used and evaluated for assessing the accuracy of campylobacter diagnostic tests: overcoming imperfect reference standards?. Epidemiology and Infection, 2018, 146 (12), pp.1556-1564. 10.1017/s0950268818001723. hal-02970821

HAL Id: hal-02970821

https://hal.science/hal-02970821

Submitted on 19 Oct 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Original Paper

*These authors contributed equally to this work.

**Cite this article:** Asselineau J, Paye A, Bessède E, Perez P, Proust-Lima C (2018). Different latent class models were used and evaluated for assessing the accuracy of campylobacter diagnostic tests: overcoming imperfect reference standards? *Epidemiology and Infection* **146**, 1556–1564. https://doi.org/ 10.1017/S0950268818001723

Received: 19 January 2018
Revised: 18 April 2018
Accepted: 27 May 2018
First published online: 27 June 2018

**Key words:**
Campylobacter; diagnostic accuracy; imperfect gold standard; latent class model; sparseness

**Author for correspondence:**
J. Asselineau,
E-mail: julien.asselineau@u-bordeaux.fr

© Cambridge University Press 2018. This is an
Open Access article, distributed under the
terms of the Creative Commons Attribution
licence (http://creativecommons.org/licenses/
by/4.0/), which permits unrestricted re-use,
distribution, and reproduction in any medium,
provided the original work is properly cited.

CAMBRIDGE
UNIVERSITY PRESS

Different latent class models were used and
evaluated for assessing the accuracy of
campylobacter diagnostic tests: overcoming
imperfect reference standards?

J. Asselineau¹,², A. Paye¹,², E. Bessède³, P. Perez¹,²,* and C. Proust-Lima¹,⁴,*

¹Bordeaux University Hospital, Public Health Department, Clinical Epidemiology Unit, F-33076 Bordeaux, France;
²INSERM, CIC 1401 Clinical Epidemiology, F-33076 Bordeaux, France; ³French National Reference Center for Campylobacter and Helicobacter, F-33076 Bordeaux, France and ⁴INSERM, UMR1219, Univ. Bordeaux, ISPED, F-33076 Bordeaux, France

Abstract

In the absence of perfect reference standard, classical techniques result in biased diagnostic accuracy and prevalence estimates. By statistically defining the true disease status, latent class models (LCM) constitute a promising alternative. However, LCM is a complex method which relies on parametric assumptions, including usually a conditional independence between tests and might suffer from data sparseness. We carefully applied LCMs to assess new campylobacter infection detection tests for which bacteriological culture is an imperfect reference standard. Five diagnostic tests (culture, polymerase chain reaction and three immu- noenzymatic tests) of campylobacter infection were collected in 623 patients from Bordeaux and Lyon Hospitals, France. Their diagnostic accuracy were estimated with standard and extended LCMs with a thorough examination of models goodness-of-fit. The model including a residual dependence specific to the immunoenzymatic tests best complied with LCM assumptions. Asymptotic results of goodness-of-fit statistics are substantially impaired by data sparseness and empirical distributions were preferred. Results confirmed moderate sen- sitivity of the culture and high performances of immunoenzymatic tests. LCMs can be used to estimate diagnostic tests accuracy in the absence of perfect reference standard. However, their implementation and assessment require specific attention due to data sparseness and limita- tions of existing software.

**Introduction**

The evaluation of new diagnostic tests requires the assessment of the diagnostic test accuracy, usually in terms of sensitivity and specificity. It consists in confronting the index test results with the presence/absence of the target condition. The reference standard is the best method available for attesting the target condition [1]. However, in many cases it is not a Gold Standard, it does not have sensitivity and specificity of 100%. This imperfection of the refer- ence standard translates into the misclassification of patients with regard to the target condi- tion and entails biased estimates of the prevalence and accuracy of new diagnostic tests [2–7]. Different solutions have been proposed and reviewed to evaluate diagnostic test accuracy in the presence of an imperfect Gold standard as the correction of accuracy parameters according to external data or the use of multiple reference standards (expert panel, composite reference standard) [4, 5]. An alternative is to rely on a purely statistical approach, the latent class frame- work, sometimes referred to as latent class models (LCM), latent class analyses (LCA) or mix- ture models [8–16]. This method defines the true target condition status as two latent classes and estimates of the sensitivities and specificities of the diagnostic tests, possibly including the imperfect Gold standard, according to these latent classes. LCMs have sparked a growing inter- est in Biostatistics with contributions evaluating their robustness to the misspecification of the assumption of independence between diagnostic tests conditionally on the true condition sta- tus [17–21], recommending methods to assess this assumption [10, 14, 22–24] and proposing extensions to take into account such misspecification [10, 12, 25, 26].

In the recent years, clinical applications of this approach have been reported in the medical and veterinary literature. However, the models and the underlying assumptions or application conditions were rarely evaluated whereas such violations can lead to biased reported estimates of diagnostic accuracy [14].

Diarrhoeal diseases are responsible for 550 million people falling ill yearly, including 220 million children under the age of 5 years and campylobacter bacteria is the most frequent cause of bacterial gastroenteritis worldwide. Campylobacter infections are generally mild but can be fatal among very young children, elderly and immunosuppressed individuals [27]. Its
diagnosis relies on stool cultures that have a moderate sensitivity because of the fragility and special culture requirement of cam-pylobacter bacteria (microaerobic environment): a too long con-tact with a not controlled atmosphere may inhibit the growth of the bacteria. By contrast, specificity is expected to be very high as it may almost only be altered by bacterial contamination. As campylobacter growth is also slow and usually takes more than 48 h, other diagnostic tests (immunoenzymatic, molecular and immunochromatographic methods) have been developed and are available. They are easy to apply and interpret and their results are obtained much more quickly than those of culture (from 30 minutes to 2 h). However, their diagnostic accuracy cannot be correctly assessed using conventional comparisons due to the imperfect reference standard of culture [28–30]. For example, true cases missed by culture but detected by the new test would be incorrectly classified as false positives. By accounting for the imperfection of culture, LCMs, by contrast, have the potential to correctly estimate the diagnostic accuracy of these new tests. In addition, LCMs can take into account a conditional depend-ence between tests, which arises for example when different tests make the same error. In campylobacter diagnosis, it might be the case when considering various immunological tests, espe-cially if based on the same campylobacter antigens.

In this work, we aimed at carefully applying the LCM meth-odology to evaluate diagnostic test accuracy through a real case study, the evaluation of new rapid diagnostic tests of campylo-bacter. Based on current recommendations [14] and statistical developments, we specifically explored the means to assess goodness of fit of LCMs, mostly regarding the sparseness of the data and the violation of the conditional independence assumption and to implement extended LCMs involving ran-dom effects.

## Methods

### Study population

Our analysis relies on data from two studies that included every stool specimen obtained from a patient with a gastrointestinal ill-ness at Bordeaux University Hospital (Bordeaux, France) from June to October 2009 and at Lyon University Hospital (Lyon, France) from February to September 2012 [28, 31]. Stools were sent to the laboratory at room temperature without transport medium. The fresh, unpreserved stools were tested for culture within 4 h after arriving at the laboratory. The remaining part of the stool samples was then frozen at -80 °C. The other tests were performed together, once a week, after the samples were thawed. For the Lyon study, culture and immunochromatographic tests were performed in Lyon, ELISAs and polymerase chain reac-tion (PCR) were performed in Bordeaux.

### Diagnostic tests

Every stool specimen was tested by five different diagnostic tests for campylobacter briefly described below. For more details, please refer to Bessède et al. [28]:

- Culture. A stool suspension was prepared, plated on a Karmali agar (Oxoid, Basingstoke, Hampshire, UK) and the plates were incubated for a maximum of 3 days in a microaerobic atmos-phere. Colonies resembling campylobacter colonies were tested with a MALDI-TOF mass spectrometer.

- Rapid immunochromatographic tests. ImmunoCardSTAT! Campy (Meridian Bioscience, Inc., Cincinnati, OH, USA) was used according to the manufacturer's instructions. It is an immunochromatographic test which detects specific cam-pylobacter antigens on a band. The result was read and vali-dated if the control line band was clearly visible.

- ELISAs. Two different tests were used: RIDASCREEN® Campylobacter (R-Biopharm AG) and Premier® Campy (Meridian Bioscience, Inc.), both according to the manufac-turer's instructions.

- Real-time PCR. The real-time PCR and hybridisation reactions were performed according to the method published by Ménard et al., using a LightCycler thermocycler (Roche Diagnostics, Meylan, France) [32].

### Statistical analysis

We considered different specifications of LCM to estimate diag-nostic accuracy parameters (sensitivity, specificity, negative and positive predictive values) of each test and the prevalence of cam-pylobacter infection in the study population.

### Standard LCM

The LCM assumes that the true target condition status, campylo-bacter infection in our case, is not observed. It is statistically defined by a binary latent variable with two modalities corre-sponding to the absence and the presence of the disease (called disease-free latent class and disease latent class, respectively). The disease prevalence is thus given by the probability P to belong to the disease latent class. Sensitivity $Se_k$ and specificity $Sp_k$ of each test $k$ ($k = 1, ..., K$) correspond to the conditional probabilities of each test result given the latent classes.

The standard LCM (LCM CI, Fig. 1a) relies on the central assumption, called ‘conditional independence assumption’, that test results are independent given the latent classes. Based on this assumption, the probability of observing a response profile for the K tests denoted $T_1$ to $T_K$ can be expressed as the sum of two terms, one per class:

$$P(T_1 = t_1, T_2 = t_2, \dots, T_K = t_K) = p \times \prod_{k=1}^{K} [Se_k^{t_k^+} \times (1 - Se_k^{t_k^-})] \quad \text{(disease latent class)} \\ + (1-p) \times \prod_{k=1}^{K} [Sp_k^{t_k^-} \times (1 - Sp_k^{t_k^+})] \quad \text{(disease-free latent class)}$$

with $t_k$ the result of test $T_k$ noted $t_k^+$ if positive and $t_k^-$ if negative.

In estimation procedures, $Se_k$ and $Sp_k$ probabilities (for $k = 1, ..., K$) are usually modelled using a probit link while the preva-lence p is usually modelled using a logit link [10, 12].

### Extended LCMs

The conditional independence assumption is rarely verified in practice so we investigated two main alternatives which consider residual dependences between tests through individual random effects [10, 12]:

- A LCM with a common residual dependence between the tests (LCM CD, Fig. 1b). In this model, an individual random
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Diagram (left panel) and corresponding profile probability (right panel) for three latent class models assuming different dependence structures, CampyLCA study, France, 2016. LCM CI, latent class model under conditional independence; LCM CD, latent class model with a residual dependence common to all tests; LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests. Ovals and rectangles indicate latent quantities and observed quantities, respectively: D = 0/1: unobserved presence/absence of campylobacter infection; T₁: Culture Karmali; T₂: Real-time PCR; T₃: Ridascreen®; T₄: Premier®Campy®; T₅: ImmunoCardStat!®Campy; u: random residual dependence which follows a standard Gaussian distribution. In the equations, tₖ⁺ and tₖ⁻ indicate a positive and negative result for test Tₖ, respectively; Φ is the standard cumulative Gaussian distribution function; parameters to estimate are (akd)k = 1,...,K, d = 0, 1 for the probit transformations of sensitivities and specificities, μ for the logit transformation of the prevalence and σ for the intensity of the individual random deviation.</figcaption></figure>

intercept, added to the probit model for the sensitivities and specificities of the tests, captures the residual correlation between tests.

- A LCM with a specific residual dependence within the three immunoenzymatic tests (Premier®Campy, Ridascreen®, ImmunoCard Stat!®Campy) (LCM SD, Fig. 1c). In this model, an individual random intercept is added to the probit models only for a subset of tests suspected to be conditionally dependent.

### Estimation of standard and extended LCMs

Parameters of LCMs were estimated in the maximum likelihood framework. Identifiability of LCMs requires that the number of parameters does not exceed $2^K-1$ ($2^K$ being the number of possible dichotomous tests combinations). So, at least three tests in LCM CI and four tests in LCM CD and SD are necessary to estimate all the parameters (2K sensitivity/specificity parameters, plus one for prevalence plus one for the intensity associated with the random intercept in LCM CD and LCM SD). We implemented the models using RandomLCA package [33] and NLMIXED procedure (SAS Institute, Inc., SAS software version 9.3, Cary, North Carolina). Because convergence towards local maxima is frequently encountered in mixture models [34], we considered 100 sets of random initial values, either completely at random (with R) or chosen within clinically plausible ranges (with SAS) to ensure convergence towards the global maximum. The introduction of a random effect in the likelihood for LCM CD and LCM SD induced an integral that had to be numerically

solved [35]. It was done with adaptive Gaussian quadratures implemented in both programs. After convergence, two-sided 95% confidence interval (95% CI) of each parameter was obtained by a Monte Carlo approximation.

### A posteriori evaluation of LCM

As recommended by some authors [10, 12, 14, 23, 24], we used a series of post-fit criteria and posterior analyses to thoroughly assess the models:

(1) Models were compared in terms of Akaike Information Criterion.

(2) Absence of residual dependence between the tests was verified using goodness-of-fit statistics which compare model predictions with observations (Pearson, Likelihood Ratio and Power Divergence statistics). Both the asymptotic Chi-square distribution and empirical distributions were considered for the statistic under the null hypothesis. Indeed, in the context of sparse data (many profiles with low frequencies), results with the asymptotic distribution may not apply and only those obtained with empirical distributions are recommended [12, 22–24, 36]. The empirical distribution was obtained by generating a large number of samples ($n = 500$) from the null assumption and computing the corresponding statistic; the P-value was deduced from the quantile which corresponded to the statistic in the observed sample.

(3) Pairwise residual correlations and bivariate residual statistics were calculated to detect potential residual dependences
<table><caption>**Table 1.** Test results profiles: observed and predicted (by the Latent Class Models) number of patients for each combination of test results, CampyLCA Study, France, 2016</caption><thead><tr><th rowspan="2">Culture<br>Karmali</th><th rowspan="2">Real-time<br>PCR</th><th rowspan="2">Rida-screen®</th><th rowspan="2">Premier®<br>Campy</th><th rowspan="2">ImmunoCard Stat!®<br>Campy</th><th rowspan="2">Observed<br>patients (%)</th><th colspan="3">Predicted patients</th></tr><tr><th>LCM CI</th><th>LCM CD</th><th>LCM SD</th></tr></thead><tbody><tr><td>—</td><td>—</td><td>—</td><td>—</td><td>—</td><td>522 (83.8)</td><td>519.6</td><td>521.2</td><td>522.1</td></tr><tr><td>—</td><td>—</td><td>—</td><td>—</td><td>+</td><td>15 (2.4)</td><td>16.9</td><td>16.4</td><td>15.6</td></tr><tr><td>—</td><td>—</td><td>—</td><td>+</td><td>+</td><td>4 (0.6)</td><td>0.2</td><td>0.7</td><td>1.4</td></tr><tr><td>—</td><td>—</td><td>+</td><td>—</td><td>—</td><td>7 (1.1)</td><td>9.6</td><td>7.6</td><td>5.2</td></tr><tr><td>—</td><td>—</td><td>+</td><td>—</td><td>+</td><td>3 (0.5)</td><td>0.4</td><td>1.8</td><td>4.1</td></tr><tr><td>—</td><td>—</td><td>+</td><td>+</td><td>+</td><td>2 (0.3)</td><td>2.6</td><td>2.4</td><td>3.0</td></tr><tr><td>—</td><td>+</td><td>—</td><td>—</td><td>—</td><td>3 (0.5)</td><td>3.4</td><td>2.4</td><td>3.0</td></tr><tr><td>—</td><td>+</td><td>+</td><td>—</td><td>—</td><td>1 (0.2)</td><td>0.1</td><td>0.5</td><td>0.2</td></tr><tr><td>—</td><td>+</td><td>+</td><td>+</td><td>—</td><td>1 (0.2)</td><td>1.7</td><td>1.9</td><td>1.2</td></tr><tr><td>—</td><td>+</td><td>+</td><td>+</td><td>+</td><td>9 (1.4)</td><td>10.8</td><td>7.1</td><td>8.7</td></tr><tr><td>+</td><td>—</td><td>—</td><td>—</td><td>—</td><td>2 (0.3)</td><td>1.9</td><td>1.5</td><td>2.0</td></tr><tr><td>+</td><td>—</td><td>—</td><td>+</td><td>—</td><td>1 (0.2)</td><td>0.1</td><td>0.2</td><td>0.0</td></tr><tr><td>+</td><td>—</td><td>+</td><td>+</td><td>—</td><td>1 (0.2)</td><td>1.2</td><td>1.2</td><td>0.8</td></tr><tr><td>+</td><td>—</td><td>+</td><td>+</td><td>+</td><td>5 (0.8)</td><td>8.1</td><td>4.3</td><td>5.4</td></tr><tr><td>+</td><td>+</td><td>+</td><td>—</td><td>—</td><td>1 (0.2)</td><td>0.2</td><td>0.2</td><td>0.8</td></tr><tr><td>+</td><td>+</td><td>+</td><td>+</td><td>—</td><td>5 (0.8)</td><td>5.2</td><td>3.5</td><td>5.7</td></tr><tr><td>+</td><td>+</td><td>+</td><td>+</td><td>+</td><td>41 (6.6)</td><td>34.2</td><td>42.4</td><td>40.2</td></tr></tbody></table>

LCM CD, latent class model with a residual dependence common to all tests; LCM CI, latent class model under conditional independence; LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests.

between pairs of diagnostic tests that were not correctly taken into account [10, 12, 24, 37].

(4) Leave-one-test-out analyses were performed by removing one by one each immunoenzymatic test in order to assess their influence on the diagnostic accuracy of the other medical tests [14].

In supplementary simulations, we assessed the type-I error rates obtained with the asymptotic distributions of statistics from point 2 in the specific case of our application data to better appreciate the performances of usual statistics in our sparseness context. The type I error rate quantifies the percentage of times when the test concludes that observations and predictions significantly differ while they actually do not. We also evaluated the power of the same statistics relying on the empirical distributions to detect a violation to the conditional independence assumption. The power quantifies the percentage of times when the test concludes that observations and predictions significantly differ and they actually do.

Statistical tests were all performed at the significance level of 5%.

## Results

From the 32 profiles of test responses possible with five dichotomous diagnostic tests (2⁵), 17 were observed and only 10 included at least three patients (Table 1) among the 623 patient samples. This underlines the sparseness of our data. The most frequent profiles were ‘all tests negative’ (83.8%) and ‘all tests positive’ (6.6%). According to the classical reference standard, bacteriological

culture, the prevalence of campylobacter infection was 9.0% (95% CI 8.8–9.3).

Predicted profiles frequencies estimated with the three LCMs were globally close to observed frequencies (Table 1) but the two models considering a residual dependence (LCM CD and LCM SD) were closer at least for the most frequent profiles. For instance, the ‘all negative tests’ profile was predicted at 519.6, 521.2 and 522.1 with LCM CI, CD and SD., respectively, for 522 patients observed; similarly, the ‘all tests positive’ profile was predicted at 34.2, 42.4 and 40.2 with LCM CI, CD and SD, respectively, for 41 patients observed. Note that among the 15 non-observed profiles, two profiles with LCM CI and CD model and one profile for LCM SD had predicted frequencies above one (data not shown).

Comparison of LCM CI, CD and SD models in terms of goodness of fit is summarised in Table 2. We only interpret in the following the statistics based on empirical distributions due to the sparseness of our data; results based on the asymptotic distributions are given only to illustrate their lack of reliability in the presence of sparseness. LCM CI under conditional independence hypothesis presented the worst Akaike information criterion and this specification was highly rejected by all statistics. The LCM CD provided an improved Akaike information criterion (by 17.6 points) but all the test statistics still rejected the adequacy of the model. LCM SD provided the best Akaike information criterion (improved by 29.6 points compared with LCM CI and 12.0 compared with LCM CD) and none of the goodness-of-fit tests rejected the specification of LCM SD even if P-values were just above the significance threshold. We explored other dependency
<table><caption>**Table 2.** Akaike information criterion and goodness-of-fit statistics for each model, CampyLCA Study, France, 2016</caption><thead><tr><th></th><th>LCM CI</th><th>LCM CD</th><th>LCM SD</th></tr></thead><tbody><tr><td>Akaike information criterion</td><td>1041.5</td><td>1023.9</td><td>1011.9</td></tr><tr><td>Pearson statistics</td><td></td><td></td><td></td></tr><tr><td>Asymptotic χ² distribution</td><td>&lt;0.001</td><td>0.012</td><td>0.021</td></tr><tr><td>Empirical distribution<sup>a</sup></td><td>&lt;0.001</td><td>0.022</td><td>0.052</td></tr><tr><td>Likelihood ratio statistics</td><td></td><td></td><td></td></tr><tr><td>Asymptotic χ² distribution</td><td>&lt;0.001</td><td>0.052</td><td>0.52</td></tr><tr><td>Empirical distribution<sup>a</sup></td><td>&lt;0.001</td><td>0.004</td><td>0.086</td></tr><tr><td>Power divergence statistics</td><td></td><td></td><td></td></tr><tr><td>Asymptotic χ² distribution</td><td>&lt;0.001</td><td>0.039</td><td>0.20</td></tr><tr><td>Empirical distribution<sup>a</sup></td><td>&lt;0.001</td><td>0.012</td><td>0.054</td></tr></tbody></table>

LCM CD, latent class model with a residual dependence common to all tests; LCM CI, latent class model under conditional independence; LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests.

structures but the latter, based on biological knowledge, remained the most satisfying one.

Evaluation of the local independence hypothesis is shown in Figure 2. While residual correlations were not highlighted for LCM CD or SD based on their 95% confidence interval, bivariate statistics rejected the local independence hypothesis at the 5% level for four pairs of tests in LCM CD and still for two pairs of tests for LCM SD.

Although it did not satisfy all the criteria, the LCM SD model presented the best evaluations. According to this model (Table 3), the prevalence of campylobacter infection was estimated at 10.5% (95% CI 8.4–13.3). The standard error of the random effect specific to the three immunoenzymatic tests was statistically different from zero (estimated at 1.6, (95% CI 0.9–2.4)). The diagnostic accuracy of the different medical tests according to LCM SD are displayed and compared with those obtained when using the culture as the reference standard in Figure 3. As expected, culture presented a moderate sensitivity (82.1%, (95% CI 70.2–90.1)). Using culture as the reference standard resulted in a systematic underestimation of other tests sensitivities and specificities. According to LCM SD model Ridascreen® and Premier®Campy tests showed the best compromise between sensitivity and specificity, both above 97%. Estimations and 95% CI of accuracy parameters of all medical tests according to all the LCMs are given in Table 3.

We note that leave-one-test-out analyses did not show relevant differences in the prevalence and diagnostic accuracy when removing each immunoenzymatic medical tests one by one (Table 4).

## Discussion

Using a reference standard whose diagnostic accuracy is known to be far from perfection necessarily leads to biased estimations of new detection tests if the imperfection is not properly taken into account. Based on a dataset of 623 patient samples, the diagnostic accuracy of five tests of campylobacter infection was estimated by using LCMs to palliate the imperfection of the

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Evaluation of local independence hypothesis by residual correlations and their 95% confidence interval, as well as by *P*-values of bivariate statistics, CampyLCA study, France, 2016. (a) Residual correlations for latent class model under conditional independence; (b) Residual correlations for latent class model with a residual dependence common to all tests; (c) Residual correlations for latent class model with a residual dependence specific to the three immunoenzymatic tests. *T*₁: Culture Karmali; *T*₂: Real-time PCR; *T*₃: Ridascreen®; *T*₄: Premier®Campy; *T*₅: ImmunoCard Stat!®Campy. Residual correlations presented with dots (point estimates) and bars (95% confidence intervals). *P*-values of bivariate statistics are provided above each pair of tests described on the horizontal axis.</figcaption></figure>

bacteriological culture. The model with a specific dependence within the three immunoenzymatic tests showed the best performances in terms of fit and compliance with LCM assumptions. The prevalence of infection was estimated about 10%. LCM results confirmed the moderate sensitivity and almost perfect specificity of the culture. Ridascreen® and Premier®Campy showed very high sensitivities (98.5% and 97.2%, respectively) and very high specificities (97.9% and 99.1%, respectively) confirming their potential usefulness for diagnosing campylobacter infection in clinical practice.

As expected with the moderate sensitivity of culture, the prevalence of infection was higher when estimated by LCM, even if the difference remained tenuous. The incorrect use of culture as a reference standard also led to the underestimation of sensitivity and specificity of all the index tests. Indeed, when using a reference standard with moderate sensitivity and perfect specificity, as a culture, the patients falsely considered as disease-free contribute to wrongly classify positive results of index tests as ‘false positives’ and negative results as ‘true negatives’. For index tests with high sensitivity, this leads to a systematic underestimation of their sensitivity and specificity (as found in our application) and, to a greater extent, of their positive predictive value. In our campylobacter case, the use of bacteriological culture as a reference

<footnote>ªThe *P*-value when using the empirical distribution was calculated as one minus the percentile of the statistic in 500 samples generated under the null assumption.</footnote>
<table><caption>**Table 3.** Diagnostic accuracy of medical tests according to LCM models, CampyLCA Study, France, 2016</caption><thead><tr><th></th><th>Reference standard</th><th>LCM CI</th><th>LCM CD</th><th>LCM SD</th></tr></thead><tbody><tr><td colspan="5" style="text-align:center; font-weight:bold;">Sensitivity, 95% CI</td></tr><tr><td>Culture Karmali</td><td>100.0</td><td>76.0</td><td>64.7-84.8</td><td>78.4</td><td>64.8-88.4</td><td>82.1</td><td>70.7-90.0</td></tr><tr><td>Real-time PCR</td><td>83.9</td><td>74.3-93.5</td><td>80.9</td><td>70.5-89.0</td><td>83.9</td><td>67.8-93.3</td><td>88.2</td><td>78.1-94.6</td></tr><tr><td>Ridascreen®</td><td>94.6</td><td>88.7-100.0</td><td>93.0</td><td>85.3-97.2</td><td>96.7</td><td>68.7-99.9</td><td>98.5</td><td>91.1-99.9</td></tr><tr><td>Premier®Campy</td><td>94.6</td><td>88.7-100.0</td><td>97.1</td><td>87.0-99.7</td><td>96.9</td><td>83.4-99.7</td><td>97.2</td><td>89.3-99.5</td></tr><tr><td>ImmunoCard Stat!®Campy</td><td>82.1</td><td>72.1-92.2</td><td>86.7</td><td>76.7-93.2</td><td>85.8</td><td>74.7-93.1</td><td>85.2</td><td>73.4-92.1</td></tr><tr><td colspan="5" style="text-align:center; font-weight:bold;">Specificity, 95% CI</td></tr><tr><td>Culture Karmali</td><td>100.0</td><td>99.6</td><td>98.7-99.9</td><td>99.5</td><td>98.3-99.9</td><td>99.6</td><td>98.7-99.9</td></tr><tr><td>Real-time PCR</td><td>97.5</td><td>96.3-98.8</td><td>99.4</td><td>98.1-99.8</td><td>99.3</td><td>98.0-99.8</td><td>99.5</td><td>98.4-99.8</td></tr><tr><td>Ridascreen®</td><td>95.9</td><td>94.3-97.6</td><td>98.2</td><td>96.7-99.1</td><td>98.2</td><td>96.6-99.1</td><td>97.9</td><td>96.3-98.9</td></tr><tr><td>Premier®Campy</td><td>97.2</td><td>95.8-98.5</td><td>100.0</td><td>NE</td><td>99.5</td><td>95.4-100.0</td><td>99.1</td><td>97.8-99.7</td></tr><tr><td>ImmunoCard Stat!®Campy</td><td>94.2</td><td>92.3-96.1</td><td>96.9</td><td>95.2-98.1</td><td>96.4</td><td>94.1-97.8</td><td>95.8</td><td>93.7-97.3</td></tr><tr><td colspan="5" style="text-align:center; font-weight:bold;">Negative predictive value, 95% CI</td></tr><tr><td>Culture Karmali</td><td>100.0</td><td>97.0</td><td>95.2-98.2</td><td>97.4</td><td>95.2-98.7</td><td>97.9</td><td>96.4-98.9</td></tr><tr><td>Real-time PCR</td><td>98.4</td><td>97.4-99.4</td><td>97.6</td><td>96.0-98.7</td><td>98.1</td><td>95.6-99.3</td><td>98.6</td><td>97.3-99.4</td></tr><tr><td>Ridascreen®</td><td>99.5</td><td>98.8-100.0</td><td>99.1</td><td>98.0-99.6</td><td>99.6</td><td>95.8-100.0</td><td>99.8</td><td>98.9-100.0</td></tr><tr><td>Premier®Campy</td><td>99.5</td><td>98.8-100.0</td><td>99.6</td><td>0.0-99.9</td><td>99.6</td><td>97.9-100.0</td><td>99.7</td><td>98.7-99.9</td></tr><tr><td>ImmunoCard Stat!®Campy</td><td>98.2</td><td>97.0-99.3</td><td>98.3</td><td>96.8-99.2</td><td>98.2</td><td>96.6-99.2</td><td>98.2</td><td>96.7-99.1</td></tr><tr><td colspan="5" style="text-align:center; font-weight:bold;">Positive predictive value, 95% CI</td></tr><tr><td>Culture Karmali</td><td>100.0</td><td>96.4</td><td>88.2-99.2</td><td>95.4</td><td>84.5-98.9</td><td>96.3</td><td>88.0-99.2</td></tr><tr><td>Real-time PCR</td><td>77.0</td><td>66.5-87.6</td><td>94.2</td><td>84.2-98.1</td><td>93.6</td><td>83.0-97.9</td><td>94.9</td><td>86.2-98.5</td></tr><tr><td>Ridascreen®</td><td>69.7</td><td>59.4-80.1</td><td>86.9</td><td>77.0-93.3</td><td>86.5</td><td>75.0-92.8</td><td>84.6</td><td>74.4-91.8</td></tr><tr><td>Premier®Campy</td><td>76.8</td><td>66.9-86.8</td><td>100.0</td><td>9.1-100.0</td><td>95.8</td><td>69.4-99.8</td><td>92.4</td><td>83.1-97.3</td></tr><tr><td>ImmunoCard Stat!®Campy</td><td>58.2</td><td>47.4-69.1</td><td>78.0</td><td>67.9-85.9</td><td>74.4</td><td>59.9-84.5</td><td>70.7</td><td>59.4-79.9</td></tr><tr><td>P</td><td>9.0</td><td>6.9-11.5</td><td>11.4</td><td>9.2-14.2</td><td>10.9</td><td>8.3-14.1</td><td>10.5</td><td>8.4-13.3</td></tr><tr><td>σ</td><td></td><td></td><td></td><td></td><td>0.9</td><td>0.5-1.2</td><td>1.7</td><td>1.0-2.4</td></tr></tbody></table>

95% CI, two-sided 95% confidence interval.

LCM CD, latent class model with a residual dependence common to all tests; LCM CI, latent class model under conditional independence; LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests.

NE, not estimated because of estimate on the boundary.

p, prevalence of campylobacter infection.

σ, random effect.

standard underestimated the positive predictive value of index tests from 12% to 18%.

Classically, LCMs rely on the assumption of conditional independence between tests. This hypothesis is implausible in many clinical situations but rarely evaluated in the literature while its violation may induce biased estimations of diagnostic accuracy [14]. It is, therefore, crucial to assess different structures of residual correlation and rely on the technical and biological mechanisms of index tests for the choice of the structures [38]. We proposed, for instance, a specific dependence between the three immunoenzymatic tests because of their common characteristics to detect campylobacter antigens. Assessing LCM models and their assumptions is not straightforward. We used different goodness-of-fit statistics and local residual dependence checking (pairwise graphs and testing) methods that were proposed in the literature. We showed that conclusions could vary according to the method, confirming the need to perform different checks

in order to obtain a body of evidence on the adequacy of the model and ensure the credibility of the results. Indeed, all the tests rejected the adequacy of conditional independence and common dependence LCMs but while the global tests did not reject the adequacy of the specific dependence LCM, some bivariate statistics still rejected the local independence assumption at the level of 5%. One may question the power of global tests compared with the bivariate statistics. Yet, with our sample size, global tests using empirical distributions showed high capabilities to detect violation of conditional independence assumption with statistical power ranging from 87% to 95% according to a supplementary simulation study (Table 5).

In our case example, three of the five diagnostic tests were immunoenzymatic tests (two ELISAs and one immunochromatographic test). We took into account the induced correlation with the specific random effect and this specification of LCM provided the best solution. However, the bigger weight of
<table><caption>**Table 4.** Diagnostic accuracy of medical tests according to leave-one-test-out analyses for LCM SD model, CampyLCA study, France, 2016</caption><thead><tr><th></th><th>without Ridascreen®</th><th>without Premier®Campy</th><th>without ImmunoCard Stat!®Campy</th></tr></thead><tbody><tr><td colspan="4">Sensitivity (95% CI)</td></tr><tr><td>Culture Karmali</td><td>82.2 (69.8-90.2)</td><td>81.1 (70.0-89.7)</td><td>80.8 (70.5-89.0)</td></tr><tr><td>Real-time PCR</td><td>87.3 (76.9-93.9)</td><td>88.8 (78.5-95.0)</td><td>86.8 (77.0-93.2)</td></tr><tr><td>Ridascreen®</td><td></td><td>100.0</td><td>98.7 (57.1-98.6)</td></tr><tr><td>Premier®Campy</td><td>98.0 (2.4-99.9)</td><td></td><td>97.2 (55.9-97.6)</td></tr><tr><td>ImmunoCard Stat!®Campy</td><td>85.8 (14.5-95.3)</td><td>86.1 (73.8-92.9)</td><td></td></tr><tr><td colspan="4">Specificity (95% CI)</td></tr><tr><td>Culture Karmali</td><td>99.7 (98.7-99.9)</td><td>99.5 (98.5-99.8)</td><td>99.6 (98.7-99.9)</td></tr><tr><td>Real-time PCR</td><td>99.3 (98.2-99.8)</td><td>99.5 (98.5-99.8)</td><td>99.4 (98.4-99.8)</td></tr><tr><td>Ridascreen®</td><td></td><td>98.1 (95.9-99.0)</td><td>98.2 (56.7-98.0)</td></tr><tr><td>Premier®Campy</td><td>99.1 (4.0-99.8)</td><td></td><td>99.2 (57.8-99.2)</td></tr><tr><td>ImmunoCard Stat!®Campy</td><td>95.8 (9.5-97.5)</td><td>95.9 (93.0-97.3)</td><td></td></tr><tr><td>P</td><td>10.6 (8.4-13.2)</td><td>10.5 (8.3-13.0)</td><td>10.7 (8.6-13.3)</td></tr><tr><td>σ</td><td>4.7 (-2.1 to -11.6)</td><td>1.2 (0.4-2.0)</td><td>0.0 (-10.3 to -10.6)</td></tr></tbody></table>

LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests.

95% CI, two-sided 95% confidence interval.

p, prevalence of campylobacter infection.

σ, random effect.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Diagnostic accuracy estimates (point estimate and 95% confidence interval) of campylobacter infection tests according to the LCM SD model and to culture as the reference standard, CampyLCA study, France, 2016. LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests; Ref Std: culture Karmali.</figcaption></figure>

immunoenzymatic tests could have still influenced the diagnostic accuracy results estimated with LCM SD. To assess to which extent the results were influenced, we re-estimated the model by excluding one immunoenzymatic test at a time in a 'leave-one-test out' procedure. We found out that the point estimates were not much different although wider confidence intervals were obtained. In our case study, patients were recruited in two hospitals which could have induced a residual dependence within the hospital. We evaluated a potential impact of the study hospital on our results by adding a study hospital variable in the final LCM. The introduction of the study hospital did not significantly modify the estimation of

prevalence of campylobacter (β = 0.29, p = 0.30) or diagnosis performances of the five tests (β = 0.34, p = 0.11).

Sparse data are almost inherent in diagnostic test evaluation due to the improvement of index tests and the limited sample sizes. Profiles with perfectly concordant responses ('all positive' and 'all negative') bring together almost all of the information (e.g. 90.4% in our application) while most of the discordant responses comprise no or a few observations only. As a result, asymptotic distributions of goodness-of-fit tests do not apply and empirical distributions under the null hypothesis have to be derived [23, 24, 36]. With the level of sparseness of our data,
<table><caption>**Table 5.** Statistical power of goodness-of-fit statistics (in %) using empirical distribution to detect violation of the conditional independence hypothesis when applying LCM CI model, CampyLCA study, France, 2016</caption><thead><tr><th>True model</th><th>Pearson statistics</th><th>Likelihood ratio statistics</th><th>Power divergence statistics</th></tr></thead><tbody><tr><td>LCM CD</td><td>86.6</td><td>92.4</td><td>91.8</td></tr><tr><td>LCM SD</td><td>93.4</td><td>95.0</td><td>95.0</td></tr></tbody></table>

LCM CI, latent class model under conditional independence; LCM CD, latent class model with a residual dependence common to all tests; LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests.

Statistical power is defined as the percentage of times the test concludes that observations and predictions significantly differ (at a 5% significance level) when they actually do.

we confirmed the impaired type-I error rates of the statistics using asymptotic distributions with either too conservative (type I error rates down to 0.002) or anticonservative (type I error rates up to 0.116) tests (Table 6) while by definition, the type-I error remained correct when using the empirical distribution. In our application, this translated into discordant results between asymptotic and empirical distributions, especially for the model with specific dependence. We also observed that P-values resulting from different statistics were more consistent when using empirical distributions. The use of quantitative test results may solve the sparseness problem and may allow more precise specification by including covariates or multiple random effects for example. However, this would also require some reflection about how to summarise quantitative tests results and provide useful criteria for the clinical decision.

Beyond their use for diagnostic accuracy assessment, LCMs remain complex models that require specific attention. The likelihood of LCM is often multimodal so that algorithms may converge to local maxima. To ensure convergence to the global maximum of Likelihood, required for correct inference, multiple sets of initial values can be used (we considered 100 sets in our application). In addition, the inclusion of random effects to account for the possible residual dependency induces a numerical integration in the likelihood which highly complicates the estimation process and may also pose convergence problems, in particular in SAS Proc NLMIXED in our experience. For instance, in the LCM with a common dependence estimated with this procedure, a simulation study highlighted biased estimations with unacceptable parameter coverage rates while the procedure under R provided correct inference with no bias and acceptable coverage rates (results not shown).

Because the target condition does not rely on a clinical definition, LCMs can be considered as a black box and make clinicians feel uncomfortable about what the results represent [5, 20]. Moreover, the statistical classification may not fully coincide with pre-existing knowledge of the target condition or it may even refer to a related, but different condition [6, 38]. This approach becomes meaningful when all index tests included in the model rely on an established clinical and biological background and the condition definition is not ambiguous. That explains why LCMs are very popular in the infectious field where the condition is clearly defined (i.e., presence or absence of the bacteria) and where tests directly identify the presence of the microorganism or of its antigens or DNA [38]. Note that other approaches dealing with the imperfection of the reference standard (discrepant analysis, composite reference standard) have been highly criticised in the literature for not satisfying basic requirements of the diagnostic accuracy assessment [5, 39, 40].

<table><caption>**Table 6.** Type-I error rates of goodness-of-fit statistics (in %) using asymptotic distribution for each model, CampyLCA study, France, 2016</caption><thead><tr><th>Models</th><th>Pearson statistics</th><th>Likelihood ratio statistics</th><th>Power divergence statistics</th></tr></thead><tbody><tr><td>LCM CI</td><td>11.6</td><td>0.2</td><td>2.4</td></tr><tr><td>LCM CD</td><td>6.2</td><td>1.4</td><td>1.8</td></tr><tr><td>LCM SD</td><td>8.0</td><td>0.2</td><td>3.6</td></tr></tbody></table>

LCM CI, latent class model under conditional independence; LCM CD, latent class model with a residual dependence common to all tests; LCM SD, latent class model with a residual dependence specific to the three immunoenzymatic tests.

Type I error rate is defined as the percentage of times the test concludes that observations and predictions significantly differ (at a 5% significance level) while they actually do not. The nominal value of type I error rate is 5%.

A critical limit of LCM approach lies in the number of available index tests needed to implement the models: three tests for the basic model and more for extended ones. Some authors recommended the use of at least 10 tests to ensure the distinction between different correlation structures [19]. In our context, with five tests, we did find differences between the LCMs structures hereby reported. Note that other LCMs structures, which performed worse than the specific dependence model, are not reported.

From a practical point of view, our feeling is that a major current drawback of LCM techniques for diagnostic accuracy assessment lies in the gap between recommendations that advise a search for specific correlation structure, posterior evaluation, goodness-of-fit statistics and graphs and programs available in standard software [14, 15, 19, 22–24, 36–38]. Model specifications are relatively limited, the correct convergence of models is not systematically ensured and a posteriori evaluation usually requires programming skills, which reduces the applicability of LCMs in the clinical epidemiology community.

In conclusion, the imperfection of the reference standard precludes the valid estimation of diagnostic accuracy parameters of new tests using standard methods and no good solution has been proposed so far. LCMs constitute a promising way to overcome it, on the condition that they are correctly specified and assessed [14]. However, this technique still requires substantial developments in usual software in particular to become a veritable solution for statisticians or epidemiologists involved in clinical epidemiology research.

**Acknowledgements.** The authors thank Olivier Dauwalder and Anne-Marie Freydière for providing the data from the Lyon study.

**Conflicts of interest:** None.

**References**

1. Bossuyt PM et al. (2016) STARD 2015: updated reporting guidelines for all diagnostic accuracy studies. *Annals of Translational Medicine* 4, 85.

2. Valenstein PN (1990) Evaluating diagnostic tests with imperfect standards. *American Journal of Clinical Pathology* 93, 252–258.

3. Ihorst G et al. (2007) The use of imperfect diagnostic tests had an impact on prevalence estimation. *Journal of Clinical Epidemiology* 60, 902–910.

4. Rutjes AWS et al. (2007) Evaluation of diagnostic tests when there is no gold standard. A review of methods. *Health Technology Assessment* 11, iii, ix-51.

5. Reitsma JB et al. (2009) A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. *Journal of Clinical Epidemiology* 62, 797–806.

6. Walter SD et al. (2012) Effect of dependent errors in the assessment of diagnostic or screening test accuracy when the reference standard is imperfect. *Statistics in Medicine* 31, 1129–1138.
7. Miller WC (2012) Commentary: reference-test bias in diagnostic-test evaluation a problem for epidemiologists, too. *Epidemiology* **23**, 83–85.

8. Hui SL and Walter SD (1980) Estimating the error rates of diagnostic tests. *Biometrics* **36**, 167–171.

9. Walter SD and Irwig LM (1988) Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. *Journal of Clinical Epidemiology* **41**, 923–937.

10. Qu Y, Tan M and Kutner MH (1996) Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. *Biometrics* **52**, 797–810.

11. Hui SL and Zhou XH (1998) Evaluation of diagnostic tests without gold standards. *Statistical Methods in Medical Research* **7**, 354–370.

12. Goetghebeur E et al. (2000) Diagnostic test analyses in search of their gold standard: latent class analyses with random effects. *Statistical Methods in Medical Research* **9**, 231–248.

13. Albert PS (2009) Estimating diagnostic accuracy of multiple binary tests with an imperfect reference standard. *Statistics in Medicine* **28**, 780–797.

14. van Smeden M et al. (2013) Latent class models in diagnostic studies when there is no reference standard - a systematic review. *American Journal of Epidemiology* **179**, 423–431.

15. Collins J and Huynh M (2014) Estimation of diagnostic test accuracy without full verification: a review of latent class methods. *Statistics in Medicine* **33**, 4141–4169.

16. Hadgu A, Dendukuri N and Hilden J (2005) Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test. A review of the statistical and epidemiologic issues. *Epidemiology* **16**, 604–612.

17. Torrance-Rynard VL and Walter SD (1997) Effects of dependent errors in the assessment of diagnostic test performance. *Statistics in Medicine* **16**, 2157–2175.

18. Alonzo TA and Pepe MS (1999) Using a combination of reference tests to assess the accuracy of a new diagnostic test. *Statistics in Medicine* **18**, 2987–3003.

19. Albert PS and Dodd LE (2004) A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. *Biometrics* **60**, 427–435.

20. Pepe MS and Janes H (2006) Insights into latent class analysis of diagnostic test performance. *Biostatistics (Oxford, England)* **8**, 474–484.

21. Menten J, Boelaert M and Lesaffre E (2008) Bayesian latent class models with conditionally dependent diagnostic tests: a case study. *Statistics in Medicine* **27**, 4469–4488.

22. Reiser M and Lin Y (1999) A goodness-of-fit test for the latent class model when expected frequencies are small. *Sociological Methodology* **29**, 81–111.

23. Formann AK (2003) Latent class model diagnostics—a review and some proposals. *Computational Statistics and Data Analysis* **41**, 549–559.

24. van Kollenburg GH, Mulder J and Vermunt JK (2015) Assessing model fit in latent class analysis when asymptotics do not hold. *Methodology* **11**, 65–79.

25. Dendukuri N, Hadgu A and Wang L (2009) Modeling conditional dependence between diagnostic tests: a multiple latent variable model. *Statistics in Medicine* **28**, 441–461.

26. Xu H and Craig BA (2009) A probit latent class model with general correlation structures for evaluating accuracy of diagnostic tests. *Biometrics* **65**, 1145–1155.

27. WHO. Mediacenter Campylobacter. Available at http://www.who.int/mediacentre/factsheets/fs255/en/ (Accessed 27 November 2017).

28. Bessède E et al. (2011) New methods for detection of campylobacteria in stool samples in comparison to culture. *Journal of Clinical Microbiology* **49**, 941–944.

29. Habib I et al. (2008) A Bayesian modelling framework to estimate Campylobacter prevalence and culture methods sensitivity: application to a chicken meat survey in Belgium. *Journal of Applied Microbiology* **105**, 2002–2008.

30. Regula G et al. (2005) Evaluation of an antimicrobial resistance monitoring program for campylobacter in poultry by simulation. *Preventive Veterinary Medicine* **70**, 29–43.

31. Freydiere AM et al. New methods for detection of campylobacteria in 319 paediatric stool samples: comparison of two immunochromatographic tests with culture, enzyme-linked immunosorbent assay (ELISA) and polymerase chain reaction (PCR) methods [abstract]. *Presented at the 23rd European Congress of Clinical Microbiology and Infectious Diseases, Berlin, Germany, April 27–30, 2013.*

32. Ménard A et al. (2005) Development of a real-time fluorescence resonance energy transfer PCR to identify the main pathogenic Campylobacter spp. *Clinical Microbiology and Infection* **11**, 281–287.

33. Beath KJ (2015) RandomLCA: random effects latent class analysis. R package version 1.0-6. Available at http://CRAN.R-project.org/package randomLCA.

34. Biernacki C, Celeux G and Govaert G (2003) Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models. *Computational Statistics and Data Analysis* **41**, 561–575.

35. Chen JH and Hung WL (2015) An automatic clustering algorithm for probability density functions. *Journal of Statistical Computation and Simulation* **85**, 3047–3063.

36. Van Smeden M et al. (2016) Problems in detecting misfit of latent class models in diagnostic research without a gold standard were shown. *Journal of Clinical Epidemiology* **74**, 158–166.

37. Sepúlveda R, Vicente-Villardón JL and Galindo MP (2008) The biplot as a diagnostic tool of local dependence in latent class models. A medical application. *Statistics in Medicine* **27**, 1855–1869.

38. Collins J and Albert PS (2016) Estimating diagnostic accuracy without a gold standard: a continued controversy. *Journal of Biopharmaceutical Statistics* **26**, 1078–1082.

39. Schiller I et al. (2016) Bias due to composite reference standards in diagnostic accuracy studies. *Statistics in Medicine* **35**, 1454–1470.

40. Hadgu A, Dendukuri N and Wang L (2012) Evaluation of screening tests for detecting Chlamydia trachomatis: bias associated with the patient-infected-status algorithm. *Epidemiology* **23**, 72–82.