# Detection of plant pathogens using real-time PCR

Marie Grosdidier, J. Aguayo, Benoit Marçais, R. Ioos

► To cite this version:

Marie Grosdidier, J. Aguayo, Benoit Marçais, R. Ioos. Detection of plant pathogens using real-time PCR: How reliable are late Ct values?. Plant Pathology, 2017, 66 (3), pp.359-367. 10.1111/ppa.12591 hal-01564563

HAL Id: hal-01564563

https://hal.science/hal-01564563

Submitted on 11 Dec 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Detection of plant pathogens using real-time PCR: how reliable are late $C_t$ values?

M. Grosdidier<sup>ab*</sup>, J. Aguayo<sup>b</sup>, B. Marçais<sup>a</sup> and R. Ioos<sup>b</sup>

<sup>a</sup>UMR IAM, INRA-Nancy, Université de Lorraine, IFR110 EFABA, 54280 Champenoux; and <sup>b</sup>ANSES Laboratoire de la Santé des Végétaux, Unité de Mycologie, IFR110 EFABA, 54220 Malzéville, France

Effective detection of pathogens from complex substrates is a challenging task. Molecular approaches such as real-time PCR can detect pathogens present even in low quantities. However, weak real-time PCR signals, as represented by high cycle threshold ($C_t$) values, may be questionable. Therefore, setting a reliable $C_t$ threshold to declare a positive reaction is important for specific detection. In this study, five methods were assessed for their performance in determining a $C_t$ cut-off value. These methods were based on the widely used probability of detection (POD) or receiver-operating characteristic (ROC) approaches. Two important forest pathogens, *Hymenoscyphus fraxineus* and *Fusarium circinatum*, were used to set up three experimental frameworks that combined two types of substrates (seed lots and spore traps) and different PCR machines. The ROC-based method emerged as the most complete and flexible method under various experimental conditions. It was demonstrated that the ROC method leads to a cut-off value below which late $C_t$ results can reliably be considered indicative of positive test results. This cut-off value must be determined for each experimental approach used. The method based on the distribution of a previously determined set of $C_t$ values corresponding to false-positives appeared to be better adapted to detecting false-negative results, and thus useful for testing potentially invasive pathogens.

**Keywords:** probability of detection, real-time PCR, receiver-operating characteristic, sensitivity, Youden index

## Introduction

Fungal diseases have a major impact on the management of natural resources, including forestry. In plant pathology, managing a disease relies first and foremost on the identification of the pathogen involved. The detection of the pathogen(s) in diseased plant tissue has conventionally been carried out using isolation techniques without any risk of false-positives. When a specific taxon is suspected, molecular-based detection techniques have superseded isolation, because they provide reliable results within a few hours. PCR-based techniques use the total DNA from the diseased plant tissue as a template to detect the target organisms. Among these techniques, real-time PCR (qPCR) can also quantify the amount of the target species (Heid et al., 1996). During the qPCR reaction, the target DNA is amplified and detected in real time via the emission of fluorescence. After several PCR cycles, the level of fluorescence emitted in a PCR tube containing the amplified DNA target will exceed a certain baseline; this level is called the cycle threshold ($C_t$) value. Despite the high sensitivity of qPCR, the detection of low quantities of target DNA, generating high or ‘late’ $C_t$ values, is often challenging and error-

prone (Kefi et al., 2003; Tse & Capeau, 2003; Pfaffl, 2004). Although late $C_t$ values may be caused by poor amplification efficiency, in many cases, they arise from technical biases inherent to PCR (McMullen & Petter, 2014; Mehle et al., 2014) and therefore may generate spurious (false-positive) results.

To reduce the risk of false-positive results, the implementation of controls that, for example, correspond to the limit of quantification (LOQ) and limit of detection (LOD), in each PCR run may be useful (Ioos & Fourier, 2011). Nevertheless, the relevancy of these controls may be unsatisfactory in the case of very low concentrations of target DNA or in the presence of cross-reacting DNA from non-target organisms (Nutz et al., 2011; McMullen & Petter, 2014). Alternatively, the reliability of the results can be verified by checking the size of the amplicon with gel electrophoresis and/or by sequencing (Kefi et al., 2003; Pfaffl, 2004; Love et al., 2006; Smith et al., 2008). However, only very low quantities of amplified DNA are available in the case of very late $C_t$ values and the above-mentioned methods are nearly impossible to use unless the amplicon is subcloned before sequencing.

The $C_t$ values generated by qPCR can be influenced by numerous factors, such as the operator conducting the reactions, the type and brand of qPCR reagents, equipment and analysis software, and the quality of the DNA extract (e.g. presence of inhibiting compounds) (Freeman et al., 1999). In addition, the number of qPCR cycles programmed for a run will also influence the range and

<footnote>*E-mail: mgrosdidier@nancy.inra.fr</footnote>
reliability of the C<sub>t</sub> values observed. For example, a high number of qPCR cycles increases the probability of obtaining late C<sub>t</sub> values generated by non-specific ampli- cons (Pfaffl, 2004). Therefore, some protocols recom- mend limiting the number of qPCR cycles (Polz & Cavanaugh, 1998; Kanagawa, 2003). Traditionally, the total number of qPCR cycles (usually 40) set for a run is considered as the cut-off value (PCR cycle number above which any response is considered as a false-positive). Moreover, when cut-off values are explicitly considered, they are frequently determined following arbitrary rules. The simplest rule involves choosing an arbitrary cut-off value between 35 and 45 cycles (Bustin & Nolan, 2004; Bustin *et al.*, 2009; Vettraino *et al.*, 2010; McMullen & Petter, 2014). In another type of arbitrary approach, Kontanis & Reed (2006) use a cut-off value based on a minimal fluorescence level above the background fluores- cence (generally between 6 and 15 cycles). Likewise, prior determination of the lowest concentration that can be detected in a serial dilution of standards can also be used to set an experimental cut-off value (Rutledge & Cote, 2003; Uribe & Martin, 2008; Deer *et al.*, 2010; McMullen & Petter, 2014). Lastly, an empirical approach calculates the cut-off as the average plus ‘X’ times the standard deviation of the C<sub>t</sub> values generated with DNA from samples naturally or artificially contami- nated (Meijerink *et al.*, 2001; Wong & Medrano, 2005; Chandelier *et al.*, 2006; Smith & De Boer, 2009). These arbitrary methods are unsatisfactory, because they are based on theoretical, unproven assumptions.

Alternatively, sounder and statistically based approaches
for identifying the cut-off threshold have been developed.
One such approach relies on the calculation of a 95%
probability of detection (POD) (5% error risk) (Dreier
et al., 2005; Garson et al., 2005; Chandelier et al.,
2010a). Other approaches are based on the receiver-oper-
ating characteristic (ROC) curve method and on calcula-
tion of the Youden index (Martin-Davila et al., 2005;
Bounaadja, 2010; Nutz et al., 2011).

Bustin (2005) denounced a lack of solutions or stan-
dards to normalize qPCR, from the development of the
qPCR reaction to the routine analysis steps. Despite the
availability of diverse approaches to determine cut-off
values in qPCR, none enjoy a consensus. A comparison
of these different methods in different experimental set-
tings may help to solve the problem of defining a cut-off
value and minimize the occurrence of false-positives and
false-negatives. More generally, a comparative study can
help determine whether late C<sub>t</sub> values are reliable or not.

Here, five methods based on POD or ROC were
assessed with three qPCR tests targeting two forest
pathogens that cause important epidemics. *Hymenoscy-
phus fraxineus* is the cause of ash dieback in Europe,
whereas *Fusarium circinatum* causes pine pitch canker in
different regions of the world. Both fungi are mainly
spread by airborne spores, and *F. circinatum* is also
spread via infected pine seeds. Levels of contamination
on two different substrates (e.g. filter air traps or seeds)
may be very low and detection of these fungi by qPCR

in such substrates is expected to generate late C<sub>t</sub> values.
The advantages of these statistically based methods over
the arbitrary methods are highlighted.

**Materials and methods**

Set up of spore filter traps in the field

In 2014, 89 spore traps were exposed for 15 days in the south-
ern part of the Rhône Valley, France (sampling zone WGS84
NW 45.9338 4.4299; NE 45.9710 5.6148; SW 44.0781 4.7320;
SE 44.1148 5.4890), an area free of both *Hymenoscyphus* die-
back and pine pitch canker. In addition, in 2014, 106 spore
traps were placed in a *H. fraxineus*-infected area located in the
northern part of the Rhone Valley (sampling zone WGS84 NW
45.9338 4.4299; NE 45.9710 5.6148; SW 45.2400 4.5953; SE
45.2437 5.5620).

Spore traps were prepared based on a modified protocol of Sch-
weigkofler et al. (2004). Briefly, traps consisted of a 12 cm deep,
17 cm wide polystyrene block, placed 1 m above ground on a
metal rod set in the ground. On the top of the block, a 150 mm
diameter no. 1 Whatman filter was pinned and sprayed with 4×
TE buffer (40 mm Tris-HCl, 4 mM EDTA, pH 8.0) to prevent fun-
gal spore germination. A negative control filter, enclosed in a plas-
tic bag, was fixed on one side of the trap. To avoid potential cross-
contamination during removal and transport of the traps to the
laboratory for analysis, each polystyrene block and its pinned filter
were hermetically enclosed in a single plastic bag.

For analysis, the control filters and the exposed filters were
-treated according to the following protocol. Each filter was
removed from the block and transferred to a plastic bag, 20 mL
of 4× TE were poured over the filter surface, and the bag was
sealed. The filter was rubbed manually through the bag to
detach the captured spores from the filter and release them into
the buffer. The spore suspension was collected in a 50 mL Fal-
con tube through an opening made in the corner of the bag
using a sterile scalpel blade. The tubes were centrifuged for
15 min at 20 376 g and the supernatant was discarded, leaving
approximately 3 mL of suspension. After a short vortex, the sus-
pension was split into two aliquots, transferred into 1.5 mL
microtubes, and centrifuged for 5 min at 20 800 g. Half of the
supernatant was discarded, leaving *c*. 750 µL in each tube. The
contents of these tubes were pooled together in a new sterile
microtube, which was centrifuged for 5 min at 20 800 g. The
supernatant was discarded, leaving *c*. 200 µL of spore suspen-
sion, which was stored at -20 °C until DNA extraction. Total
DNA was extracted using the DNeasy Plant Mini kit (QIAGEN).
AP1 lysis buffer (400 µL), 4 µL RNase, two 3-mm and twenty
2-mm glass beads were added to the tube and the sample was
ground twice using a Tissue Lyser (Retsch) set at 30 Hz for
45 s. The DNA extraction was then conducted according to the
manufacturer's recommendations, except that the lysis buffer
incubation at 65 °C was extended to 30 min. Total DNA was
eluted in 200 µL AE buffer and kept at -20 °C. A positive
control (subcloned qPCR product in a plasmid) and a negative
control (water) were included during the DNA extraction steps
as quality controls.

Analysis of naturally contaminated filter traps

*Hymenoscyphus fraxineus* target DNA from the filter was ampli-
fied by qPCR as described below. A subsample of amplicons from
filter DNA extracts that were associated with late C<sub>t</sub> values (i.e.
$C_t > 40$) was sequenced (4/36) to check that the amplified DNA corresponded to the *H. fraxineus* ITS sequence. First, each ampli- con was inserted in a pCR4-TOPO vector (Invitrogen), which was used to transform chemically competent *Escherichia coli* TOP10 cells (Invitrogen) according to the manufacturer's instructions (TOPO TA cloning; Invitrogen). Each bacterial clone was then subcultured overnight and the plasmids were purified using a Nucleospin Plasmid kit (Macherey-Nagel). The insert was then amplified by M13-F/-R PCR. The PCR reaction volume was 20 µL and consisted of 2 µL of template DNA, 1× reaction buffer from Eurogentec's Core kit for probe assay (no ROX), 1.5 mM MgCl2, 4 × 0.2 mM dNTPs, 0.3 µM of each primer (M13-F/-R), 0.025 U µL⁻¹ Taq DNA polymerase and water. Amplification was for 35 cycles and the amplicons were visualized by agarose gel electrophoresis then Sanger-sequenced using M13-F and M13-R as sequencing primers (Eurogentec).

## Production of matrix DNA free of the two target pathogens

DNA from each of the 89 filter traps set up in the Rhone valley region free of both *Hymenoscyphus* dieback and pine pitch canker was tested by *H. fraxineus*- and *F. circinatum*-specific qPCR. For 83 out of the 89 DNA extracts, results were negative with both assays. These negative extracts were selected to prepare the target pathogen-free filter matrix. From each of the 83 total DNA extracts, 50 µL were collected and pooled to constitute a bulk solution of 4.15 mL (83 × 50 µL) of *H. fraxineus*-free and *F. circinatum*-free filter matrix DNA.

The seed matrix was prepared with 1000 *Pinus pinaster* seeds from a French stand free of *F. circinatum*. Total DNA was extracted according to Ioos *et al.* (2009a) and a total of 4 mL of *F. circinatum*-free seed matrix DNA was recovered.

## Preparation of spiked matrix DNA

Total DNA from *H. fraxineus* isolate LSVM 79 and *F. circinatum* isolate LSVM 211 was extracted with the DNeasy Plant Mini kit following the manufacturer's instructions. PCR reactions were performed with the fungal DNA using primers Cfrax F/R for *H. fraxineus* and FCir F/R for *F. circinatum* as described in Ioos *et al.* (2009a,b), respectively. PCR products were each inserted into a pCR4-TOPO plasmid (Invitrogen) following the manufacturer's instructions. Plasmids were extracted from the transformed bacteria and purified using the Nucleospin Plasmid kit (Macherey-Nagel). Molecular weight and plasmid copy number were subsequently determined. Calibrated serial 10-fold dilutions ranging from 0 to 24 × 10³ plasmid copies µL⁻¹ (pc µL⁻¹) were prepared for *H. fraxineus* using the pathogen-free filter matrix DNA as a background. Similarly, calibrated serial 10-fold dilutions ranging from 0 to 24 × 10³ pc µL⁻¹ were prepared for *F. circinatum* using pathogen-free filter DNA matrix or seed DNA matrix as backgrounds.

## Real-time PCR assays

At least two non-template controls (NTC, i.e. water) were included in every qPCR run. To avoid any bias linked to the PCR run, the assays were carried out in several qPCR runs, each run including all the different target DNA concentrations. In each qPCR run, standard samples consisting of calibrated amounts of target DNA (plasmid DNA) diluted in molecular-grade water were also included to plot a standard curve. A series of six 10-fold

dilutions was used as template (240–24 × 10⁴ target DNA copies µL⁻¹).

Each of the three background matrices (two spore trap filters and one seed extract) spiked with various target concentrations was tested in qPCR in 36 replicates using a Rotor-Gene Q thermal cycler (QIAGEN) and in another 36 replicates using a QuantStudio 6 thermal cycler (Life Technologies). DNA from filters exposed in the Rhone Valley in 2014 (infected area or disease-free area) were analysed by qPCR with three replicates for each filter and using a Rotor-Gene Q thermal cycler. A filter was considered positive for a target if at least two of the three replicates yielded a $C_t$ value.

The qPCR amplifications using hydrolysis probes were carried out according to Ioos *et al.* (2009a,b) for *F. circinatum* and *H. fraxineus*, except that 45 cycles were performed for each run. The fluorescence threshold was set to 0.02 on the Rotor-Gene Q cycler for both species, whereas it was set to 100 for *F. circinatum* and 0.02 for *H. fraxineus* qPCR tests on the QuantStudio 6 cycler.

## Statistical analysis

Statistical analyses were carried out on the qPCR results produced from the spiked matrix combinations. Based on a literature review, four methods were selected to compute a cut-off value for qPCR that offered either good sensitivity or a good compromise between sensitivity and specificity. Sensitivity was defined as the likelihood of detecting low levels of target DNA, while specificity was defined as the ability to detect only the target DNA and not non-target DNA. The cut-off values were determined using two different POD approaches based only on sensitivity: Method A (Wehling *et al.*, 2011) and Method B (Wilrich & Wilrich, 2009). A third method took into consideration mainly the specificity (Method C, Chandelier *et al.*, 2010b). Finally, the ROC method (Method D) determined the cut-off values by combining both sensitivity and specificity (Martin-Davila *et al.*, 2005; Bounaadja, 2010; Nutz *et al.*, 2011).

Method A relied on fitting the relationship between the POD and the target concentration by a linear regression. POD was computed as $xN$ with $x$ and $N$ being the number of positive samples and the total number of samples tested for each target concentration, respectively. The cut-off value corresponded to the concentration where the POD equalled 95%, and its value was computed from the qPCR standard curve (i.e. $C_t$ value = a\*log(concentration) + b).

Method B computed a 95% detection limit, $D$, as follows: Detection limit $D = -\frac{\ln(1-p)}{\Lambda_0+F}$ where $\Lambda_0$ is the sampling size (PCR reaction volume in µL), $p$ is the POD taken as 0.95, and $F$ is the matrix effect (medium containing the target DNA); $F$ was computed from the formula $\sum_{j=1}^{q} \frac{y_j+d_j}{\exp(\Lambda_0+F+d_j)-1} - (nj - y_j) * d_j = 0$ where $nj$ is the number of spiked samples, $y_j$ is the number of DNA samples yielding a $C_t$ value, and $d_j$ is the level of target (number of DNA copies per PCR tube). The detection limit $D$ (in number of DNA copies µL⁻¹) was reported on the qPCR standard curve and the corresponding $C_t$ value was chosen as the cut-off value.

Method C computed the cut-off value based on the distribution of a previously known set of $C_t$ values corresponding to false-positives. The Shapiro–Wilk test was used to check the normality of the data. This involved first identifying the exponential distribution that best fitted the false-positive $C_t$ data and then estimating a cut-off for a given probability (1%). When no false-positive was detected, the cut-off was set as the number of PCR cycles programmed per run.
Method D was based on a ROC curve, which is a graphical representation of the sensitivity function of 1 - specificity (Nutz et al., 2011). For each cycle of the qPCR run, sensitivity and specificity were calculated and plotted on the graph. The sensitivity (Se) and the specificity (Sp) were calculated as follows:

$$Se = N_{TP} / (N_{TP} + N_{FN}), \text{ and}$$

$$Sp = N_{TN} / (N_{TN} + N_{FP})$$

where $N_{TP}$ is the number of true-positives; $N_{FN}$ is the number of false-negatives; $N_{TN}$ is the number of true-negatives; and $N_{FP}$ is the number of false-positives.

The cut-off value corresponded to the point on the curve where $1-Sp$, and $Se$, were closest to 0 and 1, respectively. The cut-off value was confirmed with the theoretical qPCR cycle where the Youden index $J$ (Youden, 1950) ($J = Se + Sp - 1$) was maximal.

Any DNA sample yielding a $C_t$ value later than these cut-off values, determined by the methods A to D, was considered negative.

# Results

## Analysis of naturally and artificially contaminated samples

Overall, 1170 qPCR reactions were performed on the filters exposed in 2014 in the Rhone Valley (106 traps in the *H. fraxineus*-contaminated area, 89 in the *H. fraxineus*-F. *circinatum*-free area, one exposed and one control filter per trap, three qPCR tests per filter). *Hymenoscyphus fraxineus* DNA was only detected on two filters exposed in the *H. fraxineus*-free area, with either two or three positive replicates. For four control filters, *H. fraxineus* DNA was only detected in one of the three qPCR replicates and these four filters were thus considered as negative. Thus, the rate of false-positives for the *H. fraxineus* qPCR test was 4/(3 × 195), i.e. 0.7% (95% confidence interval of 0.2–1.4%). *Hymenoscyphus fraxineus* DNA was detected in 45.6% of the qPCR assays performed with filters set up in the *H. fraxineus*-contaminated part of the Rhone Valley. DNA from 51 out of 106 filters exposed in this area yielded a $C_t$ value for at least two of the three qPCR replicates and was considered positive. These positive results showed $C_t$ values ranging from 31 to 45 (Fig. 1). With a false-positive rate of 0.7%, a very low error rate of 0.007 × 0.007 × 0.993, i.e. 0.005% can be expected with the three-qPCR-replicates decision rule.

Amplicons generated by qPCR amplification of DNA extracted from naturally *H. fraxineus*-contaminated filters were sequenced for four randomly chosen samples, with $C_t$ values ranging from 40.0 to 44.3. All the sequences showed 100% identity with the *H. fraxineus* qPCR target region (GenBank accession FJ429376) (Ioos et al., 2009b).

In all qPCR experiments with spiked matrices, and regardless of the thermal cycler used, negative controls (NTC or non-spiked matrices) yielded no $C_t$ value, confirming that, in this study, the filters and seeds used as matrices were free of *F. circinatum* and *H. fraxineus* (Fig. 2).

<figure><img src="image_2.png" /><figcaption>Figure 1 Boxplot of the $C_t$ values from the three qPCR replicates of *Hymenoscyphus fraxineus* targets on filter traps set up in the Rhone Valley in 2014 according to the area (infected or disease-free) and the filter type (sample or control). A $C_t$ of 0 means that no $C_t$ was reached and the target was not detected.</figcaption></figure>

For each thermal cycler, and for each matrix\*target combination, a standard curve ($y = ax + b$) was constructed based on the $C_t$ values generated with the standards diluted in water. The values corresponding to the slope (a) and the intercept (b) are reported in Figure 3.

## Determination of the cut-off value using four different statistical methods

For each thermal cycler and each matrix\*target combination, the target DNA concentrations or the detection limit corresponding to 95% POD produced a cut-off value using Methods A and B (Tables 1 & 2). Variations in the cut-off values were observed according to the matrix\*target combination. Moreover, the cut-off values differed significantly between the thermal cyclers.

The nature of the matrix analysed (seeds or filters) accounted for most of the variation in the cut-off values in Method B (Table 2). F values (i.e. the matrix effect) lower than 1 indicated POD values in the matrix lower than that of the ideal measurement method. The matrix effects of all the combinations were very limited (within 0.01–0.03) and differed according to the thermal cycler and the matrix used in the analyses (filter or seeds). Regardless of the thermal cycler, the matrix effect for seeds was higher than that for filters.

The $C_t$ values generated for each matrix\*thermal cycler\*target combination with spiked and non-spiked DNA samples followed a normal distribution according to the Shapiro–Wilk test ($P > 0.05$), as required for Method C. Because $C_t$ values were only observed with spiked samples and non-spiked samples never yielded any amplification, there were no false-positive results. Therefore, the specificity of each test was always equal to 100%. According to Chandelier et al. (2010b), the cut-off value is determined with 1% error-rate based on the false-positive distribution. In the conditions of field experiments in the present study, an error rate of 0.7% was used, i.e. very close to 1%. Thus, the cut-off values
<figure><img src="image_3.png" /><figcaption>**Figure 2** Detection of known concentrations of *Fusarium circinatum* and *Hymenoscyphus fraxinus* DNA in spiked seed or filter DNA matrices by qPCR. The distributions of positive results (C~t~ < 45) are shown according to DNA concentration of the target species, matrices and thermal cyclers.
*Non-spiked DNA matrix.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Figure 3** Standard curves of *Fusarium circinatum* and *Hymenoscyphus fraxinus* DNA in spiked seed or filter DNA matrices measured by qPCR using thermocyclers Rotor-Gene Q or QuantStudio 6. Values of standard curve linear regression settings (y = ax + b) for the various target*matrix*thermal cycler combinations are shown, where y is the C~t~ value and x is the log of the DNA concentration (copy number µL⁻¹).</figcaption></figure>

corresponded to the total number of PCR cycles (i.e. 45) and the probability of obtaining false-positive C~t~ values lower than 45 was nil.

The sensitivity of each test could be calculated for each qPCR cycle. Given that the specificity of the test was always 100%, the final graph of the ROC curve plotting the sensitivity function of (1-specificity) was a

straight vertical line, and according to the Youden index and the ROC Method D, the last cycle of the PCR run (i.e. 45) corresponded to the cut-off value.

In addition, the ROC Method D was improved on by plotting the frequencies of false-negative results as a function of the qPCR cycle (Method E). A decreasing curve ending in a plateau was observed for each
<table><caption>**Table 1** Target DNA concentrations and deduced cut-off values following the probability of detection-based Method A at 95% certainty (Wehling *et al.*, 2011)</caption><thead><tr><th rowspan="3">Target*matrix</th><th colspan="4">Thermal cycler</th></tr><tr><th colspan="2">Rotor-Gene Q</th><th colspan="2">QuantStudio 6</th></tr><tr><th>DNA<br>(copies<br>µL<sup>-1</sup>)</th><th>Cut-off<br>value</th><th>DNA<br>(copies<br>µL<sup>-1</sup>)</th><th>Cut-off<br>value</th></tr></thead><tbody><tr><td>Hymenoscyphus<br>fraxineus*filter</td><td>29.90</td><td>35.52</td><td>30.35</td><td>38.09</td></tr><tr><td>Fusarium<br>circinatum*filter</td><td>21.78</td><td>30.89</td><td>21.57</td><td>35.44</td></tr><tr><td>F. circinatum*seeds</td><td>20.75</td><td>32.89</td><td>20.50</td><td>35.96</td></tr></tbody></table>

<table><caption>**Table 2** Matrix effect (F), detection limit (d) and cut-off value (PCR cycle number above which any response is considered as a false positive) during qPCR, obtained following the POD-based Method B (Wilrich & Wilrich, 2009)</caption><thead><tr><th rowspan="3">Target*matrix</th><th colspan="6">Thermal cycler</th></tr><tr><th colspan="3">Rotor-Gene Q</th><th colspan="3">QuantStudio 6</th></tr><tr><th>F</th><th>d</th><th>Cut-off value</th><th>F</th><th>d</th><th>Cut-off value</th></tr></thead><tbody><tr><td>Hymenoscyphus<br>fraxineus*filter</td><td>0.014</td><td>10.6</td><td>37.3</td><td>0.013</td><td>11.9</td><td>39.8</td></tr><tr><td>Fusarium<br>circinatum*filter</td><td>0.014</td><td>12.4</td><td>34.7</td><td>0.012</td><td>10.9</td><td>39.4</td></tr><tr><td>F. circinatum*seeds</td><td>0.022</td><td>6.8</td><td>37.3</td><td>0.025</td><td>5.9</td><td>41.0</td></tr></tbody></table>

POD, probability of detection.

matrix*thermal cycler*target combination (Fig. 4). This plateau corresponded to the minimal frequency of false-negatives at the end of the qPCR run. Therefore, the earliest qPCR cycle in the plateau was set as the cut-off value. Following this rationale, the cut-off values observed on the graphs for the Rotor-Gene Q thermal cycler were 38, 38 and 40 for *F. circinatum* on seeds, *F. circinatum* on filters and *H. fraxineus* on filters, respectively. For the QuantStudio 6, no obvious plateau was observed before 45 cycles, therefore, a common cut-off value of 45 was applied irrespective of the pathogen*matrix combination. Nevertheless, a plateau was reached when the number of qPCR cycles was extended to 55 for the *H. fraxineus*filter combination (data not shown).

## Effect of matrix and thermal cyclers on cut-off values

The effect of the qPCR thermal cycler on sensitivity is shown in Figure 2. For the lower target DNA concentrations (i.e. 0.48 and 4.8 target copies per PCR tube), significantly more positive results were obtained using the QuantStudio 6 than with the Rotor-Gene Q (F = 329.7, P < 0.01, d.f. = 15). Overall, the cut-off values determined following the four different approaches varied across the types of thermal cycler and matrix*pathogen

combinations (Fig. 5). However, for the *F. circinatum*thermal cycler combinations, the cut-off values were not different between the two matrices. Conversely, for a given pathogen*matrix*thermal cycler combination, large differences of cut-off values (up to 15 cycles) were sometimes observed between the methods chosen (Fig. 5).

## Discussion

The absence of false-positive results is of paramount importance for the detection of quarantine organisms, for which there is nil tolerance. False-positives lead to unjustified and unacceptable economical losses. In addition, epidemiological studies also require reliable data to model the spatial dispersal of pathogenic airborne fungi, such as *F. circinatum* and *H. fraxineus*, especially near the disease front where inoculum levels may be very low. When the target organisms are detected by qPCR, reliable results are underpinned by a preliminary determination of reliable cut-off values. Late $C_t$ values (>40) were observed when testing DNA from some of the filter traps naturally contaminated by *H. fraxineus* in the Rhone Valley. The amount of DNA was too low to allow direct sequencing, and prior cloning of the amplicon was necessary for sequencing. During this study, cloning and analyses of amplicon sequences for some samples experimentally confirmed the reliability of the late $C_t$ values generated with DNA extracts from filters. All the qPCR amplicons were proven to originate from *H. fraxineus* DNA, thus confirming the high specificity of the test. In general, a comprehensive set of validation data should always be made publicly available for a test, so as to anticipate the rate of false-positive results due to cross-amplification of non-target species. For the present *H. fraxineus* test, the false-positive rate was determined to be between 0.2% and 1.4%. In turn, the determination of this false-positive rate made it possible to calculate a reliable cut-off value. From a practical point of view, this cloning/sequencing procedure cannot be used each time a late $C_t$ value is observed. Instead, the reliability of the late $C_t$ value has to be determined using a relevant statistical analysis.

The interpretation of the qPCR results differed with the method used and this discrepancy can have economic consequences, especially in quarantine contexts. For example, with the qPCR results on naturally contaminated filters, only 16% of filter-trap DNA extracts yielding a $C_t$ value were considered as positives with Method A, 35% with Method B, 55% with the Method E and 100% with Methods C and D. Thus, positive results can be overlooked, depending on the statistical method chosen. Therefore, an appropriate method must be chosen depending on the objective of the study.

No $C_t$ value was observed with any of the 36 qPCR replicate analyses for the non-spiked DNA extracts. However, the absence of false-positives in this spiking experiment is probably caused by limited sampling power. The false-positive rate computed from the control filters in the Rhone Valley survey of aerial spore inocula
<figure><img src="image_5.png" /></figure>

**Figure 4** Detection of false-positive and false-negative results during qPCR of samples of *Fusarium circinatum* and *Hymenoscyphus fraxineus* DNA in spiked seed or filter DNA matrices measured by qPCR using thermocyclers Rotor-Gene Q or QuantStudio 6. False-positive and false-negative values are shown according to the number of qPCR cycles for each target*matrix*thermal cycler combination.

<figure><img src="image_6.png" /></figure>

**Figure 5** Cut-off values (PCR cycle number above which any response is considered as a false-positive) obtained during qPCR of samples of *Fusarium circinatum* and *Hymenoscyphus fraxineus* DNA in spiked seed or filter DNA matrices, using thermocyclers Rotor-Gene Q or QuantStudio 6. Values determined using five different methods (A–E) are shown according to target species, matrix and thermal cycler. POD, probability of detection approach; ROC, receiver-operator characteristic approach.

was 0.7%. This rate of false-positives was taken into account by replicating the qPCR three times for each fil- ter and considering as positive only filters with at least

two C<sub>t</sub> values out of the three qPCR replicates. Thus, the false-positive rate obtained was 0.015%. This rate is based on the probability of having more than one
positive detection out of three PCRs with an estimated error rate of 0.7% (mean of 10 000 binomial sequences in three trials and a success rate of 0.007). The ROC-based Method D was tested with the field results. The qPCR replicate results obtained with 51 positive filters exposed in the contaminated area were used to compute sensitivity, whereas the control filters were used to compute specificity. The maximal value of the Youden index gave a cut-off value of 45. An identical cut-off value was determined with Methods C and D using either field samples or laboratory-spiked samples for *H. fraxineus*. This demonstrates the usefulness of the laboratory-spiked samples to determine the cut-off value when no field data are available.

The literature describes many ways to analyse and interpret qPCR detection results of DNA targets in low concentrations, based on the calculation of a cut-off value. However, to the authors' knowledge, no comparative study has ever been carried out in a common experimental set-up. By testing multiple matrix*tar-get*thermal cycler combinations, the present study assessed five statistical approaches to determine the cut-off value in variable experimental conditions, and addressed the issue of the reliability of late $C_t$ values. The data showed that results could show large variations (up to 15 cycles) according to the statistical approach chosen. This potential variation must be taken into consideration when trying to reach a consensus for qPCR standardization. Based on these results, POD-based Methods A and B were very straightforward to use but they did not consider the frequency of false-positive results. Moreover, these methods seem overly conservative, with a low cut-off value compared to the three other methods (C, D and E). In practice, methods A and B may 'miss' samples that have a low contamination level. These POD approaches calculate cut-off values based on the $C_t$ values generated by the amplification of target DNA, and also possibly non-target DNA. In contrast, Method C uses additional reference methods to confirm the status of the false-positive results. Assessing the risk of obtaining false-positive results in the analyses must be given priority. In this respect, Method C may be especially useful when positive results are doubtful and have statutory consequences. For example, Robène et al. (2015) used Method C to determine a reliable cut-off value for the detection of the regulated *Xanthomonas axonopodis* pv. *allii* from onion seed. Method C may be better for use when false-negative results cannot be afforded from a regulatory point of view, so preventing any introduction of threatening pathogens. Lastly, ROC-based Methods D and E consider the nature of results (false-positive/ true-positive) and use all results simultaneously for the calculation of the cut-off value. Method D seemed to be more flexible and well adapted when dealing with low quantities of target DNA.

The dataset presented here showed slight differences between cut-off values generated for the different get*matrix*thermal cycler combinations, following the

same method of calculation. Therefore, the treatment of a result dataset for qPCR corresponding to a given target*matrix*thermal cycler combination requires a calculation of a specific cut-off value; a single change of target, matrix or thermal cycler will affect this cut-off value. Similarly, Meijerink et al. (2001) and Wong & Medrano (2005) stress the fact that, in qPCR, all situations are unique and require special consideration. The results of this study are also consistent with the views of Freeman et al. (1999) and McMullen & Petter (2014) who claim that each $C_t$ value yielded by qPCR and the cut-off value depend greatly on the qPCR conditions and equipment. It is also obvious that it is easier to detect a particular pathogen in some matrices than in others. Different DNA extraction methods may sometimes be used for different matrices, which can also lead to differences in the suitable cut-off value. Results from the present study showed that this matrix effect is important, with the differences in *F* values computed using Method B. Detecting *F. circinatum* in the seed matrix was easier than in the filter matrix, although the efficiency of detection was rather low in both matrices. This result suggests that the DNA extraction steps should be optimized for each matrix.

In this investigation, the $C_t$ values varied with the thermal cycler when the concentrations of the target were low. Methods C, D and E were not influenced by the specificity of the test, but were limited by the sensitivity of the qPCR equipment (ability to detect low levels of target DNA fluorescence). For Method E, the minimal quantity of target detected for a given matrix*target combination differed according to the sensitivity of the thermal cycler, and in turn influenced the cut-off value. Therefore, new qPCR tests must be optimized prior to routine application and, in particular, cross-reactivity with non-target DNA must be thoroughly assessed. Each thermal cycler equipment has its own system for measuring the fluorescence emitted in the PCR tube, and, depending on the settings, either the operator and/or the analysis software can influence the $C_t$ values. In turn, this can affect the cut-off value determined using a statistical approach. All things being equal, testing the same target concentration with different thermal cyclers led to relatively high $C_t$ values, and the later the $C_t$ values were, the higher the cut-off values were. In practice, a target will be equally detected and quantified, regardless of the equipment, but with different $C_t$ values and cut-offs. In the conditions used here, no significant difference was observed between the two thermal cyclers regarding sensitivity.

Methods C and D were found to be the most appropriate when there were no false-positive results. The analysis of the amplicon sequences associated with late $C_t$ values (>40) on the Rotor-Gene Q confirmed the presence of *H. fraxineus* target in the DNA extracted from the filters exposed in the Rhone Valley. In contrast, Method E, an improvement of Method D, determined a cut-off value that may miss true-positive samples with sometimes critical consequences. In conclusion, late $C_t$ values can be reliable providing that a reliable cut-off
value is determined using a statistical approach such as
the ROC curve or the method described by Chandelier
et al. (2010b).

Acknowledgements

The authors greatly appreciate the sampling effort of
T. Scordia and the technical support of C. Husson,
A. Chandelier and C. Jeandel.

References

Bounaadja L, 2010. *Développement d'une PCR en Temps Réel pour la Détection des Brucella et Relations avec le Genre Ochrobactrum. Le Mans, France: Université de Maine, PhD thesis.*

Bustin SA, 2005. Real-time, fluorescence-based quantitative PCR: a snapshot of current procedures and preferences. *Expert Review of Molecular Diagnostics* 5, 495–8.

Bustin SA, Nolan T, 2004. Pitfalls of quantitative real-time reverse-transcription polymerase chain reaction. *Journal of Biomolecular Techniques* 15, 155–66.

Bustin SA, Benes V, Garson JA *et al.*, 2009. The MIQE guidelines: minimum information for publication of quantitative real-time PCR experiments. *Clinical Chemistry* 55, 611–22.

Chandelier A, Ivors K, Garbelotto M, Zini J, Laurent F, Cavelier M, 2006. Validation of a real-time PCR method for the detection of *Phytophthora ramorum*. *EPPO Bulletin* **36**, 409–14.

Chandelier A, André F, Laurent F, 2010a. Detection of *Chalara fraxinea* in common ash (*Fraxinus excelsior*) using real time PCR. *Forest Pathology* **40**, 87–95.

Chandelier A, Planchon V, Oger R, 2010b. Determination of cycle cut off in real-time PCR for the detection of regulated plant pathogens. *EPPO Bulletin* **40**, 52–8.

Deer DM, Lampel KA, González-Escalona N, 2010. A versatile internal control for use as DNA in real-time PCR and as RNA in real-time reverse transcription PCR assays. *Letters in Applied Microbiology* **50**, 366–72.

Dreier J, Störmer M, Kleesiek K, 2005. Use of bacteriophage MS2 as an internal control in viral reverse transcription-PCR assays. *Journal of Clinical Microbiology* **43**, 4551–7.

Freeman WM, Walker SJ, Vrana KE, 1999. Quantitative RT-PCR: pitfalls and potential. *BioTechniques* **26**, 112–25.

Garson JA, Grant PR, Ayliffe U, Ferns RB, Tedder RS, 2005. Real-time PCR quantification of hepatitis B virus DNA using automated sample preparation and murine cytomegalovirus internal control. *Journal of Virological Methods* **126**, 207–13.

Heid CA, Stevens J, Livak KJ, Williams PM, 1996. Real time quantitative PCR. *Genome Research* **6**, 986–94.

Ioos R, Fourrier C, 2011. Validation and accreditation of a duplex real-time PCR test for reliable *in planta* detection of *Chalara fraxinea*. *EPPO Bulletin* **41**, 21–6.

Ioos R, Fourrier C, Iancu G, Gordon TR, 2009a. Sensitive detection of *Fusarium circinatum* in pine seed by combining an enrichment procedure with a real-time polymerase chain reaction using dual-labeled probe chemistry. *Phytopathology* **99**, 582–90.

Ioos R, Kowalski T, Husson C, Holdenrieder O, 2009b. Rapid *in planta* detection of *Chalara fraxinea* by a real-time PCR assay using a dual-labelled probe. *European Journal of Plant Pathology* **125**, 329–35.

Kanagawa T, 2003. Bias and artifacts in multitemplate polymerase chain reactions (PCR). *Journal of Bioscience and Bioengineering* **96**, 317–23.

Kefi R, Mafart B, Spadoni JL, Stevanovitch A, Béraud-Colomb É, 2003. Application de la technique de PCR en temps réel à l'étude de l'ADN ancien. *Comptes Rendus Palevol* **2**, 125–32.

Kontanis EJ, Reed FA, 2006. Evaluation of real-time PCR amplification efficiencies to detect PCR inhibitors. *Journal of Forensic Sciences* **51**, 795–804.

Love JL, Scholes P, Gilpin B, Savill M, Lin S, Samuel L, 2006. Evaluation of uncertainty in quantitative real-time PCR. *Journal of Microbiological Methods* **67**, 349–56.

Martin-Davila P, Fortun J, Gutierrez C *et al.*, 2005. Analysis of a quantitative PCR assay for CMV infection in liver transplant recipients: an intent to find the optimal cut-off value. *Journal of Clinical Virology* **33**, 138–44.

McMullen M, Petter F, 2014. Report of the EPPO Workshop on setting Ct cut-off values for real-time PCR. *EPPO Bulletin* **44**, 165.

Mehle N, Dreo T, Jeffries C, Ravnikar M, 2014. Descriptive assessment of uncertainties of qualitative real-time PCR for detection of plant pathogens and quality performance monitoring. *EPPO Bulletin* **44**, 502–9.

Meijerink J, Mandigers C, van de Licht L, Tönnissen E, Goodsaid F, Raemaekers J, 2001. A novel method to compensate for different amplification efficiencies between patient DNA samples in quantitative real-time PCR. *Journal of Molecular Diagnostics* **3**, 55–61.

Nutz S, Döll K, Karlovsky P, 2011. Determination of the LOQ in real-time PCR by receiver operating characteristic curve analysis: application to qPCR assays for *Fusarium verticillioides* and *F. proliferatum*. *Analytical and Bioanalytical Chemistry* **401**, 717–26.

Pfaffl MW, 2004. Quantification strategies in real-time PCR. In: Bustin SA, ed. *A-Z of Quantitative PCR*. La Jolla, CA, USA: International University Line, 87–112.

Polz MF, Cavanaugh CM, 1998. Bias in template-to-product ratios in multitemplate PCR. *Applied and Environmental Microbiology* **64**, 3724–30.

Robène I, Perret M, Jouen *E* *et al.*, 2015. Development and validation of a real-time quantitative PCR assay to detect *Xanthomonas axonopodis* pv. *allii* from onion seed. *Journal of Microbiological Methods* **114**, 78–86.

Rutledge RG, Cote C, 2003. Mathematics of quantitative kinetic PCR and the application of standard curves. *Nucleic Acids Research* **31**, e93.

Schweigkofler W, O'Donnell K, Garbelotto M, 2004. Detection and quantification of airborne conidia of *Fusarium circinatum*, the causal agent of pine pitch canker, from two California sites by using a real-time PCR approach combined with a simple spore trapping method. *Applied and Environmental Microbiology* **70**, 3512–20.

Smith DS, De Boer SH, 2009. Implementation of an artificial reaction control in a TaqMan method for PCR detection of *Ralstonia solanacearum* race 3 biovar 2. *European Journal of Plant Pathology* **124**, 405–12.

Smith DS, De Boer SH, Gourley J, 2008. An internal reaction control for routine detection of *Clavibacter michiganensis* subsp. *sepedonicus* using a real-time TaqMan PCR-based assay. *Plant Disease* **92**, 684–93.

Tse C, Capeau J, 2003. Quantification des acides nucléiques par PCR quantitative en temps réel. *Annales de Biologie Clinique* **61**, 279–93.

Uribe P, Martin FN, 2008. Using sigmoidal curve-fitting in a real-time PCR detection assay to determine detection thresholds. General Technical Report PSW-GTR-214. [http://www.fs.fed.us/psw/publications/documents/psw_gtr214/psw_gtr214_075-081_uribe.pdf]. Accessed 28 July 2016.

Vettraino AM, Sukno S, Vannini A, Garbelotto M, 2010. Diagnostic sensitivity and specificity of different methods used by two laboratories for the detection of *Phytophthora ramorum* on multiple natural hosts. *Plant Pathology* **59**, 289–300.

Wehling P, LaBudde RA, Brunelle SL, Nelson MT, 2011. Probability of detection (POD) as a statistical model for the validation of qualitative methods. *Journal of AOAC International* **94**, 335–47.

Wilrich C, Wilrich P-T, 2009. Estimation of the POD function and the LOD of a qualitative microbiological measurement method. *Journal of AOAC International* **92**, 1763–72.

Wong ML, Medrano JF, 2005. Real-time PCR for mRNA quantitation. *BioTechniques* **39**, 75.

Youden WJ, 1950. Index for rating diagnostic tests. *Cancer* **3**, 32–5.