<figure><img src="image_1.png" /></figure>

HAL
open science

# Assessment of reproducibility of a VP7 Blocking ELISA diagnostic test for african horse sickness

Manuel Duran-Ferrer, Montserrat Agüero, Stéphan Zientara, Cécile Beck, Sylvie Lecollinet, Corinne Sailleau, Shirley Smith, Christiaan Potgieter, Paloma Rueda, Patricia Sastre, et al.

► To cite this version:

Manuel Duran-Ferrer, Montserrat Agüero, Stéphan Zientara, Cécile Beck, Sylvie Lecollinet, et al.,

Assessment of reproducibility of a VP7 Blocking ELISA diagnostic test for african horse sickness.

Transboundary and emerging diseases, 2019, 66 (1), pp.83-90. 10.1111/tbed.12968 . hal-02619842

HAL Id: hal-02619842

https://hal.inrae.fr/hal-02619842

Submitted on 25 May 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
ORIGINAL ARTICLE

WILEY

<figure><img src="image_3.png" /></figure>

# Assessment of reproducibility of a VP7 Blocking ELISA diagnostic test for African horse sickness

Manuel Durán-Ferrer<fnref n="1" /> | Montserrat Agüero<fnref n="1" /> | Stephan Zientara<fnref n="2" /> | Cécile Beck<fnref n="2" /> |

Sylvie Lecollinet<fnref n="2" /> | Corinne Sailleau<fnref n="2" /> | Shirley Smith<fnref n="3" /> | Christiaan Potgieter<fnref n="3" />,<fnref n="4" /> |

Paloma Rueda<fnref n="5" /> | Patricia Sastre<fnref n="5" /> | Federica Monaco<fnref n="6" /> | Ruben Villalba<fnref n="1" /> | Cristina Tena-

Tomás<fnref n="1" /> | Carrie Batten<fnref n="7" /> | Lorraine Frost<fnref n="7" /> | John Flannery<fnref n="7" /> | Simon Gubbins<fnref n="7" /> ID |

Baratang A. Lubisi<fnref n="8" /> | José Manuel Sánchez-Vizcaíno<fnref n="9" /> | Michelle Emery<fnref n="10" /> |

Tracy Sturgill<fnref n="10" /> | Eileen Ostlund<fnref n="10" /> | Javier Castillo-Olivares<fnref n="7" /> ID

**Correspondence**

Javier Castillo-Olivares
Department of Veterinary Medicine,
University of Cambridge, Madingley Road,
Cambridge, CB3 0ES, UK
Email: fjc37@cam.ac.uk

**Present address**

Javier Castillo-Olivares, Department of Veterinary Medicine, University of Cambridge, Madingley Road, Cambridge, CB3 0ES, UK.

**Funding information**

World Organisation of Animal Health (OIE), Grant/Award Number: AD/SR/2015/1885; Biotechnology and Biological Sciences Research Council, Grant/Award Number: BBS/E/I/00002536, BBS/E/I/00007036, BBS/E/I/00007037

**Abstract**

The laboratory diagnosis of African horse sickness (AHS) is important for: (a) demonstrating freedom from infection in a population, animals or products for trade (b) assessing the efficiency of eradication policies; (c) laboratory confirmation of clinical diagnosis; (d) estimating the prevalence of AHS infection; and (e) assessing postvaccination immune status of individual animals or populations. Although serological techniques play a secondary role in the confirmation of clinical cases, their use is very important for all the other purposes due to their high throughput, ease of use and good cost-benefit ratio. The main objective of this study was to support the validation of AHS VP7 Blocking ELISA up to the Stage 3 of the World Animal Health Organization (OIE) assay validation pathway. To achieve this, a collaborative ring trial, which included all OIE Reference Laboratories and other AHS-specialist diagnostic centres, was conducted in order to assess the diagnostic performance characteristics of the VP7 Blocking ELISA. In this trial, a panel of sera of different epidemiological origin and infection status was used. Through this comprehensive evaluation we can conclude that the VP7 Blocking ELISA satisfies the OIE requirements of reproducibility. The VP7 Blocking ELISA, in its commercial version is ready to enter Stage 4 of the validation pathway (Programme Implementation). Specifically, this will require testing the diagnostic performance of the assay using contemporary serum samples collected during control campaigns in endemic countries.

**KEYWORDS**

African horse sickness, antibodies, diagnostic sensitivity, diagnostic specificity, ELISA, performance characteristics, reproducibility, ring trial

<footnote n="1">¹Laboratorio Central de Veterinaria, LCV, Madrid, Spain</footnote>

<footnote n="2">²UMR, Laboratoire de Santé Animale, ANSES, INRA, ENVA, Maisons-Alfort, France</footnote>

<footnote n="3">³Deltamune, Pretoria, South Africa</footnote>

<footnote n="4">⁴Department of Biochemistry, Centre for Human Metabolomics, North-West University, Potchefstroom, South Africa</footnote>

<footnote n="5">⁵INGENASA, Madrid, Spain</footnote>

<footnote n="6">⁶IZS dell'Abruzzo e Molise, Teramo, Italy</footnote>

<footnote n="7">⁷The Pirbright Institute, Pirbright, UK</footnote>

<footnote n="8">⁸OVI, Pretoria, Gauteng, South Africa</footnote>

<footnote n="9">⁹Universidad Complutense of Madrid, Madrid, Spain</footnote>

<footnote n="10">¹⁰USDA, Ames, Iowa</footnote>
Corrections added on 24 December 2018, after first online publication: a spelling error in author Potgieter's name has been corrected and an affiliation added; authors Cécile Beck, Sylvie Lecollinet and Corinne Sailleau have been added.

# 1 | INTRODUCTION

African horse sickness (AHS) is an infectious but noncontagious viral disease affecting all species of *Equidae* and is caused by an *Orbivirus* of the family *Reoviridae* and characterized by respiratory and circulatory syndromes. Nine different serotypes of AHS virus (AHSV) have been identified. AHS is transmitted by species of *Culicoides* spp. biting midges. All serotypes of AHS occur in eastern and southern Africa, from where they occasionally spread into countries surrounding the Mediterranean and on occasions reaching India and Pakistan (Mellor & Hamblin, 2004; Zientara, Weyer, & Lecollinet, 2015). Due to its high mortality rate, which can exceed 90% in susceptible populations, the huge economic losses that it causes in endemic countries and when it spills over into nonendemic territories, and its negative impact on international trade of equids, AHS is an OIE (World Organization for Animal Health)-listed disease. Furthermore, its control is considered a priority and AHS is one of the six animal diseases currently included by the OIE in the procedure for official recognition of a country's disease-free status (OIE, 2017a).

Clinical signs are characteristic but not pathognomonic of the disease, especially in endemic countries where the infection can cause a milder clinical form of the disease. Confirmatory diagnosis in the laboratory is most often achieved through virus detection techniques in blood samples or in postmortem tissue samples when AHSV-infected horses die before specific antibodies can become detectable in clinical serum specimens (Mellor & Hamblin, 2004; Zientara et al., 2015). Currently, polymerase chain reaction (PCR) methods are the first choice for diagnosis (Agüero et al., 2008; Guthrie et al., 2013). Once the disease has been confirmed, virus characterization can be attempted by PCR typing techniques (Bachanek-Bankowska et al., 2014; Weyer et al., 2015) or by the classical pathway of virus isolation in cell cultures followed by virus neutralization testing (OIE, 2017b).

Horses that survive natural infection develop antibodies against the infecting serotype of AHSV within 8–12 days postinfection and, consequently, serology is the most practical approach for surveillance in nonendemic countries, determining disease freedom in a population or for import-export testing prior to international trade. In endemic countries, where many horses are routinely vaccinated and/or experience reinfections, the use of serology for diagnosis, import/export or surveillance, requires the analysis of serum samples collected sequentially over a period of time to determine whether an increase or decrease in antibody titres are indicative of AHSV infections have occurred in the animals involved. Several laboratory procedures have been used for such purposes, including complement

fixation (McIntosh, 1956; OIE, 2017b), agar gel immunodiffusion (Verwoerd, Huismans, & Erasmus, 1979), immunofluorescence or virus neutralization tests (Hamblin et al., 1991; Hopkins, Hazarati, & Ozawa, 1966; Mellor & Hamblin, 2004; OIE, 2017b). However, ELISA methods have prevailed and are currently considered the first choice because they are rapid, easy to standardize, easy to perform and have high-throughput capacity (Hamblin, Graham, Anderson, & Crowther, 1990; Laviada, Roy, & Sánchez-Vizcaíno, 1992; Maree & Paweska, 2005; OIE, 2017b; Rubio et al., 1998; Wade-Evans, Wolhouse, O'Hara, & Hamblin, 1993).

The VP7 protein AHSV is the outer core protein of the capsid and a group-specific antigen (Zientara et al., 2015; Maree & Paweska, 2005). The VP7 Blocking ELISA detects specific antibodies against the antigenically conserved VP7 protein. The test is currently commercially available INGEZIM AHSV COMPAC PLUS 2.0 (Ingenasa, Madrid, Spain). An indirect ELISA detecting VP7 protein is also available (Maree & Paweska, 2005). Both tests are recommended by OIE for serological diagnosis of AHS (OIE, 2017b) as well as for the implementation of checks prior to international movement of equids. Furthermore, the VP7 Blocking ELISA is one of the serological tests prescribed for the control of movement and importations to the European Union, according to requirements of Directive 2009/156/EC, as last amended (European Union, 2009).

To achieve harmonized criteria for validation of diagnostic tests, the OIE has proposed an assay validation pathway (OIE, 2017c) with the aim of declaring that a test is “fit for purpose”. This statement indicates the test is suitable and reliable for: (a) demonstrating freedom from infection in a population, animals or products for trade; (b) assessing the efficiency of eradication policies; (c) laboratory confirmation of clinical diagnosis; (d) estimating the prevalence of AHS infection; and (e) assessing postvaccination immune status of individual animals or populations. The validation pathway consists of four stages: Stage 1, determination of analytical characteristics (analytical sensitivity, specificity and repeatability estimates); Stage 2, determination of diagnostic characteristics (diagnostic sensitivity and specificity estimates); Stage 3, determination of reproducibility estimates; and Stage 4, declaration of fitness for purpose and international recognition (OIE, 2012).

The main objective of this study was to support the validation of the VP7 Blocking ELISA up to Stage 3 of the OIE validation pathway. For this, a “ring” trial, which included the participation of OIE Reference Laboratories and other specialist diagnostic laboratories, was set up to test a panel of field serum samples of different epidemiological origin and experimental sera from horses infected with AHSV and/or
vaccinated with live attenuated, inactivated or recombinant AHSV vaccines. The inter-laboratory reproducibility of the test was assessed by studying the Kappa agreement statistic (Thrusfield, 1995), Z-score index (ISO, 2005) and variation of the test's performance (sensitivity and specificity) amongst the different laboratories. Through this comprehensive evaluation, we can conclude that the VP7 Blocking ELISA assay satisfied the OIE requirements of reproducibility and maintained adequate diagnostic performance characteristics.

## 2 | MATERIALS AND METHODS

### 2.1 | Participant laboratories

The study was organized by the Laboratorio Central de Veterinaria (LCV), Algete, Madrid, Spain (EURL/OIE Reference Laboratory for AHS) under the coordination of The Pirbright Institute, Pirbright, UK (OIE Reference Laboratory for AHS). Other participant institutes in the project were: (a) Agence nationale de sécurité sanitaire de l'alimentation, de l'environnement et du travail (ANSES), Maisons-Alfort, France; (b) Onderstepoort Veterinary Institute (OVI) (OIE Reference Center for AHS), South Africa; (c) Deltamune, Centurion, South Africa; (d) VISAVET, University Complutense of Madrid, Spain (OIE Reference Center for AHS); (e) INGENASA, Madrid, Spain; (f)

National Veterinary Services Laboratory (NVSL), USDA, Ames, Iowa, USA; and (g) Istituto Zooprofilattico Sperimentale dell'Abruzzo e Molise, Italy.

### 2.2 | Diagnostic test kits

The test kits used in the study were obtained directly from the commercial manufacturer (INGENASA, Madrid, Spain). All partners used the same serum panel (Table 1) and the same tests. All the test kits belonged to the same manufacturing batches.

The VP7 Blocking ELISA (INGEZIM AHSV COMPAC PLUS), is designed to detect specific AHSV anti-VP7 antibodies in sera from animals of any equine species, that is, horses, donkeys, zebra and their crosses. The principle of this test is the blocking of the reaction between the recombinant VP7 protein adsorbed to the ELISA plate and a peroxidase-conjugated AHS-VP7-specific monoclonal antibody (Mab). VP7-specific antibodies present in serum samples block the reaction between the antigen and the conjugated Mab resulting in a reduction in enzymatic reaction between the peroxidase and the colorimetric enzyme substrate added to the plate. The optical density readouts of the reactions were determined and results expressed as the percentage of reactivity between the conjugate and the antigen that is blocked by the serum sample.

<table><caption>TABLE 1 Composition of panel of serum sample</caption><thead><tr><th>Animal Species</th><th>Status of donor</th><th>Virus serotype</th><th>Time of collection</th><th>Expected value*</th><th>Number of samples</th></tr></thead><tbody><tr><td colspan="6">AHS noninfected</td></tr><tr><td>Various</td><td>Not previously exposed to AHSV</td><td>Na</td><td>Na</td><td>Neg (n = 41)</td><td>Mule (11); Donkey (10); Horse (20)</td></tr><tr><td colspan="6">Vaccinated and not infected with AHSV</td></tr><tr><td>Horse</td><td>Vaccinated with attenuated vaccine</td><td>S-4; S-5</td><td>152-428 dpv</td><td>Neg (n = 12); Pos (n = 42)</td><td>54</td></tr><tr><td>Horse</td><td>Vaccinated with inactivated vaccine</td><td>S-1, S-2, S-4, S-9</td><td>15-115 dpv</td><td>Pos (n = 31)</td><td>31</td></tr><tr><td>Horse</td><td>Vaccinated with MVA-VP2 vaccine</td><td>S-9</td><td>42 dpv</td><td>Neg (n = 4)</td><td>4</td></tr><tr><td>Horse</td><td>Vaccinated with subunits vaccine</td><td>Na</td><td>52 dpv</td><td>Pos (n = 1)</td><td>1</td></tr><tr><td colspan="6">Infected</td></tr><tr><td>Horse</td><td>Naturally infected</td><td>Nd</td><td>Na</td><td>Pos (n = 19)</td><td>19</td></tr><tr><td>Horse</td><td>Experimentally infected</td><td>S-2; S-9</td><td>18-28 dpi</td><td>Pos (n = 4)</td><td>4</td></tr><tr><td colspan="6">Vaccinated and infected with AHSV</td></tr><tr><td>Horse</td><td>Vaccinated with attenuated vaccine and experimentally infected</td><td>S-4; S-5; S-6</td><td>152 dpv/67dpi</td><td>Pos (n = 11)</td><td>11</td></tr><tr><td>Horse</td><td>Vaccinated with inactivated vaccine and experimentally infected</td><td>S-1, S-2, S-4, S-9 (Vac)/S-2 (Infec)</td><td>129-143 dpv; 14-28 dpi</td><td>Pos (n = 9)</td><td>9</td></tr><tr><td>Horse</td><td>Vaccinated with inactivated vaccine and experimentally infected</td><td>S-9 (Vac)/S-9 (infec)</td><td>116 dpv; 7-14 dpi</td><td>Pos (n = 8)</td><td>8</td></tr><tr><td>Horse</td><td>Vaccinated with MVA-VP2* vaccine and experimentally infected</td><td>S-9 (Vac)/S-9 (Infec)</td><td>28 dpi</td><td>Pos (n = 4)</td><td>4</td></tr><tr><td colspan="4">Total</td><td>Neg (n = 57); Pos (n = 129)</td><td>186</td></tr></tbody></table>

Notes. dpi: days postinfection; dpv: days postvaccination; infec: infection; MVA: Modified Vaccinia Ankara viruses expressing single African horse sickness virus VP2 antigens; n: number of serum samples; na: nonapplicable; Na: not available; nd: no data; Nd: not done; Neg: negative; Pos: positive; Vac: vaccination.

<footnote>*Expected value for the samples according to the original results obtained, by the VP7 Blocking ELISA, at the laboratory of origin.</footnote>
The tests were conducted as indicated by the manufacturer's instructions. Briefly, serum samples and positive and negative controls were diluted in dilution buffer (PBS; 0.35 M NaCl, 0.05% Tween 20; 0.1% Kathon) and 100 µl/well added to two duplicate wells. After 1 hr incubation at 37°C the plates were washed five times with washing buffer (PBS; 0.135 M NaCl, 0.05% Tween 20). This was followed by addition of 100 µl/well of horseradish peroxidase-labelled VP7-specific Mab (supplied by manufacturer). After an incubation period of 30 min at 37°C, the plates were washed five times in washing buffer and 100 µl/well of the chromogenic substrate ABTS were added to all wells and incubated for 10 min after which the reaction was stopped by the addition of 100 µl/well of a 2% SDS solution. The optical density at 405 nm was read and results recorded. Mean values for each pair of samples were calculated, inclusive of positive and negative controls. The blocking percentage was calculated as follows: $BP = [OD Neg Control – OD Sample] \times 100/[OD Neg Control – OD Pos Control]$. Samples showing BP values >50% were considered positive; samples with BP <45% were considered negative; samples with values between 45% and 50% were doubtful.

## 2.3 | Panel of serum samples

All participant laboratories contributed to the preparation of the sera panel by providing available samples. The panel was assembled at the LCV-Algete, and it was made out of 186 serum samples representing all possible AHS sero-epidemiological status that can be found in the field (Table 1): (a) AHS noninfected; (b) vaccinated (with either attenuated or inactivated whole virus vaccines, MVA-VP2 vaccine, or recombinant subunit vaccines) and not infected with AHSV; (c) naturally or experimentally infected with AHSV; and (d) vaccinated (with either attenuated or inactivated whole virus vaccines or MVA-VP2 vaccine) and infected with AHSV. Samples were sent by donor laboratories to LCV-Algete and then collected, aliquoted and frozen at -20°C until they were distributed to each participant laboratory, blindly coded for testing. Once results were made available, they were collated and a database was created. Laboratory codes (codes A, B, C, D, E, F, G, H and I) were not broken until the exercise was completed.

## 2.4 | Assay of samples, interpretation of results and expected values

In each laboratory, the VP7 Blocking ELISA test was conducted in duplicate for each serum sample. Samples were classified, according to the manufacturer's instructions, as positive, doubtful or negative. The criterion of maximum sensitivity was selected for the final test evaluation. Thus, doubtful results were considered as positive.

To determine the diagnostic characteristics of tests (sensitivity and specificity), a reference value was assigned to each sample tested. The reference value assigned was the result previously obtained by the VP7 Blocking ELISA in the laboratory that donated the serum.

## 2.5 | Statistical methods

The assessment of reproducibility of the assays was the main objective of the study by estimating test's sensitivity (Se) and specificity (Sp) and their associated 95% confidence level interval (95% CI) calculated over the data generated by laboratories as a whole (total observations), after rejecting outlier laboratories.

For identifying outliers, two statistical approaches were complementarily used: (a) Cohen's Kappa statistic, used to compare the level of agreement of the results for each pair of laboratories (Thrusfield, 1995); and (b) The Z-Score or standard score, which determines the level of standard deviation by which the value of an observation is below/above the mean value (ISO, 2005). Observed values above the mean have positive standard scores, while values below the mean have negative standard scores. Z-score ranges from -3.0 standard deviations (which would fall to the far left of the normal distribution curve) up to +3.0 standard deviations (which would fall to the far right of the normal distribution curve). If a Z-score is 0, it represents the score is identical to the mean score. When Z-score values for a particular laboratory were above +2.0 or below -2.0, the laboratory's accuracy was considered questionable and its results were excluded from further analysis.

# 3 | RESULTS

Tests were completed successfully by all participants and results were sent to LCV-Algete who compiled the data and distributed it to all partners for sharing information and discussion.

The performance of laboratories was assessed by computing kappa statistics (Figure 1). This revealed that Laboratory F produced highly discordant results in comparison with the rest of the laboratories. Except for laboratory F, the results show that the VP7 Blocking ELISA had good or very good reproducibility between the participant laboratories (Kappa ≥ 0.7). The Z-Scores computed from the data were consistent with the kappa statistics. Thus, the Z-score range

<figure><img src="image_4.png" /><figcaption>**FIGURE 1** VP7 Blocking ELISA: inter-laboratory reproducibility estimates by Cohen's Kappa factor statistic test [Colour figure can be viewed at wileyonlinelibrary.com]</figcaption></figure>
<table><caption>TABLE 2 Estimates of Sensitivity/Specificity of VP7 Blocking ELISA</caption><thead><tr><th colspan="10">Sensitivity estimate</th></tr><tr><th>Animal status</th><th>ExV</th><th>n</th><th>Pos</th><th>Neg</th><th>Average sensitivity</th><th>95% CI LL</th><th>95% CI UL</th><th>Min</th><th>Max</th></tr></thead><tbody><tr><td colspan="10">Vaccinated and not infected with AHSV</td></tr><tr><td>Vaccinated animals with attenuated vaccine</td><td>POS</td><td>336</td><td>317</td><td>19</td><td>94.3%</td><td>92.9%</td><td>97.7%</td><td>88.1%</td><td>100.0%</td></tr><tr><td>Vaccinated animals with inactivated vaccine</td><td>POS</td><td>248</td><td>236</td><td>12</td><td>95.2%</td><td>91.7%</td><td>97.5%</td><td>90.3%</td><td>100.0%</td></tr><tr><td>Vaccinated animals with sub-unit/MVA-VP2* vaccine</td><td>POS</td><td>8</td><td>8</td><td>0</td><td>100.0%</td><td>63.1%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Overall sensitivity</td><td>POS</td><td>592</td><td>561</td><td>31</td><td>94.8%</td><td>93.5%</td><td>97.1%</td><td>89.2%</td><td>100.0%</td></tr><tr><td colspan="10">Infected with AHSV</td></tr><tr><td>Naturally or experimentally infected with AHS</td><td>POS</td><td>184</td><td>181</td><td>3</td><td>98.4%</td><td>95.3%</td><td>99.7%</td><td>95.7%</td><td>100.0%</td></tr><tr><td colspan="10">Vaccinated and experimentally infected with AHS</td></tr><tr><td>Vaccinated with attenuated vaccine and experimentally infected</td><td>POS</td><td>88</td><td>88</td><td>0</td><td>100.0%</td><td>95.9%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Vaccinated with inactivated vaccine and experimentally infected</td><td>POS</td><td>136</td><td>136</td><td>0</td><td>100.0%</td><td>97.3%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Vaccinated with MVA-VP2* vaccine and experimentally infected</td><td>POS</td><td>32</td><td>11</td><td>21</td><td>34.4%</td><td>18.6%</td><td>53.2%</td><td>25.0%</td><td>50.0%</td></tr><tr><td>Vaccinated with all vaccine types and experimentally infected</td><td>POS</td><td>256</td><td>235</td><td>21</td><td>91.8%</td><td>85.9%</td><td>93.5%</td><td>90.6%</td><td>93.8%</td></tr><tr><td>Overall sensitivity on infected animals</td><td>POS</td><td>440</td><td>416</td><td>24</td><td>94.5%</td><td>90.8%</td><td>95.6%</td><td>92.7%</td><td>96.4%</td></tr><tr><td>Overall sensitivity estimate</td><td>POS</td><td>1032</td><td>977</td><td>55</td><td>94.7%</td><td>93.1%</td><td>96.0%</td><td>90.7%</td><td>98.4%</td></tr><tr><th colspan="10">Specificity estimate</th></tr><tr><th>Animal status</th><th>ExV</th><th>n</th><th>Pos</th><th>Neg</th><th>Average specificity</th><th>95% CI LL</th><th>95% CI UL</th><th>Min</th><th>Max</th></tr><tr><td>Naïve animals (AHSV noninfected)</td><td>NEG</td><td>328</td><td>0</td><td>328</td><td>100.0%</td><td>98.9%</td><td>100.0%</td><td>100.0%</td><td>100.0%</td></tr><tr><td>Vaccinated and not infected with AHSV</td><td>NEG</td><td>128</td><td>4</td><td>124</td><td>96.9%</td><td>92.2%</td><td>99.1%</td><td>87.5%</td><td>100.0%</td></tr><tr><td>Overall specificity estimate</td><td>NEG</td><td>456</td><td>4</td><td>452</td><td>99.1%</td><td>97.8%</td><td>99.8%</td><td>96.5%</td><td>100.0%</td></tr></tbody></table>

Note. 95% CI LL: confidence interval 95% low limit; 95% CI UL: confidence interval 95% low limit; ExV: expected value according to previous ELISA results; Max: maximum value; Min: minimum value; MVA: Modified Vaccinia Ankara viruses expressing single African horse sickness virus VP2 antigens; n: number of observations.

for laboratory F was -2.7 to 1.1, whereas the Z-score range for the rest of laboratories was -1.4 to 1.1. For calculations of average sensitivity, data from laboratory F were excluded.

The sensitivity of the test in naturally or experimentally infected animals (not vaccinated) was 98.4% (181/184 [95% CI: 95.3-99.7]). If individuals were immunized prior to infection, sensitivity varied depending on the vaccine used. For animals previously vaccinated with vaccines containing VP7 antigen, the sensitivity was very high, both for individuals that received attenuated vaccines (Se: 100% [88/88; 95% CI: 95.9-100]) and for horses vaccinated with inactivated vaccines (Se: 100% [136/136; 95% CI: 97.3-100]). But when animals were not previously exposed to VP7 antigen (as with Modified Vaccinia Ankara [MVA] viruses expressing single AHSV VP2 antigens), detection of VP7 antibodies after infection was poor 34.4% (11/32 [95% CI: 18.6-53.2]). In vaccinated, noninfected animals, the sensitivity was 94.8% (561/592 [95% CI: 93.5-97.1]) with no major differences between types of vaccine used. The overall sensitivity (not vaccinated and infected plus vaccinated infected animals) of the test was 94.7% (977/1032 [95% CI: 93.1-96.0]).

The specificity of the VP7 Blocking ELISA was 100% (328/328; [95% CI: 98.9-100]) in naïve animals (AHS noninfected animals), but lower in vaccinated and not infected individuals, reaching a value of 96.9% (124/128 [95% CI: 92.2-99.1]). The overall specificity was very high 99.1% (452/456 [95% CI: 97.8-99.8]; Table 2).

4 | DISCUSSION

The clinical outcome of AHS in horses is characterized by a rapid evolution of the disease inducing syndromes that range from subacute to per-acute form. Mortality rates typically exceed 50%, but can reach 90% depending on clinical form of the disease and the immunological status of the animals (Mellor & Hamblin, 2004). The infected horse often dies shortly after infection, and thus there is limited opportunity to mount a strong antibody response. The use of attenuated vaccines in endemic areas helps prevent horses from succumbing to AHS, but makes it difficult to determine whether AHS-specific antibodies are derived from an immune response to the vaccine or to the AHSV causing the outbreak. This situation is even
more complicated in endemic countries where polyvalent vaccines are used and different AHSV serotypes circulate in the field. The difficulty of gathering a number of well characterized sera to validate the serological tests and assess their fitness for the different purposes for which the tests are to be used is an important consideration for the preparation of the serum panel used in our study. A significant number of sera representing the various epidemiological situations of the disease were collected. This, together with the organization of an international collaborative trial among reference and specialist AHS laboratories, provided the opportunity to obtain robust and reliable data on the test's reproducibility (Stage 3 of the Validation Pathway). It is important to note that for our study we used a commercial diagnostic ELISA with well characterized diagnostic performance characteristics, that had already been assessed according to the Stage 1 and Stage 2 of the OIE validation pathway (OIE, 2012).

Expression of AHS-VP7 proteins by means of recombinant baculoviruses has been used to manufacture large quantities of immunodiagnostic reagents of good quality. Previous studies have used this recombinant antigen in indirect ELISA procedures, demonstrating that VP7-based antibody detection methods are rapid, robust and highly accurate diagnostic tools, with superior diagnostic accuracy to the classical virus neutralization test (Maree & Paweska, 2005).

ELISA methods based on the competitive-blocking principle are useful for the diagnosis of viral and bacterial diseases (Gallardo et al., 2015; Praud et al., 2016) due to their good diagnostic performance and their potential to be used with sera independent of the animal species of origin. Currently, the VP7 Blocking ELISA for the diagnosis of AHS is available as a commercial kit at National Reference Laboratories of the EU, and its performance is checked annually through ring trials organized by the European Reference Laboratory for AHS (Agüero, personal communications). In many laboratories, the test is currently the basis of preexport testing for AHS, a procedure that is routinely used to support the intense and increasing movement of horses across international borders. Both indirect and blocking VP7 ELISA methods are tests recommended by OIE for serological diagnosis of AHS (OIE, 2017b) and are prescribed tests by European legislation for movement checks and importation of *Equidae* (European Union, 2009).

We conducted a thorough analysis of performance of the AHS-VP7 Blocking ELISA with the view of generating enough data to support the validation of this assay up to Stage 3 (Reproducibility) of the OIE validation pathway for diagnostic tests (OIE, 2012, 2017c,d). Data from one laboratory clearly performed as an outlier and, after investigation, was excluded from the assessment due to unidentified operational errors, after which, reproducibility was estimated by assessing the overall sensitivity and specificity and their associated uncertainty.

The good test sensitivity (98.4% [95% CI: 95.3–99.7]) of the VP7 Blocking ELISA in naturally or experimentally infected animals confirms its ability to detect nonvaccinated individuals previously exposed to virus and would support its use for trade purposes. The ability to detect antibodies in infected horses previously immunized with vaccines containing complete AHSV virions was also very good

either in animals vaccinated with attenuated (100% [95% CI: 95.9–100]) or with inactivated vaccines (100% [95% CI: 97.3–100]). This reveals the potential usefulness of the test to assess the sero-prevalence of AHS in a population of *Equidae*. In the case of a population where vaccination was performed, with either inactivated or live attenuated vaccines, it would be difficult to determine whether the seropositive animals had been exposed to field virus. In these cases, further investigations would be required, which could potentially include the routine monitoring of the serological status of the population (or of sentinel herds) or examination of the vaccination and clinical histories of the animals. The utility of serological methods for these purposes needs further analysis and evaluation, and the good performance of the VP7 Blocking ELISA would be a good candidate to determine the value of serology as a surveillance tool in endemic countries. In the case of a nonendemic country, the high sensitivity and specificity of the test indicate that the VP7 ELISA is an excellent tool for disease surveillance and corroboration of disease freedom once infection occurring during an outbreak has been eradicated.

In our study we computed any dubious result as positive. Of the total of results computed in the calculations of sensitivity and specificity (n = 1032), only 47 (4.6%) were classified as dubious. This very low percentage of dubious results is unlikely to affect negatively the use of this assay for surveillance procedures. In practice, it would be appropriate to initially classify any dubious results as provisional positives that require further investigation depending on specific epizootiological circumstances (epizootic in nonendemic country, epizootic in endemic country, post eradication surveillance in nonendemic country etc....)

Several initiatives to develop DIVA (Differentiation of Infected from Vaccinated Animals) vaccines have been developed over the years due to the unsuitability of current vaccines to discriminate between active AHS-infected and -vaccinated horses in a population. Some potential DIVA vaccines are based on the immunogenic antigen VP2 or VP2 and VP5 (Alberca et al., 2014; Castillo-Olivares et al., 2011; Chiam et al., 2009; Guthrie et al., 2009), which are the targets of virus neutralizing antibodies of the host. Because such vaccines lack VP7 in their composition, they would not induce a VP7 antibody response and therefore would not interfere with current diagnostic tests based on VP7. A combination of these VP2/VP5-based vaccines with VP7 diagnostic tests would offer useful DIVA capacity and would improve surveillance procedures. With this in mind we included in the panel some serum samples from horses vaccinated and experimentally infected with AHS. Thus, we also assessed the VP7 Blocking ELISA performance in animals experimentally infected with AHSV after being vaccinated with MVA viruses expressing single AHSV VP2 antigens (Alberca et al., 2014). Although the animals were fully protected against the disease, the level of VP7 antibodies detected was low. This finding would be consistent with immune response of individuals with high levels of protection against AHSV in which exposure to the virus (and VP7 antigen) is significantly reduced in terms of time and intensity. The availability of this type of sample was very limited, but the results of our study demonstrated the potential of VP7 Blocking ELISA to be used in
combination with VP2-based DIVA vaccines. Further work is needed to corroborate the applicability of the VP7 Blocking ELISA as a DIVA test when it is used with VP2-based vaccines.

The test also showed good capacity to monitor the response to vaccination because of its capacity to detect antibodies in vaccinated noninfected horses (94.8%; 95% CI: 93.5–97.1), with no major differences between types of vaccine used. This is extremely important when organizing preventive sanitary campaigns in order to monitor the operational activities of field vaccination programmes. The values obtained in our study are in line with those obtained by previous work when evaluating the VP7-based indirect ELISA in its ability to detect horses immunized with attenuated vaccine (Maree & Paweska, 2005). In our study, a number of serum samples that were derived from vaccinated and not infected horses produced a negative result by the VP7 Blocking ELISA (this results coincided with the original results obtained at the laboratory of origin). This can be explained from the fact that these sera were collected during the last years of the eradication campaign of the last AHS Spanish outbreak (1987–1992), when AHSV was not circulating in the field, long time after the animals were last vaccinated (as indicated in Table 1). Therefore, under these circumstances, it is not surprising that the antibody titres had waned to undetectable levels by the time the horses were sampled.

The surveillance of AHSV free territories is of great importance in containing the disease within endemic areas. For this reason, serological diagnostic tests need to be highly sensitive to accurately detect possible incursions of AHSV in nonendemic areas or to certify with confidence that a previously infected population is free of AHS after the implementation of an eradication programme. In addition, they need to be highly specific to avoid the unnecessary costs associated with the animal movement restrictions that would be imposed in the region until investigation of any false positive results is concluded. The results on test specificity obtained with VP7 Blocking ELISA in our study (100% [95% CI: 98.9–100]) combined with its high sensitivity (reviewed above) would suggest its feasibility for the surveillance of the AHS in free areas. Again, the values obtained in our study are in line with those obtained by previous work when evaluating the VP7-based indirect ELISA (Maree & Paweska, 2005).

In conclusion, through this comprehensive evaluation we can conclude that the VP7 Blocking ELISA is reproducible and satisfies the OIE requirements of reproducibility. However, it is recommended (OIE, 2012, 2017c) that monitoring of diagnostic performance of the assay is periodically performed through international ring trials, such as those that have been previously conducted by the OIE/EU reference laboratories over the years.

The VP7 Blocking ELISA, in its commercial version, is ready to enter Stage 4 of the validation pathway (Programme Implementation). Specifically, future studies should aim to test the diagnostic performance of the assay using contemporary serum samples collected during control campaigns in endemic countries. This type of analysis will help to determine positive and negative predictive values of the tests in the various epidemiological scenarios in endemic territories.

## ACKNOWLEDGEMENTS

This study was funded by the OIE through the Scheme of Scientific Activities to support the control of African horse sickness (AD/SR/2015/1885). Dr Javier Castillo-Olivares was funded by Biotechnology and Biological Sciences Research Council/Pirbright Institute Fellowship Programme (grant number: BBS/E/I/00002536). SG was funded by the Biotechnology and Biological Sciences Research Council (grant numbers: BBS/E/I/00007036 and BBS/E/I/00007037).

## CONFLICT OF INTEREST

P. Rueda and P. Sastre are employees of INGENASA, the manufacturing company of the AHS Blocking ELISA test, but this fact does not represent in itself a competing interest. INGENASA participated by supplying the test kits, and performing the ELISA tests on the samples from the panel, which were blindly coded. Interpretation and analysis of the data, collected from nine different laboratories, including INGENASA, was conducted by Drs Duran, Aguero, Gubbins and Castillo-Olivares. We therefore conclude that the objectivity of the study was not affected. Otherwise there are no competing interests.

## ORCID

Simon Gubbins ID https://orcid.org/0000-0003-0538-4173

Javier Castillo-Olivares ID https://orcid.org/0000-0003-3453-8557

## REFERENCES

Agüero, M., Gómez-Tejedor, C., Cubillo, M. A., Rubio, C., Romero, E., & Jiménez-Clavero, M. A. (2008). Real-time fluorogenic reverse transcription polymerase chain reaction assay for detection of African horse sickness virus. *Journal of Veterinary Diagnostic Investigation, 20*, 325–328. https://doi.org/10.1177/104063870802000310

Alberca, B., Bachanek-Bankowska, K., Cabana, M., Calvo-Pinilla, E., Viaplanana, E., Frost, L., ... Castillo-Olivares, J. (2014). Vaccination of horses with a recombinant modified vaccinia Ankara virus (MVA) expressing African horse sickness (AHS) virus major capsid protein VP2 provides complete clinical protection against challenge. *Vaccine*, 32, 3670–3674. https://doi.org/10.1016/j.vaccine.2014.04.036

Bachanek-Bankowska, K., Maan, S., Castillo-Olivares, J., Manning, N. M., Maan, N. S., Potgieter, A. C., ... Mertens, P. P. (2014). Real-time RT-PCR assays for detection and typing of African horse sickness virus. *PLoS ONE*, 9(4), e93758. https://doi.org/10.1371/journal.pone.0093758

Castillo-Olivares, J., Calvo-Pinilla, E., Casanova, I., Bachanek-Bankowska, K., Chiam, R., Maan, S., ... Mertens, P. P. (2011). A modified vaccinia Ankara virus (MVA) vaccine expressing African horse sickness virus (AHSV) VP2 protects against AHSV challenge in an IFNAR-/- mouse model. *PLoS ONE*, 6(1), e16503. https://doi.org/10.1371/journal.pone.0016503

Chiam, R., Sharp, E., Maan, S., Rao, S., Mertens, P., Blacklaws, B., ... Castillo-Olivares, J. (2009). Induction of antibody responses to African Horse Sickness Virus (AHSV) in ponies after vaccination with recombinant Modified Vaccinia Ankara (MVA). *PLoS ONE*, 4(6), e5997. https://doi.org/10.1371/journal.pone.0005997
European Union (2009). Council Directive 2009/156/EC of 30 November 2009 on animal health conditions governing the movement and importation from third countries of equidae, as last amended. Retrieved from https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX: 02009L0156-20161018&rid=1. Accessed 15 Mar 2018.

Gallardo, C., Nieto, R., Soler, A., Pelayo, V., Fernández-Pinero, J., Mar-
kowska-Daniel, I., ... Arias, M. (2015). Assessment of African swine
fever diagnostic techniques as a response to the epidemic outbreaks
in eastern European Union Countries: How to improve surveillance
and control programs. *Journal of Clinical Microbiology*, 3, 2555-2565.
https://doi.org/10.1128/JCM.00857-15

Guthrie, A. J., MacLachlan, N. J., Joone, C., Lourens, C. W., Weyer, C. T., Quan, M., ... Gardner, I. A. (2013). Diagnostic accuracy of a duplex real time reverse transcription quantitative PCR assay for detection of African horse sickness virus. *Journal of Virological Methods*, 189, 30-35. https://doi.org/10.1016/j.jviromet.2012.12.014

Guthrie, A. J., Quan, M., Lourens, C. W., Audonnet, J. C., Minke, J. M., Yao, J., ... Maclachlan, N. J. (2009). Protective immunization of horses with a recombinant canarypox virus vectored vaccine co-expressing genes encoding the outer capsid proteins of African horse sickness virus. Vaccine, 27, 4434-4438. https://doi.org/10.1016/j.vaccine.2009.05.044

Hamblin, C., Graham, S. D., Anderson, E. C., & Crowther, J. R. (1990). A competitive ELISA for the detection of group-specific antibodies to African horse sickness virus. *Epidemiology and Infection*, *104*, 303–312. https://doi.org/10.1017/S0950268800059483

Hamblin, C., Mellor, P. S., Graham, S. D., Hooghuis, H., Montejano, R. C., Cubillo, M. A., & Boned, J. (1991). Antibodies in horses, mules and donkeys following monovalent vaccination against African horse sickness. *Epidemiology and Infection*, *106*, 365-371. https://doi.org/10.1017/S0950268800048512

Hopkins, I. G., Hazarati, X., & Ozawa, Y. (1966). Development of plaque techniques for the titration and neutralisation tests with African horse sickness virus. *American Journal of Veterinary Research*, *27*, 96-105.

ISO (2005). ISO 13528:2005. Statistical methods for use in proficiency testing by interlaboratory comparisons.

Laviada, M. D., Roy, P., & Sánchez-Vizcaíno, J. M. (1992). Adaptation and evaluation of an indirect ELISA and immunoblotting test for African horse sickness antibody detection. In T. E. Walton & B. I. Osburn (Eds.), *Proceedings of the 2nd International Symposium on Bluetongue, African Horse Sickness and Related Orbiviruses* (pp. 646–650). Boca Raton, FL: CRC Press.

Maree, S., & Paweska, J. T. (2005). Preparation of recombinant African horse sickness virus VP7 antigen via a simple method and validation of a VP7-based indirect ELISA for the detection of group-specific IgG antibodies in horse sera. *Journal of Virological Methods*, 125, 55–65. https://doi.org/10.1016/j.jviromet.2004.12.002

McIntosh, B. M. (1956). Complement fixation with horsesickness virus. *Onderstepoort Journal of Veterinary Research*, 27, 165–169.

Mellor, P. S., & Hamblin, C. (2004). African horse sickness. *Veterinary Research*, 35, 445–466. https://doi.org/10.1051/vetres:2004021

OIE (2012). Standard operating procedure for OIE registration of diagnostic kits. Guide and administrative form. Retrieved from https://www.oie.int/doc/ged/D12069.PDF. Accessed 15 Mar 2018.

OIE (2017a). Official disease status: African horse sickness. http://www.oie.int/en/animal-health-in-the-world/official-disease-status/african-horse-sickness/. Accessed 25 Mar 2018.

OIE (2017b). African horse sickness (infection with African horse sickness virus). In *Manual of diagnostic tests and vaccines for terrestrial animals*. Chapter 2.5.11. Retrieved from http://www.oie.int/fileadmin/Home/eng/Health_standards/tahm/2.05.01_AHS.pdf. Accessed 15 Mar 2018.

OIE (2017c). Principles and methods of validation of diagnostic assays for infectious diseases. In *Manual of diagnostic tests and vaccines for terrestrial animals*. Chapter 1.1.6. http://www.oie.int/fileadmin/Home/eng/Health_standards/tahm/1.01.06_VALIDATION.pdf. Accessed 15 Mar 2018

OIE (2017d). Recommendation for validation of diagnostic tests. In *Manual of diagnostic tests and vaccines for terrestrial animals*. Section 3.6. Retrieved from http://www.oie.int/fileadmin/Home/eng/Health_standards/tahm/3.6.00_INTRODUCTION.pdf. Accessed 15 mar 2018

Praud, A., Durán-Ferrer, M., Fretin, D., Jaÿ, M., O'Connor, M., Stournara, A., ... Garin-Bastuji, B. (2016). Evaluation of three competitive ELISAs and a fluorescence polarisation assay for the diagnosis of bovine brucellosis. *Veterinary Journal*, **216**, 38–44. https://doi.org/10.1016/j.tvjl.2016.06.014

Rubio, C., Cubillo, M. A., Hooghuis, H., Sanchez-Vizcaíno, J. M., Diaz-Laviada, M., Plateau, E., ... Hamblin, C. (1998). Validation of ELISA for the detection of African horse sickness virus antigens and antibodies. *Archives of Virology*, 14, 311–315.

Thrusfield, M. (1995). Veterinary epidemiology, 2nd ed. Oxford, UK: Blackwell Science.

Verwoerd, D. W., Huismans, H., & Erasmus, B. J. (1979). Orbiviruses. In H. Fraenkal-Conrat, & R. R. Wagner (Eds.), *Comprehensive virology*, Vol. 14 (pp. 285–345). New York, NY: Plenum Press. https://doi.org/10.1007/978-1-4684-3563-4

Wade-Evans, A., Wolhouse, T., O'Hara, R., & Hamblin, C. (1993). The use of African horse sickness virus VP7 antigen, synthesised in bacteria, and anti-VP7 monoclonal antibodies in a competitive ELISA. *Journal of Virological Methods*, 45, 179-188. https://doi.org/10.1016/0166-0934(93)90102-W

Weyer, C. T., Joone, C., Lourens, C. W., Monyai, M. S., Koekemoer, O., Grewar, J. D., ... Guthrie, A. J. (2015). Development of three triplex real-time reverse transcription PCR assays for the qualitative molecular typing of the nine serotypes of African horse sickness virus. *Journal of Virological Methods*, 223, 69-74. https://doi.org/10.1016/j.jviromet.2015.07.015

Zientara, S., Weyer, C. T., & Lecollinet, S. (2015). African horse sickness. *Revue Scientifique et Technique*, 34, 315-327. https://doi.org/10.20506/rst.34.2.2359

How to cite this article: Durán-Ferrer M, Agüero M, Zientara S, et al. Assessment of reproducibility of a VP7 Blocking ELISA diagnostic test for African horse sickness. *Transbound Emerg Dis*. 2019;66:83–90. https://doi.org/10.1111/tbed.12968