1 **Annex 7. Item 5.1. – Chapter 2.2.6. Selection and use of reference samples and panels**

###### 2 MEETING OF THE WOAH BIOLOGICAL STANDARDS COMMISSION

3 **Paris, 4–8 September 2023**

4

**USA COMMENTS IN RED FONT**

**CHAP T E R  2 . 2 . 6 .**

# 6	SEL ECT I ON A N D U SE OF

7	**R EF ER EN CE SA M PLES AN D PA N EL S**

## 8	INTRODUCTION

9 *The WOAH Validation Recommendations provide detailed information and examples in support of the*

10 *~~WOAH Validation Standard that is published as~~Chapter 1.1.6*~~Principles and methods of~~Validation of

11 diagnostic assays for infectious diseases of terrestrial animals*~~this~~*~~Terrestrial Manual*, or Chapter 1.1.2 of*~~

12 *~~the~~*~~Aquatic Manual~~*. ~~The Term “WOAH Validation Standard” in this chapter should be taken as referring to~~*

13 *~~those chapters.~~*

14 *Reference samples and panels are essential from the initial proof of concept in the development laboratory*

15 *through to the maintenance and monitoring of assay performance in the diagnostic laboratory and all of the*

16 *stages in between. The critical importance of reference samples and panels cannot be over-emphasised.*

17 *The wrong choice of reference materials can lead to bias and flawed conclusions right from development*

18 *through to validation and use. Therefore, care must be exercised in selecting reference samples and*

19 *designing panels.*

20 ***Fig. 1.****Reference samples and panels grouped based on similar characteristics and composition. The*

21 *topics and alphanumeric subheadings*(*e.g. Proof of concept, A.2.1*) *refer to the relevant section in*~~the~~

22 ~~WOAH Validation Standard,~~*Chapter 1.1.6*Validation of diagnostic assays for infectious diseases of

23 terrestrial animals*.*

24 *ASp = Analytical specificity; ASe = Analytical sensitivity; DSp = diagnostic specificity; DSe = diagnostic sensitivity*

Report of the Meeting of the Biological Standards Commission / September 2023

63

25 *~~As can be seen in Figure 1,~~Reference samples ~~and/~~or panels are mentioned throughout ~~the WOAH~~*

26 *~~Validation Standard,~~chapter 1.1.6. ~~As defined in the glossary of the OIE Quality Standard and Guidelines~~*

27 *~~for Veterinary Laboratories: Infectious Diseases~~*~~,~~*‘Reference materials are “substances whose properties are*

28 *sufficiently homogenous and well established to be used for the calibration of an apparatus, the assessment*

29 *of a measurement method, or for assigning values to materials”**33**. In the context of test method validation,*

30 *reference materials or samples contain the analyte of interest in varying concentrations or ~~activities~~*

31 *reactivities and are used in developing and evaluating the candidate assay’s analytical and diagnostic*

32 *performance~~characteristics. In our case,~~. Analyte means the specific component of a test sample that is*

33 *detected or measured by the test method,  antibody, antigen or nucleic acid. ~~These~~Reference samples*

34 *may be sera, fluids, tissues, excreta, feed ~~and~~or environmental samples that contain the analyte of interest*

35 *and are usually harvested from infected animals and their environments. However, in some cases, they may*

36 *be prepared in the laboratory from an original starting material*(*a dilution of a high positive serum in*

37 *negative serum*) *or perhaps created by spiking the chosen matrix with a derived analyte*(*a bacterial or*

38 *viral culture, a recombinant/expressed protein, or a genomic construct*). *Whether natural or prepared, they*

39 *are used in experiments throughout the development process, carry over into the validation pathway and*

40 *can be used to monitor performance throughout the lifespan of the assay.*

41 *~~In Figure 1, reference samples and panels are grouped based on similar characteristics and composition~~*

42 *~~and these groupings will be the basis for the following descriptions. As a cross-reference, the appropriate~~*

43 *~~Section of the OIE Validation Standard is indicated under each particular application of the reference sample~~*

44 *~~or panel.~~*

45 *~~Reference samples may be used for multiple purposes from the initial stages of development and~~*

46 *~~optimisation, through Stage 1 and into continual monitoring and maintenance of the assay.~~ Wherever*

47 *possible, large quantities of ~~these~~reference samples should be collected or prepared and preserved for*

48 *long-term use. Switching reference samples during the validation process introduces an intractable variable*

49 *that can severely undermine interpretation of experimental data and ~~therefore,~~the integrity of the*

50 *development and validation process. For assays that may target multiple species, the samples should be*

51 *representative of the primary species of interest. It is critical that these samples reflect both the target analyte*

52 *and the matrix in which it is found in the population for which the assay is intended. The reference materials*

53 *should appropriately represent the range of analyte concentration to be detected by the assay.*

54 *~~It is important to emphasise that, no matter~~Whether reference samples are selected from natural sources*

55 *or prepared in the laboratory, all selection criteria ~~or~~and preparation procedures, as well as testing*

56 *requirements, need to be fully described and put into document control. Not only is this good quality*

57 *management practice, but it will provide both an enhanced level of continuity and confidence throughout the*

58 *lifespan of the assay. Summaries of the data to be collected and documented for reference material can be*

59 *found in Figure 2. For more detail on best practice and quality standards for the documentation of*

60 *provenance of reference material refer to Watson*et al. (*2021*)*.*

1 https://www.techlab.fr/Commun/UK_Def_MRC.asp

Report of the Meeting of the Biological Standards Commission / September 2023

64

61 ***Figure. 2.****Documentation of reference material should be thorough to ensure i*) *transparency of intended*

62 *purpose during assay development; ii*) *the correct sample types are used in all stages of assay*

63 *development and validation; iii*) *accurate replacement of depleted reagents; and iv*) *appropriate choice of*

64 *reference material during assay modification and re-validation. Minimum descriptive metadata are listed for*

65 *pathogen, animal host, tissue type and phase of infection.*

66	**A. GROUP A**

67 The question of pooling of samples to create a reference sample is often asked. If reference material is harvested from a

68 single animal, it is important to ascertain  it is representative of a typical course and stage of infection within

69 the context of the population to be tested. If not, this could lead to bias and flawed conclusions related to validation. Pooling

70 is a good alternative but it is imperative to pool from animals that are in a similar phase of infection. This is particularly

71 important for antibody detection systems. Pooling also addresses the issue of the larger quantities of reference material to

72 be stored for long term use, especially when dealing with smaller host species. Before pooling any samples, it is preferable

73 that they be independently tested to demonstrate that they are similar with respect to analyte concentration and/or

74 reactivity. There should be an assessment following pooling to ensure that unforeseen interference is not introduced by

75 the pooling of multiple samples, for example differing blood types or antibody composition within the independent samples

76 could cross-react within the pool, thus causing the pooled sample to behave differently in the specified assay than the

77 individual samples when tested independently*.*

78 It is often difficult to obtain individual samples that truly represent analyte concentrations or reactivities across the spectrum

79 of the expected range. Given the dynamics of many infections or responses to pathogens, intermediate ranges are often

80 very transient. In the case of antibody responses, early infection phases in individual animals often result in highly variable

81 and heterogeneous populations of antibody isotypes and avidities. In general, these do not make good reference samples

82 for assessing the analytical characteristics of an assay. They are nonetheless important for different types of reference

83 panels as will be discussed later. For most applications in Group A, it is acceptable to use prepared samples that are

84 spiked with known concentrations of analyte or a dilution series of a high positive in negative matrix to create a range of

85 concentrations.

86 Whether natural or prepared, reference samples should represent the anticipated range of analyte concentrations, from

87 low weak to ~~high~~strong positive, which would be expected during a typical course of infection. A negative reference sample

88 should be included as a background monitor. If a negative (matrix) is used as diluent for preparation of a positive reference

89 sample (e.g. a negative serum used to dilute a high positive serum or tissue spiked with a construct), that negative should

90 definitely be included as the negative reference sample.

91 As mentioned above, all reference samples should be well characterised. This includes documentation on both the

92 pathogen and donor host. For pathogens, this may include details related to strain, serotype, genotype, lineage, etc. The

93 source of the host material should be well described with respect to species, breed, age, sex, reproductive status,

94 vaccination history, herd history, etc. Wherever possible, the phase of infection should be noted. This could include details

95 related to clinical signs, antibody profiles, pathogen load or shedding, etc. Equally important, tests that are used to

96 determine disease/infection status need to be well documented (see Section E of this chapter for further explanation). In

97 some cases, experimental infection/exposure may be the only viable option for the production of reference material. In this

98 ~~case, all of the above considerations plus the experimental protocol should be detailed.~~

Report of the Meeting of the Biological Standards Commission / September 2023

65

99 Above all else, natural or prepared, reference materials must be unequivocal with respect to their status as representing

100 either a true positive or a true negative sample. This may require that the status be confirmed using another test or battery

101 of tests. For example, many antibody reference sera are characterised using multiple serological tests. This provides not

102 only confidence but additional documented characteristics that may be required when attempting to replace or duplicate

103 this reference material in the future.

104 Recommendations regarding stability and storage of reference materials are available: https://www.woah.org/en/what-we-

105 offer/veterinary-products/#ui-id-4

#### 106 1.	Proof of concept (~~WOAH Validation Standard,~~Chapter 1.1.6, Section A.2.1)

107 ~~The WOAH Validation Standard,~~Chapter 1.1.6states that test methods and related procedures must be appropriate for

108 specific diagnostic applications in order for the test results to be of relevance. In other words, the assay must be ‘fit for

109 purpose’. Many assays are developed with good intentions but without a specific application in mind. At the very outset, it

110 is critical that the diagnostic purpose(s) should be defined with respect to the population(s) to be tested. The most common

111 purposes are listed in broad terms in Section A of ~~the WOAH Validation Standard,~~chapter 1.1.6. As such, they are inclusive

112 of more narrow and specific applications. However, these specific purpose(s) need to be clearly defined from the outset

113 and are critically important in the context of a fully validated assay. As will be seen in the following descriptions, clearly

114 defining the application will have impact on both the selection of reference samples and panels and the design of analytical

115 and diagnostic evaluations.

#### 116 2.	Operating range (~~WOAH Validation Standard,~~Chapter 1.1.6, Section A.2.~~2~~3) and

117 **analytical sensitivity (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.1.3)**

##### 118 2.1.	~~Analytical approaches~~Operating range and analytical sensitivity

119 The operating range of the assay ~~is~~defines the lower and upper analyte detection limits and the interval ~~of analyte~~

120 ~~concentrations (amounts)~~over which the method provides suitable accuracy and precision. ~~It also defines the lower~~

121 ~~and upper detection limits the assay. To establish this range,~~The operating range is established by serial dilution, to

122 extinction, of replicates of a ~~high~~strong positive reference sample~~is selected. This high positive sample~~, either natural

123 or prepared~~, is serially diluted to extinction~~. Dilutions of the strong positive are made in ~~a~~negative matrix

124 representative of the typical sample ~~matrix of samples~~type taken from animals in the population targeted by the

125 assay. This includes antibody assays where ~~a high~~replicates of a strong positive reference serum should be diluted

126 in a negative reference serum to create the dilution series. Analytical sensitivity (ASe) is measured by replicates of

127 the lower limit of detection (LOD) of an analyte in an assay. The same ~~high~~strong positive reference sample may be

128 used to determine both the operating range and the analytical LOD.

##### 129 2.2.	Comparative approaches to analytical sensitivity

130 If the intended purpose is to detect low levels of analyte or subclinical infections, it may be difficult to obtain the

131 appropriate reference materials from early stages of the infection process. In some cases, it may be useful to

132 determine a comparative ASe by running a panel of samples on the candidate assay and on another independent

133 assay. Ideally this panel of samples would be serially collected from either naturally or experimentally infected animals

134 and should represent infected animals early after infection, ~~on~~through to the development of clinical or fulminating

135 disease, if possible. This would provide a relative comparison of ASe between the assays~~, as well as,~~and a temporal

136 comparison of the earliest point of detection relative to the pathogenesis of the disease.

137 An experiment like the one described above, provides a unique opportunity to collect reference samples representing

138 a natural range of concentrations that would be useful for other validation purposes. Care must be taken to avoid use

139 of such samples when inappropriate (consult Group D below). Wherever possible serial samples should be collected

140 from ~~at least five~~a statistically sound number of animals throughout the course of infection. In cases where sampling

141 is lethal (e.g. requiring the harvest of internal organ tissues), the number of animals required ~~would be a minimum~~

142 depends on need and fitness of ~~five per sampling event~~the experimental approach. In all cases approval from an

143 ethics committee is required. For smaller host species, ~~this~~the number may need to be increased in order to collect

144 sufficient reference material. Given that experiments like this require a high commitment of resources, it would be

145 wise to maximise the collection of not only the currently targeted reference samples but additional materials (e.g.

146 multiple tissues, fluids, etc.) that may be useful as reference materials in the future.

#### 147 3.	Optimisation (~~WOAH Validation Standard,~~Chapter 1.1.6, Section A.2.~~3~~2) and preliminary

148 **repeatability (~~WOAH Validation Standard,~~Chapter 1.1.6, Section A.2.~~6~~8)**

149 Optimisation is the process by which the most important physical, chemical and biological parameters of an assay are

150 evaluated and adjusted to ensure that the performance characteristics of the assay are best suited to the intended

151 application. At least three reference samples representing negative, ~~low~~weak and ~~high~~strong positive may be chosen

152 from either natural or prepared reference samples. Optimisation experiments are rather exhaustive especially when assays

153 with multiple preparatory and testing steps are involved. It is very important that a sufficient quantity of each reference

154 sample be available to complete all optimisation experiments. Changing reference samples during the course of

155 optimisation is not recommended as this will result in the addition of an uncontrolled variable and a disruption in the

156 continuity of optimisation evidence.

157 Assessment of repeatability should begin during assay development and optimisation stages~~. Repeatability~~and is further

158 verified during Stage 1 of assay validation (Section B.1.1 of chapter 1.1.6). The same reference samples should be used

159 ~~for both processes, again~~throughout to provide continuity of evidence.

#### 160 4.	Calibration and process controls (~~WOAH Validation Standard,~~Chapter 1.1.6, Section

161	**A.2.6)**

##### 162 4.1.	International, national or in-house analyte reference standards

163 International reference standards are highly characterised, contain defined concentrations of analyte, and are usually

164 prepared and held by international reference laboratories. They are the reagents to which all assays and/or other

165 reference materials should be standardised. National reference standards are calibrated by comparison with an

166 international standard reagent whenever possible. In the absence of an international standard, a national reference

167 standard may be selected or prepared and it then becomes the standard of comparison for the candidate assay. In

168 the absence of both of the above, an in-house standard should be selected or prepared by the development laboratory

169 within the responsible organisation. In all cases, thorough documentation of reference material should be observed

170 as summarised in Figure 2. All of the standard reagents, whether natural or prepared, must be highly characterised

171 through extensive analysis, and preferably the methods for their characterisation, preparation, and storage have been

172 published in peer-reviewed publications (Watson *et al.*, 2021). These reference standards should also be both stable

173 and innocuous.

174 Reference standards, especially antibody, are usually provided in one of two formats. They may be provided as a

175 single positive reagent of given titre with the expectation that the candidate assay will be standardised to give an

176 equivalent titre. This is a straight forward analytical approach but many of these ‘single’ standards have been prepared

177 from highly positive samples as a pre-dilution in a negative matrix in order to maximise the number of aliquots

178 available. The drawback here is that there is no accounting for any potential matrix effect in the candidate assay as

179 there is no matrix control provided. The other approach is to provide a negative and a ~~low~~weak and ~~high~~strong

180 positive set of reference standards that are of known concentrations or reactivities and are within the operating range

181 of the standard method that was used to prepare them. The negative provided in the set must be the same as the

182 negative diluent used to prepare the weak and strong positive reference standard, if the positive standards were

183 diluted. This compensates for any potentially hidden matrix effect. In addition, this set of three acts as a template for

184 the selection and/or preparation of process controls (discussed below).

185 Classically, the above standards usually have been polyclonal antibody standards and to a lesser extent, conventional

186 antigen standards used for calibration of serological assays. However, today, reference standards could also be

187 monoclonal antibodies or recombinant/expressed proteins or genomic constructs, if they are to be used to calibrate

188 assays to a single performance standard.

##### 189 4.2.	Working standards or process controls

190 Working standard reagent(s), commonly known as quality or process controls, are calibrated to international, national,

191 or in-house standard reagents. They are selected or prepared in the local matrix which is found in the population for

192 which the assay is intended. Ideally, negative and ~~low~~weak and ~~high~~strong positive working standards should be

193 selected or prepared. Concentrations and/or reactivities should be within the normal operating range of the assay.

194 Large quantities should be prepared, aliquoted and stored for routine use in each diagnostic run of the assay. The

195 intent is that these controls should mimic, as closely as possible, field samples and should be handled and tested like

196 routine samples. They are used to establish upper and lower control limits of assay performance and to monitor

197 random and/or systematic variability using various control charting methods. Their daily performance will determine

198 whether or not an assay is in control and if individual runs may be accepted. As such, these working reference

199 samples are critically important from a quality management standpoint.

#### 200 5.	Technical modifications (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.5.2.1)

201 Technical modifications to a validated assay such as changes in instrumentation, extraction protocols, and conversion of

202 an assay to a semi-automated or fully automated system using robotics will typically not necessitate full revalidation of the

203 assay. Rather, a methods comparison study may be done to determine if these minor modifications to the assay protocol

204 will affect the test results. Consult See chapter 2.2.8 *~~Comparability~~*~~of *assays after changes in a validated test method*~~for

205 description of experiments and statistical approaches ~~to assay precision in the face of technical modifications~~that are

206 appropriate for comparability testing (Bowden & Wang, 2021; Reising *et al.,*2021).

207 In general, these approaches require the use of three reference samples, a negative, a weak and ~~a low and high~~strong

208 positive~~. Again these samples~~to represent the entire operating range of both assays. Samples may be either natural or

209 prepared. The important point to re-iterate here is that the same reference samples that were used in the developmental

210 stages of the assay may be used to assess modifications after the method has been put into routine diagnostic use. This

211 provides a higher level of confidence assessing potential impacts because the performance characteristics of these

212 reference samples have been well characterised. At the very least, if new reference samples are to be used, they should

213 be selected or prepared using the same criteria or preparation procedures established for previous materials~~. Again~~as this

214 enhances the continuity of evidence.

#### 215 6.	Reagent replacement (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.5.2.3)

216 When a reagent such as a process control sample is nearing depletion, it is essential to prepare and repeatedly test a

217 replacement before such a control is depleted. The prospective replacement should be included in multiple runs of the

218 assay in parallel with the original control to establish their proportional relationship. It is important to change only one

219 control reagent at a time to avoid the compound problem of evaluating more than one variable.

220 ~~Again, it cannot be over-emphasised that any~~Replacement reference reagent should be selected or prepared using the

221 same criteria or preparation procedures established for previous materials~~. Again~~as this enhances the continuity of evidence

222 and confidence in the assay and underlines the importance of documentation of reference material data (Figure 2).

225 Analytical specificity (ASp) is the degree to which the assay distinguishes between the target analyte and other components

226 that may be detected in the assay. ~~This is a relatively broad definition that is often not well understood. ASp may be broken~~

227 ~~down into different elements as described below.~~

228 The choice of reference samples that are required to assess ASp is highly dependent on the ~~specific~~intended purpose or

229 application ~~that was originally envisaged~~defined at the development stage of the assay. Assessment of ASp is a crucial

230 element in proof of concept and verification of fitness for purpose and may be broken down into three elements: selectivity,

231 exclusivity and inclusivity.

232 Selectivity: an important element is the extent to which a method can accurately detect and or quantify the targeted analyte

233 in the milieu of nucleic acids, proteins and/or antibodies in the test matrix. This is sometimes termed ‘selectivity’. An

234 example is the use of reference samples for tests that are designed to differentiate infected from vaccinated animals (DIVA

235 tests).

236 Reference samples need to be selected and tested from i) non-infected/non-vaccinated, ii) non-infected/vaccinated, iii)

237 infected/non-vaccinated, and iv) infected/vaccinated animals. These samples may be collected under field conditions but

238 it is important that an accurate history be collected, ideally with respect to the animals, but at least to the herds involved,

239 including vaccination practices and disease occurrences (Figure 2). Alternatively, it may be necessary to produce this

240 material in experiments like those described in Section A.2.2 of this chapter, ~~but~~ including a combination of experimentally

241 vaccinated and challenged animals. ~~It~~Application of the 3 Rs (replacement, reduction and refinement) aims to avoid or

242 minimise the number of animals used in experiments. For enzyme-linked immunosorbent assays (ELISAs), it is important

243 to avoid use of the vaccine as capture antigen in the assay (e.g. indirect ELISA~~enzyme-linked immunosorbent assay [I-~~

244 ~~ELISA]~~), because carrier proteins in the vaccine may stimulate non-specific antibody responses in vaccinated animals that

245 may be detected in ELISA~~leading to false positives in the assay. Similarly to the comparative approach described above~~

246 ~~with respect to ASe, at least five animals in each group should be considered. For smaller host species, this number may~~

247 ~~need to be increased in order to collect sufficient reference material.~~, leading to false positives in the assay. Depending on

248 the DIVA test, a single experiment could be designed to assess aspects of both ASe and ASp.

249 ~~A second element, sometimes termed ‘exclusivity’,~~ Exclusivity is the capacity of the assay to detect an analyte or genomic

250 sequence that is unique to a targeted organism, and excludes all other other known organisms that are potentially cross-

251 reactive. This is especially true in serological assays where there are many examples of antigens expressed by other

252 organisms that are capable of eliciting cross-reacting antibody. An attempt should be made to obtain reference samples

253 from documented cases of infections ~~and/~~or organisms that may be cross-reactive. Depending on the type of assay, these

254 reference materials may represent the organism itself, host-derived samples, or genomic sequences. A profile for the

255 exclusivity of the assay should be established, and expanded on a continual basis as potentially cross-reactive organisms

256 arise.

257 ~~Thirdly, a critical design consideration~~Inclusivity relates to the capacity of an assay to detect one or several strains or

258 serovars of a species, several species of a genus, or a similar grouping of closely related ~~organisms~~viruses, bacteria or

259 antibodies. This defines the scope of detection and thus the fitness for purpose. ~~Reference samples are required to define~~

260 ~~the scope of the assay~~The scope of the assay defines the choice of reference samples and the results will determine proof

261 of fitness. If for example an assay is developed as a screening test to detect all known genotypes or serotypes of a virus,

262 then reference samples from each representative type should be tested. As new lineages or serotype variants arise, they

263 too should be tested as part of the test profile, which should be updated on an ongoing basis.

#### 264 2.	Analytical accuracy of ~~adjunct~~ancillary tests (~~WOAH Validation Standard,~~Chapter 1.1.6,

265 **Section B.1.4)**

266 Some test methods or procedures are solely analytical tools ~~and are usually applied~~used to further characterise an analyte

267 that has been detected in a primary assay~~, for example assays like~~. Examples are the virus neutralisation test~~s~~ used to

268 type an isolated virus or characterise an antibody response and subtyping of haemagglutinin genes by polymerase chain

269 reaction of avian influenza virus. Such ~~adjunct~~ancillary tests must be validated for analytical performance characteristics,

270 ~~but~~and differ ~~from~~to routine diagnostic tests because they do not require validation for diagnostic performance

271 characteristics. The analytical accuracy of these tests is often dependant on the use of reference ~~reagents~~material. These

272 reagents, whether they are antibody for typing strains of organisms or reference strains of the organism, etc., should be

273 thoroughly documented, as required for any other reference material (Figure 2), with respect to their source, identity and

274 performance characteristics.

275	**C. GROUP C**

276 Reference samples in Group C may be used for a number of purposes. In the initial development stages, they may be

277 used in the assessment of assay repeatability and both preliminary reproducibility in Stage 1 and the more in depth

278 assessment of reproducibility in Stage 3 of the Validation Pathway. However, these samples have a number of other

279 potential uses once the assay is transferred to the diagnostic laboratory. They may be used as panels for training and

280 qualifying of analysts, and for assessing laboratory proficiency in external ring testing programmes. Ideally, 20 or more

281 individual samples should be prepared in large volumes. About a quarter (25%) should be negative samples and the

282 remainder (75%) should represent a collection of positives spanning the operating range of the assay. They should be

283 aliquoted into individual tubes in sufficient volumes for single use only and stored for long term use (Chapter 1.1.2

284 *Collection, submission and storage of diagnostic specimens*). The number of aliquots of each that will be required will

285 depend on how many laboratories will be using the assay on a routine diagnostic basis and how often proficiency testing

286 is anticipated. Ideally, they should be prepared in an inexhaustible quantity, but this is seldom feasible. At a minimum,

287 several hundred or more aliquots of each should be prepared at a time if the assay is intended for use in multiple

288 laboratories. This allows assessment of laboratory proficiency by testing the same sample over many testing intervals – a

289 useful means of detecting systematic error (bias) that may creep into long term use of an assay.

290 These samples may be natural or prepared from either single or pooled starting material. The intent is that they should

291 mimic as closely as possible a true test sample. Because mass storage is always a problem, it may be necessary to store

292 these materials in bulk and prepare working aliquots from time to time. However, if storage space is available, it is

293 preferable to prepare and store large numbers of aliquots at one time because bulk quantities of analyte, undergoing

294 freeze–thaw cycles to prepare a few aliquots at a time, may be subject to degradation. Because this type of reference

295 material is consumed at a fairly high rate, they will need to be replaced or replenished on a continual basis. As potential

296 replacement material is identified during routine testing or during outbreaks, it is advisable to work with field counterparts

297 to obtain bulk reference material and store it for future use. Alternatively, it may be necessary to produce this material in

298 experiments like those described in Section A.2.2 of this chapter. Similar to the comparative approach described above

299 with respect to ASe, at least five animals in each group should be considered. For smaller host species, this number may

300 need to be increased in order to collect sufficient reference material.

301 **1.	Repeatability (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.1.1) and ~~preliminary~~**

302 **~~reproducibility~~provisional assay recognition (~~WOAH Validation Standard,~~Chapter 1.1.6,**

303 **Section B.2.6)**

304 Repeatability is the level of agreement between results of replicates of a sample both within and between runs of the same

305 test method in a given laboratory. Repeatability is estimated by evaluating variation in results of replicates from a minimum

306 of three (preferably five) samples representing analyte activity within the operating range of the assay. Consult Chapter

307 2.2.4 *Measurement uncertainty*for statistical approaches for measures of uncertainty for assessments of repeatability.

308 Reproducibility is the ability of a test method to provide consistent results, as determined by estimates of precision, when

309 applied to aliquots of the same samples tested in different laboratories. However, preliminary reproducibility estimates of

310 the candidate assay should be determined during developmental stages. A small panel of three (but preferably five)

311 representing negative, weak and ~~both low and high~~strong positives, like those described above, would be adequate. This

312 type of panel could also be used for a limited evaluation of reproducibility to enhance provisional acceptance status for the

313 assay. The test method is usually assessed in ~~one~~two or more laboratories with a high level of experience and proficiency

314 in assays similar to the candidate assay. The panel of ‘blind’ samples is evaluated using the candidate assay in each of

315 these laboratories, using the same protocol, same reagents and comparable equipment. This is a scaled-down version of

316 Stage 3 of assay validation. ~~Consult Chapter 2.2.4 for further explanation of the topic and its application.~~

#### 317 2.	Reproducibility (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.3)

318 Reproducibility is an important measure of the precision of an assay when used in a cross-section of laboratories located

319 in distinct or different regions or countries using the identical assay (protocol, reagents and controls). As the number of

320 laboratories increases, so do~~es~~ the number of variables encountered with respect to laboratory environments, equipment

321 differences and technical expertise. ~~These~~An overview of the factors affecting testing reproducibility is provided in Waugh

322 & Clark (2021). Reproducibility studies are a measure of an assay’s capacity to remain unaffected by substantial changes

323 or substitutions in test conditions anticipated in multi-laboratory use (e.g. shipping conditions, technology transfer, reagents

324 batches, equipment, testing platforms and/or environments). ~~Each of~~ At least three laboratories should test the same panel

325 of ‘blind’ samples containing a minimum of 20 samples, representing negative and a range of positive samples. If selected

326 negative and/or positive samples in the panel are duplicated, in the panel then it may be possible to assess both assay

327 reproducibility and within-laboratory repeatability~~estimates may be augmented by replicate testing of these samples when~~

328 ~~used in the reproducibility studies~~.

#### 329 3.	Proficiency testing (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.5.1)

330 A validated assay in routine use in multiple laboratories needs to be continually monitored to ensure uniform performance

331 and provide overall confidence in test results. This is assessed through external quality assurance programmes. Proficiency

332 testing is one measure of laboratory competence derived by means of an inter-laboratory comparison; implied is that

333 participating laboratories are using the same (or similar) test methods, reagents and controls. Results are usually

334 expressed qualitatively, i.e. either negative or positive, to determine pass/fail criteria. However, ~~for single dilution assays,~~

335 where semi-quantitative results ~~provide~~are provided, additional ~~data for assessment of~~analysis may assess non-random

336 error among the participating laboratories. Refer to Johnson & Cabuang (2021) for an overview of proficiency testing and

337 ring trials.

338 Proficiency testing programmes are varied depending on the type of assay in use. For single dilution type assays, panel

339 sizes ~~also vary but a minimum of five samples, representing negative and both low and high positives, like those described~~

340 above, would be adequate. Proficiency testing is not unlike a continuous form of reproducibility assessment. However,

341 reproducibility, by definition, is a measure of the assay’s performance in multiple laboratories; whereas proficiency testing

342 is an assessment of laboratory competence in the performance of an established and validated assay. Measurements of

343 precision can be estimated for both the reproducibility and repeatability data if replicates of the same reference sample are

344 included in this ‘blind’ panel. Consult Chapter 2.2.4 for further explanation of the topic and its application. vary but a

345 minimum of five samples, representing negative weak and strong positives, would be adequate.

346	**D. GROUP D**

347 Reference samples in Group D differ from the previous Groups in that each sample in the panel should be from a different

348 individual animal. As indicated in Chapter 2.2.8 *Comparability of assays after changes in a validated test method*,

349 experimental challenge studies often include repeated sampling of individual animals to determine the progression of

350 disease, but this is a different objective ~~than~~to comparing performance characteristics that would be associated with

351 diagnostic sensitivity (DSe) and diagnostic specificity (DSp) of a test method. Serially drawn samples, taken on different

352 days from the same animal, cannot be used as representative of individual animals in populations targeted by the assay,

353 because such samples violate the rule of independence of samples required for such studies.

354 Care must be taken in choosing the reference samples and the standard (independent) method used in this type of

355 comparison to ensure that the analytes being detected (if different) demonstrate the same type of pathogenic profile in

356 terms of time of appearance after exposure to the infectious agent, and relative abundance in the test samples chosen.

#### 357 1.	Standard method comparison and provisional recognition (~~WOAH Validation Standard,~~

358 **Chapter 1.1.6, Sections B.2.~~6~~5 and B.2.6)**

359 There are situations where it is not possible or desirable to fulfil Stage 2 of the Validation Pathway because appropriate

360 samples from the target population are scarce and animals are difficult to access (such as for exotic diseases). However,

361 a small but select panel of highly characterised test samples representing the range of analyte concentration should be

362 run in parallel in the candidate assay method and ~~by~~a WOAH standard method, as published in the WOAH *Manual~~s~~*.

363 Biobanks may be a useful resource in this context, providing well-characterised samples supported with metadata to

364 enhance transparency and provenance of samples used in method comparisons (Watson *et al.*, 2021). If the methods are

365 deemed to be comparable (Chapter 2.2.8), and depending on the intended application of the assay, the choice may be

366 made that further diagnostic validation is not required. For example, if the intended application is for screening of imported

367 animals or animal products for exotic pathogens or confirmation of clinical signs, full validation beyond a test method

368 comparison may not be feasible or warranted.

369 Experience has shown that the greatest obstacle to continuing through Stage 2 of the Validation Pathway is the number of

370 defined samples required to estimate diagnostic performance parameters with a high degree of certainty (~~WOAH Validation~~

371 ~~Standard,~~chapter 1.1.6, Section B.2). In some cases, provisional recognition by international, national or local authorities

372 may be granted for an assay that has not been completely evaluated past analytical stages. The different rationales for

373 provisional acceptance are well explained in ~~the WOAH Validation Standard,~~chapter 1.1.6. In all cases however, sound

374 evidence must exist for comparative estimates of DSp and DSe based on a small select panel of well-characterised

375 samples containing the targeted analyte.

376 Ideally, for both comparison with a standard method or provisional recognition, a panel of, for example, 60 samples could

377 be assembled to ensure sufficient sample size for statistical analysis of the resulting data. This would include 30 ‘true’

378 negatives and 30 ‘true’ positives. Wherever possible, the positives should reflect the range of analyte concentrations or

379 activities expected in the target population. As mentioned above, each sample in this panel must represent an individual

380 animal. Consult Chapter 2.2.5 for statistical approaches to determining methods comparability using diagnostic samples.

#### 381 2.	Biological modifications (~~WOAH Validation Standard,~~Chapter 1.1.6, Section B.5.2.2)

382 There may be situations where changes to some of the biologicals used in the assay may be necessary and/or warranted.

383 This may include changes to reagents themselves or a change to a different type of specimen which contains the same

384 analyte as targeted in the original validated assay (e.g. from serum to saliva). At the very least, all of the analytical criteria

385 of the validation pathway must be re-assessed before proceeding. If the analytical requisites are met, the remaining

386 question relates to whether or not a full diagnostic validation is required. A similar approach to the above using a panel of

387 60 individual reference samples may be considered. However, in this case the original test method would be considered

388 as the standard (independent) test and the modified method would be considered the candidate. Consult Chapter 2.2.5 for

389 statistical approaches to determining methods comparability using diagnostic samples.

390	**E. GROUP E**

391 Reference animals and reference samples in this Group E are well described in ~~the WOAH Validation Standard,~~chapter

392 1.1.6, Section B.2.1). However, there are a few points that are worth re-iterating here.

#### 393 1.	‘Gold standard’ 34 – diagnostic specificity and diagnostic sensitivity (~~WOAH Validation~~

394 **~~Standard,~~Chapter 1.1.6, Section B.2.1)**

395 For conventional estimates of DSp, negative reference samples refer to true negative samples, from animals that have

396 had no possible infection or exposure to the agent. In some situations, where the disease has never been reported in a

397 country or limited to certain regions of a country, identification of true negative reference samples is usually not a problem.

398 However, where the disease is endemic, samples such as these may be difficult to locate. It is often possible to obtain

2 The term “Gold Standard” is limited to a perfect reference standard as described in ~~the WOAH Validation Standard,~~chapter 1.1.6, Section B.2.1.2, and Chapter 2.2.5 *Statistical approaches to validation*, Introduction and Figure 1.

399 these samples from regions within a large country or perhaps different countries where the disease in question does not

400 occur or has ~~either~~been eradicated~~or has never had the disease in question~~.

401 ~~Again~~ For conventional estimates of DSe, positive reference samples refer to true positives. Care must be taken to ensure

402 that the sample population is representative of the population that will be the target of the validated assay. It is generally

403 problematic to find sufficient numbers of true positive reference animals, as determined by isolation of the organism. It may

404 be necessary to resort to samples from animals that have been tested by a combination of methods that unequivocally

405 classify animals as infected/exposed as discussed in ~~the WOAH Validation Standard,~~chapter 1.1.6.

406 ~~The important point here is that~~All samples, irrespective of origin, must be documented as they would for any other

407 reference sample ~~so as~~to unequivocally ~~to~~classify animals as infected or exposed, dependent on the fitness for purpose

408 and proposed use of the test. As mentioned in Section A, and summarised in Figure 2, of this chapter, all reference samples

409 should be well characterised~~. This includes documentation on both the pathogen and donor host. For pathogens, this may~~

410 include details related and data documented to ~~strain, serotype, genotype, lineage, etc. The source of the host material~~

411 ~~should be well described with respect to species, breed, age, sex, reproductive status, vaccination history, herd history,~~

412 ~~etc. Wherever possible, the phase of infection should be noted. This could include details related clinical signs, antibody~~

413 ~~profiles, pathogen load or shedding, etc. In some cases, experimental infection/exposure may be the only viable option~~

414 ensure appropriate sample selection for ~~the production of reference material (see the OIE Validation Standard, Section~~

415 ~~B.2.3). In this case, all of the above and the experimental protocol should be detailed~~intended purpose.

416 Particularly relevant to these reference samples, the tests that are used to determine their so called ‘true’ disease/infection

417 status need to be well documented in order to assess potential errors in estimates that may be carried over into the

418 estimates for the candidate assay. Indeed, when using imperfect standard assays to define reference animal or sample

419 status, the DSe and DSp performance estimates of the candidate assay may be flawed and often overestimated. Consult

420 Chapter 2.2.5 for statistical considerations. Situations where a perfect reference is available for either positive or negative

421 animals, and one where the reference is perfect for both are described for diagnostic test validation by Heuer & Stevenson

422 (2021).

426 Latent-class models are introduced in ~~the WOAH Validation Standard,~~chapter 1.1.6. They do not rely on the assumption

427 of a perfect reference (standard or independent) test but rather estimate the accuracy of the candidate test and the

428 reference standard with the combined test results. Because these statistical models are complex and require critical

429 assumptions, statistical assistance should be sought to help guide the analysis and describe the sampling from the target

430 population(s), the characteristics of other tests included in the analysis, the appropriate choice of model and the estimation

431 methods based on peer-reviewed literature. Consult Chapter 2.2.5 for statistical considerations.

432 Reference populations, not individual reference samples, used in latent-class studies need to be well described~~. This~~

433 includes documentation on both the pathogen and donor host. For pathogens, this may include details related to strain,

434 ~~serotype, genotype, lineage, etc., that may be circulating in the population. The source of the host material should be wel~~l

435 ~~described with respect to species, breed, age, sex, reproductive status, vaccination history, herd history, etc.~~ as

436 summarised in Figure 2. Wherever possible, the phase of infection in the populations should be noted with respect to

437 morbidity or mortality events, recovery, etc.

438 As a special note, if latent class models are to be used to ascribe estimates of DSe and DSp and include multiple

439 laboratories in the design, it is possible to incorporate an assessment of reproducibility into the assessment. ~~As stated~~

440 ~~above, statistical advice should be sought in this respect.~~Bayesian analysis of latent class models are complex and require adherence

441 to critical assumptions. Statistical assistance should be sought to help guide the analysis and describe the sampling from

442 the target population(s), the characteristics of other tests included in the analysis, the appropriate choice of model and the

443 estimation methods (based on peer-reviewed literature). See chapter 2.2.5 for details and Cheung *et al.,*2021.

444 

**RATIONALE:**  Latent class is a model; Bayesian is an analysis approach.

Report of the Meeting of the Biological Standards Commission / September 2023

72

445 **FURTHER READING**

446 BOWDEN T.R., CROWTHER J.R. & WANG J. (2021). Review of critical factors affecting analytical characteristics of serological

447 and molecular assays. *Rev. Sci. Tech. Off. Int. Epiz*., 40, 53–73. doi:10.20506/rst.40.1.3208.

448 CHEUNG A., DUFOUR S., JONES G., KOSTOULAS P., STEVENSON M.A., SINGANALLUR N.B. & FIRESTONE S.M. (2021). Bayesian

449 latent class analysis when the reference test is imperfect. *Rev. Sci. Tech. Off. Int. Epiz*., **40**, 271–286.

450 doi:10.20506/rst.40.1.3224

451 JOHNSON P. & CABUANG L. (2021). Proficiency testing and ring trials. *Rev. Sci. Tech. Off. Int. Epiz*., **40**, 189–203.

452 https://doi.org/10.20506/rst.40.1.3217

453 HEUER C. & STEVENSON M.A. (2021). Diagnostic test validation studies when there is a perfect reference standard. *Rev. Sci.*

454 *Tech. Off. Int. Epiz*., **40**, 261–270. doi:10.20506/rst.40.1.3223

455 REISING M.M., TONG C., HARRIS B., TOOHEY-KURTH K.L., CROSSLEY B., MULROONEY D., TALLMADGE R.L., SCHUMANN K.R., LOCK,

456 A.B. & LOIACONO C.M. (2021). A review of guidelines for evaluating a minor modification to a validated assay. *Rev. Sci.*

457 *Tech. Off. Int. Epiz*., **40**, 217–226. doi:10.20506/rst.40.1.3219

458 WATSON J.W., CLARK G.A. & WILLIAMS D.T. (2021). The value of virtual biobanks for transparency purposes with respect to

459 reagents and samples used during test development and validation. *Rev. Sci. Tech. Off. Int. Epiz*., **40**, 253–259.

460 doi:10.20506/rst.40.1.3222.

461 WAUGH C. & CLARK G. (2021). Factors affecting test reproducibility among laboratories. *Rev. Sci. Tech. Off. Int. Epiz*., **40**,

462 131–143. doi:10.20506/rst.40.1.3213	*

463	* *

464	**NB:**There is a WOAH Collaborating Centre for

465	Diagnostic Test Validation Science in the Asia-Pacific Region (please consult the WOAH Web site:

466	https://www.woah.org/en/what-we-offer/expertise-network/collaborating-centres/#ui-id-3).

467	Please contact the WOAH Collaborating Centre for any further information on validation

**NB:**FIRST ADOPTED IN 2014.

Report of the Meeting of the Biological Standards Commission / September 2023

73