**Annex 6. Item 5.1. – Chapter 2.2.4 Measurement uncertainty**

###### 2	MEETING OF THE WOAH BIOLOGICAL STANDARDS COMMISSION

3	**Paris, 4–8 September 2023**

4

**USA COMMENTS IN RED FONT**

5	

**CHAP T E R  2 . 2 . 4 .**

# 6	M E A S U RE M EN T U N CE RT A I N T Y

## 7	INTRODUCTION

8 *The WOAH Validation Recommendations provide detailed information and examples in support of*

9 *~~WOAH Validation Standard that is published as~~Chapter 1.1.6*~~Principles and methods of~~Validation of

10 diagnostic assays for infectious diseases of terrestrial animals*~~this~~*~~Terrestrial Manual*, or Chapter 1.1.2 of*~~

11 *~~the~~*~~Aquatic Manual~~*. ~~The Term “WOAH Validation Standard” in this chapter should be taken as referring to~~*

12 *~~those chapters.~~*

13 *Estimation of measurement uncertainty*(*MU*)*~~, sometimes termed measurement imprecision,~~ is a*

14 *requirement for testing laboratories based on international quality standards such as ISO/IEC 17025-~~2005,~~*

15 *2017 General requirements for the competence of testing and calibration laboratories*~~(*ISO/IEC 17025*)~~. *The*

16 *measurement process for detection of an analyte in a diagnostic sample is not entirely reproducible and*

17 *hence there is no exact value that can be associated with the measured analyte. Therefore, the result is*

18 *most accurately expressed as an estimate ~~together~~with an associated ~~level of~~imprecision level. This*

19 *imprecision is the measurement uncertainty*(*MU*). *MU is limited to the measurement process of quantitative*

20 *tests. The approach described here is known as “top-down” or “control sample” because it uses a weak*

21 *positive control sample and expresses the MU result at the ~~cut-off~~ diagnostic threshold, where it most matters.*

22 *of whether the measurement is appropriate and fit for whatever use to which it may be applied. It is not an*

23 *alternative to test validation but is rightly considered a component of that process*(*see ~~the WOAH Validation~~*

24 *~~Standard,~~chapter 1.1.6 Section B.1.1*Repeatability)*.*

**RATIONALE:** Replaced cutoff with threshold for consistency with verbiage used later in the document.

## 25	A. THE NECESSITY OF DETERMINING MU

26 To assure compliance with ISO/IEC 17025-~~2005~~2017 requirements, national accreditation bodies for diagnostic testing

27 laboratories require laboratories to calculate MU estimates for accredited test methods that produce quantitative results,

28 e.g. optical densities (OD), percentage of positivity or inhibition (PP, PI), titres, cycle threshold (CT) values, etc. This

29 includes tests where numeric results are calculated and then ~~are~~expressed as a positive or negative result at a cut-off

30 value. For the purpose of estimating MU in serology and reverse transcriptase polymerase chain reaction (RT-PCR),

31 suitable statistical measures are mean target values ± 2 standard deviations (SD), ~~which is approximately equal to a 95%~~

32 ~~confidence interval (CI),~~ relative standard deviation (RSD = SD / mean of replicates) and coefficient of variation (CV = RSD

33 × 100%). Examples provided below assume normal distribution of data.Alternative methods are available that are less sensitive to both that assumption and to the presence of outliers; they are not illustrated here.  The concept of MU does not apply to strictly binary

34 (qualitative) results (positive or negative).

**RATIONALE:** This is not approximately equal to a confidence interval, even for normally distributed data. (A confidence interval is based on the standard error, not standard deviation.) Rather, this is an approximate 95% reference interval. See discussion in Sec. 15.5 of Martin Bland (2000), An Introduction to Medical Statistics, 3d ed. (Oxford University Press). Note also that ISO/IEC Guide 98-3, Guide to the Expression of Uncertainty in Measurement (2008), Sec. 6.2.2, states that they deliberately do not use the term "confidence" in relation to this interval. 

Alternative methods are less reliant on distributional assumptions, and better handles the presence of outliers. See for example Maronna, Martin, Yohai, and Salibian-Barerra (2019), Robust Statistics: Theory and Methods, 2d ed. (Wiley).

Report of the Meeting of the Biological Standards Commission / September 2023

57

#### 35 1.	Samples for use in determining MU

36 Repeatability is the level of agreement between results of replicates of a sample both within and between runs of the same

37 test method in a given laboratory. During assay development, repeatability is estimated by evaluating variation in results

38 of independent replicates from a minimum of three (preferably five) samples representing analyte activity within the

39 operating range of the assay (see~~the WOAH Validation Standard,~~Chapter 1.1.6 *Validation of diagnostic assays for*

40 *infectious diseases of terrestrial animals*, Sections A.2.5 *Robustness*and B.1.1 *Repeatability*, and Chapter 2.2.6 *Selection*

41 *and use of reference samples and panels*, Section ~~3.1~~A.4.2). Typically, the variation in replicate results is expressed as

42 RSD or CV. The significant feature is that repeatability studies can be used to define the expected precision of the assay

43 in the detection of a range of analyte concentrations.

44 The use of internal quality or process controls over a range of expected results has become part of daily quality control

45 and quality assurance operations of accredited facilities (see ~~the WOAH Validation Standard,~~chapter 1.1.6, Sections A.2.6

46 *Calibration of the assay to standard reagents*and B.5.1 *Monitoring the assay*, and Chapter 2.2.6, Section ~~1.4~~C.1). These

47 results provide a continuous monitor relative to different aspects of repeatability, e.g. intra- and inter-assay variation, intra-

48 and inter-operator variation and intra- and inter-batch variation, which, when subjected to statistical analysis, provide an

49 expression of the level of robustness (precision) of a test procedure. The monitoring of assay quality control parameters

50 for repeatability provides evidence that the assay is or is not performing as expected. For control samples to provide valid

51 inferences about assay precision, they should be treated in exactly the same way as test samples in each run of the assay,

52 e.g. including sample preparation such as extraction steps or dilution of serum samples for an antibody enzyme-linked

53 immunosorbent assay (ELISA).

54 The variation of the results for control samples can also be used as an estimate of those combined sources of uncertainty

55 and is called the “top-down” approach. This approach recognises that the components of precision will be manifest in the

56 ultimate measurement. So monitoring the precision of the measurement over time will effectively show the combined effects

57 of the imprecision associated with component steps.

58 The imprecision or uncertainty of the measurement process associated with a test result becomes increasingly more

59 important the closer the test value is to the diagnostic cut-off value. This is because an interpretation is made relative to

60 the assay threshold regarding the status of the test result as positive, negative, or inconclusive (as will be described in the

61 following example). In this context, ~~low~~weak positive samples, like those used in repeatability studies or as the low weak

62 positive control, are most appropriate for estimation of MU. The rationale being that MU, which is a function of assay

63 precision, is most critical at decision-making points (i.e. thresholds or cut-offs), which are usually near the lower limit of

64 detection for the assay. In this chapter, the application of MU with respect to cut-off (threshold) values, whether

65 recommended by test-kit manufacturers or determined in the diagnostic laboratory, is described.

66 MU, using the top-down approach, ideally requires long-term accumulated data from a weak positive control sample after

67 multiple test runs over time, with multiple operators and variable conditions. The examples given below are based on 10

68 data points but higher numbers will increase robustness.

#### 69 2.	Example of MU calculations in ELISA serology

70 For most antibody detection tests, it is important to remember that the majority of tests are measurements of antibody

71 activity relative to a threshold against which a dichotomous interpretation of positive or negative is applied. This is important

72 because it helps to decide where application of MU is appropriate. In serology, uncertainty is frequently most relevant at

73 the threshold between positive and negative determinations. Results falling into this zone are also described as

74 intermediate, inconclusive, suspicious or equivocal (see ~~the WOAH Validation Standard,~~chapter 1.1.6, Section B.2.4

75 *Selection of a cut-off*(*threshold*) *value for classification of test results*).

76 A limited data set from a competitive ELISA for antibody to avian influenza virus is used as an example of a “top-down”

77 approach for serology. A ~~low~~weak positive control sample was used to calculate MU at the cut-off level 32.

##### 78 2.1.	Method of expression of MU

79 As the uncertainty is to be estimated at the threshold, which is not necessarily the reaction level of the ~~low~~weak

80 positive control serum, the relative standard deviation (RSD), or coefficient of variation (CV), if expressed as a

81 percentage, provides a convenient transformation:

_

1  The Australian Government, Department of Agriculture, Fisheries and Forestry, has compiled worked examples for a number ofdiagnostic tests Available online at: https://www.agriculture.gov.au/agriculture-land/animal/health/laboratories/tests/measurement-uncertainty (accessed 22 June 2023)

Report of the Meeting of the Biological Standards Commission / September 2023

58

82 where X represents the set of replicates.  To simplify assessment, ~~the~~a suitably transformed result (such as sample-to-positive ratio, percent inhibition, or background-corrected optical density) is regarded as the assay output result, which is then averaged across

83 the number of replicates (�X). In the case of this example, a competitive ELISA, results are “normalised” (as defined

84 in ~~the WOAH Validation Standard,~~chapter 1.1.6, Section A.2.7 *‘Normalising’ test results to a working standard*) to a

85 working standard by forming a ratio of all optical density (OD) values to the OD result of a non-reactive (negative)

86 control (ODN). This ratio is subtracted from 1 to set the level of antibody activity on a positive correlation scale; the

87 greater the level, the greater the calculated value. This adjusted value is expressed as a per cent and referred to as

88 the percentage inhibition or PI value. So for the low weak positive control serum (ODL), the transformation to obtain

89 the per cent inhibition values for the ~~low~~weak positive control (PIL) is:

##### RATIONALE: X needs to be defined, and the equation has been corrected. Clarified what is meant by transformed above, and the equation below.

90	PIL = 100 × [1– {ODL/ ODN}]

91	The relative standard deviation becomes:

92

##### 93 2.2.	Example

RSD (PIL) = SD (PIL)/ Mean(PIL)

94 A limited data set for the AI competitive ELISA example is shown below. In the experiment, the operator tested the

95 low weak positive control serum ten times in the same run. Ideally in the application of this “top down” method, a

96 larger data set would be used, which would enable accounting for effects on precision resulting from changes in

97 operator and assay components (other than only the control serum).

98 ***Table 1.****Top-down or control sample approach for an influenza antibody C-ELISA*

99 Mean PI = 56.3; Std Dev (SD) = 7.9; Assays (*n*) = 10

##### 100 2.3.	Calculating uncertainty

101 From the limited data set,

102 RSD (PIL) = SD/Mean = 7.9/56.3 = 0.14 (or as coefficient of variation = 14%)

103 Expanded uncertainty (U) is the statistic defining the interval within which the value of the measure and is believed

104 to lie within a specified level of confidence, usually 95%. Expanding the uncertainty is done by multiplying the RSD

105 (PIL) by a factor of 2; this allows the calculation of an approximate 95% ~~confidence~~reference interval around the threshold value

106 (in this case at PI = 50%), assuming normally distributed data.

107 U (95%~~C~~RI) = 2 × RSD = 0.28

108 This estimate can then be applied at the threshold level

109 95% ~~C~~RI = 50 ± (50 × 0.28) = 50 ± 14%

**RATIONALE:** Missing an equals sign. Changed confidence to reference based on previous comment above referencing Bland's textbook. Changed notation from CI to RI for reference interval.  A confidence interval is based on standard error not standard deviation and is not appropriate here.

##### 110 2.4.	Interpretation

111 ~~Any positive result (PI > 50%) that is less than 64% is not positive with 95% confidence. Similarly, a negative result~~

112 ~~(PI < 50%) that is higher or equal to a PI of 36 is not negative at the 95% confidence level.~~A sample with a PI between 36% and 64% is within the MU surrounding the threshold value, and thus its diagnostic status is less certain than those of samples with results further from that threshold.  This zone of lower

113 confidence may correlate with the “grey zone” or “inconclusive/suspect zone” for interpretation that should be

114 established for all tests (Greiner *et al., 1995*).

**RATIONALE:** This interpretation is too precise given the multiple approximations made and the nuances of the interpretation of a reference interval. 

115 **3.	Example of MU calculation in molecular tests**

##### 116 3.1.	Example

117 For real-time PCRs, replicates of positive controls with their respective cycle threshold (CT) values can be used to

118 estimate MU using the top-down approach (Newberry & Colling, 2021). The method of expression follows the same

119 formula as for the ELISA example above. This example uses data from replicate runs of a weak positive control

120 sample (10 runs) of an equine influenza hydrolysis probe assay.

121 ***Table 2.****Top-down or control sample approach for an equine influenza TaqMan A assay*

122 Mean = 33.36; Std Dev (SD) = 0.43; Assay n=10

##### 123 3.2.	Calculating uncertainty

124 From the limited data set,

125 RSD (PIL) = SD/Mean 0.43/33.36 = 0.0128 (or as coefficient of variation = 1.28%)

126 Expanded uncertainty (U) is the statistic defining the interval within which the value of the measure and is believed

127 to lie within a specified level of confidence, usually 95%. Expanding the uncertainty is done by multiplying the RSD

128 (PIL) by a factor of 2; this allows the calculation of an approximate 95% ~~confidence~~ reference interval around the threshold value

129 (in this case at Ct value = 37), assuming normally distributed data.

130 U (95%~~C~~RI) = 2 × RSD = 0.0255

131 This estimate can then be applied at the threshold level

132 95% ~~C~~RI = 37± (37 × 0.0255) = 37 ± 0.94

133 The mean cycle threshold (Ct) value after 10 runs is 33.36 and the standard deviation is 0.43. The relative standard

134 deviation is 0.0128. The expanded uncertainty (95% ~~C~~RI) is 2 × the relative standard deviation = 0.0255. Measurement

135 of uncertainty (MU) is most relevant at the cut-off (Ct = 37) and can be applied by multiplication (37 × 0.0255 = 0.94).

136 Subtraction from the threshold (37-0.94) provides the lower 95% ~~confidence~~reference limit (Ct = 36.06) and addition (37+0.94)

137 the upper 95% ~~confidence~~reference limit (Ct = 37.94).

**RATIONALE:**Regarding the change from confidence to reference, see above comment citing Bland 's textbook.

##### 138 3.3.	Interpretation of the results

139 A sample with a ~~PI~~Ct between 36% and ~~64~~37% is within the MU surrounding the threshold value, and thus its diagnostic status is less certain than those of samples with results further from that threshold. ~~Any positive result (Ct < 37) that is higher than 36 Ct is not positive with 95% confidence. Similarly, any negative~~

140 ~~result (Ct > 37) that is less than 38 Ct is not negative with 95% confidence.~~

**RATIONALE:**This interpretation is too precise given the multiple approximations made and the nuances of the interpretation of a reference interval. 

## 141	B. OTHER APPLICATIONS

142 The top-down approach should be broadly applicable ~~for~~to a range of diagnostic tests including molecular tests. For the

143 calculation of tests using a typical two-fold dilution series for the positive control such as virus neutralisation, complement

144 fixation and haemagglutination inhibition tests geometric mean titre (i.e. mean and SD of log base 2 titre values) of the

145 positive control serum should be calculated. Relative standard deviations based on these log scale values may then be

146 applied at the threshold (log) titre, and finally transformed (by antilog) to represent the uncertainty at the threshold. 

147 

148 **RATIONALE:** To clarify what transformation is expected.

149 

150 However, in all

cases, the approach assumes that the variance about the positive control used to estimate the RSD is proportionally similar

151 at the point of application of the MU, for example at the threshold. If the RSD varies significantly over the measurement

152 scale, the positive control serum used to estimate the MU at the threshold should be selected for an activity level close to

153 that threshold. The Australian Government, Department of Agriculture, Fisheries and ~~Water Resources~~Forestry, has

154 compiled worked examples for a number of diagnostic tests (see footnote 1). ~~(DAFF, 2010),~~~~which are available online at:~~

155 ~~http://www.agriculture.gov.au/animal/health/laboratories/tests/worked-example-measurement~~

156 ~~For quantitative real-time PCRs (qPCR) replicates of positive controls with their respective cycle threshold (CT) values can~~

157 ~~be used to estimate MU using the top-down approach.~~

158 Other approaches and variations have been described, i.e. for serological tests (Dimech *et al.,*2006; Goris *et al.,*2009;

159 Toussaint *et al.,*2007). ~~Additional work and policy~~Central documents ~~are available from the National Pathology~~

160 ~~Accreditation Advisory Group and Life Science. The central document~~to MU ~~is~~are the Guide to the expression of

161 uncertainty in measurement (GUM), ISO/IEC Guide, (1995) and Eurachem/CITAC Guide, 2012 CG 4: Quantifying

162 uncertainty in analytical measurement.

#### 163 Scope and limitations of the top-down approach

164 Methods for quantifying uncertainty (addressing MU) for tests vary. When estimating MU for quantitative, biologically based

165 diagnostic tests, where variations in the substrate or matrix have large and unpredictable effects, a top-down approach is

166 recommended (Dimech *et al.,*2006; Eurachem 2012; Goris *et al.,*2009; ISO/IEC Guide 98-3:2008; Newberry & Colling,

167 2021; Standards Council of Canada, 2021; and footnote 1). The advantage of this method is that quality control data are

168 generated during normal test runs and can be used to estimate the precision of the assay and express it at the cut-off. The

169 application at the cut-off depends on the performance of the test at different analyte concentrations, e.g. variation is likely

170 to increase at higher diluted samples. The top-down approach does not identify individual contributors to measurement

171 uncertainty but rather provides an overall estimate. Measurement uncertainty does not replace test validation; however,

172 the validation process includes assessments of repeatability through quality control samples which facilitate calculation of

173 MU.

### 174 REFERENCES

175 ~~AMERICAN ASSOCIATION FOR LABORATORY ACCREDITATION (A2LA). Policy on estimating measurement uncertainty for life~~

176 ~~science testing labs~~~~https://portal.a2la.org/policies/A2LA_P103b.pdf~~~~(accessed 22 November 2018).~~

177 ~~Australian Government, Department of Agriculture and WATER RESOURCES.. Worked examples of measurement uncertainty~~

178 ~~http://www.agriculture.gov.au/animal/health/laboratories/tests/worked-example-measurement~~~~(accessed 22 November~~

179 ~~2018).~~

180 ~~AUSTRALIAN GOVERNMENT DEPARTMENT OF HEALTH AND AGEING (2007). Requirements for the estimation of measurement~~

181 uncertainty,	National	Pathology	Accreditation	Advisory	Group.

182 ~~http://www.health.gov.au/internet/main/publishing.nsf/Content/B1074B732F32282DCA257BF0001FA218/$File/dhaeou.p~~

183 ~~df~~~~(accessed 22 November 2018).~~

184 DIMECH W., FRANCIS B., KOX J. & ROBERTS G. (2006). Calculating uncertainty of measurement for serology assays by use

185 of precision and bias. *Clin. Chem*., **52**, No. 3, 526–529.

186 ELLISON S.L.R. & WILLIAMS A. (EDS) (2012). Eurachem/CITAC Guide CG 4: Quantifying uncertainty in analytical

187 measurement,	3rd	Edition.	Eurachem,	Teddington,	United	Kingdom,	Section	3.	Available	at:

188 www.eurachem.org/images/stories/Guides/pdf/QUAM2012_P1.pdf (Accessed 10 June 2023).

189 GORIS N., VANDENBUSSCHE F., HERR, C., VILLERS, J., VAN DER STEDE, Y. & DE CLERCQ K. (2009). Validation of two real-time

190 PCR methods for foot-and-mouth disease diagnosis: RNA-extraction, matrix effects, uncertainty of measurement and

191 precision. *J. Virol. Methods,***160**, 157–162.

192 GREINER M., SOHR D. & GOEBEL P.A. (1995). Modified ROC analysis for the selection of cut-off values and the definition of

193 intermediate results of serodiagnostic tests, *J. Immunol. Methods*, **185**, 123–132.

194 ISO/IEC ~~(1995). Guide to the expression of uncertainty in measurement (GUM), ISO/IEC Guide 98:1995. Internationa~~l

195 ~~Organization for Standardization (ISO),~~~~www.iso.org~~.

196 ~~ISO/IEC (2005). ISO/IEC~~17025:~~2005~~2017. General Requirements for the Competence of Testing and Calibration

197 Laboratories. International Organization for Standardization (ISO), www.iso.org.

198 ISO/IEC GUIDE 98-3:2008 (E). Uncertainty of measurement – Part 3 Guide to the expression of uncertainty in measurement

199 (GUM:1995), International Organization for Standardization (ISO), www.iso.org (accessed 10 June 2023).

200 SCC- STANDARDS COUNCIL OF CANADA. Requirements and Guidance for method Verification and Validation in Testing

201 Laboratories (RG-MVVT), 2021-05-24. http://www.scc.ca/ (accessed 10 June 2023) .

202 TOUSSAINT J.F., ASSAM P., CAIJ B., DEKEYSER F., KNAPEN K., IMBERECHTS H., GORIS N., MOLENBERGHS G., MINTIENS K., & DE

203 CLERCQ K. (2007). Uncertainty of measurement for competitive and indirect ELISAs. *Rev. sci. tech. Off. int. Epiz.,***26**, 649–

204 656.

205 NEWBERRY K. & COLLING A (2021). Quality standards and guidelines for test validation for infectious diseases in veterinary

206 laboratories.	*Rev.	Sci.	Tech.	Off.	Int.	Epiz*.,	40,	227–237.	Available	at

207 https://doc.woah.org/dyn/portal/index.xhtml?page=alo&aloId=41245 (accessed 22 June 2023)

205	*

206	* *

207 **NB:**There is a WOAH Collaborating Centre for

208 Diagnostic Test Validation Science in the Asia-Pacific Region (please consult the WOAH Web site:

209 https://www.woah.org/en/what-we-offer/expertise-network/collaborating-centres/#ui-id-3).

210 Please contact the WOAH Collaborating Centre for any further information on validation.

211 **NB:**FIRST ADOPTED IN 2014.